flucFS(1)

SYNOPSIS

flucFS [-h] | [-l log-file] [-L log-level] [-f] [-q] [-m] [-D] [-s size] fluc-root user-dir

flucFS2 [-h] | [-l log-file] [-L log-level] [-f] [-e] [-q] [-m] [-D] [-s size] fluc-root user-dir

flucFSview [-h] | [-l log-file] [-L log-level] [-f] [-q] [-m] [-D] [-s size] fluc-root user-dir view-name

flucFS2view [-h] | [-l log-file] [-L log-level] [-f] [-e] [-q] [-m] [-D] [-s size] fluc-root user-dir view-name

Configuration tools:

flucFSconf [-c r|w] [-h|-l|-L|-d index] fluc-root

flucFSconf [-c r|w] -m index -n new-index fluc-root

flucFSconf [-c r|w] -i index [-o flags] -p pattern -t targetPattern -s fileString -f formatString fluc-root

flucFSviewconf [-h|-l|-L|-d index] fluc-root view-name

flucFSviewconf -m index -n new-index fluc-root view-name

flucFSviewconf -i index [-o flags] -p pattern -t targetPattern -s fileString -f formatString fluc-root view-name

DESCRIPTION

flucFS is using FUSE to mount an existing directory in some other folder. Everything written to the mounted folder is transparently converted, encrypted, compressed as defined in the configuration. If no match is found in its configuration, the data is passed through without any change. The configuration is done with the argument strings of the FLUC byte interface API. However, since flucFS is a filesystem and uses only the read and write functions of the API, the element access features of the API are not useable with flucFS.

flucFS2 provides the same functionality as flucFS but uses the older version 2 of FUSE. This should only be used on systems without support of FUSE version 3.

flucFSconf is a command line utility to view and modify the configuration of flucFS. A fast memory mapped database is used to store configuration and meta data. This database is stored in a subdirectory of fluc-root with the name .flucIndex. flucFSconf will check the formatString and fileString arguments for correct syntax before storing them. On error, the exit code is set accordingly and a detailed error message is printed on stderr.

A graphical user interface to view and modify flucFS configuration is part of the Frankenstein Limes Control Center flcc.

flucFSview will read only mount an existing directory in some other folder. This view of the directory can be configured separately and defines what is visible and how.

flucFS2view provides the same functionality as flucFSview but uses the older version 2 of FUSE. This should only be used on systems without support of FUSE version 3.

flucFSviewconf is a command line utility to view and modify the configuration of flucFSview.

The fuse kernel module must be loaded in order to use any fuse filesystem. This kernel module is usually installed with the linux kernel.

COMMAND-LINE OPTIONS

flucFS

-h: print usage information
-l: set name of log-file
-L: set level/amount of log output to one of: error, info, debug or trace. Error is default
-f: run in foreground
-e: allow to use nonempty user-dir (only needed if FUSE_VERSION = 2)
-q: keep quiet
-m: serve requests on several threads. Without it every request is served one after the other. Requests for one file are serialised either way, because a FLUC byte interface handle is used sequentially.
-D: commit the metadata without the second fsync. Faster, at the price described under METADATA DATABASE. Durable without it.
-s: size of the metadata database, in bytes or with a K, M or G suffix. Wins over the environment variable FL_FLUCFS_MAPSIZE. See METADATA DATABASE.

flucFSconf

-h

print usage information

-l

list all config entries

-L

list all config entries in machine readable format

-c

select read or write configuration [default: write]

-d

delete config entry with index

-m

move config entry from index to new-index

-i

index to store config entry

-o

set option flags of the entry as integer. default is 0. currntly flucFS uses two flags:

DISABLE: this entry is never used
USE_IR: the stored inverse read string is only used when this flag ist set.

-p

set pattern for file name matching

-t

set targetPattern to build physical filename

-s

set fileString

-f

set formatString

EXAMPLES

To mount the folder /var/data on the folder /home/hugo/mydata execute this command:

     flucFS -l logfile /var/data /home/hugo/mydata

If this is started from within the folder /home/hugo it can be abbreviated with

    flucFS -l logfile /var/data mydata

If no logfile is specified, flucFS will NOT run as daemon and the logging is done to stderr.

To unmount the filesystem execute

     fusermount -u /home/hugo/mydata

or, if it was not running as a daemon, just hit <Ctrl\>-c

CONFIGURATION

flucFS uses 2 configuration tables, one for read requests and one for write requests. The pattern value is applied to the filename of a request. If the filename is matching the config entry is used for the request. The first matching entry will be used. The order of the entries is controlled by the integer index value. The matching is done in increasing index order. If no matching entry is found the request is passed through without any data modification.

If the useIR flag is set the read configuration is only used for files unknown to flucFS, because they were NOT created by writing to the mounted folder. When reading files previously written to the mounted folder flucFS knows how to read them by using the meta data stored on write.

If the useIR flag is not set only the read configuration is used for all files.

Config entries where the DISABLE flag is set are ignored by flucFS.

Example output of a flucFS configuration:

    $ flucFSconf -l froot
    list entries in froot
    10,enabled,useIR:*.txt:[copy].gz:OUTPUT(SAV.FILE(CNV.GZP(CMPLEV=LEV6) FIO.BLK(NAME=''))):FORMAT.BINARY()
    30,enabled,useIR:tlog/*:[copy].xz:WRITE.BINARY(FILE='' COMPRESS.XZ()):FORMAT.BINARY()

    $ flucFSconf -c r -l froot
    list entries in froot
    10,enabled,noIR:*.gz:[base]:read.auto(file=''):FORMAT.BINARY()
    20,enabled,noIR:*.bz:[path]/[base]:READ.AUTO(FILE=''):FORMAT.BINARY()
    30,enabled,noIR:*.bin:[copy]:INPUT(SAV.FILE(FIO.BLK(NAME=''))):FORMAT.BINARY()

In this example all files with the extension .txt will be comressed with gzip. All files in the folder froot/tlog will be tranparently compressed with xz. If flucFS gets a request to read a file which it does not now about, because it was NOT written to the mounted folder but to the froot folder, it will decompress all files with the extension .gz with gzip automatically. Other files are read without any conversion.

METADATA DATABASE

Below fluc-root flucFS keeps a directory .flucIndex with an LMDB database. It holds, for every file written through the mount, the mapping between the logical name the user sees and the physical file, together with the commands that produced it and the file attributes. Without that entry a file cannot be found again through the mount, even though the physical file is still on disk.

Size

About seven entries per file are stored, measured at some 1600 bytes of database per file. The default of 64 MiB therefore holds roughly 42000 files. A mount that has to hold more says so with -s, or with the environment variable FL_FLUCFS_MAPSIZE; both take bytes or a K, M or G suffix, and -s wins over the variable:

    flucFS -s 512M /var/data /home/hugo/mydata
    FL_FLUCFS_MAPSIZE=512M flucFS /var/data /home/hugo/mydata

The value is a reservation of address space, not of disk: the database file grows with the data. An existing .flucIndex adopts a larger value on the next mount. All programs that open the same fluc-root — flucFS, flucFSconf and metadb — have to be given the same value.

When the database is full, the write fails with ENOSPC ("No space left on device") although the file system below has room. That is the signal to raise the size. Note that the database grows with every file and does not shrink when files are deleted.

Durability

By default every metadata commit is durable: the data and then the meta page are each written and flushed. That costs two disk revolutions per file and is what makes a run of many small files slow.

-D drops the second flush. On the development machine a run of 658 files on ext4 went from 13.3 to 9.3 seconds. What it risks is bounded and worth knowing exactly: after a power loss or a kernel crash — not after a crash of flucFS itself, which is not enough — LMDB falls back to the previous meta page, and the metadata of the last file written is lost. The physical file is on disk, but the mount no longer knows its logical name. The database itself stays intact.

Use -D where the mount is a conversion step that can be repeated. Leave it off where the mount is the place the data lives.

NOTICE

Due to its implementation design the logical view provided in the mounted folder might show more than one file with the same name if used in irrational ways. For example writing the file hugo.txt in the logical folder and another file with the same name to the root folder.

EXIT STATUS

flucFSconf

On success the exit status is 0, otherwise if it is:

smaller than 0: a usage error of flucFSconf like missing to specify an required argument or an invalid index value

bigger than 0: a syntax error in the given fileString or formatString

BUGS

If a mount command fails with this message:

    fuse: failed to open /dev/fuse: Permission denied

Solution:

The permissions of the device node /dev/fuse must allow read and write for any user in order to mount a FUSE filesystem by a user other than root. This can be done by executing:

    chmod ugo+rw /dev/fuse