HELP: Archive management functions
TYPE: OVERLAY
SYNTAX: > flcl ARCHIVE.{FILTER()/COMP()/DECO()/COPY()/LIST()/VERIFY()/REKEY()/UPDATE()/DELETE()/DELVSN()/DELKEY()/DELARC()/PRUNE()/CHECK()}
The ARCHIVE main command consists of subcommands to manage archives. The main features of FLAM5 archives are listed below:
- Versioning with every change (append, update, delete, insert): Each change of an archive results in a new version (replacement for GDG), which prevents data loss. Additionally, write errors of any kind, including power failures, lead to an implicit rollback to the previous state.
- Flexible indexing with any number of keys: Our solution replaces traditional VSAM/ISAM systems and enables efficient organization with full access (insert, update, delete, select) to individual records and columns as well as searches in the end-to-end compressed, encrypted and signed data (including ODATA access).
- Cryptographic security at the highest level: By using our own session keysets and proven standards such as OpenPGP, we offer comprehensive protection against data manipulation and unauthorized access (incl. PKCS#11 and IBM CCA/ICSF support for hardware security modules).
- Separation of access rights to the actual mass data from the administrative data, whereby a further distinction is made between access to the archive's directory and access to the member index (search). This allows the data owner to determine who can view the archive's table of contents, who can search in the compressed and encrypted data and compile encrypted and compressed result sets, and who can access the actual net data.
- Simple re-keying and anonymization options: Our solution makes it easy to assign rights and anonymize sensitive data (specific columns) in order to comply with data protection guidelines. This enables secure and controlled third-party utilization of data in the cloud.
- Seamless integration with various cloud providers: Our system enables connection to any key-value and object stores, giving you maximum flexibility when storing your data. The cryptographic encryption allows you to use the most favorable provider in the world, as you have 100% control over the use of the data.
- Support for various standardized cloud APIs: This enables transparent integration into native cloud applications, massively reducing the amount of data and access and thus the costs, while at the same time increasing security (encryption, integrity, authenticity) and control over the data.
- Simple change of storage cloud: The 'flambeed' data can be easily copied or moved from one storage cloud (e.g. Azure) to another (e.g. AWS/S3) or stored locally on your own disks.
- The FLAM storage is designed as a long-term archive: Since we store the data in neutralized and format-independent form, we can restore the data for any platform and in any format for decades.
- The new FLAM5 archives are a separate file system, which can be made available transparently as an I/O subsystem on the host or File System in User Space (FUSE) on Unix systems.
- Grouping of data via EOT handling according to formats when writing to the archives, when copying or duplicating the archives or when reading from the archives. In each of these phases, this allows the data to be broken down into individual archive members or files according to their format (e.g. PAIN, PAX or CAMT in SEPA).
- Grouping of search results via EOT handling according to the assigned filter name, so that the results for each query are written to a separate file.
To get syntax information, please use:
flcl SYNTAX ARCHIVE
To get help for a parameter, please use:
flcl HELP ARCHIVE.subcmd.parameter[.parameter[...]]
To read the manual page for a parameter, please use:
flcl MANPAGE ARCHIVE.subcmd.parameter[.parameter[...]]
or
flcl HELP ARCHIVE.subcmd.parameter[.parameter[...]] MAN
To generate the user manual for the command, please use:
flcl GENDOCU ARCHIVE=filename
Parameters can be defined via command line (directly or by parameter file) or via properties taken from the corresponding property file.
Below you can find a sample to create an archive, to append few members, to list the content, to verify the archive, to delete a member from the archive, to delete older versions from the archive, to prune the archive and to delete the complete archive.
; write all 'm*.txt' to a new archive with maximal compression
ARCHIVE.COMP(READ.TEXT(FILE='m*.txt' SUPTWS CHRMODE=SUBSTITUTE)
TO.NEW(STORE.FILE(NAME='my_txt_files.fl5' OVERWRITE)
FLAM(CMPSUITE=ZSTD DEFAULTS(CMPLEVEL=COMPACT))))
; add all remaining txt files with overwrite of the existing files
ARCHIVE.COMP(READ.TEXT(FILE='*.txt' SUPTWS CHRMODE=SUBSTITUTE)
TO.MODIFY(STORE.FILE(NAME='my_txt_files.fl5')
FLAM(MBRMODE=OVERWRITE)))
; list content of the archive
ARCHIVE.LIST(STORE.FILE(NAME='my_txt_files.fl5'))
; verify all checksums of the archive
ARCHIVE.VERIFY(STORE.FILE(NAME='my_txt_files.fl5'))
; copy all member '*x*.txt' to another archive without decompression
ARCHIVE.COPY(FROM(STORE.FILE(NAME='my_txt_files.fl5')
FLAM(MEMBER='*x*.txt'))
TO.DUPLICATE(STORE.FILE(NAME='my_x_txt.files.fl5' OVERWRITE)))
; decode all member from the copied archive containing all '*x*.txt' member
ARCHIVE.DECO(FROM(STORE.FILE(NAME='my_x_txt.files.fl5')
FLAM(MEMBER='*')) WRITE.TEXT())
; delete all '*v*.txt' files from the first archive (3. version of this archive)
ARCHIVE.DELETE(STORE.FILE(NAME='my_txt_files.fl5') FLAM(MEMBER='*v*.txt'))
; delete all old version from the archive
ARCHIVE.DELVSN(STORE.FILE(NAME='my_txt_files.fl5' DELETE=ALL-OLD-VERSIONS))
; prune the archive
ARCHIVE.PRUNE(STORE.FILE(NAME='my_txt_files.fl5'))
; delete the whole archive
ARCHIVE.DELARC(STORE.FILE(NAME='my_txt_files.fl5'))
Once the data format has been defined and columns have been indexed, you can define filters and search in the data or update, insert or delete records. The following is an example where the first 3 letters of each text record above are simply indexed and a subset is compiled, which is then transferred and the records searched for are decoded at the end.
; write all '*.txt' file to a new archive with indexing for me and a carrier for search
ARCHIVE.COMP(READ.TEXT(FILE='*.txt' SUPTWS CHRMODE=SUBSTITUTE)
TO.NEW(STORE.FILE(NAME='my_txt_files.fl5' OVERWRITE)
FLAM(ENCRYPT.PGP(DATA(ID='<cuser>') MEMBER(ID='carrier'))
ROW(NAME='TXTRST' COLUMN(NAME='RECORD' INDEX(METHOD=BF2 DATLEN=3 SIGLEN=SL8))))))
; create a filter searching for all strings starting with 'abc'
ARCHIVE.FILTER(DATASET(FORMAT=CSV
ROW(NAME='TXTRST' COLUMN(NAME='RECORD'))
RECORD='abc*') OUTPUT='my_filter.txt')
; the carrier copy all matching compressed segments to a result archive
ARCHIVE.COPY(FROM(STORE.FILE(NAME='my_txt_files.fl5')
FLAM(MEMBER='*' MATCH.FILTER='my_filter.txt' SUBSET))
TO.DUPLICATE(STORE.FILE(NAME='result_subset.fl5' OVERWRITE)))
; the user decode all matching records from the copied subset archive
ARCHIVE.DECO(FROM(STORE.FILE(NAME='result_subset.fl5')
FLAM(MEMBER='*' SUBSET MATCH.DATASET(FORMAT=CSV
ROW(NAME='TXTRST' COLUMN(NAME='RECORD'))
RECORD='abc*'))) WRITE.TEXT(FILE='my_abc_records.txt'))
; or direct from the original archive with the same result
ARCHIVE.DECO(FROM(STORE.FILE(NAME='my.txt.files.fl5')
FLAM(MEMBER='*' SUBSET MATCH.DATASET(FORMAT=CSV
ROW(NAME='TXTRST' COLUMN(NAME='RECORD'))
RECORD='abc*'))) WRITE.TEXT(FILE='my_abc_records.txt'))
In the example above, it makes sense to use the filter if the archive is outsourced to a third party and you only want to transfer the few still compressed and encrypted segments from there without the third party needing access to the data and you only have the relevant records determined on site. For this reason, a cryptographic distinction was made here between access to the data and access to the member index and the directory.
For more complex formats, even more complex options are possible, not all of which can be illustrated here. The two examples are only intended to provide an introduction to the use of archives.