When a file is being compressed with FLAM, one or more original files are read record by record, and a compressed file (optionally encrypted), called FLAMFILE, is created. When a file is decompressed using FLAM, the compressed file is read sequentially and one or more decompressed files are created.
The compression process consists of a series of recurring cycles. Each cycle generally entails reading a fixed number of records in the original file; these records are then buffered in the matrix buffer, compressed and output in the form of a compressed block. The number of records is between 1 and 255 or 4095, depending on the compression mode used, and is specified when the compression procedure is invoked. A compressed block containing correspondingly fewer records is generated when the end of the file is reached or if the matrix buffer overflows.
The compressed blocks created by FLAM are written into a sequential FLAMFILE.
By the record-oriented reading, the file's structural information is preserved, which is of great importance for reproducing the original file in computing environments with data management systems.
Depending on the compression mode, each record in the FLAMFILE is given either a binary checksum or a check character, which is used to verify the integrity of the data when the file is decompressed. The FLAMFILE can be saved or exchanged with another computer. When the file is decompressed, it may have a different organization, a different record format or a different record size from the original file, depending on the settings entered by the user or on the default settings.
Independently from the Frankenstein-Limes compression method FLAM establishes a concept for file conversion that satisfies compatibility requirements as much as possible. The compressed file created by FLAM is a logical image of the records of the original file. This is the basis for any conversion with FLAM.
The compressed file, the FLAMFILE, is stored by default as a sequential file in accordance with the above-mentioned principle (for random access, index sequential storage is also possible).
The problems which occur with uncompressed files when the requirements are comparable must therefore not be simply ignored due to the fact that FLAM is being used. Some of these problems are made easier to solve by the FLAM concept. Others remain despite FLAM and must therefore be solved along application-specific or organizational lines as before, the only difference being that the original file can be substituted by a FLAMFILE.
FLAM does not solve the problem of incompatible record and field structures in heterogeneous environments. Sometimes users are totally unaware about these problems. FLAM offers user exits that allow to embed special conversion routines. As FLAM is designed as an open system, it will be possible to offer standard solutions for special problem areas in the future.
FLAM requires, that the original records are passed to FLAM record by record. The chosen compression method implies that FLAM works asynchronously. N original records may result in k compressed records where n is unequal k. This can be a problem in special cases.
The FLAMFILE is always created with a fixed record length that may be specified by the user. This results usually in compressed records of equal length. This is necessary because some DP systems only support files with equal record length. This is also true for some data transmission products.
The smallest record length is 80 byte. This allows to process the FLAMFILE in the punched card format (RJE file transfer!). The upper limits are defined by the environment: where the file is stored and which products are used to transfer the file. The upper limit defined by FLAM is 32764 byte. However, the user may define the format of the record as "variable length" or "fixed length". For a fixed-length record that is not filled up, padding is applied.
It is also possible to define different block sizes to optimize the I/O operations, file transfer and storage requirements. This flexibility makes it mostly possible to find a suitable solution for all participating software and hardware environments as well as for special applications.
The FLAMFILE is principally a binary file where all 256 bit combinations are allowed. With this code the FLAMFILE can only be transmitted in transparent mode (MODE=CX8 and VR8).
If transmission on a 7-bit line is performed, file transfer products expand such binary files in a way that ASCII compatibility is guaranteed. Some products convert each half byte into one byte, other products expand 3 bytes into 4 bytes.
If the original file contains only printable characters, FLAM can provide a different code format of the compressed file (MODE=CX7). In this case characters from the original file are not combined with FLAM descriptors but simply copied into the compressed file. This mode is mostly more efficient than MODE=CX8 with following expansion 3 to 4.
The FLAM descriptors itself under MODE=CX7 consist only of printable characters that are unambiguous in the international code systems ASCII and EBCDIC. These are all small and big Latin letters, the ten digits and the blank. Any kind of control character, special characters, umlauts are excluded.
The advantage of this method is that the resulting FLAMFILE can be converted freely between ASCII and EBCDIC and vice versa at any time between compression and decompression. If the conversion is not handled by the data transmission system or on the data transmission path, the user may use the FLAM user exits and can apply the appropriate code conversions as part of the compression/decompression process.
FLAM always works in MODE=CX7 in respect to the code system of its host. If source and target are of the same code system (ASCII or EBCDIC) no conversion is necessary. If source and target are of different code systems, FLAM requires that the FLAMFILE is converted to the code system of the target host system first to decompression.
If the FLAMFILE in MODE=CX7 shall be transmitted via a 7-bit line as well as via an 8-bit line, it will be necessary to analyse the actual situation to maintain ull compatibility. It must be considered, that FLAM offers integrated code conversion functions not on all platforms. However, this problem can be solved with MODE=CX7.
Because the FLAMFILE has records of equal length, the last record is filled with a padding character if fixed record format is used. In MODE=CX7 this padding character is blank, otherwise binary zero. If a variable record format is used, no padding is done for the last record - the last record is shortened instead.
Each record in the FLAMFILE has (internal) overhead: the FLAM syntax, which organizes a frame for the compressed data to fulfil the different requirements. The overhead is the same for each record: 4 byte in 7-bit format and 6 byte in 8-bit format. The user should consider this when defining the record length for the FLAMFILE. The shorter the record length, the bigger the overhead. In addition, the FLAMFILE contains the following syntactical elements: an optional file header for each original file, a compulsory block header for each matrix, etc.
Usually the FLAMFILE starts with a file header. This file header consists of a system independent on a system dependent part. The file header contains various information about the original file. During decompression FLAM will use this information - if not provided otherwise - to create the decompressed target file.
It is possible to concatenate multiple compressed files. In this case the FLAMFILE contains multiple file header. The FLAM utility ignores these file headers during decompres-sion, and will use only the first one. However, the others appear in the protocol.
With this feature FLAM is prepared for the insertion of identical file headers into archive files, a feature that may help to identify a file even in the case of hardware faults.
If the FLAM record level interface is used, the different files can be separated during decompression.
An empty file is converted into a FLAMFILE that consists only out of a file header. This implies that an empty file no longer must be treated as a special case. The usual problems with empty files in the job command language and during file transfer belong to the past.
With a parameter it can be specified if and which file header shall be created during compression.
The decompression function can be used to only print out the file header without doing the actual decompression. This allows a quick information about the origin of the compressed file.
One block header is created per matrix. This block header contains all the necessary information for proper decompression even without a file header. However, in this case the user has to specify the target format via parameter, JCL or catalogue entry if another format than sequential and variable record length shall be created.
The block header contains also all information that is needed by the FLAM nucleus for decompression, e.g. MODE, version, matrix size, etc. This information guarantees the upward compatibility of FLAM.
The individual records of the FLAMFILE contain their length redundantly. If the FLAMFILE has variable record length an additional length field of 2- or 4-bytes length is part of each record. In MODE=CX7 on PC standard record separator characters of length 1 or 2 are used. Therefore the record length is not a physically unique size within a heterogeneous environment.
Each FLAMFILE record created in 8-bit code is protected against manipulations via a 16-bit checksum. In addition, so called block pointers allow synchronisation (and restart on block level), if data cannot be decompressed properly because of manipulation or hardware faults.
A FLAMFILE created in 7-bit code does not contain checksums, because this would inhibit code conversion from ASCII to EBCDIC and vice versa. Instead the number of bytes per record is checked. This function detects for example if a non 1:1 code conversion was applied. This may happen if printer control characters or tabulator characters are not converted 1:1. However, this does not comply with the preposition that the file must only contain printable characters.
It has a clear advantage to use the 8-bit format if it is not really necessary to work in 7-bit format.
Compression is faster, the compression ratio is better, the compressed file is better protected in respect to data security and data integrity and transmission of these files is faster in transparent mode. Also more possibilities for data encryption exist.
The reason is, that a 7-bit FLAMFILE can only be encrypted by randomising the character string. Other encryptions do not comply with the purpose for which these files were created (see above).
A FLAMFILE in 8-bit format, however, can be encrypted with all kinds of additional encryption methods to create a FLAMFILE not compatible with the market version.
For such conversions, the original state of the FLAMFILE as created by FLAM must be restored before the FLAMFILE is decompressed. In MODE=CX7 the FLAMFILE in addition must be converted into the code system of the host where the decompression is performed.
In the case that 1:1 code conversions shall be performed before compression or after decompression, FLAM offers the possibility to convert from EBCDIC to ASCII and vice versa as well from one EBCDIC dialect to another EBCDIC dialect. These conversions are implemented via code tables that can be replaced by the user. It is possible to use user defined code tables as well for encryption purposes. For all conversion problems not mentioned so far the user has the possibility to use the user exits for manipulation of uncompressed data. This is independent from the MODE parameter. The required conversions can be performed in connection with record processing.
Independently from the user exits, the record level interface provides access on record level before compression and after decompression. Using this possibility the user may process original data that otherwise could not be handled by FLAM. This interface also allows the integration of FLAM with user applications and software packages.
Also in the case when a FLAMFILE was created without a file header (HEADER=NO) FLAM can decompress this FLAMFILE.
Principally a restoration of a damaged FLAMFILE is possible but requires the consultation of an expert from the manufacturer. Such damages are caused exclusively by hardware or material faults or by unauthorised manipulation.
The ability to store several compressed files in one FLAMFILE has been realized in the form of the group file.
If several files are read during compression, FLAM generates for each input file a file header (parameter HEADER=YES,default) in the FLAMFILE. A number of FLAMFILEs are written physically, in sequential order one after the other. (If the parameter HEADER=NO is set, no details about the respective file are stored in the group file. When decompressed, this file is then no longer recognized as a FLAMFILE containing a number of compressed files and it can only be decompressed altogether.)
The file type and format of a group file can be adapted to suit any requirements, exactly as in FLAMFILEs.
The parameter SHOW=DIR can be used to display the details of all of the compressed files in this group file, without the group file being decompressed.
FLAM is able to decompress each individual file of this group file by specifying a selection rule (see chapter 3.1.4.3). The decompressed file can be specified by means of a command or FLAM creates the file dynamically and catalogues it.
Libraries are compressed into a group file by FLAM on a member-by-member basis, i.e. it would be possible to decompress each member into a separate file using an appropriate conversion rule. Accordingly, a number of individual files can be used to generate library members.
This group file allows libraries that have been generated under a range of different operating systems to be exchanged between heterogeneous operating systems in a compatible way.
If neither a selection rule nor a conversion rule is specified, the compressed files are decompressed into a specified file as in earlier FLAM versions; i.e. all of the originally different files are now positioned one after the other, decompressed. Conversion is executed in accordance with the file attributes of the output.
If FILEINFO=NO was set when the group file was being created,
no file name was stored for the compressed data in question. This also
means that there would be no file name available for creating the files.
Despite this, the compressed data can still be accessed and appropriate conversion rules compiled by means of the internal file names FILE0001 (for the 1st file) through FILE9999 (for the 9999th file).
Compressed files may be ported to a target system via file transfer or via tapes, cartridges, etc. It is not necessary that source system and target system are of the same type. However, necessary requirement is, that a file transfer feature exists or that a compatible tape or cartridge format exists.
If these requirements are fulfilled, data exchange is possible on both systems if FLAM exists and is installed.
All FLAM versions are upward compatible. That means, that systems with both FLAM older and newer version can compress/decompress in the old manner.
Since version 2.x the different formats of the compressed data are compatible on all systems where FLAM exists.
For data exchange between heterogeneous and homogeneous systems only logical data formats should be compressed with FLAM. Physical data formats cannot be reproduced identically on a physically different system.
Different methods for compression exist. With ADC, VR8 and CX8 data is compressed in 8-bit mode, with CX7 in 7-bit mode. For data exchange between mainframes any mode can be used therefore.
It is also necessary to check if the file transfer works transparently for 8-bit binary data. If yes, an 8-bit method (which can be decompressed on the target system) should be chosen for compression.
If file transfer is not transparent the CX7 mode must be selected. The file is only allowed to contain printable characters that can be converted 1:1 during the file transfer. Also in this case it should be checked if the selected compression mode is available on the target system.
For file transfer also transfer mode, record length and record format (variable or fixed) must be considered. It is possible that on the target system it is necessary to insert or delete length fields before decompression. Some file transfer products allow only certain record length and certain record formats.
One parameter that must be provided with the same value on both systems is the buffer size (MAXBUFFER) for the compression of one data block. This parameter has on mainframes the maximum value of 2.5 MB. FLAM uses on mainframes two alternating buffers, so double the memory is needed.
File attributes of the original file are meaningless for data exchange. The reason is that the file transmitted in sequential form is the FLAMFILE.
Within the target system the decompressed data can be stored in a file with any valid file organization. This may be an organization that allows for sequential, index sequential or random access.
Important is, that the data must comply to this organization (e.g., a record key must be sorted in ascending order for index sequential organization).
Files can be compressed immediately during or after processing and can remain stored in compressed form until they are transmitted. Or they can be stored in uncompressed form and are compressed directly before transmission.
During compression and decompression any 1:1 code conversions can be applied to the original data.
For the conversion from EBCDIC to ASCII a standard table is provided. It is also possible to load a user defined table by specifying its name (TRANSLATE).
Generally spoken it is better to apply code conversion during decompression because the compression algorithm treats some frequent characters (like zero or blank) in a special way. Code conversion could reduce the compression effect. Also, because of the smaller character set of the ASCII code, it may happen that characters not contained in the ASCII code set are lost during code conversion, which cannot be reconstructed for decompression.
A delicate problem during the exchange of compressed data are index sequential files. Caused by the conversion the binary or alphanumeric keys may be out of order because of different collating sequences in the code systems. No problem, however, exists for keys that consist out of printable letters or for printable numeric keys.
A special conversion before or after processing with FLAM is necessary for index sequential files that contain binary and alphanumeric keys.
Target files may be created during decompression with a file organization and record format differing from the original file.
This is especially true for compressed files received from another operating system.
If no other specifications are made by the user, all files that were compressed under the same operating system are reconstructed using the information in the file header.
In addition it is possible to convert the compressed file into any file format supported by FLAM on the target system.
However, dependencies between file organization and record format may exist:
If the file is transformed to fixed record length, the original records can be longer or shorter than the new record length.
Longer original records are truncated if parameter TRUNCATE=YES is specified.
Shorter original records are padded up to the new fixed record length with blanks (PADCHAR).
If an index sequential file is transformed into a sequential file, the keys are removed if parameter KEYDISP=DEL is specified.
If a sequential file is transformed into an index sequential file, the original records must contain a field with key properties (unique and sorted in ascending order). Otherwise a printable key or arbitrary length can be inserted at the key position (KEYDISP=NEW).
Records of length 0 or gaps within relative files are removed if the file is converted into index sequential format.
If relative files are converted into sequential files, gaps are converted into records with length 0.
For a conversion into fixed format gaps are removed.
If files are converted into relative format, records of length 0 are represented as gaps, except if records of length 0 can be represented in the relative file format.
LDS files are managed on the disk by VSAM in units of 4096 bytes. If there is an "internal" format it is known only to the user and it is not used by VSAM.
In this regard, FLAM offers support not only for compression, but also for decompression.
Through the use of the parameters IRECSIZE, IBLKSIZE, and IDSORG=LDS for input and ORECSIZE, OBLKSIZE and ODSORG=LDS for output, it is possible to specify "internal" fixed record lengths with appropriate blocking. There is no requirement for the block size to be an exact multiple of the record length, any remainder will be ignored.
These specifications enable every file to be converted into an LDS format; that is, a considerably greater degree of efficiency can be achieved during compression.