Performance (CPU/memory utilization and throughput) is an important topic when using FLAM. The following points describe some major factors that contribute to the overall performance of FLAM. They should give you a good starting point if you experience performance issues.
In order to minimize consumed CPU cycles during mass data processing, FLAM pre-calculates as much as possible before processing any data, which requires memory on the one hand and spends some CPU time on the other. If you process many small files with FLAM, this can have a negative effect on performance. FLAM is not designed and developed to process small amounts of data.
FLAM uses special instructions of the respective hardware platforms to be able to accelerate some algorithms. On IBM's mainframes, for example, this concerns the use of CPACF and zEDC.
For IBM mainframe systems, it is possible to outsource memory-to-memory functions (compression, conversion, formatting) as SRBs to so-called zIIP processors. Although this does not result in a performance advantage, it does save chargeable CPU time (MSU).
FLAM processes data in self-sufficient segments, which represent a block or a certain number of records. The amount of data that is processed in one go determines the throughput performance.
It is important to understand that there is an overhead per segment, so the segments should not be too small. However, if too much data is put into a segment, then the processing constantly takes place outside the caches, which also reduces the performance. Here, the block size or the record count must be optimally selected for the respective platform.
For historical reasons, FLAM works with a default block size of 64 KiB and a record count of 512. This is a good average value across all platforms, but not always optimal.
It should also be noted that such a segment is expanded after decompression, i.e. in this case one should always start with a smaller block size.
On many systems, 256 KiB is now a good block size. The optimal record count is the optimal block size divided by the average record length.
Character set conversion can take up a lot of CPU time. By default, FLAM checks each character for validity and stops processing with an error if invalid or unconvertible characters are found (MODE=STOP). This form of error handling is relatively expensive in terms of CPU utilization, but ensures detection of incorrectly encoded text data. Enabling error handling that leads to the replacement of these incorrect characters, processing can be accelerated. The character set conversion is faster if all characters of the input alphabet result in data of the same length in the output alphabet.
With the environment variable FL_DEFAULT_CHAR_MODE (or the system
variable &FLCHARM on z/OS), the default mode for error handling during
character set conversion can be changed from STOP to SUBSTITUTE, for
example, in order to achieve higher performance here.
On z/OS, FLAM uses the IBM Unicode Services as long as it supports the
required character set conversion and is known to be faster than FLAM's
implementation. This only works for simple conversions without mappings,
subsets, reporting or transliterations. The environment variable
FL_IBM_UNICODE_SERVICE (or the system variable &FLIBMUS) can be used
to enable/disable the use of IBM Unicode Services. By setting the
variable to ON instead of the default AUTO, the IBM Unicode Services
will also be used if STOP is selected as the character conversion mode,
which saves a lot of CPU cycles, but at the cost of no longer noticing
some errors in the character data.
FLAM writes many log messages by default, which also takes up CPU time
and I/O. The QUIET or SILENT keywords and the MESSAGE object can
be used to control the amount of output and thus limit or increase the
amount of work involved. Often, during the transition from test to
production, people forget to reduce FLAM's chattiness, which cost both
disk space and CPU cycles in production.
FLAM uses a global literal cache to save memory and CPU when comparing
strings. Here a hashtable is used. Its size can be defined via the
environment variable FL_LITERAL_CACHE_SIZE (or system variable
&FLITCS). In the debug log, the appropriate statistics are output,
which may help you to make adjustments if necessary.
When creating ZIP archives, offsets and statistics values must eventually be written into the member headers, which requires repositioning (fseek()) within the file.
This is possible for host datasets on z/OS via so-called RBA access. In a PS-VB file, this requires a tree in the background where the OS remembers which relative byte address (RBA) is in which block, in which records at which offset in the record. This is extremely time-consuming and if there are many members in the ZIP archive, this can increase the runtime considerably. The maintenance of this tree also costs a lot of CPU, which is not charged to the application.
To avoid this, you should use file types for ZIP archives that are suitable for RBA access. Under z/OS, these are primarily never files in the USS. Alternatively, FLAM also supports streamed ZIP archives, as they were developed for piping under Unix systems.
With the keyword STREAM, you can force a purely sequential write, which
then also works with a PS-VB, but you should check whether ZIP tool used
for extraction understands this streamed format.
With PS-FB datasets, the RBA access is faster because you do not need a tree, but there is padding at the end of the ZIP file. Since ZIP archives have to be read from behind, many other tools cannot cope with this padding because they expect the ZIP directory and not 0 bytes or blanks. With FLAM this is not a problem, but as already mentioned, many other tools then recognize the ZIP file as incomplete or not even as a ZIP file.
For memory allocation, don't use the realloc control feature of CEE.
_CEE_REALLOC_CONTROL=bound,percentage
FLAM is only tested with default settings. With some memory allocation strategies out of memory errors can occur.
For good performance and enough memory we recommend the IBM default runtime options except for run CEEOPTS below:
//CEEOPTS DD * ALL31(ON) STACK(1M,1M,ANYWHERE,KEEP,512K,128K) HEAP(1M,1M,ANYWHERE,KEEP,8K,4K) /*
With ALL31(OFF) and HEAP(,,BELOW,,) the available memory is too small
for the stack in some cases, especially when using the FLUC subsystem,
table or large-record support. Since version 5.1.25, FLAM uses
application-specific run-time option defaults with the CEEUOPT assembly
language source program to set these application-specific option
defaults. The CEEUOPT3 member used for 31 bit and the CEEUOPT6 member
used for 64 bits are located in hlq.FLAM.SRCLIB. We recommend building
and linking the examples if you are going to write applications that use
our APIs (e.g. FLCREC). For batch processing we recommend to create a
storage report (RPTSTG(ON)) and define the CEEOPTS like recommended in
the report.