- Generated on Fri Mar 8 2024 21:11:27 for FLCREC-API by
1.10.0
|
FLCREC-API
FLUC Record Interface
|
This interface provides record- or element-oriented sequential read and write access to original files. Original files can be binary, text or XML data sets as well as GZIP-, BZIP2-, XZ-, PGP-, ZIP, FLAM4 files or any other kind of original local or remote data set format supported by FLAM. It includes all conversion and formatting capabilities of the Frankenstein Limes Universal Converter (FLUC). These conversion capabilities are also available as an in-memory variant (FCRCONV/RUNV/FINV) without file I/O. The COBOL sample 'SOFCRIPS' shows how it can be used to convert an XML transaction to a copy book (second FCRCONP call) for SEPA instant payments.
The interface provides functions similar to the record-oriented file I/O functions used with COBOL or PL/1 on mainframes. Only the function for opening files differs in that it takes a file, a format and a state string. You can specify them using the syntax of our command line (FLCL). The FLAM return code (rtc) can be mapped to a message with FCRMSG() and the complete FLAM error trace can be fetched with FCRTRC(). Additionally, this interface provides functions for getting help and syntax information for the file and format string parameters.
There is also a logging facility which causes log messages to be collected in memory. These log messages can be queried when desired.
Additionally, a set of conversion functions is available, which can be used when accessing elements (format.element()), for example to parse XML documents in COBOL or PL/I. An example can be found in a section below.
There are also function for hash, MAC and HMAC calculations, base encoding and decoding and a few other help function for example to determine the amount of processed bytes or units (records or blocks).
The interface does support Micro Focus Enterprise Server for SystemZ (MF-EDZ). It is usable on Linux and Windows, 32 and 64 bit systems with Cobol and PL1.
There are some special functions implemented, mainly to provide a possibility for a workaround concerning the native pointer handling bug if the AMODE directive is required. In this case (e.g AMODE=31), the native pointer provided by the locate functions (FCRLOC(), FCRCONP(), ...) is not usable with the ADDRESS OF statement (invalid pointer value). The pointer is mapped with the corresponding MF-EDZ routine, but the pointer is still in native format. For this case, the function FCRCPY() is provided to implement a transparent workaround.
In the opposite direction, i.e. when writing, FCRPTR() converts a local AMODE=31 pointer to a native pointer to allow, for example, structures with length and pointer for variable length formats in table support.
Often to AMODE=31 is only required to determine job or system names from control blocks. In this case FCRSYM() can be used for it.
For use with Micro Focus Enterprise Server the following environment variables might be used.
DD:name or //'name' if the Micro Focus support is needed. Otherwise normal FLAM4 use is done.DD:name or //'name' cannot be accessed an error is returned. flcl info get.enc prints a list of all supported encodings.:EBCDIC or :ASCII might be used. If the encoding strings begins with 'IBM' big endian is assumed and binary values will be byte swapped. DD: filename filename into the environment (for FCRSYM()). filename into the environment (for FCRSYM()).Each function is provided as a separate load module. All parameters are call-by-reference and function do not have a return value, except few help functions like hash calculation, etc. All parameters must be specified. There is no support to determine a variable amount of parameter. There are mainly 4 types of parameters:
POINTER: pointer to an address (usually 32 (PIC S9(9) COMP or BIN FIXED(31,0)) or 64 bit) INTEGER: pointer to a 32 bit number in two's complement (PIC S9(9) COMP or BIN FIXED(31,0)) STRING[x]: pointer to a byte (8 bit) array of length x (PIC X(x) or CHAR(x)) STRING: pointer to a variable length byte (8 bit) array (PIC X(n) CHAR(n))
The type INTEGER has local endianness. On mainframes, this is usually big endian. This means that the most significant byte is stored first (i.e. at the lowest memory address) and the least significant byte is stored last. On x86 platforms, the byte order is little endian. This means that the byte order is reversed. Only the function FCRCNT() returns a 64 bit integer value for the amount of processed data.
Input strings may be null-terminated in which case the corresponding length pointer may be NULL. If a length value is provided, the string may not be null-terminated. All output strings are null-terminated. The output length is set only if the pointer is not NULL. This is only valid for printable string output. Binary byte arrays are not null-terminated. The return code parameter can be a NULL pointer in which case no return value is set.
When reading: Through the file definition string, you specify how the read operation interprets and transforms your original data to produce an internal neutral FLAM5 element list from it. The format string describes how this element list will then be formatted into a sequence of records. You also have the option of getting the raw elements by specifying element formatting in the format string (format.element()). An optional state string is returned when opening a file for reading which contains some metadata about the contents.
When writing: The format string defines how the provided records or elements will be formatted into a neutral FLAM5 element list. The file definition string describes how this element list will be written to a file. The state string can be used to set additional metadata that provides FLUC with additional information about the data. The converted data can be written to multiple targets in parallel by specifying multiple I/Os through the OUTPUT object. It is possible to specify multiple WRITE overlays or OUTPUT objects to write the same data in different formats to different files, reading the input only once.
To write more than one member to a ZIP archive or FLAMFILE the function FCRNEWM() can be used. This function does a reopen of the open file to add a new member. This is a lot faster than to close the file and open it again with the append switch for the next member.
This will work for all kind of files (not only archives) and could result in a concatenation or a bouquet of further files depending on the input-output name mapping specified in the write string.
The FCROPN() describes the first member and for all other members of this file you must call FCRNEWM(). This functions allows you to provide a new format and state string. If the new member is formated like the old member, the format string can be NULL. In this case the current formatting will be reused. Additionally you have the possibility to receive the interim statistics buffer.
Here is the typical program sequence in pseudo code:
FCROPN(&hdl,"write.text(... archive.zip(member='[name]'))","format.text()","state(orgf='first.member.txt')"); if (hdl==NULL) exit(1); FCRPUT(&hdl,...); ... FCRPUT(&hdl,...); FCRNEWM(&hdl,"format.record()","state(orgf='second.member.txt')"); if (hdl==NULL) exit(2); FCRPUT(&hdl,...); ... FCRPUT(&hdl,...); FCRNEWM(&hdl,NULL,"state(orgf='third.member.txt')"); if (hdl==NULL) exit(3); ... ... ... FCRCLS(&hdl);
The member definition for the archive should contain an input output name mapping, because the state string for each new member provides the original file name (ORGF), which is mapped into the member name of the archive.
When reading or writing, you can define how to format the data (binary, character, text, xml, ...), use different conversions (Base64, OpenPGP, GZIP, CHRSET, ...) executed sequentially and different kinds of I/O methods (block, record, text, FLAM4, ...). This means that you can read and write compressed, encrypted and encoded files, where you can change the character set and other things as part of the write and read operation.
Through the format string, you specify how the data is transformed into a sequential list of records or elements. For example, if the data contains text, you can specify which character set the output should be in and so on. Please use the interactive help function FCRHLP() for more information.
Beside record (default) and element formatting, you can also choose binary, character, text and XML formatting. An FCRGET() call returns the formatted content of one element. For example, a text record with delimiter or a binary block. Please be aware, that such a block could potentially contain up to 1 GB of data.
For read operations, you can use the auto detection capabilities of FLUC.
FCROPN("read.file='filename'","format.record()")
This results in any kind of file (encoded, encrypted and/or compressed) being converted to records. If the content is XML, then the XML data is pretty printed into records. If it is text, then the data is parsed based on the containing delimiters. If it is binary and record lengths are detected, then the records are provided one by one. If it is binary and no record lengths are known or detected, then the data is wrapped into records.
Record formatting is the default format. If the format string does not start with format, the format string is expected to be the inner portion of format.record(). So, the format string reclen=80 is equivalent to format.record(reclen=80). Record formatting will result in the expected results for a record-oriented I/O interface.
FLAM5 elements are parsed data elements with a type, a length, a value and more. If the element data contains printable characters, then these characters are encoded in UTF-8 by default, but can also be converted as needed.
With element formatting, you can read and write a serialized form of FLAM5 elements. For example, it can be used to tokenize an XML document and read these tokens (elements) for further processing. The serialized element format (version 0) is described by the structure FlmElmRec0. It basically consists of a set of 32 bit integers followed by element data and metadata.
The formatting modes binary, character, text or XML can also be used. The converted element data field is returned as a record, which could be a line of a text with a delimiter or the whole chunk of binary data as a block. These formatting modes are designed to be used via the byte interface, but available on the record interface as well. Be careful when using these modes as results could be not as expected.
When using element formatting, you can use several individual element data conversion modules for per-element data conversion. If no converter is used, the element data is simply copied into application memory to build the element structure (FlmElmRec0). A set of functions with 'v' at the end of the function name can be used to set a custom element data converter. A converter must be opened before using it, which can done by calling FCROPNV with a corresponding conversion string. The conversion string describes how the data is converted from the neutral format of a FLAM5 element to the representation in the application memory (when reading) or how the application memory must be interpreted to form the corresponding neutral FLAM5 element data type (when writing). The functions FCRGETV, FCRLOCV and FCRPUTV accept an additional conversion handle as parameter which is obtained from FCROPNV() to replace the default/standard conversion. The FCRGETV, FCRLOCV and FCRPUTV functions require to know the data format that will be read in advance, which is not always the case. Instead, you can also use the regular FCRGET, FCRLOC and FCRPUT functions and use the function FCRCONV to convert the data, if needed. In fact, this function works on arbitrary data and may also be used independently.
There is no limit on the amount of element converters that can be opened. The output length of most converts can be controlled in two ways: By passing appropriate length values in the conversion string or or by passing a buffer of appropriate length at call time.
The converter handle must be closed with FCRCLSV to release all associated resources.
Some usage scenarios for element converters:
With version 5.1.16 of FLAM the table support was introduced. For the record interface you can activate an end of table support if you read from a file (file string). If you activate ENDOFT in the format string at write and use FCRSTN() in conjunction with an output file name containing [table], then you can split the data in different files. But at read you must now handle the FLMRTC_EOT error like an FLMRTC_EOF. If you don't get data the reason could be EOF or EOT. With EOT the next read gives the records conform to the new table. At EOF you get still no data and still EOF. If you activate ENDOFT at read you can use the new function FCRGTN() to get the name of the current table after FCROPN() or after an EOT was signalled. This can be used to interpret the data correctly if more than one table is in a file. If you want to write more than one table to a file, you can work with the table format detection, but normally it is better to define the table format with the new function FCRSTN(), before you provide the data. If you set the table name the data must match this table format (row specification). The table format detection is disabled if FCRSTN() is used. You can reactivate the automatic table detection at write by calling FCRGTN().
For all default character conversions, it is useful to set the environment variable LANG. Other used environment variables of FLAM can be found in the FLCL manual. With version 5.1.19 a new function was introduced (FCRENV()) to load the FLAM environment . This function can be used before the first API call to establish the same environment used by FLAM utilities, subsystems and so on. This give the application developer the possibility to adjust the environment before the first real call is done. Until version 5.1.18, each opening function has read the system variables on z/OS. This is now part of the FCRENV() function to give complete control about the environment to users of the API. To fetch a symbol from the environment, the function FCRSYM() can be used.
On system using EBCDIC a special support for critical punctuation characters was implemented (see FLCL manual). This support converts the several punctuation character from a certain EBCDIC code page to the local character set defined over the LANG variable (if LANG not defined the default is 1047). Below you can find the list of character with different code points in the different supported EBCDIC code pages, which are part of the first 128 Unicode code points.
CRITICAL PUNCTUATION CHARACTERS:
! $ # @ [ \ ] ^ ` { | } ~
SUPPORTED EBCDIC CODE PAGES FOR COMMAND ENTRY:
"IBM-1140","IBM-1141","IBM-1142","IBM-1143",
"IBM-1144","IBM-1145","IBM-1146","IBM-1147",
"IBM-1148","IBM-1149","IBM-1153","IBM-1154",
"IBM-1156","IBM-1122","IBM-1047","IBM-924",
"IBM-500","IBM-273","IBM-037","IBM-875","IBM-424",
"IBM-277","IBM-278","IBM-280","IBM-284","IBM-285",
"IBM-297","IBM-871","IBM-870","IBM-1025","IBM-1112",
"IBM-1157"
This conversion is required to interpret the command syntax correctly. These CLP strings are the major part used by this interface. To work with this API the user must build such CLP string. For this often literals are used. On EBCDIC systems you can define in which CCSID (code page) the literals are provided by the compiler. For example, in COBOL the default, if the CODEPAGE parameter not defined, is 1140.
Your application could get variables from outside (e.g. file names) in the local character set (e.g. 1141). Your literals are in 1140 and you must build a CLP string with literals and variables. To support this kind of inconsistent code pages (since version 5.1.19 of FLAM) escape sequences (&xxx;) the CCSID areas (&nnnn;...&nnnn;) are supported (see FLCL manual). Below you can find an examples for a CLP command string how to use it.
WORKING-STORAGE SECTION.
01 C2-DATA PIC X(200)
VALUE 'CONV.MEMORY(READ.CHAR(TABLE='&TLD;.PAIN008.XMLTAB')
- 'WRITE.RECORD(TABLE='&TLD;.PAIN008.FIXTAB'))'.
or
WORKING-STORAGE SECTION.
01 C2-DATA PIC X(200)
VALUE '&1140;CONV.MEMORY(READ.CHAR(TABLE='~.PAIN008.XMLTAB')
- 'WRITE.RECORD(TABLE='~.PAIN008.FIXTAB'))'.
An unsupported CCSID (e.g. 0) can be used to define the local character set as code page (default case). To get an area for the literal code page you must add a CCSID escape sequence (&1140;). If you have a variable part, you must switch to the local character set (&0000;) or the correct CCSID for this variable and if the literal continued switch back to the literal CCSID (&1140;). If you need such moving characters in your literals you can use the corresponding escape sequence for it. Below you can find all escape sequences for the critical punctuation characters.
! = &EXC; - Exclamation mark
$ = &DLR; - Dollar sign
# = &HSH; - Hashtag (number sign)
@ = &ATS; - At sign
[ = &SBO; - Square bracket open
\ = &BSL; - Backslash
] = &SBC; - Square bracket close
^ = &CRT; - Caret (circumflex)
` = &GRV; - Grave accent
{ = &CBO; - Curly bracket open
| = &VBR; - Vertical bar
} = &CBC; - Curly bracket close
~ = &TLD; - Tilde
This two features provide the possibility to build CLP strings dependency on the EBCDIC code page used for literals and as local or system character set.
The interface also provides access to the info command. FCRINF() can be used, for example, to get information about files and supported CCSIDs.
The interface also provides a set of functions to calculate hash, MAC and HMAC values. FCRHASH() does everything in one call and is useful for transaction processing. FCRHINI(), FCRHADD() and FCRHFIN() can be used to calculate the checksum over several pieces of data. A sample program in Cobol using the hash function can be found in the library SRCLIB(SOFCRHSH).
Mainly for the hash function above the interface provides a set of function to encode and decode binary data. FCRBASE() does anything in one call and is useful for transaction processing. FCRBINI(), FCRBRUN() and FCRBFIN() can be used over several pieces of the data.
The interface is provided as DLL and each function of the DLL is also available as separate load module.
For dynamic linking of the load module (fetch) the hlq.FLAM.LOAD library must be part of the STEPLIB concatenation. A static link in COBOL or PL1 works as well with the separate load modules.
If you link one of the load modules statically to your application (each load module contains all required entries of the interface), you must include the DLL import files below to resolve the missing external references and set the binder option DYNAM(DLL):
hlq.FLAM.IMPORT(FLCBYT) hlq.FLAM.IMPORT(FL5CORE)
To link dynamically against the DLL you must include the import DLL file form the IMPORT library below:
hlq.FLAM.IMPORT(FLCRECLB)
Assembler programs need to activate the LE runtime system (PIPI) before any call. The calling convention is standard OS (save area and R1 points to the flagged parameter list (each parameter call be reference, R15 is not used (no return code))).
We recommend to use application-specific runtime option defaults with the CEEUOPT assembler language source program to establish application default options. You can find the recommended runtime options as member CEEUOPTx (x=3 for 31 bit and x=6 for 64 bit applications) in the hlq.FLAM.SRCLIB partitioned dataset.
The interface is only available as dynamic link library (DLL) or shared object (SO). You can link it dynamically, statically or load the library at runtime (dlopen(), LoadLibrary()) with the common DLL/SO mechanism of your operating system.
A sample program in C with name SCFCRCPY can be found as part of the installation package for mainframe systems in the library SRCLIBC(SCFCRCPY), with the corresponding compile and link step in JOBLIB(SBUILD). For other platforms (Windows, UNIX) the sample program source of SCFCRCPY is located in the sample directory and the compile and link procedures can be found in the Makefile of the same directory.
Additionally, COBOL samples with name SOFCRGET (uses FCRGET to read any kind of supported file format and writes the EBCDIC records to a host dataset), SOFCRXML (reads a clear or encoded XML file and writes the content as dump), SOFCREXV (parses an XML file based on several element converters and does operations with it) and SOFCRINT/FLT (reads records from a dataset, converts the strings to several different integer/floats values and calculates the sum) can be found in SRCLIB(SOFCRGET/SOFCRXML/SOFCREXV/SOFCRINT). How to call our LE requiring entries in a LE-less assembler program using CEEPIPI the sample SAFCRGET is provided. The corresponding compile and link procedures are also located as separate steps in JOBLIB(SBUILD). SOFCRGET/INT uses record formatting and SOFCRXML/EXV uses element formatting. SPxxxxxx are the corresponding samples in PL1.
This simple XML document contains some strings and integer numbers within some nested XML tags:
<foo>
<address>
<name> Max Herre </name>
<number> 13 </number>
</address>
</foo>
The file (FCROPN) can be opened with the file and format strings below. In this example, we use a static allocation.
FILE-STRING: "read.xml(file='DD:INPUT' nocmnt)" FORMAT-STRING: "format.element()"
If the file is compressed/encrypted/encoded in a supported format, it is decoded first. Character set conversion takes place automatically, if necessary. The internal neutral representation of XML elements is always a UTF-8 string. The nocmnt flag suppresses XML comments from the document while reading.
We need a couple of converters which are created with FCROPNV and an appropriate conversions string:
CONVERSION-STRING1: "write.string(chrset(ccsid=DEFAULT))" CONVERSION-STRING2: "write.string(chrset(ccsid=DEFAULT whitespace=collapse) padding=right)" CONVERSION-STRING3: "conv.integer(from(format.str()) to(format.bin(signed)))"
The first converter converts an element to a string with the system's default character set. This is used to convert the XML tag names in order to be able to compare them in our code.
The second converter also converts an element to a string with default CCSID, but additionally removes leading, trailing and duplicate successive whitespace and pads the string on the right to fill our output buffer that we pass to the interface functions. This is used to convert the "name" field of the document and store it in a fixed-length data structure.
The last converter is used to convert the string representation of a number to a signed number in two's complement format in system endianness. The output buffer passed must be 1, 2, 4 or 8 bytes long as these are the supported binary integer lengths.
Now that everything is setup, we can start reading XML elements with FCRLOC. If we encounter the start of an XML tag, we use the first converter to retrieve the tag name in our local charset and do a string compare. For the string data in the "name" tag, we use the second converter to store it in a COBOL data structure (COPYBOOK). If the element contains a number, we use the number converter to store the number as 32 bit signed bianry integer as "PIC 9(8) COMP".
Here is the example for reading the above XML file in a kind of simplified pseudo code:
pvFil=FCROPN("read.xml(file='DD:INPUT' nocmnt)","format.element()")
pvKyw=FCROPNV("write.string(chrset(ccsid=DEFAULT))")
pvStr=FCROPNV("write.string(chrset(ccsid=DEFAULT whitespace=collapse) padding=right)")
pvInt=FCROPNV("conv.integer(from(format.str(marker=period)) to(format.bin(signed)))")
uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
if (uiTyp=STARTELM && acKyw=="foo") then begin
uiTyp=FCRLOC(pvFil)
if (uiTyp==ENDSTARTELM) then begin
uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
if (uiTyp=STARTELM && acKyw=="address") then begin
uiTyp=FCRLOC(pvFil)
if (uiTyp==ENDSTARTELM) then begin
if (uiTyp=STARTELM && acKyw=="name") then begin
uiTyp=FCRLOC(pvFil)
if (uiTyp==ENDSTARTELM) then begin
uiTyp=FCRLOC(pvFil,acDat)
if (uiTyp=DATA) then begin
FCRCONV(pvStr,acDat,32,CB.NAME)
uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
if (uiTyp!="ENDELM" && acKyw!="name")
ERROR
end
end
end
end
uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
if (uiTyp=STARTELM && acKyw=="number") then begin
uiTyp=FCRLOC(pvFil)
if (uiTyp==ENDSTARTELM) then begin
uiTyp=FCRLOC(pvFil,acDat)
if (uiTyp=DATA) then begin
FCRCONV(pvInt,acDat,4,CB.NUMBER)
uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
if (uiTyp!="ENDELM" && acKyw!="name")
ERROR
end
end
end
end
end
end
end
end
FCRCLSV(pvInt)
FCRCLSV(pvStr)
FCRCLSV(pvKyw)
FCRCLS(pvFil)
FCRLOC returns a pointer to the neutral FL5 element. Alternatively, FCRGET can be used to retrieve a copy of the element data written to an application-provided buffer. The corresponding functions with a 'V' at the end (FCRLOCV/GETV) provide FCRLOC/FCRGET functionality, but with implicit data conversion by passing an open converter handle (created with FCROPENV). If the type of data is unknown in advance, the data can also be read with FCRLOC/FCRGET first, and then converted explicitly with FCRCONV. This function performs explicit conversions on arbitrary buffers. Therefore, it is not limited to data processed by the record interface.
1.10.0