Use of IBM Unicode Services

Since version 5.1.27, FLAM supports the use of IBM Unicode Services on system Z. The Unicode Services are used only for simple conversions: when no user table, transliteration, case mapping, subsets, whitespace handling, reporting, normalization or other complex feature is active. For such simple conversions, FLAM uses the IBM Unicode Services by default to reduce CPU and memory utilization. In edge cases where the native FLAM implementation is measurably faster, FLAM will use it instead.

It is important to note the differences in behavior between the IBM Unicode Services and the native FLAM character conversion module:

Behavior differences (AUTO and ON mode)

• If an incomplete character is encountered and the error handling is not STOP, the FLAM character conversion module ignores or substitutes this incomplete character. The IBM Unicode Services do return an EINVAL error in this case.

• IBM Unicode Services can produce errors for technically valid code points when processing UTF-8 encoded data. The FLAM character conversion module first decodes UTF-8 and verifies the code point afterwards, independent of the encoding. This means you can encode code points in multiple forms (e.g. the character 0 as 0x30 or 0xFC80808080B0). The IBM Unicode Services support only the shortest form (0x30) of an encoding for a code point and all other possible encodings are not accepted and will result in an error (invalid character).

• Depending on the Unicode Services version (APAR) a BOM character in the data might remain for UTF character sets or might be replaced by a substitution character for single byte character sets. When calling the Unicode Service FLAM will request the BOM removal, and FLAM will remove a BOM character if it is causing an invalid character error, but both will not work with certain versions of Unicode Services.

• No error is caused by a wrong BOM (invalid character) in the middle of the data and a BOM change from LE to BE or vice versa is also not detected. For data with mixed endianness the Unicode Services should be deactivated.

To prevent these changes in behavior you must set the system symbol &FLIBMUS or the environment variable FL_IBM_UNICODE_SERVICE to OFF, but this will result in more CPU and memory utilization for simple character conversions (see chapter Used Environment variables). The environment variable's default is AUTO which ensures that only the minimal behavior changes above are in place. Setting it to ON also enables the use of Unicode Services for character conversion mode STOP via the round-trip conversion technique, which causes the additional behavior changes described below.

Additional behavior differences (ON mode only)

• A lot of non-convertible characters will be converted to a unique character in the target character set without an error using the round trip conversion scheme.

• A substitution character is only converted with activated substitution. Without substitution, the substitution character 0x1A in ASCII/UTF or 0x3F in EBCDIC will cause an error. Based on that behavior FLAM will request the substitution if Unicode Services are used.

We recommend setting FL_IBM_UNICODE_SERVICE=ON to maximize performance if the limitations described above are acceptable because checking each character if it is valid or not has a significant performance impact.