TEXT

Synopsis

HELP:   Read text data from memory
TYPE:   OBJECT
SYNTAX: TEXT(BLKSIZE=num,RECLENGTH=num,SUPTWS,NELDLM,CCSID='str'/DEFAULT/ASCII/EBCDIC/BOMUTF/BOMUCS/SYSTEM/LOCAL,CHRMODE=STOP/IGNORE/SUBSTITUTE/IDENTITY/TRANSLIT,SKIPEQUAL,USRTABLE='str'/NPAS/SEPA/DELA/DLAX,ONEMAP,COMBINED=NFD/NFC/AUTO/ON/OFF,BOM,KEEPBOM,ENL2LF,RPLFFD[=num],PADCHAR=num,REGEXP(),DECODE=NONE/FIODEC/CRYDEC/CMPDEC/ALWAYS,DECRYPT[{}...],TABLE(),AVSCAN())

Description

Format text operates on blocks of text data. First, the text data is converted to the UTF-8 character set. Then the text data is split into record and rest elements based on text delimiters. The data must contain a valid text delimiter within the provided record/line length. A text record contains all text between two delimiters, optionally without trailing whitespace. The optional trailing whitespace and the delimiter make up the rest element. Taken together, the record and the rest elements represent the original data.

List of valid text delimiters (Unicode codepoints):

• 0x00 (padding character (can be changed))
• 0x0A (carriage return)
• 0x0D (line feed)
• 0x0A, 0x0D (carriage return line feed)
• 0x0A, ..., 0x0A, 0x0D (dirty delimiters)
• 0x0A, 0x0D, ..., 0x0D (dirty delimiters)
• 0x0C (form feed)
• 0x85 (next line (EBCDIC 0x15, optional if single byte ASCII))

The character data will be converted from the provided CCSID to UTF-8. If no CCSID is provided, a system dependent default CCSID is used: If available, the CCSID stored in the file system is used. If this is unset or not supported, the environment variable LANG is used. If this is also not set to a valid value, ISO-8859-1 (Latin-1) is used on ASCII and IBM-1047 (Open Systems Latin-1) on EBCDIC systems.

While text formatting, form feed characters may optionally be replaced by empty records. A padding character can be defined as special delimiter that, unlike other delimiters, is not added to the rest element after a record as it is not considered part of the original data.

Text formatting also supports:

• Different methods to define the record/line delimiter
• Suppression of trailing whitespace
• Replacement of horizontal tabs by whitespace characters
• Replacement of vertical tabs by whitespace and newline characters
• Replacement of remaining control characters
• Validation of records using regular expressions

During character conversion these options are supported:

• Stopping, ignoring or substitution when encountering an invalid or incomplete character
• A freely defined substitution character list
• Different system substitution / transliteration tables
• A user-defined substitution table to change the system tables
• Case management (upper, lower, folding and special casing)
• Transliteration between subsets (mapping) and normalization
• BOM management and automatic detection of character sets
• Reporting of ignored or substituted characters
• Auto detection of charsets

Arguments

• NUMBER: RECLENGTH=num - Maximum record length (Host) / line length (Unix/Win) for text parsing [512]
• SWITCH: SUPTWS - Suppress trailing whitespaces [FALSE]
• SWITCH: NELDLM - Activate NEL (0x85) as delimiter for ASCII character sets [FALSE]
• SWITCH: SKIPEQUAL - Skip conversion if formats are equal (e.g. UTF-8 to UTF-8)
• SWITCH: KEEPBOM - Keep byte order mark for faster conversion
• NUMBER: RPLFFD=num - Replace form feeds, filling rest of page with blank lines assuming n lines per page [60]
• NUMBER: PADCHAR=num - Padding character to be regarded as additional delimiter [0x00]
• NUMBER: DECODE=NONE/FIODEC/CRYDEC/CMPDEC/ALWAYS - Decode encoded data first (remove base encodings)
  • NONE - No automated decoding
  • FIODEC - Base decoding after file IO, before decryption
  • CRYDEC - Base decoding after decryption, before decompression
  • CMPDEC - Base decoding after decompression, before formatting if XML
  • ALWAYS - Always decode encoded data (like CMPDEC but also if TEXT)