FILTER

Synopsis

HELP:   Match against bloom filter
TYPE:   OBJECT
SYNTAX: FILTER(NAME='str',ROW='str',SIGNATURE='bin-hex',COLUMN[()...],SUBSET[=SEGMENT/RECORD/AUTO],RNNOTFND=INFO/WARNING/ERROR,CNNOTFND=INFO/WARNING/ERROR,NOINDEX=INFO/WARNING/ERROR,INVSIG=INFO/WARNING/ERROR,SIGNWC=INFO/WARNING/ERROR,INCASE=INFO/WARNING/ERROR)

Description

The FILTER object can be used to specify a binary signature together with the corresponding name of the format (row specification) for selecting segments. This data is used to form a subset of compressed and encrypted segments as a result set, which does not allow any back-keying to the data being searched for here and if decode done only the matching records for this signature are provided.

In addition, an array of match parameters must be provided to speed up the comparison of signatures. These match parameters are generated when the FILTER command is called. When using FILTER, it makes sense to include a list of the format descriptions for the respective archive, which can be generated using the ROWLIST parameter of the LIST command. This means that you do not have to specify the indexing of the individual columns, as long as they can be clearly determined from the row specifications.

The row specification name can contain wildcards, but this makes only sense, if wildcards are also used for the column names (CN). The SUBSET paramter copies only matching segments. If SUBSET is not defined, the whole member is selected. For DECODE, SUBSET=RECORD forces only signature matching records written into the output stream. This can have the effect that records are shown here where the signature matches but which do not match the data with which the signature was created. Such a record when comparing against a filter (signature) would be a 'false positive' where the hash values of the data happened to be the same. If you want to avoid this, then you have to compare against the clear data, for which MATCH.DATASET is available as an alternative. With SUBSET=AUTO, all records of a hit segment are selected for a FILTER object.

The CNNOTFND selection can be used to forces an error if no corresponding column name is found in the data for any column name in the search query. If this error not requested, this column is interpreted as a wildcard, i.e. this column always leads to a match. The fact that no error is generated here by default is useful if search fields are added later that do not exist in earlier archives, then these columns that were not indexed at the time must always lead to a match so that the sequential search over the data can then select the correct rows. In such a case, the search will simply be slower because this column was not yet indexed in this archive. Beside ERROR the selection supports also WARNING and by default an informational is written to the log. Analog error handling can be defined if a searched column is not indexed (NOINDEX), the indexing does not match (INVSIG) or wildcards used for indexed areas (SIGNWC).

The comparison of strings can be set to non-case sensitive via the NOCASE parameter. When indexing a column, you must specify when writing whether the comparison should be case sensitive or not. If a case sensitive column is not to be compared later, this is adjusted and a warning is issued. You can use the INCASE parameter to turn this into an INFO message or an error with cancellation.

Arguments