HELP: Storage location of existing archive
TYPE: OVERLAY
SYNTAX: STORE/SAVE.{FILE()/FOLDER()/S3()/AZURE()/SEGUSER()/OBJUSER()/IBMCDA()}
The STORE overlay selects where the FLAM archive is read from.
The following storage backends are available:
Please refer to the documentation for each backend for its parameters. The section below explains which of these are segment stores and which are object stores, and -- for the cloud backends -- how FLAM reaches them on each platform and what that costs.
A FLAM archive is a set of segments, and STORE decides where those segments live. What a segment is, is worth knowing before the rest of this section makes sense, because it decides how many of them there are:
Two shapes of store exist for keeping them, and the difference matters both for performance and for what a backend has to be able to do:
An object store does not write one object per segment, and that is what makes the kind practical at all. Segments of the same type are collected in a write buffer and go out together once it is full:
Parameter Collects Default Effect
--------- ------------------ -------- ------------------------------
DATBUF data segments 4 MiB One object per 4 MiB of data
instead of one per segment.
MBRBUF member segments 1 MiB Many member indexes share one
object.
DIRBUF the main segment 1 byte No collecting. There is only
one main segment, so there is
nothing to collect it with,
and any buffer at least as
large as the segment is
bypassed anyway.
So an archive of ten thousand small members does not become ten thousand objects; it becomes a handful of large ones plus the main segment. That is the simplification that lets the same object store be a folder on a local disk or a bucket in a remote cloud without the caller noticing the difference -- and it is the reason these two parameters are the first thing to reach for when a cloud archive is slow.
Within the object stores there is a second distinction. FOLDER writes into a directory of the local file system, so an object costs one file operation. The CLOUD backends -- STORE.S3, STORE.AZURE and, on z/OS, STORE.IBMCDA -- reach a remote service over the network, so an object costs a request. How many of those requests can be in flight at once is what decides the throughput of a cloud archive, and it is not the same on every platform.
On Linux, Windows, AIX, Solaris and z/Linux the cloud backends use FLAM's own HTTP client, which is built on libcurl. That client keeps a connection pool for the lifetime of the store, negotiates HTTP/2 where the server offers it, and drives several transfers at once. On top of that FLAM prefetches segments it expects to need and queues writes, so the network stays busy while the archive is being processed.
libcurl is not part of z/OS, and building it there is not something a customer should have to do. FLAM therefore uses the z/OS Client Web Enablement Toolkit (HWTHTTP) instead, which is a component of the operating system. That works, and it has two consequences that have to be planned for rather than discovered.
The first is speed. HWTHTTP speaks HTTP/1.0 and HTTP/1.1 and no more, so there is no multiplexing: one request is on the connection at a time and the next one waits for the answer to the previous. Neither the prefetch nor the write queue can hide that, because both need more than one request in flight to help. A cloud archive on z/OS is therefore markedly slower than the same archive on an open platform, and the gap grows with the number of segments rather than with their size. What does help is the write buffering described above: raising DATBUF, and MBRBUF for an archive with many members, turns many small requests into few large ones. That is the one knob with real effect on this route.
The second is that HWTHTTP requires a POSIX(ON) Language Environment enclave, and an address space can hold exactly one of those. Where FLAM already runs in such an enclave the conversation happens in process; where it does not, FLAM starts a helper program in a SECOND ADDRESS SPACE and exchanges the data with it through shared memory. Which of the two applies is decided by the environment, not by anything the user sets:
The helper is started once per address space and reused for every archive and every store in it, so its cost is paid once: one additional address space, one LE enclave, and one shared memory segment sized from DATBUF and placed below the 2 GB bar. It ends with the process. In a job log it announces itself, and its name is FLSERVHZ.
STORE.IBMCDA does not use FLAM's transport at all. It hands each operation to the DFSMSdfp Cloud Data Access component of z/OS, which the system programmer configures once with a provider file and a credential store. Two things follow from that.
It is not the faster route. Cloud Data Access sits on the same Client Web Enablement Toolkit underneath, so the HTTP/1.1 limitation above applies to it as well, and at the level available on z/OS 3.1 it has no session either -- every operation stands on its own, so not even the connection is reused. Where it does need a second address space it uses the same helper as the native route, and only MVS needs one: on z/OS UNIX the Cloud Data Access services run in process in both addressing modes.
What it is good for is configuration and security. The endpoint, the TLS material and the credentials are described once, by the system programmer, and never appear in a FLAM command or a JCL member -- a STORE.IBMCDA names a provider and a bucket and nothing else. On a system where cloud credentials must not be visible to the user who runs the archive, that is the route to take, and the slower transfer is the price.