PROVIDER

Synopsis

HELP:   How to reach the S3 service and who we are; can come from a parameter file
TYPE:   OBJECT
SYNTAX: PROVIDER(ENDPOINT='str',REGION='str',PATHSTYLE,PORT=num,CABUNDLE='str',ACCESSKEY='str',SECRETKEY='str',SESSIONTOKEN='str',IMDS,APIKEY='str',IAMENDPOINT='str')

Description

Everything that is constant for one Amazon S3 or S3-compatible service.

A provider answers two questions that do not change from archive to archive: how the service is reached, and who we are to it. The endpoint, the region, the URL style, the port and the trust anchor belong to the first, the credentials to the second. What stays outside is what varies: the BUCKET and the PREFIX name the data, and the write buffers, OVERWRITE and TIMEOUT describe the run.

TIMEOUT is deliberately not part of the provider. It says how long this particular run may make no progress, which is a property of the job and not of the service.

Why the credentials are not an object of their own

The secret parts and the public parts of the credential sit together in here, and they sit next to the endpoint. An access key id without its secret is useless and is not a secret itself, so "secret versus public" is not a useful line. And a credential belongs to an endpoint: an operator sets both up at the same time and changes them together.

The practical consequence is that the same command runs unchanged in a test and in a production environment. Only the provider differs.

Reading a provider from a parameter file

The command line parser reads any object from a file, so:

store.s3(PROVIDER='/etc/flam/awsprod.par' BUCKET=mybucket PREFIX='hr/2026')

The file holds the parameter list without the surrounding brackets:

ENDPOINT='https://s3.eu-central-1.amazonaws.com' REGION=eu-central-1 ACCESSKEY=AKIAIOSFODNN7EXAMPLE SECRETKEY='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'

The secret then never appears on a command line, which is visible in the process list and in the shell history of every user on the machine, and the file can carry its own access rights. On z/OS the file may be a data set.

Which authentication method applies

The method is chosen by which parameters are present, and the store reports its choice on the open line. The combinations are:

• ACCESSKEY and SECRETKEY -- a long-lived key pair, signed with SigV4.
• ACCESSKEY, SECRETKEY and SESSIONTOKEN -- temporary credentials that were obtained elsewhere.
• IMDS -- the credentials come from the instance metadata service of the machine FLAM runs on. Only meaningful inside a cloud instance.
• APIKEY -- IBM Cloud Object Storage with an IBM Cloud IAM api key. The key is exchanged for a token that is then renewed as needed, and the requests carry it instead of a signature. IAMENDPOINT overrides where the key is exchanged, which is only of interest for a test.
• Nothing at all -- the shared AWS configuration is used, that is the environment, then the profile files under the home directory.

APIKEY is not an addition to the pattern above but a replacement. IBM Cloud Object Storage speaks the S3 API, so it is this store and not another one, but its own credential is an api key and not a key pair. Naming both is a configuration mistake, and the api key wins -- it is the deliberate act, where a leftover AWS_ACCESS_KEY_ID in a shell is not.

A service credential for IBM Cloud Object Storage carries both: the api key and, when it was created with the HMAC option, a key pair as well. Either works. The key pair is signed with SigV4 like any other S3 service, the api key goes the way described here.

The service instance is deliberately not a parameter. IBM Cloud Object Storage wants it in a header for the two operations that address the service rather than its contents, creating a bucket and listing the buckets, and FLAM does neither: the bucket exists before an archive is written to it. An object request that carries the header anyway is refused, so there is nothing to be gained by offering it.

Why PORT exists next to ENDPOINT

ENDPOINT is optional. Without it the endpoint is derived from the region and the bucket, and then there is no URL text to carry a port. PORT reaches the derived host on a different port, a proxy or a gateway in front of the service. When ENDPOINT is given, PORT replaces a port already in it, because it is defined as an override.

Arguments

• STRING: ENDPOINT='str' - Custom endpoint URL for S3-compatible stores (e.g. MinIO) [default: AWS endpoint; env: AWS_ENDPOINT_URL]
• STRING: REGION='str' - AWS region (e.g. us-east-1) [default: us-east-1; env: AWS_DEFAULT_REGION]
• SWITCH: PATHSTYLE - Use path-style URLs instead of virtual-hosted [FALSE; env: FL_S3_PATH_STYLE]
• NUMBER: PORT=num - Port override merged into the endpoint (0 = the endpoint decides) [0; env: FL_S3_PORT]
• STRING: CABUNDLE='str' - Trust anchor for the TLS connection: PEM file, or SAF key ring on z/OS [env: FL_CA_BUNDLE; SAF: RST.CABUNDLE]
• STRING: ACCESSKEY='str' - AWS access key ID [env: AWS_ACCESS_KEY_ID]
• STRING: SECRETKEY='str' - AWS secret access key [kept secret; env: AWS_SECRET_ACCESS_KEY]
• STRING: SESSIONTOKEN='str' - AWS STS session token for temporary credentials [kept secret; env: AWS_SESSION_TOKEN]
• SWITCH: IMDS - Take the credentials from the instance metadata service [FALSE; env: FL_S3_IMDS]
• STRING: APIKEY='str' - IBM Cloud api key; selects IAM instead of SigV4 [kept secret; env: FL_IBM_APIKEY]
• STRING: IAMENDPOINT='str' - IBM Cloud IAM token endpoint [env: FL_IBM_IAM_ENDPOINT, def: https://iam.cloud.ibm.com]