export_workspace: allow setting the region when configuring the workspaces

[VictorVerhaert]

Check what region is used for setting up the s3 client to a bucket and allow for setting the region specific for export_workspace.
Otherwise the same workspaces that work on CDSE won't work on OTC which has a different default s3 endpoint

[JeroenVerstraelen]

region = endpoint

Option 1:

• Use the same bucket on OTC and WAW3-1
• Bucket credentials given in WAW3-1 should accessible in OTC

Option 2:

• Create new bucket on OTC
• Merge buckets over time

[pvbouwel]

In the draft standard for workspaces (https://github.com/Open-EO/openeo-api/blob/draft/extensions/workspaces/openapi.yaml#L41) each workspace has a workspace provider. This workspace provider could determine which region (endpoint) should be used.

[pvbouwel]

This relates to MVP1 of https://confluence.vito.be/pages/viewpage.action?spaceKey=EP&title=OpenEO+S3+access

It requires:

• feature: allow proxying multiple S3 endpoints  VITObelgium/fakes3pp#3
• feature: Add support for session tags in assumeRoleWithWebIdentity (nested claim format) VITObelgium/fakes3pp#4
• Make sure every OpenEO instance has key material available within web driver pod
• Make sure JWT token for base trust is generated and set as environment variable AWS_WEB_IDENTITY_TOKEN_FILE MVP1
• Run as a daemon set on worker nodes and bind to a host port
• Make sure profiles are available within the spark pods
• Make sure there is a fake domain name that resolves to the node ip
• Add profile names to workspace config

[pvbouwel]

It seems we do not yet have a way to configure config files for a job execution in geopyspark yet. Most of the time environment variables are used. There are already config maps that are created for a spark application (like prometheus config) but these seem to be managed by the spark operator. It seems we need one for ourselves as well to store:

• the token file
• the aws profiles
• bucket to profile mappings (future)

[pvbouwel]

First add public and private keys to each of the environments (CDSE) as that will be a pre-requisite for the web identity tokens

[pvbouwel]

Public and private keys for IDP are staged into vault. For each environment I created a new version of the vault object we keep but I added the 2 fields that are generated using:

echo -e "y\n\n\n" | ssh-keygen -t rsa -b 2048 -m PEM -f rsa &&ssh-keygen -f rsa -e -m PEM > rsa.pub && echo -e "  \"idp_private_key\": \"$(cat rsa | tr '\n' '@' | sed 's/@/\\n/g' | sed 's/\\n$//'| sed 's/\\n/\n/'g  | base64 | tr -d '\n')\",\n  \"idp_public_key\": \"$(cat rsa.pub | tr '\n' '@' | sed 's/@/\\n/g' | sed 's/\\n$//' | sed 's/\\n/\n/'g | base64 | tr -d '\n')\","

[pvbouwel]

Deployed a build-version of the STS and s3 proxy with the token trust to cdse-staging but when I run the artifacts upload workflow it fails if I don´t specify the region explicitly. So that would be a regression must fix that first.