Send data to non-AWS S3 bucket

[bvarick]

Is it possible to save to something other than AWS S3? Either a different S3 compatible destination or a local directory?

[bvarick]

When I try to put a minio url for the s3 uri in the global_config.json, I get this error:

mdb-2130-normalizer-1  | Invalid bucket name "https:": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

botocore has a built in endpoint_url property that I think is for this purpose.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

Is it as simple as adding that to the global_config.json as a new key and then editing src/normalize/normalize.py and src/normalize/compact.py to use that value instead of a generated value for the s3_bucket_path?

[AndiLi99]

Hi, we haven’t tested this with a non-Amazon S3 bucket, but as long as it’s S3 compatible with the client, I don’t see why that wouldn’t work.

Based on the docs you linked, it sounds like your suggestion should work, I would be interested in hearing how it goes!

[bvarick]

I got it working and submitted a pull request

[AndiLi99]

Thank you @bvarick for the PR!