Allow environment variable login to work with requesting s3 credentials

[xaviernogueira]

Context: We are working in an cloud-based (AWS) pipeline, where one does not have write permissions (i.e., we can't make netrc files). The environment variable login is our only choice, as manual input is also a non-option.

Problem: While env variable login works with search data, we are searching for data in one lambda function, passing the granule info as JSON to another, and then downloading the data in a different lambda function. The issue here is we need to be able to download or read the data from just the links. Therefore the first step as documented here is to send a requests.get() to get our temporary s3 credentials. However, the S3 credentials endpoint does not work with the env variable login pattern!

Is there a workaround or something?

[jrbourbeau]

Thanks for the issue @xaviernogueira. Does earthaccess.get_s3_credentials(...) work for you in that case? There should be extra logic to handle the environment variable authentication there.

Depending on what you're doing, you might also try some of the high-level APIs like earthaccess.get_s3fs_session(...) which can remove some boilerplate code. (This is a separate point from the issue you're running into, still thought I mention though)

[jrbourbeau]

FWIW I've definitely used earthaccess in cloud pipelines before using environment variable authentication (not with AWS Lambda functions, but with Coiled serverless functions). This is a pattern we'll want to support, so let us know if the above suggestion ^ doesn't work for you

[xaviernogueira]

@jrbourbeau thanks! earthaccess.get_s3_credentials(...) seems to be the solution I was looking for. We got locked out of the server for the day (it said 10 minutes but it still isn't working), so we will circle back and see if it works in the Lambda tomorrow.

Also nice work with dask and zarr (I see you are a contributor), two of my favorite libraries! I will be at the NYC Zarr sprint in Febuary, maybe I'll catch you there?

[xaviernogueira]

@jrbourbeau also note that your documentation is slightly inccorect for get_s3_credentials(...) as "endpoint" is not a valid argument (despite being in the docs)...but should be? Maybe I'll make a mini PR at some point when I am less busy for that.

[jrbourbeau]

Could you point me to where in the docs you're seeing this? FWIW there's the top level earthaccess.get_s3_credentials(...) (doesn't have endpoint=) and also an Auth.get_s3_credentials(...) method (has endpoint=)

Also nice work with dask and zarr (I see you are a contributor), two of my favorite libraries! I will be at the NYC Zarr sprint in Febuary, maybe I'll catch you there?

Glad to hear it! Unfortunately I won't be able to make it, but I hope it's a good time!

[andypbarrett]

I've added a documentation label because we need to explain what can and cannot be done at the moment as well as explain better how to do these tasks.