[BUG] Blocking behavior in AWSV4SignerAsyncAuth

[dacevedo12]

What is the bug?

Yesterday's AWS us-east-1 outage revealed that botocore talks to STS using requests, and retries them using a backoff with time.sleep, which are both blocking functions and will cause performance degradation as they hold the event-loop hostage.

In this line: sig_v4_auth.add_auth(aws_request) https://github.com/opensearch-project/opensearch-py/blob/main/opensearchpy/helpers/asyncsigner.py#L54

How can one reproduce the bug?

Try calling a search operation using AWSV4SignerAsyncAuth while simulating poor network conditions to trigger the retry with backoff mechanism in botocore

What is the expected behavior?

It should not block the event-loop

What is your host/environment?

Not relevant

Do you have any screenshots?

Do you have any additional context?

Three possible solution proposals:

• Use aiobotocore, as it already takes care of that https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/signers.py
• Document a heads-up warning to use only frozen (read-only) credentials with AWSV4SignerAsyncAuth to prevent this blocking refresh operation
• Run the sig_v4_auth.add_auth(aws_request) call in a thread pool executor

[dblock]

Ouch. Regardless of the fix we do, do you think you can write a test that reproduces this kind of behavior?

I would go with aiobotocore to delegate this kind of problems, I don't see downsides.

[dacevedo12]

Ouch. Regardless of the fix we do, do you think you can write a test that reproduces this kind of behavior?

I would go with aiobotocore to delegate this kind of problems, I don't see downsides.

Sure, I can take a look this weekend

[dblock]

@dacevedo12 Any interest of picking this up again?

[dblock]

In 2.4.0 we've added support for Urllib3 Sigv4 (it's now the default). Does that implementation have the same problem @dacevedo12?

[dacevedo12]

One could say this is mitigated by #470 as it defaults to using frozen credentials, which won't be refreshed by boto under the hood.

However, if it goes through the other branch with credentials that (for some unknown reason) don't have a get_frozen_credentials method, it will refresh them when needed, and that process is potentially blocking.

Refreshing credentials is usually a very quick network call to AWS STS so in most cases it's not concerning, but in scenarios like the one AWS experienced when I originally reported this issue, there will be retries with exponential backoff (using time.sleep) and that for sure will be blocking.

[thehesiod]

hello, maintainer of aiobotocore here, yes this should support the async version of AioSession.get_credentials. So in this case swapping to .get_credentials and checking to see if the method is async or not

[dblock]

Thanks for jumping in @thehesiod! Appreciate if someone on this thread could PR something here.

[thehesiod]

made a draft here: #834