Retry to retrieve credentials if this fails during the credential refresh.

[ldebrouwer]

We've noticed that occasionally we can fail to retrieve credentials through the WebIdentity provider when the current credentials expire.

We currently use the ChainProvider, and through that we essentially fall through to the WebIdentity Provider because we use the pod IAM web session identity to provide access.

Our current hypothesis is that during the refresh process there may be a moment where the pod has no identity attached to it, for a few milliseconds, which causes it to fall through to using the EC2 Instance metadata instead.

We see this in the form of errors as such (anonymised);

AsyncAws\Core\Exception\Http\ClientException · HTTP 403 returned for "https://sqs.eu-west-1.amazonaws.com/". Code: AccessDeniedException Message: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/.eks-hms-node/i-00000000000000000 is not authorized to perform: sqs:sendmessage on resource: arn:aws:sqs:eu-west-1:XXXXXXXXXXXX:XXXXXXXXXXXX because no identity-based policy allows the sqs:sendmessage action

We'd never expect to see it fall through to using the EC2 Instance metadata.

Of course we could explicitly use the WebIdentity provider, instead of the ChainProvider, but I believe that due to the lack of retry logic to assume the role/retrieve credentials (something which the official AWS PHP SDK does provide) we'd still be in this position upon credential expiry/refresh.