Error response received from instance metadata service - Request is failing to the call made to metadata service

[psk200]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I'm using the below js v3 sdk version

"@aws-sdk/client-secrets-manager": "^3.14.0", "@aws-sdk/client-sts": "^3.282.0", "@aws-sdk/credential-provider-imds": "^3.272.0",

Environment : EKS Pod

EC2 - worker

aws ec2 modify-instance-metadata-options --instance-id <instance_id> --http-tokens required --http-endpoint enabled --http-put-response-hop-limit 1

Pod is getting the role
from service account
the complete process of assuming role to service account is followed from below link
https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

This part of the code is assuming the pod role

const client1 = new STSClient({ region: process.env.REGION});
const command = new GetCallerIdentityCommand({});
const data = await client1.send(command);
console.log(":get_caller_identity:", data)

Response

:get_caller_identity: {
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'requestId',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  UserId: 'EXAMPLEID:aws-sdk-js-session-EXAMPLE_SESSION',
  Account: 'ACCOUNT_ID',
  Arn: 'arn:aws:sts::<ACCOUNT_ID>:assumed-role/<role_name_from_service_account>/aws-sdk-js-session-EXAMPLE_SESSION'
}

Complete code snippet

const {
    SecretsManagerClient,
    GetSecretValueCommand,
  } = require("@aws-sdk/client-secrets-manager");
  
  const { STSClient, GetCallerIdentityCommand } = require("@aws-sdk/client-sts");
  
  
  const getSecret = async () => {
    return new Promise(async (resolve, reject) => {
  
      try {
        const client1 = new STSClient({ region: process.env.REGION});
        const command = new GetCallerIdentityCommand({});
        const data = await client1.send(command);
        console.log(":get_caller_identity:", data)
      } catch (error) {
        console.log("getting sts failed ")
        return reject(error);
      }
  
      let response;
  
      try {
        const client = new SecretsManagerClient({
          region: process.env.REGION,
        });
        response = await client.send(
          new GetSecretValueCommand({
            SecretId: process.env.APP_SECRET_NAME,
            VersionStage: "AWSCURRENT",
          })
        );
      } catch (error) {
        console.log("Secret fetching failed ", error)
        return reject(error);
      }
  
      const secret_string = response.SecretString;
  
      const secrets = JSON.parse(secret_string);
  
      
      resolve(secrets);
    })
  }
  
  module.exports = { getSecret };

SDK version number

@aws-sdk/client-secrets-manager 3.14.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node v16

Reproduction Steps

Environment : EKS Pod

1 Create a eks worker node group with imdsv2 enabled and hop limit 1

aws ec2 modify-instance-metadata-options --instance-id <instance_id> --http-tokens required --http-endpoint enabled --http-put-response-hop-limit 1

2. Create service account and attach the role policy

cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default
EOF
kubectl apply -f my-service-account.yaml
aws iam attach-role-policy --role-name my-role --policy-arn=arn:aws:iam::$account_id:policy/my-policy

Deploy the Pod

the complete process of assuming role to service account is followed from below link
https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

Observed Behavior

SDK details:

@aws-sdk/client-secrets-manager 3.14.0

const client = new SecretsManagerClient({
          region: process.env.REGION,
        });
        response = await client.send(
          new GetSecretValueCommand({
            SecretId: process.env.APP_SECRET_NAME,
            VersionStage: "AWSCURRENT",
          })
        );

This above code returns the following response

 [ProviderError: Error response received from instance metadata service] {
  tryNextLink: true,
  statusCode: 401
}

Expected Behavior

The getsecretvalue should return the secret but its throwing error response from metadata service

The same is working with aws-sdk v2

Possible Solution

No response

Additional Information/Context

No response

[atif-saddique-deel]

having the same issue.

/node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/credential-provider-imds/dist-cjs/remoteProvider/httpRequest.js:26
                reject(Object.assign(new property_provider_1.ProviderError("Error response received from instance metadata service"), { statusCode }));
                                     ^

ProviderError: Error response received from instance metadata service
    at ClientRequest.<anonymous> (/node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/credential-provider-imds/dist-cjs/remoteProvider/httpRequest.js:26:38)
    at ClientRequest.emit (node:events:513:28)
    at HTTPParser.parserOnIncomingClient (node:_http_client:693:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at Socket.socketOnData (node:_http_client:534:22)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  tryNextLink: true,
  statusCode: 404,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

 "@aws-sdk/client-secrets-manager": "3.391.0",
 "@aws-sdk/credential-providers": "3.391.0",

[atif-saddique-deel]

@yenfryherrerafeliz any update on this?
We keep getting this issue almost 5% of the time.
From our current logs, I can see that we trigger around 2000 pods in one hour and we get this error on almost 90 to 100 pods. If I restarted the pod, it works fine.
We make this call in init container to get secret value and it keeps failing intermittently.
Even the maxRetries and timeout options don't seem to work properly.

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import { fromInstanceMetadata } from '@aws-sdk/credential-providers';

 const secretsManager = new SecretsManagerClient({
        region: AWS_REGION,
        credentials: fromInstanceMetadata({
            timeout: 10000,
            maxRetries: 5,
        })
    });

await secretsManager.send(new GetSecretValueCommand({SecretId}));

Error

{
  "level":"ERROR","time":1692366362804,"clientName":"SecretsManagerClient","commandName":"GetSecretValueCommand",
  "input":{"SecretId":"/my-app"},"error":{"tryNextLink":true,"name":"ProviderError","statusCode":404,
  "$metadata":{"attempts":1,"totalRetryDelay":0}},"metadata":{"attempts":1,"totalRetryDelay":0}
}

[atif-saddique-deel]

any update?
not sure if it will help but we are getting this issue on spot instances.

[yenfryherrerafeliz]

Hi @psk200, @atif-saddique-deel, this error is coming from the instance metadata service itself. I will investigate this with the proper service team related to this and I will get back to you folks.

I will provide updates soon.

Thanks!

[yenfryherrerafeliz]

Hi @psk200, I see you mentioned that the same implementation worked in v2. Can you please provide the code snippet for v2?

Thanks!

[atif-saddique-deel]

Hi @yenfryherrerafeliz
in my case the error seems to be coming from kube2iam which fails to assume iam role and causes the requests from specific pod to fail.

[yenfryherrerafeliz]

Hi @atif-saddique-deel, kube2iam is a third party tool that is not under our support scope. However, from my understanding what kube2iam does is to intercept any request done with the SDK and inject credentials there, so that it works most likely as a proxy. So, are you making this service is up before start doing any operations with the SDK?

Please let me know.

Thanks!

[yenfryherrerafeliz]

Hi @psk200, can you please confirm if you are also using kube2iam?, if so, please see my comment here.

Thanks!

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.