Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assuming role configured in AWS profile does not work outside of aws partition (China, US Gov Cloud, etc.) #6711

Open
3 of 4 tasks
csy97 opened this issue Dec 3, 2024 · 9 comments
Assignees
Labels
bug This issue is a bug. p2 This is a standard priority issue queued This issues is on the AWS team's backlog

Comments

@csy97
Copy link

csy97 commented Dec 3, 2024

Checkboxes for prior research

Describe the bug

When I call GetCallerIdentityCommand at cn-north-1 the request will be sent to the STS service at us-east-1. China's resources are isolated from global, so obviously this won't work in China.

Regression Issue

  • Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/package-name@version, ...

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.20.5

Reproduction Steps

credentials file like this

[default]
aws_access_key_id = a'k
aws_secret_access_key = sk

config file like this

[default]
region = cn-north-1
[profile tes_assume]
region = cn-north-1
role_arn = arn:aws-cn:iam::xxx:role/test_assume

The js code is very simple.When I try to execute GetCallerIdentityCommand using profile test_assume I get the error “Error fetching identity: InvalidClientTokenId: The security token included in the request is invalid”

const { STSClient, GetCallerIdentityCommand } = require("@aws-sdk/client-sts");
const { fromIni } = require("@aws-sdk/credential-providers");

async function getRoleIdentity() {
  const credentials = fromIni({ profile: "test_assume" });

  const stsClient = new STSClient({
    credentials, 
    region: "cn-north-1", 
  });

  try {
    const command = new GetCallerIdentityCommand({});
    const response = await stsClient.send(command);

    console.log("Current Role Identity:");
    console.log(`Account: ${response.Account}`);
    console.log(`UserId: ${response.UserId}`);
    console.log(`ARN: ${response.Arn}`);
  } catch (error) {
    console.error("Error fetching identity:", error);
  }
}

package.json

{
  "dependencies": {
    "@aws-sdk/client-sts": "^3.699.0",
    "@aws-sdk/credential-providers": "^3.699.0"
  }
}

I capture tcpdump request during cdk bootstrap command. The output is

[root@ip-172-31-22-83 ec2-user]# tcpdump -n port 443
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
07:29:14.889285 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [S], seq 4148898204, win 62727, options [mss 8961,sackOK,TS val 1119140418 ecr 0,nop,wscale 7], length 0
07:29:15.117433 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [S.], seq 2138776450, ack 4148898205, win 8190, options [mss 1460,nop,wscale 6,nop,nop,sackOK], length 0
07:29:15.117501 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [.], ack 1, win 491, length 0
07:29:15.118109 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [P.], seq 1:386, ack 1, win 491, length 385
07:29:15.345920 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [.], ack 1, win 980, length 0
07:29:15.346081 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [.], ack 386, win 974, length 0
07:29:15.346200 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 1:94, ack 386, win 974, length 93
07:29:15.346200 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 94:100, ack 386, win 974, length 6
07:29:15.346234 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [.], ack 94, win 491, length 0
07:29:15.346243 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [.], ack 100, win 491, length 0
07:29:15.347120 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [P.], seq 386:810, ack 100, win 491, length 424
07:29:15.363706 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 94:100, ack 386, win 974, length 6
07:29:15.363745 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [.], ack 100, win 491, options [nop,nop,sack 1 {94:100}], length 0
07:29:15.575130 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [.], ack 810, win 968, length 0
07:29:15.575278 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 100:260, ack 810, win 968, length 160
07:29:15.575308 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [.], ack 260, win 490, length 0
07:29:15.575601 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 260:292, ack 810, win 968, length 32
07:29:15.575614 IP 172.31.22.83.57130 > 67.220.245.46.https: Flags [.], ack 292, win 490, length 0
07:29:15.575688 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 292:3212, ack 810, win 968, length 2920
07:29:15.575688 IP 67.220.245.46.https > 172.31.22.83.57130: Flags [P.], seq 3212:5347, ack 810, win 968, length 2135

I capture tcpdump request during cdk bootstrap command. The output is

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:18:06.262164 IP 172.31.3.41.43922 > 72.21.206.96.https: Flags [S], seq 1915800606, win 62727, options [mss 8961,sackOK,TS val 3264941276 ecr 0,nop,wscale 7], length 0
14:18:06.494006 IP 72.21.206.96.https > 172.31.3.41.43922: Flags [S.], seq 3188368014, ack 1915800607, win 8190, options [mss 1460,nop,wscale 6,nop,nop,sackOK], length 0
14:18:06.494066 IP 172.31.3.41.43922 > 72.21.206.96.https: Flags [.], ack 1, win 491, length 0
14:18:06.494873 IP 172.31.3.41.43922 > 72.21.206.96.https: Flags [P.], seq 1:382, ack 1, win 491, length 381
14:18:06.726602 IP 72.21.206.96.https > 172.31.3.41.43922: Flags [.], ack 1, win 980, length 0
14:18:06.727233 IP 72.21.206.96.https > 172.31.3.41.43922: Flags [.], ack 382, win 976, length 0

As you can see, the ip address of STS service requested is in us-east-1 region.

Apparently, it could not work in the China region. Please fix this issue, Thanks!

Observed Behavior

[root@ip-172-31-22-83 nodejs]# node test3.js
Error fetching identity: InvalidClientTokenId: The security token included in the request is invalid
    at throwDefaultError (/root/nodejs/node_modules/@smithy/smithy-client/dist-cjs/index.js:836:20)
    at /root/nodejs/node_modules/@smithy/smithy-client/dist-cjs/index.js:845:5
    at de_CommandError (/root/nodejs/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:505:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /root/nodejs/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /root/nodejs/node_modules/@smithy/core/dist-cjs/index.js:168:18
    at async /root/nodejs/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /root/nodejs/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getRoleIdentity (/root/nodejs/test3.js:17:22) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '72193656-2308-4e43-b1b1-0b317ce6aaa1',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'InvalidClientTokenId'
}

Expected Behavior

SDK V3 Using source_profile works fine.

Possible Solution

No response

Additional Information/Context

No response

@csy97 csy97 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 3, 2024
@csy97
Copy link
Author

csy97 commented Dec 3, 2024

When I modify the code to the following it is working

My confusion is that even if I specify the region in the .config and credentials files it doesn't seem to take effect whether fromIni can read . /aws/config?

[root@ip-172-31-22-83 nodejs]# cat a.js
const { fromIni } = require("@aws-sdk/credential-providers");
const { STSClient, GetCallerIdentityCommand } = require("@aws-sdk/client-sts");

async function getRoleIdentity() {
  const credentials = fromIni({
    profile: "tes_assume",
    clientConfig: { region: "cn-north-1" } 
  });

  console.log(`credentials:`, credentials);

  const stsClient = new STSClient({
    endpoint: "https://sts.cn-north-1.amazonaws.com.cn",
    credentials, 
    region: "cn-north-1", 
  });

  try {
    // 调用 GetCallerIdentity API
    const command = new GetCallerIdentityCommand({});
    const response = await stsClient.send(command);

    console.log("Current Role Identity:");
    console.log(`response:`, response);
    console.log(`Account: ${response.Account}`);
    console.log(`UserId: ${response.UserId}`);
    console.log(`ARN: ${response.Arn}`);
  } catch (error) {
    console.error("Error fetching identity:", error);
  }
}


getRoleIdentity();

@zshzbh zshzbh removed the needs-triage This issue or PR still needs to be triaged. label Dec 4, 2024
@zshzbh zshzbh self-assigned this Dec 4, 2024
@zshzbh zshzbh added p2 This is a standard priority issue investigating Issue is being investigated and/or work is in progress to resolve the issue. labels Dec 4, 2024
@zshzbh
Copy link
Contributor

zshzbh commented Dec 5, 2024

I can't reproduce this issue

In config -

[profile codeartifact]
[default]
region = us-west-2
output = json

[profile test]
source_profile = default
role_arn       = arn:aws:iam::XXXX:role/testRole 

In credentials

[default]
aws_access_key_id=XXX
aws_secret_access_key=XXXX


[dev]
aws_access_key_id=XXX
aws_secret_access_key=XXX

Code I have

import { STSClient, GetCallerIdentityCommand } from "@aws-sdk/client-sts";
import { fromIni } from "@aws-sdk/credential-providers";

async function getRoleIdentity() {
  const credentials = fromIni({
    profile: "test",
   //  clientConfig: { region: "us-east-1" },
  });
  console.log("credentials: ", credentials);
  const stsClient = new STSClient({
    credentials,
    region: "us-east-1",
  });
  try {
    const command = new GetCallerIdentityCommand({});
    const response = await stsClient.send(command);

    console.log("Current Role Identity:");
    console.log(`Account: ${response.Account}`);
    console.log(`UserId: ${response.UserId}`);
    console.log(`ARN: ${response.Arn}`);
  } catch (error) {
    console.error("Error fetching identity:", error);
  }
}
getRoleIdentity();

I can get the result without specifying the region. I will talk about it with the team

@zshzbh
Copy link
Contributor

zshzbh commented Dec 5, 2024

Hey @csy97 ,

cn-north-1 region is in non-commercial partition, inner client region for credential provider clients needs to be specified for non-commercial partition, otherwise it will default to us-east-1, and us-east-1 does not work for non-commercial partitions.

Please let me know if you have any other questions.

Thanks
Maggie

@zshzbh zshzbh added response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. and removed investigating Issue is being investigated and/or work is in progress to resolve the issue. labels Dec 5, 2024
@zshzbh zshzbh added the investigating Issue is being investigated and/or work is in progress to resolve the issue. label Dec 9, 2024
@rix0rrr
Copy link

rix0rrr commented Dec 10, 2024

Hi @zshzbh,

We do have additional questions.

The region= itself is located in the INI file. This is the same format and mechanism the AWS CLI uses to select the region to use for the AssumeRole call, and it is the same mechanism the SDKv2 used before.

It doesn't seem correct for SDKv3 to ignore the region in the INI file?

@rix0rrr
Copy link

rix0rrr commented Dec 10, 2024

I did some research using mitmproxy to see what the different clients do, comparing the AWS CLI, SDKv2 and SDKv3.

TL;DR: The CLI and SDKv2 behave the same, SDKv3 behaves differently.

Config

The behavior depends a lot on the configuration setup. I used the following config:

~/.aws/config

[default]
region = eu-west-1

[profile Assumable]
role_arn = arn:aws:iam::993655754359:role/Assumable
source_profile = Assumert
region = ap-southeast-1

[profile Assumert]
region = eu-central-1
aws_access_key_id=****
aws_secret_access_key=******

Results

The AWS CLI (v1 and v2)

For better or worse, the AWS CLI defines the Gold Standard of how these files are supposed to behave.

#!/bin/bash
$ rm ~/.aws/cli/cache/*
$ env AWS_CA_BUNDLE=~/.mitmproxy/mitmproxy-ca.pem HTTPS_PROXY=http://localhost:8080/ \
    aws --profile Assumable sts get-caller-identity

Result, 2 calls, both to ap-southeast-1.

Call Region
AssumeRole ap-southeast-1
GetCallerIdentity ap-southeast-1

Conclusion: the AWS CLI v2 uses the region of the target profile to determine in what region to perform the AssumeRole call.

I also tested AWS CLIv1 (including AWS_STS_REGIONAL_ENDPOINTS=regional) and it behaves the same.

The SDKv2

I used the following program to test the behavior of the previous iteration of the SDK:

process.env.AWS_SDK_LOAD_CONFIG = '1';

import { STS } from 'aws-sdk';
import { SharedIniFileCredentials } from 'aws-sdk';
import { ProxyAgent } from 'proxy-agent';
import * as AWS from 'aws-sdk';

async function main() {
  AWS.config.update({
    httpOptions: { agent: new ProxyAgent() }
  });

  // Load credentials from INI file
  const credentials = new SharedIniFileCredentials({ profile: 'Assumable' });

  // Configure STS client with proxy and credentials
  const sts = new STS({
    credentials: credentials,
  });

  const identity = await sts.getCallerIdentity().promise();
  console.log('Caller Identity:', identity);
}

main().catch(e => {
  console.error(e);
  process.exitCode = 1;
});

Command line:

$ env AWS_STS_REGIONAL_ENDPOINTS=regional NODE_TLS_REJECT_UNAUTHORIZED=0 AWS_CA_BUNDLE=~/.mitmproxy/mitmproxy-ca.pem HTTPS_PROXY=http://localhost:8080/ npx tsx sdkv2.ts

Result, 2 calls, one ap-southeast-1 and one to eu-west-1 🙄

Call Region
AssumeRole ap-southeast-1
GetCallerIdentity eu-west-1

Conclusion: the SDK v2 uses the region of the target profile to determine in what region to perform the AssumeRole call. In this case I can't fault it for not doing GetCallerIdentity in the target region either, since the Client object doesn't know the profile, and it can't know the profile since there is no room in the API to pass in the profile name. There's no way for it to pick the right region so it just defaults to the region from the [default] profile.

If I pass the profile with AWS_PROFILE=Assumable as well, it does both calls in ap-southeast-1 as expected.

$ env AWS_PROFILE=Assumable ... npx tsx sdkv2.ts
Call Region
AssumeRole ap-southeast-1
GetCallerIdentity ap-southeast-1

SDK v3

Using the following program:

import { STS } from '@aws-sdk/client-sts';
import { ProxyAgent } from 'proxy-agent';
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';
import { NodeHttpHandler } from '@smithy/node-http-handler';

async function main() {
  // Configure proxy for STS client
  const clientConfig = {
    requestHandler: new NodeHttpHandler({
      httpsAgent: new ProxyAgent()
    })
  };
  const stsClient = new STS({
    credentials: fromNodeProviderChain({ profile: 'Assumable', clientConfig }),
    ...clientConfig,
  });

  const identity = await stsClient.getCallerIdentity();
  console.log('Caller Identity:', identity);
}

main().catch(e => {
  console.error(e);
  process.exitCode = 1;
});

Command line:

env NODE_TLS_REJECT_UNAUTHORIZED=0 AWS_CA_BUNDLE=~/.mitmproxy/mitmproxy-ca.pem HTTPS_PROXY=http://localhost:8080/ npx tsx sdkv3.ts

Results

Call Region
AssumeRole us-east-1
GetCallerIdentity eu-west-1

Conclusion: I guess again I can't fault the SDK for performing the GetCallerIdentity call in eu-west-1, but performing the AssumeRole call in us-east-1 seems wrong. It has enough information to do the right thing (which is being at least compatible with the AWS CLI, if not the SDKv2), but refuses to do so.

With AWS_PROFILE=Assumable being passed:

Call Region
AssumeRole us-east-1
GetCallerIdentity ap-southeast-1

At least the target region is correct, but the AssumeRole region is still wrong!

Conclusion

Disregarding the target region of the GetCallerIdentity call, we can at least say the following about the AssumeRole calls:

Client AssumeRole region
AWS CLI v1 target profile region
AWS CLI v2 target profile region
SDKv2 target profile region
SDKv3 us-east-1

One of these is not like the others... 😬

@rix0rrr
Copy link

rix0rrr commented Dec 10, 2024

For shits and giggles, I decided to give myself an aneurism and tried to pass in different sources of region information that conflict with the profile setting, using command-line flags and environment variables, to see what would happen:

Client Region override AssumeRole region GetCallerIdentity region
CLIv1 (none) target profile region target profile region
CLIv2 (none) target profile region target profile region
SDKv2 (none) target profile region target profile region (* if using $AWS_PROFILE)
SDKv3 (none) us-east-1 target profile region (* if using $AWS_PROFILE)
CLIv1 $AWS_REGION target profile region target profile region
CLIv1 $AWS_DEFAULT_REGION $AWS_DEFAULT_REGION $AWS_DEFAULT_REGION
CLIv2 $AWS_REGION $AWS_REGION $AWS_REGION
SDKv2 $AWS_REGION target profile region $AWS_REGION
SDKv3 $AWS_REGION us-east-1 $AWS_REGION
CLIv1 --region --region --region
CLIv2 --region --region --region

The CLIs will always let environment variables and command line switches override both regions. The CLIv1 ignores AWS_REGION but that's logical, since that variable didn't exist yet when it was created and it uses AWS_DEFAULT_REGION instead.

SDKv2 ignores AWS_REGION for the AssumeRole call, an always uses the region from the profile.

SDKv3 ignores AWS_REGION for the AssumeRole call, and always uses us-east-1.

@github-actions github-actions bot added potential-regression Marking this issue as a potential regression to be checked by team member and removed potential-regression Marking this issue as a potential regression to be checked by team member labels Dec 10, 2024
@zshzbh zshzbh added queued This issues is on the AWS team's backlog and removed investigating Issue is being investigated and/or work is in progress to resolve the issue. labels Dec 10, 2024
@zshzbh
Copy link
Contributor

zshzbh commented Dec 10, 2024

Hey @rix0rrr,

Thanks for the test!

The CLI is a special case as it supports command line options. The command line options in the CLI are roughly equivalent to specifying the region in a service client instantiation, so the region specified in CLI always takes precedence.

As for JS SDK V3 defaults to us-east-1 in AssumeRole, I just talked with AWS JS SDK team, and this issue is queued now for SDE team.

As for JS SDK V2, as it's maintenance mode, we only fix critical bugs and security issues.

Thanks!
Maggie

@kuhe
Copy link
Contributor

kuhe commented Dec 10, 2024

The current search order for inner STS assume role region is:

  1. (CODE) credentials provider clientConfig.region (can only be set by user)
  2. credentials provider parentClientConfig.region (set automatically only if no provider is given by the user)
    a. (CODE) initialization region in code for the outer client
    b. (ENV) environment variable AWS_REGION
    c. (FILE) config file region
  3. default commercial partition's us-east-1 (historical reasons)

For all SDKs, the general configuration precedence is the same: CODE, ENV, FILE, in that order.

// no credentials provider is set, the default credentials provider 
// will prioritize the parent client region
new Client({ region: 'us-west-2' }); 
// STS AssumeRole is called with us-west-2 (priority=2).

The UX problem here is that when you define a custom credentials provider,

import { fromIni } from "@aws-sdk/credential-providers";

const provider = fromIni({ ... });

even though this is an SDK credential provider, it is a standalone function. As such, it is considered a CODE level configuration, so the region should be set on it like so:

fromIni({ profile: "abc", clientConfig: { region: "..." } });

If you were to instead set process.env.AWS_PROFILE=abc and leave code level region and profile undefined, the region would be that specified in the profile for both AssumeRole and GetCallerIdentity.


All that said, the behavior clearly defies the expectations of the caller, so we need to think about what can be safely changed.

Initially, I would consider the following change:

fromIni() credential provider STS AssumeRole region search order

  1. (CODE) credentials provider clientConfig.region (can only be set by user)
  2. credentials provider parentClientConfig.region (set automatically only if no provider is given by the user)
    a. (CODE) initialization region in code for the outer client
    b. (ENV) environment variable AWS_REGION
    c. (FILE) config file region
  3. the region of the profile in the config file
  4. default commercial partition's us-east-1 (historical reasons)

However, this will conflict with another proposed change in which the parent client is linked to the provider even if the provider is defined by the user. We will need more time to think through this.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Dec 11, 2024
@rix0rrr
Copy link

rix0rrr commented Dec 11, 2024

We need to configure things like a proxy agent, do we have to configure in code. Yet I still want the region from the profile if that what is being asked for by the user

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p2 This is a standard priority issue queued This issues is on the AWS team's backlog
Projects
None yet
Development

No branches or pull requests

4 participants