Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws s3 cp exit code 255: botocore.awsrequest.AWSRequest object at #8988

Open
1 task
fishcharlie opened this issue Oct 18, 2024 · 8 comments
Open
1 task

aws s3 cp exit code 255: botocore.awsrequest.AWSRequest object at #8988

fishcharlie opened this issue Oct 18, 2024 · 8 comments
Assignees
Labels
bug This issue is a bug. p3 This is a minor priority issue s3

Comments

@fishcharlie
Copy link

Describe the bug

When running aws s3 cp code.zip s3://my-bucket/folder/code.zip, it exits with code 255 and prints the following:

<botocore.awsrequest.AWSRequest object at 0x7f367a0c9af0>

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

It to successfully upload the file.

Current Behavior

Fails with exit code 255 and prints:

<botocore.awsrequest.AWSRequest object at 0x7f367a0c9af0>

Reproduction Steps

Honestly I have no idea what is even going wrong. aws cli doesn't even print a good error messages. So I have no idea how to even reproduce this or what is even going wrong...

Would be more than happy to help debug further but need some guidance here since the error message is so poor.

Possible Solution

No response

Additional Information/Context

runner@fv-az1496-34:~/work/EchoServer/EchoServer$ ls -al
total 613764
drwxr-xr-x  10 runner docker      4096 Oct 18 04:06 .
drwxr-xr-x   3 runner docker      4096 Oct 18 04:05 ..
-rw-r--r--   1 runner docker 628228556 Oct 18 04:06 code.zip


runner@fv-az1496-34:~/work/Server/Server$ aws --version
aws-cli/2.18.0 Python/3.12.6 Linux/6.5.0-1025-azure exe/x86_64.ubuntu.22

CLI version used

2.18.0

Environment details (OS name and version, etc.)

Python/3.12.6 Linux/6.5.0-1025-azure exe/x86_64.ubuntu.22

@fishcharlie fishcharlie added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 18, 2024
@adev-code adev-code self-assigned this Oct 24, 2024
@adev-code adev-code added investigating This issue is being investigated and/or work is in progress to resolve the issue. s3 p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Oct 24, 2024
@adev-code
Copy link

Hello @fishcharlie, thanks for reaching out. I have replicated the command you ran and did not get the same error. For a further look, please include the full debug logs by adding --debug to your command and redacting any sensitive information. Thanks!

@adev-code adev-code added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Oct 25, 2024
Copy link

github-actions bot commented Nov 4, 2024

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

@github-actions github-actions bot added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Nov 4, 2024
@HajraRafiq28
Copy link

@fishcharlie Try to add a region while configuring the credential. It will work

@github-actions github-actions bot removed closing-soon This issue will automatically close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Nov 8, 2024
@adev-code adev-code added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 8, 2024
@dmartin-isp
Copy link

dmartin-isp commented Nov 9, 2024

Even with a bad command, the output shouldn't be <botocore.awsrequest.AWSRequest object at 0x7f367a0c9af0> but some kind of error message. I've got another aws cli command that I'm trying to use and am getting a similarly unhelpful output.

@nateprewitt
Copy link
Member

Hi @dmartin-isp, we definitely agree that's not intended output. We'll need debug logs as requested above, or a simple repro to investigate further though.

I don't believe anyone internally has been able to hit this case yet.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 9, 2024
@adev-code adev-code added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 11, 2024
@nickdbush
Copy link

nickdbush commented Nov 13, 2024

Managed to reproduce this consistently in our CI environment only (GHA). Logs: https://gist.github.com/nickdbush/36efe6752d78cdd261a07fec4e39da2a

We're mocking S3 with adobe/S3Mock. Development environment is macOS and GHA is Linux. Without --debug, we get the output @fishcharlie is seeing

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 13, 2024
@nateprewitt
Copy link
Member

Thanks for the logs, @nickdbush! We found where the erroneous message is leaking in for the CLI, there's some missing error handling for region detection through IMDS. We can improve the message being returned, but the actual failure is coming from an IMDS response on your host. Is that also being virtualized/mocked?

The request that's failing is a very simple PUT with one header to get an IMDS v2 token (more info) for added security. The IMDS instance you're contacting is returning a 400 which may be that it doesn't support this API (if third-party) or something else unexpected is happening.

2024-11-13 14:49:50,723 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-11-13 14:49:50,725 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2024-11-13 14:49:50,747 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 400 322

We'll start looking at fixing the output. While we're doing that, any more info on if anyone encountering this is using a third-party IMDS emulator? If not, could you provide more info on your specific setup. We wouldn't anticipate this happening in a standard ECS/EC2 setup.

@nateprewitt nateprewitt added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 13, 2024
@nickdbush
Copy link

nickdbush commented Nov 13, 2024

@nateprewitt yeah we were missing a region flag for the CLI on our end, which ended up unblocking us! no other services mocked except S3, and given that it was only happening in Actions, it is a bit of a puzzle of what may have been responding with that 400. cheers :^)

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p3 This is a minor priority issue s3
Projects
None yet
Development

No branches or pull requests

6 participants