[Bug]: provider-aws-s3 - Not able to create AWS S3 bucket after upgrading to v1.16.0

[vilkovtato]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

s3.aws.upbound.io/v1beta1(2) - Bucket

Resource MRs required to reproduce the bug

apiVersion: s3.aws.upbound.io/v1beta2
kind: Bucket
metadata:
  name: bump-provider-test
  namespace: crossplane
spec:
  forProvider:
    region: us-east-1

Steps to Reproduce

• install aws s3 provider:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.16.0

• this will automatically install aws-provider-family version v1.16.0
• apply above mentioned s3 bucket manifest

What happened?

bucket was not created

Relevant Error Output Snippet

observe failed: failed to observe the resource: [{0 reading S3 Bucket
        (bump-provider-test): operation error S3: HeadBucket, https
        response error StatusCode: 403, RequestID: xxx, HostID:xxx,
        api error Forbidden: Forbidden  []}]

Crossplane Version

1.17.1

Provider Version

v1.16.0

Kubernetes Version

1.29

Kubernetes Distribution

EKS

Additional Info

this is not problem of access rights:

• different provider (ec2) is working - which was confirmed by successfully creating AWS security group...
• in order to be sure, the AWS IAM Role used was given full s3 rights...
• the s3: HeadBucket action does not exists in AWS

both v1beta1/v1beta2 has the same problem

[turkenf]

Hi @vilkovtato,

Thank you, I can't reproduce this issue. Is it still valid on your side?

[atrakic]

I have observed simmilar issue within kind+localstack (using provider-aws-s3:v1.17.0).

Steps to reproduce:

1. start kind+localstack
   https://github.com/atrakic/localstack-kind/blob/main/localstack-kind.sh

2. deploy s3 bucket:
   https://github.com/atrakic/localstack-kind/blob/main/scripts/deploy-static-website.sh
   (which would deploy this manifest: https://github.com/atrakic/localstack-kind/blob/main/s3-website-bucket.yaml)

3. After running I get:

fatal error: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist 

• get buckets status block show following:

$ kubectl get buckets -n crossplane-system -o yaml

...
  status:
    atProvider: {}
    conditions:
    - lastTransitionTime: "2024-12-04T11:27:16Z"
      message: 'observe failed: failed to observe the resource: [{0 reading S3 Bucket
        (crossplane-s3-bucket): operation error S3: HeadBucket, https response error
        StatusCode: 0, RequestID: , HostID: , request send failed, Head "https://localhost.localstack.cloud/crossplane-s3-bucket":
        dial tcp: lookup localhost.localstack.cloud on 10.96.0.10:53: no such host  []}]'
      reason: ReconcileError
      status: "False"
      type: Synced
...

Do I need some additional s3 IAM policy or is it something else?
Thx in advance.

[turkenf]

Hi @atrakic and @vilkovtato,

I see a similar issue in this uptest run:

    logger.go:42: 09:55:57 | case/0-apply |     - lastTransitionTime: "2024-12-10T09:55:31Z"
    logger.go:42: 09:55:57 | case/0-apply |       message: 'observe failed: failed to observe the resource: [{0 reading S3 Bucket
    logger.go:42: 09:55:57 | case/0-apply |         (sagemaker-bucketname303): operation error S3: HeadBucket, https response
    logger.go:42: 09:55:57 | case/0-apply |         error StatusCode: 403, RequestID: F67MKV1B1DC65HAP, HostID: BQXJy2erbSSc1JN2l3QhrbQklwnnQ/5gOqgfLopBTXnuLRid8a9MM5TWyYS0Z1Q8HtyOM44uJyU=,
    logger.go:42: 09:55:57 | case/0-apply |         api error Forbidden: Forbidden  []}]'
    logger.go:42: 09:55:57 | case/0-apply |       reason: ReconcileError
    logger.go:42: 09:55:57 | case/0-apply |       status: "False"
    logger.go:42: 09:55:57 | case/0-apply |       type: Synced

At first glance, this issue seems like an authorization/permission error to me. I don't encounter any issues when I try to create a single bucket without any dependencies. Is this the same for you?

Also, you stated in the issue report that the error occurred with provider version v1.16.0. Can you confirm that the same resource configuration worked before v1.16.0?

[vilkovtato]

Hi @turkenf,

in my case, this cannot be authorization/permission error. As I wrote in description "in order to be sure, the AWS IAM Role used was given full s3 rights..." and "different provider (ec2) is working - which was confirmed by successfully creating AWS security group...", so the problem is connected only to s3 provider.

[turkenf]

in my case, this cannot be authorization/permission error. As I wrote in description "in order to be sure, the AWS IAM Role used was given full s3 rights..." and "different provider (ec2) is working - which was confirmed by successfully creating AWS security group...", so the problem is connected only to s3 provider.

Thank you for the quick response @vilkovtato. Does the same scenario work in provider v1.15.0?