helmv4: Authentication issue when pulling helm charts from private OCI registries.

[jonnylangefeld]

What happened?

helmv4 allows to download helm charts from private OCI registries by specifying username and password:

args := &helmv4.ChartArgs{
	Chart: pulumi.String("oci://myregistry.com/my-chart"),
	RepositoryOpts: helmv4.RepositoryOptsArgs{
		Username: pulumi.String("username"),
		Password: pulumi.String("password"),
	},
}

However, this results in authentication errors despite correct credentials. Different kinds for the different cloud provider registries:

on AWS using ECR

error: an unhandled error occurred: waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials

on GCP using GCR

error: an unhandled error occurred: waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: unexpected status from HEAD request to https://gcr.io/v2/gke-shared-dev/upstream/helm.cilium.io/tetragon/mainfests/1.1.2: 401 Unauthorized

We did some investigation and the following turns out to be the culprit:

When reproducing the issue with the CLI then this works:

helm registry login myregistry.com -u "<username>" -p "<token>"
helm pull oci://myregistry.com/my-chart

but this doesn’t, even though it should:

helm pull oci://myregistry.com/my-chart --username="<username>" --password="<token>"

The Pulumi helm v4 provider does the equivalent to the latter.

This is recognized as an open upstream helm issue: helm/helm#12769

The main issue is the inconsistency. helm pull with username and password without previous helm registry login works for https:// helm registries, but not for oci://.

If this is an upstream helm issue, why is this posted on the pulumi-kubernetes repo?

The main reason I'm posting this here is that the helmv4 Pulumi provider copy/pastes some private upstream functions. For instance the newRegistryClient() function that the upstream helm bugfix PR is attempting to fix here by adding the username and password to the signature, is copy/pasted into this repo because it's a private function:

pulumi-kubernetes/provider/pkg/helm/tool.go

Line 341 in f5f368d

func newRegistryClient(settings *cli.EnvSettings, certFile, keyFile, caFile string, insecureSkipTLSverify, plainHTTP bool) (*registry.Client, error) {

This means that even if the upstream helm PR is merged, the helmv4 Pulumi provider would still suffer from this issue. So it has to be fixed here as well. But we might still need to wait for the upstream fix, for the new upstream NewClient() function.

The upstream helm PR has been open for 7 months with most recent activity 1 month ago. We could help push it along.

Output of pulumi about

╰─ pulumi about
CLI
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Host
OS       darwin
Version  14.6.1
Arch     arm64

Backend
Name           pulumi.com
URL            https://app.pulumi.com/jonny-langefeld
User           jonny-langefeld
Organizations  jonny-langefeld, snowflake
Token type     personal

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[EronWright]

I believe it is a known limitation that Helm OCI support relies on ambient credentials, as set by helm registry login or by docker login. The upstream PR seeks to allow for adhoc credentials via flags, and Pulumi would assumedly incorporate that by wiring up the RepositoryOpts. At this time, the RepositoryOpts is applicable only to HTTP-based chart repositories.

[blampe]

Strictly speaking I don't think we would need to wait for the fix upstream, we could probably plumb the credentials through with what we already have.

Related #2911

[stepan-romankov]

Is it any walk around for this issue @blampe @EronWright? I'm stuck with installation of AWS Gateway AIP Controller chart as it is located only in OCI repository.

[blampe]

@stepan-romankov the AWS registry is public and doesn't need credentials.

chart: "oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart"

[stepan-romankov]

@stepan-romankov the AWS registry is public and doesn't need credentials.

chart: "oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart"

error: an unhandled error occurred: program failed:
waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: unexpected status from HEAD request to https://public.ecr.aws/v2/aws-application-networking-k8s/aws-gateway-controller-chart/manifests/v1.0.7: 403 Forbidden

	gatewayApiController, err := helmv4.NewChart(ctx, "gateway-api-controller", &helmv4.ChartArgs{
		Name:      pulumi.String("gateway-api-controller"),
		Chart:     pulumi.String("oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart"),
		Version:   pulumi.String("v1.0.7"),
		Namespace: namespace.Metadata.Name(),
		Values: pulumi.Map{
			"region":       pulumi.String(region.Name),
			"clusterVpcId": eksCluster.Core.VpcId(),
			"serviceAccount": pulumi.Map{
				"create": pulumi.Bool(false),
			},
			"log": pulumi.Map{
				"level": pulumi.String("debug"),
			},
		},
	}, pulumi.Provider(kubernetesProvider))

[blampe]

@stepan-romankov thanks for including your code. It works successfully for me:

 +   pulumi:pulumi:Stack                                                            p-k-3193-go-dev                                                                      create
 +   └─ kubernetes:helm.sh/v4:Chart                                                 gateway-api-controller                                                               create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:vpcassociationpolicies.application-networking.k8s.aws         create
 +      ├─ kubernetes:core/v1:Service                                               gateway-api-controller:p-k-3193/webhook-service                                      create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:serviceimports.application-networking.k8s.aws                 create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:dnsendpoints.externaldns.k8s.io                               create
 +      ├─ kubernetes:apps/v1:Deployment                                            gateway-api-controller:p-k-3193/gateway-api-controller-aws-gateway-controller-chart  create
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding               gateway-api-controller:gateway-api-controller-aws-gateway-controller-chart           create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:accesslogpolicies.application-networking.k8s.aws              create
 +      ├─ kubernetes:core/v1:Secret                                                gateway-api-controller:p-k-3193/webhook-cert                                         create
 +      ├─ kubernetes:core/v1:ServiceAccount                                        gateway-api-controller:p-k-3193/gateway-api-controller                               create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:serviceexports.application-networking.k8s.aws                 create
 +      ├─ kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration  gateway-api-controller:aws-appnet-gwc-mutating-webhook                               create
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole                      gateway-api-controller:gateway-api-controller-aws-gateway-controller-chart           create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:gatewayclasses.gateway.networking.k8s.io                      create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:iamauthpolicies.application-networking.k8s.aws                create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:targetgrouppolicies.application-networking.k8s.aws            create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:tlsroutes.gateway.networking.k8s.io                           create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:gateways.gateway.networking.k8s.io                            create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:grpcroutes.gateway.networking.k8s.io                          create
 +      └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:httproutes.gateway.networking.k8s.io                          create

As previously mentioned, Pulumi is currently using your ambient helm credentials. This means if you previously logged into public.ecr.aws your credentials are probably expired. I suspect helm registry logout public.ecr.aws or helm registry login will resolve your issue.