Using a k8s CRD fails due to ArgoCD App k8s service still not ready even though the ArgoCD App is marked as healthy/synced and the helm chart has parameter wait

[cooervo]

What happened?

Getting error when running pulumi update:

 Preview failed: resource "urn:pulumi:mc-predev::foo-infra::kubernetes:external-secrets.io/v1alpha1:ClusterSecretStore::
dev-cluster-secret-store" was not successfully created by the Kubernetes API server: conversion webhook for external-secrets.io/v1alpha1,
 Kind=ClusterSecretStore failed: Post "https://external-secrets-webhook.external-secrets.svc:443/convert?timeout=30s": 
service "external-secrets-webhook" not found

Example

I'm getting the above error when trying to install in same pulumi update the following 2 resources:

// The ArgoCD app installing the helm chart for external-secrets
 const externalSecretsApp = new Application.argoproj.v1alpha1.Application(
    `${env}-external-secrets-app`,
    {
      name: 'external-secrets',
      namespace: 'external-secrets',
      sources: [
        {
          path: 'kubernetes/helm/charts/external-secrets/',
          helm: {
            parameters: [
              {
                name: 'wait',
                value: 'true',
              },
            ],
            valueFiles: [`../../values/external-secrets/${env}.values.yaml`],
            ...
          repoURL: INFRA_REPO_URL,
          targetRevision: env,
        },
      ],
    ...
    },
    {
      provider,
      dependsOn: [...],
    },
  );

// The CRD I previously got from using crd2pulumi --nodejsPath ClusterSecretStore ClusterSecretStore.yaml    
  const clusterSecretStore =
    new ClusterSecretStore.external_secrets.v1alpha1.ClusterSecretStore(
      `${env}-cluster-secret-store`,
      {
        metadata: {
          name: 'gcp-cluster-secret-store',
        },
        spec: {
          provider: {
            // gcpsm = GCP  Secret Manager
            gcpsm: {
              projectID: GCP_PROJECT,
            },
          },
        },
      },
      { dependsOn: [...], provider },
    );

Even though first resource, externalSecretsApp, is dependency in dependsOn of second resource, clusterSecretStore, I still get the error:

 Preview failed: resource "urn:pulumi:mc-predev::foo-infra::kubernetes:external-secrets.io/v1alpha1:ClusterSecretStore::
dev-cluster-secret-store" was not successfully created by the Kubernetes API server: conversion webhook for external-secrets.io/v1alpha1, 
Kind=ClusterSecretStore failed: Post "https://external-secrets-webhook.external-secrets.svc:443/convert?timeout=30s": 
service "external-secrets-webhook" not found

Output of pulumi about

pulumi about
CLI          
Version      3.139.0
Go Version   go1.23.3
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  command     1.0.1
resource  gcp         8.0.0
resource  kubernetes  4.18.1
language  nodejs      unknown
resource  random      4.16.7
resource  std         1.7.3

Host     
OS       darwin
Version  14.6.1
Arch     arm64

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[cooervo]

I also tried wrapping the CRD in a custom resource but still get same error:

const clusterSecretStore = new k8s.apiextensions.CustomResource(
    `${env}-cluster-secret-store`,
    {
      apiVersion: 'external-secrets.io/v1alpha1',
      kind: 'ClusterSecretStore',
      metadata: {
        name: 'gcp-cluster-secret-store',
      },
      spec: {
        provider: {
          // gcpsm = GCP  Secret Manager
          gcpsm: {
            projectID: GCP_PROJECT,
          },
        },
      },
    },
    { dependsOn: [externalSecretsApp], provider },
  );

[EronWright]

I would guess that the issue is with the external-secrets apps not being fully ready, because Pulumi doesn't know how to check for readiness on ArgoCD's Application resource. Please take a look at this article that shows how to use the new waitFor annotation, and please try it with the Application resource.

[cooervo]

@EronWright tried using xTransform but all I get is the Application type

    `${env}-external-secrets-app`,
        ...
    {
      provider,
      dependsOn: [...],
      xTransforms: [
        (args) => {
          console.log('args.type', args.type); 
          return undefined;
        },
      ],
    },
  );

Output:

    args.type kubernetes:argoproj.io/v1alpha1:Application

Also tried using the waitfor as an annotation so the ArgoCD App wait until all pods are ready but also fails:

    + kubernetes:argoproj.io/v1alpha1:Application: (create)
        [urn=urn:pulumi:mc-predev::interval-infra::kubernetes:argoproj.io/v1alpha1:Application::mc-predev-external-secrets-app]
        [provider=urn:pulumi:mc-predev::interval-infra::pulumi:providers:kubernetes::mc-predev-k8s-main-provider::7bebb8e7-aaf3-458c-a489-1ed14d82769a]
        apiVersion: "argoproj.io/v1alpha1"
        kind      : "Application"
        metadata  : {
            annotations: {
                pulumi.com/waitFor: "condition=Ready pods --all -n external-secrets --timeout=30s"
            }
            name       : "external-secrets"
            namespace  : "argocd"
        }

seems like this waitFor transform won't work when the resources are wrapped in an abstraction such as ArgoCD not sure if it would even work with a Helm Chart/Release wrapping the resources.

[blampe]

@cooervo please try

pulumi.com/waitFor: "condition=Ready"

The syntax is not a command to run, but rather a specific condition to observe on the object. The code you posted is waiting for a status condition literally named "Ready pods --all..." which doesn't exist.