Install fails on eks 1.21

[mestuddtc]

I tried creating jenkins x cluster by:

1. templating this repo, setting version to latest (1.17.2), and setting required variables (region, cluster_version = 1.21, jx_git_url, jx_bot_username, jx_bot_token)
2. templating cluster repo from https://github.com/jx3-gitops-repositories/jx3-eks-vault and making no changes
3. running terraform apply

Secrets appear to never be created and various pods never start (similar to #16)

If I destroy the deployment, delete all state files, delete and re-template the cluster repo, change the cluster_version to 1.20, and run terraform apply again, jenkins x eventually comes up with everything running. (In contrast to report in #22. Deleting the cluster repo is required, as changes made by jx seem to prevent a new deploy from working.)

jx admin log for 1.21:

jx gitops git setup
setup git user  email [email protected]
generated Git credentials file: /workspace/xdg_config/git/credentials with username: jenkins-recognia-com email:
jx gitops apply
found last commit message: Initial commit
make regen-phase-1
make[1]: Entering directory '/workspace/source'
jx gitops git setup
setup git user  email [email protected]
generated Git credentials file: /workspace/xdg_config/git/credentials with username: jenkins-recognia-com email:
Already up to date.
jx gitops requirements merge
wrote the ConfigMap jx-requirements.yml to /tmp/671956249
WARNING: validation failures in YAML file /tmp/671956249: (root): Additional property gitops is not allowed, (root): Additional property kaniko is not allowed, (root): Additional property versionStream is not allowed
saved file /workspace/source/jx-requirements.yml
jx gitops requirements resolve -n
modified the chart repository in /workspace/source/jx-requirements.yml
WARNING: no email in secret jx-boot in namespace jx-git-operator
modified the pipeline user in /workspace/source/jx-requirements.yml
no resolve logic for kubernetes provider eks
jx gitops repository create
file does not exist: .jx/gitops/source-config.yaml so not defaulting any SourceConfig resources
jx gitops repository resolve --source-dir config-root/namespaces
jx gitops jenkins jobs
the source config file .jx/gitops/source-config.yaml does not exist
jx secret convert edit
jx gitops helmfile resolve
jx gitops image -s .jx/git-operator
helmfile --file helmfile.yaml template --validate --include-crds --output-dir-template /tmp/generate/{{.Release.Namespace}}/{{.Release.Name}}
Adding repo jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories

Adding repo bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

Adding repo stable https://charts.helm.sh/stable
"stable" has been added to your repositories

Adding repo jxgh https://jenkins-x-charts.github.io/repo
"jxgh" has been added to your repositories

Templating release=jxboot-helmfile-resources, chart=jxgh/jxboot-helmfile-resources
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-serviceaccount.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-sa.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-sa.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-basic-auth-htpasswd-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-basic-auth-user-password-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/maven-settings-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/tekton-container-registry-auth-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/tekton-git-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/ingress-config-configmap.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jenkins-x-docker-registry.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jenkins-x-extensions-config.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/kapp-config.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-clusterrole.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jenkins-x-aggregate-view.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-clusterrolebinding.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/committer-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-pipeline-activity-reporter-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-view-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/owner-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/viewer-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-rolebinding.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-rb.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-rb.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-cronjob.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-cronjob.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/700-chartmuseum-ing.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/700-hook-ing.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/700-nexus-ing.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/environments.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/environments.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/environments.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/repositories.yaml


Templating release=jenkins-x-crds, chart=jxgh/jenkins-x-crds
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/environments-crd.yaml
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/pipelineactivities-crd.yaml
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/releases-crd.yaml
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/sourcerepositories-crd.yaml


Templating release=jx-pipelines-visualizer, chart=jxgh/jx-pipelines-visualizer
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/rbac.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/rbac.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/rbac.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/service.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/deployment.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/ingress.yaml


Templating release=jx-preview, chart=jxgh/jx-preview
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-sa.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/previews.preview.jenkins.io.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-clusterrole.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-clusterrolebinding.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-role.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-rb.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-cj.yaml


Templating release=lighthouse, chart=jxgh/lighthouse
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/hmacsecret.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/oauthsecret.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/config-external-plugins.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/lighthousebreakpoints-crd.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/lighthousejobs-crd.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-service.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-service.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-service.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-cj.yaml


Templating release=nexus, chart=jxgh/nexus
wrote /tmp/generate/jx/nexus/nexus/templates/secret.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/config.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/pvc.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/service.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/deployment.yaml


Templating release=chartmuseum, chart=stable/chartmuseum
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/secret.yaml
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/pvc.yaml
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/service.yaml
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/deployment.yaml


Templating release=jx-build-controller, chart=jxgh/jx-build-controller
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/serviceaccount.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/clusterrole.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/clusterrolebinding.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/role.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/rolebinding.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/deployment.yaml


Templating release=health-checks-jx, chart=jxgh/jx-kh-check
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-bot-token.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml


Adding repo ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories

Templating release=nginx-ingress, chart=ingress-nginx/ingress-nginx
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-poddisruptionbudget.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-serviceaccount.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-configmap.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/clusterrole.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/clusterrolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-role.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-rolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-service-metrics.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-service-webhook.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-service.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-deployment.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml


Adding repo external-secrets https://external-secrets.github.io/kubernetes-external-secrets
"external-secrets" has been added to your repositories

Adding repo banzaicloud-stable https://kubernetes-charts.banzaicloud.com
"banzaicloud-stable" has been added to your repositories

Templating release=kubernetes-external-secrets, chart=external-secrets/kubernetes-external-secrets
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/crds/kubernetes-client.io_externalsecrets_crd.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/serviceaccount.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/rbac.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/rbac.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/rbac.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/service.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/deployment.yaml


Templating release=pusher-wave, chart=jxgh/pusher-wave
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/serviceaccount.yaml
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/clusterrole.yaml
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/clusterrolebinding.yaml
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/deployment.yaml


Adding repo cdf https://cdfoundation.github.io/tekton-helm-chart
"cdf" has been added to your repositories

Templating release=tekton-pipeline, chart=cdf/tekton-pipeline
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-ns.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-podsecuritypolicy.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-bot-sa.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-sa.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-sa.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/webhook-certs-secret.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-artifact-bucket-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-artifact-pvc-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-defaults-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-leader-election-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-logging-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-observability-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-registry-cert-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/feature-flags-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelines-info-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/clustertasks.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/conditions.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelineresources.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelineruns.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelines.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/runs.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/taskruns.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tasks.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-aggregate-edit-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-aggregate-view-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-cluster-access-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-tenant-access-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-cluster-access-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-cluster-access-crb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-tenant-access-crb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-cluster-access-crb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-info-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-leader-election-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-leaderelection-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-info-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-leaderelection-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-svc.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-svc.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-deploy.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-deploy.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-horizontalpodautoscaler.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/webhook.pipeline.tekton.dev-mutwebhookcfg.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config.webhook.pipeline.tekton.dev-valwebhookcfg.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/validation.webhook.pipeline.tekton.dev-valwebhookcfg.yaml


jx gitops split --dir /tmp/generate
jx gitops rename --dir /tmp/generate
jx gitops helmfile move --output-dir config-root --dir /tmp/generate --dir-includes-release-name
jx secret convert --source-dir config-root -r jx-vault
jx secret replicate --selector secret.jenkins-x.io/replica-source=true
jx gitops namespace --dir-mode --dir config-root/namespaces
jx gitops helmfile report
namespace cert-manager
ignoring release jetstack/cert-manager in namespace cert-manager as using conditional jxRequirementsIngressTLS.enabled

namespace cert-manager
ignoring release jetstack/cert-manager in namespace cert-manager as using conditional jxRequirementsIngressTLS.enabled

namespace jx
ignoring release bitnami/external-dns in namespace jx as using conditional jxRequirementsIngressExternalDNS.enabled
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/chartmuseum-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/hook-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/nexus-ingress.yaml with api version networking.k8s.io/v1
found jxboot-helmfile-resources version: 1.1.0 https://github.com/jenkins-x-charts/jxboot-helmfile-resources
WARNING: ignoring release jxgh/jenkins-x-crds in namespace jx as we cannot find any generated resources but there is no conditional
ignoring Ingress in file config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-ingress.yaml with api version networking.k8s.io/v1
found jx-pipelines-visualizer version: 1.7.5 https://github.com/jenkins-x/jx-pipelines-visualizer
found jx-preview version: 0.0.192 https://github.com/jenkins-x-plugins/jx-preview
found lighthouse version: 1.1.52 https://github.com/jenkins-x/lighthouse
found nexus version: 0.1.41 https://github.com/jenkins-x-charts/nexus
found chartmuseum version: 2.14.2 https://github.com/helm/chartmuseum
found jx-build-controller version: 0.3.9 https://jenkins-x.io/
ignoring release jxgh/acme in namespace jx as using conditional jxRequirementsIngressTLS.enabled
found jx-kh-check version: 0.0.78 https://jenkins-x.io/

namespace jx
ignoring release bitnami/external-dns in namespace jx as using conditional jxRequirementsIngressExternalDNS.enabled
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/chartmuseum-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/hook-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/nexus-ingress.yaml with api version networking.k8s.io/v1
found jxboot-helmfile-resources version: 1.1.0 https://github.com/jenkins-x-charts/jxboot-helmfile-resources
WARNING: ignoring release jxgh/jenkins-x-crds in namespace jx as we cannot find any generated resources but there is no conditional
ignoring Ingress in file config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-ingress.yaml with api version networking.k8s.io/v1
found jx-pipelines-visualizer version: 1.7.5 https://github.com/jenkins-x/jx-pipelines-visualizer
found jx-preview version: 0.0.192 https://github.com/jenkins-x-plugins/jx-preview
found lighthouse version: 1.1.52 https://github.com/jenkins-x/lighthouse
found nexus version: 0.1.41 https://github.com/jenkins-x-charts/nexus
found chartmuseum version: 2.14.2 https://github.com/helm/chartmuseum
found jx-build-controller version: 0.3.9 https://jenkins-x.io/
ignoring release jxgh/acme in namespace jx as using conditional jxRequirementsIngressTLS.enabled
found jx-kh-check version: 0.0.78 https://jenkins-x.io/

namespace kuberhealthy
ignoring release jxgh/kh-tls-check in namespace kuberhealthy as using conditional jxRequirementsTLSCheck.enabled

namespace kuberhealthy
ignoring release jxgh/kh-tls-check in namespace kuberhealthy as using conditional jxRequirementsTLSCheck.enabled

namespace nginx
found ingress-nginx version: 3.12.0 https://github.com/kubernetes/ingress-nginx

namespace nginx
found ingress-nginx version: 3.12.0 https://github.com/kubernetes/ingress-nginx

namespace secret-infra
found kubernetes-external-secrets version: 8.3.0 https://github.com/external-secrets/kubernetes-external-secrets
found pusher-wave version: 0.4.21 https://github.com/pusher/wave

namespace secret-infra
found kubernetes-external-secrets version: 8.3.0 https://github.com/external-secrets/kubernetes-external-secrets
found pusher-wave version: 0.4.21 https://github.com/pusher/wave

namespace tekton-pipelines
found tekton-pipeline version: 0.27.2 https://github.com/cdfoundation/tekton-helm-chart

namespace tekton-pipelines
found tekton-pipeline version: 0.27.2 https://github.com/cdfoundation/tekton-helm-chart

saved docs/releases.yaml
saved docs/README.md
jx gitops scheduler
jx gitops hash --pod-spec --kind Deployment -s config-root/namespaces/jx/lighthouse-config/config-cm.yaml -s config-root/namespaces/jx/lighthouse-config/plugins-cm.yaml -d config-root/namespaces/jx/lighthouse
jx gitops label --dir config-root/cluster                   gitops.jenkins-x.io/pipeline=cluster
jx gitops label --dir config-root/customresourcedefinitions gitops.jenkins-x.io/pipeline=customresourcedefinitions
jx gitops label --dir config-root/namespaces                gitops.jenkins-x.io/pipeline=namespaces
jx gitops annotate --dir config-root --selector app=pusher-wave kapp.k14s.io/change-group=apps.jenkins-x.io/pusher-wave
jx gitops annotate --dir config-root --selector app.kubernetes.io/name=ingress-nginx kapp.k14s.io/change-group=apps.jenkins-x.io/ingress-nginx
jx gitops label --dir config-root/cluster --kind=Namespace team=jx
jx gitops annotate --dir  config-root/namespaces --kind Deployment --selector app=pusher-wave --invert-selector wave.pusher.com/update-on-config-change=true
using kubectl to apply resources
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=customresourcedefinitions -R -f config-root/customresourcedefinitions
customresourcedefinition.apiextensions.k8s.io/environments.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/pipelineactivities.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/releases.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/sourcerepositories.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/previews.preview.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/lighthousebreakpoints.lighthouse.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/lighthousejobs.lighthouse.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/externalsecrets.kubernetes-client.io created
customresourcedefinition.apiextensions.k8s.io/clustertasks.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/conditions.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/pipelineresources.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/pipelineruns.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/pipelines.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/runs.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/taskruns.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/tasks.tekton.dev created
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=cluster                   -R -f config-root/cluster
namespace/jx-production created
namespace/jx-staging created
namespace/jx created
namespace/nginx created
namespace/secret-infra created
clusterrole.rbac.authorization.k8s.io/jx-build-controller-jx created
clusterrolebinding.rbac.authorization.k8s.io/jx-build-controller-jx created
clusterrole.rbac.authorization.k8s.io/jx-pipelines-visualizer created
clusterrolebinding.rbac.authorization.k8s.io/jx-pipelines-visualizer created
clusterrole.rbac.authorization.k8s.io/jx-preview-gc-jobs created
clusterrolebinding.rbac.authorization.k8s.io/jx-preview-gc-jobs created
clusterrole.rbac.authorization.k8s.io/jenkinsx-aggregate-view created
clusterrole.rbac.authorization.k8s.io/tekton-bot created
clusterrolebinding.rbac.authorization.k8s.io/tekton-bot-jx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-external-secrets-auth created
clusterrole.rbac.authorization.k8s.io/kubernetes-external-secrets created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-external-secrets created
clusterrole.rbac.authorization.k8s.io/pusher-wave-pusher-wave created
clusterrolebinding.rbac.authorization.k8s.io/pusher-wave-pusher-wave created
clusterrole.rbac.authorization.k8s.io/tekton-aggregate-edit created
clusterrole.rbac.authorization.k8s.io/tekton-aggregate-view created
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-controller-cluster-access created
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-cluster-access created
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-controller-tenant-access created
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-tenant-access created
namespace/tekton-pipelines created
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-webhook-cluster-access created
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook-cluster-access created
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=namespaces                -R -f config-root/namespaces
deployment.apps/jenkins-x-chartmuseum created
persistentvolumeclaim/jenkins-x-chartmuseum created
externalsecret.kubernetes-client.io/jenkins-x-chartmuseum created
service/jenkins-x-chartmuseum created
serviceaccount/jenkins-x-controllerbuild created
deployment.apps/jx-build-controller created
rolebinding.rbac.authorization.k8s.io/jx-build-controller created
role.rbac.authorization.k8s.io/jx-build-controller created
kuberhealthycheck.comcast.github.io/jx-bot-token created
rolebinding.rbac.authorization.k8s.io/jx-webhook-check-rb created
rolebinding.rbac.authorization.k8s.io/jx-webhook-events-check-rb created
kuberhealthycheck.comcast.github.io/jx-webhook-events created
serviceaccount/jx-webhook-events-sa created
role.rbac.authorization.k8s.io/jx-webhook-events-service-role created
kuberhealthycheck.comcast.github.io/jx-webhook created
serviceaccount/jx-webhook-sa created
role.rbac.authorization.k8s.io/jx-webhook-service-role created
deployment.apps/jx-pipelines-visualizer created
ingress.networking.k8s.io/jx-pipelines-visualizer created
serviceaccount/jx-pipelines-visualizer created
service/jx-pipelines-visualizer created
cronjob.batch/jx-preview-gc-jobs created
rolebinding.rbac.authorization.k8s.io/jx-preview-gc-jobs created
role.rbac.authorization.k8s.io/jx-preview-gc-jobs created
serviceaccount/jx-preview-gc-jobs created
ingress.networking.k8s.io/chartmuseum created
role.rbac.authorization.k8s.io/committer created
environment.jenkins.io/dev created
sourcerepository.jenkins.io/dev created
rolebinding.rbac.authorization.k8s.io/gcactivities created
role.rbac.authorization.k8s.io/gcactivities created
rolebinding.rbac.authorization.k8s.io/gcpods created
role.rbac.authorization.k8s.io/gcpods created
ingress.networking.k8s.io/hook created
configmap/ingress-config created
externalsecret.kubernetes-client.io/jenkins-maven-settings created
configmap/jenkins-x-docker-registry created
configmap/jenkins-x-extensions created
externalsecret.kubernetes-client.io/jx-basic-auth-htpasswd created
externalsecret.kubernetes-client.io/jx-basic-auth-user-password created
cronjob.batch/jx-gcactivities created
serviceaccount/jx-gcactivities created
cronjob.batch/jx-gcpods created
serviceaccount/jx-gcpods created
role.rbac.authorization.k8s.io/jx-pipeline-activity-updater created
role.rbac.authorization.k8s.io/jx-view created
configmap/kapp-config created
ingress.networking.k8s.io/nexus created
role.rbac.authorization.k8s.io/owner created
environment.jenkins.io/production created
environment.jenkins.io/staging created
rolebinding.rbac.authorization.k8s.io/tekton-bot created
role.rbac.authorization.k8s.io/tekton-bot created
serviceaccount/tekton-bot created
externalsecret.kubernetes-client.io/tekton-container-registry-auth created
externalsecret.kubernetes-client.io/tekton-git created
role.rbac.authorization.k8s.io/viewer created
service/hook created
configmap/lighthouse-external-plugins created
deployment.apps/lighthouse-foghorn created
rolebinding.rbac.authorization.k8s.io/lighthouse-foghorn created
role.rbac.authorization.k8s.io/lighthouse-foghorn created
serviceaccount/lighthouse-foghorn created
cronjob.batch/lighthouse-gc-jobs created
rolebinding.rbac.authorization.k8s.io/lighthouse-gc-jobs created
role.rbac.authorization.k8s.io/lighthouse-gc-jobs created
serviceaccount/lighthouse-gc-jobs created
externalsecret.kubernetes-client.io/lighthouse-hmac-token created
deployment.apps/lighthouse-keeper created
rolebinding.rbac.authorization.k8s.io/lighthouse-keeper created
role.rbac.authorization.k8s.io/lighthouse-keeper created
serviceaccount/lighthouse-keeper created
service/lighthouse-keeper created
externalsecret.kubernetes-client.io/lighthouse-oauth-token created
deployment.apps/lighthouse-tekton-controller created
rolebinding.rbac.authorization.k8s.io/lighthouse-tekton-controller created
role.rbac.authorization.k8s.io/lighthouse-tekton-controller created
serviceaccount/lighthouse-tekton-controller created
service/lighthouse-tekton-controller created
deployment.apps/lighthouse-webhooks created
rolebinding.rbac.authorization.k8s.io/lighthouse-webhooks created
role.rbac.authorization.k8s.io/lighthouse-webhooks created
serviceaccount/lighthouse-webhooks created
configmap/config created
configmap/jx-install-config created
configmap/plugins created
configmap/nexus created
deployment.apps/nexus-nexus created
persistentvolumeclaim/nexus-nexus created
externalsecret.kubernetes-client.io/nexus created
service/nexus created
externalsecret.kubernetes-client.io/tekton-container-registry-auth created
externalsecret.kubernetes-client.io/tekton-container-registry-auth created
service/ingress-nginx-controller-admission created
configmap/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
service/ingress-nginx-controller-metrics created
poddisruptionbudget.policy/ingress-nginx-controller created
service/ingress-nginx-controller created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
serviceaccount/ingress-nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
deployment.apps/kubernetes-external-secrets created
serviceaccount/kubernetes-external-secrets created
service/kubernetes-external-secrets created
deployment.apps/pusher-wave-pusher-wave created
serviceaccount/pusher-wave-pusher-wave created
configmap/config-artifact-bucket created
configmap/config-artifact-pvc created
configmap/config-defaults created
configmap/config-leader-election created
configmap/config-logging created
configmap/config-observability created
configmap/config-registry-cert created
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.pipeline.tekton.dev created
configmap/feature-flags created
configmap/pipelines-info created
serviceaccount/tekton-bot created
deployment.apps/tekton-pipelines-controller created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-leaderelection created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller created
role.rbac.authorization.k8s.io/tekton-pipelines-controller created
serviceaccount/tekton-pipelines-controller created
service/tekton-pipelines-controller created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-info created
role.rbac.authorization.k8s.io/tekton-pipelines-info created
role.rbac.authorization.k8s.io/tekton-pipelines-leader-election created
podsecuritypolicy.policy/tekton-pipelines created
deployment.apps/tekton-pipelines-webhook created
horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook-leaderelection created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook created
role.rbac.authorization.k8s.io/tekton-pipelines-webhook created
serviceaccount/tekton-pipelines-webhook created
service/tekton-pipelines-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.pipeline.tekton.dev created
secret/webhook-certs created
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.pipeline.tekton.dev created
jx gitops postprocess
there is no post processing Secret jx-post-process in namespace default so not performing any additional post processing steps
jx verify ingress --ingress-service ingress-nginx-controller
Waiting to find the external host name of the ingress controller Service in namespace nginx with name ingress-nginx-controller
The Ingress address ac590dd0ede7c4936a47425532479724-f436271ef857f6f8.elb.us-east-1.amazonaws.com is not an IP address. We recommend we try resolve it to a public IP address and use that for the domain to access services externally.
Waiting for ac590dd0ede7c4936a47425532479724-f436271ef857f6f8.elb.us-east-1.amazonaws.com to be resolvable to an IP address...
WARNING: retrying after error: address cannot be resolved yet ac590dd0ede7c4936a47425532479724-f436271ef857f6f8.elb.us-east-1.amazonaws.com
.
.
.
.
.
.
.
.
.
.
.
.
.

ac590dd0ede7c4936a47425532479724-f436271ef857f6f8.elb.us-east-1.amazonaws.com resolved to IP 35.170.250.137
No domain flag provided so using default 35.170.250.137.nip.io to generate Ingress rules
defaulting the domain to 35.170.250.137.nip.io and modified /workspace/source/jx-requirements.yml
now verifying docker registry ingress setup
jx gitops git setup
found git user.name jenkins-recognia-com from requirements
found git user.email  from requirements
setup git user  email [email protected]
generated Git credentials file: /workspace/xdg_config/git/credentials with username: jenkins-recognia-com email:
git add --all
git commit -m "chore: regenerated" -m "/pipeline cancel"
[main e214e20] chore: regenerated
 203 files changed, 12529 insertions(+), 108 deletions(-)
 rewrite .jx/secret/mapping/secret-mappings.yaml (61%)
 create mode 100644 config-root/cluster/namespaces/jx-production.yaml
 create mode 100644 config-root/cluster/namespaces/jx-staging.yaml
 create mode 100644 config-root/cluster/namespaces/jx.yaml
 create mode 100644 config-root/cluster/namespaces/nginx.yaml
 create mode 100644 config-root/cluster/namespaces/secret-infra.yaml
 create mode 100644 config-root/cluster/resources/jx/jx-build-controller/jx-build-controller-jx-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/jx/jx-build-controller/jx-build-controller-jx-crb.yaml
 create mode 100644 config-root/cluster/resources/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-crb.yaml
 create mode 100644 config-root/cluster/resources/jx/jx-preview/jx-preview-gc-jobs-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/jx/jx-preview/jx-preview-gc-jobs-crb.yaml
 create mode 100644 config-root/cluster/resources/jx/jxboot-helmfile-resources/jenkinsx-aggregate-view-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/jx/jxboot-helmfile-resources/tekton-bot-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/jx/jxboot-helmfile-resources/tekton-bot-jx-crb.yaml
 create mode 100644 config-root/cluster/resources/nginx/ingress-nginx-nginx-ingress/ingress-nginx-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/nginx/ingress-nginx-nginx-ingress/ingress-nginx-crb.yaml
 create mode 100644 config-root/cluster/resources/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-crb.yaml
 create mode 100644 config-root/cluster/resources/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets-auth-crb.yaml
 create mode 100644 config-root/cluster/resources/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets-crb.yaml
 create mode 100644 config-root/cluster/resources/secret-infra/pusher-wave/pusher-wave-pusher-wave-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/secret-infra/pusher-wave/pusher-wave-pusher-wave-crb.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-aggregate-edit-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-aggregate-view-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-cluster-access-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-cluster-access-crb.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-tenant-access-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-tenant-access-crb.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-ns.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-cluster-access-clusterrole.yaml
 create mode 100644 config-root/cluster/resources/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-cluster-access-crb.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/jenkins-x-crds/environments.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/jenkins-x-crds/pipelineactivities.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/jenkins-x-crds/releases.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/jenkins-x-crds/sourcerepositories.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/jx-preview/previews.preview.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/lighthouse/lighthousebreakpoints.lighthouse.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/jx/lighthouse/lighthousejobs.lighthouse.jenkins.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/secret-infra/kubernetes-external-secrets/externalsecrets.kubernetes-client.io-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/clustertasks.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/conditions.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/pipelineresources.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/pipelineruns.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/pipelines.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/runs.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/taskruns.tekton.dev-crd.yaml
 create mode 100644 config-root/customresourcedefinitions/tekton-pipelines/tekton-pipeline/tasks.tekton.dev-crd.yaml
 create mode 100644 config-root/namespaces/jx-production/jxboot-helmfile-resources/tekton-container-registry-auth-secret.yaml
 create mode 100644 config-root/namespaces/jx-staging/jxboot-helmfile-resources/tekton-container-registry-auth-secret.yaml
 create mode 100644 config-root/namespaces/jx/chartmuseum/jenkins-x-chartmuseum-deploy.yaml
 create mode 100644 config-root/namespaces/jx/chartmuseum/jenkins-x-chartmuseum-persistentvolumeclaim.yaml
 create mode 100644 config-root/namespaces/jx/chartmuseum/jenkins-x-chartmuseum-secret.yaml
 create mode 100644 config-root/namespaces/jx/chartmuseum/jenkins-x-chartmuseum-svc.yaml
 create mode 100644 config-root/namespaces/jx/jx-build-controller/jenkins-x-controllerbuild-sa.yaml
 create mode 100644 config-root/namespaces/jx/jx-build-controller/jx-build-controller-deploy.yaml
 create mode 100644 config-root/namespaces/jx/jx-build-controller/jx-build-controller-rb.yaml
 create mode 100644 config-root/namespaces/jx/jx-build-controller/jx-build-controller-role.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-bot-token-kuberhealthycheck.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-check-rb-rb.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-check-rb-rb.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-kuberhealthycheck.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-sa-sa.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-service-role-role.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-kuberhealthycheck.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-sa-sa.yaml
 create mode 100644 config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-service-role-role.yaml
 create mode 100644 config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-deploy.yaml
 create mode 100644 config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-ingress.yaml
 create mode 100644 config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-sa.yaml
 create mode 100644 config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-svc.yaml
 create mode 100644 config-root/namespaces/jx/jx-preview/jx-preview-gc-jobs-cronjob.yaml
 create mode 100644 config-root/namespaces/jx/jx-preview/jx-preview-gc-jobs-rb.yaml
 create mode 100644 config-root/namespaces/jx/jx-preview/jx-preview-gc-jobs-role.yaml
 create mode 100644 config-root/namespaces/jx/jx-preview/jx-preview-gc-jobs-sa.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/chartmuseum-ingress.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/committer-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/dev-environment.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/dev-sourcerepository.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/gcactivities-rb.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/gcactivities-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/gcpods-rb.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/gcpods-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/hook-ingress.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/ingress-config-cm.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jenkins-maven-settings-secret.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jenkins-x-docker-registry-cm.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jenkins-x-extensions-cm.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-basic-auth-htpasswd-secret.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-basic-auth-user-password-secret.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-gcactivities-cronjob.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-gcactivities-sa.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-gcpods-cronjob.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-gcpods-sa.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-pipeline-activity-updater-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/jx-view-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/kapp-config-cm.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/nexus-ingress.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/owner-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/production-environment.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/staging-environment.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/tekton-bot-rb.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/tekton-bot-role.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/tekton-bot-sa.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/tekton-container-registry-auth-secret.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/tekton-git-secret.yaml
 create mode 100644 config-root/namespaces/jx/jxboot-helmfile-resources/viewer-role.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse-config/config-cm.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse-config/install-config-cm.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse-config/plugins-cm.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/hook-svc.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-external-plugins-cm.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-foghorn-deploy.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-foghorn-rb.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-foghorn-role.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-foghorn-sa.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-gc-jobs-cronjob.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-gc-jobs-rb.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-gc-jobs-role.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-gc-jobs-sa.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-hmac-token-secret.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-keeper-deploy.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-keeper-rb.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-keeper-role.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-keeper-sa.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-keeper-svc.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-oauth-token-secret.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-tekton-controller-deploy.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-tekton-controller-rb.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-tekton-controller-role.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-tekton-controller-sa.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-tekton-controller-svc.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-webhooks-deploy.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-webhooks-rb.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-webhooks-role.yaml
 create mode 100644 config-root/namespaces/jx/lighthouse/lighthouse-webhooks-sa.yaml
 create mode 100644 config-root/namespaces/jx/nexus/nexus-cm.yaml
 create mode 100644 config-root/namespaces/jx/nexus/nexus-nexus-deploy.yaml
 create mode 100644 config-root/namespaces/jx/nexus/nexus-nexus-persistentvolumeclaim.yaml
 create mode 100644 config-root/namespaces/jx/nexus/nexus-secret.yaml
 create mode 100644 config-root/namespaces/jx/nexus/nexus-svc.yaml
 create mode 100644 config-root/namespaces/jx/source-repositories/README.md
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-controller-admission-svc.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-controller-cm.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-controller-deploy.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-controller-metrics-svc.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-controller-poddisruptionbudget.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-controller-svc.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-rb.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-role.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/ingress-nginx-sa.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/ingress-nginx-admission-valwebhookcfg.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-create-job.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-patch-job.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-rb.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-role.yaml
 create mode 100644 config-root/namespaces/nginx/ingress-nginx-nginx-ingress/templates/admission-webhooks/job-patch/ingress-nginx-admission-sa.yaml
 create mode 100644 config-root/namespaces/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets-deploy.yaml
 create mode 100644 config-root/namespaces/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets-sa.yaml
 create mode 100644 config-root/namespaces/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets-svc.yaml
 create mode 100644 config-root/namespaces/secret-infra/pusher-wave/pusher-wave-pusher-wave-deploy.yaml
 create mode 100644 config-root/namespaces/secret-infra/pusher-wave/pusher-wave-pusher-wave-sa.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-artifact-bucket-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-artifact-pvc-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-defaults-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-leader-election-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-logging-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-observability-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config-registry-cert-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/config.webhook.pipeline.tekton.dev-valwebhookcfg.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/feature-flags-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/pipelines-info-cm.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-bot-sa.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-deploy.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-leaderelection-rb.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-rb.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-role.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-sa.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-controller-svc.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-info-rb.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-info-role.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-leader-election-role.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-podsecuritypolicy.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-deploy.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-horizontalpodautoscaler.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-leaderelection-rb.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-rb.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-role.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-sa.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/tekton-pipelines-webhook-svc.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/validation.webhook.pipeline.tekton.dev-valwebhookcfg.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/webhook-certs-secret.yaml
 create mode 100644 config-root/namespaces/tekton-pipelines/tekton-pipeline/webhook.pipeline.tekton.dev-mutwebhookcfg.yaml
make[1]: Leaving directory '/workspace/source'
found last commit message: chore: regenerated

/pipeline cancel
make regen-phase-2
make[1]: Entering directory '/workspace/source'
jx verify ingress --ingress-service ingress-nginx-controller
now verifying docker registry ingress setup
jx gitops repository create
file does not exist: .jx/gitops/source-config.yaml so not defaulting any SourceConfig resources
jx gitops repository resolve --source-dir config-root/namespaces
jx gitops jenkins jobs
the source config file .jx/gitops/source-config.yaml does not exist
jx secret convert edit
jx gitops helmfile resolve
jx gitops image -s .jx/git-operator
helmfile --file helmfile.yaml template --validate --include-crds --output-dir-template /tmp/generate/{{.Release.Namespace}}/{{.Release.Name}}
Adding repo jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories

Adding repo bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

Adding repo stable https://charts.helm.sh/stable
"stable" has been added to your repositories

Adding repo jxgh https://jenkins-x-charts.github.io/repo
"jxgh" has been added to your repositories

Templating release=jxboot-helmfile-resources, chart=jxgh/jxboot-helmfile-resources
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-serviceaccount.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-sa.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-sa.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-basic-auth-htpasswd-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-basic-auth-user-password-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/maven-settings-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/tekton-container-registry-auth-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/tekton-git-secret.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/ingress-config-configmap.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jenkins-x-docker-registry.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jenkins-x-extensions-config.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/kapp-config.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-clusterrole.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jenkins-x-aggregate-view.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-clusterrolebinding.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/committer-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-pipeline-activity-reporter-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-view-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/owner-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/viewer-role.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/build-bot-rolebinding.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-rb.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-rb.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcactivities-cronjob.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/jx-gcpods-cronjob.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/700-chartmuseum-ing.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/700-hook-ing.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/700-nexus-ing.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/environments.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/environments.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/environments.yaml
wrote /tmp/generate/jx/jxboot-helmfile-resources/jxboot-helmfile-resources/templates/repositories.yaml


Templating release=jenkins-x-crds, chart=jxgh/jenkins-x-crds
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/environments-crd.yaml
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/pipelineactivities-crd.yaml
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/releases-crd.yaml
wrote /tmp/generate/jx/jenkins-x-crds/jenkins-x-crds/templates/sourcerepositories-crd.yaml


Templating release=jx-pipelines-visualizer, chart=jxgh/jx-pipelines-visualizer
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/rbac.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/rbac.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/rbac.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/service.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/deployment.yaml
wrote /tmp/generate/jx/jx-pipelines-visualizer/jx-pipelines-visualizer/templates/ingress.yaml


Templating release=jx-preview, chart=jxgh/jx-preview
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-sa.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/previews.preview.jenkins.io.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-clusterrole.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-clusterrolebinding.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-role.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-rb.yaml
wrote /tmp/generate/jx/jx-preview/jx-preview/templates/gc-jobs-cj.yaml


Templating release=lighthouse, chart=jxgh/lighthouse
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-sa.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/hmacsecret.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/oauthsecret.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/config-external-plugins.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/lighthousebreakpoints-crd.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/lighthousejobs-crd.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-role.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-rb.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-service.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-service.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-service.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/foghorn-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/keeper-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/tekton-controller-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/webhooks-deployment.yaml
wrote /tmp/generate/jx/lighthouse/lighthouse/templates/gc-jobs-cj.yaml


Templating release=nexus, chart=jxgh/nexus
wrote /tmp/generate/jx/nexus/nexus/templates/secret.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/config.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/pvc.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/service.yaml
wrote /tmp/generate/jx/nexus/nexus/templates/deployment.yaml


Templating release=chartmuseum, chart=stable/chartmuseum
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/secret.yaml
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/pvc.yaml
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/service.yaml
wrote /tmp/generate/jx/chartmuseum/chartmuseum/templates/deployment.yaml


Templating release=jx-build-controller, chart=jxgh/jx-build-controller
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/serviceaccount.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/clusterrole.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/clusterrolebinding.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/role.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/rolebinding.yaml
wrote /tmp/generate/jx/jx-build-controller/jx-build-controller/templates/deployment.yaml


Templating release=health-checks-jx, chart=jxgh/jx-kh-check
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-bot-token.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-check.yaml
wrote /tmp/generate/jx/health-checks-jx/jx-kh-check/templates/jx-webhook-events-check.yaml


Adding repo ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories

Templating release=nginx-ingress, chart=ingress-nginx/ingress-nginx
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-poddisruptionbudget.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-serviceaccount.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-configmap.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/clusterrole.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/clusterrolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-role.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-rolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-service-metrics.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-service-webhook.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-service.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/controller-deployment.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
wrote /tmp/generate/nginx/nginx-ingress/ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml


Adding repo external-secrets https://external-secrets.github.io/kubernetes-external-secrets
"external-secrets" has been added to your repositories

Adding repo banzaicloud-stable https://kubernetes-charts.banzaicloud.com
"banzaicloud-stable" has been added to your repositories

Templating release=kubernetes-external-secrets, chart=external-secrets/kubernetes-external-secrets
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/crds/kubernetes-client.io_externalsecrets_crd.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/serviceaccount.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/rbac.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/rbac.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/rbac.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/service.yaml
wrote /tmp/generate/secret-infra/kubernetes-external-secrets/kubernetes-external-secrets/templates/deployment.yaml


Templating release=pusher-wave, chart=jxgh/pusher-wave
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/serviceaccount.yaml
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/clusterrole.yaml
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/clusterrolebinding.yaml
wrote /tmp/generate/secret-infra/pusher-wave/pusher-wave/templates/deployment.yaml


Adding repo cdf https://cdfoundation.github.io/tekton-helm-chart
"cdf" has been added to your repositories

Templating release=tekton-pipeline, chart=cdf/tekton-pipeline
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-ns.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-podsecuritypolicy.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-bot-sa.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-sa.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-sa.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/webhook-certs-secret.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-artifact-bucket-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-artifact-pvc-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-defaults-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-leader-election-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-logging-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-observability-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config-registry-cert-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/feature-flags-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelines-info-cm.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/clustertasks.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/conditions.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelineresources.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelineruns.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/pipelines.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/runs.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/taskruns.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tasks.tekton.dev-crd.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-aggregate-edit-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-aggregate-view-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-cluster-access-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-tenant-access-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-cluster-access-clusterrole.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-cluster-access-crb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-tenant-access-crb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-cluster-access-crb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-info-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-leader-election-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-role.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-leaderelection-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-info-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-leaderelection-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-rb.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-svc.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-svc.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-controller-deploy.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-deploy.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/tekton-pipelines-webhook-horizontalpodautoscaler.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/webhook.pipeline.tekton.dev-mutwebhookcfg.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/config.webhook.pipeline.tekton.dev-valwebhookcfg.yaml
wrote /tmp/generate/tekton-pipelines/tekton-pipeline/tekton-pipeline/templates/validation.webhook.pipeline.tekton.dev-valwebhookcfg.yaml


jx gitops split --dir /tmp/generate
jx gitops rename --dir /tmp/generate
jx gitops helmfile move --output-dir config-root --dir /tmp/generate --dir-includes-release-name
jx secret convert --source-dir config-root -r jx-vault
jx secret replicate --selector secret.jenkins-x.io/replica-source=true
jx gitops namespace --dir-mode --dir config-root/namespaces
jx gitops helmfile report
namespace cert-manager
ignoring release jetstack/cert-manager in namespace cert-manager as using conditional jxRequirementsIngressTLS.enabled

namespace cert-manager
ignoring release jetstack/cert-manager in namespace cert-manager as using conditional jxRequirementsIngressTLS.enabled

namespace jx
ignoring release bitnami/external-dns in namespace jx as using conditional jxRequirementsIngressExternalDNS.enabled
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/chartmuseum-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/hook-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/nexus-ingress.yaml with api version networking.k8s.io/v1
found jxboot-helmfile-resources version: 1.1.0 https://github.com/jenkins-x-charts/jxboot-helmfile-resources
WARNING: ignoring release jxgh/jenkins-x-crds in namespace jx as we cannot find any generated resources but there is no conditional
ignoring Ingress in file config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-ingress.yaml with api version networking.k8s.io/v1
found jx-pipelines-visualizer version: 1.7.5 https://github.com/jenkins-x/jx-pipelines-visualizer
found jx-preview version: 0.0.192 https://github.com/jenkins-x-plugins/jx-preview
found lighthouse version: 1.1.52 https://github.com/jenkins-x/lighthouse
found nexus version: 0.1.41 https://github.com/jenkins-x-charts/nexus
found chartmuseum version: 2.14.2 https://github.com/helm/chartmuseum
found jx-build-controller version: 0.3.9 https://jenkins-x.io/
ignoring release jxgh/acme in namespace jx as using conditional jxRequirementsIngressTLS.enabled
found jx-kh-check version: 0.0.78 https://jenkins-x.io/

namespace jx
ignoring release bitnami/external-dns in namespace jx as using conditional jxRequirementsIngressExternalDNS.enabled
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/chartmuseum-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/hook-ingress.yaml with api version networking.k8s.io/v1
ignoring Ingress in file config-root/namespaces/jx/jxboot-helmfile-resources/nexus-ingress.yaml with api version networking.k8s.io/v1
found jxboot-helmfile-resources version: 1.1.0 https://github.com/jenkins-x-charts/jxboot-helmfile-resources
WARNING: ignoring release jxgh/jenkins-x-crds in namespace jx as we cannot find any generated resources but there is no conditional
ignoring Ingress in file config-root/namespaces/jx/jx-pipelines-visualizer/jx-pipelines-visualizer-ingress.yaml with api version networking.k8s.io/v1
found jx-pipelines-visualizer version: 1.7.5 https://github.com/jenkins-x/jx-pipelines-visualizer
found jx-preview version: 0.0.192 https://github.com/jenkins-x-plugins/jx-preview
found lighthouse version: 1.1.52 https://github.com/jenkins-x/lighthouse
found nexus version: 0.1.41 https://github.com/jenkins-x-charts/nexus
found chartmuseum version: 2.14.2 https://github.com/helm/chartmuseum
found jx-build-controller version: 0.3.9 https://jenkins-x.io/
ignoring release jxgh/acme in namespace jx as using conditional jxRequirementsIngressTLS.enabled
found jx-kh-check version: 0.0.78 https://jenkins-x.io/

namespace kuberhealthy
ignoring release jxgh/kh-tls-check in namespace kuberhealthy as using conditional jxRequirementsTLSCheck.enabled

namespace kuberhealthy
ignoring release jxgh/kh-tls-check in namespace kuberhealthy as using conditional jxRequirementsTLSCheck.enabled

namespace nginx
found ingress-nginx version: 3.12.0 https://github.com/kubernetes/ingress-nginx

namespace nginx
found ingress-nginx version: 3.12.0 https://github.com/kubernetes/ingress-nginx

namespace secret-infra
found kubernetes-external-secrets version: 8.3.0 https://github.com/external-secrets/kubernetes-external-secrets
found pusher-wave version: 0.4.21 https://github.com/pusher/wave

namespace secret-infra
found kubernetes-external-secrets version: 8.3.0 https://github.com/external-secrets/kubernetes-external-secrets
found pusher-wave version: 0.4.21 https://github.com/pusher/wave

namespace tekton-pipelines
found tekton-pipeline version: 0.27.2 https://github.com/cdfoundation/tekton-helm-chart

namespace tekton-pipelines
found tekton-pipeline version: 0.27.2 https://github.com/cdfoundation/tekton-helm-chart

saved docs/releases.yaml
saved docs/README.md
jx gitops scheduler
jx gitops hash --pod-spec --kind Deployment -s config-root/namespaces/jx/lighthouse-config/config-cm.yaml -s config-root/namespaces/jx/lighthouse-config/plugins-cm.yaml -d config-root/namespaces/jx/lighthouse
jx gitops label --dir config-root/cluster                   gitops.jenkins-x.io/pipeline=cluster
jx gitops label --dir config-root/customresourcedefinitions gitops.jenkins-x.io/pipeline=customresourcedefinitions
jx gitops label --dir config-root/namespaces                gitops.jenkins-x.io/pipeline=namespaces
jx gitops annotate --dir config-root --selector app=pusher-wave kapp.k14s.io/change-group=apps.jenkins-x.io/pusher-wave
jx gitops annotate --dir config-root --selector app.kubernetes.io/name=ingress-nginx kapp.k14s.io/change-group=apps.jenkins-x.io/ingress-nginx
jx gitops label --dir config-root/cluster --kind=Namespace team=jx
jx gitops annotate --dir  config-root/namespaces --kind Deployment --selector app=pusher-wave --invert-selector wave.pusher.com/update-on-config-change=true
jx gitops git setup
found git user.name jenkins-recognia-com from requirements
found git user.email  from requirements
setup git user  email [email protected]
generated Git credentials file: /workspace/xdg_config/git/credentials with username: jenkins-recognia-com email:
git add --all
git commit -m "chore: regenerated" -m "/pipeline cancel"
[main 9cfac1a] chore: regenerated
 17 files changed, 18 insertions(+), 18 deletions(-)
make[1]: Leaving directory '/workspace/source'
make regen-phase-3
make[1]: Entering directory '/workspace/source'
Already up to date.
To https://github.com/Recognia/cluster-operations-testing
   56cc328..9cfac1a  main -> main
VAULT_ADDR=https://vault.jx-vault:8200 VAULT_NAMESPACE=jx-vault jx secret populate --secret-namespace jx-vault
waiting for vault pod vault-0 in namespace jx-vault to be ready...
pod vault-0 in namespace jx-vault is ready
verifying we have vault installed
about to run: /root/.jx/plugins/bin/vault-1.8.3 version
Vault v1.8.3 (73e85c3c21dfd1e835ded0053f08e3bd73a24ad6)
verifying we can connect to vault...
about to run: /root/.jx/plugins/bin/vault-1.8.3 kv list secret
Keys
----
accounts/
dockerrepo
mysql
vault is setup correctly!

managed to verify we can connect to vault
VAULT_ADDR=https://vault.jx-vault:8200 jx secret wait -n jx
waiting for the mandatory Secrets to be populated from ExternalSecrets...
jenkins-x-chartmuseum: key secret/data/jx/adminUser missing properties: password, username
jx-basic-auth-user-password: key secret/data/jx/basic/auth/user missing properties: password, key secret/data/jx/basic/auth/user/password missing properties: username
lighthouse-hmac-token: key secret/data/lighthouse/hmac missing properties: token
lighthouse-oauth-token: key secret/data/lighthouse/oauth missing properties: token
nexus: key secret/data/nexus missing properties: password
tekton-container-registry-auth: key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
tekton-git: key secret/data/jx/pipelineUser missing properties: token, username

[ankitm123]

You have to wait for 2 PRs to be merged before you can use jx with kubernetes 1.21 with vault: jenkins-x-charts/vault-instance#12 and jenkins-x/terraform-aws-eks-jx#320

You can check the logs of the external secrets in the secret-infra namespace and vault pods in the jx-vault namespace, and see that it has issues around iss.

[ankitm123]

@mestuddtc Those PRs have been merged, can you update the version of the eks-jx module to the latest, and run an apply again? https://github.com/jenkins-x/terraform-aws-eks-jx/releases/tag/v1.17.3

[mestuddtc]

terraform apply fails with the following error:

╷
│ Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Vault.spec.ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in com.banzaicloud.vault.v1alpha1.Vault.spec.ingress.spec.rules.http.paths.backend, ValidationError(Vault.spec.ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in com.banzaicloud.vault.v1alpha1.Vault.spec.ingress.spec.rules.http.paths.backend]
│
│   with module.eks-jx.module.vault.helm_release.vault-instance[0],
│   on .terraform/modules/eks-jx/modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
│   11: resource "helm_release" "vault-instance" {
│

[mestuddtc]

Looks like an update is needed in https://github.com/jenkins-x-charts/vault-instance/blob/master/charts/vault-instance/values.yaml I could open a PR for that, but I wouldn't know how to test it.

[mestuddtc]

jenkins-x-charts/vault-instance#13

[ankitm123]

@mestuddtc the latest version of the eks-jx module has ur changes, try it and let me know if it works!

[mestuddtc]

@ankitm123 it does successfully install - the boot job completed, and the secrets are populated with no secret errors from any pods.

There are four things reported by jx health. I don't know what the result of those is.

mstudd-mac:jx3-terraform-eks mstudd$ jx health status -A
NAME                          NAMESPACE                     STATUS                        ERROR MESSAGE
daemonset                     kuberhealthy                  OK
deployment                    kuberhealthy                  OK
dns-status-internal           kuberhealthy                  OK
jx-bot-token                  jx                            ERROR                         Check execution error: pod deleted unexpectedly
jx-install                    jx-git-operator               ERROR                         Check execution error: jx-git-operator/jx-install: error when waiting for pod to start: the khcheck check pod is deleted, waiting for start failed
jx-pod-status                 kuberhealthy                  OK
jx-secrets                    kuberhealthy                  OK
jx-webhook                    jx                            ERROR                         Check execution error: jx/jx-webhook: error when waiting for pod to start: the khcheck check pod is deleted, waiting for start failed
jx-webhook-events             jx                            ERROR                         Check execution error: pod deleted unexpectedly
network-connection-check      kuberhealthy                  OK
pod-restarts                  kuberhealthy                  OK

Occasionally those pods show up, but they're always deleted before I can view them. There are FailedMount events for the four pods:

Unable to attach or mount volumes: unmounted volumes=[kube-api-access-jxgxc], unattached volumes=[kube-api-access-jxgxc]: timed out waiting for the condition

And the jxinstall, jx-webhook pods seem to have OwnerRefInvalidNamespace events:

ownerRef [v1/Pod, namespace: jx, name: kuberhealthy-84645cf59c-94dzb, uid: 03c15d18-3b5a-441c-b306-3329eb361d80] does not exist in namespace "jx"

I haven't had time yet to investigate. I'm not sure if those should be a separate bug?

[ankitm123]

I think that's a kuberhealthy issue, James is looking at it I believe, but it's just a monitoring tool, and not a blocker (and unrelated to eks), so it will get fixed soon.
For now, I think we can close this issue.
/close

[mestuddtc]

Ok, good to know.
Thanks for your help.