Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not able to configure dynamic Kubernetes Accounts using Vault in Spinnaker #28

Open
headsha opened this issue May 6, 2021 · 2 comments
Open

Not able to configure dynamic Kubernetes Accounts using Vault in Spinnaker #28

headsha opened this issue May 6, 2021 · 2 comments

Comments

@headsha
Copy link

headsha commented May 6, 2021

I had setup the config as mentioned in the doc https://docs.armory.io/docs/armory-admin/dynamic-accounts-configure/ , and configured spinnaker to access my kubernetes cloud providers accounts dynamically from vault .

I was able to successfully add kubernetes accounts as deployment targets. Deployments to those accounts were also successful.

But recently i observed, even though the accounts are getting added in the same manner, (that is, the account credentials are reflecting in the gate-endpoint/credentials url.) deployments are always happening to the cluster where Spinnaker is running( instead of the target accounts selected). Tried the same config several times but had no luck.

These are the clouddriver logs -

2021-04-29 12:36:36.772 INFO 1 --- [0.0-7002-exec-6] c.n.s.c.k.v.KubernetesValidationUtil : Validating credentials for spinpoc-GKE namespace

2021-04-29 12:36:36.786 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [ORCHESTRATION] Processing op: KubernetesDeployManifestOperation

2021-04-29 12:36:36.788 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Beginning deployment of manifest...

2021-04-29 12:36:36.789 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Swapping out artifacts in namespace demons from context...

2021-04-29 12:36:36.791 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Finding deployer for namespace...

2021-04-29 12:36:36.793 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Checking if all requested artifacts were bound...

2021-04-29 12:36:36.795 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Sorting manifests by priority...

2021-04-29 12:36:36.798 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Deploy order is: namespace demons

2021-04-29 12:36:36.800 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Finding deployer for namespace...

2021-04-29 12:36:36.802 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Annotating manifest namespace demons with artifact, relationships & moniker...

2021-04-29 12:36:36.804 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Swapping out artifacts in namespace demons from other deployments...

2021-04-29 12:36:36.815 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Submitting manifest namespace demons to kubernetes master...

2021-04-29 12:36:37.418 INFO 1 --- [tionProcessor-3] c.n.s.c.data.task.jedis.JedisTask : [DEPLOY_KUBERNETES_MANIFEST] Deploy manifest task completed successfully.

Kindly help me with this. Thanks in advance! :)
@nabuskey
Copy link

I am running into the same issue. Credentials are visible in clouddriver/credentials endpoint and they seem correct.

Spinnaker: 0SS 1.26.6
EAP: 0.1.2

clouddriver log output:

2022-03-23 20:19:15.761 ERROR 1 --- [           main] c.n.s.c.k.s.KubernetesCredentials        : Could not list namespaces for account client1-ekscluster: Failed to read [namespac
e] from : I0323 20:18:31.138966      54 request.go:621] Throttling request took 1.1600885s, request: GET:https://172.20.0.1:443/apis/aws.crossplane.io/v1alpha3?timeout=32s
I0323 20:18:41.142049      54 request.go:621] Throttling request took 2.589032099s, request: GET:https://172.20.0.1:443/apis/cert-manager.io/v1alpha2?timeout=32s
I0323 20:18:51.142081      54 request.go:621] Throttling request took 3.595912017s, request: GET:https://172.20.0.1:443/apis/autoscaling/v2beta2?timeout=32s
I0323 20:19:01.342074      54 request.go:621] Throttling request took 13.795372604s, request: GET:https://172.20.0.1:443/apis/ec2.aws.jet.crossplane.io/v1alpha2?timeout=32s
I0323 20:19:11.542274      54 request.go:621] Throttling request took 23.994753432s, request: GET:https://172.20.0.1:443/apis/infrastructure.cluster.x-k8s.io/v1beta1?timeout=32s
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:spinnaker:clouddriver" cannot list resource "namespaces" in API group "" at the cluster scope

2022-03-23 20:19:15.761  WARN 1 --- [           main] .s.KubernetesCredentialsLifecycleHandler : New account client1-ekscluster did not return any namespace and could be unreachable or misconfigured
2022-03-23 20:19:15.764  INFO 1 --- [           main] .s.KubernetesCredentialsLifecycleHandler : Adding 2 agents for new account client1-ekscluster
2022-03-23 20:20:01.692 ERROR 1 --- [           main] c.n.s.c.k.s.KubernetesCredentials        : Could not list namespaces for account client1-ekscluster-ekscluster: Failed to read [namespace] from : I0323 20:19:17.040485      66 request.go:621] Throttling request took 1.175398564s, request: GET:https://172.20.0.1:443/apis/aws.crossplane.io/v1beta1?timeout=32s
I0323 20:19:27.043962      66 request.go:621] Throttling request took 2.587310269s, request: GET:https://172.20.0.1:443/apis/iam.aws.jet.crossplane.io/v1alpha2?timeout=32s
I0323 20:19:37.243838      66 request.go:621] Throttling request took 3.795616107s, request: GET:https://172.20.0.1:443/apis/batch/v1beta1?timeout=32s
I0323 20:19:47.443897      66 request.go:621] Throttling request took 13.993248945s, request: GET:https://172.20.0.1:443/apis/acme.cert-manager.io/v1?timeout=32s
I0323 20:19:57.652947      66 request.go:621] Throttling request took 24.20128221s, request: GET:https://172.20.0.1:443/apis/cache.aws.crossplane.io/v1beta1?timeout=32s
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:spinnaker:clouddriver" cannot list resource "namespaces" in API group "" at the cluster scope

Logs from a test deployment trying to use the newly added account

2022-03-23 20:23:51.092  INFO 1 --- [nio-7002-exec-5] c.n.s.c.k.v.KubernetesValidationUtil     : Validating credentials for client1-ekscluster namespace
2022-03-23 20:23:51.308  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [ORCHESTRATION] Processing op: KubernetesDeployManifestOperation
2022-03-23 20:23:51.327  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Beginning deployment of manifest...
2022-03-23 20:23:51.341  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Swapping out artifacts in namespace testing1 from con
text...
2022-03-23 20:23:51.345  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Finding deployer for namespace...
2022-03-23 20:23:51.392  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Checking if all requested artifacts were bound...
2022-03-23 20:23:51.395  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Sorting manifests by priority...
2022-03-23 20:23:51.400  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Deploy order is: namespace testing1
2022-03-23 20:23:51.404  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Finding deployer for namespace...
2022-03-23 20:23:51.415  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Annotating manifest namespace testing1 with artifact,
 relationships & moniker...
2022-03-23 20:23:51.435  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Swapping out artifacts in namespace testing1 from oth
er deployments...
2022-03-23 20:23:51.441  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [DEPLOY_KUBERNETES_MANIFEST] Submitting manifest namespace testing1 to kubernetes
master...

2022-03-23 20:24:00.742  INFO 1 --- [tionProcessor-0] c.n.s.c.data.task.jedis.JedisTask        : [ORCHESTRATION] Orchestration failed: KubernetesDeployManifestOperation | KubectlException: [Deploy failed: I0323 20:23:53.285943     577 request.go:621] Throttling request took 1.189829442s, request: GET:https://172.20.0.1:443/apis/addons.cluster.x-k8s.io/v1alpha4?timeout=32s
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "testing1", Namespace: ""
from server for: "STDIN": namespaces "testing1" is forbidden: User "system:serviceaccount:spinnaker:clouddriver" cannot get resource "namespaces" in API group "" in the namespace "testing1"
]
2022-03-23 20:24:00.766 ERROR 1 --- [tionProcessor-0] c.n.s.c.o.DefaultOrchestrationProcessor  : com.netflix.spinnaker.clouddriver.kubernetes.op.job.KubectlJobExecutor$KubectlException: Deploy failed: I0323 20:23:53.285943     577 request.go:621] Throttling request took 1.189829442s, request: GET:https://172.20.0.1:443/apis/addons.cluster.x-k8s.io/v1alpha4?timeout=32s
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "testing1", Namespace: ""
from server for: "STDIN": namespaces "testing1" is forbidden: User "system:serviceaccount:spinnaker:clouddriver" cannot get resource "namespaces" in API group "" in the namespace "testing1"
        at com.netflix.spinnaker.clouddriver.kubernetes.op.job.KubectlJobExecutor.deploy(KubectlJobExecutor.java:440)
        at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.lambda$deploy$14(KubernetesCredentials.java:523)
        at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:633)
        at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.runAndRecordMetrics(KubernetesCredentials.java:618)
        at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials.deploy(KubernetesCredentials.java:519)
        at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.CanDeploy.deploy(CanDeploy.java:58)
        at com.netflix.spinnaker.clouddriver.kubernetes.op.manifest.KubernetesDeployManifestOperation.operate(KubernetesDeployManifestOperation.java:209)
        at com.netflix.spinnaker.clouddriver.kubernetes.op.manifest.KubernetesDeployManifestOperation.operate(KubernetesDeployManifestOperation.java:46)
        at com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation$operate.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy:118)
        at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy)

clouddriver/credentials endpoint output.

[
  {
    "accountType": "client1-ekscluster",
    "challengeDestructiveActions": false,
    "cloudProvider": "kubernetes",
    "environment": "client1-ekscluster",
    "name": "client1-ekscluster",
    "primaryAccount": false,
    "requiredGroupMembership": [],
    "type": "kubernetes"
  }
]

clouddriver config

spinnaker:
  extensibility:
    plugins-root-path: /opt/clouddriver/plugins
    plugins:
      Armory.EAP:
        enabled: true
        version: 0.1.2
    repositories:
      eap:
        enabled: true
        url: https://raw.githubusercontent.com/armory-plugins/external-accounts/master/plugins.json
armory:
  external-accounts:
    dir: /var/accounts
    file-prefix:
      default: clouddriver
      kubernetes: kube
credentials:
  poller:
    enabled: true
    types:
      kubernetes: 
        reloadFrequencyMs: 180000

Using a side car to populate local files.

bash-5.0$ cat /var/accounts/kube-dynamic-accounts.yaml
kubernetes:
  accounts:
  - name: client1-ekscluster
    kubeconfigfilepath: /var/accounts/kubeconfig-client1-ekscluster.yaml

kubeconfig file is valid:

bash-5.0$ kubectl --kubeconfig /var/accounts/kubeconfig-client1-ekscluster.yaml get pods
No resources found in spinnaker namespace.

@nabuskey
Copy link

Never mind it was my bad. In one of structs for sidecar code, I had

type Account struct {
	Name               string `json:"name"`
	KubeconfigFilePath string `json:"kubeconfigFile"`
}

Output file is not a JSON file so serialization tags are wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nabuskey @headsha