Vault Token Renewal Not Triggered in Spring Cloud Kubernetes Config Server (v3.2.0)

[arjavdongaonkar]

Description:

I am currently using Spring Cloud Kubernetes Config Server (v3.2.0), but I am facing an issue with HashiCorp Vault token renewal. When running the Spring Cloud Config Server normally (v4.2.0), the Vault token renewal activity is automatically triggered. However, with the Spring Cloud Kubernetes Config Server (v3.2.0), the token renewal is not triggered, resulting in the missing X-Config-Token header in requests.

Expected Behavior:

The Vault token renewal should be automatically triggered when using Spring Cloud Kubernetes Config Server, and the X-Config-Token header should be included in the requests.

Environment:

• Spring Cloud Kubernetes Config Server version: 3.2.0
• Spring Cloud Config Server version (when working normally): 4.2.0

Investigation:

After further investigation, I discovered that the spring-vault-core dependency must be included in the application's classpath to trigger LifecycleAwareSessionManager (which is responsible for the token renewal process).

On Spring Cloud Config Server (v4.2.0), I see the following log line:
o.s.v.a.LifecycleAwareSessionManager - Scheduling Token renewal

However, this log line is not seen when using Spring Cloud Kubernetes Config Server (v3.2.0), indicating that the token renewal is not being triggered.

logs:

` \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.4.0)

07:16:29.522 [main] INFO  o.s.c.k.c.KubernetesConfigServerApplication - Starting KubernetesConfigServerApplication v3.2.0 using Java 21.0.5 with PID 1 (/usr/src/app/spring-cloud-kubernetes-configserver-3.2.0.jar started by root in /usr/src/app) 
07:16:29.526 [main] INFO  o.s.c.k.c.KubernetesConfigServerApplication - The following 2 profiles are active: "kubernetes", "composite" 
07:16:37.401 [main] INFO  o.s.c.context.scope.GenericScope - BeanFactory id=18b381b0-f077-3064-b6f3-ccc47cc63c6b 
07:16:40.020 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port 8888 (http) 
07:16:40.112 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8888"] 
07:16:40.118 [main] INFO  o.a.catalina.core.StandardService - Starting service [Tomcat] 
07:16:40.119 [main] INFO  o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/10.1.33] 
07:16:40.424 [main] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext 
07:16:40.425 [main] INFO  o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 10693 ms 
07:16:43.305 [main] INFO  o.h.validator.internal.util.Version - HV000001: Hibernate Validator 8.0.1.Final 
07:16:51.826 [main] INFO  o.s.b.a.e.web.EndpointLinksResolver - Exposing 16 endpoints beneath base path '/actuator' 
07:16:52.821 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8888"] 
07:16:52.911 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat started on port 8888 (http) with context path '/' 
07:16:53.041 [main] INFO  o.s.c.k.c.KubernetesConfigServerApplication - Started KubernetesConfigServerApplication in 35.837 seconds (process running for 39.806) 
07:16:55.831 [http-nio-8888-exec-1] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet' 
07:16:55.833 [http-nio-8888-exec-1] INFO  o.s.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet' 
07:16:55.835 [http-nio-8888-exec-1] INFO  o.s.web.servlet.DispatcherServlet - Completed initialization in 2 ms
07:17:35.174 [http-nio-8888-exec-9] WARN  o.s.c.c.s.e.EnvironmentController - Error getting the Environment with name=communication-service profiles=postgres,rabbitmq,redis,domains,access-keys,s3,kafka,communication-service,secrets label=null includeOrigin=false 
java.lang.IllegalArgumentException: Missing required header in HttpServletRequest: X-Config-Token
`

Can someone help with this issue and provide guidance on how to enable Vault token renewal for Spring Cloud Kubernetes Config Server (v3.2.0)?

[ryanjbaxter]

Are using the docker image for the config server from DockerHub?

Can you provide your deployment yaml and any other configuration you are setting for the config server?

[arjavdongaonkar]

Hey @ryanjbaxter,

Sure, here’s the information you requested:

Docker Image:
I am building the image from the source code. Here’s the relevant snippet from the Dockerfile:

RUN git clone --branch v3.2.0 --depth 1 https://github.com/spring-cloud/spring-cloud-kubernetes.git .
WORKDIR $APP_HOME/spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configserver
RUN mvn package org.springframework.boot:spring-boot-maven-plugin:repackage -DskipTests -Dskip.build.image=true

The resulting JAR file is then executed with Java.

Deployment YAML:
Below is the Deployment YAML I’m using. It includes two active profiles: kubernetes and composite (for Vault and Git configurations).

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        image: <img_tag>
        env:
        - env:
        - name: SPRING_PROFILES_ACTIVE
          value: "composite"
        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_KUBERNETES_ROLE
          value: <vault-config-server-role>
        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_HOST
          value: <Vault_host>
        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_AUTHENTICATION
          value: "KUBERNETES"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE
          value: "vault"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_BACKEND
          value: "secret"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_KVVERSION
          value: '2'
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_HOST
          value: <Vault_host>
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_TYPE
          value: "git"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_DEFAULTLABEL
          value: ${GIT_BRANCH}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_URI
          value: ${GIT_URI}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_SEARCHPATHS
          value: ${GIT_SEARCHPATHS}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_PASSWORD
          value: ${GIT_TOKEN}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_USERNAME
          value: ${GIT_USERNAME}
        - name: SPRING_PROFILES_INCLUDE
          value: 'kubernetes'
        - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI
          value: 'true'
        - name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLEAPI
          value: 'false'
        image: <Image>

it works when we provide this as well

        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_TOKEN
          value: <Vault_token>

We have attached required service account and roles to this deployment (even joined this in role binding).
Followed this doc: https://docs.spring.io/spring-cloud-kubernetes/reference/spring-cloud-kubernetes-configserver.html
Let me know if you need further clarification or additional details!

[arjavdongaonkar]

Hi @ryanjbaxter,

I had a couple of questions that I’d love your input on:

1. Accessing Secrets in Config Server API
   When calling the Config Server to fetch configurations, we currently need to explicitly include secrets in the profile, like this:
   curl --location 'http://localhost:8888/ab-service/ab-service,secrets'
   Since Kubernetes Secrets are similar to Vault in nature, is there a way to configure the Config Server to fetch them by default without explicitly including them in the profiles (similar to how Vault configurations are handled)?
   Is it possible to treat Kubernetes Secrets as a composite type? This way, we could configure them alongside other sources, such as Vault or Git, allowing more flexibility in configuration management.

2. Secrets Priority in propertySources
   In the current setup, Kubernetes Secrets appear at the bottom of the propertySources list in the API response.
   For example:

"propertySources": [
  { "name": "vault:application" },
  { "name": "secret:ab-service-secrets" }
]

Can we configure the order of propertySources so that Kubernetes Secrets appear at the top of the list? This would ensure that values from Secrets take precedence, allowing other configurations to resolve their values using Secrets.

[ryanjbaxter]

I am going to focus on one issue at a time we can come back to your other questions after we have a good idea about your original issue.

You said

it works when we provide this as well
- name: SPRING_CLOUD_CONFIG_SERVER_VAULT_TOKEN
value: <Vault_token>

So does that mean if you set that environment variable everything works on Kubernetes?