Bugs: SecretClient: HTTP response with status code 400 #222

Thinhhoagn0211 · 2023-12-02T05:11:19Z

Thinhhoagn0211
Dec 2, 2023

When i running docker-compose.yml of repo edgex-compose tag v2.3.0 on my laptop, why i turn off my laptop more than 8h or more, when i start my laptop and all service in edgex have trouble with Error:
level=INFO ts=2023-12-02T05:06:31.568298855Z app=core-metadata source=secret.go:63 msg="Creating SecretClient"
level=INFO ts=2023-12-02T05:06:31.568303891Z app=core-metadata source=secret.go:70 msg="Reading secret store configuration and authentication token"
level=INFO ts=2023-12-02T05:06:31.568310132Z app=core-metadata source=secret.go:177 msg="load token from file"
level=INFO ts=2023-12-02T05:06:31.568360942Z app=core-metadata source=secret.go:88 msg="Attempting to create secret client"
level=INFO ts=2023-12-02T05:06:31.570047917Z app=core-metadata source=secrets.go:260 msg="ttl already <= half of the renewal period"
level=WARN ts=2023-12-02T05:06:31.570660927Z app=core-metadata source=secret.go:119 msg="Retryable failure while creating SecretClient: HTTP response with status code 400, message: failed to renew token"
level=INFO ts=2023-12-02T05:06:32.570736378Z app=core-metadata source=secret.go:70 msg="Reading secret store configuration and authentication token"
level=INFO ts=2023-12-02T05:06:32.570769964Z app=core-metadata source=secret.go:177 msg="load token from file"
level=INFO ts=2023-12-02T05:06:32.570845895Z app=core-metadata source=secret.go:88 msg="Attempting to create secret client"
level=INFO ts=2023-12-02T05:06:32.572491635Z app=core-metadata source=secrets.go:260 msg="ttl already <= half of the renewal period"
level=WARN ts=2023-12-02T05:06:32.573378577Z app=core-metadata source=secret.go:119 msg="Retryable failure while creating SecretClient: HTTP response with status code 400, message: failed to renew token"
level=INFO ts=2023-12-02T05:06:33.573591999Z app=core-metadata source=secret.go:70 msg="Reading secret store configuration and authentication token"
level=INFO ts=2023-12-02T05:06:33.573665631Z app=core-metadata source=secret.go:177 msg="load token from file"
level=INFO ts=2023-12-02T05:06:33.573766718Z app=core-metadata source=secret.go:88 msg="Attempting to create secret client"
level=INFO ts=2023-12-02T05:06:33.575905025Z app=core-metadata source=secrets.go:260 msg="ttl already <= half of the renewal period"
level=WARN ts=2023-12-02T05:06:33.577569573Z app=core-metadata source=secret.go:119 msg="Retryable failure while creating SecretClient: HTTP response with status code 400, message: failed to renew token"
level=INFO ts=2023-12-02T05:06:34.577736588Z app=core-metadata source=secret.go:70 msg="Reading secret store configuration and authentication token"
level=INFO ts=2023-12-02T05:06:34.577781603Z app=core-metadata source=secret.go:177 msg="load token from file"
level=INFO ts=2023-12-02T05:06:34.577888137Z app=core-metadata source=secret.go:88 msg="Attempting to create secret client"
level=INFO ts=2023-12-02T05:06:34.579910204Z app=core-metadata source=secrets.go:260 msg="ttl already <= half of the renewal period"
level=WARN ts=2023-12-02T05:06:34.581308499Z app=core-metadata source=secret.go:119 msg="Retryable failure while creating SecretClient: HTTP response with status code 400, message: failed to renew token"
level=INFO ts=2023-12-02T05:06:35.581496324Z app=core-metadata source=secret.go:70 msg="Reading secret store configuration and authentication token"

Please give me some helps about problem!!!
Thanks u

bnevis-i · 2023-12-03T18:39:09Z

bnevis-i
Dec 3, 2023
Maintainer

This is by design. The EdgeX secret store tokens have a 1 hour time-to-live (TTL). If the service does not renew its token by the time it expires, it is invalidated. There are various ways to adjust the TTL (by settings on Vault, and setting of default TTL periods of EdgeX's tokens) if this is a frequent occurrence, however, it is a feature of the underlying open source component (Hashicorp Vault) that tokens do not live in perpetuity. Here is also a feature by bypass this behavior for add-on services https://docs.edgexfoundry.org/3.0/security/Ch-DelayedStartServices/ but this capability is disabled for the EdgeX core services by default to save space.

Also, you should be able to restart the secretstore-setup component to regenerate fresh tokens and there is retry logic inside of the core services to pick up fresh tokens.

Hopefully, one of the above three methods (a) don't sleep, (b) adjust TTL, or (c) use delayed start features should be able to resolve your issue.

5 replies

NguyenKhanh27 Dec 6, 2023

I did notice, im using the docker compose version 2.3 of Edgex whereas vault service is implemented like this :

vault:
  cap_add:
  - IPC_LOCK
  command: server
  container_name: edgex-vault
  depends_on:
  - security-bootstrapper
  entrypoint:
  - /edgex-init/vault_wait_install.sh
  environment:
    API_GATEWAY_HOST: edgex-kong
    API_GATEWAY_STATUS_PORT: '8100'
    PROXY_SETUP_HOST: edgex-security-proxy-setup
    STAGEGATE_BOOTSTRAPPER_HOST: edgex-security-bootstrapper
    STAGEGATE_BOOTSTRAPPER_STARTPORT: '54321'
    STAGEGATE_DATABASE_HOST: edgex-redis
    STAGEGATE_DATABASE_PORT: '6379'
    STAGEGATE_DATABASE_READYPORT: '6379'
    STAGEGATE_KONGDB_HOST: edgex-kong-db
    STAGEGATE_KONGDB_PORT: '5432'
    STAGEGATE_KONGDB_READYPORT: '54325'
    STAGEGATE_READY_TORUNPORT: '54329'
    STAGEGATE_REGISTRY_HOST: edgex-core-consul
    STAGEGATE_REGISTRY_PORT: '8500'
    STAGEGATE_REGISTRY_READYPORT: '54324'
    STAGEGATE_SECRETSTORESETUP_HOST: edgex-security-secretstore-setup
    STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT: '54322'
    STAGEGATE_WAITFOR_TIMEOUT: 60s
    VAULT_ADDR: http://edgex-vault:8200
    VAULT_CONFIG_DIR: /vault/config
    VAULT_UI: "true"
  hostname: edgex-vault
  image: vault:1.11.4
  networks:
    edgex-network: {}
  ports:
  - 127.0.0.1:8200:8200/tcp
  restart: always
  tmpfs:
  - /vault/config
  user: root:root
  volumes:
  - edgex-init:/edgex-init:ro,z
  - vault-file:/vault/file:z
  - vault-logs:/vault/logs:z  vault:
  cap_add:
  - IPC_LOCK
  command: server
  container_name: edgex-vault
  depends_on:
  - security-bootstrapper
  entrypoint:
  - /edgex-init/vault_wait_install.sh
  environment:
    API_GATEWAY_HOST: edgex-kong
    API_GATEWAY_STATUS_PORT: '8100'
    PROXY_SETUP_HOST: edgex-security-proxy-setup
    STAGEGATE_BOOTSTRAPPER_HOST: edgex-security-bootstrapper
    STAGEGATE_BOOTSTRAPPER_STARTPORT: '54321'
    STAGEGATE_DATABASE_HOST: edgex-redis
    STAGEGATE_DATABASE_PORT: '6379'
    STAGEGATE_DATABASE_READYPORT: '6379'
    STAGEGATE_KONGDB_HOST: edgex-kong-db
    STAGEGATE_KONGDB_PORT: '5432'
    STAGEGATE_KONGDB_READYPORT: '54325'
    STAGEGATE_READY_TORUNPORT: '54329'
    STAGEGATE_REGISTRY_HOST: edgex-core-consul
    STAGEGATE_REGISTRY_PORT: '8500'
    STAGEGATE_REGISTRY_READYPORT: '54324'
    STAGEGATE_SECRETSTORESETUP_HOST: edgex-security-secretstore-setup
    STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT: '54322'
    STAGEGATE_WAITFOR_TIMEOUT: 60s
    VAULT_ADDR: http://edgex-vault:8200
    VAULT_CONFIG_DIR: /vault/config
    VAULT_UI: "true"
  hostname: edgex-vault
  image: vault:1.11.4
  networks:
    edgex-network: {}
  ports:
  - 127.0.0.1:8200:8200/tcp
  restart: always
  tmpfs:
  - /vault/config
  user: root:root
  volumes:
  - edgex-init:/edgex-init:ro,z
  - vault-file:/vault/file:z
  - vault-logs:/vault/logs:z

Is there anything relevant to your suggestion to config the vault service or TTL ? Thanks for any suggestions!

bnevis-i Dec 6, 2023
Maintainer

Other relevant configuration (this is from the 3.1 release... the 2.3 release may differ slightly).

security-secretstore-setup:
    container_name: edgex-security-secretstore-setup
    depends_on:
      security-bootstrapper:
        condition: service_started
      vault:
        condition: service_started
    environment:
      EDGEX_ADD_KNOWN_SECRETS: redisdb[app-rules-engine],redisdb[device-rest],message-bus[device-rest],redisdb[device-virtual],message-bus[device-virtual]
      EDGEX_ADD_SECRETSTORE_TOKENS: ""
      EDGEX_GROUP: "2001"
      EDGEX_SECURITY_SECRET_STORE: "true"
      EDGEX_USER: "2002"
      PROXY_SETUP_HOST: edgex-security-proxy-setup
      SECRETSTORE_HOST: edgex-vault
      SECUREMESSAGEBUS_TYPE: redis
      STAGEGATE_BOOTSTRAPPER_HOST: edgex-security-bootstrapper
      STAGEGATE_BOOTSTRAPPER_STARTPORT: "54321"
      STAGEGATE_DATABASE_HOST: edgex-redis
      STAGEGATE_DATABASE_PORT: "6379"
      STAGEGATE_DATABASE_READYPORT: "6379"
      STAGEGATE_PROXYSETUP_READYPORT: "54325"
      STAGEGATE_READY_TORUNPORT: "54329"
      STAGEGATE_REGISTRY_HOST: edgex-core-consul
      STAGEGATE_REGISTRY_PORT: "8500"
      STAGEGATE_REGISTRY_READYPORT: "54324"
      STAGEGATE_SECRETSTORESETUP_HOST: edgex-security-secretstore-setup
      STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT: "54322"
      STAGEGATE_WAITFOR_TIMEOUT: 60s
    hostname: edgex-security-secretstore-setup
    image: nexus3.edgexfoundry.org:10004/security-secretstore-setup-arm64:latest
    networks:
      edgex-network: null
    read_only: true
    restart: always
    security_opt:
    - no-new-privileges:true
    tmpfs:
    - /run
    - /vault
    user: root:root
    volumes:
    - type: volume
      source: edgex-init
      target: /edgex-init
      read_only: true
      volume: {}
    - type: bind
      source: /etc/localtime
      target: /etc/localtime
      read_only: true
      bind:
        create_host_path: true
    - type: bind
      source: /tmp/edgex/secrets
      target: /tmp/edgex/secrets
      bind:
        selinux: z
        create_host_path: true
    - type: volume
      source: kuiper-sources
      target: /tmp/kuiper
      volume: {}
    - type: volume
      source: kuiper-connections
      target: /tmp/kuiper-connections
      volume: {}
    - type: volume
      source: vault-config
      target: /vault/config
      volume: {}

bnevis-i Dec 6, 2023
Maintainer

For security-secretstore-setup there is a configuration https://github.com/edgexfoundry/edgex-go/blob/main/cmd/security-spiffe-token-provider/res/configuration.yaml

Writable:
  LogLevel: DEBUG
Service:
  Host: localhost
  Port: 59841
  StartupMsg: "This is the SPIFFE token provider microservice"
TokenConfig:
  DefaultTokenTTL: 1h
  DefaultJWTTTL: 15m
  DefaultJWTAudience: edgex
  UserPassMountPoint: userpass
Spiffe:
  EndpointSocket: /tmp/edgex/secrets/spiffe/public/api.sock
  TrustDomain: edgexfoundry.org

The relevant override to set in environment variables on security-secretstore-setup is TOKENCONFIG_DEFAULTTOKENTTL

There is also configuration in
https://github.com/edgexfoundry/edgex-go/blob/main/cmd/security-bootstrapper/entrypoint-scripts/vault_wait_install.sh.

This Vault configuration can be overridden by setting DEFAULT_VAULT_LOCAL_CONFIG on the vault container if overriding DefaultTokenTTL is insufficient.

bnevis-i Dec 6, 2023
Maintainer

Also, debugging your changes will be problematic.

You may find it useful to study https://github.com/edgexfoundry/edgex-go/blob/main/cmd/security-secretstore-setup/res/configuration.yaml and override SECRETSTORE_REVOKEROOTTOKENS="false" in security-secretstore-setup when first initializing Vault (the make clean target wipes out all the docker volumes) and grabbing the root token out of the resp-init.json and using it to inspect the remaining TTL's of all the tokens to make sure everything took.

By default, EdgeX services will only renew their tokens once half their TTL expires, so your actual max TTL will need to be twice whatever you need to survive when your system is sleeping.

bnevis-i Dec 7, 2023
Maintainer

Also consul this reference as well:

https://docs.edgexfoundry.org/2.3/microservices/configuration/CommonEnvironmentVariables/

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EdgeX Foundry Project

Bugs: SecretClient: HTTP response with status code 400 #222

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

EdgeX Foundry Project

Bugs: SecretClient: HTTP response with status code 400 #222

Thinhhoagn0211 Dec 2, 2023

Replies: 1 comment · 5 replies

bnevis-i Dec 3, 2023 Maintainer

NguyenKhanh27 Dec 6, 2023

bnevis-i Dec 6, 2023 Maintainer

bnevis-i Dec 6, 2023 Maintainer

bnevis-i Dec 6, 2023 Maintainer

bnevis-i Dec 7, 2023 Maintainer

Thinhhoagn0211
Dec 2, 2023

Replies: 1 comment 5 replies

bnevis-i
Dec 3, 2023
Maintainer

bnevis-i Dec 6, 2023
Maintainer

bnevis-i Dec 6, 2023
Maintainer

bnevis-i Dec 6, 2023
Maintainer

bnevis-i Dec 7, 2023
Maintainer