consul: no service named core-metadata getting this error on device service

[ankit-4129]

Hi Team,
we have written a device service using edgex C SDK 2.2.0.
We have verified that our service is able to read key-value from consul.

But we are getting empty response when trying to get 'http://edge-consul:8500/v1/catalog/service/core-metadata' using same deployment configuration.

we have added below environment:
SECRETSTORE_TOKENFILE: "/usr/local/share/secrets/edge-device-service-usb-webcam/secrets-token.json"
CONSUL_HTTP_TOKEN_FILE: "/tmp/edgex/secrets/consul-acl-token/consul_http_token"
and we are able to see this logs:

Starting USB Webcam Device Service
In constructor
Calling edgex_sdk_init()
In edgex_sdk_init
Calling devsdk_service_new
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="iot_threadpool_alloc (threads: 8 max_jobs: 0 default_priority: -1 affinity: -1)"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-0 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-1 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-2 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-3 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-4 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-5 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-6 starting"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Thread iot-0-7 starting"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="iot_scheduler_alloc (priority: -1 affinity: -1)"
Calling devsdk_service_start
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config MessageQueue/Host = edge-redis"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config SecretStore/Host = edge-vault"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config SecretStore/Port = 8200"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config SecretStore/TokenFile = /usr/local/share/secrets/edge-device-service-usb-webcam/secrets-token.json"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config Service/Host = edge-device-service-usb-webcam"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config Writable/LogLevel = DEBUG"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="vault: scheduled token refresh in 3180 seconds"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Found registry service at consul.http://edge-consul:8500"
level=DEBUG ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="iot_threadpool_add_work jobs/max: 1/4294967295"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config MessageQueue/Host = edge-redis"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config SecretStore/Host = edge-vault"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config SecretStore/Port = 8200"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config SecretStore/TokenFile = /usr/local/share/secrets/edge-device-service-usb-webcam/secrets-token.json"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config Service/Host = edge-device-service-usb-webcam"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Override config Writable/LogLevel = DEBUG"
level=INFO ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="Reconfiguring"
level=ERROR ts=2023-11-28T06:51:52Z app=edge-device-service-usb-webcam msg="consul: no service named core-metadata"

and the service gets stuck on this log consul: no service named core-metadata.

We use manual steps to get consul token from vault try getting results from 'http://edge-consul:8500/v1/catalog/service/core-metadata' then we are getting empty list in response.

Below are steps:
Get the Secret token from vault secrets-token.json:

{"request_id":"f5dd3742-6456-729d-18b6-7024f4919017","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"hvs.CAESID48mxdpdUEplzKME0pwkQaluQBvBajXV4qr_hsaEPaaGh4KHGh2cy5FY01mOTUzQWJlNE5kam80aWxWYUZqeG4","accessor":"Fkd6vuNDBOf1pE9njXF55J5J","policies":["default","edge-device-service-usb-webcam"],"token_policies":["default","edge-device-service-usb-webcam"],"metadata":null,"lease_duration":3600,"renewable":true,"entity_id":"","token_type":"service","orphan":true,"mfa_requirement":null,"num_uses":0}}

Get the consul creds of device-service-usb-webcam from vault:

curl --request GET \
  --url http://edge-vault:8200/v1/consul/creds/edge-device-service-usb-webcam \
  --header 'X-Vault-Token: hvs.CAESID48mxdpdUEplzKME0pwkQaluQBvBajXV4qr_hsaEPaaGh4KHGh2cy5FY01mOTUzQWJlNE5kam80aWxWYUZqeG4'

{"request_id":"a48ec5aa-7425-f815-dadb-be41d2997a8e","lease_id":"consul/creds/edge-device-service-usb-webcam/x5QubKlDnav2nEY6hIhwRhZD","renewable":true,"lease_duration":604800,"data":{"accessor":"361697ad-d693-eaf6-1e5b-769615add4bb","consul_namespace":"","local":false,"partition":"","token":"521c5aad-cda2-fd0c-b6ac-256b85b3e33a"},"wrap_info":null,"warnings":null,"auth":null}

Try getting core-metadata details using consul token of device-service-usb-webcam:

sh-5.1# curl --header "X-Consul-Token: 521c5aad-cda2-fd0c-b6ac-256b85b3e33a"  http://edge-consul:8500/v1/catalog/service/core-metadata
[]

And then we tried getting response from this API http://edge-consul:8500/v1/catalog/service/core-metadata using token provided in
CONSUL_HTTP_TOKEN_FILE: "/tmp/edgex/secrets/consul-acl-token/consul_http_token" then we are able to get desired result.

Get the consul creds from CONSUL_HTTP_TOKEN_FILE:

{"SecretID":"2405fab0-798a-9ed2-9b90-9ebce3a6680e","Policies":[{"ID":"00000000-0000-0000-0000-000000000001","Name":"global-management"}]}

Try getting core-metadata details using consul token from CONSUL_HTTP_TOKEN_FILE:

sh-5.1# curl --header "X-Consul-Token: 2405fab0-798a-9ed2-9b90-9ebce3a6680e"  http://edge-consul:8500/v1/catalog/service/core-metadata

[{"ID":"f805fdd0-ac25-a1b1-518c-19b538b25238","Node":"edge-consul","Address":"10.89.0.48","Datacenter":"dc1","TaggedAddresses":null,"NodeMeta":null,"ServiceKind":"","ServiceID":"core-metadata","ServiceName":"core-metadata","ServiceTags":[],"ServiceAddress":"edgex-metadata","ServiceWeights":{"Passing":1,"Warning":1},"ServiceMeta":{},"ServicePort":59881,"ServiceSocketPath":"","ServiceEnableTagOverride":false,"ServiceProxy":{"Mode":"","MeshGateway":{},"Expose":{}},"ServiceConnect":{},"CreateIndex":108,"ModifyIndex":108}]

Looks like service is not using token from CONSUL_HTTP_TOKEN_FILE and then it is not able to find service core-metadata.
Are we missing any policy in token?

[jiekechoo]

@ankit-4129
I don't understand your purpose. You wrote a device service, but it can't work in secure mode?
Could you provide more information? docker-compose.yml, device service configuration, anything.

[ankit-4129]

we are not using docker-compose instead we are using podman and to deploy containers we are using libpod APIs, vault token we are generating it manually.
and rest of the details of the issue we are facing on our custom device service is mentioned above.

can you please helps with the steps we could take to get more information about the error we are getting?

consul: no service named core-metadata

[jiekechoo]

I have two suggestions for you:

1. upgrade to v3.1.0 as @lenny-intel mentioned.
2. use docker compose as your test env, then migrate to podman.

[bnevis-i]

Looks like service is not using token from CONSUL_HTTP_TOKEN_FILE and then it is not able to find service core-metadata.

The missing information is that in secure mode, EdgeX services get their consul token from Vault. How this works is that when Consul is initialized, it gets a management token that has permission to issue other tokens, and this token is configured into Vault as the consul secrets provider. The lifetime of the two are tied together, so that if a service allows its Vault token to expire, Vault also revokes the consul token as well. CONSUL_HTTP_TOKEN_FILE is used for the consul CLI and EdgeX does not use it, and your evidence seems to suggest that the consul libraries won't check to to override the token EdgeX is passing.

Additionally, make sure you closely read
https://docs.edgexfoundry.org/2.3/security/Ch-Configuring-Add-On-Services/#optional-configure-the-acl-role-of-configurationregistry-to-use-if-the-service-depends-on-it

I think already set ADD_SECRETSTORE_TOKENS, but you also need to make sure you are using ADD_REGISTRY_ACL_ROLES in order to create the consul role that the consul secrets engine to map to. Also a concern, possibly, is the service naming. Make sure your app services are named 'app-' and devices are named 'device-' as there are some hardcoded rules in some spots of EdgeX. 'edge-device-service-usb-webcam' is compliant with neither.

[ankit-4129]

We have provided ADD_REGISTRY_ACL_ROLES in deployment file of edgex-consul and for ADD_SECRETSTORE_TOKENS we are doing it manually so we wanted to understand the flow of policies and roles.

"secret/edgex/" + serviceName + "/*": {
		"capabilities": []string{"create", "update", "delete", "list", "read"},
	},
	"consul/creds/" + serviceName: {
		"capabilities": []string{"read"},
	},

Do we need any other policy.

[bnevis-i]

Yeah, that's it, at least for v2.3.0
https://github.com/edgexfoundry/edgex-go/blob/v2.3.0/internal/security/fileprovider/defaults.go

And for the consul acl roles:
https://github.com/edgexfoundry/edgex-go/blob/v2.3.0/internal/security/bootstrapper/command/setupacl/command.go

For EdgeX 3.0, the consul role has an additional rule to read the common config service key, and the vault roles need additional access to identity/oidc/token/service-name and identity/oidc/introspect (for microservices to be able to self-issue and verify JWTs) We also flattened the consul key hierarchy in 3.0 as well to get rid of the core/ support/ app/ etc hierarchy (less "if" statements in the code).

[lenny-goodell]

@ankit-4129 , version 2.2.0 is no longer supported. Please upgrade to using the recently release 3.1.0 (Napa) which is the currently supported release.