Istio - SPIRE Integration : Istio can't fetch file-cert based TLS certificates from spire agent SDS socket

[sridharjprakasam]

Istio version: 1.23.12
Spire Version : 1.8.7
Platform : RHEL 8

**Issue Description

When SPIRE is configured to provide workload identity it replaces the default sds-cluster. However for ingress gateway, the source of the TLS certs is also moved to sds-cluster. The default SDS client, when SPIRE workload api socket is not mount, understand the specific resource name file-cert:~. However, once it is replaced by SPIRE, the above resource name is passed onto the spire-agent, which is not able to resolve it.**

This breaks the file based TLS SIMPLE mode configuration when SPIRE is the workload identity issuer.

Gateway TLS

 tls:
        mode: SIMPLE
        privateKey: /etc/ingress-certs/tls.key
        serverCertificate: /etc/ingress-certs/tls.crt

Envoy Config Dump - Downstream TLS context

"tls_certificate_sds_secret_configs": [
             {
              "name": "file-cert:/etc/ingress-certs/tls.crt~/etc/ingress-certs/tls.key",
              "sds_config": {
               "api_config_source": {
                "api_type": "GRPC",
                "grpc_services": [
                 {
                  "envoy_grpc": {
                   "cluster_name": "sds-grpc"
                  }
                 }
                ],
                "set_node_on_first_message_only": true,
                "transport_api_version": "V3"
               },
               "resource_api_version": "V3"
              }
             }
            ]

Error in ingress gateway logs, istio tries to make use of spire agent socket to fetch the file-cert based certificate

warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 3, workload is not authorized for the requested identities ["file-cert:/etc/ingress-certs/tls.crt~/etc/ingress-certs/tls.key"] thread=18

Is it possible for spire agent to recognise that the request is for file-cert ( local path based cert ) and not an SVID for workload and deliver the requested file cert from spire-agent's local path

Related Istio Issue : istio/istio#54365

[MarcosDY]

Hello @sridharjprakasam,
Thanks for creating this issue!

Could this be related to #5687 and the recent feature change where the default value of the VERIFY_CERTIFICATE_AT_CLIENT flag has been set to true?

[sridharjprakasam]

@MarcosDY
This recent feature change of defaulting VERIFY_CERT_AT_CLIENT=true is that istio sidecar proxy (as a client) will always try to verify the upstream server certificate using the sidecar's OS level CA trust if a CA explicitly is not specified.

The issue described here is that istio proxy (as a ingress-gateway) couldn't fetch the TLS certificates for TLS termination at gateway level, when integrated with spire. Istio is attempting to fetch a file-cert based certificate available in its own file system from the spire agent socket, as Istio binds to it as it sees it mounted on in its well known path var/run/secrets/workload-spiffe-uds/socket during its startup

client --(https)--> Istio Gateway Listener [TLS cert] <---(istio-sds)---> Spire agent

Moreover the same issue is there in Istio 1.19.7 while integrating with SPIRE

[bleggett]

This is not a SPIRE issue, it's just an Istio/Envoy config snafu.

[bleggett]

FWIW this will be fixed Istio-side via istio/istio#54725 by separating the file-based SDS config from the workload-CA/SPIRE based SDS config.

[evan2645]

Thank you @bleggett 🙏

Can you help us understand the impact? Are there now Istio releases that are incompatible with SPIRE?

[bleggett]

The tl;dr is that

• if you are using SPIRE to supply workload certs to your ingress gateways
• if you are using file-based TLS SIMPLE mode for that gateway

that would break prior to istio/istio#54725

That is what this issue was raised for.

I suspect #5687 is more or less the same issue as this, and the PR will fix both.