Make sure KIC in Konnect works with Kuma Mesh

[czeslavo]

Is there an existing issue for this?

• I have searched the existing issues

Does this enhancement require public documentation?

• I have added an Acceptance Criteria item for adding and/or adjusting public documentation (if applicable)

Problem Statement

No tests and documentation are covering that KIC with Konnect integration enabled works with Kuma Mesh. We'd like to:

• ensure what is required for KIC to communicate with public Konnect APIs when:
  • Kuma is installed and global passthrough mode is disabled
  • Kuma is installed and global passthrough mode is enabled
• provide users with documentation that will explain how to enable KIC in Konnect in the above setups

Proposed Solution

• Test the above scenarios with Kuma Mesh manually and prepare docs that explain how to use KIC in Konnect with Kuma Mesh with different settings
• Add an automated E2E test in KIC repository that covers Kuma and Konnect integration scenarios

Additional information

Kuma issue tracking this: https://github.com/Kong/kong-mesh/issues/6821

Acceptance Criteria

• As a KIC user, I can refer to a docs site to learn how to use KIC in Konnect in environments with Kuma Mesh installed where
  • global passthrough is enabled
  • global passthrough is disabled
• kong/ingress Helm chart doesn't annotate the controller's Pod with kuma.io/gateway
• Automated E2E tests are added that ensure Kuma and KIC in Konnect integration interoperability

[czeslavo]

I was able to make KIC+Konnect+Kuma with the global passthrough disabled and in-mesh mTLS enabled BUT workarounds that I had to employ are:

1. Change the Kong Gateway Pod’s readiness status check to /status endpoint. It returns 200s once the Gateway boots up, while we’d rather prefer to use /status/ready that waits for the Gateway to receive a non-empty configuration from the controller (so the proxy service is not served by instances with no config). In the controller, we run our own service discovery by looking at the Admin API service EndpointSlices and using those despite their status (well, we ignore the terminating ones) so we can configure the not ready instances and make them ready effectively by doing so. It appears it was a blocker for us before as Mesh, as I understand, doesn’t create inbound listeners for Pods that are not ready yet and we couldn’t establish a connection.

   It would be nice if there was a possibility in Kuma to enforce creating listeners despite the status using some annotation. Or maybe there already is? 🤔

2. A MeshPassthrough match for Kubernetes API server had to be added and I couldn’t find a better solution than putting a hardcoded IP there (CIDR could be better, but it would still be matching Pod network which is not desired).

   Is there any better way to make sure the controller deployed in Mesh can communicate with the Kube API freely?

3. As the Kong Gateway exposes not only a proxy service, but also the aformentioned Admin API, we shouldn’t annotate it with kuma.io/gateway: enabled I believe as we’d like to be able to intercept traffic between the controller and the Gateway. Because of that I went with traffic.kuma.io/exclude-inbound-ports: 8000,8443 only for the proxy ports.

   It works, but I wonder - is there any significant difference between these two approaches? How much kuma.io/gateway: enabled differs from the traffic.kuma.io/exclude-inbound-ports?

[czeslavo]

For the context/reproduction, I used Kuma v2.9.0 installed with Helm:

helm install --create-namespace --namespace kuma-system kuma kuma/kuma

Kuma objects:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    backends:
    - conf:
        caCert:
          RSAbits: 2048
          expiration: 10y
      dpCert:
        rotation:
          expiration: 1d
      name: ca-1
      type: builtin
    enabledBackend: ca-1
---
apiVersion: kuma.io/v1alpha1
kind: MeshPassthrough
metadata:
  name: disable-passthrough
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
    proxyTypes:
    - Sidecar
  default:
    passthroughMode: Matched
    appendMatch:
    - type: Domain
      value: "*.konghq.tech" # For Konnect APIs
      protocol: tls
      port: 443
    - type: IP
      value: 192.168.194.129 # For Kubernetes API server
      protocol: tls
      port: 443
---
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: allow-all
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  from:
  - targetRef:
      kind: Mesh
    default:
      action: Allow

KIC was installed with kong/ingress Helm chart:

• Gateway and Controller Pods' annotations were modified to not have kuma.io/gateway: enabled.
• The Gateway Pod was annotated with traffic.kuma.io/exclude-inbound-ports: 8000,8443 for proxy traffic.
• The Gateway's readiness probe was changed from /status/ready to /status.