failed fetching KongUpstreamPolicy v3.2.3

[balait4]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Upgraded Kong from v3.5.0 to v.3.7.1 and KIC from 3.0.2 to 3.2.3. Before upgrade we are able to use KongUpstreamPolicy, after upgrade we see the below error for all KongUpstreamPolicy. We have db-less setup and performed the manual upgrade by update the crd's and then we updated the new image in deployment.

kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/crd?ref=v3.2.3 | kubectl apply -f -
kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/rbac?ref=v3.2.3 | kubectl apply -f -

 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "northamerica-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "britishisles-least-conn", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-least-conn not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "netherlands-funda-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "westcentraleurope-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}

Expected Behavior

Should see the KongUpstreamPolicy

Steps To Reproduce

DB-less setup upgrade kong from v3.5.0 to v.3.7.1 and KIC from 3.0.2 to 3.2.3.

Kong Ingress Controller version

v3.2.3

Kubernetes version

v1.28.2

Anything else?

No response

[balait4]

Any help on this please?

[gcamus59]

We face the same issue after upgrading to Kong 3.7.1 and KIC 3.2.3. Around 10 Kubernetes services referenced a KongUpstreamPolicy to add a LB strategy with hashOn IP.
In the Kubernetes services events we have the following errors:

Events:
  Type     Reason                              Age                    From         Message
  ----     ------                              ----                   ----         -------
  Warning  KongConfigurationTranslationFailed  54m (x3 over 54m)      kong-client  failed fetching KongUpstreamPolicy: KongUpstreamPolicy apps-sbx/hashon-ip not found
  Warning  KongConfigurationTranslationFailed  53m (x4 over 53m)      kong-client  failed fetching KongUpstreamPolicy: KongUpstreamPolicy apps-sbx/hashon-ip not found
  Warning  KongConfigurationTranslationFailed  3m50s (x4 over 3m59s)  kong-client  failed fetching KongUpstreamPolicy: KongUpstreamPolicy apps-sbx/hashon-ip not found

The KongUpstreamPolicy is well created and available in the same namespace.
We noticed that the KongUpstreamPolicy resource status block is continuously updated with ancestor reordering.

...
Status:
  Ancestors:
    Ancestor Ref:
      Group:      core
      Kind:       Service
      Name:       my-api-report
      Namespace:  apps-sbx
    Conditions:
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Controller Name:         konghq.com/kic-gateway-controller
    Ancestor Ref:
      Group:      core
      Kind:       Service
      Name:       my-api-swagger
      Namespace:  apps-sbx
    Conditions:
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Controller Name:         konghq.com/kic-gateway-controller

[randmonkey]

@gcamus59 For the reordering, we found that in building ancestor status we did not sort the ancestors so the order may change when the ancestor status of the KongUpstreamPolicy needs to be changed.

For the "KongUpstreamPolicy not found" issue, I could not reproduce it with the same KIC and Kong gateway versions. Would you please provide more detail about it?

[balait4]

One thing, I have multiple service consuming the same KUP then we faced this issue. Then I created separate KUP for each service then it worked. So thats the workaround we used for now.

[randmonkey]

@balait4 Thanks. I also tried the scenario with multiple services referencing the same KUP but did not reproduce it. Would you please provide more details about your environment? Like the spec of the services/KUPs.

[balait4]

Here is the KUPs/Service
KUP_non_working_object.txt

[randmonkey]

@balait4 Looks like there are more than 16 services referencing the same KongUpstreamPolicy. I tested with such case but cannot reproduce the "upstream policy not found" error and all 17 services are correctly configured with the upstream policy. While there can be at most 16 items in status.ancestors and newest one will be ignored. So some of the services may not seen in the status of KongUpstreamPolicy.

Have you seen any other unexpected logs in your KIC pod? Also, please provide some further details of your service and KIC deployment.

[balait4]

Here is the debug log of my KIC and kong and kong configuration yaml.
debug_log_KIC.txt
debug_log_kong.txt
deployment_yaml.txt

service manifest:

apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/retries: '2'
    konghq.com/upstream-policy: kongingress-apikey-header
    konghq.com/plugins: request-transformer-plugin
#    konghq.com/plugins: request-transformer-plugin, prevent-retry-on-timeout-plugin
    konghq.com/protocol: https
    konghq.com/connect-timeout: "2000"
    konghq.com/read-timeout: "300000"
  name: australia-apikey-header
  namespace: api-proxy
spec:
  clusterIP: None
  ports:
  - name: apps
    port: 8080
    protocol: TCP
    targetPort: 8080
  type: ClusterIP