insecureSkipTLSVerify causes Out of Sync on argocd

[humutkazan]

Hi all,

I have deployed keda as argocd application to my cluster. Pods seem healthy, haven't tried scaling yet though. But what I wonder is this line seems to be causing out of sync state. insecureSkipTLSVerify=false

What can I do about it? Should I remove it from apiservice yaml, but I read that default value is true? or should I ignore it like below

    ignore:
      - group: apiregistration.k8s.io
        kind: APIService
        jsonPointers:
          - /spec/insecureSkipTLSVerify

Thanks

[asfaltboy]

Also asked in #4732 (comment)

[humutkazan]

changed it to true and now I see this log constantly

ERROR cert-rotation Error updating webhook with certificate {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "error": "APIService.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\" is invalid: spec.insecureSkipTLSVerify: Invalid value: true: may not be true if caBundle is present"}

activation/deactivation still works though. I don't need hpa at the moment but I assume it won't work since metrics server is not healthy.

[humutkazan]

@zroubalik sorry to tag you but since you answered my last question and I can't think of anything else to do.. do you have any suggestion regarding this?

[samuelb]

I experienced the same problem today when I attempted to move from Keda 2.9 to 2.12. ArgoCD kept retrying to update APIService what caused the aad-pod-identity-mic pods to run into a VMSSUpdate rate limit on Azure APIs which negatively impacted services on multiple AKS clusters (pods couldn't start because they could get volumes mounted with the secrets-store-csi-driver). With disabling auto-sync for Keda in ArgoCD, the situation calmed down.

I'm now going with KEDA 2.10 where this problem doesn't happen.

[pashtet04]

I have the same issue with Keda 2.12 and v1.27.4-eks-8ccc7ba

[humutkazan]

Since we haven't received an answer yet, I can tell you what I've tried. Removing that line solved the problem for now. It acts as false by default (AFAIU). CA bundle has been created and there is no out of sync warning. but again not 100% sure if it acts false as default but there is no Invalid value: true: may not be true if caBundle is present error and like I said CA bundle has been created

[eumel8]

same problem with Fleet. Let's try to remove it from the chart

[JorTurFer]

Default value is false, but during the swap from true to false, kubectl didn't remove the field, causing an error during the upgrade because KEDA couldn't patch the apiservice. We will remove this field in next versions as it's totally not necessary and we added it to explicitly set false (to remove the field if it was already set).
We definitively remove the explicit set in next versions, but currently the field can be ignored in ArgoCD config. Setting it to true doesn't work because it conflicts with caBundle. The options are 2, ignore the out-of-sync in argo, or ignore the field explictly.

[humutkazan]

I chose option 3 which is removing the insecureSkipTLSVerify flag from yaml for now since default value is false. thanks

[JorTurFer]

Hello,
That's not posible, sorry. We are working on the fix directly in the upstream and we will remove the flag as soon as the problem is fixed. If we remove the flag now, we can break upgrade processes for users requiring manual actions, and keeping it just as it's now can be annoying but it doesn't require any manual action for having KEDA working.
I apologize if the out-of-sync is annoying for you, but you can just ignore the setting in ArgoCD config

[humutkazan]

Hi,

I don't think I completely understood you. I am using keda charts in my project and I removed insecureSkipTLSVerify flag. argocd is not out of sync anymore and working good for almost 20 days. Are you saying that this solution is not good? I can always re-add that flag and ignore from argocd.

[JorTurFer]

did you remove the flag directly from helm chart? Are you using a fork of KEDA' chart or how can you removed the flag?

[JorTurFer]

My point is that an ArgoCD out-of-sync message is better than not being usable. I'd like to say that we can just remove the flag and that's it, but there is a group of user which can be affected during migration processes, that's why we are contributing with the upstream project to fix it permanently there and then, remove the flag from our helm chart

[humutkazan]

yes I think I got what you are saying but I copied all related yamls from keda github repo and creating my own helm chart

[eumel8]

for Fleet we ended up with this workaround in fleet.yaml

# https://github.com/rancher/fleet/issues/1386
ignore:
  conditions:
   - type: Active
     status: "False"
     reason: ScalerNotActive