Why keda-operator crash everytime it tries to startup ? #1754

rajarajan-s-imanage · 2021-04-22T16:11:58Z

rajarajan-s-imanage
Apr 22, 2021

Hi,

I'm seeing a crash of keda-operator, I'm not sure if this is an issue or if it's a simple configuration that I've missed.

I've deployed KEDA using
kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.2.0/keda-2.2.0.yaml

I'm using an Azure Service Bus Scaler

Please let me know if you need more information on this. Thanks

Here's my TriggerAuth and ScaledObject

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: {{ .Values.app.name }}-trigger-auth
  namespace: {{ .Release.Namespace }}
spec:
  hashiCorpVault:
    address: {{ .Values.vault.address }}
    authentication: kubernetes
    role: {{ .Values.keda.namespace }}-keda-role
    mount: "kubernetes"
    credential:
      serviceAccount: "/var/run/secrets/kubernetes.io/serviceaccount/token"
    secrets:
      - parameter: connection
        key: "secret"
        path: "kv/data/{{ .Release.Namespace }}/crack-task/manage"
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ .Values.app.name }}-keda-autoscaler
  namespace: {{ .Release.Namespace }}
spec:
  scaleTargetRef:
    name: {{ .Values.app.name }}
  pollingInterval: 10
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  maxReplicaCount: {{ .Values.keda.maxReplicaCount }}
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: {{.Release.Namespace}}-crack-task
        messageCount: {{ .Values.keda.messageCount | int | quote}}
      authenticationRef:
        name: {{ .Values.app.name }}-trigger-auth

keda-operator log

2021-04-22T15:57:40.740Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "source": "kind source: /, Kind="}
2021-04-22T15:57:40.740Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "source": "kind source: /, Kind="}
2021-04-22T15:57:40.740Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-04-22T15:57:40.740Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "source": "kind source: /, Kind="}
2021-04-22T15:57:40.840Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication"}
2021-04-22T15:57:40.840Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "worker count": 1}
2021-04-22T15:57:40.840Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication"}
2021-04-22T15:57:40.840Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-04-22T15:57:40.840Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-04-22T15:57:40.841Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "worker count": 1}
2021-04-22T15:57:40.941Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject"}
2021-04-22T15:57:40.941Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "worker count": 1}
2021-04-22T15:57:40.941Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "worker count": 1}
2021-04-22T15:57:40.941Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "ku-smoke", "ScaledObject.Name": "ku-cracking-service-keda-autoscaler"}
2021-04-22T15:57:40.941Z	INFO	controllers.ScaledObject	Creating a new HPA	{"ScaledObject.Namespace": "ku-smoke", "ScaledObject.Name": "ku-cracking-service-keda-autoscaler", "HPA.Namespace": "ku-smoke", "HPA.Name": "keda-hpa-ku-cracking-service-keda-autoscaler"}
E0422 15:58:01.846297       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 448 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2348d40, 0x3c45860)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x2348d40, 0x3c45860)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRef(0x2b57360, 0xc00098d6b0, 0x2b4aca0, 0xc00116bcf0, 0xc000e9a940, 0xc0013829f8, 0xc000e88ab0, 0x8, 0x8, 0xc000528de0, ...)
/workspace/pkg/scaling/resolver/scale_resolvers.go:99 +0x80d
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers(0xc00018f900, 0xc000338280, 0xc001382900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/workspace/pkg/scaling/scale_handler.go:355 +0x3b3
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalers(0xc00018f900, 0x266c240, 0xc00000a780, 0xc0007fcc00, 0x106, 0x400, 0xc000961530, 0x106)
/workspace/pkg/scaling/scale_handler.go:76 +0xee
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).getScaledObjectMetricSpecs(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0x0, 0x255cf60, 0x12, 0x0, 0x0)
/workspace/controllers/hpa.go:146 +0x65
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).newHPAForScaledObject(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0xc000961b10, 0x4, 0x2c, 0x0)
/workspace/controllers/hpa.go:47 +0x6a
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).createAndDeployNewHPA(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0xc000961b10, 0xc0005421b0, 0x2c)
/workspace/controllers/hpa.go:30 +0x225
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0xc000961b10, 0x4, 0x26c60a4, 0x2)
/workspace/controllers/scaledobject_controller.go:316 +0x35e
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).reconcileScaledObject(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0x0, 0x0, 0x23, 0x2afd4e0)
/workspace/controllers/scaledobject_controller.go:204 +0x151
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).Reconcile(0xc000878e60, 0xc000e88ab0, 0x8, 0xc001008360, 0x23, 0xc000d96590, 0xc00091e480, 0xc000576008, 0xc000576000)
/workspace/controllers/scaledobject_controller.go:160 +0x405
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a47830, 0x24100e0, 0xc0002f0000, 0x0)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244 +0x2a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a47830, 0x203000)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000a47830)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000d96560)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000d96560, 0x2aec320, 0xc0005906c0, 0x1, 0xc00064aae0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000d96560, 0x3b9aca00, 0x0, 0x1, 0xc00064aae0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000d96560, 0x3b9aca00, 0xc00064aae0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:179 +0x416
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x20cfc0d]
goroutine 448 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x2348d40, 0x3c45860)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRef(0x2b57360, 0xc00098d6b0, 0x2b4aca0, 0xc00116bcf0, 0xc000e9a940, 0xc0013829f8, 0xc000e88ab0, 0x8, 0x8, 0xc000528de0, ...)
/workspace/pkg/scaling/resolver/scale_resolvers.go:99 +0x80d
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers(0xc00018f900, 0xc000338280, 0xc001382900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/workspace/pkg/scaling/scale_handler.go:355 +0x3b3
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalers(0xc00018f900, 0x266c240, 0xc00000a780, 0xc0007fcc00, 0x106, 0x400, 0xc000961530, 0x106)
/workspace/pkg/scaling/scale_handler.go:76 +0xee
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).getScaledObjectMetricSpecs(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0x0, 0x255cf60, 0x12, 0x0, 0x0)
/workspace/controllers/hpa.go:146 +0x65
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).newHPAForScaledObject(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0xc000961b10, 0x4, 0x2c, 0x0)
/workspace/controllers/hpa.go:47 +0x6a
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).createAndDeployNewHPA(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0xc000961b10, 0xc0005421b0, 0x2c)
/workspace/controllers/hpa.go:30 +0x225
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0xc000961b10, 0x4, 0x26c60a4, 0x2)
/workspace/controllers/scaledobject_controller.go:316 +0x35e
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).reconcileScaledObject(0xc000878e60, 0x2b4aca0, 0xc000d965c0, 0xc00000a780, 0x0, 0x0, 0x23, 0x2afd4e0)
/workspace/controllers/scaledobject_controller.go:204 +0x151
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).Reconcile(0xc000878e60, 0xc000e88ab0, 0x8, 0xc001008360, 0x23, 0xc000d96590, 0xc00091e480, 0xc000576008, 0xc000576000)
/workspace/controllers/scaledobject_controller.go:160 +0x405
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a47830, 0x24100e0, 0xc0002f0000, 0x0)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244 +0x2a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a47830, 0x203000)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000a47830)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000d96560)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000d96560, 0x2aec320, 0xc0005906c0, 0x1, 0xc00064aae0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000d96560, 0x3b9aca00, 0x0, 0x1, 0xc00064aae0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000d96560, 0x3b9aca00, 0xc00064aae0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:179 +0x416

Answered by zroubalik

Apr 29, 2021

Hi, it seems like the HashiCorpVault handling is broken, my guess the path is not properly resolved? This problem should be correctly handled and not result panic though.

View full answer

zroubalik · 2021-04-29T09:49:24Z

zroubalik
Apr 29, 2021
Maintainer

Hi, it seems like the HashiCorpVault handling is broken, my guess the path is not properly resolved? This problem should be correctly handled and not result panic though.

5 replies

zroubalik Apr 29, 2021
Maintainer

It it failing here:

keda/pkg/scaling/resolver/scale_resolvers.go

Lines 92 to 99 in d46a336

    
           secret, err := vault.Read(e.Path) 
        
           if err != nil { 
        
           	logger.Error(err, "Error trying to read secret from Vault", "triggerAuthRef.Name", triggerAuthRef.Name, 
        
           		"secret.path", e.Path) 
        
           	continue 
        
           } 
        
           result[e.Parameter] = resolveVaultSecret(logger, secret.Data, e.Key)

My guess is, that the vault.Read(e.Path) call doesn't return any secret or error, caused probably by this condition:

https://github.com/hashicorp/vault/blob/afd9e759ca82f48444e43428605d3ecc0862b6e1/api/logical.go#L81

The hashicopr dependedncies should be updated anyway, but I don't have enough resources to test the change, @rajarajan-s-imanage are you willing to contribute this?

rajarajan-s-imanage May 6, 2021
Author

Hi @zroubalik, thanks for looking into this.

Actually, I think your guess is right, the secret was not existing on the path that I've specified in the configuration.
I can confirm that the issue didn't happen when we put the secret on the path mentioned in TriggerAuthentication spec.

Sorry for getting back late on this, I'm currently held up with other stuff.

zroubalik May 18, 2021
Maintainer

@rajarajan-s-imanage do you think that you can ping me on Slack? I'd like to talk about this issue, would be great if you can give me more input, so I can try to fix the error handling and error message for this specific case. Thanks!

rajarajan-s-imanage May 20, 2021
Author

Hey @zroubalik, Sure yes. I could definitely help you with this.
[email protected]
Please send an invite to your workspace. I'll reach out to you right away.

zroubalik May 20, 2021
Maintainer

@rajarajan-s-imanage I have sent you invite to https://slack.k8s.io/ #keda channel. If you don't receive the invite, feel free to join that slack and ping.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why keda-operator crash everytime it tries to startup ? #1754

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Why keda-operator crash everytime it tries to startup ? #1754

rajarajan-s-imanage Apr 22, 2021

Replies: 1 comment · 5 replies

zroubalik Apr 29, 2021 Maintainer

zroubalik Apr 29, 2021 Maintainer

rajarajan-s-imanage May 6, 2021 Author

zroubalik May 18, 2021 Maintainer

rajarajan-s-imanage May 20, 2021 Author

zroubalik May 20, 2021 Maintainer

rajarajan-s-imanage
Apr 22, 2021

Replies: 1 comment 5 replies

zroubalik
Apr 29, 2021
Maintainer

zroubalik Apr 29, 2021
Maintainer

rajarajan-s-imanage May 6, 2021
Author

zroubalik May 18, 2021
Maintainer

rajarajan-s-imanage May 20, 2021
Author

zroubalik May 20, 2021
Maintainer