Azure EventHub - unable to get eventhub client #1618

hgreving · 2021-02-16T17:38:52Z

hgreving
Feb 16, 2021

The Azure EventHub scaler documentation I found on keda.sh is not clear about what to configure when using PodIdentity. It only tells to configure: eventHubName and eventHubNamespace when using Azure as PodIdentity provider. The HPA is created but there is no scaling happening due to the following error in the KEDA operator log:

2021-02-16T17:07:25.578Z ERROR controller Reconciler error {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "example-keda-scaled-object", "namespace": "example", "error": "error getting scaler for trigger #0: unable to get eventhub client: failed to refersh token: &{{{ 0 0 0 } 0x3a45858 {{ <nil> false } { <nil> false } {http <nil> 169.254.169.254 /metadata/identity/oauth2/token false api-version=2018-02-01&resource=https%3A%2F%2Feventhubs.azure.net%2F } { <nil> false }} https://eventhubs.azure.net/ true 300000000000} 0xc000abb760 0xc0008689f0 <nil> [] 5}"}
github.com/go-logr/zapr.(*zapLogger).Error

Also found the following messages in the scaledobject:

    - message: Error checking metric name uniqueness
      reason: ScaledObjectCheckFailed
      status: 'False'
      type: Ready
    - message: ScaledObject check failed
      reason: UnkownState
      status: Unknown
      type: Active

I hope someone could point me in the right direction on how to configure it correctly?

I installed KEDA v2.1.0.

Here some manifests:

TriggerAuthentication.yaml:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: example-keda-trigger-auth
  namespace: example
spec:
  podIdentity:
    provider: azure

ScaledObject.yaml:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: example-keda-scaled-object
  namespace: example
spec:
  maxReplicaCount: 4
  minReplicaCount: 1
  scaleTargetRef:
    name: example-keda-deployment
  triggers:
    authenticationRef:
      name: example-keda-trigger-auth
    metadata:
      eventHubName: example-keda
      eventHubNamespace: dev-example-keda-log-ns
      storageConnectionFromEnv: StorageConnection
      unprocessedEventThreshold: '10'
    type: azure-eventhub

Deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-keda-deployment
  namespace: example
spec:
  selector:
    matchLabels:
      app: example-keda
  template:
    metadata:
      labels:
        app: example-keda
        aadpodidbinding: keda-scaler-identity-selector
    spec:
      containers:
      - name: example-keda
        image: --removed image--
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        envFrom:
        - configMapRef:
            name: example-keda-config
        - secretRef:
            name: example-keda-secret

tomkerkhove · 2021-02-17T06:33:51Z

tomkerkhove
Feb 17, 2021
Collaborator

@ahmelsayed Do you have an idea?

0 replies

hgreving · 2021-02-17T14:09:08Z

hgreving
Feb 17, 2021
Author

I found a misconfiguration with the managed identities permissions. After fixing that I have the following error in the keda operator log:

2021-02-17T13:56:18.778Z ERROR controller Reconciler error {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "example-keda-scaled-object", "namespace": "example", "error": "error getting scaler for trigger #0: unable to get eventhub metadata: no storage connection string given"}

and the ScaledObject has the same status:

status:
  conditions:
    - message: Error checking metric name uniqueness
      reason: ScaledObjectCheckFailed
      status: 'False'
      type: Ready
    - message: ScaledObject check failed
      reason: UnkownState
      status: Unknown
      type: Active

I don't know yet why the ScaledObjectCheck failed?

10 replies

hgreving Feb 18, 2021
Author

Operator logs:

ERROR controller Reconciler error {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "example-keda-scaled-object", "namespace": "example", "error": "error getting scaler for trigger #0: unable to get eventhub client: failed to refersh token: &{{{ 0 0 0 } 0x3a45858 {{ <nil> false } { <nil> false } {http <nil> 169.254.169.254 /metadata/identity/oauth2/token false api-version=2018-02-01&resource=https%3A%2F%2Feventhubs.azure.net%2F } { <nil> false }} https://eventhubs.azure.net/ true 300000000000} 0xc000c87680 0xc0008689f0 <nil> [] 5

tomkerkhove Feb 18, 2021
Collaborator

Are those the only logs you have?

tomkerkhove Feb 18, 2021
Collaborator

Adding @ahmelsayed who might know what's up

tomkerkhove Feb 18, 2021
Collaborator

Created #1623 to optimize the blob container validation

hgreving Feb 18, 2021
Author

More keda operator logs

2021-02-18T13:59:09.838Z INFO controllers.ScaledObject Reconciling ScaledObject {"ScaledObject.Namespace": "example", "ScaledObject.Name": "example-keda-scaled-object"}
2021-02-18T14:07:41.917Z ERROR controllers.ScaledObject Unable to fetch scalers in metric name uniqueness check {"ScaledObject.Namespace": "example", "ScaledObject.Name": "example-keda-scaled-object", "error": "error getting scaler for trigger #0: unable to get eventhub client: failed to refersh token: &{{{ 0 0 0 } 0x3a45858 {{ <nil> false } { <nil> false } {http <nil> 169.254.169.254 /metadata/identity/oauth2/token false api-version=2018-02-01&resource=https%3A%2F%2Feventhubs.azure.net%2F } { <nil> false }} https://eventhubs.azure.net/ true 300000000000} 0xc000e2f160 0xc0008689f0 <nil> [] 5}"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).validateMetricNameUniqueness
/workspace/controllers/scaledobject_controller.go:246
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).reconcileScaledObject
/workspace/controllers/scaledobject_controller.go:193
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).Reconcile
/workspace/controllers/scaledobject_controller.go:157
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-02-18T14:07:41.917Z ERROR controllers.ScaledObject Error checking metric name uniqueness {"ScaledObject.Namespace": "example", "ScaledObject.Name": "example-keda-scaled-object", "error": "error getting scaler for trigger #0: unable to get eventhub client: failed to refersh token: &{{{ 0 0 0 } 0x3a45858 {{ <nil> false } { <nil> false } {http <nil> 169.254.169.254 /metadata/identity/oauth2/token false api-version=2018-02-01&resource=https%3A%2F%2Feventhubs.azure.net%2F } { <nil> false }} https://eventhubs.azure.net/ true 300000000000} 0xc000e2f160 0xc0008689f0 <nil> [] 5}"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).Reconcile
/workspace/controllers/scaledobject_controller.go:160
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-02-18T14:07:41.956Z ERROR controller Reconciler error {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "example-keda-scaled-object", "namespace": "example", "error": "error getting scaler for trigger #0: unable to get eventhub client: failed to refersh token: &{{{ 0 0 0 } 0x3a45858 {{ <nil> false } { <nil> false } {http <nil> 169.254.169.254 /metadata/identity/oauth2/token false api-version=2018-02-01&resource=https%3A%2F%2Feventhubs.azure.net%2F } { <nil> false }} https://eventhubs.azure.net/ true 300000000000} 0xc000e2f160 0xc0008689f0 <nil> [] 5}"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure EventHub - unable to get eventhub client #1618

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 10 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Azure EventHub - unable to get eventhub client #1618

hgreving Feb 16, 2021

Replies: 2 comments · 10 replies

tomkerkhove Feb 17, 2021 Collaborator

hgreving Feb 17, 2021 Author

hgreving Feb 18, 2021 Author

tomkerkhove Feb 18, 2021 Collaborator

tomkerkhove Feb 18, 2021 Collaborator

tomkerkhove Feb 18, 2021 Collaborator

hgreving Feb 18, 2021 Author

hgreving
Feb 16, 2021

Replies: 2 comments 10 replies

tomkerkhove
Feb 17, 2021
Collaborator

hgreving
Feb 17, 2021
Author

hgreving Feb 18, 2021
Author

tomkerkhove Feb 18, 2021
Collaborator

tomkerkhove Feb 18, 2021
Collaborator

tomkerkhove Feb 18, 2021
Collaborator

hgreving Feb 18, 2021
Author