Keda pod crashes with HPA not find in version "autoscaling/v2beta2\ #2029

wamak9 · 2021-08-10T21:51:17Z

wamak9
Aug 10, 2021

Report

Keda pod stuck in crashloop with error

2021-08-10T21:38:52.839439439Z I0810 21:38:52.839231       1 request.go:655] Throttling request took 1.047519064s, request: GET:https://10.245.0.1:443/apis/keda.sh/v1alpha1?timeout=32s
2021-08-10T21:38:54.042315492Z 2021-08-10T21:38:54.041Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "HorizontalPodAutoscaler.autoscaling", "error": "no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\""}
2021-08-10T21:38:54.042358992Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
2021-08-10T21:38:54.042367992Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:143
2021-08-10T21:38:54.042373192Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
2021-08-10T21:38:54.042377292Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:184
2021-08-10T21:38:54.042381692Z sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
2021-08-10T21:38:54.042386292Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:676
2021-08-10T21:38:54.042390992Z 2021-08-10T21:38:54.042Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-08-10T21:38:54.042397593Z 2021-08-10T21:38:54.042Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication"}
2021-08-10T21:38:54.042409293Z 2021-08-10T21:38:54.042Z	ERROR	controller	Could not wait for Cache to sync	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "error": "failed to wait for triggerauthentication caches to sync: cache did not sync"}
2021-08-10T21:38:54.042417393Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
2021-08-10T21:38:54.042425593Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:184
2021-08-10T21:38:54.042430893Z sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
2021-08-10T21:38:54.042435093Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:676
2021-08-10T21:38:54.042498693Z 2021-08-10T21:38:54.042Z	ERROR	controller-runtime.manager	error received after stop sequence was engaged	{"error": "failed to wait for triggerauthentication caches to sync: cache did not sync"}
2021-08-10T21:38:54.042524093Z 2021-08-10T21:38:54.042Z	ERROR	setup	problem running manager	{"error": "no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\""}
2021-08-10T21:38:54.042530693Z runtime.main
2021-08-10T21:38:54.042536293Z 	/usr/local/go/src/runtime/proc.go:204

Not sure why its crashing at the moment there are no autoscaling or hpa objects defined

V:2.4.0
Installed using helm no custom settings defined.

Expected Behavior

Shoudnt it be constantly looking for scaledobjects if there are none ?

Actual Behavior

Instead keeps crashing with the error.

Steps to Reproduce the Problem

Install Keda using the helm3
As soon as pods are created keda goes into crashloop

Logs from KEDA operator

Keda Operator logs :

2021-08-10T21:38:32.097938615Z I0810 21:38:32.097358       1 request.go:655] Throttling request took 1.047648966s, request: GET:https://10.245.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
2021-08-10T21:38:33.302371917Z 2021-08-10T21:38:33.302Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-08-10T21:38:33.304565532Z 2021-08-10T21:38:33.304Z	INFO	controllers.ScaledObject	Running on Kubernetes 1.20	{"version": "v1.20.7"}
2021-08-10T21:38:33.304608532Z 2021-08-10T21:38:33.304Z	INFO	setup	Starting manager
2021-08-10T21:38:33.304622832Z 2021-08-10T21:38:33.304Z	INFO	setup	KEDA Version: 2.4.0
2021-08-10T21:38:33.304627732Z 2021-08-10T21:38:33.304Z	INFO	setup	Git Commit:
2021-08-10T21:38:33.304632332Z 2021-08-10T21:38:33.304Z	INFO	setup	Go Version: go1.15.13
2021-08-10T21:38:33.304652232Z 2021-08-10T21:38:33.304Z	INFO	setup	Go OS/Arch: linux/amd64
2021-08-10T21:38:33.304885434Z I0810 21:38:33.304817       1 leaderelection.go:243] attempting to acquire leader lease keda/operator.keda.sh...
2021-08-10T21:38:33.305021035Z 2021-08-10T21:38:33.304Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-08-10T21:38:50.738123983Z I0810 21:38:50.737868       1 leaderelection.go:253] successfully acquired lease keda/operator.keda.sh
2021-08-10T21:38:50.738309984Z 2021-08-10T21:38:50.738Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-08-10T21:38:50.738323384Z 2021-08-10T21:38:50.738Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "source": "kind source: /, Kind="}
2021-08-10T21:38:50.738353285Z 2021-08-10T21:38:50.738Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "source": "kind source: /, Kind="}
2021-08-10T21:38:50.738373385Z 2021-08-10T21:38:50.738Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "source": "kind source: /, Kind="}
2021-08-10T21:38:50.839023691Z 2021-08-10T21:38:50.838Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-08-10T21:38:50.839065691Z 2021-08-10T21:38:50.838Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication"}
2021-08-10T21:38:50.839072891Z 2021-08-10T21:38:50.838Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-08-10T21:38:50.839097391Z 2021-08-10T21:38:50.838Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "worker count": 1}
2021-08-10T21:38:50.839107091Z 2021-08-10T21:38:50.838Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication"}
2021-08-10T21:38:50.839122291Z 2021-08-10T21:38:50.838Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "worker count": 1}
2021-08-10T21:38:52.839439439Z I0810 21:38:52.839231       1 request.go:655] Throttling request took 1.047519064s, request: GET:https://10.245.0.1:443/apis/keda.sh/v1alpha1?timeout=32s
2021-08-10T21:38:54.042315492Z 2021-08-10T21:38:54.041Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "HorizontalPodAutoscaler.autoscaling", "error": "no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\""}
2021-08-10T21:38:54.042358992Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
2021-08-10T21:38:54.042367992Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:143
2021-08-10T21:38:54.042373192Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
2021-08-10T21:38:54.042377292Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:184
2021-08-10T21:38:54.042381692Z sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
2021-08-10T21:38:54.042386292Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:676
2021-08-10T21:38:54.042390992Z 2021-08-10T21:38:54.042Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-08-10T21:38:54.042397593Z 2021-08-10T21:38:54.042Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication"}
2021-08-10T21:38:54.042409293Z 2021-08-10T21:38:54.042Z	ERROR	controller	Could not wait for Cache to sync	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "error": "failed to wait for triggerauthentication caches to sync: cache did not sync"}
2021-08-10T21:38:54.042417393Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
2021-08-10T21:38:54.042425593Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:184
2021-08-10T21:38:54.042430893Z sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
2021-08-10T21:38:54.042435093Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:676
2021-08-10T21:38:54.042498693Z 2021-08-10T21:38:54.042Z	ERROR	controller-runtime.manager	error received after stop sequence was engaged	{"error": "failed to wait for triggerauthentication caches to sync: cache did not sync"}
2021-08-10T21:38:54.042524093Z 2021-08-10T21:38:54.042Z	ERROR	setup	problem running manager	{"error": "no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\""}
2021-08-10T21:38:54.042530693Z runtime.main
2021-08-10T21:38:54.042536293Z 	/usr/local/go/src/runtime/proc.go:204

Keda Metrics-apiserver log :

2021-08-10T21:13:23.064448077Z I0810 21:13:23.064181       1 main.go:101] keda_metrics_adapter "msg"="KEDA Version: 2.4.0"
2021-08-10T21:13:23.070365215Z I0810 21:13:23.064290       1 main.go:102] keda_metrics_adapter "msg"="KEDA Commit: "
2021-08-10T21:13:23.070377515Z I0810 21:13:23.064345       1 main.go:103] keda_metrics_adapter "msg"="Go Version: go1.15.13"
2021-08-10T21:13:23.070383715Z I0810 21:13:23.064356       1 main.go:104] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
2021-08-10T21:13:25.067513859Z I0810 21:13:25.067308       1 request.go:655] Throttling request took 1.047283988s, request: GET:https://10.245.0.1:443/apis/batch/v1beta1?timeout=32s
2021-08-10T21:13:26.270353955Z I0810 21:13:26.270071       1 provider.go:46] keda_metrics_adapter/provider "msg"="starting"
2021-08-10T21:13:26.270399355Z I0810 21:13:26.270135       1 main.go:166] keda_metrics_adapter "msg"="starting adapter..."
2021-08-10T21:13:26.270405155Z 2021-08-10 21:13:26.270134 I | Starting metrics server at :9022
2021-08-10T21:13:26.688866868Z I0810 21:13:26.688615       1 serving.go:325] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
2021-08-10T21:13:26.956132600Z I0810 21:13:26.955866       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
2021-08-10T21:13:26.956153400Z I0810 21:13:26.955915       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
2021-08-10T21:13:26.956159700Z I0810 21:13:26.955918       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2021-08-10T21:13:26.956165100Z I0810 21:13:26.955949       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2021-08-10T21:13:26.956171700Z I0810 21:13:26.955986       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2021-08-10T21:13:26.956177000Z I0810 21:13:26.956011       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2021-08-10T21:13:26.956280501Z I0810 21:13:26.956210       1 dynamic_serving_content.go:130] Starting serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key
2021-08-10T21:13:26.956817804Z I0810 21:13:26.956532       1 secure_serving.go:197] Serving securely on [::]:6443
2021-08-10T21:13:26.956832604Z I0810 21:13:26.956658       1 tlsconfig.go:240] Starting DynamicServingCertificateController
2021-08-10T21:13:27.056699352Z I0810 21:13:27.056311       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2021-08-10T21:13:27.058005560Z I0810 21:13:27.056461       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2021-08-10T21:13:27.058023860Z I0810 21:13:27.056464       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController

KEDA Version

2.4.0

Kubernetes Version

1.20

Platform

Microsoft Azure

Scaler Details

Prometheus

Anything else?

There are no HPA or scaled objects defined.
Previously used 1.5 version and upgraded to the latest version.
Have to remove crds manually based on the documentation provided.

Answered by zroubalik

Aug 11, 2021

Okay, so it needs to be enabled by administrator on that cluster. There's nothing we (KEDA) could do about it, sorry 😐

View full answer

tomkerkhove · 2021-08-11T06:30:23Z

tomkerkhove
Aug 11, 2021
Collaborator

It is having issues creating the HPA, but for some reason it's trying to use autoscaling/v2beta2 instead of autoscaling/v1.

Do you know @zroubalik?

0 replies

zroubalik · 2021-08-11T06:40:43Z

zroubalik
Aug 11, 2021
Maintainer

KEDA is using autoscaling/v2beta2, this version was introduced in k8s ~1.12-1.13.

What kubectl api-versions | grep autoscaling gives you?

0 replies

wamak9 · 2021-08-11T16:18:46Z

wamak9
Aug 11, 2021
Author

@zroubalik -- Seems like there is no autoscaling api
Below is the output I get

admission.voyager.appscode.com/v1beta1
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
azmon.container.insights/v1
batch/v1
batch/v1beta1
cassandra.datastax.com/v1beta1
certificates.k8s.io/v1
certificates.k8s.io/v1beta1
coordination.k8s.io/v1
coordination.k8s.io/v1beta1
discovery.k8s.io/v1beta1
events.k8s.io/v1
events.k8s.io/v1beta1
extensions/v1beta1
external.metrics.k8s.io/v1beta1
flowcontrol.apiserver.k8s.io/v1beta1
keda.sh/v1alpha1
linkerd.io/v1alpha1
linkerd.io/v1alpha2
metrics.k8s.io/v1beta1
networking.k8s.io/v1
networking.k8s.io/v1beta1
node.k8s.io/v1
node.k8s.io/v1beta1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1
scheduling.k8s.io/v1beta1
split.smi-spec.io/v1alpha1
storage.k8s.io/v1
storage.k8s.io/v1beta1
tap.linkerd.io/v1alpha1
v1
voyager.appscode.com/v1beta1

We are on 1.20 k8s version.

0 replies

zroubalik · 2021-08-11T16:40:23Z

zroubalik
Aug 11, 2021
Maintainer

Okay, so it needs to be enabled by administrator on that cluster. There's nothing we (KEDA) could do about it, sorry 😐

0 replies

wamak9 · 2021-08-11T18:17:09Z

wamak9
Aug 11, 2021
Author

@zroubalik -- you are right.
Now i need to figure out a way to enable those apis
thanks for looking into the issue.

0 replies

zroubalik · 2021-08-11T18:29:05Z

zroubalik
Aug 11, 2021
Maintainer

@akramwaseem happy to help. Good luck!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keda pod crashes with HPA not find in version "autoscaling/v2beta2\ #2029

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Keda pod crashes with HPA not find in version "autoscaling/v2beta2\ #2029

wamak9 Aug 10, 2021

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

Anything else?

Replies: 6 comments

tomkerkhove Aug 11, 2021 Collaborator

zroubalik Aug 11, 2021 Maintainer

wamak9 Aug 11, 2021 Author

zroubalik Aug 11, 2021 Maintainer

wamak9 Aug 11, 2021 Author

zroubalik Aug 11, 2021 Maintainer

wamak9
Aug 10, 2021

tomkerkhove
Aug 11, 2021
Collaborator

zroubalik
Aug 11, 2021
Maintainer

wamak9
Aug 11, 2021
Author

zroubalik
Aug 11, 2021
Maintainer

wamak9
Aug 11, 2021
Author

zroubalik
Aug 11, 2021
Maintainer