Unable to delete the scaled object deployed in Azure AKS and Namespace in terminating state

[nidiculageorge]

Report

Unable to delete the scaled object deployed in Azure AKS and Namespace in terminating state

Expected Behavior

As we tried deleting the namespace its in terminating state because of the above scaled object in that namespace

Actual Behavior

Namesapce should be deleted after the scaled object is deleted

Steps to Reproduce the Problem

Logs from KEDA operator

example

KEDA Version

2.11.1

Kubernetes Version

1.25

Platform

Microsoft Azure

Scaler Details

No response

Anything else?

No response

[nidiculageorge]

I tried removing the finalizer but still I am unable to remove the Scaled object the finalizer is not getting removed from the scaled object

[SpiritZhou]

Could you provide the events of scaledobject or logs of keda-operator?

[zroubalik]

Do you have KEDA operator up and running?

[nidiculageorge]

@zroubalik we have keda operator running running in the cluster.But other applications are also using this same Keda

[zroubalik]

So try to remove these scaledobjects first, then namespace

[nidiculageorge]

@zroubalik this is keda namespace .Could you please clarify.

Or do we need to delete this

[zroubalik]

just delete all scaledobjects in the namespace you are about to remote:

k delete scaledobject --all -n YOUR_NAMESPACE

And also check keda-operator pod logs to see if there are any errors or anything

[nidiculageorge]

@zroubalik I tried deleting the same and is stuck like this

[nidiculageorge]

@zroubalik I could see some errors in the keda operator log related to this namespace which we are trying to delete

2023-10-19T03:28:51Z ERROR scale_handler error getting metric for scaler {"scaledObject.Namespace": "autozoning-live-ibr-34", "scaledObject.Name": "recognitionicr-autoscaler", "scaler": "azureServiceBusScaler", "error": "scaler with id 2 not found, len = 0, cache has been probably already invalidated"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
/workspace/pkg/scaling/scale_handler.go:485
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
/workspace/pkg/metricsservice/server.go:48
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:98
google.golang.org/grpc.(*Server).processUnaryRPC
/workspace/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
/workspace/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
/workspace/vendor/google.golang.org/grpc/server.go:965
2023-10-19T03:28:51Z ERROR scale_handler error getting metric spec for the scaler {"scaledObject.Namespace": "autozoning-live-ibr-34", "scaledObject.Name": "recognitionicr-autoscaler", "scaler": "azureServiceBusScaler", "error": "scaler with id 3 not found. Len = 0"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
/workspace/pkg/scaling/scale_handler.go:446
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
/workspace/pkg/metricsservice/server.go:48
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:98
google.golang.org/grpc.(*Server).processUnaryRPC
/workspace/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
/workspace/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
/workspace/vendor/google.golang.org/grpc/server.go:965

[nidiculageorge]

@zroubalik any update

[nidiculageorge]

Hi Team,

Is there any update on the issue.

[JorTurFer]

To delete the ScaledObject, KEDA operator must be alive to remove the finalizer. Is the operator alive when you delete the ScaledObject?
You can manually edit the ScaledObject and remove the finalizer to remove the ScaledObject, despite the operator pod is down

[JuanRamino]

Manually remove the finalizer from scaledobjects is not a solution.
We are experiencing a long time wait when deleting a namespace caused by scaledobjects.
The operator is always alive when this happen.
Sometimes it manages to delete the namespace after a few minutes, others after 10 minutes but it is never a quick process
Why this happen?

[JorTurFer]

Do you see any error related with throttling in operator logs?