Trying to understand KEDA RabbitMQ scaler scale down mechanics.

[Static-Flow]

Hey everyone. Trying to use the RabbitMq Scaler and it's scaling down the deployment while work is still ongoing. This is discussed here #157 and it seems like the recommended approach is to not ack the consumed message until processing is done. What I'm trying to understand from that is whether Keda's RabbitMq scaler is hinging on Ready messages specifically or the total which includes unacked messages? From some tests it seems that it's only looking at Ready messages.

Am I just engineering this poorly? I've got a publisher that creates work items and pushes them to RabbitMq, KEDA sees a non zero message count and increases the deployment to 1, that one consumer can handle the publishers traffic so the queue is always 0, 5 minute timer elapses and since queue is 0 scales down the deployment mid work.

It's maybe a slightly different edge case because the workloads aren't long running themselves the consumers are just doing multiple of them at once. A consumer pod has multiple "workers" inside it to handle consumed messages so the consumption loop is tight resulting in a consumer being able to accept lots of messages at once even though it may not finish them all immediately. My initial plan was to ack the consumed messages when the consumer workers are done with them but KEDA only seems to focus on the Ready message count being zero, not that there are still unacked messages.

A solution would be to reduce each deployment pod to work on a single work item at once but since the work in I/O bound that seems like a really inefficient way of doing it since each pod will spend time waiting on I/O that it could be using to consume another work message.

[zroubalik]

@JorTurFer CC

[JorTurFer]

hey @Static-Flow ,
you are using amqp protocol, right? The problem with amqp is that it doesn't include unacked messages (i.e. consumed but not yet finished processing), HTTP mode does.
So I recommend to use (at least for KEDA) the http protocol

[amer-idwise]

So if I used the HTTP mode (where the mode set to be QueueLength) then it'll use the total messages rather than the Ready messages??

I couldn't see any official documentation related to that

[JorTurFer]

https://stackoverflow.com/a/63020119