How to debug channel close on NACKS RECEIVED

[bennyboer]

Hi there,

we're sometimes experiencing errors like:

Shutdown Signal: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=NACKS RECEIVED, class-id=0, method-id=0)

logged by org.springframework.amqp.rabbit.connection.CachingConnectionFactory.

This seems to be originating from here, thus I'm posting it in the rabbitmq-java-client project.

We're using RabbitMQ via Spring AMQP. The brokers logs do not show anything.

Anyone here that is able to tell us why this error might arise and/or how we might be able to debug its origins?

Thank you!

[acogoluegnes]

The client code (application or Spring AMQP) uses Channel#waitForConfirmsOrDie(long), when the timeout is reached the library closes the channel and throws this exception. The timeout may be too short or there may be something up on the server side that makes publish confirms cannot be sent to the client.

You may increase the timeout if it is really low (e.g. less than 1 second, but that may depend on the context) or find any relevant information on the server.

The application could also retry publishing unconfirmed messages.

[michaelklishin]

Alternatively, the application may be publishing to a queue that uses a length limit and overflow set to reject-publish, in which case basic.nacks are expected for everything published after the queue is at max capacity.

[bennyboer]

So it could also be that the broker just had a hiccup or the confirm showhow got lost? The error is pretty rare as it only occurs every few months.

[michaelklishin]

@bennyboer "the hiccup" must be a termination of a queue leader replica, which should be very visible in the logs. Except for overflow = reject_publish killing a queue replica with rabbitmqctl eval is the only relatively easy way to make a node send a basic.nack.

"Somehow got lost" is not specific enough to comment on but yes, publisher confirms exist exactly because there are hundreds of possible failure scenarios in distributed systems and their detection is never instantaneous.