fix: use rabbit quorum queues in lieu of ha #294
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We define the use of quorum queues via kustomize as the default queue type for the named vhosts, but the oslo_messaging_rabbit config opt of `rabbit_ha_queues: true` was set, taking precedence. We actually do not want to use HA queues, as they are being deprecated, and will be removed in newer versions of RMQ (4.x being released EOY 2024). The use of HA queues in genestack up to this point was the result of sane but no longer ideal defaults set by openstack-helm that were carried forth. This explicitly disables rabbit_ha_queues, and then enables rabbit_quorum_queue. Removing the related rabbit vhost is required for this change prior to re-deploying a given openstack service. Example of re-deploying nova when making this change; note how we remove the queue, vhost, and user: ``` kubectl -n openstack delete queues.rabbitmq.com nova-queue kubectl -n openstack delete vhosts.rabbitmq.com nova-vhost kubectl -n openstack delete users.rabbitmq.com nova helm --upgrade install nova ./nova ``` **NOTE**: Several helm upgrades may be required due to a race condition with the operator removing the vhost. Uninstalling first may be easier, but do so carefully. Other changes: - add: `rabbit_transient_quorum_queue` which is newly availably in 2024.1. We will want to begin using this to make transient queues reliable - add: `use_queue_manager` which is newly available in 2024.1 We will want to begin using this when available to de-obfuscate named queues in rabbit - add: `rabbit_interval_max` to reconnect faster after a node outage - fix: send heartbeats more frequently; clients should mark a given node as down about 30s more quickly (default was 60s) - fix: set `kombu_reconnect_delay` lower to help avoid multiple code paths not being traversed when a RMQ node goes down
cloudnull
approved these changes
Jun 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We define the use of quorum queues via kustomize as the default queue type for the named vhosts, but the oslo_messaging_rabbit config opt of
rabbit_ha_queues: truewas set, taking precedence. We actually do not want to use HA (mirrored) queues, as they are being deprecated, and will be removed in newer versions of RMQ (4.x being released EOY 2024). The use of HA queues in genestack up to this point was the result of sane but no longer ideal defaults set by openstack-helm that were carried forth.This explicitly disables rabbit_ha_queues, and then enables rabbit_quorum_queue. Removing the related rabbit vhost is required for this change prior to re-deploying a given openstack service.
Example of re-deploying nova when making this change; note how we remove the queue, vhost, and user:
NOTE: Several helm upgrades may be required due to a race condition with the operator removing the vhost. Uninstalling first may be easier, but do so carefully.
Other changes:
rabbit_transient_quorum_queuewhich is newly availably in 2024.1. We will want to begin using this to make transient queues reliableuse_queue_managerwhich is newly available in 2024.1 We will want to begin using this when available to de-obfuscate named queues in rabbitrabbit_interval_maxto reconnect faster after a node outagekombu_reconnect_delaylower to help avoid multiple code paths not being traversed when a RMQ node goes down