RabbitMQ Operations¶

Command-level companion to the Overview. Commands assume RabbitMQ 4.3 and a shell on a cluster node (prefix with sudo on package installs, or docker compose exec rabbitmq in the Lab).

Installing natively¶

Install from Team RabbitMQ's repositories rather than the distribution's own packages, which lag far behind and often pair RabbitMQ with an unsupported Erlang version. Each RabbitMQ series supports a specific range of Erlang/OTP versions; the team's repositories ship a matching Erlang, so take both from there. Repository URLs and signing keys do change; if a step below fails, check the current Debian or RPM guide.

Debian and Ubuntu¶

sudo apt-get install -y curl gnupg apt-transport-https

# Team RabbitMQ signing key
curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" \
  | sudo gpg --dearmor -o /usr/share/keyrings/com.rabbitmq.team.gpg

# Erlang and RabbitMQ repositories (Ubuntu 24.04 shown; swap "noble" for your release)
sudo tee /etc/apt/sources.list.d/rabbitmq.list <<'EOF2'
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb1.rabbitmq.com/rabbitmq-erlang/ubuntu/noble noble main
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb2.rabbitmq.com/rabbitmq-erlang/ubuntu/noble noble main
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb1.rabbitmq.com/rabbitmq-server/ubuntu/noble noble main
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb2.rabbitmq.com/rabbitmq-server/ubuntu/noble noble main
EOF2

sudo apt-get update
sudo apt-get install -y erlang-base erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
  erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key erlang-runtime-tools \
  erlang-snmp erlang-ssl erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl
sudo apt-get install -y rabbitmq-server

Pin both packages (apt-mark hold erlang-base rabbitmq-server) so an unattended upgrade can't move a node to a new series behind your back.

RHEL, Rocky and Alma (dnf)¶

sudo rpm --import https://github.com/rabbitmq/signing-keys/releases/download/3.0/rabbitmq-release-signing-key.asc

sudo tee /etc/yum.repos.d/rabbitmq.repo <<'EOF2'
[modern-erlang]
name=modern-erlang-el9
baseurl=https://yum1.rabbitmq.com/erlang/el/9/$basearch
        https://yum2.rabbitmq.com/erlang/el/9/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key

[rabbitmq-el9]
name=rabbitmq-el9
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/9/$basearch
        https://yum2.rabbitmq.com/rabbitmq/el/9/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key
       https://github.com/rabbitmq/signing-keys/releases/download/3.0/rabbitmq-release-signing-key.asc

[rabbitmq-el9-noarch]
name=rabbitmq-el9-noarch
baseurl=https://yum1.rabbitmq.com/rabbitmq/el/9/noarch
        https://yum2.rabbitmq.com/rabbitmq/el/9/noarch
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key
       https://github.com/rabbitmq/signing-keys/releases/download/3.0/rabbitmq-release-signing-key.asc
EOF2

sudo dnf makecache -y --disablerepo='*' --enablerepo='rabbitmq-el9*' --enablerepo='modern-erlang'
sudo dnf install -y socat logrotate erlang rabbitmq-server
sudo dnf versionlock add erlang rabbitmq-server   # needs python3-dnf-plugin-versionlock

First boot¶

sudo systemctl enable --now rabbitmq-server
sudo rabbitmq-plugins enable rabbitmq_management rabbitmq_prometheus

# Replace guest with a real administrator
sudo rabbitmqctl add_user admin           # prompts for the password
sudo rabbitmqctl set_user_tags admin administrator
sudo rabbitmqctl set_permissions -p / admin '.*' '.*' '.*'
sudo rabbitmqctl delete_user guest

sudo rabbitmq-diagnostics status

Raise the file-descriptor limit for the service before it carries real traffic:

sudo systemctl edit rabbitmq-server
# [Service]
# LimitNOFILE=65536
sudo systemctl restart rabbitmq-server

Firewall (firewalld)¶

Client ports can face the application network; the clustering ports should only accept the other nodes. A dedicated zone keyed on the cluster subnet does that cleanly:

# Clients: AMQP (plain + TLS) and the management UI
sudo firewall-cmd --permanent --zone=public --add-port={5672/tcp,5671/tcp,15672/tcp}

# Cluster peers only: epmd, Erlang distribution, CLI tools, Prometheus scraping
sudo firewall-cmd --permanent --new-zone=rabbitmq-cluster
sudo firewall-cmd --permanent --zone=rabbitmq-cluster --add-source=10.0.10.0/24
sudo firewall-cmd --permanent --zone=rabbitmq-cluster \
  --add-port={4369/tcp,25672/tcp,35672-35682/tcp,15692/tcp}
sudo firewall-cmd --reload

Add 5552/5551 for streams and 1883/8883 for MQTT if you enable those plugins.

Configuration¶

rabbitmq-diagnostics status lists which config files a node actually loaded, and rabbitmq-diagnostics environment prints the effective configuration. Both are the first thing to check when a setting "isn't working". A good baseline:

listeners.tcp.default = 5672
vm_memory_high_watermark.relative = 0.6
disk_free_limit.absolute = 5GB
log.file.level = info
log.connection.level = warning
heartbeat = 60
consumer_timeout = 1800000

Users, vhosts and permissions¶

rabbitmqctl list_users
rabbitmqctl add_user andrew                  # prompts; avoids the password landing in shell history
rabbitmqctl change_password andrew           # prompts
rabbitmqctl set_user_tags andrew administrator
rabbitmqctl delete_user andrew

rabbitmqctl add_vhost orders --default-queue-type quorum
rabbitmqctl list_vhosts name default_queue_type

# An application user confined to its own resource names: configure, write, read
rabbitmqctl add_user orders-svc
rabbitmqctl set_permissions -p orders orders-svc '^orders\..*' '^orders\..*' '^orders\..*'
rabbitmqctl list_user_permissions orders-svc

# A monitoring user for HTTP API checks. The tag alone gives a read-only view of
# every vhost; no resource permissions, so it can't consume or publish anything.
# (The Prometheus endpoint on 15692 needs no credentials at all.)
rabbitmqctl add_user monitor
rabbitmqctl set_user_tags monitor monitoring

# Limits
rabbitmqctl set_vhost_limits -p orders '{"max-connections": 500, "max-queues": 2000}'
rabbitmqctl set_user_limits orders-svc '{"max-connections": 50, "max-channels": 500}'

Policies¶

A queue matches at most one policy: the one with the highest priority among those whose pattern matches. Policies don't merge, so a queue matched by a "limits" policy and a "DLX" policy only gets one of them. Put all the keys a group of queues needs into a single policy.

# Durable work queues for the orders app: bounded, dead-lettered, poison-safe
rabbitmqctl set_policy -p orders --apply-to quorum_queues --priority 10 orders-work '^orders\.' \
  '{"max-length": 100000,
    "overflow": "reject-publish",
    "dead-letter-exchange": "orders.dlx",
    "dead-letter-strategy": "at-least-once",
    "delivery-limit": 5,
    "delayed-retry-type": "failed",
    "delayed-retry-min": 5000,
    "delayed-retry-max": 60000}'

# Operator policies cap what application policies may set (the stricter value wins)
rabbitmqctl set_operator_policy -p orders --apply-to queues guardrails '.*' \
  '{"max-length-bytes": 2147483648}'

rabbitmqctl list_policies -p orders
rabbitmqctl list_queues -p orders name policy operator_policy effective_policy_definition
rabbitmqctl clear_policy -p orders orders-work

--apply-to accepts queues, classic_queues, quorum_queues, streams, exchanges or all.

Definitions (topology backup)¶

Definitions are the cluster's topology and access control: vhosts, users (with password hashes), permissions, exchanges, queues, bindings and policies. They're small, they're what you need to rebuild a cluster, and they should be exported on a schedule and kept in version control or backup storage. Message contents aren't part of a definitions backup.

rabbitmqctl export_definitions /var/backups/rabbitmq/definitions-$(date +%F).json
rabbitmqctl import_definitions /var/backups/rabbitmq/definitions-2026-10-01.json

To have new nodes or clusters boot with a known topology:

definitions.import_backend = local_filesystem
definitions.local.path = /etc/rabbitmq/definitions.json

Clustering¶

Joining nodes by hand¶

Peer discovery is preferred (see the Overview and the lab's config/cluster/rabbitmq.conf), but manual joins are still useful for repairs.

# 1. Same Erlang cookie on every node (copy it from the first node), then restart
sudo install -o rabbitmq -g rabbitmq -m 0400 erlang.cookie /var/lib/rabbitmq/.erlang.cookie
sudo systemctl restart rabbitmq-server

# 2. On each node joining rabbit@rmq1
rabbitmqctl stop_app
rabbitmqctl reset                 # required if this node has ever run on its own
rabbitmqctl join_cluster rabbit@rmq1
rabbitmqctl start_app

# 3. Verify from any node
rabbitmqctl cluster_status

When a join fails:

Removing and replacing nodes¶

# Remove a node that is gone for good (run on a surviving node).
# On 4.3 this removes its quorum queue and stream replicas first.
rabbitmqctl forget_cluster_node rabbit@rmq3

# Remove a healthy node gracefully (run on the leaving node)
rabbitmq-upgrade drain
rabbitmqctl stop_app
rabbitmqctl reset

# After the replacement node joins, give it quorum queue replicas and spread leaders
rabbitmq-queues grow rabbit@rmq4 all
rabbitmq-queues rebalance quorum

Quorum queues and streams¶

rabbitmq-queues quorum_status lab.orders.payment        # members, leader, Raft indexes
rabbitmq-queues add_member lab.orders.payment rabbit@rmq3
rabbitmq-queues delete_member lab.orders.payment rabbit@rmq3
rabbitmq-queues check_if_node_is_quorum_critical         # exit code != 0 if stopping this node breaks a majority
rabbitmqctl list_queues name type leader members online

rabbitmq-streams stream_status lab.events
rabbitmq-streams add_replica lab.events rabbit@rmq3

Upgrades¶

1. Read the release notes for the target version, including the "Breaking Changes" section and the required Erlang version.
2. Upgrade to the latest patch of your current series first. 4.3 only accepts upgrades from the latest 4.2.x, with every 4.2 feature flag enabled.
3. Enable all stable feature flags and check for deprecated features you still use:

bash rabbitmqctl enable_feature_flag all rabbitmq-diagnostics check_if_any_deprecated_features_are_used

4. Export definitions.
5. One node at a time:

bash rabbitmq-queues check_if_node_is_quorum_critical # must pass rabbitmq-upgrade drain sudo systemctl stop rabbitmq-server # upgrade erlang + rabbitmq-server packages (or the container image tag) sudo systemctl start rabbitmq-server rabbitmq-diagnostics check_running && rabbitmq-diagnostics check_local_alarms rabbitmq-upgrade revive rabbitmq-queues rebalance quorum # after the last node

6. When every node runs the new version: rabbitmqctl enable_feature_flag all again.

Mixed-version clusters exist only to make rolling upgrades possible; don't leave one running for more than a few hours.

Troubleshooting playbooks¶

scripts/triage.sh in the lab runs the read-only checks below in one pass and works against native nodes too (RMQ_EXEC=sudo ./scripts/triage.sh).

Publishers are blocked or hanging¶

Symptom: publishes stall or time out; the UI shows connections as blocked or blocking.

rabbitmq-diagnostics alarms
rabbitmq-diagnostics memory_breakdown --unit mb
df -h /var/lib/rabbitmq

A memory or disk alarm blocks every publisher in the cluster until it clears. Short term: make sure consumers are running so queues drain, and purge queues that hold junk. Longer term: find what's holding memory (usually one huge queue, too many connections/channels, or management statistics on a broker with many objects), add length limits, and size disk_free_limit so the alarm fires with room to spare. Raising the watermark only moves the cliff closer to the kernel's OOM killer.

A queue keeps growing¶

rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers consumer_capacity \
  | sort -k2 -nr | head
rabbitmqctl list_consumers queue_name channel_pid prefetch_count active

Read the numbers together. Ready messages with no consumers means a service is down or consuming the wrong queue. Ready messages with consumers and low consumer_capacity (well below 100%) means consumers aren't receiving fast enough: often prefetch is too low for the round-trip time. Ready messages with consumers at full capacity means the consumers themselves are too slow: scale them out, or shard the queue. A queue that only grows during a daily batch is normal; a monotonic climb isn't.

Unacknowledged messages pile up¶

Many unacked messages with few acks per second means consumers have received work and aren't finishing it: stuck on a downstream call, deadlocked, or holding messages they'll never ack (a bug in an error path). Restarting the consumer releases them for redelivery. Quorum queue consumer timeouts (30 minutes by default, consumer-timeout policy key to change) eventually reclaim deliveries from a stuck consumer. To check the broker itself for stuck processes:

rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'

File descriptors or connections climbing¶

watch -n 5 "rabbitmq-diagnostics -q status | grep -iA4 'file descriptors'"
rabbitmqctl list_connections user peer_host client_properties | head
rabbitmqctl list_connections peer_host | sort | uniq -c | sort -nr | head

One host holding thousands of connections is a client leaking them; a steady stream of new connections from one host (watch rabbitmq_connections_opened_total) is a client connecting per operation. Both are client bugs; the fix is connection reuse. Named connections (connection_name client property) make the offender obvious in client_properties.

A node won't start or rejoin¶

Check journalctl -u rabbitmq-server and the node's log first. Common causes: the hostname changed (the data directory and the node's identity are tied to rabbit@<hostname>); the cookie file has the wrong owner or permissions (it must be 0400 or 0600, owned by rabbitmq); the Erlang version is outside the range the RabbitMQ version supports; or, on 4.x, the node can't reach a majority of its cluster peers, so it waits for them during boot (it logs that it's waiting; it isn't hung).

The cluster lost its majority¶

With two of three nodes down, metadata operations and quorum queues stop until enough nodes return. The right fix is almost always to bring the nodes back, even with degraded hardware: they rejoin and catch up through Raft. If nodes are permanently lost, rebuild from exported definitions rather than forcing the survivors; forced recovery of Raft members can lose committed data and should be a last resort following the official documentation.

Messages are "disappearing"¶

rabbitmqctl list_bindings -p / source_name destination_name routing_key | grep <exchange>

Usually unroutable: published to an exchange with no matching binding, so they're dropped (rabbitmq_global_messages_unroutable_dropped_total increases). Publish with mandatory, or add an alternate exchange. Other causes: TTL expiry, drop-head overflow on a full queue, a consumer using auto_ack that crashes, or rejection without requeue on a queue without a DLX.

Redelivery loops¶

The same messages delivered over and over with redelivered=true: a consumer failing and requeuing a poison message. On quorum queues, rejecting with basic.reject makes the delivery limit dead-letter it eventually; basic.nack doesn't (4.3+). On classic queues there's no limit at all: the consumer has to detect repeats (via the redelivered flag or its own counter) and reject without requeue.

Purging junk¶

rabbitmqctl purge_queue -p / notifications.error
rabbitmqctl delete_queue -p / old.queue --if-empty

Purging treats the symptom. The classic example is OpenStack: services emit notifications to notifications.* and versioned_notifications.* queues, and when nothing consumes them (telemetry disabled or removed) those queues grow until they trip the memory alarm. Either stop producing them ([oslo_messaging_notifications] driver = noop in each service) or bound them with a policy:

rabbitmqctl set_policy --apply-to queues notifications-cap '^(versioned_)?notifications\.' \
  '{"max-length": 10000, "message-ttl": 3600000}'

Log analysis¶

Since 3.7, RabbitMQ log lines look like 2026-10-03 14:02:11.123456+00:00 [error] <0.1234.0> message. Rotated files are compressed, so these use zgrep, which reads both.

LOGS=/var/log/rabbitmq/rabbit@*.log*

# Lines per level
zgrep -hoE '^\S+ \S+ \[[a-z]+\]' $LOGS | awk '{print $3}' | sort | uniq -c

# Top 50 distinct errors and warnings, with timestamps, pids and numbers normalised
zgrep -hE '\[(error|warning)\]' $LOGS \
  | sed -E 's/^\S+ \S+ \[[a-z]+\] <[0-9.]+> //; s/[0-9]+/N/g' \
  | sort | uniq -c | sort -rn | head -50

# Cluster membership, partition and leadership events, in order
zgrep -hiE 'nodedown|node .* (down|up)|partition|leader|election|waiting for' $LOGS | sort

# Which client hosts open the most connections (churn hunting)
zgrep -h 'accepting AMQP connection' $LOGS \
  | grep -oE '\([0-9a-f.:]+:[0-9]+ ->' | sed -E 's/^\(//; s/:[0-9]+ ->$//' \
  | sort | uniq -c | sort -rn | head

# Connections closed without a proper close (crashed clients, LB timeouts, missed heartbeats)
zgrep -hcE 'client unexpectedly closed TCP connection|missed heartbeats' $LOGS

CLI reference¶

Legacy: 3.x clusters under Pacemaker (OpenStack)¶

This section applies to RabbitMQ 3.x with Mnesia, as deployed by Red Hat OpenStack Platform (Director/TripleO) and similar installers, where Pacemaker manages RabbitMQ through a resource agent. On OSP 10-12 the resource is rabbitmq-clone; containerized releases (OSP 13 and later) use rabbitmq-bundle. Substitute accordingly. None of this applies to RabbitMQ 4.x, where Mnesia and its partition-recovery problems are gone.

The underlying failure mode was Mnesia: after a partition or an unclean shutdown the nodes could disagree about cluster membership or table contents, and RabbitMQ would refuse to form a cluster or would run in a split state. The cure was to wipe the node databases and rebuild the cluster from one node. Wiping the database destroys all queued messages; with OpenStack that's normally acceptable because services re-create their queues and RPC callers retry.

Option 1: let Pacemaker rebuild the cluster¶

# On one controller
pcs resource disable rabbitmq-clone
pcs resource unmanage rabbitmq-clone

# On ALL controllers: stop RabbitMQ and wipe the Mnesia database
pkill -u rabbitmq
ps -elf | grep [r]abbit           # confirm nothing but possibly epmd is left
rm -rf /var/lib/rabbitmq/mnesia/*

# On one controller: hand control back; Pacemaker bootstraps a fresh cluster
pcs resource manage rabbitmq-clone
pcs resource enable rabbitmq-clone
pcs status | grep -A3 rabbitmq

Option 2: rebuild by hand, then hand back to Pacemaker¶

# On one controller
pcs resource unmanage rabbitmq-clone

# On ALL controllers
pkill -u rabbitmq
rm -rf /var/lib/rabbitmq/mnesia/*
rabbitmq-server -detached

# On every controller EXCEPT the one chosen as the seed (controller-0 here)
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@overcloud-controller-0
rabbitmqctl start_app
rabbitmqctl cluster_status

# Back on one controller
pcs resource manage rabbitmq-clone

Other 3.x-era notes¶

• If rabbit_diagnostics:maybe_stuck() reported stuck processes on a 3.x node, restarting RabbitMQ on that node was usually the only fix.
• Policies using ha-mode mirrored classic queues across nodes. Those policies do nothing on 4.x; queues that need replication must be migrated to quorum queues (declare new quorum queues, move consumers and publishers over, drain the old ones; a Shovel can move leftovers).
• 3.6 and earlier wrote multi-line =ERROR REPORT==== <date> === log entries. These one-liners were for sosreports and log bundles from that era:

```bash # Notable errors per log file for R in $(find . -name 'rabbit@.log' ! -name 'sasl'); do echo "* $R" zgrep -E 'etimed|timeo|^** |^=|abru|no ex|incon|Missing|badmatch|orddict,fetch|badreturnvalue|undef|getenv|sender_death|rabbit on node.(up|down)|shutdown|killed|partition|(Starting|Stopping) R|Error desc|Master.death' "$R" \ | grep -v '^=I'; echo; done

# Count of report types per file for R in $(find . -name 'rabbit@.log' ! -name 'sasl'); do echo "* $R" zgrep -E '^=' "$R" | sed -e 's/ ... ===$//' | sort -k3 | uniq -c; done | grep -v INFO

# Node up/down and start/stop events with their report headers for R in $(find . -name 'rabbit@.log' ! -name 'sasl'); do echo "* $R" zgrep -EB1 'rabbit on node.(up|down)|down:|(Starting|Stopping) R' "$R"; done | grep -v '^--' ```