Galera Cluster: Monitoring¶

Reference: Monitoring the Cluster. These are the wsrep_* status variables worth checking regularly or as a first pass when something looks off. See Overview if any of the underlying concepts (Primary state, quorum, GTID) are unfamiliar.

Individual Checks¶

Cluster address¶

SHOW GLOBAL VARIABLES LIKE 'wsrep_cluster_address';

Should be identical on every node.

Cluster state UUID¶

SHOW STATUS LIKE 'wsrep_cluster_state_uuid';

Every node should report the same value. A node reporting a different UUID than its peers is no longer part of the same cluster state, even if it appears "up."

Cluster configuration ID¶

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_conf_id';

Should match on all nodes. A mismatch indicates the cluster is partitioned.

Cluster size¶

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';

Number of nodes the cluster currently sees — use this to spot missing members at a glance.

Cluster status¶

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';

Should read Primary on every healthy node. Anything else means the node is part of a nonoperational component — typically from a lost-quorum membership change or a split-brain condition.

Node readiness¶

SHOW GLOBAL STATUS LIKE 'wsrep_ready';

ON means the node can accept write-sets from the cluster. OFF means almost all queries will fail with Error 1047 (08S01) Unknown Command.

Node connectivity¶

SHOW GLOBAL STATUS LIKE 'wsrep_connected';

ON means the node has a network connection to at least one other cluster member. OFF means it has none.

Node state comment¶

SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';

When part of the Primary Component, expect one of: Joining, Waiting on SST, Joined, Synced, Donor. A value of Initialized means the node is part of a nonoperational component.

Checking Several at Once¶

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');

Combined Health-Check Script¶

#!/bin/bash

echo "http://galeracluster.com/documentation-webpages/monitoringthecluster.html"

echo "Cluster State UUID"
mysql -e "SHOW STATUS LIKE 'wsrep_cluster_state_uuid';"
echo ""

echo "Cluster Conf ID"
mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_conf_id';"
echo ""

echo "Checking Cluster Size"
mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';"
echo ""

echo "Checking Cluster Status"
mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"
echo ""

echo "Check Node Ready"
mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_ready';"
echo ""

echo "Check Node Connected"
mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_connected';"
echo ""

echo "Check Node Comment"
mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"
echo ""

Prebuilt Status Tool (Docker)¶

For the containerized deployment, a packaged checker wraps most of the above:

docker pull registry.gitlab.com/myteam/galera_status:latest
docker run -it --rm registry.gitlab.com/andrewmercer/galera_status:latest python galera_status.py -c all
docker run -it --rm registry.gitlab.com/andrewmercer/galera_status:latest python galera_status.py --check [ option ]

Troubleshooting for what to do when a check above comes back wrong
Crash Recovery for grastate.dat and wsrep_last_committed, used specifically when deciding which node to bootstrap from