MySQL MGR Cluster
MGR (MySQL Group Replication) provides multi-node state synchronization, supporting automatic member management and fault detection. It is recommended as the primary HA solution for production environments.
This guide describes how to build a Single-Primary mode cluster based on native Group Replication across three nodes and configure MySQL Router for transparent access.
Cluster Node Planning
| Server IP | Host Role | Services Deployed |
|---|---|---|
| 10.206.0.2 | Primary (Bootstrap Node) | MySQL, MySQL Router, MySQL Shell |
| 10.206.0.3 | Secondary | MySQL, MySQL Router, MySQL Shell |
| 10.206.0.4 | Secondary | MySQL, MySQL Router, MySQL Shell |
MySQL Service Deployment
This section covers the steps for deploying MySQL via binary packages. To ensure basic cluster consistency, the following steps must be fully executed on each node in the cluster.
-
Download Installation Package
wget https://pdpublic.mingdao.com/private-deployment/offline/common/mysql-8.0.45-linux-glibc2.17-x86_64.tar.xz -
Extract Package and Move to Installation Path
tar -xvf mysql-8.0.45-linux-glibc2.17-x86_64.tar.xzmv mysql-8.0.45-linux-glibc2.17-x86_64 /usr/local/mysql -
Add MySQL to
PATHEnvironment Variableecho 'export PATH=/usr/local/mysql/bin:$PATH' > /etc/profile.d/mysql.shsource /etc/profile.d/mysql.sh -
Create System User and Related Directories
useradd -U -r -s /sbin/nologin mysqlmkdir -p /data/mysqlmkdir -p /data/logs/mysqlchown -R mysql:mysql /data/mysql /data/logs/mysql /usr/local/mysql/ -
Create
/etc/my.cnfConfiguration FileKey Configuration PointsBefore writing the following content to the server, ensure you modify these parameters based on the actual environment of the current node:
server-id: Must be unique for each node; it is recommended to increment sequentially (e.g., 1, 2, 3).report_host: Enter the actual intranet IP address of the current server.group_replication_local_address: Enter the actual intranet IP of the current server (default port is 33061).group_replication_group_seeds: Enter the list of IPs and ports for all three nodes in the cluster.
cat > /etc/my.cnf <<'EOF'[mysqld]user = mysqlbasedir = /usr/local/mysqldatadir = /data/mysqlsocket = /usr/local/mysql/mysqld.sockpid-file = /usr/local/mysql/mysqld.pidlog-error = /data/logs/mysql/mysqld.logserver-id = 1bind-address = 0.0.0.0port = 3306mysqlx_port = 33060skip-name-resolve = ONmax_connections = 5000default_storage_engine = InnoDBinnodb_buffer_pool_size = 2Gcharacter_set_server = utf8mb4collation_server = utf8mb4_0900_ai_cislow_query_log = 1slow_query_log_file = /data/logs/mysql/mysql-slow.loglong_query_time = 1log_bin = /data/mysql/mysql-binbinlog_format = ROWsync_binlog = 1binlog_expire_logs_seconds = 2592000# --- GTID & Replication ---gtid_mode = ONenforce_gtid_consistency = ONlog_slave_updates = ONmaster_info_repository = TABLErelay_log_info_repository = TABLEtransaction_write_set_extraction = XXHASH64binlog_transaction_dependency_tracking = WRITESETreplica_parallel_type = LOGICAL_CLOCKreplica_parallel_workers = 4replica_preserve_commit_order = ON# --- Group Replication (MGR) ---report_host = 10.206.0.2report_port = 3306plugin_load_add = 'group_replication.so'group_replication_group_name = "c9f6d3f2-7b21-4e5a-9c87-3a0e9f0a43d2"group_replication_start_on_boot = OFFgroup_replication_local_address = "10.206.0.2:33061"group_replication_group_seeds = "10.206.0.2:33061,10.206.0.3:33061,10.206.0.4:33061"group_replication_single_primary_mode = ONgroup_replication_enforce_update_everywhere_checks = OFFgroup_replication_recovery_get_public_key = ON[client]port = 3306socket = /usr/local/mysql/mysqld.sock[mysql]default-character-set = utf8mb4EOF -
Configure Systemd Management File
cat > /etc/systemd/system/mysql.service <<'EOF'[Unit]Description=MySQL Database ServerDocumentation=man:mysqld(8) http://dev.mysql.com/doc/After=network.targetWants=network-online.targetAfter=network-online.target[Service]Environment=MYSQLD_PARENT_PID=1User=mysqlGroup=mysqlType=forkingPIDFile=/usr/local/mysql/mysqld.pidExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --daemonizeExecStop=/bin/kill -s SIGTERM $MAINPIDLimitNOFILE=102400LimitMEMLOCK=infinityRestart=on-failureOOMScoreAdjust=-500TimeoutSec=0[Install]WantedBy=multi-user.targetEOF -
Initialize MySQL
mysqld --no-defaults --initialize --datadir=/data/mysql/ --user=mysql --log-error=/data/logs/mysql/mysqld.log -
Start MySQL
systemctl daemon-reloadsystemctl enable mysqlsystemctl start mysql -
Log in to MySQL and Change Initial Password
Log in using the temporary password generated during initialization:
mysql -uroot -p$(grep 'temporary password' /data/logs/mysql/mysqld.log | awk '{print $NF}')After logging in, run the following commands in MySQL:
-- Change the current root user passwordALTER USER USER() IDENTIFIED BY '123456';RESET MASTER;-- Create a root user that allows remote connectionsSET SQL_LOG_BIN=0;CREATE USER 'root'@'%' IDENTIFIED BY '123456';GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;SET SQL_LOG_BIN=1;- The root user password set in the command is
123456; a strong password must be used in actual deployment. - If the password contains special characters, only
-or_are allowed; characters such as@ ! # &are prohibited to avoid compatibility issues.
- The root user password set in the command is
Build MGR Cluster
- All-node Preparation: Periodic sync user creation, channel credential settings, and other basic configurations must be executed on all nodes.
- Bootstrapping and Grouping: Cluster initialization (Bootstrap) is only performed on the first node, while other nodes only perform join operations.
-
Create Replication Synchronization User Execute the following command on each node to create the account used for group replication synchronization:
SET SQL_LOG_BIN=0;CREATE USER 'repl'@'%' IDENTIFIED BY '123456';GRANT REPLICATION SLAVE, CONNECTION_ADMIN, BACKUP_ADMIN, GROUP_REPLICATION_STREAM ON *.* TO 'repl'@'%';SET SQL_LOG_BIN=1;- The replication synchronization account
replpassword set in the command is123456; a strong password must be used in actual deployment. - If the password contains special characters, only
-or_are allowed; characters such as@ ! # &are prohibited to avoid compatibility issues.
- The replication synchronization account
-
Configure Group Replication Recovery Channel
Run the following command on each node to complete the group replication recovery channel credential configuration. Replace the password with the actual password of the synchronization account
repl:CHANGE REPLICATION SOURCE TO SOURCE_USER='repl', SOURCE_PASSWORD='123456' FOR CHANNEL 'group_replication_recovery'; -
Start and Initialize MGR Group
The MGR cluster must be "bootstrapped" by one node, after which other nodes can join. Please strictly follow this sequence:
-
A. First Node (Bootstrap Initialization) Execute the following commands to enable bootstrap mode and initialize the cluster:
SET GLOBAL group_replication_bootstrap_group=ON;START GROUP_REPLICATION;SET GLOBAL group_replication_bootstrap_group=OFF; -
B. Other Nodes (Join Existing Group) After the first node is successfully initialized, execute the following command on the remaining nodes to join the group:
START GROUP_REPLICATION;
-
-
Check Member Information
SELECT * FROM performance_schema.replication_group_members;Normal status should show all nodes with
MEMBER_STATE = ONLINE, with only onePRIMARYnode and the rest asSECONDARY. -
Modify
group_replication_start_on_bootin/etc/my.cnftoONRun this on each node. This parameter was set to
OFFduring initialization to prevent bootstrap conflicts. After the cluster is established, enable it so nodes can automatically rejoin the existing MGR replication group after a server reboot or MySQL restart:sed -ri 's|group_replication_start_on_boot.*|group_replication_start_on_boot = ON|g' /etc/my.cnfgrep group_replication_start_on_boot /etc/my.cnf -
Rolling Restart and Status Verification
Run this on each node in sequence. After restarting one node, confirm that its status has returned to
ONLINEbefore proceeding to the next node:# 1. Restart servicesystemctl restart mysql# 2. Check node status (Ensure all currently active nodes have MEMBER_STATE = ONLINE)mysql -uroot -p123456 -e 'SELECT * FROM performance_schema.replication_group_members;'
Enable InnoDB Cluster
-
All-node Operations: The MySQL Shell (
mysqlsh) package must be downloaded and installed on all nodes. -
Single-node Operations: The initialization configuration of InnoDB Cluster only needs to be executed once on the first node (Primary).
-
Download mysqlsh Installation Package
wget https://pdpublic.mingdao.com/private-deployment/offline/common/mysql-shell-8.0.45-linux-glibc2.17-x86-64bit.tar.gz -
Extract Package and Move to Installation Path
tar -xvf mysql-shell-8.0.45-linux-glibc2.17-x86-64bit.tar.gzmv mysql-shell-8.0.45-linux-glibc2.17-x86-64bit /usr/local/mysql-shell -
Add mysqlsh to
PATHEnvironment Variableecho 'export PATH=/usr/local/mysql-shell/bin:$PATH' > /etc/profile.d/mysql-shell.shsource /etc/profile.d/mysql-shell.sh -
Connect to PRIMARY Node (First Node Only)
mysqlsh --uri root@10.206.0.2:3306 -p123456 -
Initialize and Create InnoDB Cluster
var cluster = dba.createCluster('hap-mysql') -
View Cluster Status
cluster.status()- After logging back in, you need to execute
var cluster = dba.getCluster()before you can usecluster.status().
- After logging back in, you need to execute
MySQL Router Deployment
To achieve HA for the access layer, the following steps must be performed on each node in the cluster separately.
-
Download Installation Package
wget https://pdpublic.mingdao.com/private-deployment/offline/common/mysql-router-8.0.45-linux-glibc2.17-x86_64.tar.xz -
Extract Package and Move to Installation Path
tar -xvf mysql-router-8.0.45-linux-glibc2.17-x86_64.tar.xzmv mysql-router-8.0.45-linux-glibc2.17-x86_64 /usr/local/mysql-router -
Add MySQL to
PATHEnvironment Variableecho 'export PATH=/usr/local/mysql-router/bin:$PATH' > /etc/profile.d/mysql-router.shsource /etc/profile.d/mysql-router.sh -
Initialize mysqlrouter
mysqlrouter --bootstrap root:123456@10.206.0.2:3306 --user=mysql --report-host=10.206.0.2Note:
-
The IP after
--bootstrapis the Primary node IP and is used to read the cluster topology. Use the same Primary address on all cluster nodes. -
The IP after
--report-hostis the intranet IP of the current node. Change it separately on each node to its actual IP.
-
-
Increase MySQL Router Maximum Connection Configuration
sed -i '/^\[DEFAULT\]$/a max_total_connections=5000' /usr/local/mysql-router/mysqlrouter.conf -
Configure Permissions
chown -R mysql:mysql /usr/local/mysql-router -
Configure Systemd
cat > /etc/systemd/system/mysqlrouter.service <<'EOF'[Unit]Description=MySQL Router ServiceDocumentation=man:mysqlrouter(1) https://dev.mysql.com/doc/mysql-router/en/Wants=network-online.targetAfter=network-online.target[Service]Type=simpleUser=mysqlGroup=mysqlWorkingDirectory=/usr/local/mysql-routerExecStart=/usr/local/mysql-router/bin/mysqlrouter -c /usr/local/mysql-router/mysqlrouter.confRestart=on-failureRestartSec=2LimitNOFILE=102400OOMScoreAdjust=-500TimeoutSec=0[Install]WantedBy=multi-user.targetEOF -
Start mysqlrouter
systemctl daemon-reloadsystemctl enable mysqlroutersystemctl start mysqlrouter -
Connection Test
Verify connectivity through the read-write port 6446 of MySQL Router on the current node:
mysql -h 127.0.0.1 -P 6446 -u root -p123456
MySQL MGR Auto Recovery
If half or more nodes fail simultaneously in an MGR cluster, the cluster will automatically enter a Complete Outage (OFFLINE/RECOVERING) state to prevent "brain-split". In this extreme case, even if the node service process is normal, MGR defaults to being unable to rejoin automatically without manual intervention.
The core purpose of configuring the "auto recovery script" is to perform periodic checks and, when the cluster is completely interrupted, automatically select the latest node by GTID progress to bootstrap recovery and achieve fault self-healing. Complete the following steps on each node.
-
Create Directory and Grant Permissions
# 1. Create log directorymkdir -p /data/logs/mysql# 2. Ensure the directory owner is the mysql user (or adjust according to your actual running user)chown -R mysql:mysql /data/logs/mysql# 3. Set directory access permissionschmod 755 /data/logs/mysql -
Create Core Detection and Recovery Script
/usr/local/bin/mysql-auto-recover.shNote: Please ensure you verify and modify the parameters in the
Core Configurationarea of the script below (such as password, node IP list, etc.) to match your actual deployment environment.cat > /usr/local/bin/mysql-auto-recover.sh <<'EOF'#!/bin/bashset -euo pipefail # Strict mode# ============================ Core Configuration ============================MYSQL_CLI="/usr/local/mysql/bin/mysql" # File path of the MySQL clientMYSQL_SHELL="/usr/local/mysql-shell/bin/mysqlsh" # MySQL Shell file pathMYSQL_USER="root" # MySQL database administrator usernameMYSQL_PASS="123456" # MySQL database administrator passwordCLUSTER_NAME="hap-mysql" # Name of the MGR (InnoDB Cluster)LOG_FILE="/data/logs/mysql/mysql-auto-recover.log" # Log path of the auto recovery scriptLOCK_FILE="/data/logs/mysql/mysql-auto-recover.lock" # Process exclusive lock file pathNODES=("10.206.0.2" "10.206.0.3" "10.206.0.4") # IP address list of all MGR cluster nodes, space-separated and aligned with the cluster planLOG_KEEP_DAYS=180 # Local log file retention daysTRIES=3 # Continuous retry count for fault confirmation detectionSLEEP_SEC=2 # Buffer time between each fault detection retry (seconds)# ============================ Log Functions ============================# Unified log format: [YYYY-MM-DD HH:MM:SS] [LEVEL] [PID] message# Levels: INFO (normal event) / WARN (abnormal but recoverable) / ERROR (recovery failed and requires attention)_log() {local level="$1"; shiftprintf '[%s] [%-5s] [pid=%d] %s\n' "$(date '+%F %T')" "$level" "$$" "$*" >> "$LOG_FILE"}log_info() { _log "INFO" "$@"; }log_warn() { _log "WARN" "$@"; }log_error() { _log "ERROR" "$@"; }# Prefix each stdout/stderr line from external processes such as mysqlsh before writing it to disk.# Empty lines and known low-value warning noise are filtered to keep logs clean and readable.log_pipe() {local tag="$1"local linewhile IFS= read -r line; do[ -z "$line" ] && continuecase "$line" in*"Using a password on the command line"*) continue ;;*"Cannot set LC_ALL to locale"*) continue ;;esac_log "INFO" "[$tag] $line"done}# ============================ Basic Functions ============================# Concurrency control: ensures only one detection process runs at a timeexec_lock() {exec 9>"$LOCK_FILE"if ! flock -n 9; thenlog_warn "Another instance is already running; exiting."exit 0fi}# Log rotation and cleanup: split files by day, and point the main log path to today's file through a symbolic linkrotate_logs() {local log_file="$1"local keep_days="${2:-180}"local log_dir base_name today today_loglog_dir=$(dirname "$log_file")base_name=$(basename "$log_file" .log)today=$(date +%F)today_log="${log_dir}/${base_name}_${today}.log"# If today's log file does not exist, create it and point the main log path to it (atomic symlink update)if [ ! -f "$today_log" ]; then: > "$today_log"ln -sfn "$today_log" "$log_file"fi# If the main log is not a symlink or does not point to today's file, repair the symlinkif [ ! -L "$log_file" ] || [ "$(readlink -f "$log_file" 2>/dev/null || true)" != "$today_log" ]; thenln -sfn "$today_log" "$log_file"fi# Clean expired logs in the current directory only to avoid accidental deletionfind "$log_dir" -maxdepth 1 -name "${base_name}_*.log" -type f -mtime +"$keep_days" -exec rm -f {} \; 2>/dev/null || true}check_node_alive() {local node="$1"if "$MYSQL_CLI" -h "$node" -u"$MYSQL_USER" -p"$MYSQL_PASS" \--connect-timeout=3 -e "SELECT 1;" >/dev/null 2>&1; thenreturn 0elselog_warn "Node $node is unreachable."return 1fi}check_all_nodes_alive() {local ok=0for node in "${NODES[@]}"; doif check_node_alive "$node"; thenok=$((ok+1))fidoneecho "$ok"}# Return value: non-negative integer = ONLINE member count; -1 = query failed (different from a real 0)get_cluster_online_count() {local resultresult=$("$MYSQL_CLI" -u"$MYSQL_USER" -p"$MYSQL_PASS" \--connect-timeout=3 -N -e \"SELECT COUNT(*) FROM performance_schema.replication_group_members WHERE MEMBER_STATE='ONLINE';" 2>/dev/null) || {echo "-1"return 0}if [[ "$result" =~ ^[0-9]+$ ]]; thenecho "$result"elseecho "-1"fi}# Count the total committed transactions in gtid_executed, compatible with:# 1) Single transaction: uuid:1000 -> 1# 2) Range: uuid:1-1000 -> 1000# 3) Multiple ranges: uuid:1-100:200-500 -> 100 + 301 = 401# 4) Multiple sources: uuid1:1-100,uuid2:1-200 -> 100 + 200 = 300# Key point: split by ":" and skip the UUID segment to avoid counting digits/hyphens in UUIDs as transactions.# Use the "total transaction count" rather than the "maximum sequence number" as the primary selection basis.# This correctly handles cases where nodes have the same highest range but different lower range sizes# (for example 1-53986:1000005-1000010 vs 1-53988:1000005-1000010).count_gtid_transactions() {local gtid="$1"[ -z "$gtid" ] && { echo 0; return 0; }echo "$gtid" | awk 'BEGIN { total = 0 }{# Remove all whitespace charactersgsub(/[ \t\r\n]/, "", $0)# Multiple sources are separated by commasn = split($0, sources, ",")for (i = 1; i <= n; i++) {# Each source is in the form uuid:range[:range...]; the first segment is UUID and must be skippedm = split(sources[i], parts, ":")for (j = 2; j <= m; j++) {seg = parts[j]if (seg ~ /^[0-9]+$/) {# Single transactiontotal += 1} else if (seg ~ /^[0-9]+-[0-9]+$/) {# Closed interval [start, end], transaction count = end - start + 1k = split(seg, range, "-")start = range[1] + 0end = range[k] + 0if (end >= start) total += (end - start + 1)}}}print total}'}find_primary_node() {declare -A TX_MAPlocal max_tx=0local primary=""local node gtid txfor node in "${NODES[@]}"; dogtid=$("$MYSQL_CLI" -h "$node" -u"$MYSQL_USER" -p"$MYSQL_PASS" \--connect-timeout=3 -N -e \"SELECT @@GLOBAL.gtid_executed;" 2>/dev/null) || gtid=""tx=$(count_gtid_transactions "$gtid")TX_MAP[$node]=$txlog_info "Node $node GTID: executed=[$gtid] transactions=$tx"if [ "$tx" -gt "$max_tx" ]; thenmax_tx=$txprimary=$nodefidone# If all node GTIDs are 0 (new empty cluster or all queries failed), default to the first node[ -z "$primary" ] && primary="${NODES[0]}"log_info "Primary candidate elected: $primary (transactions=$max_tx)"echo "$primary"}# Check whether the current host IP matches the target node (supports multiple NICs, exact match)is_local_host() {local target="$1"local ipfor ip in $(hostname -I 2>/dev/null); do[ "$ip" = "$target" ] && return 0donereturn 1}do_recovery() {local my_ip rcmy_ip=$(hostname -I | awk '{print $1}')log_info "This node ($my_ip) is the recovery target; invoking reboot_cluster_from_complete_outage."set +e# Set LC_ALL / LANG to C.UTF-8 to avoid mysqlsh locale warnings on systems without en_US.UTF-8.# Redirect all output to log_pipe so it is written with the [MYSQLSH] prefix.# Use PIPESTATUS[0] to capture the mysqlsh exit code itself instead of log_pipe's 0.timeout 60 env LC_ALL=C.UTF-8 LANG=C.UTF-8 \"$MYSQL_SHELL" --no-wizard --py \--user="$MYSQL_USER" --password="$MYSQL_PASS" \--host=127.0.0.1 <<PYEOF 2>&1 | log_pipe "MYSQLSH"import systry:cluster = dba.reboot_cluster_from_complete_outage('$CLUSTER_NAME', {'force': True})print("Cluster rebooted successfully.")except Exception as e:print("Recovery failed:", e)sys.exit(1)PYEOFrc=${PIPESTATUS[0]}set -eif [ "$rc" -eq 0 ]; thenlog_info "Cluster reboot completed successfully."elif [ "$rc" -eq 124 ]; thenlog_error "Cluster reboot timed out after 60 seconds; will retry on next cycle."elselog_error "Cluster reboot failed (exit_code=$rc); will retry on next cycle."fi}# ============================ Main Flow ============================# Lock first, then rotate logs to avoid concurrent instances corrupting symlinks/logsexec_lockrotate_logs "$LOG_FILE" "$LOG_KEEP_DAYS"MY_IP=$(hostname -I | awk '{print $1}')log_info "===== MGR health check started (host=$MY_IP) ====="# Declare a complete cluster outage only after TRIES consecutive confirmations that all nodes are reachable and ONLINE member count is 0is_outage=falseconsecutive=0for attempt in $(seq 1 "$TRIES"); doreachable=$(check_all_nodes_alive)online_count=$(get_cluster_online_count)if [ "$reachable" -eq "${#NODES[@]}" ] && [ "$online_count" = "0" ]; thenconsecutive=$((consecutive + 1))log_warn "Attempt $attempt/$TRIES: outage signal confirmed ($consecutive/$TRIES) [reachable=$reachable, online=$online_count]"if [ "$consecutive" -ge "$TRIES" ]; thenis_outage=truebreakfielseif [ "$consecutive" -gt 0 ]; thenlog_info "Attempt $attempt/$TRIES: outage signal reset [reachable=$reachable, online=$online_count]"ficonsecutive=0fi# Do not sleep after the last attempt to avoid unnecessary waiting[ "$attempt" -lt "$TRIES" ] && sleep "$SLEEP_SEC"doneif [ "$is_outage" = true ]; thenlog_warn "Complete cluster outage detected; initiating recovery procedure."PRIMARY_NODE=$(find_primary_node)if is_local_host "$PRIMARY_NODE"; thendo_recoveryelselog_info "This node ($MY_IP) is not the recovery target ($PRIMARY_NODE); standing by."fielselog_info "Cluster is healthy or precondition not met; no action taken."filog_info "===== MGR health check finished ====="EOFGrant execution permissions
chmod +x /usr/local/bin/mysql-auto-recover.sh -
Configure Systemd Service Unit
Create a Service file to define how to run the script above.
cat > /etc/systemd/system/mysql-auto-recover.service <<'EOF'[Unit]Description=MySQL MGR Auto-recovery ServiceAfter=network.target mysql.service[Service]Type=oneshotExecStart=/usr/local/bin/mysql-auto-recover.shUser=rootTimeoutSec=0EOF -
Configure Systemd Timer
Create a Timer file, setting the script to run once every minute.
cat > /etc/systemd/system/mysql-auto-recover.timer <<'EOF'[Unit]Description=Run MySQL MGR Recovery check every minute[Timer]OnCalendar=*-*-* *:*:00Unit=mysql-auto-recover.servicePersistent=true[Install]WantedBy=timers.targetEOF -
Load Configuration and Start
# Reload systemd configurationsystemctl daemon-reload# Enable and immediately start the timersystemctl enable --now mysql-auto-recover.timer -
Common Commands
The following are common instructions for managing the auto recovery service and its timer:
# View the next execution time of the timersystemctl list-timers --all | grep mysql-auto-recover# View real-time operation logstail -f /data/logs/mysql/mysql-auto-recover.log# Manually trigger a recovery detection immediatelysystemctl start mysql-auto-recover.service# View the status of the timer servicesystemctl status mysql-auto-recover.timer# Temporarily stop auto recovery monitoringsystemctl stop mysql-auto-recover.timer