MySQL MGR Cluster

MGR (MySQL Group Replication) provides multi-node state synchronization, supporting automatic member management and fault detection. It is recommended as the primary HA solution for production environments.

This guide describes how to build a Single-Primary mode cluster based on native Group Replication across three nodes and configure MySQL Router for transparent access.

Cluster Node Planning

Server IP	Host Role	Services Deployed
10.206.0.2	Primary (Bootstrap Node)	MySQL, MySQL Router, MySQL Shell
10.206.0.3	Secondary	MySQL, MySQL Router, MySQL Shell
10.206.0.4	Secondary	MySQL, MySQL Router, MySQL Shell

MySQL Service Deployment

Deployment Instructions

This section covers the steps for deploying MySQL via binary packages. To ensure basic cluster consistency, the following steps must be fully executed on each node in the cluster.

1. Download the installation package

   wget https://pdpublic.mingdao.com/private-deployment/offline/common/mysql-8.4.10-linux-glibc2.17-x86_64.tar.xz

2. Extract the package and move it to the installation path

   tar -xvf mysql-8.4.10-linux-glibc2.17-x86_64.tar.xz
   mv mysql-8.4.10-linux-glibc2.17-x86_64 /usr/local/mysql

3. Add MySQL to the PATH environment variable

   echo 'export PATH=/usr/local/mysql/bin:$PATH' > /etc/profile.d/mysql.sh
   source /etc/profile.d/mysql.sh

4. Create the system user and required directories

   useradd -U -r -s /sbin/nologin mysql
   mkdir -p /data/mysql
   mkdir -p /data/logs/mysql
   chown -R mysql:mysql /data/mysql /data/logs/mysql  /usr/local/mysql/

5. Create the /etc/my.cnf configuration file

   Key Configuration Points

   Before writing the following content to the server, ensure you modify these parameters based on the actual environment of the current node:

   • server-id: Must be unique for each node; sequential values are recommended (for example, 1, 2, and 3).
   • report_host: Enter the actual intranet IP address of the current server.
   • group_replication_local_address: Enter the actual intranet IP address of the current server. The default port is 33061.
   • group_replication_group_seeds: Enter the list of IP addresses and ports for all three nodes in the cluster.

   cat > /etc/my.cnf <<'EOF'
   [mysqld]
   user                         = mysql
   basedir                      = /usr/local/mysql
   datadir                      = /data/mysql
   socket                       = /usr/local/mysql/mysqld.sock
   pid-file                     = /usr/local/mysql/mysqld.pid
   log-error                    = /data/logs/mysql/mysqld.log
   
   server-id                    = 1
   bind-address                 = 0.0.0.0
   port                         = 3306
   mysqlx_port                 = 33060
   skip-name-resolve            = ON
   max_connections              = 5000
   default_storage_engine       = InnoDB
   innodb_buffer_pool_size      = 2G
   character_set_server         = utf8mb4
   collation_server             = utf8mb4_0900_ai_ci
   slow_query_log               = 1
   slow_query_log_file          = /data/logs/mysql/mysql-slow.log
   long_query_time              = 1
   log_bin                      = /data/mysql/mysql-bin
   sync_binlog                  = 1
   binlog_expire_logs_seconds   = 2592000
   # --- GTID & Replication ---
   gtid_mode                    = ON
   enforce_gtid_consistency     = ON
   log_replica_updates          = ON
   replica_parallel_workers               = 4
   replica_preserve_commit_order          = ON
   
   # --- Group Replication (MGR) ---
   report_host                  = 10.206.0.2
   report_port                  = 3306
   plugin_load_add              = 'group_replication.so'
   group_replication_group_name = "c9f6d3f2-7b21-4e5a-9c87-3a0e9f0a43d2"
   group_replication_bootstrap_group = OFF
   group_replication_start_on_boot = OFF
   group_replication_local_address = "10.206.0.2:33061"
   group_replication_group_seeds   = "10.206.0.2:33061,10.206.0.3:33061,10.206.0.4:33061"
   group_replication_single_primary_mode = ON
   group_replication_enforce_update_everywhere_checks = OFF
   group_replication_recovery_get_public_key = ON
   super_read_only               = ON
   
   
   [client]
   port                 = 3306
   socket               = /usr/local/mysql/mysqld.sock
   
   [mysql]
   default-character-set = utf8mb4
   EOF

6. Configure the systemd management file

   cat > /etc/systemd/system/mysql.service <<'EOF'
   [Unit]
   Description=MySQL Database Server
   Documentation=man:mysqld(8) http://dev.mysql.com/doc/
   After=network.target
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   Environment=MYSQLD_PARENT_PID=1
   User=mysql
   Group=mysql
   Type=forking
   PIDFile=/usr/local/mysql/mysqld.pid
   ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --daemonize
   ExecStop=/bin/kill -s SIGTERM $MAINPID
   LimitNOFILE=102400
   LimitMEMLOCK=infinity
   Restart=on-failure
   OOMScoreAdjust=-500
   TimeoutSec=0
   
   [Install]
   WantedBy=multi-user.target
   EOF

7. Initialize MySQL

   mysqld --no-defaults --initialize --datadir=/data/mysql/ --user=mysql --log-error=/data/logs/mysql/mysqld.log

8. Start MySQL

   systemctl daemon-reload
   systemctl enable mysql
   systemctl start mysql

9. Log in to MySQL and change the initial password

   Log in using the temporary password generated during initialization:

   mysql -uroot -p$(grep 'temporary password' /data/logs/mysql/mysqld.log | awk '{print $NF}')

   After logging in, run the following commands in MySQL:

   -- Change the current root user password
   ALTER USER USER() IDENTIFIED BY '123456';
   -- Create a root user that allows remote connections
   SET SQL_LOG_BIN=0;
   CREATE USER 'root'@'%' IDENTIFIED BY '123456';
   GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
   SET SQL_LOG_BIN=1;

   • The root user password set in the command is 123456; a strong password must be used in actual deployments.
   • If the password contains special characters, only - or _ are allowed. Characters such as @ ! # & are prohibited to avoid compatibility issues.

Build MGR Cluster

Deployment Instructions

• All-node Preparation: Creating the synchronization user, configuring channel credentials, and other basic configuration steps must be performed on all nodes.
• Bootstrapping and Grouping: Cluster initialization (Bootstrap) is performed only on the first node. The other nodes only perform the join operation.

1. Create the replication synchronization user

   Run the following commands on each node to create the account used for Group Replication synchronization:

   SET SQL_LOG_BIN=0;
   CREATE USER 'repl'@'%' IDENTIFIED BY '123456';
   GRANT REPLICATION SLAVE, CONNECTION_ADMIN, BACKUP_ADMIN, GROUP_REPLICATION_STREAM ON *.* TO 'repl'@'%';
   SET SQL_LOG_BIN=1;

   • The password for the Group Replication synchronization account repl set in the command is 123456; a strong password must be used in actual deployments.
   • If the password contains special characters, only - or _ are allowed. Characters such as @ ! # & are prohibited to avoid compatibility issues.

2. Configure the Group Replication recovery channel

   Run the following command on each node to configure the Group Replication recovery channel credentials. Replace the password with the actual password of the synchronization account repl:

   CHANGE REPLICATION SOURCE TO SOURCE_USER='repl', SOURCE_PASSWORD='123456' FOR CHANNEL 'group_replication_recovery';

3. Start and initialize the MGR group

   The MGR cluster must be bootstrapped by one node before the other nodes can join. Follow the sequence below carefully:

   • A. First node (bootstrap initialization): Run the following commands to enable bootstrap mode and initialize the cluster:

     SET GLOBAL group_replication_bootstrap_group=ON;
     START GROUP_REPLICATION;
     SET GLOBAL group_replication_bootstrap_group=OFF;

   • B. Other nodes (join the existing cluster): After the first node has been initialized successfully, run the following command on the remaining nodes to join the group:

     START GROUP_REPLICATION;

4. Check the group member information

   SELECT * FROM performance_schema.replication_group_members;

   The normal status should show all nodes with MEMBER_STATE = ONLINE, with only one PRIMARY node and the others as SECONDARY nodes.

5. Change the group_replication_start_on_boot parameter in /etc/my.cnf to ON

   Run this on each node. This parameter is set to OFF during initialization to prevent bootstrap conflicts. After the cluster is established, enable it so that nodes automatically rejoin the MGR replication group after a restart:

   sed -ri 's|group_replication_start_on_boot.*|group_replication_start_on_boot = ON|g' /etc/my.cnf

   grep group_replication_start_on_boot /etc/my.cnf

6. Perform a rolling restart and verify node status

   Run the following commands on each node in sequence. After restarting one node, confirm that its status has returned to ONLINE before proceeding to the next node:

   # 1. Restart the service
   systemctl restart mysql
   
   # 2. Check node status (ensure MEMBER_STATE is ONLINE for all currently active nodes)
   mysql -uroot -p123456 -e 'SELECT * FROM performance_schema.replication_group_members;'

Enable InnoDB Cluster

Deployment Instructions

• All-node Operations: Download and install the MySQL Shell (mysqlsh) package on all nodes.
• Single-node Operations: Initialize InnoDB Cluster only once on the first node (Primary).

1. Download the mysqlsh installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/mysql-shell-8.4.10-linux-glibc2.17-x86-64bit.tar.gz

2. Extract the package and move it to the installation path

    tar -xvf mysql-shell-8.4.10-linux-glibc2.17-x86-64bit.tar.gz
    mv mysql-shell-8.4.10-linux-glibc2.17-x86-64bit /usr/local/mysql-shell

3. Add mysqlsh to the PATH environment variable

   echo 'export PATH=/usr/local/mysql-shell/bin:$PATH' > /etc/profile.d/mysql-shell.sh
   source /etc/profile.d/mysql-shell.sh

4. Connect to the PRIMARY node (first node only)

   mysqlsh --uri root@10.206.0.2:3306 -p123456

5. Initialize and create InnoDB Cluster

   var cluster = dba.createCluster('hap-mysql', {adoptFromGR: true})

6. View cluster status

   cluster.status()

   • After logging in again, run var cluster = dba.getCluster() before using cluster.status().

MySQL Router Deployment

Deployment Instructions

To provide HA for the access layer, perform the following steps separately on each node in the cluster.

1. Download the installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/mysql-router-8.4.10-linux-glibc2.17-x86_64.tar.xz

2. Extract the package and move it to the installation path

    tar -xvf mysql-router-8.4.10-linux-glibc2.17-x86_64.tar.xz
    mv mysql-router-8.4.10-linux-glibc2.17-x86_64 /usr/local/mysql-router

3. Add MySQL Router to the PATH environment variable

   echo 'export PATH=/usr/local/mysql-router/bin:$PATH' > /etc/profile.d/mysql-router.sh
   source /etc/profile.d/mysql-router.sh

4. Initialize mysqlrouter

   mysqlrouter --bootstrap root:123456@10.206.0.2:3306  --user=mysql  --report-host=10.206.0.2

   Note:

   • The IP address after --bootstrap is the Primary node IP, which is used to read the cluster topology. Use the same Primary address on all cluster nodes.
   • The IP address after --report-host is the intranet IP of the current node. Change it to the actual IP of each node.

5. Increase the MySQL Router maximum connection limit

   sed -i '/^\[DEFAULT\]$/a max_total_connections=5000' /usr/local/mysql-router/mysqlrouter.conf

6. Configure permissions

   chown -R mysql:mysql /usr/local/mysql-router

7. Configure systemd

   cat > /etc/systemd/system/mysqlrouter.service <<'EOF'
   [Unit]
   Description=MySQL Router Service
   Documentation=man:mysqlrouter(1) https://dev.mysql.com/doc/mysql-router/en/
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   Type=simple
   User=mysql
   Group=mysql
   WorkingDirectory=/usr/local/mysql-router
   ExecStart=/usr/local/mysql-router/bin/mysqlrouter -c /usr/local/mysql-router/mysqlrouter.conf
   Restart=on-failure
   RestartSec=2
   LimitNOFILE=102400
   OOMScoreAdjust=-500
   TimeoutSec=0
   
   [Install]
   WantedBy=multi-user.target
   EOF

8. Start mysqlrouter

   systemctl daemon-reload
   systemctl enable mysqlrouter
   systemctl start mysqlrouter

9. Test the connection

   Verify the connection through MySQL Router's read-write port 6446 on the current node:

   mysql -h 127.0.0.1 -P 6446 -u root -p123456

MySQL MGR Auto Recovery

Configuration Instructions

If half or more nodes in an MGR cluster fail simultaneously, the cluster automatically enters a Complete Outage (OFFLINE/RECOVERING) state to prevent a split-brain condition. Even if the node service processes remain normal, MGR cannot normally form the group again without manual intervention.

The purpose of the auto-recovery script is to periodically detect this condition and, when the cluster is completely interrupted, select the node with the latest GTID progress to bootstrap recovery and provide self-healing. Complete the following steps on each node.

1. Create the directory and grant permissions

   # 1. Create the log directory
   mkdir -p /data/logs/mysql
   
   # 2. Ensure that the directory is owned by the mysql user (or adjust it for your actual running user)
   chown -R mysql:mysql /data/logs/mysql
   
   # 3. Set directory permissions
   chmod 755 /data/logs/mysql

2. Create the core detection and recovery script /usr/local/bin/mysql-auto-recover.sh

   Note: Be sure to check and modify the parameters in the Core Configuration section below, such as the password and node IP list, so that they match your actual deployment environment.

   cat > /usr/local/bin/mysql-auto-recover.sh <<'EOF'
   #!/bin/bash
   set -euo pipefail  # Strict mode
   
   # ============================ Core Configuration ============================
   MYSQL_CLI="/usr/local/mysql/bin/mysql"                   # MySQL client file path
   MYSQL_SHELL="/usr/local/mysql-shell/bin/mysqlsh"        # MySQL Shell file path
   MYSQL_USER="root"                                       # MySQL database administrator username
   MYSQL_PASS="123456"                                     # MySQL database administrator password
   CLUSTER_NAME="hap-mysql"                                # MGR (InnoDB Cluster) name
   LOG_FILE="/data/logs/mysql/mysql-auto-recover.log"      # Auto-recovery script log path
   LOCK_FILE="/data/logs/mysql/mysql-auto-recover.lock"    # Process exclusive lock file path
   NODES=("10.206.0.2" "10.206.0.3" "10.206.0.4")           # IP addresses of all MGR cluster nodes, space-separated and matching the cluster plan
   LOG_KEEP_DAYS=180                  # Local log retention period in days
   TRIES=3                            # Consecutive retry count for failure confirmation
   SLEEP_SEC=2                        # Wait/buffer time between failure-detection retries, in seconds
   
   # ============================ Log Functions ============================
   # Unified log format: [YYYY-MM-DD HH:MM:SS] [LEVEL] [PID] message
   # Levels: INFO (normal event) / WARN (abnormal but recoverable) / ERROR (recovery failure requiring attention)
   _log() {
       local level="$1"; shift
       printf '[%s] [%-5s] [pid=%d] %s\n' "$(date '+%F %T')" "$level" "$$" "$*" >> "$LOG_FILE"
   }
   log_info()  { _log "INFO"  "$@"; }
   log_warn()  { _log "WARN"  "$@"; }
   log_error() { _log "ERROR" "$@"; }
   
   # Add a unified prefix to each stdout/stderr line from external processes such as mysqlsh before writing it to disk.
   # Empty lines and known low-value warning noise are filtered to keep the log clean and readable.
   log_pipe() {
       local tag="$1"
       local line
       while IFS= read -r line; do
           [ -z "$line" ] && continue
           case "$line" in
               *"Using a password on the command line"*) continue ;;
               *"Cannot set LC_ALL to locale"*)          continue ;;
           esac
           _log "INFO" "[$tag] $line"
       done
   }
   
   # ============================ Basic Functions ============================
   # Concurrency control: ensure that only one detection process runs at a time.
   exec_lock() {
       exec 9>"$LOCK_FILE"
       if ! flock -n 9; then
           log_warn "Another instance is already running; exiting."
           exit 0
       fi
   }
   
   # Log rotation and cleanup: split logs by day and point the main log path to the current day's file through a symbolic link.
   rotate_logs() {
       local log_file="$1"
       local keep_days="${2:-180}"
       local log_dir base_name today today_log
   
       log_dir=$(dirname "$log_file")
       base_name=$(basename "$log_file" .log)
       today=$(date +%F)
       today_log="${log_dir}/${base_name}_${today}.log"
   
       # Create today's log file and point the main log path to it when it does not exist.
       if [ ! -f "$today_log" ]; then
           : > "$today_log"
           ln -sfn "$today_log" "$log_file"
       fi
   
       # Repair the symlink if the main log is not a symlink or does not point to today's file.
       if [ ! -L "$log_file" ] || [ "$(readlink -f "$log_file" 2>/dev/null || true)" != "$today_log" ]; then
           ln -sfn "$today_log" "$log_file"
       fi
   
       # Clean up expired logs in the current directory only to avoid accidental deletion.
       find "$log_dir" -maxdepth 1 -name "${base_name}_*.log" -type f -mtime +"$keep_days" -exec rm -f {} \; 2>/dev/null || true
   }
   
   check_node_alive() {
       local node="$1"
       if "$MYSQL_CLI" -h "$node" -u"$MYSQL_USER" -p"$MYSQL_PASS" \
           --connect-timeout=3 -e "SELECT 1;" >/dev/null 2>&1; then
           return 0
       else
           log_warn "Node $node is unreachable."
           return 1
       fi
   }
   
   check_all_nodes_alive() {
       local ok=0
       for node in "${NODES[@]}"; do
           if check_node_alive "$node"; then
               ok=$((ok+1))
           fi
       done
       echo "$ok"
   }
   
   # Return value: non-negative integer = ONLINE member count; -1 = query failed (different from a real 0).
   get_cluster_online_count() {
       local result
       result=$("$MYSQL_CLI" -u"$MYSQL_USER" -p"$MYSQL_PASS" \
           --connect-timeout=3 -N -e \
           "SELECT COUNT(*) FROM performance_schema.replication_group_members WHERE MEMBER_STATE='ONLINE';" 2>/dev/null) || {
           echo "-1"
           return 0
       }
       if [[ "$result" =~ ^[0-9]+$ ]]; then
           echo "$result"
       else
           echo "-1"
       fi
   }
   
   # Count committed transactions in gtid_executed. Supported forms:
   #   1) Single transaction: uuid:1000                  -> 1
   #   2) Range:           uuid:1-1000                    -> 1000
   #   3) Multiple ranges: uuid:1-100:200-500             -> 100 + 301 = 401
   #   4) Multiple sources: uuid1:1-100,uuid2:1-200       -> 100 + 200 = 300
   # Key point: split by ":" and skip the UUID segment to avoid counting digits or hyphens in UUIDs as transactions.
   # Use the total transaction count rather than the maximum sequence number to select the primary node.
   # This correctly handles nodes with the same highest range but different lower-range sizes
   # (for example, 1-53986:1000005-1000010 versus 1-53988:1000005-1000010).
   count_gtid_transactions() {
       local gtid="$1"
       [ -z "$gtid" ] && { echo 0; return 0; }
       echo "$gtid" | awk '
       BEGIN { total = 0 }
       {
           # Remove all whitespace characters.
           gsub(/[ \t\r\n]/, "", $0)
           # Multiple sources are separated by commas.
           n = split($0, sources, ",")
           for (i = 1; i <= n; i++) {
               # Each source has the form uuid:range[:range...]; the first UUID segment must be skipped.
               m = split(sources[i], parts, ":")
               for (j = 2; j <= m; j++) {
                   seg = parts[j]
                   if (seg ~ /^[0-9]+$/) {
                       # Single transaction.
                       total += 1
                   } else if (seg ~ /^[0-9]+-[0-9]+$/) {
                       # Closed interval [start, end]; transaction count = end - start + 1.
                       k = split(seg, range, "-")
                       start = range[1] + 0
                       end   = range[k] + 0
                       if (end >= start) total += (end - start + 1)
                   }
               }
           }
           print total
       }'
   }
   
   find_primary_node() {
       declare -A TX_MAP
       local max_tx=0
       local primary=""
       local node gtid tx
   
       for node in "${NODES[@]}"; do
           gtid=$("$MYSQL_CLI" -h "$node" -u"$MYSQL_USER" -p"$MYSQL_PASS" \
               --connect-timeout=3 -N -e \
               "SELECT @@GLOBAL.gtid_executed;" 2>/dev/null) || gtid=""
           tx=$(count_gtid_transactions "$gtid")
           TX_MAP[$node]=$tx
           log_info "Node $node GTID: executed=[$gtid] transactions=$tx"
           if [ "$tx" -gt "$max_tx" ]; then
               max_tx=$tx
               primary=$node
           fi
       done
   
       # If all node GTIDs are 0 (a new empty cluster or all queries failed), use the first node by default.
       [ -z "$primary" ] && primary="${NODES[0]}"
   
       log_info "Primary candidate elected: $primary (transactions=$max_tx)"
       echo "$primary"
   }
   
   # Check whether the current host IP matches the target node (supports multiple NICs and exact matching).
   is_local_host() {
       local target="$1"
       local ip
       for ip in $(hostname -I 2>/dev/null); do
           [ "$ip" = "$target" ] && return 0
       done
       return 1
   }
   
   do_recovery() {
       local my_ip rc
       my_ip=$(hostname -I | awk '{print $1}')
       log_info "This node ($my_ip) is the recovery target; invoking reboot_cluster_from_complete_outage."
       set +e
       # Set LC_ALL / LANG to C.UTF-8 to avoid mysqlsh locale warnings on systems without en_US.UTF-8.
       # Redirect all output to log_pipe so that it is written with the [MYSQLSH] prefix.
       # Use PIPESTATUS[0] to capture the mysqlsh exit code itself rather than log_pipe's 0.
       timeout 60 env LC_ALL=C.UTF-8 LANG=C.UTF-8 \
           "$MYSQL_SHELL" --no-wizard --py \
           --user="$MYSQL_USER" --password="$MYSQL_PASS" \
           --host=127.0.0.1 <<PYEOF 2>&1 | log_pipe "MYSQLSH"
   import sys
   try:
       cluster = dba.reboot_cluster_from_complete_outage('$CLUSTER_NAME', {'force': True})
       print("Cluster rebooted successfully.")
   except Exception as e:
       print("Recovery failed:", e)
       sys.exit(1)
   PYEOF
       rc=${PIPESTATUS[0]}
       set -e
       if [ "$rc" -eq 0 ]; then
           log_info "Cluster reboot completed successfully."
       elif [ "$rc" -eq 124 ]; then
           log_error "Cluster reboot timed out after 60 seconds; will retry on next cycle."
       else
           log_error "Cluster reboot failed (exit_code=$rc); will retry on next cycle."
       fi
   }
   
   # ============================ Main Flow ============================
   # Lock first, then rotate logs to avoid concurrent instances racing while updating symlinks or logs.
   exec_lock
   rotate_logs "$LOG_FILE" "$LOG_KEEP_DAYS"
   
   MY_IP=$(hostname -I | awk '{print $1}')
   log_info "===== MGR health check started (host=$MY_IP) ====="
   
   # Declare a complete cluster outage only after all nodes are reachable and ONLINE member count is 0 for TRIES consecutive checks.
   is_outage=false
   consecutive=0
   for attempt in $(seq 1 "$TRIES"); do
       reachable=$(check_all_nodes_alive)
       online_count=$(get_cluster_online_count)
   
       if [ "$reachable" -eq "${#NODES[@]}" ] && [ "$online_count" = "0" ]; then
           consecutive=$((consecutive + 1))
           log_warn "Attempt $attempt/$TRIES: outage signal confirmed ($consecutive/$TRIES) [reachable=$reachable, online=$online_count]"
           if [ "$consecutive" -ge "$TRIES" ]; then
               is_outage=true
               break
           fi
       else
           if [ "$consecutive" -gt 0 ]; then
               log_info "Attempt $attempt/$TRIES: outage signal reset [reachable=$reachable, online=$online_count]"
           fi
           consecutive=0
       fi
   
       # Do not sleep after the last attempt to avoid unnecessary waiting.
       [ "$attempt" -lt "$TRIES" ] && sleep "$SLEEP_SEC"
   done
   
   if [ "$is_outage" = true ]; then
       log_warn "Complete cluster outage detected; initiating recovery procedure."
       PRIMARY_NODE=$(find_primary_node)
   
       if is_local_host "$PRIMARY_NODE"; then
           do_recovery
       else
           log_info "This node ($MY_IP) is not the recovery target ($PRIMARY_NODE); standing by."
       fi
   else
       log_info "Cluster is healthy or precondition not met; no action taken."
   fi
   
   log_info "===== MGR health check finished ====="
   EOF

   Grant execution permissions:

   chmod +x /usr/local/bin/mysql-auto-recover.sh

3. Configure the systemd service unit

   Create a Service file defining how to run the script above.

   cat > /etc/systemd/system/mysql-auto-recover.service <<'EOF'
   [Unit]
   Description=MySQL MGR Auto-recovery Service
   After=network.target mysql.service
   
   [Service]
   Type=oneshot
   ExecStart=/usr/local/bin/mysql-auto-recover.sh
   User=root
   TimeoutSec=0
   EOF

4. Configure the systemd timer

   Create a Timer file to run the script once every minute.

   cat > /etc/systemd/system/mysql-auto-recover.timer <<'EOF'
   [Unit]
   Description=Run MySQL MGR Recovery check every minute
   
   [Timer]
   OnCalendar=*-*-* *:*:00
   Unit=mysql-auto-recover.service
   Persistent=true
   
   [Install]
   WantedBy=timers.target
   EOF

5. Load the configuration and start the timer

   # Reload the systemd configuration
   systemctl daemon-reload
   
   # Enable and immediately start the timer
   systemctl enable --now mysql-auto-recover.timer

6. Common commands

   The following commands are commonly used to manage the auto-recovery service and timer:

   # View the timer's next execution time
   systemctl list-timers --all | grep mysql-auto-recover
   
   # View the live operation log
   tail -f /data/logs/mysql/mysql-auto-recover.log
   
   # Manually trigger a recovery check immediately
   systemctl start mysql-auto-recover.service
   
   # View the timer service status
   systemctl status mysql-auto-recover.timer
   
   # Temporarily stop auto-recovery monitoring
   systemctl stop mysql-auto-recover.timer