Redis Master-Slave + Keepalived HA Service Configuration

IP	Role
192.168.1.13	redis Node01
192.168.1.14	redis Node02
192.168.1.100	VIP

Configure Redis

1. Refer to the Redis master-slave documentation to complete the deployment.

2. Modify the redis.conf file on the Master node to add the slaveof parameter for synchronizing the slave nodes.

   slaveof 192.168.1.14 6379

Configure Keepalived

Install Keepalived

yum install -y keepalived

Modify Keepalived Configuration File

Redis Node01

Modify the contents of the /etc/keepalived/keepalived.conf file as follows:

global_defs {
    router_id hap-redis-ha-01
}

vrrp_script check_redis_health {
    script "/usr/local/redis/script/check_redis_health.sh"
    interval 10
}

vrrp_sync_group VG1 {
    group {
        VI_1
    }
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0          # Change to the network interface of the deployment host
    virtual_router_id 180   # Unique among all VRRP routers on the same subnet, range 0-255
    priority 100            # Priority, different for each node
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass HAP-redis-Keepalived-Auth
    }
    track_script {
        check_redis_health
    }
    notify_master /usr/local/redis/script/notify_master.sh
    notify_backup /usr/local/redis/script/notify_backup.sh
    virtual_ipaddress {
        192.168.1.100      # VIP address
    }
}

• Note that the priority values in the configuration files of the two nodes are different.
• By default, only interface and VIP address need to be modified.

Redis Node02

Modify the contents of the /etc/keepalived/keepalived.conf file as follows:

global_defs {
    router_id hap-redis-ha-01
}

vrrp_script check_redis_health {
    script "/usr/local/redis/script/check_redis_health.sh"
    interval 10
}

vrrp_sync_group VG1 {
    group {
        VI_1
    }
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0          # Change to the network interface of the deployment host
    virtual_router_id 180   # Unique among all VRRP routers on the same subnet, range 0-255
    priority 90             # Priority, different for each node
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass HAP-redis-Keepalived-Auth
    }
    track_script {
        check_redis_health
    }
    notify_master /usr/local/redis/script/notify_master.sh
    notify_backup /usr/local/redis/script/notify_backup.sh
    virtual_ipaddress {
        192.168.1.100      # VIP address
    }
}

• Note that the priority values in the configuration files of the two nodes are different.
• By default, only interface and VIP address need to be modified.

Create Redis Related Scripts

Both nodes need to create the following Redis-related scripts

Create a directory to store scripts

mkdir -p /usr/local/redis/script/

Execute vim /usr/local/redis/script/check_redis_health.sh to create the Redis status check script.

The contents of the check_redis_health.sh script are as follows:

#!/bin/bash

# Redis related information
redis_cli="/usr/local/redis/bin/redis-cli"
redis_host="127.0.0.1"
redis_port="6379"
redis_password="123456"

# Log function
redis_keepalived_log_file="/var/log/redis_keepalived.log"
log_info() {
    echo "$(date +"%Y-%m-%d %H:%M:%S") INFO: $1" >> "$redis_keepalived_log_file"
}

# Check configuration
max_retries=3
retry_interval=2

check_redis_status() {
    # Check if the Redis service is running
    $redis_cli -h $redis_host -p $redis_port -a $redis_password ping > /dev/null 2>&1
    redis_status=$?
}

# Continuously check the Redis service status, retrying a maximum of max_retries times
retry_count=0
while true; do
    check_redis_status

    if [ $redis_status -eq 0 ]; then
        # Redis service is normal, exit the loop
        exit 0
    else
        if [ $retry_count -ge $max_retries ]; then
            # Maximum retry attempts reached, execute an operation to stop Keepalived service and exit
            log_info "After $max_retries retries, Redis remains unhealthy, stop Keepalived."
            systemctl stop keepalived
            exit 0
        fi
    fi

    retry_count=$((retry_count + 1))
    sleep $retry_interval
done

Execute vim /usr/local/redis/script/notify_master.sh to create the script that Keepalived executes when this node is elected as Master.

The contents of the notify_master.sh script are as follows:

#!/bin/bash

# Redis related information
redis_host="127.0.0.1"
redis_port="6379"
redis_password="123456"
redis_cli="/usr/local/redis/bin/redis-cli"
redis_config="/usr/local/redis/redis.conf"

# Log function
redis_keepalived_log_file="/var/log/redis_keepalived.log"
log_info() {
    echo "$(date +"%Y-%m-%d %H:%M:%S") INFO: $1" >> "$redis_keepalived_log_file"
}

# Stop the replication thread on the current node
if $redis_cli -h $redis_host -p $redis_port -a $redis_password <<< "slaveof no one"; then
    log_info "The current node becomes master, successfully stopped redis master-slave replication on the current node."
else
    log_info "The current node becomes master, failed to stop redis master-slave replication on the current node."
fi

# Comment out `slaveof` on the current node
if sed -i '/^slaveof/ s/^/#/' $redis_config; then
    log_info "Successfully commented out the slaveof for the current node."
else
    log_info "Failed to comment out the slaveof for the current node."
fi

Execute vim /usr/local/redis/script/notify_backup.sh to create the script that Keepalived executes when this node is elected as Backup.

The contents of the notify_backup.sh script are as follows:

#!/bin/bash

# Redis related information
redis_host="127.0.0.1"
redis_port="6379"
redis_password="123456"
redis_cli="/usr/local/redis/bin/redis-cli"
redis_config="/usr/local/redis/redis.conf"
redis_master_host="192.168.1.13"

# Log function
redis_keepalived_log_file="/var/log/redis_keepalived.log"
log_info() {
    echo "$(date +"%Y-%m-%d %H:%M:%S") INFO: $1" >> "$redis_keepalived_log_file"
}

# Start the replication thread on the current node
if $redis_cli -h $redis_host -p $redis_port -a $redis_password <<< "slaveof $redis_master_host 6379"; then
    log_info "The current node becomes backup, successfully started redis master-slave replication on the current node."
else
    log_info "The current node becomes backup, failed to start redis master-slave replication on the current node."
fi

# Uncomment `slaveof` on the current node
if sed -i '/^#slaveof/ s/^#//' $redis_config; then
    log_info "Successfully uncommented the slaveof configuration for the current node."
else
    log_info "Failed to uncomment the slaveof configuration for the current node."
fi

• In the configuration files of the two nodes, the variable redis_master_host is not the same. Make sure to replace it with the IP address of the other Redis server.

Add execute permissions to all scripts

chmod +x /usr/local/redis/script/*.sh

Start Keepalived

systemctl start keepalived
systemctl enable keepalived

Testing

You can observe whether the setup meets expectations by restarting Keepalived.

Expected behavior:

• On the node with the VIP, Redis is in master state, and the slaveof parameter in the redis.conf file is commented out.
• On the non-VIP node, Redis is in slave synchronization state, and the slaveof parameter in the redis.conf file is not commented out.

Troubleshooting

Both Servers Have VIP

This may usually be caused by:

• Firewall or other network restrictions between the two machines

  • Ports are restricted (default is port 112)
  • Network environment does not support VRRP (Virtual Router Redundancy Protocol)

• The network between the two machines is unreachable with each other's addresses

• The priority configured on both nodes may be the same, causing failure to elect a master node

• You can troubleshoot via the /var/log/messages log file

Neither Server Has VIP

This may usually be caused by:

• The script file specified in keepalived.conf vrrp_script has execution errors and always returns a non-zero error value, causing Keepalived's state to remain uninitialized and unable to bind the VIP

  • For example, Redis is not started, or the password connecting to Redis in the script file is incorrect. You can manually execute the script for troubleshooting.

• You can troubleshoot via the /var/log/messages log file