Redis高可用方案（redis主从+keepalived+sentinel）

原创

kuankuan001vip 2015-05-05 17:56:16 ©著作权

文章标签 高可用 keepalived Redis sentinel 文章分类 Redis 数据库

©著作权归作者所有：来自51CTO博客作者kuankuan001vip的原创作品，请联系作者获取转载授权，否则将追究法律责任

架构：redis主从+keepalived+sentinel

三台机器，两台redis主从，一台配合选举sentinel leader机器。

Master: 192.168.100.135 controller 部署redis+keepalived+sentinel

Slave: 192.168.100.136 web-nb-136 部署redis+keepalived+sentinel

配合sentinel: 192.168.100.128 WEB-NB-128 配置sentinel

VIP: 192.168.100.140

redis版本: redis-2.8.6.tar.gz 自带sentinel功能

keepalived版本: keepalived-1.2.13.tar.gz

四个实例6379、6380、6381、6382

测试结论:

第一种情况:
advert_int 4
down-after-milliseconds 3000
failover-timeout 9000
failover用时: 10-15s

第二种情况:
advert_int 3
down-after-milliseconds 2000 (时间太短，会导致主从角色不稳定，有可能出现主从来回切换的现象，从而导致服务不可用,2000ms是最低极限)
failover-timeout 6000
failover用时: 5-10s 确切用时:6-7s

135上的操作:

1. 安装配置redis

（1）安装redis

[root@controller ~]# cat install_redis_2-8-6.sh

yum install tcl -y
wget http://download.redis.io/releases/redis-2.8.6.tar.gz
tar zxvf redis-2.8.6.tar.gz
cd redis-2.8.6;make;cd src
cp redis-server /usr/local/bin/
cp redis-cli /usr/local/bin/
cp redis-sentinel /usr/local/bin/
cp redis-check-aof redis-check-dump redis-benchmark /usr/local/bin/
mkdir /etc/redis /var/log/redis /var/run/redis/
mkdir -p /var/redis/redis_{6379,6380,6381,6382}

[root@controller ~]# sh install_redis_2-8-6.sh

（2）配置redis

① 实例6379:

[root@controller ~]# cd /etc/redis/

[root@controller redis]# cat redis_6379.conf
daemonize yes
pidfile "/var/run/redis/redis_6379.pid"
port 6379
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis_6379.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump_6379.rdb"
dir "/var/redis/redis_6379"
maxmemory 4gb
slave-read-only yes
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

② 实例6380:

[root@controller redis]# cat redis_6380.conf
daemonize yes
pidfile "/var/run/redis/redis_6380.pid"
port 6380
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis_6380.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump_6380.rdb"
dir "/var/redis/redis_6380"
maxmemory 2gb
slave-read-only yes
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

③ 实例6381:

[root@controller redis]# cat redis_6381.conf
daemonize yes
pidfile "/var/run/redis/redis_6381.pid"
port 6381
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis_6381.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump_6381.rdb"
dir "/var/redis/redis_6381"
maxmemory 2gb
slave-read-only yes
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

④ 实例6382:

[root@controller redis]# cat redis_6382.conf
daemonize yes
pidfile "/var/run/redis/redis_6382.pid"
port 6382
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis_6382.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump_6382.rdb"
dir "/var/redis/redis_6382"
maxmemory 2gb
slave-read-only yes
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

2.安装配置keepalived

（1）安装keepalived

[root@controller ~]# cat install_keepalived-1.2.13.sh
#!/bin/bash
wget -qO keepalived-1.2.13.tar.gz http://www.keepalived.org/software/keepalived-1.2.13.tar.gz
yum install openssl openssl-devel -y
tar zxvf keepalived-1.2.13.tar.gz;cd keepalived-1.2.13
./configure --prefix=/usr/local/keepalived
make && make install
mkdir -p /etc/keepalived/{scripts,log}
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/bin/genhash /usr/sbin/

（2）配置keepalived

[root@controller ~]# cd /etc/keepalived/

[root@controller keepalived]# cat keepalived.conf
global_defs {
lvs_id LVS_redis
}

vrrp_script chk_redis {
script "/etc/keepalived/scripts/redis_check.sh"
    weight -20
    interval 2
}

vrrp_instance VI_1 {
state backup
interface eth0
virtual_router_id 52
nopreempt
priority 200
advert_int 4

virtual_ipaddress {
    192.168.100.140
}

track_script {
    chk_redis
}

notify_master /etc/keepalived/scripts/redis_master.sh
notify_stop /etc/keepalived/scripts/keepalived_stop.sh

}

（3）编写keepalived监测以及触发脚本

① 监测脚本redis_check.sh

该脚本用于监测实例运行状态。

[root@controller scripts]# cd scripts/

[root@controller scripts]# cat redis_check.sh
#!/bin/bash
Vip=192.168.100.140
PortGroup=(6379 6380 6381 6382)
VipValue=`/sbin/ip add|grep $Vip`
RedisCli_Cmd="/usr/local/bin/redis-cli"
Count=0

case_value(){
case $1 in
     RedisRole)
        Value=`$RedisCli_Cmd -p $Port info|awk -F'[:|\r]' '/role/{print $2}'`
        ;;
     Alive)
        Value=`$RedisCli_Cmd -p $Port PING`
esac
}

sub_value(){
for Port in ${PortGroup[@]};do
     case_value $1
     if [ $Value = "$2" ];then
          let "Count = $Count + 1"
          if [ $Count -eq 4 ];then exit 0;fi
     else
          `which pkill` keepalived
          exit 1
     fi
done
}

if [ -n "$VipValue" ];then
sub_value RedisRole master
else
sub_value Alive PONG
fi

[root@controller scripts]# chmod +x redis_check.sh

② 触发脚本keepalived_stop.sh

在keepalived状态发生变化或服务异常停止时触发执行该脚本。

[root@controller scripts]# cat keepalived_stop.sh
#!/bin/bash
`which pkill` redis-server

[root@controller scripts]# chmod +x keepalived_stop.sh

③ 开机启动脚本keepalived_start.sh

用于自动解决keepalived启动顺序的问题.

[root@controller ~]# cat /etc/keepalived/scripts/keepalived_start.sh
#!/bin/bash

RedisRole=`/usr/local/bin/redis-cli -p 6379 info|awk -F'[:|\r]' '/role/{print $2}'`
KeepalivedStartCmd="/etc/init.d/keepalived start"

if [ $RedisRole = "master" ];then
        $KeepalivedStartCmd
else
        while true;
        do
                sleep 1
                ping 192.168.100.140 -c 1 >/dev/null 2>&1
                if [ $? -eq 0 ];then $KeepalivedStartCmd;break;fi
        done
fi

[root@controller ~]# chmod +x /etc/keepalived/scripts/keepalived_start.sh

3.配置sentinel

[root@controller keepalived]# cd /etc/redis/

[root@controller redis]# cat sentinel.conf

port 26379

daemonize yes

logfile "/var/log/redis/sentinel.log"

sentinel monitor MyMaster6379 192.168.100.135 6379 2
sentinel down-after-milliseconds MyMaster6379 2000
sentinel failover-timeout MyMaster6379 6000
sentinel config-epoch MyMaster6379 1

sentinel monitor MyMaster6380 192.168.100.135 6380 2
sentinel down-after-milliseconds MyMaster6380 2000
sentinel failover-timeout MyMaster6380 6000
sentinel config-epoch MyMaster6380 1

sentinel monitor MyMaster6381 192.168.100.135 6381 2
sentinel down-after-milliseconds MyMaster6381 2000
sentinel failover-timeout MyMaster6381 6000
sentinel config-epoch MyMaster6381 1

sentinel monitor MyMaster6382 192.168.100.135 6382 2
sentinel down-after-milliseconds MyMaster6382 2000
sentinel failover-timeout MyMaster6382 6000
sentinel config-epoch MyMaster6382 1

注释:三台机器的sentinel.conf的配置除端口不同之外，其他都是一样的.

需要注意的几个地方：

1、注意第一行最后一个2，意思是当有2个sentinel实例同时检测到redis异常时，才会有反应。（测试过程中使用的是1）

2、主从切换后，redis.conf、sentinel.conf内容都会改变，如果还想要原来的主从架构，需要再修改配置文件，并重新启动；

3、 master挂掉，sentinel已经选择了新的master，但是还没有将其改成master，但是已经将old master改成了slave。那么这时候如果重启old master，就会处于无主状态。所以一方面要等sentinel稳定后再启动old master，或者重新人工修改配置文件，重新启动集群。

4、sentinel只是在server端做主从切换，app端要自己开发，例如Jedis库的SentinelJedis，能够监控sentinel的状态。这样才能完整的实现高可用性的主从切换。（本测试使用keepalived的VIP漂移技术实现透明化服务）

4. 加入开机启动

[root@controller redis]# cat /etc/rc.local

# Redis 2.8.6
redis-server /etc/redis/redis_6379.conf
redis-server /etc/redis/redis_6380.conf
redis-server /etc/redis/redis_6381.conf
redis-server /etc/redis/redis_6382.conf

# Redis-sentinel

redis-sentinel /etc/redis/sentinel.conf

# Keepalived

#/etc/init.d/keepalived start

/bin/bash /etc/keepalived/scripts/keepalived_start.sh

136上的操作:

1. 安装配置redis

（1）安装redis

安装方法和135上的redis安装方法相同.

（2）配置redis

① 实例6379:

[root@controller ~]# cd /etc/redis/

slaveof 192.168.100.135 6379

② 实例6380:

slaveof 192.168.100.135 6380

③ 实例6381:

slaveof 192.168.100.135 6381

④ 实例6382:

slaveof 192.168.100.135 6382

2.安装配置keepalived

（1）安装keepalived

安装方法和135上的相同.

（2）配置keepalived

[root@web-nb-136 keepalived]# cat keepalived.conf
global_defs {
lvs_id LVS_redis
}

vrrp_script chk_redis {
script "/etc/keepalived/scripts/redis_check.sh"
weight -20
interval 2
}

vrrp_instance VI_1 {
state backup
interface eth0
virtual_router_id 52
priority 100
advert_int 4
virtual_ipaddress {
   192.168.100.140
}

track_script {
   chk_redis
}

notify_master /etc/keepalived/scripts/redis_master.sh
   notify_stop /etc/keepalived/scripts/keepalived_stop.sh
}

（3）编写keepalived监测以及触发脚本

和135上的①②③的脚本相同.

3.配置sentinel

[root@web-nb-136 redis]# cat sentinel.conf

port 26479

daemonize yes

logfile "/var/log/redis/sentinel.log"

4. 加入开机启动

[root@web-nb-136 redis]# cat /etc/rc.local

# Redis 2.8.6
redis-server /etc/redis/redis_6379.conf
redis-server /etc/redis/redis_6380.conf
redis-server /etc/redis/redis_6381.conf
redis-server /etc/redis/redis_6382.conf

# Redis-sentinel

redis-sentinel /etc/redis/sentinel.conf

# Keepalived
#/etc/init.d/keepalived start

/bin/bash /etc/keepalived/scripts/keepalived_start.sh

128上的操作:

128只是用来配合监测redis maser异常过程中进行重新选举领头（leader） Sentinel 时充当人头的.128只需要配置启动sentinel即可（前面我们已经在135和136上配置了两个sentinel）。

因为一个 Sentinel 都需要获得系统架构中多数（majority） Sentinel 的支持，才能发起一次自动故障迁移，并预留一个给定的配置纪元（configuration Epoch ，一个配置纪元就是一个新主服务器配置的版本号）。

换句话说， 在只有少数（minority） Sentinel 进程正常运作的情况下， Sentinel 是不能执行自动故障迁移的。

由于redis-2.8.6已经集合sentinel，因此需要先安装redis，在配置sentinel.

1.安装redis

安装方法和135上的redis安装方法相同.

2.配置sentinel

[root@WEB-NB-128 ~]# cat /etc/redis/sentinel.conf

port 26579

daemonize yes

logfile "/var/log/redis/sentinel.log"

3.加入开机启动

[root@WEB-NB-128 ~]# cat /etc/rc.local

# Redis-sentinel

redis-sentinel /etc/redis/sentinel.conf

需要注意几个极限的问题:

1. 整个系统部署完成最初，Redis主从在启动keepalived服务时是有顺序的；要点就是要保证先启动keepalived服务的机器是Redis Master。

2. 考虑到后期维护期间，机房意外断电的情况，当电源恢复正常后，如何保证Redis主从keepalived的先后启动顺序。

解决思路: 自定义keepalived开机启动脚本，做相应的判断，问题即可得到解决。本测试中该问题已经得到解决。

3. 当Master失效之后，故障自动切换完成，切忌不要立刻启动失效的Old Master，要等到Sentinel稳定（也就是新的Master已经成功接管读写任务）之后再启动Old Master，当然此时的Old

Master 会自动变为新Master的Slave，并向新的Master发起同步请求。

4. 线上部署时，会涉及到防火墙的配置，主要考虑到以下几个要点就不会有问题

① 允许哪些来源ip（网段）可以访问本机redis实例端口；

② 允许哪些来源ip（网段）可以访问本机的sentinel端口；

③ 允许vrrp协议可以通过本机

举例:

-A INPUT -p vrrp -j ACCEPT
-A INPUT -s 192.168.100.0/24 -p tcp -m tcp --dport 6379 -j ACCEPT
-A INPUT -s 192.168.100.0/24 -p tcp -m tcp --dport 6380 -j ACCEPT
-A INPUT -s 192.168.100.0/24 -p tcp -m tcp --dport 6381 -j ACCEPT
-A INPUT -s 192.168.100.0/24 -p tcp -m tcp --dport 6382 -j ACCEPT
-A INPUT -s 192.168.100.0/24 -p tcp -m tcp --dport 26379 -j ACCEPT