cluster redis 节点宕机处理 redis cluster state fail

(1)领着选举过程是集群中所有master参与,如果半数以上master节点与master节点通信超过(cluster-node-timeout),认为当前master节点挂掉.

(2)什么时候整个集群不可用(cluster_state:fail),当集群不可用时,所有对集群的操作做都不可用，收到((error) CLUSTERDOWN The cluster is down)错误

    a:如果集群任意master挂掉,且当前master没有slave.集群进入fail状态.

    b:如果进群超过半数以上master挂掉，无论是否有slave集群进入fail状态.

1.1、redis cluster增加节点

    随着业务的增长，有一天你发现现在的redis集群规模已不能满足业务需求了，你得向集群中增加节点来扩展redis cluster的处理能力。向一个redis cluster增加节点也是一件愉快的事情，当然首先你得准备好一个空的redis实例，就像我下边一样我先准备好了一个“redis7006”实例：

redis@slave01:~$ ls redis7003 redis7004 redis7005 redis7006 redis@slave01:~$ tree redis7006 redis7006 ├── bin │ ├── redis-benchmark │ ├── redis-check-aof │ ├── redis-check-dump │ ├── redis-cli │ ├── redis-sentinel │ ├── redis-server │ └── redis-trib.rb └── redis.conf

配置文件的修改就是修改一端口号、pid文件后缀、日志文件后缀这些信息，这里就不再给出。

接着把新的实例启动起来：

redis@slave01:~$ cd redis7006/ redis@slave01:~ /redis7006 $ bin /redis-server redis.conf

新的实例准备好了，接下就把此实例增加到集群中吧，它增加到集群中是担任master，还是担任slave呢？这都是可用参数来控制的，默认不指定时是作为master。增加节点的命令如下：

. /redis-trib .rb add-node IP:PORT IP:PORT

第一个IP:PORT是指新实例的ip地址和端口号，第二个IP:PORT是指现有集群中的任意一个节点的ip地址和端口，是master可以，是slave也可以。如果增加的节点是做salve，那命令如下：

. /redis-trib .rb add-node --slave IP:PORT IP:PORT

第一个IP:PORT是指新实例的ip地址和端口号，第二个IP:PORT是指集群中任意一个节点的ip地址和端口，采用这样的方式增加的slave节点我们并不能控制他是哪个节点的从，如果要想指定加入的节点是哪个节点的从节点，那如下命令：

. /redis-trib .rb add-node --slave --master- id ID IP:PORT IP:PORT

不再解释上边命令了，也是很好理解的。

掌握了增加节点的命令，那就把redis7006这个实例加入到现有集群中，如下操作：

redis@master:~ /redis7000 $ bin /redis-trib .rb add-node 192.168.207.130:7006 192.168.207.128:7002 >>> Adding node 192.168.207.130:7006 to cluster 192.168.207.128:7002 Connecting to node 192.168.207.128:7002: OK Connecting to node 192.168.207.130:7003: OK Connecting to node 192.168.207.130:7004: OK Connecting to node 192.168.207.128:7001: OK Connecting to node 192.168.207.128:7000: OK Connecting to node 192.168.207.130:7005: OK >>> Performing Cluster Check (using node 192.168.207.128:7002) S: 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 slots: (0 slots) slave replicates 8490b5e84bf0359871ea3fa55b97bb9877be0512 M: 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 slots:5461-10922 (5462 slots) master 1 additional replica(s) M: 9d076e70871fc7291485aba97b2623dc9fb3b3b0 192.168.207.130:7004 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slots: (0 slots) slave replicates 9d076e70871fc7291485aba97b2623dc9fb3b3b0 S: f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slots: (0 slots) slave replicates 568669e03c61b3c4edc31643dcb47e85bc2c3e23 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. Connecting to node 192.168.207.130:7006: OK >>> Send CLUSTER MEET to node 192.168.207.130:7006 to make it join the cluster. [OK] New node added correctly.

成功把节点加入到集群后再来检查一下集群状态：

192.168.207.130:7003> cluster nodes b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 master - 0 1444270807484 0 connected f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slave 568669e03c61b3c4edc31643dcb47e85bc2c3e23 0 1444270805452 6 connected 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slave 9d076e70871fc7291485aba97b2623dc9fb3b3b0 0 1444270806468 7 connected 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 slave 8490b5e84bf0359871ea3fa55b97bb9877be0512 0 1444270804437 4 connected 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 master - 0 1444270804437 2 connected 10923-16383 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 myself,master - 0 0 4 connected 5461-10922 9d076e70871fc7291485aba97b2623dc9fb3b3b0 192.168.207.130:7004 master - 0 1444270804437 7 connected 0-5460

从上边的输出中可知redis7006已是一个master节点，但此时此节点不存放任何数据，因为它没有被分配hash slot，所以想让此节点真正的工作起来，你还得做一个切分操作，把在其他节点上的hash slot转移一部份到新增的节点上来，这个操作也是通过redis-trib.rb工具来完成的，具体用法如下：

. /redis-trib .rb reshard IP:PORT

采用reshard参数表示重新切分，后边的IP:PORT是集群中任意一个节点的ip地址和端口，redis-trib.rb工具会自动寻找集群中的其他节点。

现在我们就来把7004上的2000个slot转移到7006上，如下操作：

redis@master:~ /redis7000 $ bin /redis-trib .rb reshard 192.168.207.128:7000 Connecting to node 192.168.207.128:7000: OK Connecting to node 192.168.207.130:7004: OK Connecting to node 192.168.207.130:7003: OK Connecting to node 192.168.207.128:7002: OK Connecting to node 192.168.207.128:7001: OK Connecting to node 192.168.207.130:7006: OK Connecting to node 192.168.207.130:7005: OK >>> Performing Cluster Check (using node 192.168.207.128:7000) S: 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slots: (0 slots) slave replicates 9d076e70871fc7291485aba97b2623dc9fb3b3b0 M: 9d076e70871fc7291485aba97b2623dc9fb3b3b0 192.168.207.130:7004 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 slots: (0 slots) slave replicates 8490b5e84bf0359871ea3fa55b97bb9877be0512 M: 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 slots: (0 slots) master 0 additional replica(s) S: f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slots: (0 slots) slave replicates 568669e03c61b3c4edc31643dcb47e85bc2c3e23 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 2000 #输入2000 What is the receiving node ID? b502b7efac704e92ac439933a42edf613f908fea #输入哪个节点想接收这些slot，这里就是7006的ID Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:9d076e70871fc7291485aba97b2623dc9fb3b3b0 #输入想从哪个节点移动slot，我这里输入了7004的ID Source node #2:done #输入done回车后会输出具体会移动的slot Moving slot 0 from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 .........略........ Moving slot 1995 from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 Moving slot 1996 from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 Moving slot 1997 from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 Moving slot 1998 from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 Moving slot 1999 from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 Do you want to proceed with the proposed reshard plan ( yes /no )? yes #执行计划，开始真正移动slot Moving slot 0 from 192.168.207.130:7004 to 192.168.207.130:7006: .....略..... Moving slot 1993 from 192.168.207.130:7004 to 192.168.207.130:7006: Moving slot 1994 from 192.168.207.130:7004 to 192.168.207.130:7006: Moving slot 1995 from 192.168.207.130:7004 to 192.168.207.130:7006: Moving slot 1996 from 192.168.207.130:7004 to 192.168.207.130:7006: Moving slot 1997 from 192.168.207.130:7004 to 192.168.207.130:7006: Moving slot 1998 from 192.168.207.130:7004 to 192.168.207.130:7006: Moving slot 1999 from 192.168.207.130:7004 to 192.168.207.130:7006: redis@master:~ /redis7000 $

移动完成后再查看一下集群状态，如下：

192.168.207.130:7003> cluster nodes b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 master - 0 1444273978574 8 connected 0-1999 #已有2000个slot f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slave 568669e03c61b3c4edc31643dcb47e85bc2c3e23 0 1444273977559 6 connected 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slave 9d076e70871fc7291485aba97b2623dc9fb3b3b0 0 1444273975530 7 connected 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 slave 8490b5e84bf0359871ea3fa55b97bb9877be0512 0 1444273977559 4 connected 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 master - 0 1444273975530 2 connected 10923-16383 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 myself,master - 0 0 4 connected 5461-10922 9d076e70871fc7291485aba97b2623dc9fb3b3b0 192.168.207.130:7004 master - 0 1444273976544 7 connected 2000-5460 #少了2000个solt

至此，新节点已增加到集群并能正常工作。

1.2、redis cluster删除节点

    有增加节点的操作，当然也会有删除一个节点，当业务量下降后，集群中redis节点数量服务能力大大超过了业务所带来的压力，此时我们应该移除一些节点，回收主机资源，以免造成资源的浪费。

移除一个节点用如下命令：

. /redis-trib .rb del-node IP:PORT 'node-id'

 IP:PORT表示集群中任意一个节点的ip地址和端口，node-id表示你想要移除的节点ID号

接下来就测试一下删除一个节点的操作，我们计划把7004这个节点删除，那运行如下命令：

redis@master:~ /redis7000 $ bin /redis-trib .rb del-node 192.168.207.128:7000 '9d076e70871fc7291485aba97b2623dc9fb3b3b0' >>> Removing node 9d076e70871fc7291485aba97b2623dc9fb3b3b0 from cluster 192.168.207.128:7000 Connecting to node 192.168.207.128:7000: OK Connecting to node 192.168.207.130:7004: OK Connecting to node 192.168.207.130:7003: OK Connecting to node 192.168.207.128:7002: OK Connecting to node 192.168.207.128:7001: OK Connecting to node 192.168.207.130:7006: OK Connecting to node 192.168.207.130:7005: OK [ERR] Node 192.168.207.130:7004 is not empty! Reshard data away and try again.

  上边报错了，说是这个节点不是空的，需要重新切片后再尝试，这是因为在这个节点上还有2000-5460的slot，需要把这些slot移走后才能删除此节点，我们就把7004上的所有slot移到7006后再来删除此节点试试：

redis@master:~ /redis7000 $ bin /redis-trib .rb reshard --from 9d076e70871fc7291485aba97b2623dc9fb3b3b0 --to b502b7efac704e92ac439933a42edf613f908fea --slots 3461 -- yes 192.168.207.128:7000

官方命令格式是“./redis-trib.rb reshard <host>:<port> --from <node-id> --to <node-id> --slots --yes”这样的，但在实际运用时发现“<host>:<port>”需要写在最后，这个是表示集群中任意一个节点的ip地址和端口信息。

运行上表命令并等待一会儿后，数据就迁移完毕，再来观察一下集群中7004节点的状态信息：

192.168.207.130:7004> cluster nodes f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slave 568669e03c61b3c4edc31643dcb47e85bc2c3e23 0 1444283310797 6 connected 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 master - 0 1444283307772 2 connected 10923-16383 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 master - 0 1444283306763 4 connected 5461-10922 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 slave 8490b5e84bf0359871ea3fa55b97bb9877be0512 0 1444283308780 4 connected 9d076e70871fc7291485aba97b2623dc9fb3b3b0 192.168.207.130:7004 myself,master - 0 0 7 connected 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slave 9d076e70871fc7291485aba97b2623dc9fb3b3b0 0 1444283308780 7 connected b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 master - 0 1444283309788 8 connected 0-5460

看见了吧，上边输出信息中7004已没有slot了，现在再来移除此节点试试：

redis@master:~ /redis7000 $ bin /redis-trib .rb del-node 192.168.207.128:7000 '9d076e70871fc7291485aba97b2623dc9fb3b3b0' >>> Removing node 9d076e70871fc7291485aba97b2623dc9fb3b3b0 from cluster 192.168.207.128:7000 Connecting to node 192.168.207.128:7000: OK Connecting to node 192.168.207.130:7004: OK Connecting to node 192.168.207.130:7003: OK Connecting to node 192.168.207.128:7002: OK Connecting to node 192.168.207.128:7001: OK Connecting to node 192.168.207.130:7006: OK Connecting to node 192.168.207.130:7005: OK >>> Sending CLUSTER FORGET messages to the cluster... >>> 192.168.207.128:7000 as replica of 192.168.207.130:7006 >>> SHUTDOWN the node.

移除成功了，移除后还会重新计算集群中的主从关系，并把移除的节点shutdown，这里就把7000作为7006的从节点了，再来查看集群状态就是这样了：

redis@slave01:~ /redis7004 $ bin /redis-cli -c -h 192.168.207.130 -p 7003 192.168.207.130:7003> cluster nodes b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 master - 0 1444283726189 8 connected 0-5460 f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slave 568669e03c61b3c4edc31643dcb47e85bc2c3e23 0 1444283728207 6 connected 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slave b502b7efac704e92ac439933a42edf613f908fea 0 1444283725182 8 connected #看下他的主节点是7006 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 slave 8490b5e84bf0359871ea3fa55b97bb9877be0512 0 1444283729216 4 connected 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 master - 0 1444283728207 2 connected 10923-16383 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 myself,master - 0 0 4 connected 5461-10922

1.3、redis cluster主从手动切换

    也许在一些特殊的场景你需要切换集群中主从节点，比如你想让一个主节点下线升级维护时，当然你可简单粗爆的把这个主节点shutdown，然后redis cluster会监测到集群中少了一个主节点，集群就会把他的从节点提升为主节点，然后你再把shutdown的节点启动，那他会成为一个从节点运行，这样你就可以做你想做的了。但这样粗爆的行为是会导致集群短暂的停止服务，有一个切换的时间，如果是一个可控的维护工作，我们还是希望更为平滑，那集群中手动切换工具就能满足我们的需求。

    我们先连接到集群中，查看一下集群状态信息，如下：

redis@slave01:~ /redis7004 $ bin /redis-cli -c -h 192.168.207.128 -p 7002 #连接到7002从节点 192.168.207.128:7002> cluster nodes 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 master - 0 1444285238614 4 connected 5461-10922 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 myself,slave 8490b5e84bf0359871ea3fa55b97bb9877be0512 0 0 3 connected #7002的主节点是7003 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 master - 0 1444285240634 2 connected 10923-16383 b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 master - 0 1444285237604 8 connected 0-5460 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slave b502b7efac704e92ac439933a42edf613f908fea 0 1444285239625 8 connected f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slave 568669e03c61b3c4edc31643dcb47e85bc2c3e23 0 1444285239625 6 connected

接下来就进行手动切换操作，把7002与7003的主从关系调换：

192.168.207.128:7002> cluster failover #这就是切换命令 OK 192.168.207.128:7002> cluster nodes 8490b5e84bf0359871ea3fa55b97bb9877be0512 192.168.207.130:7003 slave 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 0 1444285980634 9 connected #变成slave了 0becd52cb1fa1a69f8d3135dd98f87cd8ccf9d78 192.168.207.128:7002 myself,master - 0 0 9 connected 5461-10922 #变成master了 568669e03c61b3c4edc31643dcb47e85bc2c3e23 192.168.207.128:7001 master - 0 1444285977604 2 connected 10923-16383 b502b7efac704e92ac439933a42edf613f908fea 192.168.207.130:7006 master - 0 1444285976594 8 connected 0-5460 7dae923773125c5956605fac6159412850b100b3 192.168.207.128:7000 slave b502b7efac704e92ac439933a42edf613f908fea 0 1444285979624 8 connected f3ba7c62307e0321f5d21310d14027c403c73907 192.168.207.130:7005 slave 568669e03c61b3c4edc31643dcb47e85bc2c3e23 0 1444285978615 6 connected 192.168.207.128:7002>

在上边的主从切换测试中几乎是瞬间完成，这样业务是不受到影响的。值得注意的是“cluster failover”命令只能在slave节点上运行.