推荐学习

  • Redis让我凉在美团一面，摸爬滚打3个月，二战逆袭成功斩获Offer
  • 缓存架构技术：Redis+MongDB，阿里P7面试必跳的坑



两台服务器搭建redis集群 2台redis集群_两台服务器搭建redis集群


Redis Cluster主从模式

原理

redis cluster 为了保证数据的高可用性，加入了主从模式，一个主节点对应一个或多个从节点，主节点提供数据存取，从节点则是从主节点拉取数据。
备份，当这个主节点挂掉后，就会有这个从节点选取一个来充当主节点，从而保证集群不会挂掉。

说明

假如集群有A、B、C三个主节点, 如果这3个节点都没有加入从节点，如果B挂掉了，我们就无法访问整个集群了， A和C的slot也无法访问。
所以在集群建立的时候，一定要为每个主节点都添加了从节点,。比如，像集群包含主节点A、B、C, 以及从节点A1、B1、C1, 那么即使B挂掉，系统也可以继续正确工作 ，B1节点替代了B节点，故Redis集群将会选择B1节点，作为新的主节点，集群将会继续正确地提供服务 当B重新开启后，它就会变成B1的从节点。

注意

如果节点B和B1同时挂了，Redis集群就无法继续正确地提供服务了。

故最小的redis集群需要3个主节点(即Redis Cluster集群至少需要3个master节点，也就是说至少需要6个节点才能构建Redis cluster集群)，一台机器可运行多个redis实例(一般使用两台机器，每台启动3个redis实例，即三个主节点，三个从节点)。

很多案例使用单台服务器开6个端口，操作差不多，只是配置基本相对简单点，多台服务器更接近生产环境 。

当集群最开始创建好后，要记住各节点的主从关系(或是创建的时候指定主从关系)；若是其中一台机器重启，重启后，需重新将其加入到redis cluster集群中；这就需要将这台机器上的各节点之前的从节点变为主节点(客户端执行slaveof no one)，然后再根据新的主节点，添加这台机器的各节点到集群中，添加后变为从节点。

搭建环境

环境 : CentOS7

redis01 : 172.16.7.11 端口 : 7000、7001、7002

redis02 : 172.16.7.22 端口 : 7003、7004、7005

redis03 : 172.16.7.33 端口 : 7006、7007、7008

关闭SElinux和防火墙

安装redis

首先选择一台服务器，进行安装

初始安装redis01

[root@redis01 ~]# yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git[root@redis01 ~]#  wget http://download.redis.io/releases/redis-4.0.1.tar.gz[root@redis01 ~]# tar xf redis-4.0.1.tar.gz -C /usr/local/src/[root@redis01 ~]# cd /usr/local/src/redis-4.0.1/[root@redis01 redis-4.0.1]# make && make test && make PREFIX=/usr/local/redis install[root@redis01 redis-4.0.1]# mkdir /usr/local/redis/conf[root@redis01 redis-4.0.1]# cp *.conf /usr/local/redis/conf/[root@redis01 redis-4.0.1]# cp /usr/local/src/redis-4.0.1/src/redis-trib.rb /usr/local/redis/bin/[root@redis01 redis-4.0.1]# ln -s /usr/local/redis/bin/* /usr/local/bin/

进行操作系统基础调优设置

[root@redis01 redis-4.0.1]# echo "* - nofile 10240" >> /etc/security/limits.conf [root@redis01 redis-4.0.1]# echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf [root@redis01 redis-4.0.1]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf [root@redis01 redis-4.0.1]# sysctl -pnet.core.somaxconn = 10240vm.overcommit_memory = 1[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/enabled[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/defrag[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local[root@redis01 redis-4.0.1]# su -l

创建节点

创建集群节点目录

[root@redis01 ~]# mkdir /data/redis/redis-cluster -p[root@redis01 redis-cluster]# cd /data/redis/redis-cluster/[root@redis01 redis-cluster]# mkdir 7000 7001 7002

执行下面的脚本，配置各个节点的redis.conf

[root@redis01 redis-cluster]# for i in 0 1 2; do echo -e "cluster-enabled yesbind 0.0.0.0port 700$ipidfile /data/redis-cluster/700$i/redis.pidlogfile "/data/redis-cluster/700$i/redis.log"dir /data/redis-cluster/7000/cp-backlog 1024imeout 0cluster-node-timeout 10100cp-keepalive 0daemonize yesloglevel noticedatabases 16stop-writes-on-bgsave-error yesdbcompression yesdbchecksum yesdbfilename "dump.rdb"slave-serve-stale-data yesslave-read-only yesepl-diskless-sync noepl-diskless-sync-delay 5epl-disable-tcp-nodelay noslave-priority 100lazyfree-lazy-eviction nolazyfree-lazy-expire nolazyfree-lazy-server-del noslave-lazy-flush noappendonly noappendfilename "appendonly.aof"appendfsync everyseco-appendfsync-on-rewrite yesauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbaof-load-truncated yeslua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128latency-monitor-threshold 0otify-keyspace-events ""hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64hll-sparse-max-bytes 3000activerehashing yesclient-output-buffer-limit normal 0 0 0client-output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes" > 700$i/redis.conf; done

查看各个节点的文件树

[root@redis01 redis-cluster]# tree .├── 7000│   └── redis.conf├── 7001│   └── redis.conf└── 7002    └── redis.conf

启动redis01的集群服务

[root@redis02 redis-cluster]# mv 7000 7003[root@redis02 redis-cluster]# mv 7001 7004[root@redis02 redis-cluster]# mv 7002 7005[root@redis02 redis-cluster]# ls7003  7004  7005[root@redis02 redis-cluster]# sed -i 's#7000#7003#g' 7003/redis.conf[root@redis02 redis-cluster]# sed -i 's#7001#7004#g' 7005/redis.conf[root@redis02 redis-cluster]# sed -i 's#7002#7005#g' 7004/redis.conf

redis02 服务器的安装

redis02 ，redis03不需要完全按上面的步骤，只部分需要复制redis01的安装解压目录

复制redis01的安装解压目录

[root@node02 ~]# scp -r node01:/usr/local/redis /usr/local/


两台服务器搭建redis集群 2台redis集群_ruby_02


[root@redis02 ~]# scp -r node01:/data/redis/redis-cluster /data/redis/


两台服务器搭建redis集群 2台redis集群_ruby_03


修改某个服务器的各个节点的redis.conf文件

[root@redis02 redis-cluster]# mv 7000 7003[root@redis02 redis-cluster]# mv 7001 7004[root@redis02 redis-cluster]# mv 7002 7005[root@redis02 redis-cluster]# ls7003  7004  7005[root@redis02 redis-cluster]# sed -i 's#7000#7003#g' 7003/redis.conf[root@redis02 redis-cluster]# sed -i 's#7001#7004#g' 7005/redis.conf[root@redis02 redis-cluster]# sed -i 's#7002#7005#g' 7004/redis.conf

做运行脚本加入bin目录 

[root@node02 ~]# ln -s /usr/local/redis/bin/* /usr/local/bin/

进行操作系统基础调优设置

如同redis01的操作系统基础调优设置

启动redis02的集群服务

[root@redis02 redis-cluster]# for i in 3 4 5; do redis-server /data/redis/redis-cluster/700$i/redis.conf  done41766:C 08 Oct 22:21:42.369 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo41766:C 08 Oct 22:21:42.369 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41766, just started41766:C 08 Oct 22:21:42.369 # Configuration loaded41768:C 08 Oct 22:21:42.375 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo41768:C 08 Oct 22:21:42.375 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41768, just started41768:C 08 Oct 22:21:42.375 # Configuration loaded41770:C 08 Oct 22:21:42.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo41770:C 08 Oct 22:21:42.380 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41770, just started41770:C 08 Oct 22:21:42.380 # Configuration loaded

redis03服务器的安装

类比redis02 服务器的安装

redis01，redis02，redis03聚合创建集群

查各Redis节点的启动状态

#yum -y install net-tools[root@redis01 redis-cluster]# netstat -antup|grep redistcp        0      0 172.16.7.11:17000     0.0.0.0:*               LISTEN      68821/redis-server  tcp        0      0 172.16.7.11:17001     0.0.0.0:*               LISTEN      68823/redis-server  tcp        0      0 172.16.7.11:17002     0.0.0.0:*               LISTEN      68828/redis-server  tcp        0      0 172.16.7.11:7000      0.0.0.0:*               LISTEN      68821/redis-server  tcp        0      0 172.16.7.11:7001      0.0.0.0:*               LISTEN      68823/redis-server  tcp        0      0 172.16.7.11:7002      0.0.0.0:*               LISTEN      68828/redis-server

安装ruby

[root@redis01 redis-cluster]# yum -y install rh-ruby23[root@redis01 redis-cluster]# ruby -vruby 2.0.0p648 (2015-12-16) [x86_64-linux][root@redis01 redis-cluster]# scl enable rh-ruby23 bash[root@redis01 redis-cluster]# ruby -vruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]


两台服务器搭建redis集群 2台redis集群_redis两台机器集群_04


解决方案；

yum install centos-release-scl-rh


两台服务器搭建redis集群 2台redis集群_ruby_05


再次安装 rh-ruby23


两台服务器搭建redis集群 2台redis集群_两台服务器搭建redis集群_06


安装ruby的redis扩展

[root@redis01 redis-cluster]# gem install redisFetching: redis-4.0.2.gem (100%)Successfully installed redis-4.0.2Parsing documentation for redis-4.0.2Installing ri documentation for redis-4.0.2Done installing documentation for redis after 0 seconds1 gem installed

使用redis自带的集群管理工具redis-trib.rb创建集群

在redis01上运行以下脚本

[root@redis01 redis-cluster]# redis-trib.rb create --replicas 1 172.16.7.11:7000 172.16.7.11:7001 172.16.7.11:7002 172.16.7.22:7003 172.16.7.22:7004 172.16.7.22:7005 172.16.7.33:7006 172.16.7.33:7007 172.16.7.33:7008>>> Creating cluster>>> Performing hash slots allocation on 9 nodes...Using 4 masters:172.16.7.11:7000172.16.7.22:7003172.16.7.33:7006172.16.7.11:7001Adding replica 172.16.7.22:7004 to 172.16.7.11:7000Adding replica 172.16.7.33:7007 to 172.16.7.22:7003Adding replica 172.16.7.11:7002 to 172.16.7.33:7006Adding replica 172.16.7.22:7005 to 172.16.7.11:7001Adding replica 172.16.7.33:7008 to 172.16.7.11:7000M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000   slots:0-4095 (4096 slots) masterM: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001   slots:12288-16383 (4096 slots) masterS: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002   replicates af2e1c600ed2511155a6cad61cabeb095affff86M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003   slots:4096-8191 (4096 slots) masterS: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004   replicates d8138e15d9cb69962abb7457125a7965b735a658S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168dM: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006   slots:8192-12287 (4096 slots) masterS: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008   replicates d8138e15d9cb69962abb7457125a7965b735a658Can I set the above configuration? (type 'yes' to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join........>>> Performing Cluster Check (using node 172.16.7.11:7000)M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000   slots:0-4095 (4096 slots) master   2 additional replica(s)S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007   slots: (0 slots) slave   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004   slots: (0 slots) slave   replicates d8138e15d9cb69962abb7457125a7965b735a658M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003   slots:4096-8191 (4096 slots) master   1 additional replica(s)M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006   slots:8192-12287 (4096 slots) master   1 additional replica(s)M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001   slots:12288-16383 (4096 slots) master   1 additional replica(s)S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002   slots: (0 slots) slave   replicates af2e1c600ed2511155a6cad61cabeb095affff86S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005   slots: (0 slots) slave   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168dS: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008   slots: (0 slots) slave   replicates d8138e15d9cb69962abb7457125a7965b735a658[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

这样，三个服务器启动了9个实例，配置成4主5从,其中一个是一主两从,其他都是一主一从

集群验证

选项 -c 可连接集群, 因为配置文件中bind改成了ip，所以必须要用 -h 选项，p选项后接端口号

在redis01上的7000端口的实例上set一个key

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000172.16.7.11:7000> set name haha-> Redirected to slot [5798] located at 172.16.7.22:7003OK172.16.7.22:7003> get name"haha"

由上可见 , redis01 set name 后,将数据重定向到了redis02的7003端口的实例上

在redis03的7008实例上get一个key

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.33 -c -p 7008172.16.7.33:7008> get name-> Redirected to slot [5798] located at 172.16.7.22:7003"weihan"172.16.7.22:7003>

get name 还是会重定向到7003这个实例上 , 这说明redis cluster集群已经是可用的了

redis-trib.rb检查集群状态

4个master会分配到4个服务器上, 而且每个服务器上必须有一个master，创建集群的时候，可以指定master，若创建的时候没指定,是默认分配

[root@redis01 redis-cluster]# redis-trib.rb check 172.16.7.11:7000>>> Performing Cluster Check (using node 172.16.7.11:7000)M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000   slots:0-4095 (4096 slots) master   2 additional replica(s)S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007   slots: (0 slots) slave   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004   slots: (0 slots) slave   replicates d8138e15d9cb69962abb7457125a7965b735a658M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003   slots:4096-8191 (4096 slots) master   1 additional replica(s)M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006   slots:8192-12287 (4096 slots) master   1 additional replica(s)M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001   slots:12288-16383 (4096 slots) master   1 additional replica(s)S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002   slots: (0 slots) slave   replicates af2e1c600ed2511155a6cad61cabeb095affff86S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005   slots: (0 slots) slave   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168dS: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008   slots: (0 slots) slave   replicates d8138e15d9cb69962abb7457125a7965b735a658[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

列出集群节点

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000172.16.7.11:7000> CLUSTER NODESb1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007@17007 slave 691428ae7c1728a76520b8f9d46fa6d83930f191 0 1539016061737 8 connectede752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004@17004 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016062000 5 connected691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003@17003 master - 0 1539016064655 4 connected 4096-8191af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006@17006 master - 0 1539016062000 7 connected 8192-1228772f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001@17001 master - 0 1539016063648 2 connected 12288-16383711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002@17002 slave af2e1c600ed2511155a6cad61cabeb095affff86 0 1539016063000 7 connectedd8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000@17000 myself,master - 0 1539016063000 1 connected 0-4095ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005@17005 slave 72f44da597f73c8d23c6d3fb60fa95489eb0168d 0 1539016063000 6 connected6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008@17008 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016063000 9 connected

打印集群信息

172.16.7.11:7000> CLUSTER INFOcluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:9cluster_size:4cluster_current_epoch:9cluster_my_epoch:1cluster_stats_messages_ping_sent:1240cluster_stats_messages_pong_sent:1251cluster_stats_messages_sent:2491cluster_stats_messages_ping_received:1243cluster_stats_messages_pong_received:1240cluster_stats_messages_meet_received:8cluster_stats_messages_received:2491

备注

若指定redis cluster中的master和slave ，可以用下面的方法

# 添加主库[root@redis01 redis-cluster]# redis-trib.rb create 172.16.7.11:7000 172.16.7.22:7003 172.16.7.33:7006# 添加从库[root@redis01 redis-cluster]# redis-trib.rb add-node --slave 172.16.7.22:7001 172.16.7.11:7000    # 把172.16.7.11的7001 作为从库添加到172.16.7.11的7000

作者：阿啄debugIT