redis 集群只有db0 redis集群 node is not empty

转载

mob64ca13f7ecc9 2024-06-24 08:42:50

文章标签 redis 集群只有db0 redis 云计算缓存 3d 文章分类 Redis 数据库

环境:
redis版本: redis-3.2.12

实现: 3主 3从
   node01: 1主1从 (主从千万不要部署在一起单机单服务 (nginx php mysql python tomcat)
   node02: 1主1从
   node03: 1主1从

一. 基础环境的安装

1.安装redis 所有节点(node01 node02 node03)
 mkdir -pv /etc/redis-cluster
 yum install redis -y     # gw extra2.安装集群相关软件包
 yum install ruby ruby-devel rubygems rpm-build -y# 3.修改系统参数  (PXE装机时已经优化)
 # echo "session required /lib64/security/pam_limits.so" >> /etc/pam.d/login
 # wget http://10.15.200.8/sa_utils/21-nofile.conf -O /etc/security/limits.d/21-nofile.conf# Increased maximum number of open files to 100032 (it was originally set to 1024)
 # 内核优化: vm.overcommit_memory = 1' to /etc/sysctl.conf
 # 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'# 4.修改配置文件 将node01的配置文件 分别scp到node02 node03
 # /etc/redis-cluster/nodes_3000.conf  /etc/redis-cluster/nodes_3001.conf
 # 然后修改配置文件:
 # bind 10.15.200.101 127.0.0.1   # 只修改 10.15.200.101 将此ip修改为当前node节点对应的ip地址
 # [root@node01 ~]# scp /etc/redis-cluster/{nodes_3000.conf,nodes_3001.conf} node02:/etc/redis-cluster/
 # [root@node01 ~]# scp /etc/redis-cluster/{nodes_3000.conf,nodes_3001.conf} node03:/etc/redis-cluster/
 # [root@node02 ~]# sed -i 's/10.15.200.101/10.15.200.102/g' /etc/redis-cluster/*.conf
 # [root@node03 ~]# sed -i 's/10.15.200.101/10.15.200.103/g' /etc/redis-cluster/*.confnode01:
   wget http://10.15.200.8/files/redis_cluster/node01_nodes_3000.conf -O /etc/redis-cluster/nodes_3000.conf
   wget http://10.15.200.8/files/redis_cluster/node01_nodes_3001.conf -O /etc/redis-cluster/nodes_3001.confnode02:
   wget http://10.15.200.8/files/redis_cluster/node02_nodes_3000.conf -O /etc/redis-cluster/nodes_3000.conf
   wget http://10.15.200.8/files/redis_cluster/node02_nodes_3001.conf -O /etc/redis-cluster/nodes_3001.confnode03:
   wget http://10.15.200.8/files/redis_cluster/node03_nodes_3000.conf -O /etc/redis-cluster/nodes_3000.conf
   wget http://10.15.200.8/files/redis_cluster/node03_nodes_3001.conf -O /etc/redis-cluster/nodes_3001.conf日志文件: tailf /var/log/redis/3000.log
 如果不修改IP地址 报如下的错误:
 17491:M 28 Feb 08:46:11.156 # Creating Server TCP listening socket 10.15.200.101:3000: bind: Cannot assign requested address正常的日志, 有如下的提示:
 17608:M 28 Feb 08:48:19.287 * The server is now ready to accept connections on port 30005.启动服务 (三个节点: node01 node02 node03 都要运行如下两个命令 启动3000 3001的服务:)
 /usr/bin/redis-server /etc/redis-cluster/nodes_3000.conf
 # tailf /var/log/redis/3000.log/usr/bin/redis-server /etc/redis-cluster/nodes_3001.conf
 # tailf /var/log/redis/3001.log6.查看日志 或 查看进程 (以3000端口号为例 其它同理)
 1) [root@node01 ~]# tailf /var/log/redis/3000.log
 2) [root@node01 ~]# ps -ef | grep  30007.服务全部重启后 再次查看端口号  (校验所有的服务器的redis是否全部启动)
 [root@node01 ~]# netstat -tulanp | grep :300
 tcp        0      0 127.0.0.1:3000          0.0.0.0:*               LISTEN      19899/redis-server
 tcp        0      0 10.15.200.101:3000      0.0.0.0:*               LISTEN      19899/redis-server
 tcp        0      0 127.0.0.1:3001          0.0.0.0:*               LISTEN      20282/redis-server
 tcp        0      0 10.15.200.101:3001      0.0.0.0:*               LISTEN      20282/redis-server# 端口号 3000 13000  3001  13001 (含义)
 8.
 [root@node01 ~]# cat /etc/redis-cluster/node_3000.conf
 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 :0 myself,master - 0 0 0 connected
 vars currentEpoch 0 lastVoteEpoch 0
 [root@node01 ~]# cat /etc/redis-cluster/node_3001.conf
 947d6460c205fdb3a12445f5c442e57330b0560c :0 myself,master - 0 0 0 connected
 vars currentEpoch 0 lastVoteEpoch 0

二. 配置集群Cluster

# [root@node01 ~]# gem install redis
 # Fetching: redis-4.2.5.gem (100%)
 # ERROR:  Error installing redis:
         redis requires Ruby version >= 2.3.0.# https://rubygems.org/gems/redis/versions/3.2.2
 # wget https://rubygems.org/downloads/redis-3.2.2.gem /usr/local/srcwget http://10.15.200.8/sa_utils/redis-3.2.2.gem -O /usr/local/src/redis-3.2.2.gem
[root@node01 ~]# gem install -l /usr/local/src/redis-3.2.2.gem
 Successfully installed redis-3.2.2
 Parsing documentation for redis-3.2.2
 Installing ri documentation for redis-3.2.2
 1 gem installednode02 node03
 wget http://10.15.200.8/sa_utils/redis-3.2.2.gem -O /usr/local/src/redis-3.2.2.gem[root@node02 ~]# gem install -l /usr/local/src/redis-3.2.2.gem
 Successfully installed redis-3.2.2
 Parsing documentation for redis-3.2.2
 Installing ri documentation for redis-3.2.2
 1 gem installed[root@node03 ~]# gem install -l /usr/local/src/redis-3.2.2.gem
 Successfully installed redis-3.2.2
 Parsing documentation for redis-3.2.2
 Installing ri documentation for redis-3.2.2
 1 gem installed[root@node01 ~]# wget http://10.15.200.8/packages/redis-3.2.12.tar.gz -O /usr/local/src/redis-3.2.12.tar.gz
 [root@node01 ~]# cd /usr/local/src/ && tar -zxvf redis-3.2.12.tar.gz && cp redis-3.2.12/src/redis-trib.rb /usr/local/sbin
 [root@node01 src]# /usr/local/sbin/redis-trib.rb
 Usage: redis-trib <command> <options> <arguments ...>创建集群: replicas表示每个master需要有几个slave
[root@node01 ~]# /usr/local/sbin/redis-trib.rb create --replicas 1 10.15.200.101:3000 10.15.200.101:3001 10.15.200.102:3000 10.15.200.102:3001 10.15.200.103:3000 10.15.200.103:3001
--replicas 1: 为集群中的每个主节点创建一个从节点
 集群正常工作至少需要3个主节点: 以上是三主三从>>> Creating cluster
 [ERR] Sorry, can't connect to node 10.15.200.101:3000[root@node01 ~]# vim /usr/local/share/gems/gems/redis-3.2.2/lib/redis/client.rb +16
   8     DEFAULTS = {
   9       :url => lambda { ENV["REDIS_URL"] },
  10       :scheme => "redis",
  11       :host => "127.0.0.1",
  12       :port => 6379,
  13       :path => nil,
  14       :timeout => 5.0,
  15       :connect_timeout => 5.0,
  16       :password => 'admin',  # 修改此处的密码 (一定要加 引号)[root@node01 ~]# /usr/local/sbin/redis-trib.rb create --replicas 1 10.15.200.101:3000 10.15.200.101:3001 10.15.200.102:3000 10.15.200.102:3001 10.15.200.103:3000 10.15.200.103:3001
>>> Creating cluster
 >>> Performing hash slots allocation on 6 nodes...
 Using 3 masters:
 10.15.200.101:3000
 10.15.200.102:3000
 10.15.200.103:3000
 Adding replica 10.15.200.102:3001 to 10.15.200.101:3000
 Adding replica 10.15.200.101:3001 to 10.15.200.102:3000
 Adding replica 10.15.200.103:3001 to 10.15.200.103:3000
 M: 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 10.15.200.101:3000
    slots:0-5460 (5461 slots) master
 S: 947d6460c205fdb3a12445f5c442e57330b0560c 10.15.200.101:3001
    replicates 24f84465fdd543dd8f6a9e50375afd793db7941b
 M: 24f84465fdd543dd8f6a9e50375afd793db7941b 10.15.200.102:3000
    slots:5461-10922 (5462 slots) master
 S: 08e742f053d23c8f1d2ed8d5dd01d6dca80c9d6e 10.15.200.102:3001
    replicates 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75
 M: d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 10.15.200.103:3000
    slots:10923-16383 (5461 slots) master
 S: d5f8e89d6ecd483751991d201e1ed689cc04a28a 10.15.200.103:3001
    replicates d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11
 Can I set the above configuration? (type 'yes' to accept): yes
 >>> Nodes configuration updated
 >>> Assign a different config epoch to each node
 >>> Sending CLUSTER MEET messages to join the cluster
 Waiting for the cluster to join.....
 >>> Performing Cluster Check (using node 10.15.200.101:3000)
 M: 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 10.15.200.101:3000
    slots:0-5460 (5461 slots) master
    1 additional replica(s)
 M: 24f84465fdd543dd8f6a9e50375afd793db7941b 10.15.200.102:3000
    slots:5461-10922 (5462 slots) master
    1 additional replica(s)
 S: 947d6460c205fdb3a12445f5c442e57330b0560c 10.15.200.101:3001
    slots: (0 slots) slave
    replicates 24f84465fdd543dd8f6a9e50375afd793db7941b
 S: 08e742f053d23c8f1d2ed8d5dd01d6dca80c9d6e 10.15.200.102:3001
    slots: (0 slots) slave
    replicates 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75
 S: d5f8e89d6ecd483751991d201e1ed689cc04a28a 10.15.200.103:3001
    slots: (0 slots) slave
    replicates d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11
 M: d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 10.15.200.103:3000
    slots:10923-16383 (5461 slots) master
    1 additional replica(s)
 [OK] All nodes agree about slots configuration.
 >>> Check for open slots...
 >>> Check slots coverage...
 [OK] All 16384 slots covered.[root@node01 ~]# /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin'
 10.15.200.103:3000> cluster nodes
 08e742f053d23c8f1d2ed8d5dd01d6dca80c9d6e 10.15.200.102:3001 slave 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 0 1614518465535 4 connected
 947d6460c205fdb3a12445f5c442e57330b0560c 10.15.200.101:3001 slave 24f84465fdd543dd8f6a9e50375afd793db7941b 0 1614518464528 3 connected
 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 10.15.200.101:3000 master - 0 1614518464023 1 connected 0-5460
 d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 10.15.200.103:3000 myself,master - 0 0 5 connected 10923-16383
 d5f8e89d6ecd483751991d201e1ed689cc04a28a 10.15.200.103:3001 slave d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 0 1614518461507 6 connected
 24f84465fdd543dd8f6a9e50375afd793db7941b 10.15.200.102:3000 master - 0 1614518463520 3 connected 5461-1092210.15.200.103:3000> cluster info
 cluster_state:ok
 cluster_slots_assigned:16384
 cluster_slots_ok:16384
 cluster_slots_pfail:0
 cluster_slots_fail:0
 cluster_known_nodes:6
 cluster_size:3
 cluster_current_epoch:6
 cluster_my_epoch:5
 cluster_stats_messages_sent:7209
 cluster_stats_messages_received:7209三. 测试
1）get 和 set数据
 [root@node01 ~]# echo 'hello_world' | /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin' -x set mykey
 OK[root@node01 ~]# /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin'
 10.15.200.101:3000> get mykey
 -> Redirected to slot [14687] located at 10.15.200.103:3000
 "hello_world\n"           # 直接根据hash匹配切换到相应的slot的节点上# 将 10.15.200.103  3000 端口对应的服务停止之后 会重分布
 [root@node01 ~]# /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin'
 10.15.200.101:3000> get mykey
 -> Redirected to slot [14687] located at 10.15.200.103:3001
 "hello_world\n" 写入一定数量的数据:  # node01 直接命令行 粘贴
 for i in `seq 1 10000`
 do
     echo "hello_world_$i" | /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin' -x set "mykey_$i"
 done查看现集群分布情况:
 [root@node01 ~]# /usr/local/sbin/redis-trib.rb info 10.15.200.101:3000
 10.15.200.103:3001 (bbf0844e...) -> 3351 keys | 5461 slots | 1 slaves.
 10.15.200.102:3000 (462e7cf4...) -> 3316 keys | 5462 slots | 1 slaves.
 10.15.200.102:3001 (6ac1063d...) -> 3335 keys | 5461 slots | 1 slaves.
 [OK] 10002 keys in 3 masters.
 0.61 keys per slot on average. 增加一个主节点:
 注意: 最后面一段 地址+端口号: 10.15.200.101:3000 是现集群的任意一个正常的IP:PORT都可以[root@node01 ~]# /usr/local/sbin/redis-trib.rb add-node 10.15.200.104:3000 10.15.200.101:3000
 >>> Adding node 10.15.200.104:3000 to cluster 10.15.200.101:3000
 >>> Performing Cluster Check (using node 10.15.200.101:3000)
 S: 0d0c41cfa78c4372dc64da8ac957d5344ef69424 10.15.200.101:3000
    slots: (0 slots) slave
    replicates 6ac1063d0e014e89606a84c0cece3ddf4ee6b68e
 M: bbf0844ee9b5c25635dc8f42f674e4ce631a64f9 10.15.200.103:3001
    slots:10923-16383 (5461 slots) master
    1 additional replica(s)
 S: 86b188db102f856a129cafda55c7f6da57982881 10.15.200.101:3001
    slots: (0 slots) slave
    replicates 462e7cf49b2e9e1159ffe94439cd0afc231e9f4c
 M: 462e7cf49b2e9e1159ffe94439cd0afc231e9f4c 10.15.200.102:3000
    slots:5461-10922 (5462 slots) master
    1 additional replica(s)
 S: 107b83d08c63eed219242f7a32585d626ed47dba 10.15.200.103:3000
    slots: (0 slots) slave
    replicates bbf0844ee9b5c25635dc8f42f674e4ce631a64f9
 M: 6ac1063d0e014e89606a84c0cece3ddf4ee6b68e 10.15.200.102:3001
    slots:0-5460 (5461 slots) master
    1 additional replica(s)
 [OK] All nodes agree about slots configuration.
 >>> Check for open slots...
 >>> Check slots coverage...
 [OK] All 16384 slots covered.
 >>> Send CLUSTER MEET to node 10.15.200.104:3000 to make it join the cluster.
 [OK] New node added correctly.     # 添加成功[root@node01 ~]# /usr/local/sbin/redis-trib.rb check 10.15.200.104:3000      # 检查集群状态
 >>> Performing Cluster Check (using node 10.15.200.104:3000)
 M: e140c57f1eb4d1694c14f05b2bafc998e00fbc94 10.15.200.104:3000
    slots: (0 slots) master        # 默认刚刚新增的主节点是没有slots的, 这样在存取数据的时候不会被选择
    0 additional replica(s)[OK] All nodes agree about slots configuration.
 >>> Check for open slots...
 >>> Check slots coverage...
 [OK] All 16384 slots covered.# 查看key的分部情况
 [root@node01 ~]# /usr/local/sbin/redis-trib.rb info 10.15.200.101:3000
 10.15.200.104:3000 (e140c57f...) -> 0 keys | 0 slots | 0 slaves.
 10.15.200.103:3001 (bbf0844e...) -> 3351 keys | 5461 slots | 1 slaves.
 10.15.200.102:3000 (462e7cf4...) -> 3316 keys | 5462 slots | 1 slaves.
 10.15.200.102:3001 (6ac1063d...) -> 3335 keys | 5461 slots | 1 slaves.
 [OK] 10002 keys in 4 masters.
 0.61 keys per slot on average. [root@node01 ~]# /usr/local/sbin/redis-trib.rb reshard 10.15.200.104:3000
 >>> Performing Cluster Check (using node 10.15.200.104:3000)
 M: e140c57f1eb4d1694c14f05b2bafc998e00fbc94 10.15.200.104:3000
    slots: (0 slots) master
    0 additional replica(s) 删除节点:
 [root@node01 ~]# /usr/local/sbin/redis-trib.rb check 10.15.200.104:3000
 >>> Performing Cluster Check (using node 10.15.200.104:3000)
 M: 4319c3a09b885fc0c818c7db946293f46ec8a0b9 10.15.200.104:3000   # 注意此处的 ID号 后面删除时会用到[root@node01 ~]# /usr/local/sbin/redis-trib.rb del-node 10.15.200.104:3000 4319c3a09b885fc0c818c7db946293f46ec8a0b9
 >>> Removing node 4319c3a09b885fc0c818c7db946293f46ec8a0b9 from cluster 10.15.200.104:3000
 >>> Sending CLUSTER FORGET messages to the cluster...
 >>> SHUTDOWN the node. 报错:
 [ERR] Node 10.15.200.104:3000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.由于服务器重启原因, 导致该节点中默认生成的配置或历史存储数据不一致导致启动集群报错, 需要清除该节点的文件, 必要时清除数据库, 步骤如下:
 [root@node04 ~]# rm -fr /var/lib/redis/*
 [root@node04 ~]# redis-cli -h 127.0.0.1 -a 'admin' -p 30003.关闭redis服务
 ps -ef
 kill -9
 4.启动服务
 /usr/bin/redis-server /etc/redis-cluster/nodes_3000.conf[ERR] Calling MIGRATE: ERR Target instance replied with error: NOAUTH Authentication required.

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。