快速搭建高可用RedisCluster主从模式集群的填坑之路

Redis Cluster主从模式

原理

redis cluster 为了保证数据的高可用性，加入了主从模式，一个主节点对应一个或多个从节点，主节点提供数据存取，从节点则是从主节点拉取数据。 备份，当这个主节点挂掉后，就会有这个从节点选取一个来充当主节点，从而保证集群不会挂掉。

说明

假如集群有A、B、C三个主节点, 如果这3个节点都没有加入从节点，如果B挂掉了，我们就无法访问整个集群了， A和C的slot也无法访问。 所以在集群建立的时候，一定要为每个主节点都添加了从节点,。比如，像集群包含主节点A、B、C, 以及从节点A1、B1、C1, 那么即使B挂掉，系统也可以继续正确工作 ，B1节点替代了B节点，故Redis集群将会选择B1节点，作为新的主节点，集群将会继续正确地提供服务 当B重新开启后，它就会变成B1的从节点。

注意

如果节点B和B1同时挂了，Redis集群就无法继续正确地提供服务了。

故最小的redis集群需要3个主节点（即Redis Cluster集群至少需要3个master节点，也就是说至少需要6个节点才能构建Redis cluster集群），一台机器可运行多个redis实例（一般使用两台机器，每台启动3个redis实例，即三个主节点，三个从节点）。

很多案例使用单台服务器开6个端口，操作差不多，只是配置基本相对简单点，多台服务器更接近生产环境 。

当集群最开始创建好后，要记住各节点的主从关系（或是创建的时候指定主从关系）；若是其中一台机器重启，重启后，需重新将其加入到redis cluster集群中；这就需要将这台机器上的各节点之前的从节点变为主节点（客户端执行slaveof no one），然后再根据新的主节点，添加这台机器的各节点到集群中，添加后变为从节点。

搭建环境

环境 : CentOS7

redis01 : 172.16.7.11 端口 : 7000、7001、7002

redis02 : 172.16.7.22 端口 : 7003、7004、7005

redis03 : 172.16.7.33 端口 : 7006、7007、7008

关闭SElinux和防火墙

安装redis

首先选择一台服务器，进行安装

初始安装redis01

[root@redis01 ~]# yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git
[root@redis01 ~]#  wget http://download.redis.io/releases/redis-4.0.1.tar.gz
[root@redis01 ~]# tar xf redis-4.0.1.tar.gz -C /usr/local/src/
[root@redis01 ~]# cd /usr/local/src/redis-4.0.1/
[root@redis01 redis-4.0.1]# make && make test && make PREFIX=/usr/local/redis install
[root@redis01 redis-4.0.1]# mkdir /usr/local/redis/conf
[root@redis01 redis-4.0.1]# cp *.conf /usr/local/redis/conf/
[root@redis01 redis-4.0.1]# cp /usr/local/src/redis-4.0.1/src/redis-trib.rb /usr/local/redis/bin/
[root@redis01 redis-4.0.1]# ln -s /usr/local/redis/bin/* /usr/local/bin/

进行操作系统基础调优设置

[root@redis01 redis-4.0.1]# echo "* - nofile 10240" >> /etc/security/limits.conf 
[root@redis01 redis-4.0.1]# echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf 
[root@redis01 redis-4.0.1]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf 
[root@redis01 redis-4.0.1]# sysctl -p
net.core.somaxconn = 10240
vm.overcommit_memory = 1
[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
[root@redis01 redis-4.0.1]# su -l

创建节点

创建集群节点目录

[root@redis01 ~]# mkdir /data/redis/redis-cluster -p
[root@redis01 redis-cluster]# cd /data/redis/redis-cluster/
[root@redis01 redis-cluster]# mkdir 7000 7001 7002 

执行下面的脚本，配置各个节点的redis.conf

[root@redis01 redis-cluster]# for i in 0 1 2;
 do 
echo -e "cluster-enabled yes\n\
bind 0.0.0.0\n\
port 700$i\n\
pidfile /data/redis-cluster/700$i/redis.pid\n\
logfile "/data/redis-cluster/700$i/redis.log"\n\
dir /data/redis-cluster/7000/\n\
tcp-backlog 1024\n\
timeout 0\n\
cluster-node-timeout 10100\n\
tcp-keepalive 0\n\
daemonize yes\n\
loglevel notice\n\
databases 16\n\
stop-writes-on-bgsave-error yes\n\
rdbcompression yes\n\
rdbchecksum yes\n\
dbfilename "dump.rdb"\n\
slave-serve-stale-data yes\n\
slave-read-only yes\n\
repl-diskless-sync no\n\
repl-diskless-sync-delay 5\n\
repl-disable-tcp-nodelay no\n\
slave-priority 100\n\
lazyfree-lazy-eviction no\n\
lazyfree-lazy-expire no\n\
lazyfree-lazy-server-del no\n\
slave-lazy-flush no\n\
appendonly no\n\
appendfilename "appendonly.aof"\n\
appendfsync everysec\n\
no-appendfsync-on-rewrite yes\n\
auto-aof-rewrite-percentage 100\n\
auto-aof-rewrite-min-size 64mb\n\
aof-load-truncated yes\n\
lua-time-limit 5000\n\
slowlog-log-slower-than 10000\n\
slowlog-max-len 128\n\
latency-monitor-threshold 0\n\
notify-keyspace-events ""\n\
hash-max-ziplist-entries 512\n\
hash-max-ziplist-value 64\n\
list-max-ziplist-entries 512\n\
list-max-ziplist-value 64\n\
set-max-intset-entries 512\n\
zset-max-ziplist-entries 128\n\
zset-max-ziplist-value 64\n\
hll-sparse-max-bytes 3000\n\
activerehashing yes\n\
client-output-buffer-limit normal 0 0 0\n\
client-output-buffer-limit slave 256mb 64mb 60\n\
client-output-buffer-limit pubsub 32mb 8mb 60\n\
hz 10\n\
aof-rewrite-incremental-fsync yes" > 700$i/redis.conf;
 done

查看各个节点的文件树

[root@redis01 redis-cluster]# tree 
.
├── 7000
│   └── redis.conf
├── 7001
│   └── redis.conf
└── 7002
    └── redis.conf

启动redis01的集群服务

[root@redis02 redis-cluster]# mv 7000 7003
[root@redis02 redis-cluster]# mv 7001 7004
[root@redis02 redis-cluster]# mv 7002 7005
[root@redis02 redis-cluster]# ls
7003  7004  7005
[root@redis02 redis-cluster]# sed -i 's#7000#7003#g' 7003/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7001#7004#g' 7005/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7002#7005#g' 7004/redis.conf

redis02 服务器的安装

redis02 ，redis03不需要完全按上面的步骤，只部分需要复制redis01的安装解压目录

复制redis01的安装解压目录

[root@node02 ~]# scp -r node01:/usr/local/redis /usr/local/

image

[root@redis02 ~]# scp -r node01:/data/redis/redis-cluster /data/redis/

image

修改某个服务器的各个节点的redis.conf文件

[root@redis02 redis-cluster]# mv 7000 7003
[root@redis02 redis-cluster]# mv 7001 7004
[root@redis02 redis-cluster]# mv 7002 7005
[root@redis02 redis-cluster]# ls
7003  7004  7005
[root@redis02 redis-cluster]# sed -i 's#7000#7003#g' 7003/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7001#7004#g' 7005/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7002#7005#g' 7004/redis.conf

做运行脚本加入bin目录

[root@node02 ~]# ln -s /usr/local/redis/bin/* /usr/local/bin/ 

进行操作系统基础调优设置

如同redis01的操作系统基础调优设置

启动redis02的集群服务

[root@redis02 redis-cluster]# for i in 3 4 5;
 do redis-server /data/redis/redis-cluster/700$i/redis.conf 
 done
41766:C 08 Oct 22:21:42.369 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41766:C 08 Oct 22:21:42.369 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41766, just started
41766:C 08 Oct 22:21:42.369 # Configuration loaded
41768:C 08 Oct 22:21:42.375 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41768:C 08 Oct 22:21:42.375 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41768, just started
41768:C 08 Oct 22:21:42.375 # Configuration loaded
41770:C 08 Oct 22:21:42.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41770:C 08 Oct 22:21:42.380 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41770, just started
41770:C 08 Oct 22:21:42.380 # Configuration loaded

redis03服务器的安装

类比redis02 服务器的安装

redis01，redis02，redis03聚合创建集群

查各Redis节点的启动状态

#yum -y install net-tools
[root@redis01 redis-cluster]# netstat -antup|grep redis
tcp        0      0 172.16.7.11:17000     0.0.0.0:*               LISTEN      68821/redis-server  
tcp        0      0 172.16.7.11:17001     0.0.0.0:*               LISTEN      68823/redis-server  
tcp        0      0 172.16.7.11:17002     0.0.0.0:*               LISTEN      68828/redis-server  
tcp        0      0 172.16.7.11:7000      0.0.0.0:*               LISTEN      68821/redis-server  
tcp        0      0 172.16.7.11:7001      0.0.0.0:*               LISTEN      68823/redis-server  
tcp        0      0 172.16.7.11:7002      0.0.0.0:*               LISTEN      68828/redis-server 

安装ruby

[root@redis01 redis-cluster]# yum -y install rh-ruby23
[root@redis01 redis-cluster]# ruby -v
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
[root@redis01 redis-cluster]# scl enable rh-ruby23 bash
[root@redis01 redis-cluster]# ruby -v
ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]

image

解决方案；

yum install centos-release-scl-rh

image

再次安装 rh-ruby23

image

安装ruby的redis扩展

[root@redis01 redis-cluster]# gem install redis
Fetching: redis-4.0.2.gem (100%)
Successfully installed redis-4.0.2
Parsing documentation for redis-4.0.2
Installing ri documentation for redis-4.0.2
Done installing documentation for redis after 0 seconds
1 gem installed

使用redis自带的集群管理工具redis-trib.rb创建集群

在redis01上运行以下脚本

[root@redis01 redis-cluster]# redis-trib.rb create --replicas 1 172.16.7.11:7000 172.16.7.11:7001 172.16.7.11:7002 172.16.7.22:7003 172.16.7.22:7004 172.16.7.22:7005 172.16.7.33:7006 172.16.7.33:7007 172.16.7.33:7008
>>> Creating cluster
>>> Performing hash slots allocation on 9 nodes...
Using 4 masters:
172.16.7.11:7000
172.16.7.22:7003
172.16.7.33:7006
172.16.7.11:7001
Adding replica 172.16.7.22:7004 to 172.16.7.11:7000
Adding replica 172.16.7.33:7007 to 172.16.7.22:7003
Adding replica 172.16.7.11:7002 to 172.16.7.33:7006
Adding replica 172.16.7.22:7005 to 172.16.7.11:7001
Adding replica 172.16.7.33:7008 to 172.16.7.11:7000
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
   slots:0-4095 (4096 slots) master
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
   slots:12288-16383 (4096 slots) master
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
   replicates af2e1c600ed2511155a6cad61cabeb095affff86
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
   slots:4096-8191 (4096 slots) master
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
   replicates d8138e15d9cb69962abb7457125a7965b735a658
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
   slots:8192-12287 (4096 slots) master
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
   replicates d8138e15d9cb69962abb7457125a7965b735a658
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 172.16.7.11:7000)
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
   slots:0-4095 (4096 slots) master
   2 additional replica(s)
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
   slots: (0 slots) slave
   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
   slots:4096-8191 (4096 slots) master
   1 additional replica(s)
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
   slots:8192-12287 (4096 slots) master
   1 additional replica(s)
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
   slots: (0 slots) slave
   replicates af2e1c600ed2511155a6cad61cabeb095affff86
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
   slots: (0 slots) slave
   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

这样，三个服务器启动了9个实例，配置成4主5从,其中一个是一主两从,其他都是一主一从

集群验证

选项 -c 可连接集群, 因为配置文件中bind改成了ip，所以必须要用 -h 选项，p选项后接端口号

在redis01上的7000端口的实例上set一个key

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000
172.16.7.11:7000> set name haha
-> Redirected to slot [5798] located at 172.16.7.22:7003
OK
172.16.7.22:7003> get name
"haha"

由上可见 , redis01 set name 后,将数据重定向到了redis02的7003端口的实例上

在redis03的7008实例上get一个key

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.33 -c -p 7008
172.16.7.33:7008> get name
-> Redirected to slot [5798] located at 172.16.7.22:7003
"weihan"
172.16.7.22:7003>

get name 还是会重定向到7003这个实例上 , 这说明redis cluster集群已经是可用的了

redis-trib.rb检查集群状态

4个master会分配到4个服务器上, 而且每个服务器上必须有一个master，创建集群的时候，可以指定master，若创建的时候没指定,是默认分配

[root@redis01 redis-cluster]# redis-trib.rb check 172.16.7.11:7000
>>> Performing Cluster Check (using node 172.16.7.11:7000)
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
   slots:0-4095 (4096 slots) master
   2 additional replica(s)
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
   slots: (0 slots) slave
   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
   slots:4096-8191 (4096 slots) master
   1 additional replica(s)
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
   slots:8192-12287 (4096 slots) master
   1 additional replica(s)
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
   slots: (0 slots) slave
   replicates af2e1c600ed2511155a6cad61cabeb095affff86
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
   slots: (0 slots) slave
   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

列出集群节点

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000
172.16.7.11:7000> CLUSTER NODES
b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007@17007 slave 691428ae7c1728a76520b8f9d46fa6d83930f191 0 1539016061737 8 connected
e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004@17004 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016062000 5 connected
691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003@17003 master - 0 1539016064655 4 connected 4096-8191
af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006@17006 master - 0 1539016062000 7 connected 8192-12287
72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001@17001 master - 0 1539016063648 2 connected 12288-16383
711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002@17002 slave af2e1c600ed2511155a6cad61cabeb095affff86 0 1539016063000 7 connected
d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000@17000 myself,master - 0 1539016063000 1 connected 0-4095
ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005@17005 slave 72f44da597f73c8d23c6d3fb60fa95489eb0168d 0 1539016063000 6 connected
6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008@17008 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016063000 9 connected

打印集群信息

172.16.7.11:7000> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:4
cluster_current_epoch:9
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1240
cluster_stats_messages_pong_sent:1251
cluster_stats_messages_sent:2491
cluster_stats_messages_ping_received:1243
cluster_stats_messages_pong_received:1240
cluster_stats_messages_meet_received:8
cluster_stats_messages_received:2491

备注

若指定redis cluster中的master和slave ，可以用下面的方法

# 添加主库
[root@redis01 redis-cluster]# redis-trib.rb create 172.16.7.11:7000 172.16.7.22:7003 172.16.7.33:7006
# 添加从库
[root@redis01 redis-cluster]# redis-trib.rb add-node --slave 172.16.7.22:7001 172.16.7.11:7000    # 把172.16.7.11的7001 作为从库添加到172.16.7.11的7000