Redis第2️⃣1️⃣课集群伸缩

作者: 小超_8b2f | 来源:发表于2019-05-29 10:51 被阅读9次

redis cluster中只有db0，没有db1 ~ db16

一、集群伸缩

1）伸缩原理

伸缩原理

将6385作为一个master节点加入进来。其实就是启动6385节点，然后meet，此时还没有被分配槽slot，尚无法工作，需要进行slot迁移（现有可用节点的slot迁移到新节点6385），这会伴随着一些key-value的迁移,即槽和数据在节点间移动。

2）集群扩容

集群添加节点的作用
为它迁移槽和数据，实现 $\color{blue}{扩容}$
作为从节点，负责 $\color{blue}{故障转移}$

（1）准备新节点

配置文件是集群模式
配置和其它节点统一
启动后是孤儿节点

redis-server conf/redis-6385.conf
redis-server conf/redis-6386.conf

加入集群前是孤立节点

（2）加入集群

版本一：原生meet命令版本 $\color{red}{(无检查,不推荐使用)}$

不推荐理由：若新节点已经加入了其它集群，再meet本集群会造成不可控故障。

#让节点间可以互相感知
redis-cli -p 7000 cluster meet 127.0.0.1 7006
redis-cli -p 7000 cluster meet 127.0.0.1 7007
# 查看节点
redis-cli -p 7000 cluster nodes 
#设置2个新加入的节点为主从关系
redis-cli -h 127.0.0.1 -p 7007 cluster replicate ${node-id-7006}

加入集群后

版本二：官方工具版本 $\color{LimeGreen}{(推荐使用)}$

推荐理由： 检测并阻止其它集群的节点加入到本集群
注意：命令是：redis-cli --cluster， $\color{red}{没具体指定是哪个库，且有2横杠}$

0. 命令参数说明

redis-cli --cluster help
Cluster Manager Commands:
  create         host1:port1 ... hostN:portN
                 --cluster-replicas <arg>
  check          host:port
                 --cluster-search-multiple-owners
  info           host:port
  fix            host:port
                 --cluster-search-multiple-owners
  reshard        host:port
                 --cluster-from <arg>
                 --cluster-to <arg>
                 --cluster-slots <arg>
                 --cluster-yes
                 --cluster-timeout <arg>
                 --cluster-pipeline <arg>
                 --cluster-replace
  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace
  add-node       new_host:new_port existing_host:existing_port
                 --cluster-slave
                 --cluster-master-id <arg>
  del-node       host:port node_id
  call           host:port command arg arg .. arg
  set-timeout    host:port milliseconds
  import         host:port
                 --cluster-from <arg>
                 --cluster-copy
                 --cluster-replace
  help           

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

1. 加入主节点

$ sed 's/7000/7006/g' redis-7000.conf > redis-7006.conf
$ redis-server redis-7006.conf
$ redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

>>> Adding node 127.0.0.1:7006 to cluster 127.0.0.1:7000 # 提示正在添加7006节点
>>> Performing Cluster Check (using node 127.0.0.1:7000) #做现有集群检查
M: 1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 2d19dda2a8a790d5636a664fe3ed54aa3dd7677c 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 1ac9fbbfe11362e151204132e3d110b18139a1d9
S: 5a4f085dee8400093f45ce2cfa42cbd206167f73 127.0.0.1:7004
   slots: (0 slots) slave
   replicates a3c0d3b42da023dc402faf439d4f93a1cb44d402
M: a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 09792d31e728ad714a5a90bc7639f277d817fb4e 127.0.0.1:7005
   slots: (0 slots) slave
   replicates a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77
[OK] All nodes agree about slots configuration.
>>> Check for open slots...    #检查尚未使用的槽
>>> Check slots coverage...    #检查被使用了的槽
[OK] All 16384 slots covered.  #所有槽都已被占用
# meet操作：redis-cli -p 7000 cluster meet 127.0.0.1 7006
>>> Send CLUSTER MEET to node 127.0.0.1:7006 to make it join the cluster.
[OK] New node added correctly.

$ redis-cli -p 7006 cluster nodes
a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002@17002 master - 0 1558920859435 3 connected 10923-16383
1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000@17000 master - 0 1558920856402 1 connected 0-5460
5a4f085dee8400093f45ce2cfa42cbd206167f73 127.0.0.1:7004@17004 slave a3c0d3b42da023dc402faf439d4f93a1cb44d402 0 1558920857000 2 connected
a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001@17001 master - 0 1558920858422 2 connected 5461-10922
09792d31e728ad714a5a90bc7639f277d817fb4e 127.0.0.1:7005@17005 slave a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 0 1558920857000 3 connected
2677387c78c27c55184125161466957b0b52dd61 127.0.0.1:7006@17006 myself,master - 0 1558920854000 0 connected
2d19dda2a8a790d5636a664fe3ed54aa3dd7677c 127.0.0.1:7003@17003 slave 1ac9fbbfe11362e151204132e3d110b18139a1d9 0 1558920858000 1 connected

2. 加入从节点

$ sed 's/7000/7007/g' redis-7000.conf > redis-7007.conf
$ redis-server redis-7007.conf
$ redis-cli --cluster add-node 127.0.0.1:7007 127.0.0.1:7000 --cluster-slave --cluster-master-id 2677387c78c27c55184125161466957b0b52dd61

>>> Adding node 127.0.0.1:7007 to cluster 127.0.0.1:7000 # 提示正在添加7007节点
>>> Performing Cluster Check (using node 127.0.0.1:7000)#做现有集群检查
M: 1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 2677387c78c27c55184125161466957b0b52dd61 127.0.0.1:7006
   slots: (0 slots) master
S: 2d19dda2a8a790d5636a664fe3ed54aa3dd7677c 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 1ac9fbbfe11362e151204132e3d110b18139a1d9
S: 5a4f085dee8400093f45ce2cfa42cbd206167f73 127.0.0.1:7004
   slots: (0 slots) slave
   replicates a3c0d3b42da023dc402faf439d4f93a1cb44d402
M: a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 09792d31e728ad714a5a90bc7639f277d817fb4e 127.0.0.1:7005
   slots: (0 slots) slave
   replicates a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77
[OK] All nodes agree about slots configuration.
>>> Check for open slots...    #检查尚未使用的槽
>>> Check slots coverage...    #检查被使用了的槽
[OK] All 16384 slots covered.  #所有槽都已被占用
# meet操作：redis-cli -p 7000 cluster meet 127.0.0.1 7007
>>> Send CLUSTER MEET to node 127.0.0.1:7007 to make it join the cluster.
Waiting for the cluster to join

#redis-cli -p 7007 cluster replicate ${node-id-7006}
>>> Configure node as replica of 127.0.0.1:7006
[OK] New node added correctly.

$ redis-cli -p 7007 cluster nodes
2677387c78c27c55184125161466957b0b52dd61 127.0.0.1:7006@17006 master - 0 1558921744000 0 connected
a1c1f02eb3e1236f76bb31dbdbd82d73d9fa012c 127.0.0.1:7007@17007 myself,slave 2677387c78c27c55184125161466957b0b52dd61 0 1558921747000 0 connected
09792d31e728ad714a5a90bc7639f277d817fb4e 127.0.0.1:7005@17005 slave a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 0 1558921750223 3 connected
2d19dda2a8a790d5636a664fe3ed54aa3dd7677c 127.0.0.1:7003@17003 slave 1ac9fbbfe11362e151204132e3d110b18139a1d9 0 1558921748201 1 connected
1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000@17000 master - 0 1558921747000 1 connected 0-5460
a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001@17001 master - 0 1558921749000 2 connected 5461-10922
a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002@17002 master - 0 1558921749212 3 connected 10923-16383
5a4f085dee8400093f45ce2cfa42cbd206167f73 127.0.0.1:7004@17004 slave a3c0d3b42da023dc402faf439d4f93a1cb44d402 0 1558921747191 2 connected

（3）迁移槽和数据

1. 槽迁移计划图解

槽均分3份儿 -> 重新均分为4份

image.png

2. 迁移数据流程描述：

对目标节点发送：cluster setslot {slot} importing {sourceNodeId}命令，让目标节点准备导入槽数据
对源节点发送：cluster setslot {slot} migrating {targetNodeId}命令，让源节点准备迁出槽的数据
源节点循环执行Cluster getkeysinslot {slot} {count}命令，每次获取count个属于槽的键
在源节点上执行migrate {targetIp} {targetPort} key 0 {timeout} 命令把指定key迁移
重复执行3 ~ 4 步骤直到槽下所有键数据迁移到目标节点。
向集群内所有主节点发送cluster setslot {slot} node {targetNodeId}，通知槽分配给目标节点

3. 迁移数据流程图

迁移数据完整流程图

4. 迁移数据流程伪代码

迁移数据 - 伪代码

5. 批量迁移数据

批量执行数据迁移任务

6. 用reshard命令迁移槽

$ redis-cli --cluster reshard 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 09792d31e728ad714a5a90bc7639f277d817fb4e 127.0.0.1:7005
   slots: (0 slots) slave
   replicates a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77
S: 2d19dda2a8a790d5636a664fe3ed54aa3dd7677c 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 1ac9fbbfe11362e151204132e3d110b18139a1d9
M: a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: a1c1f02eb3e1236f76bb31dbdbd82d73d9fa012c 127.0.0.1:7007
   slots: (0 slots) slave
   replicates 2677387c78c27c55184125161466957b0b52dd61
M: 2677387c78c27c55184125161466957b0b52dd61 127.0.0.1:7006
   slots: (0 slots) master
   1 additional replica(s)
S: 5a4f085dee8400093f45ce2cfa42cbd206167f73 127.0.0.1:7004
   slots: (0 slots) slave
   replicates a3c0d3b42da023dc402faf439d4f93a1cb44d402
M: a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#16383 ÷ 3 = 5460   #16383 ÷ 4 = 4096
How many slots do you want to move (from 1 to 16384)? 4096  # 迁移4096个需要自己指定
What is the receiving node ID? 2677387c78c27c55184125161466957b0b52dd61
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all #选择的是all,即所有有槽的节点都向新节点转移槽

Ready to move 4096 slots.
  Source nodes:
    M: 1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
    M: a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001
       slots:[5461-10922] (5462 slots) master
       1 additional replica(s)
    M: a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002
       slots:[10923-16383] (5461 slots) master
       1 additional replica(s)
  Destination node:
    M: 2677387c78c27c55184125161466957b0b52dd61 127.0.0.1:7006
       slots: (0 slots) master
       1 additional replica(s)
  Resharding plan: #下面的区间范围是自己改的，实际上是一行行输出的log，只是计划log,尚未转移
    Moving slot [5461 ~  6826] from a3c0d3b42da023dc402faf439d4f93a1cb44d402
    Moving slot [0 ~ 1364] from 1ac9fbbfe11362e151204132e3d110b18139a1d9
    Moving slot [10923 ~ 12287] from a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77
Do you want to proceed with the proposed reshard plan (yes/no)? yes # 给你反悔的机会
    Moving slot [5461 ~  6826] from 127.0.0.1:7001 to 127.0.0.1:7006: 
    Moving slot [0 ~ 1364]  from 127.0.0.1:7000 to 127.0.0.1:7006:
    Moving slot [10923 ~ 12287] from 127.0.0.1:7002 to 127.0.0.1:7006:
#槽里没有数据迁移过程快，有数据迁移过程慢,每个节点都平均迁移点儿。

迁移的新节点的槽是分成了3段： 0-1364 5461-6826 10923-12287:

$ redis-cli -p 7000 cluster nodes | grep master
a3c0d3b42da023dc402faf439d4f93a1cb44d402 127.0.0.1:7001@17001 master - 0 1559094097000 2 connected 6827-10922
2677387c78c27c55184125161466957b0b52dd61 127.0.0.1:7006@17006 master - 0 1559094100841 7 connected 0-1364 5461-6826 10923-12287
a89a427b5fe8b2b0ef07ac8c6252dc3c8efa1f77 127.0.0.1:7002@17002 master - 0 1559094097000 3 connected 12288-16383
1ac9fbbfe11362e151204132e3d110b18139a1d9 127.0.0.1:7000@17000 myself,master - 0 1559094099000 1 connected 1365-5460

3）集群缩容

0. 流程图

集群缩容流程图

1. 下线迁移槽

将下线节点的槽均匀地迁到其它节点

$ redis-cli --cluster reshard --cluster-from {node_id_7006} --cluster-to {node_id_7000} --cluster-slots 1365 127.0.0.1:7006
$ redis-cli --cluster reshard --cluster-from {node_id_7006} --cluster-to {node_id_7001} --cluster-yes --cluster-slots 1366 127.0.0.1:7006
$ redis-cli --cluster reshard --cluster-from {node_id_7006} --cluster-to {node_id_7002} --cluster-yes --cluster-slots 1365 127.0.0.1:7006

2. 忘记节点

#1. 先下线从节点，再下线主节点
$ redis-cli --cluster del-node 127.0.0.1:7007 {node_id_7007}
$ redis-cli --cluster del-node 127.0.0.1:7006 {node_id_7006}
$ redis-cli -p 7000 cluster nodes #已经差不到7006，7007节点

Redis第2️⃣1️⃣课集群伸缩

一、集群伸缩

1）伸缩原理

2）集群扩容

（1）准备新节点

（2）加入集群

版本一：原生meet命令版本 $\color{red}{(无检查,不推荐使用)}$

版本二：官方工具版本 $\color{LimeGreen}{(推荐使用)}$

0. 命令参数说明

1. 加入主节点

2. 加入从节点

（3）迁移槽和数据

1. 槽迁移计划图解

2. 迁移数据流程描述：

3. 迁移数据流程图

4. 迁移数据流程伪代码

5. 批量迁移数据

6. 用reshard命令迁移槽

3）集群缩容

0. 流程图

1. 下线迁移槽

2. 忘记节点

3. 关闭节点

五、客户端路由

六、集群原理

七、开发运维常见问题

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

Redis

Redis第2️⃣1️⃣课 集群伸缩

一、 集群伸缩

1）伸缩原理

2）集群扩容

（1）准备新节点

（2）加入集群

版本一：原生meet命令版本

版本二：官方工具版本

0. 命令参数说明

1. 加入主节点

2. 加入从节点

（3）迁移槽和数据

1. 槽迁移计划图解

2. 迁移数据流程描述：

3. 迁移数据流程图

4. 迁移数据流程伪代码

5. 批量迁移数据

6. 用reshard命令迁移槽

3）集群缩容

0. 流程图

1. 下线迁移槽

2. 忘记节点

3. 关闭节点

五、 客户端路由

六、 集群原理

七、开发运维常见问题

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

Redis

Redis第2️⃣1️⃣课集群伸缩

一、集群伸缩

版本一：原生meet命令版本 $\color{red}{(无检查,不推荐使用)}$

版本二：官方工具版本 $\color{LimeGreen}{(推荐使用)}$

五、客户端路由

六、集群原理