Codis3.2集群HA高可用方案

Codis高可用方案官方推荐使用Sentinel

Redis 本身就是最终一致性的。Master 挂了，Promote Slave 成为新的 Master 需要时间（测试15秒内）。其实 Sentinel 就是这个逻辑。Codis3.2 自己没有实现 HA，而是直接依赖 Sentinel 的。

注意事项：

Sentinel需要使用原生的Redis-server，版本要等于或高于Codis3.2里面的3.2.8版本， 这里是在Redis3.2.9的下配置测试的，另外Protected-mode no这个保护模式需要关闭，Sentinel.conf默认配置不需要写主从关系，Codif-fe界面SYNC会读取主从关系。

1，每个 Group 前面的 SYNC 是刷 group 所涉及到的 Slot 的路由表给 Proxy 的，包含主从关系。

2，而 Sentinel 前面的 SYNC 是给所有 Sentinel 刷配置的，同时把 Sentinel 列表刷给每个 Proxy。所以只要在集群搭建完成之后，或者每次添加、删除 Redis server 之后（稳定之后） ，在 Sentinel 下面点一次 SYNC 即可。

3，Codis3.2 的只解决了并发迁移＋异步迁移问题。不会去做 HA 有关的工作。所以在 3.2 下解决问题的话， Sentinel 就足够了，而且只能用 Sentinel。Sentinel 负责的是 Group 内的主从切换。

4，内核优化

echo never > /sys/kernel/mm/transparent_hugepage/enabled

sysctl vm.overcommit_memory=1

一，Codis集群架构

192.168.188.120 codis120 codis-server  zookeeper sentinel

192.168.188.121 codis121 codis-server  zookeeper sentinel

192.168.188.122 codis122 codis-server  zookeeper sentinel

192.168.188.123 codis123 codis-server  codis-proxy  sentinel lvs

192.168.188.124 codis124 codis-server  codis-proxy  sentinel lvs

192.168.188.125 codis125  codis-server  codis-dashboard sentinel codis-fe

二，配置Codis-server

每台虚拟机部署3个实例：6379、6380、6381，共18个实例

192.168.188.120:6379、6380、6381

192.168.188.121:6379、6380、6381

192.168.188.122:6379、6380、6381

192.168.188.123:6379、6380、6381

192.168.188.124:6379、6380、6381

192.168.188.125:6379、6380、6381

三，配置Group

设置好1主2从关系，共6组，注意主从从3个实例需要来自3台虚拟机

Group1（主从从）

192.168.188.120:6379  192.168.188.121:6380  192.168.188.123:6381

Group2（主从从）

192.168.188.121:6379  192.168.188.120:6380  192.168.188.124:6381

Group3（主从从）

192.168.188.122:6379  192.168.188.123:6380  192.168.188.125:6381

Group4（主从从）

192.168.188.123:6379  192.168.188.122:6380  192.168.188.120:6381

Group5（主从从）

192.168.188.124:6379  192.168.188.125:6380  192.168.188.121:6381

Group6（主从从）

192.168.188.125:6379  192.168.188.124:6380  192.168.188.122:6381

四，配置Sentinel

需要配置6个Sentinel，没台虚拟机上1个

注意这个原生是在sentinel 的配置文件里配置，3.2.9的原生的sentinel配置下，另外protected-mode no这个保护模式需要关闭，sentinel.conf默认配置不需要写主从关系，Codif-fe界面SYNC会读取主从关系。

1，配置信息

[root@codis120 config]# cat sentinel.conf |grep -Ev "^#|^$"

protected-mode no

port 26379

dir "/tmp"

2，启动entinel，多个Sentinel便于选举和保证高可用

192.168.188.120:26379

192.168.188.120:26379

192.168.188.120:26379

192.168.188.120:26379

192.168.188.120:26379

192.168.188.120:26379

3，SYNC后通过日志查看哨兵记录

20980:X 19 Sep 10:53:28.965 # +monitor master codis-easypass-2 192.168.188.121 6379 quorum 2

20980:X 19 Sep 10:53:28.965 # +set master codis-easypass-2 192.168.188.121 6379 parallel-syncs 1

20980:X 19 Sep 10:53:28.965 # +set master codis-easypass-2 192.168.188.121 6379 down-after-milliseconds 30000

20980:X 19 Sep 10:53:28.965 # +set master codis-easypass-2 192.168.188.121 6379 failover-timeout 300000

20980:X 19 Sep 10:53:28.967 * +slave slave 192.168.188.120:6380 192.168.188.120 6380 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:28.969 * +slave slave 192.168.188.124:6381 192.168.188.124 6381 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:28.973 # +monitor master codis-easypass-3 192.168.188.122 6379 quorum 2

20980:X 19 Sep 10:53:28.973 # +set master codis-easypass-3 192.168.188.122 6379 parallel-syncs 1

20980:X 19 Sep 10:53:28.973 # +set master codis-easypass-3 192.168.188.122 6379 down-after-milliseconds 30000

20980:X 19 Sep 10:53:28.973 # +set master codis-easypass-3 192.168.188.122 6379 failover-timeout 300000

20980:X 19 Sep 10:53:28.997 # +monitor master codis-easypass-4 192.168.188.123 6379 quorum 2

20980:X 19 Sep 10:53:28.997 # +set master codis-easypass-4 192.168.188.123 6379 parallel-syncs 1

20980:X 19 Sep 10:53:28.997 # +set master codis-easypass-4 192.168.188.123 6379 down-after-milliseconds 30000

20980:X 19 Sep 10:53:28.997 # +set master codis-easypass-4 192.168.188.123 6379 failover-timeout 300000

20980:X 19 Sep 10:53:29.001 # +monitor master codis-easypass-5 192.168.188.124 6379 quorum 2

20980:X 19 Sep 10:53:29.002 # +set master codis-easypass-5 192.168.188.124 6379 parallel-syncs 1

20980:X 19 Sep 10:53:29.002 # +set master codis-easypass-5 192.168.188.124 6379 down-after-milliseconds 30000

20980:X 19 Sep 10:53:29.002 # +set master codis-easypass-5 192.168.188.124 6379 failover-timeout 300000

20980:X 19 Sep 10:53:29.006 # +monitor master codis-easypass-6 192.168.188.125 6379 quorum 2

20980:X 19 Sep 10:53:29.006 # +set master codis-easypass-6 192.168.188.125 6379 parallel-syncs 1

20980:X 19 Sep 10:53:29.006 # +set master codis-easypass-6 192.168.188.125 6379 down-after-milliseconds 30000

20980:X 19 Sep 10:53:29.006 # +set master codis-easypass-6 192.168.188.125 6379 failover-timeout 300000

20980:X 19 Sep 10:53:29.010 # +monitor master codis-easypass-1 192.168.188.120 6379 quorum 2

20980:X 19 Sep 10:53:29.010 # +set master codis-easypass-1 192.168.188.120 6379 parallel-syncs 1

20980:X 19 Sep 10:53:29.010 # +set master codis-easypass-1 192.168.188.120 6379 down-after-milliseconds 30000

20980:X 19 Sep 10:53:29.010 # +set master codis-easypass-1 192.168.188.120 6379 failover-timeout 300000

20980:X 19 Sep 10:53:29.029 * +slave slave 192.168.188.121:6380 192.168.188.121 6380 @ codis-easypass-1 192.168.188.120 6379

20980:X 19 Sep 10:53:29.031 * +slave slave 192.168.188.123:6381 192.168.188.123 6381 @ codis-easypass-1 192.168.188.120 6379

20980:X 19 Sep 10:53:29.033 * +slave slave 192.168.188.124:6380 192.168.188.124 6380 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:29.036 * +slave slave 192.168.188.122:6381 192.168.188.122 6381 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:29.038 * +slave slave 192.168.188.123:6380 192.168.188.123 6380 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:29.040 * +slave slave 192.168.188.125:6381 192.168.188.125 6381 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:29.042 * +slave slave 192.168.188.122:6380 192.168.188.122 6380 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:29.044 * +slave slave 192.168.188.120:6381 192.168.188.120 6381 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:29.047 * +slave slave 192.168.188.125:6380 192.168.188.125 6380 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:29.049 * +slave slave 192.168.188.121:6381 192.168.188.121 6381 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:30.958 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:30.960 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:30.962 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:30.965 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:30.976 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:30.978 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:30.981 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:30.989 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:30.991 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:30.993 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:31.001 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:31.003 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-1 192.168.188.120 6379

20980:X 19 Sep 10:53:31.005 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:31.008 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:31.010 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:31.012 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:31.015 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-1 192.168.188.120 6379

20980:X 19 Sep 10:53:31.052 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:31.055 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-1 192.168.188.120 6379

20980:X 19 Sep 10:53:31.058 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:31.059 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:31.062 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:31.069 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:31.071 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-1 192.168.188.120 6379

20980:X 19 Sep 10:53:31.276 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-4 192.168.188.123 6379

20980:X 19 Sep 10:53:31.292 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-6 192.168.188.125 6379

20980:X 19 Sep 10:53:31.303 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-3 192.168.188.122 6379

20980:X 19 Sep 10:53:31.314 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-2 192.168.188.121 6379

20980:X 19 Sep 10:53:31.322 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-5 192.168.188.124 6379

20980:X 19 Sep 10:53:31.325 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-1 192.168.188.120 6379

4，也可以通过图形界面查看

http://192.168.188.125:18080/api/topom/sentinels/info/192.168.188.120:26379/monitored

5，最后配置文件核对

新增内容已经保存到sentinel.conf里面了

[root@codis120 config]# cat sentinel.conf |grep -Ev "^#|^$"

protected-mode no

port 26379

dir "/tmp"

sentinel myid 620b655b311656aa6016e6b864b0f2a8fb03d453

sentinel monitor codis-easypass-6 192.168.188.125 6379 2

sentinel failover-timeout codis-easypass-6 300000

sentinel config-epoch codis-easypass-6 0

sentinel leader-epoch codis-easypass-6 0

sentinel known-slave codis-easypass-6 192.168.188.124 6380

sentinel known-slave codis-easypass-6 192.168.188.122 6381

sentinel known-sentinel codis-easypass-6 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404c

sentinel known-sentinel codis-easypass-6 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645

sentinel known-sentinel codis-easypass-6 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54

sentinel known-sentinel codis-easypass-6 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4

sentinel known-sentinel codis-easypass-6 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9

sentinel monitor codis-easypass-2 192.168.188.121 6379 2

sentinel failover-timeout codis-easypass-2 300000

sentinel config-epoch codis-easypass-2 0

sentinel leader-epoch codis-easypass-2 0

sentinel known-slave codis-easypass-2 192.168.188.120 6380

sentinel known-slave codis-easypass-2 192.168.188.124 6381

sentinel known-sentinel codis-easypass-2 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404c

sentinel known-sentinel codis-easypass-2 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645

sentinel known-sentinel codis-easypass-2 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54

sentinel known-sentinel codis-easypass-2 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4

sentinel known-sentinel codis-easypass-2 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9

sentinel monitor codis-easypass-1 192.168.188.120 6379 2

sentinel failover-timeout codis-easypass-1 300000

sentinel config-epoch codis-easypass-1 0

sentinel leader-epoch codis-easypass-1 0

sentinel known-slave codis-easypass-1 192.168.188.123 6381

sentinel known-slave codis-easypass-1 192.168.188.121 6380

sentinel known-sentinel codis-easypass-1 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404c

sentinel known-sentinel codis-easypass-1 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645

sentinel known-sentinel codis-easypass-1 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54

sentinel known-sentinel codis-easypass-1 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4

sentinel known-sentinel codis-easypass-1 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9

sentinel monitor codis-easypass-3 192.168.188.122 6379 2

sentinel failover-timeout codis-easypass-3 300000

sentinel config-epoch codis-easypass-3 0

sentinel leader-epoch codis-easypass-3 0

sentinel known-slave codis-easypass-3 192.168.188.123 6380

sentinel known-slave codis-easypass-3 192.168.188.125 6381

sentinel known-sentinel codis-easypass-3 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404c

sentinel known-sentinel codis-easypass-3 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645

sentinel known-sentinel codis-easypass-3 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54

sentinel known-sentinel codis-easypass-3 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4

sentinel known-sentinel codis-easypass-3 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9

sentinel monitor codis-easypass-4 192.168.188.123 6379 2

sentinel failover-timeout codis-easypass-4 300000

sentinel config-epoch codis-easypass-4 0

sentinel leader-epoch codis-easypass-4 0

sentinel known-slave codis-easypass-4 192.168.188.120 6381

sentinel known-slave codis-easypass-4 192.168.188.122 6380

sentinel known-sentinel codis-easypass-4 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404c

sentinel known-sentinel codis-easypass-4 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645

sentinel known-sentinel codis-easypass-4 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54

sentinel known-sentinel codis-easypass-4 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4

sentinel known-sentinel codis-easypass-4 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9

sentinel monitor codis-easypass-5 192.168.188.124 6379 2

sentinel failover-timeout codis-easypass-5 300000

sentinel config-epoch codis-easypass-5 0

sentinel leader-epoch codis-easypass-5 0

sentinel known-slave codis-easypass-5 192.168.188.125 6380

sentinel known-slave codis-easypass-5 192.168.188.121 6381

sentinel known-sentinel codis-easypass-5 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404c

sentinel known-sentinel codis-easypass-5 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645

sentinel known-sentinel codis-easypass-5 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54

sentinel known-sentinel codis-easypass-5 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4

sentinel known-sentinel codis-easypass-5 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9

sentinel current-epoch 0

五，Group内主从切换测试

Group3测试

Group3（主从从）

192.168.188.122:6379  192.168.188.123:6380  192.168.188.125:6381

1，首先，停止192.168.188.122:6379

[root@codis122 admin]# /usr/local/codis/admin/codis-server-admin-6379.sh stop

/usr/local/codis/admin/../config/redis-6379.conf

stopping codis-server ...

STOPPED

2，检测发现192.168.188.125:6381被选举为主了，切换过程大概15秒内

8980:X 19 Sep 11:51:17.416 # +sdown master codis-easypass-3 192.168.188.122 6379

8980:X 19 Sep 11:51:17.461 # +new-epoch 1

8980:X 19 Sep 11:51:17.462 # +vote-for-leader 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 1

8980:X 19 Sep 11:51:17.488 # +odown master codis-easypass-3 192.168.188.122 6379 #quorum 6/2

8980:X 19 Sep 11:51:17.488 # Next failover delay: I will not start a failover before Tue Sep 19 12:01:17 2017

8980:X 19 Sep 11:51:18.283 # +config-update-from sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-3 192.168.188.122 6379

8980:X 19 Sep 11:51:18.283 # +switch-master codis-easypass-3 192.168.188.122 6379 192.168.188.125 6381

8980:X 19 Sep 11:51:18.284 * +slave slave 192.168.188.123:6380 192.168.188.123 6380 @ codis-easypass-3 192.168.188.125 6381

8980:X 19 Sep 11:51:18.284 * +slave slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 6381

8980:X 19 Sep 11:51:48.285 # +sdown slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 6381

8980:X 19 Sep 11:52:07.256 # -sdown slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 6381

3，最后启动192.168.188.122:6379，变成了Group3的一个从

[root@codis122 admin]# /usr/local/codis/admin/codis-server-admin-6379.sh start

/usr/local/codis/admin/../config/redis-6379.conf

starting codis-server ...

[root@codis122 admin]# 18701:X 19 Sep 11:52:07.331 # -sdown slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 6381