zookeeper three mode

前提：

1、必须安装JAVA环境

2、同步三台机器的时间

环境

三台测试机

操作系统: centos7 ; hostname: c1 ; ip: 192.168.11.232

操作系统: centos7 ; hostname: c2 ; ip: 192.168.11.227

操作系统: centos7 ; hostname: c3 ; ip: 192.168.11.234

备注

注意要关闭防火墙

查看磁盘状况

为了避免磁盘使用不均而导致系统问题，因而先查看磁盘的分布情况。

zookeeper安装

效果图：

安装

mkdir -p   /var/download   &&   cd    /var/download 

wget -chttp://www.eu.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

tar zxvf zookeeper-3.4.6.tar.gz 

mkdir -p/var/local/server 

mv zookeeper-3.4.6      /var/local/server/zookeeper 

cd    /var/local/server/zookeeper/conf 

touch zoo.cfg

编辑zoo.cfg中加入如下内容：

tickTime=2000

dataDir=/var/data/zookeeper

dataLogDir=/var/local/server/zookeeper/log

clientPort=2181

initLimit=5

syncLimit=2

autopurge.purgeInterval=5 

autopurge.snapRetainCount=20 

# 上面设置的参数表示：间隔5个小时清理一次，log文件保存20个，snapshot快照文件保存20个。若zookeeper启动时超过设置的文件时，就会执行删除文件的操作。

server.1=c1:2888:3888

server.2=c2:2888:3888

server.3=c3:2888:3888

保存

创建数据目录和日志目录：

mkdir-p /var/data/zookeeper

mkdir-p /var/local/server/zookeeper/log

如不创建启动时会报如下异常：

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh   status

JMX enabledbydefault

Usingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfg

Errorcontacting service. It is  probably  not  running.

创建myid文件， id 与 zoo.cfg 中的序号对应

echo 1 > /var/data/zookeeper/myid

注意：

注意，如果是c2和c3中进行相应的修改

c2上应改为：echo 2 > /var/data/zookeeper/myid

c3上应改为：echo 3 > /var/data/zookeeper/myid

配置hosts文件：编辑/etc/hosts，加入如下内容：

192.168.1.80   c1

192.168.1.81   c2

192.168.1.82   c3

常用命令

启动

/var/local/server/zookeeper/bin/zkServer.sh    start

查看状态

/var/local/server/zookeeper/bin/zkServer.sh   status

重启

/var/local/server/zookeeper/bin/zkServer.sh   restart

关闭

/var/local/server/zookeeper/bin/zkServer.sh   stop

在其中一台用客户端连接

/var/local/server/zookeeper/bin/zkCli.sh   -server    c1:2181

操作实例：

启动：/var/local/server/zookeeper/bin/zkServer.sh   start

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh    start

JMX enabledbydefault

Usingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfg

Starting   zookeeper ... STARTED

查看状态：/var/local/server/zookeeper/bin/zkServer.sh   status

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh   status

JMX enabledbydefaultUsingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfg

Error   contacting service.   It  is  probably   not    running.

此时发现zookeeper并没有正常运行。

查看日志more zookeeper.out，发现如下异常：

2016-08-2509:55:54,088[myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to2at election address c2/192.168.1.81:3888java.net.NoRouteToHostException:No route to host        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:579)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)        at java.lang.Thread.run(Thread.java:745)2016-08-2509:55:54,101[myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to3at election address c3/192.168.1.82:3888java.net.ConnectException:Connection refused        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:579)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)        at java.lang.Thread.run(Thread.java:745)2016-08-2509:55:54,287[myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 2 at election address c2/192.168.1.81:3888java.net.NoRouteToHostException:No route to host        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:579)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)        at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)

这是由于2、3号机器没有正常配置导致。可以在zoo.cfg中将其注释后再次尝试。

tickTime=2000

dataDir=/var/data/zookeeperdataLog

Dir=/var/local/server/zookeeper/log

clientPort=2181

initLimit=5

syncLimit=2

server.1=c1:2888:3888

#server.2=c2:2888:3888

#server.3=c3:2888:3888

正常启动效果：

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh   start

JMX enabledbydefaultUsingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfg

Starting   zookeeper ... STARTED

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh   status

JMX enabledbydefaultUsingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode:standalone

测试

在 c1 上连接 c1 上的 zookeeper

/var/local/server/zookeeper/bin/zkCli.sh -server c1:2181

[root@sso conf]# /var/local/server/zookeeper/bin/zkCli.sh    -server    c1:2181

Connecting to c1:21812016-08-2510:06:30,235[myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on02/20/201409:09 GMT2016-08-2510:06:30,244[myid:] - INFO  [main:Environment@100] - Client environment:host.name=sso.xxx.com2016-08-2510:06:30,244[myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_792016-08-2510:06:30,250[myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation2016-08-2510:06:30,251[myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/local/jdk1.7.0_79/jre2016-08-2510:06:30,251[myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/var/local/server/zookeeper/bin/../build/classes:/var/local/server/zookeeper/bin/../build/lib/*.jar:/var/local/server/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/var/local/server/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/var/local/server/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/var/local/server/zookeeper/bin/../lib/log4j-1.2.16.jar:/var/local/server/zookeeper/bin/../lib/jline-0.9.94.jar:/var/local/server/zookeeper/bin/../zookeeper-3.4.6.jar:/var/local/server/zookeeper/bin/../src/java/lib/*.jar:/var/local/server/zookeeper/bin/../conf:2016-08-2510:06:30,251[myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2016-08-25 10:06:30,252 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp2016-08-2510:06:30,252[myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=2016-08-2510:06:30,252[myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux2016-08-2510:06:30,252[myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd642016-08-2510:06:30,253[myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-327.10.1.el7.x86_642016-08-2510:06:30,253[myid:] - INFO  [main:Environment@100] - Client environment:user.name=root2016-08-2510:06:30,253[myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root

2016-08-25 10:06:30,254 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/var/local/server/zookeeper/conf2016-08-2510:06:30,257[myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=c1:2181sessionTimeout=30000watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@194d62f1Welcome to ZooKeeper!2016-08-2510:06:30,335[myid:] - INFO  [main-SendThread(sso.xxx.com:2181):ClientCnxn$SendThread@975] - Openingsocketconnection to server sso.xxx.com/192.168.1.80:2181. Willnotattempt to authenticate using SASL (unknown error)JLine support is enabled2016-08-2510:06:30,349[myid:] - INFO  [main-SendThread(sso.xxx.com:2181):ClientCnxn$SendThread@852] - Socket connection established to sso.xxx.com/192.168.1.80:2181, initiating session2016-08-2510:06:30,412[myid:] - INFO  [main-SendThread(sso.xxx.com:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server sso.xxx.com/192.168.1.80:2181, sessionid =0x156bf70d7580000, negotiated timeout =30000WATCHER::WatchedEventstate:SyncConnected type:None path:null[zk: c1:2181(CONNECTED)0]

添加一个根节点目录zookeeper_project

create /project zookeeper_project

[zk: c1:2181(CONNECTED) 0] create /project  zookeeper_project

Created /project

[zk: c1:2181(CONNECTED) 1] get /project

zookeeper_project

cZxid = 0x2

ctime = Thu Aug 25 10:09:09 CST 2016

mZxid = 0x2

mtime = Thu Aug 25 10:09:09 CST 2016

pZxid = 0x2

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x0

dataLength = 17

numChildren = 0

此时说明单机配置成功

多机集群

取消zoo.cfg中server.2和server.3的注释，最后zoo.cfg配置文件如下：

tickTime=2000

dataDir=/var/data/zookeeper

dataLogDir=/var/local/server/zookeeper/log

clientPort=2181

initLimit=5

syncLimit=2

server.1=c1:2888:3888

server.2=c2:2888:3888

server.3=c3:2888:3888

在c2和c3中执行类似的操作

多机联合测试

c1运行结果：

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh  status  

JMX enabled by default

Using config:/var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode:leader

c2运行结果：

[root@cas zookeeper]# /var/local/server/zookeeper/bin/zkServer.sh  status 

JMX enabled by defaultUsing config:/var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode:follower

c3运行结果：

[root@localhost zookeeper]# /var/local/server/zookeeper/bin/zkServer.sh   status

JMX enabled by defaultUsing config:/var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode:follower

同步测试

c1上执行：

create /c1project c1projecttest和get /c1project

[zk: c1:2181(CONNECTED) 2] create /c1project c1projecttest

Created /c1project

[zk: c1:2181(CONNECTED) 1] get /c1project

c1projecttest

cZxid = 0x100000013

ctime = Thu Aug 25 11:03:49 CST 2016

mZxid = 0x100000013

mtime = Thu Aug 25 11:03:49 CST 2016

pZxid = 0x100000013

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x0

dataLength = 13

numChildren = 0

c2上查看同步结果：

get /c1project

[zk: c2:2181(CONNECTED) 0] get   /c1project

c1projecttest

cZxid = 0x100000013

ctime = Thu Aug 25 11:03:49 CST 2016

mZxid = 0x100000013

mtime = Thu Aug 25 11:03:49 CST 2016

pZxid = 0x100000013

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x0

dataLength = 13

numChildren = 0

c3上查看同步结果：

[zk: c3:2181(CONNECTED) 4] get /c1project

c1projecttest

cZxid = 0x100000013

ctime = Thu Aug 25 11:03:49 CST 2016

mZxid = 0x100000013

mtime = Thu Aug 25 11:03:49 CST 2016

pZxid = 0x100000013

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x0

dataLength = 13

numChildren = 0

此时说明zookeeper集群配置成功。

从节点主动升级和接管成为主节点

leader和follower的自动切换，前面我们知道c1是leader，当leader down掉后是否有从节点升级为leader。

c1上执行

/var/local/server/zookeeper/bin/zkServer.sh   stop

c2上查看状态

[root@cas ~]# /var/local/server/zookeeper/bin/zkServer.sh   status

JMX enabled by default

Using config:/var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode:follower

可以发现c2仍然为从节点

再看看c3

[root@localhost ~]# /var/local/server/zookeeper/bin/zkServer.sh   status

JMX enabled by default

Using config:/var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode:leader

此时c3升级为了主节点

再次启动c1，并查看状态

[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh  start

JMX enabledbydefaultUsingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfgStarting

zookeeper ... STARTED

[root@sso conf]#[root@sso conf]# /var/local/server/zookeeper/bin/zkServer.sh    status

JMX enabledbydefault

Usingconfig: /var/local/server/zookeeper/bin/../conf/zoo.cfg

Mode: follower

发现c1作为了从节点。