美文网首页
kafka集群搭建 - ZooKeeper

kafka集群搭建 - ZooKeeper

作者: kekefund | 来源:发表于2018-12-04 16:49 被阅读20次

一、什么是ZooKeeper？

ZooKeeper（动物园管理员），顾名思义，是用来管理Hadoop（大象）、Hive（蜜蜂）、Pig（小猪）的管理员，同时Apache HBase、Apache Solr、LinkedIn Sensei等众多项目中都采用了ZooKeeper。

ZooKeeper是一个集中式服务，用于维护配置信息、命名、提供分布式同步和提供组服务。所有这些类型的服务都以某种形式被分布式应用程序使用。每次它们被实现时，都有大量的工作用于修复不可避免的bug和竞选条件。由于实现这类服务的困难，应用程序最初通常跳过它们，这使得它们在存在更改时变得脆弱，并且难以管理。即使正确完成，这些服务的不同实现也会在部署应用程序时导致管理复杂性。

二、Zookeeper集群

Kafka使用Zookeeper来存储集群元数据以及消费者元数据。Zookeeper集群被称为群组。Zookeeper使用的是一致性协议，所以建议每个群组里应该包含奇数个节点（比如3个、5个等），因为只有当群组里的大多数节点（也就是法定人数）处于可用状态，Zookeeper才能处理外部的请求。也就是说，如果3个节点的集群，允许1个节点失效。如果5个节点的集群，允许2个节点失效。

三、Docker方式配置ZooKeeper

myid文件：标识要写到快照目录下面myid文件里

采用Docker镜像：wurstmeister/zookeeper，部署一个3节点的集群。

节点一：

  • docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./zoo.cfg:/opt/zookeeper-3.4.9/conf/zoo.cfg
      - ./data:/opt/zookeeper-3.4.9/data
      - ./logs:/opt/zookeeper-3.4.9/logs
      - ./myid:/opt/zookeeper-3.4.9/data/myid
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
    restart: always
  • myid文件
1
  • zoo.cfg文件
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper-3.4.9/data
dataLogDir=/opt/zookeeper-3.4.9/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#kafka01 (10.100.158.245)
server.1=0.0.0.0:2888:3888
#kafka02 (10.100.19.15)
server.2=10.150.19.15:2888:3888
#kafka03 (10.100.124.217)
server.3=10.100.124.217:2888:3888

节点二：

  • docker-compose.yml
version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./zoo.cfg:/opt/zookeeper-3.4.9/conf/zoo.cfg
      - ./data:/opt/zookeeper-3.4.9/data
      - ./logs:/opt/zookeeper-3.4.9/logs
      - ./myid:/opt/zookeeper-3.4.9/data/myid
    ports:
      - "2180:2181"
      - "2888:2888"
      - "3888:3888"
    restart: always
  • myid文件
2
  • zoo.cfg文件
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper-3.4.9/data
dataLogDir=/opt/zookeeper-3.4.9/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#kafka01 (10.100.158.245)
server.1=10.100.158.245:2888:3888
#kafka02 (10.100.19.15)
server.2=0.0.0.0:2888:3888
#kafka03 (10.100.124.217)
server.3=10.100.124.217:2888:3888

节点三：

  • docker-compose.yml
version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./zoo.cfg:/opt/zookeeper-3.4.9/conf/zoo.cfg
      - ./data:/opt/zookeeper-3.4.9/data
      - ./logs:/opt/zookeeper-3.4.9/logs
      - ./myid:/opt/zookeeper-3.4.9/data/myid
    ports:
      - "2179:2181"
      - "2888:2888"
      - "3888:3888"
    restart: always
  • myid文件
3
  • zoo.cfg文件
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper-3.4.9/data
dataLogDir=/opt/zookeeper-3.4.9/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#kafka01 (10.100.158.245)
server.1=10.100.158.245:2888:3888
#kafka02 (10.100.19.15)
server.2=10.100.19.15:2888:3888
#kafka03 (10.100.124.217)
server.3=0.0.0.0:2888:3888

四、查看ZooKeeper状态


# 服务器1

[root@VM_158_24_centos ~]# docker exec -it s1_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: leader
 
 
 
#服务器2
 
[root@VM_19_15_centos ~]# docker exec -it s2_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower
 
 
 
#服务器3
 
[root@VM_124_27_centos ~]# docker exec -it s3_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower

停掉leader，2个follower的状态有变化，如下：


# 服务器1
 
[root@VM_158_245_centos ~]# docker stop s1_zookeeper_1
s1_zookeeper_1
 
 
# 服务器2
 
2018-10-30 10:54:44,555 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 1 at election address /10.154.158.245:3888
java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
 at java.lang.Thread.run(Thread.java:745)
2018-10-30 10:54:44,556 [myid:2] - INFO [WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.154.158.245 to address: /10.154.158.245
2018-10-30 10:54:44,556 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x500000000 (n.zxid), 0xf4 (n.round), LOOKING (n.state), 2 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
2018-10-30 10:54:44,759 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:QuorumPeer@844] - FOLLOWING
2018-10-30 10:54:44,759 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:ZooKeeperServer@173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /opt/zookeeper-3.4.9/logs/version-2 snapdir /opt/zookeeper-3.4.9/data/version-2
2018-10-30 10:54:44,759 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 215
2018-10-30 10:54:44,760 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.105.124.217 to address: /10.105.124.217
2018-10-30 10:54:44,761 [myid:2] - WARN [QuorumPeer[myid=2]/0.0.0.0:2181:Learner@236] - Unexpected exception, tries=0, connecting to /10.105.124.217:2888
java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:228)
 at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:69)
 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)
2018-10-30 10:54:45,794 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:Learner@329] - Getting a snapshot from leader
2018-10-30 10:54:45,796 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:FileTxnSnapLog@240] - Snapshotting: 0x500000000 to /opt/zookeeper-3.4.9/data/version-2/snapshot.500000000
 
 
# 服务器3
2018-10-30 10:54:44,567 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:FastLeaderElection@818] - New election. My id = 3, proposed zxid=0x500000000
2018-10-30 10:54:44,568 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 1 at election address /10.154.158.245:3888
java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
 at java.lang.Thread.run(Thread.java:745)
2018-10-30 10:54:44,569 [myid:3] - INFO [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.154.158.245 to address: /10.154.158.245
2018-10-30 10:54:44,569 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x500000000 (n.zxid), 0xf4 (n.round), LOOKING (n.state), 3 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
2018-10-30 10:54:44,575 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x500000000 (n.zxid), 0xf4 (n.round), LOOKING (n.state), 2 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
2018-10-30 10:54:44,776 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer@856] - LEADING
2018-10-30 10:54:44,779 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@59] - TCP NoDelay set to: true
2018-10-30 10:54:44,780 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:ZooKeeperServer@173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /opt/zookeeper-3.4.9/logs/version-2 snapdir /opt/zookeeper-3.4.9/data/version-2
2018-10-30 10:54:44,780 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 219
2018-10-30 10:54:45,782 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@329] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@7f54169
2018-10-30 10:54:45,808 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@384] - Synchronizing with Follower sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x300000000
2018-10-30 10:54:45,809 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@458] - Sending SNAP
2018-10-30 10:54:45,809 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@482] - Sending snapshot last zxid of peer is 0x300000000 zxid of leader is 0x600000000sent zxid of db as 0x500000000
2018-10-30 10:54:45,820 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@518] - Received NEWLEADER-ACK message from 2
2018-10-30 10:54:45,820 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@952] - Have quorum of supporters, sids: [ 2,3 ]; starting up and setting last processed zxid: 0x600000000

再次查看status:

 
#服务器2
 
[root@VM_19_15_centos ~]# docker exec -it s2_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower
 
 
 
#服务器3
 
[root@VM_124_217_centos ~]# docker exec -it s3_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: leader

服务器3变成leader了！

重新启动服务器1，服务器1是follower

 
[root@VM_158_245_centos ~]# docker start s1_zookeeper_1
 
s1_zookeeper_1
 
[root@VM_158_245_centos ~]# docker exec -it s1_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower

参考

Configuring Zookeeper Cluster

多台虚拟机搭建zookeeper集群

搭建kafka集群详细教程

zookeeper.apache.org

相关文章

  • Kafka学习笔记（二）

    Kafka环境搭建 准备工作 Kafka集群是把状态保存在Zookeeper中的，首先要搭建Zookeeper集群...

  • 多台虚拟机搭建zookeeper集群

    zookeeper集群搭建 kafka是把状态保存在zookeeper中的，首先要搭建zookeeper集群。尽管...

  • centos7 搭建kafka集群

    搭建kafka集群，需要先搭建zookeeper集群，搭建zookeeper集群，依赖于jdk 1.下载安装jdk...

  • Docker搭建kafka集群

    搭建zookeeper集群 zookeeper集群搭建 拉取kafka镜像 运行容器 主机一： 主机二： 主机三：

  • Kafka集群搭建

    Kafka集群搭建 准备工作 Kafka 依赖 ZooKeeper ，所以需要提前安装好 ZooKeeper ,...

  • Kafka集群

    Kafka集群是把状态保存在Zookeeper中的，首先要搭建Zookeeper集群。Zookeeper集群部署请...

  • Hadoop+zookeeper+Flume+Spark+Kaf

    Hadoop zookeeper Flume Spark Kafka Hbase大数据集群搭建教程 集群搭建流程 ...

  • kafka集群搭建

    kafka集群搭建 一、zookeeper集群搭建 略，请参考其他文档，这里不再赘述。 二、下载kafka 从ka...

  • kafka集群搭建文档

    1 Kafka集群搭建 zookeeper集群搭建 https://www.jianshu.com/p/6ab2b...

  • kafka环境搭建2-broker集群+zookeeper集群

    上篇文章kafka环境搭建1-单机版介绍了kafka基本环境的搭建。kafka基于zookeeper管理集群，要在...

网友评论

      本文标题：kafka集群搭建 - ZooKeeper

      本文链接：https://www.haomeiwen.com/subject/ikkvcqtx.html

      延伸阅读

      深度阅读

      • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

      栏目导航

      热点阅读

      关于我们|服务条款|联系我们|kafka集群搭建 - ZooKeeper|投稿指南|网站地图|RSS订阅|排版工具|手机版

      提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

      Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

      备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

      本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

      所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！