Docker搭建Kafka集群

【前言】这是基于docker-compose， 单机版的部署方式，适合于功能测试。

Docker镜像选型

wurstmeister/kafka 特点：docker hub star数最多，版本更新到 Kafka 2.0 ，zookeeper与kafka分开于不同镜像。

安装过程

假设docker、docker-compose等工具已经安装。

下载zookeeper和 kafka 镜像:

docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka

用户目录下创建一个docker-compose.yml文件

vi docker-compose.yml

内容如下：

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 10.0.3.5
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

[注] 10.0.3.5 是本机（vm）地址。

启动docker-compose

docker-compose up -d

[root@kafka ~]# docker-compose up -d
Creating network "root_default" with the default driver
Creating root_kafka_1     ... done
Creating root_zookeeper_1 ... done

启动多个kafka 节点，比如3个

docker-compose scale kafka=3

root@kafka ~]# docker-compose scale kafka=3
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting root_kafka_1 ... done
Creating root_kafka_2 ... done
Creating root_kafka_3 ... done

或者启动的时候一次直接启动三个kafka

[root@kafka ~]# docker-compose up --scale kafka=3 -d
Creating network "root_default" with the default driver
Creating root_kafka_1     ... done
Creating root_kafka_2     ... done
Creating root_kafka_3     ... done
Creating root_zookeeper_1 ... done

查看状态

[root@kafka ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED              STATUS              PORTS                                                NAMES
505cdd44ae8e        wurstmeister/kafka       "start-kafka.sh"         About a minute ago   Up About a minute   0.0.0.0:32788->9092/tcp                              root_kafka_1
a7d9cf355386        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   About a minute ago   Up About a minute   22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   root_zookeeper_1
41d84d774841        wurstmeister/kafka       "start-kafka.sh"         About a minute ago   Up About a minute   0.0.0.0:32787->9092/tcp                              root_kafka_3
e410abbc3635        wurstmeister/kafka       "start-kafka.sh"         About a minute ago   Up About a minute   0.0.0.0:32786->9092/tcp                              root_kafka_2

可用性测试

进入Kafka容器: （这里随机选择进入第二个kafka容器 root_kafka_2）

[root@kafka ~]# docker exec -it root_kafka_2 bash
bash-4.4#

创建一个topic（其中 zookeeper容器名为 root_zookeeper_1，topic名为test），输入：

bash-4.4# env | grep KAFKA_HOME
KAFKA_HOME=/opt/kafka
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic test \
> --zookeeper root_zookeeper_1:2181 --replication-factor 1 --partitions 1
Created topic "test".

查看新创建的topic:

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --zookeeper root_zookeeper_1:2181 --describe --topic test
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 1003    Replicas: 1003  Isr: 1003

发布消息: （输入若干条消息后 按^C 退出发布）

往broker Kafka 2 发送消息，到 kafka 1 检查能不能收到

bash-4.4# $KAFKA_HOME/bin/kafka-console-producer.sh --topic=test \
> --broker-list root_kafka_2:9092
> send message from kafka 2
>line 2
>^Cbash-4.4#

往 Kafka 3 发送消息

bash-4.4# $KAFKA_HOME/bin/kafka-console-producer.sh --topic=test \
> --broker-list root_kafka_3:9092
>send from kafka 2 to 3
>check if I can receive from kafka 1
>^Cbash-4.4#

接收消息: (按^C 退出接收)
到 kafka 1 接收，也可以去其它kafka 接收。

bash-4.4# $KAFKA_HOME/bin/kafka-console-consumer.sh \
> --bootstrap-server root_kafka_1:9092 \
> --from-beginning --topic test
send message from kafka 2
line 2
send from kafka 2 to 3
check if I can receive from kafka 1
^CProcessed a total of 4 messages
bash-4.4#

接收到的消息来自两处：root_kafka_2 和 root_kafka_3。验证通过

参考文档：
https://github.com/wurstmeister/kafka-docker
http://wurstmeister.github.io/kafka-docker/
https://www.jianshu.com/p/ac03f126980e
https://www.jianshu.com/p/8ccd712e2599