二 用docker compose搭建kafka集群

本地一台机器，想要模拟多台机器搭建kafka集群，可以通过docker来实现。在另一篇两种方式搭建Zookeeper伪集群：纯手工或使用Docker compose里，尝试了如何用docker compose来搭建一个zookeeper集群。在此基础上，我们用docker compose搭建一个kafka集群。

准备好kafka镜像，docker-compose.yaml

1. 下载kafka镜像
   docker pull wurstmeister/kafka
2. 编写docker-compose.yaml

version: '3.1'
services:
  zoo1:
    image: zookeeper:3.4.13
    restart: always
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

  zoo2:
    image: zookeeper:3.4.13
    restart: always
    hostname: zoo2
    container_name: zoo2
    ports:
      - "2182:2181"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

  zoo3:
    image: zookeeper:3.4.13
    restart: always
    hostname: zoo3
    container_name: zoo3
    ports:
      - "2183:2181"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

  kafka1:
    image: wurstmeister/kafka
    restart: always
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka1
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  kafka2:
    image: wurstmeister/kafka
    restart: always
    hostname: kafka2
    container_name: kafka2
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka2
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  kafka3:
    image: wurstmeister/kafka
    restart: always
    hostname: kafka3
    container_name: kafka3
    ports:
      - "9094:9094"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka3
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    depends_on:
      - zoo1
      - zoo2
      - zoo3

3. 执行docker-compose命令搭建服务

$ docker-compose up
Creating zoo3
Creating zoo2
Creating zoo1
Creating kafka1
Creating kafka3
Creating kafka2

$ docker ps 
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                        NAMES
5f4e192e275c        wurstmeister/kafka   "start-kafka.sh"         34 hours ago        Up 2 hours          0.0.0.0:9093->9093/tcp                       kafka2
0569b044a9e0        wurstmeister/kafka   "start-kafka.sh"         34 hours ago        Up 2 hours          0.0.0.0:9094->9094/tcp                       kafka3
ee31d9b6dd23        wurstmeister/kafka   "start-kafka.sh"         34 hours ago        Up 2 hours          0.0.0.0:9092->9092/tcp                       kafka1
79528069f7cd        zookeeper:3.4.13     "/docker-entrypoin..."   34 hours ago        Up 2 hours          2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zoo1
a75506975f1d        zookeeper:3.4.13     "/docker-entrypoin..."   34 hours ago        Up 2 hours          2888/tcp, 3888/tcp, 0.0.0.0:2183->2181/tcp   zoo3
77fb2d918601        zookeeper:3.4.13     "/docker-entrypoin..."   34 hours ago        Up 2 hours          2888/tcp, 3888/tcp, 0.0.0.0:2182->2181/tcp   zoo2

测试kafka集群：创建topic，生产消息，消费消息

1. 进入任意一个kafka容器内部，创建topic:

$ docker exec -it kafka1 bash
bash-4.4# /opt/kafka_2.13-2.7.0/bin/kafka-topics.sh --create --topic chat --partitions 5 --zookeeper 172.19.0.11:2181 --replication-factor 3
Created topic chat.

2. 在kafka1容器内部，开启生产者，生产一条消息：

bash-4.4# /opt/kafka_2.13-2.7.0/bin/kafka-console-producer.sh --broker-list kafka1:9092 --topic chat
>hello world!

3. 进入任意kafka容器内部，开启消费者，接收到了消息。说明用docker搭建的kafka集群能正确的工作。

$ docker exec -it kafka2 bash
bash-4.4# /opt/kafka_2.13-2.7.0/bin/kafka-console-consumer.sh --bootstrap-server kafka2:9093 --topic chat --from-beginning
hello world!