kafka 部署使用

kafka 简单介绍

• Produce
• Consumer
• Broker
• Topic
  • partition

使用 docker 方式搭建

image.png

• 搭建zookeeper 环境

  # docker pull zookeeper 
  # docker run -d --restart=always --name zookeeper -p 2181:2181 -v /etc/localtime:/etc/localtime \
                  --log-driver json-file --log-opt max-size=100m --log-opt max-file=2  \
                  zookeeper
  

• 搭建kafka环境
  192.168.48.134 为我的宿主机ip，下面需要按照自己的环境做对应的修改。

  # docker pull wurstmeister/kafka
  # docker run -d --restart=always --name kafka1 -p 19092:9092 -v /etc/localtime:/etc/localtime \
                  --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
                  -e KAFKA_BROKER_ID=1 \
                  -e KAFKA_ZOOKEEPER_CONNECT=192.168.48.134:2181 \
                  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.48.134:19092 \
                  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
                  wurstmeister/kafka
  
  # docker run -d --restart=always --name kafka2 -p 19093:9092 -v /etc/localtime:/etc/localtime \
                  --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
                  -e KAFKA_BROKER_ID=2 \
                  -e KAFKA_ZOOKEEPER_CONNECT=192.168.48.134:2181 \
                  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.48.134:19093 \
                  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
                  wurstmeister/kafka
  
  # docker run -d --restart=always --name kafka3 -p 19094:9092 -v /etc/localtime:/etc/localtime \
                  --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
                  -e KAFKA_BROKER_ID=3 \
                  -e KAFKA_ZOOKEEPER_CONNECT=192.168.48.134:2181 \
                  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.48.134:19094 \
                  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
                  wurstmeister/kafka
  

• 验证是否安装成功

  # docker exec -ti kafka1 /bin/bash
  # cd /opt/kafka/bin
  
  // 创建topic：分区数为4，副本因子为3(3个broker)
  # ./kafka-topics.sh --zookeeper 192.168.48.134:2181 --create --topic test --replication-factor 3 --partitions 4 Created topic "test"
  # ./kafka-topics.sh --zookeeper 192.168.48.134:2181 --describe --topic test
    Topic: test TopicId: _A5YjE72QWeECBPEOFCquw PartitionCount: 4   ReplicationFactor: 3    Configs: 
      Topic: test   Partition: 0    Leader: 3   Replicas: 3,2,1 Isr: 3,2,1
      Topic: test   Partition: 1    Leader: 1   Replicas: 1,3,2 Isr: 1,3,2
      Topic: test   Partition: 2    Leader: 2   Replicas: 2,1,3 Isr: 2,1,3
      Topic: test   Partition: 3    Leader: 3   Replicas: 3,1,2 Isr: 3,1,2
  

  可以看出来，目前，3个kafka实例的集群环境已经搭建ok

  进一步，测试消息的发送和消费

  // 在kafka2 上发送消息
  # docker exec -ti kafka2 /bin/bash
  # cd /opt/kafka/bin
  # ./kafka-console-producer.sh --broker-list 192.168.48.134:19092,192.168.48.134:19093,192.168.48.134:19094 --topic test
  >hello, kafka!
  
  // 另启一个ssh 在kafka3 上消费消息
  # docker exec -ti kafka3 /bin/bash
  # cd /opt/kafka/bin
  # ./kafka-console-consumer.sh --bootstrap-server 192.168.48.134:19092,192.168.48.134:19093,192.168.48.134:19094 --topic test --from-beginning
  hello, kafka!
  

物理主机方式搭建

TODO

---

参考资料
1、https://www.cnblogs.com/answerThe/p/11267129.html#:~:text=%E5%9F%BA%E4%BA%8Edocker%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BAkafka%E9%9B%86%E7%BE%A4%EF%BC%88%E5%8D%95%E6%9C%BA%E7%89%88%EF%BC%89%201%20kafka%E4%BE%9D%E8%B5%96%E4%BA%8Ezookeeper%EF%BC%88%E4%BC%A0%E9%80%81%E9%97%A8https%3A%2F%20...%202%20%E5%90%AF%E5%8A%A8zookeeper%E5%AE%B9%E5%99%A8%EF%BC%88wurstmeister%2Fz%20...%203,4%20%E5%88%86%E5%88%AB%E8%BF%9B%E5%85%A5%E5%AE%B9%E5%99%A8kafka1%E3%80%81kafka2%E3%80%81kafka3%E5%88%9B%E5%BB%BA%20...%205%20%E6%B5%8B%E8%AF%95%E9%9B%86%E7%BE%A4%EF%BC%8C%E5%9C%A8kafka3%E4%B8%8A%E5%90%91test1%E5%8F%91%E9%80%81%E6%B6%88%E6%81%AF%EF%BC%8C%E5%9C%A8kaf%20...%206%20%E4%BE%9D%E6%AC%A1%E5%85%B3%E9%97%ADkafka2%E3%80%81test1%E5%90%8E%E6%9F%A5%E7%9C%8B%E9%9B%86%E7%BE%A4%E7%8A%B6%E6%80%81)