kafka_2.11-0.10.1.0部署

下载ZK和Kafka

zookeeper-3.4.5-cdh5.7.0.tar.gz
kafka_2.11-0.10.1.0.tgz
都存放在
/home/hadoop/soul/soft/

• ZK部署

解压
[hadoop@hadoop000 soft]$ tar -zxvf zookeeper-3.4.5-cdh5.7.0.tar.gz -C ../app/

进入conf目录
[hadoop@hadoop000 conf]$ pwd
/home/hadoop/soul/app/zookeeper-3.4.5-cdh5.7.0/conf
复制配置文件
cp zoo_sample.cfg  zoo.cfg
编辑配置文件
[hadoop@hadoop000 conf]$ vim zoo.cfg 
修改dataDir路径，其他默认
dataDir=/home/hadoop/soul/data/tmp/zookeeper

配置环境变量
[hadoop@hadoop000 conf]$ echo $ZK_HOME
/home/hadoop/soul/app/zookeeper-3.4.5-cdh5.7.0

启动ZK
[hadoop@hadoop000 conf]$ zkServer.sh start

jps查看是否有QuorumPeerMain进程，有则启动成功

• Kafka部署

解压
[hadoop@hadoop000 soft]$ tar -zxvf kafka_2.11-0.10.1.0.tgz -C ../app/

进入config目录
[hadoop@hadoop000 config]$ pwd
/home/hadoop/soul/app/kafka_2.11-0.10.1.0/config

修改参数为(其他默认)：
listeners=PLAINTEXT://localhost:9092
log.dirs=/home/hadoop/soul/data/logs/tmp/kafka-logs
zookeeper.connect=localhost:2181/kafka

配置到环境变量
[hadoop@hadoop000 config]$ echo $KAFKA_HOME
/home/hadoop/soul/app/kafka_2.11-0.10.1.0

• 简单使用
  1、启动

kafka-server-start.sh -daemon  /home/hadoop/soul/app/kafka_2.11-0.10.1.0/config/server.properties 

2、创建topic

kafka-topics.sh --create --zookeeper localhost:2181/kafka --replication-factor 1 --partitions 1 --topic test

3、查看所有topic

kafka-topics.sh --list --zookeeper localhost:2181/kafka

4、启动生产者并输入消息

[hadoop@hadoop000 config]$ kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

5、启动消费者接收消息

[hadoop@hadoop000 kafka-logs]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

This is a message
This is another message