这篇文章主要用来介绍Kafka & Zookeeper相关基础知识。
1. Kafka是由Linkedin开发，是一个分布式、支持分区的（partition）、多副本的（replica），基于zookeeper协调的分布式消息系统，它的最大的特性就是可以实时的处理大量数据以满足各种需求场景，之后于2010年贡献给了Apache基金会并成为顶级开源项。
2. Zookeeper是专为分布式系统设计的开源协调系统，类似一个大管家，用来管理分布式应用的数据管理问题。Zookeeper本身就是高可用的，基于Cluster（一个Leader，多个Follower）的。ZooKeeper是以Fast Paxos算法为基础，实现同步服务，配置维护和命名服务等分布式应用。
根据官方文档，我简单部署了Kafka示例。
https://kafka.apache.org/quickstart
Download the 2.0.0 release and un-tar it.
> tar-xzf kafka_2.11-2.0.0.tgz
> cdkafka_2.11-2.0.0
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance.
> bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...
Now start the Kafka server:
> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
...
Let's create a topic named "test" with a single partition and only one replica:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
We can now see that topic if we run the list topic command:
> bin/kafka-topics.sh --list --zookeeper localhost:2181
test
Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.
Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default, each line will be sent as a separate message.
Run the producer and then type a few messages into the console to send to the server.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message
Kafka also has a command line consumer that will dump out messages to standard output.
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test--from-beginning
This is a message
This is another message
If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal.
All of the command line tools have additional options; running the command with no arguments will display usage information documenting them in more detail.
其中，9092是默认的consumer和producer的端口。
一个基于Spring Boot & Kafka的示例。
https://github.com/buxiaoliang/kafka
网友评论