Kafka 架构，基于ZooKeeper

刚开始做后端没多久，遇到各种新鲜技术名词。现在了解一下kafka的架构和原理。

引用自 http://cloudurable.com/blog/kafka-architecture/index.html

Kafka框架

Kafka consists of Records, Topics, Consumers, Producers, Brokers, Logs, Partitions, and Clusters.  A Broker is a Kafka server that runs in a Kafka Cluster. Kafka Brokers form a cluster. The Kafka Cluster consists of many Kafka Brokers on many servers. Broker sometimes refer to more of a logical system or as Kafka as a whole.

Kafka needs ZooKeeper

Kafka Architecture: Core Kafka

Kafka uses Zookeeper to do leadership election of Kafka Broker and Topic Partition pairs. Kafka uses Zookeeper to manage service discovery for Kafka Brokers that form the cluster. Zookeeper sends changes of the topology to Kafka, so each node in the cluster knows when a new broker joined, a Broker died, a topic was removed or a topic was added, etc. Zookeeper provides an in-sync view of Kafka Cluster configuration. 

简而言之Kafka需要Zookeeper同步内部cluster的消息，比如加入新的topic了。

Kafka Brokers

A Kafka cluster is made up of multiple Kafka Brokers. Each Kafka Broker has a unique ID (number). Kafka Brokers contain topic log partitions. Connecting to one broker bootstraps a client to the entire Kafka cluster. For failover, you want to start with at least three to five brokers. A Kafka cluster can have, 10, 100, or 1,000 brokers in a cluster if needed.

Kafka supports replication to support failover. Recall that Kafka uses ZooKeeper to form Kafka Brokers into a cluster and each node in Kafka cluster is called a Kafka Broker. Topic partitions can be replicated across multiple nodes for failover. The topic should have a replication factor greater than 1 (2, or 3). For example, if you are running in AWS, you would want to be able to survive a single availability zone outage. If one Kafka Broker goes down, then the Kafka Broker which is an ISR (in-sync replica) can serve data.

Kafka 内部由许多Brokers node组成，Brokers包含日志主题。每个日志主题在超过一个的Brokers上存放，以达到备份，灾难恢复。对brokers的消息同步，就是通过zookeeper实现，比如leader brokers的选举，brokers之间的消息同步等。

http://cloudurable.com/blog/kafka-architecture-topics/index.html 介绍了详细的partition在brokers的存放，备份，同步。