Basic Kafka 上次关注这个居然已经是近一年前了。
业务上需要处理:如果确保消费的消息没有处理完成的话,依然可以在下次继续使用?(sonar扫描任务记录到kafka,多个consumer从kafka读取出来进行处理。但有可能处理失败)
10分钟视频介绍,官网入门介绍、核心概念 读完这几篇就有个基本的概念
Kafka in a Nutshell——这篇还没读完。
Producer:写入消息到topic;
Consumer:读取topic的消息;读取的位置有offset维护(对应的哪个topci在哪个partions上的哪个位置)
Cluster:保存topic的消息。包含多个broker,每个broker将topic维护到多个partitions,每个partition是个完整的副本)
相当有一个server端,两个client端(P&C)
同一组内的consumer(目前业务的状态)会由Kafka维护以确保每个consumer不会读到重复的event/message
- Consumers can join a group by using the samegroup.id.
- The maximum parallelism of a group is that the number of consumers in the group ← no of partitions.
- Kafka assigns the partitions of a topic to the consumer in a group, so that each partition is consumed by exactly one consumer in the group.
- Kafka guarantees that a message is only ever read by a single consumer in the group.
- Consumers can see the message in the order they were stored in the log.
Kafka - Understanding Consumer Group with examples
-
If we have three partitions for a topic and we start three consumers for the same topic then each consumer is assigned one partition automatically.
-
If we have three partitions for a topic and we start two consumers for the same topic then one consumer is assigned one partition and other is assigned two partitions.
-
If we have three partitions for a topic and we start one consumer for the same topic then one consumer is assigned all three partitions.
-
If we have three partitions for a topic and we start four consumers for the same topic then three of four consumers are assigned one partition each, and one consumer will not receive any messages.
网友评论