Kafka Indexing -> Segment Flow

Where do Kafka messages go when they Enter the Druid? Do they morph into insects? Do they exist at all?

This diagram shows the relationship between partitions on the Kafka broker, ingestion tasks, and Druid segments during Kafka Indexing. Druid parameters that influence ingestion efficiency and number of segments are noted.

The Kafka Indexing Service works differently before and after Druid 0.12 (Imply 2.5.x). The main difference is that pre-0.12, segments could not mix data from different partitions. This can have a large impact on total number of segments generated when using <0.12.

Additional references:

http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html

http://druid.io/docs/latest/tutorials/tutorial-kafka.html