How is Kafka so Fast?

How is Kafka so Fast?

• 1.
  Ricardo Paiva andHervé Rivière Understanding the design of Kafka and how it handles Criteo workload How is Kafka so fast?
• 2.
  What is Kafka?
• 3.
  3 • Apache Kafkais a distributed message queue • Open-sourced by LinkedIn in 2011 • High-throughput • Highly distributed • Fault-tolerant • Low-latency What is Kafka?
• 4.
  4 • • Usecase • GLUP pipeline (aka Kafka Local) • Streaming event processing platform (aka Kafka Stream) • Some figures : • 14 clusters / 200 servers / 7 DC • Up to 7 millions messages / sec • Up to 150 TB processed per day Kafka @ Criteo ?
• 5.
  5 • Topics, Partitionsand Offsets 7 6 5 4 3 2 1 08910 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 08 7 6 5 4 3 2 1 0891011 7 6 5 4 3 2 1 08 7 6 5 4 3 2 1 0 Partition 0 Partition 1 Partition 0 Partition 1 Partition 2 Partition 3 OldNew Writes Topic A Topic B
• 6.
  Complexity inside theclients
• 7.
  7 • Brokers • Managepartitions • Receive from producer records for a (topic, partition) • Answer to consumer asking records for (topic, partition, offset) • Manage replicas • Manage consumer coordination • Assigning good partitions to the good consumer Broker 1 Producer Broker 2 Consumer Consumer Fetch (Topic A, Partition 4, Offset 10) Bytes Fetch (Topic B, Partition 1, Offset 10) Bytes
• 8.
  8 • Producers Producer Broker (partition leader) Broker (replica) Broker (replica) ack •Producers decide what partitions to send to; • Producers can send a batch of messages; • Producers can compress a batch; • Producers wait for acknowledgement from the broker (acks=1) or broker + replica (acks=all);
• 9.
  9 • Consumers ConsumerBroker 6 78 9 10 11 12 12 offset=7 Partition 2: Partition 2, offset 6 7 8 9 1 2 3Commit offset=9 • Consumers control what offset to consume from; • Consumers commit offsets to kafka, but it’s just another Kafka topic; • Consumers can receive batched and / or compressed data; • Kafka coordinates which partitions each consumer will consume from.
• 10.
  Did you saySSD is better than HDD ?
• 11.
  11 • Faster butnot that much
• 12.
  12 • • EachKafka partition is mapped to segment files • Segment file : log append structure • Records are immutable • Broker is doing very few random disk search Only sequential I/O Kafka Active Segment file Old segment files
• 13.
  13 • • Kafkarelies on native Linux Page cache (read-ahead and write-behind) • JVM off-heap cache for free • Kafka records aren’t deserialized in Kafka JVM • No Java object memory overhead • No OutOfMemory issue • No big GC pauses Caching data for free Kafka Active Segment file Disk OS Old segments files
• 14.
  14 • Reliability withreplication • Kafka disk writes are asynchronous • Kafka replicas synchronisation (over network) is synchronous • Trusting replicas in case of data corruption / server crash Broker (partition leader) Broker (replica) Broker (previous leader)
• 15.
  Zero Copy
• 16.
  16 • Sending datafrom file to network (traditional approach) read(file, tmp_buf, len); write(socket, tmp_buf, len);
• 17.
  17 • Sending datafrom file to network (zero-copy approach) transferTo(position, count, writableChannel);
• 18.
  Make things simple
• 19.
  19 • • Paralelismbased on topic partitions; • Data compressed/uncompressed on the client; • Producers send a batch of messages; • No serialization/deserialization costs on the brokers; • Writing directly to file: • Append only (cheapers disks); • No complex data structure (no BTree or LSM tree); • Uses OS memory management; • Relies on replicas not on disks; • Zero-copy; Key takeaways
• 20.
  Thank you! #rivers