Cheat SheetsApache KafkaFundamentals

Fundamentals — Cheat Sheet

Apache Kafka · 11 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org

Fundamentals

Apache Kafka11 topicsQuick revision reference

Apache Kafka Introduction

Kafka is a distributed, fault-tolerant, high-throughput event streaming platform used for real-time data pipelines, messaging, and event-driven architectures.

✓Kafka is a distributed, durable, ordered commit log — not a traditional queue.
✓Topics are split into partitions; each partition is an ordered, immutable sequence.
✓Consumer groups enable parallel processing; one partition → one consumer per group.
✓Offsets track each consumer group's position — consumers control when they commit.
✓Records are retained for a configurable period (default 7 days) regardless of consumption.
✓Common uses: event-driven microservices, CDC, log aggregation, stream processing, event sourcing.

Java — minimal producer and consumer

// Producer: send a message
ProducerRecord<String, String> record = new ProducerRecord<>(
    "orders",        // topic
    "order-123",     // key  → determines partition
    "{"amount":99}" // value
);
producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.printf("Sent to %s[%d] at offset %d%n",
            metadata.topic(), metadata.partition(), metadata.offset());
    }
});

// Consumer: poll loop
consumer.subscribe(List.of("orders"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> r : records) {
        System.out.printf("[%d] %s = %s%n", r.offset(), r.key(), r.value());
    }
    consumer.commitSync();
}

Kafka vs Traditional Messaging

Unlike RabbitMQ or ActiveMQ, Kafka persists messages on disk as an immutable log, enables multiple consumers to replay events, and scales to millions of messages per second.

✓Kafka is a durable, append-only log — messages are NOT deleted after consumption.
✓Consumers pull at their own pace and maintain independent offsets.
✓Multiple consumer groups can independently read the same topic (fan-out without duplication).
✓RabbitMQ is push-based with per-message delivery guarantees and complex routing (exchanges).
✓Choose Kafka for high-throughput, replayable event streams; RabbitMQ for task queues and complex routing.
✓Both can coexist: Kafka as event backbone, RabbitMQ for task/notification queues.

Conceptual — architectural comparison table

// Traditional MQ (RabbitMQ) — push model
// Broker decides delivery; message deleted after ack
Producer → Exchange → Queue → Consumer (message gone after ack)

// Kafka — pull model / log
// Consumer reads at its own pace; message persisted on disk
Producer → Topic (partition 0) → log: [offset 0][offset 1][offset 2]...
                                        ↑ consumer-group-A at offset 2
                                        ↑ consumer-group-B at offset 0  (replaying from start)

// Key differences:
// ┌─────────────────────────┬─────────────────┬─────────────────┐
// │ Property                │ RabbitMQ        │ Kafka           │
// ├─────────────────────────┼─────────────────┼─────────────────┤
// │ Delivery model          │ Push            │ Pull            │
// │ Message retention       │ Until ack       │ Time-based      │
// │ Multiple consumers      │ Competing       │ Independent     │
// │ Replay                  │ No              │ Yes             │
// │ Throughput              │ ~50k msg/s      │ Millions msg/s  │
// │ Ordering                │ Per-queue       │ Per-partition   │
// │ Routing                 │ Exchanges/rules │ By topic/key    │
// └─────────────────────────┴─────────────────┴─────────────────┘

Topics & Partitions

A topic is a named log of events; partitions are ordered, immutable sequences that allow parallelism — more partitions mean more consumer throughput but higher metadata overhead.

✓A topic is a named, durable, append-only log; partitions are its physical subdivisions, each stored on a single broker leader.
✓Ordering is guaranteed only within a partition — use message keys to ensure all events for the same entity land in the same partition.
✓The number of partitions is the maximum degree of parallelism for a consumer group; you cannot have more active consumers than partitions.
✓Kafka retains records regardless of consumption — multiple consumer groups can read the same topic independently at their own offsets.
✓Each record in a partition has a unique, monotonically increasing offset; there are no global offsets across partitions.
✓Partition count is non-trivial to change after creation; increasing it breaks key-based ordering for keys that haven't been produced yet.

Java — Kafka AdminClient

// Create a topic programmatically using AdminClient
Properties adminProps = new Properties();
adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

try (AdminClient admin = AdminClient.create(adminProps)) {
    int numPartitions = 12;
    short replicationFactor = 3;

    NewTopic ordersTopic = new NewTopic("orders", numPartitions, replicationFactor);
    // Optional: set topic-level config
    ordersTopic.configs(Map.of(
        "retention.ms",    String.valueOf(7 * 24 * 60 * 60 * 1000L), // 7 days
        "compression.type","snappy"
    ));

    admin.createTopics(List.of(ordersTopic)).all().get();
    System.out.println("Topic created successfully");
}

Brokers & Clusters

A Kafka cluster is a group of broker nodes; topics are spread across brokers for fault tolerance and throughput, and a controller broker coordinates partition leadership.

✓A broker is a Kafka server; a cluster is a group of cooperating brokers.
✓Each partition has one leader broker and N−1 follower replicas.
✓Only the leader handles reads and writes; followers replicate asynchronously.
✓ISR (in-sync replicas) are followers within replica.lag.time.max.ms of the leader.
✓acks=all + min.insync.replicas=2 is the standard durability configuration.
✓KRaft (Kafka 3.x) replaces ZooKeeper with an internal Raft quorum controller.

Shell — describe topic partitions and leaders

# View cluster metadata with kafka-topics.sh
kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

# Output (example):
# Topic: orders  Partitions: 3  ReplicationFactor: 3
# Partition: 0  Leader: 2  Replicas: 2,0,1  Isr: 2,0,1
# Partition: 1  Leader: 0  Replicas: 0,1,2  Isr: 0,1,2
# Partition: 2  Leader: 1  Replicas: 1,2,0  Isr: 1,2,0
#
# Isr = in-sync replicas — only these can become leader on failover

ZooKeeper & KRaft Mode

ZooKeeper stored Kafka metadata historically; KRaft (Kafka Raft, 3.x) replaces it with an internal quorum controller, eliminating the external dependency and simplifying operations.

✓ZooKeeper stored Kafka metadata externally — a separate cluster to manage and monitor.
✓KRaft replaces ZooKeeper with an internal Raft quorum stored in __cluster_metadata topic.
✓KRaft is GA since Kafka 3.3; ZooKeeper mode is removed in Kafka 4.0.
✓KRaft supports millions of partitions vs ~200k with ZooKeeper.
✓Controller failover with KRaft is milliseconds vs 30-120 seconds with ZooKeeper.
✓kafka-storage.sh format must be run once before starting a KRaft cluster.

Shell — ZooKeeper-based Kafka setup

# Legacy ZooKeeper-based broker config (server.properties)
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka   # ZK quorum + chroot
broker.id=1
log.dirs=/var/kafka/data

# Start ZooKeeper first
bin/zookeeper-server-start.sh config/zookeeper.properties

# Then start Kafka brokers
bin/kafka-server-start.sh config/server.properties

# View metadata stored in ZooKeeper
bin/zookeeper-shell.sh zk1:2181
# > ls /kafka/brokers/ids       → active broker IDs
# > get /kafka/controller        → current controller broker
# > ls /kafka/topics             → all topics

# Limitations:
# - ZK handles ~200k partition leadership changes
# - Slow controller failover (30-120 seconds in large clusters)
# - Additional ops burden: separate ZK JVM, separate monitoring

Replication & In-Sync Replicas (ISR)

Each partition has a configurable replication factor; the ISR list tracks replicas that are fully caught up — only ISR replicas can be elected leader on failover.

✓replication.factor copies each partition to N brokers for fault tolerance
✓ISR = set of replicas caught up within replica.lag.time.max.ms of the leader
✓Leader election on failure picks only from the current ISR (unless unclean election enabled)
✓acks=all ensures all ISR replicas acknowledge; combine with min.insync.replicas=2 for RF=3
✓unclean.leader.election.enable=true risks data loss but prevents partition unavailability
✓Monitor under-replicated-partitions metric — it fires before data loss occurs

Kafka — replication factor and ISR inspection

# Create a topic with replication factor 3
kafka-topics.sh --create   --topic orders   --partitions 6   --replication-factor 3   --bootstrap-server broker1:9092

# Describe topic — see leaders, replicas, ISR per partition
kafka-topics.sh --describe --topic orders --bootstrap-server broker1:9092

# Output:
# Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
# (If broker 3 lags, partition 0 ISR becomes: 1,2)

# Under-replicated partitions alert — ISR < replication factor
kafka-topics.sh --describe --under-replicated-partitions   --bootstrap-server broker1:9092

Leaders & Followers

Every partition has one leader that handles all reads and writes; followers replicate from the leader and can be promoted to leader if the leader fails.

✓Every partition has exactly one leader; all produces and consumes go through the leader.
✓Followers replicate from the leader; ISR members have replicated all messages within lag threshold.
✓On leader failure, the controller promotes the first ISR member as the new leader.
✓unclean.leader.election.enable=false (default) prevents data loss by rejecting out-of-sync leaders.
✓acks=all + min.insync.replicas=2 guarantees writes survive one broker failure.
✓KIP-392 (Kafka 2.4+) enables consumer reads from the nearest rack-aware replica.

Shell — describe topic leaders and trigger election

# Inspect leader and ISR per partition
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

# Sample output:
# Topic: orders  Partitions: 3  ReplicationFactor: 3
# Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
# Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2
#
# Ideal: leaders evenly spread across brokers (1 per partition)
# Problem: if broker 1 is slow, ISR may shrink:
# Isr: 1   ← only leader is in-sync, followers fell behind

# Trigger preferred leader election (restores original leader distribution)
kafka-leader-election.sh \
  --bootstrap-server localhost:9092 \
  --election-type PREFERRED \
  --all-topic-partitions

Offsets & Log Structure

Each message in a partition has a monotonically increasing offset; the log is append-only and immutable, enabling deterministic replay and position-based consumption.

✓Offset is a monotonically increasing integer identifying a message's position in a partition.
✓The partition log is append-only and immutable; old segments are deleted or compacted.
✓Consumers store committed offsets in __consumer_offsets; committing means "processed up to here".
✓auto.offset.reset=earliest replays from the beginning; latest skips existing messages.
✓At-least-once is the practical default: commit after processing, deduplicate downstream.
✓Exactly-once requires enable.idempotence=true on producer + read_committed isolation on consumer.

Shell — partition log structure on disk

# On-disk layout of a partition (orders-0)
/var/kafka/data/orders-0/
  00000000000000000000.log       # messages at offsets 0–999999
  00000000000000000000.index     # offset → physical file position
  00000000000000000000.timeindex # timestamp → offset lookup
  00000000000001000000.log       # messages at offsets 1000000–1999999
  00000000000001000000.index
  00000000000002000000.log       # active (current) segment

# Each .log entry: [offset][message size][CRC][magic][attributes][timestamp][key][value]

# Retention settings (application.properties / topic config)
# Log retention — delete segments older than 7 days
log.retention.hours=168

# Or size-based retention — keep last 10 GB per partition
log.retention.bytes=10737418240

# Log compaction — keep only the latest record per key
log.cleanup.policy=compact    # useful for changelog topics (state stores)

Log Retention & Compaction

Kafka retains messages on disk regardless of consumer acknowledgement. Retention can be time-based, size-based, or compaction-based, giving full control over data lifetime.

✓Kafka retains messages after consumption — consumers are independent
✓retention.ms and retention.bytes control time/size-based deletion
✓Log compaction keeps only the latest value per key
✓Tombstones (null values) signal key deletion in compacted topics
✓cleanup.policy=compact,delete combines both strategies

Kafka CLI — setting retention

# Set retention on an existing topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter \
  --add-config retention.ms=86400000,retention.bytes=1073741824

Schema Registry & Avro

Schema Registry centralises Avro/JSON/Protobuf schemas so producers and consumers share a contract. It prevents breaking changes from silently corrupting downstream consumers.

✓Schema Registry stores schemas centrally; each message embeds only a 5-byte schema ID
✓BACKWARD: consumers with old schema can still read new messages
✓Adding a field with a default is BACKWARD compatible; removing a required field is not
✓Subject naming: {topic}-key and {topic}-value
✓Avro is compact binary; Protobuf has better language support; JSON is human-readable

Spring Boot — Avro + Schema Registry

# application.properties
spring.kafka.producer.value-serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
spring.kafka.producer.properties.schema.registry.url=http://schema-registry:8081
spring.kafka.consumer.value-deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
spring.kafka.consumer.properties.schema.registry.url=http://schema-registry:8081
spring.kafka.consumer.properties.specific.avro.reader=true

Topic Configuration Properties

Kafka topics have configuration properties controlling durability, retention, compaction, segment size, and message size. Knowing the key properties is essential for production tuning.

✓retention.ms / retention.bytes — how long / how much data to keep
✓min.insync.replicas — minimum replicas that must confirm for acks=all
✓max.message.bytes — maximum single message size (default 1 MB)
✓cleanup.policy=delete|compact|compact,delete
✓segment.bytes controls when a new log segment file is created

Kafka CLI — topic creation with key configs

kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic payments \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000      \ # 7 days
  --config retention.bytes=10737418240 \ # 10 GB
  --config cleanup.policy=delete       \
  --config min.insync.replicas=2       \
  --config max.message.bytes=1048576   \ # 1 MB
  --config compression.type=lz4

Learn this free with Aria, your AI tutor → AiCanCode.org/learn/kafka

Cheat Sheet

Fundamentals

Apache Kafka

11 topics · swipe →

AiCanCode.org

Fundamentals

Apache Kafka Introduction

✓Kafka is a distributed, durable, ordered commit log — not a traditional queue.
✓Topics are split into partitions; each partition is an ordered, immutable sequence.
✓Consumer groups enable parallel processing; one partition → one consumer per group.
✓Offsets track each consumer group's position — consumers control when they commit.
✓Records are retained for a configurable period (default 7 days) regardless of consumption.

// Producer: send a message
ProducerRecord<String, String> record = new ProducerRecord<>(
    "orders",        // topic
    "order-123",     // key  → determines partition
    "{"amount":99}" // value
);
producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.printf("Sent to %s[%d] at offset %d%n",
…

AiCanCode.org

Fundamentals

Kafka vs Traditional Messaging

✓Kafka is a durable, append-only log — messages are NOT deleted after consumption.
✓Consumers pull at their own pace and maintain independent offsets.
✓Multiple consumer groups can independently read the same topic (fan-out without duplication).
✓RabbitMQ is push-based with per-message delivery guarantees and complex routing (exchanges).
✓Choose Kafka for high-throughput, replayable event streams; RabbitMQ for task queues and complex routing.

// Traditional MQ (RabbitMQ) — push model
// Broker decides delivery; message deleted after ack
Producer → Exchange → Queue → Consumer (message gone after ack)

// Kafka — pull model / log
// Consumer reads at its own pace; message persisted on disk
Producer → Topic (partition 0) → log: [offset 0][offset 1][offset 2]...
                                        ↑ consumer-group-A at offset 2
                                        ↑ consumer-group-B at offset 0  (replaying from start)
…

AiCanCode.org

Fundamentals

Topics & Partitions

✓A topic is a named, durable, append-only log; partitions are its physical subdivisions, each stored on a single broker leader.
✓Ordering is guaranteed only within a partition — use message keys to ensure all events for the same entity land in the same partition.
✓The number of partitions is the maximum degree of parallelism for a consumer group; you cannot have more active consumers than partitions.
✓Kafka retains records regardless of consumption — multiple consumer groups can read the same topic independently at their own offsets.
✓Each record in a partition has a unique, monotonically increasing offset; there are no global offsets across partitions.

// Create a topic programmatically using AdminClient
Properties adminProps = new Properties();
adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

try (AdminClient admin = AdminClient.create(adminProps)) {
    int numPartitions = 12;
    short replicationFactor = 3;

    NewTopic ordersTopic = new NewTopic("orders", numPartitions, replicationFactor);
…

AiCanCode.org

Fundamentals

Brokers & Clusters

✓A broker is a Kafka server; a cluster is a group of cooperating brokers.
✓Each partition has one leader broker and N−1 follower replicas.
✓Only the leader handles reads and writes; followers replicate asynchronously.
✓ISR (in-sync replicas) are followers within replica.lag.time.max.ms of the leader.
✓acks=all + min.insync.replicas=2 is the standard durability configuration.

# View cluster metadata with kafka-topics.sh
kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

# Output (example):
# Topic: orders  Partitions: 3  ReplicationFactor: 3
# Partition: 0  Leader: 2  Replicas: 2,0,1  Isr: 2,0,1
…

AiCanCode.org

Fundamentals

ZooKeeper & KRaft Mode

✓ZooKeeper stored Kafka metadata externally — a separate cluster to manage and monitor.
✓KRaft replaces ZooKeeper with an internal Raft quorum stored in __cluster_metadata topic.
✓KRaft is GA since Kafka 3.3; ZooKeeper mode is removed in Kafka 4.0.
✓KRaft supports millions of partitions vs ~200k with ZooKeeper.
✓Controller failover with KRaft is milliseconds vs 30-120 seconds with ZooKeeper.

# Legacy ZooKeeper-based broker config (server.properties)
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka   # ZK quorum + chroot
broker.id=1
log.dirs=/var/kafka/data

# Start ZooKeeper first
bin/zookeeper-server-start.sh config/zookeeper.properties

# Then start Kafka brokers
…

AiCanCode.org

Fundamentals

Replication & In-Sync Replicas (ISR)

✓replication.factor copies each partition to N brokers for fault tolerance
✓ISR = set of replicas caught up within replica.lag.time.max.ms of the leader
✓Leader election on failure picks only from the current ISR (unless unclean election enabled)
✓acks=all ensures all ISR replicas acknowledge; combine with min.insync.replicas=2 for RF=3
✓unclean.leader.election.enable=true risks data loss but prevents partition unavailability

# Create a topic with replication factor 3
kafka-topics.sh --create   --topic orders   --partitions 6   --replication-factor 3   --bootstrap-server broker1:9092

# Describe topic — see leaders, replicas, ISR per partition
kafka-topics.sh --describe --topic orders --bootstrap-server broker1:9092

# Output:
# Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
…

AiCanCode.org

Fundamentals

Leaders & Followers

✓Every partition has exactly one leader; all produces and consumes go through the leader.
✓Followers replicate from the leader; ISR members have replicated all messages within lag threshold.
✓On leader failure, the controller promotes the first ISR member as the new leader.
✓unclean.leader.election.enable=false (default) prevents data loss by rejecting out-of-sync leaders.
✓acks=all + min.insync.replicas=2 guarantees writes survive one broker failure.

# Inspect leader and ISR per partition
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

# Sample output:
# Topic: orders  Partitions: 3  ReplicationFactor: 3
# Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
# Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2
#
…

AiCanCode.org

Fundamentals

Offsets & Log Structure

✓Offset is a monotonically increasing integer identifying a message's position in a partition.
✓The partition log is append-only and immutable; old segments are deleted or compacted.
✓Consumers store committed offsets in __consumer_offsets; committing means "processed up to here".
✓auto.offset.reset=earliest replays from the beginning; latest skips existing messages.
✓At-least-once is the practical default: commit after processing, deduplicate downstream.

# On-disk layout of a partition (orders-0)
/var/kafka/data/orders-0/
  00000000000000000000.log       # messages at offsets 0–999999
  00000000000000000000.index     # offset → physical file position
  00000000000000000000.timeindex # timestamp → offset lookup
  00000000000001000000.log       # messages at offsets 1000000–1999999
  00000000000001000000.index
  00000000000002000000.log       # active (current) segment

…

AiCanCode.org

Fundamentals

Log Retention & Compaction

✓Kafka retains messages after consumption — consumers are independent
✓retention.ms and retention.bytes control time/size-based deletion
✓Log compaction keeps only the latest value per key
✓Tombstones (null values) signal key deletion in compacted topics
✓cleanup.policy=compact,delete combines both strategies

# Set retention on an existing topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter \
  --add-config retention.ms=86400000,retention.bytes=1073741824

AiCanCode.org

Fundamentals

Schema Registry & Avro

✓Schema Registry stores schemas centrally; each message embeds only a 5-byte schema ID
✓BACKWARD: consumers with old schema can still read new messages
✓Adding a field with a default is BACKWARD compatible; removing a required field is not
✓Subject naming: {topic}-key and {topic}-value
✓Avro is compact binary; Protobuf has better language support; JSON is human-readable

# application.properties
spring.kafka.producer.value-serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
spring.kafka.producer.properties.schema.registry.url=http://schema-registry:8081
spring.kafka.consumer.value-deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
spring.kafka.consumer.properties.schema.registry.url=http://schema-registry:8081
spring.kafka.consumer.properties.specific.avro.reader=true

AiCanCode.org

Fundamentals

Topic Configuration Properties

✓retention.ms / retention.bytes — how long / how much data to keep
✓min.insync.replicas — minimum replicas that must confirm for acks=all
✓max.message.bytes — maximum single message size (default 1 MB)
✓cleanup.policy=delete|compact|compact,delete
✓segment.bytes controls when a new log segment file is created

kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic payments \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000      \ # 7 days
  --config retention.bytes=10737418240 \ # 10 GB
  --config cleanup.policy=delete       \
  --config min.insync.replicas=2       \
…

AiCanCode.org

Master Fundamentals

Full lessons + an AI tutor, free.

AiCanCode.org

/learn/kafka