Skip to content
notes.vinhnq.com

Search

System DesignSenior24 min read

Why Kafka is fast and durable: the log, zero-copy, batching and replication

Review 21 cards
On this page

Kafka and message brokers series: 1. Message brokers: four problems with request-response and how to choose a broker · 2. Why Kafka is fast and durable: the log, zero-copy, batching and replication · 3. Tuning Kafka for your goal: throughput, latency, durability, availability · 4. Case study: a synchronous fraud check over Kafka and Redis — good idea or not? · 5. CCDAK review: 40 Kafka questions worth remembering

“Kafka is fast” means very high throughput (hundreds of MB/s per broker), not the lowest latency. This post explains the design decisions behind that number, then fills in what the slides left empty: why Kafka is durable (replication, ISR, acks, min.insync.replicas). Read post 1 on message brokers first.

The experiments here ran on a 3-broker KRaft cluster (Confluent Platform 7.5, i.e. Kafka 3.5) with the Java client kafka-clients 4.1.

1. Kafka in one picture

Key idea: a topic is split into partitions; each partition is an append-only log copied to several brokers. A consumer only has to remember which offset it has read up to.

  • Partition: an append-only log (writes only go at the end) belonging to a topic (a named stream of events). Each record gets an increasing offset (its position). Records are never modified and aren’t deleted when read; they’re only cleaned up by retention (time, size) or compaction (keep the latest record per key).
  • Replication: each partition has copies on several brokers. One copy is the leader (takes writes, serves reads), the others are followers (they fetch from the leader).
  • The producer picks a record’s partition; consumers in the same consumer group share the partitions and track their own offsets.
  • Metadata (topics, partitions, leaders, ISR) is managed by a controller quorum running Raft (KRaft). Kafka 4.0 removed ZooKeeper entirely.
flowchart LR
  subgraph T["Topic payments, 3 partitions, replication factor 3"]
    P0["P0: leader B1, followers B2 B3"]
    P1["P1: leader B2, followers B3 B1"]
    P2["P2: leader B3, followers B1 B2"]
  end
  Prod[Producer] -->|key hash| P0
  Prod --> P1
  Prod --> P2
  P0 --> C1[Consumer 1, group G]
  P1 --> C1
  P2 --> C2[Consumer 2, group G]

2. A correction: Kafka optimises throughput, not latency

Key idea: Kafka doesn’t make each message fly through; it trades a little latency for throughput. If you need sub-millisecond p99 per message, Kafka isn’t the natural choice.

Typical end-to-end latency is a few ms to a few tens of ms, fine for most systems. The design decisions fall into three groups: broker side, client side, and parallelism through partitions.

3. Broker side

Key idea: the broker does as little as possible: it appends sequentially, sends bytes straight from the page cache to the network, doesn’t fsync and doesn’t delete on read.

3.1. Sequential I/O on an append-only log

“Disks are slow” is only true for random access. Sequential reads and writes are much faster, and the OS optimises them further (read-ahead: fetching the next chunk early; coalescing several writes into one).

The slides say 3–4× faster; in practice HDDs can differ by hundreds of times (no head seeks), SSDs less but still significantly.

Kafka only appends and reads sequentially from an offset: no B-tree, no in-place updates.

Inside a partition: segments, indexes and retention

Each partition is a directory of segments. Only the active segment is appended to; when it’s big enough (segment.bytes, 1 GB by default) or old enough, it’s rolled and a new one starts. Each segment has three files (as seen inside the lab container):

/var/lib/kafka/data/payments-3/
  00000000000000000000.log        # record data, append-only
  00000000000000000000.index      # offset -> byte position (sparse index)
  00000000000000000000.timeindex  # timestamp -> offset
  • Write: append to the end of .log.
  • Read: look up the byte position in .index (a sparse index, binary-searched), then read sequentially.
  • Retention: delete whole old segment files, not individual records.

3.2. Zero-copy

Sending a file the usual way costs four copies and four context switches: disk → page cache (kernel) → application buffer (user space) → socket buffer (kernel) → NIC. Kafka uses sendfile (in Java, FileChannel.transferTo): data goes straight from the page cache to the NIC without passing through user space.

flowchart LR
  subgraph Usual
    D1[Disk] --> K1[Page cache] --> U1[App buffer] --> S1[Socket buffer] --> N1[NIC]
  end
  subgraph Zero-copy
    D2[Disk] --> K2[Page cache] --> N2[NIC]
  end

This only works because Kafka stores batches on disk in exactly the format it sends over the wire: the broker doesn’t decode or rebuild records.

Zero-copy in Java, and why the broker wants a large page cache

3.3. Writing to the page cache, without fsync

The most surprising decision: the broker acknowledges once the record is in the page cache, without calling fsync. The page cache is RAM the OS uses to buffer files; fsync forces it to disk right now. The OS flushes later on its own; log.flush.interval.messages and log.flush.interval.ms default to “never flush explicitly”.

sequenceDiagram
  participant P as Producer
  participant B as Broker (leader)
  participant OS as Page cache
  participant D as Disk
  P->>B: produce batch
  B->>OS: write() into page cache
  B-->>P: ACK
  OS--)D: flush later (by the OS)

With a single broker this is unsafe: lose power before the flush and you lose acknowledged data. Kafka gets durability from replication: a record only counts as committed once it’s in the memory of several brokers. Three machines in three racks losing power at once is far rarer (see section 6).

Should you force fsync? And how the page cache behaves

How the page cache works (“free” RAM isn’t wasted; page cache is charged to the container, so a broker’s container_memory_usage_bytes creeps up without a leak): see Virtual memory, page cache and the JVM. The path a write() takes to disk, how far apart sequential and random I/O are, and what an fsync costs: see I/O models: blocking, epoll and file descriptors.

3.4. Cheap consumers: keep messages, store only offsets

Traditional queues delete messages once consumers ack them: the I/O jumps around (random I/O) and the broker tracks per-message state.

Kafka doesn’t delete on read. Each consumer group stores one number per partition: the next offset to read, written to the internal topic __consumer_offsets (a compacted topic). As a result:

  • An extra consumer group is almost free: ten groups reading the same topic are still sequential reads, usually served from the page cache.
  • Consumers can rewind (seek, or kafka-consumer-groups --reset-offsets) to reprocess.
  • A slow consumer doesn’t slow the broker down: the data stays until retention removes it.

4. Client side

Key idea: the bottleneck is usually the network and the request count, not the disk. Clients batch, compress whole batches and pick partitions themselves, so the broker just appends.

4.1. Batching

10,000 records sent one by one are 10,000 round trips; in batches of 500 they’re 20 requests.

The producer groups records per partition into batches (batch.size, 16 KB by default) and waits up to linger.ms for them to fill. Kafka 4.0 changed the linger.ms default from 0 to 5 ms (KIP-1030): waiting a few ms gives better throughput at almost no latency cost. Consumers fetch in batches too (fetch.min.bytes, max.poll.records). Measured linger.ms and compression numbers on Kafka 4.3 are in the producer internals post.

4.2. Batch compression

Compressing small records one by one compresses badly; compressing a whole batch of structurally similar JSON records compresses very well (often 3–10×). Batches are compressed at the producer, stored compressed on the broker and decompressed at the consumer. You win three times: less network bandwidth, less disk space, less disk I/O.

Codecs: lz4 balances speed and ratio, zstd gives the best ratio (more CPU), plus snappy and gzip (slow, rarely used).

4.3. Smart clients, simple brokers

The client batches, compresses, hashes keys to pick partitions, computes checksums, retries, and fetches metadata to send straight to each partition’s leader. The broker only appends and reads sequentially, so it can serve a lot of clients.

How the Java producer picks a partition:

  • With a key: partition = toPositive(murmur2(keyBytes)) % numPartitions. The same key always lands in the same partition, so one account’s events stay in order.
  • Without a key: “sticky partitioning” (KIP-480, refined into uniform sticky by KIP-794 in Kafka 3.3). The producer sticks to one partition until the batch is full, then moves on, giving bigger batches than spraying records round-robin.
The murmur2 experiment, and what adding partitions does

I reimplemented murmur2 and compared it with the partitions the broker actually returned for 6 keys on a 6-partition topic: all 6 matched, and resending the same key always hit the same partition. The key acc-1 lands in partition 3 with 6 partitions, but recomputing with 8 partitions gives 5.

5. Parallelism: partitions and consumer groups

Key idea: the partition is the unit of parallelism. Within a group each partition belongs to one consumer, so useful consumers top out at the partition count.

Partitions spread write load across brokers, allow horizontal scaling (add brokers, move some partitions over), and are the unit of replication.

Consumers sharing a group.id split the partitions: each partition is assigned to exactly one consumer in the group at a time; one consumer can own several partitions. The slides’ example: consumer 1 gets P0 and P1, consumer 2 gets P2 and P3, consumer 3 gets P4.

flowchart LR
  Pr1[Producer 1] --> P0 & P1 & P2
  Pr2[Producer 2] --> P3 & P4
  subgraph G[Consumer group G]
    C1[Consumer 1]
    C2[Consumer 2]
    C3[Consumer 3]
  end
  P0 --> C1
  P1 --> C1
  P2 --> C2
  P3 --> C2
  P4 --> C3

A rebalance is the group redistributing partitions. It happens when a consumer joins, leaves or dies (no heartbeat within session.timeout.ms, or no poll within max.poll.interval.ms). Remember:

  • A sixth consumer in a group reading a 5-partition topic sits idle.
  • Ordering is only guaranteed within a partition; there’s no global order. If one account’s events must be ordered, use the account id as the key.

6. Why Kafka is durable: replication, ISR, acks

Key idea: a record is only acknowledged once enough replicas have it. Durability is the combination of replication.factor=3, min.insync.replicas=2 and acks=all; drop any one and there’s a hole.

The slides promise a “Why is Kafka reliable?” section but stop after parallelism; what follows fills it in from the Kafka documentation.

6.1. Leaders, followers and the ISR

Each partition has replication.factor copies (usually 3). Producers write only to the leader. Followers keep sending fetch requests to the leader for new records, exactly like a consumer.

  • ISR (in-sync replicas): the replicas (leader included) that have caught up within replica.lag.time.max.ms (30 s by default). A follower that falls behind is dropped, and rejoins once it catches up.
  • High watermark: the highest offset every ISR replica has. Consumers only read up to it, so they never see a record that could disappear on a leader change.
sequenceDiagram
  participant P as Producer (acks=all)
  participant L as Leader B1
  participant F1 as Follower B2
  participant F2 as Follower B3
  P->>L: produce offset 100
  L->>L: append to page cache
  F1->>L: fetch from 100
  F2->>L: fetch from 100
  L-->>F1: record 100
  L-->>F2: record 100
  F1->>L: fetch from 101 (confirms it has 100)
  F2->>L: fetch from 101
  L->>L: high watermark = 101
  L-->>P: ACK

6.2. acks and min.insync.replicas

The producer’s acks decides when the leader acknowledges:

acks Leader acks when Data is lost when
0 Never waits (the producer doesn’t expect a response) Any network or broker failure
1 The leader has written it to its own log The leader dies before followers copy it
all (-1) Every replica in the current ISR has it Only if every ISR replica is lost together

While it waits for followers, an acks=all request sits in the broker’s “purgatory”, timed by a hierarchical timing wheel (how a timing wheel works).

The hole in acks=all: if the ISR shrinks to just the leader, “every replica in the ISR” is one machine, no safer than acks=1. min.insync.replicas closes it: with acks=all, if the ISR is below the threshold the leader refuses the write (NotEnoughReplicasException) rather than accepting it and risking loss.

The 3-broker lab with broker 3 stopped:

Topic RF min.insync.replicas ISR left acks=all acks=1
payments 3 2 2 Write succeeds Write succeeds
strict 3 3 2 Fails after delivery.timeout.ms Write succeeds
What the application actually sees when the ISR is too small

6.3. Unclean leader election

When a leader dies, the controller picks a new leader from the ISR, so it’s guaranteed to have every committed record. If every ISR replica is dead and only an out-of-sync (lagging) replica remains, there are two options:

6.4. The idempotent producer: no duplicates, no reordering on retry

Retries cause duplicates (the leader wrote the batch but the ack got lost, so the producer resends) and reordering (several requests in flight, max.in.flight.requests.per.connection > 1: batch 1 fails and is retried after batch 2 succeeded).

enable.idempotence=true (idempotent: doing it many times has the effect of doing it once; the default since Kafka 3.0, together with acks=all) solves both. Each producer gets a producer id, each batch a per-partition sequence number; the broker drops batches whose sequence it has seen and rejects out-of-order ones. Limits: max.in.flight.requests.per.connection ≤ 5, and it only deduplicates within one producer session (a restart gets a new id; for cross-session dedupe use a transactional.id). How transactions work inside (coordinator, control markers, LSO, fencing) is in the producer internals post.

send is asynchronous: it only puts the record in a buffer and returns a Future<RecordMetadata>. Wrapping send in try/catch won’t catch broker errors; they only arrive through the callback or future.get().

Reordering without idempotence, and a sample Java producer for payment events

A producer for payment events, keyed by account id to keep per-account order, handling the outcome in a callback (run against the lab: three records with the same key landed in partition 1, offsets 2, 3 and 4):

var props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:19092,localhost:19093");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.ACKS_CONFIG, "all");                 // wait for every in-sync replica
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);    // retries never duplicate or reorder
props.put(ProducerConfig.LINGER_MS_CONFIG, 10);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 32_768);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 60_000); // must be >= linger.ms + request.timeout.ms

try (var producer = new KafkaProducer<String, String>(props)) {
    var record = new ProducerRecord<>("payments", "acc-42", "{\"type\":\"TOPUP\",\"amount\":200000}");
    producer.send(record, (meta, ex) -> {        // runs on the producer's I/O thread; keep it light
        if (ex != null) log.error("publish failed", ex);   // alert, or let the outbox retry
        else log.info("partition={} offset={}", meta.partition(), meta.offset());
    });
}

Exercises

Exercise 1: How many partitions?

Topic clickstream needs 600 MB/s at peak. One producer writes about 100 MB/s into a single partition, and one consumer processes 30 MB/s. Load is expected to double within 2 years. Pick a partition count.

Solution to Exercise 1

Today: max(600/100, 600/30) = max(6, 20) = 20 partitions, with consumers as the limit. With double the load in 2 years: 1,200/30 = 40. Since you can’t shrink and growing breaks per-key ordering, go with about 40–48 partitions from the start (48 divides by 2, 3, 4, 6, 8 and 12, which makes balancing consumers easy). Start with about 20 consumers and scale towards 48 as needed.

Exercise 2: Losing brokers

The topic has RF=3. For each setting: when one broker dies, can an acks=all producer still write? What about two? Can acknowledged records be lost?

min.insync.replicas=1 min.insync.replicas=2 min.insync.replicas=3
1 broker down
2 brokers down
Solution to Exercise 2
min.insync.replicas=1 min.insync.replicas=2 min.insync.replicas=3
1 broker down Writes succeed Writes succeed Writes fail
2 brokers down Writes succeed, but records live on one machine Writes fail Writes fail

A record acknowledged with acks=all lives on every replica in the ISR at that moment, so it’s only lost if all of them are lost before flushing, or if unclean leader election is enabled. With min ISR 1 after two brokers die, losing the last machine loses data. (With KRaft controllers co-located on the same 3 nodes, losing 2 nodes also loses the controller quorum, so no new leader can be elected; production clusters should run dedicated controllers.)

Exercise 3: Keeping per-account order

A wallet service publishes TOPUP, PAYMENT and REFUND events to a 12-partition topic wallet-transactions. The balance consumer sometimes sees a REFUND before the PAYMENT it refunds. Find likely causes and fixes.

Solution to Exercise 3

Common causes:

  1. The producer sets no key, so records spread over partitions handled in parallel by different consumers.
  2. The key is transaction_id for PAYMENT but refund_id for REFUND, so they land in different partitions.
  3. Idempotence is disabled and retries happen with several in-flight requests.
  4. The consumer processes records of one partition in parallel with a thread pool.

Fix: use the same key for every event that must stay ordered relative to the others, here account_id, since per-account order is what the balance needs; keep enable.idempotence=true; a consumer that wants parallelism within a partition must parallelise by key, not by record.

Remember

  • Explain why “Kafka is fast” is about throughput, and how linger.ms trades latency for it.
  • List the four broker-side reasons (sequential I/O, zero-copy, page cache without fsync, offsets only) and know when zero-copy is disabled.
  • Explain why adding partitions breaks per-key ordering, and how to size partitions up front.
  • Know the three settings that must go together to avoid data loss, and why acks=1 voids min.insync.replicas.
  • Tell the ISR, the high watermark and unclean leader election apart.

Interview questions

Junior

What are topics, partitions and offsets?

A topic is a named stream of records split into partitions; an offset is a record’s position within its partition.

Each partition is an ordered append-only log. A consumer group stores its read offset per partition so it knows where to resume.

How does Kafka guarantee ordering?

Only within a partition; there’s no order across partitions.

Records in one partition are read in the order they were written. If one entity’s events must be ordered, use the entity id as the key, because the same key always goes to the same partition.

Mid

Why does Kafka achieve high throughput even though it writes to disk?

Because everything is sequential and batched: an append-only log, zero-copy, client-side compressed batches, and partitions working in parallel.

  • The append-only log makes reads and writes sequential; writes go to the page cache without fsync.
  • Consumers are served with zero-copy sendfile.
  • Nothing is deleted on read, so there’s no random I/O and only offsets are stored.
  • Clients batch and compress whole batches, so fewer requests and fewer bytes; they pick partitions themselves and send straight to the leader.
  • Partitions allow parallelism across many brokers and consumers.
How do acks=0, 1 and all differ?

They differ in how many copies the leader waits for before acking: none, just itself, or every replica in the ISR.

0: the producer doesn’t wait for a response; fastest, loses data on any failure. 1: acked once the leader has written it; lost if the leader dies before followers copy it. all: waits for every ISR replica; only lost if all of them are lost together, and paired with min.insync.replicas=2 so the ISR can’t shrink to one machine. Since Kafka 3.0 the default is all.

How do consumer groups work, and when do they rebalance?

Consumers with the same group.id share the partitions, each owned by exactly one consumer; the group rebalances when membership or the partition count changes.

Useful consumers top out at the partition count. A rebalance happens when a consumer joins or leaves, misses heartbeats for session.timeout.ms, doesn’t poll within max.poll.interval.ms, or when the topic’s partition count changes.

What goes wrong when you add partitions to a live topic?

Keys move to other partitions, so per-key ordering breaks; and you can’t go back down.

A key’s partition is murmur2(key) % partition count. After adding partitions, old and new records of the same key sit in different partitions, and key-based state stores drift. Over-provision from the start, or create a new topic and migrate.

Senior

Kafka doesn’t fsync before acking, so why is it called durable?

Kafka moves durability from “on one machine’s disk” to “in the memory of several machines”.

With acks=all a record is only acked once every ISR replica has it, and min.insync.replicas=2 guarantees at least two machines. With replicas in different racks/AZs, the chance of all of them losing power before the OS flushes is very low. Forcing an fsync per message wrecks throughput for little extra protection.

What are the ISR and the high watermark?

The ISR is the set of replicas caught up with the leader; the high watermark is the highest offset every ISR replica has.

Caught up means within replica.lag.time.max.ms; laggards are dropped and rejoin when they catch up. Leaders are only elected from the ISR (unless unclean election is enabled). Consumers only read up to the high watermark, so they never read a record that could vanish on a leader change.

What is unclean leader election and when would you enable it?

It lets an out-of-sync replica become leader when every ISR replica is dead: available again at once, but data is lost.

It’s off by default. Enable it for data you can afford to lose (logs, metrics) to favour availability; keep it off for payments and orders, where stopping beats losing data.

How does the idempotent producer work, and what are its limits?

A producer id plus per-partition sequence numbers let the broker drop duplicate batches and reject out-of-order ones, but only within one producer session.

Retries neither duplicate nor reorder, with up to 5 in-flight requests per connection. Limits: one session only, and only inside Kafka. A restarted producer, or an application calling send twice for the same business action, still creates duplicates; you need a transactional.id or an idempotent consumer.

Useful phrases for interviews

  • “Kafka isn’t low-latency by design; it trades a few milliseconds of latency for throughput.”
  • “Each partition is an append-only log, so both reads and writes are sequential.”
  • “The broker acks from the page cache without fsync — durability comes from replication, not from the disk.”
  • “For payment topics I’d use replication factor three, min.insync.replicas two, and acks=all on every producer.”
  • “min.insync.replicas is ignored for acks=1, so it only protects producers that ask for it.”
  • “We key by account id so all events for an account land in one partition and stay ordered.”
  • “Adding partitions remaps keys, so we size the topic for two years of growth up front.”
  • “With TLS, zero-copy is off, so we budget extra broker CPU.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.