Skip to content
notes.vinhnq.com

Search

System DesignSenior20 min read

Inside the Kafka producer: send(), batching, the partitioner and transactions

Review 17 cards
On this page

Kafka internals series: 1. Inside the producer · 2. Serialization and Schema Registry · 3. Consumer groups, rebalancing and liveness

The post on why Kafka is fast and durable covered acks, ISR, batching and the idempotent producer as ideas. This one opens up the Java producer: what send() really does, the numbers you get when you tune linger.ms and compression, how the partitioner picks a partition, and how Kafka transactions work underneath. Afterwards you can explain why send() can hang for 60 seconds and why one open transaction freezes read_committed consumers.

1. The path of send()

Key idea: send() only puts the record into an in-memory queue and returns a Future. The network work happens later on the sender thread, one batch per partition.

flowchart LR
  A["send(record)"] --> B["Key/value serializer"]
  B --> C["Partitioner"]
  C --> D["RecordAccumulator<br/>batches per partition"]
  D --> E["Sender thread<br/>groups batches per broker"]
  E --> F["Leader broker"]
  F -- "ack" --> E
  E -- "callback / Future" --> A

Step by step, on your thread:

  1. Wait for the topic’s metadata (which partitions, where the leaders are) if it isn’t cached yet.
  2. Serialize key and value to bytes (part 2 of this series).
  3. The partitioner picks a partition (section 3).
  4. Append the record to that partition’s open batch in the RecordAccumulator. A batch becomes ready when it reaches batch.size or has waited linger.ms.

The sender thread groups ready batches by broker: one request carries batches for many partitions led by the same broker. Each connection allows up to max.in.flight.requests.per.connection (default 5) unacknowledged requests.

Settings around the accumulator and the timeouts of one send
Setting Default Meaning
buffer.memory 32 MB Total RAM for unsent batches, shared by all partitions
max.block.ms 60,000 Longest send() (and partitionsFor, initTransactions…) may block
batch.size 16,384 bytes Maximum size of one batch for one partition
linger.ms 5 (since Kafka 4.0, 0 before) Extra wait so batches fill up
max.request.size 1 MB Largest request (and record); bigger records are rejected on the client
request.timeout.ms 30,000 Wait for one request’s response
delivery.timeout.ms 120,000 Total time from send() to success or failure, including time in the accumulator and all retries

A record bigger than batch.size can still be sent (alone in its own batch) as long as it is under max.request.size and the broker’s message.max.bytes.

Metrics worth watching (producer-metrics): record-queue-time-avg (how long records wait in the accumulator), batch-size-avg, records-per-request-avg, compression-rate-avg, buffer-available-bytes, record-error-rate, request-latency-avg.

2. Batching and compression: real numbers

Key idea: linger.ms only matters when load is below what the producer can send; then a few milliseconds of waiting cut requests by an order of magnitude. Compression works on the whole batch, so fuller batches compress better.

I measured on Kafka 4.3.2 (a single KRaft broker on my machine, Java client 4.1.1, JDK 21, acks=all). Each record is a 179-byte order JSON, sent at a steady 5,000 records/second for 3 seconds into a 3-partition topic:

linger.ms Requests Records per request Average batch Time to ack (average)
0 14,333 1.0 263 B 0.3 ms
5 554 27.3 1,828 B 3.0 ms
20 147 104.9 6,856 B 10.8 ms
20 + lz4 147 104.9 1,698 B 10.7 ms

Reading it: linger.ms=0 sends roughly one record per request. Adding 5 ms cuts requests 26 times, at the cost of about 3 ms per record. That is why Kafka 4.0 changed the default from 0 to 5 (KIP-1030).

When writing flat out (200,000 records), batches fill before linger.ms expires, so linger stops mattering and the codec decides:

Codec Compression rate (compression-rate-avg) Records per request
none 1.00 240
lz4 0.21 264
snappy 0.19 725
zstd 0.09 840
gzip 0.10 1,342

batch.size applies to the compressed (estimated) data, so a stronger codec packs more records into a 16 KB batch. On repetitive JSON, zstd and gzip shrink data to about a tenth; lz4 and snappy to about a fifth.

Why the flat-out table isn’t for comparing latency, and how to measure yourself

“Time to ack” in the flat-out run depends on queueing in the accumulator, not on the codec: 200,000 records are send()-ed almost at once, and the last one waits behind hundreds of batches. I only use that table for compression rate and records per request. To compare latency, send at a fixed rate (like the 5,000 msg/s table) and measure percentiles (p99), not averages.

The built-in tool: kafka-producer-perf-test.sh --topic t --num-records 1000000 --record-size 200 --throughput 5000 --producer-props linger.ms=5 compression.type=lz4 prints throughput and p50/p95/p99 latency. Use your real payload (--payload-file), because random data barely compresses and makes every codec look the same.

Every number in this post comes from one broker on a laptop over loopback. On a real cluster with 1–2 ms RTT between machines, batching helps even more, because each saved request is a saved network round trip.

3. The partitioner: how a partition is chosen

Key idea: the order is: partition set on the record → hash of the key (murmur2) → no key, stick to one partition for about a batch, then switch. Since Kafka 3.3 the keyless path also avoids slow brokers.

The keyed path (murmur2, and keys moving when the partition count changes) is in the post on why Kafka is fast. The keyless path went through three generations:

Era Choice without a key Effect
Before 2.4 Round-robin per record A small batch per partition, many requests, higher latency
2.4 (KIP-480) Sticky: stay on one partition until the batch is full, then pick another at random Bigger batches, fewer requests
3.3 (KIP-794) Uniform sticky: switch after about batch.size bytes to that partition; favour brokers with short queues More even spread, avoids slow brokers

Measured with 30,000 keyless records into 3 partitions: a spread of [11,666, 8,969, 9,365] with 94 partition switches (316 consecutive records per partition on average). “Even” here means even over time, not even per record.

Adaptive partitioning and a custom partitioner for hot keys

Uniform sticky tracks each broker’s queue length and picks the next partition weighted toward less busy brokers (partitioner.adaptive.partitioning.enable=true, the default). Set partitioner.availability.timeout.ms > 0 and partitions whose broker hasn’t accepted a batch within that time are skipped temporarily. This only applies to keyless records; keyed records must always go to their own partition.

A hot key (one key carrying most of the traffic, like a big merchant) overloads one partition. Two common fixes:

  • Salt the key: merchantId + "#" + (orderId % 4) spreads one key over 4 partitions. You lose ordering across that merchant’s orders; only do it when the business allows.
  • A custom partitioner: reserve a few partitions for big tenants and hash the rest as usual.
public class TenantPartitioner implements Partitioner {
  private static final Set<String> BIG = Set.of("merchant-42");
  public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    int n = cluster.partitionCountForTopic(topic);
    String k = (String) key;
    if (BIG.contains(k)) return n - 1;                       // dedicated partition for the big tenant
    return Utils.toPositive(Utils.murmur2(keyBytes)) % (n - 1); // everyone else
  }
  public void close() {}
  public void configure(Map<String, ?> configs) {}
}

I haven’t run this code; it shows the shape of the kafka-clients Partitioner interface.

4. Retries: which errors, and for how long

Key idea: the producer retries transient errors (leader moved, flaky network) within delivery.timeout.ms. Non-retriable errors (record too large, bad config, no permission) go straight to the callback.

Kind Examples What the producer does
Retriable NOT_LEADER_OR_FOLLOWER, LEADER_NOT_AVAILABLE, NOT_ENOUGH_REPLICAS, network timeouts Retries after retry.backoff.ms (100 ms, growing up to retry.backoff.max.ms 1 s)
Non-retriable RecordTooLargeException, InvalidTopicException, TopicAuthorizationException, SerializationException Fails immediately, the application must handle it

The slides suggest retries=30. I don’t follow that: retries has defaulted to Integer.MAX_VALUE since Kafka 2.1, and the real limit is time. A retry count depends on backoff and on how long the incident lasts, which is hard to predict. Asking “how long a leader outage do we tolerate?” and setting delivery.timeout.ms to match is simpler.

5. Transactions from the inside

Key idea: a transaction makes writes to several partitions (and an offset commit) succeed or abort together. It rests on three things: a stable transactional.id for fencing, a transaction coordinator that keeps a log, and control markers in each partition so consumers know what was committed.

The post on dual writes, exactly-once and ordering covers when to use transactions (consume-transform-produce) and when they aren’t enough. This section is the mechanism.

sequenceDiagram
  participant P as Producer
  participant TC as Transaction coordinator
  participant B as Partition leaders
  P->>TC: InitProducerId(transactional.id)
  TC-->>P: PID + new epoch (old instance fenced)
  P->>TC: AddPartitionsToTxn(orders-0, payments-1)
  P->>B: Produce batch (PID, epoch, sequence)
  P->>TC: EndTxn(COMMIT)
  TC->>TC: write PREPARE_COMMIT to __transaction_state
  TC->>B: WriteTxnMarkers(COMMIT)
  TC->>TC: write COMPLETE_COMMIT
  • Coordinator: the broker leading partition hash(transactional.id) % 50 of the internal __transaction_state topic. It stores each transaction as a state machine, so it survives broker failures.
  • Control marker: a special record (COMMIT or ABORT) written to every participating partition. It takes an offset, and applications never see it.
  • LSO (last stable offset): the first offset that still belongs to an open transaction. read_committed consumers read only up to the LSO; read_uncommitted consumers (the default) read up to the high watermark and see aborted records too.

Lab on Kafka 4.3.2, partition 0 of topic pl-tx: commit A1, A2; abort B1, B2; commit C1; open a transaction with D1 and leave it.

Consumer Reads Stops at
read_uncommitted 0:A1, 1:A2, 3:B1, 4:B2, 6:C1, 8:D1 9
read_committed 0:A1, 1:A2, 6:C1 8 (LSO, stuck before D1)
read_committed after a new instance calls initTransactions() 0:A1, 1:A2, 6:C1 10 (D1 aborted)

Offsets 2, 5, 7 and 9 are control markers: those are the offset “gaps” the Core Concepts slides mention. A new instance with the same transactional.id took 216 ms in initTransactions() and aborted the old instance’s open transaction. The old instance’s next send got InvalidProducerEpochException: it had been fenced (blocked because its epoch is stale).

Plain Java lab code and the Spring Kafka equivalent

The core of the lab (full code in StickyTxLab.java):

Properties p = new Properties();
p.put("bootstrap.servers", "localhost:39092");
p.put("key.serializer", StringSerializer.class.getName());
p.put("value.serializer", StringSerializer.class.getName());
p.put("transactional.id", "lab-tx-1");          // turns on idempotence, acks=all

var prod = new KafkaProducer<String, String>(p);
prod.initTransactions();                        // get PID + epoch, fence the old instance
prod.beginTransaction();
prod.send(new ProducerRecord<>("pl-tx", 0, "k", "A1"));
prod.send(new ProducerRecord<>("pl-tx", 0, "k", "A2"));
prod.commitTransaction();

prod.beginTransaction();
prod.send(new ProducerRecord<>("pl-tx", 0, "k", "B1-aborted"));
prod.flush();
prod.abortTransaction();

Error handling per the KafkaProducer docs: ProducerFencedException, OutOfOrderSequenceException and AuthorizationException are fatal, so close() the producer. For other KafkaExceptions, abortTransaction() and retry the whole transaction.

Spring Boot 3 (not run for this post):

spring:
  kafka:
    producer:
      transaction-id-prefix: payment-tx-   # enables transactions on KafkaTemplate
    consumer:
      isolation-level: read_committed
      enable-auto-commit: false
@Transactional("kafkaTransactionManager")   // or kafkaTemplate.executeInTransaction(t -> ...)
public void publish(OrderPaid e) {
  kafkaTemplate.send("orders", e.orderId(), e);
  kafkaTemplate.send("ledger", e.accountId(), toLedger(e));
}

With transaction-id-prefix, Spring Boot creates a KafkaTransactionManager; a listener container using it sends the offset of the record being processed into the same transaction (consume-transform-produce). Note: JPA’s @Transactional and Kafka’s are two different transactions; nesting them doesn’t make the DB and Kafka atomic (see the dual-write post above).

Exercises

Exercise 1: send() takes down the API

An order service calls kafkaTemplate.send() in a controller. During 2 minutes of Kafka maintenance, every order API times out, including endpoints that don’t use Kafka. Explain why and propose configuration and design changes.

Solution to Exercise 1

Why: send() blocks when it can’t get metadata or when buffer.memory is full, for up to max.block.ms = 60 s. Tomcat’s threads (200 by default) get stuck in send(); once they’re gone, every endpoint waits, related or not.

Short-term fixes:

  • max.block.ms=1000 for producers serving synchronous requests. On timeout return 503 with Retry-After instead of holding a thread.
  • delivery.timeout.ms sized to the SLA (say 30 s) and a callback that records failures in metrics.
  • A bulkhead: limit concurrent requests into the order flow (semaphore or a dedicated pool) so other flows survive.

Proper fix: a transactional outbox. The controller writes the order and an outbox row in one DB transaction; a relay pushes to Kafka later. During Kafka maintenance the outbox backs up and the API keeps working. It also removes the dual write.

Exercise 2: Reading offsets with gaps

A partition is written only by transactional producers. A read_committed consumer sees offsets 0, 1, 3, 4, 8. Explain the missing offsets, and why endOffset - committedOffset (lag computed the usual way) never reaches 0 even after the consumer has read everything.

Solution to Exercise 2
  • Offsets 2, 5, 6, 7 aren’t committed application records. One consistent reading: 0, 1 belong to T1, 2 = T1’s COMMIT marker; 3, 4 belong to T2, 5 = T2’s COMMIT marker; 6 is a record of aborted T3, 7 = T3’s ABORT marker; 8 belongs to T4. The numbers alone can’t prove it; kafka-dump-log.sh --print-data-log prints isControl: true and the marker type.
  • Lag doesn’t reach 0: there is always a marker after the last record. Having read everything, the position is the offset after the marker, but a consumer sometimes commits the offset right after the last record (before the marker), so lag shows 1. And if a transaction is still open, a read_committed consumer only reaches the LSO, so lag measured against the high watermark stays above 0.
  • Conclusion: on transactional topics, lag alerts must tolerate a small offset (1–2) and should also track LastStableOffsetLag; never write business logic that assumes “contiguous offsets”.

Exercise 3: Choosing linger.ms

Two producers: (a) access logs at 50,000 records/s that nobody reads right away; (b) money-transfer commands at 200 records/s with an end-to-end p99 SLA of 50 ms. Pick linger.ms, batch.size and a codec, and explain.

Solution to Exercise 3
  • (a) Access logs: linger.ms=20–50, batch.size=64–256 KB, compression.type=zstd, acks=1 or all depending on how much loss you accept. At 50,000 records/s batches fill on their own, so linger costs little latency; zstd cuts bandwidth and disk 5–10 times.
  • (b) Transfers: 200 records/s is low load: with linger.ms=5 a batch holds about 1 record per partition, so linger mostly adds 5 ms without batching anything. Keep linger.ms at 0–5 (the default 5 still fits a 50 ms SLA), default batch.size, lz4 or no compression (small records, little gain), acks=all, idempotence on, key = accountId. Measure the real p99 before tuning.

Remember

  • Can you draw send() → serializer → partitioner → accumulator → sender thread, and point to the two places where send() blocks?
  • Can you explain why linger.ms only matters under low load, with the 14,333 → 554 requests measurement?
  • Do you know how the partitioner chooses for keyless records since Kafka 3.3, and why not to set the old partitioner.class?
  • Do you bound retries with delivery.timeout.ms rather than retries?
  • Can you walk through a transaction (InitProducerId, AddPartitions, EndTxn, markers) and explain why an open transaction freezes read_committed consumers?

Interview questions

Junior

Is KafkaProducer.send() synchronous?

No: it puts the record in a buffer and returns a Future immediately; a background thread sends batches. But it can block when metadata is missing or the buffer is full.

To wait for the result, use send().get() (slow, defeats batching) or a callback. Spring’s KafkaTemplate.send() returns a CompletableFuture; send errors live in the future, so a try/catch around the call doesn’t catch them.

Which partition does a keyless record go to?

The producer sticks to one partition until it has sent about a batch, then moves to another (sticky), so the spread is even over time but bursty over short windows.

Before Kafka 2.4 it was round-robin per record. Since 3.3 the next partition is also weighted toward less loaded brokers.

Mid

How do linger.ms and batch.size interact?

A batch is sent when it reaches batch.size or has waited linger.ms, whichever comes first.

Under high load batches fill first and linger costs almost no latency. Under low load linger decides; my lab at 5,000 msg/s took 14,333 requests with linger 0 and 554 with linger 5. The default linger is 5 ms since Kafka 4.0.

What do you set retries to on a producer?

Leave the default (Integer.MAX_VALUE) and bound it with delivery.timeout.ms, because what you control is how long you tolerate an incident, not how many attempts.

Alongside: idempotence on so retries don’t duplicate or reorder, and a callback that handles non-retriable errors (record too large, no permission) and the TimeoutException on expiry.

Why shouldn’t the topic use a different compression codec from the producer?

The broker would have to decompress and recompress every batch, burning CPU and losing zero-copy; with compression.type=producer the broker stores batches as they are.

Pick the codec on the producer: lz4 for latency, zstd for ratio. Consumers decompress using the codec recorded in each batch, no config needed.

Senior

Explain how Kafka transactions work underneath.

A producer with a transactional.id gets a PID and epoch from the coordinator; the coordinator logs state in __transaction_state; on commit it writes COMMIT markers into every participating partition, and read_committed consumers read only up to the LSO.

  • initTransactions() bumps the epoch, fences the old instance and completes any open transaction.
  • Each new partition in the transaction is registered with the coordinator; consumer offsets go through sendOffsetsToTransaction.
  • Commit is two-phase: PREPARE_COMMIT in the coordinator’s log (the point of no return), then the markers.
  • Cost: extra round trips, markers take offsets, and open transactions block read_committed consumers.
What is a zombie producer and how does Kafka stop it?

A zombie is an old instance that thinks it’s still alive (say after a long GC pause) while a new one has replaced it; Kafka stops it with an epoch tied to the transactional.id.

The new instance’s initTransactions() bumps the epoch; every request from the old instance carries a lower epoch and is rejected (ProducerFencedException / InvalidProducerEpochException). So transactional.id must be stable per role, not random.

Consumer lag is growing but the consumer reports no errors, and the topic is transactional. What do you suspect?

An open (or hung) transaction is holding the LSO, so read_committed consumers can’t move on even though new data exists.

Check: compare the LSO with the high watermark (LastStableOffsetLag), run kafka-transactions.sh find-hanging --broker-id N. Fix: find the producer holding the transaction, lower transaction.timeout.ms, abort the hung transaction with the tool.

send() hangs for 60 seconds when the cluster is down. How do you redesign?

Lower max.block.ms on synchronous paths, isolate with a bulkhead, and ideally decouple the Kafka write from the request with an outbox.

send() blocks on metadata or a full buffer.memory. Watch buffer-available-bytes and record-queue-time-avg to see it coming; an outbox turns a Kafka incident into delay instead of API errors.

Useful phrases for interviews

  • “send() only appends to the record accumulator; a background sender thread does the network I/O.”
  • “send() can still block, on missing metadata or a full buffer, for up to max.block.ms.”
  • “In my test, five milliseconds of linger cut the request count from about fourteen thousand to five hundred.”
  • “Batch size is measured after compression, so a stronger codec packs more records per request.”
  • “Keyless records stick to one partition for about a batch, then the partitioner moves on.”
  • “I don’t tune the retry count; I bound retries with delivery timeout.”
  • “Each transaction leaves a control marker in every partition it touched, which is why offsets have gaps.”
  • “Read-committed consumers stop at the last stable offset, so one hung transaction freezes them.”
  • “A stable transactional id lets the new instance fence the zombie.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.