Skip to content
notes.vinhnq.com

Search

System DesignSenior22 min read

Tuning Kafka for your goal: throughput, latency, durability, availability

Review 19 cards
On this page

Kafka and message brokers series: 1. Message brokers: four problems with request-response and how to choose a broker · 2. Why Kafka is fast and durable: the log, zero-copy, batching and replication · 3. Tuning Kafka for your goal: throughput, latency, durability, availability · 4. Case study: a synchronous fraud check over Kafka and Redis — good idea or not? · 5. CCDAK review: 40 Kafka questions worth remembering

No set of Kafka settings maxes out throughput, latency, durability and availability at once. This post shows how to rank the four goals, then pick values for the producer, consumer, topics, brokers and Spring Boot 3, with a 3-broker lab to check them. Read post 2 first for ISR, acks and batching.

At a glance:

flowchart TD
  G{What comes first?} -->|Durability| D["acks=all, RF=3, min ISR=2"]
  G -->|Throughput| T["big linger.ms, batch.size, zstd"]
  G -->|Latency| L["linger.ms=0, fetch.min.bytes=1"]
  D -.->|trade-off| X["Broker down: partition stops writes"]
  C[Consumer] --> CM[Commit after processing]
  CM --> ID[Idempotent writes]
  B["Safe defaults on broker"] --> TP[Override per topic]
  TP --> BM["Benchmark, monitor, tune"]

1. Rank your goals before touching any setting

Key idea: the four goals pull settings in different directions, so rank them for each data flow’s business needs before you pick any value.

Goal Meaning Settings that pull towards it
Throughput How many MB or messages per second Large linger.ms, large batch.size, compression, acks=1, more partitions
Latency How long one message takes from producer to consumer linger.ms=0, no or light compression, fetch.min.bytes=1
Durability An acknowledged record is never lost acks=all, RF=3, min.insync.replicas=2, idempotence, unclean election off
Availability The system keeps accepting reads/writes when nodes die Low min ISR, unclean election on, fast failure detection (small session.timeout.ms)

ISR (in-sync replicas): the replicas keeping up with the leader. Unclean election: letting a lagging replica become leader. Idempotence: the producer numbers its batches so the broker drops duplicates on retry.

Two examples from the course material:

  • Finance: durability > availability > latency > throughput. Better to stop accepting transactions for a few seconds than to lose one that was reported as successful.
  • Loading a data warehouse: throughput > availability > durability > latency. Hundreds of thousands of events per second; losing a few or being minutes late is fine.

Priorities can change: an ingestion flow that only fed reports may later compute commissions, and then durability moves to the top.

Before the tables:

  • Defaults change by client version: since 3.0 the producer defaults to acks=all and enable.idempotence=true; since 4.0 linger.ms=5; 2.x clients still default to acks=1. Set them explicitly and keep clients up to date.
  • Throughput bands: low under 10K messages/s, moderate 10K–100K, high 100K–1M.
  • The values below are starting points: benchmark, monitor, tune.

2. Producer

Key idea: the producer decides most of the trade-off: acks and idempotence for durability; linger.ms, batch.size and compression for throughput or latency.

Setting Throughput Latency Durability
acks 1 1 all
enable.idempotence false false true
linger.ms 20 0 5–10
batch.size 32768–65536 16384 16384
compression.type zstd none or lz4 lz4

linger.ms is how long the producer waits to fill a batch; a full batch is sent immediately.

Full producer table and the delivery.timeout.ms trap
Setting Throughput Latency Durability Notes
acks 1 1 all Default all since 3.0
enable.idempotence false false true Requires acks=all; left at default with acks=1 the client disables it silently, explicitly true with acks=1 is a config error
retries / delivery.timeout.ms default low default or per SLA Control it with delivery.timeout.ms (default 120 s) and leave retries at its default
linger.ms 20 0 5–10 The course doc suggests 0 / 8 / 20 for latency / moderate / throughput
batch.size 32768–65536 16384 16384 A full batch is sent immediately
compression.type zstd none or lz4 lz4 lz4 is the balanced choice
max.in.flight.requests.per.connection 5 5 5 5 is the maximum that keeps ordering with idempotence
buffer.memory raise (64–128 MB) default default When full, send blocks up to max.block.ms
Partitioner default default default No key: uniform sticky; with a key: murmur2

Durability settings only help if the code actually sees send failures:

  • KafkaTemplate.send() is asynchronous: errors (expired delivery.timeout.ms, NotEnoughReplicasException) live in the returned CompletableFuture, so a try/catch around send() never catches them; handle them in whenComplete.
  • When the event goes with a DB write, acks=all doesn’t save you from a dual write (writing to two places where one can fail): you need a transactional outbox.

Both are analysed on real code in Case study: a synchronous fraud check over Kafka (the asynchronous send() section and the transactional outbox section).

3. Consumer

Key idea: the most important consumer decision is when to commit offsets; after that, a separate group.id per service and a cooperative assignor so rebalances don’t stop the whole group.

An offset is how far a consumer group has read in a partition. A rebalance is the group redistributing partitions among its consumers (when one joins, leaves or dies).

  • A per-service group.id (e.g. wallet-balance): two services sharing a group split the messages.
  • enable.auto.commit=false for important data (the doc suggests true).
  • auto.offset.reset only applies when the group has no committed offset or it has expired.
  • max.poll.interval.ms (5 minutes by default): processing one batch for longer gets the consumer kicked out of the group.
Full consumer table, checking auto.offset.reset, and the KIP-848 protocol
Setting Suggestion Notes
group.id A specific per-service name, e.g. wallet-balance Two services sharing a group split messages instead of each getting all of them
enable.auto.commit The doc suggests true; for important data use false and commit after processing See the pitfall below
auto.offset.reset latest or earliest Only applies when the group has no committed offset or the offset has expired
partition.assignment.strategy CooperativeStickyAssignor Incremental rebalancing, the whole group doesn’t stop
fetch.min.bytes 1 (latency) / 50,000 (throughput) Works with fetch.max.wait.ms (500 ms default)
receive.buffer.bytes 1 MB, or 4 MB if memory allows Helps on high-latency links between consumer and broker
session.timeout.ms 45,000 (default since 3.0) Lower detects dead consumers faster (availability) but risks false rebalances on long GC pauses
max.poll.interval.ms / max.poll.records 300,000 / 500 (defaults) Processing one batch for longer than this gets the consumer kicked out of the group

Checking auto.offset.reset on the lab: topic payments already held 12 records. A new group with latest read 0 of them, a new group with earliest read all 12. The consumer in section 5, run a second time with earliest, processed 0 records, because the group had committed offsets so auto.offset.reset no longer applied.

4. Topics and brokers

Key idea: RF=3, min.insync.replicas=2 and no auto-created topics are the safe defaults; the partition count is a trade-off between ordering and throughput.

  • replication.factor=3: survives one broker failure. min.insync.replicas=2: goes with acks=all.
  • unclean.leader.election.enable: false for durability, true for availability; can be overridden per topic.
  • auto.create.topics.enable=false: a typo in a topic name doesn’t create a 1-partition, RF-1 topic.
  • offsets.retention.minutes around a month (default 7 days): a group idle for longer loses its offsets.
Full topic and broker table, and the expired-offset trap
Scope Setting Suggestion Why
Topic partitions 1 if you need total ordering; more for throughput See the formula in post 2
Topic replication.factor 3 Survives one broker failure
Topic min.insync.replicas 2 Goes with acks=all
Topic compression.type producer The broker keeps batches as-is, no recompression
Broker auto.create.topics.enable false A typo in a topic name doesn’t create a 1-partition, RF-1 topic
Broker num.replica.fetchers 1 (default), 2–4 for latency More replica fetcher threads, faster commits when many partitions sit between a pair of brokers
Broker num.io.threads / num.network.threads 16 / 8 Depends on cores and disks
Broker unclean.leader.election.enable false (durability) / true (availability) Can be overridden per topic
Broker broker.rack Per rack/AZ Replicas get spread across racks
Broker offsets.retention.minutes 43,800 (about a month) Default 10,080 (7 days); a group idle longer loses its offsets and falls back to auto.offset.reset
Broker log.flush.interval.messages Default (no forced flush) Lower is more durable at a huge throughput cost

5. Spring Boot 3: configuration in application.yml

Key idea: Spring Boot maps spring.kafka.* onto the client. The Spring Kafka container commits offsets instead of auto-commit, and you need an error handler with a DLT so failed messages don’t vanish.

An example for a wallet service that prioritises durability:

spring:
  kafka:
    bootstrap-servers: kafka1:9092,kafka2:9092,kafka3:9092
    producer:
      acks: all
      compression-type: lz4
      batch-size: 32768
      properties:
        enable.idempotence: true
        linger.ms: 10
        max.in.flight.requests.per.connection: 5
        delivery.timeout.ms: 60000
    consumer:
      group-id: wallet-balance
      auto-offset-reset: earliest
      enable-auto-commit: false
      max-poll-records: 200
      properties:
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
    listener:
      ack-mode: record        # commit after each successfully processed record
      concurrency: 3          # 3 consumers per instance; total consumers shouldn't exceed partitions

A DLT (dead letter topic) holds messages that still fail after all retries, so you can inspect them later instead of blocking the flow.

Listener, error handler with a DLT, and the plain Java consumer that ran
@Component
class BalanceListener {
    private final BalanceService balances;
    BalanceListener(BalanceService balances) { this.balances = balances; }

    @KafkaListener(topics = "wallet-transactions")
    void on(ConsumerRecord<String, TransactionEvent> record) {
        balances.apply(record.value());   // idempotent: unique on event id in the DB
    }
}

@Configuration
class KafkaErrorConfig {
    @Bean
    DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        var recoverer = new DeadLetterPublishingRecoverer(template);   // publishes to <topic>.DLT by default
        var backOff = new ExponentialBackOffWithMaxRetries(5);          // bounded, never retry forever
        backOff.setInitialInterval(500);
        backOff.setMultiplier(2.0);
        var handler = new DefaultErrorHandler(recoverer, backOff);
        handler.addNotRetryableExceptions(InvalidTransactionException.class); // business error: straight to DLT
        return handler;
    }
}

The Spring code above wasn’t run: the lab only used the plain Java client, and malformed messages also need an ErrorHandlingDeserializer configured to reach the DLT. The same idea with a plain consumer, which did run, is below: auto-commit off, commitSync only after processing, shut down with wakeup().

props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CooperativeStickyAssignor.class.getName());

var consumer = new KafkaConsumer<String, String>(props);
Runtime.getRuntime().addShutdownHook(new Thread(consumer::wakeup)); // poll() throws WakeupException
try {
    consumer.subscribe(List.of("payments"));
    while (true) {
        var records = consumer.poll(Duration.ofMillis(500));
        for (var r : records) applyToBalance(r);   // idempotent
        if (!records.isEmpty()) consumer.commitSync();   // at-least-once
    }
} catch (WakeupException e) {
    // shutting down
} finally {
    consumer.close();
}

6. Case study: two configurations for two systems

Key idea: two flows on one cluster can need almost opposite configurations; rank each flow on its own before filling in the table.

Context. (A) wallet transactions, about 2,000 messages/s; a transaction reported as successful must never be lost. (B) app clickstream, 150,000 events/s at peak, for analytics; losing a few events and being minutes late is fine.

Ranking. (A) durability > availability > latency > throughput. (B) throughput > availability > durability > latency.

(A) Wallet transactions (B) Clickstream
Producer acks all 1
enable.idempotence true false (goes with acks=1; rare duplicates accepted)
linger.ms / batch.size 5 / 16 KB 50 / 128 KB
compression.type lz4 zstd
Topic partitions 12, key = account_id 48, no key (sticky)
RF / min.insync.replicas 3 / 2 3 / 1
unclean.leader.election.enable false true
Consumer commit Auto-commit off, commit after an idempotent DB write Auto-commit or per-batch commit
fetch.min.bytes 1 100,000
Retention 30 days (reconciliation, replay) 3 days

Risks and monitoring.

  • (A) stops accepting writes if two brokers die at once. That’s deliberate, so alert on UnderMinIsrPartitionCount > 0 and have a runbook.
  • (B) can lose records on an unclean leader election, so analytics numbers may be slightly off.
  • Both need consumer lag monitoring (how many messages the consumer hasn’t caught up on): (A) with a threshold of seconds, (B) of minutes.

7. Benchmark, monitor, tune

Key idea: a table won’t give you optimal values; measure with real data, watch the right metrics, change one setting at a time.

  1. Benchmark with real message sizes: kafka-producer-perf-test --topic t --num-records 1000000 --record-size 1024 --throughput -1 --producer-props acks=all linger.ms=10 and kafka-consumer-perf-test. Measure p99 latency, not just average throughput.
  2. Monitor: consumer lag per group/partition; UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount; request latency (RequestMetrics for produce/fetch); producer record-error-rate, request-latency-avg, batch-size-avg, compression-rate-avg; rebalance counts.
  3. Tune one setting at a time and record the results.

8. Lab: a 3-broker KRaft cluster with docker compose

Key idea: bring up 3 KRaft brokers (Kafka managing its own metadata, no ZooKeeper), stop one, and watch the ISR shrink while acks and min ISR decide whether writes succeed.

Docker compose creates and tears down clusters quickly, isolates resources and runs anywhere. My file ran with image confluentinc/cp-kafka:7.5.3, every node both broker and controller (production should run dedicated controllers).

The docker compose file and lab commands

Repeat the kafka1 block for kafka2 and kafka3, changing KAFKA_NODE_ID, container_name, hostname and the ports 19093 and 19094:

services:
  kafka1:
    image: confluentinc/cp-kafka:7.5.3
    container_name: kafka1
    hostname: kafka1
    ports: ["19092:19092"]
    environment:
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka1:29093,2@kafka2:29093,3@kafka3:29093"
      KAFKA_LISTENERS: "INTERNAL://:29092,CONTROLLER://:29093,EXTERNAL://:19092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka1:29092,EXTERNAL://localhost:19092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
docker compose up -d
docker exec kafka1 kafka-topics --bootstrap-server kafka1:29092 \
  --create --topic payments --partitions 6 --replication-factor 3 --config min.insync.replicas=2
docker exec kafka1 kafka-topics --bootstrap-server kafka1:29092 --describe --topic payments
docker stop kafka3        # watch the ISR shrink, then try sending with acks=all and acks=1
docker exec kafka1 kafka-consumer-groups --bootstrap-server kafka1:29092 --describe --group wallet-balance

Exercises

Exercise 1: Rank and configure

Three systems: (a) marketing push notifications; (b) syncing orders to the warehouse system; (c) shipping application logs to a log search system. Rank the four goals for each and pick acks, min.insync.replicas, linger.ms, compression.type and unclean election.

Solution to Exercise 1
  • (a) Marketing notifications: latency > availability > throughput > durability (losing one notification is fine; arriving 10 minutes late kills the promotion). acks=1, min ISR 1, linger.ms 0–5, lz4, unclean true.
  • (b) Orders to the warehouse: durability > availability > latency > throughput. acks=all, idempotence, min ISR 2, linger.ms 5, lz4, unclean false, key order_id, consumers commit after processing.
  • (c) Application logs: throughput > availability > latency > durability. acks=1, min ISR 1, linger.ms 20–50, a large batch.size, zstd, unclean true.

Exercise 2: The broker-loss lab

Bring up the lab from section 8. Create two RF=3 topics, one with min.insync.replicas=2 and one with min.insync.replicas=3. Stop one broker. Predict and verify: can acks=all and acks=1 producers write to each topic, and what error does the application see?

Solution to Exercise 2: my actual results
  • The min ISR 2 topic: both acks=all and acks=1 write fine.
  • The min ISR 3 topic: acks=1 writes, acks=all fails. The application gets TimeoutException: Expiring 1 record(s) after delivery.timeout.ms, because NotEnoughReplicasException is retried automatically.
  • --describe shows Isr: 1,2. After restarting the broker the ISR returns to 1,2,3 within seconds.

Exercise 3: A consumer that neither loses nor double-counts

Write a consumer that updates wallet balances from wallet-transactions so that a crash at any point neither loses a transaction nor applies it twice.

Solution to Exercise 3: at-least-once plus an idempotent consumer
  • No loss: disable auto-commit and commit offsets only after the DB write succeeds (the plain consumer in section 5, or Spring with ack-mode: record).
  • No double-counting: a crash after the DB write but before the offset commit means the record is redelivered, so the DB write must be idempotent. In one DB transaction: INSERT INTO processed_event(event_id) VALUES (?) ON CONFLICT DO NOTHING. If no row was inserted, the record was already applied, so skip it; otherwise update the balance.

That’s at-least-once delivery plus an idempotent consumer, i.e. an exactly-once effect. Alternative: store the offset in the same DB transaction and seek to it when partitions are assigned.

Remember

  • Rank the four goals for a concrete flow and name the matching acks, min ISR, linger.ms, compression and unclean election values.
  • Explain why durability and availability pull against each other in min.insync.replicas and unclean election.
  • Know when auto-commit loses messages, when it duplicates them, and how to commit safely for payments.
  • Tell eager from cooperative rebalancing, and know what KIP-848 changes in Kafka 4.0.
  • Explain why you keep safe broker defaults and override per topic, and which metrics you watch.

Interview questions

Junior

When does auto.offset.reset apply?

Only when the group has no committed offset for a partition, or the committed offset is no longer valid.

Invalid means expired per offsets.retention.minutes, or outside the retained range. earliest reads from the oldest retained record, latest reads only new records, none throws an exception. A group with committed offsets always resumes from them.

What are the three mandatory settings of a Kafka producer?

bootstrap.servers, key.serializer and value.serializer.

Everything else has a default, but set acks, enable.idempotence, linger.ms and compression.type explicitly, because defaults change between versions.

What happens if two different services share a group.id?

They become one consumer group and split the partitions: each message reaches only one of the two services.

If each service must receive every message, each needs its own group.id.

Mid

How would you configure a producer for financial data?

acks=all, idempotence on, and a topic with RF=3, min.insync.replicas=2 and unclean election off.

  • max.in.flight.requests.per.connection ≤ 5, delivery.timeout.ms from the SLA (and ≥ linger.ms + request.timeout.ms), a small linger.ms, lz4, and a key on the entity that needs ordering.
  • Handle send failures in the callback, and use an outbox if the event accompanies a DB write.
Is enable.auto.commit=true safe?

Only (as at-least-once) when you process synchronously inside the poll loop; with asynchronous processing it can lose messages.

It periodically commits the offsets of records already returned by poll, during a later poll. On a thread pool, offsets can be committed before processing finishes, and a crash loses those records. For important data, disable it and commit after processing, or let the Spring Kafka container commit per AckMode.

Processing one batch takes 6 minutes. What happens and how do you fix it?

It exceeds max.poll.interval.ms (5 minutes by default), so the consumer is kicked out of the group and the batch is reprocessed.

Its partitions go to another consumer and the old consumer’s commit fails. Fix it by lowering max.poll.records, raising max.poll.interval.ms to fit, or offloading the heavy work and using pause/resume on the partitions meanwhile. Patterns for long-running tasks (async, liveness tuning, pipelines) are in §5 of this post.

Senior

How does cooperative rebalancing differ from eager?

Eager revokes every partition and stops the whole group; cooperative revokes only the partitions that must move, and the rest keep running.

Cooperative (CooperativeStickyAssignor) works over two rounds, so rolling deploys and autoscaling cause much less disruption. Kafka 4.0 adds the KIP-848 protocol (group.protocol=consumer), which moves assignment to the broker and makes rebalances incremental by default.

How do you choose the partition count for a new topic?

Divide the target throughput by one consumer’s and one producer’s per-partition throughput, take the max, then add headroom for 1–2 years of growth.

You need headroom because you can’t shrink, and growing remaps keys. The consumer is usually the limit. Weigh the cost of many partitions: open files, replication work, producer memory, leader election time. If you need total ordering and the load is small, use one partition.

Where do durability and availability conflict in Kafka’s configuration?

In min.insync.replicas, unclean.leader.election.enable, acks and session.timeout.ms.

  • Higher min.insync.replicas is more durable but stops writes sooner when nodes die.
  • Unclean election on recovers faster but loses records.
  • acks=all is slower and can be refused.
  • Lower session.timeout.ms detects failures faster but triggers false rebalances.

Choose per flow from its goal ranking, with safe broker defaults and per-topic overrides.

Which metrics do you monitor on a Kafka cluster?

Consumer lag first, then replication health (UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount).

Add active controller count, produce/fetch request latency, broker disk and network, producer error and retry rates, average batch size, and rebalance counts. Alert against each flow’s SLA, not a single global threshold.

Useful phrases for interviews

  • “Before touching any setting, I rank throughput, latency, durability and availability for that specific flow.”
  • “For the ledger we accept that the topic stops taking writes if two brokers die — durability ranks above availability.”
  • “Defaults changed in Kafka 3.0 and 4.0, so we set acks, idempotence and linger.ms explicitly.”
  • “We commit offsets only after the database write, and the write itself is idempotent.”
  • “Auto-commit is only at-least-once if processing finishes inside the poll loop.”
  • “Cooperative rebalancing means a rolling deploy doesn’t stop the whole consumer group.”
  • “We keep safe defaults on the brokers and override them per topic for the throughput-first pipelines.”
  • “These numbers are a starting point — we benchmark with real payloads, monitor lag and tune one setting at a time.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.