Kafka and message brokers series: 1. Message brokers: four problems with request-response and how to choose a broker · 2. Why Kafka is fast and durable: the log, zero-copy, batching and replication · 3. Tuning Kafka for your goal: throughput, latency, durability, availability · 4. Case study: a synchronous fraud check over Kafka and Redis — good idea or not? · 5. CCDAK review: 40 Kafka questions worth remembering
No set of Kafka settings maxes out throughput, latency, durability and availability at once. This post shows how to rank the four goals, then pick values for the producer, consumer, topics, brokers and Spring Boot 3, with a 3-broker lab to check them. Read post 2 first for ISR, acks and batching.
At a glance:
flowchart TD
G{What comes first?} -->|Durability| D["acks=all, RF=3, min ISR=2"]
G -->|Throughput| T["big linger.ms, batch.size, zstd"]
G -->|Latency| L["linger.ms=0, fetch.min.bytes=1"]
D -.->|trade-off| X["Broker down: partition stops writes"]
C[Consumer] --> CM[Commit after processing]
CM --> ID[Idempotent writes]
B["Safe defaults on broker"] --> TP[Override per topic]
TP --> BM["Benchmark, monitor, tune"]
1. Rank your goals before touching any setting
Key idea: the four goals pull settings in different directions, so rank them for each data flow’s business needs before you pick any value.
| Goal | Meaning | Settings that pull towards it |
|---|---|---|
| Throughput | How many MB or messages per second | Large linger.ms, large batch.size, compression, acks=1, more partitions |
| Latency | How long one message takes from producer to consumer | linger.ms=0, no or light compression, fetch.min.bytes=1 |
| Durability | An acknowledged record is never lost | acks=all, RF=3, min.insync.replicas=2, idempotence, unclean election off |
| Availability | The system keeps accepting reads/writes when nodes die | Low min ISR, unclean election on, fast failure detection (small session.timeout.ms) |
ISR (in-sync replicas): the replicas keeping up with the leader. Unclean election: letting a lagging replica become leader. Idempotence: the producer numbers its batches so the broker drops duplicates on retry.
Two examples from the course material:
- Finance: durability > availability > latency > throughput. Better to stop accepting transactions for a few seconds than to lose one that was reported as successful.
- Loading a data warehouse: throughput > availability > durability > latency. Hundreds of thousands of events per second; losing a few or being minutes late is fine.
Priorities can change: an ingestion flow that only fed reports may later compute commissions, and then durability moves to the top.
Before the tables:
- Defaults change by client version: since 3.0 the producer defaults to
acks=allandenable.idempotence=true; since 4.0linger.ms=5; 2.x clients still default toacks=1. Set them explicitly and keep clients up to date. - Throughput bands: low under 10K messages/s, moderate 10K–100K, high 100K–1M.
- The values below are starting points: benchmark, monitor, tune.
2. Producer
Key idea: the producer decides most of the trade-off: acks and idempotence for durability; linger.ms, batch.size and compression for throughput or latency.
| Setting | Throughput | Latency | Durability |
|---|---|---|---|
acks |
1 |
1 |
all |
enable.idempotence |
false |
false |
true |
linger.ms |
20 | 0 | 5–10 |
batch.size |
32768–65536 | 16384 | 16384 |
compression.type |
zstd |
none or lz4 |
lz4 |
linger.ms is how long the producer waits to fill a batch; a full batch is sent immediately.
Full producer table and the delivery.timeout.ms trap
| Setting | Throughput | Latency | Durability | Notes |
|---|---|---|---|---|
acks |
1 |
1 |
all |
Default all since 3.0 |
enable.idempotence |
false |
false |
true |
Requires acks=all; left at default with acks=1 the client disables it silently, explicitly true with acks=1 is a config error |
retries / delivery.timeout.ms |
default | low | default or per SLA | Control it with delivery.timeout.ms (default 120 s) and leave retries at its default |
linger.ms |
20 | 0 | 5–10 | The course doc suggests 0 / 8 / 20 for latency / moderate / throughput |
batch.size |
32768–65536 | 16384 | 16384 | A full batch is sent immediately |
compression.type |
zstd |
none or lz4 |
lz4 |
lz4 is the balanced choice |
max.in.flight.requests.per.connection |
5 | 5 | 5 | 5 is the maximum that keeps ordering with idempotence |
buffer.memory |
raise (64–128 MB) | default | default | When full, send blocks up to max.block.ms |
| Partitioner | default | default | default | No key: uniform sticky; with a key: murmur2 |
Durability settings only help if the code actually sees send failures:
KafkaTemplate.send()is asynchronous: errors (expireddelivery.timeout.ms,NotEnoughReplicasException) live in the returnedCompletableFuture, so atry/catcharoundsend()never catches them; handle them inwhenComplete.- When the event goes with a DB write,
acks=alldoesn’t save you from a dual write (writing to two places where one can fail): you need a transactional outbox.
Both are analysed on real code in Case study: a synchronous fraud check over Kafka (the asynchronous send() section and the transactional outbox section).
3. Consumer
Key idea: the most important consumer decision is when to commit offsets; after that, a separate group.id per service and a cooperative assignor so rebalances don’t stop the whole group.
An offset is how far a consumer group has read in a partition. A rebalance is the group redistributing partitions among its consumers (when one joins, leaves or dies).
- A per-service
group.id(e.g.wallet-balance): two services sharing a group split the messages. enable.auto.commit=falsefor important data (the doc suggeststrue).auto.offset.resetonly applies when the group has no committed offset or it has expired.max.poll.interval.ms(5 minutes by default): processing one batch for longer gets the consumer kicked out of the group.
Full consumer table, checking auto.offset.reset, and the KIP-848 protocol
| Setting | Suggestion | Notes |
|---|---|---|
group.id |
A specific per-service name, e.g. wallet-balance |
Two services sharing a group split messages instead of each getting all of them |
enable.auto.commit |
The doc suggests true; for important data use false and commit after processing |
See the pitfall below |
auto.offset.reset |
latest or earliest |
Only applies when the group has no committed offset or the offset has expired |
partition.assignment.strategy |
CooperativeStickyAssignor |
Incremental rebalancing, the whole group doesn’t stop |
fetch.min.bytes |
1 (latency) / 50,000 (throughput) | Works with fetch.max.wait.ms (500 ms default) |
receive.buffer.bytes |
1 MB, or 4 MB if memory allows | Helps on high-latency links between consumer and broker |
session.timeout.ms |
45,000 (default since 3.0) | Lower detects dead consumers faster (availability) but risks false rebalances on long GC pauses |
max.poll.interval.ms / max.poll.records |
300,000 / 500 (defaults) | Processing one batch for longer than this gets the consumer kicked out of the group |
Checking auto.offset.reset on the lab: topic payments already held 12 records. A new group with latest read 0 of them, a new group with earliest read all 12. The consumer in section 5, run a second time with earliest, processed 0 records, because the group had committed offsets so auto.offset.reset no longer applied.
4. Topics and brokers
Key idea: RF=3, min.insync.replicas=2 and no auto-created topics are the safe defaults; the partition count is a trade-off between ordering and throughput.
replication.factor=3: survives one broker failure.min.insync.replicas=2: goes withacks=all.unclean.leader.election.enable:falsefor durability,truefor availability; can be overridden per topic.auto.create.topics.enable=false: a typo in a topic name doesn’t create a 1-partition, RF-1 topic.offsets.retention.minutesaround a month (default 7 days): a group idle for longer loses its offsets.
Full topic and broker table, and the expired-offset trap
| Scope | Setting | Suggestion | Why |
|---|---|---|---|
| Topic | partitions |
1 if you need total ordering; more for throughput | See the formula in post 2 |
| Topic | replication.factor |
3 | Survives one broker failure |
| Topic | min.insync.replicas |
2 | Goes with acks=all |
| Topic | compression.type |
producer |
The broker keeps batches as-is, no recompression |
| Broker | auto.create.topics.enable |
false |
A typo in a topic name doesn’t create a 1-partition, RF-1 topic |
| Broker | num.replica.fetchers |
1 (default), 2–4 for latency | More replica fetcher threads, faster commits when many partitions sit between a pair of brokers |
| Broker | num.io.threads / num.network.threads |
16 / 8 | Depends on cores and disks |
| Broker | unclean.leader.election.enable |
false (durability) / true (availability) |
Can be overridden per topic |
| Broker | broker.rack |
Per rack/AZ | Replicas get spread across racks |
| Broker | offsets.retention.minutes |
43,800 (about a month) | Default 10,080 (7 days); a group idle longer loses its offsets and falls back to auto.offset.reset |
| Broker | log.flush.interval.messages |
Default (no forced flush) | Lower is more durable at a huge throughput cost |
5. Spring Boot 3: configuration in application.yml
Key idea: Spring Boot maps spring.kafka.* onto the client. The Spring Kafka container commits offsets instead of auto-commit, and you need an error handler with a DLT so failed messages don’t vanish.
An example for a wallet service that prioritises durability:
spring:
kafka:
bootstrap-servers: kafka1:9092,kafka2:9092,kafka3:9092
producer:
acks: all
compression-type: lz4
batch-size: 32768
properties:
enable.idempotence: true
linger.ms: 10
max.in.flight.requests.per.connection: 5
delivery.timeout.ms: 60000
consumer:
group-id: wallet-balance
auto-offset-reset: earliest
enable-auto-commit: false
max-poll-records: 200
properties:
partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
listener:
ack-mode: record # commit after each successfully processed record
concurrency: 3 # 3 consumers per instance; total consumers shouldn't exceed partitions
A DLT (dead letter topic) holds messages that still fail after all retries, so you can inspect them later instead of blocking the flow.
Listener, error handler with a DLT, and the plain Java consumer that ran
@Component
class BalanceListener {
private final BalanceService balances;
BalanceListener(BalanceService balances) { this.balances = balances; }
@KafkaListener(topics = "wallet-transactions")
void on(ConsumerRecord<String, TransactionEvent> record) {
balances.apply(record.value()); // idempotent: unique on event id in the DB
}
}
@Configuration
class KafkaErrorConfig {
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
var recoverer = new DeadLetterPublishingRecoverer(template); // publishes to <topic>.DLT by default
var backOff = new ExponentialBackOffWithMaxRetries(5); // bounded, never retry forever
backOff.setInitialInterval(500);
backOff.setMultiplier(2.0);
var handler = new DefaultErrorHandler(recoverer, backOff);
handler.addNotRetryableExceptions(InvalidTransactionException.class); // business error: straight to DLT
return handler;
}
}
The Spring code above wasn’t run: the lab only used the plain Java client, and malformed messages also need an ErrorHandlingDeserializer configured to reach the DLT. The same idea with a plain consumer, which did run, is below: auto-commit off, commitSync only after processing, shut down with wakeup().
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CooperativeStickyAssignor.class.getName());
var consumer = new KafkaConsumer<String, String>(props);
Runtime.getRuntime().addShutdownHook(new Thread(consumer::wakeup)); // poll() throws WakeupException
try {
consumer.subscribe(List.of("payments"));
while (true) {
var records = consumer.poll(Duration.ofMillis(500));
for (var r : records) applyToBalance(r); // idempotent
if (!records.isEmpty()) consumer.commitSync(); // at-least-once
}
} catch (WakeupException e) {
// shutting down
} finally {
consumer.close();
}
6. Case study: two configurations for two systems
Key idea: two flows on one cluster can need almost opposite configurations; rank each flow on its own before filling in the table.
Context. (A) wallet transactions, about 2,000 messages/s; a transaction reported as successful must never be lost. (B) app clickstream, 150,000 events/s at peak, for analytics; losing a few events and being minutes late is fine.
Ranking. (A) durability > availability > latency > throughput. (B) throughput > availability > durability > latency.
| (A) Wallet transactions | (B) Clickstream | |
|---|---|---|
Producer acks |
all |
1 |
enable.idempotence |
true |
false (goes with acks=1; rare duplicates accepted) |
linger.ms / batch.size |
5 / 16 KB | 50 / 128 KB |
compression.type |
lz4 |
zstd |
| Topic partitions | 12, key = account_id |
48, no key (sticky) |
RF / min.insync.replicas |
3 / 2 | 3 / 1 |
unclean.leader.election.enable |
false |
true |
| Consumer commit | Auto-commit off, commit after an idempotent DB write | Auto-commit or per-batch commit |
fetch.min.bytes |
1 | 100,000 |
| Retention | 30 days (reconciliation, replay) | 3 days |
Risks and monitoring.
- (A) stops accepting writes if two brokers die at once. That’s deliberate, so alert on
UnderMinIsrPartitionCount > 0and have a runbook. - (B) can lose records on an unclean leader election, so analytics numbers may be slightly off.
- Both need consumer lag monitoring (how many messages the consumer hasn’t caught up on): (A) with a threshold of seconds, (B) of minutes.
7. Benchmark, monitor, tune
Key idea: a table won’t give you optimal values; measure with real data, watch the right metrics, change one setting at a time.
- Benchmark with real message sizes:
kafka-producer-perf-test --topic t --num-records 1000000 --record-size 1024 --throughput -1 --producer-props acks=all linger.ms=10andkafka-consumer-perf-test. Measure p99 latency, not just average throughput. - Monitor: consumer lag per group/partition;
UnderReplicatedPartitions,UnderMinIsrPartitionCount,OfflinePartitionsCount; request latency (RequestMetricsfor produce/fetch); producerrecord-error-rate,request-latency-avg,batch-size-avg,compression-rate-avg; rebalance counts. - Tune one setting at a time and record the results.
8. Lab: a 3-broker KRaft cluster with docker compose
Key idea: bring up 3 KRaft brokers (Kafka managing its own metadata, no ZooKeeper), stop one, and watch the ISR shrink while acks and min ISR decide whether writes succeed.
Docker compose creates and tears down clusters quickly, isolates resources and runs anywhere. My file ran with image confluentinc/cp-kafka:7.5.3, every node both broker and controller (production should run dedicated controllers).
The docker compose file and lab commands
Repeat the kafka1 block for kafka2 and kafka3, changing KAFKA_NODE_ID, container_name, hostname and the ports 19093 and 19094:
services:
kafka1:
image: confluentinc/cp-kafka:7.5.3
container_name: kafka1
hostname: kafka1
ports: ["19092:19092"]
environment:
CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: "broker,controller"
KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka1:29093,2@kafka2:29093,3@kafka3:29093"
KAFKA_LISTENERS: "INTERNAL://:29092,CONTROLLER://:29093,EXTERNAL://:19092"
KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka1:29092,EXTERNAL://localhost:19092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
docker compose up -d
docker exec kafka1 kafka-topics --bootstrap-server kafka1:29092 \
--create --topic payments --partitions 6 --replication-factor 3 --config min.insync.replicas=2
docker exec kafka1 kafka-topics --bootstrap-server kafka1:29092 --describe --topic payments
docker stop kafka3 # watch the ISR shrink, then try sending with acks=all and acks=1
docker exec kafka1 kafka-consumer-groups --bootstrap-server kafka1:29092 --describe --group wallet-balance
Exercises
Exercise 1: Rank and configure
Three systems: (a) marketing push notifications; (b) syncing orders to the warehouse system; (c) shipping application logs to a log search system. Rank the four goals for each and pick acks, min.insync.replicas, linger.ms, compression.type and unclean election.
Solution to Exercise 1
- (a) Marketing notifications: latency > availability > throughput > durability (losing one notification is fine; arriving 10 minutes late kills the promotion).
acks=1, min ISR 1,linger.ms0–5,lz4, uncleantrue. - (b) Orders to the warehouse: durability > availability > latency > throughput.
acks=all, idempotence, min ISR 2,linger.ms5,lz4, uncleanfalse, keyorder_id, consumers commit after processing. - (c) Application logs: throughput > availability > latency > durability.
acks=1, min ISR 1,linger.ms20–50, a largebatch.size,zstd, uncleantrue.
Exercise 2: The broker-loss lab
Bring up the lab from section 8. Create two RF=3 topics, one with min.insync.replicas=2 and one with min.insync.replicas=3. Stop one broker. Predict and verify: can acks=all and acks=1 producers write to each topic, and what error does the application see?
Solution to Exercise 2: my actual results
- The min ISR 2 topic: both
acks=allandacks=1write fine. - The min ISR 3 topic:
acks=1writes,acks=allfails. The application getsTimeoutException: Expiring 1 record(s)afterdelivery.timeout.ms, becauseNotEnoughReplicasExceptionis retried automatically. --describeshowsIsr: 1,2. After restarting the broker the ISR returns to1,2,3within seconds.
Exercise 3: A consumer that neither loses nor double-counts
Write a consumer that updates wallet balances from wallet-transactions so that a crash at any point neither loses a transaction nor applies it twice.
Solution to Exercise 3: at-least-once plus an idempotent consumer
- No loss: disable auto-commit and commit offsets only after the DB write succeeds (the plain consumer in section 5, or Spring with
ack-mode: record). - No double-counting: a crash after the DB write but before the offset commit means the record is redelivered, so the DB write must be idempotent. In one DB transaction:
INSERT INTO processed_event(event_id) VALUES (?) ON CONFLICT DO NOTHING. If no row was inserted, the record was already applied, so skip it; otherwise update the balance.
That’s at-least-once delivery plus an idempotent consumer, i.e. an exactly-once effect. Alternative: store the offset in the same DB transaction and seek to it when partitions are assigned.
Remember
- Rank the four goals for a concrete flow and name the matching
acks, min ISR,linger.ms, compression and unclean election values. - Explain why durability and availability pull against each other in
min.insync.replicasand unclean election. - Know when auto-commit loses messages, when it duplicates them, and how to commit safely for payments.
- Tell eager from cooperative rebalancing, and know what KIP-848 changes in Kafka 4.0.
- Explain why you keep safe broker defaults and override per topic, and which metrics you watch.
Interview questions
Junior
When does auto.offset.reset apply?
Only when the group has no committed offset for a partition, or the committed offset is no longer valid.
Invalid means expired per offsets.retention.minutes, or outside the retained range. earliest reads from the oldest retained record, latest reads only new records, none throws an exception. A group with committed offsets always resumes from them.
What are the three mandatory settings of a Kafka producer?
bootstrap.servers, key.serializer and value.serializer.
Everything else has a default, but set acks, enable.idempotence, linger.ms and compression.type explicitly, because defaults change between versions.
Mid
How would you configure a producer for financial data?
acks=all, idempotence on, and a topic with RF=3, min.insync.replicas=2 and unclean election off.
max.in.flight.requests.per.connection≤ 5,delivery.timeout.msfrom the SLA (and ≥linger.ms+request.timeout.ms), a smalllinger.ms,lz4, and a key on the entity that needs ordering.- Handle send failures in the callback, and use an outbox if the event accompanies a DB write.
Is enable.auto.commit=true safe?
Only (as at-least-once) when you process synchronously inside the poll loop; with asynchronous processing it can lose messages.
It periodically commits the offsets of records already returned by poll, during a later poll. On a thread pool, offsets can be committed before processing finishes, and a crash loses those records. For important data, disable it and commit after processing, or let the Spring Kafka container commit per AckMode.
Processing one batch takes 6 minutes. What happens and how do you fix it?
It exceeds max.poll.interval.ms (5 minutes by default), so the consumer is kicked out of the group and the batch is reprocessed.
Its partitions go to another consumer and the old consumer’s commit fails. Fix it by lowering max.poll.records, raising max.poll.interval.ms to fit, or offloading the heavy work and using pause/resume on the partitions meanwhile. Patterns for long-running tasks (async, liveness tuning, pipelines) are in §5 of this post.
Senior
How does cooperative rebalancing differ from eager?
Eager revokes every partition and stops the whole group; cooperative revokes only the partitions that must move, and the rest keep running.
Cooperative (CooperativeStickyAssignor) works over two rounds, so rolling deploys and autoscaling cause much less disruption. Kafka 4.0 adds the KIP-848 protocol (group.protocol=consumer), which moves assignment to the broker and makes rebalances incremental by default.
How do you choose the partition count for a new topic?
Divide the target throughput by one consumer’s and one producer’s per-partition throughput, take the max, then add headroom for 1–2 years of growth.
You need headroom because you can’t shrink, and growing remaps keys. The consumer is usually the limit. Weigh the cost of many partitions: open files, replication work, producer memory, leader election time. If you need total ordering and the load is small, use one partition.
Where do durability and availability conflict in Kafka’s configuration?
In min.insync.replicas, unclean.leader.election.enable, acks and session.timeout.ms.
- Higher
min.insync.replicasis more durable but stops writes sooner when nodes die. - Unclean election on recovers faster but loses records.
acks=allis slower and can be refused.- Lower
session.timeout.msdetects failures faster but triggers false rebalances.
Choose per flow from its goal ranking, with safe broker defaults and per-topic overrides.
Which metrics do you monitor on a Kafka cluster?
Consumer lag first, then replication health (UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount).
Add active controller count, produce/fetch request latency, broker disk and network, producer error and retry rates, average batch size, and rebalance counts. Alert against each flow’s SLA, not a single global threshold.
Useful phrases for interviews
- “Before touching any setting, I rank throughput, latency, durability and availability for that specific flow.”
- “For the ledger we accept that the topic stops taking writes if two brokers die — durability ranks above availability.”
- “Defaults changed in Kafka 3.0 and 4.0, so we set
acks, idempotence andlinger.msexplicitly.” - “We commit offsets only after the database write, and the write itself is idempotent.”
- “Auto-commit is only at-least-once if processing finishes inside the poll loop.”
- “Cooperative rebalancing means a rolling deploy doesn’t stop the whole consumer group.”
- “We keep safe defaults on the brokers and override them per topic for the throughput-first pipelines.”
- “These numbers are a starting point — we benchmark with real payloads, monitor lag and tune one setting at a time.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.