Kafka problems and patterns series: 1. Partition count, large messages, lag and long-running tasks · 2. Dual writes, exactly-once and message ordering · 3. Case study: fraud detection with Kafka and Redis
Knowing how Kafka works isn’t enough; a senior interview asks what you do when a topic is too narrow, a message is too big, a consumer falls behind or one record jams a whole partition. This post collects those problems, each with a pattern and its trade-offs. Read Why Kafka is fast and durable and Tuning Kafka for your goal first.
1. Choosing the partition count
Key idea: compute the partition count separately for writes and for reads, take the larger, then add headroom for growth. Partitions can be added but never removed.
Two facts make this hard:
- Grow-only. Adding partitions doesn’t interrupt clients, but there’s no command to remove them.
- Hashing isn’t consistent across counts. The default partitioner uses
murmur2(key) % partition_count. Change the divisor and the same key lands in a different partition, so per-key ordering breaks.
Formula: partitions = target throughput / per-partition throughput. Throughput has two directions, and the narrower one wins.
| Topic type | Examples | Per-partition throughput | Course’s suggestion |
|---|---|---|---|
| Write-heavy | IoT, clickstream | ~10 MB/s writes; ~700 B messages, ~500 B compressed | < 10k msg/s: < 4; 10k–100k: 8–16; 100k–1M: 20–32 |
| Read-heavy | payments, orders | depends on logic and DB, must load test | < 1k msg/s: ≤ 8; 1k–10k: 8–16 |
Example: a payment-events topic peaks at 5,000 msg/s, 500 B each. Writes: 5,000 × 500 B = 2.5 MB/s, one partition is enough. Reads: a load test shows one consumer (DB write, ledger call) handles 400 msg/s, so 5,000 / 400 = 12.5, i.e. 13 consumers. Pick 16: enough for 13 consumers, room to grow, and it divides evenly across 2, 4 or 8 instances.
2. Shrinking a topic: migrate to a new one
Key idea: you can’t shrink in place, so create a correctly sized topic, copy the data, then move producers and consumers over in order.
- Create a new topic with the desired partition count.
- Copy from the old topic with MirrorMaker 2 (or a hand-written consumer/producer).
- When the copy catches up, cut over: pause producers briefly so both sides match.
- Switch producers to the new topic (the course suggests double-publishing to both for a while, so consumers that haven’t moved don’t stall).
- Move consumers to the new topic and choose where they start:
auto.offset.reset=latest: start from the cutover; consumers must be idempotent because the overlap may be read twice.earliest, or reset offsets to the cutover timestamp (kafka-consumer-groups --reset-offsets --to-datetime): nothing missed, but a stretch is definitely read twice.
3. Large messages: claim check
Key idea: a record’s default size limit is about 1 MB. Don’t raise it; put the payload in object storage and send only a reference.
Example: a producer wants to send a 20 MB report file for a consumer to process.
- Option 1, raise the limit:
max.message.bytes(topic),message.max.bytes(broker), plusmax.request.sizeon the producer andmax.partition.fetch.bytes,fetch.max.byteson the consumer. The course calls this an anti-pattern with a practical ceiling of ~10 MB: big records slow replication, cut throughput, eat consumer memory, and one big record blocks every small record behind it in the partition. - Option 2, claim check: the producer uploads the file to S3 (or MinIO, GCS) and sends a message with the URL, size and checksum. The consumer reads the message and downloads the file. Simple and fast; the cost is one more storage system.
Claim check details: write order, cleanup, security
- Order: upload first, confirm, then send the message. The other way round, a consumer can receive a URL pointing to a file that doesn’t exist yet.
- Cleanup: a file uploaded whose message never got sent is orphaned. Put a lifecycle rule on the bucket (delete after N days), longer than the topic’s retention.
- Security: send an object key rather than a public URL; the consumer reads with its own credentials, or generates a short-lived presigned URL.
- Idempotency: derive the object key from the content (a hash) or a business ID, so producer retries don’t create copies.
- Medium messages (1–5 MB): try compression (
compression.type=zstd) first; JSON often compresses 5–10×.
4. Message lag: measure first, scale second
Key idea: lag = the partition’s end offset minus the group’s committed offset. When lag grows, find the cause through monitoring first; scaling consumers only helps when slow consumers are the cause, and it’s capped by the partition count.
- Offset lag:
log-end-offset − committed-offset, per partition. - Time lag:
(last_produced − last_consumed) / produce rate. This one is easier to put an SLO on (“at most 30 seconds behind”).
Sustained lag causes:
- Unacceptable latency for users.
- Data loss: if lag exceeds retention, records are deleted before anyone reads them.
- Consumers judged dead and rebalanced, if the cause is slow processing (section 5).
Common causes: producer spikes, consumers slowing down (slow DB, slow external API), long tasks, network, long GC pauses, constant rebalances.
flowchart LR
A[Lag grows] --> B{Produce rate up?}
B -- yes, temporary --> C[Wait it out, check retention]
B -- no --> D{Per-record processing time up?}
D -- yes --> E[Fix the slow dependency: DB, API, GC]
D -- no --> F{Constant rebalances?}
F -- yes --> G[See section 5: liveness]
F -- no --> H[Scale consumers, add partitions if needed]
Measuring lag: tools, metrics and Java code
- CLI:
kafka-consumer-groups.sh --bootstrap-server ... --describe --group paymentprintsCURRENT-OFFSET,LOG-END-OFFSET,LAGper partition. - Consumer-side metric:
records-lag-max(JMX, Java client) shows the worst partition that consumer owns. It can’t see partitions nobody owns. - External monitoring: Burrow (the course’s pick; it evaluates status over a sliding window rather than a fixed threshold), Kafka Lag Exporter (time lag, exported to Prometheus), Confluent Control Center.
- DIY with AdminClient:
try (Admin admin = Admin.create(props)) {
Map<TopicPartition, OffsetAndMetadata> committed =
admin.listConsumerGroupOffsets("payment").partitionsToOffsetAndMetadata().get();
Map<TopicPartition, OffsetSpec> latest = committed.keySet().stream()
.collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.latest()));
var ends = admin.listOffsets(latest).all().get();
committed.forEach((tp, om) ->
System.out.printf("%s lag=%d%n", tp, ends.get(tp).offset() - om.offset()));
}
- Good alerts: time lag above the SLO for 5 consecutive minutes, and lag above 50% of retention (data loss is coming).
5. Long-running tasks: async, liveness tuning, pipeline
Key idea: consumers should process as fast as possible. When one record needs minutes (reports, ML jobs, loan applications), there are three options: process asynchronously, loosen the liveness settings, or split the work into a multi-stage pipeline.
The problem: a consumer is considered dead if it doesn’t call poll() within max.poll.interval.ms (default 5 minutes), or doesn’t heartbeat within session.timeout.ms (default 45 seconds). Dead means the group rebalances, the partition moves to another consumer, the in-flight record is processed again, and the new consumer is just as slow: a rebalance loop. The heartbeat and coordinator mechanics, with measurements of this loop, are in consumer groups, rebalancing and liveness.
| Option | What it does | Gains | Costs |
|---|---|---|---|
| Async | poll, commit immediately, hand records to a thread pool | low latency, simple poll loop | at-most-once (crash = loss), hard error handling, out of order |
| Liveness tuning | lower max.poll.records (500), raise max.poll.interval.ms |
no code change | fragile: one unusually slow record and you rebalance again; hung consumers detected later |
| Pipeline | split into stages, each a consumer of the previous stage’s topic | at-least-once, traceable per stage, decoupled stages, ordered within one process | more topics, different processes interleave |
flowchart LR
C[(command topic)] --> S0[Stage 0: validate application]
S0 --> T1[(stage-1 topic)]
T1 --> S1[Stage 1: scoring]
S1 --> T2[(stage-2 topic)]
T2 --> S2[Stage 2: create contract]
The course also shows a leaner variant: an Initializer reads the command topic and writes the first stage to one shared stage topic; a Stage Processor reads the stage topic, performs one step, and writes the next stage back to that same topic until done.
A fourth option: pause partitions and process outside the poll loop
To keep at-least-once without building a pipeline: poll a batch, call consumer.pause(assignment), hand the batch to a worker thread, and keep calling poll() regularly (it returns nothing while paused) so you never exceed max.poll.interval.ms. When the worker finishes, commit that batch’s offsets from the polling thread (KafkaConsumer isn’t thread-safe), then resume().
You must handle onPartitionsRevoked: wait for or cancel the worker and commit what’s done before giving up the partition. Spring Kafka has similar support built in; if you write it yourself, this is where it usually goes wrong.
Another option for really long jobs (tens of minutes): Kafka only carries the command, the job records its state in a DB (a jobs table with status, attempt, heartbeat_at), and a separate scheduler reaps stuck jobs. Kafka then no longer has to keep a consumer alive for the whole job.
6. Poison pills, retry topics and dead letter topics
Key idea: a record that always fails (a poison pill), retried forever, jams its partition and lag grows without end. Retry a few times with backoff, then move it to a dead letter topic (DLT) so the partition moves on. This section is my addition; the slides don’t cover it.
Separate two kinds of failure:
- Transient (DB timeout, API 503): retrying helps.
- Permanent (can’t deserialize, business data invalid): retrying is pointless, send straight to the DLT.
With the plain client (no Spring), an undeserializable record makes poll() throw at the same offset until you seek past it; see the consumer group post.
| Retry style | How | Per-key order | When |
|---|---|---|---|
| Blocking | retry in place with capped backoff, then DLT | kept | order matters (balance events) |
| Non-blocking | move the record to orders-retry-1m, orders-retry-10m, then orders-dlt |
lost | independent records (sending emails) |
Spring Kafka: DefaultErrorHandler, @RetryableTopic and deserialization errors
Blocking retries, then DLT:
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
var recoverer = new DeadLetterPublishingRecoverer(template); // publishes to <topic>-dlt
var backoff = new ExponentialBackOffWithMaxRetries(3);
backoff.setInitialInterval(1_000);
backoff.setMultiplier(2.0);
var handler = new DefaultErrorHandler(recoverer, backoff);
handler.addNotRetryableExceptions(ValidationException.class); // permanent: DLT right away
return handler;
}
Non-blocking retries: @RetryableTopic(attempts = "4", backoff = @Backoff(delay = 60_000, multiplier = 10)) on the listener; Spring creates the retry topics and the DLT.
Deserialization poison pills: the failure happens inside poll(), before your listener, so the consumer gets stuck at that offset. Wrap the real deserializer in ErrorHandlingDeserializer so the failure becomes a record carrying the exception, which the error handler can send to the DLT.
A DLT needs an owner: alert when the DLT receives records, a dashboard over the error headers (Spring adds kafka_dlt-exception-message, the original topic and offset), and a tool to replay records from the DLT to the main topic once the bug is fixed.
7. Hot partitions
Key idea: a skewed key (a big merchant, a bot account) piles load onto one partition; adding partitions doesn’t help because that key still goes to one place. This section is my addition.
- Detect: bytes-in per partition heavily skewed, lag growing on one partition only, one consumer at high CPU while the others idle.
- Split the hot key: add a suffix (
merchant-42#0..7) to spread it across 8 partitions. That key loses ordering, so only do it when its order doesn’t matter, or re-merge later. - Separate lane: route the hot key to its own topic with its own consumers.
- Make the consumer faster for that key: batch many records with the same key into one DB write.
Exercises
Exercise 1: Size the partitions
Topic user-click: peak 200,000 events/s, 600 B each after compression; the consumer only bulk-writes to ClickHouse and one consumer handles 30,000 events/s. Topic loan-application: peak 50 applications/s, each needing 2 seconds of processing. Size each topic and name the main risk.
Solution to Exercise 1
user-click: writes 200,000 × 600 B = 120 MB/s; at ~10 MB/s per partition that’s 12. Reads 200,000 / 30,000 ≈ 6.7, so 7. The larger is 12; with growth headroom, 24 (divisible by 2, 3, 4, 6, 8, 12). Risk: brokers and disks (120 MB/s × RF 3 = 360 MB/s of writes across the cluster), not the partition count.
loan-application: one sequential consumer handles 0.5 applications/s, so you’d need 50 / 0.5 = 100 parallel consumers, i.e. 100 partitions: absurd for 50 msg/s. This is a long-running task: build a pipeline, or let Kafka carry commands while a separate worker pool runs jobs with state in a DB. Then 8–12 partitions are enough.
Exercise 2: Lag spikes at 2 a.m.
Every night at 2:00, lag on payment-events climbs from 0 to 400,000 in 20 minutes, then drains by 4:00. The produce rate doesn’t change. List your hypotheses in the order you’d check them, and the metric that confirms each.
Solution to Exercise 2
The produce rate is flat, so suspect the consumer side and its dependencies:
- A nightly batch job competing for the DB (backup, reports,
VACUUM): check the consumer’s query latency and DB load at 2:00. - Constant rebalances: look for
Revoke/Assignlog lines and therebalance-rate-per-hourmetric; automated deploys or long tasks exceedingmax.poll.interval.mscan cause them. - Long GC pauses: GC logs,
jvm_gc_pause_secondsat 2:00 (for example a cache reload). - A slow external dependency (a partner bank’s nightly maintenance): API call latency by hour.
Only if all four are clean do you consider more consumers. Also confirm 400,000 records is far from retention, so nothing gets lost.
Remember
- Size a topic from write throughput and load-tested read throughput, and explain why to pick a number with many divisors.
- Walk through shrinking a topic via a new topic, and where consumers may read duplicates.
- Explain why a claim check beats raising
max.message.bytes. - Measure lag in offsets and in time, and know the order of root-cause checks before scaling.
- Compare the three ways to handle long tasks, and know a poison pill needs capped retries then a DLT.
Interview questions
Junior
What is consumer lag and how do you measure it?
It’s the gap between a partition’s end offset and the group’s committed offset; measure it with kafka-consumer-groups --describe or the records-lag-max metric.
Also track time lag (how many seconds the consumer is behind the producer) because it’s easier to put an SLO on. Lag approaching retention means data loss is coming.
Can you reduce a topic’s partition count?
No. Create a smaller topic, copy the data (MirrorMaker 2), then move producers and consumers over.
When moving consumers, choose the starting point on the new topic explicitly (latest, earliest, or the cutover timestamp) and accept re-reading a stretch, so consumers must be idempotent.
Mid
You need to send a 50 MB file through Kafka. How?
Use a claim check: upload the file to object storage and send a message with the object key, size and checksum.
Raising max.message.bytes to ~10 MB is still an anti-pattern: slower replication, memory-hungry consumers, big records blocking small ones behind them. Upload first, then send the message, and use a lifecycle rule to clean up orphaned files.
Lag is growing. Do you add consumers right away?
No. Find the cause first: a produce spike, slower per-record processing, constant rebalances, GC.
More consumers only help if consumers are the bottleneck and some partitions have no reader. If the DB behind them is saturated, more consumers make it worse.
One record always makes the consumer throw. What happens and how do you handle it?
With unlimited retries, that partition is stuck at that offset and lag grows forever. Retry a few times with backoff, then publish it to a dead letter topic.
Separate transient errors (retry) from permanent ones (straight to the DLT). Deserialization errors happen before the listener, so wrap the deserializer in ErrorHandlingDeserializer. The DLT needs alerts and a replay tool.
Senior
How do you choose the partition count for a new payments topic?
Compute both directions and take the larger: write throughput / ~10 MB/s per partition, and peak TPS / one consumer’s TPS measured by a load test. Then add about 2× headroom.
Payment topics usually bottleneck on reads (DB, ledger), so the load test decides. Since we need per-account ordering, I over-provision up front so we never have to add partitions later and reshuffle keys. Choose numbers like 12, 16, 24 that divide evenly across instances.
A consumer needs 6–10 minutes per record. How do you design it?
Don’t make Kafka keep the consumer alive for the whole job: split it into short pipeline stages, or let Kafka carry commands while jobs run on workers with state in a DB.
Tuning max.poll.interval.ms with max.poll.records=1 works but is fragile, and blocks the whole partition while processing. Commit-first async loses at-least-once. Pausing partitions and polling regularly keeps at-least-once but needs careful revocation handling.
One merchant is 40% of traffic and its partition always lags. What do you do?
That’s a hot partition from a skewed key; more partitions won’t help. Split the hot key (add a suffix to spread it) or route it to its own topic and consumers.
Spreading the key loses ordering for that merchant, so ask the business first: if ordering is needed per order rather than per merchant, change the key to order_id. Batching writes for the hot key also speeds the consumer up.
Useful phrases for interviews
- “Topic width is a one-way decision: you can grow it but never shrink it.”
- “I size partitions for both directions and take the larger number.”
- “We over-provision partitions for keyed topics to avoid re-hashing later.”
- “Large files go to S3; the event only carries a claim check.”
- “We alert on time lag, not just offset lag.”
- “Find the root cause before scaling the consumer group.”
- “Since KIP-62, heartbeats run on a background thread, so the limit you hit is the poll interval.”
- “Long-running tasks belong in a pipeline, not in one poll loop.”
- “A poison pill blocks the partition until we move it to the dead letter topic.”
- “A skewed key created a hot partition, and more partitions won’t fix that.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.