Kafka problems and patterns series: 1. Partition count, large messages, lag and long-running tasks · 2. Dual writes, exactly-once and message ordering · 3. Case study: fraud detection with Kafka and Redis
The three “hardest” questions of any messaging system: how to write to the DB and publish an event without them drifting apart, how to process each message exactly once, and how to keep order. This post answers each with Kafka’s concrete mechanisms and their limits. The idempotent consumer and idempotent producer are covered in Idempotency part 2; here they’re just wired into the bigger picture.
1. Dual writes: two kinds
Key idea: a dual write is a process that must write to two places and needs both to succeed or neither. Two writes to Kafka are covered by Kafka transactions; a DB write plus a Kafka write has no shared transaction.
- United dual write: both writes go to Kafka, e.g. topics
ordersandaudit. Early Kafka had no way to roll back a successful write; since 0.11 there are Kafka transactions. - Separate dual write: save state to a DB, then emit an event to another system (Kafka, an email service…). The two systems aren’t linked, so there’s no shared transaction. This is the common case.
Example, paying a bill: deduct the balance in the DB, then emit BillPaid. If the service dies between the two steps, the balance is deducted but the event never goes out: the bill is never marked paid. The failure is rare, but the impact is severe and it’s very hard to detect and fix.
2. Two fixes that don’t work
Key idea: swapping the order or wrapping both in a DB transaction only moves the failure somewhere else.
- Emit first, save second: die in between and the event is out while the state never changed. The receiver marks a bill paid that wasn’t.
- Wrap in a DB transaction (
BEGIN, save, emit,COMMIT): the emit succeeds butCOMMITfails, so the DB rolls back while the event is already out. On top of that, the DB transaction stays open while waiting on Kafka.
3. Four ways out, and the transactional outbox
Key idea: there are four patterns; the outbox is the most straightforward and effective one for DB + Kafka.
The outbox only keeps one service’s DB and Kafka in step; when several services must change data you need a saga, see distributed transactions: 2PC, saga, outbox.
| Pattern | Idea | Verdict |
|---|---|---|
| Listen to yourself | only publish the event; the service itself consumes it to update its DB | state updates late, reads right after writes miss it; rarely practical |
| Event sourcing | events are the source of truth, state is rebuilt from them | powerful but complex, hard error handling, fits only some problems |
| CDC on business tables | Debezium reads the balance table’s WAL and emits changes |
simple, but the event is a “row change”, not a business event you define |
| Transactional outbox | write the state and a row in an outbox table in the same transaction; CDC reads the outbox and emits |
simple, you control the event’s content; costs latency and is at-least-once |
flowchart LR
H[Handler] -->|1 BEGIN: UPDATE balance + INSERT outbox, COMMIT| DB[(PostgreSQL)]
DB -->|2 WAL stream| CDC[Debezium on Kafka Connect]
CDC -->|3 emit event| T[(topic bill-events)]
Keep the outbox table simple, and store already-encoded bytes so the connector can ship them as-is without understanding them:
CREATE TABLE outbox (
id uuid PRIMARY KEY,
topic varchar(255) NOT NULL,
key bytea, -- e.g. account_id, decides partition and order
value bytea NOT NULL, -- the serialized event (Avro/Protobuf/JSON)
headers bytea,
created_at timestamptz NOT NULL DEFAULT now()
);
Tooling: Debezium as the connector, Kafka Connect to run it, Strimzi to manage it on Kubernetes.
What the outbox costs:
- Latency: one extra step reading the WAL, usually tens to hundreds of ms.
- The outbox table grows over time and slows the DB if you don’t clean it.
- At-least-once: the connector may re-emit after a restart, so consumers must be idempotent.
Running an outbox: ordering, cleanup, polling relay
- Ordering: Debezium reads the WAL in commit order and uses the
keycolumn as the Kafka key, so one account’s events land in one partition in commit order. Two concurrent transactions for the same account are ordered by commit, so lock the account row if business order matters. - Cleanup: Debezium reads the WAL, not the table, so you can
INSERTand thenDELETEin the same transaction: the WAL still has the insert to emit, and the table stays nearly empty. Alternatives: delete rows older than N hours in batches, or partition the table by day and drop old partitions. - Debezium Outbox Event Router: a ready-made SMT (single message transform) that reads columns like
aggregatetype,aggregateid,payloadand routes to the matching topic. - Polling relay (no CDC): a job runs
SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED, sends to Kafka, then setspublished_at. Easier to set up, but more DB load, latency equal to the poll interval, and several parallel relays can reorder events. Comparison in the request-reply case study. - MySQL/TiDB: Debezium reads MySQL’s binlog; TiDB has TiCDC, which ships straight to Kafka.
4. Exactly-once: two kinds of flow
Key idea: in an event pipeline, flaky networks, consumer crashes and retries all create duplicates. Exactly-once doesn’t come from one setting but from combining mechanisms, and the combination depends on whether the flow is Kafka → Kafka or has side effects.
flowchart LR
A[(topic A)] --> S0[stage 0]
S0 --> B[(topic B)]
B --> S1[stage 1]
S1 -->|side effect| DB[(DB)]
- Aggregation flow (topic A → stage 0 → topic B): read from Kafka, write to Kafka. Kafka transactions handle it: write the output and commit the input offsets in one transaction, and downstream consumers read with
isolation.level=read_committed. - Sink flow (topic B → stage 1 → DB): has side effects outside Kafka. Kafka transactions don’t cover it. This is the common case.
For the sink flow, the formula is: exactly-once = at-least-once (lose nothing) + an idempotent consumer (drop duplicates). The slides say “at-least-once + at-most-once”; more precisely, lose nothing at every hop, and make every repeated processing step have no extra effect.
Kafka transactions for consume-transform-produce
producer.initTransactions(); // a stable transactional.id per instance
while (running) {
var records = consumer.poll(Duration.ofMillis(200));
if (records.isEmpty()) continue;
producer.beginTransaction();
try {
for (var r : records) producer.send(new ProducerRecord<>("topic-b", r.key(), transform(r.value())));
producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
producer.commitTransaction();
} catch (ProducerFencedException e) {
throw e; // another instance has our transactional.id: stop for good
} catch (KafkaException e) {
producer.abortTransaction(); // then seek back to the committed offsets and reprocess
}
}
- Fencing: a stable
transactional.idlets the broker block “zombies” (old instances not quite dead) using an epoch. - Downstream consumers must set
isolation.level=read_committed, or they still see data from aborted transactions. - Limits: it only covers writes to Kafka and offsets. API calls, DB writes or emails inside
transformcan still repeat on abort and reprocessing. Kafka Streams turns this on withprocessing.guarantee=exactly_once_v2.
4.1. Lose nothing on the producer side
Key idea: producers lose messages when a send fails on the network without a retry, or the leader dies before followers copy. Fix it with retries, acks=all and idempotence.
retries: the slides recommend 30. Since Kafka 2.1 the default isInteger.MAX_VALUE, and the real retry window is bounded bydelivery.timeout.ms(default 120 seconds). Tune that instead of counting attempts.acks=all: the leader acknowledges only after every in-sync replica has the record. Pair it withmin.insync.replicas=2on RF 3.enable.idempotence=true(default since 3.0): the broker drops retry duplicates and keeps order within the partition.
4.2. Lose nothing on the consumer side
Key idea: consumers lose messages by committing before processing finishes, or by using a new group on every start. Commit after processing, and keep group.id stable.
The slides recommend enabling auto-commit. That’s only right when processing runs synchronously in the poll loop: auto-commit commits the previous batch’s offsets on the next poll(). If you hand records to another thread and keep polling, auto-commit commits records that aren’t processed yet. Spring Kafka disables auto-commit by default and lets the container commit after the listener (see the configuration post).
4.3. Idempotent consumer
Key idea: at-least-once guarantees duplicates, so a consumer with side effects must be idempotent: either overwrite without changing the outcome, or block duplicates with a unique constraint.
- Harmless overwrite:
UPDATE order SET status = 'PAID'run twice is still correct. Watch for an old event arriving late and overwriting a newer one; add a version condition. - Deduplicate: every message has an ID (a UUID or a business ID). The transactions table has a unique constraint on
request_id; a duplicate insert is caught and skipped, in the same transaction as the side effect.
Details (a processed_message table, how long to keep it, duplicates from a producer generating a new message) are in Idempotency part 2.
5. Message ordering
Key idea: Kafka only guarantees order within one partition. Most problems need only a partial order (per player, per order), so use the key to put related messages in one partition.
Order matters in many places: in a game, the sequence “up, jump, turn left” decides the state; for an order, “placed, paid, shipped, delivered” is its lifecycle.
- Total order: you know the exact order of every event. In Kafka you only get it with one partition, giving up parallelism.
- Partial order: you only know the order of related events. A distributed system without a global clock can only offer this, and it’s usually enough: one player’s actions don’t relate to another’s.
Don’t order by physical clocks (created_at from many machines): clocks drift and can go backwards. Use sequence numbers, versions, or offsets within a partition.
End-to-end ordering needs all three:
- Producer: same key → same partition, and
enable.idempotence=trueso retries don’t reorder (with idempotence off andmax.in.flight.requests.per.connection > 1, a later batch can land before one being retried). - Broker: order within a partition is offset order.
- Consumer: one consumer per partition in the group, processing sequentially per key. Dumping records into a thread pool loses order.
Things that silently break ordering
- Adding partitions: keys move, and a key’s new events can be processed before old ones still sitting in the old partition.
- Non-blocking retry topics (
@RetryableTopic): the failed record takes a detour and later records with the same key overtake it. For order-sensitive events, use blocking retries then a DLT. - Several producers for one key (two instances emitting events for one order): the order between them is arrival order at the broker, not business order. Give each aggregate one owner, or attach a version.
- Defense at the consumer: store the last applied version per aggregate and drop events with a version less than or equal to it. That handles both duplicates and reordering.
6. Request-reply over Kafka
Key idea: Kafka is a one-way road; asking and answering needs a reply topic, an ID to match pairs and somewhere to wait for the result. It can be done, but usually shouldn’t be.
The slides on this walk through: separate request and reply topics; message_id and original_message_id to match them; a local pool for replies (a Guava cache with TTL) with polling or a CompletableFuture callback; the multi-instance problem (a group per instance is wasteful, a shared group delivers replies to the wrong instance); a shared reply store (Redis), sending replies to a fixed partition, and a “return address” (attach the requestor’s address so the replier calls back over HTTP). I analysed the whole set, with code and a verdict, in the request-reply case study. The slides’ closing advice is worth keeping: if you must do it, package the client side as a shared library.
Exercises
Exercise 1: Review a money-transfer service
The transfer service does: @Transactional { debit account A; credit account B; kafkaTemplate.send("transfer-events", event) }. The notification consumer receives the event and sends a push notification, with auto-commit on. Point out every place things can diverge, duplicate or get lost, and propose a fixed design.
Solution to Exercise 1
Problems:
- Dual write: the event is sent inside the DB transaction. If the DB commit fails, the event is already out (a notification for a transfer that didn’t happen). If the send fails, it depends on the code: the transfer may roll back because of a Kafka error, or the error is swallowed and the event lost (
send()is async, so atry/catchdoesn’t catch broker errors). - No key: one account’s events scatter across partitions, losing order.
- Consumer: auto-commit with synchronous processing is at-least-once, so pushes can be duplicated; there’s no dedupe.
Fix:
- Write an
outboxrow (key =transfer_idoraccount_id) in the same transaction as the two balance updates; Debezium emits it totransfer-events. - Debezium’s producer:
acks=all, idempotence on. - Consumer: disable auto-commit, commit after the push is sent; dedupe on
event_idwith aprocessed_messagetable (unique) or a RedisSET NXwith TTL, since the push isn’t part of a DB transaction. Accept a small chance of a duplicate push if Redis loses the key: a push is a side effect that tolerates duplicates.
Exercise 2: Exactly-once for a reward-points pipeline
Pipeline: orders → service A computes points per order and writes points-delta → service B reads points-delta and updates a user_points table in PostgreSQL. Design it so each order adds points exactly once.
Solution to Exercise 2
- A (Kafka → Kafka): a Kafka transaction. A stable
transactional.idper instance; writepoints-deltaandsendOffsetsToTransactionforordersin the same transaction. Key =user_id. Or write A in Kafka Streams withexactly_once_v2. - B (Kafka → DB): read with
read_committed. In one PostgreSQL transaction:INSERT INTO points_applied(order_id) ...(uniqueorder_id); if it’s a duplicate, skip; otherwiseUPDATE user_points SET points = points + :delta WHERE user_id = :u. Commit the DB, then commit the offset. - A stronger variant for B: store the Kafka offsets in PostgreSQL itself, in the same transaction (a
consumer_offsets(topic, partition, offset)table), andseek()to them on startup. The DB becomes the single source of truth, with no per-order dedupe table needed; still keep the uniqueorder_idto guard against A re-emitting.
Remember
- Explain why “emit first” and “wrap in a DB transaction” both fail to fix a dual write.
- Draw the transactional outbox flow with Debezium and name its three costs.
- Tell an aggregation flow (Kafka transactions) from a sink flow (at-least-once + idempotent).
- List the producer and consumer settings that lose nothing, and where auto-commit is dangerous.
- Know the three conditions for end-to-end ordering and the things that silently break it.
Interview questions
Junior
What ordering does Kafka guarantee?
Only within a partition. To keep related messages in order, give them the same key so they land in the same partition.
A topic with many partitions has no total order. One partition does, but loses parallelism.
What is a dual write?
An operation that must write to two systems (say a DB and Kafka) with no shared transaction, so sometimes one side succeeds and the other doesn’t.
The standard fix is the transactional outbox: write the event to an outbox table in the same DB transaction, then CDC or a relay ships it to Kafka.
Mid
Why not just send to Kafka inside the DB transaction?
Because Kafka doesn’t take part in the DB transaction: the send succeeds, the commit fails, and the event is out while the DB rolls back.
The other way (send after commit), a crash between the two loses the event. On top of that, the DB transaction stays open while waiting on the broker.
What configuration makes producers and consumers lose nothing?
Producer: acks=all, idempotence on, retries within delivery.timeout.ms, topics with RF 3 and min.insync.replicas=2. Consumer: commit after processing, stable group.id.
Auto-commit is only safe when processing is synchronous in the poll loop. At-least-once comes with duplicates, so the consumer must be idempotent.
What are the downsides of the transactional outbox?
Added latency, an outbox table that grows unless cleaned, and only at-least-once, so consumers must tolerate duplicates.
With Debezium you can insert and delete in the same transaction because the connector reads the WAL. Monitor the connector: if it stops, events pile up in the WAL and the DB’s disk fills.
Senior
Kafka has exactly-once, so why must a consumer that writes to a DB still be idempotent?
Because Kafka’s exactly-once only covers writes to Kafka and offset commits; the DB write sits outside that transaction.
A consumer that dies after the DB write but before committing the offset reprocesses the record. Fix: a unique constraint on the message ID in the same transaction as the side effect, or store offsets in the DB itself.
Design it so an order’s events are processed in order end to end.
Key = order_id, an idempotent producer, and a consumer that processes each partition sequentially; add a version check in the consumer as a defense.
Avoid what breaks order: adding partitions, non-blocking retry topics, several producers for one order. If events go through an outbox, order follows DB commit order, so updates to one order must lock its row.
Outbox, CDC on tables, event sourcing: which do you pick to publish events between services?
The outbox: the event is a business contract I define, written atomically with the state, and it still uses CDC to ship.
CDC on tables suits internal data sync (search, warehouse) because it exposes the table schema. Event sourcing is only worth it when the event history itself is the business (a ledger, an audit trail), given its complexity.
Useful phrases for interviews
- “Saving to the database and then publishing is a dual write.”
- “There’s no transaction that spans both a database and a message broker.”
- “We write the event to an outbox table in the same transaction.”
- “Debezium does change data capture from the PostgreSQL WAL.”
- “Exactly-once in Kafka doesn’t cover side effects like emails or DB writes.”
- “For the sink flow, exactly-once means at-least-once delivery plus an idempotent consumer.”
- “The transactional ID lets the broker fence zombie producers.”
- “We only need a partial order per account, not a global one.”
- “Don’t order events across machines by wall-clock time.”
- “We deduplicate with a unique constraint on the request ID.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.