Skip to content
notes.vinhnq.com

Search

System DesignSenior17 min read

Kafka in practice (2): dual writes, exactly-once and message ordering

Review 12 cards
On this page

Kafka problems and patterns series: 1. Partition count, large messages, lag and long-running tasks · 2. Dual writes, exactly-once and message ordering · 3. Case study: fraud detection with Kafka and Redis

The three “hardest” questions of any messaging system: how to write to the DB and publish an event without them drifting apart, how to process each message exactly once, and how to keep order. This post answers each with Kafka’s concrete mechanisms and their limits. The idempotent consumer and idempotent producer are covered in Idempotency part 2; here they’re just wired into the bigger picture.

1. Dual writes: two kinds

Key idea: a dual write is a process that must write to two places and needs both to succeed or neither. Two writes to Kafka are covered by Kafka transactions; a DB write plus a Kafka write has no shared transaction.

  • United dual write: both writes go to Kafka, e.g. topics orders and audit. Early Kafka had no way to roll back a successful write; since 0.11 there are Kafka transactions.
  • Separate dual write: save state to a DB, then emit an event to another system (Kafka, an email service…). The two systems aren’t linked, so there’s no shared transaction. This is the common case.

Example, paying a bill: deduct the balance in the DB, then emit BillPaid. If the service dies between the two steps, the balance is deducted but the event never goes out: the bill is never marked paid. The failure is rare, but the impact is severe and it’s very hard to detect and fix.

2. Two fixes that don’t work

Key idea: swapping the order or wrapping both in a DB transaction only moves the failure somewhere else.

  • Emit first, save second: die in between and the event is out while the state never changed. The receiver marks a bill paid that wasn’t.
  • Wrap in a DB transaction (BEGIN, save, emit, COMMIT): the emit succeeds but COMMIT fails, so the DB rolls back while the event is already out. On top of that, the DB transaction stays open while waiting on Kafka.

3. Four ways out, and the transactional outbox

Key idea: there are four patterns; the outbox is the most straightforward and effective one for DB + Kafka.

The outbox only keeps one service’s DB and Kafka in step; when several services must change data you need a saga, see distributed transactions: 2PC, saga, outbox.

Pattern Idea Verdict
Listen to yourself only publish the event; the service itself consumes it to update its DB state updates late, reads right after writes miss it; rarely practical
Event sourcing events are the source of truth, state is rebuilt from them powerful but complex, hard error handling, fits only some problems
CDC on business tables Debezium reads the balance table’s WAL and emits changes simple, but the event is a “row change”, not a business event you define
Transactional outbox write the state and a row in an outbox table in the same transaction; CDC reads the outbox and emits simple, you control the event’s content; costs latency and is at-least-once
flowchart LR
  H[Handler] -->|1 BEGIN: UPDATE balance + INSERT outbox, COMMIT| DB[(PostgreSQL)]
  DB -->|2 WAL stream| CDC[Debezium on Kafka Connect]
  CDC -->|3 emit event| T[(topic bill-events)]

Keep the outbox table simple, and store already-encoded bytes so the connector can ship them as-is without understanding them:

CREATE TABLE outbox (
  id         uuid PRIMARY KEY,
  topic      varchar(255) NOT NULL,
  key        bytea,          -- e.g. account_id, decides partition and order
  value      bytea NOT NULL, -- the serialized event (Avro/Protobuf/JSON)
  headers    bytea,
  created_at timestamptz NOT NULL DEFAULT now()
);

Tooling: Debezium as the connector, Kafka Connect to run it, Strimzi to manage it on Kubernetes.

What the outbox costs:

  • Latency: one extra step reading the WAL, usually tens to hundreds of ms.
  • The outbox table grows over time and slows the DB if you don’t clean it.
  • At-least-once: the connector may re-emit after a restart, so consumers must be idempotent.
Running an outbox: ordering, cleanup, polling relay
  • Ordering: Debezium reads the WAL in commit order and uses the key column as the Kafka key, so one account’s events land in one partition in commit order. Two concurrent transactions for the same account are ordered by commit, so lock the account row if business order matters.
  • Cleanup: Debezium reads the WAL, not the table, so you can INSERT and then DELETE in the same transaction: the WAL still has the insert to emit, and the table stays nearly empty. Alternatives: delete rows older than N hours in batches, or partition the table by day and drop old partitions.
  • Debezium Outbox Event Router: a ready-made SMT (single message transform) that reads columns like aggregatetype, aggregateid, payload and routes to the matching topic.
  • Polling relay (no CDC): a job runs SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED, sends to Kafka, then sets published_at. Easier to set up, but more DB load, latency equal to the poll interval, and several parallel relays can reorder events. Comparison in the request-reply case study.
  • MySQL/TiDB: Debezium reads MySQL’s binlog; TiDB has TiCDC, which ships straight to Kafka.

4. Exactly-once: two kinds of flow

Key idea: in an event pipeline, flaky networks, consumer crashes and retries all create duplicates. Exactly-once doesn’t come from one setting but from combining mechanisms, and the combination depends on whether the flow is Kafka → Kafka or has side effects.

flowchart LR
  A[(topic A)] --> S0[stage 0]
  S0 --> B[(topic B)]
  B --> S1[stage 1]
  S1 -->|side effect| DB[(DB)]
  • Aggregation flow (topic A → stage 0 → topic B): read from Kafka, write to Kafka. Kafka transactions handle it: write the output and commit the input offsets in one transaction, and downstream consumers read with isolation.level=read_committed.
  • Sink flow (topic B → stage 1 → DB): has side effects outside Kafka. Kafka transactions don’t cover it. This is the common case.

For the sink flow, the formula is: exactly-once = at-least-once (lose nothing) + an idempotent consumer (drop duplicates). The slides say “at-least-once + at-most-once”; more precisely, lose nothing at every hop, and make every repeated processing step have no extra effect.

Kafka transactions for consume-transform-produce
producer.initTransactions(); // a stable transactional.id per instance
while (running) {
  var records = consumer.poll(Duration.ofMillis(200));
  if (records.isEmpty()) continue;
  producer.beginTransaction();
  try {
    for (var r : records) producer.send(new ProducerRecord<>("topic-b", r.key(), transform(r.value())));
    producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
    producer.commitTransaction();
  } catch (ProducerFencedException e) {
    throw e; // another instance has our transactional.id: stop for good
  } catch (KafkaException e) {
    producer.abortTransaction(); // then seek back to the committed offsets and reprocess
  }
}
  • Fencing: a stable transactional.id lets the broker block “zombies” (old instances not quite dead) using an epoch.
  • Downstream consumers must set isolation.level=read_committed, or they still see data from aborted transactions.
  • Limits: it only covers writes to Kafka and offsets. API calls, DB writes or emails inside transform can still repeat on abort and reprocessing. Kafka Streams turns this on with processing.guarantee=exactly_once_v2.

4.1. Lose nothing on the producer side

Key idea: producers lose messages when a send fails on the network without a retry, or the leader dies before followers copy. Fix it with retries, acks=all and idempotence.

  • retries: the slides recommend 30. Since Kafka 2.1 the default is Integer.MAX_VALUE, and the real retry window is bounded by delivery.timeout.ms (default 120 seconds). Tune that instead of counting attempts.
  • acks=all: the leader acknowledges only after every in-sync replica has the record. Pair it with min.insync.replicas=2 on RF 3.
  • enable.idempotence=true (default since 3.0): the broker drops retry duplicates and keeps order within the partition.

4.2. Lose nothing on the consumer side

Key idea: consumers lose messages by committing before processing finishes, or by using a new group on every start. Commit after processing, and keep group.id stable.

The slides recommend enabling auto-commit. That’s only right when processing runs synchronously in the poll loop: auto-commit commits the previous batch’s offsets on the next poll(). If you hand records to another thread and keep polling, auto-commit commits records that aren’t processed yet. Spring Kafka disables auto-commit by default and lets the container commit after the listener (see the configuration post).

4.3. Idempotent consumer

Key idea: at-least-once guarantees duplicates, so a consumer with side effects must be idempotent: either overwrite without changing the outcome, or block duplicates with a unique constraint.

  • Harmless overwrite: UPDATE order SET status = 'PAID' run twice is still correct. Watch for an old event arriving late and overwriting a newer one; add a version condition.
  • Deduplicate: every message has an ID (a UUID or a business ID). The transactions table has a unique constraint on request_id; a duplicate insert is caught and skipped, in the same transaction as the side effect.

Details (a processed_message table, how long to keep it, duplicates from a producer generating a new message) are in Idempotency part 2.

5. Message ordering

Key idea: Kafka only guarantees order within one partition. Most problems need only a partial order (per player, per order), so use the key to put related messages in one partition.

Order matters in many places: in a game, the sequence “up, jump, turn left” decides the state; for an order, “placed, paid, shipped, delivered” is its lifecycle.

  • Total order: you know the exact order of every event. In Kafka you only get it with one partition, giving up parallelism.
  • Partial order: you only know the order of related events. A distributed system without a global clock can only offer this, and it’s usually enough: one player’s actions don’t relate to another’s.

Don’t order by physical clocks (created_at from many machines): clocks drift and can go backwards. Use sequence numbers, versions, or offsets within a partition.

End-to-end ordering needs all three:

  1. Producer: same key → same partition, and enable.idempotence=true so retries don’t reorder (with idempotence off and max.in.flight.requests.per.connection > 1, a later batch can land before one being retried).
  2. Broker: order within a partition is offset order.
  3. Consumer: one consumer per partition in the group, processing sequentially per key. Dumping records into a thread pool loses order.
Things that silently break ordering
  • Adding partitions: keys move, and a key’s new events can be processed before old ones still sitting in the old partition.
  • Non-blocking retry topics (@RetryableTopic): the failed record takes a detour and later records with the same key overtake it. For order-sensitive events, use blocking retries then a DLT.
  • Several producers for one key (two instances emitting events for one order): the order between them is arrival order at the broker, not business order. Give each aggregate one owner, or attach a version.
  • Defense at the consumer: store the last applied version per aggregate and drop events with a version less than or equal to it. That handles both duplicates and reordering.

6. Request-reply over Kafka

Key idea: Kafka is a one-way road; asking and answering needs a reply topic, an ID to match pairs and somewhere to wait for the result. It can be done, but usually shouldn’t be.

The slides on this walk through: separate request and reply topics; message_id and original_message_id to match them; a local pool for replies (a Guava cache with TTL) with polling or a CompletableFuture callback; the multi-instance problem (a group per instance is wasteful, a shared group delivers replies to the wrong instance); a shared reply store (Redis), sending replies to a fixed partition, and a “return address” (attach the requestor’s address so the replier calls back over HTTP). I analysed the whole set, with code and a verdict, in the request-reply case study. The slides’ closing advice is worth keeping: if you must do it, package the client side as a shared library.

Exercises

Exercise 1: Review a money-transfer service

The transfer service does: @Transactional { debit account A; credit account B; kafkaTemplate.send("transfer-events", event) }. The notification consumer receives the event and sends a push notification, with auto-commit on. Point out every place things can diverge, duplicate or get lost, and propose a fixed design.

Solution to Exercise 1

Problems:

  1. Dual write: the event is sent inside the DB transaction. If the DB commit fails, the event is already out (a notification for a transfer that didn’t happen). If the send fails, it depends on the code: the transfer may roll back because of a Kafka error, or the error is swallowed and the event lost (send() is async, so a try/catch doesn’t catch broker errors).
  2. No key: one account’s events scatter across partitions, losing order.
  3. Consumer: auto-commit with synchronous processing is at-least-once, so pushes can be duplicated; there’s no dedupe.

Fix:

  • Write an outbox row (key = transfer_id or account_id) in the same transaction as the two balance updates; Debezium emits it to transfer-events.
  • Debezium’s producer: acks=all, idempotence on.
  • Consumer: disable auto-commit, commit after the push is sent; dedupe on event_id with a processed_message table (unique) or a Redis SET NX with TTL, since the push isn’t part of a DB transaction. Accept a small chance of a duplicate push if Redis loses the key: a push is a side effect that tolerates duplicates.

Exercise 2: Exactly-once for a reward-points pipeline

Pipeline: orders → service A computes points per order and writes points-delta → service B reads points-delta and updates a user_points table in PostgreSQL. Design it so each order adds points exactly once.

Solution to Exercise 2
  • A (Kafka → Kafka): a Kafka transaction. A stable transactional.id per instance; write points-delta and sendOffsetsToTransaction for orders in the same transaction. Key = user_id. Or write A in Kafka Streams with exactly_once_v2.
  • B (Kafka → DB): read with read_committed. In one PostgreSQL transaction: INSERT INTO points_applied(order_id) ... (unique order_id); if it’s a duplicate, skip; otherwise UPDATE user_points SET points = points + :delta WHERE user_id = :u. Commit the DB, then commit the offset.
  • A stronger variant for B: store the Kafka offsets in PostgreSQL itself, in the same transaction (a consumer_offsets(topic, partition, offset) table), and seek() to them on startup. The DB becomes the single source of truth, with no per-order dedupe table needed; still keep the unique order_id to guard against A re-emitting.

Remember

  • Explain why “emit first” and “wrap in a DB transaction” both fail to fix a dual write.
  • Draw the transactional outbox flow with Debezium and name its three costs.
  • Tell an aggregation flow (Kafka transactions) from a sink flow (at-least-once + idempotent).
  • List the producer and consumer settings that lose nothing, and where auto-commit is dangerous.
  • Know the three conditions for end-to-end ordering and the things that silently break it.

Interview questions

Junior

What ordering does Kafka guarantee?

Only within a partition. To keep related messages in order, give them the same key so they land in the same partition.

A topic with many partitions has no total order. One partition does, but loses parallelism.

What is a dual write?

An operation that must write to two systems (say a DB and Kafka) with no shared transaction, so sometimes one side succeeds and the other doesn’t.

The standard fix is the transactional outbox: write the event to an outbox table in the same DB transaction, then CDC or a relay ships it to Kafka.

Mid

Why not just send to Kafka inside the DB transaction?

Because Kafka doesn’t take part in the DB transaction: the send succeeds, the commit fails, and the event is out while the DB rolls back.

The other way (send after commit), a crash between the two loses the event. On top of that, the DB transaction stays open while waiting on the broker.

What configuration makes producers and consumers lose nothing?

Producer: acks=all, idempotence on, retries within delivery.timeout.ms, topics with RF 3 and min.insync.replicas=2. Consumer: commit after processing, stable group.id.

Auto-commit is only safe when processing is synchronous in the poll loop. At-least-once comes with duplicates, so the consumer must be idempotent.

What are the downsides of the transactional outbox?

Added latency, an outbox table that grows unless cleaned, and only at-least-once, so consumers must tolerate duplicates.

With Debezium you can insert and delete in the same transaction because the connector reads the WAL. Monitor the connector: if it stops, events pile up in the WAL and the DB’s disk fills.

Senior

Kafka has exactly-once, so why must a consumer that writes to a DB still be idempotent?

Because Kafka’s exactly-once only covers writes to Kafka and offset commits; the DB write sits outside that transaction.

A consumer that dies after the DB write but before committing the offset reprocesses the record. Fix: a unique constraint on the message ID in the same transaction as the side effect, or store offsets in the DB itself.

Design it so an order’s events are processed in order end to end.

Key = order_id, an idempotent producer, and a consumer that processes each partition sequentially; add a version check in the consumer as a defense.

Avoid what breaks order: adding partitions, non-blocking retry topics, several producers for one order. If events go through an outbox, order follows DB commit order, so updates to one order must lock its row.

Outbox, CDC on tables, event sourcing: which do you pick to publish events between services?

The outbox: the event is a business contract I define, written atomically with the state, and it still uses CDC to ship.

CDC on tables suits internal data sync (search, warehouse) because it exposes the table schema. Event sourcing is only worth it when the event history itself is the business (a ledger, an audit trail), given its complexity.

Useful phrases for interviews

  • “Saving to the database and then publishing is a dual write.”
  • “There’s no transaction that spans both a database and a message broker.”
  • “We write the event to an outbox table in the same transaction.”
  • “Debezium does change data capture from the PostgreSQL WAL.”
  • “Exactly-once in Kafka doesn’t cover side effects like emails or DB writes.”
  • “For the sink flow, exactly-once means at-least-once delivery plus an idempotent consumer.”
  • “The transactional ID lets the broker fence zombie producers.”
  • “We only need a partial order per account, not a global one.”
  • “Don’t order events across machines by wall-clock time.”
  • “We deduplicate with a unique constraint on the request ID.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.