Idempotency series: 1. Concepts, APIs and the database · 2. Kafka and payment-gateway IPNs
Part 1 explained why every channel is at-least-once, and how to deduplicate at the API and database layers. This part applies the same ideas to the two places where duplicates happen most often in production: Kafka redelivering messages, and payment gateways calling your callback again. Afterwards you can write a consumer and an IPN handler that are harmless to repeat.
Read first: part 1, and transactions and locking for conditional updates.
1. Idempotent Kafka consumers
Key idea: an at-least-once consumer can receive the same message again, so record a “processed” marker in the database in the same transaction as the business change. A redelivered message hits the primary key and is skipped.
The order: receive a message → process it → commit the offset (the position read up to in the partition). A message is delivered again when:
- The consumer crashes after processing, before the offset commit.
- A rebalance (Kafka reassigning partitions among the group’s consumers) happens midway.
- The producer sends a duplicate: an outbox republishes, or the application retries.
CREATE TABLE processed_message (
consumer_group text NOT NULL,
message_id uuid NOT NULL,
processed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer_group, message_id)
);
@KafkaListener(topics = "wallet-transactions", groupId = "notification")
@Transactional // DataSource transaction: dedupe row and business change commit together
public void onFundTransfer(FundTransferEvent event) {
int fresh = jdbc.update("""
INSERT INTO processed_message (consumer_group, message_id) VALUES ('notification', ?)
ON CONFLICT DO NOTHING""", event.messageId());
if (fresh == 0) {
log.info("duplicate {}, skip", event.messageId());
return; // let the offset be committed as usual
}
notificationRepository.save(Notification.from(event));
}
// The offset is committed after the method returns (AckMode.RECORD/BATCH). A crash before that:
// the message comes back, the dedupe INSERT hits the key and is skipped. The notification is never written twice.
Two more things:
- Ordering: an old message can arrive after a newer one (retries, multiple partitions). Store a
versionorevent_timeon the target row and only update when the incoming one is newer:UPDATE ... WHERE id = ? AND version < ?. - Cleanup: keep
processed_messagelonger than topic retention + your maximum replay window, then purge in batches or partition the table by day.
Side effects outside the database: three ways to handle them
2. Kafka’s idempotent producer
Key idea: enable.idempotence=true lets the broker recognise a batch the producer resent after a lost ack, using a PID and sequence numbers. It only covers the client’s own internal retries and doesn’t replace deduplication in the consumer.
The problem: the broker writes a batch but the ack is lost, the producer retries, and the broker writes it twice. How idempotence solves it:
- Each producer gets a Producer ID (PID) and an epoch (the producer’s generation number); each batch sent to a partition carries an increasing sequence number.
- The broker remembers the last sequence for each (PID, partition): a batch with a sequence it has already seen is dropped (and still acknowledged as success); a batch that skips ahead is rejected instead of being written out of order.
- Requirements:
acks=all,retries > 0,max.in.flight.requests.per.connection ≤ 5(unacknowledged requests per connection). - Since Kafka 3.0, idempotence is on by default in the producer.
Its scope is narrower than many people assume:
- Only within one producer session: a restart gets a new PID; to survive restarts you need a
transactional.id. - Only the client’s own internal retries. If the application calls
send()twice (a try/catch, an outbox republish), those are two distinct messages. - Nothing about the consumer side.
How transactional.id works inside (coordinator, control markers, fencing) is in the producer internals post.
Why 5, the sample wallet service’s mistakes, and why an outbox republishes
The course’s sample wallet service gets this part right (acks=all, retries=30, enable.idempotence=true, max.in.flight=5), but two details undermine it:
- It sends messages without a key, so transactions for the same account land on different partitions and lose their order. Fix: use
debit_accountortransaction_idas the key. - It wraps
kafkaTemplate.send()in a try/catch. Butsend()is asynchronous: broker errors live in the returnedCompletableFuturerather than being thrown. Fix: handle the future’s outcome.
Why an outbox republishes: the relay (the process that reads the outbox table and pushes to Kafka) sends the message and only then marks the row as sent, so dying between the two steps means sending again. The “transactional outbox” section of Case study: a synchronous fraud check over Kafka has an outbox schema, a relay query using FOR UPDATE SKIP LOCKED, and the trade-off between a polling relay and CDC (Debezium reading the WAL or binlog). CDC has lower latency and doesn’t scan the table, at the cost of running Kafka Connect. Both are at-least-once. Why you need an outbox in the first place (dual writes) is in the dual write and exactly-once post; an outbox keeps one service consistent, across services you need a saga.
3. Case study: a payment gateway’s IPN
Key idea: the gateway reports the result by two routes, and only the signed IPN is trusted. The IPN handler checks the signature and amount, only transitions out of PENDING_PAYMENT, and replies “already confirmed” to a resend.
3.1. The flow
This section only covers the idempotency side. The full integration flow (building the URL, signing and verifying the HMAC signature, response codes) is in Integrating a VNPay-style payment gateway.
With a redirect-style gateway (VNPay and similar), the backend creates an order in PENDING_PAYMENT and returns a signed URL. The user pays on the gateway’s page, then the result comes back by two routes:
sequenceDiagram
participant B as Browser
participant FE as Frontend
participant BE as Backend
participant PG as Payment gateway
BE-->>FE: signed payment URL
FE->>PG: redirect, user pays
PG->>BE: server-to-server IPN (may be resent many times)
BE-->>PG: acknowledgement code
PG->>B: redirect to return URL
B->>FE: return URL with parameters
FE->>BE: ask for order status
BE-->>FE: status from the database
- IPN (Instant Payment Notification): the gateway calls the backend directly; this is the source of truth. Without a correctly formatted reply, the gateway resends it.
- Return URL: the browser comes back, for display only. Its parameters can be edited and the tab may be closed, so the frontend must ask the backend for the real status.
3.2. An idempotent IPN handler
@GetMapping("/payment/ipn")
public IpnReply ipn(@RequestParam Map<String, String> params) {
if (!signature.verify(params)) return IpnReply.INVALID_CHECKSUM; // 1. signature first
Order order = orders.findByTxnRef(params.get("txn_ref")).orElse(null);
if (order == null) return IpnReply.ORDER_NOT_FOUND;
if (order.amountMinor() != Long.parseLong(params.get("amount"))) return IpnReply.INVALID_AMOUNT;
if (order.isFinal()) return IpnReply.ALREADY_CONFIRMED; // 2. a resend: do nothing
boolean success = "00".equals(params.get("response_code"));
int changed = tx.execute(s -> success
? orders.markPaid(order.id(), params.get("gateway_txn_no")) // UPDATE ... WHERE status = 'PENDING_PAYMENT'
: orders.markFailed(order.id(), params.get("response_code")));
// changed == 0: a concurrent IPN just finished; still reply "already confirmed"
return changed == 1 ? IpnReply.CONFIRMED : IpnReply.ALREADY_CONFIRMED;
}
- Verify the signature and the amount before trusting any parameter. Compare amounts in minor units (VNPay-style gateways send the amount multiplied by 100).
- State machine (the set of states and allowed transitions): only transition out of
PENDING_PAYMENT; final states (PAID,FAILED,EXPIRED) are never overwritten. The conditional update referees two IPNs arriving at once. - Reply codes exactly as the gateway’s spec defines them for each case (already confirmed, bad checksum, wrong amount, order not found); the gateway uses them to decide whether to resend. A resend gets “already confirmed”, not an error, or the gateway keeps retrying forever.
- A unique constraint on the gateway’s transaction number in the payment/ledger table: the last line of defence against bugs in the code.
- The IPN never arrives: a periodic reconciliation job calls the gateway’s transaction-query API for orders stuck in
PENDING_PAYMENTfor more than N minutes. Once the payment window has passed, mark themEXPIREDand release the held nights/seats.
4. What to store, what to return, how long to keep it
Key idea: every layer needs a dedupe key, but what it stores and how long it keeps it differ. Temporary keys (an Idempotency-Key, a consumer marker) expire; keys that live in the real data (records, transactions) are kept forever.
Summary table for six layers: key, stored, returned, retention
| Layer | Dedupe key | Stored | Returned to the repeat | Kept for |
|---|---|---|---|---|
| API | (client_id, Idempotency-Key) |
Body hash, status, response status + body | The original response; 422 if the body differs; 409 if in progress |
24 hours to a few days |
| Creating a record | Business key (user_id, request_id) |
The record itself | The existing record | Forever (it’s data) |
| State transition | The current state | Nothing extra | The current outcome | n/a |
| Consumer | (consumer_group, message_id) or a business id |
A “processed” marker | Skip, commit the offset | > retention + replay window |
| Kafka producer | (PID, partition, sequence) | Kept by the broker | A successful ack | For the producer session |
| IPN | Order id + final state, unique gateway transaction number | Transaction, ledger entries | The “already confirmed” code | Forever |
Exercises
Exercise 1: Review a producer config
Task: a producer has acks=1, retries=10, max.in.flight.requests.per.connection=10, no enable.idempotence, and sends messages without keys. List the risks and fix them.
Solution to Exercise 1
Risks:
acks=1andmax.in.flight=10switch idempotence off; a retry after a lost ack writes a duplicate.- More than one in-flight request without idempotence lets retries reorder writes.
acks=1loses data if the leader dies before followers copy it.- Without keys there’s no per-account ordering.
Fix: acks=all, enable.idempotence=true, max.in.flight=5, key = account_id, topic min.insync.replicas=2, and the consumer still deduplicates on a business id.
Remember
- Explain why deduplication must share a transaction with the business change, and when the offset is committed.
- Know when deduplicating on
message_idfails, and why a business key is better. - State the PID + sequence mechanism, the three config requirements and the three limits of the idempotent producer.
- Draw the gateway’s two result routes and say which one is the source of truth.
- List the steps of an IPN handler, including what happens when the IPN never arrives.
Interview questions
Junior
Why not update the payment status from the return URL?
Because the return URL travels through the browser: its parameters can be edited, and it may never arrive.
If the user closes the tab before the redirect, it never comes. Use it only for display and to make the frontend ask the backend for the real status. The status is updated from the signed server-to-server IPN, plus a reconciliation job.
Mid
How do you build an idempotent Kafka consumer?
In one database transaction: insert a “processed” marker with ON CONFLICT DO NOTHING, and only if it was inserted, write the business change; commit the offset after the transaction commits.
The key is (consumer_group, message_id or a business id) in a processed_message table. A redelivered message hits the key and is skipped. Handle ordering with a version or event_time, and purge the dedupe table after retention plus the replay window.
How long should a processed_message table be kept?
Longer than the period during which a message can be redelivered: the topic’s retention plus your largest replay window.
The replay window is when you reset offsets during an incident. Anything shorter and a replay reprocesses old messages. Purge in batches or partition the table by day so deletes are cheap; if you deduplicate on a business key stored in the data table itself, there’s nothing to purge.
Senior
How does Kafka’s idempotent producer work, and what are its limits?
The broker remembers recent sequences per (PID, partition), drops duplicate batches and rejects out-of-order ones. It only covers the client’s own internal retries, within one producer session.
- The producer has a PID and epoch, and each batch to a partition carries a sequence number.
- It needs acks=all, retries > 0, max.in.flight ≤ 5. On by default since Kafka 3.0, but silently disabled if you set acks=1 without spelling out enable.idempotence.
- Limits: a restart gets a new PID (you need a transactional.id), it doesn’t stop the application calling send twice, and it says nothing about consumers.
How would you design a payment gateway IPN handler?
Verify the signature, find the order, compare the amount, and only then update through a state-machine conditional update. If the order is already final, reply “already confirmed” so the gateway stops resending.
- Only transition out of PENDING_PAYMENT; write the transaction and ledger entries in the same transaction; keep a unique constraint on the gateway’s transaction number.
- The return URL is display-only.
- A reconciliation job queries the gateway for overdue pending orders, marks them EXPIRED and releases held resources.
Kafka has exactly-once, so why must consumers still be idempotent?
Because Kafka’s exactly-once only covers the read-from-Kafka → process → write-to-Kafka loop; database writes, API calls and emails fall outside that transaction.
Exactly-once (idempotent producer, transactions, read_committed) commits offsets and output in the same Kafka transaction. A crash between the PostgreSQL write and the offset commit still reprocesses the message. Side effects outside Kafka still need deduplication in the database, in the same transaction as the business change.
Useful phrases for interviews
- “Our consumers are at-least-once, so every handler starts by inserting the message id into a dedupe table in the same transaction.”
- “Where we have a business key, we deduplicate on that rather than on the message id.”
- “Idempotence is on by default since Kafka 3.0, but we still set it explicitly so a conflicting acks setting fails fast.”
- “The idempotent producer only protects against its own retries; it can’t tell that the application sent the same thing twice.”
- “We treat the IPN as the source of truth; the return URL only drives the UI.”
- “A resent IPN for an order that’s already final gets an ‘already confirmed’ reply, never an error.”
- “A reconciliation job queries the gateway for orders stuck in pending and expires them after the payment window.”
- “For side effects outside the database, we either accept duplicates or route them through a task table with its own idempotency key.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.