Kafka and message brokers series: 1. Message brokers: four problems with request-response and how to choose a broker · 2. Why Kafka is fast and durable: the log, zero-copy, batching and replication · 3. Tuning Kafka for your goal: throughput, latency, durability, availability · 4. Case study: a synchronous fraud check over Kafka and Redis — good idea or not? · 5. CCDAK review: 40 Kafka questions worth remembering
This post reviews a wallet service (Spring Boot 3.2) that sends a fraud-check command to Kafka and then polls Redis for the result while the user waits. Afterwards you’ll be able to point out its eight problems, know the alternatives, and answer “should we do request-reply over Kafka?”. Read first: Why Kafka is fast and durable (partitions, keys, consumer groups), Tuning Kafka for your goal (auto-commit) and Idempotency part 2 (idempotent consumers, outbox).
1. The problem and its constraints
Key idea: before money moves, the wallet must ask fraud-detection whether the payment looks suspicious. Four constraints make it hard:
- Fraud-detection only takes input from Kafka; it has no HTTP API.
- The user is waiting: a response within 1–2 seconds, leaving a few hundred milliseconds for the fraud check.
- The wallet runs several instances: the instance that sends the command isn’t necessarily the one that receives the result.
- Real money: we need a decision for when the fraud check is slow or down: let it through (fail open) or block it (fail closed).
The original code has two endpoints. POST /api/v1/payments fakes a transaction and fires a FUND_TRANSFER event (fire-and-forget). POST /api/v1/payments/sync sends a fraud-check command and waits; this post is mostly about that one.
2. The message envelope and topic naming
Key idea: every message is wrapped in an envelope (a wrapper that carries metadata) with meta, message_code and payload. A good start, but it lacks schema_version, reply_to and deadline.
Envelope fields, topic naming and where metadata belongs
meta:message_id(UUID),original_message_id(the id of the message this one answers, null otherwise),type(EVENT,COMMAND…),service_id(the sender),timestamp,auto_retry.message_code: the business type, e.g.FUND_TRANSFER.payload: the transaction (id, PAYMENT/TOPUP type, debit account, credit account, amount, status, creation time).
Topic naming: <tenant>-<owning service>-<name>-<environment>, e.g. ronin-wallet-transactions-dev, ronin-fraud-detection-result-dev. Putting the environment in the topic name only makes sense when environments share a cluster; separate clusters are safer.
The envelope already has a dedup id, a correlation id and a message type. A senior reviewer would add:
schema_versionor a schema registry, so consumers know which payload version they’re reading.reply_toanddeadline: where to reply, and after when a reply is pointless (otherwise fraud keeps scoring commands the caller gave up on).- An existing standard: CloudEvents (
id,source,type,time, and a mapping to Kafka headers).
3. Events and commands are different things
Key idea: an event reports something that happened and nobody replies; a command asks one specific party to do something and usually needs a result. The original code sends both to the same topic, ronin-wallet-transactions-dev, told apart by meta.type.
| Event | Command | |
|---|---|---|
| Example | FundTransferred |
CheckFraud |
| Meaning | something happened, past tense, immutable | “please do this” |
| Receiver | the sender neither knows nor cares who listens | one owner, can reject it |
| Reply | nobody replies | usually needs a result |
4. The original design: request-reply with Redis as the matchmaker
Key idea: the reply goes to an arbitrary instance in the ronin-wallet group, so Redis is a shared correlation store: whoever receives the reply writes it, the waiting instance polls. The idea is sound; the execution isn’t.
sequenceDiagram
participant U as User
participant W1 as Wallet instance 1
participant K as Kafka
participant FD as Fraud-detection
participant W2 as Wallet instance 2
participant R as Redis
U->>W1: POST /payments/sync
W1->>K: COMMAND (message_id = m1)
W1->>W1: sleep 5 ms, then poll Redis with backoff
K->>FD: COMMAND m1
FD->>K: RESULT (original_message_id = m1)
K->>W2: RESULT m1 (consumer group ronin-wallet)
W2->>R: SET wallet:m1 = result, TTL
W1->>R: GET wallet:m1 (result found)
W1-->>U: 200 + result
W1-)R: DEL wallet:m1 (async)
The Redis key is <service_id>:<original_message_id>. The wait loop, trimmed from KafkaConnector.sendSync:
Thread.sleep(5); // wait for consumer receiving and processing (tuning)
int leftTime = cacheTimeout;
...
while (leftTime >= 0) {
var response = getResponseFromCache(...);
if (response != null) return response;
var delay = (int) (initDelay * Math.pow(BACKOFF_MULTIPLIER, attempt++));
leftTime -= delay;
Thread.sleep(delay);
}
return null;
where cacheTimeout comes from spring.redis.timeout: 300000, initDelay = 20, and the multiplier is 2.
5. Review: eight problems
Key idea: the worst two are a five-minute wait on a servlet thread and a timeout that turns into a fail-open; the rest are about topics, keys, send-error handling, dual writes, serialisation and lost replies.
5.1. A five-minute wait on a servlet thread, with an uncapped backoff
Key idea: the wait budget is 300,000 ms and the 20 ms × 2^n delay has no cap. Even the code comment admits “biz timeout should be different from cache infra timeout”.
Replaying the polling schedule (Java 21) shows three consequences:
- Thread exhaustion (running out of threads): Tomcat has 200 threads by default. At just ~1 request per second waiting 5.5 minutes, every thread is taken after a bit over three minutes, and unrelated endpoints die too.
- Doubled latency: the gap between polls keeps growing; a reply ready at 2.6 s is only seen at 5.1 s.
- A blind spot at the end: the last poll is at 163.8 s, then the thread sleeps another 163.8 s and returns
nullwithout polling again. A reply arriving in that window is ignored even though it sits in Redis.
The simulated poll schedule and the Spring Boot 3.2 config detail
poll at ms: 5 25 65 145 305 625 1265 2545 5105 10225 20465 40945 81905 163825
returns null at ms: 327665 (polls=14)
reply ready at 700 ms -> seen at 1265 ms
reply ready at 1300 ms -> seen at 2545 ms
reply ready at 2600 ms -> seen at 5105 ms
reply ready at 170000 ms -> seen at never (timeout 327665 ms)
The project is on Spring Boot 3.2, where spring.redis.* was renamed to spring.data.redis.*. So spring.redis.timeout no longer configures the Redis client at all; it’s only read via @Value as the wait budget.
5.2. A silent fail-open when there is no result
Key idea: the caller can’t tell “not fraud”, “fraud” and “don’t know” apart, because every failure becomes null.
KafkaMessage<FraudCheckResult> response = kafkaConnector.sendSync(transactionTopic, message);
return response.getPayload();
} catch (Exception e) {
log.error("Failed to produce the message to topic: " + transactionTopic);
return null; // TODO: return the default value
5.3. Commands and events in one topic
Key idea: besides the reasons in section 3, fraud-detection must read the whole (high-volume) event stream to pick out a few commands, so event lag delays the commands someone is waiting for.
5.4. No message key
Key idea: kafkaTemplate.send(topic, request) has no key, so two payments of the same account can land in two partitions and be processed out of order. Use debit_account as the key.
Without a key the producer uses the sticky partitioner (fill one partition until the batch is full). Fraud detection, though, needs per-account order. With a key, watch for hot keys (one key getting too much traffic) from large merchant accounts.
5.5. send() is asynchronous, so try/catch misses broker errors
Key idea: send failures live in the future; they aren’t thrown right away. The fire-and-forget endpoint can lose an event while the log says “Produced a message”; the synchronous endpoint waits the full five minutes after a failed send.
5.6. Dual write: an event for a transaction with no DB transaction
Key idea: writing the DB and then sending to Kafka without a shared transaction is a dual write: one side can succeed while the other fails. The standard fix is the transactional outbox (section 7).
The /payments endpoint currently only fakes a transaction, with no database. Once a real database is involved, there are two ways to fail:
- DB write succeeds, Kafka send fails: the event is lost, accounting never finds out.
- Kafka send succeeds, DB rolls back: an event about a transaction that doesn’t exist.
5.7. Kryo without class registration, and the cost of polling Redis
Key idea: kryo.setRegistrationRequired(false) lets Kryo instantiate any class on the classpath on read, the same family of vulnerabilities as Java deserialisation. Use JSON with fixed types, or require class registration.
Values are written with writeClassAndObject, so the class name is in the data; anyone who can write to Redis can make the service instantiate arbitrary classes. The format is also tied to the class layout: a rolling deploy that changes a field leaves old and new instances unable to read each other’s data.
On load: every waiting request issues up to 14 GETs just to ask “is it there yet?”. Polling-free options: BLPOP with a timeout, Pub/Sub on a per-instance channel, or keyspace notifications.
5.8. Auto-commit and a Redis write without retries
Key idea: the reply consumer runs with enable-auto-commit: true and pushResponseToCache has no retry (// TODO: add retry logic), so replies can be lost. That matters because a timeout is a fail-open here (5.2): one lost reply is one unchecked payment.
Why replies get lost, and the service_id key bug in the contract
- Auto-commit commits offsets periodically, not tied to the Redis write having finished.
- Redis write fails: the default error handler retries a few times, then skips the record.
- Process dies after the commit but before the Redis write: the reply is lost.
A contract bug: the write key uses the service_id of the reply, the read key the service_id of the request. If fraud-detection sets service_id to its own name (natural for a “sender” field), the keys never match and every request times out. The contract must say the reply carries a reply_to/correlation_id issued by the caller, and the responder only copies it.
6. The alternatives
Key idea: wait for replies without polling: keep a correlationId → future map and complete it when the reply arrives (A, B). If the other side or the product can change, HTTP with a timeout (D) or an async UX (E) is simpler still.
| Option | Idea | Pros | Cons |
|---|---|---|---|
A. ReplyingKafkaTemplate |
Spring puts correlationId + replyTopic in headers, keeps a map of pending futures |
No polling, proper timeout, little code | Each instance must receive its own replies |
| B. Per-instance reply partition + future map | Same as A, hand-written with kafka-clients | You understand the mechanics, no Spring needed | You manage the instance → partition mapping |
C. Redis BLPOP instead of polling |
Keep the original architecture, block with a timeout | Small change | Still blocks a thread, Redis still on the critical path |
| D. HTTP/gRPC with a timeout | Fraud-detection exposes a scoring API | Simplest; circuit breakers, retries, tracing for free | Fraud-detection must change; temporal coupling |
| E. Asynchronous UX | Return 202 + PENDING, push the result via push/webhook/polling |
No blocked threads, tolerates a slow fraud check | Changes the product flow, adds a state machine |
6.1. Option B: my own version, run for real
Key idea: each instance owns one partition of the reply topic and keeps a ConcurrentHashMap<correlationId, CompletableFuture>. On a three-broker cluster: p99 23 ms, the 2 s timeout is an explicit decision, and no memory leaks.
- The partition is fixed per instance (e.g. from its StatefulSet ordinal) and read with
assign(), no consumer group needed. - The command carries
correlation-idandreply-partitionheaders; fraud-detection copiescorrelation-idand replies to that partition.
Option B code, run results and the CompletionException bug
The core:
CompletableFuture<String> check(String txId, String debitAccount, long amount, Duration timeout) {
String correlationId = UUID.randomUUID().toString();
var future = new CompletableFuture<String>();
pending.put(correlationId, future);
var record = new ProducerRecord<>(COMMANDS, debitAccount, txId + ":" + amount); // key = account
record.headers().add("correlation-id", correlationId.getBytes(UTF_8));
record.headers().add("reply-partition", String.valueOf(replyPartition).getBytes(UTF_8));
producer.send(record, (md, ex) -> { if (ex != null) future.completeExceptionally(ex); });
return future.orTimeout(timeout.toMillis(), TimeUnit.MILLISECONDS)
.whenComplete((v, ex) -> pending.remove(correlationId)); // no map leak
}
private void listen() { // one thread per instance
try (var c = consumer(null)) { // no group.id: assign the partition ourselves
var tp = new TopicPartition(REPLIES, replyPartition);
c.assign(List.of(tp));
c.seekToEnd(List.of(tp));
while (running) {
for (var r : c.poll(Duration.ofMillis(50))) {
var f = pending.get(header(r, "correlation-id"));
if (f != null) f.complete(r.value()); // late or unknown reply: ignore
}
}
}
}
The caller makes an explicit decision when time runs out:
String verdict = client.check("tx-1", "acc-1", 100_000, Duration.ofSeconds(2))
.exceptionally(ex -> {
Throwable cause = ex instanceof CompletionException ? ex.getCause() : ex;
return cause instanceof TimeoutException ? "TIMEOUT->HOLD" : "ERROR->HOLD";
})
.join();
Results on a three-broker KRaft cluster (cp-kafka 7.5.3, kafka-clients 4.1.1, acks=all, a laptop):
200 sequential checks: OK=180 REVIEW=20 p50=5 ms p99=23 ms max=26 ms
200 concurrent checks done in 60 ms, pending map size=0/0
slow responder: verdict=TIMEOUT->HOLD after 2004 ms, pending=0
late reply dropped, pending=0
No polling, the reply is seen as soon as it lands, and late replies are dropped without leaking memory. These are local numbers, not production ones.
6.2. Option A: ReplyingKafkaTemplate in Spring Boot 3
Key idea: Spring handles the future map and correlation headers for you; on the fraud-detection side @KafkaListener + @SendTo is enough. What’s left is choosing how replies reach the right instance.
ReplyingKafkaTemplate configuration (not run, per the Spring Kafka 3.x API)
@Bean
ReplyingKafkaTemplate<String, FraudCheckCommand, FraudVerdict> fraudTemplate(
ProducerFactory<String, FraudCheckCommand> pf,
ConcurrentKafkaListenerContainerFactory<String, FraudVerdict> factory,
@Value("${app.instance-id}") String instanceId) {
var replies = factory.createContainer("fraud-check.replies");
replies.getContainerProperties().setGroupId("wallet-replies-" + instanceId); // one group per instance
var template = new ReplyingKafkaTemplate<>(pf, replies);
template.setDefaultReplyTimeout(Duration.ofSeconds(2));
template.setSharedReplyTopic(true); // the instance also receives other instances' replies and ignores them
return template;
}
public FraudVerdict check(FraudCheckCommand cmd) throws Exception {
var record = new ProducerRecord<String, FraudCheckCommand>("fraud-check.commands", cmd.debitAccount(), cmd);
return fraudTemplate.sendAndReceive(record).get(2, TimeUnit.SECONDS).value();
}
The listener container factory on the fraud-detection side needs a reply template; Spring copies the correlation header for you.
7. Transactional outbox: once the transaction is really written to a database
Key idea: write the business data and the message to send into the same database in one transaction, then let a separate process (the relay) move outbox rows to Kafka. The result is at-least-once, so consumers must deduplicate by id.
An example on PostgreSQL:
CREATE TABLE outbox (
id uuid PRIMARY KEY,
aggregate_id text NOT NULL, -- used as the message key, e.g. debit_account
topic text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz
);
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;
-- inside the business transaction
BEGIN;
INSERT INTO wallet_transaction (...) VALUES (...);
INSERT INTO outbox (id, aggregate_id, topic, payload) VALUES (...);
COMMIT;
-- relay (several copies can run in parallel)
SELECT id, aggregate_id, topic, payload FROM outbox
WHERE published_at IS NULL ORDER BY created_at LIMIT 100
FOR UPDATE SKIP LOCKED;
-- send to Kafka, wait for the acks, then:
UPDATE outbox SET published_at = now() WHERE id = ANY(?);
- The relay can send and then die before the
UPDATE, so duplicates happen: consumers deduplicate byid(see Idempotency part 2). - The polling relay can be replaced by CDC (change data capture: Debezium reading the PostgreSQL WAL or the MySQL binlog), compared in the block below.
- MySQL 8.0+ supports
FOR UPDATE SKIP LOCKEDtoo.
Further reading: dual writes and outbox with Debezium; why outbox doesn’t replace a saga when several services must change data; and an asynchronous fraud detection design, with the detector behind Kafka, off the payment path.
8. Verdict: good idea or not?
Key idea: not as a default. A synchronous call over Kafka adds two broker hops, a consumer group on each side and a correlation store: you get every downside of RPC without the upside of messaging.
Proposed, in order of preference:
- If fraud-detection can change: HTTP/gRPC, a 300 ms timeout, a circuit breaker, an explicit
UNKNOWNpolicy. Still publishFundTransferredthrough the outbox so fraud-detection can learn and score after the fact. - If it can’t: option A/B, a dedicated command topic, the account as the key, a 1–2 s timeout, no waiting on servlet threads (or Java 21 virtual threads,
spring.threads.virtual.enabled=true), timeout →PENDING_REVIEW. - If the business accepts it: return
202, keep the paymentPENDING, push the result to the app. The most robust choice when fraud-detection can take several seconds.
Comparison table: how the original and proposed designs fail
| Failure mode | Original design | Proposed design |
|---|---|---|
| Fraud check takes 10 s | Thread stuck, empty 200 after 5.5 minutes | 2 s timeout → PENDING_REVIEW |
| Fraud-detection is down | Every request hangs, thread pool exhausted | Circuit breaker opens, UNKNOWN policy applied immediately |
| Reply lost (crash after commit) | Fail open | Timeout → PENDING_REVIEW, re-checked later |
| Sending the command fails | Nobody notices, still waiting | Future fails immediately |
| Duplicate reply | Overwrites the Redis key (harmless) | Future already done, second reply dropped |
| DB write succeeds, Kafka fails | Event lost | Outbox resends |
Exercises
Exercise 1: Fix the wait loop without changing the architecture
Keep Redis as the correlation store. Rewrite the wait so that the budget is 1,500 ms, the gap between checks never exceeds 50 ms, there is always a final check before reporting a timeout, and no servlet thread is held beyond the budget. Work out the maximum number of Redis commands per request.
Solution to exercise 1: BLPOP, or capped polling
The smallest change is to stop polling and block with a timeout: the instance that receives the reply runs RPUSH reply:<id> <value> then EXPIRE reply:<id> 10; the waiting instance calls BLPOP reply:<id> 1.5. One Redis command per request, the reply is seen the moment it exists, and the timeout is exact.
If you must poll: delay min(10 × 2^n, 50) ms, deadline = start + 1500, loop while (now < deadline), and one more GET after the loop. Maximum commands ≈ 4 (at 0, 10, 30, 70 ms) + (1,500 − 70)/50 ≈ 29 + 1 ≈ 34.
The thread is still held for up to 1.5 s, so cap concurrent requests on this endpoint (a bulkhead) or run it on virtual threads.
Exercise 2: Write the message contract for the fraud check
Define headers and body for the command and the reply, say who sets which field, and what fraud-detection does with an expired command.
Solution to exercise 2: headers, body and expired commands
- Command (topic
fraud-check.commands, keydebit_account). Headers set by the caller:ce_id(UUID),ce_type=wallet.fraud-check.requested.v1,correlation-id,reply-topic,reply-partition,deadline(epoch ms),traceparent. Body:transaction_id, both accounts,amount(integer, smallest currency unit),currency,created_at. - Reply (topic and partition taken from the command headers). Headers: a new
ce_id,ce_type=fraud-check.completed.v1,correlation-idcopied verbatim. Body:verdict(APPROVED|REJECTED|REVIEW),score,reasons[],model_version. - A command with
deadline < nowis skipped and counted in an “expired” metric. Scoring is idempotent bytransaction_id, so a resent command isn’t counted twice.
Remember
- Explain why request-reply over Kafka “takes the downside of RPC and loses the upside of messaging”.
- Name three ways to get a reply back to the right instance, and the cost of each.
- Trace how
null→ NPE →catch→ empty 200 turns a timeout into a fail-open, and how to modelUNKNOWN. - Know why
try/catcharoundsend()misses broker errors. - Explain dual writes, the outbox, and why consumers still have to deduplicate.
Interview questions
Junior
What’s the difference between an event and a command?
An event reports something that happened and waits for no answer; a command asks one specific party to do something and usually needs a result.
An event (FundTransferred) is immutable and the sender doesn’t know who listens. A command (CheckFraud) has one owner and can be rejected. Their lifetime, ownership and number of consumers differ, so they usually live in different topics.
What is a correlation id for in request-reply?
To match a reply with the request that is waiting for it.
The caller generates a unique id, sends it with the request (usually in a header), and the responder copies it verbatim into the reply. The caller keeps an id → future map (or a key in a shared store) and completes the future when a reply with a matching id arrives.
Mid
Several instances wait for replies over Kafka — how does each reply reach the right instance?
Three ways: a consumer group per instance, a reply partition per instance, or a shared group plus a shared correlation store.
- A separate consumer group per instance on the reply topic, each receiving every reply and discarding the rest: simple, N times the reads.
- Each instance owns a reply partition, the request carries a reply-partition header, and the instance reads it with
assign(): efficient, needs a stable identity. - A shared group plus a correlation store such as Redis (whoever receives the reply writes it, the waiting instance reads it): one more system on the critical path.
What is a dual write and how does the outbox solve it?
A dual write is writing to the database and Kafka without a shared transaction; the outbox writes both into the database in one transaction and a relay sends to Kafka.
Without a shared transaction, one side can succeed while the other fails. With the outbox, a relay (polling or CDC) reads the outbox table and sends. The result is at-least-once, so consumers must deduplicate by message id.
Your servlet thread pool is exhausted by requests waiting for replies — what do you do?
Set a short business timeout, then stop blocking platform threads: go async in the controller or use virtual threads, and add a bulkhead and a circuit breaker.
- Return a
CompletableFuture/DeferredResultfrom the controller (Spring MVC supports async processing), or use virtual threads (Java 21,spring.threads.virtual.enabled=true) so waiting is cheap. - A bulkhead caps concurrent requests on this endpoint so it can’t take others down.
- A circuit breaker for when the other side keeps failing.
Senior
Would you approve a synchronous fraud check over Kafka?
Not as a default: it adds two broker hops, a correlation store and temporal coupling, and the user still waits.
- If fraud-detection can change: call it over HTTP/gRPC with a timeout and a circuit breaker, and still publish the event through the outbox for post-hoc scoring.
- If it only speaks Kafka:
ReplyingKafkaTemplateor per-instance reply partitions, a dedicated command topic, a 1–2 s timeout, and anUNKNOWNoutcome with a clear policy. - If the product allows it: move to an asynchronous UX.
The fraud check times out — where does the payment go?
That’s a business decision and must be written down; technically, the result needs a third state, UNKNOWN.
Options: fail closed (reject), fail open (accept), or a hybrid: accept below a limit or for trusted users, put the rest into PENDING_REVIEW and re-check when fraud-detection recovers. A timeout must never turn into null or “clean”. Track the UNKNOWN rate as a risk metric.
Why is Kryo with setRegistrationRequired(false) a risk?
Because the data contains class names and Kryo instantiates any class on the classpath when reading — the same family of vulnerabilities as Java deserialisation.
Anyone who can write to Redis can make the service instantiate arbitrary classes. The format is also tied to the class layout, so a rolling deploy that changes a field leaves old and new instances unable to read each other’s data. Use JSON with fixed types, or require class registration, and protect Redis with ACLs/TLS.
The outbox relay sends duplicates — does the idempotent producer help?
No; consumers have to deduplicate by message id.
If the relay dies after Kafka acks but before marking the rows as sent, that batch is sent again. The idempotent producer only deduplicates internal retries within the same producer session (PID + sequence number); it can’t deduplicate two send calls from two sessions. A transactional producer can’t join the database transaction either. Use a processed-message table with a unique constraint, in the same transaction as the business effect.
Useful phrases for interviews
- “Request-reply over Kafka gives you the coupling of RPC without the benefits of messaging.”
- “The business timeout has to be explicit — a timeout should never be mistaken for a clean result.”
- “We model the verdict with three states: approved, rejected and unknown, and the business owns the policy for unknown.”
- “Each instance owns a reply partition, so a reply is read exactly once and matched by correlation id.”
- “An uncapped exponential backoff doubled our observed latency and left a blind spot at the end of the wait.”
- “
send()is asynchronous, so we handle failures in the callback, not in a try/catch.” - “Writing to the database and to Kafka separately is a dual write; the outbox turns it into at-least-once delivery.”
- “The relay can resend a batch, so consumers deduplicate by message id.”
- “If the product allows it, we return 202 and push the result instead of holding a thread.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.