Kafka and message brokers series: 1. Message brokers: four problems with request-response and how to choose a broker · 2. Why Kafka is fast and durable: the log, zero-copy, batching and replication · 3. Tuning Kafka for your goal: throughput, latency, durability, availability · 4. Case study: a synchronous fraud check over Kafka and Redis — good idea or not? · 5. CCDAK review: 40 Kafka questions worth remembering
Synchronous HTTP serves you well for a long time — until one order has to notify five systems, one dead service drags a whole chain down, or a marketing campaign knocks the database over. This post covers the four problems a message broker solves, what it costs, and how to choose a broker in the right order. Read first: Web and REST APIs (the request-response model) and the SKIP LOCKED section of Transactions, isolation levels and locking.
1. Request-response: synchronous, blocking, coupled
Key idea: the caller sends a request and then waits for the response, so it slows down and fails along with every service it calls.
- Synchronous: the result has to be available during the call.
- Blocking: the caller’s thread is held for the whole wait, including time spent on the network.
- Coupled: the caller must know the callee’s address and API, and both must be up at the same time.
sequenceDiagram
participant A as Service A
participant B as Service B
participant C as Service C
Note over A,C: Synchronous
A->>B: request
B-->>A: response (A is blocked)
A->>C: request
C-->>A: response (A is blocked)
Note over A: only now does other work
Note over A,C: Asynchronous
A-)B: message
A-)C: message
Note over A: does other work right away
B--)A: result (whenever ready)
C--)A: result (whenever ready)
The latency of a synchronous chain is the sum of the calls. Its success rate is the product of each service’s success rate: three services at 99.9% give a chain of roughly 99.7%.
When one link is a third party you don’t control (a payment gateway, an SMS carrier), timeouts, retries and circuit breakers are the minimum defence for synchronous calls. See the timeouts, retries and circuit breakers post.
2. Problem 01: sending the same message to several services
Key idea: the sender sends once, to the broker, and moves on; the broker delivers to each receiver.
Context. When an order is created, the Order service has to tell Inventory, Notification and Analytics. With request-response, Order makes three calls, waits three times and handles three kinds of failure.
Solution. The broker delivers the message to B, C and D in one of two ways:
- Push: the broker pushes to consumers (RabbitMQ).
- Pull: consumers fetch it themselves (Kafka).
Benefit 1: asynchronous communication. The create-order API’s latency is now just the time to write to the broker (a few ms), independent of the slowest service downstream.
3. Problem 02: everyone has to know everyone, and they fail together
Key idea: producers and consumers only know the broker’s address, not each other, so one side dying doesn’t take the other down.
Context. A has to know the addresses of B, C and D. When B fails or slows down, A’s threads are held, its connection pool runs dry and timeouts propagate back to the client. That’s a cascading failure — one failure spreading down the chain.
Solution. Split the roles:
- A producer only publishes messages to a topic (or an exchange/queue, depending on the broker).
- A consumer subscribes to the topic and processes the messages.
Benefit 2: decoupling and fault isolation. If a consumer dies, messages wait in the broker; when it comes back, it picks up where it left off. If the producer dies, consumers simply have nothing to do.
4. Problem 03: adding a new service without touching the old ones
Key idea: with pub/sub, adding a receiver is just adding a consumer that subscribes to the topic; the producer doesn’t change a single line.
Context. A new team is building loyalty points (service E) and needs to know about every order. With request-response, the Order team has to add another call, redeploy and take on another failure point. Load-balancing across several instances of E would mean A doing load balancing itself, which isn’t A’s job.
Solution: pub/sub and fan-out. A publishes an event to topic X. Whoever needs it subscribes to X.
flowchart LR
A[Order service] -->|publish OrderCreated| X[(Topic orders)]
X --> B[Inventory]
X --> C[Notification]
X --> D[Analytics]
X -.->|new subscriber, A unchanged| E[Loyalty]
Benefit 3: scalability. Every consumer group receives every message (fan-out). Within a group, the broker spreads messages across instances (load balancing).
- Kafka: that’s the consumer group, where each partition is assigned to exactly one consumer in the group.
- RabbitMQ: several consumers read the same queue (competing consumers); fan-out uses an exchange bound to several queues.
5. Problem 04: a traffic spike takes the database down
Key idea: put a queue in front of the database as a buffer: the queue absorbs the peak, and workers drain it at the rate the database can sustain, with no data loss.
Context. Events from IoT devices or user behaviour (clicks, views) are written straight to the database. Normally that’s 2,000 writes/s; at peak or during a campaign it hits 30,000 writes/s. The database handles about 8,000, the connection pool runs out and it falls over, taking unrelated features that share the database with it.
Two ways out:
- Rate limiting: reject requests above a threshold. It protects the database but loses data or forces clients to retry.
- A message queue as a buffer (load levelling): devices write to the queue, and a pool of workers reads from it and writes to the database.
flowchart LR
D1[IoT devices] --> Q[(Queue / topic)]
D2[Mobile app] --> Q
D3[Web tracking] --> Q
Q -->|steady rate, batched writes| W[Workers]
W --> DB[(Database)]
Benefit 4: reliability. The database sees a flat load instead of a spike, like a reservoir that holds back a flood and releases it slowly. Workers also write in batches (500 rows per INSERT), far cheaper than 500 single-row writes.
Rate limiting or a queue, and sizing the queue for a peak
6. Message brokers: definition, benefits and costs
Key idea: a broker is software in the middle that accepts and delivers messages. It trades four benefits for complexity, latency and operational cost.
| Benefit | Cost |
|---|---|
| Asynchronous, faster APIs | Complexity: one more system, more failure paths |
| Decoupling, fault isolation | Higher end-to-end latency, no immediate result |
| Scaling via fan-out and consumer groups | Operational overhead: clusters, monitoring, upgrades, security |
| Load levelling, database protection | Infrastructure and training cost |
In production you also pay for:
- Eventual consistency: data across services only matches after a delay.
- Duplicates: most brokers deliver at-least-once. Consumers must be idempotent: processing the same message again doesn’t change the result.
- Ordering only holds within a partition, or a queue with a single consumer.
- Harder debugging: you need correlation ids and distributed tracing.
- Poison messages (messages that keep failing) need a dead letter queue (DLQ), a separate queue for failed messages.
Two more design mistakes: a distributed monolith and publishing inside a transaction
The opposite mistake is just as common: putting the publish or an HTTP call inside the @Transactional method “to be safe”. The transaction waits on the network while holding its locks. How to spot this kind of stalled transaction (idle in transaction, Sleep in the processlist) and fix it is in Harmful transactions.
7. How an event differs from a response
Key idea: a response is an answer for exactly one party and can change over time; an event is something that happened — immutable, valuable to many parties, and replayable.
| Response | Event | |
|---|---|---|
| Sent to | Exactly the client that made the request | Anyone interested; the sender doesn’t know who |
| Content | Can vary over time (current balance) | Immutable: something that happened and can’t be changed |
| Value | To one party | A business fact valuable to several services |
| Communication | Synchronous | Asynchronous |
“The balance is 1,200,000” is a response; ask later and you get another number. “At 09:15 on 18 Nov account 42 topped up 200,000” is an event, true forever. Being immutable, events can be kept for a long time and replayed to rebuild state or reprocess after a bug fix.
An event is not a command (post 4 relies on this distinction):
- Event (“OrderPlaced”): past tense; the sender doesn’t care who handles it.
- Command (“ReserveStock”): an instruction to a specific party that expects an outcome.
Putting commands on a shared event topic is a design smell worth questioning. A common anti-pattern that comes from mixing the two up: using a broker as a synchronous call — send a command and hold the HTTP request open waiting for a reply.
Case study: a synchronous fraud check over Kafka and Redis reviews such a design: a servlet thread waiting up to 5 minutes, a timeout that turns into an empty HTTP 200, commands and events on one topic. It compares the alternatives and concludes when request-reply over Kafka is still acceptable.
Event streaming, the parts of Kafka and share groups in Kafka 4.x
Kafka describes itself as a distributed event streaming platform, not just a message broker. An event streaming platform has to handle three things: getting events in and out, storing them durably according to retention, and preserving order.
Streaming (moving a flow of events) is not stream processing (computing over it, as Kafka Streams or Flink do).
Kafka was built at LinkedIn and open-sourced in 2011. Around its storage core sit Kafka Connect (data integration), Kafka Streams/ksqlDB (stream processing) and five API families: Producer, Consumer, Streams, Connect and Admin.
8. In what order should you evaluate a broker
Key idea: don’t start with “which one is fastest?”. Filter by delivery guarantee and functionality, and only then measure performance on your own workload.
- Delivery guarantee: at-most-once (may lose), at-least-once (may duplicate), exactly-once (usually only within a narrow scope). If a broker can’t meet the guarantee the business needs, its speed is irrelevant.
- Functionality: flexible routing, delayed messages, priority, replay, ordering, long retention, multi-tenancy, transactions.
- Performance: throughput and latency, measured on your own workload.
- Scalability: adding nodes, adding partitions, limits on topic/queue counts.
- Ecosystem: clients for your team’s languages, connectors, monitoring tools, managed offerings, people who know how to run it.
The six options in summary (details in the subsections below):
| Broker | Delivery guarantee | Strengths | Good fit |
|---|---|---|---|
| Database table | exactly-once within a transaction | Consistent with business data, no new infra | Internal jobs, outbox, moderate load |
| Redis Streams | at-least-once | Low latency, already deployed | Lightweight queues, short-lived data |
| RabbitMQ | at-least-once | Routing, delays, priority, RPC | Task queues, business workflows |
| NATS (JetStream) | at-most-once (core) / at-least-once | Lightweight, low latency, subject routing | Service messaging, edge/IoT |
| Pulsar | at-least-once, with transactions | Multi-tenancy, queue + stream, geo | Multi-tenant platforms |
| Kafka | at-least-once, exactly-once within Kafka | Throughput, retention, replay, ecosystem | Event streaming, data pipelines, CDC |
8.1. A database as a queue
Use a table as the queue: producers INSERT a NEW row, a worker takes it, flips it to IN_PROCESSING, and marks it DONE when finished.
- Pros: no new system to run, and consistency: the business data and the job are written in the same transaction, so there’s no dual write. This is exactly the idea behind the outbox.
- Cons: the database becomes the bottleneck; error handling (a worker dying mid-job, retries, backoff) is yours to build; you lack fan-out, retention and consumer groups.
The trap with several workers: two of them running SELECT ... WHERE status = 'NEW' LIMIT 1 will grab the same order. The fix is to let the database lock rows and skip rows that are already locked with FOR UPDATE SKIP LOCKED (PostgreSQL 9.5+, MySQL 8.0+). The claim query commits immediately; the work happens after the commit, without holding locks.
Case study of a drinks queue: the order_queue table, SKIP LOCKED and leases
The course cites a conference paper (IEOM 2021) describing a Starbucks-style ordering system that uses a table as its queue, one row per order with order_no, status and updated_at:
| order_no | status | updated_at |
|---|---|---|
| 34 | DONE | 12:11:00 |
| 35 | IN_PROCESSING | 12:11:02 |
| 36 | IN_PROCESSING | 12:11:03 |
| 37 | NEW | 12:11:08 |
At the scale of one store a table is the right call: nothing new to run, and the pickup screen is just a SELECT. But two workers running SELECT ... LIMIT 1 will both grab order 37.
My version of the table and the claim query, run on PostgreSQL 16 with two concurrent workers: worker 1 got orders 1–5, worker 2 got 6–10, no overlap.
CREATE TABLE order_queue (
id bigserial PRIMARY KEY,
order_no bigint NOT NULL UNIQUE, -- prevents duplicate orders
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'NEW',
attempts int NOT NULL DEFAULT 0,
available_at timestamptz NOT NULL DEFAULT now(), -- for retries with backoff
locked_until timestamptz, -- lease: past it, assume the worker died
updated_at timestamptz NOT NULL DEFAULT now()
);
-- partial index: only rows still waiting, so it stays small and hot
CREATE INDEX order_queue_ready ON order_queue (available_at, id) WHERE status = 'NEW';
-- A worker claims up to 5 jobs without touching jobs other workers hold
WITH next AS (
SELECT id FROM order_queue
WHERE status = 'NEW' AND available_at <= now()
ORDER BY id
LIMIT 5
FOR UPDATE SKIP LOCKED
)
UPDATE order_queue q
SET status = 'IN_PROCESSING', attempts = q.attempts + 1,
locked_until = now() + interval '5 minutes', updated_at = now()
FROM next
WHERE q.id = next.id
RETURNING q.order_no, q.payload;
The UPDATE runs in a short transaction and commits immediately; the actual work (making the drink, or calling an external API) happens after the commit. A periodic sweeper returns expired leases to the queue:
UPDATE order_queue
SET status = 'NEW', available_at = now() + (attempts * interval '30 seconds'), updated_at = now()
WHERE status = 'IN_PROCESSING' AND locked_until < now() AND attempts < 5;
-- attempts >= 5: move to 'FAILED' for a human to look at (dead letter)
8.2. Redis
- Pros: often already there as a cache, very low latency (sub-millisecond), simple API.
- Cons: durability depends on persistence (RDB/AOF); failover can lose recent writes (async replication); RAM is expensive; few broker features.
Redis offers three mechanisms you can use as a broker: List and Pub/Sub are at-most-once, Streams is at-least-once and the closest to Kafka.
Redis’s three mechanisms: List, Pub/Sub and Streams
| Mechanism | Guarantee | Notes |
|---|---|---|
List (LPUSH + BRPOP) |
at-most-once | A worker that pops and then crashes loses the message. The “reliable queue” pattern with LMOVE/BLMOVE into an in-progress list, removed on completion, gets you at-least-once, but you must clean up the in-progress list yourself. |
| Pub/Sub | at-most-once | Fire-and-forget: a subscriber not connected at publish time never gets it. Nothing is stored. |
Streams (XADD, XREADGROUP, XACK) |
at-least-once | Consumer groups, a pending entries list (PEL), XAUTOCLAIM to take over a dead consumer’s messages. The closest thing to Kafka inside Redis. |
When to pick a list, a stream or a real broker for a given job, plus the reliable-queue pattern and a ZSET delay queue (and the read-then-delete bug that processes jobs twice), is in Redis: choosing data structures for real problems.
8.3. RabbitMQ
- Pros: flexible routing through exchanges (direct, topic, fanout, headers); at-least-once with publisher confirms and consumer acks; delayed messages (plugin), priority queues, request-reply (
reply_to+correlation_id); low latency; clients for every language (AMQP). - Cons (per the slides): messages are deleted on ack, so no replay; medium throughput; resource-hungry with long queues; a small streaming ecosystem.
- Update: since 3.9 (2021) there are RabbitMQ Streams, a log-style queue that isn’t consumed destructively and can be re-read from an offset. “No replay” only applies to classic and quorum queues. RabbitMQ 4.0 removed classic mirrored queues; for replication use quorum queues (Raft).
Choosing Kafka or RabbitMQ for backend-to-backend messaging, plus the ways to push realtime data to clients: Communication in system design, part 2.
8.4. NATS
- Pros (per the slides): high raw performance, subject-based routing (
orders.*), a single small binary, little to operate. - Cons (per the slides): queue groups distribute randomly, so ordering is lost; at-most-once.
- Update: at-most-once only describes core NATS. Since 2.2 (2021), JetStream adds durable storage, replication, acks, replay, at-least-once and dedupe by message id. “Faster than Kafka” usually compares core NATS (nothing on disk) with Kafka (a replicated log), which isn’t fair.
8.5. Pulsar
- Pros: the same topic/partition model as Kafka, but stateless brokers are separated from BookKeeper storage, so adding brokers doesn’t move data. It supports both queueing (shared subscriptions, per-message acks) and streaming, multi-tenancy, geo-replication and tiered storage.
- Cons: heavy to operate (brokers + BookKeeper + a metadata store), a smaller ecosystem than Kafka.
- Note: the slides rate its performance “below Kafka”; that’s contested and depends on who ran the benchmark. Kafka has tiered storage now too (KIP-405, GA since 3.9).
8.6. Kafka
Kafka is a distributed log: messages are appended to partitions and kept according to retention (days, weeks, or forever with log compaction) rather than deleted on read. Each consumer group tracks its own offsets (how far it has read). Kafka is popular but not a silver bullet.
- You get: very high throughput, replay, many independent consumer groups, per-partition ordering, the largest ecosystem (Connect, Streams, Schema Registry, many managed services).
- You give up: latency isn’t the lowest; no rich routing, delayed messages or priority out of the box; consumer parallelism is capped by the partition count; running a cluster isn’t trivial (although KRaft in Kafka 4.0 finally removed ZooKeeper).
An example of Kafka as the intake for 333K heartbeats/s, with each worker keeping timers per partition: liveness case study for 1 million IoT devices.
Exercises
Exercise 1: Pick a broker for three systems
Pick a broker and justify it in the order delivery guarantee → functionality → performance → scalability → ecosystem:
- (a) A small shop: one Spring Boot service + PostgreSQL, sends an email after each order, about 2,000 orders/day.
- (b) User behaviour tracking: 200,000 events/s, used by many teams, must be re-readable for 7 days.
- (c) Background tasks with priorities (VIP first) and scheduling (“run in 15 minutes”).
Solution to Exercise 1: a database table, Kafka, RabbitMQ
- (a) A database table as the queue (or outbox +
SKIP LOCKED). It needs at-least-once and consistency with the order: writing the order and the email job in one transaction is the strongest guarantee available. 2,000 orders a day is far too small to justify a broker cluster. A duplicate email is tolerable, but storingemail_sent_atavoids most of them. - (b) Kafka. High throughput, 7-day retention for replay, many teams as independent consumer groups, Connect for loading the warehouse. At-least-once is enough; you could even relax to
acks=1if losing a tiny fraction is acceptable. - (c) RabbitMQ. It has priority queues and delayed messages (the delayed message exchange plugin, or TTL + dead letter exchange), per-message acks and routing by task type. Kafka has no built-in priority or delay; faking them with multiple topics and consumers that sleep is needless complexity.
Remember
- Name the four problems of request-response and the benefit a broker brings for each.
- Explain why
save()thensend()can diverge, and how the outbox fixes it. - Tell an event, a command and a response apart with one example.
- State the five broker-selection criteria in order, and why performance isn’t first.
- Know when a database table with
SKIP LOCKEDis enough, and when to move to a broker.
Interview questions
Junior
What is a message broker and what problems does it solve?
An intermediary that accepts messages from producers and delivers them to consumers, so services don’t call each other directly.
It solves four problems of request-response: waiting (async), knowing each other and failing together (decoupling), changing the sender for each new receiver (fan-out), and traffic spikes (load levelling).
How does a queue (point-to-point) differ from pub/sub?
Queue: each message is processed by only one consumer. Pub/sub: every subscriber group gets every message.
In a queue, consumers share the work. Kafka combines both: different consumer groups are pub/sub, consumers within one group are a queue.
What’s the difference between push and pull?
Push: the broker actively sends messages to consumers. Pull: consumers ask the broker for messages.
RabbitMQ pushes, with prefetch to cap unacked messages. Kafka pulls (poll). Pull lets consumers control their own pace and batch naturally without being overwhelmed when slow. Push gives lower latency under light load but needs a backpressure mechanism (telling the sender to slow down).
What do at-most-once, at-least-once and exactly-once mean?
At-most-once: may be lost, never duplicated. At-least-once: never lost, may be duplicated. Exactly-once: each message takes effect exactly once.
- At-most-once: ack first, then process.
- At-least-once: process, then ack; a crash in between means redelivery.
- Exactly-once: in practice you only get it within a narrow scope (such as Kafka’s read-process-write transactions) or by combining at-least-once with an idempotent consumer.
Mid
How does an event differ from a command and a response?
An event is something that happened, immutable, for whoever is interested. A command is an instruction to a specific party. A response is the answer for exactly the party that asked.
A response is synchronous and its value can change over time. An event is named in the past tense (OrderPlaced), valuable to several services, and the publisher doesn’t care who handles it. A command (ReserveStock) usually expects an outcome.
Keeping them apart shapes topic design: events can fan out freely, while a command usually goes one way to the single service that owns it.
What criteria do you use to choose a message broker, and in what order?
Delivery guarantee → functionality → performance → scalability → ecosystem. Performance can be bought; guarantees and features can’t.
- Delivery guarantee: can the business tolerate loss, or duplicates.
- Functionality: routing, delays, priority, replay, ordering, retention.
- Performance: measured on my own workload — a vendor’s benchmark is no substitute.
- Finally scalability, the ecosystem and my team’s ability to operate it.
When would you use a database as a queue instead of a broker?
With moderate load, one worker pool, and a need for consistency with business data.
Moderate load means hundreds to a few thousand jobs/s. Writing the order and the job in one transaction removes the dual write. Use FOR UPDATE SKIP LOCKED, a locked_until lease to recover jobs from dead workers, and a partial index on waiting rows. Move to a broker when you need fan-out, replay, or the load strains the database.
Senior
How do you publish an event after a database write without them diverging?
Use a transactional outbox: write the event into an outbox table in the same transaction as the business data, and have a relay push it to the broker.
There’s no shared transaction between the DB and the broker, so save() followed by send() is a dual write where one side can succeed and the other fail. The relay is a poller using SKIP LOCKED, or CDC such as Debezium reading the WAL/binlog; it pushes to the broker and marks the event sent.
The relay is at-least-once, so consumers must be idempotent (a processed-message table keyed by event id, or upserts on a business key). Ordering holds if the relay publishes in order and keys messages by aggregate.
What do you pay for adding a message broker to a system?
You pay with eventual consistency, duplicates, limited ordering and operational complexity.
- Eventual consistency the business must accept.
- Duplicates, so consumers must be idempotent.
- Ordering only within a partition.
- Poison messages that need a DLQ.
- Message schemas that become versioned APIs.
- Debugging that needs tracing, and consumer lag to monitor.
Get the design wrong and you have a distributed monolith over a message bus.
How do RabbitMQ and Kafka differ at their core?
RabbitMQ is a smart broker with simple consumers; Kafka is a distributed log with a simple broker and smart clients.
- RabbitMQ (classic and quorum queues): routes through exchanges, pushes messages, deletes them on ack and tracks per-message state. It shines at routing, priority, delays and task queues.
- Kafka: messages are appended and kept per retention; consumers own their offsets and pull. It shines at throughput, replay and many independent consumer groups.
RabbitMQ Streams and Kafka share groups are bringing them closer, but their default models are still different.
Useful phrases for interviews
- “Request-response couples latency and availability: the chain is only as fast as its sum and only as reliable as its product.”
- “I’d publish an event rather than call each downstream service, so adding a consumer doesn’t touch the producer.”
- “The queue acts as a buffer for load levelling; the database sees a flat line instead of a spike.”
- “When I choose a broker, I start with the delivery guarantee and the features we need, and only then benchmark performance on our own workload.”
- “Because delivery is at-least-once, every consumer has to be idempotent.”
- “To avoid a dual write, I’d use a transactional outbox and relay the events with CDC.”
- “At this scale a table with
SKIP LOCKEDis enough; I wouldn’t introduce a Kafka cluster for two thousand jobs a day.” - “Decoupling addresses isn’t decoupling data — the message schema is still a contract we have to version.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.