Skip to content
notes.vinhnq.com

Search

System DesignSenior16 min read

Communication in system design, part 2: Kafka or RabbitMQ, real-time push and a fraud detection case study

Review 13 cards
On this page

Communication in system design series: 1. A message’s journey, REST or gRPC · 2. Async, real time and a fraud detection case study

Part 1 covered synchronous calls between backends. This part takes the other two directions: backends talking through a message broker, and servers pushing data to browsers or devices on their own. It ends with a fraud detection case study, where both styles of communication show up in one design.

1. Async between backends: Kafka or RabbitMQ

Key idea: both are message brokers, but Kafka leans toward event streaming (high throughput, retention, acceptable latency), while RabbitMQ leans toward routing and queue features (routing keys, TTL, delays, RPC).

Kafka RabbitMQ
Delivery guarantee At-most-once, at-least-once; exactly-once within Kafka At-most-once, at-least-once (publisher confirms + acks)
Throughput Very high, scales with partition count (millions of messages/second per cluster) An order of magnitude lower (tens to a few hundred thousand/second per cluster, depending on queue type)
Latency Sequential disk writes, consumers pull, typically a few ms Keeps messages in RAM first, broker pushes, low latency at moderate load
Routing Topic and key only, no built-in filtering Exchanges, routing keys, headers: flexible routing and filtering
Retention Kept by time or size, replayable Deleted once processed (except RabbitMQ Streams)
Other features Large ecosystem: Connect, Streams, schema registry Built-in dead lettering, TTL, delays, priorities, request-reply

Why you’d use a broker at all, the problems synchronous calls can’t solve, and how to choose between database-as-queue, Redis, RabbitMQ and Kafka are in message brokers and how to choose one. Why Kafka is fast is in why Kafka is fast and durable.

2. Servers pushing data to clients

Key idea: HTTP is client asks, server answers. For the server to notify on its own there are four ways, from simple to powerful: polling, long polling, Server-Sent Events, WebSocket. Choose the simplest one that meets the need.

Method How it works Fits Weakness
Polling Client asks every N seconds Infrequent updates, delay acceptable Wasted empty requests, up to N seconds late
Long polling Server holds the request until there’s data or it times out Notifications where SSE or WebSocket isn’t available A new request after every delivery
SSE One HTTP response kept open, the server writes events into it One-way server → client: notifications, prices, progress One-way; text only
WebSocket Upgrades HTTP into a two-way channel Chat, games, collaboration, order books with order entry Stateful, harder to scale and load-balance
flowchart LR
  P[Business service] --> K[Kafka or Redis pub/sub]
  K --> G1[WebSocket gateway 1]
  K --> G2[WebSocket gateway 2]
  G1 --> C1[Client A]
  G1 --> C2[Client B]
  G2 --> C3[Client C]

A common architecture: a WebSocket gateway layer that only holds connections, subscribes to Kafka or Redis pub/sub, and pushes to the right clients. Business services don’t need to know which node a client is connected to.

Scaling WebSocket to a million connections
  • Memory: each connection costs buffers and state, typically 10–50 KB depending on the framework. 1 million connections × 20 KB ≈ 20 GB, spread over 10–20 gateway nodes.
  • File descriptors: each connection is an fd. Raise ulimit -n and fs.file-max; a few hundred thousand connections per node is normal with Netty.
  • Load balancer: set the idle timeout longer than the heartbeat interval (e.g. heartbeat 30 seconds, timeout 60–120 seconds), or the LB cuts idle connections.
  • Message routing: gateways need to know which user is on which node. Simple: every gateway listens to a shared topic and filters for its own users (wasteful at scale). Better: keep a user → gateway map in Redis and send straight to that node.
  • Reconnect storms: when a node dies, hundreds of thousands of clients reconnect at once. Clients must wait randomly (backoff with jitter), and the server rate-limits handshakes.
  • Deployments: drain connections from a node before shutting it down, and tell clients to move in a controlled way.

3. Broadcasting to many devices: MQTT and order books

Key idea: when one event must reach thousands to millions of devices (switch on every IoT device, update the order book for everyone watching), use publish/subscribe through a broker, such as MQTT. Browsers speak MQTT over WebSocket.

MQTT is a lightweight pub/sub protocol built for weak devices and flaky networks: clients subscribe to topics (e.g. home/42/light), and the broker (EMQX, Mosquitto, HiveMQ) delivers messages to every subscriber. The smallest header is only 2 bytes; it’s asynchronous and supports broadcast.

An order book (an exchange’s list of buy and sell orders) changes hundreds of times a second. Pushing the whole book every time is wasteful. The common approach:

  1. The client loads a full snapshot with a sequence number.
  2. After that it only receives diffs (which price levels changed volume), each with an increasing sequence.
  3. On a sequence gap, it reloads the snapshot.
  4. The server conflates updates on a fixed beat, say every 100 ms, because human eyes can’t read faster than that.

4. QUIC, stated correctly

Key idea: QUIC runs over UDP, merges the transport and TLS 1.3 handshakes so connections start faster, and keeps streams independent so a lost packet doesn’t block other streams. It underlies HTTP/3, and EMQX uses it for MQTT on mobile networks.

  • Performance: fewer round trips to connect (1 RTT, reconnects can be 0-RTT); no TCP head-of-line blocking; switching networks (Wi-Fi to 4G) keeps the connection thanks to connection IDs.
  • Weaknesses: UDP isn’t as optimized by routers and operating systems as TCP (some corporate networks still block UDP); per-packet encryption costs more CPU.

The HTTP/1.1, 2 and 3 comparison is in the web and HTTP post.

5. Case study: a payment fraud detection system

Key idea: score risk synchronously in the payment path with a tight time budget, and compute features, learn from feedback and run investigations asynchronously through Kafka. The hardest decision is whether to let payments through or block them when scoring fails.

The slide leaves only a title for this practice section. Below is my own solution, following the four steps. The Kafka side (topics, keys, sliding windows in Redis, idempotent counting under redelivery) goes deeper in the fraud detection case study with Kafka and Redis.

Requirements.

  • FRs (pick three): (1) score each transaction before charging and return allow, review or deny; (2) the risk team adds and edits rules without a deploy; (3) record feedback (chargebacks, reported stolen cards) to improve the model.
  • NFRs: scoring p99 < 100 ms (the whole payment flow has a ~1 second budget); 99.99% availability for the scoring API; never score one transaction twice with two different outcomes; new rules take effect within 1 minute.

Estimation. 50 million transactions/day → ~600 TPS on average, peak ×5 ≈ 3K TPS. Each scoring reads ~20 counters (card transactions in the last hour, distinct cards per device…) → ~60K Redis commands/second at peak, which a small Redis cluster handles. Events ~1 KB × 50 million = 50 GB/day, ~18 TB/year into the analytics store.

API.

  • POST /v1/risk/assessments with transaction_id, user_id, amount, currency, card_fingerprint, device_id, ip, merchant_id → {decision: ALLOW|REVIEW|DENY, score, reasons[]}. Idempotent on transaction_id: calling again returns the same decision.
  • POST /v1/risk/feedback with transaction_id, label (FRAUD, LEGIT), source (chargeback, analyst).

Data model. risk_decision(transaction_id PK, score, decision, rule_hits jsonb, model_version, created_at); rule(id, expression, action, priority, enabled, version); label(transaction_id, label, source, created_at). Velocity counters live in Redis; full history lives in the analytics store.

flowchart LR
  PAY[Payment service] -->|gRPC, 100 ms deadline| RISK[Risk API]
  RISK --> RULE[Rule engine]
  RISK --> FEAT[(Redis: velocity counters)]
  RISK --> MODEL[Model scoring]
  RISK -->|store decision| DB[(PostgreSQL)]
  RISK -->|events| K[Kafka]
  K --> STREAM[Stream processor]
  STREAM --> FEAT
  K --> DW[(Analytics store)]
  DW --> TRAIN[Offline training]
  TRAIN --> MODEL

Deep dives.

  • Keep the synchronous path as short as possible: the Risk API only reads precomputed features from Redis, runs rules in memory and calls a model in the same process or on the same host. No analytics queries on this path.
  • Velocity counters: a stream processor (Kafka Streams, Flink) updates sliding-window counters after each transaction. The Risk API also increments the most important counters as it scores, because two fraudulent transactions 200 ms apart may arrive before the stream catches up.
  • Hot-reloaded rules: rules are stored in a versioned table; the Risk API reloads them every 30 seconds or on a “rules changed” event from Kafka. New rules run in “log only” (shadow) mode before they’re allowed to block.
  • Feedback loop: chargebacks arrive weeks later, become labels, and are used to retrain the model offline. New models also run in shadow, then canary.
  • REVIEW cases: the transaction is held and goes into an investigation queue; the investigation’s outcome also becomes a label.
Velocity counters in Redis and how to present it in 45 minutes

Counting a card’s transactions in the last hour with a sorted set (exact sliding window):

// Run in a pipeline or a Lua script to keep it to one round trip
String key = "vel:card:" + cardFingerprint + ":1h";
long now = System.currentTimeMillis();
redis.zadd(key, now, transactionId);                 // add the current transaction
redis.zremrangeByScore(key, 0, now - 3_600_000);     // drop what's outside the 1-hour window
long count = redis.zcard(key);                       // transactions in the last hour
redis.expire(key, 3_700);                            // clean up once the card goes quiet

A sorted set is exact but its memory grows with the number of transactions. For large counters, use per-minute buckets (INCR a per-minute key, sum the last 60) to trade a little accuracy for fixed memory. For counting distinct values (cards per device), use HyperLogLog.

Splitting 45 minutes:

Minutes Content
0–7 Three FRs, NFRs with numbers (p99 100 ms, 99.99%), out of scope: KYC, anti-money-laundering
7–11 Estimation: 3K TPS peak, 60K Redis commands/second, 50 GB/day
11–22 API, data model, synchronous and asynchronous diagram
22–38 Deep dive: latency budget, velocity counters, hot-reloaded rules, conditional fail-open
38–45 Bottlenecks (hot Redis keys for popular cards, a slow model), monitoring block rates, the feedback loop

Metrics to watch in production: block rate, REVIEW rate, fraud that slipped through (from chargebacks), Risk API p99, and how often fail-open kicks in. A sudden jump in block rate is usually a bad rule, not a fraud wave.

Bottlenecks and failures. Redis is the hottest dependency: run it as a cluster with replicas, give it a timeout of a few ms, and score with whatever features remain if some are missing. A slow model breaks the budget: cap its time and fall back to rules only when it’s exceeded. A slow Kafka only makes features a few seconds stale; it never blocks payments.

Exercises

Exercise 1: Choosing a push mechanism for three situations

Task (my own): choose polling, long polling, SSE, WebSocket or MQTT, and explain: (a) an unread-notification count on a website; (b) a document edited by many people at once; (c) a live football score for 1 million viewers in an app.

Solution to exercise 1
  • (a) SSE, or polling every 30–60 seconds for maximum simplicity. The data is one-way and small, and a few seconds of delay is fine. SSE passes through any proxy and reconnects on its own.
  • (b) WebSocket. Two-way and continuous, every keystroke is a small message, latency must be low. Add an edit-merging algorithm (OT or CRDT) and put everyone on the same document on the same node or the same pub/sub channel.
  • (c) MQTT or SSE through a fan-out layer, plus a CDN for what rarely changes. One million people receiving the same message is a broadcast problem. A pub/sub broker (MQTT, or an SSE gateway listening to Kafka) delivers the same event; conflate updates every few seconds; clients reconnect with jitter to avoid a reconnect storm when a goal is scored.

Remember

  • Compare Kafka and RabbitMQ on delivery guarantees, throughput, routing and retention, and know where Kafka’s exactly-once applies.
  • Choose between polling, long polling, SSE and WebSocket for a situation and justify it.
  • Explain MQTT’s three QoS levels and how to push an order book with a snapshot plus diffs.
  • Say correctly how QUIC removes head-of-line blocking.
  • Present the fraud detection case study: the synchronous path, the asynchronous path, and conditional fail-open.

Interview questions

Junior

How do polling, long polling and WebSocket differ?

Polling asks periodically; long polling asks and the server holds the request until there’s data; WebSocket opens a two-way channel that stays ready.

Polling is simple but wasteful and slow; WebSocket is the fastest but stateful and harder to scale.

When do you use SSE instead of WebSocket?

When you only need one-way pushes from server to client, such as notifications or price updates.

SSE is plain HTTP so it passes through proxies easily, and the browser reconnects on its own and resumes from Last-Event-ID.

Mid

Kafka or RabbitMQ for an order processing system?

Kafka if you need high throughput, many consumers reading the same events, and replay; RabbitMQ if you need complex routing, delays, TTL or priorities at the queue level.

For orders, the “order created” event is consumed by several services and may need reprocessing, so Kafka usually fits better. A “send an email after 30 minutes” task is neater with RabbitMQ delays.

Does Kafka have exactly-once?

Yes, within Kafka: the idempotent producer prevents duplicate writes, and transactions bundle reading, writing and committing offsets into one unit.

Outside Kafka (writing to a database, calling an API) it’s still at-least-once, so consumers must be idempotent, for example with a processed-messages table and a unique key.

How do you push an order book to hundreds of thousands of viewers?

Send a snapshot with a sequence number once, then only diffs with increasing sequence numbers, conflate updates roughly every 100 ms, and broadcast through pub/sub.

On a sequence gap the client reloads the snapshot. Conflation cuts bandwidth by tens of times without viewers noticing.

Senior

What do you worry about when scaling WebSocket to a million connections?

Memory and file descriptors per node, the load balancer’s idle timeout, routing messages to the node holding each user, and reconnect storms when a node dies.

Separate a gateway layer that only holds connections, keep a user → gateway map in Redis, use heartbeats, jittered client backoff, and drain connections before deploying.

The fraud scoring system times out. Should the Payment service allow or block?

Conditional fail-open: apply a minimal fallback rule set inside the Payment service, allow small transactions, require extra authentication or review for large ones, and rescore everything asynchronously.

Blocking everything loses revenue every time scoring has an incident; allowing everything lets fraudsters strike exactly then. Track how often fail-open triggers as a key metric.

Why not compute all fraud features in the stream processor?

Because the stream lags by hundreds of milliseconds to seconds, while fraudsters fire transactions very close together; the few most important counters must be incremented during scoring.

The rest (complex features, long windows) is fine to compute in the stream, since it’s off the payment path.

Useful phrases for interviews

  • “Kafka is an event streaming platform; RabbitMQ is a smart router with queue features.”
  • “Exactly-once in Kafka stops at the Kafka boundary, so the consumer still has to be idempotent.”
  • “If the client only receives updates, server-sent events are enough.”
  • “The gateway layer only holds connections and handles fan-out.”
  • “Jittered backoff on the client prevents a reconnect storm.”
  • “We send a snapshot first, then sequenced diffs, conflated every hundred milliseconds.”
  • “QUIC retransmits per stream, so one lost packet doesn’t stall the others.”
  • “Scoring sits on the payment path with a 100-millisecond deadline.”
  • “If scoring times out, we fail open for small amounts and step up authentication for large ones.”
  • “New rules run in shadow mode before they’re allowed to block.”

Notes based on the System Design course (Ronin Engineer), rewritten in my own words.