Skip to content
notes.vinhnq.com

Search

System DesignSenior18 min read

Case study: fraud detection with Kafka and Redis — topics, keys, windows and failures

Review 13 cards
On this page

Kafka problems and patterns series: 1. Partition count, large messages, lag and long-running tasks · 2. Dual writes, exactly-once and message ordering · 3. Case study: fraud detection with Kafka and Redis

An e-wallet the size of a big crypto exchange sees a huge number of transactions per minute, and plenty of fraud; the rule to catch is “an account making 100 transactions in 5 minutes raises an alert”. The general design (synchronous scoring, API, fail-open, a 45-minute plan) is already in the payment fraud detection case study; this post goes deep on the asynchronous Kafka path the slides draw: topics and keys, windows, counting without duplicates, failures and scaling. The slides only give two diagrams; the numbers and the failure handling are my own work.

1. Problem and requirements

Key idea: quickly catch accounts with abnormal transaction frequency to limit damage, without slowing down or breaking the payment flow.

  • Ingest every wallet transaction; an account with ≥ 100 transactions in any 5 minutes produces a “rule hit” for the risk team, an action service (temporary freeze, step-up authentication) and a case store.
  • Add “count/sum over a window” rules without rewriting the system.

Non-functional requirements (my numbers, since the slides only say “detect quickly”):

Item Target
Detection latency p99 < 2 seconds from transaction commit
Miss nothing every committed transaction is counted (at-least-once)
Never over-count redelivery doesn’t increase the count
Impact on payments 0 ms added to the payment path; a dead detector doesn’t block payments
Load 100 million transactions/day, peak 10× average

2. Estimation

Key idea: the load is small for Kafka; what needs careful numbers is Redis operations and memory.

  • Average: 100,000,000 / 86,400 ≈ 1,160 transactions/s.
  • Peak: × 10 (heavy market volatility) ≈ 12,000/s; design for 20,000/s with headroom.
  • Kafka bandwidth: ~300 B per event. 20,000 × 300 B = 6 MB/s in. Per day 100 million × 300 B = 30 GB, × RF 3 = 90 GB; 7 days of retention ≈ 630 GB across the cluster. Light.
  • Redis operations: one Lua script call per event, so 20,000 calls/s at peak. One Redis node runs a few tens of thousands of small scripts per second; use a 3-shard Redis Cluster for headroom and replicas.
  • Redis memory: entries inside the window = rate × window length = 20,000 × 300 s = 6 million entries at peak. Each sorted-set entry is ~100 B (member + score + skiplist and dict overhead) ≈ 600 MB. Plus ~1 million active account keys × ~100 B = 100 MB. Total ~1 GB, comfortable.
  • Partitions: a detector calling Redis sequentially at ~0.5 ms per round trip handles ~2,000 events/s. 20,000 / 2,000 = 10, round up to 12, with growth headroom 24 (divides evenly across 2, 3, 4, 6, 8, 12 instances).

3. API and data model

Key idea: two event contracts (transactions in, rule hits out) and one state structure in Redis are enough; the main flow needs no synchronous API.

Transaction event on topic wallet.transactions, key = account_id:

{
  "event_id": "8f0c…",
  "transaction_id": "T3",
  "account_id": "A",
  "type": "TRANSFER_OUT",
  "amount": "10.00",
  "currency": "USDT",
  "occurred_at": 1712128559000
}

occurred_at is the commit time taken from the wallet’s DB (a single clock source), not the app server’s clock.

Rule hit on topic fraud.rule-hits, key = account_id:

{
  "hit_id": "TX_VELOCITY_100_5M:A:1712128500000",
  "rule_id": "TX_VELOCITY_100_5M",
  "account_id": "A",
  "count": 100,
  "window_start": 1712128259000,
  "window_end": 1712128559000,
  "detected_at": 1712128559420
}

State in Redis: one sorted set per account, fraud:win:{A}, score = occurred_at, member = transaction_id. This is exactly the slide’s “A: Transactions (transaction_id, amount, timestamp)” table, with a Redis structure chosen for it.

4. Where the detector sits: behind Kafka, off the payment path

Key idea: the slides draw two options: the wallet calls the detector synchronously before recording the transaction, or the wallet records it and emits an event the detector reads through Kafka. Time-based counting rules suit the second; synchronous scoring and fail-open are analysed in the general case study.

On the Kafka path (slide 2): the wallet executes the transaction, writes the DB (TiDB), emits an event to the Transaction Topic; the Fraud Detector consumes it, counts in Redis (Rule State), emits to the Rule Hit Topic. Payments don’t depend on the detector; in exchange, transactions 1 to 100 have gone through before detection.

5. High-level design

Key idea: the wallet writes the transaction and an outbox row in one transaction, CDC ships it to Kafka, the detector counts a sliding window in Redis and emits rule hits to whoever acts on them.

flowchart LR
  U[User] -->|1 pay| W[Wallet]
  W -->|2 transaction + outbox| DB[(Wallet DB TiDB)]
  DB -->|3 CDC| T[(wallet.transactions)]
  T -->|4 consume| D[Fraud Detector x N]
  D -->|5 Lua: count window| R[(Rule State Redis)]
  D -->|6 alert| H[(fraud.rule-hits)]
  H --> A[Action service]
  H --> N[Risk team alerts]
  A -->|block flag| R
  W -.->|GET block flag| R

The slide has the wallet “emit” straight to the topic after writing the DB, which is a dual write: lose the event and fraud slips through. I use an outbox, and with TiDB, TiCDC ships it to Kafka (see part 2). The topic has 24 partitions, group fraud-detector.

6. Deep dive: sliding-window counting in Redis

Key idea: fixed windows (INCR per period) are cheap but wrong at boundaries and double count on redelivery. A time-scored sorted set gives an exact, idempotent sliding window, at the cost of memory proportional to the number of transactions.

Counting method How Exact Idempotent on redelivery Memory per account
Fixed window INCR fraud:cnt:{A}:{minute/5} + EXPIRE misses boundaries no 1 number
Approximate sliding counter 5 one-minute buckets in a hash, summed approximate no 5 numbers
Sliding log (sorted set) ZADD each transaction, trim old, ZCOUNT yes yes transactions in 5 minutes

The boundary bug: a fraudster makes 60 transactions at 10:04 and 60 at 10:06. The fixed windows [10:00, 10:05) and [10:05, 10:10) each see only 60, so nothing fires. The 5-minute sliding window ending at 10:06 sees 120.

A Lua script runs atomically inside Redis, one round trip per event:

-- KEYS[1] = fraud:win:{A}
-- ARGV[1] = occurred_at (ms), ARGV[2] = transaction_id
-- ARGV[3] = window (300000), ARGV[4] = allowed lateness (60000)
local ts, win, late = tonumber(ARGV[1]), tonumber(ARGV[3]), tonumber(ARGV[4])
redis.call('ZADD', KEYS[1], ts, ARGV[2])                      -- same member: no growth
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ts - win - late)
redis.call('PEXPIRE', KEYS[1], win + late)
return redis.call('ZCOUNT', KEYS[1], ts - win, ts)            -- transactions in the 5 minutes up to ts

The detector calls the script and emits a rule hit if the result is ≥ 100. The {A} in the key is a Redis Cluster hash tag, so all of an account’s keys live on the same shard if you later need several. Unlike the counter in the general case study (system time at synchronous scoring), the score here is the event time taken from the Kafka event, because the detector may run behind.

Event time, late events and amount-sum rules
  • Event time, not processing time: count by occurred_at. When the consumer lags 2 minutes and then catches up, transactions still land in their own windows. Counting by processing time makes 300 transactions spread over 15 minutes look like they happened within 5: a false alarm.
  • Late events: the sorted set orders by score, so a late event drops into the right place. ZCOUNT counts the window ending at that very event. The trim only removes entries older than ts − window − late, so an event up to 60 seconds late still has its neighbours to count. Anything later undercounts; add a metric for those.
  • Order within a partition is almost commit order (outbox plus account key), so late events mostly come from slow CDC, which is rare.
  • Sum rules (“withdrawals > 10,000 USD in 1 hour”): the member is the transaction_id; store amounts in a hash fraud:amt:{A} (field = transaction_id), or encode the member as T3|10.00. The script reads ZRANGEBYSCORE over the window and sums: O(n) in the window’s transaction count, still small. If n can be very large, use per-minute buckets for the sum (accepting approximation) and keep dedupe with a TTL’d set of transaction_ids.
  • Many rules: a separate window per rule means many keys; rules on the same event type can share one sliding log of the longest window, each doing ZCOUNT over its own length.

7. Deep dive: Kafka and rule hits

Key idea: keying by account gives ordering and avoids contention; at-least-once plus idempotent counting gives “miss nothing, over-count nothing”; rule hits need protection against alert storms.

  • Key = account_id: one account is handled by one detector at a time, so no two instances modify the same sorted set.
  • Alert storms: an account over the threshold crosses it again on every later transaction. Emit only one rule hit per account per rule per 5 minutes: SET fraud:hit:{A}:TX_VELOCITY NX PX 300000.
Detector consumer and producer configuration
  • One owner per key: the Lua script is already atomic, so even two instances touching one key won’t corrupt it; one owner per key still keeps order and makes reasoning easier.
  • Commits: disable auto-commit, commit after the Redis call and (if any) the rule-hit send succeed. A crash in between means reprocessing, which is safe thanks to ZADD.
  • Batch the Redis calls: for each batch from poll(), pipeline the script calls for different accounts to cut round trips; per-account order holds because a pipeline sends in order.
  • Rule-hit producer: acks=all, idempotence on.
  • A separate group per rule-hit reader: the action service, alerts and the case store, so one slow reader doesn’t hold back the others.

8. Failures and how the system survives them

Key idea: the detector keeps no local state and Redis is just state computable from the Kafka log, so most incidents are recovered by reading again.

Incident Effect Handling
Detector crash, rebalance a few events reprocessed ZADD is idempotent; rebalances are fast since there’s no local state
Lag grows (peak, slow Redis) late detection alert when time lag > 30 seconds; scale up to 24 consumers
Redis fails over to a replica the last few writes lost (async replication), slight undercount accept it, or replay briefly (next row)
All of Redis lost every window lost bring up a new Redis, reset the group’s offsets to now − 6 minutes (--to-datetime), replay; ZADD drops duplicates
Wallet loses events fraud slips through outbox + CDC instead of emitting directly
A bot makes 10,000 transactions in 5 minutes a 10,000-entry sorted set, one partition busier still small; the block flag stops transactions after the first rule hit
Detector down entirely no detection, payments keep working operational alert; that’s the price chosen in section 4

Key idea: a stream processor has built-in event-time windows, local state and Kafka → Kafka exactly-once; Redis is more flexible for dynamic rules and for synchronous reads by the wallet.

builder.stream("wallet.transactions", Consumed.with(Serdes.String(), txSerde))
  .groupByKey()
  .windowedBy(SlidingWindows.ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)))
  .count()
  .toStream()
  .filter((windowedAccount, count) -> count >= 100)
  .map((w, count) -> KeyValue.pair(w.key(), RuleHit.of(w, count)))
  .to("fraud.rule-hits");

10. Scaling the detector and changing the partition count

Key idea: the detector holds no state, so scaling is just adding consumers, up to the partition count. Because state lives in Redis by account_id rather than by partition, even adding partitions doesn’t break the count.

  • Adding consumers: at most 24 consumers have work (the partition count). Use CooperativeStickyAssignor and group.instance.id (static membership) so rolling deploys don’t stop the whole group.
  • Adding partitions: key A may move to another partition. With external state, the new consumer still reads the right sorted set fraud:win:{A}; you only want to avoid two consumers handling A during the move (the Lua script is atomic, so it stays correct anyway). With Kafka Streams, local state is split by partition, so adding input partitions forces an application reset or a repartition.
  • Monitoring: time lag of the fraud-detector group is the key metric, because lag is detection delay. Add p99 latency of the Lua script and the number of events dropped for being too late.

A 45-minute presentation plan for the whole problem is in the general case study; when asked to go deep on Kafka, I use sections 6–9 of this post.

Exercises

Exercise 1: Add a “large withdrawals” rule

Add a rule: an account whose total withdrawals exceed 10,000 USD in 1 hour raises an alert. Peak withdrawal load is 2,000/s. Design the state, compute the Redis memory, and say whether the new rule needs a new topic.

Solution to Exercise 1
  • Topic: no new topic; the detector filters type = WITHDRAW on wallet.transactions. To deploy it separately, add a consumer group fraud-withdraw reading the same topic.
  • State: a sorted set fraud:wd:{A} (score = occurred_at, member = transaction_id) and a hash fraud:wdamt:{A} (field = transaction_id, value = amount in USD). The script: ZADD, HSET, trim entries older than 1 hour + 1 minute in both, then sum the amounts in the window.
  • Memory: 2,000/s × 3,600 s = 7.2 million entries at sustained peak, × ~(100 + 80) B ≈ 1.3 GB. Acceptable, but an O(n) sum per event can be slow for very active accounts.
  • Optimisation: also keep a running total fraud:wdsum:{A}; add on insert, and on trimming expired entries subtract exactly their amounts (read from the hash before deleting). Each event then only touches the new entry and the expired ones, never re-summing the whole window.
  • Idempotent: only add to the total when ZADD returns 1 (a new member), so redelivery never over-adds.

Remember

  • Estimate the Kafka load and Redis memory for 100 million transactions/day, with the arithmetic.
  • Explain why fixed windows miss bursts and over-count, and why a sorted set doesn’t.
  • Explain why a stateless detector scales easily, and why adding partitions doesn’t break the Redis count.
  • Know the order “send the rule hit, then set the flag”, and why.
  • Describe recovering from a lost Redis by replaying from Kafka.

Interview questions

Junior

Why use account_id as the transaction topic’s key?

So all of an account’s transactions land in one partition, keep their order, and only one detector handles that account at a time.

That way no two instances modify one account’s state. Load still spreads evenly because there are millions of accounts.

How do fixed and sliding windows differ?

Fixed windows cut time into adjacent buckets (10:00–10:05, 10:05–10:10); a sliding window always looks back 5 minutes from each event.

Fixed windows miss a burst that straddles two buckets; a sliding window doesn’t, at the cost of remembering each transaction.

Mid

Does adding partitions to the transaction topic break the count?

No, because state lives in Redis by account_id, not tied to a partition; whichever consumer gets the key reads the right sorted set.

Counting with Kafka Streams local state is different: state is split by partition, so adding input partitions needs a reset or a repartition. Over-provisioning partitions up front avoids the question anyway.

A consumer crashes and Kafka redelivers events. Is the count wrong?

No, if you count with a sorted set whose member is the transaction_id: adding the same member again doesn’t add an entry.

With INCR it would be. The general approach is a naturally idempotent operation, or dedupe by ID before adding.

An account is over the threshold and crosses it again on every transaction. How do you avoid 1,000 alerts?

Emit one rule hit per account per rule per period using a Redis SET NX PX flag, and have receivers dedupe by hit_id.

Send the rule hit first and set the flag after; the other way round, a crash between the two loses the alert.

Senior

Redis loses all its data. How does the system recover?

Bring up a new Redis, reset the group’s offsets to about 6 minutes ago and read again; the count is idempotent, so replay doesn’t over-count.

The detector’s state only depends on the last 5 minutes of the log and the topic keeps 7 days, so Redis needs no backup. During replay detection is a few minutes late; existing block flags are separate data that need a backup or a rebuild from the rule-hit topic.

Why count by event time rather than when the consumer processes it?

Because when a consumer lags and then catches up, counting by processing time squeezes 15 minutes of transactions into 5 and raises false alarms.

Event time comes from the commit time in the wallet’s DB (one clock source). Add allowed lateness for late events and a metric counting events dropped for being too late.

Would you use Kafka Streams instead of Redis?

Possibly: Streams has sliding windows, local state and Kafka-to-Kafka exactly-once built in. I decide by how often rules change and who needs to read the state.

Few fixed rules at high volume: Streams. Many rules tuned by the risk team and a wallet that reads a flag synchronously: Redis. Streams needs standby replicas so rebalances don’t wait on long state restores.

Useful phrases for interviews

  • “Fraud detection runs asynchronously off the transaction stream.”
  • “Every rule hit goes to a dedicated topic, and several services act on it.”
  • “A sliding window catches bursts that straddle two fixed windows.”
  • “We keep one sorted set per account, scored by event time.”
  • “Using the transaction ID as the member makes the count idempotent.”
  • “We count by event time, so consumer lag doesn’t cause false alerts.”
  • “A per-account cooldown prevents an alert storm.”
  • “Small payments fail open if the flag lookup times out; large ones fail closed.”
  • “If Redis is lost, we replay the last six minutes from Kafka.”
  • “The detector is stateless, so rebalances are cheap.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.