Microservices in system design series: 1. Service boundaries · 2. Distributed transactions: 2PC, saga, outbox · 3. Case study: a liveness system
1 million IoT devices send a heartbeat every 3 seconds. When a device stays silent past a threshold, the system has to report it as “disconnected”. It sounds simple, but it has everything system design is about: a huge write load, timers for millions of objects, partitioned state, and false alarms when the system itself slows down. This is the homework for the microservices module, and I worked it out in full. The same problem with a deeper look at the data structures (ZSET, timing wheel) and the queue between threads (LMAX Disruptor): A liveness system: from a Redis sorted set to a timing wheel and the LMAX Disruptor.
1. Clarifying requirements
Key idea: pin down the numbers and the definition of “disconnected” first. The original problem leaves the alert latency and the threshold configuration open.
Functional
- Devices send a heartbeat periodically (every 3 s). No heartbeat past the threshold → raise a disconnected alert (the original problem mistakenly says “connected”).
- The threshold is configurable per device type (camera 10 s, temperature sensor 60 s, electricity meter 5 minutes).
- (My additions; worth asking) Report “reconnected” when a device comes back; view the current state of a device.
Non-functional
- 1M devices, possibly 5M within 2 years.
- Alert latency: a little delay is fine; I settle on ≤ 5 s after the deadline.
- No duplicate alerts: one offline = one alert.
- High availability; losing one worker must not create a flood of false alarms.
2. Estimation
Key idea: compute the two numbers that drive the design: write load and state size.
| Quantity | Calculation | Result |
|---|---|---|
| Heartbeats/s | 1,000,000 / 3 | ~333K/s (5M devices: ~1.7M/s) |
| Inbound bandwidth | 333K × ~100 B | ~33 MB/s |
| Heartbeats/day | 333K × 86,400 | ~29 billion |
| State | 1M × (16 B id + 8 B deadline + type + overhead) | ~100 MB |
| Alerts | assume 0.1% of devices drop per hour | ~1,000/hour, tiny |
Conclusion: don’t store each heartbeat in a relational database (29 billion rows/day, 333K writes/s). The state fits in one machine’s RAM, but 333K/s is enough load to split across a few workers. If you need heartbeat history for analytics, push it to cheap storage (Kafka → S3/ClickHouse in batches), off the detection path.
3. API and ingestion
Key idea: ingestion has to be lightweight and handle 333K connections/s. MQTT or UDP suit IoT; with HTTP, batch.
- MQTT (a lightweight pub/sub protocol for IoT with long-lived connections): devices publish to the topic
hb/{deviceId}. MQTT brokers (EMQX, HiveMQ) have a built-in “client disconnected” event and Last Will, but a dropped TCP connection doesn’t necessarily mean the device is dead, and many devices only connect briefly. You still need a separate liveness layer. - HTTP:
POST /v1/heartbeats {deviceId, seq, sentAt}, returning204. The gateway writes straight to Kafka and never touches a database. - Use the server receive time to compute the deadline, not the device’s
sentAt(device clocks drift).
A small config table lives in PostgreSQL: device(id, type, tenant), device_type(type, timeout_ms). Workers cache it and refresh on config-change events.
4. Four options
Key idea: all four “store a deadline and find the expired ones”; they differ in where they store it and how they search. Choose by write load and operational complexity.
| Option | Write per heartbeat | Finding expired ones | Handles 333K/s? | Complexity |
|---|---|---|---|---|
A. last_seen column in the DB + scan job |
UPDATE one row |
WHERE last_seen < now() - timeout every few seconds |
No | Low |
| B. Redis ZSET (score = deadline) | ZADD O(log n) |
ZRANGEBYSCORE -inf now every second |
Yes, with Redis Cluster | Medium |
| C. Kafka + timing wheel in workers | Update an in-memory map | Timing wheel, O(1) per tick | Yes, scales by partition | Higher |
| D. A delay message per heartbeat | Send one scheduled message | Check when the message comes due | No (333K scheduled messages/s) | Medium |
4.1. Option B: Redis ZSET
On each heartbeat: ZADD live:{shard} <now + timeout(type)> <deviceId> (overwrites the old score). One checker per shard runs every second: take the members with score ≤ now, emit DeviceOffline, then ZREM (combined in Lua so two checkers don’t emit duplicates).
- 333K
ZADD/s is more than one Redis node can take (~50K), so split into 64 ZSETslive:{0..63}byhash(deviceId) % 64, spread over 8–10 masters. - Each ZSET holds ~16K members, so
ZADDand scans are very cheap. - Simple and easy to explain; the downside is that the whole heartbeat load lands on Redis, and you have to run a cluster.
4.2. Option C: Kafka + timing wheel (chosen)
flowchart LR
D[1M devices] --> GW[Ingest gateway]
GW -->|key = device_id| K[[Kafka heartbeats 64 partitions]]
K --> W1[Liveness worker P0..P7]
K --> W2[Liveness worker P8..P15]
W1 --> E[[Kafka device-status]]
W2 --> E
E --> AL[Alert service]
E --> ST[(Current state)]
CFG[(device_type timeout)] -.-> W1
- The gateway writes heartbeats to the
heartbeatstopic with key =device_id, so every heartbeat from one device lands in the same partition and the same worker. - Each worker keeps a
Map<deviceId, deadline>for its partitions plus a hashed timing wheel: a ring of N slots, each covering a time span (say 100 ms); a device goes into the slot matching its deadline. On each tick the worker looks at one slot, and any device whose deadline has really passed (it wasn’t extended) gets aDeviceOffline. - Extending a deadline is just updating it in the map (lazily: the old slot still holds the device, and when that slot comes up the worker sees the deadline changed and skips it, or moves it to a new slot).
- State events (
ONLINE → OFFLINE,OFFLINE → ONLINE) go to thedevice-statustopic (compacted, key = device_id); the Alert service reads from it.
Why C: there’s no external store on the hot path (each heartbeat is just a Kafka write + a RAM update), it scales by adding partitions and workers, and Kafka already keeps the heartbeats, so if a worker dies another one can re-read them.
Timing wheel: why not a PriorityQueue or DelayQueue
DelayQueue/PriorityQueue is a heap: insert and poll are O(log n), and extending an element means removing and re-adding it (removing an arbitrary element from a Java heap is O(n)). With 40K devices per worker, that’s ~40K extensions/s. A heap can handle it, but it burns CPU and creates garbage for the GC.
A hashed timing wheel (Netty’s HashedWheelTimer; Kafka’s purgatory uses a hierarchical variant): insert is O(1), a tick is O(1) + the number of entries in the slot. With an acceptable delay of 1–5 s, a 100 ms tick × 1,024 slots = a 102.4 s ring. Longer deadlines (a 5-minute threshold) use extra rounds or a multi-level wheel.
The cheapest way to extend: only update deadline in the HashMap. When a slot comes up, for each device in it: if deadline > now, move it to a new slot; otherwise emit offline. The number of moves ≈ the number of times a device “would have expired”, which is tiny compared to the number of heartbeats.
void onHeartbeat(String id, long now) {
Entry e = map.computeIfAbsent(id, k -> new Entry(k));
boolean wasOffline = e.offline;
e.deadline = now + timeoutOf(id);
e.offline = false;
if (wasOffline) emit(id, "ONLINE", now);
if (!e.scheduled) { wheel.add(e, e.deadline); e.scheduled = true; }
}
void onTick(long now) {
for (Entry e : wheel.drainSlot(now)) {
if (e.deadline > now) { wheel.add(e, e.deadline); continue; } // deadline was extended
e.scheduled = false;
if (!e.offline) { e.offline = true; emit(e.id, "OFFLINE", e.deadline); }
}
}
5. False alarms: three sources and how to stop them
Key idea: the worst failure of a liveness system isn’t missing a device. It’s reporting masses of devices offline because the system itself is slow, or because of a shared network outage.
- Rebalance / a new worker takes over a partition: the new worker has no state yet. Don’t emit offline until a “grace period” equal to the largest timeout has passed, or rebuild state from the
device-statustopic (compacted) plus re-reading the last few minutes of heartbeats. - A whole region goes dark: 50,000 devices behind one cell tower drop at once. The Alert service groups by
(tenant, region, minute): when more than X% of a region’s devices drop, it sends one “region disconnected” incident instead of 50,000 messages. - Flapping: a device keeps going online and offline. Only report offline after expiry, and only report online again after N consecutive heartbeats; or cap the number of alerts per device per hour.
6. Deduplicated alerts and current state
Key idea: every state transition is an event with a deterministic id, so the alert sender can deduplicate even when an event arrives several times.
- Event
DeviceOffline{deviceId, offlineSince}: the dedup key =deviceId + offlineSince. The Alert service inserts into analerttable with a unique constraint on that key and skips duplicates (at-least-once from Kafka still yields exactly one alert). - Current state for the dashboard: a consumer of
device-statusupdates adevice_status(device_id, status, since)table. State changes are rare (~1,000/hour), so PostgreSQL has plenty of headroom. last_seenfor the detail screen: workers flush it in batches every 60 s (1M rows/minute ≈ 17K rows/s as batchedUPSERTs), or you just ask the worker through an API when needed.
7. Presenting it in 45 minutes
Key idea: spend the time on the interesting parts (timers for millions of objects, and false alarms), and don’t get lost in protocols.
| Minutes | Content |
|---|---|
| 0–5 | Clarify requirements: definition of offline, per-type thresholds, alert latency, whether to report reconnects, growth |
| 5–10 | Estimate: 333K/s, 100 MB of state, rule out “write to the DB on every heartbeat” |
| 10–20 | Options A → B → C, draw the C diagram, explain partitioning by device_id |
| 20–30 | Deep dive into the timing wheel and cheap extensions |
| 30–40 | False alarms: lag/watermark, rebalance, regional outage, flapping; alert dedup |
| 40–45 | Scaling to 5M devices, monitoring the liveness system itself (lag, alerts/minute), B vs C trade-off |
The other homework for this module (the four deadlock conditions and a simple way to avoid deadlock) is solved in the post on optimistic and pessimistic locking.
Exercises
Exercise 1: Changing thresholds through config
Problem: an admin changes the “camera” threshold from 10 s to 30 s while the system is running. Design it so the change takes effect within 10 s, with no false alarms and no missed ones.
Solution to Exercise 1
- The config change is written to
device_typeand aDeviceTypeTimeoutChanged{type, timeoutMs, version}event goes to a (compacted) config topic. Every worker reads that topic and updates its cache within a few seconds. - Apply it lazily: the new deadline only takes effect from each device’s next heartbeat (at most 3 s away), so there’s no need to walk through 1M entries.
- Raising the threshold (10 → 30 s): a device with an old (shorter) deadline can only expire before its next heartbeat if it has really been silent > 10 s. When a tick sees it expired, the worker recomputes
lastSeen + newTimeoutbefore alerting, and if that hasn’t arrived yet, moves it to a later slot. This means storinglastSeen, not justdeadline. - Lowering the threshold (30 → 10 s): also recompute
lastSeen + newTimeouton the tick. A device silent > 10 s gets reported on the next tick of its current slot, at worst after the time until that slot comes up. To be faster, when the lower-threshold event arrives, scan the devices of that type once (a few tens of thousands) and move their slots. - Put the config
versionon the offline event so you can trace why an alert fired.
Remember
- Work out 333K heartbeats/s and 100 MB of state, and explain why you don’t write to the DB on every heartbeat.
- Compare the four options (last_seen column, Redis ZSET, timing wheel, delay message).
- Explain the hashed timing wheel and how to extend deadlines cheaply.
- Name the three sources of false alarms (lag, rebalance, regional outage) and how to stop each one.
- Design deduplicated alerts with the key
deviceId + offlineSince.
Interview questions
Mid
Why not store last_seen in PostgreSQL on every heartbeat and scan it periodically?
333K UPDATEs/s is far beyond what one relational node can write (a few thousand to a few tens of thousands of writes/s), and every UPDATE creates a new row version that has to be VACUUMed.
Scanning WHERE last_seen < ... over 1M rows every few seconds is expensive too. This is transient state, so keep it in RAM or Redis, and let the DB store only state changes.
How do you detect expired devices with a Redis ZSET?
The score is the deadline: each heartbeat ZADDs and overwrites the score with now + timeout; every second a checker runs ZRANGEBYSCORE key -inf now to get the expired devices, raises alerts and ZREMs them.
Combine the fetch and the delete in Lua so two checkers don’t emit duplicates. Split into several ZSETs by hash of device_id once the load exceeds one node.
How do you make one offline produce exactly one alert?
Emit a state-transition event (only on ONLINE → OFFLINE) and deduplicate at the alert sender with the key deviceId + offlineSince under a unique constraint.
Kafka delivers at least once, so an event can arrive twice; a deterministic key makes the receiver idempotent.
Senior
A Kafka consumer is 30 seconds behind. How does your liveness system react?
It must not mass-report devices offline: deadlines are in data time and only expire up to the partition’s watermark; once lag passes a threshold, detection pauses for that partition and one infrastructure incident is raised.
After it catches up, the worker processes the backlog of heartbeats, deadlines get extended correctly, and only devices that were truly silent get reported.
Why does partitioning by device_id matter in this design?
Every heartbeat from a device lands in the same partition, so exactly one worker holds that device’s state and deadline, with no locks and no shared store.
You scale by adding partitions and workers. The price: a rebalance leaves the new worker without state, so you need a grace period or a rebuild from the compacted state topic.
A whole region loses its network and 50,000 devices go offline at once. How do you handle alerts?
Group by region/tenant over a short window: once a ratio threshold is crossed, raise one “region disconnected” incident and hold back the per-device alerts.
When the region recovers, close the incident and only report the devices that are still offline. You need region info (cell tower, gateway) in the device data to group at all.
How would you scale the design to 5 million devices?
Load rises to ~1.7M heartbeats/s and state to ~500 MB: add Kafka partitions (say 256) and workers, scale the gateway horizontally; the timing wheel stays the same.
Check bandwidth (~170 MB/s inbound), consider batching heartbeats at the gateway (one record carrying many devices), and lower the heartbeat rate for device types with long thresholds (with a 5-minute threshold, one heartbeat every 60 s is enough).
Useful phrases for interviews
- “One million devices every three seconds is about three hundred thirty thousand heartbeats per second.”
- “The state is only about a hundred megabytes; the hard part is timers, not storage.”
- “Writing last_seen to the database on every heartbeat simply doesn’t scale.”
- “We partition by device ID, so exactly one worker owns each device’s deadline.”
- “A hashed timing wheel gives O(1) timer inserts, and extending a deadline is just a map update.”
- “Every heartbeat pushes the device’s deadline forward.”
- “Timers only fire up to the partition’s watermark, never on wall-clock time alone.”
- “If consumer lag spikes, we pause offline detection instead of firing a million false alarms.”
- “A new worker waits one grace period before firing timers.”
- “Grouping by region turns an alert storm into one incident.”
Notes based on the System Design course (Ronin Engineer), rewritten in my own words.