Skip to content
notes.vinhnq.com

Search

System DesignSenior22 min read

Redis: persistence, replication, Sentinel and Cluster

Review 16 cards
On this page

Redis series: 1. Redis internals: why it is fast · 2. Persistence, replication, Sentinel and Cluster · 3. Choosing data structures for real problems

Redis runs in RAM, so it has to answer three questions: what survives a restart (persistence), what happens when the machine dies (replication, failover), and how to split once one machine isn’t enough (Cluster). After reading, you can say what each setup loses, why, and pick a deployment mode. Read part 1 first if the single-threaded event loop and the role of fork() aren’t clear yet.

1. Persistence: RDB, AOF and hybrid

Key idea: RDB snapshots the whole dataset periodically (loses minutes), AOF logs every command (loses ~1 second), hybrid combines both. Choose by the role Redis plays.

1.1. RDB: a full snapshot

Key idea: BGSAVE calls fork() so a child process writes the snapshot while the parent keeps serving; the price is that every snapshot is a heavy fork().

  • SAVE: the main thread writes the file itself and blocks every command until it finishes. Almost never used in production.
  • BGSAVE: the child writes the snapshot from its copy of memory while the parent keeps accepting commands.
  • Automatic snapshots follow save; the Redis 7 default is save 3600 1 300 100 60 10000 (≥1 change in 1 hour, 100 in 5 minutes, or 10,000 in 1 minute).
flowchart LR
  M[Main process] -->|fork| C[Child process]
  M -->|keeps accepting writes| MEM[Modified memory pages get copied]
  C -->|reads the memory snapshot taken at fork| F[dump.rdb.tmp]
  F -->|atomic rename| R[dump.rdb]

Copy-on-write — parent and child share memory pages, and only pages the parent modifies after the fork get copied. The more writes during a snapshot, the more memory grows; in the worst case it nearly doubles.

Pros: compact files, fast reloads (a sequential read, no replaying of commands), handy for backups and moving data. Cons: you lose every change since the last snapshot (possibly several minutes).

1.2. AOF: logging every command

Key idea: AOF (append-only file — a file you only ever add to the end of) logs every write command, in the order execute first, log afterwards. appendfsync decides how much you lose when the machine dies.

flowchart LR
  CMD[Write command] --> EXE[Run in memory]
  EXE --> BUF[AOF buffer]
  BUF -->|write before replying in the next loop| OS[OS page cache]
  OS -->|fsync per appendfsync| DISK[(Disk)]
appendfsync Meaning Maximum loss when the machine dies
always fsync after every event loop iteration that had writes almost nothing (a few commands from the current iteration)
everysec (default) fsync once per second on a background thread ~1 second, up to 2 seconds if the disk is slow
no let the OS flush on its own (often ~30 seconds on Linux) tens of seconds

If the Redis process dies (killed, crashed), data already write()-ten to the page cache survives and the OS flushes it. Only when the whole machine dies (power loss, kernel panic) is the un-fsync-ed part lost.

Cons: larger files than RDB, slower reloads (commands are replayed), and always or a congested disk can slow down the main thread.

Why log after executing, AOF rewrite, multi-part AOF and when a slow disk blocks the main thread

Logging after execution has two benefits the slides mention: no need to validate a command’s syntax first (a bad command is rejected when it runs, so it never reaches the log), and logging doesn’t slow down the command itself. The cost: if Redis dies between running a command and logging it, that command is lost.

The AOF keeps growing, so Redis rewrites it periodically (BGREWRITEAOF, triggered automatically by auto-aof-rewrite-percentage 100 and auto-aof-rewrite-min-size 64mb): a child process writes the current state as the fewest possible commands (1,000 INCRs become a single SET).

Since Redis 7 the AOF is multi-part: a base file, incr files and a manifest, inside the appendonlydir directory. A rewrite only creates a new base and opens a new incr file, and no longer has to keep a rewrite buffer in memory as before.

1.3. Hybrid: RDB at the head, AOF at the tail

Key idea: when the AOF is rewritten, the base is written in RDB format and new commands are appended in AOF format. It loads almost as fast as RDB and loses as little as AOF.

aof-use-rdb-preamble arrived in Redis 4.0 and has been on by default since 5.0. On startup Redis loads the RDB part, then replays the remaining commands. The downsides the slides list: the file is no longer human-readable (the head is binary), and Redis before 4.0 can’t read it; the second hardly matters today.

2. Replication

Key idea: one master accepts every write and streams it asynchronously to replicas. You gain read scaling and a hot standby; you lose consistency and may lose writes on failover.

2.1. How it works

Key idea: a replica asks to sync with PSYNC; the first time or when far behind it gets a full sync (a whole RDB), after a short disconnect a partial sync from the backlog.

Replication by itself doesn’t fail over; promoting a replica is the job of Sentinel or Cluster.

PSYNC steps, full sync, partial sync and the 1 MB backlog trap
  1. The replica (the slides call it a slave, the old name changed in 5.0) connects and sends PSYNC <replid> <offset>.
  2. The first time, or when too far behind: a full sync. The master produces an RDB and sends it, buffering new commands meanwhile and sending them once the replica has loaded the snapshot. Since Redis 7, repl-diskless-sync yes is the default: streamed straight over the socket, no file written.
  3. After that the master streams every write command, asynchronously.
  4. After a short disconnect: a partial sync. The replication backlog — a ring buffer of the most recent commands (repl-backlog-size 1mb by default). If the replica’s offset is still inside it, only the missing part is sent.

2.2. Asynchronous means data can be lost

Key idea: the master replies “OK” before the replica receives the command. If the master dies right then and a replica takes over, the write is gone, even though the client saw it succeed.

sequenceDiagram
  participant C as Client
  participant M as Master
  participant R as Replica
  C->>M: SET order:1 paid
  M-->>C: OK
  Note over M: master dies before sending the command
  Note over R: replica is elected as the new master
  C->>R: GET order:1
  R-->>C: nil

Two tools reduce the risk rather than remove it:

  • WAIT numreplicas timeout: the client blocks until the previous writes are acknowledged by at least N replicas. “Nearly synchronous”, but an acknowledgement means received, not necessarily fsynced; on timeout the write has still been applied on the master.
  • min-replicas-to-write N + min-replicas-max-lag S: the master refuses writes if fewer than N replicas have a lag of ≤ S seconds. This bounds the window of lost writes when the master is isolated (split brain — two nodes both believing they are the master).

2.3. Reading from replicas

Key idea: reading from replicas spreads the read load, but you may see data a few ms to a few seconds old.

Fine for data that tolerates lag (product catalogues, leaderboards); wrong for read-your-writes (reading back what you just wrote), such as a profile that was just edited. In Lettuce you choose with ReadFrom.REPLICA_PREFERRED or ReadFrom.UPSTREAM.

3. Sentinel: automatic failover

Key idea: sentinels are separate processes (usually 3 or 5, on different machines) that monitor the master, promote a replica when it dies, and tell clients where the current master is.

  1. Monitoring the master and replicas with PING.
  2. Automatic failover: after down-after-milliseconds without a reply, a sentinel marks the master SDOWN (subjectively down — “I think it’s dead”). When a quorum agrees, it becomes ODOWN (objectively down). The sentinels elect a leader (which needs an absolute majority, Raft-style); the leader promotes the best replica.
  3. Service discovery: clients don’t hard-code the master but ask SENTINEL get-master-addr-by-name mymaster. Lettuce and Jedis support this out of the box and switch automatically on failover.
flowchart TB
  S1[Sentinel 1] -.monitors.-> M[Master]
  S2[Sentinel 2] -.monitors.-> M
  S3[Sentinel 3] -.monitors.-> M
  M -->|replication| R1[Replica 1]
  M -->|replication| R2[Replica 2]
  APP[Application] -->|asks for the current master| S1
  APP -->|reads, writes| M

A new master is up within seconds to tens of seconds. Cons: still asynchronous (a failover can lose writes), more processes to operate, and every write still goes to a single master, so neither writes nor capacity scale.

Which replica the leader promotes

Priority order: replica-priority, then the largest replication offset (the freshest data), then the smallest run ID. The leader sends REPLICAOF NO ONE to the chosen replica, then reconfigures the remaining replicas to follow the new master.

4. Redis Cluster: spreading data over several masters

Key idea: Cluster splits the keyspace across several masters, each with its own replicas, and fails over without Sentinel. In exchange, multi-key commands only work when every key is in the same slot.

4.1. Hash slots

Key idea: there are 16,384 hash slots; a key belongs to slot CRC16(key) mod 16384, and each master owns a set of slots.

flowchart TB
  K1["CRC16(key2) mod 16384 = 4998"] --> M1["Master 1: slots 0-5460"]
  K2["CRC16(key1) mod 16384 = 9189"] --> M2["Master 2: slots 5461-10922"]
  K3["CRC16(foo) mod 16384 = 12182"] --> M3["Master 3: slots 10923-16383"]
  M1 --> S1[Replica 1]
  M2 --> S2[Replica 2]
  M3 --> S3[Replica 3]

Nodes talk over the cluster bus (data port + 10000) using gossip — each node tells a few others who is alive and who owns which slots, and the news spreads through the cluster. When a majority of masters sees a master silent for longer than cluster-node-timeout, one of its replicas stands for election and takes over.

Why 16,384 slots, and why slots instead of consistent hashing

Why 16,384 and not 65,536 (CRC16 yields up to 65,536 values)? Every heartbeat packet carries a bitmap of the sender’s slots: 16,384 bits = 2 KB, 65,536 bits = 8 KB, too heavy for packets sent constantly. Cluster is designed for at most ~1,000 masters, for which 16,384 slots is fine-grained enough.

Why slots and not consistent hashing? Slots let you move individual slots explicitly (resharding, rebalancing), and “which slot lives where” is a small table that every node and client can cache.

4.2. How the client finds the right node: MOVED and ASK

Key idea: a client can send to any node; a node that doesn’t own the slot answers with a redirect. MOVED = the slot has moved for good (update the table), ASK = the slot is mid-migration (redirect this request only).

  • -MOVED 3999 10.0.0.2:6379: the client updates its slot table and retries. Good clients (Lettuce, Jedis JedisCluster) load the slot table at startup with CLUSTER SHARDS and hit the right node from the start.
  • -ASK 3999 10.0.0.3:6379: this key is already on the new node. The client sends ASKING followed by the command to the new node once, without updating its table.

4.3. Multi-key commands and hash tags

Key idea: a command touching several keys only works when every key is in the same slot. A hash tag — a non-empty {...} part of the key — means only that part is hashed, so related keys land in the same slot.

Affected commands: MGET, SUNION, RENAME, MULTI transactions, Lua scripts. Across slots Redis returns CROSSSLOT Keys in request don't hash to the same slot.

Example: user_profile:34 and user_session:34 land in slots 11933 and 12053 (possibly two nodes), so MGET on both fails. Add the {34} tag and both land in slot 9745.

The slot hash function in Java, and the too-coarse hash tag trap

I rewrote Redis’s hashing function in Java to check it:

static int slot(String key) {
    int open = key.indexOf('{');
    if (open >= 0) {
        int close = key.indexOf('}', open + 1);
        if (close > open + 1) key = key.substring(open + 1, close); // hash only the part inside {}
    }
    return crc16(key.getBytes(StandardCharsets.UTF_8)) & 16383;     // = mod 16384
}
crc16("123456789") = 0x31C3     (matches the standard CRC16-XMODEM value in the Redis docs)
foo                  -> 12182
user_profile:34      -> 11933
user_session:34      -> 12053
user_profile:{34}    -> 9745
user_session:{34}    -> 9745
{}user:34            -> 3665    (empty braces: the whole key is hashed)

4.4. What Cluster doesn’t allow

Key idea: adding masters scales reads, writes and capacity, but you give up a few features of single-node Redis.

  • Only database 0, no SELECT.
  • Multi-key commands, transactions and Lua only within one slot.
  • KEYS/SCAN only per node; to walk the whole keyspace you iterate over every master.
  • Pub/sub messages are normally broadcast to every node (since 7.0 there is sharded pub/sub, SPUBLISH, scoped to a slot).
  • The client must support cluster mode; some old libraries and proxies don’t.

Other cons: still asynchronous (writes lost on failover), more complex operations (resharding, slot rebalancing).

5. Choosing a deployment mode

Key idea: the slides recommend Cluster; I think you should choose by need. If the data fits on one machine, Sentinel is usually enough. How to judge “fits on one machine” (commands per second, the ~32 GB per-instance limit) and where Redis sits among the cache tiers is in Caching in system design: cache tiers and large-scale counters.

Exercises

Exercise 1: choose persistence and a mode for three Redis instances

Task: a booking system has three Redis needs: (a) caching flight search results, 8 GB; (b) login sessions for 2 million users, 3 GB, where a lost session means the user has to log in again; (c) holding seats for 10 minutes before payment, 200 MB, where a loss could lead to the same seat being sold twice. Choose persistence, an HA mode and eviction for each.

Solution to Exercise 1
  • (a) Search cache: no persistence (or infrequent RDB to avoid a cold cache), allkeys-lru, Sentinel or one master + replica. Losing everything only makes the database run hot for a while; add stampede protection for restarts.
  • (b) Sessions: AOF everysec + hybrid, noeviction (sessions must not be evicted to make room), Sentinel with 3 nodes. Losing 1 second of new sessions when a machine dies is acceptable (users log in again).
  • (c) Seat holds: Redis should not be the source of truth. Seats must be held in the database with a conditional statement or SELECT ... FOR UPDATE (see concurrent seat holds); Redis only serves as a fast front-line filter or an expiry timer. The reason: an asynchronous failover can lose a seat hold that was just written, and two people buy the same seat. If you still use Redis, give it a separate instance, noeviction, AOF, min-replicas-to-write 1, and keep the database as the final check.

Exercise 2: size the backlog

Task: the master writes 8 MB/s of replication commands on average, 20 MB/s at peak. The network between two regions can flap for up to 30 seconds. What should repl-backlog-size be?

Solution to Exercise 2

For a partial sync to still work after 30 seconds at peak, you need 20 MB × 30 = 600 MB; with a safety margin, about 1 GB. The backlog is allocated only once on the master (shared by all replicas), so 1 GB is an acceptable cost compared with a full sync of a dataset of tens of GB.

At the same time, raise client-output-buffer-limit replica so a full sync (if it still happens) isn’t cut off halfway, and track master_repl_offset against each replica’s offset to alert on lag.

Remember

  • Say how much data each setup (RDB, AOF always/everysec/no, hybrid) loses when the process dies and when the whole machine dies.
  • Explain why fork() blocks the main thread and inflates memory, and how to mitigate it.
  • Explain why a write acknowledged with “OK” can still vanish on failover, and how far WAIT/min-replicas-to-write help.
  • Know why you need at least 3 sentinels, and how a quorum differs from a majority.
  • Tell MOVED from ASK, use hash tags, and know when to pick Sentinel over Cluster.

Interview questions

Mid

How do RDB and AOF differ?

RDB is a periodic snapshot (fast reload, loses changes since the last one); AOF logs every command (loses ~1 second with everysec, but the file is larger and reloads more slowly).

RDB is created by BGSAVE (forking a child process) and the file is compact. AOF fsyncs according to appendfsync (every second by default) and needs periodic rewrites. Hybrid mode (the default since 5.0 when AOF is on) writes the base in RDB format and then appends commands in AOF format.

Does BGSAVE block Redis?

Writing the file doesn’t, but fork() itself does block the main thread.

fork() copies the page table, proportionally to the amount of RAM (tens to hundreds of ms on a large instance). Copy-on-write also makes memory grow with the amount of writes during the snapshot. SAVE, on the other hand, blocks completely.

Is Redis replication synchronous or asynchronous? What are the consequences?

Asynchronous: the master replies to the client before the replica receives the command.

Consequences: reads from replicas can be stale, and when the master dies and a replica is promoted, writes acknowledged with “OK” that hadn’t reached the replica are lost. WAIT and min-replicas-to-write reduce the risk but don’t make it truly synchronous.

What is a hash tag in Redis Cluster?

A non-empty {...} part of a key: only that part is hashed with CRC16 to pick the slot.

It puts related keys (user_profile:{34}, user_session:{34}) into the same slot, so multi-key commands, transactions and Lua can run on them. A tag that is too coarse piles many keys onto one node.

Senior

How does a Sentinel failover work? How many sentinels do you need?

SDOWN → a quorum makes it ODOWN → a leader is elected by majority → the leader promotes the best replica. You need at least 3 sentinels on 3 independent machines.

Each sentinel pings the master; after down-after-milliseconds without a reply it marks it SDOWN. The leader picks a replica (priority, largest offset, run ID), promotes it, reconfigures the other replicas and informs clients through service discovery. Three sentinels so that a majority still exists after losing one machine.

Why does Redis Cluster have 16,384 slots?

Because heartbeats carry a slot bitmap: 16,384 bits is 2 KB, while 65,536 bits is 8 KB, too heavy to send constantly.

Cluster targets at most ~1,000 masters, so 16,384 slots are fine-grained enough to spread evenly. Slots also allow moving data explicitly, piece by piece, during resharding.

What is the difference between MOVED and ASK?

MOVED: the slot now belongs to another node permanently, so the client updates its slot table. ASK: the slot is being migrated, so the client redirects this one request only.

With ASK, this particular key is already on the target node; the client sends ASKING followed by the command to the target, without updating its slot table, because the slot still officially belongs to the source node until the migration finishes.

When do you choose Sentinel over Cluster?

When the data fits on one node and the write load fits one command thread.

Sentinel gives you automatic failover while keeping every multi-key command, multiple databases and Lua without slot restrictions, with simpler clients. Choose Cluster when you need to scale writes or capacity beyond one node, or want small instances so forks are fast, and you accept designing keys with hash tags.

Can Redis be used as a primary data store?

Yes, for data that fits in RAM and can tolerate a little loss; no, for data where acknowledged writes must never be lost.

You lose about 1 second with AOF everysec, more on failover because replication is asynchronous. It is a poor fit for money, inventory, or data needing complex queries, relations or multi-table transactions. Then Redis is a cache or an acceleration layer in front of the database, or you use a service with a durable log such as MemoryDB.

Useful phrases for interviews

  • “RDB takes a point-in-time snapshot in a forked child, so you can lose everything since the last snapshot.”
  • “With AOF everysec, losing the process is fine; losing the whole machine loses about one second of writes.”
  • “fork itself blocks the main thread while it copies the page table, and copy-on-write inflates memory under heavy writes.”
  • “Replication is asynchronous, so a write acknowledged by the master can disappear on failover.”
  • “WAIT and min-replicas-to-write narrow the window of data loss, but they don’t make Redis strongly consistent.”
  • “A bigger replication backlog turns a full resync into a partial one when the network flaps.”
  • “The quorum only marks the master as down; electing a leader to run the failover needs a majority of sentinels.”
  • “Every key maps to one of 16,384 hash slots, and a MOVED redirect tells the client to refresh its slot map.”
  • “We use a hash tag per user so multi-key commands work, but never a tag as coarse as a whole tenant.”
  • “If the data fits on one node, I’d start with Sentinel and only move to Cluster when writes or memory outgrow it.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.