Caching series: 1. Cache-aside and invalidation · 2. Cache failures: avalanche, breakdown, penetration · 3. Reviewing a Redis distributed lock
Redis can take an API from 80 ms to 5 ms, and it brings a new kind of bug: you change your age from 33 to 34, refresh, still see 33, and only see the right value ten minutes later. This post answers: when the data changes, how do we stop the cache from lying, and if it does lie, for how long at most. Read Transactions, isolation levels and locking first, since most of this post is about race conditions; how Redis stores data, expires keys, persists, replicates and shards is in the Redis series.
1. What a cache is and how to measure it
Key idea: a cache holds data temporarily so it can be read again faster, and it only pays off for hot data. The main metric is the hit rate (the share of reads served by the cache).
A cache holds either a copy of source data (a customer row) or a computed result (today’s order total, a search page). The cache is the database’s shield: databases are expensive and hard to scale out, so the cache absorbs most reads. The price: performance vs cost and performance vs consistency.
Hot data is the small slice that gets most of the reads (80/20). With hit rate h:
average latency ≈ t_cache + (1 − h) × t_db
database load = (1 − h) × total reads
Example with t_cache = 1 ms, t_db = 20 ms, 10,000 req/s:
| Hit rate | Average latency | Queries/s to the DB |
|---|---|---|
| 95% | 2 ms | 500 |
| 99% | 1.2 ms | 100 |
Latency barely moves, but database load drops fivefold. So when the hit rate falls (a deploy changes the key format, someone flushes the whole cache), the database dies first.
2. Where caches live
Key idea: the same data can sit in four layers at once (client, CDN/proxy, app, Redis), each expiring in its own way. That’s half the reason invalidation is hard.
flowchart LR
C[Client<br/>caches responses] --> CDN[CDN<br/>html, js, css, media]
C --> P[Reverse proxy<br/>static files]
CDN --> S
P --> S[App server<br/>local cache: config,<br/>small catalogs]
S --> R[Remote cache Redis<br/>query results, computed results,<br/>shared data]
S --> DB[(Database<br/>buffer pool)]
R -. miss .-> DB
Nobody can purge a client’s HTTP cache. MySQL dropped its query cache in 8.0 for the same reason: every write to a table had to invalidate every related result under a global mutex.
3. Read strategies
Key idea: with read-through the cache library loads on a miss; with cache-aside the app does all five steps itself. Cache-aside is the most common way to use Redis.
Before asking “how do we store it”, ask “how is the data accessed”: reads or writes.
- Read-through: the app only talks to the cache; on a miss, the cache library calls a loader to read the database and stores the result (Caffeine’s
LoadingCache, or Spring’s@Cacheableas seen from the app). - Cache-aside (read-aside): the app reads the cache itself, and on a miss reads the database and writes the cache.
sequenceDiagram
participant A as App
participant C as Cache
participant D as DB
A->>C: 1. GET key
C-->>A: 2a. hit: return value
A->>D: 2b. miss: SELECT
D-->>A: 3. row
A->>C: 4. SET key value EX ttl
A-->>A: 5. return result
Cache-aside tolerates a Redis failure (fall back to the database, provided the client has a short timeout) and can cache any data shape (a DTO that joins three tables). The price: the logic lives in the app, and there is a window of inconsistency between reading the database and writing the cache (section 6).
Cache-aside code with Spring Boot 3 and the Redis client timeout trap
Catching cache errors yourself makes “tolerates cache failures” actually true:
@Service
class CustomerQueryService {
private static final Duration TTL = Duration.ofMinutes(10);
private final StringRedisTemplate redis;
private final CustomerStore store;
private final ObjectMapper json;
static String key(long id) { return "customer:v1:" + id; }
public CustomerView get(long id) {
String key = key(id);
String cached = readCache(key); // 1. read the cache
if (cached != null) return fromJson(cached); // 2. hit
CustomerView view = store.findView(id) // 3. miss -> database
.orElseThrow(() -> new NoSuchElementException("customer " + id));
writeCache(key, toJson(view)); // 4. populate with a TTL
return view;
}
private String readCache(String key) {
try {
return redis.opsForValue().get(key);
} catch (DataAccessException e) { // Redis down or timed out: fall back to the DB
log.warn("cache read failed for {}: {}", key, e.getMessage());
return null;
}
}
// writeCache: set(key, value, TTL) in the same kind of try/catch; toJson/fromJson use ObjectMapper
}
The key carries a version (v1), so changing the DTO shape only means moving to v2; the value is JSON rather than JDK serialization.
4. Write strategies
Key idea: write the cache together with the database (write-through), write the cache and flush the database later (write-back), or write the database and invalidate the cache (write-around). For business data, write-around + cache-aside is the sensible default.
- Write-back returns immediately and flushes to the database later in batches; the name comes from CPUs (write to the cache, push to RAM later).
- Write-around decouples the cache from storage and often invalidates asynchronously; data that’s both written and read often keeps missing.
5. The common problem: inconsistency and invalidation
Key idea: every strategy has moments when the cache says Age=26 while the database already says 27. The cure is cache invalidation (removing data that’s no longer valid); it’s hard because of timing, concurrency, data relationships and caches everywhere.
Four kinds of invalidation:
- Time-based: a TTL (time to live, how long a key survives), expiring on its own.
- Command-based: after a write, the app updates or deletes the key.
- Event-based: listen for change events (application events, CDC) and delete.
- Group-based: drop a whole group of keys by tag or prefix.
Besides timing (how long should the TTL be?) and races, there are data relationships (one product lives in its detail key, listing pages, search results) and root causes that are hard to find: the bug depends on exact timing and then heals itself when the TTL runs out.
Group invalidation: KEYS, SCAN, tag sets and versioned namespaces
6. Case study: three attempts at fixing inconsistency
Key idea: updating the cache lets two writers diverge; deleting doesn’t. Against readers, “delete first” breaks often, while “commit the DB, then delete” only breaks in a very rare race, so that’s the answer.
The read side is cache-aside; the write side is write-around, choosing between update (write the new value into the cache) and delete (remove the key). Every diagram below matches a small program that replays each interleaving against a HashMap “database” and a HashMap “cache”.
Attempt 1: update the cache, before or after the database?
sequenceDiagram
participant W1 as Request 1 (age=1)
participant C as Cache
participant D as DB
participant W2 as Request 2 (age=2)
Note over W1,W2: Update cache first
W1->>C: SET age=1
W2->>C: SET age=2
W2->>D: UPDATE age=2
W1->>D: UPDATE age=1
Note over C,D: cache=2, DB=1 → the DB is wrong
Note over W1,W2: Update cache later
W1->>D: UPDATE age=1
W2->>D: UPDATE age=2
W2->>C: SET age=2
W1->>C: SET age=1
Note over C,D: DB=2, cache=1 → the cache is wrong until the TTL
Attempt 2: delete the cache
Two writers each write the database and then delete the key, and any order works: the row lock serializes the two UPDATEs, the cache is empty, and the next read reloads. A delete is idempotent (doing it twice equals doing it once) and carries no value. But readers write to the cache too (step 4), so attempt 3 pits a writer against a reader with the combination RA + DWA (Read-Aside + Delete-Write-Around).
Attempt 3: RA + DWA, a writer against a reader
Delete the cache first, then update the database:
sequenceDiagram
participant W as Writer (age 0→1)
participant C as Cache
participant D as DB
participant R as Reader
W->>C: DEL age
R->>C: GET age (miss)
R->>D: SELECT age → 0
R->>C: SET age=0
W->>D: UPDATE age=1
Note over C,D: cache=0 (stale), DB=1 — stale until the TTL
The window is wide: the whole UPDATE and the transaction around it.
Update the database first, then delete the cache:
sequenceDiagram
participant R as Reader
participant D as DB
participant C as Cache
participant W as Writer (age 0→1)
R->>C: GET age (miss)
R->>D: SELECT age → 0
W->>D: UPDATE age=1 (commit)
W->>C: DEL age
R->>C: SET age=0
Note over D,C: DB=1, cache=0 — still inconsistent
Still inconsistent, but three conditions must hold: the key isn’t cached, the reader reads the database before the writer commits, and the writer’s UPDATE + commit + DEL all fit between the reader’s database read and its SET. That gap is one Redis round-trip (under 1 ms), while a committed write (with an fsync of the redo log or WAL) takes much longer, so it’s very rare.
The answer
Update the database first, delete the cache afterwards, plus a reasonably short TTL as a safety net for that rare race or a failed delete.
Code for deleting the cache after commit with Spring
The write side publishes an event, and the delete runs after commit:
@Transactional
public void changeAge(long id, int age) {
store.updateAge(id, age);
events.publishEvent(new CustomerChanged(id)); // delivered only after COMMIT
}
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
void onChanged(CustomerChanged event) {
String key = CustomerQueryService.key(event.id());
try {
redis.delete(key);
} catch (DataAccessException e) {
retryQueue.enqueue(key); // retried asynchronously with backoff
}
}
7. Beyond the slides: closing the remaining gap
Key idea: three ways to close the “slow reader puts the old value back” race: a double delete only narrows the window; versions + tombstones and leases reject the old value deterministically.
- Delayed double delete: delete right after commit, then again after a delay longer than one database read plus replication lag (a few hundred ms to a few seconds). Run the second delete through a delay queue or job, never
Thread.sleepin the request. - Version + tombstone: each row has a
versioncolumn that increments on every write. Instead ofDEL, the writer leaves a short-lived tombstone (a “deleted” marker) saying “invalidated at version N”; a reader may only populate if the version it read is not older. - Leases (the “Scaling Memcache at Facebook” paper): on a miss, memcache hands out a lease token; a
SETis only accepted with the token, and aDELETEin the meantime invalidates it. Redis has no built-in leases; tombstones plus versions are the closest imitation.
Lua for version + tombstone, a test run and more on leases
The check and the write must be atomic, so use Lua:
-- populate: KEYS[1]=key ARGV[1]=version ARGV[2]=payload ARGV[3]=ttlMs
local cur = redis.call('HGET', KEYS[1], 'v')
if cur and tonumber(cur) > tonumber(ARGV[1]) then return 0 end
if cur and tonumber(cur) == tonumber(ARGV[1]) and redis.call('HEXISTS', KEYS[1], 'data') == 1 then return 0 end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'data', ARGV[2])
redis.call('PEXPIRE', KEYS[1], ARGV[3])
return 1
-- invalidate (newVersion): if the stored version is lower, DEL the key, HSET v=newVersion, short PEXPIRE
Run against Redis 7, with a slow reader holding version 7 and a writer bumping it to 8:
writer invalidate(v8) -> 1
stale reader populate(v7) -> 0 (0 = rejected)
fresh reader populate(v8) -> 1 {v=8, data={"age":34}}
Memcache leases also rate-limit how often a token is issued per key, which stops the thundering herd (a crowd of requests hitting the database at once) when a hot key expires (post 2).
8. Problem 01: no atomicity between the database and the cache
Key idea: if the UPDATE succeeds but the Redis DEL fails, the cache keeps the old value until the TTL. The delete has to be retried until it succeeds: retries, an outbox or CDC.
Customer X changes their age from 33 to 34 and complains that “it takes a while to show up”, even though the system does things in the right order. The DEL failed (a timeout, a Redis failover, the pod killed halfway).
- Retry: synchronously with backoff, or through a queue.
- Outbox: write a
cache_invalidation(key)row in the same transaction as theUPDATE; a job reads the outbox and deletes the keys. - CDC (change data capture, reading the database’s change log via binlog/WAL): Debezium, Canal or your own consumer deletes the matching keys, including changes made by scripts or by an admin editing rows by hand.
flowchart LR
App[App] -->|UPDATE customer| DB[(DB)]
DB -->|binlog / WAL| CDC[CDC connector<br/>Debezium]
CDC -->|change event| K[[Kafka topic]]
K --> Inv[Invalidator]
Inv -->|DEL customer:v1:42<br/>retry until it works| R[(Redis)]
App -->|cache-aside read| R
Running CDC: PostgreSQL replication slots and MySQL binlogs
Exercises
Exercise 1: pick a strategy per kind of data
Task (paraphrased): an e-commerce site has (a) the list of provinces, (b) product details (prices change a few times a day), (c) product view counts, (d) wallet balances. Choose a strategy and a TTL for each.
Solution to Exercise 1
| Data | Strategy | TTL |
|---|---|---|
| Provinces | local cache (Caffeine), loaded at startup | a few hours |
| Product details | cache-aside + delete after commit, CDC if there are many write paths | 5–15 minutes, with jitter |
| View counts | write-back: INCR in Redis, a job flushes to the DB every minute |
none |
| Wallet balance | never decide from the cache; display only, with a very short TTL | 0–5 seconds |
Exercise 2: find the race
Task (paraphrased): the code below sometimes makes a product page show an old price for hours. Find the cause and fix it.
@Transactional
public void updatePrice(long id, BigDecimal price) {
redis.delete("product:" + id);
productRepository.updatePrice(id, price);
auditService.record(id, price); // HTTP call to another service, ~300 ms
}
// product:* keys are set without a TTL
Solution to Exercise 2
Three bugs compound each other:
- It deletes before the database write, and before the commit too, so the inconsistency window spans the whole transaction, including a 300 ms HTTP call; any reader that misses in that window caches the old price.
- The keys have no TTL, so the old price lives until the next price change.
- The audit call inside the transaction makes the transaction longer.
Fix: write the DB, publish an event, delete in AFTER_COMMIT with retries, put a TTL on every key, and move the audit call out of the transaction (or through an outbox).
Remember
- Work out how much database load drops when the hit rate goes from 95% to 99%.
- Explain why writers should delete the key instead of writing the new value.
- Draw the “delete first” race and the “commit first, delete later” race, and say why the second is rare.
- Know why deleting inside
@Transactionalis wrong and how to move it after commit. - Compare retry, outbox and CDC; and double delete, version + tombstone and leases.
Interview questions
Junior
How does cache-aside work?
The app reads the cache first; on a miss it reads the database, writes the result into the cache with a TTL and returns it.
The write side is usually write-around: write the database, then delete the key. Pros: if Redis is down you can still read the database, and the cache can hold a different shape than the table. Cons: the logic lives in the app and there’s a window of stale data.
What is the cache hit rate and why does it matter?
The share of reads served by the cache; it decides how much load reaches the database.
Database load is (1 − hit rate) × reads, so going from 95% to 99% cuts database load fivefold. The hit rate is high when access is skewed (80/20). A hit rate that drops after a deploy usually means the key format changed or someone flushed the cache.
Mid
When data changes, should you update or delete the cache key? Why?
Delete, because writing values from the writer side races between two writers, while deletes can be reordered safely.
A writes the DB, B writes the DB, B sets the cache, A sets the cache: the cache holds A’s value while the DB holds B’s. A delete carries no value and is idempotent; the next read reloads from the database. Deleting also avoids computing a value for a key nobody may read.
Write the database then delete the cache, or delete first? Why?
Write (commit) the database first, delete afterwards, with a TTL; “afterwards” means after commit, not after the UPDATE statement inside the transaction.
Deleting first lets any reader that misses during the write load the old value, a window as long as the whole write. Writing first only breaks if a complete database write fits in the tiny gap between a reader’s database read and its cache set, which is rare because cache operations are much faster. The TTL bounds the worst case.
How do write-through, write-back and write-around differ?
They differ in what gets written first and when the call returns.
- Write-through writes the cache and the database synchronously before returning: the cache is always fresh, writes are slower.
- Write-back writes the cache and returns, flushing to the database later: very fast, but data is lost if the cache dies, so it suits counters.
- Write-around writes the database and invalidates the cache: the cache only holds what’s read, but data that’s both written and read often keeps missing.
Senior
The database write succeeded but the cache delete failed. How do you guarantee invalidation?
There’s no shared transaction, so make the delete retryable until it succeeds, and always keep a TTL as an upper bound.
- In-app retry with backoff (lost if the process dies).
- An outbox that records the “intent to delete” in the same transaction, and a job that performs it.
- CDC that reads the binlog/WAL and deletes keys, catching changes made outside the app too.
On Postgres, monitor the replication slot; on MySQL, use binlog_format=ROW and keep binlogs long enough.
You write the DB first and delete after commit, yet stale data lingers. What do you suspect?
First a lagging replica feeding the cache, then deletes failing silently and other cache layers.
(1) The cache is filled from a lagging read replica. (2) The delete fails silently (an exception is caught and ignored). (3) Another cache layer: per-pod local caches, a CDN, the client’s HTTP cache. (4) Derived keys weren’t invalidated. (5) The key has no TTL. Check invalidation metrics, run TTL key, and compare the commit time with the key’s last SET.
How do you fully close the “slow reader repopulates after the writer deleted” race?
To close it fully, use versions + tombstones or leases; a double delete only narrows the window.
- Delayed double delete: a second delete after a delay longer than a DB read plus replication lag.
- Versions with tombstones in Lua: accept a fill only if the version read from the DB isn’t older than the invalidated version (deterministic, needs a version column).
- Leases as in Facebook’s memcache: a set is rejected if a delete happened after the lease was issued, which also stops the thundering herd.
If the data must never be stale, don’t use the cache for that decision.
Useful phrases for interviews
- “We use cache-aside with write-around: the database is the source of truth, and writers delete the key after commit.”
- “I’d delete rather than update the cache, because two concurrent writers can land their cache writes in the opposite order.”
- “Deleting after the commit leaves a much smaller window than deleting first, and the TTL bounds the worst case.”
- “There’s no transaction spanning the database and Redis, so invalidation has to be retryable, ideally driven by CDC or an outbox.”
- “If the cache is filled from a replica, replication lag can put the old value right back, so we load from the primary on a miss.”
- “For data that must never be stale, I wouldn’t make the decision from the cache at all.”
- “The hit rate matters more for database load than for latency: going from 95 to 99 percent cuts the load fivefold.”
- “A versioned namespace lets us invalidate a whole group of keys with a single INCR instead of scanning the keyspace.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.