Skip to content
notes.vinhnq.com

Search

System DesignMid22 min read

Cache-aside, write-around and keeping the cache consistent with the database

Review 19 cards
On this page

Caching series: 1. Cache-aside and invalidation · 2. Cache failures: avalanche, breakdown, penetration · 3. Reviewing a Redis distributed lock

Redis can take an API from 80 ms to 5 ms, and it brings a new kind of bug: you change your age from 33 to 34, refresh, still see 33, and only see the right value ten minutes later. This post answers: when the data changes, how do we stop the cache from lying, and if it does lie, for how long at most. Read Transactions, isolation levels and locking first, since most of this post is about race conditions; how Redis stores data, expires keys, persists, replicates and shards is in the Redis series.

1. What a cache is and how to measure it

Key idea: a cache holds data temporarily so it can be read again faster, and it only pays off for hot data. The main metric is the hit rate (the share of reads served by the cache).

A cache holds either a copy of source data (a customer row) or a computed result (today’s order total, a search page). The cache is the database’s shield: databases are expensive and hard to scale out, so the cache absorbs most reads. The price: performance vs cost and performance vs consistency.

Hot data is the small slice that gets most of the reads (80/20). With hit rate h:

average latency ≈ t_cache + (1 − h) × t_db
database load   = (1 − h) × total reads

Example with t_cache = 1 ms, t_db = 20 ms, 10,000 req/s:

Hit rate Average latency Queries/s to the DB
95% 2 ms 500
99% 1.2 ms 100

Latency barely moves, but database load drops fivefold. So when the hit rate falls (a deploy changes the key format, someone flushes the whole cache), the database dies first.

2. Where caches live

Key idea: the same data can sit in four layers at once (client, CDN/proxy, app, Redis), each expiring in its own way. That’s half the reason invalidation is hard.

flowchart LR
  C[Client<br/>caches responses] --> CDN[CDN<br/>html, js, css, media]
  C --> P[Reverse proxy<br/>static files]
  CDN --> S
  P --> S[App server<br/>local cache: config,<br/>small catalogs]
  S --> R[Remote cache Redis<br/>query results, computed results,<br/>shared data]
  S --> DB[(Database<br/>buffer pool)]
  R -. miss .-> DB

Nobody can purge a client’s HTTP cache. MySQL dropped its query cache in 8.0 for the same reason: every write to a table had to invalidate every related result under a global mutex.

3. Read strategies

Key idea: with read-through the cache library loads on a miss; with cache-aside the app does all five steps itself. Cache-aside is the most common way to use Redis.

Before asking “how do we store it”, ask “how is the data accessed”: reads or writes.

  • Read-through: the app only talks to the cache; on a miss, the cache library calls a loader to read the database and stores the result (Caffeine’s LoadingCache, or Spring’s @Cacheable as seen from the app).
  • Cache-aside (read-aside): the app reads the cache itself, and on a miss reads the database and writes the cache.
sequenceDiagram
  participant A as App
  participant C as Cache
  participant D as DB
  A->>C: 1. GET key
  C-->>A: 2a. hit: return value
  A->>D: 2b. miss: SELECT
  D-->>A: 3. row
  A->>C: 4. SET key value EX ttl
  A-->>A: 5. return result

Cache-aside tolerates a Redis failure (fall back to the database, provided the client has a short timeout) and can cache any data shape (a DTO that joins three tables). The price: the logic lives in the app, and there is a window of inconsistency between reading the database and writing the cache (section 6).

Cache-aside code with Spring Boot 3 and the Redis client timeout trap

Catching cache errors yourself makes “tolerates cache failures” actually true:

@Service
class CustomerQueryService {
    private static final Duration TTL = Duration.ofMinutes(10);
    private final StringRedisTemplate redis;
    private final CustomerStore store;
    private final ObjectMapper json;

    static String key(long id) { return "customer:v1:" + id; }

    public CustomerView get(long id) {
        String key = key(id);
        String cached = readCache(key);                       // 1. read the cache
        if (cached != null) return fromJson(cached);          // 2. hit
        CustomerView view = store.findView(id)                // 3. miss -> database
            .orElseThrow(() -> new NoSuchElementException("customer " + id));
        writeCache(key, toJson(view));                        // 4. populate with a TTL
        return view;
    }

    private String readCache(String key) {
        try {
            return redis.opsForValue().get(key);
        } catch (DataAccessException e) {                     // Redis down or timed out: fall back to the DB
            log.warn("cache read failed for {}: {}", key, e.getMessage());
            return null;
        }
    }
    // writeCache: set(key, value, TTL) in the same kind of try/catch; toJson/fromJson use ObjectMapper
}

The key carries a version (v1), so changing the DTO shape only means moving to v2; the value is JSON rather than JDK serialization.

4. Write strategies

Key idea: write the cache together with the database (write-through), write the cache and flush the database later (write-back), or write the database and invalidate the cache (write-around). For business data, write-around + cache-aside is the sensible default.

  • Write-back returns immediately and flushes to the database later in batches; the name comes from CPUs (write to the cache, push to RAM later).
  • Write-around decouples the cache from storage and often invalidates asynchronously; data that’s both written and read often keeps missing.

5. The common problem: inconsistency and invalidation

Key idea: every strategy has moments when the cache says Age=26 while the database already says 27. The cure is cache invalidation (removing data that’s no longer valid); it’s hard because of timing, concurrency, data relationships and caches everywhere.

Four kinds of invalidation:

  • Time-based: a TTL (time to live, how long a key survives), expiring on its own.
  • Command-based: after a write, the app updates or deletes the key.
  • Event-based: listen for change events (application events, CDC) and delete.
  • Group-based: drop a whole group of keys by tag or prefix.

Besides timing (how long should the TTL be?) and races, there are data relationships (one product lives in its detail key, listing pages, search results) and root causes that are hard to find: the bug depends on exact timing and then heals itself when the TTL runs out.

Group invalidation: KEYS, SCAN, tag sets and versioned namespaces

6. Case study: three attempts at fixing inconsistency

Key idea: updating the cache lets two writers diverge; deleting doesn’t. Against readers, “delete first” breaks often, while “commit the DB, then delete” only breaks in a very rare race, so that’s the answer.

The read side is cache-aside; the write side is write-around, choosing between update (write the new value into the cache) and delete (remove the key). Every diagram below matches a small program that replays each interleaving against a HashMap “database” and a HashMap “cache”.

Attempt 1: update the cache, before or after the database?

sequenceDiagram
  participant W1 as Request 1 (age=1)
  participant C as Cache
  participant D as DB
  participant W2 as Request 2 (age=2)
  Note over W1,W2: Update cache first
  W1->>C: SET age=1
  W2->>C: SET age=2
  W2->>D: UPDATE age=2
  W1->>D: UPDATE age=1
  Note over C,D: cache=2, DB=1 → the DB is wrong
  Note over W1,W2: Update cache later
  W1->>D: UPDATE age=1
  W2->>D: UPDATE age=2
  W2->>C: SET age=2
  W1->>C: SET age=1
  Note over C,D: DB=2, cache=1 → the cache is wrong until the TTL

Attempt 2: delete the cache

Two writers each write the database and then delete the key, and any order works: the row lock serializes the two UPDATEs, the cache is empty, and the next read reloads. A delete is idempotent (doing it twice equals doing it once) and carries no value. But readers write to the cache too (step 4), so attempt 3 pits a writer against a reader with the combination RA + DWA (Read-Aside + Delete-Write-Around).

Attempt 3: RA + DWA, a writer against a reader

Delete the cache first, then update the database:

sequenceDiagram
  participant W as Writer (age 0→1)
  participant C as Cache
  participant D as DB
  participant R as Reader
  W->>C: DEL age
  R->>C: GET age (miss)
  R->>D: SELECT age → 0
  R->>C: SET age=0
  W->>D: UPDATE age=1
  Note over C,D: cache=0 (stale), DB=1 — stale until the TTL

The window is wide: the whole UPDATE and the transaction around it.

Update the database first, then delete the cache:

sequenceDiagram
  participant R as Reader
  participant D as DB
  participant C as Cache
  participant W as Writer (age 0→1)
  R->>C: GET age (miss)
  R->>D: SELECT age → 0
  W->>D: UPDATE age=1 (commit)
  W->>C: DEL age
  R->>C: SET age=0
  Note over D,C: DB=1, cache=0 — still inconsistent

Still inconsistent, but three conditions must hold: the key isn’t cached, the reader reads the database before the writer commits, and the writer’s UPDATE + commit + DEL all fit between the reader’s database read and its SET. That gap is one Redis round-trip (under 1 ms), while a committed write (with an fsync of the redo log or WAL) takes much longer, so it’s very rare.

The answer

Update the database first, delete the cache afterwards, plus a reasonably short TTL as a safety net for that rare race or a failed delete.

Code for deleting the cache after commit with Spring

The write side publishes an event, and the delete runs after commit:

@Transactional
public void changeAge(long id, int age) {
    store.updateAge(id, age);
    events.publishEvent(new CustomerChanged(id));         // delivered only after COMMIT
}

@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
void onChanged(CustomerChanged event) {
    String key = CustomerQueryService.key(event.id());
    try {
        redis.delete(key);
    } catch (DataAccessException e) {
        retryQueue.enqueue(key);                          // retried asynchronously with backoff
    }
}

7. Beyond the slides: closing the remaining gap

Key idea: three ways to close the “slow reader puts the old value back” race: a double delete only narrows the window; versions + tombstones and leases reject the old value deterministically.

  • Delayed double delete: delete right after commit, then again after a delay longer than one database read plus replication lag (a few hundred ms to a few seconds). Run the second delete through a delay queue or job, never Thread.sleep in the request.
  • Version + tombstone: each row has a version column that increments on every write. Instead of DEL, the writer leaves a short-lived tombstone (a “deleted” marker) saying “invalidated at version N”; a reader may only populate if the version it read is not older.
  • Leases (the “Scaling Memcache at Facebook” paper): on a miss, memcache hands out a lease token; a SET is only accepted with the token, and a DELETE in the meantime invalidates it. Redis has no built-in leases; tombstones plus versions are the closest imitation.
Lua for version + tombstone, a test run and more on leases

The check and the write must be atomic, so use Lua:

-- populate: KEYS[1]=key ARGV[1]=version ARGV[2]=payload ARGV[3]=ttlMs
local cur = redis.call('HGET', KEYS[1], 'v')
if cur and tonumber(cur) > tonumber(ARGV[1]) then return 0 end
if cur and tonumber(cur) == tonumber(ARGV[1]) and redis.call('HEXISTS', KEYS[1], 'data') == 1 then return 0 end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'data', ARGV[2])
redis.call('PEXPIRE', KEYS[1], ARGV[3])
return 1
-- invalidate (newVersion): if the stored version is lower, DEL the key, HSET v=newVersion, short PEXPIRE

Run against Redis 7, with a slow reader holding version 7 and a writer bumping it to 8:

writer invalidate(v8) -> 1
stale reader populate(v7) -> 0 (0 = rejected)
fresh reader populate(v8) -> 1 {v=8, data={"age":34}}

Memcache leases also rate-limit how often a token is issued per key, which stops the thundering herd (a crowd of requests hitting the database at once) when a hot key expires (post 2).

8. Problem 01: no atomicity between the database and the cache

Key idea: if the UPDATE succeeds but the Redis DEL fails, the cache keeps the old value until the TTL. The delete has to be retried until it succeeds: retries, an outbox or CDC.

Customer X changes their age from 33 to 34 and complains that “it takes a while to show up”, even though the system does things in the right order. The DEL failed (a timeout, a Redis failover, the pod killed halfway).

  • Retry: synchronously with backoff, or through a queue.
  • Outbox: write a cache_invalidation(key) row in the same transaction as the UPDATE; a job reads the outbox and deletes the keys.
  • CDC (change data capture, reading the database’s change log via binlog/WAL): Debezium, Canal or your own consumer deletes the matching keys, including changes made by scripts or by an admin editing rows by hand.
flowchart LR
  App[App] -->|UPDATE customer| DB[(DB)]
  DB -->|binlog / WAL| CDC[CDC connector<br/>Debezium]
  CDC -->|change event| K[[Kafka topic]]
  K --> Inv[Invalidator]
  Inv -->|DEL customer:v1:42<br/>retry until it works| R[(Redis)]
  App -->|cache-aside read| R
Running CDC: PostgreSQL replication slots and MySQL binlogs

Exercises

Exercise 1: pick a strategy per kind of data

Task (paraphrased): an e-commerce site has (a) the list of provinces, (b) product details (prices change a few times a day), (c) product view counts, (d) wallet balances. Choose a strategy and a TTL for each.

Solution to Exercise 1
Data Strategy TTL
Provinces local cache (Caffeine), loaded at startup a few hours
Product details cache-aside + delete after commit, CDC if there are many write paths 5–15 minutes, with jitter
View counts write-back: INCR in Redis, a job flushes to the DB every minute none
Wallet balance never decide from the cache; display only, with a very short TTL 0–5 seconds

Exercise 2: find the race

Task (paraphrased): the code below sometimes makes a product page show an old price for hours. Find the cause and fix it.

@Transactional
public void updatePrice(long id, BigDecimal price) {
    redis.delete("product:" + id);
    productRepository.updatePrice(id, price);
    auditService.record(id, price);   // HTTP call to another service, ~300 ms
}
// product:* keys are set without a TTL
Solution to Exercise 2

Three bugs compound each other:

  • It deletes before the database write, and before the commit too, so the inconsistency window spans the whole transaction, including a 300 ms HTTP call; any reader that misses in that window caches the old price.
  • The keys have no TTL, so the old price lives until the next price change.
  • The audit call inside the transaction makes the transaction longer.

Fix: write the DB, publish an event, delete in AFTER_COMMIT with retries, put a TTL on every key, and move the audit call out of the transaction (or through an outbox).

Remember

  • Work out how much database load drops when the hit rate goes from 95% to 99%.
  • Explain why writers should delete the key instead of writing the new value.
  • Draw the “delete first” race and the “commit first, delete later” race, and say why the second is rare.
  • Know why deleting inside @Transactional is wrong and how to move it after commit.
  • Compare retry, outbox and CDC; and double delete, version + tombstone and leases.

Interview questions

Junior

How does cache-aside work?

The app reads the cache first; on a miss it reads the database, writes the result into the cache with a TTL and returns it.

The write side is usually write-around: write the database, then delete the key. Pros: if Redis is down you can still read the database, and the cache can hold a different shape than the table. Cons: the logic lives in the app and there’s a window of stale data.

What is the cache hit rate and why does it matter?

The share of reads served by the cache; it decides how much load reaches the database.

Database load is (1 − hit rate) × reads, so going from 95% to 99% cuts database load fivefold. The hit rate is high when access is skewed (80/20). A hit rate that drops after a deploy usually means the key format changed or someone flushed the cache.

Mid

When data changes, should you update or delete the cache key? Why?

Delete, because writing values from the writer side races between two writers, while deletes can be reordered safely.

A writes the DB, B writes the DB, B sets the cache, A sets the cache: the cache holds A’s value while the DB holds B’s. A delete carries no value and is idempotent; the next read reloads from the database. Deleting also avoids computing a value for a key nobody may read.

Write the database then delete the cache, or delete first? Why?

Write (commit) the database first, delete afterwards, with a TTL; “afterwards” means after commit, not after the UPDATE statement inside the transaction.

Deleting first lets any reader that misses during the write load the old value, a window as long as the whole write. Writing first only breaks if a complete database write fits in the tiny gap between a reader’s database read and its cache set, which is rare because cache operations are much faster. The TTL bounds the worst case.

How do write-through, write-back and write-around differ?

They differ in what gets written first and when the call returns.

  • Write-through writes the cache and the database synchronously before returning: the cache is always fresh, writes are slower.
  • Write-back writes the cache and returns, flushing to the database later: very fast, but data is lost if the cache dies, so it suits counters.
  • Write-around writes the database and invalidates the cache: the cache only holds what’s read, but data that’s both written and read often keeps missing.

Senior

The database write succeeded but the cache delete failed. How do you guarantee invalidation?

There’s no shared transaction, so make the delete retryable until it succeeds, and always keep a TTL as an upper bound.

  • In-app retry with backoff (lost if the process dies).
  • An outbox that records the “intent to delete” in the same transaction, and a job that performs it.
  • CDC that reads the binlog/WAL and deletes keys, catching changes made outside the app too.

On Postgres, monitor the replication slot; on MySQL, use binlog_format=ROW and keep binlogs long enough.

You write the DB first and delete after commit, yet stale data lingers. What do you suspect?

First a lagging replica feeding the cache, then deletes failing silently and other cache layers.

(1) The cache is filled from a lagging read replica. (2) The delete fails silently (an exception is caught and ignored). (3) Another cache layer: per-pod local caches, a CDN, the client’s HTTP cache. (4) Derived keys weren’t invalidated. (5) The key has no TTL. Check invalidation metrics, run TTL key, and compare the commit time with the key’s last SET.

How do you fully close the “slow reader repopulates after the writer deleted” race?

To close it fully, use versions + tombstones or leases; a double delete only narrows the window.

  • Delayed double delete: a second delete after a delay longer than a DB read plus replication lag.
  • Versions with tombstones in Lua: accept a fill only if the version read from the DB isn’t older than the invalidated version (deterministic, needs a version column).
  • Leases as in Facebook’s memcache: a set is rejected if a delete happened after the lease was issued, which also stops the thundering herd.

If the data must never be stale, don’t use the cache for that decision.

Useful phrases for interviews

  • “We use cache-aside with write-around: the database is the source of truth, and writers delete the key after commit.”
  • “I’d delete rather than update the cache, because two concurrent writers can land their cache writes in the opposite order.”
  • “Deleting after the commit leaves a much smaller window than deleting first, and the TTL bounds the worst case.”
  • “There’s no transaction spanning the database and Redis, so invalidation has to be retryable, ideally driven by CDC or an outbox.”
  • “If the cache is filled from a replica, replication lag can put the old value right back, so we load from the primary on a miss.”
  • “For data that must never be stale, I wouldn’t make the decision from the cache at all.”
  • “The hit rate matters more for database load than for latency: going from 95 to 99 percent cuts the load fivefold.”
  • “A versioned namespace lets us invalidate a whole group of keys with a single INCR instead of scanning the keyspace.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.