Caching series: 1. Cache-aside and invalidation · 2. Cache failures: avalanche, breakdown, penetration · 3. Reviewing a Redis distributed lock
A Redis distributed lock looks simple (SET key value NX and you’re done), yet it’s full of bugs, because they only show up during a GC pause, a failover, or two requests racing within the same millisecond. This post reviews the course’s sample code (a flight-booking project), reproduces six bugs against Redis 7, and rewrites it. Read first: Transactions, isolation levels and locking (database locks, SELECT ... FOR UPDATE, advisory locks), because the last question here is “do you need a Redis lock at all”.
At a glance:
flowchart TD
A{"Lock for what?"} -->|"A DB row"| B["Condition, version, FOR UPDATE"]
A -->|"Work outside DB"| C["SET token NX PX ttl"]
C --> D["Many locks: sorted order"]
D --> E["Work within the lease"]
E --> F["Unlock: Lua compares token"]
E -.->|"GC pause, failover"| G["Two holders at once"]
G --> H["Fencing token at resource"]
1. Two reasons to want a lock
Key idea: the first review question is whether this lock is about saving work or about correctness. The two need very different guarantees (Martin Kleppmann’s split).
- Efficiency: avoid doing the same work twice (two pods sending the same report email). If the lock occasionally fails, you just waste some work; a single Redis node with
SET NX PXis enough. - Correctness: two holders at once means wrong data (one seat sold to two people). A Redis lock alone is not enough: you need a fencing token checked by the resource, or the database itself as the locking mechanism (section 6).
2. The right primitive
Key idea: acquire with one atomic command carrying a random token and a TTL; release and extend must both compare the token inside a Lua script.
SET lock:order:42 <random token> NX PX 30000
NX: set only if absent, so only one client gets it.PX 30000: expires after 30 seconds, so a client that dies mid-way doesn’t hold the lock forever.- A random token (UUID) per acquisition: when unlocking, the client proves the lock is still its own.
Unlocking must compare the token and delete in one step. Lua does that, because Redis runs the whole script without interleaving other commands:
if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end
return 0
Extending the lease (for work that runs longer than the TTL) compares the token too: if GET == token then PEXPIRE.
3. The code under review
Key idea: RedisService locks with the constant value 1 and unlocks with delete(key); DistributedLockService acquires several keys in parallel and unlocks asynchronously. Each of those is a bug in section 4.
lock(key, ttl)callssetIfAbsent(key, 1, ttl);unlock(key)is a plaindelete(key).lockWithRetryretries with exponential backoff (50 ms × 2^n) until the total wait exceeds a timeout.DistributedLockServicereads the TTL and timeout from config and multiplies them by 1000 in@PostConstruct.lockWithRetry(List<String>)acquires the keys in parallel on a thread pool and unlocks everything if any one fails;unlockruns asynchronously.
Excerpts from the sample code and how I rebuilt it
The lines worth looking at (quoted minimally):
return redisTemplate.opsForValue().setIfAbsent(key, DefaultValue.NUMBER_ONE, ttlMillisecond, TimeUnit.MILLISECONDS);
...
public void unlock(String key) { redisTemplate.delete(generateKey(key)); }
...
lockTTL = lockTTLInSecond * 1000; // @PostConstruct
return redisService.lock(lockKey, lockTTL * 1000); // lock()
...
if (!lockResult) { unlock(lockResultList.keySet()); } // lockWithRetry(List)
CompletableFuture.runAsync(() -> redisService.unlock(lockKey), executor);
I rebuilt each behaviour with Jedis against Redis 7 to run the scenarios in section 4.
4. Six bugs
Key idea: the first five are code bugs you can fix: an ownerless lock, a TTL in the wrong unit, unordered multi-key locking, async unlock, a broken retry loop. The sixth is in the nature of a lease, and no client code can cure it.
Bug 1: a constant lock value and a blind DEL
Every client sets the same value 1, and unlock deletes without asking whose lock it is. Scenario:
- A takes the lock with a 200 ms TTL, then pauses for 300 ms (GC, a slow query).
- The lock expires and B takes it.
- A wakes up, runs
finally { unlock }and deletes B’s lock. - C takes the lock right after: B and C are both in the critical section (the code only one party may run at a time).
A lock(200ms): true
B lock: true
C lock while B still works: true <- B and C both inside
The same scenario with a token and Lua:
A unlock with stale token: false
C lock while B works: false
B extend: 1, pttl=30000
B unlock: true
It gets worse in lockWithRetry(List). When one key can’t be acquired, the code unlocks every key in the result map, including the ones it didn’t get, which deletes locks other clients hold:
B results: {lock:seat:1=true, lock:seat:2=false}
seat 2 still locked by A? false <- A's lock deleted by B
Bug 2: the TTL is multiplied by 1000 twice, and the config keys don’t match
A variable named lockTTLInSecond is multiplied by 1000 in init() to get milliseconds, then lock() multiplies by 1000 again. The config says ttl: 300000 (most likely meant as 300,000 ms = 5 minutes, although the variable name says seconds):
intended 300000 ms = 5.0 min; actual 300000000000 ms = 9.5 years
The lockWithRetry path multiplies only once, so the same config gives an 83-hour TTL there. A pod that dies while holding a lock leaves the resource locked for 83 hours (or 9.5 years).
A side config bug: the timeout placeholder that doesn’t exist
The code also reads ${app.transaction.lock.timeout}, but the config file only has ttl and wait. A placeholder without a default stops Spring from starting (Could not resolve placeholder). Another branch of the repo patches it with a default value, which hides the bug rather than fixing it.
Bug 3: acquiring several keys in parallel, in no particular order
Locking several seats with parallel CompletableFutures means there is no order. X needs {1, 2}, Y needs {2, 1}: X holds 1, Y holds 2, both retry on the same backoff schedule (no jitter), both fail, both release, and it repeats. That’s a livelock: like a deadlock, except both keep running and neither makes progress. Combined with bug 1, each release also deletes the other side’s lock.
Fix it one of two ways:
- A global order: sort the keys and acquire them sequentially; on failure, release what you took in reverse order. Two requests with overlapping sets always fight over the smallest key first, so they can’t hold each other’s keys.
- All or nothing in one Lua script: check that no key exists, then set them all. On Redis Cluster, all keys in a script must be in the same slot, so use a hash tag:
{flight:9}:seat:1,{flight:9}:seat:2.
A locks {seat:2}: 1
B locks {seat:1, seat:2}: 0
seat:1 left behind by B? false
B unlock-all removes only its own: 0
Bug 4: async unlock on an unbounded thread pool
unlock pushes the DEL onto Executors.newCachedThreadPool() and returns immediately. Consequences:
- The method returns while the lock still exists; the same user’s next request may be rejected.
- Nobody notices a failed unlock: the exception sits in a future nobody reads.
- A pod shutting down can lose pending unlock tasks, leaving the lock in place until the TTL (9.5 years, per bug 2).
newCachedThreadPoolhas no thread limit; when Redis is slow, every lock and unlock adds a thread, and the service dies from running out of threads.
Unlock is a sub-millisecond round-trip: call it synchronously in finally (or in the close() of try-with-resources). Log when the script returns 0: the lock expired before release, meaning the critical section may have overlapped.
Bug 5: the retry loop
do {
if (setNx(key, keyTTL)) return true;
wait = INIT_INTERVAL * (long) Math.pow(MULTIPLIER, retries);
Thread.sleep(wait);
retries++;
totalWait += wait;
} while (totalWait <= retryTimeout);
- Sleeps, then doesn’t retry: the sleep that pushes
totalWaitpast the timeout is the last thing the loop does; it exits without callingsetNxagain. With a 10-second timeout, that last 6.4-second sleep is pure waste. - No jitter (random spread): clients that start together retry together and keep colliding in waves.
- No cap per wait: 50 ms × 2^n grows to tens of seconds.
totalWaitignores round-trip time; use a deadline based onSystem.nanoTime().catch (Exception e) { return false; }: see the pitfall below.
Bug 6: expiring leases, failover, and why you need fencing tokens
Even with the five bugs above fixed, a Redis lock is still a lease: it expires by itself. There are two ways it fails that no client code can prevent:
- The client pauses longer than the TTL (a stop-the-world GC, a frozen VM, a slow database query). The client doesn’t know it lost the lock and keeps writing.
- Failover: Redis replication is asynchronous. A takes the lock on the primary, the primary dies before the write reaches the replica, the replica is promoted without the lock, and B takes it.
The real fix is a fencing token: every acquisition gets an increasing number (INCR), the client sends it with every write, and the resource rejects any number lower than the highest it has seen:
UPDATE seat SET holder = :holder, fence = :token
WHERE id = :id AND fence < :token; -- 0 rows means the token is stale: stop
B writes with token 2: true
A (woke up) writes with token 1: false
stored value: B
Redlock (acquiring on a majority of N independent Redis nodes) handles case 2 if its assumptions about clocks and delays hold. But Kleppmann showed it can’t save case 1: no time-based lock can stop a client that has been paused.
Case 1 is measured in Optimistic or pessimistic locking: a Redis lock with a 5 ms TTL around 10 ms of work sold 311 seats on a 100-seat flight; the same lock plus a version check on write (acting as the fencing token) sold exactly 100.
Command history: from SETNX + EXPIRE to SET NX PX
The analysis above is for understanding the bugs; in a real PR, they have to become comments the author can act on. The “worked review 2” section of Code review: two worked reviews turns these same bugs into labelled review comments (TODO, QUESTION, FYI), each with a concrete consequence (“a 30-second lock becomes an 8-hour one”) and a fix.
5. The rewrite
Key idea: the API returns a Lease that is AutoCloseable, so with try-with-resources it can’t be forgotten. Inside: a random token, unlock and extend via token-checking Lua, retries against a deadline with jitter, “busy” as Optional.empty() and a Redis failure as an exception.
Usage:
try (var lease = lock.tryAcquire("order:" + orderId, Duration.ofSeconds(30), Duration.ofSeconds(2))
.orElseThrow(() -> new OrderBusyException(orderId))) { // mapped to 409
// critical section: short, and no outbound call without a timeout
}
With Redisson you don’t need to write this class yourself, but you still need fencing tokens for locks that must be correct.
The full RedisLock class
@Component
public class RedisLock {
private static final Logger log = LoggerFactory.getLogger(RedisLock.class);
private static final RedisScript<Long> UNLOCK = new DefaultRedisScript<>("""
if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end
return 0
""", Long.class);
private static final RedisScript<Long> EXTEND = new DefaultRedisScript<>("""
if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('PEXPIRE', KEYS[1], ARGV[2]) end
return 0
""", Long.class);
private final StringRedisTemplate redis;
public RedisLock(StringRedisTemplate redis) { this.redis = redis; }
/** Empty means "busy"; an exception means "Redis is unavailable". */
public Optional<Lease> tryAcquire(String name, Duration ttl, Duration maxWait) throws InterruptedException {
String key = "lock:" + name;
String token = UUID.randomUUID().toString();
long deadline = System.nanoTime() + maxWait.toNanos();
long backoffMs = 20;
while (true) {
if (Boolean.TRUE.equals(redis.opsForValue().setIfAbsent(key, token, ttl))) {
return Optional.of(new Lease(key, token));
}
long remainingMs = (deadline - System.nanoTime()) / 1_000_000;
if (remainingMs <= 0) return Optional.empty();
long jittered = ThreadLocalRandom.current().nextLong(backoffMs / 2, backoffMs + 1);
Thread.sleep(Math.min(jittered, remainingMs)); // InterruptedException propagates
backoffMs = Math.min(backoffMs * 2, 500);
}
}
/** Several resources: a global order, all or nothing, release only what we hold. */
public Optional<List<Lease>> tryAcquireAll(List<String> names, Duration ttl, Duration maxWait) throws InterruptedException {
List<Lease> held = new ArrayList<>();
for (String name : names.stream().distinct().sorted().toList()) {
Optional<Lease> lease = tryAcquire(name, ttl, maxWait);
if (lease.isEmpty()) {
held.reversed().forEach(Lease::close);
return Optional.empty();
}
held.add(lease.get());
}
return Optional.of(held);
}
public final class Lease implements AutoCloseable {
private final String key;
private final String token;
private Lease(String key, String token) { this.key = key; this.token = token; }
public boolean extend(Duration ttl) {
return Long.valueOf(1).equals(redis.execute(EXTEND, List.of(key), token, String.valueOf(ttl.toMillis())));
}
@Override
public void close() {
Long deleted = redis.execute(UNLOCK, List.of(key), token);
if (deleted == null || deleted == 0) {
log.warn("lock {} had expired before release: another holder may have run concurrently", key);
}
}
}
}
List.reversed() comes from Java 21 (Sequenced Collections).
6. When you don’t need a Redis lock
Key idea: most “we need a distributed lock” cases are really “two requests must not overwrite each other on the same database row”. Then the database locks better than Redis.
A database lock lives inside the transaction, is released on commit, rollback or a dropped connection, and never expires halfway through.
Exercises
Exercise 1: a local cache for the airport list (the caching lesson’s homework)
Task (paraphrased): implement a read-aside local cache with a TTL for the list of airports (code, name). Libraries are allowed; a database is optional, and you may hardcode the list.
Solution to Exercise 1: Caffeine, Spring @Cacheable and a review of the reference solution
Solution 1: plain Caffeine, to see the behaviour clearly.
record Airport(String code, String name) {}
Cache<String, List<Airport>> cache = Caffeine.newBuilder()
.expireAfterWrite(Duration.ofSeconds(1))
.maximumSize(1_000)
.recordStats()
.build();
List<Airport> airports() {
return cache.get("all", k -> loadFromDb()); // read-aside + per-key single-flight
}
A run (the loader simulates a 200 ms query, TTL 1 second):
miss -> load from DB
hit (3 airports)
hit (3 airports)
after TTL: expired, next read goes to DB
naive check-then-put, 100 threads: DB calls = 100
cache.get(key, loader), 100 threads: DB calls = 1
cached list after a caller mutated it: [HAN, SGN, XXX]
The last line is the trap: the cache holds a reference, so any caller that “just filters” the list it got back modifies the cached copy for every other request. Store List.copyOf(...) (immutable).
Solution 2: Spring Boot 3 with @Cacheable.
@Configuration
@EnableCaching
class LocalCacheConfig {
@Bean
CacheManager localCacheManager(@Value("${app.cache.airports.ttl:10m}") Duration ttl) {
CaffeineCacheManager manager = new CaffeineCacheManager("airports");
manager.setCaffeine(Caffeine.newBuilder()
.expireAfterWrite(ttl)
.maximumSize(10_000)
.recordStats()); // exposed via Actuator: cache.gets{result=hit|miss}
manager.setAllowNullValues(false);
return manager;
}
}
@Service
class AirportService {
private final AirportRepository repository;
AirportService(AirportRepository repository) { this.repository = repository; }
@Cacheable(cacheNames = "airports", key = "'all'", sync = true)
public List<Airport> findAll() {
log.info("airports: cache miss, loading from DB");
return List.copyOf(repository.findAll());
}
}
With logging.level.org.springframework.cache=TRACE, the first call logs “No cache entry for key ‘all’” along with the SQL, and later calls log “Cache entry for key ‘all’ found”.
Reviewing the course’s reference solution. It puts @Cacheable(key = "#root.methodName", cacheNames = "localCache", cacheManager = "localCacheManager") on the repository’s findAll(), uses Caffeine with expireAfterWrite(20s), and marks the CaffeineCacheManager as @Primary. That’s fine for the homework, but in production:
- Key collisions: one shared cache called
localCachekeyed by method name. Another repository with afindAll()using this cache would get the airport list when asking for airlines, and then aClassCastException. Use one cache name per kind of data and keys that describe the content. - No
sync = true: every 20-second expiry is a small stampede (100 concurrent requests = 100 queries, as measured above). - No
maximumSize: the cache is unbounded; with parameterised keys, memory grows forever. - One copy per pod: N pods expire at different times, so pod A may see a new airport while pod B doesn’t yet. Acceptable for rarely-changing data; otherwise use Redis or broadcast invalidation (Redis pub/sub, client-side caching).
- Returning a mutable list from the cache (the trap in the last line of the run).
@Primaryon the local cache manager: every@Cacheablewithout an explicitcacheManagersilently becomes a local cache, including things that should live in Redis.
Exercise 2: choose the locking for seat holds
Task (paraphrased): an API holds seats on a flight, 1–4 seats per request, stored in a PostgreSQL table seat(flight_id, seat_no, holder, held_until). Do you need a Redis lock?
Solution to Exercise 2: let PostgreSQL do the locking
No. Holding a seat means writing those very rows, so let the database lock them:
BEGIN;
SELECT seat_no FROM seat
WHERE flight_id = :f AND seat_no = ANY(:seats)
AND (holder IS NULL OR held_until < now())
ORDER BY seat_no -- fixed order: no deadlock between overlapping requests
FOR UPDATE;
-- if every seat came back, UPDATE ... SET holder = :user, held_until = now() + interval '10 minutes'; otherwise ROLLBACK
COMMIT;
For a single seat, one UPDATE ... WHERE holder IS NULL followed by a check of the affected row count is enough. A Redis lock only becomes necessary if holding a seat also calls an external system with no shared transaction; then add a fencing token stored in the seat table.
Exercise 3: two requests booking the same seat (the Redis lesson’s homework)
The task, paraphrased: two requests arrive at the same moment, both trying to book seat A34_S012C on flight FA634. Use a Redis distributed lock so only one request is processed at a time. No database or booking logic is required.
Solution to Exercise 3: Jedis, a token, and re-checking after the lock
Here’s a solution with Jedis 5.2 (no Spring), run against Redis 7.4 on JDK 21. The “database” is a ConcurrentHashMap from seat to holder, which is enough to see who wins:
enum Result { BOOKED, SEAT_BUSY, SEAT_TAKEN }
static final String UNLOCK = """
if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end
return 0
""";
static final Map<String, String> seatOwner = new ConcurrentHashMap<>(); // simulated "DB"
static Result book(JedisPooled redis, String flight, String seat, String user) throws InterruptedException {
String key = "lock:seat:" + flight + ":" + seat; // lock exactly this resource
String token = UUID.randomUUID().toString(); // a fresh token per acquisition
String ok = redis.set(key, token, SetParams.setParams().nx().px(10_000));
if (!"OK".equals(ok)) return Result.SEAT_BUSY; // someone else holds it: return 409
try {
if (seatOwner.containsKey(seat)) return Result.SEAT_TAKEN; // re-check AFTER getting the lock
Thread.sleep(100); // pretend to write the DB, call an API
seatOwner.put(seat, user);
return Result.BOOKED;
} finally {
redis.eval(UNLOCK, List.of(key), List.of(token)); // delete only if it's still our lock
}
}
Two threads wait on a CountDownLatch and then call book(redis, "FA634", "A34_S012C", ...) together. Output:
alice -> SEAT_BUSY
bob -> BOOKED
owner = bob
lock left = false
rounds with booked != 1: 0
The last line comes from a tougher check: 200 rounds, each with 8 threads racing for a new seat, and every round ends with exactly one BOOKED. The lock is cleaned up every time (lock left = false).
Three things make this correct:
- A random token plus a compare-and-delete script for unlocking (bug 1 above).
- Re-checking the seat after acquiring the lock: a losing request can grab the lock right after the winner releases it, and must then answer
SEAT_TAKENinstead of overwriting. - Answering “busy” (409, client retries) instead of waiting forever.
In a real system, as Exercise 2 says, a conditional statement in the database is still the last line of defence; the Redis lock only reduces contention before you hit the database.
Other Redis data structures and deployment modes are covered in the Redis series.
Remember
- Explain why unlock must compare the token in Lua, and the GC-pause scenario where a blind
DELdeletes someone else’s lock. - Know how to take several locks without livelock: sorted order, or all-or-nothing Lua with a hash tag.
- Point out the code bugs in the retry loop and the async unlock, and how to fix each one.
- Explain why a Redis lock is a lease, why Redlock can’t save a paused client, and how fencing tokens fix it.
- Know when to use the database’s locks instead of Redis.
Interview questions
Mid
How do you implement a basic, correct Redis distributed lock?
Acquire with SET lock:x <random token> NX PX <ttl>, release with a Lua script that compares the token before DEL.
Acquiring is atomic, only one client wins, and it expires if the client dies. Comparing the token on release means you never delete someone else’s lock; extending also compares the token. Never use SETNX followed by a separate EXPIRE.
What do you watch out for when acquiring several locks at once?
Ordering and all-or-nothing.
Acquire in a fixed sorted order so two requests with overlapping sets can’t hold each other’s keys (a deadlock, or a livelock with retries); on failure, release only the locks you took. Or use one Lua script that checks and then sets all keys, with a hash tag so the keys share a slot on Cluster.
Senior
Is a Redis lock safe across a Redis failover? What does Redlock solve?
No. Redlock survives some nodes dying, but it can’t protect against a client paused for longer than the TTL.
Replication is asynchronous, so the primary can die before the lock reaches the replica; the promoted replica has no lock and a second client takes it. Redlock acquires on a majority of N independent nodes, but relies on assumptions about clocks and delays. For locks that must be correct, you need fencing tokens checked by the resource, or the database’s own locking.
What is a fencing token?
An increasing number issued with every lock acquisition; the resource rejects any write carrying a number lower than the highest it has seen.
Sources: an INCR in Redis, or a ZooKeeper/etcd version. Example: UPDATE ... SET fence = :t WHERE id = :id AND fence < :t. A stale client that wakes up after losing the lock gets rejected. It turns an “efficiency” lock into a “correctness” lock, provided the resource can check the token.
The work in the critical section may run longer than the TTL. How do you handle it?
First make the critical section shorter; if you can’t, renew with a watchdog and add a fencing token.
- Move external calls out of the critical section, and put a timeout on every I/O inside.
- Renew periodically with a token-checking script (a watchdog like Redisson’s, every third of the TTL), and abort if renewal fails.
- Add a fencing token, because the watchdog can’t help when the whole JVM pauses.
- On release, log when the script returns 0 (the lock had expired): a sign the critical section may have overlapped.
When do you use the database’s locks instead of Redis?
When the protected resource is a database row; Redis locks are for work outside a database transaction, such as calling an external API or a job running on many pods.
In order of preference: a conditional statement or a unique constraint (no lock needed), optimistic locking with a version, SELECT ... FOR UPDATE with ORDER BY, and advisory locks like pg_advisory_xact_lock. Database locks live inside the transaction, are released on commit, rollback or a dropped connection, and never expire halfway.
You’re reviewing a lock function with catch (Exception e) { return false; }. What do you comment?
It returns the same result for “lock busy” and “Redis is down”, and it swallows InterruptedException.
The caller can’t answer with the right 409 or 503, can’t open a circuit breaker, and can’t fall back to another mechanism. Losing the interrupt flag means the thread won’t stop on shutdown. Suggest Optional.empty() for busy, letting infrastructure exceptions propagate, throws InterruptedException, and separate metrics for each case.
Useful phrases for interviews
- “The first question I ask is whether this lock is for efficiency or for correctness.”
- “Every acquisition gets a random token, and we release with a Lua script that compares it before deleting.”
- “A Redis lock is a lease: it expires whether or not you’re finished, so the critical section has to be short.”
- “On a failover with asynchronous replication, a lock that was just granted can disappear.”
- “For correctness, the resource itself has to reject stale writers, which is what fencing tokens are for.”
- “We acquire multiple locks in a fixed sorted order and release only the ones we actually hold.”
- “If the data is a database row, I’d rather use a conditional update or SELECT FOR UPDATE than a Redis lock.”
- “Durations in config should carry their unit, like 30s, so nobody multiplies by a thousand twice.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.