Redis series: 1. Redis internals: why it is fast · 2. Persistence, replication, Sentinel and Cluster · 3. Choosing data structures for real problems
Most Redis code in Java projects only ever calls opsForValue().set(key, json), throwing away the main reason to choose Redis: data structures whose commands run atomically on the server. After this post you’ll know when to use each type, where its limits are, and how to design a delay queue that auto-cancels ride requests. Encodings and O(N) commands are in part 1; hash tags and Cluster are in part 2. All commands were tried on Redis 7.4.
At a glance:
flowchart TD
A{"What is the problem?"} -->|"Cache, counter, lock"| B["String"]
A -->|"Edit single fields"| C["Hash"]
A -->|"Queue without ack"| D["List"]
A -->|"Need ack"| E["Stream"]
A -->|"Likes, dedup"| F["Set"]
A -->|"Ranking, delayed queue"| G["Sorted set"]
G --> H["Atomic claim"]
H --> I["DB stays source of truth"]
1. The opening question: a hash or a JSON string?
Key idea: if you often read or update individual fields, use a hash, because each update is one atomic command (it runs as a whole, nobody can cut in halfway). If you always read/write the whole object, a JSON string is simpler.
Take a user object {uid, username, email, age} and three needs:
- C1: read only
username(shown in the header, called very often). - C2: update only
age. - C3: fetch the whole object.
# Option A: a JSON string
SET user:1 '{"uid":1,"username":"vinh","email":"vinh@example.com","age":33}'
# Option B: a hash
HSET user:1 uid 1 username vinh email vinh@example.com age 33
| Need | JSON string | Hash |
|---|---|---|
| C1 read username | GET the whole string, parse it in the app |
HGET user:1 username, only a few bytes on the wire |
| C2 update age | GET, parse, modify, SET back: 2 round-trips and a race |
HSET user:1 age 34: one atomic command |
| C3 fetch everything | GET: one command, deserialize straight into an object |
HGETALL, map the fields onto an object |
| Memory (measured) | 136 bytes | 112 bytes (listpack) |
C2 matters most. A round-trip is one send-and-receive over the network; between JSON’s two round-trips another request can cut in, and the later write wipes out the other one’s change. With a hash, each HSET only touches its own field.
2. String
Key idea: strings fit serialized caches, counters, sessions and locks. The common trap: combining two separate commands (INCR then EXPIRE) and assuming they’re atomic.
Strings are implemented with SDS, Redis’s own string type: binary safe, O(1) length, no overflow on append (see part 1). The limit is 512 MB, but keep values under 10 KB.
- Caching objects or responses that are already serialized (JSON, protobuf, compressed).
- Counters: atomic
INCR,INCRBY,INCRBYFLOAT. View counts, API call counts, generating increasing IDs. - Sessions shared across several instances (Spring Session uses a hash, but the idea is the same).
- Distributed locks:
SET key token NX PX ttl. There are many traps; read the lock review post before writing your own.
Example: a fixed-window rate limiter, at most 100 requests per minute per user:
INCR rl:u1:202611181030 # returns 1 on the first request of the minute
EXPIRE rl:u1:202611181030 60 # the app only calls this when INCR returned 1
3. List
Key idea: lists fit recent-items lists and simple queues. They have no ack (confirmation that processing finished) and no multiple consumer groups; if you need those, use a Stream.
A list keeps elements in insertion order. Push/pop at both ends is O(1) (LPUSH, RPUSH, LPOP, RPOP), range reads use LRANGE. Underneath it is a quicklist (a linked list of listpacks).
- Recent-items lists: the 10 newest posts with
LPUSH+LTRIM 0 9; simple timelines. - Queues: the producer does
LPUSH, the consumer doesBRPOP(blocking until an element arrives).
BRPOP removes the element before the app processes it, so a consumer that dies halfway loses the message.
A list as a queue: four limits and the reliable-queue fix
The limits the slides list, stated more precisely:
- At most 2³² − 1 elements: rarely a real problem; a list that big became a big key long before.
- No multiple consumer groups: a popped element is gone and only one consumer receives it. If two services (sending email, updating statistics) both need the same event, you have to push it into two lists yourself.
- No ack: if a consumer dies right after
BRPOP, the message is lost. - Weak ordering with several consumers: the list keeps its order, but with three parallel consumers the order in which they finish is no longer guaranteed.
A Stream (Redis 5) is an append-only log with increasing IDs. It fixes the list’s limits with three things:
- Consumer groups: each group receives every message, and consumers within a group share them.
- A pending entries list: remembers messages delivered but not yet
XACK-ed. XAUTOCLAIM: another consumer can take over abandoned messages.
XADD orders * id o1 amount 250000 # producer
XGROUP CREATE orders billing 0 # the "billing" group
XREADGROUP GROUP billing c1 COUNT 10 BLOCK 5000 STREAMS orders >
XACK orders billing 1794975683643-0 # ack only after processing
XPENDING orders billing # delivered but not yet acked
4. Hash
Key idea: a hash is a field → value map inside one key. Every operation on a field is one atomic command, which makes it a great fit for shopping carts.
The encoding (how Redis stores it underneath) is listpack when small and hashtable when large. Three main uses:
- Objects read or updated per field (section 1).
- Grouped counters:
HINCRBY stats:2026-11-18 api_calls 1. - Shopping carts: a hash
cart:{userId}, fields are product IDs, values are quantities.
| Operation | Command |
|---|---|
| Add a product | HSET cart:{u1} p100 1 |
| Increase the quantity | HINCRBY cart:{u1} p100 2 (returns 3) |
| Number of lines in the cart | HLEN cart:{u1} |
| Remove a product | HDEL cart:{u1} p200 |
| Quantity of one product | HGET cart:{u1} p100 |
| The whole cart | HGETALL cart:{u1} |
It stays correct even if the user clicks “add” in two tabs at the same time, with no lock needed.
A cart in a real implementation: TTL, prices, negative quantities, hash tags
- Set a TTL on the whole cart (
EXPIRE cart:{u1} 2592000, 30 days) and extend it on every change, so abandoned carts clean themselves up. - Don’t store prices in the cart. Read prices from the catalogue when displaying and recompute them at checkout, otherwise users keep an old price forever.
HINCRBYwith a negative number can bring the quantity to 0 or below. You need a Lua script “decrement, and delete the field if ≤ 0” to keep the invariant.- The
{u1}hash tag means other keys for the same user (checkout:{u1}) land in the same slot, if you later need to operate on them together in Cluster.
5. Set
Key idea: a set is an unordered collection of unique members, good for likes, deduplication and set operations. Set operations on large sets block the main thread.
The encoding is intset when it holds only small integers, listpack when small, and hashtable when large.
- Deduplication:
SADDreturns 1 for a new member and 0 if it already exists. Counting unique users who viewed a post today, marking messages as processed. - Likes:
SADD likes:post:9 u1,SISMEMBERto check whether someone liked it,SCARDto count. - Set operations: mutual friends with
SINTER followers:a followers:b, suggestions withSDIFF, merging withSUNION. Since 7.0,SINTERCARD 2 f:a f:b LIMIT 100only counts, without returning the whole set. - Random picks:
SRANDMEMBER,SPOPfor lucky draws.
Limits: at most 2³² − 1 members. SINTER is O(N × M) in the worst case (N: size of the smallest set, M: number of sets); SUNION/SDIFF are O(total number of members). Two sets with a million members each mean hundreds of ms of blocking the main thread.
6. Sorted set (ZSET)
Key idea: each member has a score and the set is always ordered by score, so fetching the top N or “everything due before now” is cheap.
The score is a 64-bit floating point number. Underneath it is a skiplist + dict (a listpack when small); a skiplist is a multi-level linked list that gives O(log N) search. Reading by rank range or score range is O(log N + M).
Uses:
- Leaderboards, top N, priority queues.
- Delay queues: the score is the due time (section 8).
- Sliding-window rate limiters: the score is each request’s timestamp,
ZREMRANGEBYSCOREdrops old requests,ZCARDcounts. - Secondary indexes by time.
ZINCRBY lb:2026-11 50 alice # add points
ZREVRANGE lb:2026-11 0 9 WITHSCORES # top 10
ZREVRANK lb:2026-11 alice # my rank (0-based)
ZSCORE lb:2026-11 alice
Members with the same score are ordered lexicographically by member, not by who got there first. To tie-break (decide between equals) by time you have to pack a timestamp into the score, and keep the score below 2⁵³.
Tie-breaking equal scores: test results and the 2⁵³ double limit
I tried it with alice 1500, bob 1500, carol 1800: ZREVRANGE returned carol, bob, alice; bob’s ZREVRANK was 1 and alice’s was 2. Two players with the same score are ordered by reverse lexicographic order of their names.
7. Less common types worth knowing
Key idea: besides the five main types there are a few specialised ones; knowing their names and the problems they solve is enough.
| Type | Used for | Example |
|---|---|---|
| Bitmap | flags per contiguous numeric ID, daily check-ins | SETBIT login:2026-11-18 34 1, BITCOUNT |
| HyperLogLog | approximate unique counts, 12 KB | PFADD uv:home u1, PFCOUNT uv:home |
| Geo | radius search | GEOADD, GEOSEARCH ... BYRADIUS 2 km |
| Stream | log/queue with consumer groups and acks | section 3 |
| Bitfield | many small counters inside one string | BITFIELD |
8. Case study: auto-cancel a ride after 10 minutes
Key idea: use a ZSET with the orderId as member and the expiry time as score. Workers must “claim” (take ownership of) items atomically, and the cancel in the database must be conditional, because Redis is not the source of truth.
8.1. The problem
A customer requests a ride; if no driver accepts it within 10 minutes, the system cancels it automatically and says “no drivers are available right now”. Constraints:
- A few thousand rides per minute at peak; cancelling a few seconds late is acceptable.
- Never cancel twice (two notifications, two refunds) and never miss a ride.
- The service runs on several instances.
8.2. The options
This post picks a ZSET: fast, and shared between instances. The price is handling atomic claiming, crashes and idempotency (running many times has the same effect as running once) yourself. If the load allows it, a database scan is simpler and worth considering first.
Comparing 5 ways to build a delay queue, and the final choice
| Approach | Pros | Cons |
|---|---|---|
Scan the database every second WHERE status = 'WAITING' AND created_at < now() - 10 minutes |
Simple, a single source of truth | Constant database load, needs an index, several instances scanning at once need locking (SKIP LOCKED) |
| A Redis key with a 10-minute TTL + a keyspace notification on expiry | No scanning | Notifications are pub/sub with no guarantee: a disconnected subscriber loses events; key expiry can be late |
@Scheduled/in-memory timers in each instance |
No infrastructure needed | Everything is lost on restart, no work sharing between instances |
| A message broker with delays (RabbitMQ delayed exchange, SQS delays of up to 15 minutes) | The textbook fit, with acks | More infrastructure; Kafka has no built-in delay |
ZSET: the member is the orderId, the score is the expiry time |
Fast, shared between instances, O(log N) retrieval by time | You have to handle atomic claiming, crashes and idempotency yourself |
8.3. A ZSET-based design
ZADD order:pending-timeout 1794976290000 order-o1 # score = now + 10 minutes (ms)
ZREM order:pending-timeout order-o1 # a driver accepted: remove from the queue
Each instance runs a worker loop: fetch the rides with score ≤ now and cancel them. The first version everyone thinks of:
List<String> due = redis.zrangeByScore(KEY, "-inf", now, 0, 50); // 1. read the due rides
redis.zrem(KEY, due.toArray(String[]::new)); // 2. delete them
due.forEach(this::cancelOrder); // 3. cancel them
Steps 1 and 2 are separate: four workers that all read before anyone deletes get the same list. Measured with 2,000 rides expiring over 1 second, 4 workers, Jedis 5.2 on Redis 7.4:
naive processed=2000 distinct, duplicates=784, left=0
atomic processed=2000 distinct, duplicates=0, left=0
784 rides were cancelled more than once. The atomic version combines the read and the delete into one Lua script that runs atomically on the server:
static final String CLAIM = """
local due = redis.call('ZRANGE', KEYS[1], '-inf', ARGV[1], 'BYSCORE', 'LIMIT', 0, ARGV[2])
if #due > 0 then redis.call('ZREM', KEYS[1], unpack(due)) end
return due
""";
@SuppressWarnings("unchecked")
List<String> claimDue(long nowMs, int batch) {
return (List<String>) redis.eval(CLAIM, List.of(KEY), List.of(String.valueOf(nowMs), String.valueOf(batch)));
}
Two other ways without Lua:
- After
ZRANGEBYSCORE, callZREMfor each member and only process the members for whichZREMreturned 1 (whoever manages to delete it owns it). Correct, but one round-trip per ride. ZPOPMINis also atomic, but it takes elements that aren’t due yet unless you check the score.
8.4. What can still go wrong
An atomic claim only stops duplicate processing. The two biggest remaining holes:
- A worker crashes after claiming, before cancelling: the element has already left the ZSET and the ride stays
WAITINGforever. Fix it with a visibility timeout: the claim moves the element into aprocessingZSET with score = now + 60 seconds instead of deleting it. OnlyZREM processingonce done; a job pushes back overdue elements. Or keep an infrequent database scan (every 5 minutes) for ridesWAITINGlonger than 11 minutes. - A driver accepts the ride just as the worker cancels it: the accepting thread’s
ZREMraces with the worker’s claim. The database must be the source of truth, and the cancel must be conditional:UPDATE ride SET status = 'CANCELLED' WHERE id = ? AND status = 'WAITING'. Zero affected rows means the driver accepted first, so skip it. The accept statement has the opposite condition, so the handler is idempotent even if it runs twice.
Four more failure points: duplicate notifications, clock skew, losing Redis data, a giant ZSET
- Notifications sent twice: send only after the
UPDATEsucceeds (1 affected row), ideally through an outbox so it isn’t lost if you crash halfway. - Clock skew between instances: workers compare scores with their own machine’s clock. A skew of a few hundred ms only makes cancellations slightly early or late; for large skews, use Redis’s
TIMEas the shared reference. - Losing Redis data (an asynchronous failover, an accidental flush): the database safety-net scan from point 1 covers this case too. This is why Redis shouldn’t be the only place that knows “which rides need cancelling”.
- One giant ZSET on Cluster is one key on one node. With millions of elements, split it into bucketed keys (
order:pending-timeout:{0..15}by a hash of the orderId) and have each worker scan the buckets.
When timers are reset constantly (a million devices sending heartbeats, ~333K resets per second), each reset is a round trip to Redis; the ZSET is compared with an in-memory timing wheel in A liveness system: from a Redis sorted set to a timing wheel and the LMAX Disruptor.
9. Good habits when using Redis
Key idea: few round-trips, no long jobs on the server, and every key with a controlled size and TTL.
- Cut round-trips:
MGET/MSET,HMGET, and pipelining (sending many commands without waiting for each reply). 100 sequentialGETs with a 0.5 ms RTT take 50 ms; one pipeline takes about 1 ms. A pipeline isn’t atomic; if you need atomicity, useMULTI/EXECor Lua. - Don’t run long jobs on the server: Lua scripts with big loops, O(N) commands on large collections (section 5, part 1).
- Short TTLs for caches, and spread them with jitter so keys don’t expire all at once (cache failures).
- Avoid big keys: a string over 10 KB or a collection over 5,000 members is already worth a second look. Find them with
redis-cli --bigkeys,--memkeys,MEMORY USAGE; delete them withUNLINKinstead ofDEL(the big key section). - Know one node’s ceiling: roughly 50K SET/s and 80K GET/s without pipelining, and keep each instance ≤ 32 GB; the numbers and a 1-million-QPS counter case study are in Caching in system design: cache tiers and large-scale counters.
Four more habits and a complexity table for common commands
- Choose the right structure instead of stuffing everything into a JSON string.
- Hash tags for groups of keys you need to operate on together in Cluster (part 2).
- Know a command’s complexity before using it; the Redis docs state it for every command.
- Use structured key names:
service:entity:{id}:field, just short enough, easy to find and easy to restrict with prefix-based ACLs.
| Command | Complexity | Notes |
|---|---|---|
GET, SET, HGET, SADD, SISMEMBER |
O(1) | safe |
ZADD, ZRANK, ZREM |
O(log N) | safe |
ZRANGE ... LIMIT, small LRANGE |
O(log N + M) | safe when M is small |
HGETALL, SMEMBERS, LRANGE 0 -1 |
O(N) | only when you know N is small |
SINTER, SUNION, SORT |
O(N·M) or worse | careful on large collections |
KEYS, FLUSHALL (synchronous) |
O(N of the whole keyspace) | banned in production |
Exercises
Exercise 1: a monthly leaderboard with a tie-break
Task: a game has a monthly leaderboard. It needs the top 10, the current player’s rank, and on equal points whoever reached that score first ranks higher. The maximum score is 100,000. Old monthly boards are kept for 3 months and then deleted.
Solution to Exercise 1: a score packing points and timestamp, adding points with Lua
- Key
lb:{2026-11}, memberplayerId. Score =points × 10^10 + (9_999_999_999 − epochSecond when the points were reached). With points ≤ 100,000, the maximum score is about 10¹⁵ + 10¹⁰ < 2⁵³, so the double stays exact. - Adding points can’t use
ZINCRBY(the timestamp part has to change too), so use Lua: read the old score, extract the points (math.floor(score / 1e10)), add the new points, attach the current timestamp,ZADD. It runs atomically on the server. - Top 10:
ZREVRANGE lb:{2026-11} 0 9 WITHSCORES, then the app extracts the points for display. My rank:ZREVRANK+ 1. - Cleanup:
EXPIREAT lb:{2026-11}set to the end of the third month after it. If a month has tens of millions of players, this key is a big key: consider keeping only the top 10,000 with a periodicZREMRANGEBYRANK lb 0 -10001and storing full scores in the database.
Exercise 2: a distributed lock for two requests booking the same seat
Task (the Redis session’s homework, paraphrased): two concurrent requests book seat A34_S012C on flight FA634; use a Redis lock so only one request is processed, without implementing the database or the booking logic.
The full solution with runnable Java code is in the distributed lock review post, exercise 3.
Solution to Exercise 2: the key points
The code uses Jedis, SET NX PX + unlocking with a token-comparing Lua script; 200 rounds × 8 requests, with exactly one request booking each time.
- Lock exactly the resource:
lock:seat:FA634:A34_S012C. - Use a random token, and compare it on unlock.
- Re-check the seat’s state after acquiring the lock.
- In a real system, a conditional statement in the database is still the last line of defence.
Remember
- Explain why read-modify-write on a JSON string loses updates, while
HSETdoesn’t. - Know when a List is enough for a queue, and when you need a Stream or a real broker.
- Write an atomic claim for a ZSET delay queue, and remember the 784/2,000 number.
- Name the two holes left after an atomic claim (a crash after claiming, a race with the driver) and how to close them.
- Know which commands are O(N) or worse, and why they block every client.
Interview questions
Junior
Name Redis’s main data types and one use for each.
Five main types: String, List, Hash, Set, Sorted set; each has its own commands that run atomically on the server.
- String: caching JSON, counters, locks.
- List: recent-items lists, simple queues.
- Hash: objects by field, shopping carts.
- Set: likes, deduplication, mutual friends.
- Sorted set: leaderboards, delay queues.
There are also Streams (queues with acks and consumer groups), Bitmaps, HyperLogLog and Geo.
How do you build a shopping cart with Redis?
Each cart is a hash cart:{userId} with product IDs as fields and quantities as values.
HSET to add, HINCRBY to increase, HDEL to remove, HGETALL to fetch the whole cart, HLEN to count lines. Every operation is a single atomic command. Put a TTL on abandoned carts and don’t store prices in the cart.
Mid
Do you store a user object as a hash or as a JSON string?
A hash if you often read/update individual fields; a JSON string if you always read/write the whole object.
With a hash, an HSET on one field is atomic, several writers changing different fields cause no lost updates, and small hashes are more compact. A JSON string fits when the object is deeply nested or is a cache of an already serialized response. Updating JSON with read-modify-write needs Lua or WATCH; RedisJSON is a third option.
Why can using a List as a queue lose messages, and what do you use instead?
Because BRPOP removes the element before the consumer processes it, so a dying consumer loses it; use a Stream instead.
A list has no acks and no multiple consumer groups. You can patch it with BLMOVE into a processing list plus a cleanup job. Better is a Stream: consumer groups, a pending list, XACK, XAUTOCLAIM. For high volume or real durability, use Kafka/RabbitMQ.
How do you build a delay queue with Redis so multiple workers don’t process the same item?
A ZSET with the due time as the score, and workers must claim atomically.
Claim with a Lua script doing ZRANGE BYSCORE ... LIMIT and then ZREM in the same call, or only process the elements for which your own ZREM returned 1. Reading and deleting in two separate commands causes duplicates (I measured 784/2,000 with 4 workers). Add a visibility timeout for crashes, and make the handler idempotent with a conditional statement in the database.
How does a pipeline differ from MULTI/EXEC?
A pipeline cuts round-trips but isn’t atomic; MULTI/EXEC is a server-side transaction that runs as one block.
A pipeline is a client-side technique: send many commands in one go without waiting for each reply, so other clients’ commands can still interleave. MULTI/EXEC queues the commands and runs them as one uninterrupted block, but nothing is rolled back if one command fails at runtime. They are often combined: send MULTI ... EXEC inside a pipeline.
Senior
Design a leaderboard for 50 million players, with the top 100 and each player’s rank.
A sorted set for the top, but don’t put all 50 million players in one ZSET.
A ZSET with 50 million members is a big key (several GB, on one node, with heavy forks). Keep a ZSET only for the top N (tens of thousands), trimmed periodically. The rank of players outside the top comes from the database, or approximately from the score distribution (counting by score buckets, “you’re in the top 12%”). Encode the tie-break into the score while respecting the 2⁵³ limit. Split by season/region so each key stays moderately sized.
When should you not use Redis for a data problem?
When Redis would be the only source of truth for data that must not be lost.
- Acknowledged writes must never be lost (money, inventory): Redis replication is asynchronous.
- The data doesn’t fit in RAM at a reasonable cost.
- You need flexible queries, joins or transactions across many entities.
- You need to keep messages for a long time and replay them many times (Kafka).
Redis is at its best as an acceleration layer or for temporary state, with the source of truth somewhere else.
A ZSET delay queue: what happens if a worker crashes right after claiming?
The element has left the ZSET, so the job is dropped, unless the claim has a visibility timeout.
Instead of deleting it outright, the claim script moves the element into a processing ZSET with the processing deadline as the score. It is deleted only when the work is done; a job returns overdue elements. The handler must be idempotent (a conditional statement on the state) because it may run again. Add an infrequent database scan as a safety net, since Redis can also lose data on failover.
Useful phrases for interviews
- “Redis is really a data structure server, not just a key-value cache, so I pick the structure that matches the access pattern.”
- “A hash lets us update one field in a single atomic round-trip, without a read-modify-write race.”
- “Read-modify-write on a JSON blob causes lost updates under concurrency, so for a cache I’d just delete the key.”
- “A plain list has no acknowledgements, so if we need at-least-once delivery I’d move to Streams with consumer groups.”
- “We keep the leaderboard in a sorted set per month and encode the timestamp into the score as a tie-break.”
- “A delay queue on a sorted set uses the due time as the score, and workers claim due items atomically with a Lua script.”
- “If a worker dies, the item reappears after the visibility timeout, so the handler has to be idempotent.”
- “The database stays the source of truth; the cancel is a conditional update on the status.”
- “Pipelining cut the latency of the batch from fifty milliseconds to about one, but it isn’t atomic.”
- “For approximate cardinality at scale, HyperLogLog costs twelve kilobytes regardless of size.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.