The three goals system design rounds probe most are scalability, performance and availability, and they often pull in three different directions. This post gathers the foundational principles, states CAP and PACELC correctly, then walks through each goal: how to measure it, which levers to pull, where the trade-offs are. The numbers come from the back-of-the-envelope estimation post.
1. Foundational principles and common trade-offs
Key idea: set goals first, know every technology’s limits, accept trade-offs, avoid premature optimization and over-engineering. Most design decisions pick one side of a few familiar trade-off pairs.
The course’s foundational principles:
- Define goals first (with numbers, see the interview process).
- Know the limits: every technology has them, even the “great” ones. Compare those limits with your needs.
- Trade-offs: no option wins on every axis.
- Avoid premature optimization and KISS (keep it simple, don’t make people think): don’t add Kafka when an outbox table is enough.
- SOLID and 80/20 at system level: 20% of the flows carry 80% of the load, so optimize that 20%.
- Understand how existing solutions work before using them: knowing Redis is single-threaded is how you know one
KEYS *can freeze a node.
| Trade-off pair | Example |
|---|---|
| Consistency vs availability | CAP: during a partition, return an error or stale data |
| Consistency vs performance | Sync vs async; strong vs eventual consistency |
| Space vs time | Caching, compression, normalization vs pre-aggregation, batch vs stream |
| Performance vs security | Encryption and per-step authorization slow requests |
| Complexity vs flexibility | Microservices are flexible but hard to operate |
| Development speed vs quality | Ship fast with tech debt, or slow and solid |
Many trade-offs aren’t choices but can be resolved by separating a coupling in time, in space, or between parts and whole. That has its own post: analysing trade-offs like a senior.
2. CAP and PACELC, stated correctly
Key idea: in a distributed system, network partitions will happen, so CAP really asks: during a partition, do you return an error (keep C) or return possibly stale data (keep A)? PACELC adds: with no partition, you still choose between low latency and consistency.
The slide states CAP as “at most two of three”, which invites the misreading that you can “pick CA”. The three letters:
- C (consistency): every read sees the latest write, as if there were a single copy (linearizability).
- A (availability): every request to a live node gets a non-error response.
- P (partition tolerance): the system keeps working when the network between nodes drops messages or splits.
flowchart TD
A[Network partition?] -->|Yes| B{Choose}
B -->|C| C[Refuse or wait: error, timeout]
B -->|A| D[Answer now: possibly stale]
A -->|No| E{PACELC: choose}
E -->|L| F[Answer fast, replicate asynchronously]
E -->|C| G[Wait for replicas to confirm, slower]
Real examples:
| System | During a partition | Normal operation |
|---|---|---|
PostgreSQL primary + synchronous replica (synchronous_commit = on with a sync standby) |
PC: the primary waits, writes hang if the standby is gone | EC: every commit waits for the standby |
| PostgreSQL/MySQL async replicas (the default) | Failover may lose the last few transactions | EL: replica reads may be stale |
| Cassandra, DynamoDB | PA: keep accepting writes, reconcile later | EL by default, tunable with quorums |
| Redis Sentinel | Async replication: failover may lose writes | EL |
Consistency levels and quorums
Between “strong” and “eventual” there are several levels, strongest first:
- Linearizable: reads always see the latest write in real time. The most expensive.
- Sequential: all nodes see the same order of writes, possibly delayed.
- Causal: causally related things (a reply after a comment) are seen in the right order.
- Read-your-writes: you always see what you just wrote. Often done by reading from the primary for a few seconds after a write, or pinning the session to one node.
- Monotonic reads: you never see data “go back in time” between two reads.
- Eventual: once writes stop, replicas eventually converge.
Quorums in Dynamo-style systems: N replicas, a write waits for W acknowledgements, a read asks R replicas. If R + W > N, the read set and write set always overlap, so a read sees the latest version (assuming no clock problems or concurrent writes). Example N = 3, W = 2, R = 2. Lowering W or R is faster and more available, at the cost of possibly stale reads.
3. Scalability: four levers
Key idea: a system scales well when adding resources raises capacity roughly in proportion. Four levers: use resources efficiently, keep services stateless, separate (data, business, operations), and watch the shared infrastructure.
| Lever | What to do |
|---|---|
| Use resources efficiently | CPU: limit context switches. RAM: choose suitable data structures. Disk: compress, prefer sequential writes |
| Stateless | Don’t keep sessions in an instance’s RAM, so adding instances behind a load balancer just works |
| Separate the storage layer | Vertically: a database per service. Horizontally: partitioning, sharding |
| Separate the business layer | By domain (microservices, DDD); by importance (scale the core first, degrade the rest); by request source (external/internal, web/mobile) |
| Separate by operation | Write to the primary, read from replicas; cache to offload the database |
| Shared infrastructure | Database, load balancer, network bandwidth, third-party systems, cache: any of them can become the ceiling |
4. Performance: measure, find the bottleneck, then optimize
Key idea: set goals with throughput and p99 (e.g. ≥ 10K QPS, p99 < 250 ms), find the bottleneck with measurements, decide whether the load is CPU- or I/O-bound, fix that spot, then measure again. Optimization is a loop, not a step.
- Throughput: RPS, QPS, TPS, Mbps.
- Latency: use p99 (99% of requests are faster than this). Max is for reference only because one unlucky request skews it; the average hides the slow tail. Checking a p99 target with a load test that fails when it misses the SLO, without the tool under-reporting p99 (coordinated omission), is covered in Performance testing.
| CPU-bound | I/O-bound | |
|---|---|---|
| Symptoms | CPU near 100%, little waiting | Low CPU, threads mostly waiting on disk or network |
| Locate it | Profilers: perf, async-profiler, eBPF |
iostat, vmstat, APM (New Relic, Datadog), tracing |
| Optimize | Better data structures and algorithms; less work on the critical path (move it to async processing) | Fewer I/O calls (batching, no N+1); optimize the callee (indexes); non-blocking I/O |
| Thread pool | ≈ cores (+1) | ≈ cores × (1 + wait time ÷ compute time) |
At the communication layer: non-blocking, batching, compression, a compact serialization format, and a suitable network I/O model (the synchronous communication post).
Four bottleneck groups to check: CPU (context switches, I/O wait), memory (size, swapping), disk (IOPS depend on hardware, sequential writes are far faster than random, fragmentation), network (bandwidth, I/O model). How to measure each is in the I/O model post.
Why p99, and how tail latency multiplies under fan-out
A user request often calls many services or shards in parallel and waits for the slowest. If each call has a 1% chance of being slow (beyond p99), a request that makes 100 calls is slow with probability 1 − 0.99¹⁰⁰ ≈ 63%. Each component’s tail becomes the whole system’s body.
Ways to shrink the tail:
- Hedged requests: if no reply arrives by the p95 mark, send the request to another replica and take whichever answers first.
- Deadline budgets: split the request’s total time across its steps, so later steps know how much is left.
- Less fan-out: precompute or denormalize so a request touches fewer nodes.
- Watch GC pauses and noisy neighbours: two common causes of long tails on the JVM and in the cloud.
5. Availability: design for failure
Key idea: availability = MTBF / (MTBF + MTTR), where MTBF is the mean time between failures and MTTR the mean time to recover. Everything fails, so design for failure: have redundancy, detect fast, switch fast, and sacrifice the extras to protect the core.
Example: a service fails on average every 30 days and takes 1 hour to recover → 720 / 721 ≈ 99.86%. Cut MTTR to 5 minutes with automatic failover → ≈ 99.99%. Reducing MTTR is usually much cheaper than making hardware fail less.
| Group | Mechanism | Example |
|---|---|---|
| Redundancy, switchover | Leader election via a consensus protocol | Raft (etcd, Kafka KRaft), Paxos (older and harder) |
| Promote a standby | Redis Sentinel, Patroni for PostgreSQL | |
| Protection under failure | Timeouts, service degradation, rate limiting | See timeouts, retries, circuit breakers |
| Failure detection | Heartbeats, health checks | The load balancer drops nodes that don’t answer |
| Operations | Canary, blue-green releases | Send the new version 1–5% of traffic first |
| Chaos engineering | Kill nodes on purpose to prove failover works |
flowchart LR
H[Heartbeat lost] --> D[Failure detected]
D --> E[Elect a new leader or promote a standby]
E --> R[Reroute traffic]
R --> M[System keeps running, fix the dead node later]
The availability of a dependency chain multiplies: gateway 99.95%, service 99.9%, database 99.95% in series → 0.9995 × 0.999 × 0.9995 ≈ 99.8%. Two independent 99.9% copies in parallel → 1 − 0.001² = 99.9999%. That’s why every synchronous dependency you add is an availability debt.
SLI, SLO, SLA and error budgets
- SLI (service level indicator): a measured number. For example, the fraction of requests that return no 5xx and finish under 300 ms.
- SLO (objective): the internal target for an SLI. For example, 99.9% of requests over 30 days meet the SLI.
- SLA (agreement): a commitment to customers, with penalties. Always looser than the SLO (e.g. SLA 99.5% when the SLO is 99.9%) to leave a buffer.
Error budget = 100% − SLO. At 99.9% over 30 days, the budget is ~43 minutes of failure, or 0.1% of requests. The team may “spend” it on releases and experiments. When it runs out, feature releases stop and the effort goes into stability. This turns the “fast or stable” argument into one number both sides watch.
Alerting on burn rate (how fast the budget is spent) beats instantaneous thresholds: burning 2% of the monthly budget in 1 hour is worth waking someone up; a stray minute of errors isn’t.
6. Choosing technology
Key idea: prefer “boring”, proven technology the team already knows, with a large community and active development. Each team has only a few “innovation tokens”; don’t spend them where they make no difference.
Old technology has many “known unknowns” (documented bugs with workarounds) and few “unknown unknowns”. New technology is the opposite. To check whether a project is alive: commit and release frequency, number of contributors, whether issues get answers (sites like Open Hub aggregate this). The deeper analysis is in analysing trade-offs like a senior.
Exercises
Exercise 1: Availability of a call chain
Task (my own): an order request passes through an API gateway (99.95%), the order service (99.9%), synchronously calls the inventory service (99.9%) and the payment service (99.5%), and all share one PostgreSQL (99.95%). Compute the availability of the ordering flow and propose how to get it above 99.9%.
Solution to exercise 1
In series, so multiply: 0.9995 × 0.999 × 0.999 × 0.995 × 0.9995 ≈ 0.9920, i.e. ~99.2%, about 70 hours of downtime a year. The main culprit is payment at 99.5%.
How to raise it:
- Take payment off the synchronous path: create the order as “awaiting payment”, process payment asynchronously through a queue and report the result later. The flow becomes ≈ 0.9995 × 0.999 × 0.999 × 0.9995 ≈ 99.7%.
- Inventory: make reservation asynchronous, or run 2 copies in 2 AZs. Two 99.9% copies in parallel → ~99.9999%, and the chain becomes ≈ 0.9995 × 0.999 × 0.9995 ≈ 99.8%.
- Order service and database: multiple instances and automatic failover (Patroni) to cut MTTR. With order at 99.95% and the database at 99.99%, the chain reaches ≈ 99.9%.
Lesson: the number of services called synchronously in one request is the first thing to cut when you need high availability.
Exercise 2: Choosing C or A for three features
Task (my own): during a partition, should each feature favour consistency or availability? (a) shopping cart, (b) wallet balance, (c) like counter.
Solution to exercise 2
- (a) Shopping cart: A. Let items be added even if replicas aren’t in sync and merge later (union of the two carts). Losing an add-to-cart is much worse than a duplicated item. This is the classic Amazon Dynamo example.
- (b) Wallet balance: C. Refusing a payment beats spending the same money twice. Write through one primary with a non-negative balance constraint, and read the balance for decisions from the primary.
- (c) Likes: A. Being off by a few for a few seconds goes unnoticed. Count in several places and merge (distributed counters, CRDTs).
One system often makes all three choices at once: CAP is decided per kind of data, not for the whole system.
Remember
- State CAP correctly: during a partition choose C or A, and explain the “ELC” half of PACELC.
- Tell the C in CAP from the C in ACID.
- List the four scalability levers and the order in which a system scales from small to large.
- Know why p99 is used, and how tuning differs for CPU-bound and I/O-bound load.
- Compute availability from MTBF/MTTR, and for a chain of services in series or in parallel.
Interview questions
Junior
What’s the difference between scaling up and scaling out?
Scaling up replaces the machine with a bigger one; scaling out adds more machines. Scaling out needs stateless services and a load balancer.
Scaling up is simple but has a ceiling and is still a single point of failure. Scaling out has practically no ceiling and tolerates failures better, but it’s more complex for data.
Why use p99 instead of average latency?
Because the average hides the slow requests, and those are exactly the ones users remember.
p99 = 300 ms means 1% of requests take longer than 300 ms. With millions of requests a day, 1% is tens of thousands of users.
Mid
State the CAP theorem. Can you choose CA?
When the network between nodes partitions, a distributed system must choose between consistent answers (C) and always answering (A). CA only exists on a single node, not in a distributed system.
PACELC adds that without a partition you still choose between low latency and consistency, for example synchronous or asynchronous replication.
How does the C in CAP differ from the C in ACID?
ACID’s C means not breaking data constraints within one database; CAP’s C means every replica returns the same latest value.
You can satisfy ACID’s C on the primary and still read stale data from an asynchronous replica.
A service is slow. In what order do you optimize?
Set a numeric goal (p99, QPS), measure to find the bottleneck, decide whether it’s CPU or I/O, fix that spot, then measure again.
Most often it’s I/O: N+1 queries, missing indexes, calling APIs sequentially instead of in parallel. Don’t guess, and don’t optimize what you haven’t measured.
Senior
How do you take a system from 99.5% to 99.95% availability?
Cut MTTR (automatic detection and failover), remove synchronous dependencies from the main path, add redundancy at single points of failure, and release safely (canary, fast rollback).
Compute the chain’s availability first to find the weakest link. Add SLOs and an error budget to track progress, and chaos tests to prove failover really works.
When the system is overloaded, what keeps it from going down entirely?
Rate limiting at the entrance, degrading non-core features (sacrifice the pawn), timeouts and circuit breakers on every outbound call, and queues to absorb writes that can wait.
The list of features to switch off, and the switches themselves, must be prepared in advance and exercised in chaos tests, not decided while things are on fire.
What does R + W > N mean in a quorum?
With N replicas, writes waiting for W acknowledgements and reads asking R replicas, R + W > N guarantees the read set overlaps the write set, so a read can see the latest version.
N = 3, W = 2, R = 2 is a common setup. Lowering W or R is faster and more available but may read stale data; there are still edge cases with concurrent writes and clock skew, so a quorum is not linearizability.
Useful phrases for interviews
- “During a network partition we have to choose between consistency and availability.”
- “There’s no such thing as a CA distributed system; partitions will happen.”
- “Without a partition, the trade-off becomes latency versus consistency.”
- “Keeping the service stateless lets us scale out behind a load balancer.”
- “I’d rather measure first than do premature optimization.”
- “Fan-out turns tail latency into the common case, so I track p99.”
- “This workload is I/O-bound, so a bigger thread pool or non-blocking I/O helps; more CPU won’t.”
- “Automatic failover brings our MTTR down to a few minutes.”
- “Under overload we degrade gracefully by turning off recommendations.”
- “Once the error budget is spent, we freeze feature releases.”
Notes based on the System Design course (Ronin Engineer), rewritten in my own words.