Skip to content
notes.vinhnq.com

Search

System DesignSenior21 min read

Problem solving like a senior: definitions, causal diagrams, 5 Whys and a checklist

Review 14 cards
On this page

Problem solving and trade-offs series: 1. Problem solving · 2. Analysing trade-offs

A junior engineer gets a ticket and makes it work; a senior engineer gets a situation (“customers say the system is slow”, “our cloud bill doubled”) and has to work out what the real problem is and whose it is. System design interviews grade this first. After reading, you can define a problem, draw a causal diagram, and use 5 Whys, Seven-So-What and a review checklist; part 2, Analysing trade-offs like a senior, covers choosing between the solutions.

1. What is a problem?

Key idea: a problem is the gap between the reality a stakeholder perceives and the reality they expect, so there are three ways to close it, not one.

Three words do the heavy lifting:

  • Stakeholder: a problem always belongs to someone. The on-call team sees “alerts all night”; the CFO sees “the AWS invoice”.
  • Perceives: not objective reality. Users call the app “slow” because a blank spinner turns for three seconds, while the API takes 800 ms.
  • Expect: may or may not be reasonable. “The report must be real time” sometimes means “ready by 9 a.m. every day”.

Three ways to close the gap:

  1. Move reality towards the expectation: faster, cheaper, more stable. Usually the most expensive.
  2. Change how reality is perceived: progress bars, partial results, a “we’re working on it” email, cached data labelled “updated 2 minutes ago”.
  3. Adjust the expectation: negotiate the SLA, pin down how many seconds “real time” means, explain why exporting two million rows has to run in the background.
flowchart LR
  P[Perceived reality] -- gap = problem --> E[Expected reality]
  S1[1 · Change reality] -.-> P
  S2[2 · Change perception] -.-> P
  S3[3 · Adjust expectation] -.-> E

Options 2 and 3 aren’t about fooling users: many gaps are about information rather than performance.

2. Case study: the elevators at rush hour

Key idea: a busy office building has only two elevators; every morning they’re overloaded and many people clock in late. The cheapest fix turns out not to be another elevator.

2.1 Whose problem is it?

Key idea: list the stakeholders before thinking about solutions, because the person who pays and the person who uses are often different people.

Here: employees (ride the elevators), building management (receive the complaints), the owner (pays for new elevators), the tenant companies (set the clock-in time).

The course’s cartoon: the parents (the buyers) see a cot mobile’s beautiful animals from the side, while the baby (the user) lying underneath sees only… their bottoms.

2.2 A causal diagram

Key idea: a causal diagram breaks a symptom down into causes joined by AND/OR. Its leaves are candidate solutions whose costs you can compare.

flowchart TD
  F[Employees are frustrated] --> W[They wait too long]
  F --> N[Nothing to do while waiting]
  W --> T[Elevators serve slowly]
  W --> M[Too many people at once]
  T --> T1[Elevators are slow]
  T --> T2[Too few elevators]
  T1 --> S1[Upgrade elevators]
  T2 --> S2[Add elevators]
  M --> S3[Widen the clock-in window]
  N --> S4[Mirrors, TV news in the lobby]

Read it as: “frustrated” = waiting too long AND nothing to do; “waiting too long” = slow service AND too many people; “slow service” = slow elevators OR too few of them.

Direction Acts on Cost Time
Upgrade the elevators Capacity Very high Months
Add elevators Capacity Very high, sometimes impossible Months to years
Widen clock-in to 8:00–9:30 Demand Close to zero A week
Mirrors, TV in the lobby Perception Low A day

The cheapest fixes usually act on demand or perception, not capacity. Mapped onto software:

  • Adding an elevator = scaling out, buying a bigger machine.
  • Widening the clock-in window = rate limiting, queues for load levelling, staggered batch jobs, jittered cache TTLs (a random extra delay so keys don’t expire together).
  • Mirrors in the lobby = progress indicators, skeleton UIs, returning 202 Accepted with a job status, optimistic UI.

3. A problem-solving process

Key idea: seven steps, from a definition with a number to validating with data and presenting. It’s the five-step McKinsey-style process plus two final steps engineers tend to skip.

  1. Define the problem: one sentence, with a stakeholder and a number. “Search API p99 is 4 s between 9 and 10 a.m.; the target is under 800 ms” beats “search is slow”.
  2. Break it into issues: use a tree like the causal diagram. Branches should be MECE (mutually exclusive, collectively exhaustive).
  3. Eliminate non-key issues: 80/20, keep the branches likely to matter most.
  4. Analyse and plan: for each branch, the hypothesis, the data that would test it, who does it, how long.
  5. Do the work: measure, experiment, build a PoC.
  6. Validate with data: before/after numbers, not “it feels faster”.
  7. Present and share: an ADR, a postmortem, a top-down message.
flowchart LR
  A[1 · Define] --> B[2 · Break down]
  B --> C[3 · Drop minor issues]
  C --> D[4 · Analyse, plan]
  D --> E[5 · Do the work]
  E --> F[6 · Validate with data]
  F --> G[7 · Present]
  F -- hypothesis wrong --> B

In a system design interview, step 1 is clarifying requirements: pick at most three functional requirements and turn adjectives into numbers, see The system design interview: process and requirements.

3.1 A critical-thinking checklist for step 1

Key idea: a good problem statement answers the goals, scope, constraints, assumptions and the real question behind the request.

  • Goals: what does success look like, and how is it measured?
  • Scope: what’s in, what’s out?
  • Constraints: budget, deadline, mandatory technology, regulation.
  • Assumptions: what are we treating as true without having checked it?
  • Stakeholders and timeline.
  • The real question: what does the person asking to “add Redis” actually want? Usually “the home page loads in under a second”.
  • Past efforts: who tried what, and why didn’t it work?
  • New lenses: the user’s side, the operator’s side, the cost side.
  • Root causes: symptom or cause?

Watch for hidden inputs (config, the system clock, the JVM time zone, whatever is in a cache) and hidden outputs (logs, emitted events, emails, side effects on other systems) — the course draws them as dashed arrows around Input → Processor → Output. “Can’t reproduce” bugs often live in hidden inputs; incidents that spread often live in hidden outputs.

3.2 Three common traps

Key idea: jumping to solutions too early, solving the wrong problem, and only looking for answers inside the “box” you already know.

Jumping to solutions too quickly. In Kahneman’s terms, the brain has System 1 (fast, reactive) and System 2 (slow, deliberate). An incident at 2 a.m. calls for reflexes (roll back first, ask later); a redesign calls for slow thinking.

Solving the wrong problem. Two weeks of query tuning take p99 from 400 ms to 120 ms, and users still complain, because the real time goes into 6 MB of JavaScript. The fix: measure from the user’s side first.

Boxing, chess, football and more examples of optimising the wrong thing

The course compares the two thinking modes to boxing versus chess, or to football: escaping the press has to be fast and instinctive, but once you’re in space you need the calm to pick the killer pass.

Refusing to expand the problem. If the team only knows Java, every problem becomes “add another Java service”. Simpler answers often sit outside the box: an index, a CDN setting, a business-process change, or doing nothing.

4. The toolbox

Key idea: 5 Whys finds the root, Cartesian doubt and Socratic questions test assumptions, 80/20 drops minor branches, Seven-So-What surfaces consequences.

Tool Used for Sample question
5 Whys Finding the root cause or the real goal Ask “why?” until you reach something fixable at the system level
Cartesian doubt Questioning assumptions “Is this definitely true? Where’s the evidence?”
Socratic questions Expanding knowledge, correcting assumptions “If that’s true, what else must be true?”
80/20 (Pareto) Dropping non-key issues “Which 20% of causes produce 80% of the waiting?”
Seven-So-What Evaluating a solution’s consequences “We do that. So what? And then what?” repeated a few times

4.1 5 Whys on a backend incident

Key idea: a customer was charged twice; the root cause is a wrong design-time assumption (“clients never retry”), not a slow payment gateway.

  1. Why? There are two payment records for the same order.
  2. Why two records? The client resent the request after a 10-second timeout.
  3. Why the timeout? The payment gateway took 12 seconds while degraded.
  4. Why did the second request create a new payment? The API has no idempotency key and the payment table has no unique constraint per order.
  5. Why not? The original design assumed “clients never retry”.

The fix: an Idempotency-Key plus a unique constraint (see Idempotency in distributed systems), and “what happens on retry?” added to the design checklist for every write API.

4.2 Seven-So-What on a proposed solution

Key idea: ask “so what?” repeatedly about a proposal to expose its harmful effects before you build it.

The proposal “add a Redis cache to the product detail API” reveals stale data, invalidation work, one more system to run, and the database taking all the load when Redis dies.

The full Seven-So-What chain for adding Redis
  • So what? Reads get faster and the database gets lighter.
  • So what? Data can be stale until the TTL expires. Is a wrong price for a few minutes acceptable?
  • So what? If not, we must invalidate on update and handle the races between reads and writes (see Cache-aside and invalidation).
  • So what? One more system to operate and monitor, plus a plan for when Redis dies (see Cache incidents).
  • So what? When Redis dies, all the load lands on the database at peak time…

These consequences go in the “harmful effects” column when you score options with the ideality formula (part 2).

4.3 Other frameworks and a few habits

Key idea: frameworks are just scaffolding; the habits worth keeping are speaking top-down (conclusion first) and reasoning from first principles.

Systems engineering, PDSA, TRIZ, the pyramid principle, first principles and the “Einstein” quotes
  • Systems engineering, PDSA (Plan–Do–Study–Act, a continuous-improvement loop), TRIZ (the theory of inventive problem solving; part 2 borrows some of its ideas). No framework replaces practice.
  • Top-down communication (the pyramid principle): conclusion first, reasons and data after. Listeners get the message straight away; the speaker is forced to understand the problem exactly and cut the noise.
  • First-principles thinking: reason up from basic facts (how many IOPS an SSD can do, how long a network round trip takes) instead of copying “company X does it this way”.
  • Stay curious and observant: many good solutions come from noticing how other systems (elevators included) solve the same shape of problem.

5. A checklist for backend engineers

Key idea: review a distributed design with five questions (order, duplication, relationship, direction, consistency) and three mental ones (misconception, assumption, expectation).

Item Question Example bug
Order Is ordering guaranteed? Within what scope does it need to be? OrderCancelled reaches the consumer before OrderCreated because they’re on different partitions
Duplication What if a message, request or job runs twice? A retry creates two payments; Kafka’s at-least-once delivery processes a message twice
Relationship How does the data relate, who owns it, what happens to one side when the other is deleted? A user is deleted while bookings still point at them; two services write the same table
Direction Who calls whom, which way do dependencies point? Any cycles? A calls B, which calls back into A for data, creating a timeout loop
Consistency Sync or async? How long may data be out of date? Reading from a replica right after writing to the primary, so users don’t see their own change
  • Misconception: misunderstanding a concept (believing @Transactional also covers sending a Kafka message).
  • Assumption: assumptions nobody checks (“clients never retry”, “server time is Vietnam time”).
  • Expectation: expectations nobody says out loud (“real time”, “never lose data”).

Tip: paste the five questions into your design doc template or PR description to catch design bugs at the cheapest stage. The Consistency question leads straight to CAP and PACELC (C or A during a partition, latency or consistency otherwise), covered in Design principles: CAP, scalability, performance and availability.

Exercises

Exercise 1: A causal diagram for “the API is slow at 9 a.m.”

Task (paraphrased): Support reports that users complain “the app is slow” between about 8:45 and 9:30 every morning. Draw a causal diagram like the elevator case and propose at least one fix each for capacity, demand and perception.

Solution to exercise 1: the diagram and fixes by group
flowchart TD
  U[Users find the app slow] --> L[Long waits]
  U --> I[No idea what the app is doing]
  L --> C[Server processes slowly]
  L --> D[Many requests at once]
  C --> C1[Missing indexes, heavy queries]
  C --> C2[Few pods, small connection pool]
  D --> D1[Everyone logs in at the same time]
  D --> D2[Reporting batch runs at 9:00]
  I --> I1[Blank spinner, no progress]
  • Capacity: add indexes after reading EXPLAIN ANALYZE, schedule autoscaling before 8:45, grow the connection pool if you measure threads waiting for it.
  • Demand: move the reporting batch to 2 a.m. (often the biggest and cheapest win), cache the home dashboard for 60 seconds, rate-limit the export endpoint.
  • Perception: skeleton UI, load the top of the page first, make exports return 202 Accepted with a status.

Validate: compare p95/p99 by hour before and after, one change at a time.

Exercise 2: Write a problem statement

Task (paraphrased): Your manager says “the revenue report is too slow, see if we can use Elasticsearch”. Write a problem statement using the checklist in section 3.1 before discussing technology.

Solution to exercise 2: a problem statement for the revenue report
  • Stakeholders: accountants (users, daily), the CFO (client, month end).
  • Reality and expectation: the monthly report takes 45 seconds; the target is under 5 seconds during the last three days of the month; data up to one hour old is acceptable (to be confirmed).
  • Scope: the revenue report only. Constraints: no new infrastructure this quarter; data lives in PostgreSQL 16.
  • Assumption to check: the slowness is the query (not yet measured), not the Excel export step.
  • The real question: “use Elasticsearch” is an option; the goal is “monthly report under 5 seconds”. Cheap direction first: a daily summary table (a materialized view or a summary table refreshed hourly), which the “up to one hour old” expectation allows.

Exercise 3: 5 Whys for an incident

Task (paraphrased): A reminder-email job sent the same customers three copies in one night. Run 5 Whys down to a root cause you can fix at the system level.

Solution to exercise 3: a job running on three pods
  • Why three emails? All three pods ran the job.
  • Why? @Scheduled runs on every instance, and last week the HPA scaled from 1 to 3 pods.
  • Why did nobody notice? The test environment has one pod.
  • Why doesn’t the job protect itself against duplicates? The design assumed “there’s only one instance”.

Root cause: the assumption about instance count was never written down or checked. Fix: a distributed lock for the job (ShedLock on a DB table) and a “sent” marker per (reminder_id) with a unique constraint so any repeated run is harmless; add “what if this runs on N instances?” to the review checklist (the Duplication item). See also Cron jobs.

Remember

  • Can you state the definition of a “problem” and the three ways to close the gap, with an example for each?
  • Can you draw a causal diagram for an overload situation and sort its leaves into capacity, demand and perception?
  • Can you write a one-sentence problem statement with a stakeholder and a number before naming any technology?
  • Do you know when 5 Whys hasn’t reached the root (“human error”) and why it needs a causal diagram alongside?
  • Can you recall the five distributed-design review questions: order, duplication, relationship, direction, consistency?

Interview questions

Junior

How do you define a “problem” before solving it?

A problem is the gap between the reality a stakeholder perceives and the reality they expect.

So before solving it I need to know whose problem it is, what the current reality measures, and what the expectation is. The definition also gives three ways to close the gap: change reality, change perception, or adjust the expectation.

What is 5 Whys and when do you use it?

Asking “why?” repeatedly (around five times, not a strict number) to get from a symptom to a root cause, or from a surface request to the real goal.

I use it in postmortems, when analysing bugs, and when a request arrives as a solution (“add Redis”) to find the goal behind it (“home page under one second”). Stop at a cause fixable at the system level, not at “human error”.

Mid

The system is overloaded every morning at 9. How do you approach it?

Measure first to find where the time goes, then try the cheap fixes (spread demand, change perception) before adding capacity.

Measure: which endpoints, where the time goes (database, pool, CPU, external services), whether any job runs at the same time. Three families of fixes: reduce or spread demand (move batches, cache, rate-limit, queue), change perception (progress, 202 Accepted), add capacity (indexes, scheduled autoscaling). One change at a time, comparing numbers before and after. It’s the elevator problem: widening the clock-in window is far cheaper than installing another elevator.

What does a causal diagram add to 5 Whys?

5 Whys follows a single line, so it misses things when one effect has several causes; a causal diagram is a tree that shows every branch.

Each node is the result of its children, joined with AND or OR, so you can see where a branch can be cut. For example, “users are frustrated” = long waits AND nothing to do while waiting; cutting either branch is enough to reduce frustration. The leaves become a list of candidate solutions whose costs you can compare.

Your manager asks you to “add Redis to the system”. How do you respond?

Treat it as an option, not the problem: find the real goal and measure the bottleneck first.

Ask: what’s slow, how it’s measured, what the target is, how stale the data may be. Measure to find the bottleneck (a missing index? N+1 queries? an external service?). If caching really is a good option, still list the consequences (invalidation, operating Redis, what happens when it’s down). Answer top-down: “I propose X to reach goal Y; Redis is one option, but first we need to measure Z.”

When reviewing a design that uses a message queue, what do you check?

I go through order, duplication, relationship, direction and consistency, then retries, a DLQ and lag monitoring.

Order: is it needed, and within what scope (per key or partition)? Duplication: delivery is at-least-once, so consumers must be idempotent. Relationship: who owns the data, and does the event carry enough of it? Direction: who depends on whom, any cycles? Consistency: which parts are async, how much lag is acceptable, and is publishing inside the transaction (outbox)? Plus retries, a dead-letter queue and lag monitoring.

Senior

How do you present a technical proposal to a non-technical audience?

Top-down, following the pyramid principle: the conclusion first, then two or three reasons with numbers, then cost and risk.

For example: “I propose moving the reporting batch to 2 a.m. Why: it uses 60% of the IOPS at 9 a.m., exactly when users complain. Cost: reports are up to 7 hours older, and accounting has agreed.” To talk like that you have to understand the problem precisely and cut the noise, so it’s a test for yourself too.

During an incident, how do you move fast without “jumping to solutions”?

Separate two phases: mitigate with reflexes now, fix the root cause with slow thinking once things are stable.

Mitigation: roll back the latest deploy, switch off a feature flag, shift traffic, scale temporarily; cheap and reversible. Root cause: gather data, draw a causal diagram, run 5 Whys, then write a blameless postmortem. The trap is doing it the other way round: hunting for the root cause while users are still failing, or stopping at the temporary patch and never doing the slow part.

Useful phrases for interviews

  • “Before I propose anything, can I check whose problem this is and how we measure it today?”
  • “Let me restate the problem in one sentence, with a number, so we agree on what ‘done’ means.”
  • “There are three ways to close this gap: change the system, change how it’s perceived, or adjust the expectation.”
  • “This looks like a demand problem rather than a capacity problem, so I’d start by spreading the load.”
  • “I’m making an assumption here that clients don’t retry; if that’s wrong, the design changes.”
  • “Let’s separate mitigation from the root cause: roll back now, investigate once it’s stable.”
  • “Going one level deeper: why did the system allow that mistake without anyone noticing?”
  • “My recommendation first, then the reasoning: we should move the batch job, because it accounts for most of the load at nine.”
  • “Before adding Redis, I’d like to measure where the time actually goes.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.