Microservices in system design series: 1. Service boundaries · 2. Distributed transactions: 2PC, saga, outbox · 3. Case study: a liveness system
Placing an order means reserving stock in Inventory, charging money in Payment, and creating the order in Order: three databases, and no shared BEGIN ... COMMIT. This post compares the ways to keep “all or nothing” across services, when to pick which, and fixes two common myths. Transactions inside a single database are covered in the transactions series.
1. What’s left of ACID across services
Key idea: inside one database you get full ACID. Across services, we try to keep atomicity; each local transaction handles consistency and durability; isolation is mostly gone.
- Atomicity: all or nothing. This is what distributed transactions try to achieve.
- Consistency: data satisfies every constraint before and after. Each local transaction keeps its own part. (Not the same as “consistency” in CAP, where every read sees the latest write.)
- Isolation: concurrent transactions don’t see each other’s half-done state. Concurrent distributed transactions will see it, because every step commits locally right away.
- Durability: once committed, it isn’t lost. Each database handles this.
There are two kinds: internal (a distributed database like TiDB runs transactions across its own nodes) and cross-database, cross-service, which is the topic of this post.
2. Two-phase commit, and why it still isn’t truly “strong”
Key idea: in 2PC a coordinator asks every participant “can you prepare?” and only then says “commit”. It’s simple and consistent on the happy path, but it’s slow, it gets stuck when the coordinator dies, and readers can still see a skewed state between the two commits.
sequenceDiagram
participant TM as Coordinator
participant A as Service A (RM)
participant B as Service B (RM)
TM->>A: write A - 30
TM->>B: write B + 30
TM->>A: prepare
TM->>B: prepare
A-->>TM: yes (holds locks)
B-->>TM: yes (holds locks)
TM->>B: commit
TM->>A: commit
| Pros | Cons |
|---|---|
| Simple for the application (JTA/XA does the work) | Holds locks from the write to the end of phase 2: slow, high contention |
| Strong consistency on the happy path | Coordinator dies after prepare: participants hold locks and wait (blocking) |
| Few resources support XA: Kafka, Redis and partner REST APIs don’t |
Why “strong” is only theoretical: A = 60, B = 60, move 30 from A to B. The coordinator sends commit to B first; B commits, so B = 90. At that exact moment a client reads A and B: A is still 60 (it hasn’t received commit yet) and B is already 90. The total is 150, not 120. Phase 2 doesn’t happen at the same instant on every participant. Only distributed databases with a global clock or timestamp (Spanner, TiDB) give a consistent snapshot across nodes.
3. The consistency ladder: 2PC, AT, TCC, 2-phase message, saga
Key idea: the further down the ladder, the fewer global locks and the higher the concurrency. But the application has to do more work itself and accept longer windows of inconsistency.
| Model | How it works, briefly | Good fit |
|---|---|---|
| 2PC / XA | Coordinator, prepare then commit, locks held throughout | Few participants, all support XA, low load |
| AT (Seata) | Each participant commits locally right away; the framework writes an undo log to roll back, and holds a “global lock” on modified rows | You want something close to 2PC with little code change, and the participants are relational DBs |
| TCC | Each participant has 3 APIs: Try (reserve the resource), Confirm (use the reservation), Cancel (release it) | Resources that can be reserved: money, stock, seats |
| 2-phase message (DTM) | Send a “prepared message”, commit the local transaction, then confirm the message; the broker calls back if it never gets the confirmation | One local step + later steps that are sure to succeed |
| Saga | A chain of local transactions; on failure, run compensations in reverse | Many steps, many services, eventual consistency is acceptable |
TCC in detail: three classic traps
Example of charging money: Try moves 100 from available to frozen; Confirm deducts from frozen; Cancel moves frozen back to available. Three traps:
- Empty rollback:
Tryhasn’t arrived yet (slow network) butCancelalready has.Cancelmust notice “there was no Try” and do nothing, but still record that it cancelled. - Suspension:
Cancelfinishes, then a lateTryarrives and holds the money forever.Trymust check “was this already cancelled?” and refuse. - Idempotency:
Confirm/Cancelcan be called many times and must give the same result.
The common fix: a status table (global_tx_id, branch_id, state) with a unique key, and every API reads and writes it in the same local transaction. DTM and Seata ship a “barrier” that does this.
Why saga is popular in microservices: eventual consistency is usually fine (once the order is created, the user doesn’t care that stock updates a second later; a slightly delayed payment step is fine too), and saga holds no global locks, so it handles high load and scales well.
4. Saga: orchestration or choreography
Key idea: in orchestration a coordinator issues each step as a command; in choreography each service listens to events and does its own part. With few steps and few branches, choreography is neat. With many steps and branches, orchestration is easier to understand and debug.
Saga’s benefits: no global locks, no blocking, and partial recovery (a failure at step 4 compensates steps 1–3). The cost: eventual consistency, complexity, and every step and every compensation must be idempotent.
| Orchestration | Choreography | |
|---|---|---|
| Control | Order Processor calls Warehouse, Payment, Loyalty in turn | Warehouse listens to OrderPlaced and emits StockReserved; Payment listens to StockReserved… |
| Pros | Easy to debug: saga state lives in one place; the flow is readable | No central point; each team only handles its incoming event stream |
| Cons | The state machine is hard to maintain; logic piles up in the orchestrator | Hard to debug, no global state; the flow hides in subscriptions |
5. The pivot transaction
Key idea: sort saga steps into three kinds so you know which steps need a compensation and which must be retried until they finish.
Example: a food order saga.
| Step | Service | Transaction | Compensation | Kind |
|---|---|---|---|---|
| 1 | Order | createOrder() (PENDING) |
rejectOrder() |
Compensatable |
| 2 | Consumer | verifyConsumerDetails() |
— (read-only) | Compensatable |
| 3 | Kitchen | createTicket() |
rejectTicket() |
Compensatable |
| 4 | Accounting | authorizeCreditCard() |
— | Pivot |
| 5 | Kitchen | approveTicket() |
— | Retry until done |
| 6 | Order | approveOrder() |
— | Retry until done |
- Compensatable: before the pivot, there must be a way to undo.
- Pivot: the go/no-go step. If it succeeds, the saga is guaranteed to run to the end.
- Retriable: after the pivot, a step must never fail permanently; on error, retry it (idempotently) until it succeeds.
Design tip: make the step most likely to fail (payment) the pivot or put it early, and put the steps that are hard to undo (sending an email, calling a partner that can’t cancel) after the pivot.
6. A saga has no isolation: lost updates and dirty reads
Key idea: since every step commits right away, another saga can read or overwrite the half-done state. You compensate with design-level “countermeasures”.
Lost update: a “Create order” saga is running when a “Cancel order” saga starts and overwrites the status. How to handle it:
- Semantic lock: step 1 of “Create order” sets the status to
APPROVAL_PENDING. When “Cancel order” sees a*_PENDINGstatus, it either waits or fails fast so the client retries (recommended: simple, holds no resources). - Version: check the version or the expected status on update (
UPDATE ... WHERE id = ? AND version = ?), like an optimistic lock. - Commutative updates: design operations where order doesn’t matter, for example add to or subtract from a balance instead of overwriting it.
Dirty read: a saga reads data that was committed locally while the global saga isn’t done yet (and may still be compensated). How to handle it:
- Reorder: move risk-increasing steps (adding money, adding points) after the pivot, and risk-reducing steps (deducting, reserving) before it. That way readers see a “conservative” state.
- Re-read: before writing, read the data again to check it’s unchanged; if it changed, abort and start over.
7. Two myths
Key idea: distributed transactions don’t make a system faster, and the outbox or event sourcing is not a distributed transaction.
Myth 1: use distributed transactions to raise TPS. Their goal is correctness (atomicity), not speed. They only raise TPS when the data is already split (sharded, split into services) and lock contention is low, because then many nodes work in parallel. When contention is high (every transaction touches the same account), adding nodes only adds network round trips and lock hold time.
Myth 2: the outbox and event sourcing solve distributed transactions.
- Event sourcing is a storage approach: store a sequence of events instead of the current state, rebuild state by replaying, and get a “time machine” to see the state at any point. It doesn’t make several services commit together.
- The transactional outbox solves dual write: writing to the DB and then sending to Kafka touches two systems, and a crash in between leaves one updated and the other not. The outbox writes the message to an
outboxtable in the same transaction as the business data; a relay (polling, or CDC like Debezium) reads the table and pushes to the broker, at least once. Why emitting first or wrapping both in a transaction still breaks, and a working outbox with Debezium: dual write, exactly-once and ordering.
flowchart LR
subgraph Tx[Account A local transaction]
W1[Step 1: Debit A's balance] --> W2[Step 2: INSERT outbox]
W2 --> W3[Step 3: Commit]
end
W3 --> RL[Step 4: Relay reads outbox]
RL --> MQ[[Message queue]]
MQ --> CB[Step 5: Service B receives]
CB --> CB2[Step 6: Credit B, record processed_id, commit]
Combine the outbox (sender side) with an idempotent consumer (receiver side, deduplicating by message id) and you get a “local message table”: a reliable two-step choreography saga. Idempotent consumers are covered in idempotency with Kafka.
The outbox in Spring Boot
@Transactional
public void withdraw(long accountId, long amount, UUID transferId) {
accountRepo.debit(accountId, amount); // UPDATE ... WHERE balance >= ?
outboxRepo.save(new OutboxEvent(transferId, "account", // same transaction
accountId, "MoneyWithdrawn", toJson(accountId, amount)));
}
- Polling relay: a
@Scheduledjob every 200 ms runsSELECT ... FROM outbox WHERE sent_at IS NULL ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED, sends to Kafka (acks=all), and setssent_at. Simple; latency is about one poll interval. - CDC relay: Debezium reads the WAL/binlog, and the Outbox Event Router routes by
aggregate_type. Low latency, no polling load on the DB, but you have to run Kafka Connect. - Message key =
aggregate_idto keep per-account ordering. Clean up the outbox table regularly (partition by day, then drop). - Receiver side: a
processed_message(message_id PRIMARY KEY)table, inserted in the same transaction as the credit; duplicates are skipped.
Exercises
Exercise 1: An e-commerce checkout saga
Problem: checkout consists of: create the order, reserve stock, charge the wallet (or card), add loyalty points, and create a shipment with a delivery partner (which can’t be cancelled once created). Design the saga: step order, pivot, compensations, choreography or orchestration, and what happens when the user hits cancel midway.
Solution to Exercise 1
Step order and kinds:
- Order: create the order as
PENDING(semantic lock). Compensation:REJECTED. - Inventory: reserve stock (TCC-style:
reserved += qty). Compensation: release the reservation. - Payment: charge. Pivot. On failure (insufficient funds) → compensate steps 2 and 1.
- Inventory: confirm the stock deduction (
reserved → sold). Retry until done. - Loyalty: add points. Placed after the pivot because it increases value (avoids a dirty read: points granted, then taken back). Retry.
- Shipping: create the shipment with the partner. It can’t be cancelled, so it must come after the pivot. Retry with an idempotency key sent to the partner.
- Order:
CONFIRMED.
Coordination: 6–7 steps, a slow partner call, and the need to track each order → orchestration (an Order Saga storing its state in the DB or in Temporal). The orchestrator and services talk through Kafka command/reply, and each service uses an outbox + idempotent consumer.
Cancel midway: if the order is still PENDING → the cancel API returns 409 Conflict, try again later (semantic lock, fail fast). If it’s already CONFIRMED → a separate “Cancel order” saga: cancel the shipment if the partner allows it, refund (a new transaction), return the stock, deduct the points.
Timeouts: each step has a deadline. A timeout before the pivot triggers compensation; after the pivot, it raises an alert and retries.
Remember
- Explain why concurrent distributed transactions break isolation, and how ACID’s C differs from CAP’s C.
- Draw 2PC and show why a client can still read a skewed state between the two commits.
- Rank 2PC, AT, TCC, 2-phase message and saga by consistency, and say when to use each.
- Classify saga steps as compensatable, pivot, or retriable.
- List the countermeasures for lost updates and dirty reads in a saga, and say what problem the outbox actually solves.
Interview questions
Mid
How does 2PC work, and what are its downsides?
The coordinator sends prepare to every participant; if all say yes it sends commit, and if any says no it rolls back. The downsides: locks are held through both phases, and everything gets stuck if the coordinator dies after prepare.
Also: not every system supports XA (Kafka, partner REST APIs), and latency is set by the slowest participant.
What problem does the transactional outbox solve?
Dual write: writing to the DB and sending a message touch two systems, and a crash in between loses one of them. The outbox writes the message to a table in the same transaction as the data, then a relay pushes it to the broker.
The result is at-least-once delivery, so the receiver must be idempotent. The outbox does not make several services commit together.
How does a saga compensation differ from a rollback?
A rollback erases the change as if it never happened, inside one database; a compensation is a new business transaction that reverses the effect, like a refund.
Others may have seen the state before the compensation, and the compensation can fail, so it must be idempotent.
Senior
Saga orchestration or choreography? How do you choose?
A few linear steps → choreography (fewer moving parts, independent teams); many steps, branches, timeouts or audit needs → orchestration with a workflow engine.
Choreography makes the overall flow hard to see and debug; orchestration concentrates logic in one place and depends on the orchestrator’s scalability. A very complex flow can be a sign that the service boundaries are wrong.
What is a pivot transaction, and how does it shape saga design?
It’s the go/no-go step: once it succeeds, the saga must run to the end. Every step before it must be compensatable, and every step after it must be retriable.
So: make the step most likely to fail (payment) the pivot, and push steps that can’t be undone (partner calls, emails) and value-increasing steps (adding points) after it.
A saga has no isolation. How do you prevent lost updates?
A semantic lock: the first step sets a *_PENDING status, and another saga that sees it fails fast so the client retries, or waits.
On top of that: check the version on update, design commutative operations (add/subtract instead of overwrite), and reorder steps to reduce dirty reads.
A colleague suggests distributed transactions to raise TPS. How do you respond?
Distributed transactions are for correctness (atomicity), not speed; they only raise TPS when the data is already split and lock contention is low.
If transactions fight over the same resources, more nodes just add network round trips and lock hold time. Ask what the real bottleneck is (contention, I/O, CPU) before choosing sharding, caching, or a redesign.
When would you use TCC instead of a saga?
When the resource can be reserved (money, stock, seats) and you don’t want others to see a “charged, then refunded” state: Try reserves, Confirm uses, Cancel releases.
The price is three APIs per service, plus handling empty rollback, suspension and idempotency. A saga is easier when a step can’t be reserved, or when exposing intermediate states is acceptable.
Useful phrases for interviews
- “Across services we can aim for atomicity, but isolation is basically gone.”
- “Two-phase commit holds locks across both phases, and it blocks if the coordinator dies after prepare.”
- “Even with 2PC, a reader can see one side committed and the other not yet.”
- “We modeled checkout as a saga with four local transactions.”
- “A refund is the compensating transaction for a charge, not a rollback.”
- “Payment is our pivot transaction, so every later step must be retriable.”
- “APPROVAL_PENDING acts as a semantic lock on the order.”
- “Orchestration keeps the saga state in one place; with choreography, each service just reacts to events.”
- “The outbox pattern removes the dual-write problem; it doesn’t make services commit together.”
- “Distributed transactions are about correctness, not throughput.”
Notes based on the System Design course (Ronin Engineer), rewritten in my own words.