Communication in system design series: 1. A message’s journey, REST or gRPC · 2. Async, real time and a fraud detection case study
How components talk to each other is the hardest architectural decision to change after the database. This post breaks one call into layers to show what each layer decides, then covers the two most common synchronous choices: the data format (JSON or Protobuf) and the backend protocol (REST or gRPC).
1. Five things to weigh
Key idea: when two components talk, ask: will the message arrive (delivery guarantee), will changing the interface break the other side (compatibility), how fast and frugal is it (performance), what happens on failure (error handling), who can listen in (security).
- Delivery guarantee: a synchronous call without retries is at-most-once (may be lost). With retries it becomes at-least-once (may be duplicated). Effectively exactly-once requires an idempotent receiver (see idempotency in distributed systems).
- Compatibility: old clients and new servers must coexist during a rolling deploy.
- Performance, efficiency: latency, throughput, CPU spent serializing, bandwidth.
- Error handling: timeouts, retries, clear error codes, circuit breakers.
- Security: TLS or mTLS between services, authentication, authorization.
The course stresses: in practice, don’t put performance first. A protocol 20% faster that the team can’t debug is a bad choice.
2. A message’s journey
Key idea: every call passes through the same chain of layers on both sides, and each layer carries a different concern. When choosing technology, decide in this order: network I/O model → handling pattern → protocol → serialization.
flowchart LR
subgraph Client
A[Logic] --> B[Serialization]
B --> C[Protocol]
C --> D[Handling pattern]
D --> E[Network I/O]
end
E --> N((Network))
subgraph Server
F[Network I/O] --> G[Handling pattern]
G --> H[Protocol]
H --> I[Deserialization]
I --> J[Logic]
end
N --> F
| Layer | Concern | Example choices |
|---|---|---|
| Logic | Error handling | Timeouts, retries, fallbacks |
| Serialization | Compatibility, performance | JSON, Protobuf, Avro |
| Protocol | Delivery guarantee, efficiency, security | HTTP/1.1, HTTP/2, AMQP, the Kafka protocol, MQTT |
| Handling pattern | Performance | Thread-per-request, Reactor, virtual threads |
| Network I/O | Efficiency | Blocking, non-blocking, multiplexing (epoll) |
Network I/O. The five Unix I/O models and epoll have their own post: I/O models, epoll and file descriptors. What matters here: multiplexing I/O (one system call watching many connections) underlies every high-load server.
Handling pattern. Two main kinds:
- Thread-per-request (traditional): one thread per request, which both reads the socket (blocking) and processes it. Simple, but the thread count is capped by the pool, and many threads cost context switches.
- Reactor: separates waiting on I/O from computing. A main reactor accepts new connections; sub reactors (event loops) watch reads and writes on many connections; decoding, computing and encoding go to a worker pool. Netty, Nginx and Redis all follow this shape.
Java 21 virtual threads are a third path: write blocking-style code and let the runtime multiplex it onto epoll. Details on Tomcat, Netty and virtual threads are in the I/O model post.
Protocol. HTTP/2 brings four things over HTTP/1.1: multiplexing (many requests over one TCP connection), header compression (HPACK), binary framing, and server push. The full HTTP/1.1, 2, 3 comparison is in the web and HTTP post.
3. Serialization: JSON or Protobuf
Key idea: JSON is text, readable, schema optional, easy to evolve. Protobuf is binary, much smaller and faster, but needs a predefined schema, is hard to debug, and has strict compatibility rules.
| JSON (text) | Protobuf (binary) | |
|---|---|---|
| Compatibility | High: add fields freely if readers ignore unknown ones | High if you follow the rules; changing a field’s number or type breaks it |
| Complexity | Low, optional schema, human-readable | Higher: .proto files, code generation, hard to debug |
| Performance | Slower, bigger | Several times smaller, faster to parse |
| When to use | Performance needs aren’t extreme, public APIs | High-load internal services, Kafka messages with a schema registry |
Protobuf’s compatibility rules in short:
- On the wire, a field is identified by its number, not its name. Changing the number changes the data’s meaning.
- Adding a new field with a new number: safe, old readers skip unknown fields.
- Removing a field: mark both its number and name
reservedso nobody reuses them. - Changing a field’s type: almost always breaks (a few pairs such as
int32andint64can read each other but may truncate).
A safe Protobuf change and a breaking one
Version 1:
syntax = "proto3";
message Order {
string id = 1;
int64 amount = 2; // smallest currency unit
string note = 3;
}
Version 2, safe: remove note but reserve it, add new fields with new numbers.
message Order {
reserved 3;
reserved "note";
string id = 1;
int64 amount = 2;
string currency = 4; // old clients don't know field 4 and skip it
optional string coupon = 5; // optional distinguishes "not sent" from an empty string
}
A breaking version: move amount to number 4, or change it to string. Old clients send the amount in field 2; the new server reads field 4 and sees 0, or fails to parse. Catch this automatically with buf breaking in CI, or let a schema registry block it in BACKWARD mode.
On the JSON side in Spring Boot: the ObjectMapper Spring Boot configures has FAIL_ON_UNKNOWN_PROPERTIES turned off, so clients ignore unknown fields. But a hand-made new ObjectMapper() still has it on, and one new server field makes that client throw. This is the most common JSON compatibility bug.
4. Between backends: REST or gRPC
Key idea: REST works with everything (browsers, load balancers, every language) and is fast enough for most cases. gRPC is faster thanks to HTTP/2 and Protobuf, has streaming and a strict contract, at the cost of code generation and harder load balancing.
| REST | gRPC | |
|---|---|---|
| Compatibility | Browsers, load balancers, tools, every language understand it | Browsers need gRPC-Web and a proxy; fewer tools |
| Performance | Usually HTTP/1.1 with JSON | HTTP/2, Protobuf, async calls, bidirectional streaming |
| Complexity | Low | .proto files, code generation, schema versioning |
| Load balancing | L4 or L7 load balancers both work | Needs an L7 load balancer or client-side balancing |
| When to use | Performance needs aren’t extreme, public APIs | High-load internal services, streaming, polyglot teams needing a strict contract |
A middle path: REST over HTTP/2 (in Spring Boot, server.http2.enabled=true) gets multiplexing and header compression without changing how you program.
Deadlines and errors when calling gRPC from Java
gRPC propagates a deadline (an absolute cutoff) through every hop, so downstream services know how much time is left and give up when it’s gone. That’s the “timeout budget” REST has to build by hand with headers.
OrderReply reply = orderStub
.withDeadlineAfter(300, TimeUnit.MILLISECONDS) // the whole downstream chain shares this deadline
.getOrder(OrderRequest.newBuilder().setId(id).build());
On failure the client gets a StatusRuntimeException:
UNAVAILABLE: the server couldn’t be reached; safe to retry with backoff.DEADLINE_EXCEEDED: time ran out; retry only if the operation is idempotent and budget remains.INVALID_ARGUMENT,NOT_FOUND,FAILED_PRECONDITION: business errors, don’t retry.
gRPC has built-in retry configuration in the service config (attempts, retryable codes, backoff), but enable it only for idempotent methods. General timeout and retry design is in timeouts, retries and circuit breakers.
Exercises
Exercise 1: Choosing a protocol for three situations
Task (my own): choose REST or gRPC, JSON or Protobuf, and explain: (a) an API for external partners integrating payments; (b) a pricing service called by 5 other Java services at 30K RPS in total, each response a few hundred bytes; (c) a mobile app loading a product list.
Solution to exercise 1
- (a) REST + JSON. Partners use every kind of language and tool, need readable OpenAPI docs and want to try it with curl. Compatibility and ease of integration matter more than a few milliseconds. Add idempotency keys and request signing.
- (b) gRPC + Protobuf. Internal, high load, small messages: the savings in serialization CPU and multiplexing are clear. A strict contract keeps 5 teams from breaking each other. Remember to enable per-request load balancing (section 4) and set deadlines.
- (c) REST + JSON over HTTP/2 is the safe default: CDNs, HTTP caching and debugging tools all work. Consider gRPC or Protobuf only if measurements show mobile bandwidth is a real problem; even then you need a gateway layer for the web.
Exercise 2: Reviewing a .proto change
Task (my own): a PR changes the message below. Which changes break old clients?
// before
message User { string id = 1; string name = 2; int32 age = 3; string email = 4; }
// after
message User { string id = 1; string full_name = 2; string age = 3; string phone = 4; }
Solution to exercise 2
name→full_namekeeping number 2: doesn’t break the binary wire format (the number is the same), but does break anyone using Protobuf’s JSON mapping, because the JSON field name changes.agefromint32tostringkeeping number 3: breaks. The wire types differ, so old clients misparse or skip it.emailreplaced byphonereusing number 4: the worst break, because old clients silently read the phone number into the email field.
Fix: keep age as int32 and add a new field 5 if a string is needed; remove email with reserved 4; reserved "email"; and add phone = 6. Put buf breaking in CI so PRs like this are blocked automatically.
Remember
- List the five things to weigh when choosing how components communicate, and why performance isn’t first.
- Draw a message’s journey and say which concern each layer carries.
- Tell the Reactor pattern from the Actor model.
- State Protobuf’s compatibility rules: field numbers, adding, removing with
reserved, changing types. - Explain why gRPC on Kubernetes piles load onto one pod, and three ways to fix it.
Interview questions
Junior
How do JSON and Protobuf differ?
JSON is text, human-readable, with no required schema; Protobuf is binary, smaller and faster, but needs a schema file and code generation.
JSON suits public APIs and debugging; Protobuf suits high-load internal services and Kafka messages.
What does HTTP/2 improve over HTTP/1.1?
Multiplexing many requests over one TCP connection, header compression, and binary framing instead of text.
It removes head-of-line blocking at the HTTP layer but not at the TCP layer; HTTP/3 over QUIC solves the rest.
Mid
When would you choose gRPC over REST?
For internal service-to-service calls with high load and small messages, when streaming is needed, or when several languages need a strict contract.
For public APIs, browsers and partners, REST is still better because anyone can call it and the tooling is everywhere.
How do you add a field to a Protobuf message without breaking old clients?
Add the field with a new, never-used number; old clients skip unknown fields. Don’t change the number or type of existing fields.
When removing a field, mark its number and name reserved. Put buf breaking or a schema registry in CI to block breaking changes.
How does the Reactor pattern work?
One or a few event loops use epoll to watch many connections; when one is ready they call its handler, and heavy work goes to a worker pool.
The main reactor accepts connections, sub reactors handle reads and writes, workers decode, compute and encode. The golden rule: never block the event loop.
Senior
You scaled a gRPC service from 3 to 10 pods, but the 7 new pods get almost no requests. Why?
A Kubernetes Service balances per connection, and gRPC clients hold one long-lived HTTP/2 connection, so every request still goes to the old pods.
Fix it with a headless Service plus a round_robin client, an L7 proxy or service mesh, or MAX_CONNECTION_AGE so clients reconnect periodically.
A request passes through 4 services called synchronously. How do you design the timeouts?
Set one time budget for the whole request and propagate the deadline to each hop, which uses whatever is left, instead of each service having its own fixed timeout.
gRPC has deadline propagation built in; with REST you pass it in a header yourself. Retry at one layer only and only for idempotent operations, so call counts don’t multiply level by level.
Useful phrases for interviews
- “Before picking a protocol, I’d look at delivery guarantees, compatibility, error handling and security, not just speed.”
- “On the wire, Protobuf identifies fields by number, so reusing a number is a breaking change.”
- “Adding a field with a new number is backward compatible.”
- “The Reactor pattern separates waiting on I/O from doing the work.”
- “Never block the event loop with a JDBC call.”
- “I’d use REST at the edge and gRPC for a few hot internal paths.”
- “A Kubernetes Service balances connections, not gRPC requests.”
- “We enabled client-side load balancing with round robin over a headless service.”
- “Deadline propagation lets downstream services stop early when the caller has given up.”
- “HTTP/2 multiplexes requests, but TCP-level head-of-line blocking is still there.”
Notes based on the System Design course (Ronin Engineer), rewritten in my own words.