Skip to content
notes.vinhnq.com

Search

System DesignSenior14 min read

Communication in system design, part 1: a message's journey, Protobuf, REST or gRPC

Review 11 cards
On this page

Communication in system design series: 1. A message’s journey, REST or gRPC · 2. Async, real time and a fraud detection case study

How components talk to each other is the hardest architectural decision to change after the database. This post breaks one call into layers to show what each layer decides, then covers the two most common synchronous choices: the data format (JSON or Protobuf) and the backend protocol (REST or gRPC).

1. Five things to weigh

Key idea: when two components talk, ask: will the message arrive (delivery guarantee), will changing the interface break the other side (compatibility), how fast and frugal is it (performance), what happens on failure (error handling), who can listen in (security).

  • Delivery guarantee: a synchronous call without retries is at-most-once (may be lost). With retries it becomes at-least-once (may be duplicated). Effectively exactly-once requires an idempotent receiver (see idempotency in distributed systems).
  • Compatibility: old clients and new servers must coexist during a rolling deploy.
  • Performance, efficiency: latency, throughput, CPU spent serializing, bandwidth.
  • Error handling: timeouts, retries, clear error codes, circuit breakers.
  • Security: TLS or mTLS between services, authentication, authorization.

The course stresses: in practice, don’t put performance first. A protocol 20% faster that the team can’t debug is a bad choice.

2. A message’s journey

Key idea: every call passes through the same chain of layers on both sides, and each layer carries a different concern. When choosing technology, decide in this order: network I/O model → handling pattern → protocol → serialization.

flowchart LR
  subgraph Client
    A[Logic] --> B[Serialization]
    B --> C[Protocol]
    C --> D[Handling pattern]
    D --> E[Network I/O]
  end
  E --> N((Network))
  subgraph Server
    F[Network I/O] --> G[Handling pattern]
    G --> H[Protocol]
    H --> I[Deserialization]
    I --> J[Logic]
  end
  N --> F
Layer Concern Example choices
Logic Error handling Timeouts, retries, fallbacks
Serialization Compatibility, performance JSON, Protobuf, Avro
Protocol Delivery guarantee, efficiency, security HTTP/1.1, HTTP/2, AMQP, the Kafka protocol, MQTT
Handling pattern Performance Thread-per-request, Reactor, virtual threads
Network I/O Efficiency Blocking, non-blocking, multiplexing (epoll)

Network I/O. The five Unix I/O models and epoll have their own post: I/O models, epoll and file descriptors. What matters here: multiplexing I/O (one system call watching many connections) underlies every high-load server.

Handling pattern. Two main kinds:

  • Thread-per-request (traditional): one thread per request, which both reads the socket (blocking) and processes it. Simple, but the thread count is capped by the pool, and many threads cost context switches.
  • Reactor: separates waiting on I/O from computing. A main reactor accepts new connections; sub reactors (event loops) watch reads and writes on many connections; decoding, computing and encoding go to a worker pool. Netty, Nginx and Redis all follow this shape.

Java 21 virtual threads are a third path: write blocking-style code and let the runtime multiplex it onto epoll. Details on Tomcat, Netty and virtual threads are in the I/O model post.

Protocol. HTTP/2 brings four things over HTTP/1.1: multiplexing (many requests over one TCP connection), header compression (HPACK), binary framing, and server push. The full HTTP/1.1, 2, 3 comparison is in the web and HTTP post.

3. Serialization: JSON or Protobuf

Key idea: JSON is text, readable, schema optional, easy to evolve. Protobuf is binary, much smaller and faster, but needs a predefined schema, is hard to debug, and has strict compatibility rules.

JSON (text) Protobuf (binary)
Compatibility High: add fields freely if readers ignore unknown ones High if you follow the rules; changing a field’s number or type breaks it
Complexity Low, optional schema, human-readable Higher: .proto files, code generation, hard to debug
Performance Slower, bigger Several times smaller, faster to parse
When to use Performance needs aren’t extreme, public APIs High-load internal services, Kafka messages with a schema registry

Protobuf’s compatibility rules in short:

  • On the wire, a field is identified by its number, not its name. Changing the number changes the data’s meaning.
  • Adding a new field with a new number: safe, old readers skip unknown fields.
  • Removing a field: mark both its number and name reserved so nobody reuses them.
  • Changing a field’s type: almost always breaks (a few pairs such as int32 and int64 can read each other but may truncate).
A safe Protobuf change and a breaking one

Version 1:

syntax = "proto3";

message Order {
  string id = 1;
  int64 amount = 2;      // smallest currency unit
  string note = 3;
}

Version 2, safe: remove note but reserve it, add new fields with new numbers.

message Order {
  reserved 3;
  reserved "note";
  string id = 1;
  int64 amount = 2;
  string currency = 4;   // old clients don't know field 4 and skip it
  optional string coupon = 5;  // optional distinguishes "not sent" from an empty string
}

A breaking version: move amount to number 4, or change it to string. Old clients send the amount in field 2; the new server reads field 4 and sees 0, or fails to parse. Catch this automatically with buf breaking in CI, or let a schema registry block it in BACKWARD mode.

On the JSON side in Spring Boot: the ObjectMapper Spring Boot configures has FAIL_ON_UNKNOWN_PROPERTIES turned off, so clients ignore unknown fields. But a hand-made new ObjectMapper() still has it on, and one new server field makes that client throw. This is the most common JSON compatibility bug.

4. Between backends: REST or gRPC

Key idea: REST works with everything (browsers, load balancers, every language) and is fast enough for most cases. gRPC is faster thanks to HTTP/2 and Protobuf, has streaming and a strict contract, at the cost of code generation and harder load balancing.

REST gRPC
Compatibility Browsers, load balancers, tools, every language understand it Browsers need gRPC-Web and a proxy; fewer tools
Performance Usually HTTP/1.1 with JSON HTTP/2, Protobuf, async calls, bidirectional streaming
Complexity Low .proto files, code generation, schema versioning
Load balancing L4 or L7 load balancers both work Needs an L7 load balancer or client-side balancing
When to use Performance needs aren’t extreme, public APIs High-load internal services, streaming, polyglot teams needing a strict contract

A middle path: REST over HTTP/2 (in Spring Boot, server.http2.enabled=true) gets multiplexing and header compression without changing how you program.

Deadlines and errors when calling gRPC from Java

gRPC propagates a deadline (an absolute cutoff) through every hop, so downstream services know how much time is left and give up when it’s gone. That’s the “timeout budget” REST has to build by hand with headers.

OrderReply reply = orderStub
        .withDeadlineAfter(300, TimeUnit.MILLISECONDS)   // the whole downstream chain shares this deadline
        .getOrder(OrderRequest.newBuilder().setId(id).build());

On failure the client gets a StatusRuntimeException:

  • UNAVAILABLE: the server couldn’t be reached; safe to retry with backoff.
  • DEADLINE_EXCEEDED: time ran out; retry only if the operation is idempotent and budget remains.
  • INVALID_ARGUMENT, NOT_FOUND, FAILED_PRECONDITION: business errors, don’t retry.

gRPC has built-in retry configuration in the service config (attempts, retryable codes, backoff), but enable it only for idempotent methods. General timeout and retry design is in timeouts, retries and circuit breakers.

Exercises

Exercise 1: Choosing a protocol for three situations

Task (my own): choose REST or gRPC, JSON or Protobuf, and explain: (a) an API for external partners integrating payments; (b) a pricing service called by 5 other Java services at 30K RPS in total, each response a few hundred bytes; (c) a mobile app loading a product list.

Solution to exercise 1
  • (a) REST + JSON. Partners use every kind of language and tool, need readable OpenAPI docs and want to try it with curl. Compatibility and ease of integration matter more than a few milliseconds. Add idempotency keys and request signing.
  • (b) gRPC + Protobuf. Internal, high load, small messages: the savings in serialization CPU and multiplexing are clear. A strict contract keeps 5 teams from breaking each other. Remember to enable per-request load balancing (section 4) and set deadlines.
  • (c) REST + JSON over HTTP/2 is the safe default: CDNs, HTTP caching and debugging tools all work. Consider gRPC or Protobuf only if measurements show mobile bandwidth is a real problem; even then you need a gateway layer for the web.

Exercise 2: Reviewing a .proto change

Task (my own): a PR changes the message below. Which changes break old clients?

// before
message User { string id = 1; string name = 2; int32 age = 3; string email = 4; }
// after
message User { string id = 1; string full_name = 2; string age = 3; string phone = 4; }
Solution to exercise 2
  • name → full_name keeping number 2: doesn’t break the binary wire format (the number is the same), but does break anyone using Protobuf’s JSON mapping, because the JSON field name changes.
  • age from int32 to string keeping number 3: breaks. The wire types differ, so old clients misparse or skip it.
  • email replaced by phone reusing number 4: the worst break, because old clients silently read the phone number into the email field.

Fix: keep age as int32 and add a new field 5 if a string is needed; remove email with reserved 4; reserved "email"; and add phone = 6. Put buf breaking in CI so PRs like this are blocked automatically.

Remember

  • List the five things to weigh when choosing how components communicate, and why performance isn’t first.
  • Draw a message’s journey and say which concern each layer carries.
  • Tell the Reactor pattern from the Actor model.
  • State Protobuf’s compatibility rules: field numbers, adding, removing with reserved, changing types.
  • Explain why gRPC on Kubernetes piles load onto one pod, and three ways to fix it.

Interview questions

Junior

How do JSON and Protobuf differ?

JSON is text, human-readable, with no required schema; Protobuf is binary, smaller and faster, but needs a schema file and code generation.

JSON suits public APIs and debugging; Protobuf suits high-load internal services and Kafka messages.

What does HTTP/2 improve over HTTP/1.1?

Multiplexing many requests over one TCP connection, header compression, and binary framing instead of text.

It removes head-of-line blocking at the HTTP layer but not at the TCP layer; HTTP/3 over QUIC solves the rest.

Mid

When would you choose gRPC over REST?

For internal service-to-service calls with high load and small messages, when streaming is needed, or when several languages need a strict contract.

For public APIs, browsers and partners, REST is still better because anyone can call it and the tooling is everywhere.

How do you add a field to a Protobuf message without breaking old clients?

Add the field with a new, never-used number; old clients skip unknown fields. Don’t change the number or type of existing fields.

When removing a field, mark its number and name reserved. Put buf breaking or a schema registry in CI to block breaking changes.

How does the Reactor pattern work?

One or a few event loops use epoll to watch many connections; when one is ready they call its handler, and heavy work goes to a worker pool.

The main reactor accepts connections, sub reactors handle reads and writes, workers decode, compute and encode. The golden rule: never block the event loop.

Senior

You scaled a gRPC service from 3 to 10 pods, but the 7 new pods get almost no requests. Why?

A Kubernetes Service balances per connection, and gRPC clients hold one long-lived HTTP/2 connection, so every request still goes to the old pods.

Fix it with a headless Service plus a round_robin client, an L7 proxy or service mesh, or MAX_CONNECTION_AGE so clients reconnect periodically.

A request passes through 4 services called synchronously. How do you design the timeouts?

Set one time budget for the whole request and propagate the deadline to each hop, which uses whatever is left, instead of each service having its own fixed timeout.

gRPC has deadline propagation built in; with REST you pass it in a header yourself. Retry at one layer only and only for idempotent operations, so call counts don’t multiply level by level.

Useful phrases for interviews

  • “Before picking a protocol, I’d look at delivery guarantees, compatibility, error handling and security, not just speed.”
  • “On the wire, Protobuf identifies fields by number, so reusing a number is a breaking change.”
  • “Adding a field with a new number is backward compatible.”
  • “The Reactor pattern separates waiting on I/O from doing the work.”
  • “Never block the event loop with a JDBC call.”
  • “I’d use REST at the edge and gRPC for a few hot internal paths.”
  • “A Kubernetes Service balances connections, not gRPC requests.”
  • “We enabled client-side load balancing with round robin over a headless service.”
  • “Deadline propagation lets downstream services stop early when the caller has given up.”
  • “HTTP/2 multiplexes requests, but TCP-level head-of-line blocking is still there.”

Notes based on the System Design course (Ronin Engineer), rewritten in my own words.