Kafka internals series: 1. Inside the producer · 2. Serialization and Schema Registry · 3. Consumer groups, rebalancing and liveness
A Kafka broker only sees bytes; what those bytes mean is a contract between producers and consumers. This post is about that contract: where serializers sit, which format to pick, how Schema Registry keeps the contract, and how to change a schema without breaking running consumers. Java serialization (Serializable) is covered in byte streams and serialization.
1. Serializers: where they sit and where they go wrong
Key idea: a serializer turns key and value into bytes on the producer; a deserializer does the reverse on the consumer. Kafka neither knows nor checks the data type, so any mismatch only shows up when a consumer reads.
Three terms that get mixed up:
| Serialization | Parsing | Marshaling | |
|---|---|---|---|
| What | Object ↔ bytes for storage or transfer | Text → structure (JSON → tree, HTML → DOM) | Serialization for remote calls (RPC) |
| Direction | Both ways | Usually one way | Both ways |
Two ways to configure it in Java:
// 1. By property: simple, but no compile-time type check
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// 2. Pass instances: generic type safety, and you can configure the serializer yourself
var producer = new KafkaProducer<String, OrderPaid>(props,
new StringSerializer(), new OrderPaidSerializer(objectMapper));
2. Choosing a format
Key idea: JSON is easy to start with but has no explicit contract and takes more space. Avro and Protobuf have schemas, are 2–3× smaller, and let you check compatibility before deploying. When many teams read one topic, the contract matters more than the size.
Measuring one order with 5 fields (orderId, userId, amount, currency, status):
| Format | Size | Note |
|---|---|---|
| JSON | 85 bytes | Field names repeated in every message |
| Avro through Schema Registry | 29 bytes | 5-byte header + 24 bytes of data; field names live in the schema, not the message |
| Criterion | JSON (no schema) | Avro | Protobuf |
|---|---|---|---|
| Human-readable | Yes | No (needs the schema) | No (needs the schema) |
| Contract | Implicit, in each side’s code | JSON schema, evolves by rules | .proto file, evolves by field numbers |
| Automatic compatibility checks | Yes with JSON Schema + Registry | Yes | Yes |
| Fits | Few consumers, one team, lots of debugging | Kafka ecosystem (Connect, ksqlDB), data platforms | Already on gRPC, many languages |
The course slides pick JSON because it’s simple for beginners; that’s a reasonable starting point. Once three or more teams read a topic, I lean toward Avro or Protobuf with a Registry.
3. How Schema Registry works
Key idea: the producer sends the schema to the Registry once and gets an id back; each message carries only a 5-byte header (0x00 + id). The consumer reads the id, fetches the schema from the Registry (and caches it), then decodes.
sequenceDiagram
participant P as Producer
participant SR as Schema Registry
participant K as Kafka
participant C as Consumer
P->>SR: register schema for subject orders-value
SR-->>P: id = 1 (compatibility checked first)
P->>K: 0x00 + 00000001 + Avro data
C->>K: poll
K-->>C: 0x00 + 00000001 + data
C->>SR: GET /schemas/ids/1 (first time)
SR-->>C: schema, cached
Lab with Confluent’s KafkaAvroSerializer (client 8.0, an in-memory mock:// Registry): the first 8 bytes of the message are 00 00 00 00 01 12 6f 72. Byte 0 is the magic byte 0, the next 4 bytes are schema id 1 (big-endian), then the Avro data (12 is the zig-zag-encoded length of the string “ord-12345”, followed by o, r…). The subject created is orders-value.
- Subject: the name of a schema’s “version line”. By default
TopicNameStrategy:<topic>-keyand<topic>-value, so one topic carries one value type. - The Registry stores schemas in a compacted Kafka topic (
_schemas), so it is as durable as Kafka. - Clients cache schemas by id; if the Registry goes down, running producers and consumers keep working with schemas they know, but newly started instances, or an unseen id, fail.
Subject naming strategies and several event types in one topic
| Strategy | Subject name | When to use |
|---|---|---|
TopicNameStrategy (default) |
orders-value |
One topic, one event type |
RecordNameStrategy |
shop.OrderPaid |
The same event type appears in several topics, and a topic holds several types |
TopicRecordNameStrategy |
orders-shop.OrderPaid |
One topic holds several event types, each evolving separately within that topic |
Why put several event types in one topic: to keep ordering across events of the same entity (OrderCreated, OrderPaid, OrderShipped with the same orderId key must land in the same partition). The alternative is an Avro union or a Protobuf oneof wrapping every type under TopicNameStrategy, at the cost of editing the wrapper schema whenever you add an event type.
With Protobuf, the 4-byte id is followed by a list of message indexes (which message in the .proto file), so the header is longer than 5 bytes.
4. Compatibility modes: changing a schema without breaking consumers
Key idea: a compatibility mode is the rule the Registry enforces when you register a new version. It decides which changes are allowed and which side upgrades first.
| Mode | Guarantee | Upgrade first | Fits |
|---|---|---|---|
BACKWARD (default) |
The new schema can read data written with the previous schema | Consumers | Event topics where consumers reread old data |
FORWARD |
The previous schema can read data written with the new one | Producers | Slow-to-upgrade consumers, hard coordination |
FULL |
Both directions | Any order | Many teams, independent deploys |
*_TRANSITIVE |
As above, but against all older versions | Long-lived topics (long retention, compacted) | |
NONE |
No checks | Only when moving to a new topic |
I checked each kind of change with the Registry library’s AvroSchema.isCompatible() (v1 has the 5 fields above):
| Change v1 → v2 | BACKWARD | FORWARD | FULL |
|---|---|---|---|
| Add a field with a default | ✓ | ✓ | ✓ |
| Add a field without a default | ✗ | ✓ | ✗ |
Add an optional field (["null","string"], default null) |
✓ | ✓ | ✓ |
| Remove a field that has no default | ✓ | ✗ | ✗ |
| Rename a field | ✗ | ✗ | ✗ |
Rename with the old name in aliases |
✓ | ✗ | ✗ |
Change type long → double (Avro allows promotion) |
✓ | ✗ | ✗ |
Change type string → int |
✗ | ✗ | ✗ |
The Registry’s error for the second row is very clear: READER_FIELD_MISSING_DEFAULT_VALUE: The field 'note' ... has no default value and is missing in the old schema. The new reader, reading an old message without note, has nothing to fill in.
Evolution rules for Protobuf and JSON Schema
Protobuf identifies fields by number, not by name:
- Renaming a field: safe on the wire (names aren’t in the message), but breaks generated code and the JSON mapping.
- Adding a field: safe, old readers skip unknown fields.
- Removing a field: safe if you mark
reserved 5; reserved "note";so nobody reuses number 5. Reusing an old number for another type is a serious bug: old data is read as wrong values with no error. - proto3 has no “required”; every field has a zero default, so “not sent” and “sent 0” look the same (use
optionalor wrapper types).
JSON Schema in the Registry is harder: the result of a compatibility check depends on whether the content model is “open” or “closed” (additionalProperties), so the same change (adding a field) can pass or fail depending on how the schema is written. The rules are much trickier than Avro’s, which is why many teams pick Avro or Protobuf once they need a Registry anyway.
5. Spring Kafka: JSON and deserialization errors
Key idea: Spring’s JsonSerializer writes the Java class name into a __TypeId__ header by default, tying consumers to the producer’s package. And a message that can’t be deserialized makes the consumer fail forever unless wrapped in ErrorHandlingDeserializer.
spring:
kafka:
producer:
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
properties:
spring.json.add.type.headers: false # don't send the Java class name
consumer:
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
properties:
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
spring.json.value.default.type: com.shop.billing.OrderPaidEvent
spring.json.use.type.headers: false
spring.json.trusted.packages: com.shop.billing
I haven’t run this configuration for this post; property names follow the Spring Kafka 3.x docs.
ErrorHandlingDeserializer catches deserialization errors, returns null and attaches the exception in a header, so the container’s error handler decides (retrying a format error is pointless, so send it straight to a dead letter topic). DefaultErrorHandler, retry topics and DLTs are covered in partition counts, lag and long-running tasks; the plain-client mechanics are in part 3 of this series.
Exercises
Exercise 1: Renaming status to state
Topic orders uses Avro, BACKWARD mode, 7-day retention, 4 consumers owned by 3 teams. Plan the rename without breaking anyone.
Solution to Exercise 1
A direct rename is rejected in every mode (table in section 4). Two ways:
Option A: alias (BACKWARD). v2 declares {"name":"state","type":"string","aliases":["status"]}. A v2 reader can read v1 data (the alias maps the old name). But it isn’t FORWARD: a consumer still on v1 reading v2 data finds no status. So the order is: upgrade all 4 consumers to v2 first, then the producer.
Option B: expand, then contract (safe for independent deploys).
- v2: add
state(default""), the producer writes bothstatusandstate. FULL compatible. - Each team moves its consumer to read
stateon its own schedule. - Once metrics show nobody reads
statusand 7 days of retention have passed: v3 givesstatusa default, and v4 drops it.
Option B takes longer but needs no deploy coordination across 3 teams; I’d pick B for a shared topic.
Exercise 2: “Unknown magic byte!”
A consumer using KafkaAvroDeserializer starts throwing SerializationException: Unknown magic byte! at some offsets. List the likely causes and how to handle them.
Solution to Exercise 2
The value’s first byte isn’t 0x00, so the message wasn’t written by a Registry serializer:
- Another producer (a test script,
kafka-console-producer, a misconfigured new service) wrote JSON or raw strings to the topic. - A tombstone (
nullvalue) on a compacted topic: the deserializer returnsnull, and the handling code must toleratenull. - Key and value config swapped: the key is a
Stringbut the consumer setKafkaAvroDeserializerfor the key.
Handling: wrap with ErrorHandlingDeserializer and send failing messages to a DLT with their offset; find the offending producer via headers or the client.id in broker logs; stop it at the source with ACLs so only the right service can write to the topic.
Remember
- Can you explain why the deserializer must match the serializer, and why not to go through
String? - Can you describe the wire format
0x00+ 4-byte id + data, and whether running clients survive a Registry outage? - Do you know which changes pass under BACKWARD/FORWARD/FULL, and which side upgrades first in each mode?
- Do you know when you need
_TRANSITIVE? - Do you know the two traps of Spring’s
JsonSerializer: the__TypeId__header and trusted packages?
Interview questions
Junior
What does Kafka store, and who turns objects into bytes?
The broker stores only bytes; the producer’s serializer turns key and value into bytes, and the consumer’s deserializer does the reverse.
Kafka has no default serializer and doesn’t check types. Both sides must agree on the format; a mismatch fails when the consumer reads.
Why not deserialize to a String and then parse the JSON?
Because it costs an extra copy and decode per message; reading straight from bytes to an object is faster and makes less garbage.
Jackson can read directly from byte[], and Spring’s JsonDeserializer already does this.
Mid
How does Schema Registry work?
The producer registers a schema and gets an id; each message carries magic byte 0 plus a 4-byte id; the consumer looks up the id to get the schema and decode, with caching.
The Registry checks compatibility at registration and stores schemas in the _schemas topic. The default subject is <topic>-value.
How do BACKWARD and FORWARD compatibility differ?
BACKWARD: the new schema reads old data, so upgrade consumers first. FORWARD: the old schema reads new data, so upgrade producers first.
FULL is both, so deploy order doesn’t matter. Adding a field with a default passes all three.
How do you add a required field to an Avro event?
You can’t add it as “required” right away: add it with a default (or optional null), have producers fill in real values, and handle the default on consumers.
Adding a field without a default is rejected under BACKWARD because the new consumer has nothing to fill in when reading old messages.
Senior
What happens when Schema Registry goes down?
Running producers and consumers keep working with cached schemas; newly started instances, new schemas, or unseen ids fail.
So run the Registry with several instances (one leader for writes, others serving reads), and don’t roll out new schemas during a Registry incident. Turning off auto-register means producers don’t need to write to the Registry at runtime.
JSON or Avro for a new event platform?
If many teams read and topics live long: Avro or Protobuf with a Registry, because the contract is checked before deploy and messages are 2–3× smaller. One team, few consumers: JSON is fine.
My measurement: the same order is 85 bytes in JSON versus 29 bytes in Avro. The bigger win is that compatibility errors are stopped at the producer instead of breaking consumers at runtime.
Useful phrases for interviews
- “Kafka only stores bytes; the contract between producer and consumer lives in the serializer.”
- “I deserialize straight from bytes to the object instead of going through a String.”
- “The wire format is a magic byte, a four-byte schema id, then the Avro payload.”
- “Running clients survive a Registry outage thanks to the schema cache, but new instances don’t.”
- “Backward compatibility means consumers upgrade first; forward means producers go first.”
- “Every new field gets a default, otherwise the new reader can’t read old messages.”
- “For long-retention topics I use the transitive mode, because old data stays readable.”
- “A breaking change goes to a new topic with a period of dual writes.”
- “I turn off Spring’s type headers so consumers don’t depend on the producer’s class names.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.