Skip to content
notes.vinhnq.com

Search

System DesignSenior18 min read

Inside Kafka consumer groups: rebalancing, assignors, static membership and liveness

Review 15 cards
On this page

Kafka internals series: 1. Inside the producer · 2. Serialization and Schema Registry · 3. Consumer groups, rebalancing and liveness

A consumer group is how Kafka splits partitions across consumers and re-splits them when someone joins, leaves or dies. This post goes into the protocol underneath, measures the cost of each splitting strategy, and answers the two liveness questions: “is the consumer alive?” and “is it still working?”. The post on configuring Kafka by goal has the settings tables; this one is about mechanisms and measurements.

1. Coordinator and group leader

Key idea: the coordinator is a broker that manages the group’s members, offsets and liveness. In the classic protocol, though, working out “who owns which partition” is done by a consumer: the group leader.

  • Finding the coordinator: the consumer sends FindCoordinator(group.id); the answer is the broker leading partition hash(group.id) % 50 of the internal __consumer_offsets topic (50 is offsets.topic.num.partitions). Every broker coordinates some share of the groups.
  • Group leader (classic): usually the first consumer to join. It receives the member list and their subscriptions, runs the assignor, and sends the result to the coordinator.
  • Why on the client: adding or changing a strategy (even a custom assignor) needs no broker upgrade. KIP-848 reverses this decision (section 5).

2. The classic protocol: JoinGroup, SyncGroup and eager rebalancing

Key idea: a classic rebalance has two phases: every member sends JoinGroup (declaring topics and assignors) and the leader computes the assignment; then every member sends SyncGroup to receive its share. Between the phases, in eager mode, nobody owns anything.

sequenceDiagram
  participant C0 as Consumer 0 (leader)
  participant C1 as Consumer 1
  participant GC as Coordinator
  C0->>GC: JoinGroup(topics, assignors)
  C1->>GC: JoinGroup(topics, assignors)
  GC-->>C0: you are leader, member list
  GC-->>C1: leader is C0
  C0->>C0: run assignor
  C0->>GC: SyncGroup(assignment for everyone)
  C1->>GC: SyncGroup()
  GC-->>C0: C0's share
  GC-->>C1: C1's share
  • After each rebalance the group gets a new generation (a number). Commits carrying an old generation are rejected: that’s how Kafka stops a kicked-out consumer from overwriting the new owner’s progress.
  • Eager: before JoinGroup, every consumer calls onPartitionsRevoked for all of its partitions. The whole group stops until SyncGroup completes (“stop-the-world”).
  • Rebalances happen when a consumer joins, leaves (closes or gets kicked out), changes its subscription, a topic gains partitions, or a new topic matches a subscribed regex.

3. Assignors: four strategies, measured

Key idea: choosing an assignor balances even spread, few partition moves and little stopping. Cooperative-sticky (or KIP-848) wins in most cases; range is still needed when two topics must be co-partitioned.

Lab on Kafka 4.3.2, Java client 4.1.1: topics A and B with 3 partitions each. Start c1, add c2 after 4 seconds, add c3 after 6 more.

Assignor c1 + c2 c1 + c2 + c3 Revocations (c1 / c2) Partitions moved when c3 joins
Range (eager) c1: A0 A1 B0 B1, c2: A2 B2 (4/2) 2/2/2 10 / 2 4
RoundRobin (eager) 3/3 2/2/2 9 / 3 4
CooperativeSticky 3/3 2/2/2 4 / 1 2
KIP-848 (uniform) c1: A0–A2, c2: B0–B2 2/2/2 4 / 1 2

Reading it:

  • Range splits each topic separately, and the first consumer (by member id order) gets each topic’s remainder. With several topics the remainders pile onto the same consumer: 4/2 instead of 3/3.
  • Under eager, c1 revoked 6 then 4 partitions (10 total), although adding c3 only required moving 2. Range also shuffles: c2 loses A2 and B2 but gains A1 and B1.
  • Cooperative-sticky keeps almost everything: when c3 joins, only A1 (from c1) and B2 (from c2) change owners. The log shows two rounds about 3 seconds apart: round 1 revokes, round 2 hands them to c3.
ConsumerRebalanceListener: where to commit, and revoked versus lost
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
  public void onPartitionsRevoked(Collection<TopicPartition> parts) {
    // still a valid member: commit finished work for the partitions about to go
    consumer.commitSync(currentOffsets(parts));
  }
  public void onPartitionsAssigned(Collection<TopicPartition> parts) {
    // load local state, or seek to offsets stored outside Kafka (e.g. in the DB, same transaction)
  }
  public void onPartitionsLost(Collection<TopicPartition> parts) {
    // already kicked out: the partitions may belong to someone else, do NOT commit, just clean up
  }
});
  • With eager, onPartitionsRevoked gets every partition; with cooperative, only those that actually move (in the lab c1 got just [A1]).
  • onPartitionsLost is called when the consumer finds it’s no longer in the group (session expired, max.poll.interval.ms exceeded). A commit here fails, or worse, overwrites the new owner’s offset if there’s a logic bug.
  • The callbacks run on the thread calling poll(), so keep them short: the whole group’s rebalance waits for the slowest callback.
  • Spring Kafka: the container commits pending offsets on revocation (with the default BATCH AckMode); to step in, configure a ConsumerAwareRebalanceListener.

4. Static membership: restarts without rebalancing

Key idea: give each instance a stable group.instance.id (for example the pod name) and the coordinator remembers its share. An instance that returns within session.timeout.ms gets exactly the same partitions back, and nobody else is disturbed.

Lab (classic, cooperative-sticky, session.timeout.ms=20000, two instances pod-1 and pod-2):

Step Result
pod-2 off for 5 seconds, then back pod-2 gets back exactly [A1, B0, B2]; pod-1 revokes 0 partitions
pod-2 off for 25 seconds (beyond the session timeout) After about 22.8 seconds pod-1 takes over [A1, B0, B2]

A dynamic member (no group.instance.id) sends LeaveGroup on close and triggers an immediate rebalance; a static member does not, so the coordinator waits for the session timeout.

5. KIP-848: the new consumer protocol

Key idea: since Kafka 4.0, setting group.protocol=consumer makes the broker compute the assignment and deliver it to each consumer through heartbeats. No group leader, no JoinGroup/SyncGroup, no barrier making the whole group wait.

Classic KIP-848 (group.protocol=consumer)
Who computes the assignment Group leader (client) Coordinator (broker), group.remote.assignor = uniform (default) or range
Mechanism JoinGroup + SyncGroup, everyone waits for everyone Heartbeats carry a “target assignment”; each consumer releases and acquires gradually
One slow consumer Slows the whole group’s rebalance Slows only its own share
Heartbeat, session Client sets heartbeat.interval.ms, session.timeout.ms Broker sets group.consumer.heartbeat.interval.ms (5 s), group.consumer.session.timeout.ms (45 s)

In the section 3 table, KIP-848 gave the same revocation counts as cooperative-sticky (4 / 1) with no client-side assignor config. max.poll.interval.ms is still a client setting.

6. Liveness: alive and still working

Key idea: Kafka checks two different things. Heartbeats (a background thread) answer “is the process alive”; the gap between two poll() calls answers “is it making progress”. Failing either gets you kicked out of the group.

Liveness (something good eventually happens: records eventually get processed) pairs with safety (something bad never happens). The slides give “at least one consumer per partition” as the safety property; it should be at most one consumer in the group owning a partition, plus order kept within a partition.

Check Who does it Settings (default) Fails when
Alive Heartbeat thread to the coordinator heartbeat.interval.ms 3 s, session.timeout.ms 45 s Process dies, network cut, GC pause too long
Working The client measures time between poll() calls max.poll.interval.ms 300 s, max.poll.records 500 One poll’s processing takes too long, the processing thread hangs

Rule: session.timeout.ms ≥ 3 × heartbeat.interval.ms (two missed heartbeats don’t count as dead). Setting both too low keeps brokers busy and raises false alarms.

The endless-loop lab (classic, max.poll.interval.ms=4000, max.poll.records=5, 1.2 s per record, so 6 s per poll):

[  287 ms] poll -> 5 records from offset 0; processing 1.2 s each
[ 6295 ms] commitSync FAILED: CommitFailedException ... consumer was kicked out of the group
[ 6296 ms] LOST [lv-slow-0]
[ 6303 ms] assigned [lv-slow-0]
[ 6309 ms] poll -> 5 records from offset 0; processing 1.2 s each
[12315 ms] commitSync FAILED: CommitFailedException ...

Three rounds in a row, the consumer reprocessed the same 5 records from offset 0. Heartbeats were steady, so from outside the consumer looked “healthy”; lag didn’t drop, and every side effect repeated.

The same scenario under KIP-848, and static members that miss the poll deadline

Rerunning the lab with group.protocol=consumer (twice, same result): the commit after 6 seconds succeeded, then the partition was reported LOST, the consumer rejoined and continued from offset 5, with no reprocessing. My guess is that the lab group had only one member: when it left, the group was empty and the broker accepted the commit. In a group where another member has already taken the partition, that commit would very likely be rejected. I don’t rely on this behaviour; the formula in the pitfall above is still the right approach.

A static member (group.instance.id) that exceeds max.poll.interval.ms does not send LeaveGroup like a dynamic member. The coordinator only notices when session.timeout.ms expires. Meanwhile nobody makes progress on its partitions.

Quick production check: the consumer metrics time-between-poll-avg / time-between-poll-max and last-poll-seconds-ago (consumer-metrics group). time-between-poll-max approaching max.poll.interval.ms is an early warning before the loop starts.

7. Poison pills at the client level

Key idea: a poison pill is a record the consumer can never process. Without a way to skip it, the consumer stays stuck at that offset forever, and the whole partition stalls.

Two kinds:

  • Deserialization errors: the bytes aren’t in the expected format. With the plain client, poll() throws RecordDeserializationException (with topicPartition() and offset()), and the next poll throws the same error again, because the read position doesn’t move.
  • Processing errors: deserialization works, but business logic throws (bad data, a broken constraint).

Lab: a topic with 3 records, where the record at offset 1 is 5 junk bytes instead of a 4-byte integer. The consumer read offset 0, then three polls in a row threw RecordDeserializationException at lv-poison-0 offset 1. After seek(partition, 2) it carried on with offset 2.

try {
  for (var r : consumer.poll(Duration.ofMillis(500))) handle(r);
  consumer.commitSync();
} catch (RecordDeserializationException e) {
  deadLetter.send(e.topicPartition(), e.offset(), e);   // keep it for investigation
  consumer.seek(e.topicPartition(), e.offset() + 1);    // skip the broken record
}

In Spring Kafka, ErrorHandlingDeserializer + DefaultErrorHandler + DeadLetterPublishingRecoverer do this (see part 2 and the long-running tasks post).

Exercises

Exercise 1: Rolling deploys make lag spike

A service has 20 pods, one consumer each, in a group reading a 60-partition topic with the default assignor; each pod takes about 40 seconds to restart. Every deploy, lag spikes for 10 minutes. Explain and propose fixes.

Solution to Exercise 1

Why: the default [Range, CooperativeSticky] means range with eager rebalancing. Each pod shutdown causes a rebalance (a dynamic member sends LeaveGroup), and each startup another: about 40 stop-the-world rounds, each making every consumer give back everything, possibly taking seconds to tens of seconds if a consumer is in a long batch (the rebalance waits for it to call poll).

Fixes, in order:

  1. If brokers and clients are on 4.x: group.protocol=consumer. No barrier, each pod only affects its own share.
  2. If still on classic: switch to CooperativeStickyAssignor (two rolling deploys).
  3. Static membership: group.instance.id=${POD_NAME} (a StatefulSet keeps names stable), session.timeout.ms around 60 s (above the 40 s restart). A pod back within 60 s causes no rebalance.
  4. Shrink what’s left of each rebalance: a small max.poll.records so consumers call poll often, and clean shutdown (consumer.wakeup() + close()) to commit before leaving.

Measure before and after with the rebalance count (rebalance-total in consumer-coordinator-metrics) and p99 lag during deploys.

Exercise 2: Choosing max.poll.records

A consumer calls an external API for each record: p50 120 ms, p99 1.5 s, API timeout 5 s. max.poll.interval.ms is the default. Choose max.poll.records and explain why not to use p50.

Solution to Exercise 2

The bad case is a whole batch of slow records, so size by the API timeout (5 s, the real ceiling per record), not p50:

  • By timeout: 300 s / 5 s = 60 records; with a 2× margin → 30.
  • By p99: 300 / 1.5 / 2 = 100. But when the external API has an incident, every record hits the 5 s timeout, and 100 × 5 s = 500 s > 300 s: the reprocessing loop starts exactly when the system is weakest.
  • p50 gives 1,250 records, completely wrong.

Pick 30. If throughput isn’t enough, don’t raise records: process in parallel within a batch (keeping per-key order), or move the API calls to a separate queue.

Remember

  • Can you explain how the coordinator is chosen and why classic leaves assignment to the client?
  • Can you state the cost of eager rebalancing with numbers: 10 revocations versus 4?
  • Do you know range splits per topic, and when you still need range?
  • Do you know what static membership trades, and what session timeout to pick?
  • Can you tell heartbeats from max.poll.interval.ms, and compute a safe max.poll.records?

Interview questions

Junior

Can two consumers in the same group read the same partition?

No: within a group, each partition belongs to at most one consumer. With more consumers than partitions, some sit idle.

Two different groups are independent, and each reads all the data.

When does a consumer group rebalance?

When a consumer joins, leaves or is considered dead (missed heartbeats, exceeded max.poll.interval.ms), when a subscription changes, or when a topic gains partitions.

Static membership makes a quick restart stop counting as “leave then join”.

Mid

How do eager and cooperative rebalancing differ, in numbers?

Eager revokes every partition from every consumer each time; cooperative revokes only what must move, over two rounds.

In a 6-partition lab adding two consumers, eager (range) revoked 10 partitions from the first consumer, cooperative-sticky only 4. When the third consumer joined, cooperative moved only 2 partitions.

How do heartbeats and max.poll.interval.ms differ?

Heartbeats prove the process is alive and run on a background thread; max.poll.interval.ms proves the processing loop is moving, measured between poll() calls.

A process stuck in processing still heartbeats fine; only the poll check catches it. That’s why raising session.timeout.ms doesn’t rescue a slow batch.

Can you commit offsets in onPartitionsLost?

You shouldn’t: by then the consumer has been kicked out and the partition may belong to someone else, so the commit fails or overwrites the new owner’s progress.

Commits belong in onPartitionsRevoked, while the consumer is still a valid member. onPartitionsLost only cleans up.

Senior

What does KIP-848 change compared with the classic protocol?

The broker computes the assignment and delivers it via heartbeats; no group leader, no JoinGroup/SyncGroup and no group-wide barrier; each consumer releases and acquires partitions gradually.

Consequences: one slow consumer doesn’t drag the group; heartbeat and session timeout are configured on the broker; the assignor is chosen with group.remote.assignor. It needs brokers and clients on 4.0+.

Design a consumer group so a rolling deploy of 50 pods doesn’t spike lag.

KIP-848 if possible; otherwise cooperative-sticky plus static membership with a session timeout above one pod’s restart time.

Also: clean shutdown to commit before leaving, max.poll.records small enough that rebalances don’t wait on long batches, and measure rebalance-total and lag during deploys to prove it.

The consumer shows no errors, heartbeats are steady, but lag doesn’t drop and side effects repeat. What do you suspect?

The max.poll.interval.ms loop: kicked out, commit fails, the same batch comes back and gets reprocessed forever.

Check time-between-poll-max and logs for CommitFailedException. Fix by lowering max.poll.records to fit worst-case processing time, or by moving long work out of the poll loop.

How do you handle poison pills and still keep per-key ordering?

Classify errors: retry transient ones, DLT permanent ones; for order-sensitive flows, block that key and route later events for the same key to the DLT too.

Plain skipping keeps the partition moving but breaks per-key order; stopping the partition keeps order but lets one bad record block every other key. Choose by business need.

Useful phrases for interviews

  • “The coordinator is just the broker leading the group’s partition of the offsets topic.”
  • “In the classic protocol the group leader, a consumer, computes the assignment.”
  • “Eager rebalancing is stop-the-world; in my test the first consumer revoked ten partitions where four would do.”
  • “The range assignor works per topic, so remainders pile up on one consumer.”
  • “Static membership lets a pod restart within the session timeout without any rebalance.”
  • “With KIP-848 the broker owns assignment and there’s no group-wide barrier anymore.”
  • “Heartbeats prove the process is alive; polling proves it’s making progress.”
  • “If a batch takes longer than max poll interval, you get an endless reprocessing loop.”
  • “A poison pill stalls the partition until you seek past it or send it to a dead letter topic.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.