Deployments rarely fail because the code is wrong; they fail at the handover: a new pod takes traffic before it is ready, an old pod is killed mid-request, or a new schema breaks the old version. This post compares deployment strategies, how to change a schema while two versions run at once, and what to configure in Kubernetes and Spring Boot so a deploy loses no requests. Read The Twelve-Factor App first, especially Disposability.
1. Six ways to ship a new version
Key idea: strategies differ in three things: how fast you can roll back, how much extra capacity you pay for, and whether two versions run at the same time. Rolling is the sensible default, canary is for risky changes, blue-green is for when you need instant rollback.
| Strategy | How | Rollback | Extra cost | Two versions at once |
|---|---|---|---|---|
| Recreate | Stop all old pods, start new ones | Redeploy the old version (with downtime) | 0 | No, but there is downtime |
| Rolling | Replace pods in small batches | kubectl rollout undo, also gradual |
A few pods (maxSurge) | Yes, during the whole rollout |
| Blue-green | Build a full new environment (green), flip the router once | Flip back to blue: seconds | Double during the switch | Briefly, but the database is shared |
| Canary | 1–5% of traffic to the new version, grow it based on metrics | Set the weight back to 0 | Small | Yes, for a while |
| Shadow (mirroring) | Copy real requests to the new version, discard its responses | Not needed, users never see it | Double load for mirrored traffic | Yes |
| Feature flag | New code is deployed but off, turned on per user group | Turn the flag off: instant | Near 0 | Both code paths live in one binary |
Recreate only fits when the two versions truly cannot run side by side. A Kubernetes Deployment supports recreate and rolling out of the box; blue-green and canary need a routing layer on top: switching a Service selector, a weighted ingress (the canary-weight annotation in ingress-nginx), a service mesh, or Argo Rollouts and Flagger.
flowchart LR
A[Deploy canary at 0%] --> B[5% traffic]
B --> C{Compare with old version:<br/>error rate, p99, SLO burn}
C -->|good after N minutes| D[25% then 50%]
D --> E{Still good?}
E -->|yes| F[100%, remove old version]
C -->|bad| R[Weight to 0, roll back]
E -->|bad| R
Shadow traffic: where it can hurt
2. Deploy is not release: feature flags
Key idea: deploying puts a binary on machines; releasing lets users see a feature. A feature flag separates the two, so you can deploy continuously, turn a feature on per user group, and turn it off in seconds without redeploying.
A flag is an if that reads its value from a config service (Unleash, LaunchDarkly, or the open OpenFeature standard). Three kinds: release flags hide unfinished code (live for weeks), kill switches turn off heavy features under load (live long), experiment flags split users by a hash of the user id for A/B tests.
A canary splits by pod, a flag splits by user; they combine well: roll out with the flag off, then enable it for 1% of users.
Dark launches and flag debt
3. Expand/contract: changing the schema while two versions run
Key idea: during a rolling or canary deploy, versions N and N+1 read and write the same database. So every schema change is split across releases: expand (add, don’t break), migrate (move data and reads/writes), contract (remove the old part once nobody uses it).
Take renaming the column phone to phone_number. Do it in one ALTER TABLE ... RENAME COLUMN and every old pod still running fails immediately.
| Release | Schema | Application | Safe to roll back to the previous release? |
|---|---|---|---|
| 1. Expand | Add nullable phone_number |
Write both columns, still read phone |
Yes, the old version ignores the new column |
| 2. Backfill | Copy phone → phone_number in batches |
Same as release 1 | Yes |
| 3. Switch reads | No change | Read phone_number, still write both |
Yes, phone is still written |
| 4. Stop old writes | No change | Write only phone_number |
Only back to release 3 |
| 5. Contract | Drop phone |
No change | No longer needed |
The rule: migrations run before the new code and must work with the code version right before them. Backfill in small batches as when deleting old data, and an ALTER on a large table must be online DDL. APIs and messages work the same way: adding a field is fine, removing or changing its meaning needs a full cycle like the one above, much like Schema Registry compatibility modes.
Flyway scripts for each step and other common changes
-- V41__expand_phone_number.sql (release 1)
ALTER TABLE customer ADD COLUMN phone_number VARCHAR(20);
-- V42__backfill_phone_number.sql (release 2, run repeatedly in batches, or as a separate job)
UPDATE customer SET phone_number = phone
WHERE phone_number IS NULL AND id BETWEEN :from AND :to;
-- V45__contract_phone.sql (release 5, once no pod reads phone any more)
ALTER TABLE customer DROP COLUMN phone;
How to handle other changes:
| Change | Safe approach |
|---|---|
| Add a NOT NULL column | Add it nullable (or with a DEFAULT), backfill, then add the constraint |
| Drop a column | First release: code stops reading/writing it and it leaves the JPA entity. Next release: drop |
| Change a column type, split a table | Like a rename: new thing, write both, backfill, switch reads, drop the old |
| Index a large table | CREATE INDEX CONCURRENTLY (PostgreSQL) or ALGORITHM=INPLACE, LOCK=NONE (MySQL) |
With JPA, an entity mapped to a column the database doesn’t have yet fails on the first SELECT (or ddl-auto=validate refuses to start), so the migration always goes first.
4. The Kubernetes a backend developer needs
Key idea: a Deployment keeps the pod count and drives the rollout; a Service groups the ready pods behind one stable address. Probes, rollout parameters, resources and the PodDisruptionBudget decide whether a deploy loses requests.
| Probe | Question | On failure | Should check |
|---|---|---|---|
| Startup | Have I finished starting? | Restart after failureThreshold × periodSeconds |
Same as liveness, with a long allowance |
| Readiness | Should I get traffic? | Removed from Service endpoints, not restarted | Internal state, maybe an essential shared dependency |
| Liveness | Can this process still recover? | Container restarted | Internal state only (deadlock, stuck) |
The startup probe disables the other two until it succeeds, so an app that takes 60 seconds to start isn’t killed by liveness. Readiness keeps running for the pod’s whole life, not only at startup.
maxSurge(default 25%, rounded up): how many pods abovereplicasmay exist.maxUnavailable(default 25%, rounded down): how many ready pods may be missing.maxSurge: 1, maxUnavailable: 0never reduces capacity, at the price of a slower rollout.- Resources:
requestsis what the scheduler reserves,limitsis the ceiling. Exceeding the CPU limit gets you throttled (higher p99); exceeding the memory limit gets you OOMKilled. See CPU quota and heap versus memory limit. - PodDisruptionBudget (PDB): limits how many pods may be taken down at once during voluntary disruptions such as a node drain, for example
minAvailable: 2.
Full manifest: Deployment, Service and PDB for a Spring Boot service
apiVersion: apps/v1
kind: Deployment
metadata: { name: order-service }
spec:
replicas: 4
minReadySeconds: 10 # a pod must stay ready for 10s to count as available
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
selector: { matchLabels: { app: order-service } }
template:
metadata: { labels: { app: order-service } }
spec:
terminationGracePeriodSeconds: 45 # > preStop + timeout-per-shutdown-phase
containers:
- name: app
image: registry.example.com/order-service:1.42.0 # pinned tag, never latest
ports: [{ containerPort: 8080 }]
resources:
requests: { cpu: "500m", memory: "768Mi" }
limits: { memory: "768Mi" } # many teams drop the CPU limit to avoid throttling
startupProbe:
httpGet: { path: /livez, port: 8080 }
periodSeconds: 5
failureThreshold: 24 # allows up to 120s to start
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet: { path: /livez, port: 8080 }
periodSeconds: 10
failureThreshold: 3
lifecycle:
preStop:
exec: { command: ["sh", "-c", "sleep 10"] }
---
apiVersion: v1
kind: Service
metadata: { name: order-service }
spec:
selector: { app: order-service }
ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: order-service }
spec:
minAvailable: 3
selector: { matchLabels: { app: order-service } }
A few details: progressDeadlineSeconds (default 600) marks a rollout as failed when it stops progressing, but Kubernetes does not roll back by itself; the pipeline must watch kubectl rollout status and call rollout undo. With requests == limits for every container the pod gets the Guaranteed QoS class and is evicted last when a node runs short of memory. This manifest was not applied to a real cluster while writing the post.
5. Spring Boot: health groups and graceful shutdown
Key idea: Actuator provides two health groups, liveness and readiness, for the probes. On SIGTERM, Spring Boot switches readiness to REFUSING_TRAFFIC, stops accepting new requests and waits for in-flight ones, bounded by spring.lifecycle.timeout-per-shutdown-phase.
# Probes on the main port, so they fail when the main web server is broken
management.endpoint.health.probes.add-additional-paths=true # /livez, /readyz
# Readiness includes the DB because the pod is useless without it; liveness stays default
management.endpoint.health.group.readiness.include=readinessState,db
server.shutdown=graceful # the default since Spring Boot 3.4
spring.lifecycle.timeout-per-shutdown-phase=30s
Older Spring Boot versions only enable the two groups when they detect Kubernetes or when management.endpoint.health.probes.enabled=true is set. With Actuator on a separate port, the probes can stay green while the main port is dead, hence add-additional-paths.
SIGTERM and removing the pod from endpoints happen in parallel, so a preStop sleep of a few seconds is needed before the app stops accepting connections. That detail, and the shell-form ENTRYPOINT swallowing SIGTERM, are covered in The Twelve-Factor App and the operating systems post.
sequenceDiagram
participant K as Kubernetes
participant P as Pod (Spring Boot)
participant L as Service / Ingress
K->>L: Remove pod from endpoints (async)
K->>P: Run preStop: sleep 10
Note over L: Stops sending requests a few seconds later
K->>P: SIGTERM
P->>P: Readiness = REFUSING_TRAFFIC, reject new requests
P->>P: Wait for in-flight requests (up to 30s)
P-->>K: Exit
Note over K,P: SIGKILL after terminationGracePeriodSeconds
6. Sessions and sticky sessions
Key idea: deploys kill pods, so state held in a pod’s memory is lost. Sticky sessions only reduce the symptom; move sessions out (Redis, JDBC) or use stateless tokens.
- Sticky sessions (pinning a user to one pod with a cookie) skew load, and the user still loses the session when that pod is replaced.
Canaries, shared sessions and long-lived connections
- A weighted canary can bounce one user between old and new versions; if the API behaviour changes, route the canary by user (header, cookie).
- An external session must deserialize in both versions (Spring Session, sessions and JWT). Long-lived connections (WebSocket, gRPC streams) must be closed by the server on shutdown, with clients reconnecting with jitter.
7. Canary: what to watch and when to stop
Key idea: compare the canary with the old version over the same period, on signals users feel: error rate, p99 and how fast the error budget burns. Stop automatically when a pre-agreed threshold is crossed.
| Signal | Example stop threshold | Note |
|---|---|---|
| 5xx error rate | Canary > old version + 0.5 percentage points | Relative, since the baseline changes by hour |
| p99 latency | Canary > 1.2 × old version | The average hides the slow tail |
| SLO burn rate | > 10 over 10 minutes | Burn rate = error rate / (1 − SLO) |
| Saturation | CPU throttling, connection waits, GC pauses | Correct but more expensive |
| Business signal | Orders per minute, payment success rate | Catches “200 OK but wrong” |
Burn rate says how many times faster than allowed the error budget is being spent: a 99.9% SLO allows 0.1% errors, so a canary at 0.5% errors has a burn rate of 5. Google’s SRE workbook alerts on a burn rate of 14.4 sustained for an hour, which has used 2% of a 30-day budget (see SLOs and error budgets).
8. Zero-downtime checklist
Key idea: miss one detail and you lose a few requests on every deploy.
- Expand/contract migrations, run before the code, compatible with version N-1; APIs and messages only add fields.
- Three separate probes; liveness never calls a dependency.
-
maxUnavailable: 0or enough spare capacity; setminReadySeconds. - Exec-form
ENTRYPOINT, graceful shutdown, apreStopsleep, a grace period longer than the whole shutdown. - No state in pod memory; a PDB for every service with 2+ pods.
- Pinned image tags, a rollback command you have tried, numeric canary criteria.
- Warm-up if cold pods raise p99 (JIT, connection pool, local caches).
Exercises
Exercise 1: Counting pods during a rolling update
A Deployment with replicas: 10 uses the defaults maxSurge: 25%, maxUnavailable: 25%. During a rollout, what is the maximum number of pods and the minimum number available? What changes with maxSurge: 1, maxUnavailable: 0, and what does it cost?
Solution to Exercise 1
- 25% of 10 is 2.5.
maxSurgerounds up to 3,maxUnavailablerounds down to 2. - At most 10 + 3 = 13 pods exist at once, and at least 10 − 2 = 8 are available. In the first step the controller creates 3 new pods and may remove 2 old ones right away.
- With
maxSurge: 1, maxUnavailable: 0: at most 11 pods, always 10 available. Only one pod is replaced at a time and each must become ready (plusminReadySeconds), so the rollout takes about ten startup cycles. If the cluster has no room for an 11th pod, the rollout stalls. - A service running at 80% CPU under the default settings makes 8 pods carry the load of 10, each at 100%. That’s when you pick
maxUnavailable: 0.
Exercise 2: Splitting full_name
Table customer has full_name. A new requirement needs first_name and last_name, and the service runs a rolling deploy with 6 pods. Write the release plan, stating how far back each step can roll.
Solution to Exercise 2
- R1 expand: a migration adds nullable
first_name,last_name. The code writesfull_nameand the two new columns (split by one agreed rule), still readsfull_name. Rolling back to R0 is safe because R0 doesn’t know the new columns. - R2 backfill: a batch job splits
full_namefor old rows and logs uncertain names (one word, extra spaces). Reads don’t change. Rollback is safe. - R3 switch reads: the code reads
first_name/last_name, still writes all three. Put it behind a feature flag to turn off quickly if names display wrongly. Rolling back to R2 is safe becausefull_nameis still correct. - R4 stop writing
full_name: write only the new columns and dropfull_namefrom the entity. From here you can’t go back to R2 (it would read stalefull_name), only to R3. - R5 contract: after an observation period (say a week), drop
full_name. If anyone else (reports, ETL, CDC) still reads it, move them first.
Rows created during the backfill already have all three columns thanks to R1, so the backfill only needs WHERE first_name IS NULL.
Remember
- Compare deployment strategies on rollback, cost and whether two versions run side by side.
- Write an expand/contract plan for a column rename, and say how far back each step can roll.
- Tell startup, readiness and liveness apart, and explain why liveness must not check the database.
- Work out pod counts from
maxSurge/maxUnavailable, and know what a PDB protects against. - Describe the pod shutdown sequence and the numeric signals that stop a canary.
Interview questions
Junior
What is the difference between a readiness probe and a liveness probe?
A failing readiness probe removes the pod from the Service but keeps it running; a failing liveness probe restarts the container.
Readiness answers “should you send me requests?”, and matters at startup, at shutdown and under temporary overload. Liveness answers “can I still recover?”, and should fail only when the process is truly stuck. A startup probe shields both during a slow start.
What is a rolling update, and how does Kubernetes avoid losing requests?
It replaces old pods with new ones gradually, sending traffic only to ready pods, within the maxSurge and maxUnavailable limits.
To truly lose nothing, the app must also shut down gracefully: receive SIGTERM, stop taking new requests, finish in-flight ones, plus a preStop sleep so the load balancer has time to drop the pod.
Mid
You need to rename a column that is in use. How do you deploy it with no downtime?
Expand/contract across several releases: add the new column, write both, backfill, switch reads, stop writing the old column, and only then drop it.
During a rolling deploy, old and new pods share one database, so the schema must suit both. Each migration must work with the code version right before it, and the drop happens only when you’ll never roll back to a version that reads the old column.
How does blue-green differ from canary, and when do you pick each?
Blue-green switches all traffic at once to a fully prepared new environment; canary sends a small share first and grows it based on metrics.
Blue-green suits cases that need instant rollback and can afford double capacity. Canary suits risky changes on high-traffic services, because a bug only reaches some users. Both share the database, so migrations still need to be compatible.
How do feature flags help with deployments, and what do they cost?
They separate deploy from release: code ships switched off, is enabled gradually per user, and can be turned off in seconds without a redeploy.
The cost: every flag doubles the code paths, both branches need tests, and old flags become technical debt. Release flags need an expiry date; kill switches live long and should be exercised regularly.
Senior
Every deploy shows a few dozen 502s on the dashboard. How do you investigate?
It is almost always a pod receiving traffic before it is ready, or being closed while still routed to; check both ends of the pod lifecycle.
- Startup: does readiness reflect true readiness (pool, caches), is
minReadySecondsset? - Shutdown: does the JVM get SIGTERM (exec-form
ENTRYPOINT), is graceful shutdown on, is there apreStopsleep so the ingress can drop the pod, is the grace period long enough? - Keep-alive: the proxy holds connections to old pods, so its keep-alive timeout must be shorter than the app’s, or the app sends
Connection: closewhile shutting down (see reverse proxy). - Line up the error timestamps with the pod’s
KillingandReadiness probe failedevents.
Design an automated canary for a payment service.
A weighted canary with steps of 5%, 25% and 50%, each with automated analysis comparing canary and old version, and automatic rollback when a threshold is crossed.
- Signals: 5xx rate, p99, SLO burn rate, payment success rate and transactions per minute, all labelled by
version. - Enough samples: hold a step until it has enough transactions, not just enough minutes.
- Route by user so nobody switches versions in the middle of a checkout.
- Schema and message changes go first via expand/contract. Tooling: Argo Rollouts or Flagger with Prometheus metrics.
What is a PodDisruptionBudget for, and what happens when it is misconfigured?
A PDB limits how many pods can be taken down at once during voluntary disruptions like a node drain, so the service always keeps enough pods.
It doesn’t apply to a Deployment rollout or to a node dying. Setting minAvailable equal to replicas, or a PDB on a single-pod Deployment, makes kubectl drain hang and blocks cluster upgrades. Use maxUnavailable: 1 or a minAvailable below the replica count.
Useful phrases for interviews
- “During a rolling update, old and new versions share the database, so every migration has to be backward compatible.”
- “We rename a column with expand and contract: add, dual-write, backfill, switch reads, then drop.”
- “Readiness takes the pod out of the load balancer; liveness restarts it.”
- “Liveness should never depend on the database, or a slow database restarts the whole fleet.”
- “We deploy behind a feature flag, so release is a config change, not a deploy.”
- “Blue-green buys instant rollback at the cost of double capacity.”
- “The canary is compared with the baseline over the same window, and it rolls back automatically if the burn rate goes above ten.”
- “A preStop sleep gives the ingress time to stop routing before the app closes its listener.”
- “A disruption budget protects us during node drains, not during our own rollouts.”
- “We can roll back code in a minute, but a dropped column is gone, so destructive migrations come last.”
Self-compiled notes from official documentation and hands-on practice.