Skip to content
notes.vinhnq.com

Search

Git, DevOps & ToolingSenior21 min read

Deployment strategies: rolling, blue-green, canary, feature flags and zero downtime on Kubernetes

Review 17 cards
On this page

Deployments rarely fail because the code is wrong; they fail at the handover: a new pod takes traffic before it is ready, an old pod is killed mid-request, or a new schema breaks the old version. This post compares deployment strategies, how to change a schema while two versions run at once, and what to configure in Kubernetes and Spring Boot so a deploy loses no requests. Read The Twelve-Factor App first, especially Disposability.

1. Six ways to ship a new version

Key idea: strategies differ in three things: how fast you can roll back, how much extra capacity you pay for, and whether two versions run at the same time. Rolling is the sensible default, canary is for risky changes, blue-green is for when you need instant rollback.

Strategy How Rollback Extra cost Two versions at once
Recreate Stop all old pods, start new ones Redeploy the old version (with downtime) 0 No, but there is downtime
Rolling Replace pods in small batches kubectl rollout undo, also gradual A few pods (maxSurge) Yes, during the whole rollout
Blue-green Build a full new environment (green), flip the router once Flip back to blue: seconds Double during the switch Briefly, but the database is shared
Canary 1–5% of traffic to the new version, grow it based on metrics Set the weight back to 0 Small Yes, for a while
Shadow (mirroring) Copy real requests to the new version, discard its responses Not needed, users never see it Double load for mirrored traffic Yes
Feature flag New code is deployed but off, turned on per user group Turn the flag off: instant Near 0 Both code paths live in one binary

Recreate only fits when the two versions truly cannot run side by side. A Kubernetes Deployment supports recreate and rolling out of the box; blue-green and canary need a routing layer on top: switching a Service selector, a weighted ingress (the canary-weight annotation in ingress-nginx), a service mesh, or Argo Rollouts and Flagger.

flowchart LR
    A[Deploy canary at 0%] --> B[5% traffic]
    B --> C{Compare with old version:<br/>error rate, p99, SLO burn}
    C -->|good after N minutes| D[25% then 50%]
    D --> E{Still good?}
    E -->|yes| F[100%, remove old version]
    C -->|bad| R[Weight to 0, roll back]
    E -->|bad| R
Shadow traffic: where it can hurt

2. Deploy is not release: feature flags

Key idea: deploying puts a binary on machines; releasing lets users see a feature. A feature flag separates the two, so you can deploy continuously, turn a feature on per user group, and turn it off in seconds without redeploying.

A flag is an if that reads its value from a config service (Unleash, LaunchDarkly, or the open OpenFeature standard). Three kinds: release flags hide unfinished code (live for weeks), kill switches turn off heavy features under load (live long), experiment flags split users by a hash of the user id for A/B tests.

A canary splits by pod, a flag splits by user; they combine well: roll out with the flag off, then enable it for 1% of users.

Dark launches and flag debt

3. Expand/contract: changing the schema while two versions run

Key idea: during a rolling or canary deploy, versions N and N+1 read and write the same database. So every schema change is split across releases: expand (add, don’t break), migrate (move data and reads/writes), contract (remove the old part once nobody uses it).

Take renaming the column phone to phone_number. Do it in one ALTER TABLE ... RENAME COLUMN and every old pod still running fails immediately.

Release Schema Application Safe to roll back to the previous release?
1. Expand Add nullable phone_number Write both columns, still read phone Yes, the old version ignores the new column
2. Backfill Copy phone → phone_number in batches Same as release 1 Yes
3. Switch reads No change Read phone_number, still write both Yes, phone is still written
4. Stop old writes No change Write only phone_number Only back to release 3
5. Contract Drop phone No change No longer needed

The rule: migrations run before the new code and must work with the code version right before them. Backfill in small batches as when deleting old data, and an ALTER on a large table must be online DDL. APIs and messages work the same way: adding a field is fine, removing or changing its meaning needs a full cycle like the one above, much like Schema Registry compatibility modes.

Flyway scripts for each step and other common changes
-- V41__expand_phone_number.sql (release 1)
ALTER TABLE customer ADD COLUMN phone_number VARCHAR(20);

-- V42__backfill_phone_number.sql (release 2, run repeatedly in batches, or as a separate job)
UPDATE customer SET phone_number = phone
WHERE phone_number IS NULL AND id BETWEEN :from AND :to;

-- V45__contract_phone.sql (release 5, once no pod reads phone any more)
ALTER TABLE customer DROP COLUMN phone;

How to handle other changes:

Change Safe approach
Add a NOT NULL column Add it nullable (or with a DEFAULT), backfill, then add the constraint
Drop a column First release: code stops reading/writing it and it leaves the JPA entity. Next release: drop
Change a column type, split a table Like a rename: new thing, write both, backfill, switch reads, drop the old
Index a large table CREATE INDEX CONCURRENTLY (PostgreSQL) or ALGORITHM=INPLACE, LOCK=NONE (MySQL)

With JPA, an entity mapped to a column the database doesn’t have yet fails on the first SELECT (or ddl-auto=validate refuses to start), so the migration always goes first.

4. The Kubernetes a backend developer needs

Key idea: a Deployment keeps the pod count and drives the rollout; a Service groups the ready pods behind one stable address. Probes, rollout parameters, resources and the PodDisruptionBudget decide whether a deploy loses requests.

Probe Question On failure Should check
Startup Have I finished starting? Restart after failureThreshold × periodSeconds Same as liveness, with a long allowance
Readiness Should I get traffic? Removed from Service endpoints, not restarted Internal state, maybe an essential shared dependency
Liveness Can this process still recover? Container restarted Internal state only (deadlock, stuck)

The startup probe disables the other two until it succeeds, so an app that takes 60 seconds to start isn’t killed by liveness. Readiness keeps running for the pod’s whole life, not only at startup.

  • maxSurge (default 25%, rounded up): how many pods above replicas may exist. maxUnavailable (default 25%, rounded down): how many ready pods may be missing. maxSurge: 1, maxUnavailable: 0 never reduces capacity, at the price of a slower rollout.
  • Resources: requests is what the scheduler reserves, limits is the ceiling. Exceeding the CPU limit gets you throttled (higher p99); exceeding the memory limit gets you OOMKilled. See CPU quota and heap versus memory limit.
  • PodDisruptionBudget (PDB): limits how many pods may be taken down at once during voluntary disruptions such as a node drain, for example minAvailable: 2.
Full manifest: Deployment, Service and PDB for a Spring Boot service
apiVersion: apps/v1
kind: Deployment
metadata: { name: order-service }
spec:
  replicas: 4
  minReadySeconds: 10            # a pod must stay ready for 10s to count as available
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector: { matchLabels: { app: order-service } }
  template:
    metadata: { labels: { app: order-service } }
    spec:
      terminationGracePeriodSeconds: 45     # > preStop + timeout-per-shutdown-phase
      containers:
        - name: app
          image: registry.example.com/order-service:1.42.0   # pinned tag, never latest
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: "500m", memory: "768Mi" }
            limits:   { memory: "768Mi" }   # many teams drop the CPU limit to avoid throttling
          startupProbe:
            httpGet: { path: /livez, port: 8080 }
            periodSeconds: 5
            failureThreshold: 24            # allows up to 120s to start
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /livez, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              exec: { command: ["sh", "-c", "sleep 10"] }
---
apiVersion: v1
kind: Service
metadata: { name: order-service }
spec:
  selector: { app: order-service }
  ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: order-service }
spec:
  minAvailable: 3
  selector: { matchLabels: { app: order-service } }

A few details: progressDeadlineSeconds (default 600) marks a rollout as failed when it stops progressing, but Kubernetes does not roll back by itself; the pipeline must watch kubectl rollout status and call rollout undo. With requests == limits for every container the pod gets the Guaranteed QoS class and is evicted last when a node runs short of memory. This manifest was not applied to a real cluster while writing the post.

5. Spring Boot: health groups and graceful shutdown

Key idea: Actuator provides two health groups, liveness and readiness, for the probes. On SIGTERM, Spring Boot switches readiness to REFUSING_TRAFFIC, stops accepting new requests and waits for in-flight ones, bounded by spring.lifecycle.timeout-per-shutdown-phase.

# Probes on the main port, so they fail when the main web server is broken
management.endpoint.health.probes.add-additional-paths=true   # /livez, /readyz
# Readiness includes the DB because the pod is useless without it; liveness stays default
management.endpoint.health.group.readiness.include=readinessState,db
server.shutdown=graceful                         # the default since Spring Boot 3.4
spring.lifecycle.timeout-per-shutdown-phase=30s

Older Spring Boot versions only enable the two groups when they detect Kubernetes or when management.endpoint.health.probes.enabled=true is set. With Actuator on a separate port, the probes can stay green while the main port is dead, hence add-additional-paths.

SIGTERM and removing the pod from endpoints happen in parallel, so a preStop sleep of a few seconds is needed before the app stops accepting connections. That detail, and the shell-form ENTRYPOINT swallowing SIGTERM, are covered in The Twelve-Factor App and the operating systems post.

sequenceDiagram
    participant K as Kubernetes
    participant P as Pod (Spring Boot)
    participant L as Service / Ingress
    K->>L: Remove pod from endpoints (async)
    K->>P: Run preStop: sleep 10
    Note over L: Stops sending requests a few seconds later
    K->>P: SIGTERM
    P->>P: Readiness = REFUSING_TRAFFIC, reject new requests
    P->>P: Wait for in-flight requests (up to 30s)
    P-->>K: Exit
    Note over K,P: SIGKILL after terminationGracePeriodSeconds

6. Sessions and sticky sessions

Key idea: deploys kill pods, so state held in a pod’s memory is lost. Sticky sessions only reduce the symptom; move sessions out (Redis, JDBC) or use stateless tokens.

  • Sticky sessions (pinning a user to one pod with a cookie) skew load, and the user still loses the session when that pod is replaced.
Canaries, shared sessions and long-lived connections
  • A weighted canary can bounce one user between old and new versions; if the API behaviour changes, route the canary by user (header, cookie).
  • An external session must deserialize in both versions (Spring Session, sessions and JWT). Long-lived connections (WebSocket, gRPC streams) must be closed by the server on shutdown, with clients reconnecting with jitter.

7. Canary: what to watch and when to stop

Key idea: compare the canary with the old version over the same period, on signals users feel: error rate, p99 and how fast the error budget burns. Stop automatically when a pre-agreed threshold is crossed.

Signal Example stop threshold Note
5xx error rate Canary > old version + 0.5 percentage points Relative, since the baseline changes by hour
p99 latency Canary > 1.2 × old version The average hides the slow tail
SLO burn rate > 10 over 10 minutes Burn rate = error rate / (1 − SLO)
Saturation CPU throttling, connection waits, GC pauses Correct but more expensive
Business signal Orders per minute, payment success rate Catches “200 OK but wrong”

Burn rate says how many times faster than allowed the error budget is being spent: a 99.9% SLO allows 0.1% errors, so a canary at 0.5% errors has a burn rate of 5. Google’s SRE workbook alerts on a burn rate of 14.4 sustained for an hour, which has used 2% of a 30-day budget (see SLOs and error budgets).

8. Zero-downtime checklist

Key idea: miss one detail and you lose a few requests on every deploy.

  • Expand/contract migrations, run before the code, compatible with version N-1; APIs and messages only add fields.
  • Three separate probes; liveness never calls a dependency.
  • maxUnavailable: 0 or enough spare capacity; set minReadySeconds.
  • Exec-form ENTRYPOINT, graceful shutdown, a preStop sleep, a grace period longer than the whole shutdown.
  • No state in pod memory; a PDB for every service with 2+ pods.
  • Pinned image tags, a rollback command you have tried, numeric canary criteria.
  • Warm-up if cold pods raise p99 (JIT, connection pool, local caches).

Exercises

Exercise 1: Counting pods during a rolling update

A Deployment with replicas: 10 uses the defaults maxSurge: 25%, maxUnavailable: 25%. During a rollout, what is the maximum number of pods and the minimum number available? What changes with maxSurge: 1, maxUnavailable: 0, and what does it cost?

Solution to Exercise 1
  • 25% of 10 is 2.5. maxSurge rounds up to 3, maxUnavailable rounds down to 2.
  • At most 10 + 3 = 13 pods exist at once, and at least 10 − 2 = 8 are available. In the first step the controller creates 3 new pods and may remove 2 old ones right away.
  • With maxSurge: 1, maxUnavailable: 0: at most 11 pods, always 10 available. Only one pod is replaced at a time and each must become ready (plus minReadySeconds), so the rollout takes about ten startup cycles. If the cluster has no room for an 11th pod, the rollout stalls.
  • A service running at 80% CPU under the default settings makes 8 pods carry the load of 10, each at 100%. That’s when you pick maxUnavailable: 0.

Exercise 2: Splitting full_name

Table customer has full_name. A new requirement needs first_name and last_name, and the service runs a rolling deploy with 6 pods. Write the release plan, stating how far back each step can roll.

Solution to Exercise 2
  1. R1 expand: a migration adds nullable first_name, last_name. The code writes full_name and the two new columns (split by one agreed rule), still reads full_name. Rolling back to R0 is safe because R0 doesn’t know the new columns.
  2. R2 backfill: a batch job splits full_name for old rows and logs uncertain names (one word, extra spaces). Reads don’t change. Rollback is safe.
  3. R3 switch reads: the code reads first_name/last_name, still writes all three. Put it behind a feature flag to turn off quickly if names display wrongly. Rolling back to R2 is safe because full_name is still correct.
  4. R4 stop writing full_name: write only the new columns and drop full_name from the entity. From here you can’t go back to R2 (it would read stale full_name), only to R3.
  5. R5 contract: after an observation period (say a week), drop full_name. If anyone else (reports, ETL, CDC) still reads it, move them first.

Rows created during the backfill already have all three columns thanks to R1, so the backfill only needs WHERE first_name IS NULL.

Remember

  • Compare deployment strategies on rollback, cost and whether two versions run side by side.
  • Write an expand/contract plan for a column rename, and say how far back each step can roll.
  • Tell startup, readiness and liveness apart, and explain why liveness must not check the database.
  • Work out pod counts from maxSurge/maxUnavailable, and know what a PDB protects against.
  • Describe the pod shutdown sequence and the numeric signals that stop a canary.

Interview questions

Junior

What is the difference between a readiness probe and a liveness probe?

A failing readiness probe removes the pod from the Service but keeps it running; a failing liveness probe restarts the container.

Readiness answers “should you send me requests?”, and matters at startup, at shutdown and under temporary overload. Liveness answers “can I still recover?”, and should fail only when the process is truly stuck. A startup probe shields both during a slow start.

What is a rolling update, and how does Kubernetes avoid losing requests?

It replaces old pods with new ones gradually, sending traffic only to ready pods, within the maxSurge and maxUnavailable limits.

To truly lose nothing, the app must also shut down gracefully: receive SIGTERM, stop taking new requests, finish in-flight ones, plus a preStop sleep so the load balancer has time to drop the pod.

Mid

You need to rename a column that is in use. How do you deploy it with no downtime?

Expand/contract across several releases: add the new column, write both, backfill, switch reads, stop writing the old column, and only then drop it.

During a rolling deploy, old and new pods share one database, so the schema must suit both. Each migration must work with the code version right before it, and the drop happens only when you’ll never roll back to a version that reads the old column.

How does blue-green differ from canary, and when do you pick each?

Blue-green switches all traffic at once to a fully prepared new environment; canary sends a small share first and grows it based on metrics.

Blue-green suits cases that need instant rollback and can afford double capacity. Canary suits risky changes on high-traffic services, because a bug only reaches some users. Both share the database, so migrations still need to be compatible.

How do feature flags help with deployments, and what do they cost?

They separate deploy from release: code ships switched off, is enabled gradually per user, and can be turned off in seconds without a redeploy.

The cost: every flag doubles the code paths, both branches need tests, and old flags become technical debt. Release flags need an expiry date; kill switches live long and should be exercised regularly.

Senior

Every deploy shows a few dozen 502s on the dashboard. How do you investigate?

It is almost always a pod receiving traffic before it is ready, or being closed while still routed to; check both ends of the pod lifecycle.

  • Startup: does readiness reflect true readiness (pool, caches), is minReadySeconds set?
  • Shutdown: does the JVM get SIGTERM (exec-form ENTRYPOINT), is graceful shutdown on, is there a preStop sleep so the ingress can drop the pod, is the grace period long enough?
  • Keep-alive: the proxy holds connections to old pods, so its keep-alive timeout must be shorter than the app’s, or the app sends Connection: close while shutting down (see reverse proxy).
  • Line up the error timestamps with the pod’s Killing and Readiness probe failed events.
Design an automated canary for a payment service.

A weighted canary with steps of 5%, 25% and 50%, each with automated analysis comparing canary and old version, and automatic rollback when a threshold is crossed.

  • Signals: 5xx rate, p99, SLO burn rate, payment success rate and transactions per minute, all labelled by version.
  • Enough samples: hold a step until it has enough transactions, not just enough minutes.
  • Route by user so nobody switches versions in the middle of a checkout.
  • Schema and message changes go first via expand/contract. Tooling: Argo Rollouts or Flagger with Prometheus metrics.
What is a PodDisruptionBudget for, and what happens when it is misconfigured?

A PDB limits how many pods can be taken down at once during voluntary disruptions like a node drain, so the service always keeps enough pods.

It doesn’t apply to a Deployment rollout or to a node dying. Setting minAvailable equal to replicas, or a PDB on a single-pod Deployment, makes kubectl drain hang and blocks cluster upgrades. Use maxUnavailable: 1 or a minAvailable below the replica count.

Useful phrases for interviews

  • “During a rolling update, old and new versions share the database, so every migration has to be backward compatible.”
  • “We rename a column with expand and contract: add, dual-write, backfill, switch reads, then drop.”
  • “Readiness takes the pod out of the load balancer; liveness restarts it.”
  • “Liveness should never depend on the database, or a slow database restarts the whole fleet.”
  • “We deploy behind a feature flag, so release is a config change, not a deploy.”
  • “Blue-green buys instant rollback at the cost of double capacity.”
  • “The canary is compared with the baseline over the same window, and it rolls back automatically if the burn rate goes above ten.”
  • “A preStop sleep gives the ingress time to stop routing before the app closes its listener.”
  • “A disruption budget protects us during node drains, not during our own rollouts.”
  • “We can roll back code in a minute, but a dropped column is gone, so destructive migrations come last.”

Self-compiled notes from official documentation and hands-on practice.