Third-party integration series: 1. Choosing a provider, webhooks and an anti-corruption layer · 2. Timeouts, retries, circuit breakers and bulkheads
A third party getting ten times slower at peak hour is enough to take down an unprepared system: threads get stuck, retries at every layer multiply the load, and even unrelated APIs stop responding. This post is the “defensive” half of calling out: timeouts, retries, circuit breakers, bulkheads and fallbacks, what each one solves and how to configure it in Spring Boot 3. Choosing a provider, the integration style and webhooks are in part 1.
1. Timeouts: the first thing you need
Key idea: without a timeout, a call to a hung service holds its thread forever. The thread pool drains and the whole service stops responding, unrelated APIs included.
Tomcat has 200 threads by default; when a third party hangs, they get held one by one until none are left. Set three timeouts separately:
| Timeout | What it bounds | Suggested value |
|---|---|---|
| Connect timeout | establishing the TCP connection (and the TLS handshake) | 1–2 s: if you can’t connect in 2 seconds, you usually can’t connect |
| Read timeout (or a whole-request timeout) | waiting for the response | the provider’s real p99 plus a margin, not “30 seconds to be safe” |
| Deadline (overall operation timeout) | every call, retries included | within your own API’s promise, e.g. 3 s |
How a connect timeout, a refused connection and a reset differ at the TCP level is in Networking for backend engineers.
Between internal services, gRPC has deadlines built in and propagates them through every hop, so downstream services know how much time is left; see Communication in system design: REST, gRPC, HTTP/2.
HttpClient http = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(1))
.build();
HttpRequest charge = HttpRequest.newBuilder(URI.create(base + "/charge"))
.timeout(Duration.ofSeconds(2)) // timeout for this request
.header("Idempotency-Key", "order-42-attempt-1")
.POST(HttpRequest.BodyPublishers.ofString("{\"amount\":150000}"))
.build();
// Calling a 3-second endpoint with a 500 ms timeout: HttpTimeoutException after ~501 ms (verified, JDK 21)
With Spring 6.1+ and RestClient, pass the timeouts through the request factory (a sketch, not run):
@Bean
RestClient paymentClient(RestClient.Builder builder) {
var jdk = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(1)).build();
var factory = new JdkClientHttpRequestFactory(jdk);
factory.setReadTimeout(Duration.ofSeconds(3));
return builder.baseUrl("https://pay.example.com").requestFactory(factory).build();
}
2. Retries: only when safe, with backoff and jitter
Key idea: retries rescue transient failures but can also turn a small incident into a big one. Retry only when the error is worth it, resending is safe, and you wait between attempts.
A transient failure is one that clears on its own within seconds (a network blip, a 503 while the other side deploys). Three questions before retrying:
- Is this error worth retrying? Yes: timeouts, connection errors, 429, 502, 503, 504. No: 400, 401, 403, 404, 422 (sending the same thing again fails the same way). 500 depends: many APIs return 500 for data errors. On a 429, wait for the
Retry-Afterheader if there is one; how the server side produces a 429 is in Rate limiting: five algorithms and a token bucket on Redis. - Is it safe to resend? GET is safe. A POST that creates a transaction is only safe if the other side supports an idempotency key: the same key returns the previous result instead of creating a second transaction. Without one, a timeout on a money transfer is an “unknown” state: query the status before resending. Designing keys is covered in the idempotency post.
- How long to wait between attempts? Exponential backoff (the wait doubles after each attempt) with a cap, plus jitter (a random wait) so thousands of clients don’t retry at the same instant.
On your own side, deploy so callers never see those 503s: readiness, graceful shutdown and rolling updates are in Deployment strategies.
static Result callWithRetry(HttpClient http, HttpRequest req, int maxAttempts,
Duration base, Duration cap) throws Exception {
for (int attempt = 1; ; attempt++) {
try {
HttpResponse<String> r = http.send(req, HttpResponse.BodyHandlers.ofString());
boolean retryable = r.statusCode() == 429 || r.statusCode() >= 500;
if (!retryable || attempt == maxAttempts) return new Result(r.statusCode(), r.body());
} catch (HttpTimeoutException e) {
if (attempt == maxAttempts) throw e;
}
long ceiling = Math.min(cap.toMillis(), base.toMillis() * (1L << (attempt - 1)));
Thread.sleep(ThreadLocalRandom.current().nextLong(ceiling + 1)); // full jitter
}
}
Test run of the retry loop and what production needs on top
The Java code in this post (jittered retries, a minimal circuit breaker) has been run on JDK 21 against a fake server built on com.sun.net.httpserver.
The fake server returns 503 for the first two calls:
- Attempt 1 fails and sleeps 60 ms (cap 100).
- Attempt 2 fails and sleeps 133 ms (cap 200).
- Attempt 3 succeeds. The server saw the same idempotency key on all three calls.
This is a minimal version to show the mechanics. In production add: honouring the Retry-After header on 429/503, an overall deadline, and a retry-count metric.
Whether to retry inside the request or push to a queue depends on whether the user needs the answer now.
Synchronous retries or a queue
3. Circuit breakers and bulkheads
Key idea: once a third party is down, a circuit breaker stops calling it and rejects at once (fail fast) instead of waiting for timeouts. A bulkhead caps concurrent calls per provider, so one slow provider can’t eat the shared resources.
Retrying a dead service only holds more threads and keeps it from recovering, because it keeps getting hammered. A circuit breaker tracks the failure rate through three states:
- CLOSED: calls go through and failures are counted.
- OPEN: past the threshold, every call is rejected immediately for a while.
- HALF_OPEN: when the wait is over, a few trial calls are allowed; success closes the circuit, failure opens it again.
In microservices, every call between services is a “third party” in this sense: one slow service holds threads all the way up the chain; see Microservices: when to split and finding service boundaries.
stateDiagram-v2
[*] --> CLOSED
CLOSED --> OPEN: failure rate above threshold
OPEN --> HALF_OPEN: wait time elapsed
HALF_OPEN --> CLOSED: trial calls succeed
HALF_OPEN --> OPEN: trial call fails
A minimal circuit breaker in Java (verified)
static final class Breaker {
enum State { CLOSED, OPEN, HALF_OPEN }
private State state = State.CLOSED; private int failures; private Instant openedAt;
private final int threshold; private final Duration openFor;
Breaker(int threshold, Duration openFor) { this.threshold = threshold; this.openFor = openFor; }
synchronized boolean allow() {
if (state == State.OPEN && Instant.now().isAfter(openedAt.plus(openFor))) state = State.HALF_OPEN;
return state != State.OPEN;
}
synchronized void success() { failures = 0; state = State.CLOSED; }
synchronized void failure() {
if (state == State.HALF_OPEN || ++failures >= threshold) { state = State.OPEN; openedAt = Instant.now(); }
}
}
// Threshold 3: failures 1 and 2 stay CLOSED, failure 3 goes OPEN, calls 4 and 5 are rejected at once;
// after the wait it's HALF_OPEN, and one successful trial call closes it (verified)
A bulkhead (named after the sealed compartments in a ship’s hull) caps concurrent calls to each third party, with a semaphore or a dedicated pool. A slow SMS provider can then hold at most 20 threads without dragging the payment flow down. With virtual threads (Java 21), threads are no longer scarce, but bulkheads still protect connection pools, memory and the third party itself. The DB connection pool and HTTP client pools (Apache HttpClient defaults to 5 connections per host) that a bulkhead protects are covered in Connection pooling and HikariCP.
In Spring Boot 3, use Resilience4j rather than writing your own.
Resilience4j configuration in Spring Boot 3 and the order of the wrappers
The configuration below is a sketch and wasn’t run.
resilience4j:
circuitbreaker:
instances:
sms:
sliding-window-size: 50
failure-rate-threshold: 50 # percent
slow-call-duration-threshold: 2s
slow-call-rate-threshold: 80
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 5
retry:
instances:
sms:
max-attempts: 3
wait-duration: 200ms
enable-exponential-backoff: true
enable-randomized-wait: true
bulkhead:
instances:
sms:
max-concurrent-calls: 20
@CircuitBreaker(name = "sms", fallbackMethod = "sendViaBackupProvider")
@Retry(name = "sms")
@Bulkhead(name = "sms")
public SmsResult send(OtpMessage m) { return primary.send(m); }
SmsResult sendViaBackupProvider(OtpMessage m, Throwable cause) { return backup.send(m); }
Exercises
Exercise 1: Review a client that calls an SMS provider
An OTP service uses an unconfigured new RestTemplate(); on failure a for loop resends up to 5 times immediately; the gateway in front also retries 3 times; the SMS provider doesn’t support idempotency keys. Find the problems and propose fixes.
Solution to exercise 1
- The default
RestTemplatehas no timeouts: one hang at the provider drains the threads. Set connect to 1 second and read to their p99 (say 3 seconds). - Immediate retries with no backoff or jitter pile load on them while they’re weak. Use capped backoff with jitter, at most 2–3 attempts.
- Retries at two layers: 3 × 5 = 15 calls for one OTP. Drop the gateway retry for this endpoint.
- With no idempotency key, a timed-out call may already have sent the message, so users get several OTPs. For OTPs, a duplicate is less harmful than nothing, but cap it and retry only on connection errors (definitely not sent), not on read timeouts.
- Add a circuit breaker with a backup provider as the fallback, a dedicated bulkhead for SMS, and a delivery-success-rate metric.
Remember
- Name the three timeouts, what each one is based on, and why Java’s default clients are dangerous.
- Know which errors to retry, which not, and when resending a POST is safe.
- Explain why jitter is needed and why retrying at several layers causes a retry storm.
- Draw the three circuit breaker states, and say what a bulkhead protects once you have virtual threads.
- Pick a fallback for payments, exchange rates, SMS and invoicing.
Interview questions
Junior
Why must every outbound call have a timeout?
Because without one, a hung service holds threads forever, and a drained thread pool takes the whole service down.
The server’s thread pool is finite (Tomcat defaults to 200), so a few minutes later even unrelated APIs stop responding. Set a short connect timeout (1–2 s), a read timeout based on the other side’s p99, and an overall deadline that covers retries.
Which errors should you retry, and which not?
Retry transient errors (timeouts, connection errors, 429, 502, 503, 504); don’t retry errors caused by the request itself (400, 401, 403, 404, 422).
Honour Retry-After if present. Errors caused by the request fail the same way when resent. And only retry writes when they’re guaranteed idempotent, usually thanks to an idempotency key.
Mid
What are exponential backoff and jitter, and why do you need jitter?
Backoff grows the wait exponentially up to a cap; jitter randomizes the wait so clients don’t retry at the same moment.
Backoff (100 ms, 200, 400…) gives the other side time to recover. Without jitter, thousands of clients that failed together retry together, producing synchronized waves of load against an already weak service. Full jitter (random between zero and the cap) spreads load best.
How does a circuit breaker work?
Three states: CLOSED counts failures, OPEN rejects at once once the threshold is crossed, HALF_OPEN lets a few trial calls decide whether to close again or stay open.
CLOSED usually counts failures as a rate over a sliding window, slow calls included. OPEN lasts for a set time, and every call goes straight to the fallback. The benefit: you fail fast instead of waiting for timeouts, and the other side gets time to recover.
What is a bulkhead? Do you still need one with virtual threads?
A bulkhead caps concurrent calls to each dependency; you still need it with virtual threads, because resources other than threads are still finite.
It’s a semaphore or a dedicated pool, so a slow provider only consumes its own share instead of the shared pool. Virtual threads make threads cheap, but the HTTP connection pool, memory for waiting requests, and the third party’s quota and capacity are still limited. These days it’s usually a semaphore rather than a dedicated thread pool.
Senior
A provider gets 10× slower and takes your whole system down. Analyse and fix.
It’s a chain: long timeouts hold resources, multi-layer retries multiply load, and the failure spreads to everything. Fix it with timeouts, single-layer retries, a circuit breaker, bulkheads and fallbacks.
The usual chain: no timeout or one that’s too long, threads and connection pools held, retries at several layers multiplying load, user requests queuing up, health checks slowing too, instances restarted, load shifting onto the survivors.
Fix:
- Timeouts from p99 plus an overall deadline.
- Retries at one layer with jitter and a retry budget.
- A circuit breaker that also counts slow calls; a dedicated bulkhead for that provider.
- Business-driven fallbacks; move calls that don’t need to be synchronous onto a queue.
- Per-provider latency metrics and alerts.
A call to create a transaction times out. Do you resend it?
Only with an idempotency key; otherwise query the status first, because a timeout means “unknown” and the other side may already have created it.
With a key, resend with the same key and they’ll return the earlier result instead of creating a new one. Without one, don’t resend blindly: query the status by your own reference, mark the record UNKNOWN and let the reconciliation job settle it. A connection failure before the request was sent (connect timeout) is much safer to retry than a read timeout.
Useful phrases for interviews
- “Every outbound client has an explicit connect timeout, a read timeout based on the provider’s p99, and an overall deadline.”
- “We only retry transient errors, and only writes that carry an idempotency key.”
- “Retries use capped exponential backoff with full jitter, so clients don’t retry in lockstep.”
- “We retry at exactly one layer; everything above it fails fast.”
- “Once the breaker opens, we fail fast and go to the fallback instead of waiting for timeouts.”
- “Each provider gets its own bulkhead, so a slow SMS vendor can’t starve the payment flow.”
- “A timeout on a payment call is an unknown state, not a failure; we query before we resend.”
- “The fallback depends on the business: refuse for payments, serve stale data for rates, switch providers for SMS.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.