A badly designed load test produces nice numbers and a real incident a few weeks later. This post goes from goals (SLOs) to the workload model, measuring latency correctly, tools, and reading the results to find the bottleneck. The coordinated omission section includes a small JDK 21 experiment, with real numbers from my machine.
1. Five kinds of test, each answering one question
Key idea: you pick the kind of test from the question you need answered, not from the tool. It’s the same script; only the shape of the load over time changes.
| Kind | Question | Load shape | What it catches |
|---|---|---|---|
| Smoke | Do the script and system work at all? | Tiny, 1–2 minutes | Broken script, broken environment |
| Load | Does expected traffic meet the SLO? | Ramp to peak, hold 15–60 minutes | Performance regressions, misconfiguration |
| Stress | How does it fail beyond peak, and does it recover? | Ramp past peak until errors | Failure mode, timeouts, retry storms |
| Spike | What if traffic jumps within seconds? | From low to 5–10× instantly | Slow autoscaling, cold pools, full queues |
| Soak (endurance) | Does it degrade over time? | About 70–80% of peak, 4–24 hours | Memory leaks, connection leaks, full disks |
| Capacity (breakpoint) | What is the most it can take within SLO? | Step up, hold each step a few minutes | Breaking point, the first resource to run out |
A stress test also answers does it fail gracefully: quick 503s and recovery afterwards are good; every request hanging until the service dies is bad (see timeouts and circuit breakers).
2. Goals start from the SLO
Key idea: a goal is a sentence with numbers and conditions, taken from the SLO and real traffic: arrival rate, request mix, latency percentiles, error rate. Without numbers there is no pass or fail.
- Rate: the real peak from production metrics (not the daily average) times a safety factor: 300 req/s × 1.5 = 450 req/s. Without data, estimate.
- Request mix: for example 70% listings, 20% details, 10% orders. Testing a single endpoint measures a different system.
- Thresholds: p95 and p99 per endpoint group, error rate, and resource limits (CPU < 70% at peak to leave room for failover).
Put the thresholds in the script so the tool fails the pipeline on its own. “Handles load well” is not a goal; “p99 < 300 ms at 450 req/s for 30 minutes, errors < 0.1%” is.
3. Workload models: open and closed
Key idea: a closed model has N virtual users; each sends, waits for the response, pauses (think time) and sends again, so a slow server automatically gets less load. An open model sends at a fixed arrival rate regardless of responses, like real Internet users. Most public APIs should be tested with an open model.
flowchart LR
subgraph Closed["Closed: fixed N users"]
U[Virtual user] -->|send| S1[Server]
S1 -->|response| U
U -->|think time| U
end
subgraph Open["Open: fixed arrival rate"]
A[Schedule: 500 req/s] -->|send on time| S2[Server]
S2 -->|response whenever it comes| X[Record latency]
end
In a closed model, throughput = N / (response time + think time). 100 users, 1 s think time and 0.1 s responses give about 91 req/s; if the server slows to 2 s, it drops to 33 req/s. The load falls exactly when it should be piling up.
4. Coordinated omission: systematically wrong measurements
Key idea: a tool that waits for a response before sending the next request doesn’t send while the server is frozen, so it misses exactly the requests that would have waited longest. The fix: measure latency from when the request should have been sent, or use an open model.
The name comes from Gil Tene: the tool “coordinates” with the server to skip the bad samples. The experiment: the JDK’s HttpServer, about 1 ms per request, freezing for 1 second every 5 seconds (simulating a GC pause or a long lock). The load generator sends 200 req/s for 20 seconds (4,000 requests) after a 3-second warm-up.
| Measurement (ms) | avg | p50 | p90 | p99 | p99.9 | max |
|---|---|---|---|---|---|---|
| Closed, from actual send | 2.76 | 1.72 | 2.11 | 2.63 | 1,002 | 1,007 |
| Closed, from intended send | 147.6 | 2.51 | 666.9 | 974.7 | 1,003 | 1,007 |
| Open (virtual threads), from intended send | 106.8 | 2.73 | 521.7 | 956.9 | 1,003 | 1,011 |
Machine: Apple M4 with 10 cores, macOS 27.2, JDK 21.0.13, client and server on the same machine; three runs (the last one with exactly the code in this post) differed by under 3%. The first method records only a handful of slow samples (one per stall, about 0.1% of all samples), so p99 looks perfect even though every 5 seconds users wait up to a second. The open model gives p90 ≈ 0.52 s, matching theory: the server is frozen about 20% of the time, and requests arriving then wait anywhere from 0 to 1 s, evenly spread.
The 148 ms average is meaningless too: no request looks like that; most take 2–3 ms, the rest hundreds of ms.
Experiment source code and commands
// StallServer.java: ~1 ms per request, freezes for 1 s every 5 s
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.util.concurrent.Executors;
public class StallServer {
static final Object LOCK = new Object();
public static void main(String[] args) throws Exception {
HttpServer s = HttpServer.create(new InetSocketAddress(8080), 1024);
s.createContext("/", ex -> {
synchronized (LOCK) { } // blocks during a stall
try { Thread.sleep(1); } catch (InterruptedException e) { }
byte[] body = "ok".getBytes();
ex.sendResponseHeaders(200, body.length);
ex.getResponseBody().write(body);
ex.close();
});
s.setExecutor(Executors.newFixedThreadPool(8));
s.start();
Thread.ofPlatform().daemon().start(() -> {
while (true) {
try { Thread.sleep(5000); synchronized (LOCK) { Thread.sleep(1000); } }
catch (InterruptedException e) { return; }
}
});
}
}
// LoadGen.java: java LoadGen closed|open <rps> <seconds>
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.LockSupport;
public class LoadGen {
public static void main(String[] a) throws Exception {
String mode = a[0]; int rps = Integer.parseInt(a[1]); int secs = Integer.parseInt(a[2]);
HttpClient c = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build();
HttpRequest req = HttpRequest.newBuilder(URI.create("http://localhost:8080/")).build();
long interval = 1_000_000_000L / rps;
long wEnd = System.nanoTime() + 3_000_000_000L; // 3 s warm-up
while (System.nanoTime() < wEnd) c.send(req, HttpResponse.BodyHandlers.discarding());
long total = (long) rps * secs, start = System.nanoTime();
if (mode.equals("closed")) {
List<Long> naive = new ArrayList<>(), corrected = new ArrayList<>();
for (long i = 0; i < total; i++) {
long intended = start + i * interval;
long now;
while ((now = System.nanoTime()) < intended) LockSupport.parkNanos(intended - now);
long sent = System.nanoTime();
c.send(req, HttpResponse.BodyHandlers.discarding()); // waits for the response: closed
long done = System.nanoTime();
naive.add(done - sent); // suffers from coordinated omission
corrected.add(done - intended); // includes the time it should already have been sent
}
report("closed, from actual send", naive);
report("closed, from intended send", corrected);
} else {
Queue<Long> lat = new ConcurrentLinkedQueue<>();
CountDownLatch done = new CountDownLatch((int) total);
for (long i = 0; i < total; i++) {
long intended = start + i * interval;
long now;
while ((now = System.nanoTime()) < intended) LockSupport.parkNanos(intended - now);
Thread.startVirtualThread(() -> { // doesn't wait: open
try { c.send(req, HttpResponse.BodyHandlers.discarding()); } catch (Exception e) { }
lat.add(System.nanoTime() - intended); done.countDown();
});
}
done.await();
report("open, from intended send", new ArrayList<>(lat));
}
}
static void report(String name, List<Long> ns) {
long[] v = ns.stream().mapToLong(Long::longValue).sorted().toArray();
System.out.printf("%-28s n=%d avg=%.2f p50=%.2f p90=%.2f p99=%.2f p99.9=%.2f max=%.2f ms%n",
name, v.length, Arrays.stream(v).average().orElse(0) / 1e6,
p(v, 50), p(v, 90), p(v, 99), p(v, 99.9), v[v.length - 1] / 1e6);
}
static double p(long[] v, double q) { return v[(int) Math.ceil(q / 100 * v.length) - 1] / 1e6; }
}
$ javac -d out StallServer.java LoadGen.java && java -cp out StallServer &
$ java -cp out LoadGen closed 200 20
closed, from actual send n=4000 avg=2.76 p50=1.72 p90=2.11 p99=2.63 p99.9=1002.16 max=1006.61 ms
closed, from intended send n=4000 avg=147.62 p50=2.51 p90=666.90 p99=974.67 p99.9=1003.20 max=1007.32 ms
$ java -cp out LoadGen open 200 20
open, from intended send n=4000 avg=106.82 p50=2.73 p90=521.70 p99=956.94 p99.9=1003.32 max=1010.65 ms
The corrected closed run has a higher p90 than the open run (667 versus 522 ms) because after each stall a single thread must catch up on about 200 requests back to back, ~1.7 ms each, so the catch-up itself queues. The open run catches up in parallel, with 8 server threads working at once.
5. What to measure: throughput, percentiles, errors and saturation
Key idea: four numbers per load step: achieved throughput, p50/p95/p99, error rate, and how saturated each resource is. RED describes the service from the user’s side, USE describes each resource inside.
- RED (per endpoint): Rate (req/s), Errors (including client-side timeouts), Duration (the latency distribution).
- USE (per resource): Utilization (% busy), Saturation (how much work is waiting: run queue, requests waiting for a connection, threads waiting for a lock), Errors.
- Saturation warns earlier than utilization. For a simple queue (M/M/1), response time ≈ service time / (1 − utilization): 5× at 80% busy, 10× at 90%. Latency shoots up at the “knee” before CPU hits 100%.
Little’s law as a cross-check
6. Tools: k6, Gatling, JMeter
Key idea: k6 (JavaScript scripts) is compact for APIs and CI; Gatling (Java/Kotlin/Scala DSL) suits Java teams and has good HTML reports; JMeter (GUI, XML) covers many protocols but defaults to a closed model.
// k6: open model at 450 req/s for 30 minutes, fails if the SLO is missed
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate', // open model
rate: 450, timeUnit: '1s', duration: '30m',
preAllocatedVUs: 200, maxVUs: 1000, // too few VUs and iterations are dropped (dropped_iterations)
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
'http_req_duration{name:listProducts}': ['p(95)<150', 'p(99)<300'],
dropped_iterations: ['count<1'], // the target load was really generated
},
};
export default function () {
const page = Math.floor(Math.random() * 50);
const res = http.get(`${__ENV.BASE_URL}/products?page=${page}`, { tags: { name: 'listProducts' } });
check(res, { 'status 200': (r) => r.status === 200 });
}
Tool comparison and a Gatling Java DSL sketch
| Tool | Scripts in | Open model | Strengths | Weaknesses |
|---|---|---|---|---|
| k6 | JavaScript | constant-arrival-rate, ramping-arrival-rate |
Compact, thresholds as pass/fail, CI-friendly | Can’t use Java libraries in scripts |
| Gatling | Java, Kotlin, Scala DSL | injectOpen (and injectClosed) |
Same language as the team, nice reports, resource-efficient | A DSL to learn, built with Maven/Gradle |
| JMeter | GUI, saved as XML | Open Model Thread Group (since 5.5) | Many protocols (JDBC, JMS), large community | XML is hard to review, resource-hungry |
import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;
import io.gatling.javaapi.core.*;
import io.gatling.javaapi.http.*;
import java.time.Duration;
public class ProductSimulation extends Simulation {
HttpProtocolBuilder httpProtocol = http
.baseUrl(System.getProperty("baseUrl", "http://localhost:8080"))
.acceptHeader("application/json");
FeederBuilder<String> users = csv("users.csv").random(); // realistic data, not one repeated id
ScenarioBuilder browse = scenario("browse then view")
.feed(users)
.exec(http("listProducts").get("/products?page=#{page}").check(status().is(200)))
.pause(Duration.ofSeconds(1), Duration.ofSeconds(3)) // think time within a session
.exec(http("productDetail").get("/products/#{productId}").check(status().is(200)));
{
setUp(browse.injectOpen( // open model: sessions arrive at a rate
rampUsersPerSec(10).to(200).during(Duration.ofMinutes(2)),
constantUsersPerSec(200).during(Duration.ofMinutes(30))))
.protocols(httpProtocol)
.assertions(
global().responseTime().percentile(99.0).lt(300),
global().failedRequests().percent().lt(0.1));
}
}
injectOpen controls the arrival rate of sessions; inside a session, requests still follow one another like a real user. This script was not run while writing the post (no Gatling project was set up); the API was checked against the Gatling docs.
7. Environment, data and warm-up
Key idea: results only hold for an environment that matches production in configuration, data size and access pattern. Small or repetitive data fits entirely in cache and produces fake good numbers.
- Environment: same machine type, CPU/memory limits, pod count and pool settings as production (see the JVM in containers). Third parties (payments, SMS) are replaced by stubs with realistic latency.
- Data: a production-sized database, a realistic distribution (a few customers with huge order histories), random ids from a feeder instead of one fixed id.
- Warm-up: the JVM is slow at first (interpreted, then compiled gradually by the C1 and C2 JITs), and caches and pools are cold. Drop the first few minutes from the results, but measure cold start separately: that’s when new pods take traffic during a deploy or autoscaling.
8. Finding the bottleneck
Key idea: raise the load step by step and find the first resource to saturate: the one whose queue grows when throughput stops growing. Each symptom has a tool that confirms it.
| Symptom | Suspect | Confirm with | Read more |
|---|---|---|---|
Low CPU, rising latency, threads waiting in getConnection |
DB connection pool exhausted | hikaricp.connections.pending, acquire time |
Pool sizing |
| Latency creeping up in a soak, with sudden jumps | GC, memory leak | GC log, JFR, heap after each full GC | The JVM in containers |
| Throughput flat, thread pool full | Too few threads, blocking calls | Thread dump (jcmd <pid> Thread.print) |
Thread pools and virtual threads |
Many threads BLOCKED on one monitor |
Lock in the application | Thread dump, JFR lock profiling | Race conditions and deadlocks |
| High DB CPU or lock waits | Slow SQL, N+1, row locks | EXPLAIN ANALYZE, slow log, data_locks |
EXPLAIN ANALYZE, InnoDB locks |
| Low average CPU but high p99 | Throttled by the CPU limit | nr_throttled in the cgroup |
CPU quota |
| No idea which service is slow | One hop in the call chain | Traces of slow requests | Distributed tracing |
Fix one thing at a time and measure after each change; a bottleneck that moves elsewhere is progress.
9. Reporting results, and JMH
Key idea: a report must let someone else rerun and compare: environment, workload model, numbers per step, the first resource to saturate, and the verdict against the SLO. JMH answers a different question: which piece of code is faster, inside one JVM.
A results report template
- Goal and thresholds (from the SLO), and a one-sentence pass/fail verdict.
- Environment: versions, configuration, pod count, data size, where the load generator ran.
- Workload model: open or closed, request mix, warm-up and hold durations.
- A table per step: achieved req/s, p50/p95/p99/max, error rate, CPU, pools, GC.
- The bottleneck, evidence (charts, thread dumps), recommendations, and comparison with the previous run.
A JMH skeleton and common mistakes
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(2) // fresh JVMs, no interference between benchmarks
@State(Scope.Benchmark)
public class ParseBenchmark {
String input = "2026-11-18T10:15:30Z";
@Benchmark
public Instant parse() { // return the result so the JIT can't remove the code
return Instant.parse(input);
}
@Benchmark
public void parseIntoBlackhole(Blackhole bh) {
bh.consume(Instant.parse(input));
}
}
Common mistakes: timing a loop yourself with System.nanoTime() (warm-up and dead code elimination bite), a constant input the JIT folds away (constant folding), a single fork. This is a reference skeleton and was not run while writing the post.
Exercises
Exercise 1: A closed model lowers its own load
A closed-model tool runs 50 virtual users with 1 s think time. Responses normally take 200 ms. During an incident, they take 2 s. What throughput does the tool generate before and during the incident? What would real production traffic (open model, 40 req/s) see during that incident?
Solution to Exercise 1
- Before: throughput = N / (R + Z) = 50 / (0.2 + 1) ≈ 41.7 req/s.
- During: 50 / (2 + 1) ≈ 16.7 req/s. The tool cuts its load by 60% exactly when the server is weak, so the server may “recover” and the report looks comfortable.
- Production: users keep arriving at 40 req/s. If the server is slow because it’s overloaded, it can’t keep up with 40 req/s, the queue grows without end, latency rises until timeouts, and client retries add even more load. If the server could hold 2 s, Little’s law gives 40 × 2 = 80 requests in flight, more than the 50 the tool generates: the test measured a lighter load than reality.
- Lesson: to learn how the system behaves under overload, use an open model with a fixed
rate, and watch how many requests are waiting.
Exercise 2: Write a load generator without coordinated omission
Write a Java 21 load generator that sends exactly 200 req/s and records correct latency even while the server is frozen. Run it against a server that simulates stalls and compare with the naive measurement.
Solution to Exercise 2
The solution is LoadGen from section 4. Three key points:
- Each request has an intended time
start + i × interval, computed up front, independent of whether the previous request has finished. - Latency = response time − intended time (not the actual send time).
- To keep the schedule while the server is slow, each request runs on its own virtual thread; requests in flight during a stall ≈ 200 req/s × 1 s = 200 (Little’s law), which virtual threads handle easily.
The measured results are in the section 4 table: the naive p99 = 2.63 ms, the correct p99 ≈ 957–975 ms. A production version should record into HdrHistogram (which has recordValueWithExpectedInterval to correct coordinated omission) instead of a list of Long.
Remember
- Say which question each kind of test (load, stress, spike, soak, capacity) answers.
- Write a numeric goal from the SLO and turn it into thresholds that fail the test.
- Explain why a closed model hides overload.
- Explain coordinated omission with a numeric example, and how to fix it.
- Go from a symptom to the suspected resource and the tool that confirms it.
Interview questions
Junior
How do load, stress and soak tests differ?
A load test checks that expected traffic meets the SLO, a stress test pushes past it to see how the system breaks, and a soak test holds moderate load for hours to find problems that accumulate.
A spike test adds the sudden-jump case, and a capacity test finds the highest load that still meets the SLO. It’s the same script with a different load shape over time.
Why report p99 instead of average latency?
The average hides the slow tail; p99 tells you how long the slowest 1% of requests wait.
In this post’s experiment, the 148 ms average matched no request: most took 2–3 ms, the rest hundreds of ms. A page that calls many APIs hits the slow tail far more often than 1% of the time.
Mid
What is the difference between open and closed workload models, and which do you choose?
A closed model has a fixed number of users who wait for a response before sending again, so load drops when the server slows; an open model sends at a fixed arrival rate regardless of responses.
Public APIs should be tested with an open model because users don’t know the server is slow. A closed model fits systems with a bounded number of clients, like workers or batch jobs. In a closed model, throughput = N / (response time + think time).
What is coordinated omission?
A measurement error where the tool waits for a response before sending the next request: while the server is frozen, the tool stops sending, so it records one slow sample instead of hundreds.
In the experiment, a server freezing 1 s every 5 s gave a naive p99 of 2.6 ms and a p99 of 975 ms measured from the intended send time. Fix it with an open model, by measuring from the intended time, or with tools that correct for it such as wrk2 and HdrHistogram.
How do you set goals for a load test?
From the SLO and production traffic: peak rate times a safety factor, the real request mix, p95/p99 and error-rate thresholds per endpoint, plus resource thresholds.
Write them all into the script as thresholds so the test fails itself and can run in CI. Without numbers there is no conclusion.
Senior
Throughput stops at 800 req/s while the service’s CPU is at 40%. How do you investigate?
Low CPU with flat throughput means something is waiting: find the resource whose queue is growing.
- Check the load generator first: its CPU,
dropped_iterations. - Pools: requests waiting for a DB connection (
hikaricp.connections.pending), the HTTP client pool, a full server thread pool. - Locks: thread dumps with many threads
BLOCKEDon one monitor, lock waits in the database. - Downstream: the database or a slow dependency (look at traces), CPU throttling from limits.
- Little’s law: 800 req/s × latency gives requests in flight; compare that with each pool’s size to see which one is full.
You have the p99 of 10 pods. What is the p99 of the service?
You can’t compute it from those ten numbers: percentiles can’t be averaged or added. Merge histograms, then compute.
Use histogram metrics (bucket counts), sum the buckets across pods and apply histogram_quantile. The average of p99s usually understates the real p99 when one pod is much slower than the rest.
When do you use JMH and when a load test?
JMH compares two implementations of a piece of code inside one JVM; a load test checks whether the whole system meets the SLO under concurrent load.
JMH handles warm-up, forking and dead code elimination, but can’t see the database, the network or locks between requests. Profile under load first to learn which method is worth optimising, then use JMH; otherwise a 30% gain on a method that is 2% of request time won’t change p99.
Useful phrases for interviews
- “We test public APIs with an open model, because users keep arriving even when the server slows down.”
- “A closed-model test quietly lowers the load exactly when the system is struggling.”
- “Our old numbers suffered from coordinated omission: p99 was two milliseconds while users waited a full second.”
- “The goal comes straight from the SLO: four hundred fifty requests per second with p99 under three hundred milliseconds.”
- “Percentiles can’t be averaged across pods; we aggregate histograms instead.”
- “RED tells us how the service looks from outside; USE tells us which resource is saturated.”
- “Throughput plateaued while CPU sat at forty percent, so we went looking for a queue.”
- “We discard the warm-up minutes, but we measure cold start separately because that’s what new pods see.”
- “JMH is for comparing two implementations; it says nothing about the system under load.”
- “The bottleneck moved from the connection pool to the database, which is progress.”
Self-compiled notes from official documentation and hands-on practice.