Advanced REST API series: 1. Idempotent methods, URLs and pagination · 2. Async jobs, Idempotency-Key and API docs
This post solves two things a plain “request in, response out” API can’t: an operation that takes minutes (exporting a 500 MB file) and an operation that must never run twice (a retried payment). Then it covers a response envelope with consistent error codes and how to document an API that many teams share. Afterwards you can design an async job and a complete Idempotency-Key.
Read first: part 1 (safe vs idempotent, pagination) and Web and HTTP (status codes such as 202, 303, 409, 410).
1. Case study: exporting a 500 MB file
Key idea: don’t run a heavy export inside one synchronous request. Split it into “create a job” and “fetch the result”, let a worker stream the data, and serve the file straight from object storage.
1.1. The problem
Key idea: a synchronous API for minutes of work hits timeouts, runs out of memory, ties up resources and can’t resume after a network drop.
An admin clicks “Export to Excel” on a product list; for a big customer the file is about 500 MB and takes minutes to build. The naive design: query, write the file, return it in the same request.
- Timeouts: Nginx’s
proxy_read_timeoutdefaults to 60 seconds, the AWS ALB idle timeout to 60 seconds, Cloudflare gives about 100 seconds. The client gets a504but the server keeps going; the user clicks again and now two heavy jobs run. - OOM (out of memory, the heap runs out):
findAll()loads the whole result into aList, JPA’s persistence context holds every entity, the Excel library keeps the whole workbook. 500 MB of data turns into several GB of heap. - Tied-up resources: a servlet thread and a database connection are held for minutes; a few concurrent exports drain the pool.
- No way to resume: a network blip at minute 4 means starting over.
1.2. The design: an async job
Key idea: POST creates the job and returns 202 Accepted + Location, a GET checks status, and GET /result redirects to a presigned URL (a pre-signed link that expires after a few minutes).
POST /v1/exportswith body{"type": "PRODUCTS", "filter": {"name": "pen"}}→202 Accepted, headerLocation: /v1/jobs/001, body{"job_id": "001", "status": "PENDING", "issued_at": 1763443200000}.GET /v1/jobs/001→{"status": "PROCESSING", "progress": 42}, later{"status": "DONE", "updated_at": ...}or{"status": "FAILED", "error": {...}}.GET /v1/jobs/001/result→302/303to a presigned URL on object storage (S3, MinIO, GCS), valid for say 15 minutes. The file never flows through the app servers.
The worker reads and writes as a stream (handle a row, then drop it, never collect everything in RAM), so memory stays constant however big the file is.
Streaming export worker code and per-database details
@Transactional(readOnly = true) // PostgreSQL: the driver only streams with autocommit off
public void runExport(ExportJob job, OutputStream out) {
jdbcTemplate.setFetchSize(1_000); // fetch 1000 rows per round trip, not the whole result
try (var writer = new BufferedWriter(new OutputStreamWriter(out, StandardCharsets.UTF_8))) {
jdbcTemplate.query(
"SELECT id, name, price FROM product WHERE name ILIKE ? ORDER BY id",
rs -> { // RowCallbackHandler: row by row, no List
writer.write(rs.getLong(1) + "," + csv(rs.getString(2)) + "," + rs.getBigDecimal(3));
writer.newLine();
},
"%" + job.filterName() + "%");
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
// out = a streaming multipart upload to object storage, not a giant temp file on the pod's disk
- PostgreSQL: the driver only uses a server-side cursor when
autoCommit=falseandfetchSize > 0. - MySQL Connector/J: needs
useCursorFetch=true(orfetchSize = Integer.MIN_VALUEto stream row by row). - Spring Data
Stream<T>: must run inside a transaction anddetacheach entity after use, otherwise the persistence context keeps growing. - Excel: use a streaming API (Apache POI’s
SXSSFWorkbook) or export CSV.
1.3. Polling, webhooks or real-time push
Key idea: polling (the client asks periodically) is the simplest; a webhook (the server calls the client’s URL) is efficient but complex; SSE/WebSocket suits UIs that need live progress. Whatever you choose, keep the status API.
sequenceDiagram
participant C as Client
participant A as API
participant W as Worker
participant S as Object storage
C->>A: POST /v1/exports
A-->>C: 202 Accepted, Location /v1/jobs/001
A->>W: enqueue job
loop Polling with backoff
C->>A: GET /v1/jobs/001
A-->>C: PROCESSING, Retry-After 5
end
W->>S: multipart upload
W->>A: mark job DONE
C->>A: GET /v1/jobs/001
A-->>C: DONE
C->>A: GET /v1/jobs/001/result
A-->>C: 303 to presigned URL
C->>S: download the file directly
With a webhook, the client supplies a callback_url when creating the job (or registers one up front). When done, the server calls POST callback_url with {"job_id": "001", "status": "DONE"}. In practice you combine them: a webhook to notify, the status API as the fallback when a webhook is lost.
Comparison table: polling, webhooks and SSE/WebSocket
1.4. Failure modes and how to handle them
Key idea: async jobs fail in four ways: a dead worker, an expired result, a result leaked to the wrong user, and overload. Each has a concrete guard.
- A worker dies mid-job: the job is stuck in
PROCESSING. Workers hold a job through a lease (a time-limited claim,locked_until) and renew it; when the lease expires another worker picks it up (SELECT ... FOR UPDATE SKIP LOCKED, see transactions and locking). Cap the attempts, then mark itFAILED. - Expired results: a lifecycle rule deletes files after N days;
GET /resultreturns410 Gone, not500. - Authorisation:
GET /jobs/{id}checks the job belongs to the current user. Sequential ids are easy to guess: use UUIDs and still check ownership. - Overload: cap concurrent jobs per user/tenant, return
429beyond that.
Code to sign and verify a webhook HMAC signature
static String sign(byte[] secret, long ts, String body) throws Exception {
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(secret, "HmacSHA256"));
return HexFormat.of().formatHex(mac.doFinal((ts + "." + body).getBytes(StandardCharsets.UTF_8)));
}
static boolean verify(byte[] secret, String header, String rawBody, long nowEpochSec) throws Exception {
Map<String, String> kv = new HashMap<>();
for (String p : header.split(",")) { String[] x = p.split("=", 2); kv.put(x[0], x[1]); }
long ts = Long.parseLong(kv.get("t"));
if (Math.abs(nowEpochSec - ts) > 300) return false; // older than 5 minutes: treat as a replay
byte[] expected = sign(secret, ts, rawBody).getBytes(StandardCharsets.US_ASCII);
return MessageDigest.isEqual(expected, kv.get("v1").getBytes(StandardCharsets.US_ASCII));
}
// Test run: valid signature -> true; "DONE" changed to "FAILED" -> false; resent an hour later -> false
2. Case study: duplicate payment requests
Key idea: the client sends one Idempotency-Key per operation; the server stores the key with the first response and returns exactly that response on every retry.
2.1. The problem
Key idea: a client that timed out can’t tell whether the money moved, so it retries; every duplicate is real money.
A client calls POST /payment-service/v1/payments. The server debits the account, then the connection drops; the client sees a timeout and retries. Or the user double-clicks “Pay”, or an attacker captures the request and replays it.
The course’s answer: the client generates an Idempotency-Key (a UUID) per operation and sends it as a header. The server stores the key in a column with a UNIQUE constraint; a second request with the same key is rejected by the database (“Duplicated!”).
2.2. What the original solution leaves out
Key idea: a UNIQUE constraint only stops the second row. A complete idempotent API also has to answer the six questions below.
- What does the retry get back? If it gets “Duplicated!”, the client thinks the payment failed. You must store the first response and return it verbatim (same status, same body).
- Same key, different body? (a buggy client reuses a key) Store a body hash; if it differs return
422(Stripe does this), never the old payment’s result. - Two requests with the same key at once? Either make the second wait (insert the key in the same transaction so the database blocks on the unique index), or return
409 Conflictso the client tries later. - Key scope: unique per
(client_id, key), not globally. - Expiry: keep keys for 24 hours (Stripe) to a few days, then a cleanup job purges them.
- Which outcomes are stored? Validation errors (
400) are not, so the client can fix the request and resend with the same key. Settled business outcomes (success, “insufficient balance”) are stored. Infrastructure failures halfway through roll back the key too, so a retry runs again.
flowchart TD
A[Request + Idempotency-Key] --> B{Key seen before?}
B -- No --> C[Insert key IN_PROGRESS] --> D[Run business logic] --> E[Store status + body, COMPLETED] --> F[Return response]
B -- Yes, different body hash --> G[422 key reused]
B -- Yes, still in progress --> H[409 or wait]
B -- Yes, COMPLETED --> I[Return stored response]
Idempotency from APIs to Kafka and payment gateways implements and tests this flow on PostgreSQL (20 concurrent requests with the same key create exactly one transaction), plus idempotent Kafka consumers and payment-gateway IPNs.
The client should generate the key (a UUID per operation, kept across retries), because only the client knows two requests are the same intent.
Three ways to generate the key: client, server-derived, server-issued
3. Response envelope and 6-digit error codes
Key idea: an envelope (a shared meta + data wrapper) gives every response the same shape. A 6-digit error code only helps if its first 3 digits always equal the real HTTP status.
The course’s template:
{
"meta": {
"code": "200000",
"type": "SUCCESS",
"message": "Payment created",
"service_id": "payment-service",
"request_id": "c0a8012e-7f1d-4b6a-9e1b-6f3c2d1a0b9e",
"extra_meta": {}
},
"data": { "payment_id": "P-7781", "status": "COMPLETED" }
}
A business error: meta.code = "400001", type = "INSUFFICIENT_DEBIT_AMOUNT", data = null. 6-digit code = HTTP status (3 digits) + business sequence number (3 digits): 404003 reads at a glance as a “not found” kind of error. type is a stable constant for clients to switch on; message is for humans and may change.
4. Writing API documentation
Key idea: good docs describe request and response clearly, list every possible error, and include a cURL that runs; generate the mechanical parts so they can’t drift from the code.
4.1. Documenting one endpoint
Key idea: each endpoint needs a general spec, a field table and an error table. The Description column must state business rules, format and an example.
- Field table: Parameter | Type | Required | Default | Description.
- Error table: HTTP status | code | type | description, e.g.
422 | 422001 | INSUFFICIENT_DEBIT_AMOUNT | The debit account balance is too low.
The original template has two copy-paste errors (pagination parameters on a POST, a cURL taken from another API). Guard against them with OpenAPI and by running the cURL samples in CI.
Sample template for the “Make payment” endpoint and the copy-paste errors
| Item | Content |
|---|---|
| Name | Make payment |
| Description | Create a transfer between two regular accounts |
| URL | https://{{host}}/payment-service/v1/payments |
| Method | POST |
| Headers | Content-Type: application/json, Authorization: Bearer <token>, Idempotency-Key: <uuid> (required) |
| Request body | debit_account, credit_account, amount |
| Response | meta + data envelope |
A good Description for amount: “integer, in the currency’s minor unit (VND has no fractional part), > 0, ≤ 500,000,000; e.g. 150000”.
4.2. The API map
Key idea: an API map is the index that answers “which APIs does this flow call, in what order, and who owns them”, kept both by flow and by resource.
- By flow: Flow ID, flow name (Payment, Get balance), call order, link to each API’s spec, PIC (person in charge).
- By resource: Payment, Order, Account… each resource lists all of its endpoints.
Start with a spreadsheet; at scale, use a developer portal (Backstage) or an API catalogue. Broader architecture documentation (C4, ADRs) is in Writing architecture docs.
Sample API map and Spring REST Docs
| Flow | Step | API | Service | PIC |
|---|---|---|---|---|
| F01 Payment | 1 | GET /account-service/v1/accounts/{id}/balance |
account | Team A |
| F01 Payment | 2 | POST /payment-service/v1/payments |
payment | Team B |
| F01 Payment | 3 | GET /payment-service/v1/payments/{id} |
payment | Team B |
Exercises
Exercise 1: An API contract for async exports
Task: write out every endpoint, status code and job state for the export feature.
Solution to exercise 1
- States:
PENDING → PROCESSING → DONE | FAILED | EXPIRED. POST /v1/exports→202+Location; a repeated Idempotency-Key → return the existing job; over quota →429.GET /v1/jobs/{id}→200withRetry-Afterwhile not done; someone else’s job →404(don’t reveal that it exists).GET /v1/jobs/{id}/result→303to a presigned URL whenDONE,409while not done,410once the file has expired.
Exercise 2: Document the “Make payment” API
Task: using the template from section 4, write the full spec, field table, error table and cURL.
Solution to exercise 2
curl -X POST "https://$HOST/payment-service/v1/payments" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-H "Idempotency-Key: 5f0e4c1a-2b7d-4e8f-9a3c-1d2e3f4a5b6c" \
-d '{"debit_account": "1001", "credit_account": "2002", "amount": 150000}'
| HTTP | code | type | Meaning |
|---|---|---|---|
| 201 | 201000 | SUCCESS | Transaction created |
| 400 | 400001 | INVALID_AMOUNT | amount ≤ 0 or above the limit |
| 400 | 400002 | MISSING_IDEMPOTENCY_KEY | Required header missing |
| 403 | 403001 | DEBIT_ACCOUNT_LOCKED | The debit account is locked |
| 409 | 409001 | REQUEST_IN_PROGRESS | A request with the same key is still being processed |
| 422 | 422001 | INSUFFICIENT_DEBIT_AMOUNT | Balance too low |
| 422 | 422002 | IDEMPOTENCY_KEY_REUSED | Same key, different body |
Remember
- Can you explain why a several-hundred-MB export must be an async job, and name the four things that break when it’s synchronous?
- Can you draw the flow
POST→202+Location→ polling →303presigned URL, and say when to pick webhooks or SSE instead? - Do you know how to sign and verify a webhook: HMAC over the raw body, a timestamp against replays, dedupe on
event_id? - Can you answer the six questions a UNIQUE constraint leaves open for an Idempotency-Key?
- Do you know the 6-digit error-code rule and when to pick a custom envelope or Problem Details?
Interview questions
Junior
Why not use GET to create an export job?
Because GET must be safe: many things are allowed to repeat a GET, and each repeat would create another heavy job.
Crawlers, prefetching, link previews and automatic retries may all repeat a GET. Creating a job creates a resource, so use POST and return 202 Accepted with a Location header pointing at the job resource.
Mid
When do you return 202 Accepted instead of 201 Created?
201 when the resource was fully created within the request; 202 when the server has only accepted the request and will process it later.
201 comes with a Location pointing at the resource. 202 should carry a Location for the job resource so the client can track it, optionally with Retry-After. 202 makes no promise that the work will succeed.
What do 6-digit business error codes buy you, and what rules do they need?
The code tells you the error class at a glance (the first 3 digits are the HTTP status), but only if the prefix always matches the real status.
The last 3 digits are a business sequence number; type is a stable constant for clients to switch on. Rules: the prefix matches the real HTTP status, each code has exactly one meaning listed in the docs, and codes are never reused for something else. Check the prefix match with an automated test.
Senior
How would you design an API that exports a file of several hundred MB?
As an async job: POST returns 202 + Location, a worker streams the data, the result is served through a presigned URL; never synchronously.
- Synchronously you’d hit LB/gateway timeouts (often 60 seconds), run out of memory, and hold threads and connections for minutes.
- The worker streams from the database (fetch size, server-side cursor, no List) and does a multipart upload to object storage.
- The client polls with Retry-After or receives a webhook; it downloads through a time-limited presigned URL.
- Dead workers: leases and bounded retries. Also cap concurrent jobs per tenant, check ownership on jobs, and clean up files with a lifecycle rule.
How does a Stripe-style Idempotency-Key work?
The client sends one key per operation; the server stores the key with the first response and returns it verbatim on every retry.
- The server stores (client, key) under a UNIQUE constraint, with a body hash, a status and the response.
- First call: insert the key, do the work, store status + body.
- Later calls with the same key: different body → 422, still in progress → 409 (or wait), completed → exactly the stored response.
- Validation errors aren’t stored so the client can fix them; infrastructure failures roll the key back. Keys expire after about 24 hours.
Polling, webhooks, SSE/WebSocket: how do you choose for job-completion notices?
Polling for low load and any client, webhooks for B2B and payments, SSE/WebSocket for UIs that need live progress; always keep the status API as the fallback.
- Polling: simple, works for every client, wastes empty requests.
- Webhooks: efficient and near real-time, but need a receiver with a public URL, HMAC signatures, retries with backoff and an idempotent receiver.
- SSE/WebSocket: long-lived connections and fan-out of events across instances.
How do you secure a webhook receiver?
Verify an HMAC signature over the raw body, block replays with a timestamp, dedupe on event_id, and for payments call the provider back to confirm.
- Compare signatures in constant time; accept HTTPS only.
- Deduplicate on event_id because the sender will retry.
- Return 2xx fast and process asynchronously so you don’t time out and trigger a resend.
- For critical data, don’t fully trust the webhook payload: call the provider’s API to confirm the status.
What are the downsides of a meta/data response envelope?
It’s non-standard, clients have to unwrap data everywhere, and it tends to lead to HTTP 200 even for errors.
200 for errors breaks retries, caching and status-based alerting. If you use one, the HTTP status must always be correct and the business code must match it. Problem Details (RFC 9457) is the standard choice for errors, with the business code in an extension field.
Useful phrases for interviews
- “Anything that takes longer than a gateway timeout becomes an async job: we return 202 with a Location header and let the client poll.”
- “The worker streams rows with a server-side cursor and uploads in parts, so memory stays flat no matter how big the file is.”
- “Results are served through a short-lived presigned URL, so the file never passes through our app servers.”
- “Webhooks are signed with an HMAC over the raw body plus a timestamp, and the receiver deduplicates on the event id.”
- “A unique constraint alone isn’t idempotency; the retry has to get the original response back.”
- “If the same key arrives with a different payload, we reject it with a 422 instead of replaying the old result.”
- “The first three digits of our error code always match the HTTP status, and a test enforces that.”
- “We generate the mechanical part of the docs from OpenAPI and only hand-write the business rules.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.