Skip to content
notes.vinhnq.com

Search

Web & APISenior20 min read

Advanced REST API design: async exports, Idempotency-Key and API documentation

Review 16 cards
On this page

Advanced REST API series: 1. Idempotent methods, URLs and pagination · 2. Async jobs, Idempotency-Key and API docs

This post solves two things a plain “request in, response out” API can’t: an operation that takes minutes (exporting a 500 MB file) and an operation that must never run twice (a retried payment). Then it covers a response envelope with consistent error codes and how to document an API that many teams share. Afterwards you can design an async job and a complete Idempotency-Key.

Read first: part 1 (safe vs idempotent, pagination) and Web and HTTP (status codes such as 202, 303, 409, 410).

1. Case study: exporting a 500 MB file

Key idea: don’t run a heavy export inside one synchronous request. Split it into “create a job” and “fetch the result”, let a worker stream the data, and serve the file straight from object storage.

1.1. The problem

Key idea: a synchronous API for minutes of work hits timeouts, runs out of memory, ties up resources and can’t resume after a network drop.

An admin clicks “Export to Excel” on a product list; for a big customer the file is about 500 MB and takes minutes to build. The naive design: query, write the file, return it in the same request.

  • Timeouts: Nginx’s proxy_read_timeout defaults to 60 seconds, the AWS ALB idle timeout to 60 seconds, Cloudflare gives about 100 seconds. The client gets a 504 but the server keeps going; the user clicks again and now two heavy jobs run.
  • OOM (out of memory, the heap runs out): findAll() loads the whole result into a List, JPA’s persistence context holds every entity, the Excel library keeps the whole workbook. 500 MB of data turns into several GB of heap.
  • Tied-up resources: a servlet thread and a database connection are held for minutes; a few concurrent exports drain the pool.
  • No way to resume: a network blip at minute 4 means starting over.

1.2. The design: an async job

Key idea: POST creates the job and returns 202 Accepted + Location, a GET checks status, and GET /result redirects to a presigned URL (a pre-signed link that expires after a few minutes).

  1. POST /v1/exports with body {"type": "PRODUCTS", "filter": {"name": "pen"}} → 202 Accepted, header Location: /v1/jobs/001, body {"job_id": "001", "status": "PENDING", "issued_at": 1763443200000}.
  2. GET /v1/jobs/001 → {"status": "PROCESSING", "progress": 42}, later {"status": "DONE", "updated_at": ...} or {"status": "FAILED", "error": {...}}.
  3. GET /v1/jobs/001/result → 302/303 to a presigned URL on object storage (S3, MinIO, GCS), valid for say 15 minutes. The file never flows through the app servers.

The worker reads and writes as a stream (handle a row, then drop it, never collect everything in RAM), so memory stays constant however big the file is.

Streaming export worker code and per-database details
@Transactional(readOnly = true) // PostgreSQL: the driver only streams with autocommit off
public void runExport(ExportJob job, OutputStream out) {
    jdbcTemplate.setFetchSize(1_000);           // fetch 1000 rows per round trip, not the whole result
    try (var writer = new BufferedWriter(new OutputStreamWriter(out, StandardCharsets.UTF_8))) {
        jdbcTemplate.query(
            "SELECT id, name, price FROM product WHERE name ILIKE ? ORDER BY id",
            rs -> {                              // RowCallbackHandler: row by row, no List
                writer.write(rs.getLong(1) + "," + csv(rs.getString(2)) + "," + rs.getBigDecimal(3));
                writer.newLine();
            },
            "%" + job.filterName() + "%");
    } catch (IOException e) {
        throw new UncheckedIOException(e);
    }
}
// out = a streaming multipart upload to object storage, not a giant temp file on the pod's disk
  • PostgreSQL: the driver only uses a server-side cursor when autoCommit=false and fetchSize > 0.
  • MySQL Connector/J: needs useCursorFetch=true (or fetchSize = Integer.MIN_VALUE to stream row by row).
  • Spring Data Stream<T>: must run inside a transaction and detach each entity after use, otherwise the persistence context keeps growing.
  • Excel: use a streaming API (Apache POI’s SXSSFWorkbook) or export CSV.

1.3. Polling, webhooks or real-time push

Key idea: polling (the client asks periodically) is the simplest; a webhook (the server calls the client’s URL) is efficient but complex; SSE/WebSocket suits UIs that need live progress. Whatever you choose, keep the status API.

sequenceDiagram
    participant C as Client
    participant A as API
    participant W as Worker
    participant S as Object storage
    C->>A: POST /v1/exports
    A-->>C: 202 Accepted, Location /v1/jobs/001
    A->>W: enqueue job
    loop Polling with backoff
        C->>A: GET /v1/jobs/001
        A-->>C: PROCESSING, Retry-After 5
    end
    W->>S: multipart upload
    W->>A: mark job DONE
    C->>A: GET /v1/jobs/001
    A-->>C: DONE
    C->>A: GET /v1/jobs/001/result
    A-->>C: 303 to presigned URL
    C->>S: download the file directly

With a webhook, the client supplies a callback_url when creating the job (or registers one up front). When done, the server calls POST callback_url with {"job_id": "001", "status": "DONE"}. In practice you combine them: a webhook to notify, the status API as the fallback when a webhook is lost.

Comparison table: polling, webhooks and SSE/WebSocket

1.4. Failure modes and how to handle them

Key idea: async jobs fail in four ways: a dead worker, an expired result, a result leaked to the wrong user, and overload. Each has a concrete guard.

  • A worker dies mid-job: the job is stuck in PROCESSING. Workers hold a job through a lease (a time-limited claim, locked_until) and renew it; when the lease expires another worker picks it up (SELECT ... FOR UPDATE SKIP LOCKED, see transactions and locking). Cap the attempts, then mark it FAILED.
  • Expired results: a lifecycle rule deletes files after N days; GET /result returns 410 Gone, not 500.
  • Authorisation: GET /jobs/{id} checks the job belongs to the current user. Sequential ids are easy to guess: use UUIDs and still check ownership.
  • Overload: cap concurrent jobs per user/tenant, return 429 beyond that.
Code to sign and verify a webhook HMAC signature
static String sign(byte[] secret, long ts, String body) throws Exception {
    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret, "HmacSHA256"));
    return HexFormat.of().formatHex(mac.doFinal((ts + "." + body).getBytes(StandardCharsets.UTF_8)));
}

static boolean verify(byte[] secret, String header, String rawBody, long nowEpochSec) throws Exception {
    Map<String, String> kv = new HashMap<>();
    for (String p : header.split(",")) { String[] x = p.split("=", 2); kv.put(x[0], x[1]); }
    long ts = Long.parseLong(kv.get("t"));
    if (Math.abs(nowEpochSec - ts) > 300) return false;            // older than 5 minutes: treat as a replay
    byte[] expected = sign(secret, ts, rawBody).getBytes(StandardCharsets.US_ASCII);
    return MessageDigest.isEqual(expected, kv.get("v1").getBytes(StandardCharsets.US_ASCII));
}
// Test run: valid signature -> true; "DONE" changed to "FAILED" -> false; resent an hour later -> false

2. Case study: duplicate payment requests

Key idea: the client sends one Idempotency-Key per operation; the server stores the key with the first response and returns exactly that response on every retry.

2.1. The problem

Key idea: a client that timed out can’t tell whether the money moved, so it retries; every duplicate is real money.

A client calls POST /payment-service/v1/payments. The server debits the account, then the connection drops; the client sees a timeout and retries. Or the user double-clicks “Pay”, or an attacker captures the request and replays it.

The course’s answer: the client generates an Idempotency-Key (a UUID) per operation and sends it as a header. The server stores the key in a column with a UNIQUE constraint; a second request with the same key is rejected by the database (“Duplicated!”).

2.2. What the original solution leaves out

Key idea: a UNIQUE constraint only stops the second row. A complete idempotent API also has to answer the six questions below.

  1. What does the retry get back? If it gets “Duplicated!”, the client thinks the payment failed. You must store the first response and return it verbatim (same status, same body).
  2. Same key, different body? (a buggy client reuses a key) Store a body hash; if it differs return 422 (Stripe does this), never the old payment’s result.
  3. Two requests with the same key at once? Either make the second wait (insert the key in the same transaction so the database blocks on the unique index), or return 409 Conflict so the client tries later.
  4. Key scope: unique per (client_id, key), not globally.
  5. Expiry: keep keys for 24 hours (Stripe) to a few days, then a cleanup job purges them.
  6. Which outcomes are stored? Validation errors (400) are not, so the client can fix the request and resend with the same key. Settled business outcomes (success, “insufficient balance”) are stored. Infrastructure failures halfway through roll back the key too, so a retry runs again.
flowchart TD
    A[Request + Idempotency-Key] --> B{Key seen before?}
    B -- No --> C[Insert key IN_PROGRESS] --> D[Run business logic] --> E[Store status + body, COMPLETED] --> F[Return response]
    B -- Yes, different body hash --> G[422 key reused]
    B -- Yes, still in progress --> H[409 or wait]
    B -- Yes, COMPLETED --> I[Return stored response]

Idempotency from APIs to Kafka and payment gateways implements and tests this flow on PostgreSQL (20 concurrent requests with the same key create exactly one transaction), plus idempotent Kafka consumers and payment-gateway IPNs.

The client should generate the key (a UUID per operation, kept across retries), because only the client knows two requests are the same intent.

Three ways to generate the key: client, server-derived, server-issued

3. Response envelope and 6-digit error codes

Key idea: an envelope (a shared meta + data wrapper) gives every response the same shape. A 6-digit error code only helps if its first 3 digits always equal the real HTTP status.

The course’s template:

{
  "meta": {
    "code": "200000",
    "type": "SUCCESS",
    "message": "Payment created",
    "service_id": "payment-service",
    "request_id": "c0a8012e-7f1d-4b6a-9e1b-6f3c2d1a0b9e",
    "extra_meta": {}
  },
  "data": { "payment_id": "P-7781", "status": "COMPLETED" }
}

A business error: meta.code = "400001", type = "INSUFFICIENT_DEBIT_AMOUNT", data = null. 6-digit code = HTTP status (3 digits) + business sequence number (3 digits): 404003 reads at a glance as a “not found” kind of error. type is a stable constant for clients to switch on; message is for humans and may change.

4. Writing API documentation

Key idea: good docs describe request and response clearly, list every possible error, and include a cURL that runs; generate the mechanical parts so they can’t drift from the code.

4.1. Documenting one endpoint

Key idea: each endpoint needs a general spec, a field table and an error table. The Description column must state business rules, format and an example.

  • Field table: Parameter | Type | Required | Default | Description.
  • Error table: HTTP status | code | type | description, e.g. 422 | 422001 | INSUFFICIENT_DEBIT_AMOUNT | The debit account balance is too low.

The original template has two copy-paste errors (pagination parameters on a POST, a cURL taken from another API). Guard against them with OpenAPI and by running the cURL samples in CI.

Sample template for the “Make payment” endpoint and the copy-paste errors
Item Content
Name Make payment
Description Create a transfer between two regular accounts
URL https://{{host}}/payment-service/v1/payments
Method POST
Headers Content-Type: application/json, Authorization: Bearer <token>, Idempotency-Key: <uuid> (required)
Request body debit_account, credit_account, amount
Response meta + data envelope

A good Description for amount: “integer, in the currency’s minor unit (VND has no fractional part), > 0, ≤ 500,000,000; e.g. 150000”.

4.2. The API map

Key idea: an API map is the index that answers “which APIs does this flow call, in what order, and who owns them”, kept both by flow and by resource.

  • By flow: Flow ID, flow name (Payment, Get balance), call order, link to each API’s spec, PIC (person in charge).
  • By resource: Payment, Order, Account… each resource lists all of its endpoints.

Start with a spreadsheet; at scale, use a developer portal (Backstage) or an API catalogue. Broader architecture documentation (C4, ADRs) is in Writing architecture docs.

Sample API map and Spring REST Docs
Flow Step API Service PIC
F01 Payment 1 GET /account-service/v1/accounts/{id}/balance account Team A
F01 Payment 2 POST /payment-service/v1/payments payment Team B
F01 Payment 3 GET /payment-service/v1/payments/{id} payment Team B

Exercises

Exercise 1: An API contract for async exports

Task: write out every endpoint, status code and job state for the export feature.

Solution to exercise 1
  • States: PENDING → PROCESSING → DONE | FAILED | EXPIRED.
  • POST /v1/exports → 202 + Location; a repeated Idempotency-Key → return the existing job; over quota → 429.
  • GET /v1/jobs/{id} → 200 with Retry-After while not done; someone else’s job → 404 (don’t reveal that it exists).
  • GET /v1/jobs/{id}/result → 303 to a presigned URL when DONE, 409 while not done, 410 once the file has expired.

Exercise 2: Document the “Make payment” API

Task: using the template from section 4, write the full spec, field table, error table and cURL.

Solution to exercise 2
curl -X POST "https://$HOST/payment-service/v1/payments" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Idempotency-Key: 5f0e4c1a-2b7d-4e8f-9a3c-1d2e3f4a5b6c" \
  -d '{"debit_account": "1001", "credit_account": "2002", "amount": 150000}'
HTTP code type Meaning
201 201000 SUCCESS Transaction created
400 400001 INVALID_AMOUNT amount ≤ 0 or above the limit
400 400002 MISSING_IDEMPOTENCY_KEY Required header missing
403 403001 DEBIT_ACCOUNT_LOCKED The debit account is locked
409 409001 REQUEST_IN_PROGRESS A request with the same key is still being processed
422 422001 INSUFFICIENT_DEBIT_AMOUNT Balance too low
422 422002 IDEMPOTENCY_KEY_REUSED Same key, different body

Remember

  • Can you explain why a several-hundred-MB export must be an async job, and name the four things that break when it’s synchronous?
  • Can you draw the flow POST → 202 + Location → polling → 303 presigned URL, and say when to pick webhooks or SSE instead?
  • Do you know how to sign and verify a webhook: HMAC over the raw body, a timestamp against replays, dedupe on event_id?
  • Can you answer the six questions a UNIQUE constraint leaves open for an Idempotency-Key?
  • Do you know the 6-digit error-code rule and when to pick a custom envelope or Problem Details?

Interview questions

Junior

Why not use GET to create an export job?

Because GET must be safe: many things are allowed to repeat a GET, and each repeat would create another heavy job.

Crawlers, prefetching, link previews and automatic retries may all repeat a GET. Creating a job creates a resource, so use POST and return 202 Accepted with a Location header pointing at the job resource.

Mid

When do you return 202 Accepted instead of 201 Created?

201 when the resource was fully created within the request; 202 when the server has only accepted the request and will process it later.

201 comes with a Location pointing at the resource. 202 should carry a Location for the job resource so the client can track it, optionally with Retry-After. 202 makes no promise that the work will succeed.

What do 6-digit business error codes buy you, and what rules do they need?

The code tells you the error class at a glance (the first 3 digits are the HTTP status), but only if the prefix always matches the real status.

The last 3 digits are a business sequence number; type is a stable constant for clients to switch on. Rules: the prefix matches the real HTTP status, each code has exactly one meaning listed in the docs, and codes are never reused for something else. Check the prefix match with an automated test.

Senior

How would you design an API that exports a file of several hundred MB?

As an async job: POST returns 202 + Location, a worker streams the data, the result is served through a presigned URL; never synchronously.

  • Synchronously you’d hit LB/gateway timeouts (often 60 seconds), run out of memory, and hold threads and connections for minutes.
  • The worker streams from the database (fetch size, server-side cursor, no List) and does a multipart upload to object storage.
  • The client polls with Retry-After or receives a webhook; it downloads through a time-limited presigned URL.
  • Dead workers: leases and bounded retries. Also cap concurrent jobs per tenant, check ownership on jobs, and clean up files with a lifecycle rule.
How does a Stripe-style Idempotency-Key work?

The client sends one key per operation; the server stores the key with the first response and returns it verbatim on every retry.

  • The server stores (client, key) under a UNIQUE constraint, with a body hash, a status and the response.
  • First call: insert the key, do the work, store status + body.
  • Later calls with the same key: different body → 422, still in progress → 409 (or wait), completed → exactly the stored response.
  • Validation errors aren’t stored so the client can fix them; infrastructure failures roll the key back. Keys expire after about 24 hours.
Polling, webhooks, SSE/WebSocket: how do you choose for job-completion notices?

Polling for low load and any client, webhooks for B2B and payments, SSE/WebSocket for UIs that need live progress; always keep the status API as the fallback.

  • Polling: simple, works for every client, wastes empty requests.
  • Webhooks: efficient and near real-time, but need a receiver with a public URL, HMAC signatures, retries with backoff and an idempotent receiver.
  • SSE/WebSocket: long-lived connections and fan-out of events across instances.
How do you secure a webhook receiver?

Verify an HMAC signature over the raw body, block replays with a timestamp, dedupe on event_id, and for payments call the provider back to confirm.

  • Compare signatures in constant time; accept HTTPS only.
  • Deduplicate on event_id because the sender will retry.
  • Return 2xx fast and process asynchronously so you don’t time out and trigger a resend.
  • For critical data, don’t fully trust the webhook payload: call the provider’s API to confirm the status.
What are the downsides of a meta/data response envelope?

It’s non-standard, clients have to unwrap data everywhere, and it tends to lead to HTTP 200 even for errors.

200 for errors breaks retries, caching and status-based alerting. If you use one, the HTTP status must always be correct and the business code must match it. Problem Details (RFC 9457) is the standard choice for errors, with the business code in an extension field.

Useful phrases for interviews

  • “Anything that takes longer than a gateway timeout becomes an async job: we return 202 with a Location header and let the client poll.”
  • “The worker streams rows with a server-side cursor and uploads in parts, so memory stays flat no matter how big the file is.”
  • “Results are served through a short-lived presigned URL, so the file never passes through our app servers.”
  • “Webhooks are signed with an HMAC over the raw body plus a timestamp, and the receiver deduplicates on the event id.”
  • “A unique constraint alone isn’t idempotency; the retry has to get the original response back.”
  • “If the same key arrives with a different payload, we reject it with a 422 instead of replaying the old result.”
  • “The first three digits of our error code always match the HTTP status, and a test enforces that.”
  • “We generate the mechanical part of the docs from OpenAPI and only hand-write the business rules.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.