Skip to content
notes.vinhnq.com

Search

System DesignSenior17 min read

Third-party integration: choosing a provider, sync versus async, webhooks and an anti-corruption layer

Review 13 cards
On this page

Third-party integration series: 1. Choosing a provider, webhooks and an anti-corruption layer · 2. Timeouts, retries, circuit breakers and bulkheads

Every third-party integration (payment gateway, SMS OTP, eKYC, shipping) is a dependency you don’t control: slow, cryptic errors, duplicate webhooks, API changes. This post extracts the general playbook: evaluating requirements, choosing a provider, choosing an integration style, receiving webhooks safely, hiding the provider behind your own interface, logging and testing. The concrete VNPay QR example lives in the payment gateway post and the late IPNs and reconciliation post; timeouts, retries, circuit breakers and bulkheads are in part 2.

1. Requirements analysis: what to evaluate, in what order

Key idea: security, then reliability, then performance (throughput, latency), then complexity. The order follows the cost of getting it wrong: whatever loses the most when it fails is checked first.

  • Security wrong: you can lose real money you never get back.
  • Reliability wrong: orders end up in the wrong state, forcing manual reconciliation.
  • Performance poor: still fixable with caches, queues, more machines.
  • Complexity: paid gradually in engineering time.
The course’s online-payments problem

A product needs to take payments online, automatically. Requirements: automated payments, strong security, low fees, many payment methods, plus soft factors (stability, technical support, promotions, UI, integration speed, merchant portal). The answer: integrate a payment gateway rather than dealing with each bank yourself.

1.1. Choosing a provider

There is no “best” provider: each has strengths and weaknesses, and the choice depends on your requirements. Compare across seven groups: security, fees, reliability, settlement, integration, operations, and exit cost (switching to someone else). Two things are often missed: the slide lists PCI DSS 3.2.1, a retired version; and the fees on the slide are only a snapshot in time.

Provider selection criteria, the slide’s fees and current PCI DSS
Criterion Concrete questions
Security, compliance Current PCI DSS certification, at what level? How are requests signed? Fixed IPs for callbacks?
Fees Per transaction (percentage + fixed fee), setup fee, monthly fee, refund fees
Reliability Uptime SLA, maintenance windows, incident history, is there a status-query API, how are callbacks retried
Settlement Settlement (money landing in the merchant account) T+1 or T+n, reconciliation via files or an API
Integration Docs, sandbox, SDKs, API versioning, time to get the first flow running
Operations Technical support, merchant portal, partial refunds, reporting
Exit What does switching cost: are card tokens portable, any contractual lock-in

The slides list the compliance level and domestic QR fee schedule of five popular gateways at the time they were made: roughly 1.1% plus a couple of thousand dong per transaction, with some adding a setup fee and a monthly fee. These numbers change with contracts and over time. Use the table as a template to fill in yourself, not as reference data.

2. Synchronous or asynchronous

Key idea: if the third party finishes instantly, call synchronously; if it takes a while, call synchronously to create the request and receive the result asynchronously. The VNPay QR flow is the example: the payment URL is created synchronously (step 4), and the result arrives through a server-to-server IPN (step 9.1) and through the browser returning (step 9.2).

The same shape fits payments, eKYC, invoicing and shipping.

flowchart LR
    A[Our backend] -->|1. Sync call to create the request| B[Third party]
    B -->|2. Webhook when done| A
    A -->|3. Query status when the webhook is late| B
    J[Scheduled job] -->|4. Sweep stuck requests| A
Style When to use it The cost
Synchronous call, wait for the result Fast tasks (under 1–2 s) where you need the answer now: exchange rates, address validation Their latency and errors become yours
Synchronous create, result via webhook Long or user-dependent tasks: payments, eKYC A public endpoint to protect; duplicates, reordering, lost webhooks
Synchronous create, poll for status The provider has no webhooks, or you can’t expose an endpoint Extra requests, latency tied to the poll interval, easy to hit rate limits
Put it on your own queue, a worker calls out The user doesn’t need to wait: emails, SMS notifications, pushing orders to a carrier One more broker, intermediate states, a dead-letter path

A webhook is the third party calling an endpoint of yours when something happens; polling is you asking them periodically. The queue + worker style decouples the provider’s latency from the user’s request and flattens load spikes; choosing a broker is covered in the message broker post.

3. Receiving webhooks safely

Key idea: a webhook is a public endpoint anyone can call. Verify the signature on the raw body, block replays, store and answer fast, and tolerate duplicates and reordering.

The checklist, in processing order:

  1. Verify the signature on the raw body. The sender signs with an HMAC (a hash keyed with a shared secret) under a secret specific to you, or with a digital signature you verify with their public key. It guarantees integrity and origin together; HMACs versus signatures are compared in the encoding, hashing and signatures post.
  2. Replay protection (replay: an attacker resends an old valid request). A valid signature doesn’t mean it’s the first delivery. The timestamp sits inside the signed content; reject anything more than 5 minutes off. Also dedupe by event id.
  3. Filter by source IP against the provider’s published list, only as an extra layer, never instead of the signature.
  4. Store it verbatim, answer fast: write the event to the DB (or an outbox) and return 2xx; heavy processing runs later. Slow handling means redelivery.
  5. Return the right code per their docs, because the code decides whether they retry. Per the slides: 00 on success, 01 when the order isn’t found, and at the very least a try/catch so failures don’t become stack traces.

The verification code below has been run on JDK 21. MessageDigest.isEqual compares in constant time, so an attacker can’t guess the signature byte by byte from response timing.

static String sign(byte[] secret, long ts, String body) throws Exception {
    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret, "HmacSHA256"));
    return HexFormat.of().formatHex(mac.doFinal((ts + "." + body).getBytes(StandardCharsets.UTF_8)));
}

static String verifyWebhook(byte[] secret, String tsHeader, String sigHeader, String body, Clock clock) throws Exception {
    long ts;
    try { ts = Long.parseLong(tsHeader); } catch (NumberFormatException e) { return "reject: bad timestamp"; }
    if (Math.abs(clock.instant().getEpochSecond() - ts) > 300) return "reject: outside 5-minute window";
    byte[] expected = HexFormat.of().parseHex(sign(secret, ts, body));
    byte[] given;
    try { given = HexFormat.of().parseHex(sigHeader); } catch (IllegalArgumentException e) { return "reject: bad signature format"; }
    return MessageDigest.isEqual(expected, given) ? "accept" : "reject: signature mismatch";
}
// valid: accept; tampered body: signature mismatch; redelivered 10 minutes later: outside 5-minute window (verified)

How the sender retries: VNPay resends the IPN when the merchant replies 01, 04, 97, 99 or doesn’t answer in time, up to 10 times within about 5 minutes (per the slides; check the current docs). So your handler sees duplicates, sequential or concurrent, and must be idempotent and safe under concurrency. The UPDATE ... WHERE status = 'PENDING' plus unique-constraint approach is in idempotency part 2.

4. An anti-corruption layer: keep their model out of your system

Key idea: put each third party behind an anti-corruption layer: an interface you define in your own domain language, plus one adapter per provider that handles signing, calling and mapping error codes.

The common mistake: the provider’s DTOs and status codes flow straight into business code, like if (resp.getVnpResponseCode().equals("00")) scattered across the order service. Switching providers means editing everywhere. The code below is a sketch and wasn’t run.

public interface PaymentGateway {
    CheckoutSession createCheckout(PaymentAttempt attempt);          // URL or QR
    GatewayStatus queryStatus(String attemptRef);                   // for the reconciliation job
    WebhookOutcome parseCallback(Map<String, String> rawParams);    // verify signature + map
}

public sealed interface GatewayStatus {
    record Paid(long amountMinor, String providerTxnId, Instant paidAt) implements GatewayStatus {}
    record Failed(String reason) implements GatewayStatus {}
    record Pending() implements GatewayStatus {}
    record Unknown(String rawCode) implements GatewayStatus {}       // unknown code: don't guess, alert
}

The adapter is the only place that knows about vnp_ResponseCode, the ×100 amount unit, or how parameters are encoded for signing. It’s the Adapter pattern applied at a system boundary.

Other jobs of the adapter layer: versions, secrets, unknown codes
  • Pin API and library versions: start on the latest supported version, then upgrade deliberately. Turn on dependency vulnerability scanning (Dependabot, OWASP Dependency-Check), because the provider’s SDK drags in a whole tree of libraries.
  • One secret per environment and per provider, in a secret manager, rotatable.
  • Map unknown codes to Unknown and alert, never default them to success or failure.

5. Logging and tracing every step

Key idea: log the input and output of every step, on the success path, the failure path and exceptions alike. When a customer says “I paid but the order is still pending”, whoever is on call must tell within minutes how far the request got.

Check the Nginx log too, for the step where the browser comes back. At a minimum you need:

  • One structured log line per outbound call and per inbound webhook, carrying your reference and their transaction id.
  • Verbatim copies of the important requests/responses in a dedicated table, for reconciliation and disputes.
  • Per-provider metrics, and a trace id propagated on outbound calls.
Details: log fields, the verbatim table, metrics and trace ids
  • The structured log line holds: provider, operation, our reference, their transaction id, HTTP status, business code, duration, attempt number, trace id.
  • Store the important requests and responses verbatim (with sensitive data masked) in a dedicated table, because centralized logs have short retention.
  • Metrics per provider: p50/p99 latency, error rate by code, circuit-breaker state, webhooks rejected for bad signatures. Alert when p99 approaches their timeout or the error rate climbs.
  • Propagate the trace id in the traceparent header if they support it; at a minimum send your own reference so you can cross-check. Setting up tracing is covered in the distributed tracing post.

6. Testing the integration

Key idea: the sandbox for end-to-end flows, a fake server in CI for failures the sandbox can’t produce, and fixed tests for signatures and mapping so nothing breaks silently.

  • Sandbox: success, customer cancels, wrong OTP. Details in the reconciliation post.
  • Fake server (WireMock, MockWebServer, the JDK’s HttpServer as in part 2): slower than the timeout, 503 then 200, unknown codes, connections dropped mid-response, duplicate and out-of-order webhooks.
  • Fixed signature vectors: a sample payload, secret and signature from their docs.
  • Contract tests for the adapter: real (masked) sample responses must map correctly; when they change their API, the test goes red before production does.

Exercises

Exercise 1: Design an eKYC integration

An eKYC service accepts an ID-card photo and a selfie and returns a result after 5–60 seconds via webhook; it has no batch status API but does have a per-request status API. Design the flow.

Solution to Exercise 1: an asynchronous eKYC flow
  • Table kyc_request(id, user_id, provider_ref, status, created_at, updated_at), states SUBMITTED → PROCESSING → APPROVED | REJECTED | NEEDS_REVIEW.
  • Submit: the user uploads; the backend stores the images (encrypted, see the personal data post), creates the record and queues the call.
  • Worker calls the provider with kyc_request.id as the idempotency key, with a timeout and jittered retries, and stores provider_ref.
  • Webhook: verify the signature on the raw body and the timestamp, store it verbatim, apply a conditional (forward-only) state transition, reply 2xx immediately.
  • Job runs every minute to query requests stuck in PROCESSING for over 2 minutes, rate-limited so it stays within quota.
  • Frontend shows “verifying” and gets the result by polling your own backend or via server-sent events.
  • Unknown codes become NEEDS_REVIEW for manual review.
  • Metrics: time from submission to result; the lost-webhook rate (results that came from the job rather than the webhook).

Remember

  • Explain why security is evaluated first and complexity last.
  • Know when to call synchronously, when to use webhooks, polling or a queue, and why to use webhooks and polling together.
  • List the five steps of receiving a webhook safely, and why the signature must be checked on the raw body.
  • Know how to handle duplicate, concurrent and out-of-order webhooks.
  • Explain what an anti-corruption layer keeps out of business code, and why unknown codes must become Unknown.

Interview questions

Junior

What is a webhook, and how is it different from polling?

A webhook is the third party calling an endpoint of yours when something happens; polling is you asking them periodically.

Webhooks are fast and cheap in requests, but they can be lost, duplicated or reordered, and they need a public endpoint protected by a signature. Polling is more dependable but slower and eats quota. A good system uses webhooks as the main path and polling as the safety net.

Why must a webhook or IPN handler be idempotent?

Because senders deliver at least once, so the same event can arrive several times, even twice at the same moment.

They resend when they don’t get a valid reply in time (a VNPay-style gateway resends up to about 10 times in 5 minutes). A non-idempotent handler credits money twice or decrements stock twice. Dedupe by event id and apply state changes with a conditional UPDATE so the second delivery does nothing.

Mid

How do you protect a webhook endpoint?

Verify the signature on the raw body, block replays with a signed timestamp and the event id, then store and return 2xx fast.

  • Verify the HMAC (or digital signature) on the raw body, with a constant-time comparison.
  • Check a timestamp inside the signed content; dedupe by event id.
  • Filter source IPs as an extra layer.
  • Store the event verbatim, return 2xx fast, process later.
  • A forward-only state machine so duplicates and reordering are harmless.
  • Reply with the documented codes, because they decide whether the sender retries.
Why wrap a provider behind your own interface?

So the provider’s status codes, DTOs, amount units and signing rules don’t leak into business code (an anti-corruption layer).

Switching or adding a provider means adding an adapter. Business tests use a fake of the interface. Unknown codes map to an Unknown state that raises an alert instead of being mistaken for success.

A refunded webhook arrives before succeeded. How do you handle it?

Don’t apply events in arrival order: a forward-only state machine ignores an older event that arrives late instead of letting it overwrite.

Apply transitions with a conditional UPDATE on the current state. When an event doesn’t fit the current state, treat it as a signal, call the provider’s API for the latest state, and apply that through the same update function. Store every event verbatim for investigation.

Senior

How do you choose between two payment gateways, and how do you keep the option to switch?

Evaluate security, reliability, performance and complexity, plus cost, and record it in an ADR; keep switching possible with your own interface and adapters.

  • Criteria: current PCI DSS (ask for the AOC), how requests are signed and callbacks protected, SLA and incident history, a status-query API and reconciliation files, settlement timing, total fees (per transaction, fixed, refunds), sandbox quality and support.
  • Record the decision in an ADR with the date of the numbers.
  • To keep switching possible: a PaymentGateway interface, one adapter per provider, a provider-independent payment-attempt table, no card data on your side (redirect or hosted fields), contract tests per adapter.
An eKYC integration returns results after 5–60 seconds. Synchronous or asynchronous?

Asynchronous: holding a user request for 60 seconds holds a thread, tends to hit load balancer timeouts and can’t absorb retries.

Create the request through a queue and worker, with the request id as the idempotency key. Receive the result via a signed webhook, plus a job that queries the status of requests stuck past their deadline. The frontend shows “verifying” and asks your backend (polling or server-sent events), never the provider directly.

Useful phrases for interviews

  • “We evaluate integrations in this order: security first, then reliability, then performance, then complexity.”
  • “The fee table is a snapshot; we record the numbers and their date in an ADR.”
  • “The webhook is the fast path, and a status-query job is the safety net for the ones that never arrive.”
  • “We verify the signature on the raw bytes before we parse anything.”
  • “A valid signature isn’t proof of a first delivery, so we also check a signed timestamp and dedupe by event id.”
  • “Our business code only talks to our own gateway interface; the provider’s codes stay inside the adapter.”
  • “Unknown status codes map to an explicit unknown state and page someone; we never guess.”
  • “For disputes, we record not only what they sent us but what we replied and how long it took.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.