Skip to content
notes.vinhnq.com

Search

Git, DevOps & ToolingMid25 min read

Reverse proxying with Nginx: path-based routing and X-Forwarded headers

Review 21 cards
On this page

A frontend, an API and a blog usually live behind one domain; give each its own port or subdomain and you deal with CORS, a TLS certificate per endpoint, and every service doing its own rate limiting, logging and gzip. This post builds an Nginx reverse proxy (1.29.8, in Docker Compose) that routes by path to three services, and runs every request to show the real results. Afterwards you’ll avoid the four usual mistakes: a location that matches too much, a slash in proxy_pass that rewrites the path, spoofed X-Forwarded-* headers, and defaults that don’t suit production.

Worth reading first: Advanced REST API design (the section on URL conventions behind an API gateway).

1. Reverse proxy vs forward proxy, load balancer and API gateway

Key idea: a reverse proxy sits on the server side and takes requests on behalf of the services inside. Load balancers and API gateways are reverse proxies specialised in one more job: spreading load, or API-aware logic.

  • A forward proxy sits on the client side (a corporate proxy, a VPN); the destination sees the proxy’s IP.
  • A reverse proxy sits on the server side; clients don’t know how many services are behind it.
  • A load balancer: a reverse proxy spreading load across copies of the same service. It can work at layer 4 (TCP, doesn’t read HTTP) or layer 7 (reads HTTP).
  • An API gateway: a layer-7 reverse proxy plus API-aware logic: token validation, per-client rate limits, quotas, request transformation, response aggregation, API docs. Think Kong, APISIX, Spring Cloud Gateway, AWS API Gateway.

Nginx can play all three server-side roles; here it routes by path.

2. What we’re building

Key idea: one Nginx in front of three services: /api/ goes to the backend, /blog/ and /content/ to the blog, everything else to the frontend.

The course exercise draws this layout; redrawn:

flowchart LR
  C[Client] --> P[Nginx :80]
  P -- "/ (e.g. /home)" --> FE[Frontend :8000]
  P -- "/api/... (e.g. /api/orders)" --> BE[Backend :8080]
  P -- "/blog/... and /content/..." --> BL[Blog :9000]
Path the client calls Goes to Path the backend receives
/, /home frontend unchanged
/api/orders/145 backend unchanged, /api/orders/145
/blog/posts/1 blog prefix stripped: /posts/1
/content/hot blog unchanged, /content/hot

The blog gets two treatments on purpose (prefix kept, prefix stripped) so you can see how proxy_pass handles paths.

3. The Docker Compose lab

Key idea: the three services behind the proxy are fake Nginx containers that return JSON with the path and headers they received. One look at the response tells you what the proxy did.

gateway.conf (the main proxy):

upstream backend {
    server be:8080;
    keepalive 32;
}

server {
    listen 80;
    server_name _;

    client_max_body_size 10m;

    # Common headers for every proxied location
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Request-Id      $request_id;

    proxy_connect_timeout 2s;
    proxy_read_timeout    30s;

    location = /healthz {
        access_log off;
        return 200 "ok\n";
    }

    # /api/... -> backend, path kept as is (no URI part in proxy_pass)
    location /api/ {
        proxy_pass http://backend;
    }

    # /blog/... -> blog, prefix stripped (URI part "/" in proxy_pass)
    location /blog/ {
        proxy_pass http://blog:9000/;
    }

    # /content/... -> blog, path kept
    location /content/ {
        proxy_pass http://blog:9000;
    }

    # everything else -> frontend
    location / {
        proxy_pass http://fe:8000;
    }
}
The rest of the lab: echo.conf, compose.yaml, commands and a Docker Desktop note

echo.conf (for fe, be and blog):

server {
    listen 8000;
    listen 8080;
    listen 9000;
    default_type application/json;
    location / {
        return 200 '{"service":"$hostname","port":"$server_port","uri":"$request_uri","host":"$http_host","xff":"$http_x_forwarded_for","real_ip":"$http_x_real_ip","proto":"$http_x_forwarded_proto"}\n';
    }
}

compose.yaml:

services:
  gateway:
    image: nginx:1.29-alpine
    ports: ["8088:80"]
    volumes: ["./gateway.conf:/etc/nginx/conf.d/default.conf:ro"]
    depends_on: [fe, be, blog]
  fe:
    image: nginx:1.29-alpine
    hostname: fe
    volumes: ["./echo.conf:/etc/nginx/conf.d/default.conf:ro"]
  be:
    image: nginx:1.29-alpine
    hostname: be
    volumes: ["./echo.conf:/etc/nginx/conf.d/default.conf:ro"]
  blog:
    image: nginx:1.29-alpine
    hostname: blog
    volumes: ["./echo.conf:/etc/nginx/conf.d/default.conf:ro"]

Start it and test the config:

docker compose up -d
docker compose exec gateway nginx -t
# nginx: configuration file /etc/nginx/nginx.conf test is successful

curl -s -H 'X-Forwarded-For: 6.6.6.6' http://localhost:8088/api/orders/145

On Docker Desktop for Mac, the original exercise’s network_mode: host doesn’t behave as it does on Linux (the “host” is Docker’s VM, not your Mac). Using Compose’s default network and calling services by name, as above, works everywhere.

What actually came back (trimmed):

Request Service uri the backend got Note
/ fe /
/home fe /home
/api/orders/145 be /api/orders/145 prefix kept
/apiv2 fe /apiv2 doesn’t match /api/, falls through to /
/blog/posts/1 blog /posts/1 prefix stripped
/blog (proxy) 301 → /blog/ see section 5
/content/hot blog /content/hot
/healthz (proxy) ok answered by the proxy

The headers the backend saw for /api/orders/145: host: localhost, xff: 6.6.6.6, 172.22.0.1, real_ip: 172.22.0.1, proto: http. The xff line shows the proxy appends the address it sees to whatever the client sent; it doesn’t replace it (section 6).

4. Which location wins

Key idea: the longest matching prefix wins, and file order doesn’t matter. But an exact = match stops at once, and a regex (checked in file order) beats the prefix unless the prefix has ^~.

Nginx first normalises the URI (decodes %XX, resolves . and .., merges // into /), then picks:

  1. Check every prefix location and remember the one with the longest matching prefix.
  2. If a location = /path matches exactly, stop.
  3. If the longest prefix has ^~, stop and skip regexes.
  4. Check regex locations (~ case-sensitive, ~* case-insensitive) in file order; the first match wins.
  5. If no regex matches, use the prefix remembered in step 1.
Declaration Meaning
location = /healthz exactly /healthz
location ^~ /static/ prefix that beats regexes
location ~* \.(png|jpg)$ regex, checked in order
location /api/ ordinary prefix
location / catch-all

5. proxy_pass with or without a URI

Key idea: a proxy_pass without a URI (just http://host:port) sends the path exactly as the client sent it. With a URI (even just /), the part of the path that matched the location is replaced by that URI.

This is the most common mistake. Real results from a test server:

location proxy_pass Request Backend got
/api/ http://backend /api/orders /api/orders
/blog/ http://blog:9000/ /blog/posts/1 /posts/1
/v1/ http://be:8080/internal/ /v1/users /internal/users
/api http://be:8080/ /api/orders //orders
/api http://be:8080/ /apiv2 /v2
~ ^/img/(.*)$ http://fe:8000/static/$1 /img/a.png /static/a.png

The two /api rows are the classic bug: /api is replaced by /, so the remaining /orders becomes //orders, and /apiv2 becomes /v2 (a different API altogether).

The request /blog (no trailing slash) gets a 301 to /blog/, and the Location loses port 8088. The fix is in the block below.

Automatic redirects: why /blog gets a 301 without the port, absolute_redirect and proxy_redirect

The exact result: 301 Location: http://localhost/blog/. Nginx does this on its own: when a prefix location ends in / and is handled by proxy_pass, a request equal to that string minus the slash is redirected to the version with the slash. The port goes missing because Nginx builds absolute URLs from the server name and the port it listens on (80), and knows nothing about the port mapping outside.

6. X-Forwarded headers and whom to trust

Key idea: through a proxy, the backend loses three facts: the client IP, the domain the client called (Host), and HTTP vs HTTPS. Nginx carries them in headers, but clients can send headers themselves, so only trust the parts your own proxies added.

Header Value Nginx sets Used for
Host $host the original domain (virtual hosts, building URLs)
X-Real-IP $remote_addr the IP that connected directly to the proxy
X-Forwarded-For $proxy_add_x_forwarded_for IPs along each hop (each proxy on the way): previous value + , + $remote_addr
X-Forwarded-Proto $scheme https when TLS ends at the proxy
X-Request-Id $request_id a random id that links proxy logs to backend logs
  • If you set nothing, Nginx sends Host: $proxy_host (the name in proxy_pass) and no X-Forwarded-*; tested: the backend got host: be:8080 and an empty xff.
  • The newer standard is Forwarded (RFC 7239: Forwarded: for=203.0.113.7;proto=https;host=shop.vn), but X-Forwarded-* is far more common.

How far to trust X-Forwarded-For

The lab sent X-Forwarded-For: 6.6.6.6 and the backend got 6.6.6.6, 172.22.0.1. If the backend takes the first entry as the client IP (very common code), anyone can fake their IP: dodge rate limits, fake their location, pollute the logs.

The right way: walk from right to left, skip the IPs that belong to your own proxies, and the first IP that isn’t yours is the client. Nginx’s realip module does exactly this.

The realip module, tested: config and results
set_real_ip_from  172.16.0.0/12;   # only trust proxies in this range
real_ip_header    X-Forwarded-For;
real_ip_recursive on;              # skip every trusted IP from the right
Connection from X-Forwarded-For received $remote_addr after realip
127.0.0.1 (untrusted) 6.6.6.6 127.0.0.1 (header ignored)
172.22.0.2 (trusted) 1.2.3.4, 6.6.6.6 6.6.6.6 (1.2.3.4 ignored, it’s left of the first untrusted IP)

On the Spring Boot side

The backend has to read these headers for request.getRemoteAddr(), request.getScheme(), ServletUriComponentsBuilder and redirects to come out right:

# Tomcat RemoteIpValve reads X-Forwarded-For / -Proto / -Host / -Port
server.forward-headers-strategy=native
# Only trust your real proxy/LB IPs (a regex); the default is the private ranges
server.tomcat.remoteip.internal-proxies=10\\.0\\.\\d{1,3}\\.\\d{1,3}
  • The value framework uses Spring’s ForwardedHeaderFilter instead of Tomcat’s valve, which suits servers other than Tomcat.
  • With nothing configured (the default is none unless Spring Boot detects a cloud platform), an app behind TLS termination (TLS decrypted at the proxy) generates http:// links and wrong OAuth2 redirect URIs, and Secure cookies aren’t set.

7. Production settings

Key idea: Nginx’s defaults suit static websites, not necessarily APIs. Six things to revisit: timeouts, keepalive, buffering, retries, body size and upstream DNS. Logs should carry $request_id and $upstream_response_time (details in the WebSocket block below).

Setting Default Problem What to do
Timeouts connect, read and send all 60s read is measured between two reads, not over the whole response: a backend trickling one byte every 50 seconds keeps the connection forever connect 1–2s, read per the API’s SLA, slow endpoints in their own location
Upstream keepalive none a new TCP connection per request: tens of thousands of TIME_WAIT sockets, ports run out keepalive N; + proxy_http_version 1.1; + proxy_set_header Connection "";
Buffering proxy_buffering on breaks Server-Sent Events and streaming: the client gets big chunks proxy_buffering off; or the backend sends X-Accel-Buffering: no
Retries proxy_next_upstream error timeout; since 1.9.13, no retry of POST, PATCH, LOCK once sent, unless you add non_idempotent keep the default, cap tries and time
Body client_max_body_size 1m a 3 MB upload gets 413 from Nginx, backend logs are empty set it per location
Upstream DNS resolved once, when the config is read 502 after a container is recreated with a new IP see the details block below
Each default in detail: timeouts, keepalive, buffering, upstream DNS
  • Timeouts: the three are proxy_connect_timeout, proxy_read_timeout and proxy_send_timeout. Keep connect short because inside a data centre a connection takes a few milliseconds. Intentionally slow endpoints (exports, reports) are better moved to async jobs and Idempotency-Key.
  • Keepalive: the TIME_WAIT pile-up shows at a few thousand requests per second. Before 1.29.7 Nginx spoke HTTP/1.0 to upstreams by default (connection closed after each request); since 1.29.7 the default is 1.1, but writing it out still helps whoever reads the config and whatever machine runs an older build. Why TIME_WAIT sits on the side that closes first and only exhausts ports on the calling side is in Networking for backend engineers.
  • Buffering: Nginx reads the backend’s response quickly into buffers (spilling to temp files if needed) and feeds it to the client at the client’s pace, freeing the backend early. Great for normal APIs; turn it off only on streaming locations.
  • Body: the upload gets 413 Request Entity Too Large; give uploads a dedicated location with a larger limit. client_max_body_size (default 1m) must be at least Spring’s spring.servlet.multipart.max-file-size/max-request-size, or Nginx answers 413 before the request reaches Spring. Spring’s limits and storing files safely are in file upload in Spring MVC.
  • DNS: open-source Nginx resolves host names in proxy_pass/upstream when it reads the config, and won’t start if a name doesn’t resolve (host not found in upstream).
WebSocket, the shared-header include trap, and logs for observability

WebSocket needs the upgrade headers forwarded and a read timeout longer than the ping interval:

location /ws/ {
    proxy_pass http://backend;
    # this location has its own proxy_set_header, so repeat the common ones
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        "upgrade";
    proxy_read_timeout 1h;
}

You can’t include the common header file here if it contains Connection "": both Connection lines would be sent. I also tried including a file with proxy_http_version into a location that sets proxy_http_version itself: Nginx refused to start with "proxy_http_version" directive is duplicate. Headers don’t get a duplicate error like that, which is all the more reason to watch them.

Observability. Log JSON with $request_id, $upstream_addr, $upstream_status, $request_time (total time) and $upstream_response_time (backend time). The gap between the last two tells you whether the slowness is on the network to the client or in the backend. Pass X-Request-Id (or better, traceparent) to the backend to link it with application logs; see Distributed tracing with OpenTelemetry.

Exercises

Exercise 1: Route by service name, as in the course’s URL exercise

Task (paraphrased): a gateway sits in front of user-service and order-service. Public URLs look like /user-service/v1/users/34 and /order-service/v1/orders/145; each service only knows /v1/.... Write the Nginx config.

Solution to Exercise 1: a gateway that strips the prefix per service

snippets/proxy-headers.conf, shared:

proxy_set_header Connection        "";
proxy_set_header Host              $host;
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

The gateway config:

upstream user_service  { server user-service:8080;  keepalive 16; }
upstream order_service { server order-service:8080; keepalive 16; }

server {
    listen 80;
    absolute_redirect off;
    proxy_http_version 1.1;
    include /etc/nginx/snippets/proxy-headers.conf;

    location /user-service/ {
        proxy_pass http://user_service/;            # strip: /user-service/v1/users/34 -> /v1/users/34
        proxy_set_header X-Forwarded-Prefix /user-service;
        include /etc/nginx/snippets/proxy-headers.conf;   # because this location sets its own header
    }

    location /order-service/ {
        proxy_pass http://order_service/;
        proxy_set_header X-Forwarded-Prefix /order-service;
        include /etc/nginx/snippets/proxy-headers.conf;
    }

    location / { return 404; }
}

Test run (nginx -t passes, then real calls): /user-service/v1/users/34 reaches user-service as /v1/users/34 with X-Forwarded-Prefix: /user-service; /order-service/v1/orders/145 reaches order-service as /v1/orders/145; /other gets 404.

Why it works:

  • Both the location and the proxy_pass URI end in /, so the prefix is stripped cleanly.
  • Each location sets its own header (X-Forwarded-Prefix), so the common headers have to be included again there (the inheritance rule from section 6).
  • location / { return 404; } rejects any path that belongs to no service instead of quietly routing it somewhere.
  • A senior would also question whether public URLs should expose service names at all (see the trade-off in Advanced REST API design).

Task: TLS terminates at Nginx; Spring Boot behind it speaks HTTP. After logging in, users are redirected to http://shop.vn/home and the browser shows a warning. Find the cause and fix it.

Solution to Exercise 2: X-Forwarded-Proto and forward-headers-strategy

The cause: Spring builds the redirect URL from request.getScheme(), which is http because the connection from Nginx is plain HTTP.

  • Fix both ends: Nginx sends proxy_set_header X-Forwarded-Proto $scheme; (and Host $host); Spring Boot enables server.forward-headers-strategy=native with internal-proxies matching Nginx’s IP.
  • To verify, call an endpoint that returns request.getScheme() and getServerName() through the proxy; you should see https and shop.vn.
  • Add HSTS at Nginx (add_header Strict-Transport-Security "max-age=31536000" always;) so browsers never try http:// again.

Exercise 3: Read the config, predict the result

Given:

location /static/ { root /var/www; }
location ~* \.(png|jpg)$ { root /var/cdn; }
location /api { proxy_pass http://be:8080/; }

Where do GET /static/logo.png and GET /api/v2/products go?

Solution to Exercise 3: regex beats prefix, and the trailing-slash bug

I ran it, and it behaves exactly as below.

  • /static/logo.png matches the prefix /static/, but the regex \.png$ matches too and the regex wins, so the file is read from /var/cdn/static/logo.png. Fix it with location ^~ /static/.
  • /api/v2/products matches location /api, /api is replaced by /, and the backend receives //v2/products. Fix it with location /api/ (keep proxy_pass http://be:8080/ to strip the prefix, or drop its trailing slash to keep it).

Remember

  • Can you state the order in which Nginx picks a location, and why a .png regex can steal requests from location /api/?
  • Can you predict the path the backend gets for any location / proxy_pass pair, including //orders and /apiv2 becoming /v2?
  • Can you explain why the first X-Forwarded-For entry can’t be trusted, and configure it correctly in both Nginx and Spring Boot?
  • Do you know the proxy_set_header inheritance trap and how include avoids it?
  • Can you list the defaults to change for production: timeouts, keepalive, buffering, retries, body size, upstream DNS?

Interview questions

Junior

How is a reverse proxy different from a forward proxy?

A forward proxy sits on the client side and goes out on the client’s behalf; a reverse proxy sits on the server side and takes requests in on behalf of the services.

With a forward proxy (a corporate proxy), the destination only sees the proxy. With a reverse proxy, clients only see one address. Reverse proxies terminate TLS, route by path or host, balance load, compress, cache, rate-limit and hide the internal layout.

Why does a backend behind a proxy need X-Forwarded-For and X-Forwarded-Proto?

Because the connection to the backend comes from the proxy: the source IP is the proxy’s, and the scheme is HTTP if TLS ended at the proxy.

Without these headers, logs and rate limits see only the proxy’s IP, and links and redirects the app generates use the wrong http://. X-Forwarded-For carries the chain of IPs, X-Forwarded-Proto the scheme the client used.

Mid

What’s the difference between location /api/ { proxy_pass http://be:8080; } and proxy_pass http://be:8080/;?

Without a URI the path is unchanged (/api/orders); with the URI / the part matching the location is replaced, so the backend gets /orders.

They have to go together: if the location ends in a slash, so must the URI, or you get //orders, or /apiv2 turning into /v2.

In what order does Nginx choose a location?

It remembers the longest matching prefix, stops on an exact = match or if that prefix has ^~, then the first regex matching in file order wins; if none matches, it uses the remembered prefix.

The consequence: a regex catching .png can steal requests from location /api/.

What does upstream keepalive need, and why bother?

keepalive N in the upstream, proxy_http_version 1.1 and proxy_set_header Connection "", so connections are reused instead of opened per request.

Connection "" stops Nginx from sending Connection: close. You skip the TCP (and TLS, if any) handshake on every request, which cuts latency and avoids tens of thousands of TIME_WAIT sockets exhausting ports under load.

You put SSE behind Nginx and clients receive events in big bursts. Why?

Because proxy_buffering is on by default, so Nginx collects the response in buffers before sending it.

Turn it off for that location (proxy_buffering off;) or have the backend send X-Accel-Buffering: no. Also raise proxy_read_timeout above the heartbeat interval so the connection isn’t cut after 60 seconds.

Senior

How do you get the real client IP through several layers of proxies?

Read X-Forwarded-For from right to left, skip the IPs that belong to your proxies; the first untrusted one is the client.

  • Never take the first entry, because the client controls it. The outermost layer should overwrite the header with $remote_addr.
  • Nginx: set_real_ip_from with real_ip_recursive on. Spring Boot: forward-headers-strategy=native and internal-proxies set to the load balancer’s range.
  • Behind an L4 load balancer there are no HTTP headers, so use the PROXY protocol.
Should the proxy retry requests when a backend fails?

Connection errors before the request was sent are safe to retry; timeouts after sending are only safe for idempotent requests.

  • Nginx doesn’t retry POST/PATCH by default because the backend may already have processed them.
  • Even for GET, cap the number of tries and the total retry time: when the whole cluster is overloaded, retries multiply the load and spread the outage.
  • Retries should live in one layer with backoff, not in the client, the gateway and the service all at once.
After redeploying a backend in Docker, Nginx returns 502 until it’s reloaded. Explain and fix.

Open-source Nginx resolves upstream host names once at start or reload and keeps that IP; the new container has a different IP, so connections to the old one fail.

Fix: use a name with a stable IP (a Kubernetes Service), or a resolver plus a variable in proxy_pass so it re-resolves per TTL, or the resolve parameter in upstream (since 1.27.3). If you keep the static approach, add a reload to the deploy process.

When would you use an API gateway instead of Nginx?

When you need API-aware logic that’s awkward in static Nginx config; if all you need is one entry point with TLS, path routing and per-IP rate limits, Nginx is simpler.

API-aware logic means JWT or API-key auth for partners, per-client quotas, header-based canaries, frequent route changes without reloads, many teams editing routes. Nginx has fewer moving parts. Either way, keep business logic out of the gateway.

Useful phrases for interviews

  • “The reverse proxy is our single entry point: it terminates TLS and routes by path prefix.”
  • “If the location ends with a slash, the proxy_pass URI should too; otherwise the path gets mangled.”
  • “We strip the prefix at the proxy and pass X-Forwarded-Prefix so the service can still build correct links.”
  • “We only trust forwarded headers from our own load balancer’s address range.”
  • “The client IP is the first untrusted address when you read X-Forwarded-For from the right.”
  • “We turned on upstream keepalive, which removed thousands of TIME_WAIT sockets.”
  • “Retries at the proxy are limited to idempotent requests, and they’re capped so they can’t amplify an outage.”
  • “The gap between request time and upstream response time tells us whether the backend or the network is slow.”

References


Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.