“Connection reset”, “connect timed out”, a request that hangs for 15 minutes after a quiet night: a backend developer has to read these network errors on their own. This post follows one API call from DNS to the last byte, collects the TCP details that cause incidents, and shows how to narrow a fault down with curl, dig, ss, tcpdump and openssl. The numbers come from my own runs (macOS, JDK 21.0.13, curl to example.com).
1. What happens when you call an API
Key idea: before the first byte of the request can leave, the client has to find the IP (DNS), open a connection (TCP) and agree on encryption (TLS). Each step costs about one round trip (RTT), so reusing connections is the cheapest speed-up there is.
sequenceDiagram
participant C as Client
participant D as DNS resolver
participant S as Server
C->>D: A/AAAA api.example.com?
D-->>C: 104.20.23.154 (TTL 300)
C->>S: SYN
S-->>C: SYN-ACK
C->>S: ACK + TLS ClientHello (SNI, ALPN, key share)
S-->>C: ServerHello + Certificate + Finished
C->>S: Finished + GET /orders
S-->>C: 200 OK (connection kept for reuse)
Measured with curl -w (every timestamp counts from the start, not per phase):
| Call | DNS done | TCP done | TLS done | First byte (TTFB) | Total |
|---|---|---|---|---|---|
| New connection, DNS cached by the OS | 1.9 ms | 47.9 ms | 106.6 ms | 166.0 ms | 166.1 ms |
| New connection, cold DNS | 32.2 ms | 76.5 ms | 159.6 ms | 217.0 ms | 217.2 ms |
| Second request on the same connection | 0 | 0 | 0 | 51.7 ms | 52.1 ms |
RTT is about 48 ms: TCP takes 1 RTT, TLS 1.3 one more, the request one more. Keep-alive cuts 166 ms to 52 ms, which is why HTTP clients and JDBC both use a connection pool (a set of connections kept open), see Connection pooling and HikariCP.
DNS maps names to IPs; each answer has a TTL (how long it may be cached). The path: in-app cache → OS resolver → recursive resolver → root → TLD → authoritative server.
DNS in detail: configuring the JVM TTL, stale cache, Kubernetes ndots
- Set the JVM TTL by editing
$JAVA_HOME/conf/security/java.security, or callSecurity.setProperty("networkaddress.cache.ttl", "10")before the first lookup.-Dsun.net.inetaddr.ttlis the old, unofficial way. - JDK 21’s
java.securityalso hasnetworkaddress.cache.stale.ttl(default 0): it lets you keep using a stale answer while the resolver is unreachable. - Kubernetes sets
ndots:5in pods: a name with fewer than 5 dots is first tried with every search domain appended, so one external call becomes 4–5 lookups. Use a trailing dot (api.stripe.com.) or lowerndotsindnsConfig.
The TLS 1.3 handshake: SNI, ALPN, 0-RTT and what openssl showed me
- ClientHello carries a key share (an ephemeral ECDHE key) from the start, so TLS 1.3 needs 1 RTT; TLS 1.2 needs 2.
- SNI: the host name, sent in clear so the server picks the right certificate; without it you easily hit
hostname mismatch. - ALPN: chooses
h2orhttp/1.1inside the handshake. - The client checks the chain up to a trusted CA, the validity dates, and that the host name matches the SAN.
- 0-RTT (resuming with a session ticket) sends data in the first flight but can be replayed: idempotent requests only.
openssl s_client -connect example.com:443 -servername example.com -briefprinted:TLSv1.3, cipherTLS_AES_256_GCM_SHA384, key-exchange groupX25519MLKEM768(a classical plus post-quantum hybrid),Verification: OK.
Encryption, forward secrecy and why only TLS 1.2/1.3 should be enabled are covered in the web and HTTP post.
2. TCP: what a backend engineer must know
Key idea: TCP is a reliable pipe between two IP:port pairs. Six details cause most incidents: ephemeral ports, TIME_WAIT, Nagle, the backlog, and closing with FIN or RST.
| Concept | Meaning | When it bites |
|---|---|---|
| Ephemeral port | The source port the client picks, 32768–60999 by default on Linux | Many short connections to the same destination: Cannot assign requested address |
| TIME_WAIT | The side that closes first keeps the socket for 2×MSL (60 s on Linux) so stray packets can’t leak into a new connection | Tens of thousands of client sockets when there’s no keep-alive |
| Nagle | Coalesces small writes, waits for the previous ACK | Meets delayed ACK: tens of ms of delay. Fix with TCP_NODELAY |
| Backlog | Queue of handshaken connections the app hasn’t accept()ed yet |
When full, Linux drops SYNs and clients connect slowly or time out |
| FIN | Polite close: “I’m done sending” | The peer’s read() returns -1 |
| RST | Abort: “this connection doesn’t exist / drop it” | The peer gets Connection reset |
Port and file descriptor limits are covered in the I/O model and epoll.
Backlog, Nagle and delayed ACK in detail
- Linux has two queues: the SYN queue (handshake in progress) and the accept queue (done, waiting for
accept()). Accept queue size =min(backlog passed to listen(), net.core.somaxconn);somaxconndefaults to 4096 since Linux 5.4 (128 before). Tomcat passesserver.tomcat.accept-count(default 100). - Inspect:
ss -lnton a LISTEN socket showsRecv-Q= connections waiting to be accepted andSend-Q= backlog size. Overflow counter:nstat -az TcpExtListenOverflowsornetstat -s | grep -i listen. - A full accept queue usually means the app can’t keep up (no free workers, long GC); a bigger backlog only makes clients wait longer.
- Nagle + delayed ACK: the client writes the header and the body as two small writes. Nagle holds the second until an ACK arrives; the peer delays its ACK (up to about 40 ms on Linux). Every request gets a fixed extra delay. With hand-written
Socketcode, callsetTcpNoDelay(true)and write the data in one go.
3. Connect timeout, read timeout and common errors
Key idea: each network error points to a specific layer. Reading the error name correctly already halves the search area.
My JDK 21 program reproduces each case (code in Exercise 2):
| Case | Java reports | Measured | Meaning |
|---|---|---|---|
| Nothing listens on the port | ConnectException: Connection refused |
7 ms | Host reachable, the kernel answered the SYN with RST (or a firewall REJECT) |
SYN into a black hole (10.255.255.1) |
SocketTimeoutException: Connect timed out |
1001 ms (1 s timeout) | Nobody answers: firewall DROP, dead host, wrong IP, full backlog |
| Server accepts but never replies | SocketTimeoutException: Read timed out |
504 ms (500 ms timeout) | Connection is fine, the peer is slow or stuck |
Server closes with SO_LINGER=0 |
SocketException: Connection reset |
1 ms | The peer sent RST |
| Server closes normally | read() returns -1; writing on gives Broken pipe |
— | FIN, then RST when we try to write |
A connect timeout covers opening TCP (HTTP clients usually include TLS); set it to 1–2 seconds. A read timeout covers the gap between two received chunks; base it on the peer’s p99. Without a connect timeout, Linux retries the SYN (tcp_syn_retries=6) for about 2 minutes before giving up. Choosing values and retries is in Timeouts, retries and circuit breakers.
4. L4 and L7 load balancers, NAT and MTU
Key idea: between client and server there are usually load balancers, NAT and firewalls. Each keeps connection state with its own idle timeout, and once it forgets a connection it drops packets or answers with RST.
| L4 (TCP/UDP) | L7 (HTTP) | |
|---|---|---|
| Sees | IP, port | URL, headers, cookies, each HTTP/2 request |
| Balances per | Connection | Request |
| TLS | Passthrough or terminate | Terminates to read HTTP |
| Examples | AWS NLB, kube-proxy (ClusterIP Service) |
AWS ALB, Nginx, Envoy, ingress |
Because L4 balances per connection, long-lived connections (HTTP/2, gRPC, a JDBC pool) stick to one backend; this is the gRPC-on-Kubernetes trap from the synchronous communication post.
NAT and conntrack. NAT (rewriting a private IP:port to a public IP) and Linux conntrack (the connection-tracking table used by NAT and kube-proxy) keep one entry per connection. Idle longer than the timeout and the entry is gone: later packets are dropped or get an RST. When the table is full the kernel logs nf_conntrack: table full, dropping packet.
MTU (the largest packet a hop can carry): 1500 bytes on Ethernet, leaving an MSS of 1460 bytes after IPv4 + TCP headers. VPNs and overlays (VXLAN adds 50 bytes) make it smaller. If a firewall blocks ICMP “fragmentation needed”, Path MTU Discovery breaks: small requests work, large responses hang.
5. HTTP/1.1, HTTP/2, HTTP/3 at the transport level, and mTLS
Key idea: the HTTP version decides how many connections you need and how long they live. mTLS adds a step where the server checks the client’s certificate.
- HTTP/1.1: one request at a time per connection, so the client pools several connections per host.
- HTTP/2: many streams on one TCP connection, usually one connection per host. One lost packet stalls every stream (TCP head-of-line blocking), and an L4 LB can’t spread the load.
- HTTP/3: runs on QUIC (UDP), merges the transport and TLS handshakes, and identifies connections by a connection ID so switching networks doesn’t break them. Firewalls often block UDP, so clients always keep a TCP fallback.
The full comparison table is in the web and HTTP post; REST vs gRPC over HTTP/2 is in synchronous communication.
mTLS (mutual TLS): the server also demands a client certificate signed by an internal CA. It’s used between services in a zero-trust style (nobody is trusted just for being on the same network). In Java, the keystore holds your own certificate and private key, the truststore holds the CAs you trust; Spring Boot 3.1+ groups them into spring.ssl.bundle. A service mesh (Istio, Linkerd) issues and rotates certificates for you.
6. The debugging toolkit
Key idea: work bottom-up: does the name resolve (dig), can TCP/TLS be opened and how long does it take (curl, openssl), what state are the sockets in (ss), and when you’re still blind, look at the real packets (tcpdump).
| Question | Command |
|---|---|
| Which IP, how much TTL left | dig +noall +answer api.example.com, dig +trace, dig @10.0.0.2 name (ask a specific resolver) |
| Which phase is slow | curl -o /dev/null -sS -w '...time_connect...' URL, curl -v --trace-time URL |
| Skip DNS, hit one IP | curl --resolve api.example.com:443:10.0.1.5 https://api.example.com/ |
| Certificate, TLS version, ALPN | openssl s_client -connect host:443 -servername host -alpn h2 |
| Socket states | ss -tan state time-wait | wc -l, ss -tnp dst :5432, ss -ti (RTT, retransmits); macOS: netstat -an |
| Who sends RST, do SYNs leave | sudo tcpdump -i any -nn 'port 5432 and (tcp[tcpflags] & (tcp-syn|tcp-rst) != 0)' |
Run tcpdump -w file.pcap and open it in Wireshark: filter tcp.flags.reset == 1 to find who sent the RST, tcp.analysis.retransmission to find resent packets.
I didn’t run ss (not on macOS), tcpdump (needs sudo) or the Linux kernel settings; try them again on a Linux box.
Exercises
Exercise 1: Read a curl -w timing breakdown
Run the command below three times, then answer: what’s the RTT to the server, how many RTTs does TLS take, how long does the server spend? If time_connect is 3 seconds, time_appconnect − time_connect is 2 seconds, or time_starttransfer − time_appconnect is 2 seconds, what do you suspect in each case?
curl -o /dev/null -sS -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} new=%{num_connects}\n' https://example.com/
Solution to exercise 1
My result: dns=0.001857 connect=0.047921 tls=0.106648 ttfb=0.165987 total=0.166090 new=1.
The timestamps are cumulative, so subtract:
| Phase | How | Result |
|---|---|---|
| DNS | dns |
1.9 ms (cached by the OS; 32 ms when cold) |
| TCP handshake | connect − dns |
46 ms ≈ 1 RTT |
| TLS 1.3 | tls − connect |
59 ms ≈ 1 RTT + a little CPU |
| Request to first byte | ttfb − tls |
59 ms ≈ 1 RTT + server time |
| Body download | total − ttfb |
0.1 ms (small page) |
RTT is about 46–48 ms, so the server (a CDN) spends only a few ms. The three abnormal cases:
- connect = 3 s: a SYN was lost and resent (Linux waits 1 s, then backs off), the server backlog is full, or a firewall drops packets.
tcpdumpwill show several SYNs with no SYN-ACK. - TLS = 2 s: the server lacks CPU for handshakes, or the client’s OCSP/CRL check is slow. Use
openssl s_clientto take the app out of the picture. - ttfb − tls = 2 s: the network is fine, the application is slow. Switch to distributed tracing.
new=0 with the first three timestamps at zero means the request rode an existing connection (52 ms in my run).
Exercise 2: Tell “connection reset” from “connect timeout”
Write a Java program that reproduces the five cases from section 3. Then diagnose: (a) a service reports Connect timed out to the DB after a security group change; (b) a service reports scattered Connection reset errors in the morning when calling an internal API through an ALB.
Solution to exercise 2: code and diagnosis
// The core part; the other three cases (refused, read timeout, FIN) follow the same shape
try (Socket s = new Socket()) { // SYN into a black hole
start(); s.connect(new InetSocketAddress("10.255.255.1", 80), 1000);
} catch (IOException e) { System.out.println("connect : " + e + " after " + ms()); }
try (ServerSocket server = new ServerSocket(0, 50, InetAddress.getLoopbackAddress())) {
Thread srv = Thread.ofPlatform().start(() -> { // server aborts the connection with RST
try { Socket c = server.accept(); c.setSoLinger(true, 0); c.close(); } catch (Exception ignored) {}
});
try (Socket s = new Socket()) {
s.connect(server.getLocalSocketAddress(), 1000); srv.join(); Thread.sleep(100);
start(); s.getInputStream().read();
} catch (IOException e) { System.out.println("rst : " + e + " after " + ms()); }
}
// Read timeout: the server accept()s and sleeps, the client calls setSoTimeout(500) and read().
// FIN: the server close()s normally, the client's read() returns -1, a later write gives Broken pipe.
Output on JDK 21.0.13 (macOS):
refused : java.net.ConnectException: Connection refused (connect failed) after 7 ms
connect : java.net.SocketTimeoutException: Connect timed out after 1001 ms
read : java.net.SocketTimeoutException: Read timed out after 504 ms
rst : java.net.SocketException: Connection reset after 1 ms
fin : read returned -1
fin : write after FIN -> java.net.SocketException: Broken pipe
(a) Connect timeout after a security group change. Security groups drop packets rather than answer with RST, so the client sees a timeout, not a refusal. Check: does dig give the right DB IP; nc -vz -w 2 db-host 5432 from the app host itself; tcpdump shows SYNs leaving with no SYN-ACK; then review the inbound rules and NACLs. If it were Connection refused, the network would be open and the DB just not listening on that port.
(b) Scattered morning resets through an ALB. Scattered and after an idle period points to stale keep-alive connections: the ALB closes idle connections after 60 seconds while the client pool still reuses them. Confirm: only the first request on each connection fails, and tcpdump shows the RST coming from the LB. Fix: pool idle eviction shorter than the ALB idle timeout, one retry for idempotent requests. Resets under heavy load point instead to backend crashes, deployments that don’t drain connections, or a full conntrack table.
Remember
- Explain which RTTs a fresh HTTPS call costs and why keep-alive removes about two thirds of the time.
- Know how long the JVM caches DNS, where to configure it, and why pooled connections don’t see DNS changes.
- Tell refused, connect timeout, read timeout and reset apart, and name the layer behind each.
- Know how LB/NAT idle timeouts affect pools, and set
maxLifetimeand keepalive accordingly. - Use
curl -w,dig,openssl s_client,ssandtcpdumpto narrow down a network fault.
Interview questions
Junior
What happens when your service calls https://api.example.com/orders?
DNS lookup to get the IP, TCP handshake (1 RTT), TLS handshake (1 RTT with TLS 1.3), send the request and wait for the response (1 RTT + server time); the connection is kept for next time.
Measured with a 48 ms RTT: 166 ms on a new connection, 52 ms on a reused one.
What’s the difference between a connect timeout and a read timeout?
A connect timeout limits how long opening the connection may take; a read timeout limits how long you wait for data on an open connection.
Set the connect timeout to 1–2 seconds and the read timeout from the peer’s p99, plus an overall deadline if you retry.
Mid
What is TIME_WAIT and when does it become a problem?
The state where the side that closed first keeps the socket for 2×MSL (60 seconds on Linux) so late packets can’t leak into a new connection; it hurts when one client opens many short connections to the same destination.
There are only about 28,000 ephemeral ports per destination; the symptom is Cannot assign requested address. Fix it with keep-alive and connection pooling.
How do L4 and L7 load balancers differ, and when do you pick each?
L4 forwards TCP/UDP per connection without reading the content; L7 understands HTTP and balances and routes per request.
L4 is fast, protocol-agnostic and can pass TLS through. L7 routes by path/header, retries, and spreads HTTP/2 and gRPC load correctly.
The logs show Connection reset by peer. Which causes come to mind?
The peer, or a box in the middle, sent an RST: usually a reused connection that was closed for being idle, a peer process that died, or an LB/NAT that forgot the connection.
Check whether errors follow idle periods or coincide with deployments, and use tcpdump to see which IP sent the RST. Make the pool’s idle timeout shorter than the peer’s and retry idempotent requests.
Senior
The DB failed over, but the Java service kept failing for minutes. Why?
Open pooled connections still point to the old IP because DNS is only consulted for new connections, on top of the JVM DNS cache and the record’s TTL.
You need short DNS TTLs, a low networkaddress.cache.ttl, socket timeouts to detect dead connections, and maxLifetime so connections get replaced regularly.
The first query after a quiet night hangs about 15 minutes before failing. How do you investigate?
Suspect a NAT, firewall or LB idle timeout straight away: the idle connection was dropped from a state table, packets are discarded, and TCP retransmits until it gives up.
15 minutes matches tcp_retries2=15. Check idle timeouts on the path (NAT Gateway, NLB 350 seconds); tcpdump shows retransmits with no ACK. Fix: maxLifetime shorter than that timeout, the pool’s keepaliveTime, and socket timeouts.
Useful phrases for interviews
- “A fresh HTTPS call costs about three round trips before the first byte.”
- “We ran out of ephemeral ports because every request opened a new connection.”
- “The reset came from the load balancer, not from our service.”
- “Our SYNs were black-holed by the security group, so we saw connect timeouts, not refusals.”
- “The NAT gateway’s idle timeout is shorter than our pool’s max lifetime, so idle connections died silently.”
- “DNS is only resolved when a connection is opened, so the pool kept talking to the old primary.”
- “We terminate TLS at the ingress and use mTLS inside the mesh.”
- “HTTP/2 still suffers TCP-level head-of-line blocking on lossy networks.”
- “Blocking ICMP broke path MTU discovery, so large responses hung.”
- “A packet capture showed who sent the RST.”
Self-compiled notes from official documentation and hands-on practice.