Operating systems for backend engineers series: 1. Kernel, processes and threads · 2. Virtual memory and the page cache · 3. I/O models and epoll · 4. File systems and the shell
A Java pod gets OOMKilled while its heap is only half full; free -m shows 200 MB “free” and nobody cares; two threads incrementing two unrelated variables run three times slower. All three come from how the OS and the CPU manage memory. After this post you can explain each one, and size the memory limit for a Java service. Read part 1 (processes, kernel mode) first.
1. Virtual memory
Key idea: every process sees its own address space, as if it had the whole memory to itself. The MMU (the address translator inside the CPU) turns virtual addresses into physical ones on every access, using page tables the kernel builds.
Without virtual memory, process A writes x = 1 at 0x2000, process B writes x = 2 at the same spot, and A reads back B’s value. With it, the same address 0x2000 in A and B points to two different frames (physical page slots). Memory is split into pages, usually 4 KB.
flowchart LR
A[Process A: virtual address 0x2000] --> PTA[A's page table]
B[Process B: virtual address 0x2000] --> PTB[B's page table]
PTA --> F7[Physical frame 7]
PTB --> F3[Physical frame 3]
- Each page table entry has attribute bits: is it in RAM (present), writable, accessible from user mode, executable (NX). That lets the kernel block writes to code, running code from data, and user mode reading kernel memory.
- The TLB (translation lookaside buffer) is a small CPU cache of recent translations, so the page table isn’t walked on every access.
- Huge pages (2 MB or 1 GB) cut the number of TLB entries needed by 512×.
Multi-level page tables, TLB limits and transparent huge pages
On x86-64 the page table has 4 levels (5 on newer CPUs), so one translation can cost up to 4 memory reads. The TLB only has tens to a few thousand entries. With 4 KB pages, 64 TLB entries cover just 256 KB; an 8 GB heap accessed randomly misses the TLB constantly.
Page faults
Key idea: virtual memory is handed out lazily. malloc or mmap of 1 GB only reserves space; the first touch of a page with no frame makes the MMU raise a page fault, and the kernel assigns a frame and resumes.
| Kind | When | Cost |
|---|---|---|
| Minor fault | no disk: a fresh frame, or the page is already in the page cache | about a microsecond |
| Major fault | disk read: a swapped-out page, or the first read of an mmap’d file | tens of µs (NVMe) to ms (HDD) |
Measure with ps -o min_flt,maj_flt -p <pid> or pidstat -r. Constant major faults on a service mean it’s short on RAM.
Starting a JVM with no page faults on the heap
2. Needing more than RAM: overcommit, swap and the OOM killer
Key idea: Linux lets processes ask for more than the RAM it has (overcommit), because most requested memory is never touched. When RAM really runs out, the kernel drops page cache, pushes pages to swap, and finally the OOM killer kills a process.
The slides say “virtual memory = physical memory + swap”. That’s misleading.
vm.overcommit_memory: 0 is a heuristic guess (default), 1 always says yes, 2 is strict. When RAM runs out, the kernel reclaims roughly in this order:
- Drop clean page cache pages (they can be reread from disk) and write dirty pages back to disk.
- Push anonymous pages (heap, stack, no file behind them) to swap if there is any.
vm.swappiness(default 60) tunes the balance between step 1 and step 2. - If that still isn’t enough, the OOM killer picks the process with the highest
oom_score(roughly the one using the most RAM) and kills it with SIGKILL.
In a container the limit is the cgroup’s memory.max (a cgroup is a group of processes the kernel limits together; Kubernetes’ resources.limits.memory). Going over it runs the OOM killer inside that cgroup, and the pod shows OOMKilled with exit code 137.
java.lang.OutOfMemoryError: Java heap space |
Container OOMKilled (exit 137) |
|
|---|---|---|
| Who detects it | the JVM: heap full after GC | the kernel: cgroup RSS over memory.max |
| Stack trace, heap dump | yes (-XX:+HeapDumpOnOutOfMemoryError) |
no: SIGKILL, the JVM can’t react |
| Typical cause | a heap leak, -Xmx too small |
heap plus off-heap memory over the limit |
| Fix | analyze the heap dump | shrink the heap relative to the limit, measure with NMT |
3. The JVM in a container: the heap is only one part
Key idea: a JVM’s RSS is the heap plus several off-heap regions, usually a few hundred MB. By default the JVM takes 25% of the memory limit as the max heap; to size it properly, measure with Native Memory Tracking (NMT).
- Heap (
-Xmx). - Metaspace: class metadata. A large Spring Boot app can use 100–200 MB.
- Code cache: JIT-compiled code, up to 240 MB by default.
- Thread stacks: each thread reserves 1 MB of virtual space and usually touches a few hundred KB.
- Direct buffers (
ByteBuffer.allocateDirect, Netty, the Kafka client) and mmap’d regions. - GC data structures (G1’s remembered sets and card table: a few percent of the heap) and
mallocmemory (glibc arenas, native libraries).
Verifying the 25% (MaxRAMPercentage=25) in a -m 512m container (JDK 21):
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
size_t MaxHeapSize = 134217728 # 128 MB = 25% of 512 MB
$ java -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep MaxHeapSize
size_t MaxHeapSize = 402653184 # 384 MB
Where to put this flag in the image, along with a multi-stage Dockerfile and how to read OOMKilled, is in Docker for Spring Boot.
In the NMT run below, the heap has committed only 8 MB but RSS is already 152 MB. What matters is committed and RSS; the nearly 2 GB reserved (address space only held) has nothing to do with the memory limit.
NMT experiment: 64 MB of direct buffers and 200 threads in a 512 MB container
The test program has a 128 MB max heap, allocates 64 MB of direct buffers and starts 200 sleeping threads, in a -m 512m container:
$ java -Xmx128m -XX:NativeMemoryTracking=summary NativeMem &
maxHeap=123 MB, heapCommitted=7 MB
VmRSS: 155644 kB
Threads: 213
$ jcmd <pid> VM.native_memory summary scale=MB
Total: reserved=1967MB, committed=118MB
- Java Heap (reserved=128MB, committed=8MB)
- Class (reserved=1024MB, committed=0MB)
- Thread (reserved=425MB, committed=22MB)
- Code (reserved=242MB, committed=7MB)
- Other (reserved=64MB, committed=64MB)
- Metaspace (reserved=64MB, committed=1MB)
“Other” at 64 MB is the direct buffers. 213 threads reserve 425 MB but commit only 22 MB.
4. The page cache and mmap
Key idea: the kernel keeps file contents you just read or wrote in RAM (the page cache) so the next access skips the disk. That’s why “free” RAM on Linux is almost always taken by the page cache, and that’s a good thing.
- Read: the kernel reads from disk into the page cache, then copies into the application’s buffer. Next time, if the page is still cached, no disk access.
- Write:
write()only modifies the page in the page cache and marks it dirty; the kernel writes it to disk a few seconds later, unless the application callsfsync().
$ free -m
total used free shared buff/cache available
Mem: 15890 6120 212 150 9558 9310
The column to read is available (an estimate of memory that can be handed out, including droppable page cache), not free. This machine still has about 9 GB to use.
Storage systems lean on the page cache:
- PostgreSQL sets
shared_buffersto about 25% of RAM and leaves the rest to the OS page cache (effective_cache_sizetells the planner so). - Kafka appends its log sequentially into the page cache and lets consumers read it back from there, barely touching the disk when consumers keep up (see Why Kafka is fast and durable). Consequence: don’t give a Kafka broker’s JVM a huge heap; RAM left to the page cache is worth more.
mmap maps a file straight into a process’s address space: reading a byte array in memory is reading the file’s page cache, with no read syscall and no copy. Java has FileChannel.map() returning a MappedByteBuffer.
mmap in Kafka and Lucene, and the price you pay
Kafka mmaps its index files; Lucene (Elasticsearch, OpenSearch) mmaps its segments, which is why Elasticsearch requires vm.max_map_count of at least 262144.
The price: an I/O error on an mmap’d region shows up as SIGBUS rather than an IOException, and a page evicted from the cache causes a major fault right in the thread serving a request.
5. CPU caches and the memory model
Key idea: RAM is about 100× slower than L1, so the CPU loads 64-byte cache lines and keeps copies per core. Another thread may not see a new value yet; the Java Memory Model defines when it is guaranteed to.
The slides on the “memory model” are about the cache levels between the CPU and RAM: per-core L1 and L2, shared L3.
A cache line is 64 bytes on x86 and most ARM servers (Apple M chips report 128 bytes). That’s why scanning an array sequentially is fast: one load brings in 8 long elements and the prefetcher predicts the next step. Walking a LinkedList or a scattered object graph is a cache miss at every step.
Hardware keeps the copies consistent with cache coherence, per cache line.
Latency of each memory level, cache coherence and store buffers
| Level | Typical size | Approx. latency |
|---|---|---|
| Registers | a few dozen values | under 1 ns |
| L1 (per core) | 32–128 KB | ~1 ns |
| L2 (per core or core cluster) | 256 KB – a few MB | ~3–5 ns |
| L3 (shared) | a few MB – tens of MB | ~10–20 ns |
| RAM | GB | ~80–100 ns |
| Random 4 KB read from NVMe SSD | ~20–100 µs |
When several cores hold copies of the same cache line, they can see different values (as the slides say). The coherence protocol (MESI and its variants) handles this: before a core writes to a line, the copies in other cores are invalidated.
But to run fast, each core also has a store buffer (a queue of writes not yet in the cache), and both the CPU and the JIT may reorder instructions. So a thread may not yet see a value another thread just wrote.
The Java Memory Model defines when a write by one thread is guaranteed to be visible to another: volatile, synchronized, final and the classes in java.util.concurrent insert memory barriers (instructions that forbid reordering across them) to create happens-before edges (see race conditions, volatile and deadlock).
False sharing
Key idea: coherence works per cache line, not per variable. Two threads writing different variables on the same line keep invalidating each other’s copy, as slow as fighting over one shared variable.
It’s like two people writing different sentences on the same sheet of paper: every time one writes, the other has to hand the sheet over. The fix: move the variables onto different lines with padding, or use @Contended.
LongAdder and @Contended in the JDK
Exercises
Exercise 1: Sizing the memory limit for a Spring Boot service
Task. A Spring Boot service (Tomcat, Hibernate, a Kafka client) under real load shows: about 400 MB of live heap data after a full GC; 160 MB of metaspace; about 250 threads; the Kafka client and WebClient’s Netty together use about 100 MB of direct memory. Choose -Xmx and the container memory limit, and explain.
Solution to Exercise 1: add up each part, get a 2 GiB limit
| Part | Estimate | Reason |
|---|---|---|
| Heap | 1,024 MB | about 2.5× live data so GC doesn’t run constantly |
| Metaspace | 160 MB | measured |
| Code cache | 80 MB | usually 50–100 MB once warm |
| Thread stacks | 125 MB | 250 threads × about 0.5 MB committed |
| Direct buffers | 100 MB | measured, capped with -XX:MaxDirectMemorySize=128m |
| GC structures (G1) | 60 MB | a few percent of the heap |
| malloc, native libraries, other | 100 MB | headroom for glibc arenas, zip, SSL |
| Total | about 1.65 GB |
Add about 20%: limit 2 GiB, -Xmx1g (or MaxRAMPercentage=50), request equal to limit. Verify with a load test, comparing NMT with the pod’s working set. If the working set keeps growing while the heap is flat, suspect an off-heap leak (direct buffers, a growing thread count, glibc arenas; try MALLOC_ARENA_MAX=2). Off-heap leaks often show up only after hours, which is what a soak test is for (Performance testing).
Exercise 2: Measuring false sharing
Task. Write a program with two threads, each incrementing its own volatile long 200 million times. Compare the variables sitting next to each other with 56 bytes of padding between them.
On real hardware (Apple M4) padding is about 3× faster; on a 2-vCPU VM there’s almost no difference.
Solution to Exercise 2: code, numbers on an Apple M4 and a 2-vCPU VM
public class FalseSharing {
static final long ITER = 200_000_000L;
static final class Packed { // a and b side by side: same cache line
volatile long a;
volatile long b;
}
static final class Padded { // 7 longs (56 bytes) push b to another line
volatile long a;
long p1, p2, p3, p4, p5, p6, p7;
volatile long b;
}
public static void main(String[] args) throws Exception {
for (int round = 0; round < 5; round++) {
Packed packed = new Packed();
long t1 = run(() -> { for (long i = 0; i < ITER; i++) packed.a++; },
() -> { for (long i = 0; i < ITER; i++) packed.b++; });
Padded padded = new Padded();
long t2 = run(() -> { for (long i = 0; i < ITER; i++) padded.a++; },
() -> { for (long i = 0; i < ITER; i++) padded.b++; });
System.out.printf("round %d: same line %5d ms | padded %5d ms%n", round, t1, t2);
}
}
static long run(Runnable r1, Runnable r2) throws InterruptedException {
Thread a = new Thread(r1), b = new Thread(r2);
long t0 = System.nanoTime();
a.start(); b.start(); a.join(); b.join();
return (System.nanoTime() - t0) / 1_000_000;
}
}
Results (JDK 21):
# macOS, Apple M4, 10 cores
round 0: same line 1232 ms | padded 367 ms
round 4: same line 1048 ms | padded 366 ms
# Linux container on a 2-vCPU VM
round 0: same line 928 ms | padded 1056 ms
round 2: same line 753 ms | padded 696 ms
round 4: same line 716 ms | padded 728 ms
Explanation.
- Real hardware: padding is about 3× faster because the two cores no longer fight over one cache line.
- 2-vCPU VM: almost no difference. Most likely the two vCPUs aren’t pinned to separate physical cores, or the hypervisor interleaves them on the same core, so there are never two real cores writing at once.
- Caveats: the JVM may reorder fields, so field padding isn’t guaranteed (
@Contendedis the official way). The Apple M4 reports a 128-byte cache line, so only 128 bytes of padding is certain. This is a hand-rolled benchmark without JMH; trust only the order of magnitude. - Lesson: false sharing only matters for variables written very often by many threads (counters, queue indexes); ordinary business code doesn’t need to care.
Remember
- Explain what the MMU, page tables and the TLB do, and how a minor fault differs from a major one.
- Explain why a large VSZ is harmless, and the order in which the kernel reclaims memory before the OOM killer steps in.
- Tell an
OutOfMemoryErrorfromOOMKilled, and explain why a JVM shouldn’t use swap. - List the parts of RSS outside the heap and size the memory limit for a Java service.
- Explain false sharing, and why
volatileis still needed despite cache coherence.
Interview questions
Junior
What is virtual memory and what problem does it solve?
Every process sees its own virtual address space; the MMU in the CPU translates it to physical addresses through page tables managed by the kernel.
It stops processes from overwriting each other’s memory, allows per-page read/write/execute permissions, and enables lazy allocation, shared pages (libraries, copy-on-write) and mapping files into memory.
free on Linux shows very little free memory. Should you worry?
Usually not: Linux uses idle RAM as page cache and drops it when processes need memory. Read the available column, not free.
Worry only when available is low, swap-in/out is constant (si, so in vmstat) or major faults are rising.
What is a page fault? How do minor and major faults differ?
A page fault happens when a process touches a virtual page that has no physical frame or isn’t allowed. A minor fault needs no disk; a major fault needs a disk read.
Minor: a fresh page, or the page is already in the page cache, costing microseconds. Major: a swapped-out page or an unloaded mmap’d file, costing tens of µs to ms. Constant major faults mean the machine is short on RAM.
Mid
Why is a JVM’s RSS larger than -Xmx?
The heap is only one part; a Spring Boot service typically has 300–600 MB outside the heap.
The JVM also has metaspace, the code cache, a stack per thread, direct buffers and mmap’d regions, GC data structures, and malloc memory from native libraries. Measure it with -XX:NativeMemoryTracking=summary and jcmd <pid> VM.native_memory summary.
How is an OutOfMemoryError different from a pod being OOMKilled?
OutOfMemoryError is the JVM reporting a full heap; OOMKilled is the kernel killing the process because the cgroup’s RSS went over the limit.
OutOfMemoryError: Java heap space comes with a stack trace and possibly a heap dump. OOMKilled (exit 137) is a SIGKILL, so there’s no log or dump. The first is fixed by finding a heap leak; the second by shrinking the heap relative to the limit or finding what’s growing off-heap.
Why shouldn’t a JVM use swap?
GC walks most of the heap; heap pages in swap turn every GC cycle into a burst of major faults, and pauses of tens of ms become seconds.
Disable swap or set vm.swappiness very low, and size the heap so the whole process fits in RAM.
Why don’t PostgreSQL and Kafka use all the RAM for their own caches?
Because they rely on the OS page cache: it survives process restarts, puts no pressure on a GC, and the kernel manages eviction.
PostgreSQL sets shared_buffers to about 25% of RAM and leaves the rest to the page cache; Kafka reads and writes its log through the page cache and sends data to consumers with zero-copy from it.
Senior
How do you size the memory limit for a Java service on Kubernetes?
Measure, don’t guess: heap from live data, off-heap from NMT, add them up and add 20–30%.
Run real load, use the live data after a full GC to choose the heap (about 2–3×), and measure off-heap with NMT (metaspace, code cache, threads, direct buffers, GC). Set -Xmx or MaxRAMPercentage accordingly, cap MaxDirectMemorySize, enable ExitOnOutOfMemoryError, and set the request equal to the limit. Monitor the working set, not usage (usage includes page cache).
What is false sharing, and when should you care?
Two threads writing different variables on the same cache line keep invalidating each other’s copy; it only matters for variables written very often by many threads.
Cache coherence works per cache line (64 bytes, or 128 on some ARM chips), so it’s as slow as real contention; in my test on an Apple M4 it was about 3× slower. Examples worth caring about: counters, ring buffer cursors. Fixes: padding, @Contended, or just LongAdder.
CPU cache coherence keeps caches consistent, so why does Java still need volatile?
Coherence only makes cores agree on a cache line once the value has reached the cache; volatile is what creates happens-before.
Before that the value can sit in a core’s store buffer or a register, or be reordered by the JIT or the CPU. volatile and the other synchronization tools insert memory barriers and stop the JIT from optimizing reads away, creating the happens-before edges the Java Memory Model defines, so the code is correct on every CPU architecture, including ARM with its weaker memory model.
Useful phrases for interviews
- “Virtual size is just reserved address space; what the container limit cares about is RSS, or the working set.”
- “The heap is only one slice of a JVM’s footprint; metaspace, thread stacks and direct buffers add a few hundred megabytes.”
- “Exit code 137 with no stack trace tells me the kernel killed the process, not the JVM.”
- “I’d rather leave RAM to the page cache than give a Kafka broker a huge heap.”
- “I alert on the working set, because usage includes page cache the kernel can drop.”
- “Swap and garbage collection don’t mix: a full GC touching swapped-out pages turns a pause into seconds.”
- “Coherence works per cache line, so two hot counters next to each other can slow each other down.”
- “The Java Memory Model is what turns hardware coherence into guarantees I can reason about.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.