Skip to content
notes.vinhnq.com

Search

Java CoreSenior22 min read

Virtual memory, page cache, swap and the JVM in containers

Review 17 cards
On this page

Operating systems for backend engineers series: 1. Kernel, processes and threads · 2. Virtual memory and the page cache · 3. I/O models and epoll · 4. File systems and the shell

A Java pod gets OOMKilled while its heap is only half full; free -m shows 200 MB “free” and nobody cares; two threads incrementing two unrelated variables run three times slower. All three come from how the OS and the CPU manage memory. After this post you can explain each one, and size the memory limit for a Java service. Read part 1 (processes, kernel mode) first.

1. Virtual memory

Key idea: every process sees its own address space, as if it had the whole memory to itself. The MMU (the address translator inside the CPU) turns virtual addresses into physical ones on every access, using page tables the kernel builds.

Without virtual memory, process A writes x = 1 at 0x2000, process B writes x = 2 at the same spot, and A reads back B’s value. With it, the same address 0x2000 in A and B points to two different frames (physical page slots). Memory is split into pages, usually 4 KB.

flowchart LR
  A[Process A: virtual address 0x2000] --> PTA[A's page table]
  B[Process B: virtual address 0x2000] --> PTB[B's page table]
  PTA --> F7[Physical frame 7]
  PTB --> F3[Physical frame 3]
  • Each page table entry has attribute bits: is it in RAM (present), writable, accessible from user mode, executable (NX). That lets the kernel block writes to code, running code from data, and user mode reading kernel memory.
  • The TLB (translation lookaside buffer) is a small CPU cache of recent translations, so the page table isn’t walked on every access.
  • Huge pages (2 MB or 1 GB) cut the number of TLB entries needed by 512×.
Multi-level page tables, TLB limits and transparent huge pages

On x86-64 the page table has 4 levels (5 on newer CPUs), so one translation can cost up to 4 memory reads. The TLB only has tens to a few thousand entries. With 4 KB pages, 64 TLB entries cover just 256 KB; an 8 GB heap accessed randomly misses the TLB constantly.

Page faults

Key idea: virtual memory is handed out lazily. malloc or mmap of 1 GB only reserves space; the first touch of a page with no frame makes the MMU raise a page fault, and the kernel assigns a frame and resumes.

Kind When Cost
Minor fault no disk: a fresh frame, or the page is already in the page cache about a microsecond
Major fault disk read: a swapped-out page, or the first read of an mmap’d file tens of µs (NVMe) to ms (HDD)

Measure with ps -o min_flt,maj_flt -p <pid> or pidstat -r. Constant major faults on a service mean it’s short on RAM.

Starting a JVM with no page faults on the heap

2. Needing more than RAM: overcommit, swap and the OOM killer

Key idea: Linux lets processes ask for more than the RAM it has (overcommit), because most requested memory is never touched. When RAM really runs out, the kernel drops page cache, pushes pages to swap, and finally the OOM killer kills a process.

The slides say “virtual memory = physical memory + swap”. That’s misleading.

vm.overcommit_memory: 0 is a heuristic guess (default), 1 always says yes, 2 is strict. When RAM runs out, the kernel reclaims roughly in this order:

  1. Drop clean page cache pages (they can be reread from disk) and write dirty pages back to disk.
  2. Push anonymous pages (heap, stack, no file behind them) to swap if there is any. vm.swappiness (default 60) tunes the balance between step 1 and step 2.
  3. If that still isn’t enough, the OOM killer picks the process with the highest oom_score (roughly the one using the most RAM) and kills it with SIGKILL.

In a container the limit is the cgroup’s memory.max (a cgroup is a group of processes the kernel limits together; Kubernetes’ resources.limits.memory). Going over it runs the OOM killer inside that cgroup, and the pod shows OOMKilled with exit code 137.

java.lang.OutOfMemoryError: Java heap space Container OOMKilled (exit 137)
Who detects it the JVM: heap full after GC the kernel: cgroup RSS over memory.max
Stack trace, heap dump yes (-XX:+HeapDumpOnOutOfMemoryError) no: SIGKILL, the JVM can’t react
Typical cause a heap leak, -Xmx too small heap plus off-heap memory over the limit
Fix analyze the heap dump shrink the heap relative to the limit, measure with NMT

3. The JVM in a container: the heap is only one part

Key idea: a JVM’s RSS is the heap plus several off-heap regions, usually a few hundred MB. By default the JVM takes 25% of the memory limit as the max heap; to size it properly, measure with Native Memory Tracking (NMT).

  • Heap (-Xmx).
  • Metaspace: class metadata. A large Spring Boot app can use 100–200 MB.
  • Code cache: JIT-compiled code, up to 240 MB by default.
  • Thread stacks: each thread reserves 1 MB of virtual space and usually touches a few hundred KB.
  • Direct buffers (ByteBuffer.allocateDirect, Netty, the Kafka client) and mmap’d regions.
  • GC data structures (G1’s remembered sets and card table: a few percent of the heap) and malloc memory (glibc arenas, native libraries).

Verifying the 25% (MaxRAMPercentage=25) in a -m 512m container (JDK 21):

$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
   size_t MaxHeapSize = 134217728       # 128 MB = 25% of 512 MB
$ java -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep MaxHeapSize
   size_t MaxHeapSize = 402653184       # 384 MB

Where to put this flag in the image, along with a multi-stage Dockerfile and how to read OOMKilled, is in Docker for Spring Boot.

In the NMT run below, the heap has committed only 8 MB but RSS is already 152 MB. What matters is committed and RSS; the nearly 2 GB reserved (address space only held) has nothing to do with the memory limit.

NMT experiment: 64 MB of direct buffers and 200 threads in a 512 MB container

The test program has a 128 MB max heap, allocates 64 MB of direct buffers and starts 200 sleeping threads, in a -m 512m container:

$ java -Xmx128m -XX:NativeMemoryTracking=summary NativeMem &
maxHeap=123 MB, heapCommitted=7 MB
VmRSS:   155644 kB
Threads: 213
$ jcmd <pid> VM.native_memory summary scale=MB
Total: reserved=1967MB, committed=118MB
-  Java Heap (reserved=128MB, committed=8MB)
-      Class (reserved=1024MB, committed=0MB)
-     Thread (reserved=425MB, committed=22MB)
-       Code (reserved=242MB, committed=7MB)
-      Other (reserved=64MB, committed=64MB)
- Metaspace (reserved=64MB, committed=1MB)

“Other” at 64 MB is the direct buffers. 213 threads reserve 425 MB but commit only 22 MB.

4. The page cache and mmap

Key idea: the kernel keeps file contents you just read or wrote in RAM (the page cache) so the next access skips the disk. That’s why “free” RAM on Linux is almost always taken by the page cache, and that’s a good thing.

  • Read: the kernel reads from disk into the page cache, then copies into the application’s buffer. Next time, if the page is still cached, no disk access.
  • Write: write() only modifies the page in the page cache and marks it dirty; the kernel writes it to disk a few seconds later, unless the application calls fsync().
$ free -m
               total   used   free  shared  buff/cache  available
Mem:           15890   6120    212     150        9558       9310

The column to read is available (an estimate of memory that can be handed out, including droppable page cache), not free. This machine still has about 9 GB to use.

Storage systems lean on the page cache:

  • PostgreSQL sets shared_buffers to about 25% of RAM and leaves the rest to the OS page cache (effective_cache_size tells the planner so).
  • Kafka appends its log sequentially into the page cache and lets consumers read it back from there, barely touching the disk when consumers keep up (see Why Kafka is fast and durable). Consequence: don’t give a Kafka broker’s JVM a huge heap; RAM left to the page cache is worth more.

mmap maps a file straight into a process’s address space: reading a byte array in memory is reading the file’s page cache, with no read syscall and no copy. Java has FileChannel.map() returning a MappedByteBuffer.

mmap in Kafka and Lucene, and the price you pay

Kafka mmaps its index files; Lucene (Elasticsearch, OpenSearch) mmaps its segments, which is why Elasticsearch requires vm.max_map_count of at least 262144.

The price: an I/O error on an mmap’d region shows up as SIGBUS rather than an IOException, and a page evicted from the cache causes a major fault right in the thread serving a request.

5. CPU caches and the memory model

Key idea: RAM is about 100× slower than L1, so the CPU loads 64-byte cache lines and keeps copies per core. Another thread may not see a new value yet; the Java Memory Model defines when it is guaranteed to.

The slides on the “memory model” are about the cache levels between the CPU and RAM: per-core L1 and L2, shared L3.

A cache line is 64 bytes on x86 and most ARM servers (Apple M chips report 128 bytes). That’s why scanning an array sequentially is fast: one load brings in 8 long elements and the prefetcher predicts the next step. Walking a LinkedList or a scattered object graph is a cache miss at every step.

Hardware keeps the copies consistent with cache coherence, per cache line.

Latency of each memory level, cache coherence and store buffers
Level Typical size Approx. latency
Registers a few dozen values under 1 ns
L1 (per core) 32–128 KB ~1 ns
L2 (per core or core cluster) 256 KB – a few MB ~3–5 ns
L3 (shared) a few MB – tens of MB ~10–20 ns
RAM GB ~80–100 ns
Random 4 KB read from NVMe SSD ~20–100 µs

When several cores hold copies of the same cache line, they can see different values (as the slides say). The coherence protocol (MESI and its variants) handles this: before a core writes to a line, the copies in other cores are invalidated.

But to run fast, each core also has a store buffer (a queue of writes not yet in the cache), and both the CPU and the JIT may reorder instructions. So a thread may not yet see a value another thread just wrote.

The Java Memory Model defines when a write by one thread is guaranteed to be visible to another: volatile, synchronized, final and the classes in java.util.concurrent insert memory barriers (instructions that forbid reordering across them) to create happens-before edges (see race conditions, volatile and deadlock).

False sharing

Key idea: coherence works per cache line, not per variable. Two threads writing different variables on the same line keep invalidating each other’s copy, as slow as fighting over one shared variable.

It’s like two people writing different sentences on the same sheet of paper: every time one writes, the other has to hand the sheet over. The fix: move the variables onto different lines with padding, or use @Contended.

LongAdder and @Contended in the JDK

Exercises

Exercise 1: Sizing the memory limit for a Spring Boot service

Task. A Spring Boot service (Tomcat, Hibernate, a Kafka client) under real load shows: about 400 MB of live heap data after a full GC; 160 MB of metaspace; about 250 threads; the Kafka client and WebClient’s Netty together use about 100 MB of direct memory. Choose -Xmx and the container memory limit, and explain.

Solution to Exercise 1: add up each part, get a 2 GiB limit
Part Estimate Reason
Heap 1,024 MB about 2.5× live data so GC doesn’t run constantly
Metaspace 160 MB measured
Code cache 80 MB usually 50–100 MB once warm
Thread stacks 125 MB 250 threads × about 0.5 MB committed
Direct buffers 100 MB measured, capped with -XX:MaxDirectMemorySize=128m
GC structures (G1) 60 MB a few percent of the heap
malloc, native libraries, other 100 MB headroom for glibc arenas, zip, SSL
Total about 1.65 GB

Add about 20%: limit 2 GiB, -Xmx1g (or MaxRAMPercentage=50), request equal to limit. Verify with a load test, comparing NMT with the pod’s working set. If the working set keeps growing while the heap is flat, suspect an off-heap leak (direct buffers, a growing thread count, glibc arenas; try MALLOC_ARENA_MAX=2). Off-heap leaks often show up only after hours, which is what a soak test is for (Performance testing).

Exercise 2: Measuring false sharing

Task. Write a program with two threads, each incrementing its own volatile long 200 million times. Compare the variables sitting next to each other with 56 bytes of padding between them.

On real hardware (Apple M4) padding is about 3× faster; on a 2-vCPU VM there’s almost no difference.

Solution to Exercise 2: code, numbers on an Apple M4 and a 2-vCPU VM
public class FalseSharing {
    static final long ITER = 200_000_000L;

    static final class Packed {          // a and b side by side: same cache line
        volatile long a;
        volatile long b;
    }

    static final class Padded {          // 7 longs (56 bytes) push b to another line
        volatile long a;
        long p1, p2, p3, p4, p5, p6, p7;
        volatile long b;
    }

    public static void main(String[] args) throws Exception {
        for (int round = 0; round < 5; round++) {
            Packed packed = new Packed();
            long t1 = run(() -> { for (long i = 0; i < ITER; i++) packed.a++; },
                          () -> { for (long i = 0; i < ITER; i++) packed.b++; });
            Padded padded = new Padded();
            long t2 = run(() -> { for (long i = 0; i < ITER; i++) padded.a++; },
                          () -> { for (long i = 0; i < ITER; i++) padded.b++; });
            System.out.printf("round %d: same line %5d ms | padded %5d ms%n", round, t1, t2);
        }
    }

    static long run(Runnable r1, Runnable r2) throws InterruptedException {
        Thread a = new Thread(r1), b = new Thread(r2);
        long t0 = System.nanoTime();
        a.start(); b.start(); a.join(); b.join();
        return (System.nanoTime() - t0) / 1_000_000;
    }
}

Results (JDK 21):

# macOS, Apple M4, 10 cores
round 0: same line  1232 ms | padded   367 ms
round 4: same line  1048 ms | padded   366 ms

# Linux container on a 2-vCPU VM
round 0: same line   928 ms | padded  1056 ms
round 2: same line   753 ms | padded   696 ms
round 4: same line   716 ms | padded   728 ms

Explanation.

  • Real hardware: padding is about 3× faster because the two cores no longer fight over one cache line.
  • 2-vCPU VM: almost no difference. Most likely the two vCPUs aren’t pinned to separate physical cores, or the hypervisor interleaves them on the same core, so there are never two real cores writing at once.
  • Caveats: the JVM may reorder fields, so field padding isn’t guaranteed (@Contended is the official way). The Apple M4 reports a 128-byte cache line, so only 128 bytes of padding is certain. This is a hand-rolled benchmark without JMH; trust only the order of magnitude.
  • Lesson: false sharing only matters for variables written very often by many threads (counters, queue indexes); ordinary business code doesn’t need to care.

Remember

  • Explain what the MMU, page tables and the TLB do, and how a minor fault differs from a major one.
  • Explain why a large VSZ is harmless, and the order in which the kernel reclaims memory before the OOM killer steps in.
  • Tell an OutOfMemoryError from OOMKilled, and explain why a JVM shouldn’t use swap.
  • List the parts of RSS outside the heap and size the memory limit for a Java service.
  • Explain false sharing, and why volatile is still needed despite cache coherence.

Interview questions

Junior

What is virtual memory and what problem does it solve?

Every process sees its own virtual address space; the MMU in the CPU translates it to physical addresses through page tables managed by the kernel.

It stops processes from overwriting each other’s memory, allows per-page read/write/execute permissions, and enables lazy allocation, shared pages (libraries, copy-on-write) and mapping files into memory.

free on Linux shows very little free memory. Should you worry?

Usually not: Linux uses idle RAM as page cache and drops it when processes need memory. Read the available column, not free.

Worry only when available is low, swap-in/out is constant (si, so in vmstat) or major faults are rising.

What is a page fault? How do minor and major faults differ?

A page fault happens when a process touches a virtual page that has no physical frame or isn’t allowed. A minor fault needs no disk; a major fault needs a disk read.

Minor: a fresh page, or the page is already in the page cache, costing microseconds. Major: a swapped-out page or an unloaded mmap’d file, costing tens of µs to ms. Constant major faults mean the machine is short on RAM.

Mid

Why is a JVM’s RSS larger than -Xmx?

The heap is only one part; a Spring Boot service typically has 300–600 MB outside the heap.

The JVM also has metaspace, the code cache, a stack per thread, direct buffers and mmap’d regions, GC data structures, and malloc memory from native libraries. Measure it with -XX:NativeMemoryTracking=summary and jcmd <pid> VM.native_memory summary.

How is an OutOfMemoryError different from a pod being OOMKilled?

OutOfMemoryError is the JVM reporting a full heap; OOMKilled is the kernel killing the process because the cgroup’s RSS went over the limit.

OutOfMemoryError: Java heap space comes with a stack trace and possibly a heap dump. OOMKilled (exit 137) is a SIGKILL, so there’s no log or dump. The first is fixed by finding a heap leak; the second by shrinking the heap relative to the limit or finding what’s growing off-heap.

Why shouldn’t a JVM use swap?

GC walks most of the heap; heap pages in swap turn every GC cycle into a burst of major faults, and pauses of tens of ms become seconds.

Disable swap or set vm.swappiness very low, and size the heap so the whole process fits in RAM.

Why don’t PostgreSQL and Kafka use all the RAM for their own caches?

Because they rely on the OS page cache: it survives process restarts, puts no pressure on a GC, and the kernel manages eviction.

PostgreSQL sets shared_buffers to about 25% of RAM and leaves the rest to the page cache; Kafka reads and writes its log through the page cache and sends data to consumers with zero-copy from it.

Senior

How do you size the memory limit for a Java service on Kubernetes?

Measure, don’t guess: heap from live data, off-heap from NMT, add them up and add 20–30%.

Run real load, use the live data after a full GC to choose the heap (about 2–3×), and measure off-heap with NMT (metaspace, code cache, threads, direct buffers, GC). Set -Xmx or MaxRAMPercentage accordingly, cap MaxDirectMemorySize, enable ExitOnOutOfMemoryError, and set the request equal to the limit. Monitor the working set, not usage (usage includes page cache).

What is false sharing, and when should you care?

Two threads writing different variables on the same cache line keep invalidating each other’s copy; it only matters for variables written very often by many threads.

Cache coherence works per cache line (64 bytes, or 128 on some ARM chips), so it’s as slow as real contention; in my test on an Apple M4 it was about 3× slower. Examples worth caring about: counters, ring buffer cursors. Fixes: padding, @Contended, or just LongAdder.

CPU cache coherence keeps caches consistent, so why does Java still need volatile?

Coherence only makes cores agree on a cache line once the value has reached the cache; volatile is what creates happens-before.

Before that the value can sit in a core’s store buffer or a register, or be reordered by the JIT or the CPU. volatile and the other synchronization tools insert memory barriers and stop the JIT from optimizing reads away, creating the happens-before edges the Java Memory Model defines, so the code is correct on every CPU architecture, including ARM with its weaker memory model.

Useful phrases for interviews

  • “Virtual size is just reserved address space; what the container limit cares about is RSS, or the working set.”
  • “The heap is only one slice of a JVM’s footprint; metaspace, thread stacks and direct buffers add a few hundred megabytes.”
  • “Exit code 137 with no stack trace tells me the kernel killed the process, not the JVM.”
  • “I’d rather leave RAM to the page cache than give a Kafka broker a huge heap.”
  • “I alert on the working set, because usage includes page cache the kernel can drop.”
  • “Swap and garbage collection don’t mix: a full GC touching swapped-out pages turns a pause into seconds.”
  • “Coherence works per cache line, so two hot counters next to each other can slow each other down.”
  • “The Java Memory Model is what turns hardware coherence into guarantees I can reason about.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.