Operating systems for backend engineers series: 1. Kernel, processes and threads · 2. Virtual memory and the page cache · 3. I/O models and epoll · 4. File systems and the shell
The OS shows up when things go wrong: a load average of 40 with idle CPUs, unable to create native thread with plenty of free heap, a pod throttled at low CPU usage, requests cut off on every deploy. This post covers the kernel, system calls, processes, threads, context switches and the scheduler, and connects them to the JVM and to containers. If Java threads are new to you, read Threads in Java first.
1. The operating system and the kernel
Key idea: the kernel is the core of the OS: it runs with the highest privilege and is the only part that touches the hardware directly. Applications reach the hardware through system calls.
An operating system manages the hardware (CPU, RAM, disks, network cards) and provides common services to programs. The kernel is always resident in memory.
flowchart TB
A[Applications: JVM, PostgreSQL, Nginx] -->|system call| K[Kernel]
K --> C[CPU]
K --> M[Memory]
K --> D[Disks, network cards]
There are three reasons for a layer in the middle:
- Abstraction: an application just calls
read(fd, buf, n)and the driver deals with each device’s protocol. - Protection: applications run unprivileged and can’t write into another process’s memory, so a buggy app only kills itself.
- Sharing: hundreds of processes share a few cores, and the kernel divides up CPU time and memory.
The kernel does four jobs: manage processes and threads, manage memory, manage devices, and provide system calls.
2. User mode, kernel mode and system calls
Key idea: applications run in user mode, where the CPU refuses privileged instructions. To read a file or send a packet, they make a system call that switches into kernel mode and back.
On x86-64 the kernel runs in ring 0 (kernel mode) and applications in ring 3 (user mode). Memory is split into kernel space and user space as well. On Linux the kernel is mapped into every process, but those pages are marked “kernel mode only” in the page table (part 2).
sequenceDiagram
participant App as Application
participant Libc as glibc
participant K as Kernel
App->>Libc: read(fd, buf, 4096)
Libc->>K: syscall instruction, number in a register
Note over K: check permissions, then run read
K-->>Libc: sysret with the byte count
Libc-->>App: result or errno
Each system call costs ~100 ns to a few µs, plus the cache lines kernel code evicts. The damage comes from making too many of them: FileInputStream.read() one byte at a time is one syscall per byte; wrapping it in a BufferedInputStream (8 KB per call) cuts that by a factor of 8,000. To see what a process calls, use strace -c -f -p <pid> (which slows it down a lot) or the lighter perf trace.
vDSO: reading the clock without entering the kernel
3. How Linux is designed
Key idea: Linux is a multitasking kernel where all CPUs are peers, and it’s monolithic: most services (scheduler, file systems, network stack, drivers) run in kernel mode. That’s why optimizing I/O on Linux means crossing the user/kernel boundary less often.
Linux’s four design traits, and monolithic vs microkernel vs hybrid
- Multitasking: many tasks take turns on the CPU.
- SMP: all CPUs are peers; any of them can run kernel or application code, and they share memory.
- ELF: the format for executables and libraries. It stands for Executable and Linkable Format (the slide says “Executable File Link Format”, which is wrong).
file $(which java)printsELF 64-bit. - Monolithic kernel: the scheduler, memory manager, file systems, network stack and drivers all run in kernel mode.
4. Programs, processes and threads
Key idea: a process is a running program with its own memory; a thread runs inside it and shares that memory. Every Java platform thread is a real OS thread, so the thread count is limited by the OS, not by the heap.
- Program: an executable file on disk; static data.
- Process: a running program with its own address space and its own open files.
- Thread: a flow of execution inside a process. A JVM has dozens of threads before you create any (GC, JIT, Finalizer…).
stateDiagram-v2
[*] --> Created: fork
Created --> Ready: admit
Ready --> Running: scheduled
Running --> Ready: time slice used up
Running --> Waiting: waits for I/O, lock, sleep
Waiting --> Ready: woken up
Running --> Terminated: exit
Terminated --> [*]
The PCB, ps states, fork and copy-on-write
The kernel describes each process with a PCB (process control block); on Linux it’s task_struct: PID, user ID, state, priority, a pointer to the page table, the file descriptor table, resource limits, and the register values saved when the process was stopped (including the program counter, the address of the next instruction). You can read most of it in /proc/<pid>/status, limits and fd/.
States in ps/top:
R: running or waiting for a CPU.S: sleeping on an event (socket, lock).D: uninterruptible I/O wait (disk, NFS).Z: zombie (dead, but the parent hasn’t collected the exit code).T: stopped.
Linux creates processes with fork() (clone the current process) followed by exec() (replace it with another program). Cloning memory is cheap thanks to copy-on-write: parent and child share pages, and a page is copied only when one side writes to it. Redis snapshots (BGSAVE) rely on exactly this, which is why memory usage can nearly double during a snapshot under heavy writes.
Threads in a process share code, heap, files and sockets; each thread has its own registers, stack and thread-locals.
| Process | Thread | |
|---|---|---|
| Address space | its own | shared |
| Communication | IPC: pipes, sockets | shared memory, needs synchronization |
| One side crashes | the other is fine | can kill the whole process |
| Context switch | page table switch, TLB loss | no page table switch |
(TLB: the CPU’s cache of virtual-to-physical address translations.)
Each Java platform thread maps 1:1 onto an OS thread (pthread_create, then clone). Consequences:
- Every thread has a native stack, 1 MB of virtual memory by default on Linux x64 (
-Xss); only the touched part uses real RAM. - Scheduling is the kernel’s call (
setPrioritydoes almost nothing on Linux by default). - The thread count is limited by the OS, not by the heap.
Virtual threads (Java 21) are an M:N model: many virtual threads run on a handful of carrier threads (one per CPU by default). When a virtual thread waits on I/O, the JVM unmounts it from its carrier and parks its stack on the heap; the kernel never does a context switch. Details in ExecutorService, thread pools and virtual threads.
5. Context switches and the scheduler
Key idea: a core runs one thread at a time; switching threads is a context switch, costing 1–5 µs directly and more through cold caches. The Linux scheduler shares the CPU fairly in time slices of a few ms.
During a context switch, the kernel saves the old thread’s registers and program counter into its task_struct, then loads the new thread’s registers. Between processes it also switches the page table and loses part of the TLB. The indirect cost is larger because the L1/L2 caches are still full of the old thread’s data.
There are two kinds:
- Voluntary: the thread gives up the CPU because it has to wait (I/O, a lock,
park). Lots of these point to I/O waits or lock contention. - Involuntary: the thread still wants to run but its time slice is over. Lots of these point to oversubscription (more threads wanting CPU than there are cores) or a CPU quota.
vmstat 1 # cs: context switches/s machine-wide, r: tasks waiting for a CPU
pidstat -w -t -p <pid> 1 # cswch/s and nvcswch/s per thread
grep ctxt /proc/<pid>/status # running totals for the main thread
The Linux scheduler: CFS, EEVDF, time slices and nice
From 2007 to Linux 6.5 it was CFS: each task has a vruntime (weighted time already run), runnable tasks sit in a red-black tree, and the task with the smallest vruntime runs next. Since Linux 6.6 (2023) it’s EEVDF: still fair sharing, plus a “virtual deadline” so latency-sensitive tasks get picked earlier.
The common ideas:
- Each CPU has a run queue (the queue of tasks waiting for that CPU).
- A time slice is a few ms and shrinks as more tasks wait.
- nice from -20 to 19 changes a task’s weight. Run batch jobs with
nice -n 10so they yield to the main service.
Load average is the average number of tasks running or waiting for a CPU, and on Linux it also counts D tasks (uninterruptible disk waits). So a load of 40 on an 8-core box with idle CPUs is usually a disk or NFS bottleneck: check the wa and b columns of vmstat, then iostat -x 1.
Concurrency means several tasks make progress over the same period (possibly by taking turns on one core); parallelism means several tasks run at the same instant on several cores. The slide states a principle: on one core, running A then B is always faster than interleaving them. That’s only half true.
6. Processes in containers: CPU quotas, CPU counts and signals
Key idea: a CPU limit is a quota per 100 ms period, so a container can be frozen even at low average usage. Java has to be PID 1 (or have SIGTERM forwarded to it) to shut down gracefully.
A Kubernetes resources.limits.cpu: 1 becomes cpu.max = 100000 100000 in the cgroup: in every 100 ms period, all of the container’s threads together may use 100 ms of CPU. Many cores don’t help: 8 threads on 8 cores burn the quota in 12.5 ms, and then the whole container is frozen for the remaining 87.5 ms.
Since JDK 10 (and 8u191) the JVM is container-aware: availableProcessors() is derived from the cgroup quota, not from the node’s core count. That number drives the GC thread count, the size of ForkJoinPool.commonPool(), the number of virtual-thread carriers and Netty’s event loops. Override it with -XX:ActiveProcessorCount=N. Only the CPU limit affects it; without a limit the JVM sees every core on the node. Packaging it into an image properly (multi-stage Dockerfile, MaxRAMPercentage, non-root, exec-form ENTRYPOINT) is in Docker for Spring Boot.
Signals and PID 1. To stop a pod, Kubernetes sends SIGTERM to PID 1 in the container, waits terminationGracePeriodSeconds (30 s by default), then sends SIGKILL. Exit code 143 means death by SIGTERM; 137 means SIGKILL (or the OOM killer, see part 2). The JVM catches SIGTERM and runs shutdown hooks; with server.shutdown=graceful, Spring Boot stops accepting new requests, waits for in-flight ones (up to spring.lifecycle.timeout-per-shutdown-phase), then closes its beans. How SIGTERM, readiness and preStop combine into a rolling update that drops no requests is in Deployment strategies.
Exercises
The code runs on JDK 21; the numbers from my machines only show orders of magnitude (use JMH for numbers you can trust).
Exercise 1: The cost of creating and waking threads
Task. Measure (a) the time to create, start and join a platform thread versus a virtual thread; (b) the time for one “hand-off” between two platform threads (one goes to sleep, the other is woken up).
Solution 1: code, measurements and explanation
import java.util.concurrent.locks.LockSupport;
public class ThreadCost {
static final int N = 10_000;
static volatile int turn = 0;
public static void main(String[] args) throws Exception {
for (int r = 0; r < 3; r++)
System.out.printf("platform %.1f us, virtual %.2f us, handoff %.2f us%n",
startJoin(Thread.ofPlatform()), startJoin(Thread.ofVirtual()), pingPong());
}
static double startJoin(Thread.Builder builder) throws InterruptedException {
Thread[] ts = new Thread[N];
long t0 = System.nanoTime();
for (int i = 0; i < N; i++) ts[i] = builder.start(() -> {});
for (Thread t : ts) t.join();
return (System.nanoTime() - t0) / 1_000.0 / N;
}
static double pingPong() throws InterruptedException {
int rounds = 100_000;
Thread[] pair = new Thread[2];
for (int id = 0; id < 2; id++) {
int me = id;
pair[id] = Thread.ofPlatform().unstarted(() -> {
for (int i = 0; i < rounds; i++) {
while (turn != me) LockSupport.park(); // sleep until it's my turn
turn = 1 - me;
LockSupport.unpark(pair[1 - me]); // wake the other thread
}
});
}
turn = 0;
long t0 = System.nanoTime();
pair[0].start(); pair[1].start();
pair[0].join(); pair[1].join();
return (System.nanoTime() - t0) / 1_000.0 / (2.0 * rounds);
}
}
| Environment | Platform thread | Virtual thread | One hand-off |
|---|---|---|---|
| macOS, Apple M4, 10 cores | 23–28 µs | 0.2–1.7 µs | 1.7–1.9 µs |
Linux container --cpus=1 |
20–124 µs | 0.8–2.8 µs | 0.9–3.2 µs |
Explanation. A platform thread needs clone(), a native stack and a task_struct, so it costs 20–50 times more than a virtual thread (an object on the heap). A 1–3 µs hand-off is the price of “sleep, get woken, get scheduled again”. Takeaway: don’t create a platform thread per request; use a pool, or virtual threads when the work is mostly waiting on I/O.
Exercise 2: Sequential or interleaved on one CPU
Task. Test the “on one core, sequential beats interleaved” principle: 16 identical CPU-bound tasks, run with a 1-thread pool and a 16-thread pool in a container limited to 1 CPU. Measure the total time, when the first task finishes, and the average completion time.
Solution 2: code, two runs and explanation
import java.util.*;
import java.util.concurrent.*;
// docker run --rm --cpus=1 -v $PWD:/lab -w /lab eclipse-temurin:21-jdk java TimeSlicing.java
public class TimeSlicing {
static long work() {
long x = 0;
for (int i = 0; i < 40_000_000; i++) x += (x ^ i) * 31 + 7;
return x;
}
public static void main(String[] args) throws Exception {
work(); work(); // warm up the JIT
for (int pool : new int[] {1, 16}) {
ExecutorService ex = Executors.newFixedThreadPool(pool);
long t0 = System.nanoTime();
long[] done = new long[16];
List<Future<?>> fs = new ArrayList<>();
for (int i = 0; i < 16; i++) {
int id = i;
fs.add(ex.submit(() -> { work(); done[id] = System.nanoTime(); }));
}
for (Future<?> f : fs) f.get();
var s = Arrays.stream(done).mapToDouble(d -> (d - t0) / 1e6).summaryStatistics();
System.out.printf("pool=%2d total=%4.0f ms | first=%4.0f ms | avg=%4.0f ms%n",
pool, s.getMax(), s.getMin(), s.getAverage());
ex.shutdown();
}
}
}
Two runs (availableProcessors=1):
pool= 1 total= 818 ms | first= 54 ms | avg= 437 ms
pool=16 total= 847 ms | first= 448 ms | avg= 800 ms
pool= 1 total= 977 ms | first= 82 ms | avg= 546 ms
pool=16 total=1087 ms | first= 616 ms | avg=1021 ms
Explanation. The totals are nearly equal (interleaving is 3–10% slower because of context switches and cold caches), but the average completion time is almost twice as bad: 16 tasks share one CPU, so every one of them finishes near the end.
After the run, the container’s cpu.stat showed nr_periods 24, nr_throttled 13, throttled_usec 1062903: the VM has 2 vCPUs, and the JIT, GC and worker threads ran in parallel and burned the quota fast.
Exercise 3: Fix the Dockerfile so graceful shutdown works
Task. Every time the Dockerfile below is deployed to Kubernetes, about 0.5% of requests fail with 502 and the log never shows “Commencing graceful shutdown”. Why, and how do you fix it?
FROM eclipse-temurin:21-jre
COPY app.jar /app/app.jar
ENTRYPOINT java $JAVA_OPTS -jar /app/app.jar
Solution 3: exec form and graceful shutdown settings
The shell form makes /bin/sh -c PID 1; SIGTERM never reaches Java, Spring never shuts down gracefully, and the pod is SIGKILLed after 30 seconds. Change it to ENTRYPOINT ["sh", "-c", "exec java $JAVA_OPTS -jar /app/app.jar"].
Also set:
server.shutdown: graceful,spring.lifecycle.timeout-per-shutdown-phase: 20s,terminationGracePeriodSeconds: 40.- A
preStopsleep 5, because removing the pod from the Service’s endpoints happens in parallel with SIGTERM.
Verify with kubectl delete pod: the log shows “Graceful shutdown complete” and the exit code is 143 or 0, not 137.
Remember
- Explain why a system call is a trap rather than an interrupt, and why unbuffered byte-by-byte reads are slow.
- Tell apart processes, threads (as the Linux kernel sees them), platform threads and virtual threads.
- Know when a high load average means you’re short on CPU and when it means a disk bottleneck.
- Explain why a pod at 40% of its CPU limit still gets throttled, and how many CPUs the JVM sees in a container.
- Know why a shell-form
ENTRYPOINTbreaks graceful shutdown and how to fix it.
Interview questions
Junior
What’s the difference between a process and a thread?
A process has its own address space and is isolated from other processes; a thread runs inside a process and shares memory with its other threads.
A thread shares the heap, code and open files, but has its own stack and registers. Threads are cheaper to create and switch, but shared memory needs synchronization, and one thread corrupting shared state can bring down the whole process.
What is a system call? Give a Java example.
It’s how a user-mode application asks the kernel to do privileged work: read, write, socket, connect, mmap, clone.
In Java, FileInputStream.read, Socket.connect and new Thread().start() all end up as system calls through the JVM and glibc. Each call switches into kernel mode and back, so calling too often (unbuffered byte-by-byte reads) is slow.
What’s the difference between concurrency and parallelism?
Concurrency means several tasks make progress over the same period, possibly just by taking turns on one core; parallelism means several tasks run at the same instant on several cores.
A server handling 1,000 requests on 4 cores is concurrent (1,000 requests in flight) and parallel (at most 4 running at once).
Mid
What does a context switch cost, and why is switching threads cheaper than switching processes?
About 1–5 µs directly, with a usually bigger indirect cost from cold caches and TLB; threads of one process share a page table, so the TLB stays useful.
The direct part is entering the kernel, running the scheduler, and saving and loading registers. Two processes need a page table switch. Measure with the cs column of vmstat and pidstat -w (voluntary/involuntary per thread).
Does a load average of 30 on an 8-core machine mean we’re out of CPU?
Not necessarily: on Linux the load average counts tasks waiting for a CPU and D tasks in uninterruptible I/O.
High idle plus high wa points to a disk or NFS bottleneck. Only when CPU is near 100% and the r column of vmstat is well above the core count are you short on CPU. Check iostat -x and count D tasks too.
How do Java threads map to OS threads, and how are virtual threads different?
Platform threads are 1:1 with OS threads; virtual threads are M:N on a few carrier threads.
A platform thread has a native stack (1 MB virtual by default), is scheduled by the kernel, costs tens of µs to create, and is limited by the OS and the cgroup. When a virtual thread waits on I/O, the JVM unmounts it and parks its stack on the heap, so the kernel does no context switch. Great for I/O-heavy work, no help for CPU-bound work.
Why can unable to create native thread happen with free heap?
Because the OS refused to create a native thread; the heap isn’t involved.
Causes: ulimit -u, the cgroup’s pids.max, kernel.threads-max, or no virtual memory left for stacks. The root cause is usually unbounded thread creation. Count threads via /proc/<pid>/task or a thread dump, then use bounded pools or virtual threads; raising -Xmx doesn’t help.
Senior
A pod uses 40% of its CPU limit but p99 spikes in bursts. How do you investigate?
Suspect CPU throttling: the limit is a quota per 100 ms period, while usage is averaged per minute.
Parallel GC, the JIT and big pools can burn the quota in the first few ms of a period. Check nr_throttled and throttled_usec in cpu.stat or container_cpu_cfs_throttled_periods_total, and line them up with the GC log. Fixes: fewer parallel threads to match the limit (ParallelGCThreads, ActiveProcessorCount, pool sizes), a higher limit, or requests without limits for latency-sensitive services.
How do you size a thread pool, given what you know about the scheduler?
Separate the kinds of work: CPU-bound pools at about availableProcessors, I/O-bound pools larger in proportion to wait/compute time, then measure and adjust.
- CPU-bound: about
availableProcessors(which already reflects the CPU limit); more threads only add context switches. - I/O-bound: roughly cores × (1 + wait time / compute time), capped by downstream resources (connection pools, partner rate limits); or virtual threads with a semaphore.
- Measure: high
nvcswch/sand a long run queue mean too many threads; low CPU with high latency means too few or a downstream bottleneck.
Every deploy produces a few 502s and the log has no graceful shutdown line. Why?
Java isn’t receiving SIGTERM, usually because a shell-form ENTRYPOINT makes sh PID 1 and it doesn’t forward signals; the pod is SIGKILLed after the grace period (exit 137).
Fix it with the exec form, exec java ... or tini. Add server.shutdown=graceful, a Spring timeout shorter than terminationGracePeriodSeconds, and a short preStop sleep, because endpoint removal runs in parallel with SIGTERM.
Useful phrases for interviews
- “A system call is a trap into the kernel; on x86-64 it’s the
syscallinstruction, not a hardware interrupt.” - “To the Linux kernel, a thread is just a task that shares its address space with other tasks.”
- “Context switches have a direct cost of a few microseconds and an indirect cost in cold caches, which is usually bigger.”
- “On a single core, interleaving CPU-bound work doesn’t save time; it only spreads completion times out.”
- “I size CPU-bound pools to the number of cores the JVM sees, which already accounts for the container’s CPU limit.”
- “Low average CPU with high p99 makes me check
cpu.statfor throttling.” - “Graceful shutdown only works if Java is PID 1 or something forwards SIGTERM to it.”
- “On Linux, load average includes tasks blocked on disk, so a high load doesn’t always mean we need more CPU.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.