Skip to content
notes.vinhnq.com

Search

ConcurrencySenior20 min read

ExecutorService, thread pools and virtual threads in Java

Review 14 cards
On this page

Threads and concurrency in Java series: 1. Creating threads and the lifecycle · 2. Race conditions, volatile, deadlocks · 3. ExecutorService and virtual threads

Tomcat, @Async, @Scheduled and Kafka consumers all run on thread pools, and most thread-related production incidents (an OOM from a growing queue, an app that won’t shut down, user A seeing user B’s data) come down to pool configuration. After this post you can configure a ThreadPoolExecutor, size it, shut it down cleanly, and know when to use virtual threads. Read part 1 (interruption) and part 2 (race conditions, happens-before) first; every demo was run on JDK 21.

1. ExecutorService and thread pools

Key idea: a thread pool keeps a set of threads and a queue of tasks, so you don’t create an OS thread per task. ExecutorService is the interface for submitting tasks and managing the pool’s lifecycle.

A new Thread() per task has three problems:

  • Expensive: creating and destroying OS threads costs ~1 MB of stack and system calls.
  • Unbounded: 10,000 requests = 10,000 threads = out of memory or past the OS limit.
  • No central control: naming, error handling, clean shutdown.
ExecutorService pool = ...;
pool.execute(runnable);                 // fire and forget
Future<T> f = pool.submit(callable);    // get the result through a Future
List<Future<T>> all = pool.invokeAll(tasks, 2, TimeUnit.SECONDS); // wait for all, with a deadline

ThreadPoolExecutor: the real parameters

Key idea: the Executors.newXxx() factories are just shortcuts that build a ThreadPoolExecutor. In production, spell the parameters out and know the order in which the pool handles a task.

ThreadPoolExecutor pool = new ThreadPoolExecutor(
        2, 4,                                   // corePoolSize = 2, maximumPoolSize = 4
        30, TimeUnit.SECONDS,                   // non-core threads idle for 30s are destroyed
        new ArrayBlockingQueue<>(2),            // a BOUNDED queue
        named("report"),                        // ThreadFactory: named threads for logs/thread dumps
        new ThreadPoolExecutor.AbortPolicy());  // when full: throw RejectedExecutionException

What happens when you submit a task (the part most people get wrong):

  1. Fewer than core threads → start a new thread, even if others are idle.
  2. core threads running → put it in the queue.
  3. Queue full → start extra threads, up to max.
  4. max threads and a full queue → call the rejection policy.
flowchart TD
    A[Submit task] --> B{threads < core?}
    B -- yes --> C[Start a new thread]
    B -- no --> D{Room in the queue?}
    D -- yes --> E[Enqueue]
    D -- no --> F{threads < max?}
    F -- yes --> G[Start an extra thread]
    F -- no --> H[Rejection policy]
Demo with 7 tasks and a ThreadFactory that names threads and catches exceptions

Submitting 7 slow (300 ms) tasks to the pool above:

task 1 -> threads=1, queued=0
task 2 -> threads=2, queued=0
task 3 -> threads=2, queued=1
task 4 -> threads=2, queued=2
task 5 -> threads=3, queued=2
task 6 -> threads=4, queued=2
task 7 -> rejected

The thread factory that names threads and catches exceptions from tasks submitted with execute:

static ThreadFactory named(String prefix) {
    AtomicInteger n = new AtomicInteger();
    return r -> {
        Thread t = new Thread(r, prefix + "-" + n.incrementAndGet());
        t.setUncaughtExceptionHandler((th, ex) ->
                System.out.println("uncaught in " + th.getName() + ": " + ex.getMessage()));
        return t;
    };
}

Rejection policies

Key idea: the rejection policy decides what happens when both the pool and the queue are full. The default throws; CallerRunsPolicy gives back-pressure.

Policy When the pool and queue are full Use it when
AbortPolicy (default) Throws RejectedExecutionException to the submitter You want a clear failure: return 503/429, record it for a retry
CallerRunsPolicy The submitting thread runs the task itself You want back-pressure (pushing back on the sender): the submitter slows down, so it submits less
DiscardPolicy Silently drops the new task Almost never; work disappears and nobody knows
DiscardOldestPolicy Drops the oldest queued task and resubmits the new one Only the latest value matters (positions, prices)

Sizing a pool

Key idea: there’s no single right number. Start from a formula, cap it by downstream resources, then measure under real load.

  • CPU-bound: threads ≈ cores (or +1). More just adds context switches.
  • I/O-bound: threads ≈ cores × (1 + wait time / compute time). 8 cores, 90 ms waiting on the database, 10 ms computing: 8 × (1 + 9) = 80.
  • Little’s law: concurrent work = throughput × latency. 400 requests/s at 0.25 s each → about 100 in flight. Using it to estimate threads, connections and servers: Back-of-the-envelope estimation.
  • Downstream limits: 200 threads calling the database through a 20-connection HikariCP pool means 180 threads waiting (see ORM and JPA for connection pools).

“Measure again under real load” means a load test with targets from the SLO, watching p99 and pool saturation; how to do it right is in Performance testing.

Split pools by kind of work (bulkheads, like watertight compartments in a ship): one pool for payments, one for email. Slow payments exhaust only their own pool. Spring’s @Scheduled uses a single thread by default; see the Cron jobs post.

At millions of tasks per second, the queue between threads itself becomes the bottleneck (ArrayBlockingQueue uses one ReentrantLock for both put and take). The LMAX Disruptor replaces it with a pre-allocated ring buffer, CAS and padding against false sharing; the comparison and when it’s worth it are in A liveness system: a timing wheel and the LMAX Disruptor.

Shutting a pool down properly

Key idea: pool threads are non-daemon, so a forgotten pool keeps the JVM alive. Shut down in three steps: shutdown(), wait with a deadline, then shutdownNow() if it expires.

static void shutdownGracefully(ExecutorService pool) {
    pool.shutdown();                                   // stop accepting new tasks, queued ones still run
    try {
        if (!pool.awaitTermination(30, TimeUnit.SECONDS)) {
            List<Runnable> neverRan = pool.shutdownNow(); // interrupt running tasks, return the ones never started
            log.warn("{} tasks dropped on shutdown", neverRan.size());
            if (!pool.awaitTermination(10, TimeUnit.SECONDS)) {
                log.error("pool did not terminate");
            }
        }
    } catch (InterruptedException e) {
        pool.shutdownNow();
        Thread.currentThread().interrupt();
    }
}
  • shutdown(): no new tasks (further submissions are rejected); queued tasks still run.
  • shutdownNow(): interrupts running threads and returns the tasks that never started. Tasks only really stop if they respect interruption (see part 1, section 4).
Spring and try-with-resources (Java 19)

In Spring, the executor is a bean (ThreadPoolTaskExecutor), so it’s shut down with the context. Set spring.task.execution.shutdown.await-termination=true and an await-termination-period so running tasks get time to finish during a deploy.

2. Virtual threads (Java 21)

Key idea: a virtual thread is a JVM-managed thread, cheap enough to create millions. When it blocks on I/O, the JVM takes it off the OS thread so the OS thread can run other work, which lets plain blocking code serve huge numbers of connections.

A platform thread (the traditional kind) wraps an OS thread: ~1 MB of stack, slow to create, and the OS handles a few thousand. A thread waiting 100 ms on the database still occupies its OS thread, so people moved to reactive code (WebFlux), paying with code that’s harder to read and debug.

A virtual thread (JEP 444, final in Java 21):

  • Keeps its stack on the heap, starting at a few hundred bytes to a few KB.
  • Runs on a small set of carrier threads (background threads that carry virtual threads; a ForkJoinPool with one thread per core by default).
  • On a blocking call (sleep, socket, BlockingQueue, ReentrantLock), the JVM unmounts it from its carrier and parks its stack on the heap. When the I/O completes, it’s mounted on some carrier and continues.
  • Under the hood, its socket is registered with a poller that uses epoll on Linux; see I/O models: blocking, non-blocking, epoll and file descriptors.
import java.time.Duration;
import java.util.concurrent.*;
import java.util.stream.IntStream;

public class VirtualThreads {
    static int callRemote(int i) {
        try { Thread.sleep(Duration.ofSeconds(1)); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        return i;
    }

    static long run(ExecutorService executor, int tasks) throws Exception {
        long start = System.nanoTime();
        try (executor) {
            var futures = IntStream.range(0, tasks).mapToObj(i -> executor.submit(() -> callRemote(i))).toList();
            for (var f : futures) f.get();
        }
        return Duration.ofNanos(System.nanoTime() - start).toMillis();
    }

    public static void main(String[] args) throws Exception {
        int tasks = 10_000;
        System.out.println("platform pool(200): " + run(Executors.newFixedThreadPool(200), tasks) + " ms");
        System.out.println("virtual per task : " + run(Executors.newVirtualThreadPerTaskExecutor(), tasks) + " ms");
    }
}
// platform pool(200): 50210 ms
// virtual per task : 1050 ms

10,000 tasks each waiting 1 second on I/O: 200 platform threads need 50 rounds (50 s); 10,000 virtual threads wait at once and finish in ~1 s. Create them with Thread.ofVirtual().start(r), Thread.startVirtualThread(r) or Executors.newVirtualThreadPerTaskExecutor(). In Spring Boot 3.2+, spring.threads.virtual.enabled=true makes Tomcat and @Async use virtual threads.

Virtual threads do not make code faster: they raise throughput for work that waits, don’t lower each task’s latency, and don’t help CPU-bound work.

Pinning

Key idea: a virtual thread is pinned when it can’t unmount from its carrier while blocking: inside a synchronized block (Java 21–23) or inside a native/JNI frame. Each pinned blocking call holds a carrier hostage, and there are only as many carriers as cores.

static void blockingCallInsideSynchronized() {
    Object perTaskMonitor = new Object();
    synchronized (perTaskMonitor) {          // Java 21-23: the virtual thread is pinned to its carrier
        try { Thread.sleep(Duration.ofMillis(100)); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }
}

static void blockingCallInsideReentrantLock() {
    ReentrantLock lock = new ReentrantLock();
    lock.lock();
    try {
        try { Thread.sleep(Duration.ofMillis(100)); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    } finally {
        lock.unlock();
    }
}
// 1,000 virtual threads, 10-core machine, JDK 21:
// synchronized  : 10394 ms   (only 10 run at once: 100 rounds x 100 ms)
// ReentrantLock : 108 ms

Each task uses its own monitor, so there’s no contention, yet the synchronized version is almost 100 times slower, purely because the 10 carriers are pinned. The fix: replace synchronized around I/O with ReentrantLock, or upgrade the JDK.

Finding pinning on JDK 21–23

Run with -Djdk.tracePinnedThreads=short to have the JVM print where pinning happens (reason:MONITOR plus the line of code), or enable the JFR event jdk.VirtualThreadPinned.

3. ThreadLocal and leaks in thread pools

Key idea: ThreadLocal<T> gives each thread its own copy of a value. Pool threads don’t die after a task, so a value that isn’t remove()d carries over to the next task on that thread.

Frameworks use it heavily: SecurityContextHolder (the logged-in user), TransactionSynchronizationManager (the current transaction), the logging MDC (trace id).

import java.util.concurrent.*;

public class ThreadLocalLeak {
    static final ThreadLocal<String> CURRENT_USER = new ThreadLocal<>();

    static void handle(String user) {
        if (user != null) CURRENT_USER.set(user);       // anonymous requests don't set it
        System.out.println("request by " + user + " -> audit as " + CURRENT_USER.get());
    }

    static void handleSafely(String user) {
        try {
            if (user != null) CURRENT_USER.set(user);
            System.out.println("request by " + user + " -> audit as " + CURRENT_USER.get());
        } finally {
            CURRENT_USER.remove();
        }
    }

    public static void main(String[] args) throws Exception {
        try (ExecutorService pool = Executors.newSingleThreadExecutor()) {
            pool.submit(() -> handle("alice")).get();
            pool.submit(() -> handle(null)).get();      // the reused thread still carries "alice"
            System.out.println("--- with remove() ---");
            pool.submit(() -> handleSafely("bob")).get();
            pool.submit(() -> handleSafely(null)).get();
        }
    }
}
// request by alice -> audit as alice
// request by null -> audit as alice      <- the anonymous request is audited as alice
// --- with remove() ---
// request by bob -> audit as bob
// request by null -> audit as null

The flip side is context that gets lost when work hops threads: the MDC trace id and OpenTelemetry’s current span live in a ThreadLocal, so a task handed to a hand-made executor logs without a trace id and creates spans with no parent.

Where context gets lost and how to carry it

The “where context gets lost” section of Distributed tracing with OpenTelemetry lists those places (ExecutorService, CompletableFuture.supplyAsync, @Async, message brokers, jobs reading an outbox) and how to carry the context along with Context.current().wrap(...) or Context.taskWrapping(executor).

Exercises

Exercise 1: Fetching prices in parallel with a limit and a deadline

The task: you have 100 product ids and a fetchPrice(id) function that calls a supplier API, takes 100–300 ms, sometimes fails, and very rarely hangs. The supplier allows at most 10 concurrent requests. Fetch prices for all 100 ids within 4 seconds for the whole batch, and return the successes plus the failures (with a reason).

Solution to Exercise 1: virtual threads + Semaphore + invokeAll with a timeout
record Result(Map<Integer, Long> prices, Map<Integer, String> failures) {}

static Result fetchAll(List<Integer> ids, int maxConcurrent, long timeoutMs) throws InterruptedException {
    Semaphore permits = new Semaphore(maxConcurrent);
    List<Callable<Long>> tasks = ids.stream().<Callable<Long>>map(id -> () -> {
        permits.acquire();
        try {
            return fetchPrice(id);
        } finally {
            permits.release();
        }
    }).toList();

    Map<Integer, Long> prices = new TreeMap<>();
    Map<Integer, String> failures = new TreeMap<>();
    try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
        // invokeAll with a timeout: when time is up, unfinished tasks are cancel(true)-ed
        List<Future<Long>> futures = executor.invokeAll(tasks, timeoutMs, TimeUnit.MILLISECONDS);
        for (int i = 0; i < ids.size(); i++) {
            Future<Long> f = futures.get(i);
            switch (f.state()) {                                          // Java 19+
                case SUCCESS -> prices.put(ids.get(i), f.resultNow());
                case FAILED -> failures.put(ids.get(i), f.exceptionNow().getMessage());
                case CANCELLED -> failures.put(ids.get(i), "timeout");
                case RUNNING -> throw new AssertionError("invokeAll returns only finished futures");
            }
        }
    }
    return new Result(prices, failures);
}
// ok=94 failed=6 maxInFlight=10 took=4006 ms
// {17=supplier error for 17, 34=supplier error for 34, 42=timeout, 51=..., 68=..., 85=...}
  • One virtual thread per request; Semaphore(10) is the real concurrency limit (measured peak: 10 requests in flight).
  • invokeAll with a timeout returns when every task is done or the deadline passes; unfinished tasks are cancel(true)-ed, so their threads get interrupted and leave sleep.
  • Future.state(), resultNow() and exceptionNow() (Java 19) classify results without try/catch. One failure doesn’t spoil the batch.
  • With platform threads, use Executors.newFixedThreadPool(10) and drop the semaphore, because then the pool size is the limit.

Exercise 2: Sizing a pool

The task: a service runs on an 8-core machine. (a) An image-resizing job, purely CPU. (b) Each request spends 40 ms waiting on the database and 10 ms of CPU; HikariCP has 30 connections. Pick the thread counts.

Solution to Exercise 2
  • (a) CPU-bound: 8 threads (or 9). More only adds context switching.
  • (b) By the formula: 8 × (1 + 40/10) = 40 threads. But there are only 30 connections, so thread 31 onwards just waits; use about 30 threads for the database work, a bounded queue, and measure under real load (CPU, connection wait time in Hikari’s metrics, p99 latency). Going higher means growing the connection pool, which is limited on the database side too.

Remember

  • Walk through the 4 steps ThreadPoolExecutor takes for a task, and why newFixedThreadPool never rejects.
  • Know when to pick AbortPolicy vs CallerRunsPolicy, and how to size a queue from the SLA.
  • Size a pool for CPU-bound and I/O-bound work, and name the downstream resource that caps it.
  • Explain what virtual threads improve and what they don’t, why you don’t pool them, and what pinning is.
  • Explain why ThreadLocal leaks data in a pool and how to clean it up.

Interview questions

Junior

Why use a thread pool instead of new Thread() per task?

Because OS threads are expensive and unbounded; a pool reuses a fixed set of threads and gives you one place to manage them.

Creating and destroying OS threads costs about 1 MB of stack and system calls; a traffic spike creates threads until memory runs out. A pool has a queue and explicit limits, and is the one place to name threads, catch errors and shut down cleanly. Virtual threads (Java 21) are the exception: they’re cheap, so you don’t pool them and you cap concurrency with a Semaphore instead.

Mid

How do execute() and submit() behave when the task throws?

execute lets the exception kill the thread and reach the UncaughtExceptionHandler; submit stores it in the Future, where it only surfaces on get().

execute(Runnable) returns nothing; the default handler prints to stderr and the pool creates a replacement thread. submit(...) returns a Future; the exception surfaces as an ExecutionException when you call get(). If nobody calls get(), the error is lost.

How do you choose the size of a thread pool?

CPU-bound ≈ cores; I/O-bound ≈ cores × (1 + wait/compute); then cap it by downstream resources and measure.

Little’s law also works (concurrency = throughput × latency). Downstream resources include the database connection pool and API rate limits; measure under real load. Split pools by kind of work (bulkheads) so one slow dependency doesn’t take the whole app down.

What’s the difference between shutdown() and shutdownNow()?

shutdown() stops accepting new tasks but lets running and queued ones finish; shutdownNow() interrupts running threads and returns the tasks that never started.

Tasks only really stop if they respect interruption. Neither one waits; for that you call awaitTermination(timeout). The standard pattern: shutdown(), wait with a deadline, and if that expires, shutdownNow() and wait a little longer.

Senior

Why shouldn’t you use Executors.newFixedThreadPool or newCachedThreadPool in production?

Fixed uses an unbounded queue, so overload becomes a silent OOM; Cached has no thread limit, so a spike spawns thousands of threads.

newFixedThreadPool and newSingleThreadExecutor use an unbounded LinkedBlockingQueue: no back-pressure, and maximumPoolSize and the rejection policy never take effect (the pool only grows past core when the queue is full). newCachedThreadPool has a max of Integer.MAX_VALUE. Build your own ThreadPoolExecutor with a bounded queue, a deliberate rejection policy (Abort to fail fast, CallerRuns for back-pressure), a naming ThreadFactory, and metrics for queue size and rejected tasks.

What do virtual threads help with, what don’t they help with, and what are the traps?

They raise throughput for high-concurrency I/O-bound work while keeping sequential blocking code; they don’t help CPU-bound work or reduce per-request latency.

The JVM unmounts a virtual thread from its carrier when it blocks. Traps:

  • Pinning when blocking inside synchronized (Java 21–23, fixed in 24 by JEP 491) or native code.
  • Never pool them; limit concurrency with a Semaphore so you don’t overwhelm the database or downstream APIs.
  • Heavy ThreadLocals multiply with the thread count; consider ScopedValue.
How does ThreadLocal cause leaks in a thread pool?

Pool threads are reused, so a value that was set and never removed carries over to the next task: wrong data and memory leaks.

The next request sees the previous request’s user or tenant; the value is a strong reference in the ThreadLocalMap of a thread that lives forever (in app servers it can also pin a classloader). Always remove() in finally; when passing context (MDC, SecurityContext) to another executor, use a TaskDecorator to copy it in and clean up afterwards.

Useful phrases for interviews

  • “I never use Executors.newFixedThreadPool in production because its queue is unbounded; I’d rather reject work visibly than run out of memory silently.”
  • “The pool only grows past its core size once the queue is full, which surprises a lot of people.”
  • “Virtual threads improve throughput for I/O-bound work, not latency, and you limit concurrency with a semaphore instead of a pool.”
  • “Anything we put in a ThreadLocal gets removed in a finally block, because pool threads are reused across requests.”
  • “I size an I/O-bound pool from the wait-to-compute ratio, then cap it by the narrowest resource downstream, usually the connection pool.”
  • “With submit, an exception sits inside the Future until somebody calls get, so fire-and-forget tasks use execute or log inside the task.”
  • “On shutdown we call shutdown, wait with a timeout, and only then fall back to shutdownNow.”
  • “On Java 21 to 23, blocking inside synchronized pins a virtual thread to its carrier; Java 24 fixed that.”

Notes based on the Hỏi Dân IT Java Core material plus my own research, rewritten in my own words.