Thread Pools — Algorithmics

A thread costs roughly a megabyte of stack and a millisecond to create. A web request takes five milliseconds to handle.

Spawning a thread per request means 20% of your time is thread creation, and 10,000 concurrent requests means 10 GB of stacks. The machine dies of bookkeeping.

The pattern

Create a fixed set of threads once. Give them a queue. They take work from it forever.

class ThreadPool {
  private queue: Task[] = [];

  constructor(size: number) {
    for (let i = 0; i < size; i += 1) {
      spawn(() => {
        while (true) {
          const task = this.take();     // blocks until something arrives
          try { task(); } catch (error) { report(error); }   // ← see below
        }
      });
    }
  }
}

Thread creation happens size times, at startup. After that, submitting work is an enqueue.

How big?

Two questions, and the second one is the one people skip.

How many cores? That is the ceiling for work that actually computes.

How much of each task is waiting? A task blocked on a database call is not using its core, so another thread could be.

\text{threads} \approx \text{cores} \times \left(1 + \frac{\text{wait time}}{\text{compute time}}\right)

CPU cores 8time each task spends waiting 0%

pool size to aim for: 8
threads per core: 1.0

Pure computation: one thread per core. Extra threads cannot compute anything — they only add context switches.

The queue is a policy decision

Unbounded. Never rejects, and hides overload until you run out of memory. Under sustained excess load the queue grows without limit and latency grows with it — every request eventually times out having waited behind a queue of requests that also timed out.

Bounded. Rejects when full, which is backpressure and is what you want. A fast, honest failure beats a slow, hopeful one.

What to do on rejection is also a choice: drop the task, block the submitter (which pushes back up the chain, usually correct), or run it on the calling thread.

Never submit a task and wait for it from inside the pool

pool.submit(() => {
  const part = pool.submit(subtask);
  return part.get();                    // ← waiting, while occupying a worker
});

With enough of these, every worker is waiting for a task that has no worker to run on. The pool is fully occupied and totally idle — a deadlock with no lock in it.

The fixes: a separate pool for subtasks, a work-stealing pool that can run the subtask on the waiting thread, or composing with futures instead of blocking.

Work stealing

A shared queue is a contention point: every worker hits the same lock to take a task.

Work-stealing pools give each worker its own deque. It pushes and pops its own end with no synchronisation, and only when empty does it steal from another worker’s far end.

That is Java’s ForkJoinPool, Go’s scheduler, Rust’s Rayon, and .NET’s default pool — and it is a much better fit for recursive divide-and-conquer work, where tasks spawn subtasks.

Where the model is changing

Virtual threads (Java 21), goroutines, and async/await all attack the same premise: that a thread is expensive.

Make the thread cheap — a few hundred bytes, scheduled in userspace — and you can have a million of them, blocking freely, with no pool at all. The runtime multiplexes them onto a small number of real threads.

That is genuinely simpler code, and it does not repeal Amdahl’s law or make the CPU-bound sizing question go away. It removes the pooling problem, not the parallelism problem.