Sync over Async: Diagnose .NET Thread Pool Starvation

Somewhere in your codebase, right now, there is probably a line like this:

var customer = _customerService.GetCustomerAsync(id).Result;

It was written for an understandable reason. An async method needed calling from a synchronous spot (a constructor, an old interface, a method whose signature somebody didn't want to change), and .Result made the compiler happy in four keystrokes. It worked on the dev machine. It worked in the demo. It has worked every single day since, because the trap in sync-over-async isn't a bug you can reproduce with one request. It's a debt that only comes due under load.

What .Result actually does

await and .Result read like synonyms ("get me the value out of the task") and behave like opposites. When you await a task that isn't finished, your method gets out of the way: it returns to its caller, the thread goes back to useful work, and the rest of the method is scheduled to resume when the result arrives. When you call .Result on a task that isn't finished, the thread stands there. It blocks, holding every resource it represents, doing nothing, until the task completes. (.Wait() and .GetAwaiter().GetResult() are the same standing-still with different exception wrapping.)

await

thread returns to the pool during I/O

3 ms work · 40 ms I/O on no thread · 3 ms continuation

.Result

thread blocked for the entire wait

one thread held hostage for all 46 ms

One blocked thread is invisible. The failure modes come from what that blocked thread was for, and they differ by decade.

Failure mode one: the classic deadlock

In runtimes with a synchronization context (WPF and WinForms UI threads, classic ASP.NET), the context is a queue that says "continuations run back on this thread." That makes .Result a perfect self-inflicted deadlock:

public string GetGreeting()
{
    // UI or classic ASP.NET context:
    return GreetAsync().Result;   // blocks the context thread, waiting for the task...
}

private async Task<string> GreetAsync()
{
    var name = await LoadNameAsync();  // ...whose continuation is queued to run
    return $"Hello, {name}";           //    on the thread we just blocked.
}

The thread is waiting for the task; the task's continuation is waiting for the thread. Neither yields. The app hangs forever, no exception, no log line, and this exact shape has burned .NET developers for a decade. (This is also the real reason library authors write ConfigureAwait(false): it lets the continuation run elsewhere, so a library awaiting internally doesn't participate in the caller's deadlock. It is not a blessing to block.)

Failure mode two: the modern starvation cliff

ASP.NET Core removed the synchronization context, and a generation of developers concluded that .Result is now safe. The deadlock is gone; the cost moved somewhere subtler. In ASP.NET Core, requests are served by thread pool threads, and the pool is sized for threads that work, not threads that stand around. Every request that blocks on .Result pins one pool thread for the full duration of the I/O it's "waiting" on.

Now apply load. Fifty concurrent requests each block a thread for a 200 ms downstream call, and fifty pool threads are standing still. New requests arrive and there's no thread to run them, so they queue. The pool notices and grows, but growth cannot make blocked work free. Newer runtimes detect some blocking patterns and inject threads faster, which can soften the cliff without removing the extra scheduling, memory, and context-switching cost. Latency doesn't degrade gracefully; it's fine, fine, fine, and then the queue outruns the trickle and p99 goes vertical. The classic incident shape: "the app falls over at traffic spikes and recovers on its own," and nothing in any stack trace looks wrong, because standing still isn't an error.

The async version of the same endpoint releases the thread at every await. The same fifty in-flight requests hold roughly zero threads while their I/O is out; threads only run code that's actually running. That's the entire point of async in a server: it's not about making one request faster, it's about not paying a thread per waiting request.

Async I/O does not make the dependency faster. It keeps the waiting request from occupying a worker thread that could serve somebody else.

How to diagnose .NET Thread Pool starvation

The production symptom is usually broader than one slow endpoint. Request latency climbs during a traffic spike, cheap health checks slow down with expensive requests, CPU remains surprisingly low, and the service recovers after traffic falls. That combination is a useful lead, not a diagnosis: lock contention, synchronous I/O, and other blocking work can starve the same pool.

Start with live runtime counters while the service is slow:

dotnet-counters monitor -n Checkout.Api --counters System.Runtime

The exact labels differ on older runtimes, but the useful relationship is stable:

• Worker threads: dotnet.thread_pool.thread.count keeps climbing because the runtime is adding workers to compensate for blocked ones.
• Queued work: dotnet.thread_pool.queue.length stays elevated or grows because work is arriving faster than free workers can start it.
• CPU usage: remains well below saturation because the process lacks runnable workers, not processor capacity.
• Request latency: rises with the queue because even cheap work waits behind blocked requests.

A large thread count by itself is not proof. The revealing moment is the ramp: threads rising, queued work waiting, and CPU still available. Since .NET 6, the pool responds faster to several known blocking Task APIs, so the starvation window may be shorter, but the blocked threads and their memory still exist. Microsoft's current ThreadPool starvation tutorial uses the same counter pattern before moving from detection to stacks.

Capture what the workers are waiting on

If the incident is continuous, inspect the process immediately:

dotnet-stack report -n Checkout.Api

Group repeated stacks mentally rather than hunting for one exotic frame. Dozens of worker stacks ending in Task<T>.Result, Task.Wait, GetAwaiter().GetResult(), SemaphoreSlim.Wait, or a synchronous network/database API tell you where the pool is being consumed. Confirm that the stacks belong to Thread Pool workers; a dedicated thread waiting by design is not starvation.

Intermittent incidents need evidence collected during the stall. On .NET 9 or later, capture the runtime's wait events for a bounded window:

dotnet-trace collect -n Checkout.Api \
  --clrevents waithandle \
  --clreventlevel verbose \
  --duration 00:00:30

The resulting trace preserves the call stacks that blocked even if the service recovers before you attach a debugger. For older runtimes, collect a general runtime trace or a dump during the incident and group the worker stacks in PerfView, Visual Studio, or dotnet-dump.

Treat a higher minimum worker-thread count as a diagnostic experiment, not the repair. It can shorten the pool's ramp-up and buy incident time, but it leaves every blocking call in place and moves the failure threshold to a larger traffic spike.

Test the repair under sustained concurrency, not with a single request and a stopwatch. Throughput should recover, queue length should remain controlled, and tail latency should have headroom. That evidence distinguishes a real async fix from a signature-only rewrite around work that is still synchronous underneath.

Unwinding it

The fix is unglamorous: make it async all the way up. Find the blocking call, make its containing method async, await instead, and let the signature change ripple upward until it reaches something that's already async (in ASP.NET Core, your endpoint or handler already is). The ripple is annoying precisely once, which is cheaper than the alternative in every currency.

// Before: sync facade over async guts
public Invoice GetInvoice(Guid id)
{
    var invoice = _repo.LoadAsync(id).Result;
    var pdf = _pdfService.RenderAsync(invoice).Result;
    invoice.AttachPdf(pdf);
    return invoice;
}

// After: the signature tells the truth
public async Task<Invoice> GetInvoiceAsync(Guid id)
{
    var invoice = await _repo.LoadAsync(id);
    var pdf = await _pdfService.RenderAsync(invoice);
    invoice.AttachPdf(pdf);
    return invoice;
}

Two boundaries deserve honesty rather than tricks. Constructors can't be async; move the work to a factory method or lazy initialization instead of blocking in new. And a genuinely synchronous third-party interface you must implement is the one place a documented, quarantined block may be the least-bad option; the sin is scattering casual .Result through code you do control because propagating async felt like a chore.

Do not wrap synchronous I/O in Task.Run inside an ASP.NET Core request and call that asynchronous. The work still occupies a Thread Pool thread, plus an extra scheduling hop. Prefer the dependency's async API; if none exists, bound that concurrency explicitly or move long-running work behind a queue. Microsoft's ASP.NET Core performance guidance makes the same distinction between asynchronous I/O and merely relocating a block.

The test suite will never tell you

Here's what makes this pattern durable in real codebases: it is completely invisible to functional testing. One request at a time, the blocking version returns the same bytes as the async version, often with no obvious difference. Ordinary functional tests rarely create enough sustained concurrency to exhaust the pool. The bug is concurrency, and a feedback loop that never applies load will not expose it before production does.

That gap, between "passes the tests" and "survives production," is the one Katabench exists to close. The grading measures what production measures (wall-clock against a budget, allocations, the queries you fired) on the reasoning that you get good at exactly what your feedback loop grades, and a loop that only checks correctness trains you to write code that is merely correct. The how-it-works page shows what we measure and how. Sync-over-async is a concurrency lesson, and the habit that catches it (distrust code that's only ever been proven correct, one request at a time) is the habit every track is built to train. If you are auditing a service rather than responding to an incident, use the broader C# async/await mistakes checklist to catch the adjacent failure modes before load makes them visible.