A job that normally finished in under a minute sat for thirty-six. Nothing crashed. Nothing logged an error. The process just… waited. By the time we understood why, we had learned something that has nothing to do with the model being smart or dumb, and everything to do with how AI calls behave differently from every other network call your system makes.
Here is the post-mortem, and the rule we now apply to every LLM call we ship.
The symptom
We run an AI system that reads documents and produces judgments about them. Most of the work is fast — pull a value here, classify a field there, calls that return in a couple of seconds. One stage is slow by design: it hands the model a large, messy input and asks it to reason its way to a structured result. That call takes minutes on purpose.
One afternoon, a routine run hung. No exception, no failure, no partial output. The worker was alive; the queue behind it was backing up; the job was simply parked. Thirty-six minutes later it either completed or got killed — depending on which run you looked at — with no signal in between that distinguished "working hard" from "wedged."
That last part is the real problem. A slow call and a dead call looked identical from the outside.
The investigation
The first instinct was to blame the model or the input. It was neither. The input was well within limits, and when we replayed the same request in isolation it came back in the expected few minutes. The stall only showed up under real conditions — occasional network flakiness, a provider hiccup, a connection that half-opened and then went quiet.
So we stopped looking at the model and started looking at the transport. That is where we found it.
The root cause: inherited defaults
The provider's SDK ships with a default request timeout of 600 seconds, and a default retry policy of two automatic retries. Nobody chose those numbers for our workload. They came with the client, and we had never overridden them.
Do the arithmetic on a call that wedges instead of failing cleanly:
- First attempt hangs until the 600-second timeout fires.
- Retry one: another 600 seconds.
- Retry two: another 600 seconds.
That is twenty minutes of a single logical call sitting on a connection that was never going to return anything useful. Chain that behind the downstream step that was waiting on its output, plus the surrounding queue and backoff, and you land at the thirty-six-minute stall we saw. Every one of those seconds was the system patiently honoring a default it inherited rather than a limit we had reasoned about.
Worse, the retries were silent. A 600-second wait followed by a retry looks, from the outside, exactly like a model that is thinking. There was no heartbeat, nothing emitting "still waiting on attempt 2 of 3." The pipeline had no way to tell the difference between progress and paralysis.
The insight that actually matters: AI calls come in classes
Here is the part that generalizes beyond one bad default.
We reached for a fix — "set a sane timeout" — and immediately hit a wall. What is a sane timeout? Our fast extraction calls should return in seconds; if one takes thirty, something is wrong and we want to bail. But our slow reasoning call takes minutes by design; a thirty-second timeout would murder it on every single run.
There is no single correct timeout for "an LLM call," because an LLM call is not one kind of thing. AI calls come in latency classes, and they are wildly different:
- Extraction / classification calls — small input, small output, tight scope. Expected latency: seconds. If it is slow, it is broken.
- Reasoning / authoring calls — large input, open-ended generation, real work. Expected latency: minutes. Slow is the normal case.
A global timeout is wrong for both at once. Set it low enough to protect the fast calls and you guillotine the slow ones. Set it high enough to permit the slow ones and every fast call can now hang for ten minutes before you notice. The single knob cannot be in two positions.
This is the mistake behind the whole incident. We had been treating "call the model" as one operation with one set of transport settings, when in reality we were running two operations with opposite latency profiles through the same client.
The fix
Three changes, in order of importance.
1. Per-class timeouts and retries. We stopped configuring "the LLM client" and started configuring each class of call. Fast calls get short timeouts and aggressive failure — a few seconds, then give up, because a slow extraction is a signal, not something to wait out. Slow calls get generous but bounded timeouts sized to their real distribution, plus a retry budget chosen deliberately rather than inherited.
# Wrong: one client, one policy, inherited defaults
client = LLMClient() # timeout=600s, retries=2, for everything
# Right: policy per latency class
extract = LLMClient(timeout=8, retries=1) # seconds; slow == broken == fail fast
reason = LLMClient(timeout=180, retries=1) # minutes by design, but boundedThe specific numbers matter less than the principle: you own the timeout, you do not inherit it, and you set it per class.
2. Heartbeats on long calls. For any call expected to run for minutes, we emit a periodic signal while it is in flight — a token stream, a keepalive, a "still working, elapsed 45s" log. This is what lets the system, and the on-call human, tell "working hard" apart from "wedged." Without a heartbeat, a long call and a dead call are indistinguishable, and you will always assume the optimistic one until the queue is on fire.
3. Fail loud on retry. Every retry now surfaces — a metric, a log line you do not have to know to grep for, an alarm if retries exceed a threshold. A silent retry is a stall you will only discover from the symptoms downstream.
Why this matters if you are building with LLMs
Nothing here required a better model. The model was fine. The failure lived entirely in the plumbing — in the assumption that an AI call is like any other API call and can be governed by one timeout, one retry policy, one mental model.
It cannot. And this gets more important as your system grows more agentic, not less: as you add more model calls, more tools, and more chained steps, you multiply the number of places a wedged call can silently freeze everything behind it. More autonomy means more call sites, and every call site that inherits a default is a thirty-six-minute stall waiting to happen.
The transferable rules:
- Never inherit SDK defaults into production. The vendor's defaults are tuned for a demo, not for your workload.
- Set explicit, per-class timeouts and retries. "An LLM call" is not one kind of operation. Treat extraction and reasoning as the different beasts they are.
- Give long calls a heartbeat. If you cannot distinguish "working" from "wedged," you will always guess wrong.
- Make retries loud. Silent retries are stalls in disguise.
We found this one at thirty-six minutes. You can find it in a config review this afternoon.
We build and debug AI systems that have to work in production — not just in the demo. If your LLM pipeline is fast until it mysteriously is not, book a strategy call. No pitch deck, no sales pressure — just a conversation about what is breaking.