Retry is not failover, and HTTP 200 is not a finished answer

01HTTP/1.1 200 OK

Retry and failover are not the same move

Start with the definitions, because almost every confused incident report in this area starts with the word "retry" being used for both.

A retry is the same request, sent again, to the same place. Same provider. Same base URL. Same model. Same prompt. The only thing that changes is the attempt number and, if the client is well built, the wait before it. The assumption underneath a retry is: the route is fine, the moment was bad. A packet was lost, a load balancer recycled a connection, a server had a hiccup, a queue was momentarily full. If you ask again in a moment, you will probably get the answer you wanted.

A failover is a different request path. Different provider, or a different model on a different endpoint, or a different region. The assumption underneath a failover is: the route is broken, the moment is irrelevant. The primary is down, throttled, misconfigured, out of credit, or answering 200 with nothing in it. Waiting will not help, because waiting re-buys the same broken thing.

These are not two settings on one dial. They are two independent layers, and each layer needs things the other does not share:

The two layers interact in one specific, dangerous way: whichever layer runs first spends the clock that the second layer needed. If the retry layer is allowed to run to exhaustion before the failover layer is even consulted, then the failover budget is not measured in attempts — it is measured in whatever time is left. On a slow failure, there may be no time left at all.

02HTTP/1.1 200 OK

The vocabulary you need before we go further

Five terms, defined once, used for the rest of the lecture.

Timeout. A deadline attached to an operation: if the answer has not arrived by time t, stop waiting and treat it as a failure. A timeout is not an error report from the other side. It is your own decision to stop. This distinction matters more than it sounds, because a timeout tells you nothing about what the server did — only that you stopped listening.

Idempotent. An operation is idempotent if doing it twice has the same effect as doing it once. Setting a value to 7 is idempotent; adding 7 to a counter is not. RFC 9110 §9.2.2 defines it for HTTP as "the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request", and notes such requests "can be repeated automatically" after a communication failure. Idempotency is the precondition that makes a retry safe; without it, a retry gambles that the first attempt did not already take effect.

Circuit breaker. A wrapper around a call that counts recent failures and, past a threshold, stops calling for a while. Three states: closed (normal), open (calls fail immediately without touching the network), half-open (a few trial calls test whether the far side recovered). Azure's documentation adds a detail worth keeping: the closed-state counter is time-based and resets periodically, so occasional failures do not trip the breaker — only a rate of failure does.

Jitter. Randomness added to the wait between retries. Without it, every client that failed at the same moment waits the same amount and retries at the same moment, producing a second, self-inflicted spike. With it, the retries smear out. The AWS Architecture Blog's 2015 article on the topic shows the effect directly: in a simulation of 100 contending clients, adding full jitter more than halved the total number of calls compared with un-jittered exponential backoff, and also reduced the time to completion.

SSE (Server-Sent Events). A way of sending a response body as a long series of small text events over a single ordinary HTTP response. Each event is a few lines of text ending in a blank line; the payload is prefixed with data:. The WHATWG HTML Standard defines the wire format and the browser EventSource API that consumes it. A streamed model completion is exactly this: an HTTP response whose body is a sequence of SSE events, each carrying a token or a metadata update.

Two more terms will come up in Part II and are defined there: TTFB (time to first byte) and chunked transfer encoding.

03HTTP/1.1 200 OK

What actually happens when a stream drops in Hermes

This section walks the code. Nothing here is inferred from behaviour; it is read from /usr/local/lib/hermes-agent on 2026-10-18, with file and line references so you can check it yourself.

Layer 1 — the stream retry loop. In agent/chat_completion_helpers.py, _call() reads HERMES_STREAM_RETRIES with a default of 2, then loops while _stream_attempt < _max_stream_retries, starting at −1. Three attempts, then. Each attempt opens a fresh connection to the same base_url with the same payload. When an attempt fails before any visible text has been delivered, _handle_stream_error decides whether to loop again.

Layer 2 — the classification. When the stream retry budget is exhausted, _handle_stream_error logs a kind="exhausted" diagnostic, sets result["error"], and returns False. That error propagates to the outer API retry loop, which calls classify_api_error (agent/error_classifier.py:974). The classifier runs a list of pattern-matching stages; the first that matches wins, and if none match the verdict is _V_UNKNOWN (error_classifier.py:479). _V_UNKNOWN is _v(FailoverReason.unknown) with no hints, which means it inherits the ClassifiedError defaults: retryable=True, should_fallback=False (error_classifier.py:81-84).

Read that again, because it is the whole lecture in one line: an error nobody recognises is, by default, retried and not failed over. That default is not stupid — most unrecognised errors really are transient — but it is a choice, and it is the choice that cost this box fifty minutes.

Layer 3 — the outer API retry loop. _run_api_retry_loop (agent/conversation_loop.py:1497) is while s.retry_count < s.max_retries, with max_retries set per API-call block from agent._api_max_retries, itself read from agent.api_max_retries with a default of 3 (agent/agent_init.py:1437). Each iteration calls the model including the whole stream retry loop from layer 1. So the two layers are nested, not sequential: three stream attempts inside each of three API attempts.

Layer 4 — the failover. Only when retry_count >= max_retries does agent/turn_api_error.py reach the failover branch. Lines 377–380: if a fallback is pending, buffer the status line ⚠️ Max retries (N) exhausted — trying fallback..., then call _try_activate_fallback(reason=classified.reason, ...). If that returns true, the turn restarts against the fallback model. Note the detail in the comment at the top of that handler: retry status is buffered and only flushed "when every retry+fallback is exhausted". So in a turn that succeeds on the fallback, the human may never see the "max retries exhausted" line at all. The symptom is a slow answer, not a visible error.

The arithmetic. Three stream attempts × three API attempts is a worst case of nine requests to the primary before a single request reaches claude-haiku-4.5. The audit that prompted this lecture said six. Six is what actually occurred on 2026-10-15 before the failure changed shape; nine is the ceiling the code allows. Both numbers are worth knowing, and section 4 shows the receipts.

04HTTP/1.1 200 OK · bytes=0 · chunks=0 · ttfb=-

The arithmetic that bites: a slow failure is not a cheap failure

Here is the sequence from /root/.hermes/logs/errors.log, session 20261014_093323_c971f3fb, model deepseek-v4-pro, context ~305k tokens. I am quoting timestamps and the fields the log actually printed.

2026-10-15 02:15:35  Stream stale for 600s (threshold 600s) — no chunks received.
2026-10-15 02:15:35  Stream drop on attempt 1/3 — retrying.  http_status=200 bytes=0 chunks=0
                     elapsed=600.13s ttfb=- finish_reason_seen=False
2026-10-15 02:25:36  Stream drop on attempt 2/3 — retrying.  elapsed=600.20s bytes=0 chunks=0
2026-10-15 02:35:38  Stream exhausted on attempt 3/3 — retrying. elapsed=600.05s bytes=0
2026-10-15 02:35:38  API call failed (attempt 1/3) error_type=ReadError
2026-10-15 02:35:38  Retrying API call in 2.3057117637756166s (attempt 1/3)
2026-10-15 02:45:40  Stream drop on attempt 1/3 — retrying.  elapsed=600.02s bytes=0 chunks=0
2026-10-15 02:55:42  Stream drop on attempt 2/3 — retrying.  elapsed=600.26s
2026-10-15 03:05:44  Stream exhausted on attempt 3/3 — retrying. elapsed=600.02s
2026-10-15 03:05:44  API call failed (attempt 2/3) error_type=ReadError

Six attempts. Fifty minutes of wall clock. Zero bytes delivered on any of them, and ttfb=- throughout, meaning no first byte ever arrived. The backoff between the two API attempts was 2.3 seconds — a sensible, jittered, well-behaved backoff that is, in this failure mode, a rounding error. The expensive thing was never the wait between attempts. It was the timeout inside each attempt.

That is the arithmetic that bites. A retry re-buys the full timeout. If each attempt is allowed 600 seconds to produce its first chunk, and the failure mode is "produces nothing, then the socket dies", then N attempts cost N × 600 seconds before the failover layer is consulted. The retry count is not the budget. The retry count multiplied by the timeout is the budget, and it has to fit inside whatever the job can afford.

Where does 600 come from? Not from a run budget — the audit's phrasing ("the 600 s run budget") is off. agent/reasoning_timeouts.py keeps a per-model stale-timeout floor for known reasoning models, and deepseek-v4-pro is in the 600-second group, because reasoning models routinely think longer than the 180-second default before the first token arrives. The stale detector logs "Stream stale for 600s (threshold 600s)" and kills the connection. So the 600 seconds is a deliberate, defensible patience setting for a thinking model. It is also, in a hanging failure, the price of one retry.

05HTTP/1.1 200 OK

The theory, properly

This is the part most agent-stack users skip, because their stack already has retries wired up and the defaults usually work. They usually do work — until the failure is slow, or ambiguous, or returns 200 with nothing in it. Then the defaults decide your night. Learn the theory and the defaults stop being mysterious.

5.1 At-least-once and at-most-once

Every system that can fail mid-request has to answer one question: when the outcome is unknown, do you risk doing the work twice, or risk not doing it at all?

At-most-once means: send it once, never retry. You can lose the work, but never duplicate it. Safe for charging a card or sending an email.

At-least-once means: keep trying until you get an acknowledgement. You can never lose the work, but you can duplicate it. Safe where a duplicate is harmless or detectable — a cache warm, a metrics write.

There is no third option that is both, without cooperation from the receiver. The marketing name for that cooperation is exactly-once, and it is not a property of a retry loop; it is a property of an idempotent receiver plus a deduplication key. If the receiver can recognise "I have already seen this request", the sender can retry freely and the system behaves as exactly-once. If it cannot, then at-least-once means "at least once, possibly more, and you will not necessarily know".

Model API calls sit in an interesting spot here, which is why this belongs in an agent course. A completion request is not idempotent at the provider — you will be billed again for a second full generation — but it is harmless to repeat in the sense that a duplicate answer does not corrupt state by itself. Where it stops being harmless is one level up: if the model's output is the input to a tool call that charges a card or writes a file, then retrying the model call retries the side effect. The idempotency question is not about the model. It is about the thing the model is about to do.

The RFC's guidance is the general form of this: retry automatically only when the request is idempotent, or when you have "some means to detect that the original request was never applied" (RFC 9110 §9.2.2). A tool call has no such means unless you build one.

5.2 Bounded retries

A retry loop with no bound is a denial-of-service attack on your own stack, executed politely and one attempt at a time. Every production retry policy has three bounds, and they are independent:

1. Attempt bound — a maximum number of tries. AWS's SDK documentation makes the convention explicit: max_attempts counts "total attempts including the initial request", so a value of 3 means one request and up to two retries. Note the off-by-one: retries is one less than attempts, and status messages in the wild use both words for both meanings.
2. Time bound — a deadline for the whole operation, not per attempt. This is the one agent stacks usually lack. "Three attempts" sounds bounded; "three attempts at a 600-second timeout each" is a 30-minute commitment, which is not what anyone means by bounded.
3. Concurrency or cost bound — how many retries may be in flight or how much they may cost. AWS's newer SDK retry behaviour implements this as a retry quota: a token bucket that deducts tokens per retry and refills on success. When the bucket is empty, the SDK returns the error instead of retrying, "so your application fails fast instead of waiting through retries that are unlikely to succeed." That is a budget in the strict sense: a global resource that retries spend.

A retry budget generalises that third bound: instead of a fixed count per call, cap retries as a fraction of total traffic. When the backend is healthy you never notice; when it is unhealthy, the cap stops your retry traffic from being the thing that keeps it unhealthy. Google's SRE Book lists retrying on failures among the changes that can "improve the chance of causing a large-scale failure."

5.3 Backoff, and backoff with jitter

Backoff means waiting longer between successive retries: 1 second, then 2, then 4, then 8, up to a cap. The point is not politeness. It is that the failure you are retrying through is, in the aggregate case, caused by load — and hammering it at a constant rate is the one thing guaranteed to keep it loaded. Wikipedia's article on exponential backoff gives the canonical description: the pause is multiplied by a fixed factor per adverse event, and the frequency falls inversely.

Jitter is randomisation of that pause. AWS's article is the best free treatment, and its simulation results are the part worth remembering: capped exponential backoff alone still produces clusters of calls, because every client multiplies by the same constant and arrives at the same instants. Add jitter and "the gaps are gone". Three named variants:

• Full jitter: sleep a uniform random amount between zero and the computed backoff. Least work, slightly more time.
• Equal jitter: keep half the backoff and randomise the other half. Avoids very short sleeps; in the simulation it did slightly more work than full jitter and took longer.
• Decorrelated jitter: increase the random ceiling based on the last random value. More calls than full jitter, but more spread out.

Hermes uses a jittered backoff for the outer loop: jittered_backoff(retry_count, base_delay=2.0, max_delay=60.0) in agent/turn_recovery.py, called from compute_error_backoff. That is why the 2026-10-15 log shows Retrying API call in 2.3057117637756166s for the first retry — base 2 seconds plus jitter. It respects Retry-After when the provider sends one, capped. This part of the stack is textbook-correct. The problem was never the backoff.

Two cautions. Backoff only helps when attempts are cheap or the failure is load-induced; against a hang, the timeout dominates and the backoff is noise. And jitter solves synchronisation, not volume — AWS's own conclusion is that it does "not fundamentally change the N² nature of the work to be done."

5.4 Circuit breakers, and the half-open state that earns them

A retry loop's weakness is that it is stateless per call. It has no memory that the last twelve calls to this provider failed. So a retry loop will keep politely asking a dead endpoint, one request at a time, forever.

A circuit breaker is the memory. Azure's documentation describes the three states and the useful details:

• Closed: calls pass through; every failure increments a time-based counter. The counter resets periodically, so a rare failure never trips the breaker.
• Open: calls fail immediately, without touching the network. A timer runs.
• Half-open: when the timer expires, a limited number of trial requests are allowed through. Success closes the breaker and resets the counter; any failure re-opens it and restarts the timer. The docs note why this matters: half-open "helps prevent a recovering service from suddenly being flooded with requests."

The half-open state is the design idea worth stealing even if you never write a breaker. It is the formal statement of "probe before you trust". A recovering provider can often serve one request before it can serve a hundred, and the breaker's job is to find out with one.

Azure's page also draws the distinction this lecture is about: retry assumes the operation will eventually succeed; a breaker assumes it is "likely to fail" and stops asking. And a warning that maps onto nested agent loops: if a task with a retry policy invokes another with a retry policy, "this extra layer of retries can add long delays", and the lower layer may be better off failing fast and reporting upward.

5.5 Retryable versus terminal

Every error belongs to one of three classes, and the class — not the code, not the message — decides the action.

Retryable. The same request, sent again, has a reasonable chance. Connection reset, brief 5xx, throttling with Retry-After, a stalled stream. Retry with backoff, in a bounded loop.

Terminal. The same request fails the same way forever. Bad key, model not found, unsupported parameter, policy filter, context too long. Retrying is waste — and actively harmful when it burns the budget another error needed.

Unknown. You cannot tell which of the two it is — most errors, in practice, because error surfaces are messy: HTML error pages, rewritten status codes, SDK-specific exception types, and streams that simply stop.

The cost is asymmetric. Treat unknown as retryable and a terminal error eats the entire budget. Treat it as terminal and a transient blip kills a turn one more attempt would have saved. Decide by asking what does one more attempt cost, and what does a wrong handover cost — not by defaulting.

06HTTP/1.1 200 OK

What makes an error "known" enough to act on

A classifier is a list of patterns and a default. That is all it is. The quality question is not "how many patterns", it is "what happens to the ones that match nothing", and the answer has to be designed rather than inherited.

Look at what a classifier can actually see, in order of reliability:

1. Structural signals from the protocol. Status code, finish_reason present or absent, content-length versus bytes received, whether TLS completed, whether the connection closed cleanly. Near-truth: the server or the transport asserted them.
2. Provider-declared error codes. insufficient_quota, model_not_found, overloaded_error. Reliable when present, absent more often than you would like, inconsistent between providers.
3. Message text. "maximum context length is 163840 tokens". Useful, brittle, and the place where regex classifiers quietly rot as providers reword.
4. Exception type. ReadTimeout versus ConnectTimeout versus ReadError versus RemoteProtocolError — where in the exchange it broke, which often picks the recovery.
5. Nothing at all. A socket that closed. An empty body. A 200 with one frame.

Known errors are cheap because the action is written down: 401 → rotate or fail over; 429 → wait the declared time, then retry; context overflow → compress, then retry; model not found → fail over, do not retry. The whole point of classification is to convert an ambiguous failure into one of those rows.

Unknown errors are expensive for one reason: the default action for unknown is almost always "retry the same thing", and that is the action with the worst worst-case. Retrying a known-terminal error wastes a bounded amount of time. Retrying an unknown error can waste the entire budget, because you do not know which failure you are in — and the expensive failures are exactly the ones that produce no information.

That is the asymmetry the 2026-10-15 incident exposes. The failure produced http_status=200 bytes=0 chunks=0 ttfb=-, which is a very specific and very damning signature: the connection was established, the response headers arrived, and then nothing, for ten minutes. A classifier that read bytes == 0 and chunks == 0 and ttfb is None and elapsed >= timeout would know something no message-text regex could: this route is delivering headers and nothing else. That is not a blip. That is a route-level failure, and it deserves a failover, not a third attempt at the same URL.

6.1 When should "unknown" become a failover?

The design question, answered plainly.

Promote an unclassified error to failover when at least one of these is true:

• It produces no bytes. Zero bytes, zero chunks, no TTFB, after the connection was established. Headers-only is a route problem. (Compare: 150 KB delivered and then a reset is a different thing — see section 9.)
• It is expensive to retry. The failure costs a full timeout, or a large fraction of the operation's deadline. A cheap unknown gets one more try; a 600-second unknown does not.
• It has already happened twice in a row. Two identical unknown failures from the same route in the same turn is not bad luck, it is a pattern. Retrying the same thing a third time is not evidence-gathering, it is habit.
• The remaining budget cannot afford another attempt plus the handover. If the fallback needs time to work, the decision has to be made while that time still exists.
• The failure is route-shaped, not request-shaped. Hanging with no bytes is characteristic of a specific edge path (in our logs, a CloudFront-fronted ELB). A different route is not just a different server; it is a different failure surface.

Do not promote when:

• The error is explicitly retryable and cheap (429, connection reset before headers, a single 502).
• Nothing has been tried yet — one attempt is not a diagnosis.
• The fallback is strictly worse for the task (different tool-calling support, different context window, different modality). A failover that produces a wrong answer is worse than a retry that produces none.
• The failure was caused by you (a cancellation, an interrupt, a client-side eviction). Section 9 has a real example of exactly that.

6.2 What "fail fast" means concretely

"Fail fast" is one of those phrases that sounds like a virtue and specifies nothing. Concretely, in this stack, it means five changes:

1. A deadline for the whole operation, computed before the first attempt: deadline = now + budget. Every attempt and every backoff sleeps against it. When it is gone, stop — do not start an attempt you cannot finish.
2. A per-attempt timeout that is a fraction of the budget, not a constant that dwarfs it. If the operation has 20 minutes, three attempts at 600 seconds is defensible; three attempts at 600 seconds inside a 10-minute budget is a hang with extra steps.
3. A rule that unknown-but-expensive means failover. Written into the classifier, not into the operator's head at 3 a.m.
4. A failover that does not re-run the whole retry ladder. One attempt on the fallback, then a second provider, then stop. If the fallback also needs three stream retries and three API attempts, you have not failed over, you have doubled the hang.
5. A visible terminal state. When everything is exhausted, the turn must end with a reason a human can read, not with completed: true and an empty message. This is the subject of Part II.

07HTTP/1.1 200 OK

Four layers, four different meanings of "the stream ended"

When you call a model over HTTP with streaming enabled, you are not making one request. You are making four nested agreements, and each one can end independently of the others. Almost every confusing incident report in this area comes from collapsing them into one.

Layer 1 — TCP and TLS. A socket is open and TLS negotiated. "The connection ended" means the socket closed or reset — Connection reset by peer, Broken pipe, a TLS close_notify, a timeout. Nothing at this layer knows what an HTTP response is.

Layer 2 — the HTTP response. A status line and headers arrived. "The response ended" means the body is complete according to its framing: the octet count promised by Content-Length, or the terminating zero-length chunk of a chunked body, or the old ambiguous way — the connection closed and the client decided that was the end. RFC 9110 §6.1: a response ended by connection close "is considered complete even though it might be indistinguishable from an incomplete response, unless a transport-level error indicates that it is not complete."

Layer 3 — the SSE event stream. The body is a sequence of text events. "The event stream ended" means the body ended, which is not the conversation ending: the WHATWG spec has EventSource reconnect automatically, and lets a server say "do not reconnect" with a 204. In a completion, the stream is expected to end with a terminal event carrying the completion status.

Layer 4 — the assistant message. The application object: text, tool calls, usage. "The message ended" means the server declared it complete by naming why generation stopped.

Here is the sentence this lecture exists to install: HTTP 200 is a layer-2 statement. RFC 9110 §15.3.1 defines it as "the request has succeeded", and in a streaming completion the request succeeding means the server agreed to start answering — headers, status, a body that will arrive. The body can then end anywhere. The status code was already spent.

The practical consequence: any client that treats "200" as "answer" has a hole in it that is exactly as large as the network's ability to drop a body. On this box that hole has been observed at three different sizes, from one SSE frame to 150 KB, and the largest one looked, to the code, exactly like a successful short answer.

08HTTP/1.1 200 OK

finish_reason, stop_reason, and what the server is supposed to tell you

Different vendors use different names for the same idea. Both are a small enum sent in the last event of a stream, saying why generation stopped.

OpenAI (Chat Completions). Each streamed chunk has choices[].delta and an optional finish_reason. The API reference defines the field as "optional … or null", with the value documented as: stop if the model hit a natural stop point or a provided stop sequence, length if the maximum number of tokens was reached, content_filter if content was omitted by a content filter, tool_calls if the model called a tool, plus the deprecated function_call. In practice: the normal chunks carry null, and the final chunk carries the value. A stream that ends without a non-null finish_reason somewhere did not complete a message.

Anthropic (Messages). The final event is message_stop, and the stop_reason rides on a message_delta event just before it: end_turn, max_tokens, stop_sequence, tool_use, refusal, pause_turn. The streaming docs are worth reading for one detail beyond the enum: the API may send error events inside the stream, for example overloaded_error where a non-streaming call would have returned HTTP 529. So "the stream ended" and "the answer ended" diverge even when there is no transport failure at all — the server can tell you, mid-body, that it gave up. The docs also describe explicit recovery: capture the partial response and resume from where it stopped, rather than re-generating the whole thing.

That recovery detail is the single most useful idea in Part II, and it applies to every provider with a chat-completions-compatible surface: a partially delivered answer is state you can keep. If 3,386 chunks arrived before the reset, you have a prefix. The question "should I resume or retry" is a different and much better question than "should I retry".

8.1 The distinction the enum encodes, in plain language

stop (or end_turn) means: the model decided it was finished. length (or max_tokens) means: we cut it off at your limit. tool_calls (or tool_use) means: the model stopped to ask you to do something. content_filter (or refusal) means: we stopped it for policy reasons. Every one of these is a completed message in the protocol sense, and three of the four are not a completed answer in the task sense.

That is a second, subtler trap, and it is worth naming because it is invisible to retry logic. A completion that ends with finish_reason=length is perfectly well-formed HTTP, perfectly well-formed SSE, and a truncated answer. If your code only asks "did the stream end cleanly?", the answer is yes. If it asks "is the work done?", the answer is no. Agent stacks need both questions, asked separately.

09HTTP/1.1 200 OK · bytes=40 / 149,940 / 0

Three real shapes, three different recoveries

These are from this machine's logs, all in the week of 2026-10-12 to 2026-10-18. I am quoting the fields as printed, then saying what each one means and what recovery it deserves. This is the most useful table in the lecture; the rest of the section is why.

(a) One frame and nothing else. 40 bytes is roughly one small SSE frame — enough for a role delta or an empty-choices event, nowhere near enough for an answer. Elapsed 0.82 s equals TTFB 0.82 s, so the server answered immediately and then stopped. This is not a slow provider; it is a broken completion. Compare it to a legitimate very short answer, which would still carry a finish_reason. The absence of that field is what separates "40 bytes of nothing" from "40 bytes of 'Yes.'". Recovery: this is cheap (under a second), so one retry is defensible; if the second attempt returns another frame-only stream, fail over. It is also the case where a same-provider retry has the best odds, because the failure is fast and looks like a per-request hiccup rather than a route outage.

(b) 150 KB and then a reset. 3,386 chunks over 29.17 seconds, first byte at 0.59 s, then ECONNRESET. Something was genuinely being generated for half a minute — and here the honesty requirement bites, because the surrounding log lines are not a provider story at all: at 17:13:20,571 the shared client was retired for cache eviction with tcp_force_closed=1, at 17:13:20,604 the turn ended with reason=interrupted_during_api_call, and the gateway was processing a /reset. The drop was client-initiated. A session reset killed a live stream, and the logger dutifully recorded a ReadError with http_status=200.

That is the most instructive of the three, because it shows why "unknown" is not the same as "provider fault". A classifier reading only error_type sees a network error and considers retrying; there is nothing to retry, because we closed the socket. Recovery for (b) is neither retry nor failover: it is cancellation handling. The lesson: the classifier must know who initiated the close.

(c) Nothing ever arrived, six times. Zero bytes, zero chunks, no TTFB, 600 seconds each, six times, over fifty minutes. The socket was established (200 and headers), the body never started. This is the hanging failure from section 4. Recovery: no. Not a sixth retry, not a seventh. This shape is the strongest possible argument for an early failover: the server had ten minutes to send one byte, six times in a row, and did not. Continuing is not persistence; it is a fixed cost with a known result.

The three shapes have almost the same log signature — http_status=200, finish_reason_seen=False — and completely different correct responses. That is why a single boolean "did the call succeed" is not enough, and why the diagnostic fields exist.

10HTTP/1.1 200 OK

The transport under it: framing, keep-alive, TTFB

Three mechanics that explain why a body can end without anyone lying.

Content-Length versus chunked. A response body is delimited either by a byte count (Content-Length: 1234) or by chunked transfer encoding, where the body is a series of length-prefixed chunks ending in a zero-length chunk. RFC 9112 §7.1: "The chunked transfer coding is complete when a chunk with a chunk-size of zero is received, possibly followed by a trailer section, and finally terminated by an empty line." And §6.3 sets the precedence for how a recipient decides where a body ends. For a streamed completion, Content-Length is impossible in principle — the server does not know how long the answer will be — so SSE responses are chunked or connection-delimited. Which means: the only in-band way for the server to say "this is the whole body" is the zero chunk, and the only in-band way to say "this message is whole" is finish_reason/message_stop. Those are different statements at different layers, and a truncated body can produce the first without the second.

Keep-alive and idle timeouts. HTTP connections can persist across requests (Connection: keep-alive). Intermediaries enforce idle timeouts because a connection that is open and doing nothing still costs them a slot. A streaming response that goes quiet — a reasoning model thinking, a provider buffering — looks identical to an idle connection, which is why providers send ping events (Anthropic does exactly this) and why clients implement their own stale detectors. Hermes' detector is the one in these logs: "Stream stale for 600s (threshold 600s) — no chunks received. Killing connection." That kill is our decision, and it is the correct one — but note what it produces: a ReadError (Broken pipe) that looks like a network fault. The error type describes the mechanism of the abort, not its cause. The cause was a timeout we chose.

TTFB is not elapsed. TTFB (time to first byte) is the interval between sending the request and receiving the first byte of the response. Total elapsed is how long the whole thing took. They measure different things: TTFB measures how long the server took to start, elapsed measures how long it took to finish. MDN's PerformanceResourceTiming.responseStart documents the browser-side equivalent — the timestamp "immediately after the browser receives the first byte of the response" — which is how the same distinction is measured on the web.

Why it matters for classification: in the 2026-10-15 case, ttfb=- (no first byte at all) with elapsed=600s means the failure was before generation even began. In case (b), ttfb=0.59s with elapsed=29.17s means the server was healthy for half a minute and something interrupted the middle. Same error class in the log, opposite diagnoses. A single elapsed field would have hidden that; TTFB is what turns "it took 600 seconds" into "it never started".

11HTTP/1.1 200 OK

Connecting the two halves

Put the two parts together and the 2026-10-15 incident becomes one sentence: the signal that would have justified a failover was present in the log and absent from the classifier. Part I said unknown errors default to retry-same-thing, and retrying a 600-second hang is the most expensive thing the stack can do. Part II said a stream with no bytes, no chunks, no TTFB and no finish_reason is not unknown — it has a shape. It is a route that accepts connections and does not generate.

The classifier's input should include what the transport knows: http_status, bytes, chunks, ttfb, finish_reason_seen, elapsed, and who closed the socket. With those, the taxonomy stops being a guess:

• bytes > 0, finish_reason absent, elapsed short → likely a per-request hiccup or a provider-side error event; retry once, then fail over.
• bytes > 0, finish_reason absent, elapsed long → partial completion; resume if the provider supports it, otherwise fail over and say so.
• bytes == 0, chunks == 0, ttfb absent, elapsed ≈ timeout → route failure; fail over immediately, do not retry.
• finish_reason present and non-stop → not a transport failure at all; handle length/content_filter at the task layer.

Note that only the first of those four is a retry. The default today makes all four a retry, and the fourth is not even an error.

12HTTP/1.1 200 OK

Nine rules for an agent stack

These are stated as rules because they are meant to be checked against a running system, not admired.

Rule 1 — Give the operation a deadline, not the attempt. Compute deadline = now + budget before the first request and check it before every retry. Three attempts at 600 seconds is a 30-minute operation whether or not the caller asked for one.

Rule 2 — Make the retry you can afford the retry you run. Fast failures are cheap and deserve retries; slow failures are expensive and deserve one. Otherwise the budget is spent by whichever failure is slowest, which is the one least likely to heal.

Rule 3 — Classify on structure first, text second. Status code, bytes, chunks, TTFB, finish_reason presence, who closed the socket, elapsed versus timeout. Only then declared codes, message text, exception type. Otherwise your classifier is a pile of regexes written against last year's error messages.

Rule 4 — Unknown plus expensive means fail over. A cheap unknown (sub-second, one attempt) can be retried; an expensive unknown (a full timeout, zero bytes) should hand over. Write the rule where the code is, not where the operator's intuition is.

Rule 5 — Do not fail over on errors you caused. A user cancel, a /reset, a client eviction, an interrupt — all log as ReadError, none deserve a retry. Check the cancellation flag before classifying.

Rule 6 — A completion is finished only when the protocol says so. Require finish_reason (or message_stop) before treating a message as whole, and treat length / content_filter / refusal as completed but not successful.

Rule 7 — Keep the partial. If chunks arrived, you have a prefix. Resume where the provider supports it; where it does not, record what arrived before failing.

Rule 8 — One retry ladder, not two. Nested retries multiply: three inside three is nine, and nobody said "nine" out loud. Pick the layer that owns the budget and make the other fail fast into it.

Rule 9 — An empty successful answer is a failure, and it must be loud. If the product is text and the text is empty, the job failed, whatever the status was. Otherwise you get cron jobs that "succeed" every night and produce nothing — worse than crashes, because nothing pages anyone.

13HTTP/1.1 200 OK

Applying the rules to this stack

13.1 Config changes available today

Lower the stream retry count on the reasoning primary. HERMES_STREAM_RETRIES defaults to 2, meaning three stream attempts; with deepseek-v4-pro's 600-second reasoning stale floor, each can cost ten minutes. Setting it to 1 drops the ceiling from nine attempts to six. The trade-off is real: the 0.82-second single-frame stream (case a) is exactly what a second attempt fixes cheaply, so the right value depends on the failure mix.

Bound the outer loop per workload. agent.api_max_retries: 3 is global. A cron job that must finish before the next hour cannot afford the same ladder as an interactive turn with a human watching a spinner.

Move latency-critical cron work off the reasoning primary. A nightly digest needs to finish, not to think for ten minutes. Running it on the first fallback (anthropic/claude-haiku-4.5 via OpenRouter) removes the 600-second floor from the critical path — a routing change, not a failure-handling change, and the cheapest fix in this document.

Set a stale timeout per provider where the model does not need the floor. providers.<id>.stale_timeout_seconds overrides the floor. For local Ollama endpoints a shorter value turns a hang into a fast, cheap failure. Do not lower it for remote reasoning models; they legitimately think for minutes.

13.2 Changes that belong in the code

These are written as defects because that is what they are.

The empty-stream verdict. classify_api_error has no stage that recognises EmptyStreamError, bytes == 0, or finish_reason_seen == False, so it produces reason=unknown, retryable=True, should_fallback=False. An empty stream is not unknown: the transport knows exactly what happened and the log prints it. A stage reading chunks == 0 and ttfb is None and elapsed >= timeout and returning a failover-worthy verdict would turn a 50-minute hang into a 10-minute one.

The cancellation predicate. The stream layer already checks self._request_cancelled before retrying, which is why case (b) did not spiral. The classifier cannot see that flag, so a cancelled stream still reaches it as a ReadError. Passing "was this cancelled by us" into the classification would stop the outer loop treating a /reset as a provider fault.

The success-side completion assertion. finish_reason_seen is logged on drops. It should also be checked on success: a turn whose response carried no finish_reason and no text is not a completed turn. This is rule 9, and it is the difference between "the model said nothing" being a mystery and being an alert.

13.3 What the failure-mode table looks like when the rules are applied

Two of those six rows are a retry. Two are a failover. Two are neither. The default behaviour of most stacks — retry everything that is not a 4xx — treats all six as row one.

14HTTP/1.1 200 OK

Key takeaways

1. Retry and failover are different layers with different jobs. Retry re-buys the same route; failover changes route. Whichever runs first spends the clock the other needed. [§1]
2. The retry budget is attempts × timeout, not attempts. A 600-second timeout makes three attempts a 30-minute operation, and the backoff between them is noise. [§4]
3. HTTP 200 means the headers arrived. It is a layer-2 statement. Completion is declared by finish_reason / message_stop, and without it you have a fragment, however many bytes it holds. [§7][§8]
4. "Unknown" is the expensive class, and the default for it is "retry the same thing". That default is only correct for cheap failures. Expensive unknowns should hand over. [§6]
5. An empty successful answer is a failure. If the product is text and the text is empty, the job failed — and if nothing says so, the failure is silent, which is the worst kind. [§12]

15HTTP/1.1 200 OK

Go deeper

Every link was fetched on 2026-10-18. Where the document was read it says READ; where only its resolution was confirmed it says RESOLVES, because a link you have not read is not a citation.

1. 01 RFC 9110, HTTP Semantics https://www.rfc-editor.org/rfc/rfc9110.html §15.3.1, §6.1, §9.2.2. READ · fetched 2026-10-18
2. 02 RFC 9112, HTTP/1.1 https://www.rfc-editor.org/rfc/rfc9112.html §6.3 body length, §7.1 chunked coding. READ · fetched 2026-10-18
3. 03 WHATWG HTML Standard, §9.2 Server-sent events https://html.spec.whatwg.org/multipage/server-sent-events.html READ · fetched 2026-10-18
4. 04 MDN, Using server-sent events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events further reading. RESOLVES · fetched 2026-10-18
5. 05 MDN, PerformanceResourceTiming.responseStart https://developer.mozilla.org/en-US/docs/Web/API/PerformanceResourceTiming/responseStart ; the web definition of first byte. READ · fetched 2026-10-18

1. 06 OpenAI API reference, Chat streaming https://platform.openai.com/docs/api-reference/chat-streaming finish_reason is "optional … or null". READ · fetched 2026-10-18
2. 07 OpenAI, Streaming API responses https://platform.openai.com/docs/guides/streaming-responses READ · fetched 2026-10-18
3. 08 Anthropic, Streaming messages https://docs.anthropic.com/en/docs/build-with-claude/streaming message_stop, ping, in-stream overloaded_error, capture-and-resume. READ · fetched 2026-10-18
4. 09 Anthropic Messages API reference https://docs.anthropic.com/en/api/messages The stop_reason enum. READ · fetched 2026-10-18

1. 10 AWS Architecture Blog, Exponential Backoff And Jitter https://www.awsarchitectureblog.com/2015/03/backoff.html in full. Full/equal/decorrelated jitter, 100-client simulation. READ · fetched 2026-10-18
2. 11 AWS SDKs and Tools, Retry behavior https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html Error classification and the retry-quota token bucket. READ · fetched 2026-10-18
3. 12 AWS Builders' Library, Timeouts, retries, and backoff with jitter https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ JavaScript-rendered; I confirmed the URL and title only, so treat it as further reading. RESOLVES · fetched 2026-10-18
4. 13 Wikipedia, Exponential backoff https://en.wikipedia.org/wiki/Exponential_backoff READ · fetched 2026-10-18

1. 14 Google SRE Book, ch. 21 Handling Overload https://sre.google/sre-book/handling-overload/ READ · fetched 2026-10-18
2. 15 Google SRE Book, ch. 22 Addressing Cascading Failures https://sre.google/sre-book/addressing-cascading-failures/ READ · fetched 2026-10-18
3. 16 Google SRE Workbook, ch. 11 Managing Load https://sre.google/workbook/managing-load/ The Pokémon GO retry spike, 20× the previous RPS peak. READ · fetched 2026-10-18

1. 17 Azure Architecture Center, Circuit Breaker pattern https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker READ · fetched 2026-10-18
2. 18 Azure Architecture Center, Retry pattern https://learn.microsoft.com/en-us/azure/architecture/patterns/retry Idempotency, fail-fast, nested retries. READ · fetched 2026-10-18
3. 19 Wikipedia, Idempotence https://en.wikipedia.org/wiki/Idempotence READ · fetched 2026-10-18
4. 20 Wikipedia, Circuit breaker design pattern https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern READ · fetched 2026-10-18

16HTTP/1.1 200 OK

Appendix — evidence ledger

Code (read 2026-10-18, /usr/local/lib/hermes-agent)

• agent/chat_completion_helpers.py:3739,3747 — HERMES_STREAM_RETRIES default 2; the three-attempt loop.
• agent/chat_completion_helpers.py:3568 — _handle_stream_error: empty-stream retry branch, exhaustion diagnostics, propagation.
• agent/error_classifier.py:974,479,81-84 — classify_api_error, _V_UNKNOWN, ClassifiedError defaults.
• agent/conversation_loop.py:1497 and agent/agent_init.py:1437 — outer loop and api_max_retries default 3.
• agent/turn_api_error.py:363-380 — exhaustion → buffered fallback message → _try_activate_fallback.
• agent/reasoning_timeouts.py — the 600-second reasoning stale floor group containing deepseek-v4-pro.
• agent/stream_diag.py — http_status, bytes, chunks, elapsed, ttfb, finish_reason_seen.

Config (/root/.hermes/config.yaml, read 2026-10-18)

• primary deepseek-v4-pro / provider deepseek / https://api.deepseek.com.
• fallback_providers: openrouter / anthropic/claude-haiku-4.5, then openrouter / deepseek/deepseek-chat.
• agent.api_max_retries: 3.

Logs

• /root/.hermes/logs/errors.log:12848-12860 — 2026-10-15 02:15-03:05: six attempts, 600.13 / 600.20 / 600.05 / 600.02 / 600.26 s, bytes=0 chunks=0 ttfb=-, API call failed (attempt 1/3) then (attempt 2/3), Retrying API call in 2.3057…s.
• /root/.hermes/logs/agent.log.3:170 — 2026-10-16 15:54:59 EmptyStreamError, bytes=40 chunks=1 elapsed=0.82s ttfb=0.82s.
• /root/.hermes/logs/agent.log.3:3804 and neighbours — 2026-10-16 17:13:20 ReadError (ECONNRESET), bytes=149940 chunks=3386 elapsed=29.17s ttfb=0.59s, with client eviction and /reset at the same second.

Arithmetic I performed (not observed)

• "Nine attempts" = 3 stream × 3 API attempts, derived from the two loop bounds. The observed instance stopped at six because the failure became a 400 context-length error at 03:05:45.
• "30 / 90 minutes worst case" = 3 × 600 s and 9 × 600 s. Derived, not measured end to end.

Not verified this run

• No fault injection was run to reproduce the nine-attempt ceiling.
• The AWS Builders' Library article body (JavaScript-rendered page).
• Claims about how Mnemosyne and Open Design behave are illustrative of the pattern, not measurements of those systems.