ajayr 51bc0f160e
fix(llm-error): retry provider errors reported inside a 200 stream (#6897)
* fix(llm-error): retry provider errors reported inside a 200 stream

An OpenAI-compatible gateway that answers 200 early, to keep a slow
request alive with SSE keepalives, reports a later provider failure (a
504, say) as an SSE error event. The openai SDK raises that as the base
APIError, with no status code and no response, so _classify_error found
no status, no transient class name and no busy text, and called it
generic: the run failed on its first attempt.

An error with no status now takes an effective status from its own
contents, and the first source that yields one decides: a numeric code
on the error, or a numeric code, status or status_code in its body or
the body's error object; an explicit status in its message ([504],
HTTP 502, status 503); or type server_error (500). Its busy check also
reads each run of _, - and whitespace as one space, so rate-limit and
RATE_LIMIT_EXECUTION_TIMEOUT match "rate limit". Retry budget, backoff
and circuit breaker then treat it as the equivalent HTTP error. Errors
that carry a status are classified exactly as before, and the quota,
auth and burst-rate checks still run first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(llm-error): note how status-less errors are classified

Add one sentence to item 9 of the middlewares guide, and an llm entry
under Fixed in both changelogs, for the retry of provider errors
reported inside a 200 stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(llm-error): pin the in-band status source order and leftmost match

Two rules of _extract_in_band_status had no test: a numeric status
field decides before the message is read, and within the message the
leftmost in-range status wins. Either rule could be inverted with every
test still green. Add one stays-generic case for each: code 400 with
"[503]" in the message, and "HTTP 400 after [503]".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(llm-error): do not read an array index as an in-band status

The bracketed alternative of the in-band status regex had no lookbehind,
so `messages[502]` or `tools[0][503]` matched and their number decided
the error's status. The bracketed form now starts with the lookbehind
`(?<![\w\]])`: a `[` directly after a word character or a `]` opens an
index, not a status. The `http` and `status` alternatives are unchanged;
`[504]: ...` in running text still counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(llm-error): let an in-band status preempt the separator busy check

The separator-normalised busy check ran for every status-less error,
even when the in-band lookup had already found a status: "rate-limit
header malformed" with `code: 400` was retried as busy, while HTTP 400
is not. It now runs only when no source yields a status, so an in-band
4xx decides like an HTTP one. The same text with no status anywhere
still counts as busy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(llm-error): let only 4xx and 5xx values decide the in-band status

The in-band lookup accepted 100-599 values from its first two sources,
so a 1xx-3xx status could decide the classification: `HTTP 200` in a
message whose server_error type means 500, or `code: 200` alongside
`status: 503`. Only 400-599 now count: `_as_http_status` skips a
smaller value as it skips a non-numeric one, so the candidate loop
moves on, and an out-of-range message match is skipped so the scan
continues. An error that carries a status stays as it was, and the
server_error source is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(llm-error): describe the narrowed in-band status rules

The guide, the CHANGELOG and the CHANGELOG_zh now describe the three
narrowed rules: only a 4xx or 5xx value decides, an index such as
`messages[502]` is not a status, and the separator-matched busy check
runs only when no source yields a status. Only the sentences that
describe the affected forms changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(llm-error): pin the remaining in-band status guards

Cover the nested error.status_code field, a code above 599 that yields
to a later status field, a code written in non-ASCII digits, and an
in-band 4xx whose plain busy text still decides, as it does for an HTTP
4xx. These pin current behaviour; no code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(llm-error): say which in-band fields are read on the error itself

Only `code` is read on the exception; `status` and `status_code` are
read in its body and in the body's `error` object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): link the in-stream retry entry to #6897

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(llm-error): require a separator and ASCII digits in an in-band status

The message regex read `status503` and `statuscode503` as a status
because the separator after `status` was optional, and it read
Arabic-Indic or fullwidth digits because `\d` and `int()` accept every
Unicode decimal digit. A status-less error with such text was retried
as a 503.

The `status` form now needs spaces or tabs, or a `:` or `=`, before
the number, and all three forms read only ASCII digits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(llm-error): pin the retry of an in-stream error after partial output

An `error` event that follows content chunks is retried like one that
comes first: the whole call re-runs and its result holds only the new
answer, while the tokens the first attempt already streamed to the
callbacks stay delivered, as with a mid-stream `ReadError`. The new
test pins that outcome, and a comment in `_classify_error` notes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Handle transient errors based on status codes

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-10-15 08:39:17 +08:00
..