* fix(sandbox): force UTF-8 console for PowerShell so CJK output is not garbled
LocalSandbox captures PowerShell output through a UTF-8 pipe reader
(errors=replace), but Windows PowerShell 5.1 writes console output in
the legacy OEM codepage (GBK on zh-CN Windows) unless told otherwise,
so every CJK character in tool output arrives as mojibake and the
decode never raises. Prepend a UTF-8 preamble
([Console]::InputEncoding/[Console]::OutputEncoding/$OutputEncoding)
to the -Command payload so both directions of the console are UTF-8
before the user command runs.
* fix(sandbox): pair PowerShell UTF-8 capture and guard console setup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): platform-aware Lark CLI runtime validation for Windows hosts
The managed Lark CLI sandbox runtime validation and its tests assumed POSIX
semantics that Windows hosts cannot satisfy, breaking the focused AIO/Lark CLI
suites (5 failures on current main).
- _validate_lark_cli_sandbox_runtime keeps the strict executable-bit contract
on POSIX; on Windows it validates the Linux-only artifacts by content
instead (ELF/PE/Mach-O image magic for linux-*/lark-cli, shebang for the
bin/lark-cli launcher), since NTFS cannot represent the exec bit.
- The AIO runtime-mounts test now asserts the explicit Windows credential
contract: an owner-only inheritable DACL (via the existing PowerShell ACL
resolvers) instead of exact 0o700 modes, which remain asserted on POSIX.
- Accept-path runtime tests stage ELF-prefixed payloads so both platforms
exercise realistic artifact content; the extractor's mode assertion is
POSIX-only with a writability check on Windows.
Focused suites: 5 failed / 162 passed -> 167 passed, 3 skipped on Windows 11;
POSIX behavior unchanged (POSIX branches keep the original assertions).
* refactor(tests): share Windows ACL resolvers via a helper module and cover the Windows shebang gate
Review follow-ups on #5442:
- Move the PowerShell ACL resolvers (_windows_acl_env/_windows_acl_sids/
_windows_acl_protected/_windows_acl_owner_sid) from
tests/test_lark_cli_integration.py into tests/_windows_acl_helpers.py,
following the existing shared-helper convention, so the aio suite no longer
imports the full lark-cli integration module (which drags in app.gateway
routers and the FastAPI TestClient at collection time).
- Add test_managed_sandbox_runtime_rejects_launcher_without_shebang_on_windows:
a launcher without a shebang plus ELF-magic binaries, with lark_cli.os
monkeypatched via the existing Windows stub so the shebang-missing reject
branch of _runtime_artifact_is_executable is covered on every platform.
- Comment the rejects-non-executable prestaged-binary test to record that on
Windows the rejection comes from the payload's non-magic content, since
chmod() cannot clear the exec bit there.
Focused suites: 168 passed / 3 skipped on Windows 11.
* fix(agents): keep token budget signals for runs without a run_id
#5410 moved every invocation without a non-empty string run_id onto
str(id(runtime)). Two things break on that key:
- SubagentExecutor passes the parent's run_id, None when the parent run has
none (LangGraph Server, direct create_deerflow_agent callers), and reads the
stop reason back with that None. The hard stop stored it under the id string,
so a token-capped subagent reported a clean completion to the lead.
- LangGraph gives each graph node its own Runtime wrapper, so the key changed
between after_model and the next model call: the budget warning was never
delivered, and each after_model counted every AIMessage in the thread.
Key those invocations by Runtime.control, as LoopDetectionMiddleware does,
release it in after_agent, and store the stop reason under the context run_id
as given.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): keep an active invocation's budget key when the anchor map is full
The fallback anchor map was FIFO, so with 1000 run_id-less invocations on a
shared instance an active one could lose its anchor mid-run and restart with a
fresh budget. Move the anchor to the end on every lookup, as loop detection
does, and note why execution_info.run_id is not consulted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(skills): close SkillScan bypasses in the skill review gate
The public skill review gate re-materialized a package snapshot into a
temp directory for SkillScan, but copied only entries the reader had
decoded as text and skipped every file under any evals/fixtures/
directory. Executable binaries and nested archives never reached the
package rules, and a fixture-shaped path hid any script from the scan.
Readers now keep binary bytes as content_base64, the analyzer writes
every non-symlink file byte for byte, and only eval fixture SKILL.md
samples stay exempt. Files are created exclusively, so a duplicate
archive member or a case-folded name fails the scan closed instead of
overwriting an earlier file.
SkillScan itself skipped any file that was not NUL-free UTF-8. One
Latin-1 byte in a comment hid a reverse shell from the review gate, and
a NUL byte skipped static analysis at install. Code files that fail
strict decoding now raise package-undecodable-script (HIGH) and are
analyzed over a lossy decode, so CRITICAL matches keep blocking.
"Code file" and "executable magic" were defined separately in the
installer and SkillScan and had drifted: SkillScan missed 32-bit
little-endian and fat Mach-O variants the installer blocks. Both rules
now live in skills/package_files.py, shared by the installer, the export
guard, and SkillScan.
* docs(changelog): link the skill review gate fix to #5431
* fix(skills): fail closed on bytes-less snapshot entries and skip text rules for executables
The review analyzer skipped any snapshot entry it could not turn into
bytes. Readers only emit such entries for oversized files, and they also
mark the snapshot truncated, but content_base64 is optional in the
contract, so a reader regression or a hand-built snapshot would silently
drop a file from SkillScan. An entry without bytes now fails the scan
closed (not_assessed: skillscan) unless the snapshot is truncated, and a
text entry without content no longer materializes as an empty file.
A real executable under scripts/ is a code file, so SkillScan decoded it
lossily and ran the text rules over its string tables. An OpenSSH binary
produced a CRITICAL secret-private-key finding from the key-format
banner it embeds. An undecodable file with executable magic still
reports package-undecodable-script, and its CRITICAL
package-executable-binary finding already blocks it, so it now skips the
text rules. Decodable files keep full text analysis.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
McpRoutingMiddleware picked the latest user message with
is_real_user_message, which rejects every hide_from_ui message. A Human
Input Card reply is hidden but is still the user's current request, so
when the routing keyword lived only in the clarification answer the
deferred MCP tool was never auto-promoted and the model had to call
tool_search by hand.
Switch _latest_user_message to is_genuine_user_message, the same
predicate summarization_middleware already uses for this reason (#5416).
Fixes#5425
* fix(goal): stand the goal down once the run has hit its token budget
Since #5410 goal continuations share the run's token budget, so a
continuation queued after the budget's hard stop only spends one more
model call before its tool calls are stripped. Pass the run's stop_reason
into the goal loop and stand the goal down with "token_capped" in that
case. The evaluator still runs first, so a goal the capped run satisfied
is cleared as before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(goal): list the token-budget stop among goal-loop preconditions
Also pin that a satisfied goal is cleared, not stood down, when the run
hit its token budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(sandbox): report truncated remote glob and grep results
BoxLite, Tenki, E2B, and OpenSandbox run find/grep in the sandbox, cap
the raw output with `| head`, and then filter those lines in Python:
ignored directories such as node_modules are dropped and grep's glob
scope is applied. They reported truncated only when max_results matches
survived the filter. When the capped lines were mostly filtered out, a
search with real matches past the cap came back short or empty with
truncated=False, and glob_tool/grep_tool rendered it as "No files
matched" / "No matches found". With the default max_results=200 and
1,200 files under node_modules, glob("**/*.py") reported no matches for
a workspace that has src/app.py.
remote_search_command now lets one line past its limit through, and
parse_remote_search_output(..., limit=) returns RemoteSearchOutput(text,
truncated): the first `limit` lines and whether the extra line arrived.
Exactly `limit` lines stays a complete result. Each provider passes the
cap it already computed to both calls and returns that truncated from
glob and grep when fewer than max_results results survive filtering.
The glob and grep tools now describe an empty truncated result as
incomplete instead of reporting no matches, which also covers AIO grep's
forwarded truncated flag. Sandbox.glob/grep document truncated as "the
matches may be incomplete".
* docs(changelog): reference #5427 in the remote search truncation entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Document that read permission expiry and source deletion do not erase
text already copied into the destination conversation, and that reads
follow the source's current visible history. Truncated results now tell
the agent to acknowledge the omission and ask for the missing material
before claiming every requirement is covered.
Pages were filled to 20,000 text characters by cutting the last message
that did not fit, and that suffix could never be paged back. They could
also exceed the default 12,000-character tool-output budget, which
externalized the page to a file. Pages are now sized by their serialized
length against the read_conversation tool-output budget; a message that
does not fit starts the next page intact, so only a message over 4,000
characters (or one whose escaped JSON alone exceeds the budget) is cut.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(scheduled-tasks): filter run history by occurrence status
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(scheduled-tasks): share occurrence status contract
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): mask every host path in a colon-joined list
Host-to-virtual output masking matched a host root and then consumed the
path tail up to whitespace or shell punctuation, but not `:`. A
`:`-joined list such as $PATH or $PYTHONPATH was therefore swallowed into
the first match's tail, and scanning resumed after it, so every later
entry under the same root reached the model as a raw host path. The regex
matcher (process-stable skill roots) and the direct scanner (per-thread
roots, LocalSandbox) shared the gap. Each redundant masking pass --
separator variants, the realpath spelling, the /mnt/user-data root
mapping, LocalSandbox's own reverse resolution -- happened to recover one
entry, which hid the leak for short lists: bash output leaked from the
fourth entry, single-pass consumers from the third.
The shared tail in path_patterns.py now ends at `:` in both matchers.
`;`, the Windows list separator, already ended it. A `:` inside one path
(grep -n output, a file name) only shortens the match; the remaining text
is copied through verbatim.
Shortening the match exposed a second leak. LocalSandbox reverse
resolution realpaths the matched path and returned that realpath when no
mount contained it, so a symlink inside a mount whose target lies outside
every mount was shown as the target's host path. grep -n lines used to
hide this only because the whole line resolved as one nonexistent file;
whitespace-terminated output and LocalSandbox.glob results already leaked
it on main. Reverse resolution now falls back to the link's own spelling,
normalized so `mount/../x` does not pass, before giving up. A symlink into
another mount still reports that mount's path.
* docs(changelog): reference #5418 in the colon-joined path masking entry
* docs(changelog): split the #5418 and #5419 entries fused by the merge
Resolving the CHANGELOG conflict when main was merged in dropped the
opener of the #5419 entry, so the BoxLite grep fix continued inside this
PR's bullet in both CHANGELOG.md and CHANGELOG_zh.md. Restore it as its
own bullet; the #5419 entry is byte-identical to main again.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): scope BoxLite grep globs to the search root
BoxliteBox.grep omits grep --include for busybox portability and applies
the glob in Python, but it kept only the glob's last path segment and
matched it against each file's basename. A scoped pattern therefore lost
its directory part: grep(glob="src/*.js") returned every .js file in the
tree, including vendor/ and nested src/ subdirectories that glob() with
the same pattern excludes.
The glob now goes through path_matches against the path relative to the
search root, with the file's basename when the root is a single file --
the same scope glob() uses and the one Tenki, E2B, OpenSandbox, AIO and
LocalSandbox already enforce. Like those providers, an empty glob is now
passed to path_matches instead of being treated as no filter.
* docs(changelog): reference #5419 in the BoxLite grep glob scope entry
Summarization rescues the latest user message by id so the current
request survives compaction (#4882). It picked that message with
is_real_user_message, which rejects every hide_from_ui HumanMessage,
including Human Input Card replies. When a run starts from a card
reply, the older visible request was kept and the user's answer was
summarized away.
Use is_genuine_user_message, which accepts hidden messages that carry a
valid human_input_response and still skips other hidden injections.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): keep the token budget across goal continuations of a run
TokenBudgetMiddleware cleared its usage in after_agent, and before_agent
marks every existing message as seen. A Gateway run re-enters the graph
for hidden goal continuations under the same run_id, so each
continuation started from zero and could spend another full budget,
even after the user turn had already hit the hard stop.
Keep the run's usage and warning state across graph entries, as
LoopDetectionMiddleware does since #5344. Only the per-message seen map
is dropped, and before_agent rebuilds it. Invocations without a run_id
in the context still clear everything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(agents): normalize token budget run identity and test cleanup
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): reject forged framework-injection markers in run input
`is_genuine_user_message` treats `hide_from_ui` and a human `name="summary"`
as proof the framework authored a message, and `InputSanitizationMiddleware`
skips those — escaping a real reminder's blocks would corrupt trusted context.
Neither marker was server-owned, so an external caller could set either one and
place a raw `<system-reminder>` outside the user-input boundary markers, which
the lead-agent system prompt declares trusted internal framework data. The
`hide_from_ui` variant is also filtered out of the thread UI, so the forgery
was invisible where it landed.
Both markers are now stripped from untrusted input, on the run path and on the
thread-state mutation route that writes straight into a checkpoint. Framework
injection happens inside the graph and never crosses this boundary, so nothing
the framework does is affected, and `trusted_internal` callers (IM channels,
the MCP task-notification launcher) keep writing hidden messages.
HumanInputCard replies are the one legitimate external `hide_from_ui`: the
frontend sends it alongside a `human_input_response` payload, so a message
carrying a valid one keeps the marker. That buys no bypass — the predicate
already classifies those as genuine, so they stay sanitized.
The name check reuses the predicate's own `_SUMMARY_MESSAGE_NAME` rather than a
fourth copy of the literal, and matches by `isinstance` exactly as the predicate
does: `HumanMessageChunk` is a `HumanMessage` whose `type` is not `"human"`, so
a type-based check would leave that subclass's marker settable. `name` is only
reserved on human messages — on a ToolMessage it is the tool's own name.
Three tests in test_gateway_services.py and test_message_provenance.py asserted
that a caller-supplied `hide_from_ui` survives. That assumption was the bypass;
they now pin the opposite, with a genuinely caller-owned key kept alongside to
prove the stripper is surgical.
* fix(agents): sanitize every genuine user message, not only the newest
The input guardrail scanned backwards for the first genuine user message and
returned, so only the newest turn was ever sanitized. The transformation is
request-scoped (`wrap_model_call`, never written to state), so thread state
keeps the raw text: once a newer turn arrived, the previous turn's payload was
replayed to the model verbatim, outside the boundary markers the lead-agent
prompt declares trusted framework data. The guardrail therefore held for
exactly one model call.
Reaching it needed no forged metadata and no crafted request body — type the
payload in one turn, then send anything at all in the next. A single request
carrying two user messages did it in one shot, since every message but the last
was skipped.
`_process_request` now walks the whole list and `_sanitize_message` owns the
per-message work; every existing branch (the `original_user_content` split for
upload turns, the multimodal rfind fallback, the metadata repair) is unchanged.
Framework-injected messages stay excluded by `is_genuine_user_message`.
Unexpected errors now fail open per message rather than per request. Iterating
history widened the old blast radius: one unprocessable row would have dropped
sanitization for the whole request, handing an attacker the newest turn by
crafting an older one. `GraphBubbleUp` still propagates.
Side effect worth noting: each turn's rendering is now stable across model
calls. Previously a turn was wrapped on its own call and unwrapped on the next,
changing the prompt prefix behind the newest turn and defeating prompt caching.
test_only_processes_last_user_message pinned the old scope; it now pins that
every turn is processed, and keeps driving the `wrap_model_call` entry point.
* docs: record the message-metadata trust boundary and sanitization scope
`agents/middlewares/AGENTS.md` owns the depth for InputSanitizationMiddleware
and documented only the `original_user_content` half of its trust boundary. Left
alone it would teach an agent that `hide_from_ui` is caller-owned and that the
guardrail covers one turn — and the usual failure mode is an agent "restoring"
the behaviour it believes was lost. The entry now carries both markers, the
HumanInputCard exception, the whole-history scope, and the per-message fail-open
rule.
The note lives only there. The root and `backend/AGENTS.md` layers are
orientation that points at the module guides owning the depth, and
`backend/AGENTS.md` is inherited by every backend chain — prose added there
inflates more than twenty of them, and `scripts/check_agent_guidance.py` shows
the middlewares chain has about a kilobyte of room against its hard limit.
CHANGELOG.md and CHANGELOG_zh.md record it under Security, continuing the
existing prompt-injection lineage.
* fix(gateway): mark caller-hidden messages instead of stripping the marker
Review follow-up. Stripping a caller-supplied `hide_from_ui` closed the bypass
but broke three frontend senders that use the marker purely to keep a context
message out of the transcript: the quoted conversation context
(`buildHiddenConversationQuoteMessage`), the sidecar context prompt
(`buildHiddenSidecarContextMessage`), and the agent save command. None carries a
`human_input_response`, so the HumanInputCard carve-out did not cover them, and
nothing else hides them — `_is_branch_visible_message` and the frontend's
`isHiddenFromUIMessage` both key solely on `hide_from_ui`, and no backend reads
`conversation_quote_context` or `sidecar_context`. All three would have rendered
as user-visible chat bubbles.
The marker plays two roles and only one of them is a vulnerability. The security
requirement is that a caller-supplied marker cannot skip sanitization, not that
it cannot hide a message. So the roles are separated instead of the marker being
removed: the Gateway keeps it and stamps the server-owned `UNTRUSTED_INPUT_KEY`,
and the guardrail now asks `requires_input_sanitization` — the mark, else the
genuine-user test. Hidden stays hidden; untrusted content is sanitized either
way. The reserved `summary` name is handled the same way and no longer rewritten.
Marking rather than removing is also the safer shape in general: `hide_from_ui`
is read for presentation, journal persistence, memory filtering and IM outbound
as well, and this boundary should not silently change any of them.
`is_genuine_user_message` is deliberately left alone. `ToolReceiptMiddleware`
uses it for turn-boundary detection, where a caller's hidden context message must
keep counting as not user-authored; widening it there would move the ledger's
turn window. `requires_input_sanitization` sits beside it in `message_utils` so
the two questions can be compared.
The three tests that asserted a caller-supplied `hide_from_ui` is removed now
assert it survives and carries the mark — which restores the original intent of
the two provenance cases, whose comment already read "caller-owned keys must
survive".
* docs(gateway): rewrite normalize_input's docstring around the mark
Review follow-up. The paragraph still described the pre-4a3344f6 strip model and
contradicted both the implementation and the middlewares AGENTS.md paragraph
updated in that same commit: it called `hide_from_ui` server-owned, said
carrying it skips sanitization entirely, and repeated the premise this branch
disproved — that HumanInputCard replies are the only legitimate external use.
It now describes what the code does: the markers stay caller-owned and are
preserved because three frontend senders rely on `hide_from_ui` for hiding
alone, the message is stamped with `untrusted_input` instead, and
`requires_input_sanitization` sanitizes it anyway. `untrusted_input` joins the
server-owned inventory in the preceding paragraph, which is what makes the stamp
unforgeable and unclearable.
The three surrounding docstrings now also say that these functions mark as well
as strip; `_strip_external_message_metadata` had advertised only the removal,
leaving a reader no way to find the stamp from the name.
* fix(gateway): mark state writes whose message omits additional_kwargs
Review follow-up. The state-route half of the fix missed the most natural
request shape. `_strip_external_metadata_from_message_like` returned early when
`additional_kwargs` was absent or not a dict — there was nothing to strip — and
that early return also skipped the mark. A `POST /threads/{id}/state` body of
`{"values": {"messages": [{"role": "user", "name": "summary", "content":
"<system-reminder>…</system-reminder>"}]}}` therefore reached the checkpoint
unmarked. The messages reducer's `convert_to_messages` then supplies
`additional_kwargs={}`, so at model-call time `requires_input_sanitization` fell
back to `is_genuine_user_message`, which a `summary` name fails, and the forged
tag reached the model raw and outside the boundary markers.
A missing or non-dict `additional_kwargs` is now treated as empty for both the
strip and the mark. The identity return is kept for the case where nothing
changes, so an ordinary key-omitted message does not gain an empty dict just by
passing through. The run path was never affected: `normalize_input` coerces to
BaseMessage first, which always carries the dict.
Every existing state-write test supplied an `additional_kwargs` dict, which is
why this shape slipped through; the regression now covers it at the route and
end to end through the reducer into the guardrail.
While checking the neighbouring shapes, `_skips_input_guardrail` keyed off key
presence where `is_genuine_user_message` keys off truthiness, so
`hide_from_ui: False` — already covered without a mark — would have been
stamped. It now mirrors the predicate exactly, as its docstring claimed.
* docs(middlewares): compress the sanitization note to fit the guidance chain
main's growth left the middlewares AGENTS.md chain 84 bytes under its hard
limit, and the fuller wording did not fit. The load-bearing facts stay — the
markers are marked rather than stripped, and the scan covers every turn — since
those are the two an agent editing this middleware could otherwise get wrong.
The full model lives in the normalize_input, _mark_untrusted_framework_markers
and requires_input_sanitization docstrings.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(client): emit streamed tool calls once with complete args
When a model streams a tool call as chunks (name and id first, then
argument fragments), DeerFlowClient.stream() emitted a tool_calls event
per chunk, each parsed from that fragment alone. Consumers got
`args={}` and then a call with no name or id, and never the complete
call, because the values snapshot skipped the already-streamed id.
Hold tool calls from AIMessageChunk.tool_call_chunks and emit them once
from the values snapshot, where the message is complete.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(client): pin additional_kwargs on the deferred tool_calls event
Also note why a tool-call chunk without a message id keeps the per-chunk
event.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(models): reuse the Claude Code OAuth token read from a file descriptor
ClaudeChatModel accepts a Claude Code OAuth token through
CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR, but that handoff can be drained
only once: a pipe returns EOF after the first read and a file descriptor
keeps its advanced offset. _read_secret_from_file_descriptor read it again
on every call and kept nothing. Every ClaudeChatModel instance loads
credentials in model_post_init, and create_chat_model builds fresh
instances per run, so with a descriptor-only handoff the first model
authenticated and every model after it -- including the title model of
the very first run -- had no credential. The Anthropic SDK then raised
"Could not resolve authentication method" before sending a request.
A secret read from a descriptor is now kept for the life of the process,
keyed by (env_var, fd), so a different descriptor is still read fresh.
The read happens under a lock so two threads building their first model
concurrently cannot race one of them to EOF. Empty reads and OSError are
not cached and behave as before; lookup order, config keys, and log
messages are unchanged.
* docs(changelog): reference #5411 in the Claude Code OAuth descriptor fix entry
* test(models): pin that a closed descriptor handoff keeps its token
Review follow-up on #5411: the descriptor secret cache is keyed on the
fd number, which the OS recycles. Folding os.fstat identity into the key
would break the property the cache exists for -- once the handoff fd is
closed after the first read, fstat raises EBADF and every later model
would lose the token again -- and it would still miss a regular file
rewritten in place, which keeps its st_dev/st_ino.
The handoff is fixed at process start, so keep the number as the key and
state the invariant instead: a closed handoff keeps serving its token, a
secret placed on a recycled number is not re-read, and anything handing
over a new secret in-process must clear the cache. A new test pins the
closed-handoff behavior; an fstat-fingerprinted key fails exactly that
test.
* fix(sandbox): reverse-resolve forward-slash spellings of Windows host paths
Forward resolution deliberately spells resolved paths with forward
slashes in commands and file content (#3869: backslashes break bash
escapes), but the reverse scanner anchored its matches on the native
backslash base, so on Windows every forward-resolved path that came back
in command output or agent-written files leaked the raw host path
instead of mapping to its container path. Match separator-agnostically
in LocalSandbox like sandbox.tools already does, align the two
regex-cache tests with the documented spellings, and refresh the
path_patterns rationale comments that described the old asymmetry.
* test(sandbox): pin the reverse mask to separator-agnostic matching
The flag is the entire Windows fix but is invisible on POSIX CI, so
assert the routing kwargs in the direct-helper wiring test — the same
pin test_tools_mask_patterns_route_through_the_helper already applies to
the sandbox.tools copy. A revert to separator-exact matching now fails
on every platform instead of silently reintroducing the host-path
leak on Windows.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(settings): persist account preferences across browsers
* docs(settings): scope preference guidance to user persistence
* fix(settings): preserve SSR and fence custom-agent defaults
* test: include user persistence in scoped guidance inventory
* fix(settings): sync explicit edits and preserve local tab updates
* feat(memory): add deterministic near-duplicate fact gate
Add an opt-in write-side gate for DeerMem (issue #5252): a proposed NEW
fact whose bounded token-Jaccard similarity to an existing fact in the
same user/agent scope AND category reaches
fact_dedup_similarity_threshold merges into that fact instead of being
appended — the existing id/content/createdAt are kept, confidence is
raised to the maximum, and the source is refreshed, with one
facts_merged_dedup metric increment recording the merge.
Defaults preserve the legacy behavior exactly; targeted updates by fact
id and the exact-content key check are untouched. Companion write-side
step to the read-side relevance/diversity work in #5251.
Refs #5252
Signed-off-by: pwd11 <fvdsrc@163.com>
* fix(memory): preserve corrections during fact deduplication
Signed-off-by: pwd11 <fvdsrc@163.com>
---------
Signed-off-by: pwd11 <fvdsrc@163.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(worker): don't fail the parent run on a subagent error carried by task_running
The worker scans root frames for the LLM error-fallback marker to decide the
parent run's status, and custom frames are root frames. task_running events
carry each delegated subagent message, including a subagent's marked error
fallback, so a subagent that failed after retries marked the whole parent run
as error (and an edit-and-rerun rolled back) even though the executor already
reported task_failed and the lead answered. Skip custom frames in both the
single-mode and multi-mode stream paths.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(worker): cover the single-mode stream loop for task_running fallbacks
Run the task_running regression test with stream_modes=["custom"] too,
which takes the worker's single-mode loop. Without the single-mode
guard only that case fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(models): stop reasoning_effort from reaching the constructor twice
The regular lead-agent build forwards reasoning_effort to
create_chat_model even when neither the request nor the custom agent
chose one. The factory spread that kwarg next to the profile settings,
so any profile that also yielded reasoning_effort -- a top-level value,
when_thinking_enabled, when_thinking_disabled, or the minimal effort the
extra_body.thinking disable path injects -- made the constructor raise
"got multiple values for keyword argument 'reasoning_effort'", and the
lead agent could not be built for that model. #2017 moved the
factory-injected value out of kwargs but left the caller-supplied one.
The requested effort now leaves kwargs once and layers like
model_overrides: a non-None value replaces the profile value, None keeps
it, and the thinking transforms applied afterwards still decide the
final value. Codex keeps resolving the requested value itself, so a
level it does not accept and a profile without effort support still
fall back to medium.
* docs(changelog): reference #5403 in the reasoning_effort collision fix entry
* fix(runtime): resolve omitted run owners before idempotent reuse
HTTP run admissions do not pass a user_id. The SQL run store stamps the
request user from the contextvar onto the row, but RunManager kept None
on its process-local RunRecord, so the two disagreed about who owns the
run. A keyed retry that reached a peer worker, or the owning worker after
cleanup() released its local record, hydrated the stamped row, compared
its owner with None, and raised "Run idempotency key resolved to a
different thread or user", which start_run surfaced as a 500.
MemoryRunStore stored None on both sides and never hit the check.
The same mismatch hid HTTP runs from owner-scoped history reads, which
filter local records by the current user, and skipped the worker's MCP
background_tasks projection, which only runs for records with an owner.
create() and _admit_thread_operation() now resolve an omitted owner from
the current user before building the record, and keep None when no user
is in context instead of falling back to the default bucket, so the
local record and every store agree on the owner. HTTP runs now receive
the background_tasks projection, so the replay golden's values frames
gain that key.
* docs(changelog): reference #5401 in the keyed retry owner fix entry
* test(runtime): close the SQL engine when peer-reuse test setup fails
Review follow-up on #5401: the sql case of the two-worker start_run test
initialized the engine above the try whose finally calls close_engine().
init_engine() assigns the module-global engine and session factory before
bootstrapping the schema, so a failure there skipped the teardown and left
a stale engine for later tests in the same process.
Store setup and the RunManager workers now live inside the try, so the
teardown runs whether setup or the test body fails.
* fix(runtime): stop idempotent reuse from blocking the thread on a peer worker
When an idempotent run admission lands on a worker that does not own the
run, RunManager hydrates the stored row and returns it as the reused
record. It also registered that row in the worker's local run map, but
only the owning worker's task lifecycle finalizes and cleans up local
records, so the copy kept its admission-time pending/running status for
the life of the process. On that worker every later reject-strategy
admission for the thread returned 409 until a restart, run reads kept
reporting the stale status, orphan reconciliation skipped the run as
locally live if the owner crashed, and a cancel took the local-owner
path and marked the owner's still-running row interrupted.
Return the hydrated row as a detached store-only handle instead of
registering it. A local record for the key is already returned before
the store insert, so the removed lookup of an existing local record was
unreachable. get(), cancel() and reconciliation now read the durable row
on the peer, matching the documented non-owner contract.
* test(runtime): pin keyed retries of a terminal reused run on the SQL store
Review follow-up on #5393: the post-cleanup release relied on a keyed
retry resolving through the terminal row's idempotency conflict, but
only MemoryRunStore pinned that path, and no test retried on the owner
after its local record was cleaned up.
The SQL repository test now retries the key on the peer and on the
owner once the run is terminal and cleaned up, asserting both get the
same run back as a store-only reused handle before the keyless
follow-up is admitted.
* feat(agents): elide superseded write_file payloads from model-bound requests
Step 2 of #5328. After a successful write_file the file on disk is the source
of truth, and the read-before-write gate forces a read_file before the next
modification of that path, so once a later successful read or write of the
same path exists the historical `content` argument is redundant with it. Long
report-writing runs (append-in-chunks) therefore carried every section twice,
once as the write argument and once as the following read output, until
summarization compacted the whole turn.
- ToolOutputBudgetMiddleware's model-call hooks now replace such superseded
content with a short deterministic placeholder pointing at read_file, in the
model-bound request only: state["messages"], checkpoints, receipts, loop
detection, and the run journal keep the original arguments, and nothing is
externalized to disk. The newest `keep_recent_writes` successful writes
(default 1) always stay visible; str_replace payloads are never touched; a
same-turn read never supersedes (parallel calls run in no fixed order); only
results stamped deerflow_tool_meta.status == "success" count, so failed,
gate-blocked, partial, or unstamped writes are never candidates.
- New `tool_output.elide_superseded_writes` (default on),
`tool_output.superseded_write_min_chars` (default 2000), and
`tool_output.keep_recent_writes` (default 1); config_version 41 -> 42 in
config.example.yaml and the Helm chart.
- The per-occurrence call/result pairing the gate introduced in #5329 moves
into the shared `tool_call_args.pair_tool_call_results` helper so both
policies pair the same way; the gate now uses it.
* fix(agents): scope tool-call result pairing to the issuing turn
Review finding on #5374 (P2): pair_tool_call_results consumed results from a
history-wide per-id queue, so an interrupted write_file with no result whose
tool-call id a later turn reused inherited that later call's success. With
the default elision the unconfirmed draft was then replaced by a placeholder
claiming the write succeeded, and the gate's blocked-call pairing had the
mirror-image hole.
Pair results the way DanglingToolCallMiddleware does: walk in document order,
open each AIMessage's calls, and let a ToolMessage answer only a still-open
call of the most recent preceding AIMessage. A result never answers a call
from an earlier turn, so the interrupted call stays unanswered (never a
candidate, never labeled blocked) and stray or duplicate results are ignored.
Regressions cover the helper, the superseded-write policy, and the gate.
* fix(agents): never rewrite tool-call ids duplicated within one AIMessage
Review finding on #5374 (P2): the policies select calls per occurrence, but
every provider surface is addressed by tool-call id, so when a malformed
provider payload repeats an id inside one assistant turn the rewriter could
only replace all of its occurrences at once. A failed write_file sibling then
took on the superseded successful call's path and elided content and was
presented as a success; the gate's blocked-call elision had the mirror-image
hole (a successful sibling rewritten into the blocked call).
rewrite_messages_tool_call_args now never offers an id that repeats within
its message to the selector and leaves those calls untouched on every
surface. Both policies are covered by the shared helper; regressions cover
the helper, the superseded-write policy, and the gate.
* fix(agents): skip unhashable tool-call ids in the duplicate-id guard
Review finding on #5374 (round 3): _duplicated_call_ids fed every id into a
Counter before the string guard, so a list or dict id from a malformed
provider payload raised TypeError out of wrap_model_call and failed the whole
model call whenever the history also held a rewrite candidate. The pre-PR
loop and pair_tool_call_results skip such ids; only this helper regressed.
Count non-empty string ids only, and pin it with regressions for the helper,
the superseded-write policy, and the gate.
* docs(agents): keep middleware guidance within size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
_get_firecrawl_client only read api_key from the tool config and ignored
base_url, so self-hosted Firecrawl deployments were unreachable: the SDK
defaulted to https://api.firecrawl.dev and raised 'No API key provided'.
Now base_url is forwarded as api_url; api_url is omitted when unset so
cloud behavior is unchanged.
Also document the optional self-hosted base_url on both Firecrawl entries
in config.example.yaml and reconcile their headers to the fastCRW house
style (Cloud requires FIRECRAWL_API_KEY; self-host may need no key.).
* feat: add opt-in task notes and compacted history recall
* fix: validate task continuity state and preserve user answers
Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence.
* fix: align Helm config version with task continuity schema
* fix: preserve mixed task history and declare continuity policies
* fix: recover malformed history and evict archives atomically
* feat(extensions): add in-place upgrade that keeps private config
Replace a managed local snapshot or re-pin an already-installed
requirement without going through remove, which dropped plugins[].config.
* fix(extensions): keep snapshot, enabled, and git re-pin on upgrade
Rollback keys off staging_root so a failed snapshot rename cannot
rmtree the live tree. Upgrade preserves plugins[].enabled. Re-pin
identification uses tool.uv.sources so git upgrades adopt the
existing plugin record instead of failing closed after uv already
switched the revision.
* test(extensions): cover requirement re-pin identification on upgrade
Re-pinning deerflow-extension-demo==2.0.0 to ==3.0.0 leaves added_names
empty, so identification must take the added_specs fallback. Assert
private config/required/enabled survive and the lock records 3.0.0.
* fix(extensions): reject upgrade of an uninstalled git source
Bare git+ URLs are not named Requirements, so the pre-uv-add installed
check was skipped and upgrade acted as install. Resolve them against
[tool.uv.sources] in the extensions group before uv add.
* fix(sandbox): stop remote grep/glob from reporting failures as no matches
E2B, OpenSandbox, BoxLite and Tenki ran grep/find behind `2>/dev/null | head`, so a missing search root, a missing grep/find binary or an unreadable tree exited 0 with empty stdout and the tools reported "No matches found". Wrap the search in sandbox/remote_search.py, which checks the root first and records the search's own status after head, as remote_list_dir does for list_dir: a missing root raises FileNotFoundError, a failed search raises OSError, and a genuine no-match still returns []. glob's find gains -H for symlinked roots, OpenSandbox's BusyBox fallback keeps the primary grep status, and E2B no longer swallows client errors. Regression tests run each provider's real command in a local POSIX sh.
Fixes#5376
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(sandbox): fail remote grep/glob on partial traversal errors
grep 2 / find 1 after some results were printed (an unreadable file or
subdirectory) were returned as a complete search. Callers have no
partial-result channel, and #5376 asks for permission and command
failures to raise, so these statuses now raise OSError like any other
failure. Only grep 0/1/141 and find 0/141 pass.
The error for grep 2 / find 1 says that some files or directories could
not be read and asks for a narrower path, so the agent can retry instead
of giving up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(uploads): bound document outline and preview text
Bound each document outline title to 200 characters and each fallback preview to 2000 characters across its lines, including omission markers.
Refs #5322
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* test(uploads): cover exact preview budget and restore title guidance
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
Toggling a skill wrote resolved secrets into extensions_config.json. The
Gateway skill toggle and DeerFlowClient.update_skill loaded the file with
ExtensionsConfig.from_file(), which replaces every "$VAR" string with the
environment value (and an unset variable with ""), then serialized that
model back through to_file_dict(). A "$GITHUB_TOKEN" reference was
persisted as the plaintext token, and an unset reference was erased for
good. DeerFlowClient.update_mcp_config had the same flaw for every key
other than mcpServers.
Every writer now does a raw read-modify-write, the way the MCP router
already did: read_raw_extensions_config reads the on-disk JSON,
set_raw_skill_enabled changes only the target entry, and
validate_raw_extensions_config checks the candidate the way the runtime
will load it before the atomic write. When the file does not exist yet,
the Gateway seeds it with the cached skill states only, never the
resolved cached model. The MCP router's raw loader and candidate
validation delegate to the same helpers, so the rule lives in one place,
and to_file_dict() is removed so the unsafe serialization has no entry
point left.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): allow constructor kwargs on config-declared middlewares
extensions.middlewares entries may be a class-path string or {class, kwargs}.
String entries keep the zero-argument constructor. Unknown fields and blank
class paths fail at config validation. Constructor errors still fail at
agent creation.
Fixes#5311
* fix(extensions): coerce middleware kwargs to JSON types
YAML timestamps became datetime objects while JSON kept strings, so
constructors and to_file_dict() json.dump saw different types. Validate
kwargs as JSON types at config load, stringify dates, reject NaN and
other non-JSON values, and cover the raw-dict loader branch.
* style(extensions): wrap middleware Field description for ruff E501
make lint failed: the middlewares description was 289 chars (limit 240).
Wrap it and run ruff format on the two files this PR last touched.
* docs: compact configured middleware guidance to satisfy size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(title): use filenames for attachment-only conversations
* docs: trim upload guidance to fit inherited size budget
* fix(title): bound attachment-only fallback titles
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tools): run tool assembly off-loop at async entry points
get_available_tools() may block on MCP cache initialization while it is
called on async agent-assembly paths (task_tool, durable batch execution),
stalling the calling event loop for the full discovery duration.
Dispatch the (unchanged, synchronous) assembly call to a worker thread via
asyncio.to_thread at the two async entry points so the loop keeps processing
requests, SSE frames, cancellations, and timers.
Fixes#5172
* fix(tools): offload lead-agent assembly off-loop and pin with blocking-io anchors
Review follow-up for #5224:
- run_agent now dispatches agent_factory(...) through asyncio.to_thread, so
lead-agent assembly (including both get_available_tools call sites in
_assemble_lead_agent) runs off the event loop — the Gateway headline
scenario from issue #5172.
- _ensure_sync_invocable_tool takes a double-checked threading.Lock, making
the in-place tool.func wrap on the shared tool singletons explicitly
single-shot now that assembly can run concurrently on worker threads.
- Add backend/tests/blocking_io/test_tool_assembly_offloop.py: blocking-probe
anchors for task_tool and SubagentBatchService._execute_item under the
strict Blockbuster gate, plus a meta-check proving the gate trips on the
exact syscall class (ExtensionsConfig.from_file on the loop). Verified the
anchor goes red when the offload is flattened back to a plain call.
* fix(gateway): build checkpoint state accessor off-loop; anchor run_agent offload
Review follow-up for #5224:
- Add abuild_checkpoint_state_accessor (asyncio.to_thread around the
unchanged sync builder) and switch every async call site to it: the
stateless_wait route, thread_runs, both threads call sites, and the
build_thread_checkpoint_state_accessor boundary. The agent-factory
assembly re-enters get_available_tools() and may block on MCP cache
initialization; repeat calls hit _state_accessor_graph_cache and only
pay the thread hop.
- Add a third blocking-io anchor driving the real run_agent with minimal
RunManager/bridge stubs; the factory performs a real production blocking
read (ExtensionsConfig.from_file()) and the test asserts assembly never
runs on the main thread. Verified the anchor goes red when the run_agent
offload is flattened back to a plain call.
- Adapt the test_threads_router checkpoint-builder patch sites to the new
async name.
* refactor(tools): carry assembly offloads on a dedicated bounded pool
Review follow-up for #5224:
- Add utils/assembly_io.py: a dedicated ThreadPoolExecutor (default 8
workers, DEER_FLOW_ASSEMBLY_WORKERS-overridable, mirroring
utils/file_io.py and tools/sync.py) with run_assembly(), which copies
contextvars explicitly. A hung stdio MCP server parks its worker for
the full MCP timeout; carrying assembly hops on the loop's default
executor would let a few parked assemblies queue every other
to_thread/run_in_executor(None, ...) caller behind them.
- Switch all four offloads (run_agent, task_tool, batch _execute_item,
abuild_checkpoint_state_accessor) to run_assembly().
- State the cold-path behavior in the accessor docstring: the graph
cache validates factory identity, so non-identity-stable factories may
duplicate lead-agent assembly across concurrent readers (MCP discovery
stays process-wide single-flight); the pool bounds the duplicates.
- Add a fourth blocking-io anchor driving build_thread_checkpoint_state_
accessor with a per-resolution fresh factory (always a cache miss) and
the real production blocking read; enumerate all four offloads in the
gate's module docstring. Verified the anchor goes red when
abuild_checkpoint_state_accessor is flattened back to a plain call.
* fix(subagents): revalidate batch item before launch; make assembly pool observable
Review follow-up for #5224:
- _execute_item() revalidates the durable state right after assembly and
before executor.execute_async(): renew_item_lease() returns valid=False
when cancel_batch() terminalized the item or the lease was lost while
assembly was parked, and the launch is skipped (the canceller already
finalized the item). Previously the launch was unconditional and the
poll loop's cancellation checks only started after execution began.
- Regression test driving the real SQLite repository: a blocking assembly
probe parks _execute_item, cancel_batch() lands, and the launch is
skipped with the item staying cancelled. Verified the test goes red
when the revalidation is removed.
- run_assembly() tracks pending assemblies and logs a throttled WARNING
once the pending count exceeds the worker count, so assembly starvation
(workers parked on a hung MCP server) is distinguishable from idle.
- The run_agent blocking-io anchor now binds a sentinel extension
snapshot via ctx.extensions and asserts the factory observed it through
get_agent_build_extensions(), pinning run_assembly()'s ContextVar
propagation. Verified red when ctx.run is dropped.
- Document the assembly pool in backend/AGENTS.md.
* fix(utils): decrement the assembly pending count on the pool thread
The pending-assembly counter behind the starvation warning decremented
from the asyncio future's done callback, which never fires once the
submitting loop is closed while its worker is still running: the count
ratcheted up permanently and eventually fired the starvation warning
with no starvation behind it (reproduced at 97dc9bec by review).
Decrement instead from the dispatched work item: run_assembly() wraps
func so a finally drops the count under the pending lock on the pool
thread, and the done callback is gone.
Pin the counter with tests/test_assembly_io.py: a healthy call returns
the count to zero, and an abandoned loop (stopped while the worker is
parked) does not wedge it — the abandoned case goes red against the old
done-callback decrement.
* docs(utils): fix the pending-counter comment after the decrement move
The comment still described the removed done-callback decrement,
contradicting _work()'s own comment; state the actual mechanism
(increment on the loop before dispatch, decrement from the dispatched
work item's finally on a pool thread).
* test(gateway): retarget checkpoint-accessor stubs to the services seam
thread_runs and runs now call abuild_checkpoint_state_accessor, so the
upstream wait-reader, regenerate-prepare, and idempotency tests must stub
the sync builder where abuild resolves it (app.gateway.services); stubbing
the removed router re-exports fails with AttributeError at setup. The
async seam semantics are unchanged: run_assembly invokes the stubbed
sync builder off-loop and propagates its return values and exceptions.
Move the agent/tool assembly off-load note from backend/AGENTS.md to
deerflow/utils/AGENTS.md (next to assembly_io.py) so the effective
instruction chain for agents/middlewares no longer grows past the AG002
hard limit.
* fix(runtime): serialize same-key accessor assembly and release queued-cancel slots
Address the three review follow-ups on the assembly off-load:
- assembly_io: a job cancelled while still queued never runs its work
item, so the dispatched finally never fired and _pending_assemblies
stayed elevated until a false starvation warning. Exactly-once cleanup
now rides the concurrent future's cancelled() state — cancel() only
succeeds before the executor starts the item, so cancelled() is true
precisely when the finally will never run — plus a submit-failure
release; the one-worker queued-cancellation case is pinned red/green.
- services: overlapping cold readers sharing one cache key could both
run full agent assembly. _state_accessor_graph now serializes per key
through a thread-side KeyedLockTable (pool threads, no running loop)
and re-validates factory/app-config identity under the lock, so the
factory runs exactly once while identity changes still rebuild. Cache
dict access is lock-guarded now that construction runs off-loop.
- guidance inventory: register deerflow/utils/AGENTS.md in
EXPECTED_GUIDANCE_PATHS so test_repository_has_the_approved_scoped_
guidance_shape matches the relocated assembly note (CI shard 4).
* test(keyed-lock): pin KeyedLockTable reclamation and waiter bypass directly
Thread-side counterparts of the async table's own tests: overlapping
hold() calls serialize (a late arrival joins the live entry instead of
creating a second lock that bypasses a queued waiter), the last check-in
pops the entry, and many unique keys leave the registry empty. Both
regressions verified red — popping unconditionally trips the late-arrival
test, never reclaiming trips the many-keys test.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): reconcile stuck once tasks from committed run outcome
Restart recovery (cancel_stuck_once_tasks and the multi-instance
reconcile_stuck_once_tasks) blindly flipped every stuck once-task to
'cancelled'. When handle_run_completion crashed between its two
transactions, a once-task whose run had already committed 'success'
was permanently reported as cancelled.
Both reconciliation paths now read the latest scheduled_task_runs row
without a status filter and finalize the parent to match:
success -> completed (last_error cleared),
failed -> failed with the run's error,
interrupted -> cancelled with the run's error when present,
skipped -> cancelled (no work performed).
Active occurrences (queued/launching/running) are left untouched — a
concurrent completion or a later recovery pass will finalize them once
the run reaches a terminal state.
Tasks without a terminal run row keep the previous generic cancellation.
Review follow-ups (willem-bd / Huixin615):
- Extract _finalise_once_task_from_run() so both recovery paths share one
outcome mapping (no more drift between single- and multi-instance paths).
Returns bool (True = finalised, False = active/no-op) for explicit
counter management at call sites.
- Fix a no-op (`run_row.error or None` -> `run_row.error`) in the skipped
branch.
- Drop the unused `status` parameter from the test task helpers.
- Use TERMINAL_RUN_STATUSES / ACTIVE_RUN_STATUSES constants (local copies
to avoid circular import; kept in sync with scheduled_task_runs.sql).
- [P1] Read the latest run AFTER acquiring the parent task row lock, not from
a pre-lock batch snapshot. The latest-run lookup now runs per task under
the lock with populate_existing so a concurrently committed status is read
back fresh.
- [P2] Race tests now use monkeypatch to actually enter the race window:
_intercepted_fetch commits success in a separate session at the moment the
per-task fetch fires, so a reverted pre-lock batch implementation fails the
test, while the current post-lock implementation passes.
- [P1] Do not finalize parent for active occurrences. A non-terminal
scheduled occurrence means the run is still in progress — the parent must
be left untouched until the completion path or a later recovery pass
establishes a terminal outcome.
- [P2] Add cancel_stuck_once_tasks to the single-instance poll loop so
stuck once-tasks are not left permanently "running" when the startup sweep
fails (mirrors multi-instance _reconcile_active_state behavior).
- Fix stale docstrings in cancel_stuck_once_tasks and _fetch_latest_run.
Adds regression tests for multiple historical runs (older success +
newer skipped/active) on both paths, monkeypatch-based race tests that
prove a concurrent completion committing success is reflected as
completed, and active-run tests that verify the parent is left
unchanged. Documents the behavior in AGENTS.md.
Fixes#5034
* fix(scheduler): address review comments on completion-consistency fix
- _fetch_latest_run: drop arbitrary id DESC tie-break; order by
scheduled_for DESC (deterministic recency on schedule position)
- _finalise_once_task_from_run: annotate bool return type
- Centralize TERMINAL/ACTIVE_RUN_STATUSES in scheduled_tasks/model.py;
stop duplicating them in scheduled_tasks/sql.py and
scheduled_task_runs/sql.py (removes stale circular-import workaround)
- cancel_stuck_once_tasks: run unconditionally in single-instance poll
loop (remove try/except swallow)
- tests: pin created_at/scheduled_for in _create_run so recency ordering
is actually exercised; correct docstrings that described the
active-occurrence branch as 'generic cancel' instead of 'left unchanged'
* fix(scheduler): correct finalizer return annotation
* fix: order scheduled task runs by creation time
* fix(scheduler): stabilize latest run reconciliation ordering
* fix(scheduler): order latest runs by creation time
* test: update trace scheduler stub
* fix(scheduler): clarify reconciliation diagnostics
Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(scheduler): fail closed on startup recovery
Keep single-instance parent reconciliation at startup so it cannot race manual admission. Propagate recovery failures through the Gateway lifespan before channel startup, preventing a half-started scheduler.
Tests cover both recovery failure stages and a queued occurrence that survives startup before the ordinary poll drain launches it.
Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(scheduler): order occurrences and fence stale parent writes
Allocate per-task occurrence sequences under the parent lock and guard parent projection across launch, recovery, completion, and queue failure paths. Track launch accounting separately so stale occurrences are counted once without replacing newer results. Commit completion and accounting atomically, preserve legacy history, and cover migrations and reordered execution on SQLite and PostgreSQL.
* fix(scheduler): tighten completion projection and launch fencing diagnostics
Share the once-task outcome mapping between completion and both recovery paths, validate the terminal status before opening the completion transaction, leave cron parent status untouched on completion, log the fenced launch update when an occurrence does not belong to the launched run, and drop the README capability line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scheduler): compare caller time only against unsequenced occurrences
Among sequenced rows the parent-locked occurrence_seq is the only recency key. An unsequenced row can only be legacy history or an admission by a pre-upgrade Gateway writer, so recovery prefers it over the sequence winner only when its caller timestamp is later, which is the previous ordering for that pair. A rolling upgrade therefore degrades to the pre-sequence behaviour instead of ranking every pre-upgrade admission below every sequenced one. Document that boundary instead of requiring every Gateway writer to stop before the upgrade.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scheduler): gate once-task recovery on the same projection rule
Recovery now finalises a once-task parent only from the occurrence that can_project() accepts: the highest sequenced occurrence whenever one exists, or the timestamp-latest row for a task whose history is entirely unsequenced. An unsequenced row admitted by a pre-upgrade writer can no longer cancel a parent whose sequenced occurrence is still live, nor stall finalisation of a parent whose sequenced occurrence already completed. Document that pre-upgrade instances project their own admissions during a rolling upgrade.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scheduler): defer once-task recovery while any occurrence is live
uq_scheduled_task_run_active allows one non-terminal occurrence per task, so a live row is the task's newest admission whatever its caller clock and whether it carries a sequence. Both once-task recovery paths now probe for any active occurrence after the fresh latest-run read and leave the parent untouched while one exists; cancel_stuck_once_tasks also locks the parent row so admission cannot insert a queued occurrence between that probe and the commit. Once no occurrence is live, the sequence winner decides and a terminalised unsequenced row never overrides it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(persistence): follow the local head past canonical 0019
Main's forward-revision tests assumed 0019_thread_incarnations was the local chain head. With 0022_scheduled_occurrence_seq chained after it, seed the canonical-0019 shape explicitly, assert the real head where a database is upgraded, derive the 0020 rollback binary's revision set from the ancestors of its head, and step the PostgreSQL restart scenario back to canonical 0019 before the rollback binary restarts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(migrations): describe the chain through 0022_scheduled_occurrence_seq
The rolling-forward section still ended the local chain at canonical 0019; it now names 0022_scheduled_occurrence_seq as the head and lists it among the revisions the 0020 rollback-floor binary does not know.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(scheduler): accept CI's sync Postgres URL in occurrence fixtures
CI hands over TEST_POSTGRES_URI as postgresql://...?sslmode=disable. The occurrence, ordering and 0022 migration fixtures built async engines from it directly, so SQLAlchemy chose psycopg2, which is not installed. Normalize the scheme to postgresql+asyncpg and drop libpq-only query keys, matching the existing 0019 migration tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(scheduler): keep the backend AGENTS.md chain within its budget
The middlewares guidance chain was already above the hard limit on main, so any added byte in backend/AGENTS.md fails the agent guidance check. Leave backend/AGENTS.md identical to main and record the recovery projection rule in the 0022 migration entry, which already describes the occurrence fields.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tools): filter list_uploaded_files by name and extension
Add optional query and extensions so historical upload discovery can
find older matching files instead of dropping them behind the default
20-item mtime cap.
Fixes#5339
* fix(tools): strip glob stars from list_uploaded_files extensions
Model-supplied tokens like *.pdf were prefixed to .*.pdf and never
matched Path.suffix. Also run ruff format so the backend format gate
passes.
* feat(agents): elide blocked write payloads from model-bound requests
A write_file / str_replace call rejected by the read-before-write gate never
runs, yet its payload (up to 80 KB for a non-append write, unbounded for
append) stayed verbatim in every later model request: nothing in the chain
rewrites AIMessage tool-call arguments, and ToolOutputBudgetMiddleware only
budgets ToolMessage output. The gate demands a re-read plus a fresh call, so
the model re-emits the content anyway and the original is pure dead weight.
- ReadBeforeWriteMiddleware stamps `deerflow_write_block` on the blocked
ToolMessage and, in wrap_model_call, replaces the paired call's payload
fields (content / old_str / new_str) with a short deterministic placeholder
in the model-bound request only. state["messages"], receipts, loop
detection, and the run journal keep the original arguments; nothing is
externalized to disk, since a file reference to content the model must
re-derive after reading the target would only invite bypassing the gate.
- New `tool_call_args` helper rewrites every provider surface together
(structured tool_calls, raw additional_kwargs.tool_calls, tool_use content
blocks, tool_call_chunks) so strict providers never see them disagree; the
gate only supplies the policy (which calls, what placeholder).
- `read_before_write.elide_blocked_payloads` (default on) and
`read_before_write.elide_min_chars` (default 2000) configure it; the
runtime builder passes the config through and the middleware declares it
via release_policy_parameters.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(agents): condense middleware guide entry 11 to fit the guidance budget
The agent-guidance CI check failed: the effective AGENTS.md chain for
agents/middlewares was 99673 bytes against a 98304-byte hard limit. The
chain already sat at 98459 on main, so the ReadBeforeWrite entry could not
grow. Rewrite entry 11 so it states the same facts (gate, lock scope,
fail-open, authorization scope, blocked-payload elision, shared
tool_call_args helper) in 1229 bytes instead of 2640; the chain is now
98262 bytes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(config): bump config_version for the read_before_write elision keys
Review follow-ups on #5329:
- `read_before_write.elide_blocked_payloads` / `elide_min_chars` are new
user-settable YAML keys, i.e. a config schema change, so bump
`config_version` 40 -> 41 in config.example.yaml; without it an existing
config.yaml gets no outdated-config warning and `make config-upgrade` has
nothing to signal.
- Say in the `elide_min_chars` description (and the example comment) that the
threshold and the placeholder's size figure are Python character counts,
not tokens: the same value spans roughly 3-4x in real context cost between
ASCII and CJK text.
- The builder wiring test now asserts only the wired `elide_min_chars` value
instead of the whole `ReadBeforeWriteConfig` dump, so future knobs do not
have to edit an unrelated test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore(helm): bump chart config_version to 41
validate-chart's config_version drift check failed after config.example.yaml
moved to 41 in ef9ee267. Bare bump of the chart's embedded `config:` block
and the README example; the chart does not mirror the read_before_write
section, so no field changes are needed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(agents): rewrite Responses and v1 content-block arguments too
Review finding on #5329 (P2): the content rewriter only handled Anthropic
`tool_use` blocks. With `use_responses_api=true` and
`output_version='responses/v1'`, AIMessage.content carries `function_call`
blocks whose `arguments` still hold the full write payload, and
langchain_openai's Responses input builder emits that block instead of the
rewritten structured call whose `call_id` it already carries. Standard `v1`
`tool_call` blocks likewise keep `extras.arguments`, which the v1->Responses
translator prefers over the structured args. So the blocked payload was
still sent on every later Responses API request.
`tool_call_args` now rewrites every content dialect that carries its own
copy of the arguments: Anthropic `tool_use` (input, drop partial_json),
Responses `function_call` (arguments, matched by call_id, `fc_...` item id
and status preserved), and v1 `tool_call` / `tool_call_chunk` (args plus
`extras.arguments`). Tests assert against the real adapter serializers:
`_construct_responses_api_input` for responses/v1, v1, and v0 messages,
`_convert_message_to_dict` for chat completions, and Anthropic
`_format_messages` for native and v1 content, plus an end-to-end probe
through the gate's wrap_model_call.
* fix(agents): pair blocked writes per call occurrence and defeat Responses chaining
Two review findings on #5329:
- Tool-call ids may repeat across assistant turns (DanglingToolCallMiddleware
pairs them with per-id queues). The gate matched blocked results against a
history-wide id set, so a successful write sharing an id with a later (or
earlier) blocked one also lost its payload and was labelled as blocked.
`_blocked_call_occurrences` now pairs ToolMessages with call occurrences
the same FIFO-per-id way and the selector keys on (message, call id).
- With `use_previous_response_id`, the OpenAI adapter sends only the messages
after the last AIMessage carrying a `resp_` response id and lets the server
rebuild the rest from its stored copy, which still holds the original
arguments and cannot be edited; every later response chains back to it.
`rewrite_messages_tool_call_args` now drops every `resp_` id from the
model-bound copy whenever it rewrote anything, so the adapter replays the
full rewritten history (the `use_previous_response_id=False` request
shape). OpenAI bills chained input tokens as input either way, so replay
costs no more; the state keeps its ids.
Tests cover success-before-block and block-before-success histories through
the Chat Completions serializer, and chaining through
`ChatOpenAI._get_request_payload` with `use_previous_response_id=True`:
unrewritten history chains and omits the call, rewritten history is replayed
with the placeholder and no `previous_response_id`.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
