* feat(knowledge): add independent Wiki and entity relation retrieval tools
* fix(knowledge): address review diagnostics and CI catalog coverage
* fix(knowledge): deduplicate graph evidence and update catalog e2e
* fix(knowledge): clarify bounded evidence and pin browser CI runners
* Update e2e-tests to run on ubuntu-latest
* Update workflow to use ubuntu-latest
* test(ci): stop pinning maintainer-controlled browser runner images
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): extension tool authorization PR2 — middleware-declared tool path
Layer 1 now covers tools declared by middlewares, which LangChain merges
into the bound tool set after the host's explicit-list filter:
- new agents/middlewares/tool_declarations.py: record the ordinary pass's
verdict, collect declarations from the built stack, decide only the
delta seeded by the first pass (a name denied for this build stays
denied), narrow the stack build-locally on independent state-preserving
copies (direct __dict__ write, refused up front for tools data
descriptors, never mutating caller-owned instances), and verify the
exact bound stack after the final normalization copy.
- IsolatedMiddleware ships a state-preserving __copy__ so narrowing
survives full/delta normalization and copies of copies.
- TodoMiddleware degrades when write_todos is denied: no system-prompt
injection, no completion/context-loss reminders, no jump_to.
- Layer 2 forwards host-resolved tool provenance into AuthzRequest.context
and binds the infrastructure exemption to the concrete host-created
tool object instead of a name set (tool=None goes to the provider).
- Wired into all four assembly paths (lead x2, native subagent with the
decision offloaded via asyncio.to_thread, DeerFlowClient).
Tests: new test_middleware_declared_tools.py (35) and
test_layer2_tool_provenance.py (7); blocking-IO anchor pinning the
subagent offload (teeth verified red->green); placement-guarantee and
enforcement suites extended; executor test doubles converted for the
async _create_agent. Behavior change called out in CHANGELOG: with
authorization enabled and a tools policy that excludes write_todos,
plan-mode builds no longer bind it (built-in RBAC default unchanged).
* fix(authz): close residual gaps from review — narrow non-collectable declarations, dedupe candidates, warn on skipped pass
- apply_declared_tool_view now removes non-BaseTool declarations from the
bound view when authorization is enabled (they would otherwise bind via
LangChain's create_tool auto-conversion with no provider decision);
verify_declared_tool_view fails loudly if one survives normalization
- decide_declared_tools dedupes never-submitted names so the provider sees
each candidate once
- subagent executor logs a warning when the provider is set but the seeded
Layer-1 state is missing, instead of skipping the declaration pass silently
- docs: middleware-declared tools must be BaseTool instances with usable names
* fix(authz): fail loudly on non-list/tuple middleware tools declarations
_iter_declared warned and returned () for a 'tools' attribute that is not
a list/tuple, but LangChain's factory iterates the attribute as-is — so a
set/generator/dict_values declaration bound with no Layer-1 decision.
Raise DeclaredToolViewError instead, matching the module's fail-loud
standard for every other un-narrowable shape. Disabled path remains a
strict no-op. Docs updated to match.
* fix(subagents): retain completed model usage on cancellation
* docs(subagents): fit guidance within chain budget
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(agents): persist tool artifacts as resolvable handles (#4676)
MCP and sandbox tools return file paths, URLs, and task ids that are
lost when context compaction summarizes the conversation. This adds a
per-thread artifact handle registry so the model can reference
artifacts by short deterministic handles (art_xxxxxxxx) that survive
compaction and resolve to real references at the tool-call boundary.
- ThreadState.tool_artifacts: reducer channel capped at 100 entries,
same-handle latest-wins, survives summarization compaction
- artifact_registry: deterministic handle derivation from
(thread_id, tool_call_id, call_index); extraction precedence
structured_content -> file/image blocks -> conservative text scan;
HTML-escaped model-context rendering with char budget
- ArtifactCaptureMiddleware: before_model hook captures lightweight
metadata (never content) and tracks handle consumption
- ArtifactResolutionMiddleware: rewrites handles in tool-call args to
real references before execution; unknown handles untouched
- DurableContextMiddleware: projects the registry into the hidden
durable-context data block ([available]/[consumed] status)
- config: new tool_artifacts section (all switches default on)
- frontend: artifact badges on tool-call steps keyed by tool_call_id
Fixes#4676
* fix(agents): harden artifact registry after review (#4676)
Review findings on #4929:
- capture/consumption updates emitted in the same before_model call are
now concatenated instead of dict-merged; dict-merging clobbered the
shared tool_artifacts key and permanently dropped fresh captures
- handles embed a per-result ordinal so multiple references inside one
tool result never collide (two paths in one text block previously
collapsed into one handle)
- structured_content extraction pulls concrete refs from known keys
(file/path/url/task_id/..., string or list-of-string values) instead
of storing a truncated JSON preview, falls back to an untruncated
whole-object data entry, and no longer skips file/image content blocks
- plain-string tool results (e.g. bash/write_file output) are scanned
for sandbox paths / remote URLs too, gated by detect_refs_in_text
- wire max_entries through configure_tool_artifact_max_entries at
assembly (clamped 10..1000) and gate detect_refs_in_text scans
- resolution requires both enabled and resolve_handles_in_args,
matching its docs
- add real ToolCallRequest.override coverage and the spec 14.4
integration cycle (capture -> inject -> resolve -> consumption)
* fix(agents): address second-round artifact registry review (#4676)
Follow-up findings on #4929:
- drop generic `output`/`outputs` from structured ref keys: prose under
those keys no longer becomes a bogus file entry (falls back to the
whole-object data entry; path-shaped refs are still caught by the
free-text scan)
- strip closing quotes, brackets, and backticks from detected refs so
markdown- and JSON-wrapped paths resolve to clean real references
- remove the process-wide cap setter: the reducer enforces only an
absolute ceiling (1000) and ArtifactCaptureMiddleware applies the
configured max_entries per agent when emitting updates, so agents
with different configs in one process never share or flap a cap
- skip extraction for tool messages whose tool_call_id already has
registry entries and memoize known-empty results (bounded FIFO), so
steady-state capture cost drops to the new message tail instead of a
full-history rescan per model call
- correct the inverted ordering claim: capture is position-independent
in the wrap chain because it is a before_model hook reading state
messages; fixed the assembly comment, AGENTS.md item 15, and the
ordering test docstring
CI parity verified locally: ruff check + format --check, full offline
suite, blocking-io suite, collect-only, uv lock --check,
check_agent_guidance, frontend format/lint/typecheck/build/tests.
* fix(agents): sliding-window cap and URI gates for artifact registry (#4676)
Third-round review findings on #4929:
- max_entries is now a sliding retention window: fresh captures are
always registered, and when the projection exceeds the configured
cap the middleware appends a trailing {"op": "trim_to", "keep": N}
directive that the reducer applies by evicting the oldest entries.
The previous remaining-budget emission froze the registry at the cap,
permanently dropping post-compaction artifacts — the exact failure
mode #4676 targets — and silently disabled the round-2 rescan
optimizations at the point threads grew longest
- _track_consumption memoizes settled tool-call args in a bounded FIFO
alongside the existing empty-result memo, so historical AIMessage
args are not regex-rescanned every model call
- file/image content-block URLs must be http(s) or path-shaped;
data:/blob: URIs (MCP embedded resources carrying base64 payloads)
can no longer enter thread state, tool args, or crowd out the render
budget; protocol-relative //host forms are rejected as well
- structured-content ref values go through the same URI-shape gate as
content blocks, closing the embedded-payload hole via known keys such
as url/urls
- tool_artifacts.enabled now gates the whole middleware including
consumption tracking via an early return in before_model
Docs updated to describe the sliding window, URI gates, enabled scope,
and both bounded memos.
* fix(agents): final evictions and pending-aware consumption for artifact registry (#4676)
Fourth-round review findings on #4929:
- the sliding window interacted with the seen_call_ids skip to
resurrect evicted entries every round: once an entry was trimmed, its
tool_call_id left the seen set while its ToolMessage stayed in
context, so the next model call re-extracted it, re-registered it as
newest, and evicted something else — a never-converging rotation.
The empty-result memo is now a unified _processed_results memo that
records every extraction regardless of outcome, making evictions
final; steady state over a saturated thread is quiescent (pinned by
a multi-round test asserting no updates and zero extractions)
- the consumption scan built its handle map from pre-capture state, so
a call referencing a handle captured in the same before_model call
missed, got settled as quiet, and its consumption mark was lost
forever. _track_consumption now receives the pending capture entries
and resolves against state + pending; a call settles as quiet only
after every referenced handle resolved, so transient misses retry
next round instead of permanently suppressing the mark
AGENTS.md entry 15 updated to describe both memos and the
eviction-is-final semantics.
* fix(artifacts): address occurrence identity, PII and URL review findings
* fix(artifacts): enforce resolved policy checks and bounded registry handling
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(prompts): add operator overlays for core system prompts
* test(prompts): isolate SOUL forwarding test from operator config
* fix(prompts): wrap complete subagent system message with overlay
* fix(subagents): serialize prompt overlays in durable batches
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(utils): return empty text for content-less messages
message_content_to_text fell through to str(content), so a message whose
content is None yielded the truthy literal "None". Two call sites already
work around it with a local `or ""` and name the helper in the comment; the
subagent executor's `text if text else "No response generated"` fallback and
the archive's `if not text: continue` skip cannot work around it, so a
content-less terminal turn was reported as an answer of "None" and a
content-less LLM error fallback surfaced "None" instead of its error_detail.
* test(utils): cover non-None content compatibility and document fallbacks
* test(utils): cover contentless task history and refresh guard comments
---------
Co-authored-by: Lengshuang <90967079+Lesereingrape@users.noreply.github.com>
Co-authored-by: JasonH <4430962+yang0228@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): scale max_turns into the graph's super-step budget
max_turns was handed to LangGraph as recursion_limit, but the two count
different things. recursion_limit counts super-steps, one per graph node,
and create_agent compiles a node for every middleware lifecycle hook, so
one turn costs before_model + model + after_model + tools nodes — seven to
eight through the subagent chain. The built-in general-purpose agent's
max_turns=150 therefore bought about 18 tool-using turns before failing as
turn_capped, and every middleware added to the chain shrank the effective
budget again.
Resolve the limit from the chain each subagent was actually assembled with
(subagents/turn_budget.py) instead of passing the turn count through, so
raising max_turns buys the turns it names.
No config keys or defaults changed; existing max_turns values now grant
their full budget, bounded as before by subagents.timeout_seconds and
subagents.token_budget.
* fix(subagents): warn when a counted hook can jump the agent loop
Review follow-up. The per-turn cost is a flat multiplier over the straight
before_model -> model -> tools loop. A hook that declares can_jump_to and
returns {"jump_to": ...} re-enters the loop without traversing tools,
spending another before_model + model + after_model pass that buys no tool
result, so the resolved limit becomes a lower bound rather than an exact
budget — silently re-creating the short budget this translation fixes.
Measured against a compiled graph: with one jumping after_model hook,
three tool turns need the resolved limit plus one jump pass, and the run
raises GraphRecursionError at the resolved limit.
How often a jump fires is data-dependent and unbounded, so it cannot be
folded into the arithmetic. find_jumping_hooks reports the condition off
the same __can_jump_to__ attribute the factory reads, and the executor
warns when a counted hook declares one. Nothing in today's subagent chain
does, so this changes no budget.
* fix(subagents): detect jumps declared on agent-level hooks too
Review follow-up. find_jumping_hooks exempted before_agent/after_agent on
the grounds that a jump out of them lands in the loop the budget already
pays for. That does not hold on langchain 1.3.14:
- after_agent jumps re-enter the loop after it finished, and the hook runs
again on the next exit, so the extra passes are unbounded. Even
jump_to "end" is routed to exit_node, the head of the after_agent chain,
so it reruns the chain; destinations are no safe filter.
- a before_agent hook that stages a tool call and jumps to tools runs a
tools step no model turn paid for. It is O(1), but the resolved limit
has zero headroom, so one step caps the last turn.
The detector now scans every hook pair the factory wires jump edges for.
The compiled-graph pin is parametrized over after_model->model,
after_agent->model, after_agent->end and before_agent->tools, each raising
GraphRecursionError at the resolved limit and completing once the jump's
cost is added. No middleware in the subagent chain declares a jump on any
hook, so this changes no budget.
* fix(tests): await future completion before asserting done()
`test_run_on_isolated_subagent_loop_survives_caller_loop_teardown`
signals from inside the coroutine:
async def deferred_work() -> None:
completed.set()
`run_on_isolated_subagent_loop` is `asyncio.run_coroutine_threadsafe`,
whose `concurrent.futures.Future` is marked done by the loop only after
the coroutine returns. The main thread can therefore wake from
`completed.wait()` while the future is still pending, and
`assert handles[0].done()` fails:
assert False
+ where False = done()
+ where done = <Future at 0x7f62241b22d0 state=pending>.done
Observed on main at a2808e82 (shard 2) and on an unrelated PR at
852a94dd (shard 4) sixteen seconds apart, so it tracks runner load
rather than any change under test.
Assert the result first — `Future.result(timeout=10)` blocks until the
future completes — then assert `done()`. Both assertions keep their
original meaning and no sleep is introduced.
Fixes#5298
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test: drop the now-redundant done() assertion
Review follow-up. Once `result(timeout=10)` has returned normally the
future is guaranteed to be FINISHED, so the `done()` assertion below it
could no longer fail — it documented intent rather than checking
anything.
`result()` alone proves both halves of what the test is about: that the
coroutine body ran after caller-loop teardown, and that the future
resolved. The `completed.wait()` guard above still covers the "work
never ran" case with a descriptive message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
TestBashExecutionHarvest's shell-persistence tests report
shell_persistent=None whenever deerflow.sandbox.overwrite is not already
cached in sys.modules. _setup_executor_classes replaces the
"deerflow.sandbox" parent package with a MagicMock, so a later
`from deerflow.sandbox.overwrite import unwrap_sandbox` inside
_harvest_shell_persistence can no longer locate the submodule through
the mocked parent ("'deerflow.sandbox' is not a package") when the leaf
module is not already in sys.modules. The helper's
`except Exception: return None` silently converts that ImportError into
an UNKNOWN provenance stamp.
Whether the leaf module was cached depended on which tests ran earlier
in the session, making the outcome order-dependent: green in CI by
collection-order luck, red when the module runs alone or first.
Fix it the same way the fixture already handles audit_context and
tool_search: preload the real leaf modules (deerflow.sandbox.sandbox_provider
and deerflow.sandbox.overwrite) before installing the mocked parent
package, pin them in sys.modules for the duration of the test, and
restore the previous state afterwards.
## Why
DeerFlow declares one timestamp convention in deerflow/utils/time.py: every lifecycle timestamp is UTC (now_iso / datetime.now(UTC)). SubagentResult writers in subagents/executor.py still used naive datetime.now(), so on any non-UTC host the in-memory lifecycle metadata (started_at / completed_at) was local wall-clock time. The sibling durable-batch path (subagents/batch_service.py) already stamps datetime.now(UTC), so the same run model carried two different conventions depending on which path wrote it.
## What changed
- Added an executor-local _utcnow() helper that stamps datetime.now(UTC).
- SubagentResult.completed_at default in try_set_terminal(), result.started_at in _aexecute(), and the started_at default in _aexecute_admitted() now route through _utcnow().
- Explicit caller-supplied timestamps (completed_at=...) still pass through unchanged.
- Added regression tests asserting the default writers produce UTC-aware datetimes.
## Surface area
- [x] Backend runtime (deerflow.subagents.executor) - internal dataclass lifecycle metadata, no wire format change
- [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies / Default behavior change
## Bug fix verification
- New tests: tests/test_subagent_executor.py::test_timestamp_writers_stamp_utc_aware_datetimes and test_utcnow_helper_returns_utc_aware_datetime encode the convention.
- Updated BlockingDateTime.now() in the terminal-publication-order test to mirror datetime.now's optional tz argument.
## Validation
- cd backend && python -m pytest tests/test_subagent_executor.py: 136 passed; 2 pre-existing TestBashExecutionHarvest failures reproduce identically on clean main (Windows sandbox env), unrelated to this change.
- ruff format + ruff check clean on both changed files.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** analysis of the timestamp conventions, implementation, and regression tests authored with AI assistance; change reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
* feat(ci): split backend unit tests into parallel CI shards
Split the single offline backend `make test` job into four GitHub Actions
matrix shards (SPLITS=4, GROUP=1..4) via pytest-split, so the ~12k-test suite
runs in parallel instead of in one 15-minute job. Each shard runs on its own
runner with its own Postgres/Redis services; fail-fast: false lets a failing
shard report its owned tests without cancelling its peers.
`make test` stays the canonical full-suite entry point; CI now calls the new
`make test-shard SPLITS=4 GROUP=N`. tests/blocking_io remains owned solely by
the dedicated blocking-I/O workflow (excluded via --ignore), extending #5105.
Fixes#5088
* test(ci): make backend test shards duration-aware and pin the contract
Make `make test-shard` an explicit least_duration split that READS
backend/.test_durations (read-only for shards, so concurrent CI jobs never
race writes on it), and add `make test-shard-durations` to regenerate that
file from the full offline suite. Update the CI unit-test workflow contract to
call `make test-shard SPLITS=4 GROUP=<n>` and assert the shard command carries
--splits 4, --group 2, -m "not live", --ignore=tests/blocking_io and
--splitting-algorithm least_duration. Verified on the real 13,140-test normal
suite that the four shards are pairwise disjoint and their union equals the
unsplit suite.
Refs #5088
* test(ci): fail fast when the duration baseline is missing
`make test-shard` now requires backend/.test_durations and exits with a clear
error instead of letting pytest-split silently degrade to an even (count-based)
split. Harden the CI contract test to pin `--durations-path=.test_durations` and
to assert the repo ships the committed duration baseline.
Refs #5088
* docs: trim backend/AGENTS.md within guidance budget
* test(ci): add backend test duration baseline
Add the duration baseline generated by a full offline backend run on a
GitHub-hosted ubuntu-latest runner (the same runner type the shards use), so
`make test-shard` balances the four matrix shards by real wall-clock cost.
Refs #5088
* test(ci): make the duration writer honor DURATIONS_FILE
`test-shard-durations` now writes `--durations-path=$(DURATIONS_FILE)` instead of a
hard-coded .test_durations, so the reader and writer stay consistent when the
path is overridden.
Refs #5088
* test(ci): address sharding review feedback
* test: isolate subagent execution capacity state
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(gateway): issue request trace ids unconditionally
The request trace id was gated behind logging.enhance.enabled at every
entry point, so downstream code had to keep asking whether one existed:
a header-provenance flag in its own ContextVar, a precedence resolver,
and three-level carrier fallbacks at each consumer.
Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP;
ensure_trace_context covers the entry points that never touch ASGI --
scheduled occurrences, MCP task notification runs, IM channel messages,
and the embedded client -- each scoped to one unit of work so a
long-lived worker task cannot leak one occurrence's id into the next.
The ContextVar becomes the only source; the response header, runtime
context, run metadata and log records are derived outputs.
Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and
drop their presence guards. Removed: resolve_deerflow_trace_id, the
header-provenance flag and its three helpers, set/reset_current_trace_id,
is_trace_correlation_enabled and its gateway alias.
BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and
it cannot be turned off; logging.enhance.enabled controls log output
only. Installations on the default enabled: false will start seeing the
header. No config keys were added or removed.
* fix(gateway): stop persisting a caller-supplied trace id on the run record
body.metadata forks two ways: through build_run_config into the live run
config, which the run worker restamps, and through create_or_reject into
the run record that the runs API echoes verbatim. Only the first was
covered, so a client sending metadata.deerflow_trace_id made the most
durable and most visible surface of a run disagree with the X-Trace-Id
and the log lines the same request produced -- a correlation id that
does not match the logs is worse than none.
Stamp the server-issued id once at the trust boundary so both forks
receive it, preserving the caller's own metadata keys. Close the same
gap on config.context, which reaches the runtime context by a separate
path: _build_runtime_context no longer merges server-owned keys from the
caller, and _install_runtime_context assigns rather than setdefaults.
A thread's metadata is no longer seeded with the run-scoped id of
whichever run created it -- one thread spans many runs and as many
trace ids.
Found by driving a real run through the Gateway and reading the run back
from the runs API; every unit test built its metadata by hand and so
could not see it.
* fix(gateway): expose X-Trace-Id to split-origin browser clients
X-Trace-Id is not on the CORS safelist, so a browser client served from
a separate origin could not read it -- and those are exactly the clients
that cannot read the Gateway's logs either, leaving them with nothing to
quote in a bug report. Same-origin nginx deployments were unaffected,
which is why this stayed hidden.
Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing
TRACE_ID_HEADER rather than repeating the literal.
* fix(gateway): keep X-Trace-Id on unhandled-exception 500s
Starlette's ServerErrorMiddleware sits outside every user middleware and
emits unhandled-exception 500s through the raw send, so those responses
never pass TraceMiddleware's header-writing wrapper. The 500 for a server
bug is exactly the response a user most needs to correlate with a log line,
and it was the one response that shipped without the id.
TraceMiddleware now tracks whether http.response.start has been sent. On an
exception with no response started it emits its own plain 500 carrying the
header, then re-raises: the outer ServerErrorMiddleware sees the response
already started and only re-raises too, so the server's exception logging is
untouched. An exception mid-stream keeps propagating unchanged — a second
response start cannot be sent, and the already-written header stands.
The trace id is printable ASCII by construction (normalize_trace_id /
generate_trace_id), which is what makes the raw latin-1 header encoding
safe.
* fix(gateway): strip the forged trace id from the persisted request echo
The run-record fix stopped a forged metadata.deerflow_trace_id on the
authoritative metadata surface, but the raw request echo still carried one:
create_or_reject persists body.config verbatim as runs.kwargs_json, which
the runs API serves back. A client posting config.context.deerflow_trace_id
therefore still got its forged value stored and echoed on one API surface
while the header, logs, run metadata, and checkpoint all carried the real
id — the id is ignored as input there, so echoing it back only manufactures
disagreement.
Two changes close it. redact_config_secrets — already the shared scrub for
that echo, applied at admission and again at serve time, so historical
records are covered too — now also drops deerflow_trace_id from
config.metadata and config.context. And build_run_config now merges run
metadata onto a copy of the caller's config["metadata"] instead of updating
it in place: the nested values of the request config are reference copies,
so the in-place merge was writing the server-stamped key through into
body.config, contaminating the "what the client sent" record before it was
persisted (and incidentally masking the forged-value echo on the metadata
container).
The regression test posts a forged id through body.metadata,
config.metadata, and config.context at once and reads the kwargs echo back
off the run record, failing if either leak returns.
* docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence
The trace section of the harness AGENTS.md now covers the two fixes that
close the derived-output rule (the kwargs-echo scrub in
redact_config_secrets plus build_run_config's copy merge, and
TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains
their Fixed entries.
It also writes down the one accepted divergence: a crash-recovered
scheduled launch reuses the durable run through its idempotency key, and
start_run returns early on idempotency_reused without restamping — so the
run record keeps the first attempt's deerflow_trace_id while the retry's
own log lines carry the freshly minted id of its ensure_trace_context
binding. The divergence is confined to the crash-recovery window and is
accepted rather than fixed: restamping on reuse would rewrite a persisted
record for a run that already exists, which is worse than two ids that each
correlate their own attempt's logs. Written down so the next reader of the
scheduler recovery path does not diagnose it as a bug.
* docs(config): align the logging.enhance schema note with the unconditional trace id
The config-module AGENTS.md still described logging.enhance as the gate for
the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is
gone: ids are issued unconditionally and this block decides log output only.
Left as-is, the stale wording invites an agent to "restore" a header gate it
believes was lost. Reworded to match the sibling AGENTS.md files and
config.example.yaml, with a pointer to the Request Trace Context section
that owns the full model.
* docs(changelog): link the trace entries to #5119
The five new entries pointed at the ([#XXXX]) placeholder with no reference
definition, rendering as literal text instead of a link — and RELEASING.md
step 2 relies on those references when the section becomes release notes.
All five now point at #5119, with the definition appended to the reference
block.
* refactor(harness): rename _stream_without_trace_context to _stream_turn
The name asserted the opposite of what the method now does. It was accurate
while logging.enhance.enabled could route stream() around the trace scope;
with the gate gone it is the only stream implementation left, and it binds
the id itself via ensure_trace_id(). Private, so the rename touches only the
definition and the one stream() call site.
* docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget
The expanded Request Trace Context section pushed the effective AGENTS.md
chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit
scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the
section from 7,359 to 4592 bytes with no facts removed: the entry-point
table, the derived-output rule and its enforcement points, the accepted
scheduled-retry divergence, the two resolution helpers, the stream()
binding rationale, the log-output-only gate, the CORS listing, the 500
fallback, and the test map all remain.
Sized against the merge, not just the branch: current main grew the same
chain by ~724 bytes, so the check was verified on the merged tree as well
(97,772 bytes; branch tree 97,048).
* fix(gateway): declare content-length on the fallback 500
The pre-response 500 declared content-type but no content-length, leaving
the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on
HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response
it replaces, which sends content-length: 21. The explicit header keeps the
fallback byte-identical to what clients saw before.
* docs(readme): drop the trace-correlation condition from the translations
The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id
matches X-Trace-Id "when request trace correlation is enabled". The id now
always matches and that condition no longer exists, so each bullet states
the unconditional match and that logging.enhance.enabled only controls
whether the id is printed into logs — the one piece of the feature a user
can still configure.
* test(gateway): pin TraceMiddleware wiring through create_app()
Every X-Trace-Id test exercised a hand-built four-route app, so the real
stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting
it — or short-circuiting above it — passed CI while silently dropping both
the response header and the ambient id the run-record stamp and enhanced log
records derive from. One case now drives /health through create_app() and
asserts the inbound id round-trips; mutation-checked by removing the wiring
line, which fails exactly this test.
* docs(gateway): note the fallback 500 is CORS-opaque
The pre-response 500 is emitted outside CORSMiddleware — the exception has
already unwound past it — so it carries no Access-Control-Allow-Origin and
a split-origin browser client cannot read the id on this one response,
unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the
class and in the CHANGELOG entry rather than fixed: replicating the origin
allowlist outside CORSMiddleware would let the two policies drift.
* fix(harness): keep abandoned-stream cleanup inside the trace binding
stream() binds the turn's id around each next(inner) and resets it before
yielding, but the finally's inner.close() ran after that binding was gone.
Abandoning the stream therefore drove the inner LangGraph generator's
GeneratorExit/finally path with no trace id — or an unrelated ambient one
from whichever context ran the close — so cancellation and finalization
logs and callbacks did not correlate with the turn they belong to.
inner.close() is now wrapped in a local bind/reset of the same turn id. The
token is set and reset in the same frame, never across a yield, so the
per-step cross-context safety is preserved even when GC closes the
generator from another Context — pinned by the existing copy_context close
test, which now exercises this path. The regression test records the id
from the inner generator's finally and fails without the binding.
* test(harness): teach the worker-trace fake about RunManager.cleanup
Upstream #5112 (bound gateway memory after terminal runs) added a
run_manager.cleanup(run_id) call to run_agent's finalization, so the
merge-commit CI run failed all five worker-trace-binding tests with
AttributeError on this PR's _FakeRunManager. The fake gains the same no-op
shape as its other methods.
* docs(gateway): bring the gateway AGENTS.md back under its soft budget
Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over
the 40,960 soft budget that
test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes
enforces — its Unit Tests run on main was cancelled by push concurrency, so
main is currently red on that test and every PR merge-run inherits the
failure. Two whitespace/wording trims in the row #5092 touched (a doubled
space, and "its configured `context_window`" → "its `context_window`")
bring the file to 40,953 with no content change.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2)
PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a
subagent completes, so objectively checkable requirements can never be
silently passed by a self-report.
- subagents/acceptance_checks.py: deterministic leaf families —
file:<path> exists|non-empty and file_written:<path> read through
read_current_file_content scoped to the shared thread workspace; the
read uses the sandbox-native virtual path form (the local read
validator and provider mount tables resolve /mnt/user-data/... paths,
not host paths); the scope decision canonicalizes with realpath on the
local sandbox so workspace symlinks cannot escape into uploads; a
remote provider's "Error: ..." return string is normalized to a
failed check (provider-typed via is_local_sandbox); a
UnicodeDecodeError marks a binary deliverable as existing and
non-empty; out-of-scope paths degrade to UNVERIFIED.
tests_passed:<command> anchors to a matching recorded bash execution
with status=success and a test-summary shape; matching is
shell-structure aware with control-flow attribution (span must end at
the last segment with provable execution), negating-option values are
ineligible evidence and a target negated anywhere in the command
degrades the match, extra flags must be selection-preserving, extra
positionals widen only after a path-scoped criterion, truncated
commands degrade via command_truncated, the summary shape is read
only from output attributable to the matched segment (preceding
segments provably silent by invocation form), and pass shapes require
a nonzero passed count. Criterion text is neutralized with
neutralize_untrusted_tags before storage/rendering. Anything else
renders UNVERIFIED, never silently passed.
- executor: accumulate bounded bash command/output evidence per streamed
chunk (merged by tool_call_id, newest-capped) so subagent
summarization compacting earlier messages cannot erase a recorded
execution; the recorded status is the actual shell exit status parsed
from the output's exit marker (signed codes included; the remote
Command exited with code N form is accepted only as the whole trimmed
output), falling back to deerflow_tool_meta only when no marker
exists.
- sandbox providers: e2b/opensandbox/tenki/boxlite append the
LocalSandbox-style "Exit Code: N" marker on nonzero exit even with
non-empty output; aio propagates the SDK's structured exit_code on
both exec paths the same way; local timeouts append Exit Code: 124;
and _truncate_bash_output always preserves a trailing exit marker
(signed included) inside its budget, with a 32-char floor raising any
smaller configured limit, so the actual shell outcome always survives
in the output text.
- task_tool: run the checklist offloaded (asyncio.to_thread) on the
completed branch, failure-isolated; stamp the verdict into result
metadata and render the per-criterion section into the model-visible
result text.
- status contract: additive subagent_acceptance_verdict transport with
read-side structural validation.
- delegation ledger: entry carries the verdict and renders a compact
acceptance segment; gateway strips caller-forged verdicts from both
ledger entries and message metadata, like the citation verdict.
- blocking-IO anchor pins the offload (teeth proven red->green); leaf
read errors catch only OSError/SandboxError so unexpected errors reach
the task-tool-level isolation instead of being mislabeled.
* fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4)
- negating options: overlap with a matched criterion target is now
checked by path/nodeid prefix, not exact token equality — excluding a
sub-path of the criterion's selection (pytest tests --deselect
tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds
- output attribution: any redirection token in the matched final segment
makes the recorded tail non-attributable (> / >> / 2> are word
characters to the parser, so redirection was invisible to the matcher)
- silent-source allowlist narrowed from any *activate suffix to the
*/bin/activate shape
- status_contract docstring: restore the shared-fixture sentence and
note subagent_acceptance_verdict is deliberately outside the fixture
- executor: update_bash_executions publishes [] (stream carried no
bash-family calls) instead of collapsing it into None, mirroring
update_tool_receipts
* fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4)
- tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error"
and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass
shape when the exit status is swallowed (|| true) or absent; zero-error
counts stay clean.
- file leaves: bound the deliverable read — a "wc -c" shell size probe
answers files above 50k bytes without loading ~2x their size, honoring
the host-bash kill switch and falling back to the full read on any
non-integer rendering, so verdicts never get less sound.
- executor: record the exit marker text as status_marker on harvested bash
evidence; the leaf detail now reports the marker actually seen instead of
asserting a failure indistinguishable from the command's own trailing text.
- extend the blocking-IO anchor to drive the probe branch inside the
offload; teeth re-verified red->green.
* fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4)
- file leaves: never read unbounded — size is established first (os.stat on
the validated local host path, so the host-bash-disabled configuration
needs no shell; a guarded wc -c on remote providers that renders
missing/unreadable in its own words). Above the 50k cap the leaf answers
from the size alone, at/below it the full read runs, and an
unestablishable size degrades to UNVERIFIED instead of an unlimited
fallback read.
- output attribution: source/. prefixes are never provably silent — a
crafted */bin/activate path shape says nothing about what the script
prints, so sourced segments can no longer lend a passing summary.
- executable identity: an explicitly path-spelled criterion now requires
the same normalized executable path; the basename rule stays only for
deliberately bare criterion commands.
* fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4)
- remote probe no longer runs in the sandbox's persistent shell: a fresh
env -i /bin/sh with absolute-path stat/realpath (poisoned functions,
aliases, PATH, exported functions, IFS, locale cannot steer it), plus a
marker env routing AIO onto a fresh per-call bash.exec session.
- metadata-only: stat never opens content, so a FIFO deliverable cannot
block the parent for the provider's idle timeout; non-regular files
(fifo/dir/symlink) degrade to UNVERIFIED.
- containment canonicalized against the literal mount root: a
final-component symlink or a swapped parent directory (root included)
cannot redirect the check outside shared storage; unprovable layouts
degrade to UNVERIFIED.
* fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4)
Literal-root equality made every remote file leaf permanently UNVERIFIED
on e2b and Tenki, which realize /mnt/user-data as a symlink to the home
dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink).
Containment now compares the file's realpath against the mount root's
realpath — exactly what the provider's own read path resolves, so probe
and read-back stay consistent; final-component symlinks stay rejected by
the non-dereferencing stat, and an intermediate dir-link escape under a
sane root still lands ESCAPED. The inner script is a module constant and
the suite now executes the composed probe for real against on-disk
layouts (real dir, symlinked prefix, final symlink, fifo, missing,
dir-link escape), which the canned-output stub could not see.
* fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4)
- matching: a criterion with no positional selection target (bare pytest,
make test) stands for the runner's default selection, so ANY negating
option (--ignore/--deselect/...) makes the recorded run a different
selection — unprovable. The overlap guard only sees consumed criterion
tokens, which a bare criterion does not have; scoped criteria keep the
unrelated-exclusion behavior.
- attribution: cd is no longer blanket-silent — CDPATH makes cd print the
resolved (subagent-chosen) destination and the pass shapes match as
substrings, so one mkdir 'all tests passed' plus an export minted a pass
for any quiet command. A cd argument or CDPATH= value (export or leading
assignment) carrying any summary shape makes the segment non-silent;
shape-free cd dir wrappers keep matching.
- docs: _truncate_bash_output states the effective 32-char floor (the
guarantee previously read as an unconditional max_chars bound).
* fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4)
Self-audit in the shape of the last review rounds — channels the matcher
classified as accounted-for that can change what runs, narrow the
selection, or lend the summary text:
- env assignments are no longer blanket-stripped: only an allowlist of
inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a
matched span, and a non-allowlisted assignment in any preceding segment
(pure-assignment or export NAME=) is state pollution — PATH redirects
the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code,
PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs,
BASH_ENV runs arbitrary shell startup. All degrade to unprovable.
- runtime expansions: any span token carrying /$( )/backticks, any
negating-option value carrying an expansion or glob (unknown excluded
set), and any extra executed token carrying glob metacharacters
(crafted option-looking filenames narrow invisibly) are unprovable.
Criterion-side globs stay self-consistent (literal match).
- cd: an argument carrying a runtime expansion or glob is non-silent
(unknown destination, unknown print); CDPATH= assignments are now
handled as state pollution at the match layer, subsuming the
value-shape special case.
* fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4)
- tests_passed: on a persistent-shell provider (new
Sandbox.persistent_shell_sessions capability, set by AioSandbox) every
leaf degrades to UNVERIFIED — any earlier call in the shared session
could have mutated the state the clean-looking run executed in, and
only a fresh controlled session (RFC section 6 verifier) can prove
otherwise. The flag is read from the provider registry without
acquiring a sandbox.
- env assignments: the allowlist is gone — no variable is provably inert
across repositories (CI/DEBUG are routinely read by tests). The span's
assignment prefix must equal the criterion's exactly (values included,
order-insensitive); any assignment or export NAME= in a preceding
segment is state pollution.
- scoping: positional targets are now read by option arity, so a path
embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never
counts as a selection target and an extra positional after such a
criterion narrows the default selection it denotes.
* fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4)
* fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4)
* fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4)
* fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4)
* fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4)
* fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4)
* fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4)
* fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4)
* fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4)
* fix(subagents): clean up background task entry on unexpected poller exit
* fix(subagents): pin deferred cleanup to the persistent subagent loop
The non-terminal fallback scheduled the deferred registry cleanup with
asyncio.create_task on the poller's own loop. Under synchronous tool
invocation the sync wrapper runs the tool coroutine through
asyncio.run(), which cancels caller-loop tasks at teardown, so the
cleanup died before executing and the _background_tasks entry leaked —
the same lifecycle leak the terminal path already fixed.
Schedule the deferred cleaner on the process-owned persistent subagent
loop instead, via the new public executor helper
run_on_isolated_subagent_loop (asyncio.run_coroutine_threadsafe). The
cleaner only touches thread-safe registry helpers, so it is
loop-agnostic. A caller-loop fallback remains for the unreachable case
where the persistent loop cannot be obtained, so scheduling never
raises out of an unwind path that is already handling an error. The
polling-timeout return path, which shares the scheduler, is fixed the
same way.
Tests no longer stub the scheduler: the non-terminal fallback test
drives the real scheduling wrapper on an equivalent long-lived loop and
asserts cleanup runs after asyncio.run() tears the caller loop down,
and run_on_isolated_subagent_loop itself is covered against the real
persistent loop with the caller loop closed underneath.
* fix(subagents): harden interrupted finalization against failing status path
Three edge cases from review on the unexpected-exit unwind:
1. Finalization no longer depends on the failing status accessor.
_peek_subagent_result distinguishes a gone entry from an unreadable
one instead of letting the accessor's exception abort the unwind;
_finalize_interrupted_subagent never raises (so the original poller
exception is preserved) and attaches the deferred cleaner, whose
last resort force-removes a persistently unreadable entry via the
new executor force_cleanup_background_task.
2. The generic-error unwind waits only a short grace period
(_UNEXPECTED_EXIT_GRACE_SECONDS) instead of the full execution
timeout before re-raising; the remaining lifecycle stays with the
deferred cleaner on the persistent subagent loop.
3. The deferred cleaner reports the subagent's final usage (deltas
since the unwind snapshot included, via final=True bypassing
usage_reported; the journal dedupes by source_run_id) before
removing the terminal entry. The report is transferred in a plain
worker thread so the RunJournal's loop-bound progress flush is
skipped rather than scheduled on a foreign loop.
* fix(subagents): pin deferred final usage delivery to the parent run loop
_report_deferred_final_usage ran record_external_llm_usage_records in a
to_thread worker, making it the first cross-thread RunJournal writer: the
unlocked accumulators can lose token updates and _tokens_by_model mutations
race get_completion_data() iteration on the parent loop. Capture the parent
loop at unwind time (it is alive in every path that continues the run) and
deliver the final report onto it with call_soon_threadsafe, serialized with
all other journal access; when that loop is already closed (asyncio.run
teardown) the report is dropped on purpose — the run has persisted and
nothing reads the counters back.
The live-loop test exercises the real recorder path (journal captures the
running loop of every call) instead of stubbing _report_subagent_usage, so a
cross-thread report would surface as a wrong-loop entry. Also gates the two
teardown tests on the caller loop actually closing (the deferred cleaner
could otherwise legitimately deliver while that loop is still winding down),
and adds a task_tool-level regression test for execute_async submit failure
leaving no registry residue (rolled back inside execute_async since #5086).
* docs(subagents): document the reverse loop boundary; observability + test fixes
Extend subagents/AGENTS.md's Isolated-loop callback boundary with the
reverse-direction contract from #5069: deferred registry cleanup is pinned
to the persistent subagent loop via run_on_isolated_subagent_loop, and the
final usage report is handed back onto the parent run's loop captured at
unwind time — never invoked from the persistent loop or a worker thread,
which would silently reintroduce the journal accumulator/iteration race.
Dropping the final report (closed parent loop) is the one path where a
subagent's tail usage goes permanently unaccounted, so both drop branches
now log at info with the execution id and the unaccounted record count.
Restore the retention assert in test_deferred_cleanup_task_retained_and_
survives_gc: the bounded wait observes the production done-callback discard,
the assert (not a manual discard) is what fails if that callback is
deleted.
* fix(subagents): honour grace-wait cancellation and shrink deferred-cleaner captures
Two review follow-ups on the unwind:
- The shared unwind absorbs CancelledError (never-raise contract), so a
graph-node cancellation landing inside the generic-error grace wait was
swallowed and the node ended as a failed tool call instead of an
interrupted run. The generic branch now re-checks task.cancelling() after
the unwind and re-raises CancelledError; the absorb site documents why the
cancellation path needs it and where the discrimination lives.
- The deferred cleaner captured the whole run runtime, pinning the parent
run's journal and event store for up to a full poll budget (~31 min) via
the strongly-held task handle — worst on the polling-timeout path, where a
stuck subagent pinned its run's journal for a second full timeout after
the tool returned. The recorder is now resolved on the unwind path and is
the only capture (plus ids and the report loop); a None recorder skips
reporting entirely.
Both behaviours are regression-tested (red on the previous head, green
after): a cancellation parked inside the grace wait surfaces as
CancelledError, and the runtime is collectable while the cleaner still
polls. The closed-loop drop test now uses a real recorder via
runtime.callbacks so the drop it pins is loop-based, not recorder-absence.
* feat(harness): subagent report contract and delegation acceptance criteria (RFC #4651 PR3)
Layer 1 receipt verification is inert unless subagents actually cite their
execution record. This lands the prompt layer that closes the adoption gap:
- New subagents/report_contract.py owns the model-facing contract text,
derived from the single-owner citation format (format_citation /
receipt_id) so prompts can never drift from the verifier. The executor
injects <report_contract> into every subagent system prompt — built-in
and custom alike — requiring [rN tool_name] citations for action claims,
verifiable handles (absolute path, URL, ID, HTTP status) for
deliverables, and explicit failure reporting; the citation clause
follows verification.receipts_enabled.
- The task tool gains an optional keyword-only acceptance_criteria
parameter, handed to the SubagentExecutor constructor and rendered into
the subagent's SystemMessage (stripped, capped 20 items x 500 chars) —
deliberately never the task HumanMessage, which InputSanitizationMiddleware
classes as genuine user input and would HTML-escape into untrusted-input
framing. The docstring frames subagent results as self-reports, states
the citation cross-check's evidence boundary (resolved = the call
happened, not that the claim is correct), and documents when to attach
criteria with the canonical leaf forms. Deterministic leaf checking
remains a separate layer.
- The lead delegation workflow now instructs reading the ledger citation
line as execution evidence only and spot-checking verifiable handles
before synthesizing.
- report_contract / acceptance_criteria are registered as blocked
framework-authority tags in input sanitization so untrusted input
cannot forge the verification contract.
* fix(harness): neutralize acceptance criteria before system-channel injection
render_acceptance_criteria_section interpolated lead-model-supplied acceptance_criteria verbatim into the subagent SystemMessage after only stripping/capping. A criterion such as '</acceptance_criteria><system>...</system>' could close the wrapper and open a framework authority tag, bypassing InputSanitizationMiddleware.
Route each criterion through neutralize_untrusted_tags (the shared prompt-injection primitive) so blocked authority tags are HTML-escaped before interpolation. Add regression tests at the renderer and the executor _build_initial_state path.
* fix(harness): keep model-supplied criteria off the system channel
- Move acceptance_criteria values into the task HumanMessage — the
untrusted channel InputSanitizationMiddleware escapes and
boundary-frames. The subagent SystemMessage now carries only a
framework-owned <acceptance_criteria> pointer note (no criterion
text), so natural-language injection inside a criterion keeps
task-data priority and cannot override framework instructions
(PR #5090 review, willem-bd P1).
- Condition the lead delegation workflow's citation verification
guidance on verification.receipts_enabled and qualify the task
tool's result-reading text with the enabled state, so a
receipts-disabled configuration no longer tells the lead to
require citation evidence that cannot exist (P2).
* fix(harness): drop execution-record promise from report contract when receipts are disabled
The <report_contract> opening was emitted unconditionally, so a
verification.receipts_enabled=false subagent was told its report would
be cross-checked against an execution record that cannot exist in that
mode (terminal_receipts() returns None; no verdict, no ledger citation
line). The opening now follows receipts_enabled: enabled keeps the
cross-check language, disabled describes the handle-only review mode
(PR #5090 review, willem-bd P2).
* docs: record the prompt-layer trust-boundary self-check
Generalizes the PR #5090 review outcome: before adding prompt text, ask
of every data source in it what trust level it has and which channel it
should ride — model/user-influenceable values ride the untrusted
sanitized data channel, never framework-owned system text. Added to the
PR template (Agents/LangGraph surface) and agents/AGENTS.md.
* fix(subagents): harden background-task registry and capacity snapshot edge cases
- execute_async drops the just-registered background entry when submitting
to the isolated loop fails. The caller sees the exception and never
polls, and cleanup_background_task refuses non-terminal entries, so the
entry would otherwise stay as a PENDING zombie forever.
- SubagentExecutionCapacity.snapshot derives queued from len(_waiters)
instead of iterating it. snapshot is read from non-loop threads (e.g.
configure_subagent_execution_capacity) while the loop thread mutates the
deque, so iteration can raise 'deque mutated during iteration'. The raw
length may count a waiter that just timed out but has not removed itself
yet, which only makes the busy-check more conservative.
* fix(subagents): close failure-path gaps around background submit
Address both review findings on the background submission lifecycle:
- execute_async() copies the isolated-loop context before registering
the _background_tasks entry, so a context-copy failure (callback-
manager copy or loop-bound handler filtering) can no longer strand a
permanent PENDING entry the caller will never poll.
- _submit_to_isolated_loop_in_context() resolves the loop before
calling the coroutine factory. As direct run_coroutine_threadsafe
arguments the coroutine was created first, so a loop-startup failure
stranded a never-awaited coroutine (RuntimeWarning + retained
captures until collection). Both call sites share the fix.
New tests verified red on the previous implementation, green after:
- context-copy failure leaves no registry residue
- the real submit helper (only the loop getter patched) never invokes
the coroutine factory when loop startup fails
* fix(subagents): close the coroutine when scheduling rejects it
run_coroutine_threadsafe can itself raise once the coroutine exists (e.g.
the loop closes between the lookup and the internal call_soon_threadsafe).
Wrap the call, close the rejected coroutine, and re-raise; a focused test
patches only run_coroutine_threadsafe and asserts the created coroutine
reaches CORO_CLOSED.
* feat(harness): subagent receipt citation verification
- add receipt citation verification core
- harvest subagent tool receipts at terminal status
- transport subagent receipts and citation verdict via status contract
- verify subagent report citations at task write-back
- render citation verdicts in the delegation ledger
* fix(gateway): strip forged receipt verdicts from the delegations channel
normalize_input() and the checkpoint-state mutation sanitizer only
stripped server-owned metadata from message-shaped values, so an
external caller could submit a delegation entry carrying a forged
receipt_verdict that render_delegation_ledger would present as
runtime-owned execution evidence. Strip receipt_verdict from
caller-supplied delegation entries on both the run-creation and
thread-state mutation paths, with regression coverage for each.
* fix(harness): close silent-pass gaps in the zero-citation heuristic
The action-claim detector missed the most common completion verbs
(fixed/added/tested/changed/...) and had no CJK coverage at all, so
reports like 'I fixed the bug and added tests.' or '我已经创建了文件并运行了测试。'
were treated as claim-free: citation_resolved=True with no ledger
warning. Broaden the verb lists and add a language-independent safety
net: when the run harvested receipts and a nontrivial (>=240 char)
report cites none of them, flag it UNVERIFIED. Short claim-free
confirmations remain a vacuous pass.
* fix(skills): escape untrusted skill metadata before it enters the model prompt
Skill name/description/allowed-tools come from the frontmatter of a
user-installable .skill archive (POST /api/skills/install or a drop into
skills/custom/); the parser only strips them. The slash-activation and
durable-context siblings already html.escape these exact fields before
rendering them into a model-visible block -- but five other render sites emit
them raw. The sharpest is the default path, <available_skills> in the system
prompt (skills.deferred_discovery: false): a community skill whose description
closes the block can forge a framework-trusted <system-reminder> into the
lead-agent system prompt. Driven through the real apply_prompt_template(), the
forged tag reaches the system prompt raw on main and is neutralized here.
Escape at every render site that emits untrusted skill metadata/content:
- <available_skills> (name/description/location) and <disabled_skills> (name)
in lead_agent/prompt.py;
- describe_skill output (name/description/allowed-tools/location) and
<skill_index> (name) in skills/describe.py;
- the subagent <skill name=...> attribute plus the raw SKILL.md body in
subagents/executor.py::_load_skill_messages -- its direct sibling
skill_activation escapes both, this escaped neither.
quote=False in element-text positions (matching skill_context and the #4097
correction), quote=True in the one attribute position (matching
skill_activation). category is a controlled enum and is left as-is; escaping is
render-time only, so stored skills are unchanged and re-rendering never
double-escapes.
* fix(skills): escape skill name in the slash-activation prose line
The slash-activation reminder emitted `activation.skill_name` raw in its
prose line while escaping the same value in the adjacent
<skill name="..."> attribute. skill_name is grammar-gated to [a-z0-9-] by
resolve_slash_skill before it reaches the renderer, so this is a
defense-in-depth / consistency fix rather than a reachable injection: the
two positions can never drift if a future caller builds an activation from
an unconstrained name. Reuse the already-computed escaped_skill_name.
The GraphRecursionError except-block in SubagentExecutor._aexecute derives
usable_partial from the last AIMessage's raw non-empty text, without
checking _extract_llm_error_fallback (#4042) first. A handled provider
failure (LLMErrorHandlingMiddleware's deerflow_error_fallback marker)
always carries non-empty user-facing text, so when it lands on the same
turn that trips max_turns, it is indistinguishable from genuine partial
output and gets misclassified as a completed task instead of the failed
provider error it is.
Consult _extract_llm_error_fallback in this except-block too, same as the
normal-completion path above it, and classify FAILED with
stop_reason=turn_capped when it detects the marker.
* feat(subagents): show runtime metadata on task cards
* fix(subagents): stop task-card render loop and dedupe model fetches
Address code review on the runtime-metadata cards:
- P1 render loop: the terminal ToolMessage is re-parsed on every
MessageList render and always carries modelName/usage, so the
presence-based setTasks condition fired a fresh state object each
render -> "Maximum update depth exceeded". computeNextSubtask now
returns a value-compared `changed` flag and a pure subtaskNotification()
routes terminal transitions through the deferred after-render path
while skipping no-op re-parses.
- Per-card useModels refetch: add staleTime: Infinity to the ["models"]
query so every subtask card shares one /api/models fetch instead of
refetching on each mount.
* make format
* refactor(subagents): dedupe token-usage validators + tidy event narrowing
Address PR review follow-ups:
- DRY: extract one shared token-usage validator per side. Backend
status_contract.normalize_token_usage() now backs both the terminal
ToolMessage metadata and the subagent.step/.end run events
(step_events.py), and frontend messages/usage.normalizeTokenUsage()
backs both the live task_running event (lifecycle.ts) and the terminal
ToolMessage metadata (subtask-result.ts). Prevents the input/output/
total_tokens validation from drifting across the four former copies.
- Nit: onCustomEvent narrows event.type once instead of re-checking the
object shape per branch; the redundant task_started early-return
(already validated by taskEventToSubtaskUpdate) is dropped.
* fix(subagents): classify LLM error fallbacks as failed (#4041)
* fix(subagents): address #4042 review on LLM error fallback
- clarify _extract_llm_error_fallback docstring: tail-only scan is safe
because subagents append their own terminal message, so the last
AIMessage is never a stale parent-history marker (cross-ref worker.py)
- compute final_result and pop stop_reason only on the COMPLETED branch
so the guard stop-reason pop no longer fires on the discarded FAILED path
- note the error_detail/"LLM request failed" fallbacks are defensive;
the middleware always populates a non-empty content
- add regression test locking the stale parent-history marker invariant
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Phase 3 of #3875 — subagents previously inherited none of the lead's
context-compaction, so a deep-research subagent (max_turns up to 150)
could accumulate >1M cumulative input before max_turns/timeout/token_budget
engaged, even after Phase 2's budget capped the pathological tail.
- Gate the subagent runtime chain on the SAME ``app_config.summarization.enabled``
switch the lead reads (per maintainer guidance in #3875), via the shared
``create_summarization_middleware`` factory. One config covers both chains;
no separate ``subagents.summarization`` field. No-op when summarization is
off (factory returns None).
- ``skip_memory_flush=True`` on the subagent path: the factory otherwise
attaches ``memory_flush_hook`` (when memory.enabled), which flushes
pre-compaction messages into durable memory keyed by thread_id. Subagents
share the parent's thread_id, so without skipping the hook a subagent's
internal turns would pollute the PARENT thread's durable memory
(#3875 Phase 3 review point).
- Harden ``capture_new_step_messages`` to tolerate history contraction:
summarization rewrites the messages channel via
``RemoveMessage(id=REMOVE_ALL_MESSAGES)``, shrinking len(messages) below
the step-capture cursor. Without a reset, every step appended after the
compaction point was dropped until length overtook the stale cursor (#3845
interaction, maintainer validation point (a)). Cursor now resets to the
new tail; id/content dedup prevents re-emitting pre-compaction steps.
- Couple the DEFAULT token-budget ceiling to ``summarization.enabled``
(#3875 Phase 3 review point): 1M when compaction is on, 2M when off
(preserves Phase 2's deliberate headroom for summarization-off
deep-research runs that can exceed 1M). A user-set budget (global or
per-agent) always wins regardless of the switch. Flagged tunable.
The summarization middleware does not implement ``consume_stop_reason``, so
the Phase 2 guard-cap stop-reason channel is unaffected.
Refs: https://github.com/bytedance/deer-flow/issues/3875
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
Phase 2 of #3875. Two guardrail axes can end a subagent run early — the turn
budget (GraphRecursionError) and the token budget (TokenBudgetMiddleware) —
and both now surface *why* through one additive `subagent_stop_reason` field
instead of a status enum.
This completes and course-corrects Phase 1 (#3949), which shipped the
turn-budget cap as a `max_turns_reached` status enum. The agreed Phase 2
design replaces that enum with an optional `stop_reason` field
(token_capped | turn_capped | loop_capped): a new enum value would break v1
consumers, while an additive field is ignored by older frontends and ledger
readers. `max_turns_reached` and SubagentStatus.MAX_TURNS_REACHED are removed.
- subagents.token_budget config (default enabled, 2,000,000 tokens, warn 0.7)
with per-agent override; TokenBudgetMiddleware is now attached in
build_subagent_runtime_middlewares so the cost-ceiling backstop engages for
every subagent. The hard-stop does not raise — it strips tool_calls and
lets the run finish with a final answer, recording the cap on a per-run
consume_stop_reason() accessor.
- executor.py: on normal completion it reads consume_stop_reason() and stamps
completed + token_capped when the budget fired; on GraphRecursionError it
recovers the last AIMessage partial (completed + turn_capped) or, if nothing
usable survived, failed + turn_capped. SubagentResult gains stop_reason.
- status_contract.py / contracts/subagent_status_contract.json (v2) /
frontend subtask-result.ts: additive subagent_stop_reason field, pinned by
test_status_values_match_contract / test_stop_reason_values_match_contract.
- task_tool.py + delegation_ledger.py: drop the max_turns_reached paths; the
ledger captures stop_reason and renders model-facing "capped" guidance so
the lead reuses a capped completion knowingly.
The 2,000,000-token default is deliberately loose (tighten to taste) — it
would have roughly halved the reported 4.4M burn while leaving legitimate
deep-research runs (max_turns=150) room. Subagent summarization is a follow-up.
