mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-30 23:20:35 +00:00
65 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
NanPan
|
78150e9b30
|
fix(subagents): keep isolated-loop shutdown ownership retryable (#6316)
* fix(subagents): retain isolated-loop shutdown ownership * fix(subagents): reap stopped isolated loop before replacement * test(subagents): cover pending-loop recovery after worker exit * docs(subagents): document pending-loop recovery * fix(subagents): revalidate pending loop recovery |
||
|
Zeren Wang
|
d3e8e78f3f
|
feat(authz): middleware-declared tool path Layer 1 now covers tools declared by middlewares (#6105)
* feat(authz): extension tool authorization PR2 — middleware-declared tool path Layer 1 now covers tools declared by middlewares, which LangChain merges into the bound tool set after the host's explicit-list filter: - new agents/middlewares/tool_declarations.py: record the ordinary pass's verdict, collect declarations from the built stack, decide only the delta seeded by the first pass (a name denied for this build stays denied), narrow the stack build-locally on independent state-preserving copies (direct __dict__ write, refused up front for tools data descriptors, never mutating caller-owned instances), and verify the exact bound stack after the final normalization copy. - IsolatedMiddleware ships a state-preserving __copy__ so narrowing survives full/delta normalization and copies of copies. - TodoMiddleware degrades when write_todos is denied: no system-prompt injection, no completion/context-loss reminders, no jump_to. - Layer 2 forwards host-resolved tool provenance into AuthzRequest.context and binds the infrastructure exemption to the concrete host-created tool object instead of a name set (tool=None goes to the provider). - Wired into all four assembly paths (lead x2, native subagent with the decision offloaded via asyncio.to_thread, DeerFlowClient). Tests: new test_middleware_declared_tools.py (35) and test_layer2_tool_provenance.py (7); blocking-IO anchor pinning the subagent offload (teeth verified red->green); placement-guarantee and enforcement suites extended; executor test doubles converted for the async _create_agent. Behavior change called out in CHANGELOG: with authorization enabled and a tools policy that excludes write_todos, plan-mode builds no longer bind it (built-in RBAC default unchanged). * fix(authz): close residual gaps from review — narrow non-collectable declarations, dedupe candidates, warn on skipped pass - apply_declared_tool_view now removes non-BaseTool declarations from the bound view when authorization is enabled (they would otherwise bind via LangChain's create_tool auto-conversion with no provider decision); verify_declared_tool_view fails loudly if one survives normalization - decide_declared_tools dedupes never-submitted names so the provider sees each candidate once - subagent executor logs a warning when the provider is set but the seeded Layer-1 state is missing, instead of skipping the declaration pass silently - docs: middleware-declared tools must be BaseTool instances with usable names * fix(authz): fail loudly on non-list/tuple middleware tools declarations _iter_declared warned and returned () for a 'tools' attribute that is not a list/tuple, but LangChain's factory iterates the attribute as-is — so a set/generator/dict_values declaration bound with no Layer-1 decision. Raise DeclaredToolViewError instead, matching the module's fail-loud standard for every other un-narrowable shape. Disabled path remains a strict no-op. Docs updated to match. |
||
|
Guo-Yixin
|
4d5607229e
|
fix(subagents): preserve direct-return tool results (#6083)
* fix(subagents): preserve direct-return tool results * fix(subagents): include middleware tools and surface direct failures |
||
|
PeaceMaker-best
|
66597ae89d
|
fix(subagents): retain completed model usage on cancellation (#5910)
* fix(subagents): retain completed model usage on cancellation * docs(subagents): fit guidance within chain budget Signed-off-by: Willem Jiang <willem.jiang@gmail.com> --------- Signed-off-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Inference_
|
17f6b93bcb
|
fix(subagents): preserve native document blocks in context snapshots (#5948)
* fix(subagents): preserve native document blocks in context snapshots Signed-off-by: Inference1 <3617237115@qq.com> * fix(subagents): neutralize native document snapshot text Signed-off-by: Inference1 <3617237115@qq.com> * docs: fit subagent guidance within inherited limit --------- Signed-off-by: Inference1 <3617237115@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
李靖
|
10bd065615
|
feat(agents): persist tool artifacts as resolvable handles (#4676) (#4929)
* feat(agents): persist tool artifacts as resolvable handles (#4676) MCP and sandbox tools return file paths, URLs, and task ids that are lost when context compaction summarizes the conversation. This adds a per-thread artifact handle registry so the model can reference artifacts by short deterministic handles (art_xxxxxxxx) that survive compaction and resolve to real references at the tool-call boundary. - ThreadState.tool_artifacts: reducer channel capped at 100 entries, same-handle latest-wins, survives summarization compaction - artifact_registry: deterministic handle derivation from (thread_id, tool_call_id, call_index); extraction precedence structured_content -> file/image blocks -> conservative text scan; HTML-escaped model-context rendering with char budget - ArtifactCaptureMiddleware: before_model hook captures lightweight metadata (never content) and tracks handle consumption - ArtifactResolutionMiddleware: rewrites handles in tool-call args to real references before execution; unknown handles untouched - DurableContextMiddleware: projects the registry into the hidden durable-context data block ([available]/[consumed] status) - config: new tool_artifacts section (all switches default on) - frontend: artifact badges on tool-call steps keyed by tool_call_id Fixes #4676 * fix(agents): harden artifact registry after review (#4676) Review findings on #4929: - capture/consumption updates emitted in the same before_model call are now concatenated instead of dict-merged; dict-merging clobbered the shared tool_artifacts key and permanently dropped fresh captures - handles embed a per-result ordinal so multiple references inside one tool result never collide (two paths in one text block previously collapsed into one handle) - structured_content extraction pulls concrete refs from known keys (file/path/url/task_id/..., string or list-of-string values) instead of storing a truncated JSON preview, falls back to an untruncated whole-object data entry, and no longer skips file/image content blocks - plain-string tool results (e.g. bash/write_file output) are scanned for sandbox paths / remote URLs too, gated by detect_refs_in_text - wire max_entries through configure_tool_artifact_max_entries at assembly (clamped 10..1000) and gate detect_refs_in_text scans - resolution requires both enabled and resolve_handles_in_args, matching its docs - add real ToolCallRequest.override coverage and the spec 14.4 integration cycle (capture -> inject -> resolve -> consumption) * fix(agents): address second-round artifact registry review (#4676) Follow-up findings on #4929: - drop generic `output`/`outputs` from structured ref keys: prose under those keys no longer becomes a bogus file entry (falls back to the whole-object data entry; path-shaped refs are still caught by the free-text scan) - strip closing quotes, brackets, and backticks from detected refs so markdown- and JSON-wrapped paths resolve to clean real references - remove the process-wide cap setter: the reducer enforces only an absolute ceiling (1000) and ArtifactCaptureMiddleware applies the configured max_entries per agent when emitting updates, so agents with different configs in one process never share or flap a cap - skip extraction for tool messages whose tool_call_id already has registry entries and memoize known-empty results (bounded FIFO), so steady-state capture cost drops to the new message tail instead of a full-history rescan per model call - correct the inverted ordering claim: capture is position-independent in the wrap chain because it is a before_model hook reading state messages; fixed the assembly comment, AGENTS.md item 15, and the ordering test docstring CI parity verified locally: ruff check + format --check, full offline suite, blocking-io suite, collect-only, uv lock --check, check_agent_guidance, frontend format/lint/typecheck/build/tests. * fix(agents): sliding-window cap and URI gates for artifact registry (#4676) Third-round review findings on #4929: - max_entries is now a sliding retention window: fresh captures are always registered, and when the projection exceeds the configured cap the middleware appends a trailing {"op": "trim_to", "keep": N} directive that the reducer applies by evicting the oldest entries. The previous remaining-budget emission froze the registry at the cap, permanently dropping post-compaction artifacts — the exact failure mode #4676 targets — and silently disabled the round-2 rescan optimizations at the point threads grew longest - _track_consumption memoizes settled tool-call args in a bounded FIFO alongside the existing empty-result memo, so historical AIMessage args are not regex-rescanned every model call - file/image content-block URLs must be http(s) or path-shaped; data:/blob: URIs (MCP embedded resources carrying base64 payloads) can no longer enter thread state, tool args, or crowd out the render budget; protocol-relative //host forms are rejected as well - structured-content ref values go through the same URI-shape gate as content blocks, closing the embedded-payload hole via known keys such as url/urls - tool_artifacts.enabled now gates the whole middleware including consumption tracking via an early return in before_model Docs updated to describe the sliding window, URI gates, enabled scope, and both bounded memos. * fix(agents): final evictions and pending-aware consumption for artifact registry (#4676) Fourth-round review findings on #4929: - the sliding window interacted with the seen_call_ids skip to resurrect evicted entries every round: once an entry was trimmed, its tool_call_id left the seen set while its ToolMessage stayed in context, so the next model call re-extracted it, re-registered it as newest, and evicted something else — a never-converging rotation. The empty-result memo is now a unified _processed_results memo that records every extraction regardless of outcome, making evictions final; steady state over a saturated thread is quiescent (pinned by a multi-round test asserting no updates and zero extractions) - the consumption scan built its handle map from pre-capture state, so a call referencing a handle captured in the same before_model call missed, got settled as quiet, and its consumption mark was lost forever. _track_consumption now receives the pending capture entries and resolves against state + pending; a call settles as quiet only after every referenced handle resolved, so transient misses retry next round instead of permanently suppressing the mark AGENTS.md entry 15 updated to describe both memos and the eviction-is-final semantics. * fix(artifacts): address occurrence identity, PII and URL review findings * fix(artifacts): enforce resolved policy checks and bounded registry handling --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
842f561054
|
fix(runtime): drain agent stream close across cancellation (#5806)
* fix(runtime): drain agent stream close across cancellation * docs(runtime): condense stream teardown guidance * fix(runtime): surface stream close failures on cancellation --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yang rui
|
1da284bb3a
|
feat(prompts): add operator overlays for core system prompts (#5794)
* feat(prompts): add operator overlays for core system prompts * test(prompts): isolate SOUL forwarding test from operator config * fix(prompts): wrap complete subagent system message with overlay * fix(subagents): serialize prompt overlays in durable batches --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
e01314442c
|
fix(mcp): scope sessions and task access by thread incarnation (#5556)
* fix(mcp): scope sessions and task access by thread incarnation * fix(mcp): preserve thread incarnation in delegated subagents * fix(mcp): preserve incarnation in durable batches * fix(mcp): bind standalone graph lifecycle context * fix(studio): preserve implicit thread creation metadata --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
JasonH
|
82cf57a9c3
|
fix(utils): return empty text for content-less messages (#5563)
* fix(utils): return empty text for content-less messages message_content_to_text fell through to str(content), so a message whose content is None yielded the truthy literal "None". Two call sites already work around it with a local `or ""` and name the helper in the comment; the subagent executor's `text if text else "No response generated"` fallback and the archive's `if not text: continue` skip cannot work around it, so a content-less terminal turn was reported as an answer of "None" and a content-less LLM error fallback surfaced "None" instead of its error_detail. * test(utils): cover non-None content compatibility and document fallbacks * test(utils): cover contentless task history and refresh guard comments --------- Co-authored-by: Lengshuang <90967079+Lesereingrape@users.noreply.github.com> Co-authored-by: JasonH <4430962+yang0228@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
94110e5dce
|
fix(subagents): close stream before releasing resources (#5221)
Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> |
||
|
Hyeonsang Cho
|
53798b44cd
|
fix(subagents): scale max_turns into the graph's super-step budget (#5485)
* fix(subagents): scale max_turns into the graph's super-step budget
max_turns was handed to LangGraph as recursion_limit, but the two count
different things. recursion_limit counts super-steps, one per graph node,
and create_agent compiles a node for every middleware lifecycle hook, so
one turn costs before_model + model + after_model + tools nodes — seven to
eight through the subagent chain. The built-in general-purpose agent's
max_turns=150 therefore bought about 18 tool-using turns before failing as
turn_capped, and every middleware added to the chain shrank the effective
budget again.
Resolve the limit from the chain each subagent was actually assembled with
(subagents/turn_budget.py) instead of passing the turn count through, so
raising max_turns buys the turns it names.
No config keys or defaults changed; existing max_turns values now grant
their full budget, bounded as before by subagents.timeout_seconds and
subagents.token_budget.
* fix(subagents): warn when a counted hook can jump the agent loop
Review follow-up. The per-turn cost is a flat multiplier over the straight
before_model -> model -> tools loop. A hook that declares can_jump_to and
returns {"jump_to": ...} re-enters the loop without traversing tools,
spending another before_model + model + after_model pass that buys no tool
result, so the resolved limit becomes a lower bound rather than an exact
budget — silently re-creating the short budget this translation fixes.
Measured against a compiled graph: with one jumping after_model hook,
three tool turns need the resolved limit plus one jump pass, and the run
raises GraphRecursionError at the resolved limit.
How often a jump fires is data-dependent and unbounded, so it cannot be
folded into the arithmetic. find_jumping_hooks reports the condition off
the same __can_jump_to__ attribute the factory reads, and the executor
warns when a counted hook declares one. Nothing in today's subagent chain
does, so this changes no budget.
* fix(subagents): detect jumps declared on agent-level hooks too
Review follow-up. find_jumping_hooks exempted before_agent/after_agent on
the grounds that a jump out of them lands in the loop the budget already
pays for. That does not hold on langchain 1.3.14:
- after_agent jumps re-enter the loop after it finished, and the hook runs
again on the next exit, so the extra passes are unbounded. Even
jump_to "end" is routed to exit_node, the head of the after_agent chain,
so it reruns the chain; destinations are no safe filter.
- a before_agent hook that stages a tool call and jumps to tools runs a
tools step no model turn paid for. It is O(1), but the resolved limit
has zero headroom, so one step caps the last turn.
The detector now scans every hook pair the factory wires jump edges for.
The compiled-graph pin is parametrized over after_model->model,
after_agent->model, after_agent->end and before_agent->tools, each raising
GraphRecursionError at the resolved limit and completing once the jump's
cost is added. No middleware in the subagent chain declares a jump on any
hook, so this changes no budget.
|
||
|
PeaceMaker-best
|
14c9d44440
|
feat(runtime): persist tool-progress phase transitions (#5214)
* feat(runtime): persist tool-progress phase transitions Record bounded warn, block, and recover decisions for lead and task subagent runs while preserving event-loop isolation, fail-open behavior, and concurrent transition order. * fix(runtime): trust server-owned tool progress attribution * fix(runtime): centralize trusted audit attribution * fix(runtime): preserve complete tool progress audit state * docs: trim tool progress guidance to pass size check * fix(runtime): fence subagent audit recorder loop * docs(readme): sync tool-progress event coverage Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: 嗜鵼 <hy2010hy2010@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wenchao An
|
6d5d7bb1d5
|
feat(subagents): add opt-in parent context snapshots (#5367)
* feat(subagents): add opt-in parent context snapshots * test(subagents): package synthetic snapshot evaluation * fix(subagents): preserve output text and defer snapshot capture * docs(subagents): keep snapshot guidance within chain budget * fix(subagents): omit unpaired tool calls from snapshots * fix(subagents): safely omit unserializable snapshot media |
||
|
Parthiban Sivakumar
|
f5e51b1d4f
|
fix(tests): await future completion before asserting done() (#5299)
* fix(tests): await future completion before asserting done()
`test_run_on_isolated_subagent_loop_survives_caller_loop_teardown`
signals from inside the coroutine:
async def deferred_work() -> None:
completed.set()
`run_on_isolated_subagent_loop` is `asyncio.run_coroutine_threadsafe`,
whose `concurrent.futures.Future` is marked done by the loop only after
the coroutine returns. The main thread can therefore wake from
`completed.wait()` while the future is still pending, and
`assert handles[0].done()` fails:
assert False
+ where False = done()
+ where done = <Future at 0x7f62241b22d0 state=pending>.done
Observed on main at a2808e82 (shard 2) and on an unrelated PR at
852a94dd (shard 4) sixteen seconds apart, so it tracks runner load
rather than any change under test.
Assert the result first — `Future.result(timeout=10)` blocks until the
future completes — then assert `done()`. Both assertions keep their
original meaning and no sleep is introduced.
Fixes #5298
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test: drop the now-redundant done() assertion
Review follow-up. Once `result(timeout=10)` has returned normally the
future is guaranteed to be FINISHED, so the `done()` assertion below it
could no longer fail — it documented intent rather than checking
anything.
`result()` alone proves both halves of what the test is about: that the
coroutine body ran after caller-loop teardown, and that the future
resolved. The `completed.wait()` guard above still covers the "work
never ran" case with a descriptive message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
theater
|
18b34298cc
|
fix(tests): preload sandbox leaf modules before mocking their parent package (#5215)
TestBashExecutionHarvest's shell-persistence tests report
shell_persistent=None whenever deerflow.sandbox.overwrite is not already
cached in sys.modules. _setup_executor_classes replaces the
"deerflow.sandbox" parent package with a MagicMock, so a later
`from deerflow.sandbox.overwrite import unwrap_sandbox` inside
_harvest_shell_persistence can no longer locate the submodule through
the mocked parent ("'deerflow.sandbox' is not a package") when the leaf
module is not already in sys.modules. The helper's
`except Exception: return None` silently converts that ImportError into
an UNKNOWN provenance stamp.
Whether the leaf module was cached depended on which tests ran earlier
in the session, making the outcome order-dependent: green in CI by
collection-order luck, red when the module runs alone or first.
Fix it the same way the fixture already handles audit_context and
tool_search: preload the real leaf modules (deerflow.sandbox.sandbox_provider
and deerflow.sandbox.overwrite) before installing the mocked parent
package, pin them in sys.modules for the duration of the test, and
restore the previous state afterwards.
|
||
|
PeaceMaker-best
|
3c36217a51
|
feat(observability): persist deferred tool promotions (#5183)
* feat(observability): persist deferred tool promotions Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(ci): trim agent guidance chain Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
PeaceMaker-best
|
fb28ed0122
|
feat(subagents): enable historical upload discovery (#5170)
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
Michael
|
85ffb66d6e
|
fix(subagents): stamp UTC-aware datetimes on SubagentResult lifecycle (#5153)
## Why DeerFlow declares one timestamp convention in deerflow/utils/time.py: every lifecycle timestamp is UTC (now_iso / datetime.now(UTC)). SubagentResult writers in subagents/executor.py still used naive datetime.now(), so on any non-UTC host the in-memory lifecycle metadata (started_at / completed_at) was local wall-clock time. The sibling durable-batch path (subagents/batch_service.py) already stamps datetime.now(UTC), so the same run model carried two different conventions depending on which path wrote it. ## What changed - Added an executor-local _utcnow() helper that stamps datetime.now(UTC). - SubagentResult.completed_at default in try_set_terminal(), result.started_at in _aexecute(), and the started_at default in _aexecute_admitted() now route through _utcnow(). - Explicit caller-supplied timestamps (completed_at=...) still pass through unchanged. - Added regression tests asserting the default writers produce UTC-aware datetimes. ## Surface area - [x] Backend runtime (deerflow.subagents.executor) - internal dataclass lifecycle metadata, no wire format change - [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies / Default behavior change ## Bug fix verification - New tests: tests/test_subagent_executor.py::test_timestamp_writers_stamp_utc_aware_datetimes and test_utcnow_helper_returns_utc_aware_datetime encode the convention. - Updated BlockingDateTime.now() in the terminal-publication-order test to mirror datetime.now's optional tz argument. ## Validation - cd backend && python -m pytest tests/test_subagent_executor.py: 136 passed; 2 pre-existing TestBashExecutionHarvest failures reproduce identically on clean main (Windows sandbox env), unrelated to this change. - ruff format + ruff check clean on both changed files. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis of the timestamp conventions, implementation, and regression tests authored with AI assistance; change reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. |
||
|
Aari
|
9e0fbd60fa
|
fix(sandbox): isolate concurrent subagent shell sessions (#5134)
* fix(sandbox): isolate concurrent subagent shell sessions * fix(sandbox): make execution acquire idempotent * fix(sandbox): close execution lifecycle gaps * fix(sandbox): serialize retained client lifecycle * fix(sandbox): close remaining client lifecycle gaps * fix(sandbox): unwind failed client lookup * fix(sandbox): protect internal lease identities * fix(sandbox): make cancellation reconciliation durable * fix(sandbox): fence cancelled workers and IM uploads |
||
|
spud
|
fe379c4486
|
feat(ci): split backend unit tests into parallel shards (#5137)
* feat(ci): split backend unit tests into parallel CI shards Split the single offline backend `make test` job into four GitHub Actions matrix shards (SPLITS=4, GROUP=1..4) via pytest-split, so the ~12k-test suite runs in parallel instead of in one 15-minute job. Each shard runs on its own runner with its own Postgres/Redis services; fail-fast: false lets a failing shard report its owned tests without cancelling its peers. `make test` stays the canonical full-suite entry point; CI now calls the new `make test-shard SPLITS=4 GROUP=N`. tests/blocking_io remains owned solely by the dedicated blocking-I/O workflow (excluded via --ignore), extending #5105. Fixes #5088 * test(ci): make backend test shards duration-aware and pin the contract Make `make test-shard` an explicit least_duration split that READS backend/.test_durations (read-only for shards, so concurrent CI jobs never race writes on it), and add `make test-shard-durations` to regenerate that file from the full offline suite. Update the CI unit-test workflow contract to call `make test-shard SPLITS=4 GROUP=<n>` and assert the shard command carries --splits 4, --group 2, -m "not live", --ignore=tests/blocking_io and --splitting-algorithm least_duration. Verified on the real 13,140-test normal suite that the four shards are pairwise disjoint and their union equals the unsplit suite. Refs #5088 * test(ci): fail fast when the duration baseline is missing `make test-shard` now requires backend/.test_durations and exits with a clear error instead of letting pytest-split silently degrade to an even (count-based) split. Harden the CI contract test to pin `--durations-path=.test_durations` and to assert the repo ships the committed duration baseline. Refs #5088 * docs: trim backend/AGENTS.md within guidance budget * test(ci): add backend test duration baseline Add the duration baseline generated by a full offline backend run on a GitHub-hosted ubuntu-latest runner (the same runner type the shards use), so `make test-shard` balances the four matrix shards by real wall-clock cost. Refs #5088 * test(ci): make the duration writer honor DURATIONS_FILE `test-shard-durations` now writes `--durations-path=$(DURATIONS_FILE)` instead of a hard-coded .test_durations, so the reader and writer stay consistent when the path is overridden. Refs #5088 * test(ci): address sharding review feedback * test: isolate subagent execution capacity state --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
AoHanBei
|
9b32b5d841
|
feat(observability): persist loop detection events (#5127)
* feat(observability): persist loop detection events * fix(observability): persist subagent loop events * fix(observability): narrow subagent loop event bridge * fix(observability): attribute subagent loop events * fix(tests): isolate subagent executor imports |
||
|
Hyeonsang Cho
|
9146bfa03d
|
feature(gateway): issue request trace ids unconditionally (#5119)
* refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
a06a6fed7e
|
feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2) (#5109)
* feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2) PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a subagent completes, so objectively checkable requirements can never be silently passed by a self-report. - subagents/acceptance_checks.py: deterministic leaf families — file:<path> exists|non-empty and file_written:<path> read through read_current_file_content scoped to the shared thread workspace; the read uses the sandbox-native virtual path form (the local read validator and provider mount tables resolve /mnt/user-data/... paths, not host paths); the scope decision canonicalizes with realpath on the local sandbox so workspace symlinks cannot escape into uploads; a remote provider's "Error: ..." return string is normalized to a failed check (provider-typed via is_local_sandbox); a UnicodeDecodeError marks a binary deliverable as existing and non-empty; out-of-scope paths degrade to UNVERIFIED. tests_passed:<command> anchors to a matching recorded bash execution with status=success and a test-summary shape; matching is shell-structure aware with control-flow attribution (span must end at the last segment with provable execution), negating-option values are ineligible evidence and a target negated anywhere in the command degrades the match, extra flags must be selection-preserving, extra positionals widen only after a path-scoped criterion, truncated commands degrade via command_truncated, the summary shape is read only from output attributable to the matched segment (preceding segments provably silent by invocation form), and pass shapes require a nonzero passed count. Criterion text is neutralized with neutralize_untrusted_tags before storage/rendering. Anything else renders UNVERIFIED, never silently passed. - executor: accumulate bounded bash command/output evidence per streamed chunk (merged by tool_call_id, newest-capped) so subagent summarization compacting earlier messages cannot erase a recorded execution; the recorded status is the actual shell exit status parsed from the output's exit marker (signed codes included; the remote Command exited with code N form is accepted only as the whole trimmed output), falling back to deerflow_tool_meta only when no marker exists. - sandbox providers: e2b/opensandbox/tenki/boxlite append the LocalSandbox-style "Exit Code: N" marker on nonzero exit even with non-empty output; aio propagates the SDK's structured exit_code on both exec paths the same way; local timeouts append Exit Code: 124; and _truncate_bash_output always preserves a trailing exit marker (signed included) inside its budget, with a 32-char floor raising any smaller configured limit, so the actual shell outcome always survives in the output text. - task_tool: run the checklist offloaded (asyncio.to_thread) on the completed branch, failure-isolated; stamp the verdict into result metadata and render the per-criterion section into the model-visible result text. - status contract: additive subagent_acceptance_verdict transport with read-side structural validation. - delegation ledger: entry carries the verdict and renders a compact acceptance segment; gateway strips caller-forged verdicts from both ledger entries and message metadata, like the citation verdict. - blocking-IO anchor pins the offload (teeth proven red->green); leaf read errors catch only OSError/SandboxError so unexpected errors reach the task-tool-level isolation instead of being mislabeled. * fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4) - negating options: overlap with a matched criterion target is now checked by path/nodeid prefix, not exact token equality — excluding a sub-path of the criterion's selection (pytest tests --deselect tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds - output attribution: any redirection token in the matched final segment makes the recorded tail non-attributable (> / >> / 2> are word characters to the parser, so redirection was invisible to the matcher) - silent-source allowlist narrowed from any *activate suffix to the */bin/activate shape - status_contract docstring: restore the shared-fixture sentence and note subagent_acceptance_verdict is deliberately outside the fixture - executor: update_bash_executions publishes [] (stream carried no bash-family calls) instead of collapsing it into None, mirroring update_tool_receipts * fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4) - tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error" and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass shape when the exit status is swallowed (|| true) or absent; zero-error counts stay clean. - file leaves: bound the deliverable read — a "wc -c" shell size probe answers files above 50k bytes without loading ~2x their size, honoring the host-bash kill switch and falling back to the full read on any non-integer rendering, so verdicts never get less sound. - executor: record the exit marker text as status_marker on harvested bash evidence; the leaf detail now reports the marker actually seen instead of asserting a failure indistinguishable from the command's own trailing text. - extend the blocking-IO anchor to drive the probe branch inside the offload; teeth re-verified red->green. * fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4) - file leaves: never read unbounded — size is established first (os.stat on the validated local host path, so the host-bash-disabled configuration needs no shell; a guarded wc -c on remote providers that renders missing/unreadable in its own words). Above the 50k cap the leaf answers from the size alone, at/below it the full read runs, and an unestablishable size degrades to UNVERIFIED instead of an unlimited fallback read. - output attribution: source/. prefixes are never provably silent — a crafted */bin/activate path shape says nothing about what the script prints, so sourced segments can no longer lend a passing summary. - executable identity: an explicitly path-spelled criterion now requires the same normalized executable path; the basename rule stays only for deliberately bare criterion commands. * fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4) - remote probe no longer runs in the sandbox's persistent shell: a fresh env -i /bin/sh with absolute-path stat/realpath (poisoned functions, aliases, PATH, exported functions, IFS, locale cannot steer it), plus a marker env routing AIO onto a fresh per-call bash.exec session. - metadata-only: stat never opens content, so a FIFO deliverable cannot block the parent for the provider's idle timeout; non-regular files (fifo/dir/symlink) degrade to UNVERIFIED. - containment canonicalized against the literal mount root: a final-component symlink or a swapped parent directory (root included) cannot redirect the check outside shared storage; unprovable layouts degrade to UNVERIFIED. * fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4) Literal-root equality made every remote file leaf permanently UNVERIFIED on e2b and Tenki, which realize /mnt/user-data as a symlink to the home dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink). Containment now compares the file's realpath against the mount root's realpath — exactly what the provider's own read path resolves, so probe and read-back stay consistent; final-component symlinks stay rejected by the non-dereferencing stat, and an intermediate dir-link escape under a sane root still lands ESCAPED. The inner script is a module constant and the suite now executes the composed probe for real against on-disk layouts (real dir, symlinked prefix, final symlink, fifo, missing, dir-link escape), which the canned-output stub could not see. * fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4) - matching: a criterion with no positional selection target (bare pytest, make test) stands for the runner's default selection, so ANY negating option (--ignore/--deselect/...) makes the recorded run a different selection — unprovable. The overlap guard only sees consumed criterion tokens, which a bare criterion does not have; scoped criteria keep the unrelated-exclusion behavior. - attribution: cd is no longer blanket-silent — CDPATH makes cd print the resolved (subagent-chosen) destination and the pass shapes match as substrings, so one mkdir 'all tests passed' plus an export minted a pass for any quiet command. A cd argument or CDPATH= value (export or leading assignment) carrying any summary shape makes the segment non-silent; shape-free cd dir wrappers keep matching. - docs: _truncate_bash_output states the effective 32-char floor (the guarantee previously read as an unconditional max_chars bound). * fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4) Self-audit in the shape of the last review rounds — channels the matcher classified as accounted-for that can change what runs, narrow the selection, or lend the summary text: - env assignments are no longer blanket-stripped: only an allowlist of inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a matched span, and a non-allowlisted assignment in any preceding segment (pure-assignment or export NAME=) is state pollution — PATH redirects the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code, PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs, BASH_ENV runs arbitrary shell startup. All degrade to unprovable. - runtime expansions: any span token carrying /$( )/backticks, any negating-option value carrying an expansion or glob (unknown excluded set), and any extra executed token carrying glob metacharacters (crafted option-looking filenames narrow invisibly) are unprovable. Criterion-side globs stay self-consistent (literal match). - cd: an argument carrying a runtime expansion or glob is non-silent (unknown destination, unknown print); CDPATH= assignments are now handled as state pollution at the match layer, subsuming the value-shape special case. * fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4) - tests_passed: on a persistent-shell provider (new Sandbox.persistent_shell_sessions capability, set by AioSandbox) every leaf degrades to UNVERIFIED — any earlier call in the shared session could have mutated the state the clean-looking run executed in, and only a fresh controlled session (RFC section 6 verifier) can prove otherwise. The flag is read from the provider registry without acquiring a sandbox. - env assignments: the allowlist is gone — no variable is provably inert across repositories (CI/DEBUG are routinely read by tests). The span's assignment prefix must equal the criterion's exactly (values included, order-insensitive); any assignment or export NAME= in a preceding segment is state pollution. - scoping: positional targets are now read by option arity, so a path embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never counts as a selection target and an extra positional after such a criterion narrows the default selection it denotes. * fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4) * fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4) * fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4) * fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4) * fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4) * fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4) * fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4) * fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4) * fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4) |
||
|
hataa
|
17cac3420a
|
fix(subagents): clean up background task entry on unexpected poller exit (#5069)
* fix(subagents): clean up background task entry on unexpected poller exit * fix(subagents): pin deferred cleanup to the persistent subagent loop The non-terminal fallback scheduled the deferred registry cleanup with asyncio.create_task on the poller's own loop. Under synchronous tool invocation the sync wrapper runs the tool coroutine through asyncio.run(), which cancels caller-loop tasks at teardown, so the cleanup died before executing and the _background_tasks entry leaked — the same lifecycle leak the terminal path already fixed. Schedule the deferred cleaner on the process-owned persistent subagent loop instead, via the new public executor helper run_on_isolated_subagent_loop (asyncio.run_coroutine_threadsafe). The cleaner only touches thread-safe registry helpers, so it is loop-agnostic. A caller-loop fallback remains for the unreachable case where the persistent loop cannot be obtained, so scheduling never raises out of an unwind path that is already handling an error. The polling-timeout return path, which shares the scheduler, is fixed the same way. Tests no longer stub the scheduler: the non-terminal fallback test drives the real scheduling wrapper on an equivalent long-lived loop and asserts cleanup runs after asyncio.run() tears the caller loop down, and run_on_isolated_subagent_loop itself is covered against the real persistent loop with the caller loop closed underneath. * fix(subagents): harden interrupted finalization against failing status path Three edge cases from review on the unexpected-exit unwind: 1. Finalization no longer depends on the failing status accessor. _peek_subagent_result distinguishes a gone entry from an unreadable one instead of letting the accessor's exception abort the unwind; _finalize_interrupted_subagent never raises (so the original poller exception is preserved) and attaches the deferred cleaner, whose last resort force-removes a persistently unreadable entry via the new executor force_cleanup_background_task. 2. The generic-error unwind waits only a short grace period (_UNEXPECTED_EXIT_GRACE_SECONDS) instead of the full execution timeout before re-raising; the remaining lifecycle stays with the deferred cleaner on the persistent subagent loop. 3. The deferred cleaner reports the subagent's final usage (deltas since the unwind snapshot included, via final=True bypassing usage_reported; the journal dedupes by source_run_id) before removing the terminal entry. The report is transferred in a plain worker thread so the RunJournal's loop-bound progress flush is skipped rather than scheduled on a foreign loop. * fix(subagents): pin deferred final usage delivery to the parent run loop _report_deferred_final_usage ran record_external_llm_usage_records in a to_thread worker, making it the first cross-thread RunJournal writer: the unlocked accumulators can lose token updates and _tokens_by_model mutations race get_completion_data() iteration on the parent loop. Capture the parent loop at unwind time (it is alive in every path that continues the run) and deliver the final report onto it with call_soon_threadsafe, serialized with all other journal access; when that loop is already closed (asyncio.run teardown) the report is dropped on purpose — the run has persisted and nothing reads the counters back. The live-loop test exercises the real recorder path (journal captures the running loop of every call) instead of stubbing _report_subagent_usage, so a cross-thread report would surface as a wrong-loop entry. Also gates the two teardown tests on the caller loop actually closing (the deferred cleaner could otherwise legitimately deliver while that loop is still winding down), and adds a task_tool-level regression test for execute_async submit failure leaving no registry residue (rolled back inside execute_async since #5086). * docs(subagents): document the reverse loop boundary; observability + test fixes Extend subagents/AGENTS.md's Isolated-loop callback boundary with the reverse-direction contract from #5069: deferred registry cleanup is pinned to the persistent subagent loop via run_on_isolated_subagent_loop, and the final usage report is handed back onto the parent run's loop captured at unwind time — never invoked from the persistent loop or a worker thread, which would silently reintroduce the journal accumulator/iteration race. Dropping the final report (closed parent loop) is the one path where a subagent's tail usage goes permanently unaccounted, so both drop branches now log at info with the execution id and the unaccounted record count. Restore the retention assert in test_deferred_cleanup_task_retained_and_ survives_gc: the bounded wait observes the production done-callback discard, the assert (not a manual discard) is what fails if that callback is deleted. * fix(subagents): honour grace-wait cancellation and shrink deferred-cleaner captures Two review follow-ups on the unwind: - The shared unwind absorbs CancelledError (never-raise contract), so a graph-node cancellation landing inside the generic-error grace wait was swallowed and the node ended as a failed tool call instead of an interrupted run. The generic branch now re-checks task.cancelling() after the unwind and re-raises CancelledError; the absorb site documents why the cancellation path needs it and where the discrimination lives. - The deferred cleaner captured the whole run runtime, pinning the parent run's journal and event store for up to a full poll budget (~31 min) via the strongly-held task handle — worst on the polling-timeout path, where a stuck subagent pinned its run's journal for a second full timeout after the tool returned. The recorder is now resolved on the unwind path and is the only capture (plus ids and the report loop); a None recorder skips reporting entirely. Both behaviours are regression-tested (red on the previous head, green after): a cancellation parked inside the grace wait surfaces as CancelledError, and the runtime is collectable while the cleaner still polls. The closed-loop drop test now uses a real recorder via runtime.callbacks so the drop it pins is loop-based, not recorder-absence. |
||
|
Zeren Wang
|
22b0456e45
|
feat(harness): subagent report contract and delegation acceptance criteria (#5090)
* feat(harness): subagent report contract and delegation acceptance criteria (RFC #4651 PR3) Layer 1 receipt verification is inert unless subagents actually cite their execution record. This lands the prompt layer that closes the adoption gap: - New subagents/report_contract.py owns the model-facing contract text, derived from the single-owner citation format (format_citation / receipt_id) so prompts can never drift from the verifier. The executor injects <report_contract> into every subagent system prompt — built-in and custom alike — requiring [rN tool_name] citations for action claims, verifiable handles (absolute path, URL, ID, HTTP status) for deliverables, and explicit failure reporting; the citation clause follows verification.receipts_enabled. - The task tool gains an optional keyword-only acceptance_criteria parameter, handed to the SubagentExecutor constructor and rendered into the subagent's SystemMessage (stripped, capped 20 items x 500 chars) — deliberately never the task HumanMessage, which InputSanitizationMiddleware classes as genuine user input and would HTML-escape into untrusted-input framing. The docstring frames subagent results as self-reports, states the citation cross-check's evidence boundary (resolved = the call happened, not that the claim is correct), and documents when to attach criteria with the canonical leaf forms. Deterministic leaf checking remains a separate layer. - The lead delegation workflow now instructs reading the ledger citation line as execution evidence only and spot-checking verifiable handles before synthesizing. - report_contract / acceptance_criteria are registered as blocked framework-authority tags in input sanitization so untrusted input cannot forge the verification contract. * fix(harness): neutralize acceptance criteria before system-channel injection render_acceptance_criteria_section interpolated lead-model-supplied acceptance_criteria verbatim into the subagent SystemMessage after only stripping/capping. A criterion such as '</acceptance_criteria><system>...</system>' could close the wrapper and open a framework authority tag, bypassing InputSanitizationMiddleware. Route each criterion through neutralize_untrusted_tags (the shared prompt-injection primitive) so blocked authority tags are HTML-escaped before interpolation. Add regression tests at the renderer and the executor _build_initial_state path. * fix(harness): keep model-supplied criteria off the system channel - Move acceptance_criteria values into the task HumanMessage — the untrusted channel InputSanitizationMiddleware escapes and boundary-frames. The subagent SystemMessage now carries only a framework-owned <acceptance_criteria> pointer note (no criterion text), so natural-language injection inside a criterion keeps task-data priority and cannot override framework instructions (PR #5090 review, willem-bd P1). - Condition the lead delegation workflow's citation verification guidance on verification.receipts_enabled and qualify the task tool's result-reading text with the enabled state, so a receipts-disabled configuration no longer tells the lead to require citation evidence that cannot exist (P2). * fix(harness): drop execution-record promise from report contract when receipts are disabled The <report_contract> opening was emitted unconditionally, so a verification.receipts_enabled=false subagent was told its report would be cross-checked against an execution record that cannot exist in that mode (terminal_receipts() returns None; no verdict, no ledger citation line). The opening now follows receipts_enabled: enabled keeps the cross-check language, disabled describes the handle-only review mode (PR #5090 review, willem-bd P2). * docs: record the prompt-layer trust-boundary self-check Generalizes the PR #5090 review outcome: before adding prompt text, ask of every data source in it what trust level it has and which channel it should ride — model/user-influenceable values ride the untrusted sanitized data channel, never framework-owned system text. Added to the PR template (Agents/LangGraph surface) and agents/AGENTS.md. |
||
|
hataa
|
2f8d1cfc21
|
fix(subagents): harden background-task registry and capacity snapshot edge cases (#5086)
* fix(subagents): harden background-task registry and capacity snapshot edge cases - execute_async drops the just-registered background entry when submitting to the isolated loop fails. The caller sees the exception and never polls, and cleanup_background_task refuses non-terminal entries, so the entry would otherwise stay as a PENDING zombie forever. - SubagentExecutionCapacity.snapshot derives queued from len(_waiters) instead of iterating it. snapshot is read from non-loop threads (e.g. configure_subagent_execution_capacity) while the loop thread mutates the deque, so iteration can raise 'deque mutated during iteration'. The raw length may count a waiter that just timed out but has not removed itself yet, which only makes the busy-check more conservative. * fix(subagents): close failure-path gaps around background submit Address both review findings on the background submission lifecycle: - execute_async() copies the isolated-loop context before registering the _background_tasks entry, so a context-copy failure (callback- manager copy or loop-bound handler filtering) can no longer strand a permanent PENDING entry the caller will never poll. - _submit_to_isolated_loop_in_context() resolves the loop before calling the coroutine factory. As direct run_coroutine_threadsafe arguments the coroutine was created first, so a loop-startup failure stranded a never-awaited coroutine (RuntimeWarning + retained captures until collection). Both call sites share the fix. New tests verified red on the previous implementation, green after: - context-copy failure leaves no registry residue - the real submit helper (only the loop getter patched) never invokes the coroutine factory when loop startup fails * fix(subagents): close the coroutine when scheduling rejects it run_coroutine_threadsafe can itself raise once the coroutine exists (e.g. the loop closes between the lookup and the internal call_soon_threadsafe). Wrap the call, close the rejected coroutine, and re-raise; a focused test patches only run_coroutine_threadsafe and asserts the created coroutine reaches CORO_CLOSED. |
||
|
Zeren Wang
|
3b592c2053
|
feat(harness): subagent receipt citation verification (#5076)
* feat(harness): subagent receipt citation verification - add receipt citation verification core - harvest subagent tool receipts at terminal status - transport subagent receipts and citation verdict via status contract - verify subagent report citations at task write-back - render citation verdicts in the delegation ledger * fix(gateway): strip forged receipt verdicts from the delegations channel normalize_input() and the checkpoint-state mutation sanitizer only stripped server-owned metadata from message-shaped values, so an external caller could submit a delegation entry carrying a forged receipt_verdict that render_delegation_ledger would present as runtime-owned execution evidence. Strip receipt_verdict from caller-supplied delegation entries on both the run-creation and thread-state mutation paths, with regression coverage for each. * fix(harness): close silent-pass gaps in the zero-citation heuristic The action-claim detector missed the most common completion verbs (fixed/added/tested/changed/...) and had no CJK coverage at all, so reports like 'I fixed the bug and added tests.' or '我已经创建了文件并运行了测试。' were treated as claim-free: citation_resolved=True with no ledger warning. Broaden the verb lists and add a language-independent safety net: when the run harvested receipts and a nontrivial (>=240 char) report cites none of them, flag it UNVERIFIED. Short claim-free confirmations remain a vacuous pass. |
||
|
Aari
|
ff0a6768c2
|
feat(subagents): add unified capacity and durable batch execution (#4998)
* feat(subagents): add capacity controls and durable batches * fix(helm): sync subagent config schema version * fix(subagents): preserve batch history without worker * fix(subagents): support explicit factory runtimes * fix: address durable batch review findings |
||
|
ChiHaYa
|
88252e9b31
|
fix(subagents): isolate background tasks from reused tool call IDs (#4758)
* fix(subagents): isolate background execution IDs * fix(subagents): preserve correlation scope and isolate usage * fix(subagents): make usage attribution idempotent |
||
|
Aari
|
a5059b8284
|
fix(subagents): isolate callbacks and activate skills lazily (#4497) | ||
|
hataa
|
7857fa0cce
|
feat(authz): enforce tool authorization at assembly and runtime (#4370)
* feat(authz): enforce tool authorization at assembly and runtime * fix(middleware): guard deferred tool setup lookup (#4370) --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
March7
|
8c78d1f41f
|
fix(subagents): load user-scoped skills (#4356) | ||
|
hataa
|
10890e10a8
|
feat(authz): propagate trusted authorization principal context (#4203) | ||
|
Huixin615
|
65afc9b1d2
|
fix(skills): apply allowed-tools only to active skills (#4098)
* fix(skills): scope allowed-tools to active skills * fix(skills): tolerate stale active skill paths * chore: retrigger CI * fix(skills): document policy activation limits * perf(skills): reuse per-step tool policy decisions * fix(skills): harden runtime tool policy contracts * fix(skills): redact cached policy decisions * fix(skills): make slash tool policy authoritative * fix(skills): preserve policy-safe discovery tools * test(skills): cover explicit task delegation policy |
||
|
Nan Gao
|
de55982c5a
|
fix(subagents): preserve parent checkpoint namespace (#4215)
* fix(subagents): preserve parent checkpoint namespace * test(subagents): align stream isolation coverage |
||
|
Aari
|
42544755ac
|
fix(skills): escape untrusted skill metadata before it enters the model prompt (#4128)
* fix(skills): escape untrusted skill metadata before it enters the model prompt Skill name/description/allowed-tools come from the frontmatter of a user-installable .skill archive (POST /api/skills/install or a drop into skills/custom/); the parser only strips them. The slash-activation and durable-context siblings already html.escape these exact fields before rendering them into a model-visible block -- but five other render sites emit them raw. The sharpest is the default path, <available_skills> in the system prompt (skills.deferred_discovery: false): a community skill whose description closes the block can forge a framework-trusted <system-reminder> into the lead-agent system prompt. Driven through the real apply_prompt_template(), the forged tag reaches the system prompt raw on main and is neutralized here. Escape at every render site that emits untrusted skill metadata/content: - <available_skills> (name/description/location) and <disabled_skills> (name) in lead_agent/prompt.py; - describe_skill output (name/description/allowed-tools/location) and <skill_index> (name) in skills/describe.py; - the subagent <skill name=...> attribute plus the raw SKILL.md body in subagents/executor.py::_load_skill_messages -- its direct sibling skill_activation escapes both, this escaped neither. quote=False in element-text positions (matching skill_context and the #4097 correction), quote=True in the one attribute position (matching skill_activation). category is a controlled enum and is left as-is; escaping is render-time only, so stored skills are unchanged and re-rendering never double-escapes. * fix(skills): escape skill name in the slash-activation prose line The slash-activation reminder emitted `activation.skill_name` raw in its prose line while escaping the same value in the adjacent <skill name="..."> attribute. skill_name is grammar-gated to [a-z0-9-] by resolve_slash_skill before it reaches the renderer, so this is a defense-in-depth / consistency fix rather than a reachable injection: the two positions can never drift if a future caller builds an activation from an unconstrained name. Reuse the already-computed escaped_skill_name. |
||
|
Daoyuan Li
|
2bd0f56a0f
|
fix(subagents): classify recursion-capped LLM error fallbacks as failed (#4056)
The GraphRecursionError except-block in SubagentExecutor._aexecute derives usable_partial from the last AIMessage's raw non-empty text, without checking _extract_llm_error_fallback (#4042) first. A handled provider failure (LLMErrorHandlingMiddleware's deerflow_error_fallback marker) always carries non-empty user-facing text, so when it lands on the same turn that trips max_turns, it is indistinguishable from genuine partial output and gets misclassified as a completed task instead of the failed provider error it is. Consult _extract_llm_error_fallback in this except-block too, same as the normal-completion path above it, and classify FAILED with stop_reason=turn_capped when it detects the marker. |
||
|
Nan Gao
|
aafd5077b2
|
feat(subagents): show effective model and token usage on task cards (#4049)
* feat(subagents): show runtime metadata on task cards * fix(subagents): stop task-card render loop and dedupe model fetches Address code review on the runtime-metadata cards: - P1 render loop: the terminal ToolMessage is re-parsed on every MessageList render and always carries modelName/usage, so the presence-based setTasks condition fired a fresh state object each render -> "Maximum update depth exceeded". computeNextSubtask now returns a value-compared `changed` flag and a pure subtaskNotification() routes terminal transitions through the deferred after-render path while skipping no-op re-parses. - Per-card useModels refetch: add staleTime: Infinity to the ["models"] query so every subtask card shares one /api/models fetch instead of refetching on each mount. * make format * refactor(subagents): dedupe token-usage validators + tidy event narrowing Address PR review follow-ups: - DRY: extract one shared token-usage validator per side. Backend status_contract.normalize_token_usage() now backs both the terminal ToolMessage metadata and the subagent.step/.end run events (step_events.py), and frontend messages/usage.normalizeTokenUsage() backs both the live task_running event (lifecycle.ts) and the terminal ToolMessage metadata (subtask-result.ts). Prevents the input/output/ total_tokens validation from drifting across the four former copies. - Nit: onCustomEvent narrows event.type once instead of re-checking the object shape per branch; the redundant task_started early-return (already validated by taskEventToSubtaskUpdate) is dropped. |
||
|
Nan Gao
|
bbb3deb231
|
fix(subagents): classify LLM error fallbacks as failed (#4042)
* fix(subagents): classify LLM error fallbacks as failed (#4041) * fix(subagents): address #4042 review on LLM error fallback - clarify _extract_llm_error_fallback docstring: tail-only scan is safe because subagents append their own terminal message, so the last AIMessage is never a stale parent-history marker (cross-ref worker.py) - compute final_result and pop stop_reason only on the COMPLETED branch so the guard stop-reason pop no longer fires on the discarded FAILED path - note the error_detail/"LLM request failed" fallbacks are defensive; the middleware always populates a non-empty content - add regression test locking the stale parent-history marker invariant --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
hataa
|
266883b3dd
|
fix(subagents): inherit summarization middleware and harden step capture (#3875 Phase 3) (#4009)
Phase 3 of #3875 — subagents previously inherited none of the lead's context-compaction, so a deep-research subagent (max_turns up to 150) could accumulate >1M cumulative input before max_turns/timeout/token_budget engaged, even after Phase 2's budget capped the pathological tail. - Gate the subagent runtime chain on the SAME ``app_config.summarization.enabled`` switch the lead reads (per maintainer guidance in #3875), via the shared ``create_summarization_middleware`` factory. One config covers both chains; no separate ``subagents.summarization`` field. No-op when summarization is off (factory returns None). - ``skip_memory_flush=True`` on the subagent path: the factory otherwise attaches ``memory_flush_hook`` (when memory.enabled), which flushes pre-compaction messages into durable memory keyed by thread_id. Subagents share the parent's thread_id, so without skipping the hook a subagent's internal turns would pollute the PARENT thread's durable memory (#3875 Phase 3 review point). - Harden ``capture_new_step_messages`` to tolerate history contraction: summarization rewrites the messages channel via ``RemoveMessage(id=REMOVE_ALL_MESSAGES)``, shrinking len(messages) below the step-capture cursor. Without a reset, every step appended after the compaction point was dropped until length overtook the stale cursor (#3845 interaction, maintainer validation point (a)). Cursor now resets to the new tail; id/content dedup prevents re-emitting pre-compaction steps. - Couple the DEFAULT token-budget ceiling to ``summarization.enabled`` (#3875 Phase 3 review point): 1M when compaction is on, 2M when off (preserves Phase 2's deliberate headroom for summarization-off deep-research runs that can exceed 1M). A user-set budget (global or per-agent) always wins regardless of the switch. Flagged tunable. The summarization middleware does not implement ``consume_stop_reason``, so the Phase 2 guard-cap stop-reason channel is unaffected. Refs: https://github.com/bytedance/deer-flow/issues/3875 |
||
|
Ryker_Feng
|
ebc09ce130
|
feat(mcp): auto-promote deferred MCP tools from routing hints (#4019)
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
|
||
|
hataa
|
c9fb9768d4
|
fix(subagents): unify guardrail caps on additive stop_reason + add token_budget (#3875 Phase 2) (#3980)
Phase 2 of #3875. Two guardrail axes can end a subagent run early — the turn budget (GraphRecursionError) and the token budget (TokenBudgetMiddleware) — and both now surface *why* through one additive `subagent_stop_reason` field instead of a status enum. This completes and course-corrects Phase 1 (#3949), which shipped the turn-budget cap as a `max_turns_reached` status enum. The agreed Phase 2 design replaces that enum with an optional `stop_reason` field (token_capped | turn_capped | loop_capped): a new enum value would break v1 consumers, while an additive field is ignored by older frontends and ledger readers. `max_turns_reached` and SubagentStatus.MAX_TURNS_REACHED are removed. - subagents.token_budget config (default enabled, 2,000,000 tokens, warn 0.7) with per-agent override; TokenBudgetMiddleware is now attached in build_subagent_runtime_middlewares so the cost-ceiling backstop engages for every subagent. The hard-stop does not raise — it strips tool_calls and lets the run finish with a final answer, recording the cap on a per-run consume_stop_reason() accessor. - executor.py: on normal completion it reads consume_stop_reason() and stamps completed + token_capped when the budget fired; on GraphRecursionError it recovers the last AIMessage partial (completed + turn_capped) or, if nothing usable survived, failed + turn_capped. SubagentResult gains stop_reason. - status_contract.py / contracts/subagent_status_contract.json (v2) / frontend subtask-result.ts: additive subagent_stop_reason field, pinned by test_status_values_match_contract / test_stop_reason_values_match_contract. - task_tool.py + delegation_ledger.py: drop the max_turns_reached paths; the ledger captures stop_reason and renders model-facing "capped" guidance so the lead reuses a capped completion knowingly. The 2,000,000-token default is deliberately loose (tighten to taste) — it would have roughly halved the reported 4.4M burn while leaving legitimate deep-research runs (max_turns=150) room. Subagent summarization is a follow-up. |
||
|
hataa
|
0664ea2243
|
fix(subagents): surface turn-budget cap as MAX_TURNS_REACHED with partial result (#3875 Phase 2) (#3949)
* fix(subagents): surface turn-budget cap as MAX_TURNS_REACHED with partial result (#3875) Phase 2 of #3875. When a subagent exhausts its turn budget (recursion_limit == max_turns), LangGraph raises GraphRecursionError from agent.astream. The generic except Exception in _aexecute misclassified it as FAILED and discarded the partial work already streamed into final_state, so the lead could not tell 'broken subagent' from 'out of budget' and got an empty failure. Catch GraphRecursionError specifically (before the generic handler) and set a distinct SubagentStatus.MAX_TURNS_REACHED terminal status, recovering the partial result from the last streamed chunk via a shared _extract_final_result helper (refactored out of the normal-completion path so both paths render content identically). Extend the cross-language status contract so the new value travels on additional_kwargs.subagent_status: a capped run is result-bearing, so make_subagent_additional_kwargs / read_subagent_result_metadata carry subagent_result_brief + subagent_result_sha256 (the recovered work, like completed) AND the cap notice on subagent_error -- the one status that carries both. task_tool.py returns it via the shared _task_result_command; the delegation ledger prefers the partial result_brief and renders model-facing guidance (reuse / retry tighter / raise max_turns). Frontend collapses max_turns_reached to the failed pill with the cap notice on error. No agent-loop, runner, or persistence behavior touched; default max_turns is unchanged. * refactor(subagents): consolidate content-stringify onto shared helper Address review feedback on #3949 (willem-bd, copilot-pull-request-reviewer): - executor.py: drop the private `_stringify_message_content` — a third near-duplicate of `utils/messages.py::message_content_to_text`. `_extract_final_result` now delegates to that canonical helper; the "No response generated" sentinel is pushed down to the consumer (the shared helper returns "" for no-text, matching every other call site). - task_tool.py: align the live `task_failed` event's error string with the canonical "Reached max_turns=N" used by the logger, the structured `error=`, and the executor (was "Reached max turns (N)"). Behavior for real AIMessage content is unchanged; only atypical edge inputs (consecutive bare-string list items; empty content) now match the canonical helper that every other call site already uses. `extract_response_text` is intentionally left as-is: it filters by OpenAI content-block `type`, a different shape with many callers and its own tests. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Tianye Song
|
15454b6fec
|
feat(skills): deferred skill discovery via describe_skill tool (#3775)
Replace the full-metadata <available_skills> system-prompt block with a compact <skill_index> (names only) and an on-demand describe_skill tool when skills.deferred_discovery: true (default: false / backward compat). New modules: - skills/catalog.py — SkillCatalog (immutable, searchable; select: has no cap, keyword/prefix search caps at MAX_RESULTS=5) - skills/describe.py — build_describe_skill_tool(catalog) closure; build_skill_search_setup() wires SkillSearchSetup into both the LangGraph agent factory (agent.py) and DeerFlowClient (client.py) Changes: - Skill @dataclass(frozen=True); allowed_tools/required_secrets list→tuple - Skill First prompt line gated on skill_names (deferred vs legacy wording) - get_skills_prompt_section: short-circuit storage on deferred path; merge user_id (upstream) + skill_names (this PR) params - describe_skill tool parameter named "name" (matches prompt wording) - select: branch removes [:MAX_RESULTS] cap (exact request, not ranking) - AGENTS.md: document deferred_discovery config field + new modules Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Xinmin Zeng
|
576577bd32
|
feat(channels): expose IM channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID (#3926)
* feat(channels): expose channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID IM-channel skills need the sender's platform identity (Feishu open_id, Slack Uxxx, ...). The channel manager already writes channel_user_id into body.context, but the Gateway whitelist dropped it. Forward it into the runtime context only (never configurable, which is checkpointed), and have bash_tool export it as a fixed env var through a shell-quoted command prefix. The identity deliberately does not ride execute_command(env=...): that channel is reserved for request-scoped secrets, and a non-empty env switches AioSandbox onto the bash.exec path (fresh session per call, image >= 1.9.3 required), which would have broken every IM bash command on older sandbox images and abandoned persistent-shell semantics on new ones. A command-string export keeps the legacy path, stays visible in audit logs (it is an identifier, not a secret), and gives per-call correctness in group chats where one thread and sandbox are shared by senders with different platform ids. Skipped on the Windows local sandbox, whose PowerShell/cmd.exe fallback has no POSIX export. Part of #3914 * feat(channels): propagate channel_user_id to subagents; cap value length Review findings from the pre-PR verification pass: - Subagent delegation dropped the sender identity: task_tool now captures channel_user_id from the parent runtime context and the executor forwards it into the subagent's context, mirroring the guardrail attribution fields (user_role/oauth_*/run_id). Without this, bash commands delegated via task lost the group-chat sender's id. - body.context is client-writable on web requests, so values over 256 chars are ignored instead of bloating every command string sent to the sandbox. * fix(channels): set-or-unset channel_user_id so identity is per-call regardless of AIO session persistence Review (willem-bd): the identity export could leak across senders in a shared group-chat AIO sandbox. The AIO no-env path reuses a persistent shell session (the class-lock reason, #1433), and the 256-char/type guard made some commands carry no prefix — so a dropped-id command could resolve the id a previous sender exported. Make per-call correctness independent of session semantics: an IM-channel command (channel_user_id present in context) now always carries an explicit prefix — export VAR=<quoted> for a valid id, or unset VAR for an unusable one (empty / non-str / over the cap). Non-IM runs (no key) are untouched. A prefix unset has none of the '& ; unset' suffix hazard raised earlier. Verified on a real AIO 1.11.0 container: the no-id shell path auto-creates a session per call (does not persist today), but an explicit shared session DOES persist (export stale-A -> readback [stale-A]); the unset prefix clears it (-> []). So the fix holds even on an image whose no-id path persists. Regression tests cover the dropped-id group-chat window and the non-IM passthrough. Part of #3914 * test(channels): align channel_user_id task test with new Command return shape The merge from main changed task_tool to return a Command(update=...) instead of a plain string; update the assertion to extract the tool message via the existing _task_tool_message helper, matching the sibling tests. Fixes the CI backend-unit-tests failure introduced by the merge. |
||
|
AnoobFeng
|
e3e5c73b03
|
feat(observability): add trace-id correlation and enhanced logging (#3902)
* feat(observability): add trace-id correlation and enhanced logging - add opt-in gateway request trace correlation via X-Trace-Id - enhance logging with configurable trace_id-aware formatting - propagate deerflow_trace_id into runtime context and Langfuse metadata - keep enhanced logging disabled by default to preserve existing behavior * fix: harden trace correlation wiring - Make logging enhancement a restart-required startup snapshot and remove per-request config reads from TraceMiddleware - Restrict trace ids to printable ASCII before writing them to response headers, logs, and Langfuse metadata - Gate implicit DeerFlowClient trace-id creation behind logging.enhance.enabled while preserving explicit caller opt-in - Bind embedded client trace context per stream step to avoid generator ContextVar leaks and cross-context reset errors - Rebind memory update trace ids in Timer/executor worker paths so enhanced logs keep the captured correlation id - Remove unrelated __run_journal context overwrite from the trace-correlation change set * fix(gateway): avoid eager app construction on package import * fix(gateway): avoid config load during app import Keep Gateway app construction import-safe when config.yaml is absent by disabling TraceMiddleware only for that construction-time fallback path. Startup lifespan still performs strict config loading before serving. |
||
|
Nan Gao
|
4fcb4bc366
|
feat(subagents): persist and display subagent step history (#3779) (#3845)
* feat(subagents): persist and display subagent step history (#3779) Capture both assistant turns and tool outputs during subagent execution, stream them in task_running events, and persist them as subagent.* run events so the subtask card's step timeline survives a reload. Backend: - step_events.py: pure layer (capture_step_message, build_subagent_step, subagent_run_event) shared by streaming and persistence - executor.py: capture ToolMessage outputs, not just AIMessage turns - worker.py: persist task_* custom events to RunEventStore (category "subagent" keeps them out of the thread feed; list_events backfills) Frontend: - core/tasks/steps.ts + api.ts: SubtaskStep model, messageToStep, eventsToSteps, mergeSteps, fetchSubtaskSteps - subtask card accumulates live steps and backfills on expand - carry run_id onto history content messages for the events endpoint * fix(subagents): show AI turns in subtask card + paginate step backfill (#3779) Two follow-ups to the subagent step-history feature: Problem 1 — reload backfill could silently truncate the step timeline because list_events capped at 500 events (seq-ASC) across the whole run. Add task_id filtering + an after_seq forward cursor to list_events (all three stores + abstract base + the /events route), and make fetchSubtaskSteps page through one task's subagent.step events until a short page. No schema migration: the DB filter rides the existing run-scoped index via event_metadata["task_id"]. Problem 2 — the card only rendered tool steps, so persisted AI turns were never shown. Replace toolStepsForDisplay with stepsForDisplay: interleave AI reasoning turns (with text) and tool steps by message_index, drop blank-text AI turns, and drop the trailing final-answer AI turn when completed (already shown as result). Card renders AI steps as muted clamped markdown with a sparkles icon. Tests: store task_id/after_seq filtering + pagination across memory/db/jsonl, the /events route forwarding, stepsForDisplay rules, and fetchSubtaskSteps pagination. Docs updated in both AGENTS.md. * make format * fix(subagents): capture full multi-tool step tail, batch step persistence, cap tool-call args (#3779) Address PR review findings on the subagent step-history feature: 1. executor.py streamed on stream_mode="values" and captured only messages[-1] per chunk, so a multi-tool-call turn (ToolNode appends one ToolMessage per call in a single super-step) lost all but the last tool output in both the live task_running stream and the persisted history. Replace with capture_new_step_messages, which walks the newly-appended tail (and still re-checks the trailing message on no-growth chunks so id-less in-place replacements survive). 2. worker.py persisted each step with the store's low-frequency put() (a per-thread advisory lock per call); a deep subagent (max_turns=150) emits hundreds of steps on the hot stream loop. Replace with _SubagentEventBuffer, which batches via put_batch (flush on terminal subagent.end, at FLUSH_THRESHOLD, and in the worker finally). 3. build_subagent_step capped only text; tool_calls[].args were copied verbatim, so a large write_file/bash payload produced an unbounded subagent.step row. Cap each call's serialized args at SUBAGENT_STEP_MAX_CHARS, flagged args_truncated. Tests updated/added for all three; AGENTS.md refreshed. * fix(subagents): merge backfill into latest subtask state; reuse message_content_to_text (#3779) Address the remaining two PR review findings: 4. subtask-card's fetchSubtaskSteps().then(updateSubtask) closed over a stale tasks snapshot: a late-resolving backfill wrote setTasks({...stale}), clobbering SSE steps/status and sibling subtasks that arrived during the fetch. useUpdateSubtask now reads/writes through a tasksRef mirroring the latest state (ref-to-latest), and the pure per-subtask transition is extracted to core/tasks/subtask-update.ts::computeNextSubtask (unit-tested). 5. step_events._content_to_text duplicated deerflow.utils.messages. message_content_to_text; call the shared helper instead (guarding None content with 'or ""' so a tool-call-only turn still renders as ""). Tests added for computeNextSubtask and the None-content case; AGENTS.md docs updated. |
||
|
ly-wang19
|
9535a4f1c2
|
perf(subagents): dedup streamed AI messages via a seen-id set (O(n^2) -> O(n)) (#3687)
_aexecute collects AI messages from agent.astream(stream_mode="values"), which re-yields the full state every super-step. The duplicate check rescanned the append-only ai_messages list on every chunk -- any(m["id"] == message_id) -- so a run with M messages did O(M^2) work, and M reaches max_turns=150 for the general-purpose / deep-research subagent. Track an id-keyed set alongside ai_messages: id-bearing messages become O(1) set lookups, and the id-less full-dict-compare fallback is preserved. Behavior is unchanged. Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Miracle778
|
5a699e24a1
|
feat(guardrails): expose authenticated runtime context in GuardrailRequest (#3665)
* docs: guardrail runtime attribution spec * docs: guardrail request attribution implementation plan * feat(guardrails): add runtime user context and attribution fields to GuardrailRequest Extend GuardrailRequest with optional runtime attribution fields so that pluggable GuardrailProviders can access authenticated user context and tool-call-level attribution: - Gateway injects user_role, oauth_provider, oauth_id into runtime context alongside the existing user_id (server-authenticated only, client spoofing prevented) - GuardrailRequest gains: user_id, user_role, oauth_provider, oauth_id, run_id, tool_call_id (all optional, backward compatible) - GuardrailMiddleware reads these from ToolCallRequest.runtime.context - thread_id now actually populated from context (was always None before) - Tests: 15 new/expanded tests covering Gateway injection, runtime context reading, partial/missing fields, and client spoofing prevention - Docs: new Runtime Attribution section in GUARDRAILS.md with provider example and YAML policy illustration * fix(guardrails): propagate attribution to subagents * fix(guardrails): complete subagent attribution propagation --------- Co-authored-by: Miracle778 <miracle778@no-reply.com> |