* fix(subagents): prohibit task tool in general-purpose system prompt (#4159)
The general-purpose subagent correctly lists `task` in disallowed_tools
to prevent recursive nesting. However, the system prompt did not
explicitly tell the LLM that `task` is unavailable. When the subagent
sees the parent agent use `task`, it infers the tool is available and
attempts to call it, triggering a LangGraph tool validation error.
Add an explicit <tool_restrictions> block to the system prompt stating
that `task` is NOT available and the subagent must NEVER attempt to
call it. This prevents the LLM from attempting the call in the first
place, rather than relying on runtime rejection.
Add a regression test verifying the prompt contains the prohibition.
* fix(security): register tool_restrictions in input sanitization denylist
PR #4161 added <tool_restrictions> to general_purpose.py subagent prompt
but did not register it in _BLOCKED_TAG_NAMES. The anti-drift test
test_denylist_covers_framework_authority_blocks caught this: forging
<tool_restrictions> in untrusted input could trick the model into
believing it has (or lacks) tool restrictions it does not.
Add 'tool_restrictions' to _BLOCKED_TAG_NAMES alongside the other
subagent authority blocks (file_editing_workflow / guidelines /
output_format / working_directory).
* fix(skills): escape untrusted skill metadata before it enters the model prompt
Skill name/description/allowed-tools come from the frontmatter of a
user-installable .skill archive (POST /api/skills/install or a drop into
skills/custom/); the parser only strips them. The slash-activation and
durable-context siblings already html.escape these exact fields before
rendering them into a model-visible block -- but five other render sites emit
them raw. The sharpest is the default path, <available_skills> in the system
prompt (skills.deferred_discovery: false): a community skill whose description
closes the block can forge a framework-trusted <system-reminder> into the
lead-agent system prompt. Driven through the real apply_prompt_template(), the
forged tag reaches the system prompt raw on main and is neutralized here.
Escape at every render site that emits untrusted skill metadata/content:
- <available_skills> (name/description/location) and <disabled_skills> (name)
in lead_agent/prompt.py;
- describe_skill output (name/description/allowed-tools/location) and
<skill_index> (name) in skills/describe.py;
- the subagent <skill name=...> attribute plus the raw SKILL.md body in
subagents/executor.py::_load_skill_messages -- its direct sibling
skill_activation escapes both, this escaped neither.
quote=False in element-text positions (matching skill_context and the #4097
correction), quote=True in the one attribute position (matching
skill_activation). category is a controlled enum and is left as-is; escaping is
render-time only, so stored skills are unchanged and re-rendering never
double-escapes.
* fix(skills): escape skill name in the slash-activation prose line
The slash-activation reminder emitted `activation.skill_name` raw in its
prose line while escaping the same value in the adjacent
<skill name="..."> attribute. skill_name is grammar-gated to [a-z0-9-] by
resolve_slash_skill before it reaches the renderer, so this is a
defense-in-depth / consistency fix rather than a reachable injection: the
two positions can never drift if a future caller builds an activation from
an unconstrained name. Reuse the already-computed escaped_skill_name.
The GraphRecursionError except-block in SubagentExecutor._aexecute derives
usable_partial from the last AIMessage's raw non-empty text, without
checking _extract_llm_error_fallback (#4042) first. A handled provider
failure (LLMErrorHandlingMiddleware's deerflow_error_fallback marker)
always carries non-empty user-facing text, so when it lands on the same
turn that trips max_turns, it is indistinguishable from genuine partial
output and gets misclassified as a completed task instead of the failed
provider error it is.
Consult _extract_llm_error_fallback in this except-block too, same as the
normal-completion path above it, and classify FAILED with
stop_reason=turn_capped when it detects the marker.
* feat(subagents): show runtime metadata on task cards
* fix(subagents): stop task-card render loop and dedupe model fetches
Address code review on the runtime-metadata cards:
- P1 render loop: the terminal ToolMessage is re-parsed on every
MessageList render and always carries modelName/usage, so the
presence-based setTasks condition fired a fresh state object each
render -> "Maximum update depth exceeded". computeNextSubtask now
returns a value-compared `changed` flag and a pure subtaskNotification()
routes terminal transitions through the deferred after-render path
while skipping no-op re-parses.
- Per-card useModels refetch: add staleTime: Infinity to the ["models"]
query so every subtask card shares one /api/models fetch instead of
refetching on each mount.
* make format
* refactor(subagents): dedupe token-usage validators + tidy event narrowing
Address PR review follow-ups:
- DRY: extract one shared token-usage validator per side. Backend
status_contract.normalize_token_usage() now backs both the terminal
ToolMessage metadata and the subagent.step/.end run events
(step_events.py), and frontend messages/usage.normalizeTokenUsage()
backs both the live task_running event (lifecycle.ts) and the terminal
ToolMessage metadata (subtask-result.ts). Prevents the input/output/
total_tokens validation from drifting across the four former copies.
- Nit: onCustomEvent narrows event.type once instead of re-checking the
object shape per branch; the redundant task_started early-return
(already validated by taskEventToSubtaskUpdate) is dropped.
* fix(subagents): classify LLM error fallbacks as failed (#4041)
* fix(subagents): address #4042 review on LLM error fallback
- clarify _extract_llm_error_fallback docstring: tail-only scan is safe
because subagents append their own terminal message, so the last
AIMessage is never a stale parent-history marker (cross-ref worker.py)
- compute final_result and pop stop_reason only on the COMPLETED branch
so the guard stop-reason pop no longer fires on the discarded FAILED path
- note the error_detail/"LLM request failed" fallbacks are defensive;
the middleware always populates a non-empty content
- add regression test locking the stale parent-history marker invariant
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Phase 3 of #3875 — subagents previously inherited none of the lead's
context-compaction, so a deep-research subagent (max_turns up to 150)
could accumulate >1M cumulative input before max_turns/timeout/token_budget
engaged, even after Phase 2's budget capped the pathological tail.
- Gate the subagent runtime chain on the SAME ``app_config.summarization.enabled``
switch the lead reads (per maintainer guidance in #3875), via the shared
``create_summarization_middleware`` factory. One config covers both chains;
no separate ``subagents.summarization`` field. No-op when summarization is
off (factory returns None).
- ``skip_memory_flush=True`` on the subagent path: the factory otherwise
attaches ``memory_flush_hook`` (when memory.enabled), which flushes
pre-compaction messages into durable memory keyed by thread_id. Subagents
share the parent's thread_id, so without skipping the hook a subagent's
internal turns would pollute the PARENT thread's durable memory
(#3875 Phase 3 review point).
- Harden ``capture_new_step_messages`` to tolerate history contraction:
summarization rewrites the messages channel via
``RemoveMessage(id=REMOVE_ALL_MESSAGES)``, shrinking len(messages) below
the step-capture cursor. Without a reset, every step appended after the
compaction point was dropped until length overtook the stale cursor (#3845
interaction, maintainer validation point (a)). Cursor now resets to the
new tail; id/content dedup prevents re-emitting pre-compaction steps.
- Couple the DEFAULT token-budget ceiling to ``summarization.enabled``
(#3875 Phase 3 review point): 1M when compaction is on, 2M when off
(preserves Phase 2's deliberate headroom for summarization-off
deep-research runs that can exceed 1M). A user-set budget (global or
per-agent) always wins regardless of the switch. Flagged tunable.
The summarization middleware does not implement ``consume_stop_reason``, so
the Phase 2 guard-cap stop-reason channel is unaffected.
Refs: https://github.com/bytedance/deer-flow/issues/3875
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
Phase 2 of #3875. Two guardrail axes can end a subagent run early — the turn
budget (GraphRecursionError) and the token budget (TokenBudgetMiddleware) —
and both now surface *why* through one additive `subagent_stop_reason` field
instead of a status enum.
This completes and course-corrects Phase 1 (#3949), which shipped the
turn-budget cap as a `max_turns_reached` status enum. The agreed Phase 2
design replaces that enum with an optional `stop_reason` field
(token_capped | turn_capped | loop_capped): a new enum value would break v1
consumers, while an additive field is ignored by older frontends and ledger
readers. `max_turns_reached` and SubagentStatus.MAX_TURNS_REACHED are removed.
- subagents.token_budget config (default enabled, 2,000,000 tokens, warn 0.7)
with per-agent override; TokenBudgetMiddleware is now attached in
build_subagent_runtime_middlewares so the cost-ceiling backstop engages for
every subagent. The hard-stop does not raise — it strips tool_calls and
lets the run finish with a final answer, recording the cap on a per-run
consume_stop_reason() accessor.
- executor.py: on normal completion it reads consume_stop_reason() and stamps
completed + token_capped when the budget fired; on GraphRecursionError it
recovers the last AIMessage partial (completed + turn_capped) or, if nothing
usable survived, failed + turn_capped. SubagentResult gains stop_reason.
- status_contract.py / contracts/subagent_status_contract.json (v2) /
frontend subtask-result.ts: additive subagent_stop_reason field, pinned by
test_status_values_match_contract / test_stop_reason_values_match_contract.
- task_tool.py + delegation_ledger.py: drop the max_turns_reached paths; the
ledger captures stop_reason and renders model-facing "capped" guidance so
the lead reuses a capped completion knowingly.
The 2,000,000-token default is deliberately loose (tighten to taste) — it
would have roughly halved the reported 4.4M burn while leaving legitimate
deep-research runs (max_turns=150) room. Subagent summarization is a follow-up.
* fix(subagents): surface turn-budget cap as MAX_TURNS_REACHED with partial result (#3875)
Phase 2 of #3875. When a subagent exhausts its turn budget
(recursion_limit == max_turns), LangGraph raises
GraphRecursionError from agent.astream. The generic
except Exception in _aexecute misclassified it as FAILED and
discarded the partial work already streamed into final_state, so the
lead could not tell 'broken subagent' from 'out of budget' and got an
empty failure.
Catch GraphRecursionError specifically (before the generic handler)
and set a distinct SubagentStatus.MAX_TURNS_REACHED terminal status,
recovering the partial result from the last streamed chunk via a shared
_extract_final_result helper (refactored out of the normal-completion
path so both paths render content identically).
Extend the cross-language status contract so the new value travels on
additional_kwargs.subagent_status: a capped run is result-bearing, so
make_subagent_additional_kwargs / read_subagent_result_metadata
carry subagent_result_brief + subagent_result_sha256 (the
recovered work, like completed) AND the cap notice on subagent_error
-- the one status that carries both. task_tool.py returns it via the
shared _task_result_command; the delegation ledger prefers the
partial result_brief and renders model-facing guidance (reuse / retry
tighter / raise max_turns). Frontend collapses max_turns_reached to the
failed pill with the cap notice on error.
No agent-loop, runner, or persistence behavior touched; default
max_turns is unchanged.
* refactor(subagents): consolidate content-stringify onto shared helper
Address review feedback on #3949 (willem-bd, copilot-pull-request-reviewer):
- executor.py: drop the private `_stringify_message_content` — a third
near-duplicate of `utils/messages.py::message_content_to_text`.
`_extract_final_result` now delegates to that canonical helper; the
"No response generated" sentinel is pushed down to the consumer (the
shared helper returns "" for no-text, matching every other call site).
- task_tool.py: align the live `task_failed` event's error string with
the canonical "Reached max_turns=N" used by the logger, the structured
`error=`, and the executor (was "Reached max turns (N)").
Behavior for real AIMessage content is unchanged; only atypical edge inputs
(consecutive bare-string list items; empty content) now match the canonical
helper that every other call site already uses.
`extract_response_text` is intentionally left as-is: it filters by OpenAI
content-block `type`, a different shape with many callers and its own tests.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(gateway): cache-aware cost accounting + /api/console observability endpoints
- Capture prompt-cache hits (usage_metadata.input_token_details.cache_read)
in RunJournal and SubagentTokenCollector as a sparse cache_read_tokens key
in token_usage_by_model (JSON field — no schema migration; legacy bucket
shapes unchanged)
- New read-only /api/console router: GET /stats (headline counters),
GET /runs (cross-thread paginated history joined with thread titles),
GET /usage (zero-filled daily token series + per-model breakdown);
user-scoped, 503 on the memory database backend
- Optional models[*].pricing (currency, input_per_million,
output_per_million, input_cache_hit_per_million) powers real spend
estimation; cache-hit input tokens are billed at the hit price (omitted
hit price falls back to the miss price as a conservative upper bound);
unpriced models yield cost: null
- create_chat_model strips the presentation-only pricing block so it never
reaches the provider client (unknown kwargs are forwarded into the
completion payload and break live calls)
- Tests: console router SQLite round-trips, journal/collector cache capture
incl. a DeepSeek raw-usage pin test, factory strip regression
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: address review feedback on cost sum and sparse cache_read_tokens
- console.py: replace the walrus-in-generator total-cost sum with an
explicit loop (review noted the multi-line form reads ambiguously)
- token_collector.py: omit cache_read_tokens from usage records when the
provider reported no cache hits, matching the journal's sparse
per-model bucket shape; absent is treated as 0 downstream
- add a regression test pinning the sparse record shape
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: coffeeFish <codeingforcoffee@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(channels): expose channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID
IM-channel skills need the sender's platform identity (Feishu open_id,
Slack Uxxx, ...). The channel manager already writes channel_user_id into
body.context, but the Gateway whitelist dropped it. Forward it into the
runtime context only (never configurable, which is checkpointed), and have
bash_tool export it as a fixed env var through a shell-quoted command
prefix.
The identity deliberately does not ride execute_command(env=...): that
channel is reserved for request-scoped secrets, and a non-empty env
switches AioSandbox onto the bash.exec path (fresh session per call,
image >= 1.9.3 required), which would have broken every IM bash command
on older sandbox images and abandoned persistent-shell semantics on new
ones. A command-string export keeps the legacy path, stays visible in
audit logs (it is an identifier, not a secret), and gives per-call
correctness in group chats where one thread and sandbox are shared by
senders with different platform ids.
Skipped on the Windows local sandbox, whose PowerShell/cmd.exe fallback
has no POSIX export.
Part of #3914
* feat(channels): propagate channel_user_id to subagents; cap value length
Review findings from the pre-PR verification pass:
- Subagent delegation dropped the sender identity: task_tool now captures
channel_user_id from the parent runtime context and the executor
forwards it into the subagent's context, mirroring the guardrail
attribution fields (user_role/oauth_*/run_id). Without this, bash
commands delegated via task lost the group-chat sender's id.
- body.context is client-writable on web requests, so values over 256
chars are ignored instead of bloating every command string sent to the
sandbox.
* fix(channels): set-or-unset channel_user_id so identity is per-call regardless of AIO session persistence
Review (willem-bd): the identity export could leak across senders in a
shared group-chat AIO sandbox. The AIO no-env path reuses a persistent
shell session (the class-lock reason, #1433), and the 256-char/type guard
made some commands carry no prefix — so a dropped-id command could resolve
the id a previous sender exported.
Make per-call correctness independent of session semantics: an IM-channel
command (channel_user_id present in context) now always carries an explicit
prefix — export VAR=<quoted> for a valid id, or unset VAR for an unusable
one (empty / non-str / over the cap). Non-IM runs (no key) are untouched.
A prefix unset has none of the '& ; unset' suffix hazard raised earlier.
Verified on a real AIO 1.11.0 container: the no-id shell path auto-creates
a session per call (does not persist today), but an explicit shared session
DOES persist (export stale-A -> readback [stale-A]); the unset prefix
clears it (-> []). So the fix holds even on an image whose no-id path
persists. Regression tests cover the dropped-id group-chat window and the
non-IM passthrough.
Part of #3914
* test(channels): align channel_user_id task test with new Command return shape
The merge from main changed task_tool to return a Command(update=...)
instead of a plain string; update the assertion to extract the tool
message via the existing _task_tool_message helper, matching the sibling
tests. Fixes the CI backend-unit-tests failure introduced by the merge.
* feat: emit structured runtime metadata
* fix: avoid subagent import cycle in replay gateway
* fix: preserve legacy subtask result parsing
* refactor: tighten runtime metadata contracts
* fix(middleware): keep recovery hint on task exception wrapper content
The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.
* fix(subagents): require lowercase hex for result_sha256 reader
Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(observability): add trace-id correlation and enhanced logging
- add opt-in gateway request trace correlation via X-Trace-Id
- enhance logging with configurable trace_id-aware formatting
- propagate deerflow_trace_id into runtime context and Langfuse metadata
- keep enhanced logging disabled by default to preserve existing behavior
* fix: harden trace correlation wiring
- Make logging enhancement a restart-required startup snapshot and remove per-request config reads from TraceMiddleware
- Restrict trace ids to printable ASCII before writing them to response headers, logs, and Langfuse metadata
- Gate implicit DeerFlowClient trace-id creation behind logging.enhance.enabled while preserving explicit caller opt-in
- Bind embedded client trace context per stream step to avoid generator ContextVar leaks and cross-context reset errors
- Rebind memory update trace ids in Timer/executor worker paths so enhanced logs keep the captured correlation id
- Remove unrelated __run_journal context overwrite from the trace-correlation change set
* fix(gateway): avoid eager app construction on package import
* fix(gateway): avoid config load during app import
Keep Gateway app construction import-safe when config.yaml is absent by
disabling TraceMiddleware only for that construction-time fallback path.
Startup lifespan still performs strict config loading before serving.
* feat(subagents): persist and display subagent step history (#3779)
Capture both assistant turns and tool outputs during subagent execution,
stream them in task_running events, and persist them as subagent.* run
events so the subtask card's step timeline survives a reload.
Backend:
- step_events.py: pure layer (capture_step_message, build_subagent_step,
subagent_run_event) shared by streaming and persistence
- executor.py: capture ToolMessage outputs, not just AIMessage turns
- worker.py: persist task_* custom events to RunEventStore (category
"subagent" keeps them out of the thread feed; list_events backfills)
Frontend:
- core/tasks/steps.ts + api.ts: SubtaskStep model, messageToStep,
eventsToSteps, mergeSteps, fetchSubtaskSteps
- subtask card accumulates live steps and backfills on expand
- carry run_id onto history content messages for the events endpoint
* fix(subagents): show AI turns in subtask card + paginate step backfill (#3779)
Two follow-ups to the subagent step-history feature:
Problem 1 — reload backfill could silently truncate the step timeline because
list_events capped at 500 events (seq-ASC) across the whole run. Add task_id
filtering + an after_seq forward cursor to list_events (all three stores +
abstract base + the /events route), and make fetchSubtaskSteps page through one
task's subagent.step events until a short page. No schema migration: the DB
filter rides the existing run-scoped index via event_metadata["task_id"].
Problem 2 — the card only rendered tool steps, so persisted AI turns were never
shown. Replace toolStepsForDisplay with stepsForDisplay: interleave AI reasoning
turns (with text) and tool steps by message_index, drop blank-text AI turns, and
drop the trailing final-answer AI turn when completed (already shown as result).
Card renders AI steps as muted clamped markdown with a sparkles icon.
Tests: store task_id/after_seq filtering + pagination across memory/db/jsonl,
the /events route forwarding, stepsForDisplay rules, and fetchSubtaskSteps
pagination. Docs updated in both AGENTS.md.
* make format
* fix(subagents): capture full multi-tool step tail, batch step persistence, cap tool-call args (#3779)
Address PR review findings on the subagent step-history feature:
1. executor.py streamed on stream_mode="values" and captured only
messages[-1] per chunk, so a multi-tool-call turn (ToolNode appends
one ToolMessage per call in a single super-step) lost all but the last
tool output in both the live task_running stream and the persisted
history. Replace with capture_new_step_messages, which walks the
newly-appended tail (and still re-checks the trailing message on
no-growth chunks so id-less in-place replacements survive).
2. worker.py persisted each step with the store's low-frequency put()
(a per-thread advisory lock per call); a deep subagent (max_turns=150)
emits hundreds of steps on the hot stream loop. Replace with
_SubagentEventBuffer, which batches via put_batch (flush on terminal
subagent.end, at FLUSH_THRESHOLD, and in the worker finally).
3. build_subagent_step capped only text; tool_calls[].args were copied
verbatim, so a large write_file/bash payload produced an unbounded
subagent.step row. Cap each call's serialized args at
SUBAGENT_STEP_MAX_CHARS, flagged args_truncated.
Tests updated/added for all three; AGENTS.md refreshed.
* fix(subagents): merge backfill into latest subtask state; reuse message_content_to_text (#3779)
Address the remaining two PR review findings:
4. subtask-card's fetchSubtaskSteps().then(updateSubtask) closed over a
stale tasks snapshot: a late-resolving backfill wrote setTasks({...stale}),
clobbering SSE steps/status and sibling subtasks that arrived during the
fetch. useUpdateSubtask now reads/writes through a tasksRef mirroring the
latest state (ref-to-latest), and the pure per-subtask transition is
extracted to core/tasks/subtask-update.ts::computeNextSubtask (unit-tested).
5. step_events._content_to_text duplicated deerflow.utils.messages.
message_content_to_text; call the shared helper instead (guarding None
content with 'or ""' so a tool-call-only turn still renders as "").
Tests added for computeNextSubtask and the None-content case; AGENTS.md docs updated.
_aexecute collects AI messages from agent.astream(stream_mode="values"),
which re-yields the full state every super-step. The duplicate check rescanned
the append-only ai_messages list on every chunk -- any(m["id"] == message_id) --
so a run with M messages did O(M^2) work, and M reaches max_turns=150 for the
general-purpose / deep-research subagent.
Track an id-keyed set alongside ai_messages: id-bearing messages become O(1)
set lookups, and the id-less full-dict-compare fallback is preserved. Behavior
is unchanged.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(gateway): attribute token usage to actual models
Capture per-call model names from LLM response metadata for lead, middleware, and subagent calls.
Persist a per-run token_usage_by_model breakdown and aggregate by that map in both SQL and memory stores, with legacy fallback to the run-level model_name for older rows.
Add regression coverage for by_model totals, caller consistency, active progress snapshots, store parity, and SubagentTokenCollector model propagation.
* fix(gateway): harden by-model token aggregation
Use usage.get("total_tokens", 0) when reducing per-model token usage maps so aggregation tolerates partially written or manually edited JSON blobs without changing behavior for journal-written rows.
* docs(gateway): clarify by-model run count semantics
Document that by_model[*].runs counts the number of runs in which a model appeared, so multi-model runs can increment multiple model buckets.
The subagent execution path did not call inject_langfuse_metadata(...)
and built its model with attach_tracing=True, so subagent LLM/tool
spans landed in Langfuse as isolated top-level traces carrying fresh
session ids and the default user. They were findable in the unfiltered
trace list but did not group under the parent thread's session card,
and Langfuse cost attribution for subagent traffic did not line up
with the parent conversation — even though DeerFlow's internal token
accounting (SubagentTokenCollector) was already correct.
Extend the lead-agent tracing wiring to the subagent path so a single
subagent run produces one trace that shares the parent thread's
session_id and user_id, with a subagent:<name> trace name:
- subagents/executor.py: append build_tracing_callbacks() output to
run_config["callbacks"] (preserving SubagentTokenCollector) and
call inject_langfuse_metadata(...) with thread_id, user_id, and
the normalized subagent:<name> trace name. Build the model with
attach_tracing=False so model-level tracing does not double-count
with the graph-root callbacks — the same pairing the lead agent
uses.
- tools/builtins/task_tool.py: resolve user_id via
resolve_runtime_user_id(runtime) at the parent tool layer (before
the background thread starts) and thread it through
SubagentExecutor.__init__, because the _current_user contextvar
is not guaranteed to survive the _execution_pool boundary.
Trace topology is unchanged: subagent traces remain separate top-level
traces in the same session, not nested as child spans under the lead
trace (Plan B follow-up).
Tests: tests/test_subagent_executor.py::TestSubagentTracingWiring
covers the callback append, the session/user/trace-name injection,
the disabled-langfuse no-op, the DEFAULT_USER_ID fallback, the
empty-name trace-name fallback, and the env-tag emission. Existing
test_create_agent_threads_explicit_app_config_to_model_and_middlewares
now also asserts attach_tracing=False.
Docs: CLAUDE.md Tracing System section documents subagents/executor.py
as a third injection point alongside worker.py and client.py.
* fix(subagents): raise general-purpose max_turns to 150 and default timeout to 30min
Deep-research subtasks failed out of the box with GraphRecursionError (Recursion limit of 100 reached): the built-in general-purpose subagent caps at max_turns=100. Raise it to 150 and bump the default subagent timeout from 900s (15min) to 1800s (30min) so the extra turns have time to run instead of shifting the failure to a timeout. The lead agent recursion_limit (100) is unchanged; the failures are subagent-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(subagents): clarify lead recursion_limit is independent of subagent max_turns
Add comments at both lead recursion_limit=100 sites (gateway services + channel manager) explaining the lead's LangGraph super-step budget is separate from subagent depth, so the two 100s are not conflated. Comment-only, no behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(subagents): clarify built-in vs custom timeout scope; pin bash max_turns in test
Review follow-ups: (1) clarify SubagentConfig docstring + global timeout field/comment that the 1800 default applies to built-in subagents (custom agents keep their own timeout_seconds); (2) pin bash.max_turns==60 in the defaults regression test so the config.example.yaml doc cannot drift; (3) rename test_default_timeout_preserved_when_no_config -> test_explicit_global_timeout_propagates_to_general_purpose since it intentionally exercises an explicit non-default 900. No runtime behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Subagent _create_agent() now passes checkpointer=False to prevent
inheriting the parent run's synchronous checkpointer via copy_context(),
which would cause NotImplementedError when aget_tuple() is called on
the async path. Subagents are one-shot delegations that never resume,
so persistence is unnecessary.
* feat(subagents): extend deferred MCP tool loading to subagents (#3341)
Subagents now reuse the lead agent's deferred-tool path: when
tool_search.enabled, MCP tool schemas are withheld from the model and
surfaced by name in <available-deferred-tools>, fetched on demand via the
generated tool_search helper. DeferredToolFilterMiddleware deterministically
rewrites request.tools to hide the deferred schemas (the prompt section is
discovery only, not enforcement).
Consolidates the assembly into deerflow.tools.builtins.tool_search, now the
single home for both assemble_deferred_tools (centralized fail-closed guard,
replacing the lead-only private _assemble_deferred) and the relocated
get_deferred_tools_prompt_section. Shared by every build path: lead agent,
embedded client, and subagent executor.
tool_search is appended after the subagent's name-level tool policy and is
treated as infrastructure: its catalog is built from the already
policy-filtered list, so it can never surface a tool the policy denied.
Follow-up to #3370. Fixes#3341.
* test(subagents): assert the real middleware builder emits a working deferred filter (#3341)
The existing recipe test hand-constructs DeferredToolFilterMiddleware, so it
cannot catch a regression in how build_subagent_runtime_middlewares (the call
executor._create_agent actually makes) wires the deferred setup into the
filter. Add a test that sources the filter from the real builder given a real
setup and runs it through a graph: a wrong catalog hash would silently stop
promotion, a dropped filter would stop hiding — both now caught.
Running the full real middleware stack is intentionally avoided (the other
runtime middlewares need sandbox/thread infra to execute, which would make the
test flaky); their attachment + ordering before Safety stays locked in
test_tool_error_handling_middleware.py.
* test(subagents): keep executor tests config-free in CI
* chore: trigger ci
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(subagent): structured subagent_status field over text parsing
Closes#3146.
## Why
The frontend used to derive subtask card state by string-matching the
leading text of the `task` tool's result. That contract surface was
fragile — `#3107` BUG-007 and the `#3131` review both surfaced cases
where new backend wording (`Task cancelled by user.`,
`Task polling timed out after N minutes`, `ToolErrorHandlingMiddleware`
exception wrappers) silently broke the card lifecycle. The frontend
fallback kept growing more prefixes; any future rewording would break
it again.
## Design
1. **Backend → frontend contract**: `ToolMessage.additional_kwargs`
carries `subagent_status` (one of `completed | failed | cancelled |
timed_out | polling_timed_out`) and an optional `subagent_error`
blob. The frontend prefers it over parsing `content`.
2. **Centralised stamping, not 8 sprinkled stamps**: rather than have
each of `task_tool.py`'s 5 normal-return + 3 pre-execution `Error:`
paths remember to set `additional_kwargs`, `ToolErrorHandlingMiddleware`
stamps the field after every task-tool call. Adding a new return
path in `task_tool.py` cannot now skip the stamp.
3. **Cross-language contract fixture**: the prefix→status mapping is
the one piece both sides must agree on. The shared fixture at
`contracts/subagent_status_contract.json` lists every backend return
string, the expected status, and what the error substring should
contain. Backend test (`backend/tests/test_subagent_status_contract.py`)
and frontend test (`frontend/tests/unit/core/tasks/subtask-result.test.ts`)
both load that fixture and assert the same cases. A wording drift on
either side fails the matching language's test.
4. **Round-trip serialisation pinned**: the round-trip test asserts
`ToolMessage.model_dump_json()` → `model_validate_json()` preserves
`additional_kwargs.subagent_status`. Catches the case where a future
LangChain or Pydantic upgrade silently strips unknown kwargs.
5. **Frontend status collapse documented**: the backend has five status
values, the frontend card has three (`completed | failed |
in_progress`). `cancelled` / `timed_out` / `polling_timed_out` all
collapse to `failed` with the original status preserved in `error`.
`parseSubtaskResult` returns `in_progress` for unknown values so a
backend that ships a new enum variant before the frontend upgrades
degrades to the legacy prefix fallback instead of getting pinned.
## Changes
Backend:
- `deerflow.subagents.status_contract` — new module exporting
`SUBAGENT_STATUS_KEY`, `SUBAGENT_ERROR_KEY`,
`SUBAGENT_STATUS_VALUES`, `extract_subagent_status(content)`, and
`make_subagent_additional_kwargs(status, error)`.
- `ToolErrorHandlingMiddleware`: new `_stamp_task_subagent_status`
helper centralises the stamp; `wrap_tool_call` / `awrap_tool_call`
stamp on the success path; `_build_error_message` stamps on the
wrapper path (carrying `ExcClass: detail` into `subagent_error`).
Non-task tools are untouched.
- New tests: `test_subagent_status_contract.py` (19 cases from the
shared fixture + status-enum / blank-error / unknown-status
rejection) and `test_tool_error_handling_subagent_stamp.py`
(middleware integration: terminal-content stamps, non-terminal
doesn't, non-task tools untouched, async path mirrors sync,
existing additional_kwargs survive, JSON round-trip preserved).
Frontend:
- `parseSubtaskResult(text, additionalKwargs?)` — prefers the
structured stamp; falls back to the legacy prefix matcher for
historical threads / unknown future status values.
- `STRUCTURED_STATUS_TO_SUBTASK` documents the five→three collapse.
- `message-list.tsx` passes `message.additional_kwargs` through.
- `subtask-result.test.ts` adds a structured-status block + a
fixture-driven contract block; legacy prefix tests stay green for
the fallback path.
Contract:
- `contracts/subagent_status_contract.json` — single source of truth
both languages load. Whitespace variants, varied N for polling
timeouts, the 3 pre-execution `Error:` returns task_tool produces,
and the middleware wrapper shape are all in there.
## Test plan
- `make lint` clean (backend + frontend).
- `pytest tests/test_subagent_status_contract.py
tests/test_tool_error_handling_subagent_stamp.py` → 37 passed.
- `pnpm test --run` → 103 passed (was 76, +27 new).
## Migration / fallback retirement
The text-prefix fallback stays in place until backend telemetry shows
the frontend never hits it for newly produced messages. At that point
a follow-up PR can drop the prefix branches and keep only the
structured-status branch.
Refs: bytedance/deer-flow#3138 (split summary), #3107 (origin), #3131
(prior prefix-only fix), #3146 (this issue).
* fix(subtask): back-fill result/error from text when structured status present
Three follow-ups on the PR #3154 review:
1. `readStructuredStatus` no longer short-circuits the prefix parse.
The backend currently stamps only the `subagent_status` enum value;
the human-facing `result` body and wrapped-error message still live
in `ToolMessage.content`. Dropping the text parse meant successful
tasks rendered empty completed pills and wrapped failures lost their
diagnostic. Now both shapes get composed: structured status wins,
`result`/`error` come from text when both sides agree, and a lying
success body under a `failed` stamp is dropped instead of leaking.
2. Replace the ESM-incompatible `__dirname` fixture lookup in
subtask-result.test.ts with `fileURLToPath(new URL(..., import.meta.url))`.
The frontend package is `"type": "module"`, so the previous path
would have thrown at runtime if anything ever changed under the
contract directory.
3. Drop the `$schema` reference from contracts/subagent_status_contract.json
pointing at a file that doesn't exist in the tree.
Three new tests cover the structured + text composition: completed
back-fills the success body, failed back-fills the wrapper text, and
unrecognised content under a `failed` stamp stays empty rather than
echoing noise.
* fix(#3189): prevent write_file streaming timeout on long reports
Adds a layered defense against StreamChunkTimeoutError caused by oversized
single-shot write_file tool calls:
- factory: default stream_chunk_timeout to 240s for OpenAI-compatible
clients (overridable via ModelConfig.stream_chunk_timeout in config.yaml)
- sandbox/tools: server-side 80 KB length guard on non-append write_file
calls (configurable via DEERFLOW_WRITE_FILE_MAX_BYTES env var, 0 disables);
rejects oversized payloads with a structured error pointing the model at
str_replace or append=True
- middleware: classify StreamChunkTimeoutError as transient but cap retries
at 1 via per-exception _RETRY_BUDGET_OVERRIDES (same-payload retry on a
chunk-gap timeout buffers the same way upstream; full 3-attempt loop
would stack 6-12 min of dead air)
- middleware: surface an actionable user-facing message for stream-drop
exceptions instead of leaking the raw langchain stack
- prompts: add a routing-style File Editing Workflow hint to both lead_agent
and general_purpose subagent prompts, pointing the model at str_replace
for incremental edits (mirrors Claude Code's Edit / Codex's apply_patch)
- tests: behavioural coverage for size guard, retry budget override,
stream-drop user message, factory default injection
Refs #3189
* fix(#3189): drop stream_chunk_timeout for non-OpenAI providers
Address CR feedback on PR #3195:
- factory: pop `stream_chunk_timeout` from kwargs for any model_use_path other than `langchain_openai:ChatOpenAI` instead of returning early. `ModelConfig.stream_chunk_timeout` is part of the shared schema, so a user-supplied value on a non-OpenAI provider would otherwise be forwarded to its constructor and raise `TypeError: unexpected keyword argument`.
- factory: rewrite docstring to describe the actual `exclude_none=True` behaviour (explicit null is excluded and falls back to the default) instead of the misleading "None falling out via exclude_none=True keeps its value".
- tests: add regression coverage asserting the kwarg is stripped before reaching a non-OpenAI provider's constructor.
Refs: bytedance#3189
* fix(#3189): restrict stream-drop user copy to StreamChunkTimeoutError only
Per CR on #3195: narrow _STREAM_DROP_EXCEPTIONS to StreamChunkTimeoutError. Generic httpx RemoteProtocolError / ReadError fall back to the standard 'temporarily unavailable' copy, since they routinely fire on transient network blips where the 'split the output' guidance is misleading. Retry/backoff classification is unchanged — both remain transient/retriable. Tests updated to reflect new copy, plus a symmetric regression test for ReadError.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Guard subagent terminal state transitions
* fix: publish subagent terminal status last
* Fix subagent timeout test to avoid blocking event loop
* Fix subagent timeout test tracking
* Refine subagent terminal state handling
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): consolidate system_prompt and skills into single SystemMessage
Some LLM APIs (vLLM, Xinference, Chinese LLM providers) reject multiple
system messages with \”System message must be at the beginning.\” The
subagent executor was sending separate SystemMessages for the configured
system_prompt and each loaded skill, which caused failures when calling
task tool with sub-agents.
Merge system_prompt and all skill content into one SystemMessage in the
initial state, and pass system_prompt=None to create_agent() so the
factory doesn't prepend a second one.
Fixes#2693
* fix(subagents): update SubagentConfig.system_prompt to str | None and add astream regression test
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/2ee03a26-e19b-4106-abc5-c76a2906383b
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
* fixed the lint error
* fix the lint error in the backend
* fix the unit test error of test_subagent_executor
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: bucket subagent token usage into RunRow.subagent_tokens
Add caller-bucketed token tracking to RunJournal so subagent and
middleware LLM calls are written to the correct RunRow columns instead
of all falling into lead_agent_tokens (default 0).
- RunJournal: accumulate _lead_agent_tokens / _subagent_tokens /
_middleware_tokens in on_llm_end, deduped by langchain run_id.
Add record_external_llm_usage_records() for external sources
(respects track_token_usage flag). Return caller buckets from
get_completion_data().
- SubagentTokenCollector: new lightweight callback handler that
collects LLM usage within subagent execution.
- SubagentExecutor: wire collector into subagent run_config and sync
records to SubagentResult on every chunk (timeout/cancel safe).
- SubagentResult: add token_usage_records and usage_reported fields.
- task_tool: report subagent usage to parent RunJournal on every
terminal status (COMPLETED/FAILED/CANCELLED/TIMED_OUT), including
the CancelledError path, guarded against double-reporting.
No DB migration needed — RunRow columns already exist.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix: address token usage review feedback
* Address review follow-ups
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* refactor: thread app config through lead prompt
* fix: honor explicit app config across runtime paths
* style: format subagent executor tests
* fix: thread resolved app config and guard subagents-only fallback
Address two PR review findings:
1. _create_summarization_middleware passed the original (possibly None)
app_config into create_chat_model, forcing the model factory back to
ambient get_app_config() and risking config drift between the
middleware's resolved view and the model's view. Pass the resolved
AppConfig instance through end-to-end.
2. get_available_subagent_names accepted Any-typed config and forwarded
it to is_host_bash_allowed, which reads ``.sandbox``. A
SubagentsAppConfig (also accepted upstream as a sum-type input) has
no ``.sandbox`` attribute and would be silently treated as "no
sandbox configured", incorrectly disabling the bash subagent. Guard
on hasattr and fall back to ambient lookup otherwise.
Adds regression tests for both paths.
* chore: simplify hasattr guard and tighten regression tests
- Collapse if/else into ternary in get_available_subagent_names; hasattr(None, ...) is False so the explicit None check was redundant.
- Drop comments that narrate the change rather than explain non-obvious WHY (test names already convey intent).
- Replace stringly-typed sentinel "no-arg" in regression test with direct args tuple comparison.
---------
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
* fix(subagents): use model override for tools and middleware
* fix(config): resolve effective subagent model
* fix(subagents): defer app config loading
* fix(subagents): fully defer config.yaml load in executor __init__
The previous attempt only relocated the explicit get_app_config() call,
but left resolve_subagent_model_name(...) running eagerly in __init__.
That helper has its own internal get_app_config() fallback, which still
fired when both app_config and parent_model were None and
config.model == "inherit" — exactly the path unit tests hit, breaking
21 tests in CI with FileNotFoundError: config.yaml.
Skip the eager resolve in __init__ when it would require loading the
config file, and defer to _create_agent (which already has the
app_config or get_app_config() fallback).
* refactor: thread app_config through middleware factories
Continues the incremental config-refactor sequence (#2611 root, #2612 lead
path) one layer deeper into the middleware factories. Two ambient lookups
inside _build_runtime_middlewares are eliminated and the LLMErrorHandling
band-aid removed:
- _build_runtime_middlewares / build_lead_runtime_middlewares /
build_subagent_runtime_middlewares now require app_config: AppConfig.
- get_guardrails_config() inside the factory is replaced with
app_config.guardrails (semantically identical — same default-factory
GuardrailsConfig — verified by direct equality check).
- LLMErrorHandlingMiddleware.__init__ now requires app_config and reads
circuit_breaker fields directly. The class-level
circuit_failure_threshold / circuit_recovery_timeout_sec defaults are
removed along with the try/except (FileNotFoundError, RuntimeError):
pass band-aid — the let-it-crash invariant the rest of the refactor
enforces.
Caller chain (already-resolved app_config sources):
- _build_middlewares in lead_agent/agent.py: reorder so
resolved_app_config = app_config or get_app_config() is computed BEFORE
build_lead_runtime_middlewares is called, then passed as kwarg.
- SubagentExecutor: optional app_config parameter (mirrors the lead-agent
pattern); _create_agent does the same `or get_app_config()` fallback at
agent-build time, so task_tool callers don't need to plumb app_config
through yet (typed-context plumbing for tool runtimes is a separate
refactor).
Tests:
- test_llm_error_handling_middleware: _make_app_config helper using
AppConfig(sandbox=SandboxConfig(use="test")) — same minimal-config
pattern conftest already uses. Three direct LLMErrorHandlingMiddleware()
calls each followed by post-construction circuit_breaker mutation fold
cleanly into _build_middleware(circuit_failure_threshold=...,
circuit_recovery_timeout_sec=...).
Verification:
- tests/test_llm_error_handling_middleware.py — 14 passed
- tests/test_subagent_executor.py — 28 passed
- tests/test_tool_error_handling_middleware.py — 6 passed
- tests/test_task_tool_core_logic.py — 18 passed (verifies task_tool
unchanged behavior)
- Full suite: 2697 passed, 3 skipped. The single intermittent failure in
tests/test_client_e2e.py::test_tool_call_produces_events is pre-existing
LLM flakiness (the test asserts the model decided to call a tool;
reproduces 1/3 on unchanged main as well).
* fix: address middleware app config review comments
* fix: satisfy app config annotation lint
* test: cover explicit app config middleware wiring
---------
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
* feat(subagents): support per-subagent skill loading and custom subagent types (#2230)
Add per-subagent skill configuration and custom subagent type registration,
aligned with Codex's role-based config layering and per-session skill injection.
Backend:
- SubagentConfig gains `skills` field (None=all, []=none, list=whitelist)
- New CustomSubagentConfig for user-defined subagent types in config.yaml
- SubagentsAppConfig gains `custom_agents` section and `get_skills_for()`
- Registry resolves custom agents with three-layer config precedence
- SubagentExecutor loads skills per-session as conversation items (Codex pattern)
- task_tool no longer appends skills to system_prompt
- Lead agent system prompt dynamically lists all registered subagent types
- setup_agent tool accepts optional skills parameter
- Gateway agents API transparently passes skills in CRUD operations
Frontend:
- Agent/CreateAgentRequest/UpdateAgentRequest types include skills field
- Agent card displays skills as badges alongside tool_groups
Config:
- config.example.yaml documents custom_agents and per-agent skills override
Tests:
- 40 new tests covering all skill config, custom agents, and registry logic
- Existing tests updated for new get_skills_prompt_section signature
Closes#2230
* fix: address review feedback on skills PR
- Remove stale get_skills_prompt_section monkeypatches from test_task_tool_core_logic.py
(task_tool no longer imports this function after skill injection moved to executor)
- Add key prefixes (tg:/sk:) to agent-card badges to prevent React key collisions
between tool_groups and skills
* fix(ci): resolve lint and test failures
- Format agent-card.tsx with prettier (lint-frontend)
- Remove stale "Skills Appendix" system_prompt assertion — skills are now
loaded per-session by SubagentExecutor, not appended to system_prompt
* fix(ci): sort imports in test_subagent_skills_config.py (ruff I001)
* fix(ci): use nullish coalescing in agent-card badge condition (eslint)
* fix: address review feedback on skills PR
- Use model_fields_set in AgentUpdateRequest to distinguish "field omitted"
from "explicitly set to null" — fixes skills=None ambiguity where None
means "inherit all" but was treated as "don't change"
- Move lazy import of get_subagent_config outside loop in
_build_available_subagents_description to avoid repeated import overhead
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): allow model override per subagent in config.yaml
Wire the existing SubagentConfig.model field to config.yaml so users
can assign different models to different subagent types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(subagents): cover model override in SubagentsAppConfig + registry
Addresses review feedback on #2064:
- registry.py: update stale inline comment — the block now applies
timeout, max_turns AND model overrides, not just timeout.
- test_subagent_timeout_config.py: add coverage for model override
resolution across SubagentOverrideConfig, SubagentsAppConfig
(get_model_for + load), and registry.get_subagent_config:
- per-agent model override is applied to registry-returned config
- omitted `model` keeps the builtin value
- explicit `model: null` in config.yaml is equivalent to omission
- model override on one agent does not affect other agents
- model override preserves all other fields (name, description,
timeout_seconds, max_turns)
- model override does not mutate BUILTIN_SUBAGENTS
Copilot's suggestion (3) "setting model to 'inherit' forces inheritance"
is skipped intentionally: there is no 'inherit' sentinel in the current
implementation — model is `str | None`, and None already means
"inherit from parent". Adding a sentinel would be a new feature, not
test coverage for this PR.
Tests run locally: 51 passed (37 existing + 14 new / expanded).
* test(subagents): reject empty-string model at config load time
Addresses WillemJiang's review comment on #2064 (empty-string edge case):
- subagents_config.py: add `min_length=1` to the `model` field on
SubagentOverrideConfig. `model: ""` in config.yaml would otherwise
bypass the `is not None` check and reach create_chat_model(name="")
as a confusing runtime error. This is symmetric with the existing
`ge=1` guards on timeout_seconds / max_turns, so the validation style
stays consistent across all three override fields.
- test_subagent_timeout_config.py: add test_rejects_empty_model
mirroring the existing test_rejects_zero / test_rejects_negative
cases; update the docstring on test_model_accepts_any_string (now
test_model_accepts_any_non_empty_string) to reflect the new guard.
Not addressing the first comment (validating `model` against the
`models:` section at load time) in this PR. `SubagentsAppConfig` is
scoped to the `subagents:` block and cannot see the sibling `models:`
section, so proper cross-section validation needs a second pass or a
structural change that is out of scope here — and the current behavior
is consistent with how timeout_seconds / max_turns work today. Happy to
track this as a follow-up issue covering cross-section validation
uniformly for all three fields.
Tests run locally: 52 passed in this file; 1847 passed, 18 skipped
across the full backend suite. Ruff check + format clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix event loop conflict in SubagentExecutor.execute()
When SubagentExecutor.execute() is called from within an already-running
event loop (e.g., when the parent agent uses async/await), calling
asyncio.run() creates a new event loop that conflicts with asyncio
primitives (like httpx.AsyncClient) that were created in and bound to
the parent loop.
This fix detects if we're already in a running event loop, and if so,
runs the subagent in a separate thread with its own isolated event loop
to avoid conflicts.
Fixes: sub-task cards not appearing in Ultra mode when using async parent agents
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(subagent): harden isolated event loop execution
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(subagents): add cooperative cancellation for subagent threads
Subagent tasks run inside ThreadPoolExecutor threads with their own
event loop (asyncio.run). When a user clicks stop, RunManager cancels
the parent asyncio.Task, but Future.cancel() cannot terminate a running
thread and asyncio.Event does not propagate across event loops. This
causes subagent threads to keep executing (writing files, calling LLMs)
even after the user explicitly stops the run.
Fix: add a threading.Event (cancel_event) to SubagentResult and check
it cooperatively in _aexecute()'s astream iteration loop. On cancel,
request_cancel_background_task() sets the event, and the thread exits
at the next iteration boundary.
Changes:
- executor.py: Add cancel_event field to SubagentResult, check it in
_aexecute loop, set it on timeout, add request_cancel_background_task
- task_tool.py: Call request_cancel_background_task on CancelledError
* fix(subagents): guard cancel status and add pre-check before astream
- Only overwrite status to FAILED when still RUNNING, preserving
TIMED_OUT set by the scheduler thread.
- Add cancel_event pre-check before entering the astream loop so
cancellation is detected immediately when already signalled.
* fix(subagents): guard status updates with lock to prevent race condition
Wrap the check-and-set on result.status in _aexecute with
_background_tasks_lock so the timeout handler in execute_async
cannot interleave between the read and write.
* fix(subagents): add dedicated CANCELLED status for user cancellation
Introduce SubagentStatus.CANCELLED to distinguish user-initiated
cancellation from actual execution failures. Update _aexecute,
task_tool polling, cleanup terminal-status sets, and test fixtures.
* test(subagents): add cancellation tests and fix timeout regression test
- Add dedicated TestCooperativeCancellation test class with 6 tests:
- Pre-set cancel_event prevents astream from starting
- Mid-stream cancel_event returns CANCELLED immediately
- request_cancel_background_task() sets cancel_event correctly
- request_cancel on nonexistent task is a no-op
- Real execute_async timeout does not overwrite CANCELLED (deterministic
threading.Event sync, no wall-clock sleeps)
- cleanup_background_task removes CANCELLED tasks
- Add task_tool cancellation coverage:
- test_cancellation_calls_request_cancel: assert CancelledError path
calls request_cancel_background_task(task_id)
- test_task_tool_returns_cancelled_message: assert CANCELLED polling
branch emits task_cancelled event and returns expected message
- Fix pre-existing test infrastructure issue: add deerflow.sandbox.security
to _MOCKED_MODULE_NAMES (fixes ModuleNotFoundError for all executor tests)
- Add RUNNING guard to timeout handler in executor.py to prevent
TIMED_OUT from overwriting CANCELLED status
- Add cooperative cancellation granularity comment documenting that
cancellation is only detected at astream iteration boundaries
---------
Co-authored-by: lulusiyuyu <lulusiyuyu@users.noreply.github.com>
* fix(security): disable host bash by default in local sandbox
* fix(security): address review feedback for local bash hardening
* fix(ci): sort live test imports for lint
* style: apply backend formatter
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: normalize ToolMessage structured content in serialization
When models return ToolMessage content as a list of content blocks
(e.g. [{"type": "text", "text": "..."}]), the UI previously displayed
the raw Python repr string instead of the extracted text.
Replace str(msg.content) with the existing _extract_text() helper in
both _serialize_message() and stream() to properly normalize
list-of-blocks content to plain text.
Fixes#1149
Also fixes the same root cause as #1188 (characters displayed one per
line when tool response content is returned as structured blocks).
Added 11 regression tests covering string, list-of-blocks, mixed,
empty, and fallback content types.
* fix(memory): extract text from structured LLM responses in memory updater
When LLMs return response content as list of content blocks
(e.g. [{"type": "text", "text": "..."}]) instead of plain strings,
str() produces Python repr which breaks JSON parsing in the memory
updater. This caused memory updates to silently fail.
Changes:
- Add _extract_text() helper in updater.py for safe content normalization
- Use _extract_text() instead of str(response.content) in update_memory()
- Fix format_conversation_for_update() to handle plain strings in list content
- Fix subagent executor fallback path to extract text from list content
- Replace print() with structured logging (logger.info/warning/error)
- Add 13 regression tests covering _extract_text, format_conversation,
and update_memory with structured LLM responses
* fix: address Copilot review - defensive text extraction + logger.exception
- client.py _extract_text: use block.get('text') + isinstance check (prevent KeyError/TypeError)
- prompt.py format_conversation_for_update: same defensive check for dict text blocks
- executor.py: type-safe text extraction in both code paths, fallback to placeholder instead of str(raw_content)
- updater.py: use logger.exception() instead of logger.error() for traceback preservation
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: preserve chunked structured content without spurious newlines
* fix: restore backend unit test compatibility
---------
Co-authored-by: Exploreunive <Exploreunive@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* refactor: extract shared utils to break harness→app cross-layer imports
Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: split backend/src into harness (deerflow.*) and app (app.*)
Physically split the monolithic backend/src/ package into two layers:
- **Harness** (`packages/harness/deerflow/`): publishable agent framework
package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
models, MCP, skills, config, and all core infrastructure.
- **App** (`app/`): unpublished application code with import prefix `app.*`.
Contains gateway (FastAPI REST API) and channels (IM integrations).
Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure
Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: add harness→app boundary check test and update docs
Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.
Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add config versioning with auto-upgrade on startup
When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.
- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix comments
* fix: update src.* import in test_sandbox_tools_security to deerflow.*
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle empty config and search parent dirs for config.example.yaml
Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
looking next to config.yaml, fixing detection in common setups
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: correct skills root path depth and config_version type coercion
- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
after harness split, file lives at packages/harness/deerflow/skills/
so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
_check_config_version() to prevent TypeError when YAML stores value
as string (e.g. config_version: "1")
- tests: add regression tests for both fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test imports from src.* to deerflow.*/app.* after harness refactor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
