* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(skillscan): detect exfil through instance/dataflow network clients
* fix(skillscan): resolve client handles with Python lexical scopes
_walk_client_nested_scope copied every live handle into a nested scope after
excluding only parameters. That is not Python's name resolution: a class
namespace is not a closure scope for its methods, a function-local binding
shadows an enclosing name across the whole body, and comprehensions bind their
targets in a scope of their own. Benign skills were therefore blocked as
python-env-dump-exfil (CRITICAL) even where the outbound-looking call provably
cannot run on the tracked client.
Derive each scope's bindings lexically instead:
- a class body reads its enclosing scope, but the names it binds are not passed
into the methods defined in it;
- comprehensions are their own scope, so only the outermost iterable is
evaluated outside and every for-target shadows first;
- a function-local prepass excludes names the body binds anywhere, with
global/nonlocal opting back out.
Detection is unchanged where the client really is reachable: a comprehension
calling an unshadowed handle, a method closing over an enclosing function's
handle, and a global-declared module handle all still block.
* fix(skillscan): apply target rebinding in Python evaluation order
The generic ast.iter_child_nodes() fallback yields fields in declaration order,
which for `for`/`async for`, `:=` and `+=` puts the binding target ahead of the
expression Python evaluates to produce it. Visiting the target first dropped the
handle before the call that actually runs on it, so `for s in s.post(host, ...)`
reported nothing even though Python calls post() on the client before binding the
loop target. Match captures bind through a plain `name` string rather than a Name
node, so the Store branch never saw them and a rebound name kept a stale handle.
Assignment expressions inside a comprehension were applied only to the
comprehension-local map, leaving the containing scope stale as well.
Give every bind-after-evaluate construct its own branch:
- `for`/`async for` walk the iterable against the pre-loop binding, then rebind
the target, then walk the body;
- `:=` walks its value first and binds into the containing scope too, since PEP
572 puts a comprehension's walrus target there;
- `+=` walks its value before dropping the target;
- match captures drop the handle, matching how a rebind under `if`/`try` that may
equally not execute is already treated.
The bypasses this closes are the reason detection widens here; the comprehension
walrus and match cases narrow it back where the client provably cannot be the
receiver.
* fix(skillscan): preserve scoped client handle semantics
* fix(skillscan): scan sinks inside assignment target expressions
The Assign/AnnAssign, AugAssign, For/AsyncFor, with, and comprehension
branches evaluated only the value and rebound the target, so a client
call placed in an attribute receiver or a subscript value/index -- both
evaluated at bind time -- was never scanned. os.environ could exfil
through a tracked client while python-env-dump-exfil reported nothing.
Add _walk_client_target_exprs to walk the executable parts of a binding
target (attribute receiver, subscript value+slice, recursing through
tuple/list/starred) in Python evaluation order, without treating Store
name leaves as reads. The AnnAssign annotation expression is walked too.
Name-leaf invalidation is unchanged.
* fix(skillscan): apply assignment targets and annotations in runtime order
The round of assignment-target scanning walked every target then rebound
the names in one batch, and always walked an AnnAssign annotation before
the target. Python instead binds chained and destructured targets left to
right (so `session = out[session.post(...)] = cfg` runs the subscript on
the already-rebound name), evaluates a variable annotation only in module
or class scope (never in a function, never under `from __future__ import
annotations`), and evaluates an executed annotation after the target.
The scanner therefore hard-blocked benign skills.
Bind each target left to right via _bind_client_targets (walk its
executable sub-expressions against the current bindings, then rebind
before the next target), and walk an annotation only for the nodes
_evaluated_annotation_nodes marks as actually evaluated, after the
target. Target-expression scanning and name-leaf invalidation are
otherwise unchanged.
* fix(skillscan): scan client sinks in evaluated function-signature annotations
A function's parameter and return annotations are evaluated at def time
in the enclosing scope, like its decorators and defaults, so a tracked
client sink placed in one is a real egress. _client_scope_prelude walked
the decorators and defaults but not the annotations, so os.environ could
exfil through `def f() -> session.post(host, json=dict(os.environ))`
while python-env-dump-exfil reported nothing.
Record function/async-function defs in _evaluated_annotation_nodes (their
signatures evaluate at def time unless from __future__ import annotations
postpones them) and, for those nodes, add the parameter and return
annotation expressions to the enclosing-scope prelude walk.
* fix(skillscan): match runtime evaluation order for annotations and except handlers
* fix(skillscan): scope try/match branches to their own selection state
* fix(skillscan): propagate branch bindings to everything that observes them
A branch's net effect has to be visible exactly where Python makes it visible.
The walker isolated `except`/`else`/`match` bodies into scope copies and then
discarded them, so a client created on the branch was invisible to `finally`,
to the code after the statement, and to anything defined inside the branch,
while a name the branch replaced stayed a sink receiver.
- `except*` clauses are sequential, not alternatives: thread one scope through
body, clause types and clause bodies in source order instead of reusing the
mutually exclusive copies ordinary `except` needs.
- Fold each `except`/`else`/`match` branch's net effect back into the scope that
`finally` and the following code read, and make the branch scope what nested
definitions close over.
- Keep a fallthrough scope across `match` guards, so a guard that returned false
still hands its side effects to the next case, while pattern captures stay
isolated to their own case.
- Keep an `as` target path-local: Python unbinds it only on the path that ran, so
dropping it for every path erased a live handle where no such handler executed.
Adds a 14-case runtime-oracle regression covering both directions at every
branch site; each of the ten guards was deleted on its own to confirm the test
that pins it goes red.
* fix(skillscan): join alternative branches instead of overwriting one with another
Only one of a statement's alternative branches runs, but the walker folded each
one into a single destructive binding map. Whichever branch was visited last
therefore decided the state: a handler that rebinds the name erased a sibling
that leaves the client in place (missing a client Python really calls), and a
handler that builds one was credited on paths where it never ran (inventing a
CRITICAL sink). Alias targets had the mirror problem, since only key presence
was compared, so replacing `import x as name` on a live branch was ignored.
- Join alternatives into a may-state: a name stays a sink receiver when any
feasible branch leaves it a client, and stops being one only when every
feasible branch replaced it. Alias targets join toward the target that can
still name a constructor.
- Treat each `except*` clause as optional rather than threading every clause
body unconditionally, so a clause whose type never matched cannot erase a
handle the next clause calls.
- Keep the fall-through state (no exception raised, no case matched) as one more
alternative, and drop it only where the source decides the outcome: a literal
always-raising body selecting one handler, a literal exception group choosing
`except*` clauses, a wildcard or literal-equal `match` case.
Also corrects an existing match-capture test that asserted the non-exhaustive
case is benign: the runtime oracle shows the original client still takes the
call on the path where nothing matched. Adds runtime-oracle regressions for both
directions at every alternative site; each of the nine guards was deleted on its
own to confirm the test pinning it goes red.
* fix(skillscan): model feasible conditional client flow
* fix(skillscan): preserve feasible control-flow outcomes
* fix(skillscan): model expression evaluation paths
* fix(skillscan): narrow instance-client detection to lexical statement order
The construction-to-use signal had grown into a path-aware interpreter:
exception selection, except* subgroup consumption, match capture timing,
finally override, comprehension laziness, annotation evaluation order, and
may-state joins over feasible branches. That is the heavyweight analysis
RFC #2634 rules out of Phase 5, and because the signal feeds a CRITICAL
rule it hard-blocks skill installation, so every ambiguity it resolved by
over-reporting cost a benign skill instead of a human review.
Replace it with ordinary statement order over a one-level handle map: a
known constructor bound to a simple name (including `with ... as`), a
direct outbound method call on that name in the same lexical scope,
rebinding invalidation, and name-to-name alias propagation so `s = session`
does not shed the handle. A sink is recorded at the call, so a rebind after
the call cannot retract it.
Compound statements are not interpreted. Every name an if/try/except*/loop/
match may bind is dropped before its bodies are walked, and each body is
walked from an isolated copy. Dropping before rather than after is what
keeps a `finally` that runs after a handler rebound the name, a later
except* clause, and a second loop iteration from reporting a client the
runtime never calls. Bodies are still walked, or wrapping any construction
in `if True:` would be a universal bypass.
Lexical scoping is unchanged: class namespaces are not closures for their
methods, comprehension targets and function-local bindings shadow, and
alias visibility stays per scope.
The cases this gives up are false negatives by construction and are
recorded in #4296, pinned by a test that asserts the runtime really calls
the client while the scanner stays silent.
Verified: 102 SkillScan tests; full suite 8 failed / 7851 passed, the same
network-dependent web-fetch tests that fail on clean main; per-clause red
check 12/12 guards red; 925 repo-owned files scanned branch vs main with 0
new and 0 lost CRITICAL findings; #4158 bypass BLOCKED and #4153 false
positive allowed with 0 findings through the real enforce_static_scan gate.
* test(skillscan): pin closure boundary
* refactor(skillscan): narrow client handle analysis
* fix(skillscan): close client handle correctness gaps
* fix(skillscan): require proven client imports
* Add Monocle tracing
Enable Monocle (OpenTelemetry tracing for LLM apps) with one setup call plus the monocle_apptrace dependency. setup_monocle_telemetry auto-instruments the frameworks already in use and writes traces to .monocle/. Additive; no changes to application logic.
* Config-gate Monocle telemetry in the Gateway lifespan
Addresses review on #4024: moves setup_monocle_telemetry out of agents/__init__ import time into the Gateway lifespan, gated by MonocleTracingConfig (MONOCLE_TRACING env, default off). Warns on the Langfuse/global-OTel-provider conflict and relies on monocle_apptrace's own duplicate-setup guard and existing-provider attach. Pins monocle_apptrace>=0.8.8 (+ uv.lock), adds .monocle/ to .gitignore, adds tests (default-off / toggle-on / no import-time setup), and documents exporters, Okahu, and the VS Code viewer in README, config.example.yaml, and backend/AGENTS.md.
* Clarify Monocle/Langfuse single-provider guidance
Make the docstring, warning, and AGENTS.md consistent with the README: only one library can own the global OpenTelemetry provider; Monocle initializes at startup before Langfuse's per-run handler, so enabling both drops Langfuse's spans — enable one OTel tracer (LangSmith, a callback, coexists fine).
* Address review: optional extra, exporter validation, off-box warning, tests
Responds to the second review round.
- Make monocle_apptrace an optional extra (deerflow-harness[monocle], re-exposed
as deer-flow[monocle]) following the boxlite/tui precedent, so a default
install no longer pulls the OpenTelemetry stack. It stays pinned in the dev
group for the tracing tests, and enabling MONOCLE_TRACING without the extra
raises a clear install error.
- Warn loudly at startup whenever any exporter other than `file` is configured,
since those move prompts, tool inputs/outputs, and completions beyond the
local .monocle/ directory.
- Validate MONOCLE_EXPORTERS against the known exporter names and require
OKAHU_API_KEY when okahu is selected, mirroring the Langfuse pattern.
Validation runs from Monocle's own init (not validate_enabled) so a config
typo can never fail agent runs; errors surface at Gateway startup instead.
- Grow the tests from 5 to 13: caplog coverage for the Langfuse-conflict and
off-box warnings, exporter validation cases, a stronger import-time
regression that asserts the global TracerProvider is not replaced, and a
subprocess double-invoke test exercising the real check_duplicate_setup.
- Docs: config.example.yaml block retitled to a dedicated tracing header;
README documents the [monocle] install and scopes tracing to Gateway runs.
* docs: align Monocle README section with the other tracing providers
Lead with what Monocle is and captures, drop the install step (the dev
group already ships monocle_apptrace via uv sync; unusual installs get
the RuntimeError), and point the missing-package error at the repo-native
command (uv sync --extra monocle / deerflow-harness[monocle]).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: verified Langfuse coexistence, lifespan test, scope docs
Responds to the third review round.
The Langfuse conflict claim was wrong, verified empirically against langfuse
4.5.1 in both init orders: whichever library initializes second reuses the
existing global TracerProvider and attaches its own span processor, so neither
side loses spans. Dropped the warning and its tests, corrected the README,
AGENTS.md, and config.example.yaml statements, and pinned the verified behavior
with test_coexists_with_langfuse (real monocle + real langfuse in a subprocess,
no mocks). One honest caveat documented: both processors see all spans, so
Monocle's exporters also capture Langfuse's spans when both are enabled.
Also from the review:
- Document the Gateway-only scope in AGENTS.md: the lifespan is the sole call
site, so the embedded DeerFlowClient and TUI are not instrumented; embedded
users call setup_monocle_tracing_if_enabled() themselves.
- Add test_gateway_lifespan_initializes_monocle pinning the lifespan wiring.
- Comment why MonocleTracingConfig.is_configured is intentionally coarser than
LangSmith/Langfuse (composite validation lives in validate() at startup).
- Note that monocle_exporters_list takes the comma-separated string as-is.
- Module-level importorskip("monocle_apptrace") so minimal installs collect
the test module cleanly.
* docs: reword Monocle intro sentence
* fix(tests): run the import-time regression in a subprocess
test_no_import_time_setup deleted deerflow.agents* from sys.modules and
re-imported to force __init__ to re-execute. The re-import creates new module
objects, and restoring the old sys.modules entries afterwards leaves the parent
package's attribute bindings pointing at the new ones, so any later test that
resolves a deerflow.agents.* dotted path (monkeypatch.setattr in
test_summarization_middleware, test_thread_data_middleware, and others) failed
with "module 'deerflow.agents' has no attribute ...".
Run the check in a subprocess instead: the import is genuinely fresh, the
assertion is stronger (the provider must still be the SDK-less proxy, proving
nothing was installed at any point), and no module identity leaks into the
rest of the suite.
* Address review: console warning scope, embedded hint, honest naming, doc alignment
Responds to the post-approval review round:
- Scope the off-box exporter warning to the remote exporters (okahu, s3,
blob, gcs): console writes to local stdout and no longer trips it.
config.example.yaml's data-handling note now distinguishes file /
console / remote likewise.
- Rename MonocleTracingConfig.is_configured to is_enabled so the boolean
reads as what it checks; the exporter-dependent credential check stays
in validate(), run at Gateway startup.
- Hint on the embedded path: build_tracing_callbacks() logs a debug line
when MONOCLE_TRACING is set but setup never ran in this process, so
embedded DeerFlowClient/TUI users are not left with silent no-op
tracing. Backed by a process-global setup flag.
- Re-export setup_monocle_tracing_if_enabled from deerflow.tracing,
matching the package convention.
- Note the deliberate fail-open-at-startup contrast with
LangSmith/Langfuse in the lifespan, and the OTel SDK-internals
dependency in the coexistence test.
- Test hygiene: clear MONOCLE_* env in the tracing config/factory
fixtures; reset the setup flag in the monocle test fixture; reword the
README Langfuse-spans claim as the shared-provider inference it is.
- Document that .monocle/ trace files are never rotated or cleaned up.
* fix(tests): pin the factory logger level in the embedded-hint tests
configure_logging() from earlier tests in the full suite pins an explicit
INFO level on the logger hierarchy, so a root-level caplog.at_level(DEBUG)
never sees the factory's debug hint. Scope caplog to
deerflow.tracing.factory so the test is independent of suite ordering.
* Address review: co-export disclosure, lifespan failure test, exporter parse dedup
- Off-box warning now notes that Langfuse's spans are exported too when
both providers are enabled and share the global OTel provider; pinned
both ways by tests.
- Pin the lifespan fail-open contract: a raising Monocle setup is logged
and the Gateway keeps serving (pragma dropped now that the path is
exercised). README notes a config error is reported at startup and
tracing stays off until restart.
- Hoist exporter parsing into MonocleTracingConfig.exporter_list so
validate() and the off-box warning cannot diverge, and note the
upstream coupling on the exporter allow-list.
- Reduce config.example.yaml's Monocle block to a pointer; the capture,
retention, and data-handling detail lives in README's Monocle section.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix subagent total delegation cap
* fix embedded subagent run cap context
* fix subagent cap config consistency
* fix resumed subagent run cap boundary
* fix legacy resume subagent boundary
* address subagent cap review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): show runtime metadata on task cards
* fix(subagents): stop task-card render loop and dedupe model fetches
Address code review on the runtime-metadata cards:
- P1 render loop: the terminal ToolMessage is re-parsed on every
MessageList render and always carries modelName/usage, so the
presence-based setTasks condition fired a fresh state object each
render -> "Maximum update depth exceeded". computeNextSubtask now
returns a value-compared `changed` flag and a pure subtaskNotification()
routes terminal transitions through the deferred after-render path
while skipping no-op re-parses.
- Per-card useModels refetch: add staleTime: Infinity to the ["models"]
query so every subtask card shares one /api/models fetch instead of
refetching on each mount.
* make format
* refactor(subagents): dedupe token-usage validators + tidy event narrowing
Address PR review follow-ups:
- DRY: extract one shared token-usage validator per side. Backend
status_contract.normalize_token_usage() now backs both the terminal
ToolMessage metadata and the subagent.step/.end run events
(step_events.py), and frontend messages/usage.normalizeTokenUsage()
backs both the live task_running event (lifecycle.ts) and the terminal
ToolMessage metadata (subtask-result.ts). Prevents the input/output/
total_tokens validation from drifting across the four former copies.
- Nit: onCustomEvent narrows event.type once instead of re-checking the
object shape per branch; the redundant task_started early-return
(already validated by taskEventToSubtaskUpdate) is dropped.
* fix(subagents): classify LLM error fallbacks as failed (#4041)
* fix(subagents): address #4042 review on LLM error fallback
- clarify _extract_llm_error_fallback docstring: tail-only scan is safe
because subagents append their own terminal message, so the last
AIMessage is never a stale parent-history marker (cross-ref worker.py)
- compute final_result and pop stop_reason only on the COMPLETED branch
so the guard stop-reason pop no longer fires on the discarded FAILED path
- note the error_detail/"LLM request failed" fallbacks are defensive;
the middleware always populates a non-empty content
- add regression test locking the stale parent-history marker invariant
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): inject durable context before compaction
* fix(subagents): coalesce system messages after durable-context injection
Address #4040 review:
- append SystemMessageCoalescingMiddleware innermost on the subagent chain
so the SystemMessage(authority) DurableContextMiddleware injects is merged
into one leading system_message; otherwise the durable fix trades #4039's
assistant-first 400 for a duplicate-system 400 on strict backends
- add a two-system regression guard driving the real builder output through
a strict model; assert exactly one leading SystemMessage
- assert single-leading-system in the compaction integration test too
- update the middleware count/last-element assertion (coalescer is now
unconditionally last, removing the summarization-dependence ambiguity)
- compare _skills_root against posixpath.normpath(container_path)
- document the coalescer on the subagent chain in backend/AGENTS.md
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
* feat: add composer input polishing
* Revert "Merge branch 'main' into feat/input-polish"
This reverts commit 5b6ceccf0db3092bc62fde3b05e7816829601756, reversing
changes made to 45fbc57fef5fa5fd878cf0176c37f3e3bc7ebef6.
* Merge main into feat/input-polish
* style(frontend): format input helper polish guard
* fix(input-polish): address composer polish review findings
Frontend
- Add a cancel affordance to the in-flight polish status pill that calls
abortInputPolishRequest(), so a slow/hung provider no longer hard-locks the
composer for up to stream_chunk_timeout with a page reload (and draft loss)
as the only escape.
- Reset promptHistoryIndexRef/promptHistoryDraftRef when a rewrite is applied
(and on undo), so a stale history-browse index can no longer let the next
ArrowDown silently overwrite the polished draft.
- Disable polishing while an open human-input card is present, matching the
frontend/AGENTS.md rule that composer entry points defer to the card so
card-reply metadata is preserved.
- canPolishInput now reuses parseGoalCommand/parseCompactCommand instead of a
third hardcoded reserved-command regex, and drops the phantom /help entry
(no /help parser exists in the composer), so future builtins only need to be
taught to the existing parsers.
Backend
- Extract the non-graph one-shot LLM path (build model + inject Langfuse
metadata + system/user invoke + text extract) into
deerflow.utils.oneshot_llm.run_oneshot_llm, shared by the input-polish and
suggestions routers so tracing-metadata and invocation shape cannot drift
between the two copies.
- strip_think_blocks gains truncate_unclosed (default True, preserving the
suggestions/goal JSON-prep behavior); input polish passes False so a draft
that legitimately contains a literal <think> substring is no longer
truncated into a partial rewrite or a spurious 503.
- Validate the empty-check and max_chars boundary against the same stripped
view of the draft that is sent to the model, so the user-facing length
boundary and the model input can no longer disagree.
Tests / docs
- Backend: literal-<think> preservation, whitespace-only rejection, and
normalized-length/model-input agreement cases; suggestions tests repoint the
create_chat_model patch to the shared helper module.
- Frontend: helper unit tests updated for the /help/reserved-command change; a
new Playwright case covers cancelling an in-flight polish request.
- backend/AGENTS.md documents the shared one-shot helper and the polish
normalization/think-tag behavior.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Add phase 1 skill static scanning
* Rework SkillScan phase 1 as native scanner
* refactor(skillscan): align phase 1 with trimmed RFC contract
- SecurityFinding: 7 fields (rule_id, severity, file, line, message,
remediation, evidence); category/analyzer derive from the rule_id
prefix, confidence/column/fingerprint/metadata removed
- scan_archive_preflight()/scan_skill_dir() are pure functions: no
ScanContext, no policy schema; CRITICAL-blocks is a code constant and
skill_scan.enabled is applied by enforce_static_scan()/callers
- secret-* evidence is redacted before findings leave the scanner
- de-dup keys on (rule_id, file, line) so repeated occurrences keep
distinct locations for agent self-correction
- cloud-metadata detection consolidated into network-cloud-metadata
- nested zip members get a one-level stdlib magic-byte peek; an
executable member escalates package-nested-archive to CRITICAL
- install metadata sidecar removed (Phase 7 decides if it is needed)
- rule specs moved next to their analyzers; skillscan/rules/ removed
- tests updated + new anchors: redaction, dedup lines, nested-zip
escalation, single cloud-metadata rule, bundled-skill zero-CRITICAL
* fix(skillscan): tighten reverse-shell/secret/archive scan rules from review
Address PR #3033 review feedback on the native SkillScan analyzers:
- Reverse-shell false positives: split shell detection by signal strength
(/dev/tcp/, nc -e stay CRITICAL; bash -i, mkfifo -> new HIGH
shell-reverse-shell-heuristic, warn->LLM). The Python check is now
AST-anchored on real socket.socket/os.dup2/subprocess call sites instead
of raw-text substring matching, so prose/docstrings no longer hard-block.
- Secret evidence: _redact_secret_evidence returns [redacted] with no secret
bytes (was value[:6], which leaked 2 real token bytes past the prefix).
- Archive DoS: cap outer archive member count (_MAX_ARCHIVE_MEMBERS=4096);
scan_archive_preflight early-aborts with a package-too-many-members CRITICAL
finding (routes through the existing blocked->400 fail-closed path).
- shell-destructive-command: broaden the rm -rf matcher to sensitive system
roots (/home, /usr, /*, --no-preserve-root /) while leaving safe subpaths
unflagged.
- Dead code: collapse _decode_text_for_analysis to a single decode path and
drop the unused _TEXT_SUFFIXES set and _has_text_shebang helper.
- local_skill_storage: document why the host_path branch keeps app_config
possibly-None (lazy kill-switch resolution; avoids eager get_app_config in
config-free environments such as CI).
Tests: new negative/positive coverage in test_skillscan_native.py. Full
backend suite 6616 passed, 26 skipped.
* feat: add assistant turn branching
* fix(threads): skip workspace clone when branching from historical turn
Workspace files are not checkpointed, so cloning them onto a branch
rooted at an older assistant turn leaked files created in a later
timeline. Restrict the best-effort workspace copy to branches taken from
the latest turn; historical-turn branches now report
workspace_clone_mode="skipped_historical_turn" and keep only the
restored message history.
* style(frontend): fix prettier formatting in e2e mock-api
Collapse the branch-title normalization chain onto a single line to
satisfy the frontend lint (prettier --check) CI gate.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add workspace change review
* chore: format workspace change files
* fix: optimize workspace change summaries
* style: refine workspace change badge scale
* fix: restore workspace change user context import
* fix(frontend): gate workspace change badge to assistant messages
Only pass run_id to assistant MessageListItems so the workspace-change
badge can never render under a user's prompt, which carries the same
run_id. Assert single badge render in the E2E flow.
* fix(workspace-changes): address review feedback on diff parsing and badge
- Restrict unified-diff header detection to "+++ "/"--- " (trailing space)
in both backend _count_diff_lines and frontend getWorkspaceChangeLineClass
so content lines beginning with +++/--- are counted/styled correctly
- Gate WorkspaceChangeBadge to ai messages so tool messages folded into an
assistant group don't render a duplicate badge
- Add regression tests for both diff-classification fixes
- Apply prettier formatting to workspace-changes files flagged by CI
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add scheduled tasks MVP
* fix: harden scheduled task execution semantics
* feat(scheduled-tasks): preset-driven schedule form with timezone and live preview
Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom)
plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local
for one-time tasks, a timezone selector defaulting to the browser timezone, and a
live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and
edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers).
* feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin
Localize the rest of the scheduled-tasks page (filters, detail pane, actions,
edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four
built-in recipe templates (GitHub Trending, news digest, issue triage, weekly
report) exposed as a chip row that pre-fills title + prompt + schedule. Pin
Playwright locale to en-US so E2E selectors stay stable against i18n. No backend
change, no new deps.
* fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test
Merge with main surfaced three CI failures:
- 0003_scheduled_tasks create_table collided with legacy test seeds that
build from full metadata; guard with inspector.has_table so the revision
no-ops when the table already exists (0004/0005 are already idempotent via
_helpers.py).
- persistence bootstrap concurrency/regression tests pinned HEAD to main's
0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable.
- once-task router test used a fixed past run_at and tripped the
must-be-in-the-future validation; use a future date.
* address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences
- fetchThreadScheduledTasks now checks response.ok like the other fetchers.
- trigger endpoint returns 502 (not 409) when dispatch fails outright, so
clients can distinguish a real conflict from a server-side failure.
- E2E mock normalizes scheduled-task objects with context_mode/last_thread_id
and nullable thread_id, matching the backend contract the UI renders against.
- Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its
revision id (file was renamed in spirit already; filename now follows).
- CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence
after the Scheduler notes so the sections render correctly.
* fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review
* fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review
- defer a once task's terminal status to the run-completion hook; the task
stays running until the real outcome, and a startup sweep cancels once
tasks orphaned by a crash (launch-time 'completed' could stick forever)
- record interrupted runs as a distinct 'interrupted' run status with a
readable message; an interrupted once task ends 'cancelled', not 'failed'
- enforce overlap_policy=skip for fresh_thread_per_run via an active-run
pre-check (same-thread ConflictError can never fire across fresh threads)
- protect terminal run statuses from the late launch-path 'running' write
- honor context.non_interactive only for internally-authenticated callers;
arbitrary clients can no longer strip ask_clarification
- fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the
offset at the resolved instant (once tasks fired an hour late around
spring-forward and the create->edit round-trip diverged)
- drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway
API error helper between channels and scheduled-tasks frontends
* fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics
- scrub internal-only context keys (non_interactive) from the assembled run
config for non-internal callers: gating body.context alone left the same
key smuggle-able through the free-form body.config copied verbatim by
build_run_config
- guard update_after_launch with protect_terminal so the launch bookkeeping
write cannot clobber a once task already finalized by a fast-failing run's
completion hook (parent-row sibling of the run-row guard)
- reject a manual trigger while the task has an active run (409) instead of
launching a duplicate concurrent run on fresh_thread_per_run
- re-arm a terminal once task to enabled when PATCH pushes run_at into the
future; previously the endpoint returned 200 with a next_run_at that could
never be claimed
- make max_concurrent_runs a real global cap: each poll claims only into the
remaining budget of active (queued/running) scheduled runs
- paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the
thread filter of /threads/{id}/scheduled-tasks into SQL
- stamp context.user_id on scheduler-launched runs, matching IM channels, so
user-scoped guardrail providers see the owning user
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(mcp): add per-server tool_call_timeout for MCP tool calls
Add a configurable timeout for individual MCP tool calls to prevent
agent runs from blocking indefinitely when an MCP server becomes
unresponsive (e.g., rate-limited HTTP API, hung subprocess).
Uses the MCP SDK's built-in read_timeout_seconds parameter on
ClientSession.call_tool, which handles the timeout within the
session's own task — avoiding cross-task cancellation issues with
the session pool (ref #3379, #3203).
Config field is named tool_call_timeout (not timeout) to avoid
collision with langchain-mcp-adapters' existing timeout field on
HTTP/SSE connections.
Closes#3840
* fix(mcp): read tool_call_timeout from McpServerConfig, not connection dict
The previous implementation put tool_call_timeout into the connection dict
returned by build_server_params, which langchain's create_session then
passed to _create_stdio_session(), causing TypeError. Now reads the timeout
directly from ExtensionsConfig.mcp_servers where the wrapper is built,
keeping it out of the connection dict entirely.
Fixes P1 bug from review on #3843.
* test(mcp): regression test for tool_call_timeout not leaking into connection dict
Adds two tests:
- test_build_server_params_excludes_tool_call_timeout: verifies the connection
dict returned by build_server_params() does NOT contain tool_call_timeout
- test_stdio_tool_call_timeout_does_not_raise_typeerror: end-to-end test that
get_mcp_tools() with a stdio server having tool_call_timeout configured loads
tools without TypeError from _create_stdio_session()
Regression for PR #3843 P1 bug.
* fix(mcp): only pass read_timeout_seconds when tool_call_timeout is set
When tool_call_timeout is None, don't pass read_timeout_seconds=None to
session.call_tool(). This avoids breaking existing tests that assert on
exact call_tool arguments without the extra kwarg.
* docs(mcp): clarify stdio tool timeout
* implement goal continuations
* fix(goal): address review findings for goal continuations
- goal: key the no-progress breaker on a signature of the latest visible
assistant evidence instead of the evaluator's volatile free-text, so it
actually fires on stalled turns; thread the signature through every
worker persist / no-progress call site
- goal: align _stand_down_reason default caps with should_continue_goal
(8 / 2) so the two gate functions agree on goals missing the fields
- runtime: offload the synchronous checkpointer fallback via
asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop
- frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types)
- frontend: extract pure composer helpers into input-box-helpers.ts with
unit tests (parseGoalCommand, readGoalResponseError, skill suggestions)
- tests: cover the evidence-based no-progress and default-cap behavior
- docs: align backend/AGENTS.md goal paragraph with actual behavior
- e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): hide goal continuation counter until the agent continues
The goal status bar rendered a raw "0/8" before any auto-continuation, which
read as a mysterious score. Now the counter is hidden until
continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining
the auto-continuation cap.
- Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests
- Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(goal): address review findings for goal continuations
Frontend correctness
- Fix the optimistic /goal result permanently shadowing server goal state:
the streamed continuation counter never surfaced for a goal set in-session.
Extract a shared useActiveGoal hook (used by both chat pages) that reconciles
the optimistic copy with server state via a goalReconciliationKey, de-duping
the copy-pasted goal block across the two pages.
- Stop /goal status|clear failures from escaping handleSubmit as unhandled
rejections (handleGoalCommand now returns success; the run only starts when a
goal was actually saved).
- Use a function replacer for the goal-status toast so an objective containing
$&/$1 isn't treated as a replacement pattern.
Backend cleanliness / correctness
- De-duplicate four byte-identical helpers (_call_checkpointer_method,
_message_type, _additional_kwargs, _is_visible_message) by importing them
from runtime.goal instead of re-defining them in the run worker.
- Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has
no tasks field) and document that pending_writes is the durability signal.
- Decompose the 176-line _prepare_goal_continuation_input: extract
_reread_goal_and_checkpoint and a _persist closure so the thread-unchanged
guard and stand-down persistence aren't open-coded three times. Document the
last-writer-wins write-window limitation as a follow-up.
- Add a shared parse_goal_command helper and use it from the TUI and IM-channel
/goal handlers (one place for the status/clear/set semantics).
Tests
- Restore the 11 command-registry tests dropped by the previous goal change
(filter_commands ranking/description, build_registry builtins/skills, resolve
cases) alongside the new goal tests.
- Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal
handler, parse_goal_command, and goalReconciliationKey.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix goal review feedback
* fix goal continuation checkpoint races
* prioritize goal commands while streaming
Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run.
Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check
* preserve goal status during clarification
Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint.
Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* style: format active goal hook
Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate.
Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* fix goal review followups
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(observability): add trace-id correlation and enhanced logging
- add opt-in gateway request trace correlation via X-Trace-Id
- enhance logging with configurable trace_id-aware formatting
- propagate deerflow_trace_id into runtime context and Langfuse metadata
- keep enhanced logging disabled by default to preserve existing behavior
* fix: harden trace correlation wiring
- Make logging enhancement a restart-required startup snapshot and remove per-request config reads from TraceMiddleware
- Restrict trace ids to printable ASCII before writing them to response headers, logs, and Langfuse metadata
- Gate implicit DeerFlowClient trace-id creation behind logging.enhance.enabled while preserving explicit caller opt-in
- Bind embedded client trace context per stream step to avoid generator ContextVar leaks and cross-context reset errors
- Rebind memory update trace ids in Timer/executor worker paths so enhanced logs keep the captured correlation id
- Remove unrelated __run_journal context overwrite from the trace-correlation change set
* fix(gateway): avoid eager app construction on package import
* fix(gateway): avoid config load during app import
Keep Gateway app construction import-safe when config.yaml is absent by
disabling TraceMiddleware only for that construction-time fallback path.
Startup lifespan still performs strict config loading before serving.
