RunCreateRequest declared stream_resumable as None-only, but langgraph_sdk
defaults it to False (not None) and its payload filter only drops None, so
every SDK request carried "stream_resumable": false and was rejected with 422.
False means "no resumable stream", which is what DeerFlow already serves.
This broke every IM channel run (runs.stream and runs.create both send the
field; only runs.wait omits it) and any external langgraph_sdk client. The web
frontend was unaffected because its compatibility wrapper does not send it.
An explicit true still returns 422. The new regression test drives a real SDK
client to capture its default payload and posts that to the HTTP boundary, so a
future non-None SDK default cannot regress this silently.
Fixes#4466
* fix(summarization): own the run model for compaction; bound failure
With summarization.model_name: null the summary model resolved to
config.models[0] while the executing model is selected per run; when they
differ and models[0]'s provider is broken (expired key, quota, outage)
compaction silently failed every triggered turn and context grew unbounded
until the main provider 400s the run (#3103's shape), even though the run's
own model was healthy.
Model ownership is now sourced from the builders, not re-derived at runtime:
- The lead, subagent, and manual /compact builders each pass the resolved run
model into create_summarization_middleware(run_model_name=...). The middleware
no longer reads runtime.context / get_config(), which do not carry a custom
agent's or a subagent's resolved model, so a custom-agent lead run and a
distinct-model subagent now summarize with their own model, not models[0] /
the parent's. Runtime re-resolution and the per-name model cache are removed.
- model_name: null summarizes with the run's own model; an explicitly configured
summary model generates and falls back to the run model on failure. The
fallback is built lazily after the primary fails and its construction is
guarded, so a broken fallback cannot skip a healthy primary or escape the
automatic failure boundary.
Failure is bounded and side-effect-safe:
- An empty or whitespace-only response is treated as a generation failure, not a
valid summary, so compaction never removes all history for an empty replacement.
- compact_state/acompact_state take raise_on_failure independent of force: the
manual /compact path always surfaces a generation failure (even force=false)
and routes it to the existing ContextCompactionFailed path (HTTP 500 ->
frontend error toast) instead of an unconsumed response reason. The automatic
path leaves compaction state unchanged.
- before_summarization hooks fire only after a replacement summary exists.
SummarizationConfig.model_name, config.example.yaml, and docs/summarization.md
document the final lead/subagent/manual ownership rules.
Part of RFC #4346 (section A). Evaluating fraction/triggers against the run
model's profile (profile ownership) is a separate follow-up.
* fix(summarization): manual /compact model ownership + fail-open construct/parse
Manual /compact carried only agent_name, so it derived the run model from the
custom-agent model or config.models[0] and missed the request-selected model the
run path uses (request -> custom-agent -> default). Carry model_name through
ThreadCompactRequest and the frontend compact call, resolve with the same
precedence, and move the custom-agent config read off the event loop (asyncio
.to_thread) with user_id so the strict blocking-IO gate is not bypassed by the
broad except.
Make one summary attempt own its full lifecycle so the fail-open boundary covers
construction and response parsing, not just invocation: build each candidate model
lazily and guarded (a raising constructor falls through to the healthy run model
instead of breaking agent construction), build the model_name:null primary from the
run model rather than config.models[0], and run response text extraction inside the
invocation try so a failing .text accessor falls back instead of escaping compaction.
Adds factory-level constructor-failure, response-extraction-failure (sync/async), and
route-path model-ownership tests.
Queue comments received while a run is active, then submit one deduplicated follow-up after it finishes. Failed drains are requeued and watcher tasks stop cleanly with the channel manager.
* feat(agents): database-backed storage for custom agent definitions
Add an agent_storage.backend switch (default file, behaviour-unchanged) with a
db backend that stores each custom agent as a row in the shared SQL persistence
layer, so a multi-instance deployment sees the same agents on every node
(#4331, #4357). Introduces an AgentStore interface routing all read/write
surfaces, an agents table + migration 0006, startup validation, and a file->db
importer. Follows the thread_meta store / run_events backend-switch /
0003_scheduled_tasks migration patterns; no new dependency.
* fix(agents): make db storage path production-ready (review round 1)
Addresses review feedback on the db/sync agent-storage path:
- sql.py: mirror the async engine's per-connection SQLite PRAGMAs on the sync
engine (busy_timeout=30000, synchronous=NORMAL, foreign_keys=ON, WAL) so both
engines behave identically against the shared DB; guard the engine cache with
a lock (double-checked) so concurrent first-touch cannot build duplicate
engines or register the connect listener twice.
- routers/agents.py + routers/assistants_compat.py: offload the sync-store reads
that ran on the event loop (list/get/check, update's pre-read + legacy guard +
refresh, and assistants_compat's four list routes) via asyncio.to_thread — on
db+postgres each was a network round trip stalling the loop. Writes were
already offloaded.
- file.py: translate the create() mkdir(exist_ok=False) race FileExistsError
into AgentExistsError (router 409, matching SqlAgentStore's IntegrityError
path); correct the _write docstring — per-file atomic replace, two commits
sequential not transactional.
Tests: sync-engine PRAGMA + engine-cache reuse assertions; file create-race ->
AgentExistsError; strict Blockbuster anchor over the read endpoints so a
regression back onto the loop fails CI.
* fix(agents): address round-2 review on the db store path
- update_agent tool: align the docstring/inline comment with FileAgentStore._write.
Cross-field write atomicity is db-only; the file backend commits config then
soul via two sequential os.replace (a crash between them can leave a fresh
config.yaml beside a stale SOUL.md). The dropped partial-write *reporting* is
an intentional tradeoff — the stage-then-replace safety is preserved
(test_update_agent_soul_failure_does_not_replace_config still holds).
- SqlAgentStore.update(): true upsert. Catch IntegrityError on the
insert-on-missing branch, re-fetch and apply, so two concurrent first-time
writes (e.g. two setup_agent handshakes) converge instead of surfacing a raw
UNIQUE(user_id, name) violation as a 500. Symmetric with create().
- get_agent_store(): document the graph-subprocess config-resolution invariant
(the except->file fallback is a genuine no-config path, not a mask for a
misconfigured graph process) and pin it with two tests driving the real
get_app_config() file resolution: db resolves from an on-disk config.yaml,
file fallback when config is unresolvable.
* test(agents): cover SqlAgentStore.update() write-race upsert recovery
Mandatory-TDD test for the round-2 fix in 0680340a: two concurrent first-time
update()s where the loser's insert hits UNIQUE(user_id, name). Deterministically
forces the IntegrityError recovery path by making the first _row probe miss the
committed winner, and asserts last-writer-wins instead of a surfaced 500.
* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
binding, Internal Auth) and group IM bindings with direct HTTP under a
shared platform-trust model. Explain Internal Auth trust boundaries:
DeerFlow validates only the platform token, does not manage end-user
accounts in users, and relies on the channel to maintain owner identity.
Cross-link AUTH_DESIGN, API.md, and docs README
* docs(api): document stateless /api/runs/stream endpoint
Record that clients can start a conversation without pre-creating a thread,
and that Gateway returns thread_id and run_id via the Content-Location header.
Co-authored-by: yym36991@gmail.com
* docs(api): show on_run_created for stateless stream continuation
The Python SDK example now captures thread_id/run_id via on_run_created
before documenting the follow-up call with config.configurable.thread_id,
matching the TS/cURL examples and langgraph-sdk 0.3.x behavior.
Co-authored-by: yym36991@gmail.com
* fix subagent total delegation cap
* fix embedded subagent run cap context
* fix subagent cap config consistency
* fix resumed subagent run cap boundary
* fix legacy resume subagent boundary
* address subagent cap review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
* Add phase 1 skill static scanning
* Rework SkillScan phase 1 as native scanner
* refactor(skillscan): align phase 1 with trimmed RFC contract
- SecurityFinding: 7 fields (rule_id, severity, file, line, message,
remediation, evidence); category/analyzer derive from the rule_id
prefix, confidence/column/fingerprint/metadata removed
- scan_archive_preflight()/scan_skill_dir() are pure functions: no
ScanContext, no policy schema; CRITICAL-blocks is a code constant and
skill_scan.enabled is applied by enforce_static_scan()/callers
- secret-* evidence is redacted before findings leave the scanner
- de-dup keys on (rule_id, file, line) so repeated occurrences keep
distinct locations for agent self-correction
- cloud-metadata detection consolidated into network-cloud-metadata
- nested zip members get a one-level stdlib magic-byte peek; an
executable member escalates package-nested-archive to CRITICAL
- install metadata sidecar removed (Phase 7 decides if it is needed)
- rule specs moved next to their analyzers; skillscan/rules/ removed
- tests updated + new anchors: redaction, dedup lines, nested-zip
escalation, single cloud-metadata rule, bundled-skill zero-CRITICAL
* fix(skillscan): tighten reverse-shell/secret/archive scan rules from review
Address PR #3033 review feedback on the native SkillScan analyzers:
- Reverse-shell false positives: split shell detection by signal strength
(/dev/tcp/, nc -e stay CRITICAL; bash -i, mkfifo -> new HIGH
shell-reverse-shell-heuristic, warn->LLM). The Python check is now
AST-anchored on real socket.socket/os.dup2/subprocess call sites instead
of raw-text substring matching, so prose/docstrings no longer hard-block.
- Secret evidence: _redact_secret_evidence returns [redacted] with no secret
bytes (was value[:6], which leaked 2 real token bytes past the prefix).
- Archive DoS: cap outer archive member count (_MAX_ARCHIVE_MEMBERS=4096);
scan_archive_preflight early-aborts with a package-too-many-members CRITICAL
finding (routes through the existing blocked->400 fail-closed path).
- shell-destructive-command: broaden the rm -rf matcher to sensitive system
roots (/home, /usr, /*, --no-preserve-root /) while leaving safe subpaths
unflagged.
- Dead code: collapse _decode_text_for_analysis to a single decode path and
drop the unused _TEXT_SUFFIXES set and _has_text_shebang helper.
- local_skill_storage: document why the host_path branch keeps app_config
possibly-None (lazy kill-switch resolution; avoids eager get_app_config in
config-free environments such as CI).
Tests: new negative/positive coverage in test_skillscan_native.py. Full
backend suite 6616 passed, 26 skipped.
ModelConfig is `extra="allow"`, so a config key like `api_base` (which
config.example.yaml uses for other model classes, e.g. PatchedChatDeepSeek and
Moonshot) gets copied onto a `langchain_openai:ChatOpenAI` model by users.
LangChain's OpenAI client does not reject the unknown kwarg — it transfers it
into `model_kwargs` (with a UserWarning), which is then spread into every
`Completions.create()` call and rejected by the OpenAI SDK at REQUEST time with
an opaque `unexpected keyword argument 'api_base'` error. The endpoint override
is also silently dropped, so the model targets the wrong base URL.
Changes in factory.py, mirroring the existing OpenAI-compatible helpers:
- `_normalize_openai_base_url`: renames `api_base` -> `base_url` for the
OpenAI-compatible family (ChatOpenAI + PatchedChatOpenAI); when an endpoint
key is already present, drops the alias with a warning. Runs before the
stream_usage/stream_chunk_timeout heuristics so they see the canonical key.
- `_warn_unknown_model_settings`: scoped to the same OpenAI-compatible family
(where the model_kwargs divert-and-crash actually happens and the field/alias
set is accurate), logs an actionable warning for unrecognized config keys.
- The three OpenAI-compatible helpers now share `_OPENAI_COMPAT_USE_PATHS`
instead of disagreeing on the literal class string.
Adds a note to docs/CONFIGURATION.md and 9 tests covering normalization
(ChatOpenAI + PatchedChatOpenAI, both-set precedence for base_url and
openai_api_base, non-OpenAI class untouched, no-op when unset) and the
unknown-key warning (fires on a typo, silent on a clean config and on
non-OpenAI providers like ChatAnthropic).
* feat: add scheduled tasks MVP
* fix: harden scheduled task execution semantics
* feat(scheduled-tasks): preset-driven schedule form with timezone and live preview
Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom)
plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local
for one-time tasks, a timezone selector defaulting to the browser timezone, and a
live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and
edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers).
* feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin
Localize the rest of the scheduled-tasks page (filters, detail pane, actions,
edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four
built-in recipe templates (GitHub Trending, news digest, issue triage, weekly
report) exposed as a chip row that pre-fills title + prompt + schedule. Pin
Playwright locale to en-US so E2E selectors stay stable against i18n. No backend
change, no new deps.
* fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test
Merge with main surfaced three CI failures:
- 0003_scheduled_tasks create_table collided with legacy test seeds that
build from full metadata; guard with inspector.has_table so the revision
no-ops when the table already exists (0004/0005 are already idempotent via
_helpers.py).
- persistence bootstrap concurrency/regression tests pinned HEAD to main's
0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable.
- once-task router test used a fixed past run_at and tripped the
must-be-in-the-future validation; use a future date.
* address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences
- fetchThreadScheduledTasks now checks response.ok like the other fetchers.
- trigger endpoint returns 502 (not 409) when dispatch fails outright, so
clients can distinguish a real conflict from a server-side failure.
- E2E mock normalizes scheduled-task objects with context_mode/last_thread_id
and nullable thread_id, matching the backend contract the UI renders against.
- Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its
revision id (file was renamed in spirit already; filename now follows).
- CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence
after the Scheduler notes so the sections render correctly.
* fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review
* fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review
- defer a once task's terminal status to the run-completion hook; the task
stays running until the real outcome, and a startup sweep cancels once
tasks orphaned by a crash (launch-time 'completed' could stick forever)
- record interrupted runs as a distinct 'interrupted' run status with a
readable message; an interrupted once task ends 'cancelled', not 'failed'
- enforce overlap_policy=skip for fresh_thread_per_run via an active-run
pre-check (same-thread ConflictError can never fire across fresh threads)
- protect terminal run statuses from the late launch-path 'running' write
- honor context.non_interactive only for internally-authenticated callers;
arbitrary clients can no longer strip ask_clarification
- fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the
offset at the resolved instant (once tasks fired an hour late around
spring-forward and the create->edit round-trip diverged)
- drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway
API error helper between channels and scheduled-tasks frontends
* fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics
- scrub internal-only context keys (non_interactive) from the assembled run
config for non-internal callers: gating body.context alone left the same
key smuggle-able through the free-form body.config copied verbatim by
build_run_config
- guard update_after_launch with protect_terminal so the launch bookkeeping
write cannot clobber a once task already finalized by a fast-failing run's
completion hook (parent-row sibling of the run-row guard)
- reject a manual trigger while the task has an active run (409) instead of
launching a duplicate concurrent run on fresh_thread_per_run
- re-arm a terminal once task to enabled when PATCH pushes run_at into the
future; previously the endpoint returned 200 with a next_run_at that could
never be claimed
- make max_concurrent_runs a real global cap: each poll claims only into the
remaining budget of active (queued/running) scheduled runs
- paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the
thread filter of /threads/{id}/scheduled-tasks into SQL
- stamp context.user_id on scheduler-launched runs, matching IM channels, so
user-scoped guardrail providers see the owning user
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: emit structured runtime metadata
* fix: avoid subagent import cycle in replay gateway
* fix: preserve legacy subtask result parsing
* refactor: tighten runtime metadata contracts
* fix(middleware): keep recovery hint on task exception wrapper content
The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.
* fix(subagents): require lowercase hex for result_sha256 reader
Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Images older than all-in-one-sandbox 1.9.x have no /v1/bash/* routes, so
every env-bearing command (skills declaring required-secrets) surfaced a
raw nginx 404 that the model kept retrying. Detect the 404, remember the
capability gap per sandbox instance, and return an actionable error that
names the minimum image version and the sandbox.image remediation.
No fallback through the legacy shell path on purpose: /v1/shell/exec has
no env parameter, and every workaround puts the secret values back into
the command string or on disk — the exact leak surfaces the
request-scoped secrets design closed.
Closes#3921
* feat(mcp): add per-server tool_call_timeout for MCP tool calls
Add a configurable timeout for individual MCP tool calls to prevent
agent runs from blocking indefinitely when an MCP server becomes
unresponsive (e.g., rate-limited HTTP API, hung subprocess).
Uses the MCP SDK's built-in read_timeout_seconds parameter on
ClientSession.call_tool, which handles the timeout within the
session's own task — avoiding cross-task cancellation issues with
the session pool (ref #3379, #3203).
Config field is named tool_call_timeout (not timeout) to avoid
collision with langchain-mcp-adapters' existing timeout field on
HTTP/SSE connections.
Closes#3840
* fix(mcp): read tool_call_timeout from McpServerConfig, not connection dict
The previous implementation put tool_call_timeout into the connection dict
returned by build_server_params, which langchain's create_session then
passed to _create_stdio_session(), causing TypeError. Now reads the timeout
directly from ExtensionsConfig.mcp_servers where the wrapper is built,
keeping it out of the connection dict entirely.
Fixes P1 bug from review on #3843.
* test(mcp): regression test for tool_call_timeout not leaking into connection dict
Adds two tests:
- test_build_server_params_excludes_tool_call_timeout: verifies the connection
dict returned by build_server_params() does NOT contain tool_call_timeout
- test_stdio_tool_call_timeout_does_not_raise_typeerror: end-to-end test that
get_mcp_tools() with a stdio server having tool_call_timeout configured loads
tools without TypeError from _create_stdio_session()
Regression for PR #3843 P1 bug.
* fix(mcp): only pass read_timeout_seconds when tool_call_timeout is set
When tool_call_timeout is None, don't pass read_timeout_seconds=None to
session.call_tool(). This avoids breaking existing tests that assert on
exact call_tool arguments without the extra kwarg.
* docs(mcp): clarify stdio tool timeout
* implement goal continuations
* fix(goal): address review findings for goal continuations
- goal: key the no-progress breaker on a signature of the latest visible
assistant evidence instead of the evaluator's volatile free-text, so it
actually fires on stalled turns; thread the signature through every
worker persist / no-progress call site
- goal: align _stand_down_reason default caps with should_continue_goal
(8 / 2) so the two gate functions agree on goals missing the fields
- runtime: offload the synchronous checkpointer fallback via
asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop
- frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types)
- frontend: extract pure composer helpers into input-box-helpers.ts with
unit tests (parseGoalCommand, readGoalResponseError, skill suggestions)
- tests: cover the evidence-based no-progress and default-cap behavior
- docs: align backend/AGENTS.md goal paragraph with actual behavior
- e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): hide goal continuation counter until the agent continues
The goal status bar rendered a raw "0/8" before any auto-continuation, which
read as a mysterious score. Now the counter is hidden until
continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining
the auto-continuation cap.
- Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests
- Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(goal): address review findings for goal continuations
Frontend correctness
- Fix the optimistic /goal result permanently shadowing server goal state:
the streamed continuation counter never surfaced for a goal set in-session.
Extract a shared useActiveGoal hook (used by both chat pages) that reconciles
the optimistic copy with server state via a goalReconciliationKey, de-duping
the copy-pasted goal block across the two pages.
- Stop /goal status|clear failures from escaping handleSubmit as unhandled
rejections (handleGoalCommand now returns success; the run only starts when a
goal was actually saved).
- Use a function replacer for the goal-status toast so an objective containing
$&/$1 isn't treated as a replacement pattern.
Backend cleanliness / correctness
- De-duplicate four byte-identical helpers (_call_checkpointer_method,
_message_type, _additional_kwargs, _is_visible_message) by importing them
from runtime.goal instead of re-defining them in the run worker.
- Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has
no tasks field) and document that pending_writes is the durability signal.
- Decompose the 176-line _prepare_goal_continuation_input: extract
_reread_goal_and_checkpoint and a _persist closure so the thread-unchanged
guard and stand-down persistence aren't open-coded three times. Document the
last-writer-wins write-window limitation as a follow-up.
- Add a shared parse_goal_command helper and use it from the TUI and IM-channel
/goal handlers (one place for the status/clear/set semantics).
Tests
- Restore the 11 command-registry tests dropped by the previous goal change
(filter_commands ranking/description, build_registry builtins/skills, resolve
cases) alongside the new goal tests.
- Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal
handler, parse_goal_command, and goalReconciliationKey.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix goal review feedback
* fix goal continuation checkpoint races
* prioritize goal commands while streaming
Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run.
Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check
* preserve goal status during clarification
Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint.
Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* style: format active goal hook
Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate.
Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* fix goal review followups
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
build_run_config() copied every top-level request key (except
configurable/context) verbatim into the LangGraph RunnableConfig, including
recursion_limit. The server default of 100 was fully overridable by the caller
with no upper bound, so a request like {"config": {"recursion_limit": 100000000}}
could make a single run execute effectively unbounded LangGraph super-steps
(each >= 1 LLM call), enabling runaway API cost / DoS.
Validate the client value server-side and clamp it into a safe range:
- valid positive ints are capped at a configurable ceiling
(AppConfig.max_recursion_limit, default 1000 to match the existing
frontend/public-skill default so legitimate deep runs are unaffected)
- invalid/non-positive/bool/None values fall back to the 100 server default
- applied on both the configurable and the LangGraph >= 0.6.0 context paths
- WARNING logged on clamp for observability
Add unit tests (including a configurable-ceiling case), expose
max_recursion_limit in config.example.yaml (config_version 16), and document
the ceiling/fallback in backend/docs/API.md.
Co-authored-by: DengY11 <DengY11@users.noreply.github.com>
* feat(community): add Crawl4AI web_fetch provider
Crawl4AI is a self-hosted, no-API-key web fetcher: it runs headless
Chromium and returns server-cleaned "fit" markdown directly via its
POST /md endpoint, so no client-side readability extraction is needed.
It sits alongside the existing self-hosted Browserless provider.
- deerflow.community.crawl4ai: async Crawl4AiClient + web_fetch_tool
(reads base_url/timeout_s/token/filter from config; "Error:" string
convention; 4096-char cap), mirroring the browserless provider
- tests: 17 unit cases (success, HTTP error, success:false, empty,
timeout, request error, token header, truncation, config reads)
- config.example.yaml: commented web_fetch example
- doctor: register as a no-key (free) web_fetch provider
- setup wizard: add to WEB_FETCH_PROVIDERS (no API key)
- docs: README, CONTRIBUTING, CONFIGURATION, AGENTS provider lists
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(community): address Crawl4AI provider review feedback
- timeout: robust _coerce_timeout (bool / non-numeric -> default) mirroring
jina, so 'timeout: off' no longer becomes 0.0 and times out every request
- read web_fetch config once per invocation and pass values into the client,
so a concurrent hot-reload can't split base_url from filter
- rename config key timeout_s -> timeout to match jina/infoquest (the
default providers); update config.example.yaml + setup wizard
- validate + normalize the markdown filter against {fit,raw,bm25,llm};
unknown values fall back to fit with a warning instead of an opaque HTTP 400
- client: a non-JSON 200 body (reverse proxy / auth wall) now reports the
content-type + snippet instead of a generic JSONDecodeError
- tests: 22 cases (added non-JSON-200, _coerce_timeout, _coerce_filter,
invalid-filter fallback, read-config-once)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Add Brave image search community tool
* fix(community): length-cap Brave web_search queries
Apply _clean_query in web_search_tool so over-long queries are trimmed
to Brave's 400-char limit before the API call, matching image_search_tool
and avoiding HTTP 422 from the Brave Search API.
* fix(community): harden Brave image search SSRF guard and dimension mapping
Address PR review findings:
- Catch ValueError from urlparse so a malformed bracketed-IPv6 URL skips
one item instead of crashing the whole image_search call
- Reject IPv6 literals embedding a non-global IPv4 (IPv4-mapped, 6to4,
NAT64, IPv4-compatible), closing the loopback/private SSRF bypass
- Report width/height from the dict of the URL actually returned, so a
surviving thumbnail no longer reports the dropped original's dimensions
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: generate title for interrupted first turn
* test(title): cover partial-exchange + dict-form messages
Harden the interrupted-run fallback path added in 19fc34fd:
- TitleMiddleware._should_generate_title now accepts a lone first-turn
user message when allow_partial_exchange=True, so the worker can still
derive a title if cancellation lands before any AI chunk is checkpointed.
- runtime/runs/worker._ensure_interrupted_title computes the next
checkpoint step defensively (treat missing/non-int step as 0) and
renames a shadowed ckpt_config local for readability.
- Add four unit tests in tests/test_title_middleware_core_logic.py:
partial-exchange allows user-only, partial-exchange still respects an
existing title, dict-form messages are recognized, and the sync
fallback path derives a title from dict-form messages — matching what
channel_values stores in the checkpoint.
Refs #3859.
* fix: persist interrupted-title via channel_versions bump
Address PR #3874 review feedback: ``_ensure_interrupted_title`` previously
called ``aput(..., new_versions={})``. LangGraph's DB-backed savers
(``PostgresSaver`` and the v4 ``SqliteSaver`` blob layout) strip inline
``channel_values`` from ``put`` and only persist blobs for channels named
in ``new_versions`` — so the fallback ``title`` channel was dropped on
read-back and ``threads_meta.display_name`` stayed ``"Untitled"`` after
refresh on those backends. The original in-memory e2e passed because
``InMemorySaver`` keeps the inline snapshot verbatim.
Fix mirrors ``_rollback_to_pre_run_checkpoint`` in the same file: bump
``channel_versions["title"]`` (via ``checkpointer.get_next_version`` when
available, else int/string fallbacks), persist the new version on the
checkpoint, and declare it in ``new_versions`` so the DB savers actually
write the blob.
Regression coverage in ``tests/test_run_worker_rollback.py``:
- ``test_ensure_interrupted_title_bumps_channel_version_and_declares_it_in_new_versions``
— exact ``aput`` invariants: ``new_versions == {"title": 1}``, the
written checkpoint's ``channel_versions["title"]`` is bumped, and the
pre-existing ``messages`` version is preserved.
- ``test_ensure_interrupted_title_bumps_existing_string_version`` —
string-shaped prior version (some savers use UUID-style versions);
bumped value must differ from the prior, no overwrite-in-place.
- ``test_ensure_interrupted_title_skips_when_title_already_set`` — title
short-circuit; no extra ``aput``.
- ``test_ensure_interrupted_title_returns_none_when_no_checkpoint`` —
no checkpoint yet; returns ``None`` without writing.
- ``test_ensure_interrupted_title_round_trip_with_real_sqlite_checkpointer``
— full round-trip against a real ``AsyncSqliteSaver`` on a disk-backed
DB, then closes and re-opens the saver to simulate a fresh
connection. The fallback title must still be present on the second
``aget_tuple``. This is the exact scenario the review flagged.
Validated locally with the full backend suite: 5195 passed, 18 skipped.
Refs #3859. Addresses review on #3874.
* test(worker, title): harden interrupted-title fallback for every saver
Defensive coverage on top of the channel_versions fix (commit 05253957),
addressing edge cases surfaced during a second-pass review of #3874.
Worker:
- Extract version bump into ``_bump_channel_version(checkpointer, current)``
with explicit fallbacks for int / float / numeric-string / UUID-shaped
string / None / bool, AND a wrap-around defense when the saver's
``get_next_version`` raises or returns an unchanged value. The
invariant is: returned version MUST differ from the prior. Without
this, a saver bug (or a custom backend) could leave
``new_versions={"title": v}`` no-op on DB savers — the very class of
bug the original review pointed out.
Title middleware:
- Coerce ``state.get("messages")`` from ``None`` to ``[]`` on both
``_should_generate_title`` and ``_build_title_prompt``. A
partially-initialized checkpoint can carry ``messages=None`` on the
channel_values channel (the worker reads raw channel_values, not
BaseMessages), and the default kwarg only protects against a missing
key. Repro: ``TypeError: 'NoneType' object is not iterable`` from
the next() generator — confirmed by reverting the fix and watching
``test_*_handles_none_messages_channel`` go red.
Tests (TDD-verified red→green for the new asserts):
- ``test_run_worker_rollback.py``:
* ``_bump_channel_version`` — 8 tests covering every version type
(int, float, numeric string, UUID-style string, None, bool) and
every saver-side fault mode (no ``get_next_version`` / raising /
stuck on identity).
* ``test_ensure_interrupted_title_*`` — 5 additional helper
boundary tests: title.enabled=false short-circuit; empty
messages list; messages=None; aput-error propagation (helper
contract: caller swallows, not the helper); idempotency on a
real InMemorySaver across two invocations.
* ``test_ensure_interrupted_title_preserves_non_title_channel_versions``
— pins that ``new_versions`` only contains ``"title"`` and that
other channels' versions are untouched (regression anchor for a
sloppier draft that bumped every channel).
* ``test_worker_finally_block_swallows_helper_exceptions`` — pins
the integration contract: even if the helper raises, the worker's
threads_meta status sync still runs and ``publish_end`` is still
awaited so the SSE stream closes cleanly.
- ``test_title_middleware_core_logic.py``:
* 4 additional tests: ``messages=None`` on both
``_should_generate_title`` and ``_build_title_prompt``; the
``role: user`` / ``role: assistant`` (OpenAI-style) dict
normalization; partial-exchange path with a dict-form message.
Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
→ 5215 passed, 18 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Red/green TDD verification: temporarily reverted the
``new_versions={}`` fix → 4 new tests went red as expected; restored
and the suite is green again. Same red/green dance for the
``messages=None`` coercion.
Refs #3859. Addresses second-pass review on #3874.
* fix(title): ignore dict context reminders in fallback
* fix(worker): link interrupted-title checkpoint to its parent
The title-bump checkpoint written by ``_ensure_interrupted_title`` was
landing without a ``parent_checkpoint_id`` — a real orphan in the
LangGraph history graph. Reproduction (disk-backed AsyncSqliteSaver):
[seed] checkpoint_id = 1f173dbc...
[helper] wrote title = "Why is the sky blue?"
[issue 1] new checkpoint = 1f173dbc..., parent = None
[issue 1] is new checkpoint orphaned? True
Root cause: ``_ensure_interrupted_title`` built ``write_config`` as
``{"thread_id": ..., "checkpoint_ns": ...}`` only. ``BaseCheckpointSaver``
implementations read ``configurable.checkpoint_id`` from that config as
the *parent* id when inserting (see ``langgraph/checkpoint/sqlite/aio.py``
``aput``: ``config["configurable"].get("checkpoint_id")`` becomes the
``parent_checkpoint_id`` column). With no value, the saver writes NULL —
the new checkpoint is a tree root.
Consequences:
- Any future LangGraph ``runs.resume_from`` / time-travel feature has no
backward edge to walk past the title-bump.
- History-visualization UIs built on ``alist()`` render the title-bump as
a sibling of the prior checkpoint, not its descendant.
Fix: read ``checkpoint_id`` off the tuple's own config and thread it into
``write_config["configurable"]["checkpoint_id"]`` before calling ``aput``,
the same pattern every middleware-driven write uses.
Three new regression tests against real ``AsyncSqliteSaver`` (disk-backed,
fresh connections so we exercise the on-disk read path):
- ``test_ensure_interrupted_title_links_new_checkpoint_to_its_parent`` —
asserts ``latest.parent_config["configurable"]["checkpoint_id"]`` equals
the seeded checkpoint id. TDD red-green verified: reverting the fix
flips this test red with ``AssertionError: title-bump checkpoint must
have a parent_config``.
- ``test_ensure_interrupted_title_appears_in_history_with_audit_marker``
— pins the audit contract: the title-bump entry in ``alist()`` carries
``metadata.source == "update"`` and ``metadata.writes`` contains
``runtime_interrupt_title``. This is a deliberate design choice — we
do NOT hide the entry from history (audit trail belongs in the saver),
but its source and writes marker MUST be unambiguous so UIs/tools can
identify it.
- ``test_ensure_interrupted_title_survives_immediate_next_turn`` —
cancel → immediate user follow-up scenario. Simulates the agent's next
turn appending a (user, ai) pair without touching the title channel,
then opens a fresh saver and verifies the title is still present after
the next-turn checkpoint write. Pins the channel-version-blob
invariant established by commit 05253957 — without the
``new_versions={"title": v}`` declaration there, the title blob would
vanish from the DB and this test would read back ``None``.
Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
→ 5222 passed, 15 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Reproduction script confirms ``parent_checkpoint_id`` is now non-null
and the next-turn read-back preserves the fallback title.
Refs #3859.
* Revert "fix(worker): link interrupted-title checkpoint to its parent"
This reverts commit c763ed9334781db1acdce0f5f33d663d8d5f80ff.
* test: trim over-engineered test coverage
Reduce review surface area on PR #3874 by dropping defensive tests that
don't pin a real invariant. After self-review:
- ``_bump_channel_version``: 8 tests → 2 (happy path + saver-error
fallback). Dropped float / bool / numeric-string / UUID-string /
missing-get-next-version / stuck-get-next-version branches — those
are speculative scaffolding for savers we don't ship.
- ``_ensure_interrupted_title``: dropped ``returns_none_when_title_disabled``,
``returns_none_with_no_user_message``, ``returns_none_when_no_checkpoint``
— boundary guards already exercised by the e2e test and the
``handles_none_messages_channel`` regression anchor.
Net: -107 lines of test code. Remaining coverage still pins every
red-green-verified invariant (channel_versions bump, string-version
bump, idempotency, sqlite round-trip, non-title channel preservation,
aput-error contract, worker finally swallowing, partial-exchange).
Verification: 5209 passed, 15 skipped.
* fix: harden interrupted title finalization
* fix: serialize interrupted title finalization
* fix: preserve interrupt semantics during title finalization
* fix: preserve delayed interrupted title recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
Adds ``deerflow.community.e2b_sandbox.E2BSandboxProvider`` with parity
to AioSandboxProvider: metadata-keyed per-thread persistence, server-
side idle timeout, warm-pool reclaim with liveness checks, /mnt/user-data
bootstrap symlinks, dead-sandbox auto-rebuild, and release-time mirror
of agent outputs back to the host artifact directory.
Signed-off-by: joey <zchengjoey@gmail.com>
* feat(tui): add Hermes-like terminal workbench backed by DeerFlowClient
Implements the `deerflow` TUI from RFC #3540: a terminal-native, embedded
workbench over the existing harness (no Gateway/frontend/nginx/Docker), built
Python-native with Textual and learning UX patterns from tao-pi.
Architecture — every layer except the Textual app is pure and unit-tested:
- view_state.py: ViewState + reduce(state, action), the testable heart
- runtime.py: StreamEvent -> reducer actions (pure translate + threaded driver)
- message_format / command_registry / input_history / render / theme: pure
- app.py: Textual App; runs the sync DeerFlowClient.stream() on a worker thread
and marshals actions back to the UI thread. Slash command palette, model and
thread modal pickers, ↑/↓ history, Ctrl+C interrupt, TTY-aware fallback.
- cli.py: pure launch-mode planning + headless --print/--json + `deerflow`
console script (textual is an optional [tui] extra; degrades to headless help)
Web UI visibility (the RFC's key decision): persistence.py writes a threads_meta
row under the local default user into the same DB the Gateway reads, so terminal
sessions appear in the Web UI sidebar without running the Gateway. Best-effort,
no-op on the memory backend; all DB work on one long-lived background loop.
Tests: 95 TUI tests — pure layers via pytest, app/palette/overlays via Textual's
pilot harness with a fake session, and a threads_meta read/write round-trip.
ruff clean; respects the harness->app import boundary. Docs: backend/docs/TUI.md
plus CLAUDE.md/README updates and preview screenshots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): de-duplicate streamed assistant text and tool cards; keep Tab in composer
Self-test surfaced three issues, all root-caused to consuming non-strict
streaming from DeerFlowClient (proven by the client's own
test_dedup_requires_messages_before_values_invariant, which shows the client can
re-emit a message id's full content twice):
- Assistant text was doubled (e.g. "answer answer") because the reducer blindly
concatenated same-id deltas. Now merges by content: a re-send or cumulative
snapshot replaces; only genuine increments append.
- Tool activity showed duplicate and empty "gear" cards from partial/re-emitted
tool-call chunks. ToolStarted now de-dupes by tool_call_id, drops id-less
noise chunks, and fills the name on a later chunk; a tool result with no prior
card still surfaces as a completed card.
- Tab moved focus off the composer to the scroll region (felt like broken cursor
logic). Tab is now consumed by the composer (completes a command when the
palette is open, no-op otherwise).
Adds reducer tests for each case plus a Tab-focus test; 102 TUI tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): make Esc interrupt an active run (matches the status hint)
The status line advertised "esc interrupt" but Esc was only wired to close the
slash palette, so it did nothing during a run. Esc now: closes the palette when
open, interrupts the active run when streaming, and is a no-op when idle. The
interrupt logic is shared with Ctrl+C via _interrupt_run(). Adds a regression
test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): stop prior answers duplicating on threads with history
On a thread with history, DeerFlowClient re-emits every prior message on each
new turn (its streamed_ids dedup is per-stream-call), and a re-emitted older
message can arrive after a newer message has already started. The reducer only
matched the *most recent* assistant row by id and otherwise appended, so each
re-emitted older answer was duplicated verbatim at the end of the transcript.
Match an assistant row by id anywhere in the transcript and merge in place.
Tool cards already de-dupe by call id globally, so they were unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): correct CJK cursor drift in the composer
Confirmed a Textual Input bug (latest 8.2.7): Input._cursor_offset adds an
unconditional +1 at the end of the value, overshooting by one cell after
double-width (CJK) characters. That misplaces the hardware/IME cursor — the
drift seen when typing Chinese in iTerm2 (the on-screen block cursor, drawn
separately in render_line, is fine; English doesn't use an IME so it looks
correct). Reproduced with a bare Input, so it's upstream, not our layout.
Add ComposerInput(Input) overriding _cursor_offset to the true cell position and
use it for the composer. Numeric tests pin the CJK end/mid and ASCII cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(tui): render finalized assistant messages as Markdown
The transcript showed raw Markdown (literal **bold**, ## headings, - lists,
links). Finalized assistant messages now render as Rich Markdown — headings,
bold/italic, lists, inline code + code blocks, blockquotes, horizontal rules and
links — with the ● speaker marker aligned to the top of the body.
The actively-streaming message stays plain text so partial Markdown doesn't
reflow/jump, then snaps to its rendered form when the run ends. Transcript
re-renders are coalesced on a ~60ms timer (dirty flag) so per-token Markdown
re-parsing stays smooth on long threads. Tests cover both the rendered and the
streaming-plain paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style(tui): apply ruff format
CI lint runs `ruff format --check` via uvx (latest ruff); apply the formatter so
the lint-backend job passes. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(tui): address code-quality review comments
From github-code-quality[bot] on #3760:
- runtime.py: give the `_ClientLike` Protocol method a docstring body instead of
a bare `...` (flagged as a no-effect statement), matching the harness
convention for Protocol stubs (e.g. SafetyTerminationDetector).
- test_tui_cli_main.py: drop the unnecessary `lambda: _FakeSession()` wrappers in
monkeypatch.setattr; pass `_FakeSession` directly (same behavior).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): keep history Markdown-rendered when a follow-up run starts
Previously the transcript rendered "the last assistant row" as plain text while
streaming. But when a follow-up turn starts, the last assistant row is the
*previous, finalized* answer until the new message begins — and the client
re-emits prior messages early in the turn — so sending a follow-up reverted the
previous answer from rendered Markdown back to raw text.
Track the actively-streaming message id in ViewState instead: it's reset on
RunStarted, set only when an AssistantDelta actually adds new content (history
re-emits are no-ops and don't mark it), and cleared on RunEnded. The renderer
keeps only that one message plain; all history stays Markdown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(readme): add Terminal Workbench (TUI) section to root README
Mention the new `deerflow` TUI alongside the Embedded Python Client in the root
README.md and README_zh.md (install, launch/headless commands, feature summary,
Web UI visibility), with a ToC entry and a preview screenshot. Links to
backend/docs/TUI.md for the full guide.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): address review feedback (willem-bd)
Ten findings from the TUI code review:
1. /resume was dead-ended — registered + in /help + tested as a builtin, but no
dispatch branch. Wired it to thread resolution / the switcher.
2. --resume <title> was forwarded raw into the checkpointer (blank thread).
Added Session.resolve_ref() to resolve id-or-title via list_threads; used by
--resume and /resume.
3. str(get("id","")) returned "None" for an explicit id:None (truthy), defeating
the empty-id guard so unrelated null-id tool calls collapsed into one card.
Coerce via a None-safe helper.
4. Headless --print/--json no longer spin up the persistence loop/engine/pool
(open_session(persistence=False)).
5. _LoopThread + engine are now closed: Session.close() (dispose engine + stop
loop) called from a try/finally around app.run().
6. --cli --continue (and piped --cli) now run headless instead of erroring.
7. Cancelled runs no longer persist a truncated title (guard on _cancelled).
8. Palette highlight resets to the top when the filter set changes.
9. Dropped the never-populated tools count from the header.
10. Documented the `not row.error` merge guard.
Adds regression tests for each; 126 TUI tests pass, ruff check + format clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): harden runtime credential management APIs
* fix(channels): address review feedback on credential hardening
Follow-up to the runtime credential-hardening pass, resolving five review
findings:
- WeChat auth persistence now writes through a 0o600 NamedTemporaryFile +
Path.replace instead of write_text-then-chmod, so the iLink bot_token is
never briefly readable at umask defaults (mirrors ChannelRuntimeConfigStore).
- The post-write chmod is split into its own try/except: a chmod failure on a
filesystem without POSIX perms now logs at debug instead of masquerading as
a "failed to persist" warning.
- Extracted the three near-identical _require_admin_user helpers (mcp,
channel_connections, channels) into a single require_admin_user(request, *,
detail) in app/gateway/deps.py; each router supplies its own detail string.
- Strengthened the runtime-config-store chmod coverage: a new test injects a
temp-file chmod failure and asserts it is logged at debug while the
destination is still owner-only (mutation-verified to fail if the chmod is
dropped), plus a loose-pre-existing-file case.
- Removed the unused _FakeRepo from the blocking-io test: its isinstance gate
routes through the repo-less 503 path, so neither stub was ever invoked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): make channel connect flow deterministic
* make format
* fix(channels): apply connect-code before allowed_users on telegram and wechat
The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and
WeChat still gate _check_user/allowed_users before connect-code dispatch, so
a newly allowlisted-but-unbound user is silently rejected when binding via the
browser deep-link / connect-code flow — the same deadlock the PR fixes.
- telegram: consume the /start deep-link token before the allowed_users gate.
- wechat: handle the /connect code before the allowed_users gate, and defer
inbound file extraction + context-token tracking past the gate so blocked
senders no longer trigger CDN downloads or token bookkeeping.
Adds regression tests for both adapters mirroring the slack/dingtalk coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): enforce single-active-owner invariant at the DB layer
_revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row
lock or constraint covering active rows. Under READ COMMITTED, two concurrent
connect-code consumes for the same (provider, external_account_id, workspace_id)
from different owners could each observe "no other active owner" and both commit
a connected row, leaving find_connection_by_external_identity nondeterministic.
- Add a partial unique index on (provider, external_account_id, workspace_id)
WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the
database guarantees at most one non-revoked row per external identity.
- Reorder upsert_connection to revoke other owners' active rows before the new
connected row is flushed (so the index is satisfied at commit), wrapped in a
bounded rollback-and-retry loop. A losing concurrent writer now retries
against the now-visible state instead of committing a duplicate.
Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden connect-status polling primitive
pollChannelConnectionUntilResolved was a free-floating recursive setTimeout
started from onSuccess with no cancellation, no per-provider dedup, a redundant
second endpoint per tick, and an unbounded loop on a non-finite expires_in.
- Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls
only listChannelConnections() and invalidates the providers query once when the
bind resolves, instead of fetching both endpoints every tick.
- Guard expires_in with a finite check + default window so undefined/NaN can no
longer produce a poll loop that runs until the page closes.
- Track one active poll handle per provider in useConnectChannelProvider via a
ref Map: a new connect cancels the prior poll for that provider, and a useEffect
cleanup cancels all polls on unmount.
Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics
Moving the allowed_users gate past _extract_text meant the parsed-message INFO
log (text=%r, first 100 chars) fired for senders that allowed_users would have
rejected, defeating the filter's noise/privacy role. Move that log to after the
allowed_users gate so blocked senders' message text never reaches INFO logs.
Also document the two operator-relevant semantic changes in backend/CLAUDE.md:
connect-code dispatch runs before allowed_users (so allowed_users is no longer a
bind-time defense; the model relies on code confidentiality + 600s TTL + one-time
consumption), and the single-active-owner-per-external-identity transfer semantics
now backed by the partial unique index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide
Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md:
connect codes are consumed before allowed_users (so a not-yet-allowlisted user can
still complete a first bind, and allowed_users is not a bind-time defense), and an
external identity has at most one active owner with last-bind-wins transfer enforced
at the DB layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): lift connect-code dispatch into Channel base class
Each adapter duplicated the ordering-sensitive boilerplate of extracting a
/connect code and guarding on the connection repo before its allowed_users gate.
The duplication is what let telegram/wechat drift and keep the gate ahead of the
bind. Centralize it:
- Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments).
- Add Channel._pending_connect_code(text), which guards on the repo and extracts
the code, documenting that adapters MUST consult it before authorization so a
browser-initiated bind can bootstrap a not-yet-authorized identity.
- Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper.
This also fixes a latent inconsistency where slack dispatched a bind even when
no connection repo was configured.
Pure refactor — the full channel suite stays green; adds a direct unit test for
the base helper's contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
* fix(channels): redact DingTalk parsed-message INFO log content
Log text_len instead of the first 100 chars of message text, so message
content never reaches INFO logs (the after-gate move already keeps blocked
senders out entirely). This takes over the redaction from #3584 so only this
PR touches dingtalk.py, letting the two PRs merge in any order conflict-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(community): add Serper Google Images provider for image_search
Add a Serper-backed `image_search` tool alongside the existing Serper
`web_search` provider, so users with a SERPER_API_KEY can pull Google
Images results as reference images for downstream image generation.
- Share request/response handling between web_search and image_search
via `_serper_post` / `_response_items`, with bounded `max_results`
(capped at 10) and query normalization.
- Add a best-effort SSRF guard (`_safe_public_url`) that rejects
non-http(s), localhost and private/non-global IP image URLs; filtered
entries are dropped and never consume the result limit.
- doctor: flag literal `api_key` values in config as a warning and steer
users toward `.env` + `$SERPER_API_KEY`.
- Docs/config: document the Serper image_search provider and SERPER_API_KEY,
and discourage committing literal keys to config.yaml.
- Tests: cover the provider end-to-end (100% line coverage on tools.py)
and the doctor literal-key warning path.
* fix(community): block obfuscated IPv4 literals in Serper image SSRF guard
The image_search SSRF guard only rejected dotted-decimal IP literals; encoded
forms such as decimal (http://2130706433/), hex (0x7f000001) and octal
(0177.0.0.1) raised ValueError in ip_address() and were allowed through, even
though many HTTP clients resolve them to private addresses like 127.0.0.1.
Add _decode_ipv4() to permissively decode these inet_aton-style encodings and
apply the same is_global check; hostnames that do not decode to an IP (e.g.
cafe.com) are still treated as hosts and left to fetch-time re-validation.
Addresses PR review feedback. Tests cover decimal/hex/octal loopback and
private encodings plus non-IP edge cases; tools.py stays at 100% line coverage.
* test(community): cover IPv4-mapped IPv6 URL filtering
* fix(community): address Serper image search review feedback
- Block trailing-dot hostname SSRF bypass (localhost./127.0.0.1.) in
_safe_public_url by stripping the FQDN root label before checks.
- Keep a filtered image/thumbnail URL empty instead of collapsing onto
its counterpart, preserving the high-res/preview contract.
- Evaluate the SSRF guard once per field rather than twice.
- Treat a null-typed organic/images field as "no results" rather than a
malformed payload.
- doctor.py: when a config $VAR is unset, fall through to the default env
var before reporting it as not set.
* fix(channels): require bound identity for user-owned IM messages
* make format
* docs: document bound identity channel config
* refactor: reuse channel connection config
* refactor _requires_bound_identity()
* refactor from_app_config()
* make format
* fix: reject unbound channel chats before semaphore
* security enhancement
* make format
* fix: enforce bound-identity admission at command entry point
The bound-identity gate only ran for non-command messages in
_handle_message() and as a fallback inside _handle_chat(). Commands had
no equivalent boundary, so an unbound platform user could send /new and
reach _create_thread() directly, creating an unowned Gateway thread and
empty checkpoint. Info commands (/status, /models, /memory) likewise
leaked Gateway state to unbound users.
Add the same _requires_bound_identity() check at the top of
_handle_command(), rejecting via _reject_unbound_channel_message() before
any thread creation or Gateway query. The gate is a no-op in legacy
open-bot mode (require_bound_identity=False) and auth-disabled mode.
Provider-level binding flows (/connect, /start) are consumed by the
provider adapter before reaching the manager, so they are unaffected.
Tests:
- unbound auth-enabled /new is rejected before threads.create
- bound auth-enabled /new still creates the thread
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): carry workspace fallback decision on inbound messages
* fix(channels): recheck bound identity by normalized workspace
* fix(channels): avoid duplicate bound identity checks
* fix(channels): preserve verified routing for bound identity rejects
* fix(channels): clarify bound identity upgrade failures
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
