Images older than all-in-one-sandbox 1.9.x have no /v1/bash/* routes, so
every env-bearing command (skills declaring required-secrets) surfaced a
raw nginx 404 that the model kept retrying. Detect the 404, remember the
capability gap per sandbox instance, and return an actionable error that
names the minimum image version and the sandbox.image remediation.
No fallback through the legacy shell path on purpose: /v1/shell/exec has
no env parameter, and every workaround puts the secret values back into
the command string or on disk — the exact leak surfaces the
request-scoped secrets design closed.
Closes#3921
* feat(mcp): add per-server tool_call_timeout for MCP tool calls
Add a configurable timeout for individual MCP tool calls to prevent
agent runs from blocking indefinitely when an MCP server becomes
unresponsive (e.g., rate-limited HTTP API, hung subprocess).
Uses the MCP SDK's built-in read_timeout_seconds parameter on
ClientSession.call_tool, which handles the timeout within the
session's own task — avoiding cross-task cancellation issues with
the session pool (ref #3379, #3203).
Config field is named tool_call_timeout (not timeout) to avoid
collision with langchain-mcp-adapters' existing timeout field on
HTTP/SSE connections.
Closes#3840
* fix(mcp): read tool_call_timeout from McpServerConfig, not connection dict
The previous implementation put tool_call_timeout into the connection dict
returned by build_server_params, which langchain's create_session then
passed to _create_stdio_session(), causing TypeError. Now reads the timeout
directly from ExtensionsConfig.mcp_servers where the wrapper is built,
keeping it out of the connection dict entirely.
Fixes P1 bug from review on #3843.
* test(mcp): regression test for tool_call_timeout not leaking into connection dict
Adds two tests:
- test_build_server_params_excludes_tool_call_timeout: verifies the connection
dict returned by build_server_params() does NOT contain tool_call_timeout
- test_stdio_tool_call_timeout_does_not_raise_typeerror: end-to-end test that
get_mcp_tools() with a stdio server having tool_call_timeout configured loads
tools without TypeError from _create_stdio_session()
Regression for PR #3843 P1 bug.
* fix(mcp): only pass read_timeout_seconds when tool_call_timeout is set
When tool_call_timeout is None, don't pass read_timeout_seconds=None to
session.call_tool(). This avoids breaking existing tests that assert on
exact call_tool arguments without the extra kwarg.
* docs(mcp): clarify stdio tool timeout
* implement goal continuations
* fix(goal): address review findings for goal continuations
- goal: key the no-progress breaker on a signature of the latest visible
assistant evidence instead of the evaluator's volatile free-text, so it
actually fires on stalled turns; thread the signature through every
worker persist / no-progress call site
- goal: align _stand_down_reason default caps with should_continue_goal
(8 / 2) so the two gate functions agree on goals missing the fields
- runtime: offload the synchronous checkpointer fallback via
asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop
- frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types)
- frontend: extract pure composer helpers into input-box-helpers.ts with
unit tests (parseGoalCommand, readGoalResponseError, skill suggestions)
- tests: cover the evidence-based no-progress and default-cap behavior
- docs: align backend/AGENTS.md goal paragraph with actual behavior
- e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): hide goal continuation counter until the agent continues
The goal status bar rendered a raw "0/8" before any auto-continuation, which
read as a mysterious score. Now the counter is hidden until
continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining
the auto-continuation cap.
- Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests
- Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(goal): address review findings for goal continuations
Frontend correctness
- Fix the optimistic /goal result permanently shadowing server goal state:
the streamed continuation counter never surfaced for a goal set in-session.
Extract a shared useActiveGoal hook (used by both chat pages) that reconciles
the optimistic copy with server state via a goalReconciliationKey, de-duping
the copy-pasted goal block across the two pages.
- Stop /goal status|clear failures from escaping handleSubmit as unhandled
rejections (handleGoalCommand now returns success; the run only starts when a
goal was actually saved).
- Use a function replacer for the goal-status toast so an objective containing
$&/$1 isn't treated as a replacement pattern.
Backend cleanliness / correctness
- De-duplicate four byte-identical helpers (_call_checkpointer_method,
_message_type, _additional_kwargs, _is_visible_message) by importing them
from runtime.goal instead of re-defining them in the run worker.
- Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has
no tasks field) and document that pending_writes is the durability signal.
- Decompose the 176-line _prepare_goal_continuation_input: extract
_reread_goal_and_checkpoint and a _persist closure so the thread-unchanged
guard and stand-down persistence aren't open-coded three times. Document the
last-writer-wins write-window limitation as a follow-up.
- Add a shared parse_goal_command helper and use it from the TUI and IM-channel
/goal handlers (one place for the status/clear/set semantics).
Tests
- Restore the 11 command-registry tests dropped by the previous goal change
(filter_commands ranking/description, build_registry builtins/skills, resolve
cases) alongside the new goal tests.
- Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal
handler, parse_goal_command, and goalReconciliationKey.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix goal review feedback
* fix goal continuation checkpoint races
* prioritize goal commands while streaming
Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run.
Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check
* preserve goal status during clarification
Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint.
Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* style: format active goal hook
Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate.
Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* fix goal review followups
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
build_run_config() copied every top-level request key (except
configurable/context) verbatim into the LangGraph RunnableConfig, including
recursion_limit. The server default of 100 was fully overridable by the caller
with no upper bound, so a request like {"config": {"recursion_limit": 100000000}}
could make a single run execute effectively unbounded LangGraph super-steps
(each >= 1 LLM call), enabling runaway API cost / DoS.
Validate the client value server-side and clamp it into a safe range:
- valid positive ints are capped at a configurable ceiling
(AppConfig.max_recursion_limit, default 1000 to match the existing
frontend/public-skill default so legitimate deep runs are unaffected)
- invalid/non-positive/bool/None values fall back to the 100 server default
- applied on both the configurable and the LangGraph >= 0.6.0 context paths
- WARNING logged on clamp for observability
Add unit tests (including a configurable-ceiling case), expose
max_recursion_limit in config.example.yaml (config_version 16), and document
the ceiling/fallback in backend/docs/API.md.
Co-authored-by: DengY11 <DengY11@users.noreply.github.com>
* feat(community): add Crawl4AI web_fetch provider
Crawl4AI is a self-hosted, no-API-key web fetcher: it runs headless
Chromium and returns server-cleaned "fit" markdown directly via its
POST /md endpoint, so no client-side readability extraction is needed.
It sits alongside the existing self-hosted Browserless provider.
- deerflow.community.crawl4ai: async Crawl4AiClient + web_fetch_tool
(reads base_url/timeout_s/token/filter from config; "Error:" string
convention; 4096-char cap), mirroring the browserless provider
- tests: 17 unit cases (success, HTTP error, success:false, empty,
timeout, request error, token header, truncation, config reads)
- config.example.yaml: commented web_fetch example
- doctor: register as a no-key (free) web_fetch provider
- setup wizard: add to WEB_FETCH_PROVIDERS (no API key)
- docs: README, CONTRIBUTING, CONFIGURATION, AGENTS provider lists
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(community): address Crawl4AI provider review feedback
- timeout: robust _coerce_timeout (bool / non-numeric -> default) mirroring
jina, so 'timeout: off' no longer becomes 0.0 and times out every request
- read web_fetch config once per invocation and pass values into the client,
so a concurrent hot-reload can't split base_url from filter
- rename config key timeout_s -> timeout to match jina/infoquest (the
default providers); update config.example.yaml + setup wizard
- validate + normalize the markdown filter against {fit,raw,bm25,llm};
unknown values fall back to fit with a warning instead of an opaque HTTP 400
- client: a non-JSON 200 body (reverse proxy / auth wall) now reports the
content-type + snippet instead of a generic JSONDecodeError
- tests: 22 cases (added non-JSON-200, _coerce_timeout, _coerce_filter,
invalid-filter fallback, read-config-once)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Add Brave image search community tool
* fix(community): length-cap Brave web_search queries
Apply _clean_query in web_search_tool so over-long queries are trimmed
to Brave's 400-char limit before the API call, matching image_search_tool
and avoiding HTTP 422 from the Brave Search API.
* fix(community): harden Brave image search SSRF guard and dimension mapping
Address PR review findings:
- Catch ValueError from urlparse so a malformed bracketed-IPv6 URL skips
one item instead of crashing the whole image_search call
- Reject IPv6 literals embedding a non-global IPv4 (IPv4-mapped, 6to4,
NAT64, IPv4-compatible), closing the loopback/private SSRF bypass
- Report width/height from the dict of the URL actually returned, so a
surviving thumbnail no longer reports the dropped original's dimensions
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: generate title for interrupted first turn
* test(title): cover partial-exchange + dict-form messages
Harden the interrupted-run fallback path added in 19fc34fd:
- TitleMiddleware._should_generate_title now accepts a lone first-turn
user message when allow_partial_exchange=True, so the worker can still
derive a title if cancellation lands before any AI chunk is checkpointed.
- runtime/runs/worker._ensure_interrupted_title computes the next
checkpoint step defensively (treat missing/non-int step as 0) and
renames a shadowed ckpt_config local for readability.
- Add four unit tests in tests/test_title_middleware_core_logic.py:
partial-exchange allows user-only, partial-exchange still respects an
existing title, dict-form messages are recognized, and the sync
fallback path derives a title from dict-form messages — matching what
channel_values stores in the checkpoint.
Refs #3859.
* fix: persist interrupted-title via channel_versions bump
Address PR #3874 review feedback: ``_ensure_interrupted_title`` previously
called ``aput(..., new_versions={})``. LangGraph's DB-backed savers
(``PostgresSaver`` and the v4 ``SqliteSaver`` blob layout) strip inline
``channel_values`` from ``put`` and only persist blobs for channels named
in ``new_versions`` — so the fallback ``title`` channel was dropped on
read-back and ``threads_meta.display_name`` stayed ``"Untitled"`` after
refresh on those backends. The original in-memory e2e passed because
``InMemorySaver`` keeps the inline snapshot verbatim.
Fix mirrors ``_rollback_to_pre_run_checkpoint`` in the same file: bump
``channel_versions["title"]`` (via ``checkpointer.get_next_version`` when
available, else int/string fallbacks), persist the new version on the
checkpoint, and declare it in ``new_versions`` so the DB savers actually
write the blob.
Regression coverage in ``tests/test_run_worker_rollback.py``:
- ``test_ensure_interrupted_title_bumps_channel_version_and_declares_it_in_new_versions``
— exact ``aput`` invariants: ``new_versions == {"title": 1}``, the
written checkpoint's ``channel_versions["title"]`` is bumped, and the
pre-existing ``messages`` version is preserved.
- ``test_ensure_interrupted_title_bumps_existing_string_version`` —
string-shaped prior version (some savers use UUID-style versions);
bumped value must differ from the prior, no overwrite-in-place.
- ``test_ensure_interrupted_title_skips_when_title_already_set`` — title
short-circuit; no extra ``aput``.
- ``test_ensure_interrupted_title_returns_none_when_no_checkpoint`` —
no checkpoint yet; returns ``None`` without writing.
- ``test_ensure_interrupted_title_round_trip_with_real_sqlite_checkpointer``
— full round-trip against a real ``AsyncSqliteSaver`` on a disk-backed
DB, then closes and re-opens the saver to simulate a fresh
connection. The fallback title must still be present on the second
``aget_tuple``. This is the exact scenario the review flagged.
Validated locally with the full backend suite: 5195 passed, 18 skipped.
Refs #3859. Addresses review on #3874.
* test(worker, title): harden interrupted-title fallback for every saver
Defensive coverage on top of the channel_versions fix (commit 05253957),
addressing edge cases surfaced during a second-pass review of #3874.
Worker:
- Extract version bump into ``_bump_channel_version(checkpointer, current)``
with explicit fallbacks for int / float / numeric-string / UUID-shaped
string / None / bool, AND a wrap-around defense when the saver's
``get_next_version`` raises or returns an unchanged value. The
invariant is: returned version MUST differ from the prior. Without
this, a saver bug (or a custom backend) could leave
``new_versions={"title": v}`` no-op on DB savers — the very class of
bug the original review pointed out.
Title middleware:
- Coerce ``state.get("messages")`` from ``None`` to ``[]`` on both
``_should_generate_title`` and ``_build_title_prompt``. A
partially-initialized checkpoint can carry ``messages=None`` on the
channel_values channel (the worker reads raw channel_values, not
BaseMessages), and the default kwarg only protects against a missing
key. Repro: ``TypeError: 'NoneType' object is not iterable`` from
the next() generator — confirmed by reverting the fix and watching
``test_*_handles_none_messages_channel`` go red.
Tests (TDD-verified red→green for the new asserts):
- ``test_run_worker_rollback.py``:
* ``_bump_channel_version`` — 8 tests covering every version type
(int, float, numeric string, UUID-style string, None, bool) and
every saver-side fault mode (no ``get_next_version`` / raising /
stuck on identity).
* ``test_ensure_interrupted_title_*`` — 5 additional helper
boundary tests: title.enabled=false short-circuit; empty
messages list; messages=None; aput-error propagation (helper
contract: caller swallows, not the helper); idempotency on a
real InMemorySaver across two invocations.
* ``test_ensure_interrupted_title_preserves_non_title_channel_versions``
— pins that ``new_versions`` only contains ``"title"`` and that
other channels' versions are untouched (regression anchor for a
sloppier draft that bumped every channel).
* ``test_worker_finally_block_swallows_helper_exceptions`` — pins
the integration contract: even if the helper raises, the worker's
threads_meta status sync still runs and ``publish_end`` is still
awaited so the SSE stream closes cleanly.
- ``test_title_middleware_core_logic.py``:
* 4 additional tests: ``messages=None`` on both
``_should_generate_title`` and ``_build_title_prompt``; the
``role: user`` / ``role: assistant`` (OpenAI-style) dict
normalization; partial-exchange path with a dict-form message.
Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
→ 5215 passed, 18 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Red/green TDD verification: temporarily reverted the
``new_versions={}`` fix → 4 new tests went red as expected; restored
and the suite is green again. Same red/green dance for the
``messages=None`` coercion.
Refs #3859. Addresses second-pass review on #3874.
* fix(title): ignore dict context reminders in fallback
* fix(worker): link interrupted-title checkpoint to its parent
The title-bump checkpoint written by ``_ensure_interrupted_title`` was
landing without a ``parent_checkpoint_id`` — a real orphan in the
LangGraph history graph. Reproduction (disk-backed AsyncSqliteSaver):
[seed] checkpoint_id = 1f173dbc...
[helper] wrote title = "Why is the sky blue?"
[issue 1] new checkpoint = 1f173dbc..., parent = None
[issue 1] is new checkpoint orphaned? True
Root cause: ``_ensure_interrupted_title`` built ``write_config`` as
``{"thread_id": ..., "checkpoint_ns": ...}`` only. ``BaseCheckpointSaver``
implementations read ``configurable.checkpoint_id`` from that config as
the *parent* id when inserting (see ``langgraph/checkpoint/sqlite/aio.py``
``aput``: ``config["configurable"].get("checkpoint_id")`` becomes the
``parent_checkpoint_id`` column). With no value, the saver writes NULL —
the new checkpoint is a tree root.
Consequences:
- Any future LangGraph ``runs.resume_from`` / time-travel feature has no
backward edge to walk past the title-bump.
- History-visualization UIs built on ``alist()`` render the title-bump as
a sibling of the prior checkpoint, not its descendant.
Fix: read ``checkpoint_id`` off the tuple's own config and thread it into
``write_config["configurable"]["checkpoint_id"]`` before calling ``aput``,
the same pattern every middleware-driven write uses.
Three new regression tests against real ``AsyncSqliteSaver`` (disk-backed,
fresh connections so we exercise the on-disk read path):
- ``test_ensure_interrupted_title_links_new_checkpoint_to_its_parent`` —
asserts ``latest.parent_config["configurable"]["checkpoint_id"]`` equals
the seeded checkpoint id. TDD red-green verified: reverting the fix
flips this test red with ``AssertionError: title-bump checkpoint must
have a parent_config``.
- ``test_ensure_interrupted_title_appears_in_history_with_audit_marker``
— pins the audit contract: the title-bump entry in ``alist()`` carries
``metadata.source == "update"`` and ``metadata.writes`` contains
``runtime_interrupt_title``. This is a deliberate design choice — we
do NOT hide the entry from history (audit trail belongs in the saver),
but its source and writes marker MUST be unambiguous so UIs/tools can
identify it.
- ``test_ensure_interrupted_title_survives_immediate_next_turn`` —
cancel → immediate user follow-up scenario. Simulates the agent's next
turn appending a (user, ai) pair without touching the title channel,
then opens a fresh saver and verifies the title is still present after
the next-turn checkpoint write. Pins the channel-version-blob
invariant established by commit 05253957 — without the
``new_versions={"title": v}`` declaration there, the title blob would
vanish from the DB and this test would read back ``None``.
Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
→ 5222 passed, 15 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Reproduction script confirms ``parent_checkpoint_id`` is now non-null
and the next-turn read-back preserves the fallback title.
Refs #3859.
* Revert "fix(worker): link interrupted-title checkpoint to its parent"
This reverts commit c763ed9334781db1acdce0f5f33d663d8d5f80ff.
* test: trim over-engineered test coverage
Reduce review surface area on PR #3874 by dropping defensive tests that
don't pin a real invariant. After self-review:
- ``_bump_channel_version``: 8 tests → 2 (happy path + saver-error
fallback). Dropped float / bool / numeric-string / UUID-string /
missing-get-next-version / stuck-get-next-version branches — those
are speculative scaffolding for savers we don't ship.
- ``_ensure_interrupted_title``: dropped ``returns_none_when_title_disabled``,
``returns_none_with_no_user_message``, ``returns_none_when_no_checkpoint``
— boundary guards already exercised by the e2e test and the
``handles_none_messages_channel`` regression anchor.
Net: -107 lines of test code. Remaining coverage still pins every
red-green-verified invariant (channel_versions bump, string-version
bump, idempotency, sqlite round-trip, non-title channel preservation,
aput-error contract, worker finally swallowing, partial-exchange).
Verification: 5209 passed, 15 skipped.
* fix: harden interrupted title finalization
* fix: serialize interrupted title finalization
* fix: preserve interrupt semantics during title finalization
* fix: preserve delayed interrupted title recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
Adds ``deerflow.community.e2b_sandbox.E2BSandboxProvider`` with parity
to AioSandboxProvider: metadata-keyed per-thread persistence, server-
side idle timeout, warm-pool reclaim with liveness checks, /mnt/user-data
bootstrap symlinks, dead-sandbox auto-rebuild, and release-time mirror
of agent outputs back to the host artifact directory.
Signed-off-by: joey <zchengjoey@gmail.com>
* feat(tui): add Hermes-like terminal workbench backed by DeerFlowClient
Implements the `deerflow` TUI from RFC #3540: a terminal-native, embedded
workbench over the existing harness (no Gateway/frontend/nginx/Docker), built
Python-native with Textual and learning UX patterns from tao-pi.
Architecture — every layer except the Textual app is pure and unit-tested:
- view_state.py: ViewState + reduce(state, action), the testable heart
- runtime.py: StreamEvent -> reducer actions (pure translate + threaded driver)
- message_format / command_registry / input_history / render / theme: pure
- app.py: Textual App; runs the sync DeerFlowClient.stream() on a worker thread
and marshals actions back to the UI thread. Slash command palette, model and
thread modal pickers, ↑/↓ history, Ctrl+C interrupt, TTY-aware fallback.
- cli.py: pure launch-mode planning + headless --print/--json + `deerflow`
console script (textual is an optional [tui] extra; degrades to headless help)
Web UI visibility (the RFC's key decision): persistence.py writes a threads_meta
row under the local default user into the same DB the Gateway reads, so terminal
sessions appear in the Web UI sidebar without running the Gateway. Best-effort,
no-op on the memory backend; all DB work on one long-lived background loop.
Tests: 95 TUI tests — pure layers via pytest, app/palette/overlays via Textual's
pilot harness with a fake session, and a threads_meta read/write round-trip.
ruff clean; respects the harness->app import boundary. Docs: backend/docs/TUI.md
plus CLAUDE.md/README updates and preview screenshots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): de-duplicate streamed assistant text and tool cards; keep Tab in composer
Self-test surfaced three issues, all root-caused to consuming non-strict
streaming from DeerFlowClient (proven by the client's own
test_dedup_requires_messages_before_values_invariant, which shows the client can
re-emit a message id's full content twice):
- Assistant text was doubled (e.g. "answer answer") because the reducer blindly
concatenated same-id deltas. Now merges by content: a re-send or cumulative
snapshot replaces; only genuine increments append.
- Tool activity showed duplicate and empty "gear" cards from partial/re-emitted
tool-call chunks. ToolStarted now de-dupes by tool_call_id, drops id-less
noise chunks, and fills the name on a later chunk; a tool result with no prior
card still surfaces as a completed card.
- Tab moved focus off the composer to the scroll region (felt like broken cursor
logic). Tab is now consumed by the composer (completes a command when the
palette is open, no-op otherwise).
Adds reducer tests for each case plus a Tab-focus test; 102 TUI tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): make Esc interrupt an active run (matches the status hint)
The status line advertised "esc interrupt" but Esc was only wired to close the
slash palette, so it did nothing during a run. Esc now: closes the palette when
open, interrupts the active run when streaming, and is a no-op when idle. The
interrupt logic is shared with Ctrl+C via _interrupt_run(). Adds a regression
test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): stop prior answers duplicating on threads with history
On a thread with history, DeerFlowClient re-emits every prior message on each
new turn (its streamed_ids dedup is per-stream-call), and a re-emitted older
message can arrive after a newer message has already started. The reducer only
matched the *most recent* assistant row by id and otherwise appended, so each
re-emitted older answer was duplicated verbatim at the end of the transcript.
Match an assistant row by id anywhere in the transcript and merge in place.
Tool cards already de-dupe by call id globally, so they were unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): correct CJK cursor drift in the composer
Confirmed a Textual Input bug (latest 8.2.7): Input._cursor_offset adds an
unconditional +1 at the end of the value, overshooting by one cell after
double-width (CJK) characters. That misplaces the hardware/IME cursor — the
drift seen when typing Chinese in iTerm2 (the on-screen block cursor, drawn
separately in render_line, is fine; English doesn't use an IME so it looks
correct). Reproduced with a bare Input, so it's upstream, not our layout.
Add ComposerInput(Input) overriding _cursor_offset to the true cell position and
use it for the composer. Numeric tests pin the CJK end/mid and ASCII cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(tui): render finalized assistant messages as Markdown
The transcript showed raw Markdown (literal **bold**, ## headings, - lists,
links). Finalized assistant messages now render as Rich Markdown — headings,
bold/italic, lists, inline code + code blocks, blockquotes, horizontal rules and
links — with the ● speaker marker aligned to the top of the body.
The actively-streaming message stays plain text so partial Markdown doesn't
reflow/jump, then snaps to its rendered form when the run ends. Transcript
re-renders are coalesced on a ~60ms timer (dirty flag) so per-token Markdown
re-parsing stays smooth on long threads. Tests cover both the rendered and the
streaming-plain paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style(tui): apply ruff format
CI lint runs `ruff format --check` via uvx (latest ruff); apply the formatter so
the lint-backend job passes. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(tui): address code-quality review comments
From github-code-quality[bot] on #3760:
- runtime.py: give the `_ClientLike` Protocol method a docstring body instead of
a bare `...` (flagged as a no-effect statement), matching the harness
convention for Protocol stubs (e.g. SafetyTerminationDetector).
- test_tui_cli_main.py: drop the unnecessary `lambda: _FakeSession()` wrappers in
monkeypatch.setattr; pass `_FakeSession` directly (same behavior).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): keep history Markdown-rendered when a follow-up run starts
Previously the transcript rendered "the last assistant row" as plain text while
streaming. But when a follow-up turn starts, the last assistant row is the
*previous, finalized* answer until the new message begins — and the client
re-emits prior messages early in the turn — so sending a follow-up reverted the
previous answer from rendered Markdown back to raw text.
Track the actively-streaming message id in ViewState instead: it's reset on
RunStarted, set only when an AssistantDelta actually adds new content (history
re-emits are no-ops and don't mark it), and cleared on RunEnded. The renderer
keeps only that one message plain; all history stays Markdown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(readme): add Terminal Workbench (TUI) section to root README
Mention the new `deerflow` TUI alongside the Embedded Python Client in the root
README.md and README_zh.md (install, launch/headless commands, feature summary,
Web UI visibility), with a ToC entry and a preview screenshot. Links to
backend/docs/TUI.md for the full guide.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): address review feedback (willem-bd)
Ten findings from the TUI code review:
1. /resume was dead-ended — registered + in /help + tested as a builtin, but no
dispatch branch. Wired it to thread resolution / the switcher.
2. --resume <title> was forwarded raw into the checkpointer (blank thread).
Added Session.resolve_ref() to resolve id-or-title via list_threads; used by
--resume and /resume.
3. str(get("id","")) returned "None" for an explicit id:None (truthy), defeating
the empty-id guard so unrelated null-id tool calls collapsed into one card.
Coerce via a None-safe helper.
4. Headless --print/--json no longer spin up the persistence loop/engine/pool
(open_session(persistence=False)).
5. _LoopThread + engine are now closed: Session.close() (dispose engine + stop
loop) called from a try/finally around app.run().
6. --cli --continue (and piped --cli) now run headless instead of erroring.
7. Cancelled runs no longer persist a truncated title (guard on _cancelled).
8. Palette highlight resets to the top when the filter set changes.
9. Dropped the never-populated tools count from the header.
10. Documented the `not row.error` merge guard.
Adds regression tests for each; 126 TUI tests pass, ruff check + format clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): harden runtime credential management APIs
* fix(channels): address review feedback on credential hardening
Follow-up to the runtime credential-hardening pass, resolving five review
findings:
- WeChat auth persistence now writes through a 0o600 NamedTemporaryFile +
Path.replace instead of write_text-then-chmod, so the iLink bot_token is
never briefly readable at umask defaults (mirrors ChannelRuntimeConfigStore).
- The post-write chmod is split into its own try/except: a chmod failure on a
filesystem without POSIX perms now logs at debug instead of masquerading as
a "failed to persist" warning.
- Extracted the three near-identical _require_admin_user helpers (mcp,
channel_connections, channels) into a single require_admin_user(request, *,
detail) in app/gateway/deps.py; each router supplies its own detail string.
- Strengthened the runtime-config-store chmod coverage: a new test injects a
temp-file chmod failure and asserts it is logged at debug while the
destination is still owner-only (mutation-verified to fail if the chmod is
dropped), plus a loose-pre-existing-file case.
- Removed the unused _FakeRepo from the blocking-io test: its isinstance gate
routes through the repo-less 503 path, so neither stub was ever invoked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): make channel connect flow deterministic
* make format
* fix(channels): apply connect-code before allowed_users on telegram and wechat
The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and
WeChat still gate _check_user/allowed_users before connect-code dispatch, so
a newly allowlisted-but-unbound user is silently rejected when binding via the
browser deep-link / connect-code flow — the same deadlock the PR fixes.
- telegram: consume the /start deep-link token before the allowed_users gate.
- wechat: handle the /connect code before the allowed_users gate, and defer
inbound file extraction + context-token tracking past the gate so blocked
senders no longer trigger CDN downloads or token bookkeeping.
Adds regression tests for both adapters mirroring the slack/dingtalk coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): enforce single-active-owner invariant at the DB layer
_revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row
lock or constraint covering active rows. Under READ COMMITTED, two concurrent
connect-code consumes for the same (provider, external_account_id, workspace_id)
from different owners could each observe "no other active owner" and both commit
a connected row, leaving find_connection_by_external_identity nondeterministic.
- Add a partial unique index on (provider, external_account_id, workspace_id)
WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the
database guarantees at most one non-revoked row per external identity.
- Reorder upsert_connection to revoke other owners' active rows before the new
connected row is flushed (so the index is satisfied at commit), wrapped in a
bounded rollback-and-retry loop. A losing concurrent writer now retries
against the now-visible state instead of committing a duplicate.
Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden connect-status polling primitive
pollChannelConnectionUntilResolved was a free-floating recursive setTimeout
started from onSuccess with no cancellation, no per-provider dedup, a redundant
second endpoint per tick, and an unbounded loop on a non-finite expires_in.
- Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls
only listChannelConnections() and invalidates the providers query once when the
bind resolves, instead of fetching both endpoints every tick.
- Guard expires_in with a finite check + default window so undefined/NaN can no
longer produce a poll loop that runs until the page closes.
- Track one active poll handle per provider in useConnectChannelProvider via a
ref Map: a new connect cancels the prior poll for that provider, and a useEffect
cleanup cancels all polls on unmount.
Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics
Moving the allowed_users gate past _extract_text meant the parsed-message INFO
log (text=%r, first 100 chars) fired for senders that allowed_users would have
rejected, defeating the filter's noise/privacy role. Move that log to after the
allowed_users gate so blocked senders' message text never reaches INFO logs.
Also document the two operator-relevant semantic changes in backend/CLAUDE.md:
connect-code dispatch runs before allowed_users (so allowed_users is no longer a
bind-time defense; the model relies on code confidentiality + 600s TTL + one-time
consumption), and the single-active-owner-per-external-identity transfer semantics
now backed by the partial unique index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide
Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md:
connect codes are consumed before allowed_users (so a not-yet-allowlisted user can
still complete a first bind, and allowed_users is not a bind-time defense), and an
external identity has at most one active owner with last-bind-wins transfer enforced
at the DB layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): lift connect-code dispatch into Channel base class
Each adapter duplicated the ordering-sensitive boilerplate of extracting a
/connect code and guarding on the connection repo before its allowed_users gate.
The duplication is what let telegram/wechat drift and keep the gate ahead of the
bind. Centralize it:
- Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments).
- Add Channel._pending_connect_code(text), which guards on the repo and extracts
the code, documenting that adapters MUST consult it before authorization so a
browser-initiated bind can bootstrap a not-yet-authorized identity.
- Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper.
This also fixes a latent inconsistency where slack dispatched a bind even when
no connection repo was configured.
Pure refactor — the full channel suite stays green; adds a direct unit test for
the base helper's contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
* fix(channels): redact DingTalk parsed-message INFO log content
Log text_len instead of the first 100 chars of message text, so message
content never reaches INFO logs (the after-gate move already keeps blocked
senders out entirely). This takes over the redaction from #3584 so only this
PR touches dingtalk.py, letting the two PRs merge in any order conflict-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(community): add Serper Google Images provider for image_search
Add a Serper-backed `image_search` tool alongside the existing Serper
`web_search` provider, so users with a SERPER_API_KEY can pull Google
Images results as reference images for downstream image generation.
- Share request/response handling between web_search and image_search
via `_serper_post` / `_response_items`, with bounded `max_results`
(capped at 10) and query normalization.
- Add a best-effort SSRF guard (`_safe_public_url`) that rejects
non-http(s), localhost and private/non-global IP image URLs; filtered
entries are dropped and never consume the result limit.
- doctor: flag literal `api_key` values in config as a warning and steer
users toward `.env` + `$SERPER_API_KEY`.
- Docs/config: document the Serper image_search provider and SERPER_API_KEY,
and discourage committing literal keys to config.yaml.
- Tests: cover the provider end-to-end (100% line coverage on tools.py)
and the doctor literal-key warning path.
* fix(community): block obfuscated IPv4 literals in Serper image SSRF guard
The image_search SSRF guard only rejected dotted-decimal IP literals; encoded
forms such as decimal (http://2130706433/), hex (0x7f000001) and octal
(0177.0.0.1) raised ValueError in ip_address() and were allowed through, even
though many HTTP clients resolve them to private addresses like 127.0.0.1.
Add _decode_ipv4() to permissively decode these inet_aton-style encodings and
apply the same is_global check; hostnames that do not decode to an IP (e.g.
cafe.com) are still treated as hosts and left to fetch-time re-validation.
Addresses PR review feedback. Tests cover decimal/hex/octal loopback and
private encodings plus non-IP edge cases; tools.py stays at 100% line coverage.
* test(community): cover IPv4-mapped IPv6 URL filtering
* fix(community): address Serper image search review feedback
- Block trailing-dot hostname SSRF bypass (localhost./127.0.0.1.) in
_safe_public_url by stripping the FQDN root label before checks.
- Keep a filtered image/thumbnail URL empty instead of collapsing onto
its counterpart, preserving the high-res/preview contract.
- Evaluate the SSRF guard once per field rather than twice.
- Treat a null-typed organic/images field as "no results" rather than a
malformed payload.
- doctor.py: when a config $VAR is unset, fall through to the default env
var before reporting it as not set.
* fix(channels): require bound identity for user-owned IM messages
* make format
* docs: document bound identity channel config
* refactor: reuse channel connection config
* refactor _requires_bound_identity()
* refactor from_app_config()
* make format
* fix: reject unbound channel chats before semaphore
* security enhancement
* make format
* fix: enforce bound-identity admission at command entry point
The bound-identity gate only ran for non-command messages in
_handle_message() and as a fallback inside _handle_chat(). Commands had
no equivalent boundary, so an unbound platform user could send /new and
reach _create_thread() directly, creating an unowned Gateway thread and
empty checkpoint. Info commands (/status, /models, /memory) likewise
leaked Gateway state to unbound users.
Add the same _requires_bound_identity() check at the top of
_handle_command(), rejecting via _reject_unbound_channel_message() before
any thread creation or Gateway query. The gate is a no-op in legacy
open-bot mode (require_bound_identity=False) and auth-disabled mode.
Provider-level binding flows (/connect, /start) are consumed by the
provider adapter before reaching the manager, so they are unaffected.
Tests:
- unbound auth-enabled /new is rejected before threads.create
- bound auth-enabled /new still creates the thread
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): carry workspace fallback decision on inbound messages
* fix(channels): recheck bound identity by normalized workspace
* fix(channels): avoid duplicate bound identity checks
* fix(channels): preserve verified routing for bound identity rejects
* fix(channels): clarify bound identity upgrade failures
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Brave Search web search tool
Add a community web_search provider backed by the official Brave Search
API (https://api.search.brave.com/res/v1/web/search). API key is read
from the tool config (inline api_key) or the BRAVE_SEARCH_API_KEY env
var. Output schema (title/url/content) matches existing search tools.
No new dependencies (uses the existing httpx). Also wires up the setup
wizard, doctor health check, config example, and EN/ZH docs.
* refactor(community): drop redundant [:count] slice in Brave search
The Brave API already caps results via the `count` request param, so
client-side slicing was redundant. Tests now simulate the API honoring
`count` instead of relying on the slice. Addresses PR review nit.
* style(tests): apply ruff format to test_doctor.py
Collapse multiline write_text calls onto single lines to satisfy the
CI ruff formatter (lint-backend was failing on format --check).
* Add user-owned IM channel connections
* Fix dev startup and channel connect popup
* Use async channel connect flow
* Harden dev service daemon startup
* Support local IM channel connections
* Align IM connections with local channels
* Fix safe user id digest algorithm
* Address Copilot IM channel feedback
* Address IM channel review comments
* Support all integrated IM channel connections
* Format additional channel connection tests
* Keep unavailable channel connect buttons clickable
* Fix IM channel provider icons
* Add runtime setup for enabled IM channels
* Guard global shortcut key handling
* Keep configured IM channels editable
* Avoid password autofill for channel secrets
* Make channel threads visible to connection owners
* Persist IM runtime config locally
* Allow disconnecting runtime IM channels
* Route no-auth channel sessions to local user
* Use default user for auth-disabled local mode
* Show IM channel source on threads
* Prefill IM channel runtime config
* Reflect IM channel runtime health
* Ignore Feishu message read events
* Ignore Feishu non-content message events
* Let setup wizard enable IM channels
* Fix frontend formatting after merge
* Stabilize backend tests without local config
* Isolate channel runtime config tests
* Address channel connection review comments
* Use sha256 user buckets with legacy migration
* Ensure runtime IM channels are ready after restart
* Persist disconnected IM channel state
* Address channel connection review comments
* Address channel connection review findings
Frontend connect flow:
- Open the runtime-config dialog only when a provider still needs
credentials; configured providers go straight to the connect flow, so
the binding-code/deep-link path is reachable from the UI again.
- After saving credentials, continue into the connect flow when a user
binding is still required (multi-user mode) instead of stopping at a
"Connected" toast.
- Extract shared provider-state helpers to core/channels/provider-state
and add unit + e2e coverage for the direct-connect and
configure-then-connect paths.
Provider status semantics:
- Report connection_status from the user's newest connection row;
with no binding it is not_connected, except in auth-disabled local
mode where a configured running channel is effectively connected.
Concurrency and event-loop correctness:
- Offload ChannelRuntimeConfigStore construction and writes, channel
service construction, and Slack connection replies to threads; add a
tests/blocking_io/ anchor for the runtime-config handlers.
- Consume binding codes with a conditional UPDATE so a code can only be
used once under concurrent workers; retry upsert_connection as an
update when a concurrent insert wins the unique constraint.
- Serialize ensure_channel_ready per channel so concurrent provider
polls cannot double-start a channel worker.
Config and migration hardening:
- Stop mutating the get_app_config()-cached Telegram provider config;
the runtime store now owns the UI-entered bot username.
- Register channel_connections in STARTUP_ONLY_FIELDS with the
standardized startup-only Field description.
- Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name
so another user's same-prefix bucket can never be migrated.
- Remove the unused Telegram process_webhook_update path and document
src/core/channels in the frontend docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address PR review comments on authz scoping and channel runtime
Security (review feedback from ShenAC-SAC):
- Scope internal-token callers to the connection owner carried in
X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright,
in both require_permission(owner_check=True) and the stateless run
endpoints. Internal callers keep access to their own and
shared/legacy threads, and may claim a default-owned channel thread
for its real owner, but a leaked internal token no longer grants
cross-user thread access.
- Require admin privileges for POST/DELETE /api/channels/{provider}/
runtime-config: runtime credentials and channel workers are
instance-wide shared state (same model as the MCP config API).
Read-only provider listing stays available to all users.
Performance (review feedback from willem-bd):
- Skip the redundant thread channel-metadata PATCH after the first
successful backfill per thread.
- Reuse the per-connection Slack WebClient until its token changes
instead of constructing one per outbound message.
- Reconcile channel readiness for all providers concurrently in
GET /api/channels/providers.
Also resolve the code-quality unused-import flag in the blocking-io
anchor by pre-importing the channel service via importlib.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix prettier formatting in provider-state test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reconcile UI runtime channel config with config reload on restart
Main now reloads a channel's config.yaml entry on restart_channel()
(#3514, issue #3497). Adapt the user-owned connection flow to coexist:
- configure_channel() restarts with reload_config=False — the caller
just supplied the authoritative config (browser-entered credentials
that are never written to config.yaml), so a file reload must not
clobber it with the stale on-disk entry.
- _load_channel_config() re-applies the UI runtime-store overlay used
at startup, so an operator-triggered restart keeps browser-entered
credentials for channels without a config.yaml entry and does not
resurrect a channel disconnected from the UI.
- Offload the reload's disk IO (config.yaml + runtime store) with
asyncio.to_thread, matching the blocking-IO policy on this branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(blocking-io): add changed-lines blocking-IO scanner (L1)
* feat(blocking-io): add scan-changed CLI wrapper
* feat(skill): add blocking-io-guard developer SOP skill
* docs(blocking-io): point contributors at the blocking-io-guard skill
* style(blocking-io): apply ruff format to scanner and tests
* docs(backend): document changed-lines blocking-IO scanner in CLAUDE.md
* feat(skill): add post-fix re-scan check and PR batching policy
* refactor(skill): fix SOP step ordering, align template with repo conventions
- Move re-scan into an explicit 'apply the fix' step (was wedged after
anchor generation while telling you to go back before the anchor)
- Renumber steps 0-6; drop undefined 'L1' jargon
- Mode A: document that the diff is <base>...HEAD (commit first)
- Mode B: prefer make detect-blocking-io + findings JSON file
- anchor template: module-level pytestmark per tests/blocking_io convention
- CLAUDE.md: fix 'git diff --base' phrasing
* fix(skill): catch findings introduced without touching the blocking line
Review follow-up: changed-line intersection alone misses the case where a
new async caller exposes an old sync helper — the static finding sits on
the untouched blocking line, so Mode A returned empty and the SOP stopped
on a false 'no blocking-IO surface'.
Selection is now a union over the changed files:
- findings on added lines of git diff <base>...HEAD (kept: a second
identical symbol in an already-flagged function collides on the stable
key and only this selection sees it);
- findings new versus the merge base, matched by (path, function,
symbol) — never line numbers.
Base sources are materialized via git show <merge-base>:<path>; files
absent at base count every head finding as new. SKILL.md now states the
residual same-file-only blind spot (cross-file async callers) instead of
treating an empty list as proof of zero exposure, and only requires
reading sop-skeleton.md when generalizing to another detector domain.
* docs(skill): examples teach test-writing, the teeth check defines the rule
All examples in the references/template are filesystem-flavored; make
explicit that they are instances, not the SOP's boundary — the same rules
apply to every detector category (FILE_IO, HTTP, SUBPROCESS, SLEEP) and
acceptance is always red/green teeth, never similarity to an example.
Neutralize the template's arrange comment accordingly.
* fix(blocking-io): harden changed-lines scanner per review
- Dedup the union selection by the stable key (path, function, symbol)
instead of dict identity, so a future selector returning copied dicts
cannot silently empty the result.
- parse_changed_lines now handles any unified diff: context lines advance
the new-file counter, \-markers and deletions do not, and the counter
resets at each +++ header. Previously correct only for --unified=0.
- Add blocking_io_static.scan_source (in-memory scan); base-version
comparison no longer round-trips through temp files.
- Empty Mode A report now prints the same-file-only reachability caveat
at the point of use instead of relying on the SOP text alone.
* docs(skill): bound best-effort cleanup when the offload sits in finally
Lesson from the #3505 review: the SOP routinely drives 'offload the
cleanup branch' transformations, and an awaited cleanup in finally can
mask or stall the primary exception. One sentence in Step 2 closes that
gap at the point where the fix is written.
* feat(memory): add memory.token_counting config to avoid tiktoken network dependency (#3429)
Add a `memory.token_counting` option (`tiktoken` | `char`) so deployments in
network-restricted environments can opt out of tiktoken entirely. In `char`
mode the memory-injection budget uses a network-free character-based estimate
and never triggers the BPE download from openaipublic.blob.core.windows.net,
which could otherwise block for tens of minutes (see #3402).
Also harden the default `tiktoken` path:
- cache an in-flight LOADING sentinel so concurrent callers fall back
immediately instead of spawning more blocking get_encoding threads when the
first load is still running (e.g. under the 5s startup warm-up timeout);
- cache failures with a timestamp and retry after a cooldown so a transient
network outage self-heals back to accurate counting without a restart;
- skip startup warm-up entirely in char mode.
The new config is surfaced via the memory config API and config.example.yaml
(config_version bumped). Default remains `tiktoken`, so existing deployments
are unaffected.
* fix(memory): use CJK-aware char token estimate and address review feedback
- Replace the flat len(text)//4 fallback with a CJK-aware estimate so
Chinese/Japanese/Korean memory content does not over-fill the injection budget
- Document the internal tiktoken retry cooldown and char-mode escape hatch
- Sync CLAUDE.md / config.example.yaml / MEMORY_IMPROVEMENTS.md wording
- Fix MemoryConfigResponse mocks/assertions and add CJK estimate tests
* add caller identity in replay e2e
* make format
* fix(replay-e2e): stabilize title caller replay
* fix(replay-e2e): use captured caller without run manager
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`client.py` imported the private `_build_middlewares` from `agent.py` across a
module boundary and called it as public API. Because the `_` name signals
"module-private, no external callers", any future rename or signature change
silently breaks the embedded `DeerFlowClient` path — and the test suite even
monkeypatched `deerflow.client._build_middlewares`, baking the leak in.
`DeerFlowClient` is a lead-agent variant that genuinely needs the lead agent's
full middleware composition, so make the dependency honest: promote the helper
to a documented public entry point `build_middlewares` and update every in-repo
caller. Found during #3341 review; #3341 already removed one such leak
(`_assemble_deferred` -> public `assemble_deferred_tools`) and left this one out
of scope on purpose.
- agent.py: rename def + both internal call sites; expand the docstring into a
public-entry-point contract and document the previously-undocumented
model_name / app_config / deferred_setup params
- client.py: import + call site now use the public name (removes the last
cross-module private import)
- scripts/tool-error-degradation-detection.sh: update its import + call site
- tests (5 files): update monkeypatch/patch targets and direct calls
- docs (backend/CLAUDE.md, plan_mode_usage.md, middlewares.mdx): sync the live
references that describe the symbol as current API
Pure mechanical rename, no behavior change. Historical design docs (rfc,
superpowers spec) intentionally keep the old name as point-in-time records.
Closes#3431
* docs(spec): MiniMax integration for generation skills + new music skill
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(plan): MiniMax generation providers implementation plan
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(skills): add importlib loader + FakeResp for skill tests
* test(skills): register loaded module in sys.modules; raise requests.HTTPError in FakeResp
* feat(image-generation): add MiniMax provider with env auto-detect
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(image-generation): guard unknown provider, derive ref MIME, strengthen tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(video-generation): add MiniMax provider with async poll/download
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(video-generation): surface base_resp errors while polling; add timeout test
* feat(podcast-generation): add MiniMax t2a_v2 provider with env auto-detect
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(podcast-generation): restore TTS credential guard; add volcengine + voice tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(music-generation): new MiniMax music skill via skill-creator
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(music-generation): treat empty lyrics as absent; test no-audio-data path
* refactor(skills): add request timeouts to MiniMax network calls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* fix(models): strip inconsistent user-message names for MiniMax chat
DeerFlow middlewares tag user messages with provenance names (user-input, summary, loop_warning); langchain serializes them into the OpenAI-compatible payload and MiniMax rejects mismatched user-message names with "user name must be consistent (2013)". PatchedChatMiniMax now drops the per-message name from user-role messages. Point the config.example MiniMax models at PatchedChatMiniMax so they also get reasoning_content mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(image-generation): MiniMax sends JSON prompt field, guard 1500-char limit
MiniMax image-01 takes one text string capped at 1500 chars, but the skill was sending the whole structured JSON. The MiniMax provider now extracts the JSON `prompt` field (relying on prompt_optimizer to expand it) and fails fast with a clear error before calling the API when that field exceeds 1500 chars. Authoring stays provider-agnostic; Gemini still receives the full JSON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(podcast-generation): per-provider TTS concurrency and retry/backoff
Each TTS provider owns its concurrency internally — MiniMax runs single-threaded to reduce rate-limit failures, Volcengine keeps 4 workers — with automatic retry and backoff on transient HTTP and base_resp errors. No caller-facing concurrency knob.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(skills): address Copilot review comments on generation skills
- video: add raise_for_status + timeout to the Gemini download/POST/poll calls so non-2xx responses surface as clear HTTP errors instead of JSON/KeyError or hangs
- video: check the task Fail status before the generic base_resp check so the failure keeps its task_id context
- video/image: create the output file parent directory before writing (matching music-generation) so nested output paths do not raise FileNotFoundError
- music: require a non-empty prompt and fail fast with ValueError instead of sending an empty prompt to the API
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(scripts): reclaim dev ports across worktrees in make stop/dev
All deer-flow worktrees (main checkout + linked worktrees) hardcode the same dev ports (8001/3000/2026), so a service started from any worktree must be reclaimable from another. stop_all now resolves the set of worktree roots (DEERFLOW_ROOTS) and treats a process as deer-flow-owned when its open files live under any of them. It also force-kills survivors on 2026 alongside 8001/3000, fixing `make dev` aborting on the nginx port preflight when a prior nginx lingered on 2026.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(view-image): hide the injected image-context message from the UI
ViewImageMiddleware injects a HumanMessage (text + base64 images) so the vision model can see viewed images, but it was the only internal injector that set neither hide_from_ui nor a hidden name, so it leaked into the chat UI (and IM channels) as a user bubble reading "Here are the images you've viewed:". Mark it with additional_kwargs={"hide_from_ui": True}, matching todo/dynamic_context injections, which the frontend isHiddenFromUIMessage and the channel sender already honor. The model still receives the full content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(minimax): mark M2.7 models as text-only (no vision)
MiniMax M2.7 / M2.7-highspeed do not support vision; only M3 does. The
provider config asserted vision support for M2.7 in four places.
- config.example.yaml: 4 M2.7 entries -> supports_vision: false
- backend/docs/CONFIGURATION.md: M2.7 + highspeed -> supports_vision: false
- wizard: add LLMProvider.model_vision_overrides + extra_config_for() so
selecting an M2.7 model writes supports_vision: false while M3 (default)
keeps vision; wire it through setup_wizard.py
- tests: M2.7-highspeed fixture -> supports_vision=False; add
test_minimax_vision_is_per_model
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* fix(replay-e2e): match by conversation, not the living system prompt
The model-replay match key hashed the full input including the lead-agent
system prompt. That prompt is edited frequently (e.g. #3195 added a "File
Editing Workflow" section), so the committed fixture went stale the moment
the prompt changed on main — turning the Layer-2 render gate RED on every
unrelated PR (#3430, #3432, ...). This was a self-inflicted false positive.
Root-cause fix:
- replay_provider._canonical_messages now EXCLUDES the system message from
the hash. The conversation (human/ai/tool) is the stable contract that
identifies a recorded turn; the system prompt is an internal detail not
part of the front-back contract under test. (Mirrors how open-design keys
its mock picker on the user prompt, not the system internals.) Proven
robust: injecting a prompt edit no longer causes a replay miss.
- Layer-1 golden was BLIND to replay misses: the gateway swallows a miss
into an assistant error message, so the shape-only golden stayed green on
a stale fixture. It now inspects replay_provider.replay_misses() and fails
loud. (Layer-2 already fails on a miss.)
- Re-recorded write_read_file.ultra fixture + regenerated golden under the
new conversation-only hash.
- Layer-2 render spec: assert the in-graph auto-title (deterministic); the
follow-up suggestion is fired async and depends on a clean JSON model
output, so assert it only when the fixture captured one — never gate on
its absence (recording flakiness must not block CI).
- docs: REPLAY_E2E.md updated.
Verified: Layer-1 golden green (no miss), Layer-2 both specs green,
CI=true make test 4033 passed / 0 failed, frontend pnpm check clean.
* test(replay-e2e): restore suggestions coverage with a reliable capture
Addresses review feedback (the suggestion path was dropped from Layer-2):
- record spec now waits for the `/suggestions` response before checking
capture stability, so the recorded fixture reliably includes the
frontend-fired suggestions turn (previously the stability window could
return before suggestions fired, yielding a fixture without it).
- Re-recorded write_read_file.ultra: 5 turns (write_file, auto-title,
read_file, answer, suggestions). Golden unchanged — suggestions is a
separate /suggestions call, not part of the /runs/stream SSE sequence.
- Layer-2 spec: restore the hard `EXPECTED_SUGGESTION` assertion. With the
record spec now waiting for /suggestions, a fixture missing the suggestion
turn means a broken recording and must fail loud, not pass silently.
Verified: Layer-1 golden green (no miss), Layer-2 both specs green
(auto-title + suggestion render), frontend pnpm check clean.
* ci: re-trigger (flaky Docker Hub image pull in sandbox e2e, unrelated)
backend-unit-tests failed only in test_sandbox_orphan_reconciliation_e2e.py
with 'docker pull busybox:latest ... context deadline exceeded' — a CI-runner
network flake reaching Docker Hub, not related to this docs/tests-only change.
Empty commit to re-run CI.
---------
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
* test(e2e): record/replay front-back contract verification
Guards the front-back contract with a deterministic, key-free record/replay
harness (mirrors open-design's golden-trace approach):
- ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a
normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path
so one fixture replays across days and from both the browser and direct-POST
paths; a miss raises loudly (no silent divergence).
- Recording is record-through-browser (scripts/record_gateway.py +
build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven
through the real frontend so captured inputs match exactly what the browser
sends; fixtures contain no API key.
- Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real
gateway, assert the SSE event sequence == committed golden.
- Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js +
real gateway (replay model) + Chromium; assert the replayed auto-title and
follow-up suggestions render. DOM assertions are the gate; visual regression is
a local dev gate (CI uploads the render as an artifact).
- CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of
the contract (frontend/** or backend gateway/harness/fixtures).
* test(e2e): multi-run render-order cross-stack scenario (#3352)
Guards the dangerous front-back class where a backend ordering change
silently breaks a frontend assumption while both sides' unit tests stay
green. Reproduces issue #3352: backend list_by_thread returns runs
newest-first (#2932) and the frontend prepended per-run pages, inverting
chronological order once the checkpoint no longer held the older messages.
- tests/seed_runs_router.py: test-only seeder, mounted on the replay
gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production
app). Seeds a thread with >=2 runs + per-run message events and no
checkpoint -- the #3352 precondition -- so the frontend per-run reload
path is the sole source of truth and the prepend inversion is observable.
- frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real
frontend against the real gateway, asserts the first run renders above
the second. Reverting the #3354 fix turns it red.
- replay-e2e.yml: trigger on the new replay test-infra paths.
- docs: REPLAY_E2E.md cross-stack scenario section.
* test(e2e): address Copilot review on the replay harness
- Fix stale recorder references (scripts/record_traces.py ->
scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in
replay_provider.py, test_replay_golden.py, _replay_fixture.py.
- MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the
frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the
hashed input (Layer 1 golden still green), but the table now matches the
real frontend context it claims to mirror.
- replay_provider.py docstring: stop claiming memory is recorded-enabled;
the replay config disables memory/summarization for determinism (title
stays, as an in-graph deterministic call).
- record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead
of setdefault, so an outer value can't leak into the hermetic harness.
- record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a
bare KeyError).
- playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when
set, so the gateway raises a clear 'missing env' error instead of getting ''.
* test(e2e): address Copilot review round 2
- seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai']
so a bad value is a clean 422 at the boundary instead of a 500
(KeyError on _EVENT_TYPE).
- record-write-read-file.spec.ts: waitForCaptureStable now throws on
timeout instead of returning the last count, so a truncated/partial
recording can't pass silently.
- real-backend-render.spec.ts: guard the suggestions JSON.parse; a
bracket-prefixed non-JSON turn falls back to '' so the existing
not.toBe('') assertion fails clearly instead of a generic parse throw.
- Add MiniMax-M3 to model list and set as default
- Keep MiniMax-M2.7 and MiniMax-M2.7-highspeed
- Remove older models (M2.5)
- Update related tests
Co-authored-by: octo-patch <octo-patch@github.com>
UploadsMiddleware defines only the sync `before_agent` hook. LangChain wires a
sync-only hook as `RunnableCallable(before_agent, None)`, and LangGraph's
`ainvoke` runs it directly on the event loop when `afunc is None` — so the
per-message uploads-directory scan (`exists`/`iterdir`/`stat` plus reading
sibling `.md` outlines) blocks the asyncio event loop on every message that has
an uploads directory.
Add `abefore_agent` that offloads the scan to a worker thread via
`run_in_executor`; it copies the current context, preserving the `user_id`
contextvar read by `get_effective_user_id()`.
Add a runtime anchor under `tests/blocking_io/` that drives the real
`create_agent` graph via `ainvoke` under the strict Blockbuster gate, so a
regression back onto the event loop fails CI. Update blocking-IO docs.
* test(runtime): add Blockbuster runtime anchor for JsonlRunEventStore async IO
#3084 offloaded `JsonlRunEventStore`'s file IO via `asyncio.to_thread` and added
a mock-based offload assertion (`tests/test_jsonl_event_store_async_io.py`) that
covers `put()` only. That guard is not part of the Blockbuster runtime gate
(`tests/blocking_io/`) run by `backend-blocking-io-tests.yml`.
Add a runtime anchor that drives the full async surface (`put`, `put_batch`,
`list_messages`, `list_events`, `list_messages_by_run`, `count_messages`,
`delete_by_run`, `delete_by_thread`) under the strict Blockbuster gate, so any
blocking IO reintroduced on the event loop in any of these methods fails CI —
not only removal of a specific `to_thread` call. Verified each offloaded method
goes red when its offload is reverted. Test-only; no production change.
* test(runtime): exercise list_events event_types filter branch
Per review feedback: the anchor called list_events without event_types,
so the filter branch never ran after _read_run_events' filesystem IO.
Add a second list_events call with event_types=["message"] so the full
read path -- including the filter branch -- executes under the gate.
* fix(loop-detection): defer warn injection to wrap_model_call
The warn branch in LoopDetectionMiddleware injected a HumanMessage
into state from after_model. The tools node had not yet produced
ToolMessage responses to the previous AIMessage(tool_calls=...), so
the new HumanMessage landed *between* the assistant's tool_calls and
their responses. OpenAI/Moonshot reject the next request with
"tool_call_ids did not have response messages" because their
validators require tool_calls to be followed immediately by tool
messages.
Detection now runs in after_model as before, but only enqueues the
warning into a per-thread list. Injection happens in wrap_model_call,
where every prior ToolMessage is already present in request.messages.
The warning is appended at the end as HumanMessage(name="loop_warning")
— pairing intact, AIMessage semantics untouched, no SystemMessage
issues for Anthropic.
Closes#2029, addresses #2255#2293#2304#2511.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(channels): remove loop warning display filter
* feat(loop-detection): scope pending warnings by run
* docs(loop-detection): update docs
* test(loop-detection): assert deferred warnings are queued
* fix(loop-detection): cap transient warning state
* docs: update docs
* add async awrap_model_call test coverage
* docs(loop-detection): document transient warnings
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(auth): persist auto-generated JWT secret to survive restarts
When AUTH_JWT_SECRET is not set, the auto-generated secret is now
written to .deer-flow/.jwt_secret (mode 0600) and reused on subsequent
starts. This prevents session invalidation on every restart while still
allowing explicit AUTH_JWT_SECRET in .env to take precedence.
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix the lint errors of backend
---------
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* docs: document auth design and user isolation
* docs: align auth docs with current storage and reset behavior
---------
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
* fix(nginx): defer cors to gateway allowlist
Remove proxy-level wildcard CORS handling so browser origins are controlled by the Gateway allowlist and stay aligned with CSRF origin checks.
* docs: document gateway cors allowlist
Clarify that same-origin nginx access needs no CORS headers while split-origin or port-forwarded browser clients must opt in with GATEWAY_CORS_ORIGINS.
* docs(gateway): record cors source of truth
Document that Gateway CORSMiddleware and CSRFMiddleware share GATEWAY_CORS_ORIGINS as the split-origin source of truth.
* fix(gateway): align cors origin normalization
* docs: clarify gateway langgraph routing
* docs(gateway): update runtime routing note
* fix(harness): resolve runtime paths from project root
* docs(config): update
* fix(config): address runtime path review feedback
* test(config): fix skills path e2e root
* test(config): cover legacy config fallback when project root lacks config files
Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no
config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back
to the legacy backend/repo-root candidates — the backward-compat path
requested in PR #2642 review.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
