init_engine runs on the FastAPI lifespan event loop, but created the SQLite
data directory with a synchronous os.makedirs (a stat + mkdir syscall),
blocking startup. Dispatch it via asyncio.to_thread, mirroring the #1912 fix
for the checkpointer's ensure_sqlite_parent_dir.
Adds a Blockbuster-gated regression test in tests/blocking_io/ that drives the
real init_engine path with a not-yet-existing sqlite_dir; it trips
BlockingError if the makedirs regresses onto the event loop.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): reset new chat on client-side navigation
Drive the chat reset effect with Next.js's reactive pathname instead of the render-time window.location-derived flag.
During App Router transitions, window.location may still point to the previous thread until commit, leaving chat state stale until another UI interaction triggers a render. Preserve the stale 'new' param guard so created thread UUIDs are not overwritten.
* test(frontend): add e2e to cover history-only new chat reset
* docs(frontend): clarify new chat pathname synchronization
* feat: persist AI turn duration in backend and UI
* chore: restore uv.lock to match main
* refactor(frontend): use Math.floor instead of Math.ceil for reasoning timer
Follow-up to #3301 / #3344. The Gateway-embedded runtime is the standard
topology — there is no standalone LangGraph service — but two stale
references slipped past the cleanup guard:
- .github/copilot-instructions.md still told contributors that `make dev`
"Starts LangGraph (2024)" and wrote logs/langgraph.log.
- SandboxAuditMiddleware's docstring pointed audit logs at langgraph.log.
Update both to the Gateway-embedded reality (gateway.log) and extend
test_gateway_runtime_cleanup.py to pin the agent-instruction docs so the
standalone references can't reappear.
Stays within the safe scope of #3304: does not touch langgraph_auth.py,
langgraph.json, or the langgraph-api / langgraph-cli / langgraph-runtime-inmem
deps, which the issue gates on maintainer confirmation of Studio / direct
LangGraph Server support.
Dependabot security update #3620 bumped cryptography to 48.0.1 in uv.lock
but left the harness manifest at >=43.0.0, since the dependency lives in
the workspace member packages/harness, not backend/pyproject.toml. The
drift caused 'uv lock' to keep rewriting the recorded specifier. Align the
source constraint with the locked version.
* fix(scripts): avoid lsof hang during make dev cleanup on NFS
`_is_deerflow_pid` and `_report_reclaimed_ports` call `lsof -p <pid>` to
enumerate a process's open files. On hosts whose working tree or home is
on a network filesystem (NFS/autofs), `lsof -p` blocks indefinitely on the
kernel stat calls, so `make dev` / `make stop` hang forever at
"Stopping all services...".
Add `-b` (avoid kernel blocking functions) and `-w` (suppress the
resulting warnings) to both calls. The network-only `lsof -nP -iTCP`
probes are unaffected and already returned quickly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(nginx): set global error_log so local-dev nginx starts as non-root
nginx.local.conf only declared `error_log` inside the `http {}` block.
nginx opens its compiled-in default error log (on Debian/Ubuntu builds,
the absolute /var/log/nginx/error.log) at startup, before it reaches the
http-block directive. When `make dev` launches nginx as a non-root user
that path is not writable, so startup fails with:
[emerg] open() "/var/log/nginx/error.log" failed (13: Permission denied)
Declare a global (main-context) `error_log logs/nginx-error.log warn;`.
Combined with the existing `-p $REPO_ROOT`, logging resolves to the
repo-local logs/ directory and nginx starts without elevated privileges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
should_ignore_name runs once per directory entry during glob/grep tree
walks and looped over ~57 IGNORE_PATTERNS calling fnmatch per pattern.
Precompute the literal names into a frozenset (O(1)) and the few glob
patterns into one combined compiled regex via fnmatch.translate; per name
it's now one normcase + a set lookup + at most one regex match.
os.path.normcase preserves fnmatch's platform case behavior, so results
are identical (covered by an equivalence test against the old loop).
Fixes#3655.
* fix(mcp): migrate local MCP-produced files into sandbox outputs (#3597)
Stdio MCP servers (e.g. Playwright) write files to host paths that the
sandbox/artifact API cannot resolve, since it only serves paths under
/mnt/user-data. Copy local files referenced by ResourceLink results into
the thread's sandbox outputs dir and rewrite their URIs to
/mnt/user-data/outputs/... so they become readable.
Also scope pooled MCP sessions by user_id:thread_id instead of thread_id
alone, matching the per-(user_id, thread_id) filesystem isolation.
* fix(mcp): restrict file migration to trusted source roots (#3597)
Add a source-root allowlist to the MCP file-migration path so a
malicious or buggy MCP server cannot have us copy arbitrary host files
(e.g. /etc/passwd) into a thread's outputs directory, from where the
artifact API would serve them. Files are migrated only when located
under the OS temp dir (Playwright's default), the thread's own
user-data tree, or an operator-configured root via
DEERFLOW_MCP_MIGRATION_SOURCE_ROOTS.
Expand test coverage with allowlist/security cases (path escape refusal,
trusted-root acceptance), URL-encoded file:// paths, converter content
branches (image/embedded/error/structured), and copy/resolve failure
fallbacks.
* fix(mcp): harden local file migration into sandbox outputs
Address robustness and security gaps in the MCP ResourceLink file
migration:
- Set migrated files to 0o644 so a differently-UID sandbox container can
read them, instead of inheriting the source's (possibly 0o600) mode.
- Enforce the 100MB size cap during the copy (chunked, byte-counted)
rather than from a prior stat(), closing the grow-after-stat TOCTOU.
- Create the destination atomically with O_CREAT|O_EXCL to remove the
check-then-create name-collision race.
- Document the shared-$TMPDIR multi-tenant read surface and mitigation.
Add regression tests: symlink escape refusal, explicit $TMPDIR source
migration, 0o644 mode, and the outputs/user-data resolve() OSError
fallback branches.
* fix(mcp): migrate playwright text file outputs
* fix(mcp): translate MCP file outputs to virtual paths instead of copying (#3597)
Pin stdio MCP subprocess cwd and TMPDIR/TMP/TEMP under the thread workspace
so produced files always land in the mounted user-data tree, then rewrite
returned references via deterministic host->virtual path translation. Free
text is best-effort only: a reference is rewritten only when it resolves to
an existing file inside the thread's tree, and bare filenames are matched
against files created/modified by the same tool call. Replaces the previous
copy-into-outputs + regex approach (which missed cases like temp/page-*.yml).
* style(mcp): apply ruff format to mcp path translation tests
* perf(mcp): offload stdio FS work off event loop and gate on transport
Address review on #3600:
- Wrap the workspace dir prep, snapshot diff, and per-token path
resolution in asyncio.to_thread so they no longer block the event
loop (matches the repo's blocking-IO gate convention).
- Gate the cwd/temp pinning and snapshots on stdio transport only;
SSE/HTTP servers skip the filesystem work entirely.
- Skip the post-call snapshot diff when the result has no text content.
* test(mcp): cover stdio transport gating and text-content after-walk skip
Add unit/integration coverage for the new review-driven behavior:
- _prepare_stdio_workspace dir/temp/snapshot bundle
- _result_has_text_content detection (text, embedded text, image, empty)
- non-stdio transport skips cwd/temp pinning and touches no workspace dirs
- post-call snapshot diff is skipped without text content and runs with it
* fix(mcp): address stdio path rewrite review feedback
- Restrict the stdio MCP temp directory to 0700 instead of 0777.
- Preserve operator-provided stdio cwd values while keeping injected cwd values as strings.
- Add debug logging for deterministic path rewrites and bare-filename rewrite decisions.
- Document the stdio cwd/temp pinning, virtual-path translation, and user/thread session scope.
- Cover explicit cwd preservation and temp-dir permissions in session-pool tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* perf(sandbox): cache LocalSandbox path-rewrite regexes per instance
LocalSandbox re-sorted path_mappings and re-compiled its path-rewrite
regex on every tool call: _resolve_paths_in_command (every bash),
_resolve_paths_in_content (every write_file), and
_reverse_resolve_paths_in_output (every bash output / read_file). The
inputs are derived solely from self.path_mappings, which is assigned once
in __init__ and never mutated, so the work is identical every call.
Compile the patterns once per sandbox via functools.cached_property and
reuse them; hoist 'import re' to module scope. Behavior is unchanged —
only the per-call sort+escape+compile on the agent's hot path is removed.
Fixes#3647. Adds tests covering caching identity, unchanged rewriting,
path-segment boundary matching, and the empty-mappings pass-through.
* perf(sandbox): also cache resolved local paths and sorted mapping views
Beyond the regex compilation, _find_path_mapping, _is_read_only_path,
_resolve_path_with_mapping and _reverse_resolve_path re-sorted
path_mappings and re-ran Path(local_path).resolve() (a filesystem
syscall) on every call. Since path_mappings is immutable, cache the
resolved local root per mapping and the two sorted views via
cached_property and reuse them. Behavior is unchanged; the reverse-output
pattern builder now reuses the same resolved-path cache.
* fix(frontend): render full content for multi-part AI messages
Gemini streams the first content block as a {type:text} object carrying
the thinking signature, then emits continuation deltas as bare strings.
The content extractors only kept {type:text} objects and dropped the
string parts, truncating each AI message to its first block.
Handle string elements in extractContentFromMessage, extractTextFromMessage,
and textOfMessage so the full message renders.
Fixes#1000
* test(frontend): cover multi-part AI message content extraction
Add a regression test for the #1000 truncation: an AI message whose
content is [{type:text, signature}, "bare string"] (Gemini's shape after
LangChain merge_content). Fails on main, passes with the extractor fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(frontend): cover textOfMessage multi-part content extraction
Add tests for the bare-string continuation case and the null-when-empty
contract, and document why textOfMessage joins flat ("") while the body
extractors join with "\n". Addresses review feedback on #3649.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(gateway): attribute token usage to actual models
Capture per-call model names from LLM response metadata for lead, middleware, and subagent calls.
Persist a per-run token_usage_by_model breakdown and aggregate by that map in both SQL and memory stores, with legacy fallback to the run-level model_name for older rows.
Add regression coverage for by_model totals, caller consistency, active progress snapshots, store parity, and SubagentTokenCollector model propagation.
* fix(gateway): harden by-model token aggregation
Use usage.get("total_tokens", 0) when reducing per-model token usage maps so aggregation tolerates partially written or manually edited JSON blobs without changing behavior for journal-written rows.
* docs(gateway): clarify by-model run count semantics
Document that by_model[*].runs counts the number of runs in which a model appeared, so multi-model runs can increment multiple model buckets.
Base.to_dict() and __repr__() ran sqlalchemy.inspect(type(self)).mapper
reflection on every call, but to_dict() is invoked once per row when
serializing ORM results (e.g. every event in a messages/events page).
The mapped columns are fixed at class-definition time, so cache the
column keys per class with functools.cache and iterate the cached tuple.
Behavior is unchanged.
Fixes#3653.
* feat(suggest_agent): stop frontend from fetching when suggestions disabled
* wait for suggestions config to load before fetching
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): resolve stale subagent running state after stop
Subtask cards were recreated from task tool calls as in_progress whenever
the thread was loading. If a user stopped a run before the task tool returned
a ToolMessage, the card could remain running after reload, and could flip back
to running when a later user turn started streaming.
Derive pending subtask state from the current assistant turn instead of the
global thread loading flag. A task now stays in progress only while its own
turn is loading or while a matching ToolMessage exists for result parsing;
otherwise it is marked failed.
Add unit coverage for task ToolMessage matching and for the later-turn loading
regression.
* fix(frontend): refresh subagent card on terminal status transition
- notify React via a ref + post-render effect when a subtask flips to
completed/failed, so SubtaskCard updates without relying on an
unrelated MessageList re-render
- tighten the current-turn heuristic to `groupIndex === lastGroupIndex`
(precomputed once) — fixes the rare case where an earlier subagent
group in the same turn was kept in_progress, and drops the per-group
slice+some scan to O(1)
- rename `getPendingSubtaskStatus` → `derivePendingSubtaskStatus`
- narrow `toolCall.id` at MessageList call sites; skip task tool calls
without an id instead of `id!`
- add Playwright e2e covering reload-after-stop showing `Subtask failed`
The SSE stream's `values` snapshots serialize the full state, including the
hide_from_ui human messages that ViewImageMiddleware fills with base64 image
payloads. #3535 stripped those from the REST wait/history/state endpoints via
serialize_channel_values_for_api, but the streaming path (worker publishes
serialize(chunk, mode="values")) still went through serialize_channel_values,
so the same base64 leaked to the frontend over SSE. Route values-mode
serialization through serialize_channel_values_for_api as well. Non-hidden
messages and https image URLs are left untouched.
* fix(channels): scope IM files and helper commands to owner
* fix(memory): honor bound IM owner for /memory gateway endpoints
The channel manager already attaches X-DeerFlow-Owner-User-Id for /memory
and /models, but the memory router resolved user_id solely from
get_effective_user_id(), which returns the synthetic internal user
(DEFAULT_USER_ID) for channel workers. A bound IM /memory therefore read
the default/internal memory instead of the connection owner's.
Resolve the owner via _resolve_memory_user_id(request) across all
/api/memory* endpoints: trusted internal callers act for the owner header,
browser/API callers fall back to get_effective_user_id(). Mirrors the
threads router's get_trusted_internal_owner_user_id pattern, completing
acceptance criterion #3 of #3539.
Add end-to-end tests asserting the resolved user_id (not just that the
header is sent) and that a spoofed owner header from a browser user is
ignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): align memory bucket and reuse cached storage owner
Address PR #3579 review feedback:
- Memory router now sanitizes the trusted owner header via make_safe_user_id
before routing, matching the channel file pipeline
(_safe_user_id_for_run/prepare_user_dir_for_raw_id). A bound owner id needing
sanitization now resolves to the same bucket as its files/uploads instead of
500ing in _validate_user_id.
- _handle_chat reuses the storage_user_id cached at the top of the method for
artifact delivery instead of re-deriving _channel_storage_user_id(msg), so
uploads and outputs cannot drift to different buckets if a channel rewrites
the InboundMessage in receive_file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stage unbound IM files under the run's user bucket
Address PR #3579 review feedback (#5): _channel_storage_user_id now mirrors
_resolve_run_params' identity policy, falling back to safe(msg.user_id) instead
of returning None for unbound auth-enabled channels.
Previously an unbound msg ran under safe(platform_user_id) but staged uploads
under get_effective_user_id() in the dispatcher task (unset contextvar ->
"default"), so files landed in users/default/... while the agent read from
users/{safe_platform_user_id}/.... Bound and unbound channels now write where
the agent reads. Returns None only when no identity is available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): reuse cached storage owner in streaming artifact delivery
Address PR #3579 review feedback (#6): thread the storage_user_id resolved in
_handle_chat into _handle_streaming_chat instead of re-deriving
_channel_storage_user_id(msg) in the finally block. Avoids re-running
_safe_user_id_for_run (and its possible filesystem touch) on the streaming-error
path and guarantees artifact delivery targets the same bucket as the uploads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): document owner-scoped IM file storage
Address PR #3579 review feedback (#4): the IM Channels and File Upload sections
still described pre-PR default-bucket behaviour. Document that receive_file,
_ingest_inbound_files/ensure_uploads_dir/get_uploads_dir, and
_resolve_attachments/_prepare_artifact_delivery are owner-scoped via the user_id
kwarg, and that the bucket matches the memory bucket from _resolve_memory_user_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): unify run identity and storage bucket resolution
Address PR #3579 review feedback (#3): _resolve_run_params no longer duplicates
the owner-resolution rule inline. After the #5 fix the inline block and
_channel_storage_user_id computed the identical sanitized-with-platform-fallback
value, so the run identity now calls the same helper, making it the single
source of truth for run_context["user_id"] and the file/artifact storage bucket.
_owner_headers stays deliberately separate: it sends the raw owner id over HTTP
for the gateway to re-resolve (no sanitize, no platform fallback), documented on
both helpers. test_run_identity_matches_storage_bucket pins the two together so
they cannot drift again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden runtime credential management APIs
* fix(channels): address review feedback on credential hardening
Follow-up to the runtime credential-hardening pass, resolving five review
findings:
- WeChat auth persistence now writes through a 0o600 NamedTemporaryFile +
Path.replace instead of write_text-then-chmod, so the iLink bot_token is
never briefly readable at umask defaults (mirrors ChannelRuntimeConfigStore).
- The post-write chmod is split into its own try/except: a chmod failure on a
filesystem without POSIX perms now logs at debug instead of masquerading as
a "failed to persist" warning.
- Extracted the three near-identical _require_admin_user helpers (mcp,
channel_connections, channels) into a single require_admin_user(request, *,
detail) in app/gateway/deps.py; each router supplies its own detail string.
- Strengthened the runtime-config-store chmod coverage: a new test injects a
temp-file chmod failure and asserts it is logged at debug while the
destination is still owner-only (mutation-verified to fail if the chmod is
dropped), plus a loose-pre-existing-file case.
- Removed the unused _FakeRepo from the blocking-io test: its isinstance gate
routes through the repo-less 503 path, so neither stub was ever invoked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): make channel connect flow deterministic
* make format
* fix(channels): apply connect-code before allowed_users on telegram and wechat
The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and
WeChat still gate _check_user/allowed_users before connect-code dispatch, so
a newly allowlisted-but-unbound user is silently rejected when binding via the
browser deep-link / connect-code flow — the same deadlock the PR fixes.
- telegram: consume the /start deep-link token before the allowed_users gate.
- wechat: handle the /connect code before the allowed_users gate, and defer
inbound file extraction + context-token tracking past the gate so blocked
senders no longer trigger CDN downloads or token bookkeeping.
Adds regression tests for both adapters mirroring the slack/dingtalk coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): enforce single-active-owner invariant at the DB layer
_revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row
lock or constraint covering active rows. Under READ COMMITTED, two concurrent
connect-code consumes for the same (provider, external_account_id, workspace_id)
from different owners could each observe "no other active owner" and both commit
a connected row, leaving find_connection_by_external_identity nondeterministic.
- Add a partial unique index on (provider, external_account_id, workspace_id)
WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the
database guarantees at most one non-revoked row per external identity.
- Reorder upsert_connection to revoke other owners' active rows before the new
connected row is flushed (so the index is satisfied at commit), wrapped in a
bounded rollback-and-retry loop. A losing concurrent writer now retries
against the now-visible state instead of committing a duplicate.
Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden connect-status polling primitive
pollChannelConnectionUntilResolved was a free-floating recursive setTimeout
started from onSuccess with no cancellation, no per-provider dedup, a redundant
second endpoint per tick, and an unbounded loop on a non-finite expires_in.
- Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls
only listChannelConnections() and invalidates the providers query once when the
bind resolves, instead of fetching both endpoints every tick.
- Guard expires_in with a finite check + default window so undefined/NaN can no
longer produce a poll loop that runs until the page closes.
- Track one active poll handle per provider in useConnectChannelProvider via a
ref Map: a new connect cancels the prior poll for that provider, and a useEffect
cleanup cancels all polls on unmount.
Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics
Moving the allowed_users gate past _extract_text meant the parsed-message INFO
log (text=%r, first 100 chars) fired for senders that allowed_users would have
rejected, defeating the filter's noise/privacy role. Move that log to after the
allowed_users gate so blocked senders' message text never reaches INFO logs.
Also document the two operator-relevant semantic changes in backend/CLAUDE.md:
connect-code dispatch runs before allowed_users (so allowed_users is no longer a
bind-time defense; the model relies on code confidentiality + 600s TTL + one-time
consumption), and the single-active-owner-per-external-identity transfer semantics
now backed by the partial unique index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide
Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md:
connect codes are consumed before allowed_users (so a not-yet-allowlisted user can
still complete a first bind, and allowed_users is not a bind-time defense), and an
external identity has at most one active owner with last-bind-wins transfer enforced
at the DB layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): lift connect-code dispatch into Channel base class
Each adapter duplicated the ordering-sensitive boilerplate of extracting a
/connect code and guarding on the connection repo before its allowed_users gate.
The duplication is what let telegram/wechat drift and keep the gate ahead of the
bind. Centralize it:
- Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments).
- Add Channel._pending_connect_code(text), which guards on the repo and extracts
the code, documenting that adapters MUST consult it before authorization so a
browser-initiated bind can bootstrap a not-yet-authorized identity.
- Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper.
This also fixes a latent inconsistency where slack dispatched a bind even when
no connection repo was configured.
Pure refactor — the full channel suite stays green; adds a direct unit test for
the base helper's contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
* fix(channels): redact DingTalk parsed-message INFO log content
Log text_len instead of the first 100 chars of message text, so message
content never reaches INFO logs (the after-gate move already keeps blocked
senders out entirely). This takes over the redaction from #3584 so only this
PR touches dingtalk.py, letting the two PRs merge in any order conflict-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): add operational guardrails
* make format
* fix(channels): converge with #3582 to avoid merge-order conflicts
Drop this PR's DingTalk INFO-log redaction and hand it to #3582, which
already restructures that handler and will redact the same log there. This
PR no longer touches dingtalk.py, so the two PRs can merge to main in any
order without a conflict.
For WeChat, drop the contested thread_ts priority reorder (review #3) and
keep only what inbound dedupe needs: a server-stable message_id in the
inbound metadata (message_id/msg_id, no client_id per review #6). This is a
single added line inside the metadata dict, a region #3582 never touches, so
it auto-merges regardless of order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): address three correctness review findings
1. Connect-code cap was racy (willem #1): _create_state ran delete-expired,
count, and insert as three separate transactions, so concurrent connect
POSTs from one owner could each see count < cap and all insert past it. Add
ChannelConnectionRepository.create_oauth_state_within_cap which does
delete+count+insert in a single transaction serialized per (owner,
provider) — Postgres via pg_advisory_xact_lock, SQLite via the write lock
the leading DELETE takes — and have the router use it.
2. Inbound dedupe key fell back to "" workspace (willem #3): two workspaces
delivering without team/guild/aibotid would collapse to the same key and
dedupe each other's messages. _inbound_dedupe_key now fails closed
(returns None) when no workspace identifier is present.
3. Dedupe key was recorded on receipt and never released on failure
(ShenAC #1): a transient error (DB blip, Gateway 503) left the key in place
for the full TTL, so a provider redelivery of the same message_id — exactly
the retry dedupe should absorb — was silently dropped. _handle_message now
releases the key in the unexpected-exception branch so redelivery can
recover, while keeping record-on-receipt so retries during handling are
still deduped.
Tests: repo cap enforcement incl. concurrent-issuance non-leak; dedupe
fail-closed; dedupe key release-on-failure redelivery recovery.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): address cleanup/efficiency and test review findings
Efficiency / cleanup:
- Dedupe key set drops client-generated ids (client_msg_id, client_id);
keep only server-stable event_id/message_id/msg_id, which a provider's own
redelivery preserves (ShenAC #6). Every provider already emits message_id.
- TTL/overflow pruning of _recent_inbound_events is now O(k): switch to an
OrderedDict and popitem(last=False) from the front instead of scanning all
4096 entries on every inbound (willem #4).
- Log "received inbound" only after the dedupe check so a provider retrying N
times no longer logs N accepts; document that manager dedupe covers the
agent run/final answer, not provider ack side-effects (willem #5, ShenAC #2).
- Slack drops the redundant `team_id or event.get("team")` fallback the caller
already resolved (willem #6).
- create_oauth_state_within_cap prunes only this owner/provider's expired codes
instead of a global DELETE on every connect POST; global cleanup still runs
on consume_oauth_state (willem #7).
Tests:
- Dedupe test uses tmp_path instead of a leaked mkdtemp, uses distinct objects
per publish, and adds a negative control: a different message_id is still
processed, catching over-dedupe regressions (willem #8, ShenAC #4).
- Slack HTTP-mode rejection test supplies app_token so the missing-token early
return can't mask the guard, giving the state assertions teeth (ShenAC #3).
- count_oauth_states test pins that the active row survives, not just the count
(ShenAC #5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): stop flagging string-literal path fragments as unsafe paths
The host-bash absolute-path guard scans the raw command string, so /segment
sequences inside string literals, f-strings, and templates were treated as
absolute path arguments and rejected — e.g. python -c "print(f'/端口{port}')"
or a REST template /devices/{id}/port. Whether a fragment tripped the guard
depended on the character right before the slash (a word char suppressed the
match), so the same literal could pass or fail unpredictably, pushing the model
into retry loops that bloat context and wall-clock time.
Exempt matches carrying non-ASCII characters or format braces: real host paths
a command would open contain neither, so these are text, not paths. The guard
is best-effort (not a security boundary), and plain ASCII host paths like
/etc/passwd — including ones written inside a code string such as
open('/etc/passwd') — stay rejected.
* fix(sandbox): only exempt identifier-template braces, not bash brace expansion
The literal-fragment exemption exempted any path fragment containing { or },
which let bash brace expansion (cat /etc/{passwd,shadow}) and ${VAR} expansion
reconstitute real host paths past validate_local_bash_command_paths. Tighten
the brace branch to only exempt fragments where every {...} block is a single
identifier-like placeholder (/devices/{id}/port, f-string /{port}); reject
${VAR} shell-variable expansion. Add parametrized regression tests for the
brace-expansion and shell-var bypasses.
* feat(community): add Serper Google Images provider for image_search
Add a Serper-backed `image_search` tool alongside the existing Serper
`web_search` provider, so users with a SERPER_API_KEY can pull Google
Images results as reference images for downstream image generation.
- Share request/response handling between web_search and image_search
via `_serper_post` / `_response_items`, with bounded `max_results`
(capped at 10) and query normalization.
- Add a best-effort SSRF guard (`_safe_public_url`) that rejects
non-http(s), localhost and private/non-global IP image URLs; filtered
entries are dropped and never consume the result limit.
- doctor: flag literal `api_key` values in config as a warning and steer
users toward `.env` + `$SERPER_API_KEY`.
- Docs/config: document the Serper image_search provider and SERPER_API_KEY,
and discourage committing literal keys to config.yaml.
- Tests: cover the provider end-to-end (100% line coverage on tools.py)
and the doctor literal-key warning path.
* fix(community): block obfuscated IPv4 literals in Serper image SSRF guard
The image_search SSRF guard only rejected dotted-decimal IP literals; encoded
forms such as decimal (http://2130706433/), hex (0x7f000001) and octal
(0177.0.0.1) raised ValueError in ip_address() and were allowed through, even
though many HTTP clients resolve them to private addresses like 127.0.0.1.
Add _decode_ipv4() to permissively decode these inet_aton-style encodings and
apply the same is_global check; hostnames that do not decode to an IP (e.g.
cafe.com) are still treated as hosts and left to fetch-time re-validation.
Addresses PR review feedback. Tests cover decimal/hex/octal loopback and
private encodings plus non-IP edge cases; tools.py stays at 100% line coverage.
* test(community): cover IPv4-mapped IPv6 URL filtering
* fix(community): address Serper image search review feedback
- Block trailing-dot hostname SSRF bypass (localhost./127.0.0.1.) in
_safe_public_url by stripping the FQDN root label before checks.
- Keep a filtered image/thumbnail URL empty instead of collapsing onto
its counterpart, preserving the high-res/preview contract.
- Evaluate the SSRF guard once per field rather than twice.
- Treat a null-typed organic/images field as "no results" rather than a
malformed payload.
- doctor.py: when a config $VAR is unset, fall through to the default env
var before reporting it as not set.
read_file decodes with UTF-8. Binary uploads (.xlsx, images, ...) raise
UnicodeDecodeError in the sandbox layer. UnicodeDecodeError is a ValueError
subclass, not an OSError, so it bypassed the typed handlers and fell through
to the generic except, surfacing a vague "Unexpected error reading file"
message. The model could not tell the file was binary, so it retried
read_file instead of switching to bash + pandas/openpyxl, burning LLM
round-trips and bloating context with repeated failures.
Add a dedicated UnicodeDecodeError handler that tells the model the file is
binary and to use bash with a suitable library (or view_image for images).
The subagent execution path did not call inject_langfuse_metadata(...)
and built its model with attach_tracing=True, so subagent LLM/tool
spans landed in Langfuse as isolated top-level traces carrying fresh
session ids and the default user. They were findable in the unfiltered
trace list but did not group under the parent thread's session card,
and Langfuse cost attribution for subagent traffic did not line up
with the parent conversation — even though DeerFlow's internal token
accounting (SubagentTokenCollector) was already correct.
Extend the lead-agent tracing wiring to the subagent path so a single
subagent run produces one trace that shares the parent thread's
session_id and user_id, with a subagent:<name> trace name:
- subagents/executor.py: append build_tracing_callbacks() output to
run_config["callbacks"] (preserving SubagentTokenCollector) and
call inject_langfuse_metadata(...) with thread_id, user_id, and
the normalized subagent:<name> trace name. Build the model with
attach_tracing=False so model-level tracing does not double-count
with the graph-root callbacks — the same pairing the lead agent
uses.
- tools/builtins/task_tool.py: resolve user_id via
resolve_runtime_user_id(runtime) at the parent tool layer (before
the background thread starts) and thread it through
SubagentExecutor.__init__, because the _current_user contextvar
is not guaranteed to survive the _execution_pool boundary.
Trace topology is unchanged: subagent traces remain separate top-level
traces in the same session, not nested as child spans under the lead
trace (Plan B follow-up).
Tests: tests/test_subagent_executor.py::TestSubagentTracingWiring
covers the callback append, the session/user/trace-name injection,
the disabled-langfuse no-op, the DEFAULT_USER_ID fallback, the
empty-name trace-name fallback, and the env-tag emission. Existing
test_create_agent_threads_explicit_app_config_to_model_and_middlewares
now also asserts attach_tracing=False.
Docs: CLAUDE.md Tracing System section documents subagents/executor.py
as a third injection point alongside worker.py and client.py.
* fix(sandbox): create shell session before retrying on a fresh id
The AIO sandbox recovery path generated a UUID and passed it straight to
exec_command(id=...). The sandbox image only auto-creates a session when
exec_command is called with *no* id; an exec carrying an unknown id returns
HTTP 404 "Session not found". So every ErrorObservation recovery itself
404'd, turning a transient session lapse into an unrecoverable tool error
that looped the run up to the LangGraph recursion limit.
Explicitly create_session(id=fresh_id) before targeting that id on retry.
create_session is idempotent (returns the existing session if the id already
exists), so this is safe under the serializing lock.
Updated the regression test to assert the retry targets exactly the
created session id rather than a fabricated, uncreated one.
* fix(sandbox): release the one-shot recovery session after retry
The fresh session created on the ErrorObservation recovery path is used for
exactly one command -- the next execute_command runs with no id and returns
to the default session. Under persistent session corruption every command
would create another session that is never reused or released, accumulating
sessions on the container.
Release it best-effort with cleanup_session() in a finally, swallowing any
cleanup error so it never masks a successful retry.
Addresses review feedback on #3577.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
