Add deerflow.community.serply.tools:web_search_tool, a Google SERP
provider for the web_search slot that also covers Google News and Google
Scholar through an optional `vertical` config option. Reads the key from
api_key in config.yaml or SERPLY_API_KEY, clamps max_results to Serply's
1-100 range, and returns the same structured JSON errors as the Serper
and Brave tools.
Register the provider in config.example.yaml, scripts/doctor.py,
scripts/wizard/providers.py, .env.example, backend/docs/CONFIGURATION.md,
the en/zh tools.mdx provider tabs, and tools/AGENTS.md. Tests mock httpx.
normalize_filename accepts names up to 255 UTF-8 bytes, but
claim_unique_filename appended _N to the stem without re-checking the
budget. A duplicate at maximum length therefore produced a 257-byte
name, and the write path (open_upload_file_no_symlink ->
normalize_filename) rejected it with ValueError. In the Gateway upload
route that error falls into the generic handler: the whole request
fails with a 500 and files already written in the same batch are rolled
back — including unrelated ones. The same helper backs the Feishu and
DingTalk channel downloads and client-side attachment staging.
Truncate the stem on a UTF-8 code-point boundary when appending the
dedupe tag would exceed 255 bytes, so the result always round-trips
through normalize_filename. Names short enough to fit keep the exact
dedupe shape they had before.
Tests: red on main, green here —
- unit: max-length dedupe stays within the limit and round-trips;
repeated collisions stay unique; multibyte stems truncate on a
code-point boundary; short names keep the historical _N shape
- router: a batch with a max-length duplicate now succeeds and keeps
every file instead of failing with a 500
Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
* fix(skills): accept portable frontmatter forms
* fix(skills): normalize portable tool names
* Safely preserve parenthesized portable skill tool patterns
Portable Agent Skills declarations such as Bash(tvly *) contain spaces inside a command pattern. Keep those patterns as single literal entries while preserving exact names from the existing YAML-list form, so skill loading no longer fragments valid metadata or rewrites mixed-case MCP tools.
Constraint: DeerFlow's current skill policy matches exact tool names and does not inspect Bash arguments
Constraint: Agent Skills scalar syntax uses whitespace-separated entries with parenthesized command patterns
Rejected: raw.split() | fragments Bash(tvly *) into unrelated tool names
Rejected: normalize YAML-list entries | breaks case-sensitive MCP/runtime tool names
Rejected: map Bash(...) to bash | broadens command-scoped declarations into unrestricted shell access
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Bash(...) entries literal and inactive until DeerFlow has an explicit command-pattern authorization model
Tested: 175 focused parser, validation, installer, review, loader, and tool-policy tests; Ruff check and format; compileall; git diff --check
Not-tested: Full backend suite stopped at pre-existing Windows mode assertion test_runtime_config_store_file_is_owner_only
Related: #4912
* Preserve exact custom tool names in portable skill parsing
Portable scalar frontmatter needs alias normalization for known DeerFlow-compatible names, but generic case conversion corrupts MCP and custom tool identifiers. The tokenizer also treated quoted or escaped parentheses as structural delimiters, rejecting valid command patterns. Preserve unknown names and parse quoted or escaped patterns without broadening Bash(...) into bash.
Constraint: Runtime skill policy uses exact tool-name matching
Constraint: Parenthesized patterns remain literal because argument-level authorization is not implemented
Rejected: Generic CamelCase-to-snake_case for every scalar | rewrites custom/MCP names
Rejected: Map Bash(...) to bash | broadens command-scoped declarations into unrestricted shell access
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Add an explicit alias before supporting another portable tool name; keep command-pattern authorization separate
Tested: 225 skills tests passed, 1 skipped; Ruff check; Ruff format --check; compileall; git diff --check
Not-tested: Full backend suite remains affected by unrelated Windows permissions/path and missing Lark CLI tests
Related: #4984; #4912
* Preserve case-sensitive exact tool authorities
Case-folding a scalar declaration before alias lookup can turn literal write into write_file, substituting a different runtime authority. Keep exact portable spellings as aliases and preserve lowercase, custom, and MCP names; strengthen activation coverage for spaced Bash patterns and command fragments.
Constraint: Runtime skill policy uses exact tool-name matching
Constraint: Bash(...) remains literal and inactive because command-pattern authorization is not implemented
Rejected: Case-insensitive alias lookup | maps lowercase runtime tools onto built-in authorities
Rejected: Broaden the parser into command-pattern authorization | outside this PR's scope
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Add aliases only for documented portable spellings; preserve all other scalar names verbatim
Tested: 226 skills tests passed, 1 skipped; Ruff check; Ruff format --check; compileall; git diff --check
Not-tested: Full backend suite remains affected by unrelated Windows permissions/path and missing Lark CLI tests; GitNexus index refresh remains stale
Related: #4984; #5016297602
* Support portable Glob and Grep skill aliases
Portable Agent Skills commonly declare Glob and Grep, but DeerFlow exposes the runtime tools as glob and grep. Add explicit exact-spelling aliases and activation coverage so imported skills retain search-tool access without broad normalization.
Constraint: Runtime skill policy uses exact tool-name matching
Constraint: Alias conversion is limited to documented portable spellings
Rejected: Case-fold all scalar names | can substitute custom or MCP authorities
Rejected: Map arbitrary names by convention | breaks exact runtime compatibility
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep the alias table explicit and preserve unknown scalar names verbatim
Tested: 228 skills tests passed, 1 skipped; Ruff check; Ruff format --check; compileall; git diff --check
Not-tested: Full backend suite has unrelated environment failures on Windows; GitNexus index reports stale line mappings
Related: #4984; #5026257899
---------
Co-authored-by: kriptoburak <kriptoburak@users.noreply.github.com>
The English Security Notice has four subsections; the Chinese one had two.
Translate the two that were missing:
- Gateway Admin Is Equivalent to Code Execution
- Deployment Defaults
Both describe deployment-time security behavior, so a stale translation
leaves Chinese-speaking operators without the loopback-default and
first-run-setup guidance that English readers get.
Verified against the code rather than translated blind:
- stdio MCP allowlist defaults to {npx, uvx} and is extended via
DEER_FLOW_MCP_STDIO_COMMAND_ALLOWLIST
(backend/app/gateway/routers/mcp.py)
- entry port publishes as ${BIND_HOST:-127.0.0.1}:${PORT:-2026}
(docker/docker-compose.yaml)
- /setup exists as the first-run admin creation route
(frontend/src/app/(auth)/setup/page.tsx)
Co-authored-by: zaoshangduziteng <309590849+zaoshangduziteng@users.noreply.github.com>
* fix(sandbox): harden local Docker sandbox containers and port binding
Root causes (security audit SBX-1/SBX-2) in the local container backend:
- _resolve_docker_bind_host published sandbox ports on 0.0.0.0 whenever
DEER_FLOW_SANDBOX_HOST was non-loopback (docker-compose defaults to
host.docker.internal), exposing the unauthenticated /v1/shell/* exec
API on every host interface.
- _start_container ran every sandbox with seccomp=unconfined and no
capability, privilege-escalation, or resource limits, so untrusted
model-authored code could exhaust the host, escalate privileges, and
reach internal networks / cloud metadata endpoints directly.
Hardening changes and defaults:
- Port binding: non-loopback sandbox hosts now bind the Docker default
bridge gateway instead of 0.0.0.0, discovered dynamically via
`docker network inspect bridge` with a static 172.17.0.1 fallback.
host.docker.internal resolves to that gateway through host-gateway,
so DooD gateways and the Docker host still reach the sandbox while
external interfaces no longer see the port.
DEER_FLOW_SANDBOX_BIND_HOST=0.0.0.0 restores the legacy broad bind.
- seccomp=unconfined is no longer unconditional: sandboxes run with
Docker's default seccomp profile; opt back in with
DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=1, only when the sandbox image
is verified to require syscalls the default profile blocks.
- Add --cap-drop=ALL and --security-opt no-new-privileges (Docker only;
the Apple Container CLI does not support these flags).
- Bounded resources with env overrides: --memory 2g
(DEER_FLOW_SANDBOX_MEMORY), --cpus 2 (DEER_FLOW_SANDBOX_CPUS),
--pids-limit 512 (DEER_FLOW_SANDBOX_PIDS_LIMIT); each also accepts
"0"/"none" to disable the limit.
- No --user is forced by default (the default AIO sandbox image's user
is upstream-controlled and unverified), but
DEER_FLOW_SANDBOX_CONTAINER_USER passes one through for deployments
that know their image.
- DEER_FLOW_SANDBOX_NETWORK passes --network so sandboxes can be
attached to a dedicated egress-controlled network; default networking
is unchanged.
backend/docs/CONFIGURATION.md documents the new bind behavior and every
override; tests cover each default and escape hatch.
* fix(sandbox): follow host-gateway mapping for binds; keep image-required seccomp default
Review follow-ups on the hardening change:
- Bind: resolve the sandbox host itself and bind that address, instead of
assuming the default bridge IPv4. host.docker.internal follows the
daemon host-gateway-ip mapping (customizable, possibly IPv6), so the
resolved address is exactly where the gateway connects — the published
port and advertised URL always match. IPv6 is bracketed for docker -p,
zone ids stripped, wildcard resolutions ignored; unresolved hosts fall
back to the bridge gateway with a warning pointing at
DEER_FLOW_SANDBOX_BIND_HOST.
- seccomp: the shipped AIO image needs seccomp=unconfined for its
Chromium browser (upstream quick-start always passes it; the upstream
FAQ documents the browser failing under Docker default profile), so
that option returns as the default. Tightening stays possible via
DEER_FLOW_SANDBOX_SECCOMP_PROFILE=<path to a restricted,
Chromium-compatible profile> or DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=0
for images verified to work with Docker's default profile.
- cap-drop/no-new-privileges and the resource limits are unchanged.
- Tests updated for both behaviors; 37 pass.
* fix(sandbox): bracket bare IPv6 bind overrides; state seccomp default accurately
DEER_FLOW_SANDBOX_BIND_HOST was returned verbatim, so a bare IPv6 literal
like fd00::1 produced an invalid publish spec (fd00::1:port:8080); Docker
requires the bracketed form. Normalize raw and already-bracketed IPv6
literals (IPv4/hostnames untouched), with resolver-level and argv-level
tests covering the explicit IPv6 override.
The CONFIGURATION.md overview claimed Docker's default seccomp profile
stays active, contradicting the seccomp=unconfined default the table (and
the code) actually ship for the Chromium-based image; spell out the relaxed
default and where to change it.
* style(sandbox): apply ruff format to local_backend
* fix(sandbox): reject host networking, force builtin seccomp opt-out, resolve hostname binds
Review follow-up on #4986 (willem-bd):
- P1: DEER_FLOW_SANDBOX_NETWORK=host (and container:<name>) now raise a
RuntimeError at start instead of silently voiding the hardened port
bind — Docker discards -p/--publish in host mode and shares the
network namespace for container:<name>, which would re-expose the
unauthenticated exec API on the host's interfaces. Two regression
tests cover both rejections.
- P2: the seccomp opt-out now passes seccomp=builtin explicitly instead
of omitting the option, so a daemon configured with an unconfined or
custom default cannot weaken the documented opt-out; the test asserts
the flag.
- P2: hostname values in DEER_FLOW_SANDBOX_BIND_HOST resolve to an
address before use (Docker publish specs require an IP literal as the
host part, so host.docker.internal previously produced an invalid
spec that prevented every sandbox from starting); unresolvable names
raise a clear configuration error. Tests cover resolution and
rejection; CONFIGURATION.md updated for all three behaviors.
43/43 pass in tests/test_aio_sandbox_local_backend.py; ruff check +
format clean.
* fix(sandbox): reject DEER_FLOW_SANDBOX_NETWORK=none (loopback-only, breaks published API port)
* fix(sandbox): validate the effective Docker network target; normalize IPv6 sandbox hosts once
name=host / name=none dodge raw-string checks but attach like the bare
words; strip name= prefixes and validate the effective target (network IDs
keep passing). Bracketed IPv6 sandbox hosts now resolve for the bind and
bare IPv6 hosts produce bracketed URL authorities — both input forms give
identical bind and URL addresses.
* fix(sandbox): parse the full Docker network long syntax before validating
Docker accepts comma-separated key=value fields in any order (name=, gw-priority=,
alias=, ...); a name=host field hides the host network behind surrounding fields.
Parse the CSV and validate the parsed name= target (last occurrence wins, fields
lowercased, mirroring opts/network.go); no-name values fall through like Docker's
own rejection.
* fix(sandbox): keep CHOWN/SETUID/SETGID through cap-drop=ALL for the default image
The shipped image's entrypoint starts as root, creates the gem user,
chowns /opt/jupyter and drops to that user via su; without those three
capabilities the set -e script dies before the readiness endpoint exists.
no-new-privileges stays (it blocks gaining privileges via exec, not using
the added caps). Adds a docker-gated real-image startup smoke test.
* fix(sandbox): let pre-initialized non-root images drop the startup capabilities
The CHOWN/SETUID/SETGID re-add only exists for the shipped image's root
entrypoint handoff. A custom image that never runs as root gets an explicit
opt-out (DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0) so those capabilities are
not left available to sandboxed code (chown on bind mounts, UID/GID
impersonation).
* test(sandbox): gate the real-image smoke test behind the live marker
The default offline suite (make test = -m 'not live') must not depend on a
third-party registry: mark the smoke test live, probe the daemon inside the
test body (never at collection time), and allow pinning the image reference
via DEER_FLOW_SANDBOX_SMOKE_IMAGE for a dedicated integration job.
* test/docs: isolate DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS in tests; add table row; split custom-image guidance
_clear_hardening_env now clears the new knob so a developer shell or .env
preset cannot flip the default-path tests. CONFIGURATION.md gains the table
row, and the custom-image guidance becomes its own paragraph with the
no-new-privileges scope stated correctly (it does not mitigate the retained
CAP_SETUID/SETGID risk).
* test(sandbox): make the live smoke test diagnosable
300s readiness budget (cold pull + cold start must not be conflated with
broken capabilities) and dump the container's last 40 log lines on failure
so the next live run tells us whether the capability set is incomplete
(chown/useradd/su errors) or the services are merely slow.
* test(ci): align the smoke test with the 60s provider deadline; add a dedicated live smoke workflow
Single-source the readiness deadline as SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT
(used by both provider paths and the smoke test) so the validation cannot
drift from the production contract again. New sandbox-image-smoke.yml runs
the live test on a dedicated job, with the image reference pinnable via the
SANDBOX_SMOKE_IMAGE repository variable (digest resolved and recorded in the
job summary when falling back to :latest).
* test(sandbox): pull the failing program's own logs on smoke failure
supervisord only surfaces exit codes in docker logs; nginx's stderr lands in
files inside the container. Dump supervisor program logs, nginx -t, and the
nginx error log on failure so the next run names the exact broken line.
* ci(sandbox): export an immutable repo@digest reference for the smoke run
docker pull once on the runner platform, resolve RepoDigests[0], and pass
that immutable reference to the test via GITHUB_ENV — the recorded and
executed images can no longer diverge when the tag moves, and platform
selection is left to the daemon instead of jq over the manifest index.
* fix(sandbox): add DAC_OVERRIDE — the root nginx master writes gem-owned logs
The image's root nginx master opens /var/log/nginx/{access,error}.log,
which belong to the gem user, for the container's lifetime; without
CAP_DAC_OVERRIDE it dies with 'open() failed (13: Permission denied)' on
every start (FATAL under supervisord) and readiness never arrives. Four
capabilities now: CHOWN/SETUID/SETGID for the entrypoint handoff plus this
runtime log-write need.
* fix(channels): retain strong reference to Discord ack-reaction tasks
The event loop keeps only weak references to scheduled tasks, so the bare
asyncio.create_task(self._add_reaction(message)) at both message-acceptance
sites could be garbage-collected mid-flight and silently drop the ✅
acknowledgment. The module-level retention set holds each task until
completion and logs failures, mirroring the deferred subagent cleanup fix
in #4928 and the stream bridge cleanup fix in #4931.
* fix(channels): drain in-flight ack reactions on Discord channel shutdown
Adversarial follow-up to the retention fix: stop() cancelled typing tasks
but left in-flight ack-reaction tasks in the module retention set when the
channel stopped mid-HTTP-call, pinning the channel instance and the discord
Message object graph across restart cycles. _cancel_ephemeral_tasks /
_discard_ephemeral_tasks now cover both families at every stop() cleanup
point and the _run_client finally. Also tests the done-callback error-log
branch and documents the GC test as a contract pin.
* test(channels): pin Discord ack-reaction drain wiring and sharpen wording
Second adversarial pass found the drain test cancelled a never-started
task (no yield between schedule and cancel), nothing exercised the real
stop() cross-loop wiring, and the cleanup log messages still said
"typing-task" while now covering both families. The drain test now yields
so the task genuinely suspends first, a new test drives stop() through a
real background loop and fails if the wiring reverts to typing-only
cleanup, wording covers ephemeral tasks, and the discard path snapshots
the set before iterating.
* refactor(channels): scope Discord ack-reaction retention to the channel instance
The retention set was module-level, so one DiscordChannel's stop() would
cancel every other instance's in-flight ack reactions. Instance-level
matches the typing-task bookkeeping and keeps shutdown scoped; pinned by
a two-channel isolation test.
* test(channels): drive two-channel ack isolation through full stop()
Independent review of the instance-scoping refactor confirmed the P2 fix
but suggested matching the review's letter exactly: the isolation test now
shuts the first channel down via stop() instead of the drain helper, and
one stale docstring reference to the former module-level set is corrected.
* Stabilize Windows frontend development startup
Next.js Turbopack can infer a user-level workspace root and hit an internal Rust task panic on Windows. Keep the frontend root explicit and select Webpack on Windows while preserving Turbopack elsewhere.
Constraint: The existing non-Windows development workflow should keep Turbopack.
Rejected: Disable Turbopack on every platform | unnecessary performance regression.
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Revisit the Windows Webpack fallback after the upstream Turbopack panic is resolved.
Tested: Prettier, ESLint, TypeScript, 139 frontend test files with 1060 tests, and Windows pnpm dev smoke test on port 3010.
Not-tested: Production build and browser E2E suite.
Related: #4957
* Make the frontend bundler fallback diagnosable
Keep the Windows Webpack fallback while allowing local diagnosis to select either supported Next.js bundler. Resolve the Turbopack root through the Node 20-compatible URL API so the explicit workspace boundary remains effective across the declared Next.js engine range.
Constraint: Windows defaults to Webpack because of the observed Turbopack panic, while non-Windows defaults to Turbopack.
Rejected: Accept arbitrary bundler values | Next.js only supports the two explicit flags and invalid values should fail early.
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Revisit the default Windows fallback when the upstream TaskGuard issue is resolved; the override is intentionally retained for diagnosis.
Tested: Frontend format, ESLint, TypeScript, 139-file/1062-test Rstest suite, target launcher tests, and next.config.js absolute-root load check.
Not-tested: Production build and browser E2E suite.
* feat(mcp): map request-scoped secrets to HTTP/SSE headers
`user_auth` binds a credential to a configured DeerFlow user, so a caller
that picks the credential per request — a multi-tenant gateway, a per-run
API key, one shared MCP server fronting several environments — had to
register one MCP server entry per credential.
Add a declarative `mcpServers.<server>.headers_from_context` block mapping
HTTP header names to keys of the run request's `config.context.secrets`
carrier. A new built-in interceptor resolves the mapping on every tool call
and rewrites those headers, mirroring `user_scoped_auth`. The config file
stores names only, never a credential, so the Gateway returns the block
unmasked.
Registered after OAuth and `user_auth` in the interceptor chain: the later
interceptor runs closer to the transport, and the value chosen for this one
request is the most specific, so it wins. Fail-closed by default — a mapped
key missing from the request raises a `ToolException` naming only that key,
because falling back to the server's discovery credential would send one
tenant's call under another tenant's authority. `on_missing: "passthrough"`
opts out.
Durable background tasks are excluded: `McpTaskToolCaller` drives status and
cancel polls after the Agent run ends, where no run context exists, so the
fail-closed interceptor would deny every poll. Those calls keep using
server-level credentials, and a server declaring both `headers_from_context`
and `task_toolsets` now logs a warning.
Also corrects the custom-interceptor example in docs/MCP_SERVER.md (and the
matching claim in skills/AGENTS.md), which read request secrets from
`langgraph.config.get_config()["context"]`. That key is `None` inside a tool
call — the run context rides the LangGraph runtime, not the RunnableConfig
propagated to child runnables — so interceptors written from that example
never saw a value. The example now reads `request.runtime`, and
tests/test_mcp_context_headers.py pins LangGraph's runtime-injection rule by
driving a real langchain-mcp-adapters tool through a real graph with the
ambient-runtime fallback disabled.
Closes#5005
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): resolve credential headers case-insensitively, carry them on durable submit
Review follow-ups on `headers_from_context`.
HTTP field names are case-insensitive, but every dict on the path to the wire
is not: `build_server_params` copies the operator's static `headers` spelling
verbatim, and langchain-mcp-adapters merges interceptor overrides into the
connection with a plain `{**connection_headers, **override_headers}` splat. A
static `authorization` and an injected `Authorization` therefore both reached
httpx as separate field lines, and a server reading the field with a
single-value accessor got the static discovery credential — inverting the
documented `headers` < `oauth` < `user_auth` < `headers_from_context`
precedence and running a per-request call under the shared credential.
Normalizing inside the interceptor cannot fix that on its own: the adapter
builds the request with `headers=None`, so an interceptor never sees the
connection's static headers and cannot displace them however it spells its own
key. A new `mcp/headers.py::apply_header_overrides` therefore drops any key
differing only in case and emits the spelling the connection already uses.
Applied to `headers_from_context`, `user_auth`, the OAuth interceptor, the
OAuth discovery-header write, and the durable-task connection merge, which all
carried the same collision. `headers_from_context.headers` now also rejects one
header mapped under two spellings at config load, in both the harness model and
the Gateway mirror.
Durable submit now carries the mapped headers, as docs/MCP_SERVER.md already
promised. `McpTaskToolCaller` disabled the interceptor for the whole caller, but
that caller serves submit as well as the polls, and submit is awaited inline
inside the Agent's tool call — where the run's LangGraph runtime is still the
ambient contextvar, so no secret has to be threaded through `TaskSubmitRequest`
or reach durable storage. The caller builds one chain and keeps a second view of
it without the context-headers interceptor; `call_tool` takes
`request_scoped_headers`, set only by `OrdinaryMcpTaskDriver.submit`. Status and
cancel keep server-level credentials, so background polls still cannot fail
closed, and the startup warning now describes the half it actually covers.
`_merge_preserving_secrets` restores masked extras inside `headers_from_context`
instead of writing the `***` sentinel back over the stored value, matching the
treatment `user_auth` extras and server-level extras already get; extras a PUT
omits carry over as well, while the declared mapping still replaces verbatim so
a round trip can remove an entry. `extra="allow"` plus name-based sensitivity
detection means the usual casualty is a name-valued key such as `tokenHeader`,
not only a credential.
The existing override test seeded the static header onto `request.headers`,
which production never does, so it modelled a merge that really happens one
layer down; the new tests drive a real adapter tool through a real connection
and assert on the headers the session is opened with, and the durable-submit
test runs through a real tool node with no runtime patching.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): reject case-insensitive duplicate static header names
* fix(mcp): preserve omitted headers_from_context fields on partial updates
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ViewImageMiddleware injected the viewed-image message from before_model and
removed it again from after_model. before_model, model, and after_model are
separate graph nodes, so every view_image turn cost two extra nodes and two
state writes, and up to 20MB of base64 sat in two checkpoints for the duration
of the model call. A run interrupted in that window (user cancel, restart)
stranded the payload in history for good.
Inject from wrap_model_call instead, so the message lives only in
ModelRequest.messages and is never returned as a state update:
- before_model/after_model (and the async pair) are replaced by
wrap_model_call/awrap_model_call; _remove_image_context_messages and its
RemoveMessage bookkeeping go with them. The async hook keeps the existing
asyncio.to_thread offload for the file read and base64 encode.
- _should_inject_image_message gates on request.messages rather than state, so
the decision is made against what the model will actually see.
- _inject sweeps this middleware's own message out of the request before
rebuilding it. Dropping after_model also drops the cleanup it did on every
call, so without the sweep a payload stranded by an older interrupted run
would ride along in every later request for the life of the thread. Matching
requires both the reserved id prefix and the server-owned marker, and Gateway
strips that marker from client input, so a user message is never dropped.
Chain position is unchanged, and wrap_model_call nests first-registered
outermost, so TokenBudgetMiddleware still sees the image message and enforces
the input budget against it.
Checkpoint rows that already hold a stranded payload keep it on disk. It is
inert -- never sent to a provider, and strip_data_url_image_blocks keeps it off
the wire -- and reclaiming it would mean keeping the node this change removes.
tests/test_view_image_middleware.py is rewritten around the new hook (43
tests): sync/async at unit and graph level, the stranded sweep, and the
client-message protection. Docs: middleware chain entry 23, Vision Support, the
middleware-execution-flow hook matrix and diagrams, and the
strip_data_url_image_blocks docstring.
* Preserve Windows CLI compatibility for local sandbox commands
MSYS path conversion must remain disabled for DeerFlow virtual paths, but applying a blanket environment override to every POSIX command breaks host-native CLI shims on Windows. Limit MSYS argument-conversion exclusions to safe non-root virtual path prefixes, omit values that would broaden the exclusion pattern, and document the contract.
Constraint: Preserve the virtual-path protection introduced by #2765/#2766
Rejected: Disable MSYS conversion for every command | breaks Windows CLI shims
Rejected: Toggle blanket conversion only for commands containing virtual paths | host CLIs can receive virtual-path arguments and still need normal conversion for their own paths
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: Keep regression coverage for virtual-path arguments, root mounts, and host-native CLI launchers
Tested: test_local_sandbox_encoding.py (12 passed); related sandbox suite (197 passed, 8 skipped, 7 failures matching origin/main); ruff check; ruff format --check; git diff --check; direct LocalSandbox CLI and virtual-path smoke tests
Not-tested: Full offline suite completion; stopped at 6% after unrelated Windows and optional-runtime failures
Related: #2765
Related: #2766
* Keep MSYS regression tests portable across CI operating systems
The Windows-shell environment tests patched os.name to nt while mounting Windows-specific paths. On Linux and macOS, pathlib then attempted to construct WindowsPath during command resolution or output masking, so the backend merge gate failed before exercising the environment contract. Stub the exclusion boundary in execute-command tests and retain mapping-specific filtering coverage in the helper test.
Constraint: Backend unit tests run on Linux, while the behavior under test is Windows-only
Rejected: Skip the tests outside Windows | would remove CI coverage of the environment contract
Rejected: Patch pathlib internals | couples tests to implementation details and hides the platform boundary
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep OS-specific subprocess assertions independent from host-path resolution
Tested: test_local_sandbox_encoding.py (12 passed); ruff check; ruff format --check; git diff --check
Not-tested: Linux runner execution locally because Docker Desktop is unavailable and WSL cannot access this linked worktree
Related: #5003
Related: https://github.com/bytedance/deer-flow/pullrequestreview-5013380238
* fix(gateway): preserve exact history run attribution
* fix(gateway): make history migration authoritative
* docs(runtime): keep history contract within guidance budget
* fix(runtime): fence final run duration write
* fix(frontend): restore sanitization in custom streamdown rehype chains
Streamdown 2.5 replaces its entire default rehype chain
[rehype-raw, rehype-sanitize, rehype-harden] with whatever array the
caller passes via the rehypePlugins prop. Every custom chain in this
repo therefore rendered LLM/stored markdown without any sanitization:
- Artifact markdown previews (markdown-preview-plugins.ts +
artifact-file-detail.tsx) parse raw HTML via rehypeRaw, so a
generated .md artifact could inject <style>/<iframe>/on* handlers
into the workspace DOM (stored XSS; only javascript: anchors were
blocked by the ArtifactLink component).
- The memory settings summary (memory-settings-page.tsx) spread the
shared preset without component overrides, so a hostile
<a href="javascript:..."> in stored memory content rendered as a
clickable anchor.
Fix strategy:
- Add rehype-sanitize (already resolved in the lockfile via streamdown)
as a direct dependency and re-insert a [rehypeSanitize, schema] step
in the shared preset (core/streamdown/plugins.ts). It runs after
rehypeRaw (raw HTML must be parsed into hast before it can be
cleaned) and before rehypeKatex/rehypeSlug (their output is trusted
and would otherwise be filtered or clobbered) - the same
raw -> sanitize -> math ordering streamdown itself uses.
- The schema extends rehype-sanitize's GitHub-style defaultSchema (the
base of streamdown's own sanitize schema) so legitimate authored
artifact HTML (tables, details, images, alignment/size attributes)
keeps working while script/iframe/style, on* handlers and
non-allow-listed URL schemes (javascript:, data:, ...) are dropped.
The only extensions are tel: hrefs and the math-inline/math-display
class markers remark-math emits and rehype-katex detects.
- Position rehypeSlug after the sanitize step in the artifact chain so
sanitize's id clobbering (id="x" -> id="user-content-x") cannot break
the heading anchors it creates.
- Pass a: createMarkdownLinkComponent() on the memory settings page as
defense in depth, matching the chat rendering path.
Unit tests feed a hostile payload (<a href="javascript:...">,
<img onerror>, <script>, <iframe>, <style>, ontoggle) through both
render paths and assert no executable/clickable equivalent survives,
plus regression guards for heading anchors, legitimate HTML and KaTeX
math rendering.
* fix(frontend): keep the sanitize clobber prefix on heading anchors; minimal lockfile
Review follow-ups on the sanitization change:
- Heading anchors: rehypeScopedSlug replaces rehype-slug in the artifact
chain. It runs after the sanitize step (so raw-HTML headings are also
anchored) but keeps rehype-sanitize's user-content- id prefix — an
untrusted heading like "## current" cannot mint an unprefixed
id="current" (the DOM-clobbering shape the sanitizer guards against).
In-page fragment links are translated to the prefixed anchors so they
still resolve; external URLs, bare "#", already-prefixed fragments and
sanitize-prefixed raw-HTML ids are left untouched.
- Lockfile: regenerated as a minimal diff — only the two direct-dependency
importer entries (rehype-sanitize, github-slugger for the scoped slug)
are added; the libc platform selectors on the 64 native package records
are preserved byte-for-byte instead of being dropped by lockfile
normalization.
Full frontend suite: 1034 tests passing; tsc and prettier clean.
* style: reorder github-slugger import ahead of the hast type import
* test(e2e): expect the clobber-prefixed heading anchor in artifact preview
The scoped slug plugin gives generated heading ids rehype-sanitize's
user-content- prefix and translates fragment links to match, so the
anchor-scroll test must locate the prefixed id.
* fix(frontend): reset the scoped slugger per tree; keep footnote anchors single-prefixed
Review follow-ups:
- The scoped slug attacher holds one GithubSlugger, but streamdown
caches the unified processor by plugin name, so the instance survived
across parses and repeated renders of the same heading grew -1/-2
suffixes (the artifact-anchor e2e could not find the id on re-render).
The transformer now resets the slugger per tree, as rehype-slug does;
a regression test renders identical artifact markdown twice.
- remark-rehype emits GFM footnote anchors already clobber-prefixed
(user-content-fn-1); the sanitize step prefixed those ids again while
their hrefs stayed single-prefixed, breaking footnote navigation in
every chain built on the shared preset. A new rehypeClobberFragments
step runs right after sanitize: double-prefixed ids are normalized
back to one prefix, and unprefixed fragment hrefs are translated to
the prefixed form (already-prefixed and external links untouched).
The artifact slug now inserts after this step; covered by a footnote
regression test on the shared render path.
Unit suite 1036 passing; artifact-preview e2e verified locally
(9/9, including the heading-anchor scroll test).
* fix(docker): create compose env files and keep Windows compose paths relative
Windows Docker reports a generic file-not-found when env_file targets are missing, or when compose paths are doubled. Make docker-start copy .env examples and invoke compose with filenames relative to docker/.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): make dev compose env files optional and repair test fixture
Address review feedback on #4956.
[P1] prepare_compose_env aborted before the mocked COMPOSE_CMD in
test_compose_commands_set_deer_flow_root_before_compose, because the
temp root had no compose file or .env examples. Seed them in the
fixture so the preflight reaches the mock.
[P2] .env is gitignored, so a fresh clone has none and a direct
`docker compose -f docker/docker-compose-dev.yaml up --build` aborts on
Windows before scripts/docker.sh can help. Mark the dev env_file entries
`required: false` so a missing .env is not fatal, and document that
direct Compose must be run from the repository root.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): declare Compose 2.24 floor and keep non-start commands read-only
Address the second review round on #4956.
- Document Compose >= 2.24 (CONTRIBUTING, README, compose header) and fail
early from make docker-start with an actionable message; probe both
`docker compose` and the hyphenated `docker-compose` binary.
- Document DEER_FLOW_ROOT for direct Compose callers (bash + PowerShell);
leave the variable without a $PWD fallback because PowerShell/cmd do not
export it.
- Split prepare_compose_env: compose_preflight is shared and read-only;
ensure_env_files runs only from start.
- Expand tests for version boundaries, hyphenated fallback, env-file
creation, and read-only stop/logs/restart behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): reuse the probed Compose binary for wrapper operations
The version probe could accept a standalone docker-compose install while
COMPOSE_CMD stayed hardcoded to `docker compose`, so preflight passed and
start/logs/stop/restart then failed. Keep the selected executable in
COMPOSE_BIN (array), refresh COMPOSE_CMD from it in the current shell, and
extend the fallback test through an actual stop invocation.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix skill moderation parsing for Responses API content blocks
Normalize LangChain Responses API text blocks before parsing the security moderation decision, while preserving the existing fail-closed behavior for unavailable or invalid moderation results. Add regression coverage for mixed content blocks and document the compatibility boundary.
Constraint: Responses API AIMessage content is list-shaped while Chat Completions content is string-shaped
Rejected: Disable security scanning | would weaken the skill write safety boundary
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep moderation parsing provider-format tolerant without including reasoning or tool blocks in the decision payload
Tested: 27 security scanner tests; ruff check; ruff format check
Not-tested: Live moderation request against the configured external endpoint
* Reuse shared LLM response text normalization
Route skill moderation responses through the existing provider-format normalizer so only text and output_text blocks participate in JSON parsing. Strengthen regression coverage with reasoning and tool blocks that contain misleading text fields.\n\nConstraint: Responses API content is shared across multiple harness consumers\nRejected: Keep a private normalizer | duplicated provider-shape policy diverges and can reintroduce reasoning-block contamination\nConfidence: high\nScope-risk: narrow\nReversibility: clean\nDirective: Extend the shared normalizer when a new provider content shape is verified; do not add divergent local parsers\nTested: 118 related backend tests; regression test red against the previous parser; Ruff check and format check\nNot-tested: Live GitHub CLA status refresh
* Restore trusted external skill package loading
Skill discovery follows one-level package-directory symlinks, but activation path validation rejected the resolved external path. Restore that compatibility for configured custom-skill category roots while keeping file-level symlinks and deeper escapes blocked. Add regression coverage for local and user-scoped storage plus slash activation, and document the boundary.
Constraint: Existing skill discovery follows directory symlinks and operator-managed external packages must remain loadable
Rejected: Allow arbitrary resolved paths | would weaken the skill path trust boundary
Confidence: high
Scope-risk: moderate
Directive: Keep the final SKILL.md file symlink-free and preserve one-level category-root validation
Tested: 79 targeted skill storage, loader, slash activation, and user-scoped tests passed; Ruff check and format check passed; GitNexus staged change detection reported low risk
Not-tested: Real symlink activation on this Windows host lacks SeCreateSymbolicLinkPrivilege and is skipped
Related: Skill projection copies sources into sandbox-visible views
* Exercise real filesystem symlink boundaries in skill storage tests
Replace global Path.resolve/is_symlink mocks with real directory and file symlinks, preserving the Windows privilege skip. Add regression coverage for deeper custom-root escapes and symlinks under non-custom categories so the one-level allowance remains explicit.
Constraint: Symlink creation requires SeCreateSymbolicLinkPrivilege on some Windows runners
Rejected: Keep global path-method mocks | they validate the mock behavior rather than filesystem semantics
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep security-boundary tests on real filesystem primitives; skip only when the runner lacks symlink privilege
Tested: 76 targeted loader/storage/slash tests; Ruff check; Ruff format check
Not-tested: Windows symlink-enabled execution on this host
Related: #4936
* Pin the actual nested symlink escape boundary
Place the second symlink below a real custom package directory so the test reaches the one-level relative-parent guard instead of returning early on a non-symlink parent. Keep the public-category rejection coverage unchanged.
Constraint: The security boundary depends on both symlink depth and category root
Rejected: Link the outer package directory directly | the parent is not a symlink at validation time, so the depth guard is never evaluated
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep this regression tied to the exact relative_parent.parts depth check
Tested: Targeted storage, loader, and slash suites; GitNexus staged detection
Not-tested: Symlink-enabled execution on this Windows host
Related: #4936
* Make the nested symlink regression reach the depth guard
The test now validates the SKILL.md directly through the nested symlink, so the symlink is the immediate parent and the relative-parent depth check is executed.
Constraint: Windows test execution may skip when symlink privilege is unavailable
Rejected: Keep the extra nested path segment | it bypasses the symlink-depth guard through an early return
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Mutation tests must fail when the depth restriction is removed
Tested: Targeted test (skipped on this Windows host without symlink privilege); Ruff check and format check
Not-tested: Real symlink execution on Windows; Linux CI will exercise the case
Related: #4936
* Keep sandbox projections fresh for linked external skill packages
The storage layer intentionally accepts one-level custom package-directory symlinks, but projection freshness previously hashed only the link inode. Follow the permitted target tree during custom and legacy source-signature scans so edits to SKILL.md, scripts, references, or assets trigger a rebuild before sandbox use.
Constraint: Preserve the existing one-level custom/legacy symlink boundary and do not follow public, integration, nested, or file symlinks
Rejected: Invalidate projections only from /api/skills/reload | sandbox acquisition must also detect edits made directly in external targets
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep target-tree traversal limited to the storage paths that explicitly permit external package-directory links
Tested: 77 projection, user-scoped storage, and lifecycle tests passed; Ruff check and format check passed; git diff --check passed; GitNexus staged detection reported low risk
Not-tested: Real external symlink execution on this Windows host without SeCreateSymbolicLinkPrivilege; existing tests skip that platform limitation
Sandbox is an execution environment, not a named resource: multiple tools
(bash, read_file, write_file, glob, grep, ...) depend on it, all funneled
through ensure_sandbox_initialized / ensure_sandbox_initialized_async. Gate
the single acquisition entry point (single source of truth) instead of
maintaining a sandbox-tool-name set in middleware:
- authorize_sandbox_execution helper (authz/sandbox_authz.py) checks
authorize("sandbox", "execute", target="*") — a binary judgment
(can this role use the sandbox at all); RBAC allow:"*"/true permits,
allow:[]/false denies.
- lazy path: ensure_sandbox_initialized (+ async) calls the gate before
provider.acquire.
- eager path: SandboxMiddleware.before_agent / abefore_agent call the gate
before _acquire_sandbox.
- deny raises SandboxAuthorizationError (SandboxError subclass) which
propagates through tool execution as a friendly ToolMessage (RFC §9:
'not a crash').
- authorization.enabled: false is a no-op everywhere; provider errors
follow fail_closed (deny) / fail_open (allow).
12 tests in tests/test_sandbox_authorization.py cover disabled/allow/deny/
deny-via-bool/no-policy-unrestricted/provider-error-fail-closed/open/
internal-caller + ensure_sandbox_initialized deny (never acquires) and
allow (acquires) integration paths.
* fix: DeerFlow brand casing in disclaimer, 'What's New' title, setup-sandbox .PHONY
- en-US disclaimer said 'Deerflow'; every other user-facing string
brands it 'DeerFlow' (rendered under the chat composer)
- landing section title 'Whats New in DeerFlow 2.0' -> 'What's New'
- setup-sandbox was the only Makefile target missing from .PHONY
* test: align disclaimer assertions with DeerFlow brand casing
The unit and e2e suites still expected the old "Deerflow" spelling in
inputBox.disclaimer; update both to the corrected locale string so the
i18n load test and the chat footer assertion pass again.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): serve SHA-256 via ETag so preview/edit work on non-secure contexts
crypto.subtle is only available in secure contexts (HTTPS or localhost). The frontend fell back to it to compute an artifact's SHA-256 when the Gateway did not return one, which threw on http://<lan-ip>:<port> and broke both artifact preview and inline editing (issue #4864).
- Gateway now returns the real SHA-256 as an ETag header for inline text and active-content artifact responses (and skill-archive members).
- Frontend prefers the ETag and only computes a hash as a last resort, falling back gracefully (FNV-1a) instead of throwing when crypto.subtle is missing.
* fix(artifacts): address PR review feedback for #4864
- Cache SHA-256 digests by (path, mtime_ns, size) so the many small Range
requests a browser issues while scrubbing/paginating a preview do not each
re-hash a potentially huge artifact from scratch (performance).
- Gate inline editing on a real 64-hex revision: hasRevision requires
sha256.length === 64, so the FNV-1a fallback on non-secure origins keeps
preview working but no longer 422s on save (contract).
- Anchor and lowercase the ETag regex and accept the weak W/ prefix gzip
emits, so uppercase hex and longer digests (sha-384/512) can't masquerade
as sha-256.
- Cover the forced-download ETag on the backend and add frontend tests for
weak-ETag parsing and the non-secure-context FNV fallback.
Feedback from reviewer willem-bd on PR #4865.
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: reformat artifact-file-detail.tsx for prettier with tailwind class ordering
* fix: invalidate SHA-256 cache after artifact edit
Clear the LRU cache after os.replace() so the next preview request
computes the new digest. Edits are rare, so clearing the whole
256-entry cache costs nothing (addressing PR review comment #5).
* fix(artifacts): skip ETag for oversized files + CRLF->LF + cache invalidation (#4865)
* fix(loader): use real empty-content SHA-256 for empty 416 range (#4865)
* test(artifacts): assert oversized artifacts carry no SHA-256 ETag (#4865)
* style(frontend): format long sha256 constant (prettier)
* test(backend): fix oversized-artifact ETag assertions and formatting (ruff)
* test(backend): keep oversized-payload line within ruff 240-col config
* fix(clarification): drop sibling tool calls before interrupt
- Rewrite the AIMessage in ClarificationMiddleware.after_model so a
parallel bash/write_file cannot run before the user answers
- langchain return_direct only inspects the last ToolMessage; siblings
both execute and can keep the agent loop alive
- Skip the rewrite when disable_clarification is set
- Prompt and tool docs: do not call other tools in the same turn
Fixes#4906
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(clarification): enhance sibling tool call handling in ClarificationMiddleware
- Update ClarificationMiddleware to ensure sibling tool calls are dropped when `ask_clarification` is invoked, preventing unintended execution before user input.
- Modify documentation to clarify that the `return_direct` router now inspects all client-side tool calls of the last AIMessage, ensuring proper routing behavior.
- Introduce a new integration test to validate that sibling tools do not execute when `ask_clarification` is present in the same turn.
This change addresses potential issues with tool execution order and improves the overall reliability of the middleware.
Fixes#4906
* fix(clarification): enhance tool call filtering in ClarificationMiddleware
- Update _filter_content_tool_use to handle Gemini-style function_call blocks by matching on name when no id is present, ensuring proper filtering of tool calls.
- Modify ClarificationMiddleware to maintain sibling tool call integrity by dropping unnecessary blocks, improving the clarity of the AIMessage content.
- Add a new test to validate the correct stripping of idless function call content blocks, ensuring that sibling tool calls do not execute prematurely.
This change improves the robustness of the middleware and addresses potential execution order issues.
Fixes#4906
* fix(clarification): drop siblings when ask_clarification is malformed
LangChain parks invalid args on invalid_tool_calls independently, so a
valid sibling would otherwise still execute before the user answers.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by
max_recursion_limit) so scheduled background runs can use a different
recursion limit than the web UI. The value is read at dispatch time, so
a YAML edit applies to the next scheduled run without a Gateway restart.
Also logs a warning when the resolver falls back to the default or
clamps the configured value.
The cancel endpoint resolved McpTaskService whenever SQL persistence was
configured, even with mcp_tasks.enabled=false, and acknowledged the
request by recording cancel_requested_at. The background loop that owns
the remote cancel call only runs when enabled, so the fence was never
claimed and the remote task kept running indefinitely.
Gate the endpoint on app.state.mcp_tasks_available (set only after the
service is started) and return 503 before writing the fence, consistent
with the background-loop ownership contract. Read-only list/detail
endpoints remain available while the worker is stopped.
* feat(harness): add deterministic tool receipts with model-visible ledger
Stamp an immutable per-call fact record (tool name, status, args/output
hashes, byte count, timestamp) onto every tool result via a new
ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN)
into the model context so subagent reports can cite executed actions.
- tool_receipt.py: receipt core (make/extract/render), newest-first
budget eviction, ids derived from the append-only message stream
- ToolReceiptMiddleware: stamps ToolMessages directly or inside
Command-wrapped results; hidden ledger injection mirrors
DurableContextMiddleware; sits between ToolProgress and
ToolErrorHandling with a build-time ordering guard
- config: new verification section (receipts on, judge off), config
version 32 -> 33 with example/helm/docs updates
* feat(harness): split receipt rendering from stamping; address PR review
Review fixes (PR #4659):
- output_sha256 now uses sort_keys=True for structured content, matching
the order-invariant args fingerprint
- stamping failures log at warning (silent ledger gaps would corrupt
citations); tool execution remains never blocked
- _insert_after_leading_system_messages extracted to shared public
message_utils.insert_after_leading_system_messages; both middlewares
depend on it instead of a private cross-module helper
- code comments in English
RFC #4651 revision-2 alignment:
- receipts_render_mode config ('always' | 'delegation_only'): subagent
chains always render the ledger (citations are produced there); the
lead chain renders only while processing subagent results, removing
the always-on token tax from ordinary turns
- receipts gain bounded args_preview/output_preview (<=200 chars, tail
for output) so later typed claim bindings (tests_passed) can anchor
to a specific recorded execution
* docs(harness): state receipt freshness caveat and vocabulary layering in module docstring
* merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews
- backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the
ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the
verification.* hot-reload mention into config/AGENTS.md
- config.example.yaml + helm values/README: config_version 33 -> 34 so existing
v33 configs get the outdated-config prompt (review: willem-bd)
- tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads
them; re-add with the Layer 2 claim-binding consumer (review: willem-bd)
* docs(harness): cover receipt id renumbering after compaction in module docstring
Positional display ids are stable only while history is append-only;
compaction drops ToolMessages and the survivors renumber, so Layer 2
citation verification must resolve [rN] against the ledger as of the
citing turn (review: willem-bd, doc-only).
* chore(config): bump config_version to 35
main reached 34 via #4780 without the verification section; publishing
the new schema at the same number would silently skip the outdated-config
prompt for configs synced from main in that window (review: willem-bd).
* fix(skills): restore errno import dropped upstream in #4830
upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed
the import, so the PR merge ref fails lint-backend (F821).
* fix(harness): harden tool receipts against forgery and turn-scope delegation_only
Address willem-bd's pre-merge review on #4659:
1. Untrusted receipt metadata: the gateway now strips the server-owned
deerflow_tool_receipt key from external input messages; stamping always
overwrites any tool-supplied value instead of preserving it; and
extract_tool_receipts validates persisted receipt shapes (required typed
fields, unknown keys ignored) so malformed entries are skipped instead of
crashing render or passing as runtime-stamped evidence.
2. delegation_only no longer sticks on: _should_render now scopes the
subagent_status scan to the current turn (messages after the latest
genuine user message), so an old completed delegation stops rendering the
ledger on later ordinary turns. The genuine-user predicate moves to
message_utils.is_genuine_user_message, shared with input sanitization.
* fix(harness): stamp receipts outside short-circuiting tool middlewares
Address willem-bd's review on #4659: ToolReceiptMiddleware was registered
inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which
can return a ToolMessage without invoking its handler — blocked calls
(e.g. a read-before-write-denied write_file) never got a receipt, silently
gapping the ledger on a default-enabled path. SandboxAudit additionally
rebuilds medium-risk results, dropping an inner stamp.
ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the
runtime tail. Normal results still carry deerflow_tool_meta (stamped by
ToolErrorHandling on the inner return path); short-circuit messages
self-stamp meta or fall back to message.status. The new invariant is
declared as ordering constraints in deerflow.extensions.ordering, with
composed-chain regression tests for a blocked write and a warn-rebuilt
bash result.
* feat(mcp): per-user credential injection for shared MCP servers
A single HTTP/SSE MCP server entry can now serve several users, each
authenticated to the remote service with their own credential. A server
opts in with a user_auth block mapping user ids to credential header
values ($ENV_VAR references supported):
"user_auth": {
"header": "Authorization",
"users": { "<user-id>": "$SERVICE_TOKEN_ALICE" }
}
The built-in user-scoped auth interceptor resolves the authenticated
runtime user on every tool call (request runtime -> ambient LangGraph
runtime -> auth config -> request-scoped user ContextVar) and rewrites
the configured header via request.override(), the same per-call
mechanism as the OAuth interceptor. It registers after OAuth in the
shared assembly so its per-user value wins the header when a server
declares both. The entry's static headers are used only for startup
tool discovery.
Fail-closed by default: an unmapped user - including the anonymous
default-user fallback - or a credential whose env reference resolved
empty gets an actionable ToolException instead of another user's
credential; on_missing: "passthrough" opts out per server. Combined
with the existing per-(user, thread) MCP session scoping this gives
credential isolation on shared servers.
Gateway API: user_auth.users values are masked in GET responses, and
PUT round-trips preserve stored credentials for masked values (same
contract as env/headers/oauth secrets); a masked value for a user id
not already stored is rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): address review — preserve stored user_auth sub-fields on partial PUT, warn-and-skip stdio, allow extras
Review findings from #4868:
1. A partial user_auth payload (e.g. {"enabled": false}) merged to
users={} and default on_missing, irreversibly wiping stored
credentials on PUT. The merge is now sub-field-aware via
model_fields_set — omitted sub-fields carry the stored values, an
explicitly sent users map still replaces (so full-round-trip removal
works), masked values still swap back for stored credentials.
2. user_auth on a stdio server was a silent no-op: rewritten headers go
to call meta, never a transport header, while deny errors still fired.
The interceptor builder now warns and skips non-sse/http servers,
matching the tool_call_timeout transport-mismatch convention.
3. McpUserScopedAuthConfigResponse now allows extra keys like the
harness-side model, and extras survive masking and merge, matching
the server-level model_extra handling.
Adds four regression tests (partial-PUT preservation, explicit-map
replacement, extras round-trip, stdio warn-and-skip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): reject blank user_auth.header at the gateway
A blank header passed the gateway response model, was persisted, then
failed the harness-side ExtensionsConfig validator on reload — the PUT
returned 500 after the write and every later config load/startup failed
until the file was hand-edited. Mirror the harness non-blank validator
on McpUserScopedAuthConfigResponse so the PUT fails with 422 before
anything is written.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format extensions_config.py
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): harden the trust chain and masking around user-scoped credential selection
Address review round 4:
- Scrub client-supplied user_id from run context/configurable for
external callers in inject_authenticated_user_context, before every
early return, and restamp only from request.state.user. Now that
user_id selects which user's credential user-scoped MCP auth injects,
a forged value must not survive any future path that skips the
restamp. Internal callers (IM channels, scheduler) keep supplying
end-user identity as before (PR #3294 contract). Regression tests pin
both the scrub and that a forged body.context.user_id can never
resolve as another user through merge + inject ordering.
- Include the caller's own resolved user id in the fail-closed deny
message so operators can copy the exact users key (it differs by
deployment path), and document the key formats in the mcp.mdx doc.
- Mask sensitive extra keys inside user_auth on GET like server-level
extras, and swap masked sentinels back for stored values on PUT via
_merge_extra_value_preserving_masked.
- Extract the interceptor wrap loop into compose_tool_interceptors and
pin the security property functionally: an OAuth interceptor that
actually sets Authorization loses the final header to the per-user
credential through the same composition the session-pool path uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(docker): let the Gateway write extensions_config.json in production
AGENTS.md states extensions_config.json may be edited at runtime through the
Gateway API, and the Gateway implements that for the MCP enable switch,
PUT/PATCH /api/mcp/config and the skill update route. Two properties of the
production compose stack made every one of those writes fail:
- the file was mounted read-only, and
- Docker mounts it as its own mount point, so the temp-file-plus-rename in
atomic_write_extensions_config hit EBUSY. Linux refuses rename() over a
mount point whether or not the mount is writable, so making the mount
read-write alone is not enough.
Mount it read-write and fall back to an in-place overwrite on EBUSY only.
The fallback is deliberately non-atomic and says so in a warning; it is
reached only where the atomic route cannot work, and any other errno still
propagates. config.yaml stays read-only: no API writes it.
docker-compose-dev.yaml mounts the whole project directory, so the
destination is an ordinary file there and this never surfaced in development.
* test(docker): parse mount options instead of matching a :ro suffix
Docker's short-syntax options segment is comma-separated, so a read-only
mount can legally be spelled ":ro,z" or ":z,ro" — common with SELinux
relabelling. Matching the raw string for a ":ro" suffix reads those as
writable, which silently defeats the guard: the writability assertion would
pass on a read-only mount, and the config.yaml assertion would fail on a
correctly read-only one.
Parse the options segment and test membership instead, and cover the parser
with the spellings that broke the suffix check.
* fix(config): harden mutable extensions config
Move SQL-dependent connection identity and /start bind work onto the
Gateway main loop via _submit_threadsafe_coroutine, and send PTB replies
through _run_on_telegram_loop so the Telegram worker never blocks or
touches SQLAlchemy/HTTP across event loops.
When the main loop is not running (e.g. during gateway shutdown), the
bind path logs a warning and returns False instead of running SQLAlchemy
on the wrong loop, matching the Feishu bind pattern.
* feat(extensions): let an out-of-tree extension observe what the agent did
DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:
* The middleware chain injects and rewrites a lot of context — date
reminders, recalled memory, compaction summaries, durable-context data,
image payloads, activated skill bodies. Downstream, none of it is
attributable: at the model-call boundary an injected HumanMessage is
indistinguishable from the user's own, and anything wanting to tell them
apart has to pattern-match prompt wording, which breaks on the next copy
edit.
* Two runs of "the same agent" are only comparable if the chain enforced the
same limits, prompts and thresholds. Recovering that from outside means
reading private attributes and guessing which of them change behaviour — a
guess that rots silently as middlewares gain fields.
* The lead-agent factory resolves a model after runtime overrides, renders a
prompt, filters tools through authorization and composes a stack, all
inside one synchronous call, and none of it survives: a middleware sees its
neighbours but not the prompt, the run worker sees a graph but not what
went into it.
* Summarization is destructive by design. N messages leave the context and
one summary enters it; afterwards only the summary exists, so "which
messages became this?" is not reconstructible.
This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.
Message provenance
Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
messages they inject or rewrite. Stamping is unconditional — a fact whose
presence depends on whether an observer is installed is not a fact — and the
keys are server-owned, so provenance cannot be forged from a request.
Middleware self-description
Twelve middlewares declare their own behaviour-affecting parameters through
a duck-typed `release_policy_parameters()`. Long text is hashed rather than
embedded: a declaration is an identity, not a copy of the prompt.
Agent assembly descriptor
`assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
answers "did anything about this agent change between these two runs?".
`make_lead_agent()` keeps its graph-only signature — it is the LangGraph
Server ABI declared in langgraph.json. Tools and skills are sorted before
hashing because their assembly order is incidental; middlewares are not,
because stack order decides what wraps what. Host build identity is reported
but excluded from the fingerprint, so a redeploy does not invalidate every
agent's identity.
Context compaction observation
Summarization emits the content hashes of the messages it is about to remove
joined to the summary that replaced them. Content is the only identity
available at that seam: the summary does not become a message, and what later
projects it into a request renders it bounded and escaped rather than
verbatim.
Neutral policy, transform and MCP-source facts
Guardrail decisions are published to runtime context under a `__`-prefixed
key; result-rewriting middlewares append a declared, ordered transform trail;
MCP tools carry their credential-free logical origin.
Extension route identity
Contributed routes are session-authenticated and cannot opt out, but
"logged in" and "administrator" are different questions. Extensions get a
neutral projection of the caller rather than the host's auth context, and
`require_admin` fails closed when identity cannot be determined.
Extension-owned tables
An extension that persists data owns its own MetaData and migration chain, so
its tables are absent from Base.metadata and `alembic revision --autogenerate`
proposes dropping them. Extensions declare a table prefix, which is rejected
at registration if it would shadow a host table.
The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.
uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.
* fix(backend): sort gateway service imports
