* feat(auth): add personal access tokens for programmatic API access (#4849)
Backend-first implementation of the PAT contract from #4849: show-once
dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT,
is_internal=false), digest-only storage (migration 0017), strict
credential precedence (invalid Bearer is a 401, never cookie fallback),
CSRF double-submit skipped only for Bearer requests while
auth-endpoint origin checks still run, scopes intersecting the authz
route permissions, session-auth-only PAT management and password
changes, and throttled best-effort last_used_at stamps.
* fix(auth): harden PAT scope boundary and schema parity from adversarial review
Independent review of the initial draft found: (1) scopes only constrained
the threads/runs permission axis while admin routes treated a PAT as its
(possibly admin) owner — is_admin_user now rejects PAT callers outright
since no scope grants admin capability; (2) the model declared a column
UNIQUE constraint while migration 0017 created a named unique index, so
downgrade failed on create_all-bootstrapped DBs — both now use the named
unique index; (3) auth-disabled mode is an operator override and now stays
ahead of the Bearer check so a stray Authorization header cannot 401 an
E2E sandbox; plus wiring the previously-unused constants, bounding the
last_used_at stamp cache, and four new tests (middleware-level expiry,
expires_in_days, admin-capability rejection with session control, and the
auth-disabled precedence).
* docs(api): document personal access tokens for programmatic API access
* fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression)
P1-1: scope intersection only constrains @require_permission routes, so
undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark
credential switching, channel config) accepted a PAT holding a single read
scope. AuthMiddleware now enforces a default-deny route policy in
auth/pat.py: PAT requests are admitted only to the thread/run lifecycle
routes the v1 scopes govern; everything else answers 403 regardless of
scopes. Session-cookie callers are unaffected.
P1-2: the extension principal resolver projected is_admin/roles from the
raw system_role, so an admin-owned PAT passed
deerflow_extension_api.require_admin on contributed routes despite the
documented no-admin guarantee. The projection is now PAT-aware and
suppresses every admin signal for PAT callers, mirroring
deps.is_admin_user.
Both fixes carry regression tests (route outside policy 403 + session
control; production resolver admin suppression), and API.md documents the
default-deny boundary.
* fix(auth): enforce PAT scopes on stateless run entry and harden decorator
Follow-up hardening from an independent audit of the P1 fixes:
- POST /api/runs/stream and /api/runs/wait were the only allowlisted run
entrypoints without @require_permission, so a threads:read-only PAT
could still start runs (same bug class as P1-1, now closed): both now
carry @require_permission("runs", "create"). POST /api/threads and
POST /api/threads/search gain threads:write / threads:read for the
same reason. Authorization-disabled deployments see no change (the
permission set resolves to all permissions).
- require_permission now binds the wrapped signature to locate a
positionally-passed request before injecting the test stub, fixing
'got multiple values for argument' on direct positional unit-test
calls.
- API.md: the intro PAT example used GET /api/models, which the new
default-deny policy 403s — replaced with GET /api/threads; the
default-deny route list now spells out method sets.
Regression test: threads:read-only PAT is 403 on the decorated stateless
entry while a runs:create PAT passes.
* fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example)
- CSRFMiddleware treats an explicitly empty Authorization header as
present (is None), so an invalid credential always reaches
AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by
method/CSRF state. Regression: empty-header request dies at auth.
- PATCreateRequest strips the name and rejects whitespace-only values
before token generation; created names are stored trimmed.
- API.md intro PAT example now uses the implemented
POST /api/threads/search endpoint (GET /api/threads does not exist).
- AGENTS.md trimmed back under the guidance soft budget after the
upstream merge.
* fix(auth): tighten PAT route policy to implemented methods only
The allowlist admitted GET /api/threads, a method no router implements.
Pre-authorizing a dead method weakens the default-deny boundary: a
future GET collection route added without a permission decorator would
become PAT-reachable without an explicit policy change. Restrict the
rule to POST, fix the stale GET description in API.md's PAT
constraints, and document the default-deny boundary accurately in the
gateway AGENTS.md guidance (only the threads/runs allowlist is
PAT-reachable; every other authenticated route 403s PAT callers).
Audited every remaining rule against the mounted routers: all other
method+path entries map to real routes. Regression:
test_pat_policy_does_not_pre_authorize_unimplemented_methods.
* test(auth): guarantee the negative digest test mutates the token
token[:-1] + "X" is identical to the original whenever the generated
token already ends in X (1/62), making the negative digest assertion
fail intermittently. Choose the replacement character based on the
existing tail so the mutated token always differs.
* fix(auth): require runs:cancel for cancel-then-stream requests
stream_existing_run is gated at runs:read so action-less stream joins
work with read-only credentials, but its ?action=interrupt|rollback
branch cancels the run — a separate permission. A runs:read-only PAT
passed both the PAT route policy and the route decorator and could
interrupt or roll back an active run, bypassing the runs:cancel scope.
Decorators cannot express query-parameter-conditional permissions, so
the check lives in require_cancel_permission_when_action(), applied at
the top of the handler. Regression drives the real helper through the
production middleware: runs:read-only PAT + action is 403, the same
token joins action-less, runs:read+cancel passes, session control
unaffected.
* docs(changelog): add the PAT feature entry
* docs(readme): add personal access tokens section
Repo documentation-update policy requires user-facing features to
update README.md in the same changeset; the PAT feature previously
touched only backend/docs/API.md and the gateway AGENTS.md.
* fix(auth): require runs:cancel for mutating multitask strategies
All five run-creation entrypoints were gated only by runs:create, but
RunCreateRequest.multitask_strategy accepts interrupt/rollback and
start_run forwards it to create_or_reject, which terminates an
already-active run. A runs:create-only PAT could therefore kill an
existing run through a create request, bypassing runs:cancel.
Decorators cannot express body-parameter-conditional permissions, and
per-route checks leave the same hole for the next entrypoint, so the
gate lives in start_run itself — the single choke point every
run-creation path (HTTP routes and internal launchers) flows through.
Regenerate launches pass multitask_strategy="reject" and are
unaffected; requests without a stamped auth context (internal/test
compositions) skip the gate.
The check is the shared authz.require_cancel_permission_if primitive;
require_cancel_permission_when_action now delegates to it, so every
request dimension that carries cancel capability (query action, body
strategy) flows through one gate.
Regression drives the real middleware stack: runs:create-only PAT +
interrupt/rollback is 403 with the exact detail, reject (explicit and
default) stays available, runs:create+cancel passes, session control
unaffected; a source anchor pins the gate inside start_run.
* fix(runs): keep observer joins from applying creator cancel-on-disconnect
sse_consumer's finally block applied the record's on_disconnect=cancel
policy on ANY consumer's disconnect. The join surfaces (GET /join and
the action-less GET/POST stream join) feed it the existing RunRecord,
so anyone with thread read access — including a runs:read-only PAT —
could cancel a locally-owned running run simply by closing the SSE
connection, without runs:cancel. The policy expresses the creator's
intent for their own connection; an observer's disconnect must never
be read as that intent.
sse_consumer gains apply_on_disconnect (default True). The two join
surfaces pass False; the creating endpoints (thread-scoped and
stateless create-and-stream) keep the creator semantics unchanged.
wait_for_run_completion needs no change: its callers are creator-side
or post-explicit-cancel paths only.
Regression exercises a real generator close — the same machinery
Starlette drives on client disconnect — against the production
sse_consumer: creator stream disconnect cancels, observer join
disconnect does not; a wiring anchor pins both join call sites and the
creator defaults. API.md documents the cancel-capability constraint
(this fix plus the action/strategy gates) in PAT Constraints.
* test(auth): pin the multitask gate behaviorally; state wait invariant
Independent adversarial review of the round-5 fixes found the P1-a
regression only mirror-pinned: the source anchor could be satisfied by
a comment, and deleting the gate from start_run would not fail the
suite. This drives the production start_run directly — a create-only
auth context gets 403 with the exact detail for interrupt, and a
reject request with no cancel permission at all proceeds past the gate
(never a permission 403).
Also documents wait_for_run_completion's creator-side invariant
(every caller is the creating endpoint or post-explicit-cancel) so a
future observer wiring thinks twice before reusing it — the one-caller-
away variant of the observer-disconnect P1.
* docs(changelog): correct the PAT entry's digest and route-policy description
The entry said HMAC digests (the implementation stores SHA-256 digests,
as documented in API.md and pinned by the repository tests) and claimed
the route policy admits 'implemented stateless endpoints' (it admits
the thread/run lifecycle routes, narrowing further by scopes). Also
notes the cancel-capability gate now covering action and multitask
strategies.
* fix(auth): enumerate the PAT runs route policy per implemented subroute
The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it
pre-authorized every current and future subroute under /runs, including
methods the router never implemented (e.g. GET /runs/stream), which is
the same latent default-deny weakening the threads collection rule was
tightened for: a future route added under /runs would become
PAT-reachable without an explicit policy change.
The wildcard is replaced with six segment-precise rules covering exactly
the 14 implemented method+path combinations; the {run_id} slot
necessarily matches any single segment, so the POST-only collection
names (stream, wait, regenerate, edit-regenerate) are excluded from the
GET run-id rule via negative lookahead — no dead method stays
pre-authorized. Behavior for implemented routes is unchanged.
test_pat_runs_policy_admits_exactly_the_mounted_routes derives the
expected set from the mounted thread_runs router instead of a
hand-maintained list: every implemented GET/POST route under /runs must
be admitted, routes in this router outside the subtree stay denied, and
representative unimplemented neighbors are denied — so adding a route
under /runs now fails CI until it is explicitly allowlisted, and a
removed route leaves a dead rule visible. API.md's PAT constraints list
the enumerated routes and drops a feedback mention that belonged to the
stateless /api/runs axis.
* docs(migration): add the 0017 renumbering coordination note to 0017
The PR's migration-coordination comment states each migration file
carries the note; the file did not. Adds it: numbering was generated
against main head 0016 alongside #5078 and #4843; whoever merges first
keeps the slot, the others renumber on rebase (revision/down_revision
plus the bootstrap head assertions).
* fix(auth): pad base62 tokens to a fixed 43-char width
int.from_bytes discards leading zero bytes, so the unpadded encoder
returned a variable-length body — empty for all-zero input, and shorter
than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving
test_generate_pat_token_format probabilistically flaky and the token
body without stable width (review round 6, P3).
_base62 now left-pads with "0" to _base62_width(len(data)) — the exact
integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The
format test asserts the exact fixed width instead of a probabilistic
floor, and a new unit test pins the all-zero, leading-zero-byte, and
max-value edges deterministically.
* fix(runtime): prevent IndexError in MemoryStreamBridge._make_gap on empty events buffer
* fix(runtime): handle empty stream replay gap bounds across backend and frontend
- Clamp MemoryStreamBridge queue_maxsize at 1 and validate StreamBridgeConfig.queue_maxsize >= 1
- Update StreamGap docstring to clarify None retained bounds
- Allow StreamReplayGapData and parseStreamReplayGap in frontend to accept string | null bounds, safely resuming when bounds are null
- Add backend and frontend regression unit tests for queue clamping and null bounds replay gap
* docs(stream-bridge): bump config_version and document empty buffer replay gap behavior
* docs: document nullable gap bounds and sync helm config_version to 37
* feat(harness): subagent receipt citation verification
- add receipt citation verification core
- harvest subagent tool receipts at terminal status
- transport subagent receipts and citation verdict via status contract
- verify subagent report citations at task write-back
- render citation verdicts in the delegation ledger
* fix(gateway): strip forged receipt verdicts from the delegations channel
normalize_input() and the checkpoint-state mutation sanitizer only
stripped server-owned metadata from message-shaped values, so an
external caller could submit a delegation entry carrying a forged
receipt_verdict that render_delegation_ledger would present as
runtime-owned execution evidence. Strip receipt_verdict from
caller-supplied delegation entries on both the run-creation and
thread-state mutation paths, with regression coverage for each.
* fix(harness): close silent-pass gaps in the zero-citation heuristic
The action-claim detector missed the most common completion verbs
(fixed/added/tested/changed/...) and had no CJK coverage at all, so
reports like 'I fixed the bug and added tests.' or '我已经创建了文件并运行了测试。'
were treated as claim-free: citation_resolved=True with no ledger
warning. Broaden the verb lists and add a language-independent safety
net: when the run harvested receipts and a nontrivial (>=240 char)
report cites none of them, flag it UNVERIFIED. Short claim-free
confirmations remain a vacuous pass.
Add deerflow.community.serply.tools:web_search_tool, a Google SERP
provider for the web_search slot that also covers Google News and Google
Scholar through an optional `vertical` config option. Reads the key from
api_key in config.yaml or SERPLY_API_KEY, clamps max_results to Serply's
1-100 range, and returns the same structured JSON errors as the Serper
and Brave tools.
Register the provider in config.example.yaml, scripts/doctor.py,
scripts/wizard/providers.py, .env.example, backend/docs/CONFIGURATION.md,
the en/zh tools.mdx provider tabs, and tools/AGENTS.md. Tests mock httpx.
normalize_filename accepts names up to 255 UTF-8 bytes, but
claim_unique_filename appended _N to the stem without re-checking the
budget. A duplicate at maximum length therefore produced a 257-byte
name, and the write path (open_upload_file_no_symlink ->
normalize_filename) rejected it with ValueError. In the Gateway upload
route that error falls into the generic handler: the whole request
fails with a 500 and files already written in the same batch are rolled
back — including unrelated ones. The same helper backs the Feishu and
DingTalk channel downloads and client-side attachment staging.
Truncate the stem on a UTF-8 code-point boundary when appending the
dedupe tag would exceed 255 bytes, so the result always round-trips
through normalize_filename. Names short enough to fit keep the exact
dedupe shape they had before.
Tests: red on main, green here —
- unit: max-length dedupe stays within the limit and round-trips;
repeated collisions stay unique; multibyte stems truncate on a
code-point boundary; short names keep the historical _N shape
- router: a batch with a max-length duplicate now succeeds and keeps
every file instead of failing with a 500
Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
* fix(skills): accept portable frontmatter forms
* fix(skills): normalize portable tool names
* Safely preserve parenthesized portable skill tool patterns
Portable Agent Skills declarations such as Bash(tvly *) contain spaces inside a command pattern. Keep those patterns as single literal entries while preserving exact names from the existing YAML-list form, so skill loading no longer fragments valid metadata or rewrites mixed-case MCP tools.
Constraint: DeerFlow's current skill policy matches exact tool names and does not inspect Bash arguments
Constraint: Agent Skills scalar syntax uses whitespace-separated entries with parenthesized command patterns
Rejected: raw.split() | fragments Bash(tvly *) into unrelated tool names
Rejected: normalize YAML-list entries | breaks case-sensitive MCP/runtime tool names
Rejected: map Bash(...) to bash | broadens command-scoped declarations into unrestricted shell access
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Bash(...) entries literal and inactive until DeerFlow has an explicit command-pattern authorization model
Tested: 175 focused parser, validation, installer, review, loader, and tool-policy tests; Ruff check and format; compileall; git diff --check
Not-tested: Full backend suite stopped at pre-existing Windows mode assertion test_runtime_config_store_file_is_owner_only
Related: #4912
* Preserve exact custom tool names in portable skill parsing
Portable scalar frontmatter needs alias normalization for known DeerFlow-compatible names, but generic case conversion corrupts MCP and custom tool identifiers. The tokenizer also treated quoted or escaped parentheses as structural delimiters, rejecting valid command patterns. Preserve unknown names and parse quoted or escaped patterns without broadening Bash(...) into bash.
Constraint: Runtime skill policy uses exact tool-name matching
Constraint: Parenthesized patterns remain literal because argument-level authorization is not implemented
Rejected: Generic CamelCase-to-snake_case for every scalar | rewrites custom/MCP names
Rejected: Map Bash(...) to bash | broadens command-scoped declarations into unrestricted shell access
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Add an explicit alias before supporting another portable tool name; keep command-pattern authorization separate
Tested: 225 skills tests passed, 1 skipped; Ruff check; Ruff format --check; compileall; git diff --check
Not-tested: Full backend suite remains affected by unrelated Windows permissions/path and missing Lark CLI tests
Related: #4984; #4912
* Preserve case-sensitive exact tool authorities
Case-folding a scalar declaration before alias lookup can turn literal write into write_file, substituting a different runtime authority. Keep exact portable spellings as aliases and preserve lowercase, custom, and MCP names; strengthen activation coverage for spaced Bash patterns and command fragments.
Constraint: Runtime skill policy uses exact tool-name matching
Constraint: Bash(...) remains literal and inactive because command-pattern authorization is not implemented
Rejected: Case-insensitive alias lookup | maps lowercase runtime tools onto built-in authorities
Rejected: Broaden the parser into command-pattern authorization | outside this PR's scope
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Add aliases only for documented portable spellings; preserve all other scalar names verbatim
Tested: 226 skills tests passed, 1 skipped; Ruff check; Ruff format --check; compileall; git diff --check
Not-tested: Full backend suite remains affected by unrelated Windows permissions/path and missing Lark CLI tests; GitNexus index refresh remains stale
Related: #4984; #5016297602
* Support portable Glob and Grep skill aliases
Portable Agent Skills commonly declare Glob and Grep, but DeerFlow exposes the runtime tools as glob and grep. Add explicit exact-spelling aliases and activation coverage so imported skills retain search-tool access without broad normalization.
Constraint: Runtime skill policy uses exact tool-name matching
Constraint: Alias conversion is limited to documented portable spellings
Rejected: Case-fold all scalar names | can substitute custom or MCP authorities
Rejected: Map arbitrary names by convention | breaks exact runtime compatibility
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep the alias table explicit and preserve unknown scalar names verbatim
Tested: 228 skills tests passed, 1 skipped; Ruff check; Ruff format --check; compileall; git diff --check
Not-tested: Full backend suite has unrelated environment failures on Windows; GitNexus index reports stale line mappings
Related: #4984; #5026257899
---------
Co-authored-by: kriptoburak <kriptoburak@users.noreply.github.com>
* fix(sandbox): harden local Docker sandbox containers and port binding
Root causes (security audit SBX-1/SBX-2) in the local container backend:
- _resolve_docker_bind_host published sandbox ports on 0.0.0.0 whenever
DEER_FLOW_SANDBOX_HOST was non-loopback (docker-compose defaults to
host.docker.internal), exposing the unauthenticated /v1/shell/* exec
API on every host interface.
- _start_container ran every sandbox with seccomp=unconfined and no
capability, privilege-escalation, or resource limits, so untrusted
model-authored code could exhaust the host, escalate privileges, and
reach internal networks / cloud metadata endpoints directly.
Hardening changes and defaults:
- Port binding: non-loopback sandbox hosts now bind the Docker default
bridge gateway instead of 0.0.0.0, discovered dynamically via
`docker network inspect bridge` with a static 172.17.0.1 fallback.
host.docker.internal resolves to that gateway through host-gateway,
so DooD gateways and the Docker host still reach the sandbox while
external interfaces no longer see the port.
DEER_FLOW_SANDBOX_BIND_HOST=0.0.0.0 restores the legacy broad bind.
- seccomp=unconfined is no longer unconditional: sandboxes run with
Docker's default seccomp profile; opt back in with
DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=1, only when the sandbox image
is verified to require syscalls the default profile blocks.
- Add --cap-drop=ALL and --security-opt no-new-privileges (Docker only;
the Apple Container CLI does not support these flags).
- Bounded resources with env overrides: --memory 2g
(DEER_FLOW_SANDBOX_MEMORY), --cpus 2 (DEER_FLOW_SANDBOX_CPUS),
--pids-limit 512 (DEER_FLOW_SANDBOX_PIDS_LIMIT); each also accepts
"0"/"none" to disable the limit.
- No --user is forced by default (the default AIO sandbox image's user
is upstream-controlled and unverified), but
DEER_FLOW_SANDBOX_CONTAINER_USER passes one through for deployments
that know their image.
- DEER_FLOW_SANDBOX_NETWORK passes --network so sandboxes can be
attached to a dedicated egress-controlled network; default networking
is unchanged.
backend/docs/CONFIGURATION.md documents the new bind behavior and every
override; tests cover each default and escape hatch.
* fix(sandbox): follow host-gateway mapping for binds; keep image-required seccomp default
Review follow-ups on the hardening change:
- Bind: resolve the sandbox host itself and bind that address, instead of
assuming the default bridge IPv4. host.docker.internal follows the
daemon host-gateway-ip mapping (customizable, possibly IPv6), so the
resolved address is exactly where the gateway connects — the published
port and advertised URL always match. IPv6 is bracketed for docker -p,
zone ids stripped, wildcard resolutions ignored; unresolved hosts fall
back to the bridge gateway with a warning pointing at
DEER_FLOW_SANDBOX_BIND_HOST.
- seccomp: the shipped AIO image needs seccomp=unconfined for its
Chromium browser (upstream quick-start always passes it; the upstream
FAQ documents the browser failing under Docker default profile), so
that option returns as the default. Tightening stays possible via
DEER_FLOW_SANDBOX_SECCOMP_PROFILE=<path to a restricted,
Chromium-compatible profile> or DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=0
for images verified to work with Docker's default profile.
- cap-drop/no-new-privileges and the resource limits are unchanged.
- Tests updated for both behaviors; 37 pass.
* fix(sandbox): bracket bare IPv6 bind overrides; state seccomp default accurately
DEER_FLOW_SANDBOX_BIND_HOST was returned verbatim, so a bare IPv6 literal
like fd00::1 produced an invalid publish spec (fd00::1:port:8080); Docker
requires the bracketed form. Normalize raw and already-bracketed IPv6
literals (IPv4/hostnames untouched), with resolver-level and argv-level
tests covering the explicit IPv6 override.
The CONFIGURATION.md overview claimed Docker's default seccomp profile
stays active, contradicting the seccomp=unconfined default the table (and
the code) actually ship for the Chromium-based image; spell out the relaxed
default and where to change it.
* style(sandbox): apply ruff format to local_backend
* fix(sandbox): reject host networking, force builtin seccomp opt-out, resolve hostname binds
Review follow-up on #4986 (willem-bd):
- P1: DEER_FLOW_SANDBOX_NETWORK=host (and container:<name>) now raise a
RuntimeError at start instead of silently voiding the hardened port
bind — Docker discards -p/--publish in host mode and shares the
network namespace for container:<name>, which would re-expose the
unauthenticated exec API on the host's interfaces. Two regression
tests cover both rejections.
- P2: the seccomp opt-out now passes seccomp=builtin explicitly instead
of omitting the option, so a daemon configured with an unconfined or
custom default cannot weaken the documented opt-out; the test asserts
the flag.
- P2: hostname values in DEER_FLOW_SANDBOX_BIND_HOST resolve to an
address before use (Docker publish specs require an IP literal as the
host part, so host.docker.internal previously produced an invalid
spec that prevented every sandbox from starting); unresolvable names
raise a clear configuration error. Tests cover resolution and
rejection; CONFIGURATION.md updated for all three behaviors.
43/43 pass in tests/test_aio_sandbox_local_backend.py; ruff check +
format clean.
* fix(sandbox): reject DEER_FLOW_SANDBOX_NETWORK=none (loopback-only, breaks published API port)
* fix(sandbox): validate the effective Docker network target; normalize IPv6 sandbox hosts once
name=host / name=none dodge raw-string checks but attach like the bare
words; strip name= prefixes and validate the effective target (network IDs
keep passing). Bracketed IPv6 sandbox hosts now resolve for the bind and
bare IPv6 hosts produce bracketed URL authorities — both input forms give
identical bind and URL addresses.
* fix(sandbox): parse the full Docker network long syntax before validating
Docker accepts comma-separated key=value fields in any order (name=, gw-priority=,
alias=, ...); a name=host field hides the host network behind surrounding fields.
Parse the CSV and validate the parsed name= target (last occurrence wins, fields
lowercased, mirroring opts/network.go); no-name values fall through like Docker's
own rejection.
* fix(sandbox): keep CHOWN/SETUID/SETGID through cap-drop=ALL for the default image
The shipped image's entrypoint starts as root, creates the gem user,
chowns /opt/jupyter and drops to that user via su; without those three
capabilities the set -e script dies before the readiness endpoint exists.
no-new-privileges stays (it blocks gaining privileges via exec, not using
the added caps). Adds a docker-gated real-image startup smoke test.
* fix(sandbox): let pre-initialized non-root images drop the startup capabilities
The CHOWN/SETUID/SETGID re-add only exists for the shipped image's root
entrypoint handoff. A custom image that never runs as root gets an explicit
opt-out (DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0) so those capabilities are
not left available to sandboxed code (chown on bind mounts, UID/GID
impersonation).
* test(sandbox): gate the real-image smoke test behind the live marker
The default offline suite (make test = -m 'not live') must not depend on a
third-party registry: mark the smoke test live, probe the daemon inside the
test body (never at collection time), and allow pinning the image reference
via DEER_FLOW_SANDBOX_SMOKE_IMAGE for a dedicated integration job.
* test/docs: isolate DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS in tests; add table row; split custom-image guidance
_clear_hardening_env now clears the new knob so a developer shell or .env
preset cannot flip the default-path tests. CONFIGURATION.md gains the table
row, and the custom-image guidance becomes its own paragraph with the
no-new-privileges scope stated correctly (it does not mitigate the retained
CAP_SETUID/SETGID risk).
* test(sandbox): make the live smoke test diagnosable
300s readiness budget (cold pull + cold start must not be conflated with
broken capabilities) and dump the container's last 40 log lines on failure
so the next live run tells us whether the capability set is incomplete
(chown/useradd/su errors) or the services are merely slow.
* test(ci): align the smoke test with the 60s provider deadline; add a dedicated live smoke workflow
Single-source the readiness deadline as SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT
(used by both provider paths and the smoke test) so the validation cannot
drift from the production contract again. New sandbox-image-smoke.yml runs
the live test on a dedicated job, with the image reference pinnable via the
SANDBOX_SMOKE_IMAGE repository variable (digest resolved and recorded in the
job summary when falling back to :latest).
* test(sandbox): pull the failing program's own logs on smoke failure
supervisord only surfaces exit codes in docker logs; nginx's stderr lands in
files inside the container. Dump supervisor program logs, nginx -t, and the
nginx error log on failure so the next run names the exact broken line.
* ci(sandbox): export an immutable repo@digest reference for the smoke run
docker pull once on the runner platform, resolve RepoDigests[0], and pass
that immutable reference to the test via GITHUB_ENV — the recorded and
executed images can no longer diverge when the tag moves, and platform
selection is left to the daemon instead of jq over the manifest index.
* fix(sandbox): add DAC_OVERRIDE — the root nginx master writes gem-owned logs
The image's root nginx master opens /var/log/nginx/{access,error}.log,
which belong to the gem user, for the container's lifetime; without
CAP_DAC_OVERRIDE it dies with 'open() failed (13: Permission denied)' on
every start (FATAL under supervisord) and readiness never arrives. Four
capabilities now: CHOWN/SETUID/SETGID for the entrypoint handoff plus this
runtime log-write need.
* feat(mcp): map request-scoped secrets to HTTP/SSE headers
`user_auth` binds a credential to a configured DeerFlow user, so a caller
that picks the credential per request — a multi-tenant gateway, a per-run
API key, one shared MCP server fronting several environments — had to
register one MCP server entry per credential.
Add a declarative `mcpServers.<server>.headers_from_context` block mapping
HTTP header names to keys of the run request's `config.context.secrets`
carrier. A new built-in interceptor resolves the mapping on every tool call
and rewrites those headers, mirroring `user_scoped_auth`. The config file
stores names only, never a credential, so the Gateway returns the block
unmasked.
Registered after OAuth and `user_auth` in the interceptor chain: the later
interceptor runs closer to the transport, and the value chosen for this one
request is the most specific, so it wins. Fail-closed by default — a mapped
key missing from the request raises a `ToolException` naming only that key,
because falling back to the server's discovery credential would send one
tenant's call under another tenant's authority. `on_missing: "passthrough"`
opts out.
Durable background tasks are excluded: `McpTaskToolCaller` drives status and
cancel polls after the Agent run ends, where no run context exists, so the
fail-closed interceptor would deny every poll. Those calls keep using
server-level credentials, and a server declaring both `headers_from_context`
and `task_toolsets` now logs a warning.
Also corrects the custom-interceptor example in docs/MCP_SERVER.md (and the
matching claim in skills/AGENTS.md), which read request secrets from
`langgraph.config.get_config()["context"]`. That key is `None` inside a tool
call — the run context rides the LangGraph runtime, not the RunnableConfig
propagated to child runnables — so interceptors written from that example
never saw a value. The example now reads `request.runtime`, and
tests/test_mcp_context_headers.py pins LangGraph's runtime-injection rule by
driving a real langchain-mcp-adapters tool through a real graph with the
ambient-runtime fallback disabled.
Closes#5005
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): resolve credential headers case-insensitively, carry them on durable submit
Review follow-ups on `headers_from_context`.
HTTP field names are case-insensitive, but every dict on the path to the wire
is not: `build_server_params` copies the operator's static `headers` spelling
verbatim, and langchain-mcp-adapters merges interceptor overrides into the
connection with a plain `{**connection_headers, **override_headers}` splat. A
static `authorization` and an injected `Authorization` therefore both reached
httpx as separate field lines, and a server reading the field with a
single-value accessor got the static discovery credential — inverting the
documented `headers` < `oauth` < `user_auth` < `headers_from_context`
precedence and running a per-request call under the shared credential.
Normalizing inside the interceptor cannot fix that on its own: the adapter
builds the request with `headers=None`, so an interceptor never sees the
connection's static headers and cannot displace them however it spells its own
key. A new `mcp/headers.py::apply_header_overrides` therefore drops any key
differing only in case and emits the spelling the connection already uses.
Applied to `headers_from_context`, `user_auth`, the OAuth interceptor, the
OAuth discovery-header write, and the durable-task connection merge, which all
carried the same collision. `headers_from_context.headers` now also rejects one
header mapped under two spellings at config load, in both the harness model and
the Gateway mirror.
Durable submit now carries the mapped headers, as docs/MCP_SERVER.md already
promised. `McpTaskToolCaller` disabled the interceptor for the whole caller, but
that caller serves submit as well as the polls, and submit is awaited inline
inside the Agent's tool call — where the run's LangGraph runtime is still the
ambient contextvar, so no secret has to be threaded through `TaskSubmitRequest`
or reach durable storage. The caller builds one chain and keeps a second view of
it without the context-headers interceptor; `call_tool` takes
`request_scoped_headers`, set only by `OrdinaryMcpTaskDriver.submit`. Status and
cancel keep server-level credentials, so background polls still cannot fail
closed, and the startup warning now describes the half it actually covers.
`_merge_preserving_secrets` restores masked extras inside `headers_from_context`
instead of writing the `***` sentinel back over the stored value, matching the
treatment `user_auth` extras and server-level extras already get; extras a PUT
omits carry over as well, while the declared mapping still replaces verbatim so
a round trip can remove an entry. `extra="allow"` plus name-based sensitivity
detection means the usual casualty is a name-valued key such as `tokenHeader`,
not only a credential.
The existing override test seeded the static header onto `request.headers`,
which production never does, so it modelled a merge that really happens one
layer down; the new tests drive a real adapter tool through a real connection
and assert on the headers the session is opened with, and the durable-submit
test runs through a real tool node with no runtime patching.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): reject case-insensitive duplicate static header names
* fix(mcp): preserve omitted headers_from_context fields on partial updates
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ViewImageMiddleware injected the viewed-image message from before_model and
removed it again from after_model. before_model, model, and after_model are
separate graph nodes, so every view_image turn cost two extra nodes and two
state writes, and up to 20MB of base64 sat in two checkpoints for the duration
of the model call. A run interrupted in that window (user cancel, restart)
stranded the payload in history for good.
Inject from wrap_model_call instead, so the message lives only in
ModelRequest.messages and is never returned as a state update:
- before_model/after_model (and the async pair) are replaced by
wrap_model_call/awrap_model_call; _remove_image_context_messages and its
RemoveMessage bookkeeping go with them. The async hook keeps the existing
asyncio.to_thread offload for the file read and base64 encode.
- _should_inject_image_message gates on request.messages rather than state, so
the decision is made against what the model will actually see.
- _inject sweeps this middleware's own message out of the request before
rebuilding it. Dropping after_model also drops the cleanup it did on every
call, so without the sweep a payload stranded by an older interrupted run
would ride along in every later request for the life of the thread. Matching
requires both the reserved id prefix and the server-owned marker, and Gateway
strips that marker from client input, so a user message is never dropped.
Chain position is unchanged, and wrap_model_call nests first-registered
outermost, so TokenBudgetMiddleware still sees the image message and enforces
the input budget against it.
Checkpoint rows that already hold a stranded payload keep it on disk. It is
inert -- never sent to a provider, and strip_data_url_image_blocks keeps it off
the wire -- and reclaiming it would mean keeping the node this change removes.
tests/test_view_image_middleware.py is rewritten around the new hook (43
tests): sync/async at unit and graph level, the stranded sweep, and the
client-message protection. Docs: middleware chain entry 23, Vision Support, the
middleware-execution-flow hook matrix and diagrams, and the
strip_data_url_image_blocks docstring.
* Preserve Windows CLI compatibility for local sandbox commands
MSYS path conversion must remain disabled for DeerFlow virtual paths, but applying a blanket environment override to every POSIX command breaks host-native CLI shims on Windows. Limit MSYS argument-conversion exclusions to safe non-root virtual path prefixes, omit values that would broaden the exclusion pattern, and document the contract.
Constraint: Preserve the virtual-path protection introduced by #2765/#2766
Rejected: Disable MSYS conversion for every command | breaks Windows CLI shims
Rejected: Toggle blanket conversion only for commands containing virtual paths | host CLIs can receive virtual-path arguments and still need normal conversion for their own paths
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: Keep regression coverage for virtual-path arguments, root mounts, and host-native CLI launchers
Tested: test_local_sandbox_encoding.py (12 passed); related sandbox suite (197 passed, 8 skipped, 7 failures matching origin/main); ruff check; ruff format --check; git diff --check; direct LocalSandbox CLI and virtual-path smoke tests
Not-tested: Full offline suite completion; stopped at 6% after unrelated Windows and optional-runtime failures
Related: #2765
Related: #2766
* Keep MSYS regression tests portable across CI operating systems
The Windows-shell environment tests patched os.name to nt while mounting Windows-specific paths. On Linux and macOS, pathlib then attempted to construct WindowsPath during command resolution or output masking, so the backend merge gate failed before exercising the environment contract. Stub the exclusion boundary in execute-command tests and retain mapping-specific filtering coverage in the helper test.
Constraint: Backend unit tests run on Linux, while the behavior under test is Windows-only
Rejected: Skip the tests outside Windows | would remove CI coverage of the environment contract
Rejected: Patch pathlib internals | couples tests to implementation details and hides the platform boundary
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep OS-specific subprocess assertions independent from host-path resolution
Tested: test_local_sandbox_encoding.py (12 passed); ruff check; ruff format --check; git diff --check
Not-tested: Linux runner execution locally because Docker Desktop is unavailable and WSL cannot access this linked worktree
Related: #5003
Related: https://github.com/bytedance/deer-flow/pullrequestreview-5013380238
* fix(gateway): preserve exact history run attribution
* fix(gateway): make history migration authoritative
* docs(runtime): keep history contract within guidance budget
* fix(runtime): fence final run duration write
* Fix skill moderation parsing for Responses API content blocks
Normalize LangChain Responses API text blocks before parsing the security moderation decision, while preserving the existing fail-closed behavior for unavailable or invalid moderation results. Add regression coverage for mixed content blocks and document the compatibility boundary.
Constraint: Responses API AIMessage content is list-shaped while Chat Completions content is string-shaped
Rejected: Disable security scanning | would weaken the skill write safety boundary
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep moderation parsing provider-format tolerant without including reasoning or tool blocks in the decision payload
Tested: 27 security scanner tests; ruff check; ruff format check
Not-tested: Live moderation request against the configured external endpoint
* Reuse shared LLM response text normalization
Route skill moderation responses through the existing provider-format normalizer so only text and output_text blocks participate in JSON parsing. Strengthen regression coverage with reasoning and tool blocks that contain misleading text fields.\n\nConstraint: Responses API content is shared across multiple harness consumers\nRejected: Keep a private normalizer | duplicated provider-shape policy diverges and can reintroduce reasoning-block contamination\nConfidence: high\nScope-risk: narrow\nReversibility: clean\nDirective: Extend the shared normalizer when a new provider content shape is verified; do not add divergent local parsers\nTested: 118 related backend tests; regression test red against the previous parser; Ruff check and format check\nNot-tested: Live GitHub CLA status refresh
* Restore trusted external skill package loading
Skill discovery follows one-level package-directory symlinks, but activation path validation rejected the resolved external path. Restore that compatibility for configured custom-skill category roots while keeping file-level symlinks and deeper escapes blocked. Add regression coverage for local and user-scoped storage plus slash activation, and document the boundary.
Constraint: Existing skill discovery follows directory symlinks and operator-managed external packages must remain loadable
Rejected: Allow arbitrary resolved paths | would weaken the skill path trust boundary
Confidence: high
Scope-risk: moderate
Directive: Keep the final SKILL.md file symlink-free and preserve one-level category-root validation
Tested: 79 targeted skill storage, loader, slash activation, and user-scoped tests passed; Ruff check and format check passed; GitNexus staged change detection reported low risk
Not-tested: Real symlink activation on this Windows host lacks SeCreateSymbolicLinkPrivilege and is skipped
Related: Skill projection copies sources into sandbox-visible views
* Exercise real filesystem symlink boundaries in skill storage tests
Replace global Path.resolve/is_symlink mocks with real directory and file symlinks, preserving the Windows privilege skip. Add regression coverage for deeper custom-root escapes and symlinks under non-custom categories so the one-level allowance remains explicit.
Constraint: Symlink creation requires SeCreateSymbolicLinkPrivilege on some Windows runners
Rejected: Keep global path-method mocks | they validate the mock behavior rather than filesystem semantics
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep security-boundary tests on real filesystem primitives; skip only when the runner lacks symlink privilege
Tested: 76 targeted loader/storage/slash tests; Ruff check; Ruff format check
Not-tested: Windows symlink-enabled execution on this host
Related: #4936
* Pin the actual nested symlink escape boundary
Place the second symlink below a real custom package directory so the test reaches the one-level relative-parent guard instead of returning early on a non-symlink parent. Keep the public-category rejection coverage unchanged.
Constraint: The security boundary depends on both symlink depth and category root
Rejected: Link the outer package directory directly | the parent is not a symlink at validation time, so the depth guard is never evaluated
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep this regression tied to the exact relative_parent.parts depth check
Tested: Targeted storage, loader, and slash suites; GitNexus staged detection
Not-tested: Symlink-enabled execution on this Windows host
Related: #4936
* Make the nested symlink regression reach the depth guard
The test now validates the SKILL.md directly through the nested symlink, so the symlink is the immediate parent and the relative-parent depth check is executed.
Constraint: Windows test execution may skip when symlink privilege is unavailable
Rejected: Keep the extra nested path segment | it bypasses the symlink-depth guard through an early return
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Mutation tests must fail when the depth restriction is removed
Tested: Targeted test (skipped on this Windows host without symlink privilege); Ruff check and format check
Not-tested: Real symlink execution on Windows; Linux CI will exercise the case
Related: #4936
* Keep sandbox projections fresh for linked external skill packages
The storage layer intentionally accepts one-level custom package-directory symlinks, but projection freshness previously hashed only the link inode. Follow the permitted target tree during custom and legacy source-signature scans so edits to SKILL.md, scripts, references, or assets trigger a rebuild before sandbox use.
Constraint: Preserve the existing one-level custom/legacy symlink boundary and do not follow public, integration, nested, or file symlinks
Rejected: Invalidate projections only from /api/skills/reload | sandbox acquisition must also detect edits made directly in external targets
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep target-tree traversal limited to the storage paths that explicitly permit external package-directory links
Tested: 77 projection, user-scoped storage, and lifecycle tests passed; Ruff check and format check passed; git diff --check passed; GitNexus staged detection reported low risk
Not-tested: Real external symlink execution on this Windows host without SeCreateSymbolicLinkPrivilege; existing tests skip that platform limitation
Sandbox is an execution environment, not a named resource: multiple tools
(bash, read_file, write_file, glob, grep, ...) depend on it, all funneled
through ensure_sandbox_initialized / ensure_sandbox_initialized_async. Gate
the single acquisition entry point (single source of truth) instead of
maintaining a sandbox-tool-name set in middleware:
- authorize_sandbox_execution helper (authz/sandbox_authz.py) checks
authorize("sandbox", "execute", target="*") — a binary judgment
(can this role use the sandbox at all); RBAC allow:"*"/true permits,
allow:[]/false denies.
- lazy path: ensure_sandbox_initialized (+ async) calls the gate before
provider.acquire.
- eager path: SandboxMiddleware.before_agent / abefore_agent call the gate
before _acquire_sandbox.
- deny raises SandboxAuthorizationError (SandboxError subclass) which
propagates through tool execution as a friendly ToolMessage (RFC §9:
'not a crash').
- authorization.enabled: false is a no-op everywhere; provider errors
follow fail_closed (deny) / fail_open (allow).
12 tests in tests/test_sandbox_authorization.py cover disabled/allow/deny/
deny-via-bool/no-policy-unrestricted/provider-error-fail-closed/open/
internal-caller + ensure_sandbox_initialized deny (never acquires) and
allow (acquires) integration paths.
* fix(clarification): drop sibling tool calls before interrupt
- Rewrite the AIMessage in ClarificationMiddleware.after_model so a
parallel bash/write_file cannot run before the user answers
- langchain return_direct only inspects the last ToolMessage; siblings
both execute and can keep the agent loop alive
- Skip the rewrite when disable_clarification is set
- Prompt and tool docs: do not call other tools in the same turn
Fixes#4906
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(clarification): enhance sibling tool call handling in ClarificationMiddleware
- Update ClarificationMiddleware to ensure sibling tool calls are dropped when `ask_clarification` is invoked, preventing unintended execution before user input.
- Modify documentation to clarify that the `return_direct` router now inspects all client-side tool calls of the last AIMessage, ensuring proper routing behavior.
- Introduce a new integration test to validate that sibling tools do not execute when `ask_clarification` is present in the same turn.
This change addresses potential issues with tool execution order and improves the overall reliability of the middleware.
Fixes#4906
* fix(clarification): enhance tool call filtering in ClarificationMiddleware
- Update _filter_content_tool_use to handle Gemini-style function_call blocks by matching on name when no id is present, ensuring proper filtering of tool calls.
- Modify ClarificationMiddleware to maintain sibling tool call integrity by dropping unnecessary blocks, improving the clarity of the AIMessage content.
- Add a new test to validate the correct stripping of idless function call content blocks, ensuring that sibling tool calls do not execute prematurely.
This change improves the robustness of the middleware and addresses potential execution order issues.
Fixes#4906
* fix(clarification): drop siblings when ask_clarification is malformed
LangChain parks invalid args on invalid_tool_calls independently, so a
valid sibling would otherwise still execute before the user answers.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by
max_recursion_limit) so scheduled background runs can use a different
recursion limit than the web UI. The value is read at dispatch time, so
a YAML edit applies to the next scheduled run without a Gateway restart.
Also logs a warning when the resolver falls back to the default or
clamps the configured value.
* feat(harness): add deterministic tool receipts with model-visible ledger
Stamp an immutable per-call fact record (tool name, status, args/output
hashes, byte count, timestamp) onto every tool result via a new
ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN)
into the model context so subagent reports can cite executed actions.
- tool_receipt.py: receipt core (make/extract/render), newest-first
budget eviction, ids derived from the append-only message stream
- ToolReceiptMiddleware: stamps ToolMessages directly or inside
Command-wrapped results; hidden ledger injection mirrors
DurableContextMiddleware; sits between ToolProgress and
ToolErrorHandling with a build-time ordering guard
- config: new verification section (receipts on, judge off), config
version 32 -> 33 with example/helm/docs updates
* feat(harness): split receipt rendering from stamping; address PR review
Review fixes (PR #4659):
- output_sha256 now uses sort_keys=True for structured content, matching
the order-invariant args fingerprint
- stamping failures log at warning (silent ledger gaps would corrupt
citations); tool execution remains never blocked
- _insert_after_leading_system_messages extracted to shared public
message_utils.insert_after_leading_system_messages; both middlewares
depend on it instead of a private cross-module helper
- code comments in English
RFC #4651 revision-2 alignment:
- receipts_render_mode config ('always' | 'delegation_only'): subagent
chains always render the ledger (citations are produced there); the
lead chain renders only while processing subagent results, removing
the always-on token tax from ordinary turns
- receipts gain bounded args_preview/output_preview (<=200 chars, tail
for output) so later typed claim bindings (tests_passed) can anchor
to a specific recorded execution
* docs(harness): state receipt freshness caveat and vocabulary layering in module docstring
* merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews
- backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the
ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the
verification.* hot-reload mention into config/AGENTS.md
- config.example.yaml + helm values/README: config_version 33 -> 34 so existing
v33 configs get the outdated-config prompt (review: willem-bd)
- tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads
them; re-add with the Layer 2 claim-binding consumer (review: willem-bd)
* docs(harness): cover receipt id renumbering after compaction in module docstring
Positional display ids are stable only while history is append-only;
compaction drops ToolMessages and the survivors renumber, so Layer 2
citation verification must resolve [rN] against the ledger as of the
citing turn (review: willem-bd, doc-only).
* chore(config): bump config_version to 35
main reached 34 via #4780 without the verification section; publishing
the new schema at the same number would silently skip the outdated-config
prompt for configs synced from main in that window (review: willem-bd).
* fix(skills): restore errno import dropped upstream in #4830
upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed
the import, so the PR merge ref fails lint-backend (F821).
* fix(harness): harden tool receipts against forgery and turn-scope delegation_only
Address willem-bd's pre-merge review on #4659:
1. Untrusted receipt metadata: the gateway now strips the server-owned
deerflow_tool_receipt key from external input messages; stamping always
overwrites any tool-supplied value instead of preserving it; and
extract_tool_receipts validates persisted receipt shapes (required typed
fields, unknown keys ignored) so malformed entries are skipped instead of
crashing render or passing as runtime-stamped evidence.
2. delegation_only no longer sticks on: _should_render now scopes the
subagent_status scan to the current turn (messages after the latest
genuine user message), so an old completed delegation stops rendering the
ledger on later ordinary turns. The genuine-user predicate moves to
message_utils.is_genuine_user_message, shared with input sanitization.
* fix(harness): stamp receipts outside short-circuiting tool middlewares
Address willem-bd's review on #4659: ToolReceiptMiddleware was registered
inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which
can return a ToolMessage without invoking its handler — blocked calls
(e.g. a read-before-write-denied write_file) never got a receipt, silently
gapping the ledger on a default-enabled path. SandboxAudit additionally
rebuilds medium-risk results, dropping an inner stamp.
ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the
runtime tail. Normal results still carry deerflow_tool_meta (stamped by
ToolErrorHandling on the inner return path); short-circuit messages
self-stamp meta or fall back to message.status. The new invariant is
declared as ordering constraints in deerflow.extensions.ordering, with
composed-chain regression tests for a blocked write and a warn-rebuilt
bash result.
* feat(mcp): per-user credential injection for shared MCP servers
A single HTTP/SSE MCP server entry can now serve several users, each
authenticated to the remote service with their own credential. A server
opts in with a user_auth block mapping user ids to credential header
values ($ENV_VAR references supported):
"user_auth": {
"header": "Authorization",
"users": { "<user-id>": "$SERVICE_TOKEN_ALICE" }
}
The built-in user-scoped auth interceptor resolves the authenticated
runtime user on every tool call (request runtime -> ambient LangGraph
runtime -> auth config -> request-scoped user ContextVar) and rewrites
the configured header via request.override(), the same per-call
mechanism as the OAuth interceptor. It registers after OAuth in the
shared assembly so its per-user value wins the header when a server
declares both. The entry's static headers are used only for startup
tool discovery.
Fail-closed by default: an unmapped user - including the anonymous
default-user fallback - or a credential whose env reference resolved
empty gets an actionable ToolException instead of another user's
credential; on_missing: "passthrough" opts out per server. Combined
with the existing per-(user, thread) MCP session scoping this gives
credential isolation on shared servers.
Gateway API: user_auth.users values are masked in GET responses, and
PUT round-trips preserve stored credentials for masked values (same
contract as env/headers/oauth secrets); a masked value for a user id
not already stored is rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): address review — preserve stored user_auth sub-fields on partial PUT, warn-and-skip stdio, allow extras
Review findings from #4868:
1. A partial user_auth payload (e.g. {"enabled": false}) merged to
users={} and default on_missing, irreversibly wiping stored
credentials on PUT. The merge is now sub-field-aware via
model_fields_set — omitted sub-fields carry the stored values, an
explicitly sent users map still replaces (so full-round-trip removal
works), masked values still swap back for stored credentials.
2. user_auth on a stdio server was a silent no-op: rewritten headers go
to call meta, never a transport header, while deny errors still fired.
The interceptor builder now warns and skips non-sse/http servers,
matching the tool_call_timeout transport-mismatch convention.
3. McpUserScopedAuthConfigResponse now allows extra keys like the
harness-side model, and extras survive masking and merge, matching
the server-level model_extra handling.
Adds four regression tests (partial-PUT preservation, explicit-map
replacement, extras round-trip, stdio warn-and-skip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): reject blank user_auth.header at the gateway
A blank header passed the gateway response model, was persisted, then
failed the harness-side ExtensionsConfig validator on reload — the PUT
returned 500 after the write and every later config load/startup failed
until the file was hand-edited. Mirror the harness non-blank validator
on McpUserScopedAuthConfigResponse so the PUT fails with 422 before
anything is written.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format extensions_config.py
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): harden the trust chain and masking around user-scoped credential selection
Address review round 4:
- Scrub client-supplied user_id from run context/configurable for
external callers in inject_authenticated_user_context, before every
early return, and restamp only from request.state.user. Now that
user_id selects which user's credential user-scoped MCP auth injects,
a forged value must not survive any future path that skips the
restamp. Internal callers (IM channels, scheduler) keep supplying
end-user identity as before (PR #3294 contract). Regression tests pin
both the scrub and that a forged body.context.user_id can never
resolve as another user through merge + inject ordering.
- Include the caller's own resolved user id in the fail-closed deny
message so operators can copy the exact users key (it differs by
deployment path), and document the key formats in the mcp.mdx doc.
- Mask sensitive extra keys inside user_auth on GET like server-level
extras, and swap masked sentinels back for stored values on PUT via
_merge_extra_value_preserving_masked.
- Extract the interceptor wrap loop into compose_tool_interceptors and
pin the security property functionally: an OAuth interceptor that
actually sets Authorization loses the final header to the per-user
credential through the same composition the session-pool path uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(docker): let the Gateway write extensions_config.json in production
AGENTS.md states extensions_config.json may be edited at runtime through the
Gateway API, and the Gateway implements that for the MCP enable switch,
PUT/PATCH /api/mcp/config and the skill update route. Two properties of the
production compose stack made every one of those writes fail:
- the file was mounted read-only, and
- Docker mounts it as its own mount point, so the temp-file-plus-rename in
atomic_write_extensions_config hit EBUSY. Linux refuses rename() over a
mount point whether or not the mount is writable, so making the mount
read-write alone is not enough.
Mount it read-write and fall back to an in-place overwrite on EBUSY only.
The fallback is deliberately non-atomic and says so in a warning; it is
reached only where the atomic route cannot work, and any other errno still
propagates. config.yaml stays read-only: no API writes it.
docker-compose-dev.yaml mounts the whole project directory, so the
destination is an ordinary file there and this never surfaced in development.
* test(docker): parse mount options instead of matching a :ro suffix
Docker's short-syntax options segment is comma-separated, so a read-only
mount can legally be spelled ":ro,z" or ":z,ro" — common with SELinux
relabelling. Matching the raw string for a ":ro" suffix reads those as
writable, which silently defeats the guard: the writability assertion would
pass on a read-only mount, and the config.yaml assertion would fail on a
correctly read-only one.
Parse the options segment and test membership instead, and cover the parser
with the spellings that broke the suffix check.
* fix(config): harden mutable extensions config
* feat(extensions): let an out-of-tree extension observe what the agent did
DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:
* The middleware chain injects and rewrites a lot of context — date
reminders, recalled memory, compaction summaries, durable-context data,
image payloads, activated skill bodies. Downstream, none of it is
attributable: at the model-call boundary an injected HumanMessage is
indistinguishable from the user's own, and anything wanting to tell them
apart has to pattern-match prompt wording, which breaks on the next copy
edit.
* Two runs of "the same agent" are only comparable if the chain enforced the
same limits, prompts and thresholds. Recovering that from outside means
reading private attributes and guessing which of them change behaviour — a
guess that rots silently as middlewares gain fields.
* The lead-agent factory resolves a model after runtime overrides, renders a
prompt, filters tools through authorization and composes a stack, all
inside one synchronous call, and none of it survives: a middleware sees its
neighbours but not the prompt, the run worker sees a graph but not what
went into it.
* Summarization is destructive by design. N messages leave the context and
one summary enters it; afterwards only the summary exists, so "which
messages became this?" is not reconstructible.
This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.
Message provenance
Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
messages they inject or rewrite. Stamping is unconditional — a fact whose
presence depends on whether an observer is installed is not a fact — and the
keys are server-owned, so provenance cannot be forged from a request.
Middleware self-description
Twelve middlewares declare their own behaviour-affecting parameters through
a duck-typed `release_policy_parameters()`. Long text is hashed rather than
embedded: a declaration is an identity, not a copy of the prompt.
Agent assembly descriptor
`assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
answers "did anything about this agent change between these two runs?".
`make_lead_agent()` keeps its graph-only signature — it is the LangGraph
Server ABI declared in langgraph.json. Tools and skills are sorted before
hashing because their assembly order is incidental; middlewares are not,
because stack order decides what wraps what. Host build identity is reported
but excluded from the fingerprint, so a redeploy does not invalidate every
agent's identity.
Context compaction observation
Summarization emits the content hashes of the messages it is about to remove
joined to the summary that replaced them. Content is the only identity
available at that seam: the summary does not become a message, and what later
projects it into a request renders it bounded and escaped rather than
verbatim.
Neutral policy, transform and MCP-source facts
Guardrail decisions are published to runtime context under a `__`-prefixed
key; result-rewriting middlewares append a declared, ordered transform trail;
MCP tools carry their credential-free logical origin.
Extension route identity
Contributed routes are session-authenticated and cannot opt out, but
"logged in" and "administrator" are different questions. Extensions get a
neutral projection of the caller rather than the host's auth context, and
`require_admin` fails closed when identity cannot be determined.
Extension-owned tables
An extension that persists data owns its own MetaData and migration chain, so
its tables are absent from Base.metadata and `alembic revision --autogenerate`
proposes dropping them. Extensions declare a table prefix, which is rejected
at registration if it would shadow a host table.
The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.
uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.
* fix(backend): sort gateway service imports
* fix(mcp): keep grant_type authoritative over extra_token_params
_fetch_token built the token request body as
{"grant_type": oauth.grant_type, **oauth.extra_token_params}, so an
operator-supplied extra_token_params that happened to contain
"grant_type" silently overwrote the value sent to the token endpoint
while the branch logic below still keyed off oauth.grant_type — the
sent grant_type and the chosen auth flow would disagree, and the
provider would almost certainly reject the request.
Spread extra_token_params first and set grant_type (and the other
reserved fields, which were already set after the spread) afterward, so
operator-supplied params can populate arbitrary extra fields but never
override the reserved ones the flow depends on.
* test(mcp): cover extra OAuth token parameters
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(e2b): preserve trailing whitespace in filenames and survive mtime overflow
_sync_outputs_to_host iterated the NUL-delimited find output with
entry.strip() on each record. NUL already guarantees record boundaries,
so the strip is redundant and harmful: a filename that legitimately ends
in whitespace (e.g. "report ") had its trailing space trimmed, pointing
host_path at the wrong file and recording a manifest key that can never
match — the file was re-downloaded on every release.
The same host-write block wrapped only os.utime in the outer
except OSError, but os.utime raises OverflowError (not an OSError) when
the ns value is out of range (a far-future remote mtime, e.g.
`touch -d '99999 years'`). That escaped the loop, skipping the manifest
write and forcing a full re-download next release. Wrap os.utime in its
own (OSError, OverflowError) so only the timestamp restoration is
dropped; the file is still written and the manifest still updated.
* test(e2b): rely on monkeypatch cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): exclude internal temp files from workspace changes
* fix(mcp): address review — shared tmp subdir constant, any-depth docs, nested test
- Export MCP_TMP_SUBDIR from constants.py and import it in both stdio
launch paths (tools.py, task_tool_caller.py) so the "/tmp" suffix is
composed once.
- Document that the .mcp exclusion matches by directory name at any
depth (like .git/node_modules) in README.md and mcp/AGENTS.md —
subagent work dirs below the workspace root get their own .mcp/tmp.
- Pin the any-depth semantic in test_workspace_changes.py with a nested
workspace/project/.mcp assertion.
* docs(mcp): correct any-depth exclusion rationale; re-home tmp pinning comment
The previous commit justified the any-depth `.mcp` exclusion with subagent
work dirs sitting below the workspace root — a mechanism that doesn't
exist: subagents share the parent's thread_id and both stdio launch paths
resolve sandbox_work_dir(thread_id), so `.mcp/tmp` is always pinned at the
workspace root. Reword mcp/AGENTS.md and the test comment to the real
justification (consistency with the other reserved dir names, robustness
against a server creating a relative `.mcp` from another cwd).
Also move the orphaned "pinning the process temp dir" rationale from
tools.py to constants.py next to MCP_TMP_SUBDIR, where both importers see it.
* fix(sandbox): bound aggregate E2B mount upload work
* fix(sandbox): preserve mount guards on upload failure
* fix(sandbox): cover mount preflight with deadline
* refactor(sandbox): clarify mount deadline checks
* refactor(sanbox): deduplicate mount deadline reason
* fix(sandbox): evaluate mount deadline reason lazily
* feat: integrate MiniMax Code as an ACP agent
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(skills): fail closed on drifted projection namespace on all platforms
* test(skills): add regression test simulating swallowed unlink on drifted namespace
