* fix(docker): create compose env files and keep Windows compose paths relative
Windows Docker reports a generic file-not-found when env_file targets are missing, or when compose paths are doubled. Make docker-start copy .env examples and invoke compose with filenames relative to docker/.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): make dev compose env files optional and repair test fixture
Address review feedback on #4956.
[P1] prepare_compose_env aborted before the mocked COMPOSE_CMD in
test_compose_commands_set_deer_flow_root_before_compose, because the
temp root had no compose file or .env examples. Seed them in the
fixture so the preflight reaches the mock.
[P2] .env is gitignored, so a fresh clone has none and a direct
`docker compose -f docker/docker-compose-dev.yaml up --build` aborts on
Windows before scripts/docker.sh can help. Mark the dev env_file entries
`required: false` so a missing .env is not fatal, and document that
direct Compose must be run from the repository root.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): declare Compose 2.24 floor and keep non-start commands read-only
Address the second review round on #4956.
- Document Compose >= 2.24 (CONTRIBUTING, README, compose header) and fail
early from make docker-start with an actionable message; probe both
`docker compose` and the hyphenated `docker-compose` binary.
- Document DEER_FLOW_ROOT for direct Compose callers (bash + PowerShell);
leave the variable without a $PWD fallback because PowerShell/cmd do not
export it.
- Split prepare_compose_env: compose_preflight is shared and read-only;
ensure_env_files runs only from start.
- Expand tests for version boundaries, hyphenated fallback, env-file
creation, and read-only stop/logs/restart behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): reuse the probed Compose binary for wrapper operations
The version probe could accept a standalone docker-compose install while
COMPOSE_CMD stayed hardcoded to `docker compose`, so preflight passed and
start/logs/stop/restart then failed. Keep the selected executable in
COMPOSE_BIN (array), refresh COMPOSE_CMD from it in the current shell, and
extend the fallback test through an actual stop invocation.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix skill moderation parsing for Responses API content blocks
Normalize LangChain Responses API text blocks before parsing the security moderation decision, while preserving the existing fail-closed behavior for unavailable or invalid moderation results. Add regression coverage for mixed content blocks and document the compatibility boundary.
Constraint: Responses API AIMessage content is list-shaped while Chat Completions content is string-shaped
Rejected: Disable security scanning | would weaken the skill write safety boundary
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep moderation parsing provider-format tolerant without including reasoning or tool blocks in the decision payload
Tested: 27 security scanner tests; ruff check; ruff format check
Not-tested: Live moderation request against the configured external endpoint
* Reuse shared LLM response text normalization
Route skill moderation responses through the existing provider-format normalizer so only text and output_text blocks participate in JSON parsing. Strengthen regression coverage with reasoning and tool blocks that contain misleading text fields.\n\nConstraint: Responses API content is shared across multiple harness consumers\nRejected: Keep a private normalizer | duplicated provider-shape policy diverges and can reintroduce reasoning-block contamination\nConfidence: high\nScope-risk: narrow\nReversibility: clean\nDirective: Extend the shared normalizer when a new provider content shape is verified; do not add divergent local parsers\nTested: 118 related backend tests; regression test red against the previous parser; Ruff check and format check\nNot-tested: Live GitHub CLA status refresh
* Restore trusted external skill package loading
Skill discovery follows one-level package-directory symlinks, but activation path validation rejected the resolved external path. Restore that compatibility for configured custom-skill category roots while keeping file-level symlinks and deeper escapes blocked. Add regression coverage for local and user-scoped storage plus slash activation, and document the boundary.
Constraint: Existing skill discovery follows directory symlinks and operator-managed external packages must remain loadable
Rejected: Allow arbitrary resolved paths | would weaken the skill path trust boundary
Confidence: high
Scope-risk: moderate
Directive: Keep the final SKILL.md file symlink-free and preserve one-level category-root validation
Tested: 79 targeted skill storage, loader, slash activation, and user-scoped tests passed; Ruff check and format check passed; GitNexus staged change detection reported low risk
Not-tested: Real symlink activation on this Windows host lacks SeCreateSymbolicLinkPrivilege and is skipped
Related: Skill projection copies sources into sandbox-visible views
* Exercise real filesystem symlink boundaries in skill storage tests
Replace global Path.resolve/is_symlink mocks with real directory and file symlinks, preserving the Windows privilege skip. Add regression coverage for deeper custom-root escapes and symlinks under non-custom categories so the one-level allowance remains explicit.
Constraint: Symlink creation requires SeCreateSymbolicLinkPrivilege on some Windows runners
Rejected: Keep global path-method mocks | they validate the mock behavior rather than filesystem semantics
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep security-boundary tests on real filesystem primitives; skip only when the runner lacks symlink privilege
Tested: 76 targeted loader/storage/slash tests; Ruff check; Ruff format check
Not-tested: Windows symlink-enabled execution on this host
Related: #4936
* Pin the actual nested symlink escape boundary
Place the second symlink below a real custom package directory so the test reaches the one-level relative-parent guard instead of returning early on a non-symlink parent. Keep the public-category rejection coverage unchanged.
Constraint: The security boundary depends on both symlink depth and category root
Rejected: Link the outer package directory directly | the parent is not a symlink at validation time, so the depth guard is never evaluated
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep this regression tied to the exact relative_parent.parts depth check
Tested: Targeted storage, loader, and slash suites; GitNexus staged detection
Not-tested: Symlink-enabled execution on this Windows host
Related: #4936
* Make the nested symlink regression reach the depth guard
The test now validates the SKILL.md directly through the nested symlink, so the symlink is the immediate parent and the relative-parent depth check is executed.
Constraint: Windows test execution may skip when symlink privilege is unavailable
Rejected: Keep the extra nested path segment | it bypasses the symlink-depth guard through an early return
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Mutation tests must fail when the depth restriction is removed
Tested: Targeted test (skipped on this Windows host without symlink privilege); Ruff check and format check
Not-tested: Real symlink execution on Windows; Linux CI will exercise the case
Related: #4936
* Keep sandbox projections fresh for linked external skill packages
The storage layer intentionally accepts one-level custom package-directory symlinks, but projection freshness previously hashed only the link inode. Follow the permitted target tree during custom and legacy source-signature scans so edits to SKILL.md, scripts, references, or assets trigger a rebuild before sandbox use.
Constraint: Preserve the existing one-level custom/legacy symlink boundary and do not follow public, integration, nested, or file symlinks
Rejected: Invalidate projections only from /api/skills/reload | sandbox acquisition must also detect edits made directly in external targets
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep target-tree traversal limited to the storage paths that explicitly permit external package-directory links
Tested: 77 projection, user-scoped storage, and lifecycle tests passed; Ruff check and format check passed; git diff --check passed; GitNexus staged detection reported low risk
Not-tested: Real external symlink execution on this Windows host without SeCreateSymbolicLinkPrivilege; existing tests skip that platform limitation
Sandbox is an execution environment, not a named resource: multiple tools
(bash, read_file, write_file, glob, grep, ...) depend on it, all funneled
through ensure_sandbox_initialized / ensure_sandbox_initialized_async. Gate
the single acquisition entry point (single source of truth) instead of
maintaining a sandbox-tool-name set in middleware:
- authorize_sandbox_execution helper (authz/sandbox_authz.py) checks
authorize("sandbox", "execute", target="*") — a binary judgment
(can this role use the sandbox at all); RBAC allow:"*"/true permits,
allow:[]/false denies.
- lazy path: ensure_sandbox_initialized (+ async) calls the gate before
provider.acquire.
- eager path: SandboxMiddleware.before_agent / abefore_agent call the gate
before _acquire_sandbox.
- deny raises SandboxAuthorizationError (SandboxError subclass) which
propagates through tool execution as a friendly ToolMessage (RFC §9:
'not a crash').
- authorization.enabled: false is a no-op everywhere; provider errors
follow fail_closed (deny) / fail_open (allow).
12 tests in tests/test_sandbox_authorization.py cover disabled/allow/deny/
deny-via-bool/no-policy-unrestricted/provider-error-fail-closed/open/
internal-caller + ensure_sandbox_initialized deny (never acquires) and
allow (acquires) integration paths.
Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by
max_recursion_limit) so scheduled background runs can use a different
recursion limit than the web UI. The value is read at dispatch time, so
a YAML edit applies to the next scheduled run without a Gateway restart.
Also logs a warning when the resolver falls back to the default or
clamps the configured value.
* feat(harness): add deterministic tool receipts with model-visible ledger
Stamp an immutable per-call fact record (tool name, status, args/output
hashes, byte count, timestamp) onto every tool result via a new
ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN)
into the model context so subagent reports can cite executed actions.
- tool_receipt.py: receipt core (make/extract/render), newest-first
budget eviction, ids derived from the append-only message stream
- ToolReceiptMiddleware: stamps ToolMessages directly or inside
Command-wrapped results; hidden ledger injection mirrors
DurableContextMiddleware; sits between ToolProgress and
ToolErrorHandling with a build-time ordering guard
- config: new verification section (receipts on, judge off), config
version 32 -> 33 with example/helm/docs updates
* feat(harness): split receipt rendering from stamping; address PR review
Review fixes (PR #4659):
- output_sha256 now uses sort_keys=True for structured content, matching
the order-invariant args fingerprint
- stamping failures log at warning (silent ledger gaps would corrupt
citations); tool execution remains never blocked
- _insert_after_leading_system_messages extracted to shared public
message_utils.insert_after_leading_system_messages; both middlewares
depend on it instead of a private cross-module helper
- code comments in English
RFC #4651 revision-2 alignment:
- receipts_render_mode config ('always' | 'delegation_only'): subagent
chains always render the ledger (citations are produced there); the
lead chain renders only while processing subagent results, removing
the always-on token tax from ordinary turns
- receipts gain bounded args_preview/output_preview (<=200 chars, tail
for output) so later typed claim bindings (tests_passed) can anchor
to a specific recorded execution
* docs(harness): state receipt freshness caveat and vocabulary layering in module docstring
* merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews
- backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the
ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the
verification.* hot-reload mention into config/AGENTS.md
- config.example.yaml + helm values/README: config_version 33 -> 34 so existing
v33 configs get the outdated-config prompt (review: willem-bd)
- tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads
them; re-add with the Layer 2 claim-binding consumer (review: willem-bd)
* docs(harness): cover receipt id renumbering after compaction in module docstring
Positional display ids are stable only while history is append-only;
compaction drops ToolMessages and the survivors renumber, so Layer 2
citation verification must resolve [rN] against the ledger as of the
citing turn (review: willem-bd, doc-only).
* chore(config): bump config_version to 35
main reached 34 via #4780 without the verification section; publishing
the new schema at the same number would silently skip the outdated-config
prompt for configs synced from main in that window (review: willem-bd).
* fix(skills): restore errno import dropped upstream in #4830
upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed
the import, so the PR merge ref fails lint-backend (F821).
* fix(harness): harden tool receipts against forgery and turn-scope delegation_only
Address willem-bd's pre-merge review on #4659:
1. Untrusted receipt metadata: the gateway now strips the server-owned
deerflow_tool_receipt key from external input messages; stamping always
overwrites any tool-supplied value instead of preserving it; and
extract_tool_receipts validates persisted receipt shapes (required typed
fields, unknown keys ignored) so malformed entries are skipped instead of
crashing render or passing as runtime-stamped evidence.
2. delegation_only no longer sticks on: _should_render now scopes the
subagent_status scan to the current turn (messages after the latest
genuine user message), so an old completed delegation stops rendering the
ledger on later ordinary turns. The genuine-user predicate moves to
message_utils.is_genuine_user_message, shared with input sanitization.
* fix(harness): stamp receipts outside short-circuiting tool middlewares
Address willem-bd's review on #4659: ToolReceiptMiddleware was registered
inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which
can return a ToolMessage without invoking its handler — blocked calls
(e.g. a read-before-write-denied write_file) never got a receipt, silently
gapping the ledger on a default-enabled path. SandboxAudit additionally
rebuilds medium-risk results, dropping an inner stamp.
ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the
runtime tail. Normal results still carry deerflow_tool_meta (stamped by
ToolErrorHandling on the inner return path); short-circuit messages
self-stamp meta or fall back to message.status. The new invariant is
declared as ordering constraints in deerflow.extensions.ordering, with
composed-chain regression tests for a blocked write and a warn-rebuilt
bash result.
* feat(extensions): let an out-of-tree extension observe what the agent did
DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:
* The middleware chain injects and rewrites a lot of context — date
reminders, recalled memory, compaction summaries, durable-context data,
image payloads, activated skill bodies. Downstream, none of it is
attributable: at the model-call boundary an injected HumanMessage is
indistinguishable from the user's own, and anything wanting to tell them
apart has to pattern-match prompt wording, which breaks on the next copy
edit.
* Two runs of "the same agent" are only comparable if the chain enforced the
same limits, prompts and thresholds. Recovering that from outside means
reading private attributes and guessing which of them change behaviour — a
guess that rots silently as middlewares gain fields.
* The lead-agent factory resolves a model after runtime overrides, renders a
prompt, filters tools through authorization and composes a stack, all
inside one synchronous call, and none of it survives: a middleware sees its
neighbours but not the prompt, the run worker sees a graph but not what
went into it.
* Summarization is destructive by design. N messages leave the context and
one summary enters it; afterwards only the summary exists, so "which
messages became this?" is not reconstructible.
This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.
Message provenance
Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
messages they inject or rewrite. Stamping is unconditional — a fact whose
presence depends on whether an observer is installed is not a fact — and the
keys are server-owned, so provenance cannot be forged from a request.
Middleware self-description
Twelve middlewares declare their own behaviour-affecting parameters through
a duck-typed `release_policy_parameters()`. Long text is hashed rather than
embedded: a declaration is an identity, not a copy of the prompt.
Agent assembly descriptor
`assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
answers "did anything about this agent change between these two runs?".
`make_lead_agent()` keeps its graph-only signature — it is the LangGraph
Server ABI declared in langgraph.json. Tools and skills are sorted before
hashing because their assembly order is incidental; middlewares are not,
because stack order decides what wraps what. Host build identity is reported
but excluded from the fingerprint, so a redeploy does not invalidate every
agent's identity.
Context compaction observation
Summarization emits the content hashes of the messages it is about to remove
joined to the summary that replaced them. Content is the only identity
available at that seam: the summary does not become a message, and what later
projects it into a request renders it bounded and escaped rather than
verbatim.
Neutral policy, transform and MCP-source facts
Guardrail decisions are published to runtime context under a `__`-prefixed
key; result-rewriting middlewares append a declared, ordered transform trail;
MCP tools carry their credential-free logical origin.
Extension route identity
Contributed routes are session-authenticated and cannot opt out, but
"logged in" and "administrator" are different questions. Extensions get a
neutral projection of the caller rather than the host's auth context, and
`require_admin` fails closed when identity cannot be determined.
Extension-owned tables
An extension that persists data owns its own MetaData and migration chain, so
its tables are absent from Base.metadata and `alembic revision --autogenerate`
proposes dropping them. Extensions declare a table prefix, which is rejected
at registration if it would shadow a host table.
The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.
uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.
* fix(backend): sort gateway service imports
* fix(mcp): exclude internal temp files from workspace changes
* fix(mcp): address review — shared tmp subdir constant, any-depth docs, nested test
- Export MCP_TMP_SUBDIR from constants.py and import it in both stdio
launch paths (tools.py, task_tool_caller.py) so the "/tmp" suffix is
composed once.
- Document that the .mcp exclusion matches by directory name at any
depth (like .git/node_modules) in README.md and mcp/AGENTS.md —
subagent work dirs below the workspace root get their own .mcp/tmp.
- Pin the any-depth semantic in test_workspace_changes.py with a nested
workspace/project/.mcp assertion.
* docs(mcp): correct any-depth exclusion rationale; re-home tmp pinning comment
The previous commit justified the any-depth `.mcp` exclusion with subagent
work dirs sitting below the workspace root — a mechanism that doesn't
exist: subagents share the parent's thread_id and both stdio launch paths
resolve sandbox_work_dir(thread_id), so `.mcp/tmp` is always pinned at the
workspace root. Reword mcp/AGENTS.md and the test comment to the real
justification (consistency with the other reserved dir names, robustness
against a server creating a relative `.mcp` from another cwd).
Also move the orphaned "pinning the process temp dir" rationale from
tools.py to constants.py next to MCP_TMP_SUBDIR, where both importers see it.
The Star History charts in the README files no longer render because the underlying chart service relies on GitHub stargazer data that is currently restricted. This switches the charts to a working alternative that needs no API token, updating the English, Simplified Chinese, Japanese, French, and Russian READMEs at the same time.
Co-authored-by: OctoBored <212877535+OctoBored@users.noreply.github.com>
* fix(sandbox): bound aggregate E2B mount upload work
* fix(sandbox): preserve mount guards on upload failure
* fix(sandbox): cover mount preflight with deadline
* refactor(sandbox): clarify mount deadline checks
* refactor(sanbox): deduplicate mount deadline reason
* fix(sandbox): evaluate mount deadline reason lazily
* feat: integrate MiniMax Code as an ACP agent
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(mcp): add durable task runtime foundation
* fix(chart): sync embedded config version
* fix(mcp): isolate task polls during shutdown
* feat(mcp): track consecutive poll errors on mcp_tasks
poll_attempt_count grows on every claim (successful polls included), so it
cannot drive a failure backoff without misjudging normal long tasks. Add
consecutive_poll_error_count: incremented when a claim is released after a
poll error, reset to zero by any applied snapshot. The backoff/terminal
policy that consumes it lands with the first concrete driver.
* fix(mcp): harden durable task lifecycle
* feat(mcp): add ordinary durable task driver
* test(mcp): address durable task review feedback
* fix(mcp): preserve submit tool descriptions
* fix(mcp): bound remote task calls
* fix(mcp): bound persisted task payloads
* fix(mcp): preserve task tool error details
* fix(mcp): enforce durable task boundaries
* test(mcp): cover task config snapshot lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The Honcho backend landed in #4730 without user-facing docs: the main
README's Long-Term Memory section covers the other opt-in backends
(mem0, openviking) but never mentions honcho, and unlike mem0 the
backend shipped no guide README.
- Add backends/honcho/README.md mirroring the mem0 guide structure:
configuration (with the plain-HTTP api_key guard), workspace-per-user
isolation and fail-closed identity, recall/search behavior per mode,
limitations (no fact CRUD -> gateway 501, no DeerMem migration), and
async/failure-policy semantics.
- Add a short honcho paragraph + guide link to the README Long-Term
Memory section, alongside the existing mem0 paragraph.
* feat(extensions): add gateway services and routers
* feat(extensions): add standalone reference extension
* fix(extensions): harden contributed gateway routes
* docs(extensions): document gateway contribution points
* feat(extensions): add operator CLI for packaged extension management
Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.
Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.
Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.
Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.
Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.
Co-authored-by: Codex <codex@openai.com>
* fix(extensions): stop the managed plugins rewrite from destroying config
Two data-safety defects in the managed `plugins:` block writer.
The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.
The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.
Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.
* fix(extensions): harden the manager transaction and correct its docs
Follow-up hardening on the extension package manager.
Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
`UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
the interpreter that the entry-point probe then imports and calls, and every
later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
validation the HTTPS-only source rule depends on. Neither is an index, proxy,
cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
`ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
on case transitions, so only the separated spellings were caught. Short
generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
the package's build backend, so a config the manager could never write to must
fail before that code runs rather than afterwards through rollback.
Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
runs without `--locked` when the checkout had no lock, so uv writes one while
resolving; if that sync then failed, the restore was skipped and the operator
kept a lock file they never had. A failing recovery sync now also reports the
original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
dependency resolve invites a second interrupt that escapes the handler and
strands the checkout mid-transaction; the declarations are already restored
and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
builder is a different machine, but unlike an environment-driven wheelhouse
resolution this is a source the operator typed deliberately, so it is reported
rather than rolled back. Private-network indexes are untouched: a builder on
that network can reach them.
Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.
* test(gateway): pin the request-path projection agreement
`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.
Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:
- projection edge cases, including the segment-boundary guard that keeps
root_path="/api" from slicing "/apifoo/models" into a string the router
would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
mounted under the "/health" public prefix must still 401, and a POST
mounted under "/api/webhooks" must still require a CSRF token
Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).
Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.
* ci: pin uv to the version production ships
ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.
backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.
Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.
Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.
* fix(gateway): state the extension route auth limit and abort a failed dev sync
Two scoped review follow-ups.
README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.
docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.
Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.
* fix(extensions): point Git SSH shorthand at the HTTPS correction
Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.
The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.
* docs: keep the root extension summary within its new budget
#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.
---------
Co-authored-by: Codex <codex@openai.com>
* feat(extensions): observe task lifecycle and system model calls
PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.
Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).
`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.
Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.
Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.
System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.
The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.
Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.
Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.
* fix(extensions): decide notification fail-open by origin, observe cancellation
`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.
`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.
DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.
* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown
Review response on #4684:
- The memory observation bridge caught BaseException, which would swallow
a teardown signal raised while dispatching; it now catches Exception,
matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
hook-failure path and logged an asyncio-internal traceback; it now logs
a warning like the pre-hook budget skip, while a TimeoutError a
contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
so log lines said "task goal/title/..."; they now carry the task
scope id alongside the kind.
* feat(mcp): add durable task runtime foundation
* fix(chart): sync embedded config version
* fix(mcp): isolate task polls during shutdown
* feat(mcp): track consecutive poll errors on mcp_tasks
poll_attempt_count grows on every claim (successful polls included), so it
cannot drive a failure backoff without misjudging normal long tasks. Add
consecutive_poll_error_count: incremented when a claim is released after a
poll error, reset to zero by any applied snapshot. The backoff/terminal
policy that consumes it lands with the first concrete driver.
* fix(mcp): harden durable task lifecycle
* fix(mcp): preserve tracked task on dedup conflict
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add Buzz (Nostr) channel connector
Adds a Buzz (https://github.com/block/buzz) channel so DeerFlow can join a
Nostr-relay workspace as a member: it answers @mentions in channels, replies
to DMs, and streams answers by editing one message in place.
* app/channels/buzz_nostr.py — pure NIP-01 helpers: canonical event ids,
BIP-340 signing/verification, chat/edit/auth builders, relay frames.
* app/channels/buzz.py — BuzzChannel: one NIP-42-authenticated websocket,
channel discovery (kind 39000) with one subscription per channel, live
membership tracking (44100/44101), per-channel replay watermarks, and
replies posted once then edited in place (kind 40003).
* app/channels/buzz_run_policy.py — same-thread serialization, mirroring
the Feishu precedent.
Inbound is gated in order: signature verification, self-drop, /connect
bind-and-return, pubkey allowlist, then mention / DM / mention-free /
thread-follow. Off by default; needs the new optional `buzz` extra
(coincurve, lazily imported), which detect_uv_extras resolves from
channels.buzz.enabled the same way it already handles channels.discord.
Two relay behaviours drove the design and are worth knowing when reviewing:
a global {"kinds":[9]} subscription receives nothing from buzz-relay and a
multi-value "#h" filter receives nothing either, so one REQ per channel is
required; and a single global `since` cursor skips quiet channels, so
watermarks are per channel.
Signed-off-by: Ajay R <ajayr@formbuddy.com>
* fix(channels): only publish assistant messages from the IM stream
`_accumulate_stream_text` decided what streamed `messages-tuple` payloads
become displayable text by rejecting ONLY payloads whose `type` contained
"tool", so it published everything else. DeerFlow writes hidden model
context into the messages channel as ordinary messages -- memory recall and
the rewritten user turn as hidden HumanMessages (DynamicContextMiddleware),
the `<durable_context_data>` block as another (DurableContextMiddleware) --
and LangGraph fans those state writes out on the messages stream, so they
reached every streaming IM channel as the assistant's reply.
Proved live on a Buzz relay: the connector published a `<memory>` fact block
and, in another run, a verbatim echo of the user's own inbound message.
Affects Feishu, Telegram, WeCom and Buzz; worst on Buzz, where each update
is an immutable public Nostr event that a corrective edit cannot unpublish.
Invert the filter to an allowlist of assistant message types. Two new pure
helpers keep it testable:
- `_stream_payload_type` resolves the type from both shapes the function
already handles: the `model_dump()` shape the gateway emits, and
LangChain's `to_json()` constructor shape whose own `type` is the literal
"constructor" and whose class name is the tail of the `id` path.
- `_is_assistant_stream_type` matches "ai"/"assistant" by PREFIX, not
substring -- ordinary words contain "ai" ("chain", "domain"), and a
substring test would admit a foreign type name by accident.
The bare-`str` branch is removed: an untyped payload cannot be attributed to
the assistant, nothing in DeerFlow produces one (serialize_messages_tuple
always emits `[message_dict, metadata]`), and a runtime that emitted raw text
deltas would emit hidden context the same way. Per-message-id buffering and
merging are unchanged.
Tests pin both directions, including multi-chunk merging across one message
id, so the allowlist cannot silently kill streaming, plus an end-to-end
`_handle_streaming_chat` test asserting the live payload never reaches an
outbound message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ajay R <ajayr@formbuddy.com>
* chore(helm): bump config_version to 33 in chart values and README
config.example.yaml moved to 33 for the buzz channel block; the chart's
embedded config example and its README copy track it (config_version only
drives the outdated-config warning, per scripts/check_config_version.sh).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ajay R <ajayr@formbuddy.com>
---------
Signed-off-by: Ajay R <ajayr@formbuddy.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(extensions): add middleware plugin foundation
* fix(extensions): stop config resolution from masking extension loading
`create_app()` resolved the configured plugin list inside the fail-open
guard around `load_extensions()`. CI has no `config.yaml` (gitignored and
never generated by the workflow), so `get_app_config()` raised
`FileNotFoundError` there and was swallowed as an extension failure --
`load_extensions()` never ran at all, and the four `create_app()` tests in
`test_extension_app_loading.py` passed locally but failed on every runner.
Resolve the plugin list before the guard. Only an absent `config.yaml` is
tolerated, mirroring `_resolve_trace_enabled_for_app_construction()`:
`create_app()` runs at import time, and lifespan still performs strict
config loading before serving. A `config.yaml` that exists but fails to
parse or validate now propagates instead of being reported as an extension
failure -- reporting it as the latter silently dropped a `required: true`
extension rather than failing the boot.
Make the tests config-independent with an autouse `stub_app_config`
fixture, following the existing pattern in `test_gateway_lifespan_shutdown.py`,
and cover both new branches of the config-resolution boundary.
* fix(extensions): bind the run's extension snapshot through subagent delegation
The lead-agent path resolves one immutable loaded-extension snapshot per run
and binds it through task-store allocation and graph construction, but the
subagent path re-read the process-wide singleton at execution time. In
production both are the same object, yet a `set_loaded_extensions()` between
the lead run's start and a subagent's execution (test teardown, a future
hot-reload path) would let one run mix two extension generations — exactly what
the documented invariant exists to prevent.
The graph-build binding is a ContextVar scoped to synchronous construction, so
it has already exited by the time a tool delegates; the snapshot has to travel
through runtime context instead. The run worker publishes it under the
host-internal `EXTENSION_SNAPSHOT_CONTEXT_KEY` (written after the caller merge,
popped when the run has none, so a caller-supplied value is never
authoritative), `task_tool` reads it back through the type-checking
`resolve_run_extensions()`, and `SubagentExecutor` binds it at construction.
Callers outside the Gateway run path — embedded `DeerFlowClient`, standalone
LangGraph Server — install no snapshot and keep the existing
`get_loaded_extensions()` fallback.
* refactor(extensions): defer the ordering table by call, not by a lying tuple
`CORE_ORDERING_CONSTRAINTS` was a `tuple` subclass that overrode only
`__iter__` and resolved into a class-level `_resolved` side channel. A tuple
cannot populate its own storage after construction, so the instance stayed the
empty tuple it was built as: `len()` was 0, `bool()` was False, `in` was always
False, indexing raised, slicing and `reversed()` came back empty, and it
compared unequal to the plain tuples tests substitute for it — all while
iteration yielded the real constraints. Only `assert_ordering` consumed it, and
only by iterating, so the split went unnoticed.
The sibling `_AnchorTable(dict)` uses the same idea soundly because dict is
mutable: `self.update()` fills the real storage, making every inherited
operation correct. That trick does not survive the port to an immutable type.
Replace it with `core_ordering_constraints()`, matching how `stack.py` defers
the same kind of table via `_anchors()`. The deferral is kept — it is about
dependency direction, not just cycles: `extensions/` is the layer the
middleware layer calls into, so a module-scope `agents.middlewares` import here
points the dependency backwards and closes a cycle as soon as any middleware
imports something under `extensions/` at module level. Resolution stays at
`assert_ordering` time, which already runs inside the middleware builder.
Tests pin both halves: the returned value is a plain tuple whose len/bool/
membership/indexing/reversal/equality agree with iteration, and a subprocess
probe asserts importing `extensions.ordering` does not load the middleware
layer while calling the function does.
* feat(checkpoint-cache): delta-mode checkpoint history cache with recursive compose
Read-only, invalidation-free cache for LangGraph delta-channel history
({writes, seed}) at the get_delta_channel_history choke point:
- database.checkpoint_cache config (memory|redis; max_entries 0=disabled;
redis bounded by TTL, Gateway/async only)
- memory LRU backend (copy-on-read, zero-serde hit path) and redis backend
(lazy import, degrades to all-miss on outage)
- CachedHistorySaver: recursive composition from the nearest warm ancestor
(depth budget 8), caching each level; depth-0 cold chains delegate one
inner fast-path walk. Entries keyed by immutable
(db, thread, ns, checkpoint_id, channel) — no invalidation, coherent
across workers
- provider wiring: wraps in delta mode only (async + sync), full mode
untouched; sync path is memory-only
- bench opt-in: DEERFLOW_CHECKPOINT_BENCH_HISTORY_CACHE=1
sqlite bench (500 updates, payload 2KB): write phase 2.28x at f=250,
1.32x at f=10; one delegated walk per thread cold start.
* chore(config): bump config_version to 32 for database.checkpoint_cache
The checkpoint history cache feature added the database.checkpoint_cache
section to config.example.yaml; bump the schema version so existing
deployments get the outdated-config warning and can run make config-upgrade.
* chore(helm): bump config_version to 32 in chart values and README
* fix(checkpoint-cache): purge thread history entries on delete paths
Addresses review on #4638: delete_thread/prune removed source-of-truth
checkpoints but left the thread's materialized history payloads in the
cache (memory: until LRU eviction; redis: until TTL, default 1 day) — a
data-lifecycle gap for tenant offboarding / GDPR-style erasure.
- Cache contract gains thread-scoped adelete_thread/delete_thread
(lifecycle purge, not invalidation; entries remain immutable)
- Memory backend: stem scan over the LRU map; redis: SCAN MATCH + UNLINK,
outage degrades to TTL-bounded retention without raising
- CachedHistorySaver purges on delete_thread/adelete_thread and
prune/aprune (prune rewrites chains, so pre-prune histories must go);
delete_for_runs stays delegation-only (run->thread mapping unavailable,
no in-tree callers), documented in code
- ttl_seconds description documents the residual-retention window
- Tests: thread-scoped purge on both backends, saver-level delete/prune
purge, prefix-safety (t1 vs t10), redis outage degradation, and the
pinned no-purge behavior of delete_for_runs
* fix(checkpoint-cache): stable db identity, prefix-aware sync singleton, explicit zero TTL
Addresses Copilot review on #4638:
- checkpoint_cache_db_hash now hashes the credential-free postgres
identity (host:port/database + schema): credential rotation no longer
changes the cache namespace (cold cache + orphaned keys until TTL).
Unparseable URLs fall back to the raw string.
- The sync-path memory cache singleton is also keyed by its key_prefix:
a namespace change (db identity change or operator override) recreates
the cache instead of leaving stale-prefix entries unreachable and
unpurgeable.
- ttl_seconds=0 is now an explicit, documented opt-out of redis expiry
(SET without EX; redis maxmemory policy only) instead of a silent
'ttl_seconds or None' coercion.
Tests: credential-rotation hash stability, unparseable-URL fallback,
prefix-change singleton recreation, same-prefix singleton reuse, and
zero-TTL wire behavior (ex=None).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): constrain stdio launcher args and env at the config API
The API-managed stdio allowlist validated only `command`. Since `npx`/`uvx`
exist to fetch and run code, allowlisting the command name alone named a
binary without constraining what that binary ran, so the restriction did not
match the intent stated in `_validate_mcp_update_request`'s own docstring:
reducing the blast radius of a compromised authenticated browser session.
Screen two more fields on the stdio path, shared by `PUT /api/mcp/config` and
the enable branch of `PATCH`:
- `_ARBITRARY_EXEC_ARGS` rejects the flags that make a launcher evaluate a
string (`-c`, `--call`, `-e`, `--eval`, `--print`, `--shell`, `--node-arg`,
`--node-options`), matching `--flag=value` as well as `--flag value`, for
every allowlisted command rather than just `npx`/`uvx`. `-p` is deliberately
absent: it means `--package` for npx and `--python` for uv.
- `_CODE_INJECTING_ENV_VARS` rejects env names that inject code at process
startup (`NODE_OPTIONS`, `LD_PRELOAD`, `PYTHONSTARTUP`, ...), which would
otherwise sidestep the argument check.
Remote transports skip both -- they spawn nothing. Rejection messages echo
only the normalized flag, never the caller's value.
This is defense in depth, not a trust boundary: `npx`/`uvx` fetch and execute
remote packages by design, so an admin can still point one at a package they
published. Gateway admin remains equivalent to code execution on the host, and
the code, backend/AGENTS.md, and README all say so explicitly to keep the
check from being mistaken for a boundary.
`npx`/`uvx` stay in the default allowlist: nearly every official MCP server is
`npx -y @modelcontextprotocol/server-*`, so dropping them would break the
primary UI flow for adding one.
Tests cover each rejected flag spelling, case-insensitivity, `python -c` under
an extended allowlist, env injection, and the PATCH enable path, plus nine
positive cases pinning that real-world `npx`/`uvx`/`python -m` invocations
still validate.
* fix(mcp): screen PYTHONPATH and interpreter short-flag clusters
Review follow-up on the stdio launcher screen.
`PYTHONPATH` bypassed the env denylist on the *default* allowlist: `site`
imports `sitecustomize.py` from any `sys.path` entry before the tool's
entry point runs, so a caller-controlled directory is code execution
under plain `uvx` (verified against the real launcher). `PYTHONHOME` is
the same class via a repointed stdlib. Both are now rejected.
`PYTHONSTARTUP` is inert for non-interactive launchers and stays only as
belt-and-braces, now documented as such.
`-p` was exempted for every command, but it is node's `--print` --- the
long spelling was blocked while the short one passed once an operator
extended the allowlist. Whole-token matching also missed combined short
options (`node -pe`, `perl -we`, `python -Ic`). Both rules now apply to
commands outside `{npx, uvx}`, which are interpreters rather than package
runners; scoping them that way leaves the default allowlist unchanged,
since npx/uvx do not cluster short options and their trailing arguments
belong to a third-party server's own CLI where `-name` is ordinary.
`LD_LIBRARY_PATH`/`DYLD_LIBRARY_PATH` are recorded as an accepted
residual: conditional on the process loading a shadowable library, and
legitimately set by native-dependency servers.
Still defense in depth, not a trust boundary.
* docs(mcp): record NODE_PATH as an accepted search-path residual
Reviewer asked whether NODE_PATH belongs in _CODE_INJECTING_ENV_VARS for
parity with NODE_OPTIONS/PYTHONPATH. It does not, and the mechanism in the
report does not hold: verified on node v22, NODE_PATH is searched *after* the
local node_modules chain (the resolver unshifts the requiring module's paths
ahead of it), so it cannot shadow an installed dependency, and ESM import
ignores it entirely. It can only supply a CJS module that would otherwise
fail to resolve.
That puts it with LD_LIBRARY_PATH as a conditional search path, not with the
unconditional startup execution every entry in the set provides. Widen the
residual note from native-library shadowing to the general class, name
NODE_PATH in it, and pin the boundary with a test so it reads as a decision.
* fix(mcp): scope the stdio arg screen to the launcher's option region
`_arbitrary_exec_arg` inspected every element of `args`, justified by the claim
that the launchers in scope treat the token after `--` as a package name. That
is inaccurate: only the *first* token after `--` is the package name, and with
or without the separator every token from the package name onward is handed to
the third-party server's own argv, where `-c` is routinely "config" and `-e`
"env". So `npx -y @scope/server -c config.json` was rejected even though npm
never parses that `-c`, and the positive cases stayed green only because they
were picked to dodge the denylist. Over-blocking, not a bypass -- a regression
against the PR's own "the primary UI flow must keep working" goal.
Screen the launcher's own option region instead. Finding that region needs each
launcher's option *arity*, because a value is not a positional:
`npx -p <pkg> -c '<command>'` runs the command -- `-p` is `npm exec`'s
`--package` (it overrides the global `--parseable` shorthand), so `<pkg>` is
its value and npm keeps parsing its own flags. Ending the region at the first
non-flag token, the obvious rule, walks straight past that and turns an
over-block into a real eval bypass. The tables are therefore generated, not
guessed: `_NPX_BOOLEAN_ARGS` from `@npmcli/config` (npm 10.9.4) with the `-p`
override applied, `_UVX_VALUE_ARGS` from `uvx --help` (uv 0.11.1).
The unknown-option default is deliberately opposite per launcher, following the
exec set rather than symmetry. npx owns real exec flags, so an unknown option
consumes a value and keeps the region open; npm errors on options it does not
define, so that direction cannot reject an invocation that would otherwise
work. uvx owns no string-eval flag at all, so its screen is a tripwire rather
than a control, and an unknown option consumes nothing -- which keeps uv's
large and growing boolean surface from over-blocking. uvx also drops the short
spellings from its exec set, since `-c` is uv's `--constraints <file>`: an
ordinary documented option the old screen rejected outright.
Short options are matched case-sensitively now, because their case selects a
different option -- npm reads `-C` as `--prefix`, and folding it onto `-c`
rejected it. Long spellings stay case-insensitive: npm accepts `--CALL` and
runs it. Commands outside the package-launcher table are interpreters rather
than package runners and keep the unchanged whole-args screen, including the
`-p` rule and cluster decomposition.
Verified against the real launchers rather than by reading their docs. For
every argument vector in the new tests the validator's verdict matches whether
`npx` actually executes the string or passes it through to the server, and the
bypass-guard cases were confirmed to fail under the naive first-non-flag-token
boundary before being pinned.
Reported by @willem-bd on #4617. Two further false positives found while
confirming it are covered here: `uvx -c constraints.txt <tool>` above, and
`docker run -i --rm -e KEY <image>` -- the canonical GitHub MCP server
invocation -- which an operator who extends the allowlist with `docker` still
hits, since the conservative non-package-launcher path is unchanged.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): unify thread ID validation at the API boundary
Thread ID entry points accepted arbitrary strings while downstream
consumers (filesystem paths, Kubernetes Provisioner, JSONL event store)
each enforced different character restrictions, so invalid IDs were
persisted first and only failed later during sandbox/workspace init.
Centralize validation in deerflow.utils.thread_id (pattern
^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest,
scheduler dispatch, paths.py, JSONL store, embedded client, and align
the Provisioner pattern (pinned by a parity test). UUIDs are still
generated only when no ID is supplied; caller-supplied opaque IDs stay
supported.
Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy
cleanup escape hatch (filesystem cleanup guarded), read-only
client.get_thread stays unvalidated, and scheduler rows with legacy
invalid IDs record a failed dispatch instead of raising out of the
poll loop.
* docs: document canonical thread ID contract
README: caller-supplied thread IDs need not be UUIDs; the canonical
pattern and per-endpoint behavior. AGENTS.md: the shared
deerflow.utils.thread_id contract, its enforcement boundaries, and the
legacy-ID escape hatches.
* fix(gateway): close thread ID validation gaps at remaining entry points
Follow-up to the canonical thread ID contract: a full audit found the
uniform-422 coverage only reached about half of the thread_id surfaces.
- routers: 18 routes still took a bare thread_id: str — 13 in
thread_runs.py (including the five messages/events/workspace-changes
reads that returned 500 on the JSONL event store vs 404/empty on the
DB store), 4 read routes in threads.py, and the suggestions route
flagged in review. DELETE /api/threads/{id} keeps str as the declared
legacy-cleanup escape hatch.
- client: upload_files/delete_upload/list_uploads/get_artifact now
validate up front, fulfilling the RFC's 'all mutating entry points'
clause (get_thread stays unvalidated as the declared legacy read path).
- tui: the /resume literal-ref fallback validates against the canonical
contract and reports a descriptive error instead of failing deep in
the client.
- scripts/support_bundle.py: replace the drifted dot-allowing pattern
with a byte-identical copy of THREAD_ID_PATTERN (kept local so the
script still runs with a broken venv).
* test(gateway): guard the canonical thread ID contract against regressions
- test_thread_id_route_contract.py: static AST sweep asserting every
route handler with a thread_id parameter annotates ThreadId
(whitelist: the DELETE escape hatch), plus a runtime sweep hitting
all 44 thread_id routes with a non-canonical ID and asserting a 422
that names thread_id, plus a websocket upgrade-rejection case.
- test_thread_id_validation.py: client entry-point validation,
support_bundle pattern parity, and TUI literal-ref fallback tests.
- Align two tests that encoded the old contract (dotted IDs).
* feat(artifacts): inline editing for text artifacts in the panel
Add a PUT /api/threads/{id}/artifacts/{path} endpoint that atomically
replaces an existing UTF-8 text file under /mnt/user-data/outputs after
verifying its SHA-256 revision. Active runs conflict (409); binary,
symlink, oversized, and non-output paths are rejected.
Frontend: edit/save/discard buttons, draft state with conflict detection,
CodeEditor onChange/onSave, loader SHA-256 from ETag, i18n, beforeunload guard.
Backend: PUT endpoint with thread reservation, atomic temp-file replacement,
sandbox sync for non-mounted providers, rollback on failure, ETag on GET.
Tests: 8 backend + 1 blocking-IO + 3 frontend test files.
* fix(artifacts): scope replacement permissions and release sandboxes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
