* fix(clarification): drop sibling tool calls before interrupt
- Rewrite the AIMessage in ClarificationMiddleware.after_model so a
parallel bash/write_file cannot run before the user answers
- langchain return_direct only inspects the last ToolMessage; siblings
both execute and can keep the agent loop alive
- Skip the rewrite when disable_clarification is set
- Prompt and tool docs: do not call other tools in the same turn
Fixes#4906
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(clarification): enhance sibling tool call handling in ClarificationMiddleware
- Update ClarificationMiddleware to ensure sibling tool calls are dropped when `ask_clarification` is invoked, preventing unintended execution before user input.
- Modify documentation to clarify that the `return_direct` router now inspects all client-side tool calls of the last AIMessage, ensuring proper routing behavior.
- Introduce a new integration test to validate that sibling tools do not execute when `ask_clarification` is present in the same turn.
This change addresses potential issues with tool execution order and improves the overall reliability of the middleware.
Fixes#4906
* fix(clarification): enhance tool call filtering in ClarificationMiddleware
- Update _filter_content_tool_use to handle Gemini-style function_call blocks by matching on name when no id is present, ensuring proper filtering of tool calls.
- Modify ClarificationMiddleware to maintain sibling tool call integrity by dropping unnecessary blocks, improving the clarity of the AIMessage content.
- Add a new test to validate the correct stripping of idless function call content blocks, ensuring that sibling tool calls do not execute prematurely.
This change improves the robustness of the middleware and addresses potential execution order issues.
Fixes#4906
* fix(clarification): drop siblings when ask_clarification is malformed
LangChain parks invalid args on invalid_tool_calls independently, so a
valid sibling would otherwise still execute before the user answers.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by
max_recursion_limit) so scheduled background runs can use a different
recursion limit than the web UI. The value is read at dispatch time, so
a YAML edit applies to the next scheduled run without a Gateway restart.
Also logs a warning when the resolver falls back to the default or
clamps the configured value.
* feat(harness): add deterministic tool receipts with model-visible ledger
Stamp an immutable per-call fact record (tool name, status, args/output
hashes, byte count, timestamp) onto every tool result via a new
ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN)
into the model context so subagent reports can cite executed actions.
- tool_receipt.py: receipt core (make/extract/render), newest-first
budget eviction, ids derived from the append-only message stream
- ToolReceiptMiddleware: stamps ToolMessages directly or inside
Command-wrapped results; hidden ledger injection mirrors
DurableContextMiddleware; sits between ToolProgress and
ToolErrorHandling with a build-time ordering guard
- config: new verification section (receipts on, judge off), config
version 32 -> 33 with example/helm/docs updates
* feat(harness): split receipt rendering from stamping; address PR review
Review fixes (PR #4659):
- output_sha256 now uses sort_keys=True for structured content, matching
the order-invariant args fingerprint
- stamping failures log at warning (silent ledger gaps would corrupt
citations); tool execution remains never blocked
- _insert_after_leading_system_messages extracted to shared public
message_utils.insert_after_leading_system_messages; both middlewares
depend on it instead of a private cross-module helper
- code comments in English
RFC #4651 revision-2 alignment:
- receipts_render_mode config ('always' | 'delegation_only'): subagent
chains always render the ledger (citations are produced there); the
lead chain renders only while processing subagent results, removing
the always-on token tax from ordinary turns
- receipts gain bounded args_preview/output_preview (<=200 chars, tail
for output) so later typed claim bindings (tests_passed) can anchor
to a specific recorded execution
* docs(harness): state receipt freshness caveat and vocabulary layering in module docstring
* merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews
- backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the
ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the
verification.* hot-reload mention into config/AGENTS.md
- config.example.yaml + helm values/README: config_version 33 -> 34 so existing
v33 configs get the outdated-config prompt (review: willem-bd)
- tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads
them; re-add with the Layer 2 claim-binding consumer (review: willem-bd)
* docs(harness): cover receipt id renumbering after compaction in module docstring
Positional display ids are stable only while history is append-only;
compaction drops ToolMessages and the survivors renumber, so Layer 2
citation verification must resolve [rN] against the ledger as of the
citing turn (review: willem-bd, doc-only).
* chore(config): bump config_version to 35
main reached 34 via #4780 without the verification section; publishing
the new schema at the same number would silently skip the outdated-config
prompt for configs synced from main in that window (review: willem-bd).
* fix(skills): restore errno import dropped upstream in #4830
upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed
the import, so the PR merge ref fails lint-backend (F821).
* fix(harness): harden tool receipts against forgery and turn-scope delegation_only
Address willem-bd's pre-merge review on #4659:
1. Untrusted receipt metadata: the gateway now strips the server-owned
deerflow_tool_receipt key from external input messages; stamping always
overwrites any tool-supplied value instead of preserving it; and
extract_tool_receipts validates persisted receipt shapes (required typed
fields, unknown keys ignored) so malformed entries are skipped instead of
crashing render or passing as runtime-stamped evidence.
2. delegation_only no longer sticks on: _should_render now scopes the
subagent_status scan to the current turn (messages after the latest
genuine user message), so an old completed delegation stops rendering the
ledger on later ordinary turns. The genuine-user predicate moves to
message_utils.is_genuine_user_message, shared with input sanitization.
* fix(harness): stamp receipts outside short-circuiting tool middlewares
Address willem-bd's review on #4659: ToolReceiptMiddleware was registered
inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which
can return a ToolMessage without invoking its handler — blocked calls
(e.g. a read-before-write-denied write_file) never got a receipt, silently
gapping the ledger on a default-enabled path. SandboxAudit additionally
rebuilds medium-risk results, dropping an inner stamp.
ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the
runtime tail. Normal results still carry deerflow_tool_meta (stamped by
ToolErrorHandling on the inner return path); short-circuit messages
self-stamp meta or fall back to message.status. The new invariant is
declared as ordering constraints in deerflow.extensions.ordering, with
composed-chain regression tests for a blocked write and a warn-rebuilt
bash result.
* feat(mcp): per-user credential injection for shared MCP servers
A single HTTP/SSE MCP server entry can now serve several users, each
authenticated to the remote service with their own credential. A server
opts in with a user_auth block mapping user ids to credential header
values ($ENV_VAR references supported):
"user_auth": {
"header": "Authorization",
"users": { "<user-id>": "$SERVICE_TOKEN_ALICE" }
}
The built-in user-scoped auth interceptor resolves the authenticated
runtime user on every tool call (request runtime -> ambient LangGraph
runtime -> auth config -> request-scoped user ContextVar) and rewrites
the configured header via request.override(), the same per-call
mechanism as the OAuth interceptor. It registers after OAuth in the
shared assembly so its per-user value wins the header when a server
declares both. The entry's static headers are used only for startup
tool discovery.
Fail-closed by default: an unmapped user - including the anonymous
default-user fallback - or a credential whose env reference resolved
empty gets an actionable ToolException instead of another user's
credential; on_missing: "passthrough" opts out per server. Combined
with the existing per-(user, thread) MCP session scoping this gives
credential isolation on shared servers.
Gateway API: user_auth.users values are masked in GET responses, and
PUT round-trips preserve stored credentials for masked values (same
contract as env/headers/oauth secrets); a masked value for a user id
not already stored is rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): address review — preserve stored user_auth sub-fields on partial PUT, warn-and-skip stdio, allow extras
Review findings from #4868:
1. A partial user_auth payload (e.g. {"enabled": false}) merged to
users={} and default on_missing, irreversibly wiping stored
credentials on PUT. The merge is now sub-field-aware via
model_fields_set — omitted sub-fields carry the stored values, an
explicitly sent users map still replaces (so full-round-trip removal
works), masked values still swap back for stored credentials.
2. user_auth on a stdio server was a silent no-op: rewritten headers go
to call meta, never a transport header, while deny errors still fired.
The interceptor builder now warns and skips non-sse/http servers,
matching the tool_call_timeout transport-mismatch convention.
3. McpUserScopedAuthConfigResponse now allows extra keys like the
harness-side model, and extras survive masking and merge, matching
the server-level model_extra handling.
Adds four regression tests (partial-PUT preservation, explicit-map
replacement, extras round-trip, stdio warn-and-skip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): reject blank user_auth.header at the gateway
A blank header passed the gateway response model, was persisted, then
failed the harness-side ExtensionsConfig validator on reload — the PUT
returned 500 after the write and every later config load/startup failed
until the file was hand-edited. Mirror the harness non-blank validator
on McpUserScopedAuthConfigResponse so the PUT fails with 422 before
anything is written.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format extensions_config.py
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): harden the trust chain and masking around user-scoped credential selection
Address review round 4:
- Scrub client-supplied user_id from run context/configurable for
external callers in inject_authenticated_user_context, before every
early return, and restamp only from request.state.user. Now that
user_id selects which user's credential user-scoped MCP auth injects,
a forged value must not survive any future path that skips the
restamp. Internal callers (IM channels, scheduler) keep supplying
end-user identity as before (PR #3294 contract). Regression tests pin
both the scrub and that a forged body.context.user_id can never
resolve as another user through merge + inject ordering.
- Include the caller's own resolved user id in the fail-closed deny
message so operators can copy the exact users key (it differs by
deployment path), and document the key formats in the mcp.mdx doc.
- Mask sensitive extra keys inside user_auth on GET like server-level
extras, and swap masked sentinels back for stored values on PUT via
_merge_extra_value_preserving_masked.
- Extract the interceptor wrap loop into compose_tool_interceptors and
pin the security property functionally: an OAuth interceptor that
actually sets Authorization loses the final header to the per-user
credential through the same composition the session-pool path uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(docker): let the Gateway write extensions_config.json in production
AGENTS.md states extensions_config.json may be edited at runtime through the
Gateway API, and the Gateway implements that for the MCP enable switch,
PUT/PATCH /api/mcp/config and the skill update route. Two properties of the
production compose stack made every one of those writes fail:
- the file was mounted read-only, and
- Docker mounts it as its own mount point, so the temp-file-plus-rename in
atomic_write_extensions_config hit EBUSY. Linux refuses rename() over a
mount point whether or not the mount is writable, so making the mount
read-write alone is not enough.
Mount it read-write and fall back to an in-place overwrite on EBUSY only.
The fallback is deliberately non-atomic and says so in a warning; it is
reached only where the atomic route cannot work, and any other errno still
propagates. config.yaml stays read-only: no API writes it.
docker-compose-dev.yaml mounts the whole project directory, so the
destination is an ordinary file there and this never surfaced in development.
* test(docker): parse mount options instead of matching a :ro suffix
Docker's short-syntax options segment is comma-separated, so a read-only
mount can legally be spelled ":ro,z" or ":z,ro" — common with SELinux
relabelling. Matching the raw string for a ":ro" suffix reads those as
writable, which silently defeats the guard: the writability assertion would
pass on a read-only mount, and the config.yaml assertion would fail on a
correctly read-only one.
Parse the options segment and test membership instead, and cover the parser
with the spellings that broke the suffix check.
* fix(config): harden mutable extensions config
* feat(extensions): let an out-of-tree extension observe what the agent did
DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:
* The middleware chain injects and rewrites a lot of context — date
reminders, recalled memory, compaction summaries, durable-context data,
image payloads, activated skill bodies. Downstream, none of it is
attributable: at the model-call boundary an injected HumanMessage is
indistinguishable from the user's own, and anything wanting to tell them
apart has to pattern-match prompt wording, which breaks on the next copy
edit.
* Two runs of "the same agent" are only comparable if the chain enforced the
same limits, prompts and thresholds. Recovering that from outside means
reading private attributes and guessing which of them change behaviour — a
guess that rots silently as middlewares gain fields.
* The lead-agent factory resolves a model after runtime overrides, renders a
prompt, filters tools through authorization and composes a stack, all
inside one synchronous call, and none of it survives: a middleware sees its
neighbours but not the prompt, the run worker sees a graph but not what
went into it.
* Summarization is destructive by design. N messages leave the context and
one summary enters it; afterwards only the summary exists, so "which
messages became this?" is not reconstructible.
This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.
Message provenance
Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
messages they inject or rewrite. Stamping is unconditional — a fact whose
presence depends on whether an observer is installed is not a fact — and the
keys are server-owned, so provenance cannot be forged from a request.
Middleware self-description
Twelve middlewares declare their own behaviour-affecting parameters through
a duck-typed `release_policy_parameters()`. Long text is hashed rather than
embedded: a declaration is an identity, not a copy of the prompt.
Agent assembly descriptor
`assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
answers "did anything about this agent change between these two runs?".
`make_lead_agent()` keeps its graph-only signature — it is the LangGraph
Server ABI declared in langgraph.json. Tools and skills are sorted before
hashing because their assembly order is incidental; middlewares are not,
because stack order decides what wraps what. Host build identity is reported
but excluded from the fingerprint, so a redeploy does not invalidate every
agent's identity.
Context compaction observation
Summarization emits the content hashes of the messages it is about to remove
joined to the summary that replaced them. Content is the only identity
available at that seam: the summary does not become a message, and what later
projects it into a request renders it bounded and escaped rather than
verbatim.
Neutral policy, transform and MCP-source facts
Guardrail decisions are published to runtime context under a `__`-prefixed
key; result-rewriting middlewares append a declared, ordered transform trail;
MCP tools carry their credential-free logical origin.
Extension route identity
Contributed routes are session-authenticated and cannot opt out, but
"logged in" and "administrator" are different questions. Extensions get a
neutral projection of the caller rather than the host's auth context, and
`require_admin` fails closed when identity cannot be determined.
Extension-owned tables
An extension that persists data owns its own MetaData and migration chain, so
its tables are absent from Base.metadata and `alembic revision --autogenerate`
proposes dropping them. Extensions declare a table prefix, which is rejected
at registration if it would shadow a host table.
The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.
uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.
* fix(backend): sort gateway service imports
* fix(mcp): keep grant_type authoritative over extra_token_params
_fetch_token built the token request body as
{"grant_type": oauth.grant_type, **oauth.extra_token_params}, so an
operator-supplied extra_token_params that happened to contain
"grant_type" silently overwrote the value sent to the token endpoint
while the branch logic below still keyed off oauth.grant_type — the
sent grant_type and the chosen auth flow would disagree, and the
provider would almost certainly reject the request.
Spread extra_token_params first and set grant_type (and the other
reserved fields, which were already set after the spread) afterward, so
operator-supplied params can populate arbitrary extra fields but never
override the reserved ones the flow depends on.
* test(mcp): cover extra OAuth token parameters
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(e2b): preserve trailing whitespace in filenames and survive mtime overflow
_sync_outputs_to_host iterated the NUL-delimited find output with
entry.strip() on each record. NUL already guarantees record boundaries,
so the strip is redundant and harmful: a filename that legitimately ends
in whitespace (e.g. "report ") had its trailing space trimmed, pointing
host_path at the wrong file and recording a manifest key that can never
match — the file was re-downloaded on every release.
The same host-write block wrapped only os.utime in the outer
except OSError, but os.utime raises OverflowError (not an OSError) when
the ns value is out of range (a far-future remote mtime, e.g.
`touch -d '99999 years'`). That escaped the loop, skipping the manifest
write and forcing a full re-download next release. Wrap os.utime in its
own (OSError, OverflowError) so only the timestamp restoration is
dropped; the file is still written and the manifest still updated.
* test(e2b): rely on monkeypatch cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): exclude internal temp files from workspace changes
* fix(mcp): address review — shared tmp subdir constant, any-depth docs, nested test
- Export MCP_TMP_SUBDIR from constants.py and import it in both stdio
launch paths (tools.py, task_tool_caller.py) so the "/tmp" suffix is
composed once.
- Document that the .mcp exclusion matches by directory name at any
depth (like .git/node_modules) in README.md and mcp/AGENTS.md —
subagent work dirs below the workspace root get their own .mcp/tmp.
- Pin the any-depth semantic in test_workspace_changes.py with a nested
workspace/project/.mcp assertion.
* docs(mcp): correct any-depth exclusion rationale; re-home tmp pinning comment
The previous commit justified the any-depth `.mcp` exclusion with subagent
work dirs sitting below the workspace root — a mechanism that doesn't
exist: subagents share the parent's thread_id and both stdio launch paths
resolve sandbox_work_dir(thread_id), so `.mcp/tmp` is always pinned at the
workspace root. Reword mcp/AGENTS.md and the test comment to the real
justification (consistency with the other reserved dir names, robustness
against a server creating a relative `.mcp` from another cwd).
Also move the orphaned "pinning the process temp dir" rationale from
tools.py to constants.py next to MCP_TMP_SUBDIR, where both importers see it.
* fix(sandbox): bound aggregate E2B mount upload work
* fix(sandbox): preserve mount guards on upload failure
* fix(sandbox): cover mount preflight with deadline
* refactor(sandbox): clarify mount deadline checks
* refactor(sanbox): deduplicate mount deadline reason
* fix(sandbox): evaluate mount deadline reason lazily
* feat: integrate MiniMax Code as an ACP agent
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(skills): fail closed on drifted projection namespace on all platforms
* test(skills): add regression test simulating swallowed unlink on drifted namespace
* fix(middleware): target the latest user message on first-turn fallback injection
When an earlier turn ends without any dynamic-context reminder — e.g.
the async abefore_agent degraded path times out and skips injection
(issue #3402's guard) — the next turn enters the first-injection branch
(last_date is None) on a history that already holds several turns.
That branch scanned from the start and attached the ID-swap to the
FIRST user message. The swap's {id}__user copy is appended by
add_messages, so the stale first prompt moved to the tail of history,
ahead of the current question — and the model answered the old prompt
as if it were the current turn.
Scan from the end instead (matching the midnight-crossing branch) so
the reminder attaches to the latest user message and history order is
preserved. Genuine first turns are unaffected: they have exactly one
message, which is both first and last. The pre-existing
test_injects_only_into_first_human_message_not_later_ones case encoded
the buggy target selection and is updated to the corrected contract.
* refactor: rename first_idx to target_idx after reversed scan
The branch now scans from the end, so the local holds the LAST user
injection target; first_idx read misleadingly. Match the
midnight-crossing branch's naming convention and clarify the log line
accordingly. No behavior change.
* feat(mcp): add durable task runtime foundation
* fix(chart): sync embedded config version
* fix(mcp): isolate task polls during shutdown
* feat(mcp): track consecutive poll errors on mcp_tasks
poll_attempt_count grows on every claim (successful polls included), so it
cannot drive a failure backoff without misjudging normal long tasks. Add
consecutive_poll_error_count: incremented when a claim is released after a
poll error, reset to zero by any applied snapshot. The backoff/terminal
policy that consumes it lands with the first concrete driver.
* fix(mcp): harden durable task lifecycle
* feat(mcp): add ordinary durable task driver
* test(mcp): address durable task review feedback
* fix(mcp): preserve submit tool descriptions
* fix(mcp): bound remote task calls
* fix(mcp): bound persisted task payloads
* fix(mcp): preserve task tool error details
* fix(mcp): enforce durable task boundaries
* test(mcp): cover task config snapshot lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The Honcho backend landed in #4730 without user-facing docs: the main
README's Long-Term Memory section covers the other opt-in backends
(mem0, openviking) but never mentions honcho, and unlike mem0 the
backend shipped no guide README.
- Add backends/honcho/README.md mirroring the mem0 guide structure:
configuration (with the plain-HTTP api_key guard), workspace-per-user
isolation and fail-closed identity, recall/search behavior per mode,
limitations (no fact CRUD -> gateway 501, no DeerMem migration), and
async/failure-policy semantics.
- Add a short honcho paragraph + guide link to the README Long-Term
Memory section, alongside the existing mem0 paragraph.
* feat(extensions): add gateway services and routers
* feat(extensions): add standalone reference extension
* fix(extensions): harden contributed gateway routes
* docs(extensions): document gateway contribution points
* feat(extensions): add operator CLI for packaged extension management
Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.
Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.
Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.
Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.
Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.
Co-authored-by: Codex <codex@openai.com>
* fix(extensions): stop the managed plugins rewrite from destroying config
Two data-safety defects in the managed `plugins:` block writer.
The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.
The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.
Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.
* fix(extensions): harden the manager transaction and correct its docs
Follow-up hardening on the extension package manager.
Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
`UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
the interpreter that the entry-point probe then imports and calls, and every
later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
validation the HTTPS-only source rule depends on. Neither is an index, proxy,
cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
`ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
on case transitions, so only the separated spellings were caught. Short
generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
the package's build backend, so a config the manager could never write to must
fail before that code runs rather than afterwards through rollback.
Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
runs without `--locked` when the checkout had no lock, so uv writes one while
resolving; if that sync then failed, the restore was skipped and the operator
kept a lock file they never had. A failing recovery sync now also reports the
original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
dependency resolve invites a second interrupt that escapes the handler and
strands the checkout mid-transaction; the declarations are already restored
and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
builder is a different machine, but unlike an environment-driven wheelhouse
resolution this is a source the operator typed deliberately, so it is reported
rather than rolled back. Private-network indexes are untouched: a builder on
that network can reach them.
Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.
* test(gateway): pin the request-path projection agreement
`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.
Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:
- projection edge cases, including the segment-boundary guard that keeps
root_path="/api" from slicing "/apifoo/models" into a string the router
would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
mounted under the "/health" public prefix must still 401, and a POST
mounted under "/api/webhooks" must still require a CSRF token
Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).
Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.
* ci: pin uv to the version production ships
ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.
backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.
Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.
Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.
* fix(gateway): state the extension route auth limit and abort a failed dev sync
Two scoped review follow-ups.
README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.
docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.
Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.
* fix(extensions): point Git SSH shorthand at the HTTPS correction
Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.
The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.
* docs: keep the root extension summary within its new budget
#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.
---------
Co-authored-by: Codex <codex@openai.com>
* docs: govern agent guidance size
* refactor: split agent guidance by code scope
* Clarify virtual path handling in AGENTS.md
Updated the translation section to clarify the role of `LocalSandboxProvider` and the handling of virtual paths in the tool layer.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): honcho backend config parsing
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(memory): honcho v3 http client
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(memory): honcho memory manager (workspace-per-user, fail-closed identity, async offload)
- HonchoMemoryManager implements the MemoryManager contract (add/get_context/
search/get_memory/shutdown_flush + aadd/aget_context/asearch offloaded via
asyncio.to_thread), signatures verified against manager.py's tier-1/tier-2/
async abstracts.
- Workspace resolution: workspace_overrides[user_id] else
workspace_prefix + sanitize_id(user_id); missing/empty user_id fails closed
(no-op write, empty read) rather than falling back to a shared workspace.
User peer: user_peer_overrides[user_id] else sanitize_id(user_id).
- get_context self-truncates to max_injection_chars and raises
MemoryManagerError only under failure_policy.read=fail_closed; default is
log-and-return "".
- Restore backends/honcho/__init__.py to the noop direct-import convention
(MANAGER_CLASS = HonchoMemoryManager) now that honcho_manager.py exists,
replacing Task 10's temporary lazy __getattr__ scaffold.
- Fix Task 10 deferred docstring minor: sanitize_id docstring now states the
grammar allows up to 100 chars while this helper caps at 64.
- 19 new tests appended to test_honcho_memory_backend.py (write/read/async/
lifecycle/factory-discovery); 27/27 pass. Verified end-to-end that
manager.py's drop-in backend scanner resolves "honcho" with no core edits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(memory): collision-resistant identity derivation, exception containment, passive-writes flag
Task review findings (2 Critical + 1 Important), all fixed in the same worktree:
- CRITICAL (cross-user bleed): sanitize_id is lossy -- "user.name@example.com"
and "user-name@example.com" both sanitized to the same string, merging two
users' memory into one workspace/peer. Add _stable_id() (sanitize_id output
+ 8-hex-char SHA-256 suffix of the raw id) and use it on the default
(non-override) path in _workspace/_user_peer; workspace_overrides /
user_peer_overrides still match on the raw key, unchanged. The hash suffix
also guarantees a non-empty result for a raw id that sanitizes to "" (e.g.
"!!!"), so _user_peer can no longer return "". Documented in the manager's
isolation docstring.
- CRITICAL (exception containment): client.py's _post() called response.json()
outside the try block, so a 200 with a non-JSON body raised a bare
JSONDecodeError that would escape add() with no upstream handler. Wrap the
parse and raise HonchoRequestError (mirrors Mem0Client._request). Broadened
the manager's four boundary excepts from `except HonchoRequestError` to
`except Exception` (mirrors openviking_manager.py's broad-guard precedent),
with `except MemoryManagerError: raise` first so a contract error is never
swallowed or double-wrapped.
- IMPORTANT: added requires_passive_writes_in_tool_mode: ClassVar[bool] = True
-- Honcho's only write path is passive add() (no fact CRUD hooks), so tool
mode must keep MemoryMiddleware writes flowing to the deriver. Mirrors
mem0_manager.py's identical flag/rationale.
Minors addressed: get_memory(user_id=None) empty-shape-with-no-calls test;
empty-string user_id tests for add()/get_context(); dedicated collision test
proving two colliding raw ids resolve to different workspaces/peers.
10 new tests (37/37 total pass); RED verified by stashing only the
implementation files (tests import the not-yet-existing _stable_id, so the
whole module fails to collect) before restoring the fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(memory): blocking-io anchor for honcho backend; docs + config example
- Adds test_honcho_memory_backend.py in tests/blocking_io/ with fake-client blocking IO
- Mirrors openviking anchor structure and conftest conventions
- Updates backends/README.md with honcho row and config keys section
- Updates config.example.yaml with honcho commented block
- Updates backend/AGENTS.md with honcho memory backend bullet
- Documents workspace resolution (prefix + collision-resistant sanitized id)
- Documents tool mode passive write retention via MemoryMiddleware
- Documents async entrypoint offloading via asyncio.to_thread
- Documents fail_closed vs fail_open recall failure policy
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(memory): wire close() to shutdown hook; correct honcho README defaults and tool-mode note
- HonchoMemoryManager.close() releases the HTTP client, mirroring
mem0_manager.py's pattern and the base MemoryManager.close() shutdown hook.
- README: fix workspace_prefix (deerflow-u-), message_char_limit (8000),
max_injection_chars (6000), and base_url (default http://localhost:8000,
not required) against backends/honcho/config.py; add missing
timeout_seconds/connect_timeout_seconds rows; replace the "middleware
mode only" claim with wording matching reality (tool mode supported,
search implemented, passive writes retained via MemoryMiddleware).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(memory): honor failure_policy.read on all honcho recall paths; review nits
Addresses PR #4730 review feedback:
- search() and get_memory() now route through a _read_or_fallback policy
gate (mem0's pattern), so failure_policy.read: fail_closed raises
MemoryManagerError on every recall path as documented; get_context()
uses the same helper, preventing future drift.
- Session ids use the collision-resistant _stable_id derivation; bare
sanitize_id would merge threads like "t.1"/"t-1" into one session.
- HonchoClient accepts a transport kwarg (Mem0Client precedent) so tests
inject httpx.MockTransport through the constructor.
- Config: empty/null workspace/peer override values fail fast at parse
time instead of silently falling through to the default derivation.
- _UTC_NOW_FIELDS 1-tuple replaced by a plain _UTC_NOW_FORMAT constant.
- README: user_peer_overrides row described the wrong target (it
overrides the user's own peer, not assistant_peer); document the
non-empty constraint on override values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(memory): qualify honcho isolation claim for shared workspace_overrides
The module docstring claimed users cannot see each other's memory by
construction, unconditionally. That holds for the default
one-workspace-per-user derivation, but a workspace_overrides entry mapping
several users to one workspace shares that workspace's search index:
search() uses Honcho's workspace-scoped /search (no peer filter), while
get_context()/get_memory() stay peer-scoped via working_representation.
State the asymmetry in the docstring, the README Workspace Resolution
section, and the workspace_overrides table row.
Docs-only; no behavior change (review follow-up on #4730).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Admission derived the live-entry count as `HLEN - 3`, where 3 was the number
of `meta:*` fields written 35 lines earlier in initialize(). Nothing tied the
two together, so adding a fourth meta field would shift the capacity ceiling
by one.
Name the offset `META_FIELD_COUNT` next to initialize(), and add a guard test
asserting a freshly initialized ledger holds exactly those three fields, plus
one pinning that a hard_limit of N admits exactly N reservations.
References #4575
Co-authored-by: icn5381 <255778606+icn5381@users.noreply.github.com>
* feat(extensions): observe task lifecycle and system model calls
PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.
Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).
`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.
Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.
Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.
System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.
The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.
Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.
Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.
* fix(extensions): decide notification fail-open by origin, observe cancellation
`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.
`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.
DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.
* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown
Review response on #4684:
- The memory observation bridge caught BaseException, which would swallow
a teardown signal raised while dispatching; it now catches Exception,
matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
hook-failure path and logged an asyncio-internal traceback; it now logs
a warning like the pre-hook budget skip, while a TimeoutError a
contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
so log lines said "task goal/title/..."; they now carry the task
scope id alongside the kind.
