* fix(examples): keep the extension example's tests passing as the contract grows
The README's package-test commands failed one test on main:
isinstance(registry, ExtensionRegistry) is False because the test's
FakeRegistry does not implement knowledge_provider(), which
extension-api 0.2.7 added to the runtime-checkable protocol. The same
assertion broke earlier when plugin() was added. Nothing in CI runs
these tests, so each addition to the contract broke them silently.
FakeRegistry now records knowledge_provider() and plugin() calls, and
the test asserts the example registers neither, as its README says.
Extension Package Check, which already runs when the contract changes,
now installs each bundled example that has a tests/ directory next to
the checkout's extension-api and runs the README's pytest and ruff
commands. A backend test pins that step and checks that every tested
example declares those tools in its dev extra.
The example README and the extensions guide now count nine
contribution kinds, including knowledge providers.
* docs(changelog): reference #7364 for the extension example test fix
* test(extensions): name the missing package-test step in the wiring pin
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(plugins): add sectioned long-form report writing example
A standalone example extension whose save_section model tool stores a
万字-scale report section by section (SQLite, per-user and per-thread)
and assembles every section into one markdown document on a final
section_index=999 call. The assembled document is returned as a tool
artifact with only a short status line re-entering the model context, so
the full report never consumes the context budget.
Assembly drops a section's leading heading when it restates the section
title (models habitually open each section with '# <title>' while the
assembler already emits '## <title>'), which otherwise doubles every
heading in the document. The outline section passes through unwrapped
and keeps its own heading as the document title.
* fix(frontend): register report sections in extension catalog
* docs(report-sections): correct the test command path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(knowledge): add independent Wiki and entity relation retrieval tools
* fix(knowledge): address review diagnostics and CI catalog coverage
* fix(knowledge): deduplicate graph evidence and update catalog e2e
* fix(knowledge): clarify bounded evidence and pin browser CI runners
* Update e2e-tests to run on ubuntu-latest
* Update workflow to use ubuntu-latest
* test(ci): stop pinning maintainer-controlled browser runner images
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Allow ordinary Agent tasks the same 16,000 JSON bytes already accepted for
file-reference tasks, keeping the 12,000-byte output cap and bounded rejection.
Cover ASCII, escaped Unicode, maximum output and pre-admission rejection.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* fix(workflows): stop idle agent waits from consuming worker turns
While a script step waited on a child Agent run, the supervisor sent an
empty response frame to the QuickJS worker every 20 ms and each round
trip counted toward the 10,000-turn cap, so any workflow whose Agent
calls kept it waiting for about 209 s failed with "Worker turn limit"
regardless of `timeout_seconds`.
The worker now reports when its job queue has drained, and the
supervisor waits on the pending calls locally while the worker is idle,
still re-checking cancellation and faults every 20 ms, without
exchanging frames. `timeout_seconds` is the only wall-clock bound again;
scripts that spin the job queue still hit the turn cap. The cap is a
module constant so tests can lower it.
Fixes bytedance/deer-flow#7022
* fix(workflows): default a frame without idle to the polling path
Review follow-up: read `idle` with `frame.get("idle", False)` so an
outcome-less frame that lacks the key degrades to the previous poll
behaviour instead of failing the run with a KeyError, and give the
cancel-latency regression test a 3 s budget so store writes and process
reaping on a loaded CI runner cannot flake it.
* fix(workflows): detect worker exit during idle Agent waits
Check worker exit in the existing local poll and clean up pending child runs. Adapt the work-frame budget regression from PR #7025 while retaining precise idle detection.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(extensions): isolate team workers and rotate admissions
* fix(extensions): index and batch pending team snapshots
* fix(agent-teams): retain unfinished owner rotation rounds
Carry the remaining owner queue across planning passes, including exhausted owners and newcomers. Read legacy rows completely before updating their pending projection and distinguish dispatch reservations from sorted entries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add example guide and packaging checks
* fix(extensions): align packaging checks with installer
Reject Windows drive-qualified wheel paths before extraction and reuse the installer entry-point validator. Cover unsafe paths, dotted attributes, and valid dotted modules in regression tests.
* fix(extensions): skip unused wheel extraction
Only extract wheels when browser manifests need validation, with a backend-only regression test. Link the three remaining example READMEs in the developer guide.
* docs(frontend): trim guidance intro below size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs: list the summarize event tag and scope the extension example's claims
RUN_EVENT_STREAM.md listed seven middleware tags, but the catalog and the
run event stream contract carry eight; `summarize` is emitted by
SummarizationMiddleware for every successful summary result. Add the tag,
describe the event, and pin the documented list to MIDDLEWARE_EVENT_TAGS so
the next tag cannot drift silently. summarization.md now lists all eight
`changes` fields of that event.
The extension example said it showed all five (README, pyproject, module
docstrings, test name) or every contribution kind, while ExtensionRegistry
has eight. Say it demonstrates five, name the three it does not, point to
the observers guide and the bookmarks plugin, and assert in the example's
test that the two observer kinds stay unregistered.
* docs(changelog): reference #6582
* docs: clarify extension example package layout scope
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(extensions): measure the team response size gate in UTF-8 bytes
json.dumps escapes non-ASCII by default, so a CJK clarification answer
counted ~6 escaped bytes per character and an emoji 12, while the plugin
documents 4,000 characters / 8,000 UTF-8 bytes and its own UI validates
exactly those two bounds. A 漢 answer of 2,000 characters (6,000 UTF-8
bytes, inside the bound) was refused by the gate.
Serialize with ensure_ascii=False so the gate measures the UTF-8 bytes it
names, matching the budget walkers in the same module.
* fix(agent-teams): honor the raw UTF-8 response boundary
Measure string answers without JSON quotes or escapes, retain the structured JSON byte limit, and reject malformed Unicode with the normal response validation error. Add boundary and state-preservation regressions and document the response budgets.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add full Custom Agent team collaboration plugin
* fix(extensions): register agent teams in the capability catalog
* fix(agent-teams): recover incomplete teams and preserve request capacity
* fix(agent-teams): preserve clarification waits and active handoff lineage
* fix(agent-teams): align mention and input text limits
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client
- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
configuration: one connection-precedence rule, credential fingerprint, wire-size
counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
DeerMem through one injected judge hook (both off by default, `enforce` gated by
the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
`config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
per-stratum gates, censoring and the no-network baseline.
Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
upload-event scrub that precedes persistence, only removals/reinforcements of ids
that existed, and persisted near-duplicate merges (matched by id, or by identity
for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
`typesafe` block changes (including a credential rotation); publication is
serialized and revalidates the config generation, so a build started before an
`enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
and correction.
* fix(typesafe): resolve one credential per layer and refuse unusable connection values
Review follow-ups on the shared TypeSafe client (#5906):
- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
shared `typesafe:` block: the first layer that sets either credential setting
decides it, and inside one layer the literal key wins (the standalone
semantics). A consumer whose endpoint does not accept the block's key no longer
sends it on every call, and the missing-credential message keeps naming the
variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
fragment, embedded credentials, whitespace or control characters, and a
malformed port fails at construction instead of on every request (httpx raises
`UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
a secret mounted from a file, a non-printable character — is refused when the
connection resolves. It used to construct fine and then put the whole
`Bearer <key>` value into h11's `LocalProtocolError` message, which the
guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
so that message can no longer carry the credential through the cause chain; the
exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
path: `max_attempts: true` was silently read as `1`. An integer stays a usable
float, so `timeout: 5` keeps working.
Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.
* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence
Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.
Typesafe client (the reply is treated as hostile input too):
- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
`Content-Encoding`, and read through a bounded `iter_bytes` that stops past
`MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
decoded in full and, on the gate path, allowed the tool call unexamined; it is
now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
escaped the module's taxonomy, so the memory consumers lost the round's record
instead of recording a fallback.
Memory judge:
- A request-level failure is recorded as `request_failed`, never `no_verdict`
(`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
endpoint was indistinguishable from a provider with no usable answer). The
adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
and log; `no_verdict` stays for a question-level failure, and an ineligible side
keeps its own reason even when the round's request failed.
Shadow evaluation (the enablement gate reads the evidence, not one number):
- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
meet the 1% target; a sample that only meets the point estimate is
`INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
treated as absent rather than scored: `-3` read as "nothing lost", adding a
phantom saving to the numerator and the denominator.
Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.
Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.
* fix(memory-eval): pair the baseline comparison on the same records
Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).
Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).
- `Evidence` carries the paired numbers (`baseline_paired_skips`,
`baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
when the annotation covers only a subset of the scored skips (the message carries
the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
`backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.
Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.
* fix(typesafe): bound the sync body read by the deadline
`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.
`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.
* fix(memory): read the YAML off spelling, and veto a skip over the whole batch
Two review findings on the memory judging path.
`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.
The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.
Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
Follow-up to #5833 after an independent review of the merged code.
- A malformed answer (deeply nested JSON raising RecursionError, or a
huge noul integer raising OverflowError outside the try) escaped the
fail-open path, and in the per-message task group it cancelled the
other messages' requests and dropped every flag of that tool call.
Parsing now happens inside the try, which also catches those errors.
- The 16 KiB response cap counted decoded bytes, so a compressed body
could expand far past it first. Requests now ask for identity encoding
and drop encoded responses.
- A plain-HTTP loopback endpoint followed HTTP_PROXY/ALL_PROXY, sending
the bearer key and excerpt to the proxy in cleartext. Such endpoints
now ignore proxy environment variables; HTTPS endpoints keep them.
Tests cover each case, including a real local server behind an
unreachable proxy, and pin previously untested behavior: independent
per-request deadlines, the sync hook without a task store, the >=
threshold and all four named remote tools, warnings for messages that
share a tool-call ID, consumed flags, and scanning past host messages.
The root README no longer says the excerpt is always PII-redacted, and
the example README documents the new limits and that the sync path's
deadline does not cover a stalled DNS lookup.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(examples): add opt-in fetched-content screening extension
* fix(examples): screen sync and async results through supported middleware hooks
* fix(examples): preserve empty message IDs when applying screening warnings
* fix(examples): include screening policy in assembly identity
* feat(examples): load fetched-content screening through the extension entry
Package the screening example as a plugins: extension instead of an
operator-configured middleware. install() validates the private config
and, once enabled, contributes one TOOL_VISIBLE middleware for lead
agents and subagents. The package imports only deerflow_extension_api.
Extension tool wrappers are observational, so detection and the warning
use two supported contracts: the tool wrapper classifies the visible
result and records flagged tool-call IDs in the run's task store, and
before_model replaces the matching messages of the latest tool step by
ID. The excerpt sent to Jev is now the redacted, sanitized text the
model sees. Runs without a task store send nothing. Unexpected local
errors surface as isolation diagnostics; provider errors fail open.
Tests load the package through load_extensions() and the host isolation
wrapper and run real lead/subagent graphs with a task store bound as the
Gateway worker binds it. The README records what the example exercised
in the extension API and the gaps it found.
* fix(examples): classify each tool message of a screened result on its own
A Command result was classified as one concatenated excerpt and the flag
landed on its first text message, so a benign first result could take the
warning while an injected later result reached the model unmarked.
Each text message now gets its own bounded excerpt and Jev request, for at
most eight messages per tool call. The requests share one client and run
concurrently in a task group, each under its own deadline, so one slow or
failed request cannot hide another message's flag. Every flagged message's
tool-call ID is recorded. The message cap joins the declared policy, and the
README describes the per-message requests and data sharing.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(examples): add a standalone text classification plugin
A deployment-installed package contributes one model tool, classify_texts,
and a status action through the full-stack plugin contract. The tool labels a
list of texts against caller-supplied categories with a deployment-configured
backend: Jev (one choice question per item over a shared batch state, checked
per item) or an OpenAI-compatible chat endpoint (a JSON label list validated
in code). Items are packed into requests under a size limit and sent with
bounded concurrency; every failure is explicit per call or per item;
credentials come only from deployment-named environment variables; item text
never enters instructions, question keys, results or errors, and the served
model name is kept to a short identifier. Each batch runs under an asyncio
timeout bounded by the call deadline, so the whole call stays inside the
host's 30-second limit, and ids and category names are bounded in bytes so
300 results fit the host's 64 KiB output limit.
* docs(examples): add an opt-in live smoke test for the text classification plugin
The script loads the package through the real extension path and classifies 20
synthetic English and Chinese sentences plus a blank and an over-long item with
the configured backend, printing labels and statuses only. Against real Jev it
agreed on all 20 labels in two requests and 1.6 seconds, with the served model
reported as jev-1.13.0.
* fix(examples): measure request size as sent and cover transport failures
httpx serializes json= bodies with the json.dumps default, escaping every
non-ASCII character as six bytes, while the size estimators measured raw
UTF-8, so CJK batches could reach about twice the 256 KiB limit. The
estimators now measure the same encoding; a test with CJK categories and
texts checks the bytes the transport actually received. The network status
gets a test through connect and protocol errors, and the deadline test's
wall-clock slack is widened for loaded runners.
* fix(examples): match classifier request wire encoding
* fix(examples): bound classification output keys by JSON size
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(extensions): let an out-of-tree extension observe what the agent did
DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:
* The middleware chain injects and rewrites a lot of context — date
reminders, recalled memory, compaction summaries, durable-context data,
image payloads, activated skill bodies. Downstream, none of it is
attributable: at the model-call boundary an injected HumanMessage is
indistinguishable from the user's own, and anything wanting to tell them
apart has to pattern-match prompt wording, which breaks on the next copy
edit.
* Two runs of "the same agent" are only comparable if the chain enforced the
same limits, prompts and thresholds. Recovering that from outside means
reading private attributes and guessing which of them change behaviour — a
guess that rots silently as middlewares gain fields.
* The lead-agent factory resolves a model after runtime overrides, renders a
prompt, filters tools through authorization and composes a stack, all
inside one synchronous call, and none of it survives: a middleware sees its
neighbours but not the prompt, the run worker sees a graph but not what
went into it.
* Summarization is destructive by design. N messages leave the context and
one summary enters it; afterwards only the summary exists, so "which
messages became this?" is not reconstructible.
This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.
Message provenance
Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
messages they inject or rewrite. Stamping is unconditional — a fact whose
presence depends on whether an observer is installed is not a fact — and the
keys are server-owned, so provenance cannot be forged from a request.
Middleware self-description
Twelve middlewares declare their own behaviour-affecting parameters through
a duck-typed `release_policy_parameters()`. Long text is hashed rather than
embedded: a declaration is an identity, not a copy of the prompt.
Agent assembly descriptor
`assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
answers "did anything about this agent change between these two runs?".
`make_lead_agent()` keeps its graph-only signature — it is the LangGraph
Server ABI declared in langgraph.json. Tools and skills are sorted before
hashing because their assembly order is incidental; middlewares are not,
because stack order decides what wraps what. Host build identity is reported
but excluded from the fingerprint, so a redeploy does not invalidate every
agent's identity.
Context compaction observation
Summarization emits the content hashes of the messages it is about to remove
joined to the summary that replaced them. Content is the only identity
available at that seam: the summary does not become a message, and what later
projects it into a request renders it bounded and escaped rather than
verbatim.
Neutral policy, transform and MCP-source facts
Guardrail decisions are published to runtime context under a `__`-prefixed
key; result-rewriting middlewares append a declared, ordered transform trail;
MCP tools carry their credential-free logical origin.
Extension route identity
Contributed routes are session-authenticated and cannot opt out, but
"logged in" and "administrator" are different questions. Extensions get a
neutral projection of the caller rather than the host's auth context, and
`require_admin` fails closed when identity cannot be determined.
Extension-owned tables
An extension that persists data owns its own MetaData and migration chain, so
its tables are absent from Base.metadata and `alembic revision --autogenerate`
proposes dropping them. Extensions declare a table prefix, which is rejected
at registration if it would shadow a host table.
The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.
uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.
* fix(backend): sort gateway service imports
* feat(extensions): add gateway services and routers
* feat(extensions): add standalone reference extension
* fix(extensions): harden contributed gateway routes
* docs(extensions): document gateway contribution points
* feat(extensions): add operator CLI for packaged extension management
Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.
Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.
Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.
Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.
Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.
Co-authored-by: Codex <codex@openai.com>
* fix(extensions): stop the managed plugins rewrite from destroying config
Two data-safety defects in the managed `plugins:` block writer.
The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.
The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.
Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.
* fix(extensions): harden the manager transaction and correct its docs
Follow-up hardening on the extension package manager.
Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
`UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
the interpreter that the entry-point probe then imports and calls, and every
later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
validation the HTTPS-only source rule depends on. Neither is an index, proxy,
cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
`ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
on case transitions, so only the separated spellings were caught. Short
generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
the package's build backend, so a config the manager could never write to must
fail before that code runs rather than afterwards through rollback.
Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
runs without `--locked` when the checkout had no lock, so uv writes one while
resolving; if that sync then failed, the restore was skipped and the operator
kept a lock file they never had. A failing recovery sync now also reports the
original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
dependency resolve invites a second interrupt that escapes the handler and
strands the checkout mid-transaction; the declarations are already restored
and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
builder is a different machine, but unlike an environment-driven wheelhouse
resolution this is a source the operator typed deliberately, so it is reported
rather than rolled back. Private-network indexes are untouched: a builder on
that network can reach them.
Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.
* test(gateway): pin the request-path projection agreement
`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.
Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:
- projection edge cases, including the segment-boundary guard that keeps
root_path="/api" from slicing "/apifoo/models" into a string the router
would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
mounted under the "/health" public prefix must still 401, and a POST
mounted under "/api/webhooks" must still require a CSRF token
Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).
Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.
* ci: pin uv to the version production ships
ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.
backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.
Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.
Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.
* fix(gateway): state the extension route auth limit and abort a failed dev sync
Two scoped review follow-ups.
README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.
docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.
Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.
* fix(extensions): point Git SSH shorthand at the HTTPS correction
Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.
The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.
* docs: keep the root extension summary within its new budget
#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.
---------
Co-authored-by: Codex <codex@openai.com>
