feat(extensions): add gateway contribution points and packaged extension management (#4780)

* feat(extensions): add gateway services and routers

* feat(extensions): add standalone reference extension

* fix(extensions): harden contributed gateway routes

* docs(extensions): document gateway contribution points

* feat(extensions): add operator CLI for packaged extension management

Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.

Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.

Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.

Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.

Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.

Co-authored-by: Codex <codex@openai.com>

* fix(extensions): stop the managed plugins rewrite from destroying config

Two data-safety defects in the managed `plugins:` block writer.

The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.

The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.

Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.

* fix(extensions): harden the manager transaction and correct its docs

Follow-up hardening on the extension package manager.

Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
  `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
  the interpreter that the entry-point probe then imports and calls, and every
  later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
  validation the HTTPS-only source rule depends on. Neither is an index, proxy,
  cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
  `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
  on case transitions, so only the separated spellings were caught. Short
  generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
  the package's build backend, so a config the manager could never write to must
  fail before that code runs rather than afterwards through rollback.

Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
  runs without `--locked` when the checkout had no lock, so uv writes one while
  resolving; if that sync then failed, the restore was skipped and the operator
  kept a lock file they never had. A failing recovery sync now also reports the
  original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
  dependency resolve invites a second interrupt that escapes the handler and
  strands the checkout mid-transaction; the declarations are already restored
  and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
  gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
  contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
  line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
  builder is a different machine, but unlike an environment-driven wheelhouse
  resolution this is a source the operator typed deliberately, so it is reported
  rather than rolled back. Private-network indexes are untouched: a builder on
  that network can reach them.

Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.

* test(gateway): pin the request-path projection agreement

`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.

Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:

- projection edge cases, including the segment-boundary guard that keeps
  root_path="/api" from slicing "/apifoo/models" into a string the router
  would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
  mounted under the "/health" public prefix must still 401, and a POST
  mounted under "/api/webhooks" must still require a CSRF token

Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).

Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.

* ci: pin uv to the version production ships

ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.

backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.

Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.

Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.

* fix(gateway): state the extension route auth limit and abort a failed dev sync

Two scoped review follow-ups.

README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.

docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.

Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.

* fix(extensions): point Git SSH shorthand at the HTTPS correction

Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.

The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.

* docs: keep the root extension summary within its new budget

#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.

---------

Co-authored-by: Codex <codex@openai.com>
This commit is contained in:
Nan Gao 2026-08-13 23:55:30 +08:00 committed by GitHub
parent e4a7a04719
commit c542185a7f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
62 changed files with 7565 additions and 138 deletions

View File

@ -69,3 +69,10 @@ backend/.coverage
!README.md
!frontend/README.md
!backend/README.md
# Extension manager snapshots must enter the backend builder intact. A Python
# package may require README metadata, native modules, or package assets that
# the general image-context exclusions above intentionally omit elsewhere.
!backend/extensions/
!backend/extensions/sources/
!backend/extensions/sources/**

View File

@ -35,7 +35,10 @@ jobs:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v3
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Install backend dependencies
working-directory: backend

View File

@ -33,6 +33,9 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Install backend dependencies (documented default)
working-directory: backend
@ -85,6 +88,9 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Install backend dependencies
working-directory: backend

View File

@ -30,6 +30,9 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Sync labels
run: uv run --with pyyaml python scripts/sync_labels.py

View File

@ -50,6 +50,9 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Check uv.lock is in sync
working-directory: backend

View File

@ -57,6 +57,9 @@ jobs:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Install backend dependencies
working-directory: backend
run: uv sync --group dev
@ -77,6 +80,9 @@ jobs:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Install backend dependencies (replay gateway)
working-directory: backend
run: uv sync --group dev

View File

@ -48,6 +48,9 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
version: "0.11.1"
- name: Install backend dependencies
working-directory: backend

View File

@ -57,6 +57,7 @@ deer-flow/
├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
├── backend/ # Python backend — see backend/AGENTS.md
│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
│ ├── extensions/sources/ # Deployable snapshots of locally installed Python extensions
│ ├── packages/extension-api/ # deerflow-extension-api package (import: deerflow_extension_api.*) — public extension contract
│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
@ -66,6 +67,7 @@ deer-flow/
│ # Managed integration skill packs are global at .deer-flow/integrations/skills/{provider}/
│ # Integration credentials and enabled state remain per-user
├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
├── examples/deerflow-extension-example/ # Standalone package demonstrating all extension contribution kinds
├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
└── docs/ # Cross-cutting docs, plans, and design notes
@ -73,8 +75,15 @@ deer-flow/
Third-party extensions are loaded from a top-level `plugins:` list in `config.yaml`
(operator-controlled on purpose — that list causes code to be imported, so it is deliberately
kept out of the API-writable `extensions_config.json`). See the Extension System section in
[backend/AGENTS.md](backend/AGENTS.md).
kept out of the API-writable `extensions_config.json`). Packaged extensions can contribute
middleware, task lifecycle, system-model observers, Gateway services, and FastAPI HTTP
routers; the [reference extension](examples/deerflow-extension-example/) demonstrates all
five. Manage them with `deerflow extensions install/list/enable/disable/remove` or the root
`make extension-*` wrappers. Every mutation requires a Gateway restart, and both build
hooks and extension code execute with Gateway privileges, so only trusted operator sources
belong in this path. The manager transaction, accepted source forms, lock discipline, and
contribution contract live in
[the extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md).
Runtime config lives at the **repo root**: copy `config.example.yaml``config.yaml`
(main app config) and `extensions_config.example.json``extensions_config.json` (MCP
@ -105,6 +114,11 @@ make support-bundle # Generate redacted troubleshooting summary, AI issue draft
make config # Generate local config files from the examples
make check # Check that required tools are installed
make install # Install all dependencies (frontend + backend + pre-commit hooks)
make extension-install SOURCE=... # Install and enable a trusted Python extension
make extension-list # List configured Python extensions
make extension-enable NAME=... # Enable an installed extension (restart required)
make extension-disable NAME=... # Disable without uninstalling (restart required)
make extension-remove NAME=... # Remove package and config entry (restart required)
make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
make start # Start all services in production mode (local, optimized)
make stop # Stop all running services

View File

@ -1,6 +1,6 @@
# DeerFlow - Unified Development Environment
.PHONY: help config config-upgrade check check-agent-guidance install setup doctor support-bundle detect-thread-boundaries detect-blocking-io dev dev-daemon start start-daemon nginx stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway docker-logs-redis
.PHONY: help config config-upgrade check check-agent-guidance install extension-install extension-list extension-enable extension-disable extension-remove setup doctor support-bundle detect-thread-boundaries detect-blocking-io dev dev-daemon start start-daemon nginx stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway docker-logs-redis
BASH ?= bash
BACKEND_UV_RUN = cd backend && uv run
@ -30,6 +30,11 @@ help:
@echo " make detect-thread-boundaries - Inventory backend executor/thread/event-loop boundaries"
@echo " make detect-blocking-io - Inventory blocking IO that may block the backend event loop"
@echo " make install - Install all dependencies (frontend + backend + pre-commit hooks)"
@echo " make extension-install SOURCE=... - Install and enable a trusted Python extension"
@echo " make extension-list - List configured Python extensions"
@echo " make extension-enable NAME=... - Enable an installed extension"
@echo " make extension-disable NAME=... - Disable an extension without uninstalling it"
@echo " make extension-remove NAME=... - Uninstall a managed extension"
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
@echo " make dev - Start all services in development mode (with hot-reloading)"
@echo " make dev-daemon - Start dev services in background (daemon mode)"
@ -84,7 +89,7 @@ check-agent-guidance:
# Install all dependencies
install:
@echo "Installing backend dependencies..."
@cd backend && uv sync
@cd backend && uv sync --locked
@echo "Installing frontend dependencies..."
@cd frontend && $(FRONTEND_PNPM) install
@echo "Installing pre-commit hooks..."
@ -100,6 +105,29 @@ install:
@echo " make setup-sandbox"
@echo ""
extension-install: export DEER_FLOW_EXTENSION_SOURCE := $(value SOURCE)
extension-install:
$(if $(and $(filter command line,$(origin SOURCE)),$(strip $(value SOURCE))),,$(error usage: make extension-install SOURCE=<package|git-url|dir>))
@cd backend && uv run --frozen --no-group extensions deerflow extensions install --source-env __deerflow_extension_source__
extension-list:
@cd backend && uv run --frozen --no-group extensions deerflow extensions list
extension-enable: export DEER_FLOW_EXTENSION_NAME := $(value NAME)
extension-enable:
$(if $(and $(filter command line,$(origin NAME)),$(strip $(value NAME))),,$(error usage: make extension-enable NAME=<extension>))
@cd backend && uv run --frozen --no-group extensions deerflow extensions enable --name-env __deerflow_extension_name__
extension-disable: export DEER_FLOW_EXTENSION_NAME := $(value NAME)
extension-disable:
$(if $(and $(filter command line,$(origin NAME)),$(strip $(value NAME))),,$(error usage: make extension-disable NAME=<extension>))
@cd backend && uv run --frozen --no-group extensions deerflow extensions disable --name-env __deerflow_extension_name__
extension-remove: export DEER_FLOW_EXTENSION_NAME := $(value NAME)
extension-remove:
$(if $(and $(filter command line,$(origin NAME)),$(strip $(value NAME))),,$(error usage: make extension-remove NAME=<extension>))
@cd backend && uv run --frozen --no-group extensions deerflow extensions remove --name-env __deerflow_extension_name__
# Pre-pull sandbox Docker image (optional but recommended)
setup-sandbox:
@$(RUN_WITH_GIT_BASH) ./scripts/setup-sandbox.sh

126
README.md
View File

@ -835,21 +835,117 @@ Advanced deployments can enable pluggable authorization with `authorization.enab
Advanced deployments can also extend the agent runtime itself by declaring zero-argument `AgentMiddleware` classes under `extensions.middlewares` in `config.yaml` or `extensions_config.json`. DeerFlow loads the same configured class list into the lead-agent and subagent pipelines after their built-in runtime middlewares and loop/token guards, but before the terminal-response/safety/clarification tail, so enterprise forks can add domain guardrails, tool-call governance, or observability hooks without patching the built-in middleware builders. Missing packages, invalid classes, and broken modules fail loudly at agent creation. Treat `config.yaml` and `extensions_config.json` as trusted operator-controlled files: middleware paths are code execution, just like custom tool, model, sandbox, guardrail, MCP server, and MCP interceptor declarations. Gateway skill/MCP toggle endpoints preserve this field but do not expose an API write path for `extensions.middlewares`. Per-context parameterization and separate lead-only/subagent-only middleware lists are not supported yet.
For packaged and configurable runtime integrations, use the top-level `plugins:` list in
`config.yaml`. A plugin exposes `module.path:install`, depends only on the standalone
`deerflow-extension-api` contract package, and can register exactly three contribution
kinds: isolated middleware at semantic lead/subagent model or tool positions, lead and
subagent task-lifecycle hooks, and observers for DeerFlow-owned system model calls such as
goal evaluation, memory extraction, title generation, and summarization. DeerFlow allocates
a task-scoped extension store only when one of those contribution kinds is registered and
uses the Gateway's canonical notification loop for lifecycle and system-model callbacks,
including subagents that execute on isolated loops. Plugin order is deterministic,
per-plugin configuration is passed to `install()`, and `required: true` makes load failure
abort startup; otherwise failures are reported and skipped. Plugins load once when the
Gateway app is constructed, so changes require a restart. Because this imports Python code,
`plugins:` is intentionally unavailable through the API-writable
`extensions_config.json`. In Docker deployments, install the plugin in the Gateway image
rather than only in the host environment. See `config.example.yaml` for configuration.
For packaged and configurable runtime integrations, use DeerFlow's extension manager.
It accepts a Python package requirement, a public HTTPS Git URL, or a local directory, installs the
package into the backend's dedicated `extensions` dependency group, updates
`backend/uv.lock`, and adds an enabled entry to the startup-only top-level `plugins:` list
in `config.yaml`:
```bash
# PyPI — pin a version for a reproducible deployment
make extension-install SOURCE="deerflow-extension-acme==1.2.3"
# Public HTTPS Git — pin an immutable commit
make extension-install \
SOURCE="git+https://github.com/acme/deerflow-extension-acme.git@0123456789abcdef0123456789abcdef01234567"
# Local package — an absolute path avoids Make's backend-relative working directory
make extension-install SOURCE="$PWD/examples/deerflow-extension-example"
make extension-list
make extension-disable NAME=acme
make extension-enable NAME=acme
make extension-remove NAME=acme
```
Installation is interactive because package installation can execute Python build hooks,
and the loaded extension later runs with Gateway privileges. For an already-reviewed
source, automation can acknowledge that boundary explicitly with
`cd backend && uv run --frozen --no-group extensions deerflow extensions install <source> --yes`.
The manager requires uv 0.8.0 or newer; the provided Docker images pin uv 0.11.1.
The other direct
commands are `deerflow extensions list`, `enable NAME`, `disable NAME`, and `remove NAME`;
`NAME` may be the extension name, Python distribution, or `module:install` value. Do not
put credentials in a source URL — a URL carrying embedded userinfo or a credential-looking
query parameter is rejected before uv runs. Remote Git sources must use public HTTPS; SSH
Git URLs are rejected because the stock Docker builder does not forward host SSH
credentials. Installing from a loopback URL is allowed for local tooling but warns, because
`127.0.0.1` recorded in the lock is a different machine inside the Docker builder.
A managed package declares exactly one standard PEP 621 entry point:
```toml
[project.entry-points."deerflow.extensions"]
acme = "acme_deerflow_extension:install"
```
That callable uses the standalone `deerflow-extension-api` contract and can register five
contribution kinds: isolated middleware at semantic lead/subagent model or tool positions,
lead and subagent task-lifecycle hooks, observers for DeerFlow-owned model calls that are
not wrapped by middleware model-call hooks (goal, memory, title, and summarization),
Gateway-lifetime services, and eager FastAPI HTTP routers. The contract package has no
framework dependencies; extensions must declare FastAPI, LangChain, LangGraph, or other
libraries they import.
DeerFlow allocates a task-scoped extension store only for middleware, lifecycle, or
system-model observation. Services receive app-scoped runtime dependencies after Gateway
persistence is ready and stop in reverse order after active runs drain. Extension HTTP
routers are mounted after every host route; definite shadows and routes entering the
host's authentication- or CSRF-exempt paths are rejected with attributed diagnostics,
while unrelated routers continue to load. Because the host's public paths are a reserved
prefix list that extensions cannot enter, **every contributed endpoint requires an
authenticated session** — there is currently no way for an extension to expose an
unauthenticated route, so inbound provider webhooks and public status endpoints are out of
scope for this release. Router startup/shutdown hooks, custom lifespans,
Mounts, and WebSocket routes are not accepted; lifetime resources belong in
`ExtensionService`, and WebSocket contributions require a future host-owned
authentication/Origin wrapper. Lifecycle and system-model callbacks use the Gateway's
canonical notification loop, including subagents on isolated loops.
Plugin order is deterministic, per-plugin configuration is passed to `install()`, and
`required: true` makes load failure abort startup; otherwise failures are reported and
skipped. `enabled: false` skips resolution and import. The manager preserves the extension's
private `config` when toggling it and writes `name`, `package`, `use`, `enabled`, and
`required` metadata for managed installs. Installs are recorded `required: false` so a
later broken extension is reported rather than blocking Gateway startup; pass
`extensions install <source> --required` when the package's absence should abort startup
instead. Plugins load once when the Gateway app is
constructed, so install, enable, disable, remove, and manual `plugins:` edits all require a
Gateway restart. Because this imports Python code, `plugins:` is intentionally unavailable
through the API-writable `extensions_config.json`.
Management commands bootstrap the checkout environment without the extension group via
`uv run --frozen --no-group extensions`. Frozen mode lets `disable` and `remove` start even
when an installed extension's remote source or managed snapshot has become unavailable,
while a fresh checkout can still create the non-extension environment from the existing lock. The
manager itself owns the subsequent locked dependency transaction.
Mutations for one checkout are serialized through a process lock. The initial manager
surface is create/remove rather than in-place upgrade: to change an installed source, save
its private `plugins[].config`, remove it, reinstall the new pin, and restore that config.
Local-directory installs are copied into
`backend/extensions/sources/<normalized-distribution>/`; this deployable snapshot, rather
than the original directory, is recorded in the lock. Git metadata, virtual environments,
bytecode caches, symbolic links, and likely credential files are not accepted as snapshot
content. Review what you install anyway: filtering accidental files does not sandbox an
extension, its build backend, or its runtime code.
Local `make dev`/`make start`, Docker development, and the production Gateway image all
consume the same `backend/pyproject.toml` and `backend/uv.lock`. Local and Docker-dev
launchers perform a locked sync before starting; the production image performs that sync
during its build and includes managed local snapshots in the build context. Gateway runtime
commands then use the already-created environment without resolving or installing packages.
Local and Docker-development pre-start syncs may download missing locked artifacts. A
production deployment instead downloads them only during the explicit install or image
build; starting the resulting production Gateway container never resolves or installs
extensions from the network. A local wheel or `file://` Git URL is rejected because it
would not exist in the Docker build context; pass a source directory to create a managed
snapshot instead. Because environment configuration (such as a `UV_FIND_LINKS` wheelhouse)
can still resolve a plain package name to a local wheel, the manager audits every new lock
before enabling the extension: any local reference the stock image build cannot reproduce
rolls back the entire install or removal.
Rebuild with `make up` after changing the managed extension set. See
`config.example.yaml` and the
[reference extension](examples/deerflow-extension-example/) for a complete example.
Gateway-generated follow-up suggestions now normalize both plain-string model output and block/list-style rich content before parsing the JSON array response, so provider-specific content wrappers do not silently drop suggestions.

View File

@ -90,6 +90,11 @@ When making code changes, you MUST update the relevant documentation:
```bash
make check # Check system requirements
make install # Install all dependencies (frontend + backend)
make extension-install SOURCE=... # Install and enable a trusted Python extension
make extension-list # List configured Python extensions
make extension-enable NAME=... # Enable an installed extension
make extension-disable NAME=... # Disable an extension without uninstalling it
make extension-remove NAME=... # Remove a managed extension
make detect-thread-boundaries # Inventory backend executor/thread/event-loop boundaries
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
make start # Start production services locally

View File

@ -4,7 +4,7 @@
# Stage 3 (runtime): clean image without compiler toolchain for production
# UV source image (override for restricted networks that cannot reach ghcr.io)
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.7.20
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.1
FROM ${UV_IMAGE} AS uv-source
# ── Stage 1: Builder ──────────────────────────────────────────────────────────
@ -28,6 +28,7 @@ RUN if [ -n "${APT_MIRROR}" ]; then \
# Install build tools + Node.js (build-essential needed for native Python extensions)
RUN apt-get update && apt-get install -y \
curl \
git \
build-essential \
gnupg \
ca-certificates \
@ -70,7 +71,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
esac; \
extras_flags="$extras_flags --extra $extra"; \
done; \
UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync --extra redis $extras_flags'
UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync --locked --extra redis $extras_flags'
# UTF-8 locale prevents UnicodeEncodeError on Chinese/emoji content in minimal
# containers where locale configuration may be missing and the default encoding is not UTF-8.
@ -88,7 +89,7 @@ COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
EXPOSE 8001
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
# ── Stage 3: Runtime ──────────────────────────────────────────────────────────
# Clean image without build-essential — reduces size (~200 MB) and attack surface.

View File

@ -3,11 +3,11 @@ DEER_FLOW_HOME := $(abspath $(DEER_FLOW_HOME))
BACKEND_SANDBOX_HOME := $(abspath $(CURDIR)/sandbox)
install:
uv sync
uv sync --locked
dev:
mkdir -p "$(DEER_FLOW_HOME)" "$(BACKEND_SANDBOX_HOME)"
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 DEER_FLOW_HOME="$(DEER_FLOW_HOME)" uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 \
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 DEER_FLOW_HOME="$(DEER_FLOW_HOME)" uv run --locked uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 \
--reload \
--reload-include='*.yaml' \
--reload-include='.env' \
@ -17,7 +17,7 @@ dev:
--reload-exclude="$(DEER_FLOW_HOME)"
gateway:
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run --locked uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
test:
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live" tests/ -v

View File

@ -614,6 +614,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
ExtensionLoadError,
initialize_runtime_diagnostics,
load_extensions,
record_runtime_diagnostics,
set_loaded_extensions,
)
@ -738,6 +739,14 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
"""
return {"status": "healthy", "service": "deer-flow-gateway"}
# Extension routes are deliberately last: FastAPI/Starlette dispatches in
# registration order, so every host route (including conditional routes
# and /health) keeps precedence. Definite shadows are rejected with an
# attributed diagnostic while unrelated extension routers still mount.
from deerflow.extensions.gateway import include_contributed_routers
record_runtime_diagnostics(include_contributed_routers(app, loaded_extensions))
return app

View File

@ -26,6 +26,7 @@ from app.gateway.auth_disabled import (
)
from app.gateway.authz import AuthContext, resolve_route_permissions
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
from app.gateway.request_path import get_request_route_path
from deerflow.runtime.user_context import reset_current_user, set_current_user
# Paths that never require authentication.
@ -86,7 +87,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
if _is_public(request.url.path):
if _is_public(get_request_route_path(request)):
return await call_next(request)
internal_user = None

View File

@ -17,10 +17,13 @@ from starlette.types import ASGIApp
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.session_cookie_state import SESSION_COOKIE_ISSUED_STATE_ATTR, SESSION_COOKIE_MAX_AGE_STATE_ATTR, SESSION_COOKIE_SECURE_STATE_ATTR, SKIP_AUTH_CSRF_COOKIE_STATE_ATTR
from app.gateway.auth_disabled import is_auth_disabled
from app.gateway.request_path import get_request_route_path
CSRF_COOKIE_NAME = "csrf_token"
CSRF_HEADER_NAME = "X-CSRF-Token"
CSRF_TOKEN_LENGTH = 64 # bytes
_CSRF_STATE_CHANGING_METHODS: frozenset[str] = frozenset({"POST", "PUT", "DELETE", "PATCH"})
_CSRF_EXEMPT_EXACT_PATHS: frozenset[str] = frozenset({"/api/v1/auth/me"})
def is_secure_request(request: Request) -> bool:
@ -39,19 +42,20 @@ def should_check_csrf(request: Request) -> bool:
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
"""
if request.method not in ("POST", "PUT", "DELETE", "PATCH"):
if request.method not in _CSRF_STATE_CHANGING_METHODS:
return False
if is_auth_disabled():
return False
path = request.url.path.rstrip("/")
# Exempt /api/v1/auth/me endpoint
if path == "/api/v1/auth/me":
route_path = get_request_route_path(request)
path = route_path.rstrip("/")
# Exempt host-owned endpoints that implement their own request posture.
if path in _CSRF_EXEMPT_EXACT_PATHS:
return False
# Inbound webhooks authenticate themselves via provider-specific signatures
# (e.g. GitHub's X-Hub-Signature-256), not the CSRF double-submit cookie.
if request.url.path.startswith("/api/webhooks/"):
if route_path.startswith("/api/webhooks/"):
return False
return True
@ -71,7 +75,7 @@ def is_auth_endpoint(request: Request) -> bool:
Auth endpoints don't need CSRF validation on first call (no token).
"""
return request.url.path.rstrip("/") in _AUTH_EXEMPT_PATHS
return get_request_route_path(request).rstrip("/") in _AUTH_EXEMPT_PATHS
def _host_with_optional_port(hostname: str, port: int | None, scheme: str) -> str:

View File

@ -22,7 +22,7 @@ import logging
import os
from collections.abc import AsyncGenerator, Callable
from contextlib import AsyncExitStack, asynccontextmanager
from typing import TYPE_CHECKING, TypeVar, cast
from typing import TYPE_CHECKING, Any, TypeVar, cast
from fastapi import FastAPI, HTTPException, Request
from langgraph.types import Checkpointer
@ -420,6 +420,9 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
# Initialize persistence engine BEFORE checkpointer so that
# auto-create-database logic runs first (postgres backend).
# Own cleanup before initialization so partial startup and host
# cancellation cannot strand an engine created along the way.
stack.push_async_callback(close_engine)
await init_engine_from_config(config.database)
app.state.checkpointer = await stack.enter_async_context(make_checkpointer(config))
@ -439,6 +442,35 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
app.state.run_store = MemoryRunStore()
app.state.feedback_repo = None
# Services are app-scoped. Capture this app's immutable extension set
# once and close over the same object for teardown; the process-wide
# singleton may be replaced by another app/test before shutdown.
from deerflow.extensions import EMPTY_EXTENSIONS, record_runtime_diagnostics
from deerflow.extensions.gateway import start_services, stop_services
extensions = getattr(app.state, "extensions", EMPTY_EXTENSIONS)
attempted_services: list[tuple[str, Any]] = []
async def stop_extension_services() -> None:
record_runtime_diagnostics(
await stop_services(
extensions,
service_entries=attempted_services,
)
)
# Register cleanup before starting: start() can partially acquire
# resources and then fail or be cancelled.
stack.push_async_callback(stop_extension_services)
record_runtime_diagnostics(
await start_services(
extensions,
config,
sf,
attempted_services=attempted_services,
)
)
from deerflow.persistence.thread_meta import make_thread_store
app.state.thread_store = make_thread_store(sf, app.state.store)
@ -544,7 +576,6 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
),
),
)
await close_engine()
# ---------------------------------------------------------------------------

View File

@ -0,0 +1,9 @@
"""Canonical request-path projection shared by routing security middleware."""
from starlette._utils import get_route_path
from starlette.requests import Request
def get_request_route_path(request: Request) -> str:
"""Return the same root-path-adjusted value Starlette routes match."""
return get_route_path(request.scope)

View File

@ -41,9 +41,13 @@ def install_newer_minor_api(registry: ExtensionRegistry, config: Mapping[str, An
def install_partial_then_raise(registry: ExtensionRegistry, config: Mapping[str, Any]) -> None:
"""Registers two contributors, then fails — exercises rollback."""
registry.middlewares(_Contributor("partial"))
registry.task_lifecycle(_Contributor("partial"))
"""Register every contribution kind, then fail to exercise five-bucket rollback."""
partial = _Contributor("partial")
registry.middlewares(partial)
registry.task_lifecycle(partial)
registry.system_model_observer(partial)
registry.service(partial)
registry.routers((partial,))
raise ValueError("boom")

View File

@ -1,8 +1,8 @@
"""Public contracts for DeerFlow extensions.
This package MUST NOT import `deerflow`. Everything an extension needs to
integrate lives here, so an extension depends on this package alone and can
be released independently of the host.
This package MUST NOT import `deerflow`. Every host contract an extension
needs lives here, while framework imports remain direct extension dependencies;
extensions can therefore be released independently of the host.
"""
from __future__ import annotations
@ -10,6 +10,8 @@ from __future__ import annotations
from deerflow_extension_api.contracts import (
ExtensionInstall,
ExtensionRegistry,
ExtensionRuntimeDeps,
ExtensionService,
HostPolicySnapshot,
MiddlewareContributor,
SystemModelCallObserver,
@ -33,11 +35,9 @@ from deerflow_extension_api.runtime_bridge import (
)
from deerflow_extension_api.state import ExtensionData
#: Contract version. Pre-1.0: the contract surface is observational only
#: (contributors and observers), so minors may break and only patches promise
#: to be additive. From 1.0 on, bump the major on any breaking change; see the
#: spec's evolution rules for what counts as additive.
API_VERSION = "0.1.1"
#: Contract version. Before 1.0, minors may break and patches are additive.
#: From 1.0 on, bump the major for breaking changes.
API_VERSION = "0.1.2"
__all__ = [
"API_VERSION",
@ -47,6 +47,8 @@ __all__ = [
"ExtensionData",
"ExtensionInstall",
"ExtensionRegistry",
"ExtensionRuntimeDeps",
"ExtensionService",
"HostPolicySnapshot",
"MiddlewareContributor",
"MiddlewarePlacement",

View File

@ -10,7 +10,7 @@ Compatibility rules enforced throughout this module:
from __future__ import annotations
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from dataclasses import dataclass, field
from enum import StrEnum
from typing import TYPE_CHECKING, Any, Literal, Protocol, TypeVar, runtime_checkable
@ -152,6 +152,26 @@ class MiddlewareContributor(Protocol):
return ()
# --- Extension services ----------------------------------------------------
@dataclass(frozen=True)
class ExtensionRuntimeDeps:
"""Host capabilities bound after Gateway infrastructure is ready."""
app_store: ExtensionData | None = None
policy: HostPolicySnapshot = field(default_factory=HostPolicySnapshot)
session_factory: Any | None = None
class ExtensionService(Protocol):
async def start(self, deps: ExtensionRuntimeDeps) -> None:
return None
async def stop(self) -> None:
return None
# --- Registration surface ---------------------------------------------------
@ -159,11 +179,10 @@ class MiddlewareContributor(Protocol):
class ExtensionRegistry(Protocol):
"""The write-only registration surface handed to ``install()``.
Structural and minimal on purpose. This first capability slice exposes
middleware contribution only; later slices can add defaulted registration
methods without breaking existing implementations. The host's concrete
registry additionally carries host-only machinery (attribution, positional
rollback, build) that is deliberately absent here.
Structural and minimal on purpose. Every method has a default so additive
contract releases remain compatible with older registry implementations.
The host's concrete registry additionally carries host-only machinery
(attribution, positional rollback, build) that is deliberately absent here.
"""
def middlewares(self, contributor: MiddlewareContributor) -> None:
@ -175,6 +194,18 @@ class ExtensionRegistry(Protocol):
def system_model_observer(self, observer: SystemModelCallObserver) -> None:
return None
def service(self, service: ExtensionService) -> None:
return None
def routers(self, routers: Sequence[Any]) -> None:
"""Register HTTP routers constructed eagerly during extension install.
Router types stay ``Any`` so this contract package has no FastAPI
dependency. The host validates supported route shapes before mounting;
runtime resources belong in a separately registered service.
"""
return None
#: The install() entry point signature every extension exposes.
ExtensionInstall = Callable[[ExtensionRegistry, Mapping[str, Any]], None]

View File

@ -1,6 +1,6 @@
[project]
name = "deerflow-extension-api"
version = "0.1.1"
version = "0.1.2"
description = "Public contracts for DeerFlow extensions"
requires-python = ">=3.12"
# Keep the contract package import-light and independent from the host. Public

View File

@ -1,4 +1,4 @@
### Python Extension System (Runtime Slice)
### Python Extension System (Runtime and Distribution)
Third-party Python packages can expose an `install(registry, config)` function and be
loaded, in deterministic order, from the startup-only top-level `plugins:` list in
@ -7,12 +7,142 @@ through Gateway APIs, while importing Python entry points is an operator-control
execution boundary. A plugin marked `required: true` fails Gateway construction when it
cannot load; optional plugins fail open with attributed diagnostics.
The public package is `packages/extension-api/` and must never import `deerflow`. Its
registry contract exposes exactly three contribution kinds: middleware contributors,
task-lifecycle contributors, and system-model-call observers. Middleware contributions
declare lead/subagent scope, stable order, and a semantic placement (`MODEL_LOGICAL`,
`MODEL_PHYSICAL`, `TOOL_VISIBLE`, `TOOL_RAW`, or `STANDARD`) rather than a fragile list
index. `extensions/stack.py` is the single final composition point; do not inject inside
Packaged extensions use one PEP 621 entry point in the
`deerflow.extensions` group, for example
`example = "deerflow_extension_example:install"`. The operator CLI is dispatched from
the existing `deerflow` console script to `extensions/cli.py` and exposes only these
surfaces: `install SOURCE [--yes]`, `list`, `enable NAME`, `disable NAME`, and
`remove NAME`. `NAME` resolves against the entry-point name, distribution name, or
`module:install` value. The root `make extension-*` targets are convenience wrappers;
because they execute from `backend/`, documentation should use absolute local source
paths with `SOURCE=` unless backend-relative behavior is intentional.
`ExtensionManager` owns the package/config transaction. Install runs a controlled
`uv add --project <backend> --group extensions --no-workspace --no-sync -- <source>`, updates the dedicated
`[dependency-groups].extensions` list and `uv.lock`, discovers exactly one packaging entry
point, and inserts or adopts one
managed `plugins:` record with `name`, `package`, `use`, `enabled`, `required`, and
private `config`. New records are written `required: false`, matching the loader default:
`required: true` turns any later load failure — a broken wheel, a missing native library, a
deleted snapshot — into a Gateway startup abort recoverable only through shell access, so
it is an explicit `install --required` opt-in rather than the managed default. Adoption of
an existing hand-written record preserves whatever `required` the operator already chose.
Enable/disable changes only the host-level `enabled` flag and preserves
private configuration. Remove runs `uv remove --group extensions`, removes the plugin
record, and deletes its managed source snapshot. Install validates the selected config
file before running any uv command, because `uv add`/`uv sync` execute the package's build
backend: a config this manager could never write to must fail before that code runs, not
afterwards through rollback.
Failed install/remove operations restore `pyproject.toml` and `uv.lock` and resynchronize
the restored environment; that second restore runs even when the recovery sync itself fails
(a recovery sync without `--locked` writes a lock while resolving), and a failing recovery
sync reports the original failure alongside it. The restore is deliberately not blanket:
when recovery detects a concurrent external edit to the dependency files or the config it
preserves that edit and raises instead, and `remove` leaves the plugin deactivated in that
case rather than reviving a record whose package declaration may already be gone. A
cancellation skips the recovery sync entirely — the declarations are already restored and
the next locked startup sync reconciles the environment, whereas blocking an interrupt on a
full dependency resolve invites a second interrupt that escapes the handler mid-transaction.
Package mutation is deferred from environment mutation: after `uv add/remove`
updates the declaration and lock, one `uv sync --locked --all-packages` preserves the same
config-/environment-detected optional extras as normal startup. All three uv calls pin the
backend project explicitly and discard UV environment overrides that could redirect the
project, working directory, sync mode, lock policy, or target environment — including
`UV_PYTHON`, which would swap the interpreter that then loads the extension entry point,
and `UV_INSECURE_HOST`, which would remove the TLS validation the HTTPS-only source rule
depends on; index, proxy, cache, and credential-provider settings remain available.
The `--no-workspace` boundary requires uv 0.8.0 or newer. The stock Docker paths pin uv
0.11.1, and the manager fails before mutation when the host uv is older.
All install/remove/enable/disable mutations for a checkout hold the cross-process
`.deer-flow/extension-manager.lock`; remove deactivates config before changing the package
declaration, and rollback preserves a concurrent external config edit instead of replacing
it. The MVP has no in-place upgrade: operators retain private config, remove the old
package, install the new source pin, and restore that config.
Local-directory installs are snapshots, not editable links. The manager validates the
source, derives the destination from the normalized distribution name, and copies it to
`backend/extensions/sources/<distribution>/`. It ignores Git metadata, virtual
environments, Python caches, and bytecode; rejects symbolic links, path-escaping
distribution names, and likely credential files; and the root `.dockerignore` explicitly
re-includes the entire managed tree so package READMEs, native modules, and assets reach
the backend builder. These checks prevent common packaging accidents, not malicious
code. Both Python build hooks and imported extension code execute with Gateway
privileges, so the CLI requires confirmation (or explicit `--yes`) and accepts only
trusted operator sources; source URLs containing embedded credentials are rejected.
Remote direct references are limited to HTTPS, and remote Git sources must use public
Git-over-HTTPS (with loopback HTTP accepted for local tooling). SSH Git URLs are rejected
because the stock Docker builder does not forward host SSH credentials; relative paths and
local wheels must use the managed directory snapshot path instead. Git's SCP-like shorthand
(`git@host:org/repo.git`) carries no URL scheme, so it is detected before the scheme rules
and reported with the same public-HTTPS correction rather than the local-path message.
Local wheel and `file://` sources are rejected because they cannot be reproduced inside the
Docker build context; local code must enter through the directory-snapshot path. Stock
production builds support public package indexes and public HTTPS Git sources reachable by
the builder; authenticated source configuration must not be embedded in the recorded URL.
Source validation alone cannot catch environment-driven resolution (for example a
`UV_FIND_LINKS` wheelhouse turning a plain package requirement into a local wheel
reference), so after every `uv add/remove` the manager audits the new lock before syncing
or enabling anything. Any local reference that the stock backend image build cannot
reproduce — absolute paths, `file:` URLs, or relative paths outside the project root, its
exact workspace members, and the managed `extensions/sources/` snapshots — fails the whole
transaction and rolls back the dependency files, config, snapshot, and environment. A
loopback URL recorded in the lock is warned about rather than rolled back: `127.0.0.1`
inside the image builder is a different machine, so the reference is just as
non-reproducible, but unlike an environment-driven wheelhouse resolution it is a source the
operator typed deliberately. A private-network index is left alone entirely — a builder on
that network can reach it. A
config with duplicate top-level `plugins:` keys is rejected outright rather than managed
against one block while the Gateway reads another.
The managed `plugins:` block is rewritten in place, and both of its boundaries come from
the YAML parser rather than a key-shaped pattern. `AppConfig` allows extra top-level keys,
so a neighbouring section may be named anything YAML accepts (`my.key`, `2fa`, `$schema`, a
non-ASCII word); a pattern that fails to recognize the next key does not fail loudly, it
reports "no next section" and the rewrite replaces that neighbour and its whole subtree.
Trailing comments below a file-final block are preserved for the same reason — the manager
appends `plugins:` at end of file, so that is the steady-state shape.
Dependency synchronization has one lock authority: the manager's `uv add/remove` calls
are the only extension workflow allowed to update `backend/uv.lock`, and each mutation is
followed by the local-source audit described above. The `extensions`
group is included in `[tool.uv].default-groups` alongside `dev`. Root/backend install
targets use `uv sync --locked`; direct backend `make dev`/`make gateway` use
`uv run --locked`; the local full-stack launcher and Docker-dev entrypoint perform one
locked sync and then launch with `uv run --no-sync`; the production Docker builder syncs
the same copied backend project and lock, and both image runtime commands use
`--no-sync`. Thus production may download locked remote artifacts while building an
image, but production container startup never resolves or installs an extension from the
network. Local and Docker-dev pre-start syncs may fetch missing locked artifacts.
`docker/dev-entrypoint.sh` retries a failed sync once after recreating `.venv`, but keeps
`--locked` on the retry: that repairs a broken virtualenv, not a stale lock. A second
failure aborts with recovery instructions instead of starting uvicorn against an
environment that does not match the lock, because startup must never silently resolve
dependencies.
That discipline assumes the uv writing the lock and the uv reading it stay compatible, so
uv is pinned rather than floating: `backend/Dockerfile`'s `UV_IMAGE` is the single source of
truth, both compose defaults repeat it, and every `astral-sh/setup-uv` step pins the same
version so CI exercises the manager against the binary production actually runs. Otherwise a
newer uv can bump `uv.lock`'s `revision` (or make `uv lock --check` disagree with a lock
generated elsewhere) while CI stays green, and the pinned uv in the production image then
fails on the committed lock. `backend/tests/test_ci_uv_version_pin.py` keeps the four
locations in step, which makes a uv upgrade one deliberate, reviewable change.
Rebuild the Gateway image after changing the managed set. Every install, enable, disable,
remove, or config mutation also requires a Gateway restart because plugin loading is
startup-only.
The root management wrappers bootstrap the checkout environment without the extension group
via `uv run --frozen --no-group extensions`, so a broken or disappeared extension source cannot
trigger project validation before the operator can list, disable, or remove it, while a
fresh checkout can still install the non-extension environment from the existing lock. After CLI
entry, the manager owns the controlled locked sync.
The public package is `packages/extension-api/` and must never import `deerflow` or carry
framework dependencies. Extensions declare any FastAPI, LangChain, or LangGraph imports
themselves. Its registry contract exposes five contribution kinds: middleware
contributors, task-lifecycle contributors, system-model-call observers, Gateway-lifetime
services, and eager routers. Middleware contributions declare lead/subagent scope, stable
order, and a semantic placement (`MODEL_LOGICAL`, `MODEL_PHYSICAL`, `TOOL_VISIBLE`,
`TOOL_RAW`, or `STANDARD`) rather than a fragile list index. `extensions/stack.py` is the
single final composition point; do not inject inside
the shared base builder because the lead builder appends more middleware afterward.
`extensions/ordering.py` owns host ordering invariants and validates the final composed
stack. Nothing under `extensions/` may import `agents.middlewares` at module scope: the
@ -32,8 +162,9 @@ capability, so a single-sided wrapper receives a pass-through counterpart; imple
both sides when the extension must observe both synchronous and asynchronous execution
paths.
Lead runs and subagents allocate an `ExtensionData` task store only when at least one of
the three contribution kinds is registered. Middleware and system-call sites recover the
Lead runs and subagents allocate an `ExtensionData` task store only when middleware,
task-lifecycle, or system-model observation is registered; services and routers are
app-scoped and do not allocate one. Middleware and system-call sites recover the
live store through `EXTENSION_TASK_STORE_KEY` / `task_store_from_runtime()`; lifecycle
contributors receive that same store directly. Each task resolves the immutable
loaded-extension snapshot once and binds that same object through task-store allocation,
@ -84,6 +215,47 @@ subagent's isolated loop, while synchronous system callbacks submit fire-and-for
there. Shutdown stops accepting detached observations before the memory shutdown flush and
resets the loop only after in-flight run/subagent drain ordering is complete.
Gateway services start in registration order after the persistence engine and session
factory are ready. Each receives the same `ExtensionRuntimeDeps` snapshot containing the
app store, projected host policy, and session factory. Start failures are attributed and
fail open. The runtime captures `app.state.extensions` once, registers cleanup before the
start batch, and stops the attempted service prefix in reverse order after run/subagent
drain but before store, checkpointer, and engine teardown. Each stop has an independent
bounded timeout; failures do not starve later cleanup. A service-originated
`CancelledError` fails open, while a new cancellation of the host task still propagates
through the exit stack. Runtime diagnostics must be appended through
`record_runtime_diagnostics()` so `app.state.extension_diagnostics` remains the canonical
live list.
Routers are constructed eagerly during `install()` and mounted only after all host routes,
so host handlers always win. The Gateway rejects a contributed router atomically when an
earlier host or extension route provably covers one of its paths for the same HTTP method.
The conservative matcher proves common shadows through normalized parameter names,
static-vs-dynamic matching, known built-in-converter containment, supported compound
segments, full-segment `path` catch-alls, and `Mount` descendants reducible to those same
rules. Relationships requiring general regex-language inclusion are allowed rather than
guessed. Host WebSocket routes do not collide with contributed HTTP routes, but contributed
WebSocket routes are rejected until the host can supply authentication and Origin checks.
Because `include_router()` recompiles contributed routes, preflight projects the converter
registry at include time. Nonstandard converters fail closed against reserved security
paths but otherwise prove a shadow only when their normalized matchers are identical.
Host authentication- and CSRF-exempt paths are reserved, and contributed Mounts, unsupported
route items, startup/shutdown hooks, and custom router lifespans are rejected; lifetime
resources must use `ExtensionService`. Auth and CSRF classify
`get_request_route_path(request)`, the same root-path-adjusted ASGI path Starlette routes
match; do not switch those security predicates back to reconstructed `request.url.path`.
That helper delegates to the private `starlette._utils.get_route_path` on purpose. Its
requirement is not "strip `root_path` correctly" but "return exactly what the router is
matching on", so importing the dispatcher's own implementation keeps the two in lockstep by
construction. Do not vendor a local copy: a private import that disappears fails loudly at
startup, while a stale copy diverges silently at a security boundary. `starlette` is
therefore a declared, bounded direct dependency so the bump is visible in review, and
`tests/test_gateway_request_path.py` pins the agreement independently of the mechanism.
Any preflight, conflict, or include failure rolls back the whole router without preventing
later routers from mounting. Do not introduce a framework-bound `RouterContributor`
contract: the public registry accepts `Sequence[Any]`
to keep extension-api dependency-free.
The memory kind reaches those observers through a different shape, and the difference is
deliberate rather than an oversight to be "aligned" away. DeerMem must stay vendorable and
cannot import the extension API, so it reports through the `MemoryCallbacks.on_memory_llm_result`
@ -96,7 +268,8 @@ The host hook wrapper around the callback stays at `Exception`: only the hook's
are non-fatal, and an observability path must not swallow `SystemExit` / `KeyboardInterrupt`.
Gateway `create_app()` loads plugins once, stores the immutable registry on `app.state`
and in the process-wide singleton, and installs one canonical live diagnostics list.
and in the process-wide singleton, mounts contributed routers last, and installs one
canonical live diagnostics list.
Changing `plugins` requires a restart. Any future contribution kind must be added to the
public contract and host runtime in the same slice; never accept a registration method
that the current host silently ignores.

View File

@ -0,0 +1,127 @@
"""Command-line interface for installing and managing trusted extensions."""
from __future__ import annotations
import argparse
import os
import subprocess
import sys
from collections.abc import Sequence
from pathlib import Path
from deerflow.extensions.manager import ExtensionManager
_NAME_ENV = "DEER_FLOW_EXTENSION_NAME"
_SOURCE_ENV = "DEER_FLOW_EXTENSION_SOURCE"
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="deerflow extensions",
description="Install and manage trusted Python extensions for this DeerFlow checkout.",
)
commands = parser.add_subparsers(dest="command", required=True)
install = commands.add_parser("install", help="install an extension and enable it in config.yaml")
install.add_argument("--source-env", action="store_true", help=argparse.SUPPRESS)
install.add_argument("source", help="local directory, Python package requirement, or Git URL")
install.add_argument(
"--yes",
action="store_true",
help="acknowledge that installing an extension executes trusted third-party code",
)
install.add_argument(
"--required",
action="store_true",
help="abort Gateway startup when this extension fails to load (default: report and skip)",
)
disable = commands.add_parser("disable", help="disable an extension without uninstalling it")
disable.add_argument("--name-env", action="store_true", help=argparse.SUPPRESS)
disable.add_argument("name", help="extension name, distribution, or module:install entry point")
enable = commands.add_parser("enable", help="enable an installed extension")
enable.add_argument("--name-env", action="store_true", help=argparse.SUPPRESS)
enable.add_argument("name", help="extension name, distribution, or module:install entry point")
commands.add_parser("list", help="list configured extensions")
remove = commands.add_parser("remove", help="uninstall an extension and remove its config entry")
remove.add_argument("--name-env", action="store_true", help=argparse.SUPPRESS)
remove.add_argument("name", help="extension name, distribution, or module:install entry point")
return parser
def main(argv: Sequence[str] | None = None) -> int:
args = build_parser().parse_args(list(argv) if argv is not None else None)
try:
root = find_project_root()
configured_path = os.environ.get("DEER_FLOW_CONFIG_PATH")
manager = ExtensionManager(root, config_path=configured_path)
if args.command == "install":
source = _source_argument(args)
trusted = args.yes
if not trusted:
print("Warning: a Python extension executes code with Gateway privileges.")
try:
trusted = input("Install this trusted source? [y/N] ").strip().lower() in {"y", "yes"}
except EOFError:
trusted = False
if not trusted:
print("Extension installation cancelled.", file=sys.stderr)
return 2
installed = manager.install(source, yes=trusted, required=args.required)
print(f"Installed and enabled {installed.name} ({installed.distribution}). Restart DeerFlow to load it.")
return 0
if args.command == "disable":
name = manager.set_enabled(_name_argument(args), enabled=False)
print(f"Disabled {name}. Restart DeerFlow to apply the change.")
return 0
if args.command == "enable":
name = manager.set_enabled(_name_argument(args), enabled=True)
print(f"Enabled {name}. Restart DeerFlow to apply the change.")
return 0
if args.command == "list":
configured = manager.list_configured()
print("NAME\tSTATE\tPACKAGE\tENTRY POINT")
for extension in configured:
state = "enabled" if extension.enabled else "disabled"
print(f"{extension.name}\t{state}\t{extension.distribution}\t{extension.use}")
return 0
if args.command == "remove":
name = manager.remove(_name_argument(args))
print(f"Removed {name}. Restart DeerFlow to apply the change.")
return 0
except (OSError, RuntimeError, ValueError, subprocess.CalledProcessError) as exc:
print(f"extension command failed: {exc}", file=sys.stderr)
return 1
raise AssertionError(f"unhandled extension command: {args.command}")
def _name_argument(args: argparse.Namespace) -> str:
if not args.name_env:
return args.name
name = os.environ.get(_NAME_ENV)
if name is None or not name.strip():
raise ValueError(f"{_NAME_ENV} must contain an extension name")
return name
def _source_argument(args: argparse.Namespace) -> str:
if not args.source_env:
return args.source
source = os.environ.get(_SOURCE_ENV)
if source is None or not source.strip():
raise ValueError(f"{_SOURCE_ENV} must contain an extension source")
return source
def find_project_root() -> Path:
configured = os.environ.get("DEER_FLOW_PROJECT_ROOT")
if configured:
candidate = Path(configured).expanduser().resolve()
if (candidate / "backend" / "pyproject.toml").is_file():
return candidate
raise FileNotFoundError(f"DEER_FLOW_PROJECT_ROOT is not a DeerFlow checkout: {candidate}")
candidates = (Path.cwd(), *Path.cwd().parents)
for candidate in candidates:
candidate = candidate.resolve()
if (candidate / "backend" / "pyproject.toml").is_file():
return candidate
raise FileNotFoundError("could not find a DeerFlow checkout; set DEER_FLOW_PROJECT_ROOT")

View File

@ -0,0 +1,637 @@
"""Gateway-side plumbing for app-scoped extension contributions."""
from __future__ import annotations
import asyncio
import logging
import re
from dataclasses import dataclass
from typing import Any
from deerflow_extension_api import ExtensionRuntimeDeps
from deerflow.extensions.loader import Diagnostic
from deerflow.extensions.policy import project_host_policy
from deerflow.extensions.registry import LoadedExtensions
logger = logging.getLogger(__name__)
DEFAULT_STOP_TIMEOUT_SECONDS = 30.0
_PATH_PARAMETER_GROUP = re.compile(r"\(\?P<[^>]+>")
_PATH_PARAMETER = re.compile(r"{([a-zA-Z_][a-zA-Z0-9_]*)(?::([a-zA-Z_][a-zA-Z0-9_]*))?}")
_RouteMethods = frozenset[str] | None
_RouteScopes = frozenset[str]
_HOST_PUBLIC_PATH_PREFIXES = (
"/health",
"/docs",
"/redoc",
"/openapi.json",
"/api/v1/auth/oauth/",
"/api/v1/auth/callback/",
"/api/webhooks/",
)
_HOST_PUBLIC_EXACT_PATHS = frozenset(
{
"/api/v1/auth/login/local",
"/api/v1/auth/register",
"/api/v1/auth/logout",
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
"/api/v1/auth/providers",
}
)
_HOST_CSRF_EXEMPT_EXACT_PATHS = frozenset({"/api/v1/auth/me"})
_CSRF_STATE_CHANGING_METHODS = frozenset({"POST", "PUT", "DELETE", "PATCH"})
_STANDARD_CONVERTOR_REGEXES = {
"str": "[^/]+",
"path": ".*",
"int": "[0-9]+",
"float": r"[0-9]+(\.[0-9]+)?",
"uuid": "[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}",
}
@dataclass(frozen=True)
class _RouteClaim:
path: str
matcher: str
methods: _RouteMethods
scopes: _RouteScopes
mount_prefix: str | None = None
standard_convertors: bool = True
_RouteOwners = list[tuple[_RouteClaim, str]]
def _cancellation_count() -> int:
task = asyncio.current_task()
return task.cancelling() if task is not None else 0
def _route_path_matcher(route: Any) -> str | None:
path = getattr(route, "path", None)
if path is None:
return None
pattern = getattr(getattr(route, "path_regex", None), "pattern", None)
if not pattern:
return path
return _PATH_PARAMETER_GROUP.sub("(?:", pattern)
def _route_methods(route: Any) -> _RouteMethods:
methods = getattr(route, "methods", None)
return frozenset(methods) if methods else None
def _route_scopes(route: Any) -> _RouteScopes:
from starlette.routing import Mount, Route, WebSocketRoute
if isinstance(route, WebSocketRoute):
return frozenset({"websocket"})
if isinstance(route, Route):
return frozenset({"http"})
if isinstance(route, Mount):
return frozenset({"http", "websocket"})
return frozenset({"http", "websocket"})
def _route_claim(route: Any, *, recompile: bool = False) -> _RouteClaim | None:
from starlette.routing import Mount, compile_path
path = getattr(route, "path", None)
if recompile and path is not None:
path_regex, _path_format, param_convertors = compile_path(path)
matcher = _PATH_PARAMETER_GROUP.sub("(?:", path_regex.pattern)
else:
matcher = _route_path_matcher(route)
param_convertors = getattr(route, "param_convertors", {})
if path is None or matcher is None:
return None
is_mount = isinstance(route, Mount)
return _RouteClaim(
path=path,
matcher=matcher,
methods=_route_methods(route),
scopes=_route_scopes(route),
mount_prefix=path.rstrip("/") if is_mount else None,
standard_convertors=_uses_standard_convertors(
path,
param_convertors,
is_mount=is_mount,
),
)
def _uses_standard_convertors(
path: str,
param_convertors: Any,
*,
is_mount: bool,
) -> bool:
for match in _PATH_PARAMETER.finditer(path):
parameter_name = match.group(1)
convertor_name = match.group(2) or "str"
expected_regex = _STANDARD_CONVERTOR_REGEXES.get(convertor_name)
actual_regex = getattr(param_convertors.get(parameter_name), "regex", None)
if expected_regex is None or actual_regex != expected_regex:
return False
if is_mount:
return getattr(param_convertors.get("path"), "regex", None) == _STANDARD_CONVERTOR_REGEXES["path"]
return True
def _convertor_can_extend_prefix(convertor: str, prefix: str) -> bool:
"""Return a proven built-in-convertor witness beginning with ``prefix``.
Public-route protection is a security boundary, so an unknown custom
convertor fails closed when its value could begin inside a public prefix.
Shadow detection remains separate and continues to allow relationships it
cannot prove.
"""
if convertor == "path":
return True
if convertor == "str":
return "/" not in prefix
if convertor == "int":
return all(character in "0123456789" for character in prefix)
if convertor == "float":
if not prefix:
return True
return bool(re.fullmatch(r"[0-9]+", prefix) or re.fullmatch(r"[0-9]+\.", prefix) or re.fullmatch(r"[0-9]+\.[0-9]+", prefix))
if convertor == "uuid":
groups = (8, 4, 4, 4, 12)
for mask in range(1 << (len(groups) - 1)):
shape = ""
for index, width in enumerate(groups):
shape += "h" * width
if index < len(groups) - 1 and mask & (1 << index):
shape += "-"
if len(prefix) <= len(shape) and all((expected == "h" and character in "0123456789abcdefABCDEF") or character == expected for character, expected in zip(prefix, shape, strict=False)):
return True
return False
return True
def _path_template_can_start_with(path: str, prefix: str) -> bool:
"""Whether a built-in route template has a concrete path under ``prefix``."""
from functools import cache
tokens: list[tuple[str, str]] = []
cursor = 0
for match in _PATH_PARAMETER.finditer(path):
literal = path[cursor : match.start()]
if literal:
tokens.append(("literal", literal))
tokens.append(("parameter", match.group(2) or "str"))
cursor = match.end()
trailing_literal = path[cursor:]
if trailing_literal:
tokens.append(("literal", trailing_literal))
@cache
def can_match(token_index: int, prefix_index: int) -> bool:
if prefix_index == len(prefix):
return True
if token_index == len(tokens):
return False
kind, value = tokens[token_index]
remaining = prefix[prefix_index:]
if kind == "literal":
if value.startswith(remaining):
return True
if remaining.startswith(value):
return can_match(token_index + 1, prefix_index + len(value))
return False
registered_regex = _STANDARD_CONVERTOR_REGEXES.get(value)
if registered_regex is None:
return False
for end_index in range(prefix_index, len(prefix) + 1):
concrete = prefix[prefix_index:end_index]
if re.fullmatch(registered_regex, concrete) is not None and can_match(
token_index + 1,
end_index,
):
return True
return _convertor_can_extend_prefix(value, remaining)
return can_match(0, 0)
def _claim_can_enter_prefix(claim: _RouteClaim, prefix: str) -> bool:
if claim.standard_convertors:
return _path_template_can_start_with(claim.path, prefix)
literal_prefix = claim.path.partition("{")[0]
return claim.path.startswith(prefix) or prefix.startswith(literal_prefix)
def _claim_can_enter_exact_path(claim: _RouteClaim, exact_path: str) -> bool:
"""Whether a route can dispatch to ``exact_path`` plus trailing slashes."""
if not claim.standard_convertors:
return _claim_can_enter_prefix(claim, exact_path)
# Auth and CSRF normalize with rstrip("/"), so two or more trailing
# slashes are just as exempt as one. For built-in convertors, a shortest
# slash-only witness is bounded by the template's literal length because
# ``path`` may be empty and every other built-in rejects slash.
for slash_count in range(len(claim.path) + 2):
if (
re.fullmatch(
claim.matcher,
exact_path + "/" * slash_count,
)
is not None
):
return True
# Custom regex language inclusion is deliberately not guessed at a
# security boundary. If its template can reach the exact-path prefix,
# reject it fail-closed; shadow matching below remains fail-open.
return False
def _router_routes(router: Any) -> list[_RouteClaim]:
from fastapi.routing import _DefaultLifespan
from starlette.routing import Mount, Route, WebSocketRoute
if getattr(router, "on_startup", ()) or getattr(router, "on_shutdown", ()):
raise TypeError("contributed router lifecycle hooks are not supported; register an ExtensionService instead")
lifespan_context = getattr(router, "lifespan_context", None)
if lifespan_context is not None and not isinstance(lifespan_context, _DefaultLifespan):
raise TypeError("contributed router lifespan is not supported; register an ExtensionService instead")
claims: list[_RouteClaim] = []
for route in getattr(router, "routes", []):
if isinstance(route, Mount):
raise TypeError("contributed router contains a Starlette Mount, which FastAPI.include_router() ignores")
if isinstance(route, WebSocketRoute):
raise TypeError("contributed WebSocket routes are not supported until the host can apply authentication and Origin checks")
if not isinstance(route, Route):
raise TypeError(f"contributed router contains an unsupported route item: {type(route).__name__}")
# FastAPI.include_router() reconstructs every route from ``route.path``
# using the converter registry at include time. Preflight must project
# those same semantics, not the router object's older compiled regex.
claim = _route_claim(route, recompile=True)
if claim is not None:
enters_public_exact_path = any(_claim_can_enter_exact_path(claim, public_path) for public_path in _HOST_PUBLIC_EXACT_PATHS)
enters_csrf_exact_path = _methods_overlap(
claim.methods,
_CSRF_STATE_CHANGING_METHODS,
) and any(_claim_can_enter_exact_path(claim, exempt_path) for exempt_path in _HOST_CSRF_EXEMPT_EXACT_PATHS)
enters_public_prefix = any(_claim_can_enter_prefix(claim, public_prefix) for public_prefix in _HOST_PUBLIC_PATH_PREFIXES)
if enters_public_prefix:
raise TypeError(f"contributed route {claim.path} can enter a host public namespace")
if enters_public_exact_path or enters_csrf_exact_path:
raise TypeError(f"contributed route {claim.path} can enter a host-reserved exact path")
claims.append(claim)
return claims
def _methods_overlap(left: _RouteMethods, right: _RouteMethods) -> bool:
return left is None or right is None or not left.isdisjoint(right)
def _dispatches_overlap(left: _RouteClaim, right: _RouteClaim) -> bool:
shared_scopes = left.scopes & right.scopes
if "websocket" in shared_scopes:
return True
return "http" in shared_scopes and _methods_overlap(left.methods, right.methods)
def _path_shape(path: str) -> tuple[tuple[str, ...], tuple[str, ...]]:
literals: list[str] = []
convertors: list[str] = []
cursor = 0
for match in _PATH_PARAMETER.finditer(path):
literals.append(path[cursor : match.start()])
convertors.append(match.group(2) or "str")
cursor = match.end()
literals.append(path[cursor:])
return tuple(literals), tuple(convertors)
def _convertor_covers(owner: str, candidate: str) -> bool:
if owner == candidate:
return True
if owner == "path":
return candidate in {"int", "float", "uuid"}
if owner == "str":
return candidate in {"str", "int", "float", "uuid"}
if owner == "float":
return candidate in {"float", "int"}
return False
def _path_segments(path: str) -> tuple[tuple[str, str], ...] | None:
if not path.startswith("/"):
return None
if path == "/":
return ()
segments: list[tuple[str, str]] = []
for segment in path[1:].split("/"):
match = _PATH_PARAMETER.fullmatch(segment)
if match is not None:
segments.append(("parameter", match.group(2) or "str"))
elif _PATH_PARAMETER.search(segment):
segments.append(("compound", segment))
else:
segments.append(("literal", segment))
return tuple(segments)
def _static_segment_matches(convertor: str, value: str) -> bool:
registered_regex = _STANDARD_CONVERTOR_REGEXES.get(convertor)
return registered_regex is not None and re.fullmatch(registered_regex, value) is not None
def _compound_segment_covers(owner: str, candidate: str) -> bool:
owner_literals, owner_convertors = _path_shape(owner)
candidate_literals, candidate_convertors = _path_shape(candidate)
if (
owner_literals == candidate_literals
and len(owner_convertors) == len(candidate_convertors)
and all(
_convertor_covers(owner_convertor, candidate_convertor)
for owner_convertor, candidate_convertor in zip(
owner_convertors,
candidate_convertors,
strict=True,
)
)
):
return True
if len(owner_convertors) != 1 or owner_convertors[0] != "str":
return False
owner_prefix, owner_suffix = owner_literals
minimum_candidate_length = sum(map(len, candidate_literals)) + sum(0 if convertor == "path" else 1 for convertor in candidate_convertors)
return (
candidate_literals[0].startswith(owner_prefix)
and candidate_literals[-1].endswith(owner_suffix)
and minimum_candidate_length > len(owner_prefix) + len(owner_suffix)
and all(convertor in {"str", "int", "float", "uuid"} for convertor in candidate_convertors)
)
def _compound_is_ascii_digits(compound: str) -> bool:
literals, convertors = _path_shape(compound)
return all(character in "0123456789" for literal in literals for character in literal) and all(convertor == "int" for convertor in convertors)
def _segment_excludes_newline(segment: tuple[str, str]) -> bool:
kind, value = segment
if kind == "literal":
return "\n" not in value
if kind == "parameter":
return value in {"path", "int", "float", "uuid"}
literals, convertors = _path_shape(value)
return all("\n" not in literal for literal in literals) and all(convertor in {"path", "int", "float", "uuid"} for convertor in convertors)
def _compound_segment_matches_static(compound: str, value: str) -> bool:
pattern = ""
cursor = 0
for match in _PATH_PARAMETER.finditer(compound):
pattern += re.escape(compound[cursor : match.start()])
convertor = match.group(2) or "str"
registered_regex = _STANDARD_CONVERTOR_REGEXES.get(convertor)
if registered_regex is None:
return False
pattern += f"(?:{registered_regex})"
cursor = match.end()
pattern += re.escape(compound[cursor:])
return re.fullmatch(pattern, value) is not None
def _segment_covers(owner: tuple[str, str], candidate: tuple[str, str]) -> bool:
owner_kind, owner_value = owner
candidate_kind, candidate_value = candidate
if owner_kind == "literal":
return candidate_kind == "literal" and owner_value == candidate_value
if owner_kind == "parameter" and candidate_kind == "compound":
if owner_value == "path":
return _segment_excludes_newline(candidate)
if owner_value == "str":
return all((match.group(2) or "str") in {"str", "int", "float", "uuid"} for match in _PATH_PARAMETER.finditer(candidate_value))
if owner_value in {"int", "float"}:
return _compound_is_ascii_digits(candidate_value)
return False
if owner_kind == "compound":
if candidate_kind == "literal":
return _compound_segment_matches_static(owner_value, candidate_value)
if candidate_kind == "compound":
return _compound_segment_covers(owner_value, candidate_value)
return False
if candidate_kind == "compound":
return False
if candidate_kind == "literal":
return _static_segment_matches(owner_value, candidate_value)
return _convertor_covers(owner_value, candidate_value)
def _segmented_path_covers(owner_path: str, candidate_path: str) -> bool:
owner = _path_segments(owner_path)
candidate = _path_segments(candidate_path)
if owner is None or candidate is None:
return False
if owner and owner[-1] == ("parameter", "path"):
prefix = owner[:-1]
return (
len(candidate) > len(prefix)
and all(
_segment_covers(owner_segment, candidate_segment)
for owner_segment, candidate_segment in zip(
prefix,
candidate,
strict=False,
)
)
and all(_segment_excludes_newline(candidate_segment) for candidate_segment in candidate[len(prefix) :])
)
return len(owner) == len(candidate) and all(_segment_covers(owner_segment, candidate_segment) for owner_segment, candidate_segment in zip(owner, candidate, strict=True))
def _matcher_covers(owner: _RouteClaim, candidate: _RouteClaim) -> bool:
"""Return whether every candidate path is consumed by an earlier owner."""
if owner.matcher == candidate.matcher:
return True
if not owner.standard_convertors or not candidate.standard_convertors:
return False
owner_literals, owner_convertors = _path_shape(owner.path)
candidate_literals, candidate_convertors = _path_shape(candidate.path)
if not candidate_convertors:
return re.fullmatch(owner.matcher, candidate.path) is not None
if owner.mount_prefix is not None:
if owner.mount_prefix == "":
candidate_segments = _path_segments(candidate.path)
return candidate_segments is not None and all(_segment_excludes_newline(segment) for segment in candidate_segments)
return _segmented_path_covers(
f"{owner.mount_prefix}/{{mount_path:path}}",
candidate.path,
)
if _segmented_path_covers(owner.path, candidate.path):
return True
return (
owner_literals == candidate_literals
and len(owner_convertors) == len(candidate_convertors)
and all(
_convertor_covers(owner_convertor, candidate_convertor)
for owner_convertor, candidate_convertor in zip(
owner_convertors,
candidate_convertors,
strict=True,
)
)
)
def _find_route_clash(
routes: list[_RouteClaim],
owners: _RouteOwners,
candidate_holder: str,
) -> tuple[str, str] | None:
tentative_owners = list(owners)
for route in routes:
for owner, holder in tentative_owners:
if _dispatches_overlap(route, owner) and _matcher_covers(owner, route):
return route.path, holder
tentative_owners.append((route, candidate_holder))
return None
def include_contributed_routers(app: Any, extensions: LoadedExtensions) -> list[Diagnostic]:
"""Mount reachable routers in order and reject definite shadows atomically."""
diagnostics: list[Diagnostic] = []
if not extensions.routers:
return diagnostics
mounted: list[str] = []
owners: _RouteOwners = []
for route in getattr(app, "routes", []):
claim = _route_claim(route)
if claim is not None:
owners.append((claim, "host"))
for source, router in extensions.routers:
try:
routes = _router_routes(router)
if not routes:
raise TypeError(f"contributed router exposes no routes: {router!r}")
clash = _find_route_clash(routes, owners, source)
if clash is not None:
path, holder = clash
message = f"router path {path} is already served by {holder}; this router was not mounted"
diagnostics.append(Diagnostic.error(source, message))
logger.error("Extension %s: %s", source, message)
continue
app_routes = getattr(getattr(app, "router", None), "routes", None)
route_mark = len(app_routes) if isinstance(app_routes, list) else None
try:
app.include_router(router)
except BaseException:
# FastAPI copies one route at a time. If a later copy fails,
# remove every route added by this attempt before either
# continuing fail-open or propagating a host-level exception.
if route_mark is not None:
del app_routes[route_mark:]
raise
for route in routes:
owners.append((route, source))
mounted.append(f"{source} -> {route.path}")
except Exception as exc:
message = f"router could not be mounted; continuing without it: {exc}"
diagnostics.append(Diagnostic.error(source, message))
logger.exception("Extension %s: %s", source, message)
if mounted:
logger.info("Extension routers mounted: %s", "; ".join(mounted))
return diagnostics
async def start_services(
extensions: LoadedExtensions,
app_config: Any,
session_factory: Any | None,
*,
attempted_services: list[tuple[str, Any]] | None = None,
) -> list[Diagnostic]:
"""Start extension services in registration order, failing open per item."""
diagnostics: list[Diagnostic] = []
if not extensions.services:
return diagnostics
deps = ExtensionRuntimeDeps(
app_store=extensions.app_store,
policy=project_host_policy(app_config),
session_factory=session_factory,
)
for entry in extensions.services:
source, service = entry
if attempted_services is not None:
# Record before awaiting start(): a service may acquire resources
# and then fail or be cancelled, so it still owns stop().
attempted_services.append(entry)
cancellation_count = _cancellation_count()
try:
await service.start(deps)
except asyncio.CancelledError:
if _cancellation_count() > cancellation_count:
raise
message = "service start() raised CancelledError; continuing without it"
diagnostics.append(Diagnostic.error(source, message))
logger.exception("Extension %s: %s", source, message)
except Exception as exc:
message = f"service start() failed; continuing without it: {exc}"
diagnostics.append(Diagnostic.error(source, message))
logger.exception("Extension %s: %s", source, message)
return diagnostics
async def stop_services(
extensions: LoadedExtensions,
timeout_seconds: float = DEFAULT_STOP_TIMEOUT_SECONDS,
*,
service_entries: tuple[tuple[str, Any], ...] | list[tuple[str, Any]] | None = None,
) -> list[Diagnostic]:
"""Stop services in reverse order with an independent budget per item."""
diagnostics: list[Diagnostic] = []
entries = extensions.services if service_entries is None else service_entries
for source, service in reversed(entries):
cancellation_count = _cancellation_count()
timeout = asyncio.timeout(timeout_seconds)
try:
async with timeout:
await service.stop()
except TimeoutError as exc:
if timeout.expired():
message = f"service stop() timed out after {timeout_seconds}s; continuing shutdown"
diagnostics.append(Diagnostic.error(source, message))
logger.error("Extension %s: %s", source, message)
else:
message = f"service stop() failed; continuing shutdown: {exc}"
diagnostics.append(Diagnostic.error(source, message))
logger.exception("Extension %s: %s", source, message)
except asyncio.CancelledError:
if _cancellation_count() > cancellation_count:
raise
message = "service stop() raised CancelledError; continuing shutdown"
diagnostics.append(Diagnostic.error(source, message))
logger.exception("Extension %s: %s", source, message)
except Exception as exc:
message = f"service stop() failed; continuing shutdown: {exc}"
diagnostics.append(Diagnostic.error(source, message))
logger.exception("Extension %s: %s", source, message)
return diagnostics

View File

@ -29,6 +29,18 @@ class ExtensionSpec(BaseModel):
model_config = ConfigDict(extra="forbid")
enabled: bool = Field(
default=True,
description="When false, skip the extension without resolving or importing it",
)
name: str | None = Field(
default=None,
description="Stable operator-facing name recorded by the extension manager",
)
package: str | None = Field(
default=None,
description="Installed Python distribution recorded by the extension manager",
)
use: str = Field(description="Entry point path, e.g. 'my_extension:install'")
config: dict[str, Any] = Field(
default_factory=dict,
@ -85,8 +97,8 @@ def _parse_version(version: object) -> tuple[int, ...] | None:
def _compatible(declared: str, current: str) -> bool:
"""One-directional, with the semver window for the contract's life stage.
Pre-1.0 the contract surface is observational only and minors may break,
so the window is same major.minor with patches additive: host >= declared.
Pre-1.0 minors may break, so the window is same major.minor with patches
additive: host >= declared.
From 1.0 on contracts only grow within a major, so a newer host stays
compatible with older extensions while an extension written against a
newer minor is refused it would reach for contract additions the host
@ -130,6 +142,9 @@ def load_extensions(specs: Sequence[ExtensionSpec]) -> tuple[LoadedExtensions, l
loaded_sources: list[str] = []
for spec in specs:
if not spec.enabled:
continue
try:
install = resolve_variable(spec.use)
except Exception as exc:

File diff suppressed because it is too large Load Diff

View File

@ -7,13 +7,14 @@ runtime projection.
from __future__ import annotations
from collections.abc import Iterator
from collections.abc import Iterator, Sequence
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any
from deerflow_extension_api import (
ExtensionData,
ExtensionService,
MiddlewareContributor,
SystemModelCallObserver,
TaskLifecycleContributor,
@ -35,6 +36,8 @@ class LoadedExtensions:
middleware_contributors: tuple[tuple[str, MiddlewareContributor], ...] = ()
task_lifecycle: tuple[tuple[str, TaskLifecycleContributor], ...] = ()
system_model_observers: tuple[tuple[str, SystemModelCallObserver], ...] = ()
services: tuple[tuple[str, ExtensionService], ...] = ()
routers: tuple[tuple[str, Any], ...] = ()
# Precomputed attributes, not methods: hook sites read one attribute to
# short-circuit, so the zero-extension path constructs nothing.
@ -57,6 +60,8 @@ class ExtensionRegistry(ExtensionRegistryContract):
self._middlewares: list[_Entry] = []
self._task_lifecycle: list[_Entry] = []
self._system_model_observers: list[_Entry] = []
self._services: list[_Entry] = []
self._routers: list[_Entry] = []
self._current_source: str | None = None
@contextmanager
@ -83,6 +88,13 @@ class ExtensionRegistry(ExtensionRegistryContract):
def system_model_observer(self, observer: SystemModelCallObserver) -> None:
self._system_model_observers.append((self._source(), observer))
def service(self, service: ExtensionService) -> None:
self._services.append((self._source(), service))
def routers(self, routers: Sequence[Any]) -> None:
source = self._source()
self._routers.extend((source, router) for router in routers)
def discard(self, source: str) -> None:
"""Remove every entry registered by ``source``.
@ -100,18 +112,22 @@ class ExtensionRegistry(ExtensionRegistryContract):
self._middlewares,
self._task_lifecycle,
self._system_model_observers,
self._services,
self._routers,
):
bucket[:] = [entry for entry in bucket if entry[0] != source]
def mark(self) -> tuple[int, int, int]:
def mark(self) -> tuple[int, int, int, int, int]:
"""Snapshot bucket lengths so one install() can be undone positionally."""
return (
len(self._middlewares),
len(self._task_lifecycle),
len(self._system_model_observers),
len(self._services),
len(self._routers),
)
def rollback_to(self, mark: tuple[int, int, int]) -> None:
def rollback_to(self, mark: tuple[int, int, int, int, int]) -> None:
"""Undo every registration made since ``mark``.
Positional rather than source-keyed: two specs may legitimately share
@ -123,6 +139,8 @@ class ExtensionRegistry(ExtensionRegistryContract):
self._middlewares,
self._task_lifecycle,
self._system_model_observers,
self._services,
self._routers,
),
mark,
strict=True,
@ -135,6 +153,8 @@ class ExtensionRegistry(ExtensionRegistryContract):
middleware_contributors=tuple(self._middlewares),
task_lifecycle=tuple(self._task_lifecycle),
system_model_observers=tuple(self._system_model_observers),
services=tuple(self._services),
routers=tuple(self._routers),
has_middleware_contributors=bool(self._middlewares),
has_task_lifecycle=bool(self._task_lifecycle),
has_system_model_observers=bool(self._system_model_observers),

View File

@ -49,6 +49,7 @@ def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="deerflow",
description="DeerFlow terminal workbench — a TUI over the embedded DeerFlow harness.",
epilog="Extension management: deerflow extensions --help",
add_help=True,
)
parser.add_argument("message", nargs="*", help="initial prompt for the TUI, or message in --cli mode")
@ -204,6 +205,7 @@ deerflow — DeerFlow terminal workbench
deerflow --json "question" stream newline-delimited JSON events
deerflow --recursion-limit N --print "question"
set the headless agent-loop super-step limit
deerflow extensions --help install and manage trusted Python extensions
echo "question" | deerflow --print
"""
@ -222,6 +224,10 @@ def _run_overrides(plan: LaunchPlan) -> dict[str, int]:
def main(argv: Sequence[str] | None = None) -> int:
argv = list(sys.argv[1:] if argv is None else argv)
if argv and argv[0] == "extensions":
from deerflow.extensions.cli import main as extensions_main
return extensions_main(argv[1:])
plan = plan_launch(
argv,
stdin_isatty=sys.stdin.isatty(),

View File

@ -11,7 +11,7 @@ dependencies = [
# the contract version it implements, extensions declare ranges. A range
# here would let pip resolve a newer contract package than this harness
# implements, making newer extensions look supported at runtime.
"deerflow-extension-api==0.1.1",
"deerflow-extension-api==0.1.2",
"dotenv>=0.9.9",
"exa-py>=1.0.0",
"httpx>=0.28.0",
@ -31,6 +31,7 @@ dependencies = [
"langgraph-runtime-inmem>=0.28.0",
"markdownify>=1.2.2",
"markitdown[all,xlsx]>=0.0.1a2",
"packaging>=24.2",
"pydantic>=2.12.5",
"pyyaml>=6.0.3",
"readabilipy>=0.3.0",

View File

@ -10,6 +10,16 @@ dependencies = [
"httpx>=0.28.0",
"python-multipart>=0.0.31",
"sse-starlette>=2.1.0",
# Direct dependency on purpose, even though FastAPI already pulls it in:
# app/gateway/request_path.py imports the private
# `starlette._utils.get_route_path` so the auth and CSRF predicates
# classify the exact string Starlette's router matches on. A private
# import is the safest option here because it fails loudly (ImportError at
# startup) instead of silently drifting from the dispatcher at a security
# boundary -- but it does mean a Starlette bump is a security-relevant
# change. Declaring and bounding it here makes that bump visible in the
# diff; tests/test_gateway_request_path.py pins the agreement itself.
"starlette>=1.3.1,<2",
"uvicorn[standard]>=0.34.0",
"lark-oapi>=1.4.0",
"slack-sdk>=3.33.0",
@ -34,6 +44,10 @@ browser = ["deerflow-harness[browser]"]
memory-zh = ["deerflow-harness[memory-zh]"]
[dependency-groups]
# Managed extension packages are added here by `deerflow extensions install`.
# Keeping them separate from development tooling lets every startup mode sync
# the same locked runtime set without promoting those packages to core deps.
extensions = []
dev = [
"blockbuster>=1.5.26,<1.6",
"hypothesis>=6.100,<7",
@ -64,6 +78,7 @@ markers = [
[tool.uv]
index-url = "https://pypi.org/simple"
default-groups = ["dev", "extensions"]
# langgraph-sdk 0.4.2 (pulled in by langgraph 1.2.9 for DeltaChannel) pins
# `websockets<16,>=14`, silently downgrading websockets 16.0 -> 15.0.1. The
# pin is not grounded in any API incompatibility: websockets 16's only

View File

@ -216,6 +216,58 @@ def test_public_auth_path_no_cookie(client):
assert res.status_code == 200
@pytest.mark.parametrize(
"encoded_path",
[
"/api/v1/auth/setup-sta%0Atus",
"/api/v1/auth/setup-sta%0Dtus",
"/api/v1/auth/setup-sta%09tus",
"/api/v1/auth/setup-status%23private",
"/api/v1/auth/setup-status%3Fprivate",
],
)
def test_url_reconstruction_cannot_turn_a_protected_route_path_public(
monkeypatch,
encoded_path,
):
from fastapi import FastAPI
monkeypatch.setenv("DEER_FLOW_AUTH_DISABLED", "")
app = FastAPI()
app.add_middleware(AuthMiddleware)
@app.get("/api/v1/auth/setup-sta{gap}tus")
async def control_gap(gap: str):
return {"gap": gap}
@app.get("/api/v1/auth/setup-status{suffix}")
async def delimiter_suffix(suffix: str):
return {"suffix": suffix}
response = TestClient(app).get(encoded_path)
assert response.status_code == 401
def test_auth_uses_the_same_root_path_projection_as_the_router(monkeypatch):
from fastapi import FastAPI
monkeypatch.setenv("DEER_FLOW_AUTH_DISABLED", "")
child = FastAPI()
child.add_middleware(AuthMiddleware)
@child.get("/health")
async def health():
return {"ok": True}
parent = FastAPI()
parent.mount("/prefix", child)
response = TestClient(parent).get("/prefix/health")
assert response.status_code == 200
def test_protected_auth_path_no_cookie(client):
"""/auth/me requires cookie even though it's under /api/v1/auth/."""
res = client.get("/api/v1/auth/me")

View File

@ -70,6 +70,7 @@ class _FakeRequest:
def __init__(self, path: str, method: str = "POST"):
self.method = method
self.scope = {"path": path, "root_path": ""}
class _URL:
def __init__(self, p):

View File

@ -0,0 +1,111 @@
"""Regression test pinning CI's uv binary to the version production ships.
``ExtensionManager`` is not a consumer of uv the build tool -- it is a program
whose whole job is driving ``uv`` as a subprocess, depending on its CLI
behavior (``--no-workspace``, ``--no-sync``, what ``uv add`` writes into
``[dependency-groups] extensions``) and on the ``uv.lock`` serialization
format. uv is therefore closer to a runtime dependency with a contract than to
incidental tooling.
``backend/Dockerfile`` pins that binary to an exact version, but every
``astral-sh/setup-uv`` step used to install whatever was latest at run time, so
CI exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps ``uv.lock``'s ``revision``,
CI stays green because the same uv reads back what it wrote, and the pinned uv
in the production image then cannot read the committed lock. ``uv lock
--check`` is version-sensitive for the same reason -- it verifies the lock is
what *this* uv would produce, and two versions can emit equivalent but
non-identical output.
Pinning is only half of it; without this test the two pins drift apart again
the next time someone bumps one of them. Upgrading uv should be one explicit
change that touches the Dockerfile, the workflows, and this test together.
"""
from __future__ import annotations
import re
from pathlib import Path
import pytest
import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
BACKEND_DOCKERFILE = REPO_ROOT / "backend" / "Dockerfile"
WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows"
COMPOSE_PATHS = (
REPO_ROOT / "docker" / "docker-compose.yaml",
REPO_ROOT / "docker" / "docker-compose-dev.yaml",
)
_UV_IMAGE_REFERENCE = re.compile(r"ghcr\.io/astral-sh/uv:(?P<version>\d+\.\d+\.\d+)")
_SETUP_UV_ACTION = re.compile(r"^astral-sh/setup-uv@(?P<ref>[^\s]+)$")
def _pinned_uv_version() -> str:
"""The single source of truth: the uv image the backend image builds from."""
match = _UV_IMAGE_REFERENCE.search(BACKEND_DOCKERFILE.read_text(encoding="utf-8"))
assert match is not None, f"{BACKEND_DOCKERFILE} no longer pins a ghcr.io/astral-sh/uv version"
return match.group("version")
def _workflow_paths() -> list[Path]:
return sorted(path for path in WORKFLOWS_DIR.iterdir() if path.suffix in {".yml", ".yaml"})
def _setup_uv_steps() -> list[tuple[str, str, dict]]:
"""Return (workflow name, action ref, step mapping) for each setup-uv step."""
steps: list[tuple[str, str, dict]] = []
for path in _workflow_paths():
workflow = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
for job in (workflow.get("jobs") or {}).values():
if not isinstance(job, dict):
continue
for step in job.get("steps") or []:
if not isinstance(step, dict):
continue
uses = step.get("uses")
if not isinstance(uses, str):
continue
match = _SETUP_UV_ACTION.match(uses.strip())
if match is not None:
steps.append((path.name, match.group("ref"), step))
return steps
def test_the_repository_still_has_setup_uv_steps_to_check():
"""Guard against the other assertions silently passing on an empty list."""
assert _setup_uv_steps(), "no astral-sh/setup-uv steps found; this test needs updating"
def test_every_setup_uv_step_pins_the_uv_version_production_ships():
expected = _pinned_uv_version()
unpinned: list[str] = []
mismatched: list[str] = []
for workflow_name, _ref, step in _setup_uv_steps():
with_block = step.get("with")
version = with_block.get("version") if isinstance(with_block, dict) else None
if version is None:
unpinned.append(f"{workflow_name}: {step.get('name', '<unnamed step>')}")
elif str(version) != expected:
mismatched.append(f"{workflow_name}: {version!r} != {expected!r}")
assert not unpinned, f"setup-uv steps install whatever uv is latest, so CI would not exercise the uv production ships ({expected}): {unpinned}"
assert not mismatched, f"setup-uv steps pin a uv other than the one backend/Dockerfile ships ({expected}): {mismatched}"
def test_every_setup_uv_step_uses_the_same_action_version():
refs = {ref for _workflow_name, ref, _step in _setup_uv_steps()}
assert len(refs) == 1, f"astral-sh/setup-uv is referenced at mixed action versions, so the steps do not share caching or input behavior: {sorted(refs)}"
@pytest.mark.parametrize("compose_path", COMPOSE_PATHS, ids=lambda path: path.name)
def test_compose_uv_image_default_matches_the_backend_dockerfile(compose_path: Path):
"""The compose override defaults must not drift from the image they build."""
expected = _pinned_uv_version()
versions = set(_UV_IMAGE_REFERENCE.findall(compose_path.read_text(encoding="utf-8")))
assert versions, f"{compose_path.name} no longer references a pinned uv image"
assert versions == {expected}, f"{compose_path.name} defaults to uv {sorted(versions)} while backend/Dockerfile ships {expected!r}"

View File

@ -22,9 +22,50 @@ def _make_app() -> FastAPI:
async def protected_mutation():
return {"ok": True}
@app.post("/api/v1/auth/log{gap}in/local")
async def control_gap(gap: str):
return {"gap": gap}
@app.post("/api/v1/auth/me{suffix}")
async def delimiter_suffix(suffix: str):
return {"suffix": suffix}
return app
def test_url_reconstruction_cannot_create_a_csrf_exemption():
client = TestClient(_make_app(), base_url="https://deerflow.example")
for encoded_path in (
"/api/v1/auth/log%0Ain/local",
"/api/v1/auth/log%0Din/local",
"/api/v1/auth/log%09in/local",
"/api/v1/auth/me%23private",
"/api/v1/auth/me%3Fprivate",
):
response = client.post(encoded_path)
assert response.status_code == 403, encoded_path
def test_csrf_uses_the_same_root_path_projection_as_the_router():
child = FastAPI()
child.add_middleware(CSRFMiddleware)
@child.post("/api/v1/auth/login/local")
async def login_local():
return {"ok": True}
parent = FastAPI()
parent.mount("/prefix", child)
response = TestClient(
parent,
base_url="https://deerflow.example",
).post("/prefix/api/v1/auth/login/local")
assert response.status_code == 200
def test_auth_post_rejects_cross_origin_browser_request():
"""CSRF-exempt auth routes must not accept hostile browser origins.

View File

@ -49,6 +49,7 @@ def test_backend_dockerfile_expands_multiple_uv_extras(tmp_path):
assert capture.read_text(encoding="utf-8").splitlines() == [
"sync",
"--locked",
"--extra",
"redis",
"--extra",

View File

@ -18,14 +18,26 @@ REPO_ROOT = Path(__file__).resolve().parents[2]
ENTRYPOINT = REPO_ROOT / "docker" / "dev-entrypoint.sh"
def _run(uv_extras: str | None) -> subprocess.CompletedProcess[str]:
"""Invoke `dev-entrypoint.sh --print-extras` with UV_EXTRAS set."""
def _run(
uv_extras: str | None,
*,
config_path: Path | None = None,
stream_bridge_redis_url: str | None = None,
) -> subprocess.CompletedProcess[str]:
"""Invoke the entrypoint's public extras-resolution dry run."""
env = os.environ.copy()
env.pop("UV_EXTRAS", None)
env.pop("DEER_FLOW_CONFIG_PATH", None)
env.pop("DEER_FLOW_STREAM_BRIDGE_REDIS_URL", None)
if uv_extras is not None:
env["UV_EXTRAS"] = uv_extras
if config_path is not None:
env["DEER_FLOW_CONFIG_PATH"] = str(config_path)
if stream_bridge_redis_url is not None:
env["DEER_FLOW_STREAM_BRIDGE_REDIS_URL"] = stream_bridge_redis_url
return subprocess.run(
["sh", str(ENTRYPOINT), "--print-extras"],
cwd=ENTRYPOINT.parent,
env=env,
capture_output=True,
text=True,
@ -54,12 +66,32 @@ def test_entrypoint_excludes_runtime_state_from_uvicorn_reload():
assert "--reload-exclude=/app/backend/.deer-flow" in content
def test_failed_sync_recreates_a_clean_virtual_environment():
content = ENTRYPOINT.read_text(encoding="utf-8")
assert "uv venv --clear .venv" in content
assert "uv venv --allow-existing .venv" not in content
def test_no_uv_extras_yields_empty_flags():
proc = _run(None)
assert proc.returncode == 0
assert proc.stdout.strip() == ""
def test_no_explicit_extras_uses_the_runtime_selected_config(tmp_path: Path):
config_path = tmp_path / "deployment.yaml"
config_path.write_text(
"database:\n backend: postgres\ntools:\n - name: browser_navigate\n",
encoding="utf-8",
)
proc = _run(None, config_path=config_path)
assert proc.returncode == 0, proc.stderr
assert proc.stdout.strip() == "--extra browser --extra postgres"
def test_single_extra():
proc = _run("postgres")
assert proc.returncode == 0
@ -84,6 +116,26 @@ def test_multi_extra_mixed_separators():
assert proc.stdout.strip() == "--extra postgres --extra ollama"
def test_explicit_extras_override_config_and_are_deduplicated(tmp_path: Path):
config_path = tmp_path / "deployment.yaml"
config_path.write_text("database:\n backend: postgres\n", encoding="utf-8")
proc = _run("redis,redis browser redis", config_path=config_path)
assert proc.returncode == 0, proc.stderr
assert proc.stdout.strip() == "--extra redis --extra browser"
def test_explicit_extras_keep_runtime_required_redis_without_duplicates():
proc = _run(
"postgres,postgres",
stream_bridge_redis_url="redis://redis:6379/0",
)
assert proc.returncode == 0, proc.stderr
assert proc.stdout.strip() == "--extra postgres --extra redis"
def test_empty_string_yields_empty_flags():
proc = _run("")
assert proc.returncode == 0
@ -114,3 +166,97 @@ def test_underscores_and_hyphens_in_name_are_allowed():
proc = _run("post_gres,post-gres")
assert proc.returncode == 0
assert proc.stdout.strip() == "--extra post_gres --extra post-gres"
# ── Dependency-sync failure branch ──────────────────────────────────────────
#
# The self-heal retry reuses `--locked`, so a lock that genuinely disagrees with
# the environment fails the same way twice. `set -e` already stops the script
# there -- these tests pin that the handoff to uvicorn is never reached, and
# that the operator is told what to do instead of reading a bare uv traceback.
#
# `/app/backend` only exists inside the container, so the sync block is sliced
# out of the real script and run against a stub `uv`. The block is read from
# the file rather than duplicated here: editing the script changes what runs.
_SYNC_BLOCK_START = "# ── Sync dependencies (with self-heal) ──"
_SYNC_BLOCK_END = "# ── Hand off to uvicorn ──"
_STUB_UV_ALWAYS_FAILS_SYNC = """#!/bin/sh
# `uv venv` succeeds so the retry is actually reached; every sync fails.
case "$1" in
sync) exit 1 ;;
*) exit 0 ;;
esac
"""
_STUB_UV_SUCCEEDS = """#!/bin/sh
exit 0
"""
_STUB_UV_FAILS_THEN_SUCCEEDS = """#!/bin/sh
case "$1" in
sync)
if [ -f "$STUB_UV_STATE/first_sync_done" ]; then exit 0; fi
: > "$STUB_UV_STATE/first_sync_done"
exit 1
;;
*) exit 0 ;;
esac
"""
def _sync_block() -> str:
content = ENTRYPOINT.read_text(encoding="utf-8")
start = content.index(_SYNC_BLOCK_START)
end = content.index(_SYNC_BLOCK_END)
return content[start:end]
def _run_sync_block(tmp_path: Path, stub_uv: str) -> subprocess.CompletedProcess[str]:
"""Execute the script's real sync block with a stubbed `uv` on PATH."""
bin_dir = tmp_path / "bin"
bin_dir.mkdir()
uv_stub = bin_dir / "uv"
uv_stub.write_text(stub_uv, encoding="utf-8")
uv_stub.chmod(0o755)
state_dir = tmp_path / "state"
state_dir.mkdir()
# `cd` is shadowed because /app/backend does not exist outside the
# container; everything else in the block runs verbatim.
script = f'set -e\ncd() {{ :; }}\nEXTRAS_FLAGS=""\n{_sync_block()}\necho "REACHED_HANDOFF"\n'
env = os.environ.copy()
env["PATH"] = f"{bin_dir}{os.pathsep}{env['PATH']}"
env["STUB_UV_STATE"] = str(state_dir)
return subprocess.run(["sh", "-c", script], capture_output=True, text=True, check=False, env=env, cwd=tmp_path)
def test_successful_sync_reaches_the_uvicorn_handoff(tmp_path: Path):
proc = _run_sync_block(tmp_path, _STUB_UV_SUCCEEDS)
assert proc.returncode == 0, proc.stderr
assert "REACHED_HANDOFF" in proc.stdout
def test_self_heal_retry_still_reaches_the_handoff(tmp_path: Path):
proc = _run_sync_block(tmp_path, _STUB_UV_FAILS_THEN_SUCCEEDS)
assert proc.returncode == 0, proc.stderr
assert "recreating .venv" in proc.stdout
assert "REACHED_HANDOFF" in proc.stdout
def test_failed_retry_aborts_before_starting_uvicorn(tmp_path: Path):
proc = _run_sync_block(tmp_path, _STUB_UV_ALWAYS_FAILS_SYNC)
assert proc.returncode != 0, "startup continued past an unsatisfied lock"
assert "REACHED_HANDOFF" not in proc.stdout, "uvicorn would have been started against a stale or missing environment"
def test_failed_retry_tells_the_operator_how_to_recover(tmp_path: Path):
proc = _run_sync_block(tmp_path, _STUB_UV_ALWAYS_FAILS_SYNC)
assert "make install" in proc.stderr, f"no recovery guidance on stderr: {proc.stderr!r}"

View File

@ -21,6 +21,8 @@ from deerflow_extension_api import (
ExtensionData,
ExtensionInstall,
ExtensionRegistry,
ExtensionRuntimeDeps,
ExtensionService,
HostPolicySnapshot,
MiddlewareContributor,
MiddlewarePlacement,
@ -63,6 +65,7 @@ def test_middleware_placement_defaults():
"cls",
[
HostPolicySnapshot,
ExtensionRuntimeDeps,
AgentBuildContext,
TaskInfo,
SystemModelRequest,
@ -77,7 +80,7 @@ def test_every_dataclass_is_frozen(cls):
@pytest.mark.parametrize(
"cls",
[HostPolicySnapshot, TaskInfo, SystemModelRequest, SystemModelResult],
[HostPolicySnapshot, ExtensionRuntimeDeps, TaskInfo, SystemModelRequest, SystemModelResult],
)
def test_additive_dataclasses_are_constructible_with_required_fields_only(cls):
"""Fields added later must carry defaults, or old extensions break on upgrade.
@ -114,6 +117,7 @@ def test_agent_build_context_optional_fields_keep_their_defaults():
"protocol",
[
ExtensionRegistry,
ExtensionService,
MiddlewareContributor,
TaskLifecycleContributor,
SystemModelCallObserver,
@ -195,16 +199,18 @@ def test_system_model_request_normalizes_messages_into_an_immutable_sequence():
assert SystemModelRequest(messages=("already", "a", "tuple")).messages == ("already", "a", "tuple")
def test_future_contribution_points_are_not_advertised_before_the_host_supports_them():
"""A merged slice must not silently accept registrations it cannot run."""
def test_gateway_contribution_points_are_part_of_the_public_surface():
import deerflow_extension_api
for name in (
"ExtensionRuntimeDeps",
"ExtensionService",
):
assert name not in deerflow_extension_api.__all__
assert not hasattr(deerflow_extension_api, name)
assert name in deerflow_extension_api.__all__
assert hasattr(deerflow_extension_api, name)
assert callable(ExtensionRegistry.service)
assert callable(ExtensionRegistry.routers)
assert not hasattr(deerflow_extension_api, "RouterContributor")
def test_task_store_from_runtime_reads_the_host_key():
@ -263,6 +269,15 @@ def test_distribution_marks_the_contract_package_as_typed():
assert marker.is_file()
def test_contract_package_keeps_runtime_dependencies_empty():
import tomllib
from pathlib import Path
pyproject = Path(__file__).parent.parent / "packages" / "extension-api" / "pyproject.toml"
assert tomllib.loads(pyproject.read_text())["project"]["dependencies"] == []
def test_harness_pins_the_contract_package_exactly():
"""The version contract (extension-system design): the host pins the
contract package exactly, extensions use ranges. A range here would let an
@ -289,5 +304,17 @@ def test_runtime_api_version_matches_the_installed_contract_package():
"""Every additive contract slice bumps both gates together."""
from importlib.metadata import version
assert API_VERSION == "0.1.1"
assert API_VERSION == "0.1.2"
assert API_VERSION == version("deerflow-extension-api")
def test_extension_service_contract_is_public_and_defaults_to_noop():
class _Bare:
pass
deps = ExtensionRuntimeDeps()
assert deps.app_store is None
assert deps.session_factory is None
assert asyncio.run(ExtensionService.start(_Bare(), deps)) is None
assert asyncio.run(ExtensionService.stop(_Bare())) is None

View File

@ -85,6 +85,53 @@ def test_create_app_exposes_one_canonical_live_diagnostics_list(monkeypatch):
]
def test_create_app_mounts_extension_routers_after_all_host_routes(monkeypatch):
from fastapi import APIRouter
from fastapi.testclient import TestClient
import deerflow.extensions as extensions_module
conflict = APIRouter()
good = APIRouter()
@conflict.get("/health")
async def shadow_health():
return {"status": "extension"}
@good.get("/api/extension-test/ping")
async def ping():
return {"ok": True}
registry = ExtensionRegistry()
with registry.attributed_to("router:install"):
registry.routers((conflict, good))
loaded = registry.build()
monkeypatch.setattr(
extensions_module,
"load_extensions",
lambda plugins: (loaded, []),
)
from app.gateway.app import create_app
app = create_app()
paths = [getattr(route, "path", None) for route in app.routes]
assert paths.count("/health") == 1
assert "/api/extension-test/ping" in paths
assert len(app.state.extension_diagnostics) == 1
assert app.state.extension_diagnostics == extensions_module.get_runtime_diagnostics()
assert app.state.extension_diagnostics[0].source == "router:install"
assert "host" in app.state.extension_diagnostics[0].message
client = TestClient(app)
assert client.get("/health").json() == {
"status": "healthy",
"service": "deer-flow-gateway",
}
assert client.get("/api/extension-test/ping").status_code == 401
def test_create_app_fails_open_when_extension_loading_raises_unexpectedly(monkeypatch):
import deerflow.extensions as extensions_module

View File

@ -0,0 +1,298 @@
"""Contracts for installing declared Python extensions in every startup mode."""
from __future__ import annotations
import os
import re
import subprocess
import tomllib
import zipfile
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
BACKEND_ROOT = REPO_ROOT / "backend"
def _make_recipe(path: Path, target: str) -> str:
content = path.read_text(encoding="utf-8")
match = re.search(rf"^{re.escape(target)}:[^\n]*\n(?P<recipe>(?:\t[^\n]*\n)+)", content, re.MULTILINE)
assert match is not None, f"missing {target!r} target in {path}"
return match.group("recipe")
def test_extensions_dependency_group_is_part_of_the_default_sync() -> None:
project = tomllib.loads((BACKEND_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert isinstance(project["dependency-groups"]["extensions"], list)
assert set(project["tool"]["uv"]["default-groups"]) == {"dev", "extensions"}
def test_backend_make_targets_never_mutate_the_extension_lock() -> None:
makefile = BACKEND_ROOT / "Makefile"
assert "uv sync --locked" in _make_recipe(makefile, "install")
for target in ("dev", "gateway"):
recipe = _make_recipe(makefile, target)
assert "uv run --locked uvicorn" in recipe, target
assert "uv sync" not in recipe, target
def test_root_local_startup_syncs_the_locked_backend_before_runtime() -> None:
root_install = _make_recipe(REPO_ROOT / "Makefile", "install")
serve = (REPO_ROOT / "scripts" / "serve.sh").read_text(encoding="utf-8")
assert "cd backend && uv sync --locked" in root_install
sync_at = serve.find("uv sync --locked")
runtime_at = serve.find("uv run --no-sync uvicorn app.gateway.app:app")
assert sync_at != -1
assert runtime_at > sync_at
def test_root_makefile_exposes_extension_management_commands() -> None:
makefile = REPO_ROOT / "Makefile"
install = _make_recipe(makefile, "extension-install")
assert "deerflow extensions install" in install
assert "--source-env __deerflow_extension_source__" in install
assert "DEER_FLOW_EXTENSION_SOURCE" not in install
assert "$(SOURCE)" not in install
assert "uv run --frozen --no-group extensions" in install
assert "--yes" not in install
for target, command in (
("extension-list", "deerflow extensions list"),
("extension-enable", "deerflow extensions enable"),
("extension-disable", "deerflow extensions disable"),
("extension-remove", "deerflow extensions remove"),
):
recipe = _make_recipe(makefile, target)
assert command in recipe
assert "uv run --frozen --no-group extensions" in recipe
def test_extension_management_bootstrap_does_not_resolve_a_broken_extension_source() -> None:
makefile = REPO_ROOT / "Makefile"
for target in (
"extension-install",
"extension-list",
"extension-enable",
"extension-disable",
"extension-remove",
):
recipe = _make_recipe(makefile, target)
assert "uv run --frozen --no-group extensions" in recipe, target
assert "--locked" not in recipe, target
def test_frozen_management_bootstrap_installs_core_without_reading_a_missing_extension(
tmp_path: Path,
) -> None:
project = tmp_path / "backend"
wheels = tmp_path / "wheels"
project.mkdir()
wheels.mkdir()
def _write_wheel(distribution: str, module: str) -> Path:
wheel = wheels / f"{distribution.replace('-', '_')}-1.0.0-py3-none-any.whl"
dist_info = f"{distribution.replace('-', '_')}-1.0.0.dist-info"
records = {
f"{module}/__init__.py": "VALUE = 'installed'\n",
f"{dist_info}/METADATA": f"Metadata-Version: 2.1\nName: {distribution}\nVersion: 1.0.0\n",
f"{dist_info}/WHEEL": "Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: py3-none-any\n",
}
records[f"{dist_info}/RECORD"] = "".join(f"{name},,\n" for name in (*records, f"{dist_info}/RECORD"))
with zipfile.ZipFile(wheel, "w") as archive:
for name, content in records.items():
archive.writestr(name, content)
return wheel
core_wheel = _write_wheel("bootstrap-core", "bootstrap_core")
missing_extension = _write_wheel("broken-extension", "broken_extension")
(project / "pyproject.toml").write_text(
f"""\
[project]
name = "bootstrap-host"
version = "0.0.0"
requires-python = ">=3.12"
dependencies = ["bootstrap-core @ {core_wheel.as_uri()}"]
[dependency-groups]
extensions = ["broken-extension @ {missing_extension.as_uri()}"]
""",
encoding="utf-8",
)
environment = os.environ.copy()
environment["UV_CACHE_DIR"] = str(tmp_path / "uv-cache")
subprocess.run(["uv", "lock"], cwd=project, env=environment, check=True, capture_output=True)
missing_extension.unlink()
completed = subprocess.run(
[
"uv",
"run",
"--frozen",
"--no-group",
"extensions",
"python",
"-c",
"import bootstrap_core; assert bootstrap_core.VALUE == 'installed'",
],
cwd=project,
env=environment,
check=False,
capture_output=True,
text=True,
)
assert completed.returncode == 0, completed.stderr
def test_root_extension_shortcuts_are_cross_platform_and_keep_trust_confirmation() -> None:
makefile = REPO_ROOT / "Makefile"
for target in (
"extension-install",
"extension-enable",
"extension-disable",
"extension-remove",
):
recipe = _make_recipe(makefile, target)
assert "test -n" not in recipe, target
assert "usage: make" in recipe, target
assert "--yes" not in _make_recipe(makefile, "extension-install")
def test_root_extension_shortcuts_reject_ambient_environment_arguments() -> None:
environment = os.environ.copy()
for target, variable in (("extension-install", "SOURCE"), ("extension-enable", "NAME")):
environment[variable] = "ambient-value"
result = subprocess.run(
["make", "--no-print-directory", "-n", target],
cwd=REPO_ROOT,
env=environment,
capture_output=True,
text=True,
check=False,
)
assert result.returncode == 2, target
assert f"usage: make {target}" in result.stderr, target
@pytest.mark.parametrize(
("target", "variable", "env_option"),
[
("extension-install", "SOURCE", "--source-env __deerflow_extension_source__"),
("extension-enable", "NAME", "--name-env __deerflow_extension_name__"),
("extension-disable", "NAME", "--name-env __deerflow_extension_name__"),
("extension-remove", "NAME", "--name-env __deerflow_extension_name__"),
],
)
def test_root_extension_shortcuts_keep_command_line_arguments_out_of_the_shell_recipe(
target: str,
variable: str,
env_option: str,
) -> None:
marker = "EXTENSION_WRAPPER_INJECTION"
malicious_value = f'"; printf {marker}; #$(shell printf {marker})'
result = subprocess.run(
["make", "--no-print-directory", "-n", target, f"{variable}={malicious_value}"],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=False,
)
assert result.returncode == 0, result.stderr
assert marker not in result.stdout
assert env_option in result.stdout
assert "$DEER_FLOW_EXTENSION_" not in result.stdout
assert "%DEER_FLOW_EXTENSION_" not in result.stdout
@pytest.mark.parametrize(
("target", "variable", "env_option"),
[
("extension-install", "SOURCE", "--source-env __deerflow_extension_source__"),
("extension-enable", "NAME", "--name-env __deerflow_extension_name__"),
("extension-disable", "NAME", "--name-env __deerflow_extension_name__"),
("extension-remove", "NAME", "--name-env __deerflow_extension_name__"),
],
)
def test_root_extension_shortcuts_keep_values_out_of_the_cmd_recipe_on_windows(
target: str,
variable: str,
env_option: str,
) -> None:
marker = "EXTENSION_WRAPPER_INJECTION"
malicious_value = f'"; printf {marker}; #$(shell printf {marker})'
result = subprocess.run(
[
"make",
"--no-print-directory",
"-n",
"OS=Windows_NT",
target,
f"{variable}={malicious_value}",
],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=False,
)
assert result.returncode == 0, result.stderr
assert marker not in result.stdout
assert env_option in result.stdout
assert "$DEER_FLOW_EXTENSION_" not in result.stdout
assert "%DEER_FLOW_EXTENSION_" not in result.stdout
def test_docker_dev_entrypoint_syncs_the_lock_before_runtime() -> None:
entrypoint = (REPO_ROOT / "docker" / "dev-entrypoint.sh").read_text(encoding="utf-8")
sync_at = entrypoint.find("uv sync --locked --all-packages")
runtime_at = entrypoint.find("uv run --no-sync uvicorn app.gateway.app:app")
assert sync_at != -1
assert runtime_at > sync_at
def test_docker_image_builds_from_the_lock_and_never_syncs_at_runtime() -> None:
dockerfile = (BACKEND_ROOT / "Dockerfile").read_text(encoding="utf-8")
production_compose = (REPO_ROOT / "docker" / "docker-compose.yaml").read_text(encoding="utf-8")
assert "uv sync --locked --extra redis" in dockerfile
assert "ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.1" in dockerfile
assert dockerfile.count("uv run --no-sync uvicorn app.gateway.app:app") == 2
assert "uv run --no-sync uvicorn app.gateway.app:app" in production_compose
for compose_name in ("docker-compose.yaml", "docker-compose-dev.yaml"):
compose = (REPO_ROOT / "docker" / compose_name).read_text(encoding="utf-8")
assert "ghcr.io/astral-sh/uv:0.11.1" in compose
def test_docker_context_keeps_every_managed_extension_artifact() -> None:
dockerignore = (REPO_ROOT / ".dockerignore").read_text(encoding="utf-8")
reinclusion = "!backend/extensions/sources/**"
assert reinclusion in dockerignore
assert dockerignore.rfind(reinclusion) > dockerignore.rfind("*.md")
assert dockerignore.rfind(reinclusion) > dockerignore.rfind("assets/")
assert dockerignore.rfind(reinclusion) > dockerignore.rfind("*.so")
def test_docker_builder_can_resolve_locked_git_extensions() -> None:
dockerfile = (BACKEND_ROOT / "Dockerfile").read_text(encoding="utf-8")
builder_packages = re.search(
r"apt-get install -y \\\n(?P<packages>.*?) && mkdir -p /etc/apt/keyrings",
dockerfile,
re.DOTALL,
)
assert builder_packages is not None
assert re.search(r"^\s*git\s+\\$", builder_packages.group("packages"), re.MULTILINE)

View File

@ -0,0 +1,985 @@
"""Gateway binding tests for app-scoped extension contributions."""
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from types import SimpleNamespace
import pytest
from deerflow.extensions.registry import ExtensionRegistry
class _Service:
def __init__(
self,
name: str,
events: list[str],
*,
fail_start: bool = False,
fail_stop: bool = False,
) -> None:
self.name = name
self.events = events
self.fail_start = fail_start
self.fail_stop = fail_stop
self.deps = None
async def start(self, deps) -> None:
self.events.append(f"start:{self.name}")
self.deps = deps
if self.fail_start:
raise RuntimeError(f"{self.name} failed")
async def stop(self) -> None:
self.events.append(f"stop:{self.name}")
if self.fail_stop:
raise RuntimeError(f"{self.name} failed")
@pytest.mark.asyncio
async def test_services_start_in_order_with_narrow_deps_and_fail_open():
from deerflow.extensions.gateway import start_services
events: list[str] = []
first = _Service("first", events, fail_start=True)
second = _Service("second", events)
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(first)
with registry.attributed_to("second:install"):
registry.service(second)
extensions = registry.build()
session_factory = object()
config = SimpleNamespace(
token_budget=SimpleNamespace(enabled=False, max_tokens=999),
subagents=SimpleNamespace(max_total_per_run=4),
)
diagnostics = await start_services(extensions, config, session_factory)
assert events == ["start:first", "start:second"]
assert first.deps is second.deps
assert second.deps.app_store is extensions.app_store
assert second.deps.session_factory is session_factory
assert second.deps.policy.token_budget_enabled is False
assert second.deps.policy.max_total_tokens is None
assert second.deps.policy.max_subagents_per_run == 4
assert [(diagnostic.source, diagnostic.level) for diagnostic in diagnostics] == [("first:install", "error")]
@pytest.mark.asyncio
async def test_service_originated_cancelled_error_does_not_abort_start_batch():
from deerflow.extensions.gateway import start_services
events: list[str] = []
class _CancelsItself(_Service):
async def start(self, deps) -> None:
self.events.append(f"start:{self.name}")
raise asyncio.CancelledError()
first = _CancelsItself("first", events)
second = _Service("second", events)
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(first)
with registry.attributed_to("second:install"):
registry.service(second)
diagnostics = await start_services(registry.build(), SimpleNamespace(), None)
assert events == ["start:first", "start:second"]
assert len(diagnostics) == 1
assert diagnostics[0].source == "first:install"
assert "CancelledError" in diagnostics[0].message
@pytest.mark.asyncio
async def test_services_stop_in_reverse_order_and_fail_open():
from deerflow.extensions.gateway import stop_services
events: list[str] = []
first = _Service("first", events)
second = _Service("second", events, fail_stop=True)
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(first)
with registry.attributed_to("second:install"):
registry.service(second)
diagnostics = await stop_services(registry.build())
assert events == ["stop:second", "stop:first"]
assert [(diagnostic.source, diagnostic.level) for diagnostic in diagnostics] == [("second:install", "error")]
@pytest.mark.asyncio
async def test_service_originated_cancelled_error_does_not_abort_stop_batch():
from deerflow.extensions.gateway import stop_services
events: list[str] = []
class _CancelsItself(_Service):
async def stop(self) -> None:
self.events.append(f"stop:{self.name}")
raise asyncio.CancelledError()
first = _Service("first", events)
second = _CancelsItself("second", events)
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(first)
with registry.attributed_to("second:install"):
registry.service(second)
diagnostics = await stop_services(registry.build())
assert events == ["stop:second", "stop:first"]
assert len(diagnostics) == 1
assert diagnostics[0].source == "second:install"
assert "CancelledError" in diagnostics[0].message
@pytest.mark.asyncio
async def test_each_service_stop_has_its_own_timeout_budget():
from deerflow.extensions.gateway import stop_services
events: list[str] = []
class _HangingService(_Service):
async def stop(self) -> None:
self.events.append(f"stop:{self.name}")
await asyncio.Event().wait()
first = _Service("first", events)
second = _HangingService("second", events)
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(first)
with registry.attributed_to("second:install"):
registry.service(second)
diagnostics = await stop_services(registry.build(), timeout_seconds=0.01)
assert events == ["stop:second", "stop:first"]
assert len(diagnostics) == 1
assert diagnostics[0].source == "second:install"
assert "timed out" in diagnostics[0].message
@pytest.mark.asyncio
async def test_service_originated_timeout_error_is_reported_as_failure_not_budget_expiry():
from deerflow.extensions.gateway import stop_services
events: list[str] = []
class _RaisesTimeout(_Service):
async def stop(self) -> None:
self.events.append(f"stop:{self.name}")
raise TimeoutError("extension deadline")
first = _Service("first", events)
second = _RaisesTimeout("second", events)
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(first)
with registry.attributed_to("second:install"):
registry.service(second)
diagnostics = await stop_services(registry.build(), timeout_seconds=1.0)
assert events == ["stop:second", "stop:first"]
assert diagnostics[0].source == "second:install"
assert "failed" in diagnostics[0].message
assert "timed out" not in diagnostics[0].message
@pytest.mark.parametrize(
("owner_path", "owner_protocol", "candidate_path", "candidate_protocol", "rejected"),
[
("/exact", "GET", "/exact", "GET", True),
("/items/{item_id}", "GET", "/items/{id}", "GET", True),
("/items/{item_id}", "GET", "/items/new", "GET", True),
("/items/{item_id}", "GET", "/items/prefix-{id}", "GET", True),
("/pre{tenant}", "GET", "/prefoo{id}", "GET", True),
("/items/new", "GET", "/items/{id}", "GET", False),
("/records/{value}", "GET", "/records/{id:int}", "GET", True),
("/records/{value:int}", "GET", "/records/0{id:int}", "GET", True),
("/records/{value:int}", "GET", "/records/new", "GET", False),
("/files/{rest:path}", "GET", "/files/{id:int}", "GET", True),
("/files/{rest:path}", "GET", "/files/{id}", "GET", False),
("/x/{rest:path}/tail", "GET", "/x/a/{id}/tail", "GET", False),
("/items/{item_id}", "GET", "/items/{id}", "POST", False),
("/live/{item_id}", "WS", "/live/{id}", "GET", False),
],
ids=[
"exact",
"renamed-parameter",
"dynamic-shadows-static",
"dynamic-shadows-compound",
"compound-trailing-str-shadows-narrower-compound",
"static-does-not-shadow-dynamic",
"str-covers-int",
"int-shadows-digit-compound",
"int-does-not-cover-static",
"path-covers-descendant",
"path-does-not-cover-newline-capable-str",
"nonterminal-path-does-not-cover-newline-capable-str",
"disjoint-http-methods",
"websocket-does-not-shadow-http",
],
)
def test_router_conflicts_follow_starlette_dispatch_order(
owner_path,
owner_protocol,
candidate_path,
candidate_protocol,
rejected,
):
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
app = FastAPI()
if owner_protocol == "WS":
app.add_api_websocket_route(owner_path, endpoint)
else:
app.add_api_route(owner_path, endpoint, methods=[owner_protocol])
router = APIRouter()
if candidate_protocol == "WS":
router.add_api_websocket_route(candidate_path, endpoint)
else:
router.add_api_route(candidate_path, endpoint, methods=[candidate_protocol])
registry = ExtensionRegistry()
with registry.attributed_to("candidate:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert bool(diagnostics) is rejected
if rejected:
assert diagnostics[0].source == "candidate:install"
assert "host" in diagnostics[0].message
assert candidate_path in diagnostics[0].message
else:
assert any(getattr(route, "path", None) == candidate_path for route in app.routes)
@pytest.mark.parametrize("convertor_name", ["flip", "int"])
def test_re_registered_convertor_does_not_create_a_false_shadow(
monkeypatch,
convertor_name,
):
from fastapi import APIRouter, FastAPI
from starlette.convertors import CONVERTOR_TYPES, Convertor
from starlette.routing import Match
from deerflow.extensions.gateway import include_contributed_routers
class DigitsConvertor(Convertor[str]):
regex = "[0-9]+"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
class LettersConvertor(Convertor[str]):
regex = "[A-Z]+"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
async def endpoint(value: str):
return {"value": value}
monkeypatch.setitem(CONVERTOR_TYPES, convertor_name, DigitsConvertor())
route_path = f"/owned/{{value:{convertor_name}}}"
app = FastAPI()
app.add_api_route(route_path, endpoint, methods=["GET"])
owner = app.routes[-1]
monkeypatch.setitem(CONVERTOR_TYPES, convertor_name, LettersConvertor())
router = APIRouter()
router.add_api_route(route_path, endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("candidate:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert diagnostics == []
candidate = app.routes[-1]
scope = {
"type": "http",
"path": "/owned/A",
"method": "GET",
"root_path": "",
}
assert owner.matches(scope)[0] is Match.NONE
assert candidate.matches(scope)[0] is Match.FULL
def test_router_claim_uses_converter_semantics_at_include_time(monkeypatch):
from fastapi import APIRouter, FastAPI
from starlette.convertors import CONVERTOR_TYPES, Convertor
from starlette.routing import Match
from deerflow.extensions.gateway import include_contributed_routers
class DigitsConvertor(Convertor[str]):
regex = "[0-9]+"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
class LettersConvertor(Convertor[str]):
regex = "[A-Z]+"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
async def endpoint(value: str):
return {"value": value}
monkeypatch.setitem(CONVERTOR_TYPES, "flip", DigitsConvertor())
route_path = "/owned/{value:flip}"
app = FastAPI()
app.add_api_route(route_path, endpoint, methods=["GET"])
owner = app.routes[-1]
router = APIRouter()
router.add_api_route(route_path, endpoint, methods=["GET"])
monkeypatch.setitem(CONVERTOR_TYPES, "flip", LettersConvertor())
registry = ExtensionRegistry()
with registry.attributed_to("candidate:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert diagnostics == []
candidate = app.routes[-1]
scope = {
"type": "http",
"path": "/owned/A",
"method": "GET",
"root_path": "",
}
assert owner.matches(scope)[0] is Match.NONE
assert candidate.matches(scope)[0] is Match.FULL
def test_recompiled_converter_cannot_enter_a_public_namespace(monkeypatch):
from fastapi import APIRouter, FastAPI
from starlette.convertors import CONVERTOR_TYPES, Convertor
from deerflow.extensions.gateway import include_contributed_routers
class PublicPathConvertor(Convertor[str]):
regex = r"webhooks/.+"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
async def endpoint(value: str):
return {"value": value}
router = APIRouter()
router.add_api_route("/api/{value:int}", endpoint, methods=["GET"])
monkeypatch.setitem(CONVERTOR_TYPES, "int", PublicPathConvertor())
registry = ExtensionRegistry()
with registry.attributed_to("candidate:install"):
registry.routers((router,))
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["candidate:install"]
assert "public namespace" in diagnostics[0].message
assert not any(getattr(route, "path", None) == "/api/{value:int}" for route in app.routes)
def test_host_mount_claims_descendant_http_paths():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
app = FastAPI()
app.mount("/assets", FastAPI())
router = APIRouter()
@router.get("/assets/{name:int}")
async def asset(name: int):
return {"name": name}
registry = ExtensionRegistry()
with registry.attributed_to("assets:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert len(diagnostics) == 1
assert diagnostics[0].source == "assets:install"
assert "host" in diagnostics[0].message
def test_dynamic_host_mount_claims_matching_descendants():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
app = FastAPI()
app.mount("/pre{tenant}", FastAPI())
router = APIRouter()
@router.get("/prefoo/{item_id:int}")
async def item(item_id: int):
return {"item_id": item_id}
registry = ExtensionRegistry()
with registry.attributed_to("mount:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["mount:install"]
assert "host" in diagnostics[0].message
@pytest.mark.parametrize(
("mount_path", "candidate_path", "witness"),
[
("/assets", "/assets/{name}", "/assets/a\nb"),
("/pre{tenant}", "/prefoo{id}/{child}", "/prefoo1/a\nb"),
],
)
def test_host_mount_does_not_claim_newline_capable_str_descendants(
mount_path,
candidate_path,
witness,
):
from fastapi import APIRouter, FastAPI
from starlette.routing import Match, Mount
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
app = FastAPI()
app.mount(mount_path, FastAPI())
host_mount = next(route for route in app.routes if isinstance(route, Mount) and route.path == mount_path)
router = APIRouter()
router.add_api_route(candidate_path, endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("mount:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert diagnostics == []
candidate_route = next(route for route in app.routes if getattr(route, "path", None) == candidate_path)
scope = {
"type": "http",
"path": witness,
"method": "GET",
"root_path": "",
}
assert host_mount.matches(scope)[0] is Match.NONE
assert candidate_route.matches(scope)[0] is Match.FULL
def test_contributed_websocket_route_is_rejected_until_host_auth_wraps_it():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def websocket_endpoint(websocket):
await websocket.close()
router = APIRouter()
router.add_api_websocket_route("/extension-ws", websocket_endpoint)
registry = ExtensionRegistry()
with registry.attributed_to("websocket:install"):
registry.routers((router,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["websocket:install"]
assert "WebSocket" in diagnostics[0].message
assert not any(getattr(route, "path", None) == "/extension-ws" for route in app.routes)
@pytest.mark.parametrize(
"path",
[
"/health-extension",
"/docs-private",
"/redoc-private",
"/api/webhooks/extension",
"/api/{rest:path}",
"/api/{section}/extension",
],
)
def test_extension_routes_cannot_enter_host_public_namespaces(path):
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
router = APIRouter()
router.add_api_route(path, endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("public:install"):
registry.routers((router,))
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["public:install"]
assert "public namespace" in diagnostics[0].message
assert not any(getattr(route, "path", None) == path for route in app.routes)
@pytest.mark.parametrize(
"path",
[
"/api",
"/heal",
"/api/{item_id:int}",
"/api/{item_id}",
],
)
def test_extension_routes_that_cannot_enter_a_public_namespace_are_allowed(path):
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
router = APIRouter()
router.add_api_route(path, endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("private:install"):
registry.routers((router,))
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
diagnostics = include_contributed_routers(app, registry.build())
assert diagnostics == []
assert any(getattr(route, "path", None) == path for route in app.routes)
def test_unknown_convertor_near_a_public_namespace_fails_closed(monkeypatch):
from fastapi import APIRouter, FastAPI
from starlette.convertors import CONVERTOR_TYPES, Convertor
from deerflow.extensions.gateway import include_contributed_routers
class UppercaseConvertor(Convertor[str]):
regex = "[A-Z]+"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
monkeypatch.setitem(CONVERTOR_TYPES, "uppercase", UppercaseConvertor())
router = APIRouter()
router.add_api_route(
"/api/{value:uppercase}",
lambda: {"ok": True},
methods=["GET"],
)
registry = ExtensionRegistry()
with registry.attributed_to("custom-public:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(
FastAPI(docs_url=None, redoc_url=None, openapi_url=None),
registry.build(),
)
assert [diagnostic.source for diagnostic in diagnostics] == ["custom-public:install"]
assert "public namespace" in diagnostics[0].message
def test_private_custom_convertor_with_named_backreference_is_allowed(monkeypatch):
from fastapi import APIRouter, FastAPI
from starlette.convertors import CONVERTOR_TYPES, Convertor
from deerflow.extensions.gateway import include_contributed_routers
class DoubledLetterConvertor(Convertor[str]):
regex = r"(?P<char>[A-Z])(?P=char)"
def convert(self, value: str) -> str:
return value
def to_string(self, value: str) -> str:
return value
monkeypatch.setitem(CONVERTOR_TYPES, "doubled", DoubledLetterConvertor())
router = APIRouter()
router.add_api_route(
"/private/{value:doubled}",
lambda value: {"value": value},
methods=["GET"],
)
registry = ExtensionRegistry()
with registry.attributed_to("custom-private:install"):
registry.routers((router,))
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
diagnostics = include_contributed_routers(app, registry.build())
assert diagnostics == []
assert any(getattr(route, "path", None) == "/private/{value:doubled}" for route in app.routes)
def test_extension_public_paths_track_auth_middleware_public_paths():
from app.gateway.auth_middleware import (
_PUBLIC_EXACT_PATHS,
_PUBLIC_PATH_PREFIXES,
_is_public,
)
from deerflow.extensions.gateway import (
_HOST_PUBLIC_EXACT_PATHS,
_HOST_PUBLIC_PATH_PREFIXES,
)
assert _HOST_PUBLIC_PATH_PREFIXES == _PUBLIC_PATH_PREFIXES
assert _HOST_PUBLIC_EXACT_PATHS == _PUBLIC_EXACT_PATHS
assert all(_is_public(f"{path}//") for path in _HOST_PUBLIC_EXACT_PATHS)
@pytest.mark.parametrize(
("host_path", "host_method", "candidate_path", "candidate_method"),
[
(
"/api/v1/auth/login/local",
"POST",
"/api/v1/auth/login/local",
"GET",
),
(
"/api/v1/auth/login/local",
"POST",
"/api/v1/auth/login/local/",
"GET",
),
(
"/api/v1/auth/login/local",
"POST",
"/api/v1/auth/login/local//",
"GET",
),
("/api/v1/auth/me", "GET", "/api/v1/auth/me", "POST"),
("/api/v1/auth/me", "GET", "/api/v1/auth/me/", "POST"),
],
)
def test_extension_routes_cannot_claim_reserved_exact_paths_with_a_disjoint_method(
host_path,
host_method,
candidate_path,
candidate_method,
):
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
app = FastAPI()
app.add_api_route(host_path, endpoint, methods=[host_method])
router = APIRouter()
router.add_api_route(candidate_path, endpoint, methods=[candidate_method])
registry = ExtensionRegistry()
with registry.attributed_to("public-exact:install"):
registry.routers((router,))
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["public-exact:install"]
assert "reserved" in diagnostics[0].message
assert not any(getattr(route, "path", None) == candidate_path and getattr(route, "methods", set()) == {candidate_method} for route in app.routes)
@pytest.mark.parametrize(
"path",
[
"/api/v1/auth/me",
"/api/v1/auth/me/",
"/api/v1/auth/me//",
],
)
def test_extension_csrf_reserved_exact_paths_track_csrf_exemption(
monkeypatch,
path,
):
from starlette.requests import Request
from app.gateway import csrf_middleware
from deerflow.extensions.gateway import (
_CSRF_STATE_CHANGING_METHODS,
_HOST_CSRF_EXEMPT_EXACT_PATHS,
_HOST_PUBLIC_EXACT_PATHS,
)
monkeypatch.setattr(csrf_middleware, "is_auth_disabled", lambda: False)
request = Request(
{
"type": "http",
"method": "POST",
"scheme": "http",
"path": path,
"raw_path": path.encode(),
"query_string": b"",
"headers": [],
"server": ("testserver", 80),
}
)
assert path.rstrip("/") in _HOST_CSRF_EXEMPT_EXACT_PATHS
assert _HOST_CSRF_EXEMPT_EXACT_PATHS == csrf_middleware._CSRF_EXEMPT_EXACT_PATHS
assert _CSRF_STATE_CHANGING_METHODS == csrf_middleware._CSRF_STATE_CHANGING_METHODS
assert csrf_middleware._AUTH_EXEMPT_PATHS <= _HOST_PUBLIC_EXACT_PATHS
assert csrf_middleware.should_check_csrf(request) is False
def test_safe_method_at_csrf_exempt_exact_path_is_not_reserved():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
router = APIRouter()
router.add_api_route(
"/api/v1/auth/me",
lambda: {"ok": True},
methods=["GET"],
)
registry = ExtensionRegistry()
with registry.attributed_to("safe-csrf:install"):
registry.routers((router,))
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
diagnostics = include_contributed_routers(app, registry.build())
assert diagnostics == []
assert any(getattr(route, "path", None) == "/api/v1/auth/me" for route in app.routes)
def test_router_with_one_conflict_is_rejected_atomically_and_names_first_owner():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
first = APIRouter()
first.add_api_route("/shared", endpoint, methods=["GET"])
second = APIRouter()
second.add_api_route("/would-have-been-reachable", endpoint, methods=["GET"])
second.add_api_route("/shared", endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.routers((first,))
with registry.attributed_to("second:install"):
registry.routers((second,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
paths = [getattr(route, "path", None) for route in app.routes]
assert "/shared" in paths
assert "/would-have-been-reachable" not in paths
assert len(diagnostics) == 1
assert diagnostics[0].source == "second:install"
assert "first:install" in diagnostics[0].message
def test_router_is_rejected_when_its_own_earlier_route_shadows_a_later_one():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
router = APIRouter()
router.add_api_route("/same/{value}", endpoint, methods=["GET"])
router.add_api_route("/same/fixed", endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("self-shadow:install"):
registry.routers((router,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["self-shadow:install"]
assert "self-shadow:install" in diagnostics[0].message
assert not any(getattr(route, "path", "").startswith("/same/") for route in app.routes)
def test_contributed_mount_is_rejected_but_does_not_starve_later_router():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
bad = APIRouter()
bad.mount("/nested", FastAPI())
good = APIRouter()
@good.get("/extension-good")
async def extension_good():
return {"ok": True}
registry = ExtensionRegistry()
with registry.attributed_to("bad:install"):
registry.routers((bad,))
with registry.attributed_to("good:install"):
registry.routers((good,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["bad:install"]
assert "Mount" in diagnostics[0].message
assert any(getattr(route, "path", None) == "/extension-good" for route in app.routes)
def test_router_with_unsupported_route_item_is_rejected_atomically():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
router = APIRouter()
router.add_api_route("/otherwise-valid", endpoint, methods=["GET"])
router.routes.append(object())
registry = ExtensionRegistry()
with registry.attributed_to("unsupported:install"):
registry.routers((router,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["unsupported:install"]
assert "unsupported route" in diagnostics[0].message
assert not any(getattr(route, "path", None) == "/otherwise-valid" for route in app.routes)
def test_include_router_failure_rolls_back_partial_routes_before_continuing():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
broken = APIRouter()
broken.add_api_route("/partial", endpoint, methods=["GET"])
broken.add_api_route("/explodes", endpoint, methods=["GET"])
broken.routes[-1].endpoint = None
later = APIRouter()
later.add_api_route("/partial", endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("broken:install"):
registry.routers((broken,))
with registry.attributed_to("later:install"):
registry.routers((later,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
partial_routes = [route for route in app.routes if getattr(route, "path", None) == "/partial"]
assert [diagnostic.source for diagnostic in diagnostics] == ["broken:install"]
assert len(partial_routes) == 1
def test_router_lifecycle_hooks_are_rejected_in_favor_of_extension_service():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
async def endpoint():
return {"ok": True}
async def startup_hook():
raise RuntimeError("must never be installed")
bad = APIRouter()
bad.add_api_route("/has-lifecycle", endpoint, methods=["GET"])
bad.add_event_handler("startup", startup_hook)
good = APIRouter()
good.add_api_route("/after-lifecycle", endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("lifecycle:install"):
registry.routers((bad,))
with registry.attributed_to("good:install"):
registry.routers((good,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
paths = [getattr(route, "path", None) for route in app.routes]
assert [diagnostic.source for diagnostic in diagnostics] == ["lifecycle:install"]
assert "ExtensionService" in diagnostics[0].message
assert "/has-lifecycle" not in paths
assert "/after-lifecycle" in paths
assert startup_hook not in app.router.on_startup
def test_router_custom_lifespan_is_rejected_in_favor_of_extension_service():
from fastapi import APIRouter, FastAPI
from deerflow.extensions.gateway import include_contributed_routers
@asynccontextmanager
async def lifespan(_app):
yield
async def endpoint():
return {"ok": True}
router = APIRouter(lifespan=lifespan)
router.add_api_route("/custom-lifespan", endpoint, methods=["GET"])
registry = ExtensionRegistry()
with registry.attributed_to("lifespan:install"):
registry.routers((router,))
app = FastAPI()
diagnostics = include_contributed_routers(app, registry.build())
assert [diagnostic.source for diagnostic in diagnostics] == ["lifespan:install"]
assert "ExtensionService" in diagnostics[0].message
assert not any(getattr(route, "path", None) == "/custom-lifespan" for route in app.routes)

View File

@ -28,6 +28,30 @@ def test_no_specs_yields_empty_result():
assert loaded.has_middleware_contributors is False
def test_host_disabled_required_extension_is_skipped_before_resolution(monkeypatch):
def _must_not_resolve(path: str):
raise AssertionError(f"disabled extension was resolved: {path}")
monkeypatch.setattr("deerflow.extensions.loader.resolve_variable", _must_not_resolve)
loaded, diagnostics = load_extensions([ExtensionSpec(use="missing_extension:install", enabled=False, required=True)])
assert diagnostics == []
assert loaded.has_middleware_contributors is False
def test_manager_metadata_is_accepted_without_reaching_the_install_hook():
spec = ExtensionSpec(
name="demo",
package="deerflow-extension-demo",
use=f"{_FIXTURE}:install_ok",
enabled=False,
)
assert spec.name == "demo"
assert spec.package == "deerflow-extension-demo"
def test_successful_install_registers_and_attributes():
spec = ExtensionSpec(use=f"{_FIXTURE}:install_ok")
loaded, diagnostics = load_extensions([spec])
@ -88,6 +112,9 @@ def test_install_failure_rolls_back_partial_registration():
assert sources == {f"{_FIXTURE}:install_ok"}
assert len(loaded.middleware_contributors) == 1, "rollback must clear every partial registration"
assert loaded.task_lifecycle == (), "rollback must clear partial lifecycle registrations too"
assert loaded.system_model_observers == ()
assert loaded.services == ()
assert loaded.routers == ()
def test_rollback_does_not_remove_a_different_specs_registrations_sharing_the_same_use():

File diff suppressed because it is too large Load Diff

View File

@ -80,6 +80,34 @@ def test_system_model_observers_are_attributed_and_require_task_storage():
assert loaded.needs_task_store is True
def test_services_are_attributed_without_allocating_task_storage():
registry = ExtensionRegistry()
service = _Contributor("service")
with registry.attributed_to("service_ext:install"):
registry.service(service)
loaded = registry.build()
assert loaded.services == (("service_ext:install", service),)
assert loaded.needs_task_store is False
def test_routers_are_flattened_in_registration_order_without_task_storage():
registry = ExtensionRegistry()
first = _Contributor("first-router")
second = _Contributor("second-router")
with registry.attributed_to("router_ext:install"):
registry.routers((first, second))
loaded = registry.build()
assert loaded.routers == (
("router_ext:install", first),
("router_ext:install", second),
)
assert loaded.needs_task_store is False
def test_rollback_restores_all_registration_buckets_positionally():
registry = ExtensionRegistry()
with registry.attributed_to("keep:install"):
@ -89,6 +117,8 @@ def test_rollback_restores_all_registration_buckets_positionally():
registry.middlewares(_Contributor("drop-middleware"))
registry.task_lifecycle(_Contributor("drop-lifecycle"))
registry.system_model_observer(_Contributor("drop-observer"))
registry.service(_Contributor("drop-service"))
registry.routers((_Contributor("drop-router"),))
registry.rollback_to(mark)
loaded = registry.build()
@ -96,6 +126,8 @@ def test_rollback_restores_all_registration_buckets_positionally():
assert [contributor.tag for _, contributor in loaded.middleware_contributors] == ["keep"]
assert loaded.task_lifecycle == ()
assert loaded.system_model_observers == ()
assert loaded.services == ()
assert loaded.routers == ()
def test_registration_order_is_preserved():
@ -120,11 +152,15 @@ def test_discard_removes_every_entry_of_one_source():
with registry.attributed_to("bad:install"):
registry.middlewares(drop)
registry.system_model_observer(drop)
registry.service(drop)
registry.routers((drop,))
registry.discard("bad:install")
loaded = registry.build()
assert loaded.middleware_contributors == (("good:install", keep),)
assert loaded.task_lifecycle == (("good:install", keep),)
assert loaded.system_model_observers == ()
assert loaded.services == ()
assert loaded.routers == ()
def test_registering_outside_attributed_to_raises():

View File

@ -0,0 +1,203 @@
"""Regression tests for Gateway ownership of extension services."""
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from types import SimpleNamespace
import pytest
from fastapi import FastAPI
from deerflow.extensions import (
get_runtime_diagnostics,
initialize_runtime_diagnostics,
reset_runtime_diagnostics,
)
from deerflow.extensions.registry import ExtensionRegistry
@pytest.fixture(autouse=True)
def _isolate_runtime_diagnostics():
reset_runtime_diagnostics()
yield
reset_runtime_diagnostics()
def _database_config() -> SimpleNamespace:
return SimpleNamespace(
backend="memory",
checkpoint_channel_mode="full",
checkpoint_delta=SimpleNamespace(snapshot_frequency=10),
)
@asynccontextmanager
async def _resource(value):
yield value
def _patch_runtime_resources(monkeypatch, events: list[str]) -> None:
async def init_engine(_database) -> None:
return None
async def close_engine() -> None:
events.append("engine_close")
monkeypatch.setattr("deerflow.runtime.make_stream_bridge", lambda _config: _resource(object()))
monkeypatch.setattr("deerflow.runtime.make_store", lambda _config: _resource(object()))
monkeypatch.setattr("deerflow.runtime.checkpointer.async_provider.make_checkpointer", lambda _config: _resource(object()))
monkeypatch.setattr("deerflow.persistence.engine.init_engine_from_config", init_engine)
monkeypatch.setattr("deerflow.persistence.engine.close_engine", close_engine)
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", lambda: None)
@pytest.mark.asyncio
async def test_runtime_owns_engine_cleanup_before_initialization(monkeypatch):
from app.gateway.deps import langgraph_runtime
events: list[str] = []
async def fail_engine_init(_database) -> None:
events.append("engine_init")
raise RuntimeError("schema bootstrap failed")
async def close_engine() -> None:
events.append("engine_close")
monkeypatch.setattr("deerflow.runtime.make_stream_bridge", lambda _config: _resource(object()))
monkeypatch.setattr("deerflow.persistence.engine.init_engine_from_config", fail_engine_init)
monkeypatch.setattr("deerflow.persistence.engine.close_engine", close_engine)
with pytest.raises(RuntimeError, match="schema bootstrap failed"):
async with langgraph_runtime(
FastAPI(),
SimpleNamespace(database=_database_config()),
):
pytest.fail("runtime must not yield")
assert events == ["engine_init", "engine_close"]
@pytest.mark.asyncio
async def test_later_startup_failure_stops_same_snapshot_and_appends_diagnostics(monkeypatch):
import deerflow.extensions as extensions_module
from app.gateway.deps import langgraph_runtime
events: list[str] = []
class _Service:
def __init__(self, name: str, *, fail_start: bool = False, fail_stop: bool = False) -> None:
self.name = name
self.fail_start = fail_start
self.fail_stop = fail_stop
async def start(self, _deps) -> None:
events.append(f"start:{self.name}")
if self.fail_start:
raise RuntimeError("start exploded")
async def stop(self) -> None:
events.append(f"stop:{self.name}")
if self.fail_stop:
raise RuntimeError("stop exploded")
registry = ExtensionRegistry()
with registry.attributed_to("bad-start:install"):
registry.service(_Service("bad-start", fail_start=True))
with registry.attributed_to("bad-stop:install"):
registry.service(_Service("bad-stop", fail_stop=True))
snapshot = registry.build()
other_registry = ExtensionRegistry()
with other_registry.attributed_to("other:install"):
other_registry.service(_Service("other"))
monkeypatch.setattr(extensions_module, "_loaded", other_registry.build())
_patch_runtime_resources(monkeypatch, events)
monkeypatch.setattr(
"deerflow.persistence.thread_meta.make_thread_store",
lambda _sf, _store: (_ for _ in ()).throw(RuntimeError("thread store failed")),
)
app = FastAPI()
app.state.extensions = snapshot
live_diagnostics = initialize_runtime_diagnostics([])
app.state.extension_diagnostics = live_diagnostics
with pytest.raises(RuntimeError, match="thread store failed"):
async with langgraph_runtime(
app,
SimpleNamespace(database=_database_config()),
):
pytest.fail("runtime must not yield")
assert events == [
"start:bad-start",
"start:bad-stop",
"stop:bad-stop",
"stop:bad-start",
"engine_close",
]
assert all("other" not in event for event in events)
assert app.state.extension_diagnostics is live_diagnostics
assert app.state.extension_diagnostics == get_runtime_diagnostics()
assert [diagnostic.source for diagnostic in live_diagnostics] == [
"bad-start:install",
"bad-stop:install",
]
@pytest.mark.asyncio
async def test_cancellation_during_service_start_propagates_after_cleanup(monkeypatch):
from app.gateway.deps import langgraph_runtime
events: list[str] = []
blocking_start_entered = asyncio.Event()
class _Service:
def __init__(self, name: str, *, block: bool = False) -> None:
self.name = name
self.block = block
async def start(self, _deps) -> None:
events.append(f"start:{self.name}")
if self.block:
blocking_start_entered.set()
await asyncio.Event().wait()
async def stop(self) -> None:
events.append(f"stop:{self.name}")
registry = ExtensionRegistry()
with registry.attributed_to("first:install"):
registry.service(_Service("first"))
with registry.attributed_to("blocking:install"):
registry.service(_Service("blocking", block=True))
with registry.attributed_to("never-started:install"):
registry.service(_Service("never-started"))
_patch_runtime_resources(monkeypatch, events)
app = FastAPI()
app.state.extensions = registry.build()
async def run_runtime() -> None:
async with langgraph_runtime(
app,
SimpleNamespace(database=_database_config()),
):
pytest.fail("runtime must not yield while service start is blocked")
task = asyncio.create_task(run_runtime())
await asyncio.wait_for(blocking_start_entered.wait(), timeout=1.0)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
assert events == [
"start:first",
"start:blocking",
"stop:blocking",
"stop:first",
"engine_close",
]

View File

@ -0,0 +1,180 @@
"""Conformance tests for the shared request-path projection.
``app.gateway.request_path.get_request_route_path()`` exists to answer one
question: *which path string is Starlette's router matching right now?* Auth
and CSRF classify that value, so the security predicates and the dispatcher
must agree on it exactly. When they disagree, a route mounted under a
public-looking prefix can be classified public while the router dispatches to
a protected handler.
These tests pin the agreement itself rather than the mechanism that produces
it, so they stay meaningful whether the projection keeps delegating to
Starlette or is ever reimplemented. Nothing here asserts *how* the value is
computed -- only that middleware and router see the same string.
"""
import pytest
from fastapi import FastAPI
from starlette.requests import Request
from starlette.testclient import TestClient
from app.gateway.auth_middleware import AuthMiddleware
from app.gateway.csrf_middleware import CSRFMiddleware, is_auth_endpoint, should_check_csrf
from app.gateway.request_path import get_request_route_path
from deerflow.config.authorization_config import AuthorizationConfig
@pytest.fixture(autouse=True)
def _default_route_authorization_config(monkeypatch):
"""Keep minimal middleware apps independent of a repository config.yaml."""
monkeypatch.setattr(
"app.gateway.authz._get_route_authorization_config",
lambda: AuthorizationConfig(),
)
@pytest.fixture(autouse=True)
def _auth_enabled(monkeypatch):
"""Every case here is about the enabled-auth path."""
monkeypatch.delenv("DEER_FLOW_AUTH_DISABLED", raising=False)
def _request(path: str, root_path: str = "", method: str = "GET") -> Request:
return Request(
{
"type": "http",
"method": method,
"path": path,
"root_path": root_path,
"query_string": b"",
"headers": [],
}
)
# ── Projection edge cases ────────────────────────────────────────────────────
@pytest.mark.parametrize(
("path", "root_path", "expected"),
[
# No mount: the raw ASGI path is already what the router matches.
("/api/models", "", "/api/models"),
# Mounted: root_path is stripped on the segment boundary.
("/prefix/api/models", "/prefix", "/api/models"),
# Nested mounts accumulate into root_path; strip all of it.
("/outer/inner/health", "/outer/inner", "/health"),
# The mount itself was requested; the router matches the empty path.
("/prefix", "/prefix", ""),
# root_path is not a prefix at all -- never strip.
("/other/models", "/prefix", "/other/models"),
# Prefix collides mid-segment. Stripping here would produce "foo",
# a string the router would never match.
("/apifoo/models", "/api", "/apifoo/models"),
("/apifoo", "/api", "/apifoo"),
],
)
def test_projection_strips_root_path_only_on_segment_boundaries(path: str, root_path: str, expected: str):
assert get_request_route_path(_request(path, root_path)) == expected
# ── Agreement with the router ────────────────────────────────────────────────
def test_projection_matches_the_path_the_router_dispatched_on():
"""The middleware-visible string equals the route's declared path."""
seen: dict[str, str] = {}
child = FastAPI()
@child.get("/health")
async def health(request: Request):
seen["projection"] = get_request_route_path(request)
return {"ok": True}
middle = FastAPI()
middle.mount("/inner", child)
parent = FastAPI()
parent.mount("/outer", middle)
assert TestClient(parent).get("/outer/inner/health").status_code == 200
# The router matched the declared "/health", not the wire path.
assert seen["projection"] == "/health"
def test_public_route_stays_public_under_nested_mounts():
"""Availability direction: a mounted /health must not start 401-ing."""
child = FastAPI()
child.add_middleware(AuthMiddleware)
@child.get("/health")
async def health():
return {"ok": True}
middle = FastAPI()
middle.mount("/inner", child)
parent = FastAPI()
parent.mount("/outer", middle)
assert TestClient(parent).get("/outer/inner/health").status_code == 200
# ── The bypass these predicates exist to prevent ─────────────────────────────
def test_mounting_under_a_public_prefix_does_not_expose_protected_routes():
"""Security direction: the mount prefix must not leak into classification.
Classifying the raw wire path would see "/health/api/models", match the
"/health" public prefix, and skip authentication entirely -- while the
router dispatches to the protected "/api/models" handler.
"""
child = FastAPI()
child.add_middleware(AuthMiddleware)
@child.get("/api/models")
async def models():
return {"models": []}
parent = FastAPI()
parent.mount("/health", child)
assert TestClient(parent).get("/health/api/models").status_code == 401
def test_csrf_is_enforced_for_routes_mounted_under_the_webhook_prefix():
"""CSRF's webhook exemption keys off the projection, not the wire path."""
child = FastAPI()
child.add_middleware(CSRFMiddleware)
@child.post("/action")
async def action():
return {"ok": True}
parent = FastAPI()
parent.mount("/api/webhooks", child)
response = TestClient(parent).post("/api/webhooks/action")
assert response.status_code == 403
assert "CSRF token missing" in response.json()["detail"]
# ── CSRF predicates read the same projection ─────────────────────────────────
def test_csrf_exemptions_follow_the_projection():
# Genuine host webhook: no mount, exempt.
assert should_check_csrf(_request("/api/webhooks/github", method="POST")) is False
# Same wire path, but the router is matching "/github" inside a mount --
# not the host's webhook namespace, so CSRF still applies.
assert should_check_csrf(_request("/api/webhooks/github", "/api/webhooks", method="POST")) is True
def test_auth_endpoint_detection_follows_the_projection():
assert is_auth_endpoint(_request("/api/v1/auth/login/local", method="POST")) is True
# A mount whose prefix is itself the auth namespace: the router matches
# "/local", which is not the host's exempt endpoint.
assert is_auth_endpoint(_request("/api/v1/auth/login/local", "/api/v1/auth/login", method="POST")) is False
# The mount point itself was requested; the router matches the empty path.
assert is_auth_endpoint(_request("/api/v1/auth/login/local", "/api/v1/auth/login/local", method="POST")) is False

View File

@ -148,7 +148,7 @@ async def test_shutdown_is_noop_without_inflight_runs():
@pytest.mark.asyncio
async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeypatch):
"""The wiring order lock for #3373: drain in-flight runs, THEN close the pool.
"""Drain runs before services, then close runtime resources in stack order.
Patches every ``langgraph_runtime`` collaborator down to trivial stand-ins so
only the bootstrap/teardown ordering runs. The checkpointer probe records when
@ -158,6 +158,7 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
from fastapi import FastAPI
from app.gateway.deps import langgraph_runtime
from deerflow.extensions.registry import ExtensionRegistry
events: list[str] = []
@ -170,17 +171,27 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
@asynccontextmanager
async def fake_stream_bridge(_config):
yield object()
try:
yield object()
finally:
events.append("stream_bridge_closed")
@asynccontextmanager
async def fake_store(_config):
yield object()
try:
yield object()
finally:
events.append("store_closed")
async def fake_init_engine(_db):
events.append("engine_initialized")
def fake_session_factory():
events.append("session_factory_resolved")
return None
async def fake_close_engine():
return None
events.append("engine_closed")
async def spy_shutdown(self, *, timeout): # noqa: ANN001
events.append("runs_drained")
@ -197,7 +208,7 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
monkeypatch.setattr("deerflow.runtime.make_store", fake_store)
monkeypatch.setattr("deerflow.persistence.engine.init_engine_from_config", fake_init_engine)
monkeypatch.setattr("deerflow.persistence.engine.close_engine", fake_close_engine)
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", lambda: None)
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", fake_session_factory)
monkeypatch.setattr("deerflow.runtime.events.store.make_run_event_store", lambda _cfg: object())
monkeypatch.setattr("deerflow.persistence.thread_meta.make_thread_store", lambda _sf, _store: object())
monkeypatch.setattr(RunManager, "shutdown", spy_shutdown, raising=False)
@ -205,16 +216,36 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
monkeypatch.setattr("deerflow.extensions.notify.reset_extension_notify_loop", spy_reset_extension_notify_loop)
app = FastAPI()
registry = ExtensionRegistry()
class _Service:
async def start(self, _deps):
events.append("service_started")
async def stop(self):
events.append("service_stopped")
with registry.attributed_to("service:install"):
registry.service(_Service())
app.state.extensions = registry.build()
startup_config = SimpleNamespace(database=SimpleNamespace(backend="memory", checkpoint_channel_mode="full", checkpoint_delta=SimpleNamespace(snapshot_frequency=10)), run_events=None)
async with langgraph_runtime(app, startup_config):
pass
assert "runs_drained" in events, "langgraph_runtime never drained in-flight runs on shutdown"
assert "service_started" in events
assert "service_stopped" in events
assert "checkpointer_closed" in events
assert events.index("runs_drained") < events.index("checkpointer_closed"), f"runs must be drained before the checkpointer pool is closed; got order {events}"
assert events.index("engine_initialized") < events.index("session_factory_resolved")
assert events.index("session_factory_resolved") < events.index("service_started")
assert events.index("runs_drained") < events.index("service_stopped")
assert events.index("service_stopped") < events.index("store_closed")
assert events.index("store_closed") < events.index("checkpointer_closed")
assert events.index("checkpointer_closed") < events.index("engine_closed")
assert events.index("engine_closed") < events.index("stream_bridge_closed")
assert events[0] == "extension_loop_set"
assert events.index("checkpointer_closed") < events.index("extension_loop_reset"), f"extension loop reset must be the final runtime teardown; got order {events}"
assert events.index("stream_bridge_closed") < events.index("extension_loop_reset"), f"extension loop reset must be the final runtime teardown; got order {events}"
@pytest.mark.asyncio

View File

@ -105,7 +105,10 @@ def test_backend_make_dev_gateway_reload_excludes_runtime_state_with_absolute_di
assert "DEER_FLOW_HOME := $(abspath $(DEER_FLOW_HOME))" in makefile
assert "BACKEND_SANDBOX_HOME := $(abspath $(CURDIR)/sandbox)" in makefile
assert 'mkdir -p "$(DEER_FLOW_HOME)" "$(BACKEND_SANDBOX_HOME)"' in makefile
assert 'DEER_FLOW_HOME="$(DEER_FLOW_HOME)" uv run uvicorn' in makefile
# The launch line may carry runtime-only uv flags (`--locked` pins the
# extension lock); what this guards is that DEER_FLOW_HOME is exported on it,
# so the reload-excludes below resolve to the same absolute directories.
assert re.search(r'DEER_FLOW_HOME="\$\(DEER_FLOW_HOME\)" uv run(?: --(?:locked|no-sync))? uvicorn', makefile)
assert '--reload-exclude="$(DEER_FLOW_HOME)"' in makefile
assert '--reload-exclude="$(BACKEND_SANDBOX_HOME)"' in makefile

View File

@ -2,13 +2,17 @@
import pytest
from deerflow.tui.cli import LaunchPlan, plan_launch
from deerflow.tui.cli import LaunchPlan, build_parser, plan_launch
def plan(argv, *, stdin_tty=True, stdout_tty=True, env=None):
return plan_launch(argv, stdin_isatty=stdin_tty, stdout_isatty=stdout_tty, env=env or {})
def test_top_level_help_points_to_extension_management():
assert "deerflow extensions --help" in build_parser().format_help()
def test_bare_command_on_tty_launches_tui():
p = plan([])
assert p.mode == "tui"

View File

@ -157,14 +157,15 @@ def test_sandbox_mkdir_precedes_uvicorn_launch(name):
``_mkdir_dirs`` only proves the mkdir is present somewhere; this pins script
order so a future edit can't move (or guard) the mkdir below the launch and
silently reintroduce the #3454 crash on a fresh checkout. ``uv run uvicorn``
matches the launch but not serve.sh's ``stop_all`` kill line.
silently reintroduce the #3454 crash on a fresh checkout. The ``uv run``
matcher allows runtime-only flags while still excluding serve.sh's
``stop_all`` kill line.
"""
lines = LAUNCHERS[name].read_text(encoding="utf-8").splitlines()
launch_idx = next((i for i, ln in enumerate(lines) if "uv run uvicorn" in ln), None)
launch_idx = next((i for i, ln in enumerate(lines) if re.search(r"\buv run(?: --(?:no-sync|locked))? uvicorn\b", ln)), None)
mkdir_idx = next((i for i, ln in enumerate(lines) if re.search(r"\bmkdir\b", ln) and "sandbox" in ln.lower()), None)
assert launch_idx is not None, f"{name}: could not locate the 'uv run uvicorn' launch line"
assert launch_idx is not None, f"{name}: could not locate the uvicorn launch line"
assert mkdir_idx is not None, f"{name}: could not locate the sandbox mkdir line"
assert mkdir_idx < launch_idx, f"{name}: sandbox mkdir (line {mkdir_idx + 1}) must precede uvicorn launch (line {launch_idx + 1})"

11
backend/uv.lock generated
View File

@ -821,6 +821,7 @@ dependencies = [
{ name = "python-telegram-bot" },
{ name = "slack-sdk" },
{ name = "sse-starlette" },
{ name = "starlette" },
{ name = "uvicorn", extra = ["standard"] },
{ name = "wecom-aibot-python-sdk" },
]
@ -886,6 +887,7 @@ requires-dist = [
{ name = "python-telegram-bot", specifier = ">=21.0" },
{ name = "slack-sdk", specifier = ">=3.33.0" },
{ name = "sse-starlette", specifier = ">=2.1.0" },
{ name = "starlette", specifier = ">=1.3.1,<2" },
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.34.0" },
{ name = "wecom-aibot-python-sdk", specifier = ">=0.1.6" },
]
@ -904,10 +906,11 @@ dev = [
{ name = "ruff", specifier = ">=0.14.11" },
{ name = "textual", specifier = ">=0.80" },
]
extensions = []
[[package]]
name = "deerflow-extension-api"
version = "0.1.1"
version = "0.1.2"
source = { editable = "packages/extension-api" }
[[package]]
@ -946,6 +949,7 @@ dependencies = [
{ name = "langgraph-sdk" },
{ name = "markdownify" },
{ name = "markitdown", extra = ["all", "xlsx"] },
{ name = "packaging" },
{ name = "pydantic" },
{ name = "pyyaml" },
{ name = "readabilipy" },
@ -1028,6 +1032,7 @@ requires-dist = [
{ name = "markdownify", specifier = ">=1.2.2" },
{ name = "markitdown", extras = ["all", "xlsx"], specifier = ">=0.0.1a2" },
{ name = "monocle-apptrace", marker = "extra == 'monocle'", specifier = ">=0.8.8" },
{ name = "packaging", specifier = ">=24.2" },
{ name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" },
{ name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.3.3" },
{ name = "psycopg-pool", marker = "extra == 'postgres'", specifier = ">=3.3.0" },
@ -4514,8 +4519,8 @@ name = "standard-aifc"
version = "3.13.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "audioop-lts" },
{ name = "standard-chunk" },
{ name = "audioop-lts", marker = "python_full_version >= '3.13'" },
{ name = "standard-chunk", marker = "python_full_version >= '3.13'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c4/53/6050dc3dde1671eb3db592c13b55a8005e5040131f7509cef0215212cb84/standard_aifc-3.13.0.tar.gz", hash = "sha256:64e249c7cb4b3daf2fdba4e95721f811bde8bdfc43ad9f936589b7bb2fae2e43", size = 15240, upload-time = "2024-10-30T16:01:31.772Z" }
wheels = [

View File

@ -41,7 +41,8 @@ logging:
# lead-only vs subagent-only configuration is not expressible yet. Treat these
# files as trusted operator config because middleware classes execute code.
# Uncomment this block to define middlewares in config.yaml. Leaving it commented
# lets extensions_config.json remain the source of truth for extension packages.
# lets extensions_config.json remain the source of truth for this legacy
# config-declared middleware list. Packaged plugins use the `plugins:` block below.
# extensions:
# middlewares:
# - my_company.deerflow_middlewares:DomainGuardMiddleware
@ -2531,8 +2532,27 @@ authorization:
# entry names an install entry point as "module.path:install"; the `config`
# block is handed to that package verbatim and validated by the package itself.
#
# Loading is explicit on purpose: an installed package does nothing until it is
# listed here, and the list order is what fixes middleware ordering.
# Prefer the extension manager over editing this list by hand:
# make extension-install SOURCE="deerflow-extension-acme==1.2.3"
# make extension-install SOURCE="git+https://github.com/acme/deerflow-extension-acme.git@<commit>"
# make extension-install SOURCE="/absolute/path/to/local-extension"
# make extension-list
# make extension-enable NAME=acme
# make extension-disable NAME=acme
# make extension-remove NAME=acme
# The installer updates backend/pyproject.toml, backend/uv.lock, and this block
# together. It also asks the operator to confirm that the source is trusted;
# extensions and their build systems execute with Gateway privileges.
#
# Every install, enable, disable, remove, or manual edit requires a Gateway
# restart. Development launchers may fetch missing locked artifacts before
# handoff; a built production Gateway never installs plugins during startup.
# Local directories are copied to backend/extensions/sources as deployable
# snapshots; use a pinned version or public HTTPS Git commit for reproducible
# remote sources. SSH Git URLs are rejected because the stock Docker builder
# does not forward host SSH credentials.
# Loading remains explicit: a package installed outside the manager does nothing
# until it is listed here, and list order fixes contribution ordering.
#
# This is deliberately separate from the `extensions:` block above (MCP servers,
# skills, config-declared middlewares). That one is backed by
@ -2540,20 +2560,16 @@ authorization:
# endpoint; a list that causes code to be imported must stay in this
# operator-controlled file only.
#
# This initial extension-system slice accepts middleware contributors. Their
# install functions register semantically placed middlewares for the lead and/or
# subagent stack; the examples below intentionally cover only that surface.
# Plugins can contribute semantically placed middleware, task-lifecycle hooks,
# system-model observers, Gateway-lifetime services, and eager FastAPI HTTP routers.
# See examples/deerflow-extension-example for a package that exercises all five.
#
# plugins:
# - use: acme_request_observer:install
# - name: example # PEP 621 entry-point name
# package: deerflow-extension-example # Python distribution managed by uv
# use: deerflow_extension_example:install
# enabled: true # false skips import and registration
# required: false # true makes load failure abort Gateway startup
# # (install --required opts in)
# config:
# enabled: true
# placement: model_logical
# - use: acme_tool_observer:install
# config:
# enabled: true
# placement: tool_raw
# # `required: true` turns a load failure into a startup failure. Use it for
# # packages whose absence changes behaviour rather than just observability.
# - use: acme_required_middleware:install
# required: true
# label: example # extension-private values, if any

View File

@ -5,13 +5,17 @@
# (PR #2767, addressing review on Issue #2754).
#
# Responsibilities:
# 1. Resolve `--extra X` flags from UV_EXTRAS (comma- or whitespace-separated,
# mirroring scripts/detect_uv_extras.py for parity with local `make dev`).
# 1. Resolve `--extra X` flags through scripts/detect_uv_extras.py, using the
# selected config plus explicit UV_EXTRAS and runtime-required backends.
# 2. Validate each extra against [A-Za-z][A-Za-z0-9_-]* so a stray shell
# metacharacter in `.env` cannot reach `uv sync`.
# 3. `uv sync --all-packages` so workspace member extras (deerflow-harness's
# 3. `uv sync --locked --all-packages` so the declared extension group and
# workspace member extras (deerflow-harness's
# postgres extra in particular) are installed — see PR #2584.
# 4. Self-heal: if the first sync fails, recreate .venv and retry once.
# 4. Self-heal: if the first sync fails, recreate .venv and retry once. The
# retry stays `--locked`, so it repairs a broken .venv but not a stale
# lock; a second failure aborts with recovery instructions rather than
# starting uvicorn against an environment that does not match the lock.
# 5. Hand off to uvicorn with reload, replacing this shell so uvicorn becomes
# PID 1 inside the container.
#
@ -38,22 +42,78 @@ fi
# ── Resolve extras ──────────────────────────────────────────────────────────
EXTRAS_FLAGS=""
EXTRA_NAMES=""
set -f
append_extra() {
extra_name="$1"
case "$extra_name" in
[!A-Za-z]* | *[!A-Za-z0-9_-]*)
echo "[startup] UV_EXTRAS entry '$extra_name' is invalid (must match [A-Za-z][A-Za-z0-9_-]*) — aborting" >&2
exit 1
;;
esac
case " $EXTRA_NAMES " in
*" $extra_name "*) return ;;
esac
EXTRA_NAMES="$EXTRA_NAMES $extra_name"
EXTRAS_FLAGS="$EXTRAS_FLAGS --extra $extra_name"
}
# Validate explicit input before the detector normalizes it. The shared
# detector deliberately drops invalid names with a warning, while container
# startup fails closed so malformed .env input cannot be silently ignored.
if [ -n "${UV_EXTRAS:-}" ]; then
# Normalize comma → space, then split on whitespace via the unquoted `for`.
for raw in $(printf '%s' "$UV_EXTRAS" | tr ',' ' '); do
[ -z "$raw" ] && continue
# Reject anything that does not look like an identifier.
# Two patterns: leading non-letter, or any non-[A-Za-z0-9_-] character.
case "$raw" in
[!A-Za-z]* | *[!A-Za-z0-9_-]*)
echo "[startup] UV_EXTRAS entry '$raw' is invalid (must match [A-Za-z][A-Za-z0-9_-]*) — aborting" >&2
exit 1
;;
esac
EXTRAS_FLAGS="$EXTRAS_FLAGS --extra $raw"
[ -z "$raw" ] || append_extra "$raw"
done
fi
# Docker dev mounts the host checkout at /app/project while
# DEER_FLOW_PROJECT_ROOT points at /app for runtime path translation. Prefer
# both locations, then the checkout-relative path used by direct invocations.
ENTRYPOINT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
DETECTOR_PATH=""
for candidate in \
"${DEER_FLOW_PROJECT_ROOT:+$DEER_FLOW_PROJECT_ROOT/scripts/detect_uv_extras.py}" \
/app/project/scripts/detect_uv_extras.py \
"$ENTRYPOINT_DIR/../scripts/detect_uv_extras.py"
do
if [ -n "$candidate" ] && [ -f "$candidate" ]; then
DETECTOR_PATH="$candidate"
break
fi
done
if [ -z "$DETECTOR_PATH" ]; then
echo "[startup] scripts/detect_uv_extras.py is unavailable" >&2
exit 1
fi
if command -v python3 >/dev/null 2>&1; then
DETECTOR_PYTHON=python3
elif command -v python >/dev/null 2>&1; then
DETECTOR_PYTHON=python
else
echo "[startup] Python is required to resolve optional dependencies" >&2
exit 1
fi
if ! DETECTED_FLAGS=$("$DETECTOR_PYTHON" "$DETECTOR_PATH"); then
echo "[startup] detect_uv_extras.py failed" >&2
exit 1
fi
# The detector emits only validated `--extra NAME` pairs. Parse that small
# interface instead of evaluating shell text, and validate again at the final
# shell boundary before any value can reach uv.
set -- $DETECTED_FLAGS
while [ "$#" -gt 0 ]; do
if [ "$1" != "--extra" ] || [ "$#" -lt 2 ]; then
echo "[startup] detect_uv_extras.py returned invalid output" >&2
exit 1
fi
append_extra "$2"
shift 2
done
if [ "$PRINT_EXTRAS_ONLY" = "1" ]; then
# Trim leading space for tidier output, then exit.
printf '%s\n' "${EXTRAS_FLAGS# }"
@ -78,22 +138,31 @@ mkdir -p "$DEER_FLOW_HOME" /app/backend/.deer-flow /app/backend/sandbox
cd /app/backend
# `--all-packages` propagates extras into workspace members (PR #2584).
# `--extra redis` is always installed because docker-compose-dev defaults the
# stream bridge to Redis (DEER_FLOW_STREAM_BRIDGE_REDIS_URL); redis is an
# optional extra elsewhere. It is kept out of EXTRAS_FLAGS so the --print-extras
# contract (UV_EXTRAS-derived flags only) stays unchanged.
# docker-compose-dev's default DEER_FLOW_STREAM_BRIDGE_REDIS_URL is translated
# to `--extra redis` by the shared detector, alongside config and UV_EXTRAS.
# `$EXTRAS_FLAGS` intentionally unquoted so each `--extra X` becomes its own arg.
# shellcheck disable=SC2086 # word-splitting is intentional here
if ! uv sync --all-packages --extra redis $EXTRAS_FLAGS; then
if ! uv sync --locked --all-packages $EXTRAS_FLAGS; then
echo "[startup] uv sync failed; recreating .venv and retrying once"
uv venv --allow-existing .venv
uv venv --clear .venv
# The retry keeps `--locked` on purpose: it repairs a corrupt or partial
# .venv, not a lock that disagrees with pyproject.toml. Startup must never
# silently resolve dependencies, so a second failure is fatal rather than
# something uvicorn limps past and reports later as an import error.
# `set -e` would already stop here; abort explicitly so the operator gets
# the fix instead of a bare uv exit code.
# shellcheck disable=SC2086
uv sync --all-packages --extra redis $EXTRAS_FLAGS
if ! uv sync --locked --all-packages $EXTRAS_FLAGS; then
echo "[startup] uv sync --locked failed again after recreating .venv." >&2
echo "[startup] backend/uv.lock does not match backend/pyproject.toml, or a locked artifact is unreachable." >&2
echo "[startup] Run 'make install' on the host to refresh the lock, then restart this container." >&2
exit 1
fi
fi
# ── Hand off to uvicorn ─────────────────────────────────────────────────────
PYTHONPATH=. exec uv run uvicorn app.gateway.app:app \
PYTHONPATH=. exec uv run --no-sync uvicorn app.gateway.app:app \
--host 0.0.0.0 --port 8001 \
--reload \
--reload-include='*.yaml' \

View File

@ -153,7 +153,7 @@ services:
# cache_from disabled - requires manual setup: mkdir -p /tmp/docker-cache-gateway
args:
APT_MIRROR: ${APT_MIRROR:-}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.11.1}
UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple}
NPM_REGISTRY: ${NPM_REGISTRY:-}
LARK_CLI_NPM_VERSION: ${LARK_CLI_NPM_VERSION:-1.0.65}

View File

@ -90,7 +90,7 @@ services:
dockerfile: backend/Dockerfile
args:
APT_MIRROR: ${APT_MIRROR:-}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.11.1}
UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple}
UV_EXTRAS: ${UV_EXTRAS:-}
NPM_REGISTRY: ${NPM_REGISTRY:-}
@ -102,7 +102,7 @@ services:
# reconnect, but run cancel, request dedup, and per-worker IM channel
# services remain worker-local. Override GATEWAY_WORKERS only when the
# Redis stream bridge is enabled and those limitations are acceptable.
command: sh -c "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1}"
command: sh -c "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1}"
volumes:
- ${DEER_FLOW_CONFIG_PATH}:/app/backend/config.yaml:ro
- ${DEER_FLOW_EXTENSIONS_CONFIG_PATH}:/app/backend/extensions_config.json:ro

View File

@ -0,0 +1,149 @@
# DeerFlow extension example
This directory is a compact, standalone Python package showing all five DeerFlow
extension contribution kinds. It depends on the public
`deerflow-extension-api` contract and never imports `deerflow.*` or `app.*`.
The contract package intentionally has no framework dependencies. An extension
must therefore declare every framework it imports itself; this example explicitly
depends on FastAPI, LangChain, and LangGraph in `pyproject.toml`.
## What it demonstrates
| Contribution | Example behavior |
| --- | --- |
| Middleware | Counts tool calls through one `TOOL_VISIBLE` middleware for lead agents and subagents |
| Task lifecycle | Creates task-scoped stats on start and folds them into app scope on stop |
| System-model observer | Counts DeerFlow-owned model calls, including failures |
| Service | Binds `ExtensionRuntimeDeps` only while the Gateway is running |
| Router | Eagerly declares `GET /api/extension-example/stats` during `install()` |
The middleware reads task scope only through `task_store_from_runtime()`. It
passes through unchanged when no task store exists. The router and service use
the same `ExampleService` object: its FastAPI dependency returns `503` before
`start()`, after `stop()`, or when no app store was bound. This keeps the route
topology stable while runtime capabilities arrive later.
## Run the package tests
`deerflow-extension-api` is currently sourced from this checkout. Install it
first, then install this independent package:
```bash
cd examples/deerflow-extension-example
uv venv --python 3.12
uv pip install -e ../../backend/packages/extension-api
uv pip install -e ".[dev]"
uv run --no-project pytest -q
uv run --no-project ruff check .
uv run --no-project ruff format --check .
```
The tests use only the public contract plus this package's declared dependencies;
the DeerFlow harness and Gateway application are not imported.
## Install and load it in DeerFlow
From the DeerFlow checkout root, install this directory through the extension
manager. Use an absolute path because the Make wrapper invokes the manager from
`backend/`:
```bash
make extension-install SOURCE="$PWD/examples/deerflow-extension-example"
make extension-list
```
After the trust prompt is accepted, the manager:
- copies a deployable snapshot to
`backend/extensions/sources/deerflow-extension-example/`;
- adds that snapshot to `backend/pyproject.toml`'s `extensions` dependency
group and updates `backend/uv.lock`;
- installs the locked environment; and
- adds and enables this startup-only entry in the selected `config.yaml`.
```yaml
plugins:
- name: example
package: deerflow-extension-example
use: deerflow_extension_example:install
enabled: true
required: false
config: {}
```
Start or restart DeerFlow after installation:
```bash
make dev
```
The Gateway imports extensions only while constructing the application. Install,
enable, disable, remove, and manual `plugins:` changes therefore take effect only
after a restart. These commands manage the example afterward:
```bash
make extension-disable NAME=example
make extension-enable NAME=example
make extension-remove NAME=example
```
The manager can also install a PyPI requirement or a pinned public HTTPS Git
URL. SSH Git URLs are rejected because the stock Docker builder does not
forward host SSH credentials. The direct CLI surface, run from `backend/`, is:
```text
uv run --frozen --no-group extensions deerflow extensions install <source> [--yes] [--required]
uv run --frozen --no-group extensions deerflow extensions list
uv run --frozen --no-group extensions deerflow extensions enable <name>
uv run --frozen --no-group extensions deerflow extensions disable <name>
uv run --frozen --no-group extensions deerflow extensions remove <name>
```
`--yes` is intended only for automation that has already reviewed and trusted
the source: extension build hooks and runtime code execute with Gateway
privileges. `--required` records `required: true`, which turns any later load
failure into a Gateway startup abort; leave it off unless the application is
wrong without this extension.
The local snapshot is included in Docker builds. Local `make dev`, Docker dev,
and the production Gateway image all consume the same `backend/uv.lock`.
Development launchers may download missing locked artifacts before handing off
to the Gateway; a built production container does not. Rebuild the production
image with `make up` after changing the installed set.
After one or more runs, request the extension route:
```bash
curl -s http://localhost:2026/api/extension-example/stats
```
The response contains aggregated task outcomes, tool-call counts, system-model
call counts, the app scope id, and a small projection of the host policy. The
route passes through the Gateway's normal authentication middleware; use an
authenticated browser session when authentication is enabled.
## Packaging entry point
Managed packages expose exactly one standard PEP 621 entry point in the
`deerflow.extensions` group. This example declares:
```toml
[project.entry-points."deerflow.extensions"]
example = "deerflow_extension_example:install"
```
The entry-point name (`example`) is the stable operator-facing name accepted by
`enable`, `disable`, and `remove`; those commands also accept the distribution
name or the `module:install` value.
## Package layout
```text
deerflow_extension_example/
├── __init__.py # version-stamped install() entry point
└── plugin.py # state plus all five small contribution implementations
tests/
├── test_entry_point.py
└── test_plugin.py
```

View File

@ -0,0 +1,35 @@
"""A compact, standalone DeerFlow extension exercising every contribution kind."""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from deerflow_extension_api import ExtensionInstall, ExtensionRegistry, extension
from deerflow_extension_example.plugin import (
ExampleMiddlewareContributor,
ExampleService,
ExampleSystemObserver,
ExampleTaskLifecycle,
build_router,
)
__all__ = ["install"]
@extension(api="0.1.2", name="example")
def install(registry: ExtensionRegistry, config: Mapping[str, Any]) -> None:
"""Register one example of each supported contribution kind."""
if config.get("enabled", True) is False:
return
service = ExampleService()
registry.middlewares(ExampleMiddlewareContributor())
registry.task_lifecycle(ExampleTaskLifecycle())
registry.system_model_observer(ExampleSystemObserver())
registry.service(service)
registry.routers((build_router(service),))
_entry_point: ExtensionInstall = install

View File

@ -0,0 +1,176 @@
"""The example's five deliberately small contribution implementations."""
from __future__ import annotations
from collections.abc import Awaitable, Callable, Sequence
from dataclasses import dataclass, field
from threading import Lock
from typing import Any
from deerflow_extension_api import (
AgentBuildContext,
AgentScope,
ExtensionData,
ExtensionRuntimeDeps,
MiddlewarePlacement,
Placement,
SystemModelRequest,
SystemModelResult,
SystemOperationKind,
TaskInfo,
TaskOutcome,
task_store_from_runtime,
)
from fastapi import APIRouter, Depends, HTTPException
from langchain.agents.middleware import AgentMiddleware
from langgraph.prebuilt.tool_node import ToolCallRequest
@dataclass
class ExampleStats:
"""Small extension-owned value used in both app and task stores."""
tool_calls: int = 0
tasks: dict[str, int] = field(default_factory=dict)
system_model_calls: dict[str, dict[str, int]] = field(default_factory=dict)
_lock: Lock = field(default_factory=Lock, repr=False, compare=False)
def note_tool_call(self) -> None:
with self._lock:
self.tool_calls += 1
def task_tool_calls(self) -> int:
with self._lock:
return self.tool_calls
def absorb_task(self, tool_calls: int, outcome: TaskOutcome) -> None:
with self._lock:
self.tool_calls += tool_calls
key = outcome.value
self.tasks[key] = self.tasks.get(key, 0) + 1
def note_system_call(self, kind: SystemOperationKind, *, failed: bool) -> None:
with self._lock:
entry = self.system_model_calls.setdefault(
kind.value,
{"calls": 0, "errors": 0},
)
entry["calls"] += 1
if failed:
entry["errors"] += 1
def snapshot(self) -> dict[str, Any]:
with self._lock:
return {
"tasks": dict(self.tasks),
"tool_calls": self.tool_calls,
"system_model_calls": {kind: dict(counts) for kind, counts in self.system_model_calls.items()},
}
def _stats(store: ExtensionData) -> ExampleStats:
return store.get_or_init(ExampleStats, ExampleStats)
class ExampleMiddleware(AgentMiddleware):
async def awrap_tool_call(
self,
request: ToolCallRequest,
handler: Callable[[ToolCallRequest], Awaitable[Any]],
) -> Any:
task_store = task_store_from_runtime(getattr(request, "runtime", None))
task_stats = task_store.get(ExampleStats) if task_store is not None else None
if task_stats is not None:
task_stats.note_tool_call()
return await handler(request)
class ExampleMiddlewareContributor:
def contribute_middlewares(
self,
app_store: ExtensionData,
ctx: AgentBuildContext,
) -> Sequence[MiddlewarePlacement]:
return (
MiddlewarePlacement(
ExampleMiddleware(),
Placement.TOOL_VISIBLE,
AgentScope.BOTH,
),
)
class ExampleTaskLifecycle:
async def on_task_start(
self,
app_store: ExtensionData,
task_store: ExtensionData,
info: TaskInfo,
) -> None:
task_store.set(ExampleStats())
async def on_task_stop(
self,
app_store: ExtensionData,
task_store: ExtensionData,
info: TaskInfo,
outcome: TaskOutcome,
) -> None:
task_stats = task_store.remove(ExampleStats)
_stats(app_store).absorb_task(
task_stats.task_tool_calls() if task_stats is not None else 0,
outcome,
)
class ExampleSystemObserver:
async def on_system_model_call(
self,
app_store: ExtensionData,
task_store: ExtensionData,
kind: SystemOperationKind,
request: SystemModelRequest,
result: SystemModelResult,
) -> None:
_stats(app_store).note_system_call(kind, failed=result.error is not None)
class ExampleService:
def __init__(self) -> None:
self._deps: ExtensionRuntimeDeps | None = None
async def start(self, deps: ExtensionRuntimeDeps) -> None:
self._deps = deps
async def stop(self) -> None:
self._deps = None
async def require_deps(self) -> ExtensionRuntimeDeps:
deps = self._deps
if deps is None or deps.app_store is None:
raise HTTPException(
status_code=503,
detail="extension-example is not running",
)
return deps
def build_router(service: ExampleService) -> APIRouter:
"""Build paths during registration, before runtime dependencies exist."""
router = APIRouter(prefix="/api/extension-example", tags=["extension-example"])
@router.get("/stats")
async def read_stats(
deps: ExtensionRuntimeDeps = Depends(service.require_deps),
) -> dict[str, Any]:
assert deps.app_store is not None
return {
"scope_id": deps.app_store.scope_id,
"session_factory_available": deps.session_factory is not None,
"host_policy": {
"max_subagents_per_run": deps.policy.max_subagents_per_run,
},
**_stats(deps.app_store).snapshot(),
}
return router

View File

@ -0,0 +1,43 @@
[project]
name = "deerflow-extension-example"
version = "0.1.0"
description = "A compact standalone example covering every DeerFlow extension contribution kind"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"deerflow-extension-api>=0.1.2,<0.2",
"fastapi>=0.115.0,<1",
"langchain>=1.3,<2",
"langgraph>=1.2.9,<1.3",
]
[project.entry-points."deerflow.extensions"]
example = "deerflow_extension_example:install"
[project.optional-dependencies]
dev = [
"httpx>=0.28,<1",
"pytest>=9,<10",
"ruff>=0.14,<1",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["deerflow_extension_example"]
[tool.ruff]
line-length = 240
target-version = "py312"
[tool.ruff.lint]
select = ["E", "F", "I", "UP"]
[tool.ruff.lint.isort]
known-first-party = ["deerflow_extension_example"]
[tool.ruff.format]
quote-style = "double"
indent-style = "space"

View File

@ -0,0 +1,11 @@
from importlib.metadata import distribution
def test_installed_distribution_exposes_deerflow_extension_entry_point() -> None:
entry_points = [entry_point for entry_point in distribution("deerflow-extension-example").entry_points if entry_point.group == "deerflow.extensions"]
assert [(entry_point.name, entry_point.value) for entry_point in entry_points] == [("example", "deerflow_extension_example:install")]
install = entry_points[0].load()
assert install.__deerflow_api__ == "0.1.2"
assert install.__deerflow_name__ == "example"

View File

@ -0,0 +1,165 @@
from __future__ import annotations
import asyncio
from dataclasses import dataclass, field
from typing import Any
from deerflow_extension_api import (
EXTENSION_TASK_STORE_KEY,
AgentBuildContext,
AgentScope,
ExtensionData,
ExtensionRegistry,
ExtensionRuntimeDeps,
HostPolicySnapshot,
SystemModelRequest,
SystemModelResult,
SystemOperationKind,
TaskInfo,
TaskOutcome,
)
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
from deerflow_extension_example import install
class FakeRegistry:
def __init__(self) -> None:
self.middleware_contributors: list[Any] = []
self.task_lifecycle_contributors: list[Any] = []
self.system_model_observers: list[Any] = []
self.services: list[Any] = []
self.contributed_routers: list[Any] = []
def middlewares(self, contributor: Any) -> None:
self.middleware_contributors.append(contributor)
def task_lifecycle(self, contributor: Any) -> None:
self.task_lifecycle_contributors.append(contributor)
def system_model_observer(self, observer: Any) -> None:
self.system_model_observers.append(observer)
def service(self, service: Any) -> None:
self.services.append(service)
def routers(self, routers: Any) -> None:
self.contributed_routers.extend(routers)
@dataclass
class FakeRuntime:
context: dict[str, Any] = field(default_factory=dict)
@dataclass
class FakeToolRequest:
runtime: FakeRuntime
def test_install_registers_all_five_contribution_kinds() -> None:
registry = FakeRegistry()
install(registry, {})
assert isinstance(registry, ExtensionRegistry)
assert len(registry.middleware_contributors) == 1
assert len(registry.task_lifecycle_contributors) == 1
assert len(registry.system_model_observers) == 1
assert len(registry.services) == 1
assert len(registry.contributed_routers) == 1
assert [route.path for route in registry.contributed_routers[0].routes] == ["/api/extension-example/stats"]
assert install.__deerflow_api__ == "0.1.2"
assert install.__deerflow_name__ == "example"
def test_disabled_extension_registers_nothing() -> None:
registry = FakeRegistry()
install(registry, {"enabled": False})
assert registry.middleware_contributors == []
assert registry.task_lifecycle_contributors == []
assert registry.system_model_observers == []
assert registry.services == []
assert registry.contributed_routers == []
def test_registered_contributions_publish_one_shared_stats_snapshot() -> None:
registry = FakeRegistry()
install(registry, {})
app_store = ExtensionData("app")
task_store = ExtensionData("task-1")
task = TaskInfo(
task_id="task-1",
run_id="run-1",
thread_id="thread-1",
kind="lead",
)
async def exercise_contributions() -> tuple[int, int, dict[str, Any], int]:
lifecycle = registry.task_lifecycle_contributors[0]
await lifecycle.on_task_start(app_store, task_store, task)
placement = registry.middleware_contributors[0].contribute_middlewares(
app_store,
AgentBuildContext(scope=AgentScope.LEAD),
)[0]
async def tool_handler(_request: object) -> str:
return "tool-result"
request = FakeToolRequest(runtime=FakeRuntime(context={EXTENSION_TASK_STORE_KEY: task_store}))
assert await placement.middleware.awrap_tool_call(request, tool_handler) == "tool-result"
await registry.system_model_observers[0].on_system_model_call(
app_store,
task_store,
SystemOperationKind.TITLE,
SystemModelRequest(messages="title prompt"),
SystemModelResult(error=RuntimeError("provider unavailable")),
)
await lifecycle.on_task_stop(
app_store,
task_store,
task,
TaskOutcome.COMPLETED,
)
app = FastAPI()
app.include_router(registry.contributed_routers[0])
async with AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test",
) as client:
before_start = await client.get("/api/extension-example/stats")
await registry.services[0].start(
ExtensionRuntimeDeps(
app_store=app_store,
policy=HostPolicySnapshot(max_subagents_per_run=6),
session_factory=object(),
)
)
response = await client.get("/api/extension-example/stats")
await registry.services[0].stop()
after_stop = await client.get("/api/extension-example/stats")
return (
before_start.status_code,
response.status_code,
response.json(),
after_stop.status_code,
)
before_start, status_code, body, after_stop = asyncio.run(exercise_contributions())
assert before_start == 503
assert status_code == 200
assert after_stop == 503
assert body == {
"scope_id": "app",
"session_factory_available": True,
"host_policy": {"max_subagents_per_run": 6},
"tasks": {"completed": 1},
"tool_calls": 1,
"system_model_calls": {"title": {"calls": 1, "errors": 1}},
}

View File

@ -390,7 +390,7 @@ if ! $SKIP_INSTALL; then
# `--all-packages` propagates extras into workspace members (deerflow-harness
# in particular). Required for postgres extras — see PR #2584.
# Intentionally unquoted to splat multiple `--extra X` pairs.
(cd backend && uv sync --quiet --all-packages $UV_EXTRAS_FLAGS) || { echo "✗ Backend dependency install failed"; exit 1; }
(cd backend && uv sync --locked --quiet --all-packages $UV_EXTRAS_FLAGS) || { echo "✗ Backend dependency install failed"; exit 1; }
(cd frontend && "$DEERFLOW_PNPM_PYTHON" "$DEERFLOW_PNPM_RUNNER" install --silent) || { echo "✗ Frontend dependency install failed"; exit 1; }
echo "✓ Dependencies synced"
else
@ -464,7 +464,7 @@ mkdir -p temp/client_body_temp temp/proxy_temp temp/fastcgi_temp temp/uwsgi_temp
# 1. Gateway API
run_service "Gateway" \
"cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 $GATEWAY_EXTRA_FLAGS > ../logs/gateway.log 2>&1" \
"cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 $GATEWAY_EXTRA_FLAGS > ../logs/gateway.log 2>&1" \
8001 30
# 2. Frontend