mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-09 13:39:26 +00:00
feat(extensions): add gateway contribution points and packaged extension management (#4780)
* feat(extensions): add gateway services and routers * feat(extensions): add standalone reference extension * fix(extensions): harden contributed gateway routes * docs(extensions): document gateway contribution points * feat(extensions): add operator CLI for packaged extension management Add `deerflow extensions install/list/enable/disable/remove` plus the root `make extension-*` wrappers, backed by an `ExtensionManager` that owns one transaction over backend/pyproject.toml, backend/uv.lock, the managed source snapshot, the uv environment, and the `plugins:` block in config.yaml. Install accepts a package requirement, a public HTTPS Git URL, or a local directory. Local directories are copied to backend/extensions/sources/ as deployable snapshots rather than editable installs, and the root .dockerignore re-includes that tree so snapshots reach the backend builder. Remote sources are HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock Docker builder cannot reproduce them. Because environment configuration can still resolve a plain package name to a local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed by an audit of the new lock: any local reference the stock image build cannot reproduce rolls back the whole transaction. A config carrying duplicate top-level `plugins:` keys is rejected outright rather than managed against one block while the Gateway reads another. Dependency synchronization now has one lock authority. The `extensions` dependency group joins [tool.uv].default-groups, every startup path syncs the same lock with --locked and launches with --no-sync, and the Docker images move to uv 0.11.1 for the --no-workspace boundary the manager needs. Loader gains `enabled`, `name` and `package` fields so a disabled extension is skipped before resolution and import. Co-authored-by: Codex <codex@openai.com> * fix(extensions): stop the managed plugins rewrite from destroying config Two data-safety defects in the managed `plugins:` block writer. The "next top-level key" boundary was a regex matching only `[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a config may legally carry any top-level key, and a key the pattern cannot recognize did not fail loudly — it read as "no next section", and the rewrite replaced that neighbour and its entire subtree with the managed block. `my.key`, `2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a plain `extension-enable`/`disable`. Both boundaries now come from the YAML parser's node marks, so key shape is irrelevant. The file-final branch never consulted the trailing-comment scan the has-next-key branch used, so any comment below the block was dropped. Since the manager appends `plugins:` at end of file, that is the steady-state shape for most installs: an operator note below the block was destroyed on the next toggle. Separately, every managed install wrote `required: true` while the loader defaults to false. That turned any later load failure — broken wheel, missing native library, deleted snapshot — into a Gateway startup abort recoverable only with shell access. New records are now written `required: false`, with an explicit `install --required` opt-in; adopting an existing hand-written record still preserves the operator's own choice. * fix(extensions): harden the manager transaction and correct its docs Follow-up hardening on the extension package manager. Security posture, which the docs already claimed: - Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps the interpreter that the entry-point probe then imports and calls, and every later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS validation the HTTPS-only source rule depends on. Neither is an index, proxy, cache or credential-provider setting, so neither was covered by the carve-out. - Recognize run-together and all-caps secret query parameters (`accesstoken`, `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires on case transitions, so only the separated spellings were caught. Short generic words stay boundary-anchored, so `?keyword=` remains installable. - Validate the config before running any uv command. `uv add`/`uv sync` execute the package's build backend, so a config the manager could never write to must fail before that code runs rather than afterwards through rollback. Transaction integrity: - Run the second dependency-file restore from a `finally`. The recovery sync runs without `--locked` when the checkout had no lock, so uv writes one while resolving; if that sync then failed, the restore was skipped and the operator kept a lock file they never had. A failing recovery sync now also reports the original failure instead of replacing it. - Skip the recovery sync on cancellation. Answering Ctrl-C with a full dependency resolve invites a second interrupt that escapes the handler and strands the checkout mid-transaction; the declarations are already restored and the next locked startup sync reconciles the environment. - Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so contention surfaced as `Permission denied` rather than serializing. - Locate the entry-point probe's JSON payload instead of parsing stdout's first line, so a `sitecustomize`/`.pth` banner cannot roll back a good install. - Warn when the lock records a loopback source. `127.0.0.1` inside the image builder is a different machine, but unlike an environment-driven wheelhouse resolution this is a source the operator typed deliberately, so it is reported rather than rolled back. Private-network indexes are untouched: a builder on that network can reach them. Docs: the blanket claim that failed operations restore the config file was wrong — the conflict branches deliberately preserve a concurrent external edit and leave `remove` deactivated. Document that, the `required: false` default, the config preflight, the interrupt behaviour, and where the plugins-block boundaries come from. * test(gateway): pin the request-path projection agreement `get_request_route_path()` imports the private `starlette._utils.get_route_path` so the auth and CSRF predicates classify the exact string Starlette's router matches on. Its requirement is not "strip root_path correctly" but "return what the dispatcher is matching", so delegating to the router's own implementation keeps the two in lockstep by construction. Keep the private import rather than vendoring a copy: an import that disappears fails loudly at startup, while a stale copy diverges silently at a security boundary. Cover the property directly instead of the mechanism, so the tests survive a future reimplementation: - projection edge cases, including the segment-boundary guard that keeps root_path="/api" from slicing "/apifoo/models" into a string the router would never match - agreement with the router under nested mounts - the two bypasses these predicates exist to prevent: a protected route mounted under the "/health" public prefix must still 401, and a POST mounted under "/api/webhooks" must still require a CSRF token Both are verified to fail when the projection is reverted to `request.url.path` (9/13 red) and when a plausible vendored copy omits the boundary guard (the 2 boundary cases red). Declare starlette as a bounded direct dependency so a bump — which is security-relevant here — shows up in review rather than arriving silently through FastAPI. * ci: pin uv to the version production ships ExtensionManager is not a consumer of uv the build tool -- it is a program whose whole job is driving `uv` as a subprocess, depending on its CLI behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into `[dependency-groups] extensions`) and on the `uv.lock` serialization format. uv is closer to a runtime dependency with a contract than to incidental tooling. backend/Dockerfile pins that binary to 0.11.1, but all eight astral-sh/setup-uv steps installed whatever was latest at run time, so CI exercised the manager against a uv that is not the uv production runs. The sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI stays green because the same uv reads back what it wrote, and the pinned uv in the production image cannot read the committed lock. `uv lock --check` is version-sensitive for the same reason -- it verifies the lock is what *this* uv would produce, and two versions can emit equivalent but non-identical output. Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so the steps share input and caching behavior. Pinning alone drifts apart again on the next bump, so add a constraint test in the style of test_compose_default_bind_host.py: the Dockerfile's UV_IMAGE tag is the single source of truth, and both compose defaults plus every setup-uv step must match it. Verified to fail when a pin drifts, when a step omits `version`, and -- the real scenario -- when the Dockerfile is bumped alone, which lights up the workflows and both compose files at once. * fix(gateway): state the extension route auth limit and abort a failed dev sync Two scoped review follow-ups. README: contributed routers cannot enter the host's reserved public prefixes, which makes every extension endpoint session-authenticated -- there is no way to expose an unauthenticated route. The rejection rule was documented but its consequence was not, so inbound provider webhooks and public status endpoints read as merely undocumented rather than out of scope for this release. docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e` already stopped the script there -- uvicorn was not being started against a stale environment -- but it exited on a bare uv exit code with no indication of what to do. Abort explicitly with the cause and the fix. Tests slice the sync block out of the real script and run it against a stub uv, so they exercise the shipped code rather than a copy of it (/app/backend only exists inside the container). They cover the success path, the retry that recovers, the abort, and the guidance. Verified against the pre-fix script: only the guidance case goes red, confirming the abort itself was already correct. * fix(extensions): point Git SSH shorthand at the HTTPS correction Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git` reached the scheme rules looking like a bare path and was rejected with "local path references are not deployable; pass a local directory so DeerFlow can snapshot it". The operator asked for a remote source, so that guidance points at the wrong fix. Detect the shorthand ahead of the scheme rules and report the public-HTTPS correction instead. The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose `host` reads as a URL scheme and produced the generic HTTPS message. Both spellings now share one message, as does the PEP 508 named form. * docs: keep the root extension summary within its new budget #4799 split the depth out of the module guides and added a size gate; the root file's job is now orientation, and this branch had pushed it 192 bytes past the soft limit. The manager transaction, source rules, and lock discipline are already stated in full in the extensions guide, so the root keeps the one-line orientation and points there instead of restating them. --------- Co-authored-by: Codex <codex@openai.com>
This commit is contained in:
parent
e4a7a04719
commit
c542185a7f
@ -69,3 +69,10 @@ backend/.coverage
|
||||
!README.md
|
||||
!frontend/README.md
|
||||
!backend/README.md
|
||||
|
||||
# Extension manager snapshots must enter the backend builder intact. A Python
|
||||
# package may require README metadata, native modules, or package assets that
|
||||
# the general image-context exclusions above intentionally omit elsewhere.
|
||||
!backend/extensions/
|
||||
!backend/extensions/sources/
|
||||
!backend/extensions/sources/**
|
||||
|
||||
@ -35,7 +35,10 @@ jobs:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
|
||||
6
.github/workflows/backend-unit-tests.yml
vendored
6
.github/workflows/backend-unit-tests.yml
vendored
@ -33,6 +33,9 @@ jobs:
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
|
||||
- name: Install backend dependencies (documented default)
|
||||
working-directory: backend
|
||||
@ -85,6 +88,9 @@ jobs:
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
|
||||
3
.github/workflows/label-sync.yml
vendored
3
.github/workflows/label-sync.yml
vendored
@ -30,6 +30,9 @@ jobs:
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
|
||||
- name: Sync labels
|
||||
run: uv run --with pyyaml python scripts/sync_labels.py
|
||||
|
||||
3
.github/workflows/lint-check.yml
vendored
3
.github/workflows/lint-check.yml
vendored
@ -50,6 +50,9 @@ jobs:
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
|
||||
- name: Check uv.lock is in sync
|
||||
working-directory: backend
|
||||
|
||||
6
.github/workflows/replay-e2e.yml
vendored
6
.github/workflows/replay-e2e.yml
vendored
@ -57,6 +57,9 @@ jobs:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
@ -77,6 +80,9 @@ jobs:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
- name: Install backend dependencies (replay gateway)
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
3
.github/workflows/skill-review-ci.yml
vendored
3
.github/workflows/skill-review-ci.yml
vendored
@ -48,6 +48,9 @@ jobs:
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
# Must match backend/Dockerfile's UV_IMAGE tag; pinned by backend/tests/test_ci_uv_version_pin.py
|
||||
version: "0.11.1"
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
|
||||
18
AGENTS.md
18
AGENTS.md
@ -57,6 +57,7 @@ deer-flow/
|
||||
├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
|
||||
├── backend/ # Python backend — see backend/AGENTS.md
|
||||
│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
|
||||
│ ├── extensions/sources/ # Deployable snapshots of locally installed Python extensions
|
||||
│ ├── packages/extension-api/ # deerflow-extension-api package (import: deerflow_extension_api.*) — public extension contract
|
||||
│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
|
||||
│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
|
||||
@ -66,6 +67,7 @@ deer-flow/
|
||||
│ # Managed integration skill packs are global at .deer-flow/integrations/skills/{provider}/
|
||||
│ # Integration credentials and enabled state remain per-user
|
||||
├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
|
||||
├── examples/deerflow-extension-example/ # Standalone package demonstrating all extension contribution kinds
|
||||
├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
|
||||
├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
|
||||
└── docs/ # Cross-cutting docs, plans, and design notes
|
||||
@ -73,8 +75,15 @@ deer-flow/
|
||||
|
||||
Third-party extensions are loaded from a top-level `plugins:` list in `config.yaml`
|
||||
(operator-controlled on purpose — that list causes code to be imported, so it is deliberately
|
||||
kept out of the API-writable `extensions_config.json`). See the Extension System section in
|
||||
[backend/AGENTS.md](backend/AGENTS.md).
|
||||
kept out of the API-writable `extensions_config.json`). Packaged extensions can contribute
|
||||
middleware, task lifecycle, system-model observers, Gateway services, and FastAPI HTTP
|
||||
routers; the [reference extension](examples/deerflow-extension-example/) demonstrates all
|
||||
five. Manage them with `deerflow extensions install/list/enable/disable/remove` or the root
|
||||
`make extension-*` wrappers. Every mutation requires a Gateway restart, and both build
|
||||
hooks and extension code execute with Gateway privileges, so only trusted operator sources
|
||||
belong in this path. The manager transaction, accepted source forms, lock discipline, and
|
||||
contribution contract live in
|
||||
[the extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md).
|
||||
|
||||
Runtime config lives at the **repo root**: copy `config.example.yaml` → `config.yaml`
|
||||
(main app config) and `extensions_config.example.json` → `extensions_config.json` (MCP
|
||||
@ -105,6 +114,11 @@ make support-bundle # Generate redacted troubleshooting summary, AI issue draft
|
||||
make config # Generate local config files from the examples
|
||||
make check # Check that required tools are installed
|
||||
make install # Install all dependencies (frontend + backend + pre-commit hooks)
|
||||
make extension-install SOURCE=... # Install and enable a trusted Python extension
|
||||
make extension-list # List configured Python extensions
|
||||
make extension-enable NAME=... # Enable an installed extension (restart required)
|
||||
make extension-disable NAME=... # Disable without uninstalling (restart required)
|
||||
make extension-remove NAME=... # Remove package and config entry (restart required)
|
||||
make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
|
||||
make start # Start all services in production mode (local, optimized)
|
||||
make stop # Stop all running services
|
||||
|
||||
32
Makefile
32
Makefile
@ -1,6 +1,6 @@
|
||||
# DeerFlow - Unified Development Environment
|
||||
|
||||
.PHONY: help config config-upgrade check check-agent-guidance install setup doctor support-bundle detect-thread-boundaries detect-blocking-io dev dev-daemon start start-daemon nginx stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway docker-logs-redis
|
||||
.PHONY: help config config-upgrade check check-agent-guidance install extension-install extension-list extension-enable extension-disable extension-remove setup doctor support-bundle detect-thread-boundaries detect-blocking-io dev dev-daemon start start-daemon nginx stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway docker-logs-redis
|
||||
|
||||
BASH ?= bash
|
||||
BACKEND_UV_RUN = cd backend && uv run
|
||||
@ -30,6 +30,11 @@ help:
|
||||
@echo " make detect-thread-boundaries - Inventory backend executor/thread/event-loop boundaries"
|
||||
@echo " make detect-blocking-io - Inventory blocking IO that may block the backend event loop"
|
||||
@echo " make install - Install all dependencies (frontend + backend + pre-commit hooks)"
|
||||
@echo " make extension-install SOURCE=... - Install and enable a trusted Python extension"
|
||||
@echo " make extension-list - List configured Python extensions"
|
||||
@echo " make extension-enable NAME=... - Enable an installed extension"
|
||||
@echo " make extension-disable NAME=... - Disable an extension without uninstalling it"
|
||||
@echo " make extension-remove NAME=... - Uninstall a managed extension"
|
||||
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
|
||||
@echo " make dev - Start all services in development mode (with hot-reloading)"
|
||||
@echo " make dev-daemon - Start dev services in background (daemon mode)"
|
||||
@ -84,7 +89,7 @@ check-agent-guidance:
|
||||
# Install all dependencies
|
||||
install:
|
||||
@echo "Installing backend dependencies..."
|
||||
@cd backend && uv sync
|
||||
@cd backend && uv sync --locked
|
||||
@echo "Installing frontend dependencies..."
|
||||
@cd frontend && $(FRONTEND_PNPM) install
|
||||
@echo "Installing pre-commit hooks..."
|
||||
@ -100,6 +105,29 @@ install:
|
||||
@echo " make setup-sandbox"
|
||||
@echo ""
|
||||
|
||||
extension-install: export DEER_FLOW_EXTENSION_SOURCE := $(value SOURCE)
|
||||
extension-install:
|
||||
$(if $(and $(filter command line,$(origin SOURCE)),$(strip $(value SOURCE))),,$(error usage: make extension-install SOURCE=<package|git-url|dir>))
|
||||
@cd backend && uv run --frozen --no-group extensions deerflow extensions install --source-env __deerflow_extension_source__
|
||||
|
||||
extension-list:
|
||||
@cd backend && uv run --frozen --no-group extensions deerflow extensions list
|
||||
|
||||
extension-enable: export DEER_FLOW_EXTENSION_NAME := $(value NAME)
|
||||
extension-enable:
|
||||
$(if $(and $(filter command line,$(origin NAME)),$(strip $(value NAME))),,$(error usage: make extension-enable NAME=<extension>))
|
||||
@cd backend && uv run --frozen --no-group extensions deerflow extensions enable --name-env __deerflow_extension_name__
|
||||
|
||||
extension-disable: export DEER_FLOW_EXTENSION_NAME := $(value NAME)
|
||||
extension-disable:
|
||||
$(if $(and $(filter command line,$(origin NAME)),$(strip $(value NAME))),,$(error usage: make extension-disable NAME=<extension>))
|
||||
@cd backend && uv run --frozen --no-group extensions deerflow extensions disable --name-env __deerflow_extension_name__
|
||||
|
||||
extension-remove: export DEER_FLOW_EXTENSION_NAME := $(value NAME)
|
||||
extension-remove:
|
||||
$(if $(and $(filter command line,$(origin NAME)),$(strip $(value NAME))),,$(error usage: make extension-remove NAME=<extension>))
|
||||
@cd backend && uv run --frozen --no-group extensions deerflow extensions remove --name-env __deerflow_extension_name__
|
||||
|
||||
# Pre-pull sandbox Docker image (optional but recommended)
|
||||
setup-sandbox:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/setup-sandbox.sh
|
||||
|
||||
126
README.md
126
README.md
@ -835,21 +835,117 @@ Advanced deployments can enable pluggable authorization with `authorization.enab
|
||||
|
||||
Advanced deployments can also extend the agent runtime itself by declaring zero-argument `AgentMiddleware` classes under `extensions.middlewares` in `config.yaml` or `extensions_config.json`. DeerFlow loads the same configured class list into the lead-agent and subagent pipelines after their built-in runtime middlewares and loop/token guards, but before the terminal-response/safety/clarification tail, so enterprise forks can add domain guardrails, tool-call governance, or observability hooks without patching the built-in middleware builders. Missing packages, invalid classes, and broken modules fail loudly at agent creation. Treat `config.yaml` and `extensions_config.json` as trusted operator-controlled files: middleware paths are code execution, just like custom tool, model, sandbox, guardrail, MCP server, and MCP interceptor declarations. Gateway skill/MCP toggle endpoints preserve this field but do not expose an API write path for `extensions.middlewares`. Per-context parameterization and separate lead-only/subagent-only middleware lists are not supported yet.
|
||||
|
||||
For packaged and configurable runtime integrations, use the top-level `plugins:` list in
|
||||
`config.yaml`. A plugin exposes `module.path:install`, depends only on the standalone
|
||||
`deerflow-extension-api` contract package, and can register exactly three contribution
|
||||
kinds: isolated middleware at semantic lead/subagent model or tool positions, lead and
|
||||
subagent task-lifecycle hooks, and observers for DeerFlow-owned system model calls such as
|
||||
goal evaluation, memory extraction, title generation, and summarization. DeerFlow allocates
|
||||
a task-scoped extension store only when one of those contribution kinds is registered and
|
||||
uses the Gateway's canonical notification loop for lifecycle and system-model callbacks,
|
||||
including subagents that execute on isolated loops. Plugin order is deterministic,
|
||||
per-plugin configuration is passed to `install()`, and `required: true` makes load failure
|
||||
abort startup; otherwise failures are reported and skipped. Plugins load once when the
|
||||
Gateway app is constructed, so changes require a restart. Because this imports Python code,
|
||||
`plugins:` is intentionally unavailable through the API-writable
|
||||
`extensions_config.json`. In Docker deployments, install the plugin in the Gateway image
|
||||
rather than only in the host environment. See `config.example.yaml` for configuration.
|
||||
For packaged and configurable runtime integrations, use DeerFlow's extension manager.
|
||||
It accepts a Python package requirement, a public HTTPS Git URL, or a local directory, installs the
|
||||
package into the backend's dedicated `extensions` dependency group, updates
|
||||
`backend/uv.lock`, and adds an enabled entry to the startup-only top-level `plugins:` list
|
||||
in `config.yaml`:
|
||||
|
||||
```bash
|
||||
# PyPI — pin a version for a reproducible deployment
|
||||
make extension-install SOURCE="deerflow-extension-acme==1.2.3"
|
||||
|
||||
# Public HTTPS Git — pin an immutable commit
|
||||
make extension-install \
|
||||
SOURCE="git+https://github.com/acme/deerflow-extension-acme.git@0123456789abcdef0123456789abcdef01234567"
|
||||
|
||||
# Local package — an absolute path avoids Make's backend-relative working directory
|
||||
make extension-install SOURCE="$PWD/examples/deerflow-extension-example"
|
||||
|
||||
make extension-list
|
||||
make extension-disable NAME=acme
|
||||
make extension-enable NAME=acme
|
||||
make extension-remove NAME=acme
|
||||
```
|
||||
|
||||
Installation is interactive because package installation can execute Python build hooks,
|
||||
and the loaded extension later runs with Gateway privileges. For an already-reviewed
|
||||
source, automation can acknowledge that boundary explicitly with
|
||||
`cd backend && uv run --frozen --no-group extensions deerflow extensions install <source> --yes`.
|
||||
The manager requires uv 0.8.0 or newer; the provided Docker images pin uv 0.11.1.
|
||||
The other direct
|
||||
commands are `deerflow extensions list`, `enable NAME`, `disable NAME`, and `remove NAME`;
|
||||
`NAME` may be the extension name, Python distribution, or `module:install` value. Do not
|
||||
put credentials in a source URL — a URL carrying embedded userinfo or a credential-looking
|
||||
query parameter is rejected before uv runs. Remote Git sources must use public HTTPS; SSH
|
||||
Git URLs are rejected because the stock Docker builder does not forward host SSH
|
||||
credentials. Installing from a loopback URL is allowed for local tooling but warns, because
|
||||
`127.0.0.1` recorded in the lock is a different machine inside the Docker builder.
|
||||
|
||||
A managed package declares exactly one standard PEP 621 entry point:
|
||||
|
||||
```toml
|
||||
[project.entry-points."deerflow.extensions"]
|
||||
acme = "acme_deerflow_extension:install"
|
||||
```
|
||||
|
||||
That callable uses the standalone `deerflow-extension-api` contract and can register five
|
||||
contribution kinds: isolated middleware at semantic lead/subagent model or tool positions,
|
||||
lead and subagent task-lifecycle hooks, observers for DeerFlow-owned model calls that are
|
||||
not wrapped by middleware model-call hooks (goal, memory, title, and summarization),
|
||||
Gateway-lifetime services, and eager FastAPI HTTP routers. The contract package has no
|
||||
framework dependencies; extensions must declare FastAPI, LangChain, LangGraph, or other
|
||||
libraries they import.
|
||||
|
||||
DeerFlow allocates a task-scoped extension store only for middleware, lifecycle, or
|
||||
system-model observation. Services receive app-scoped runtime dependencies after Gateway
|
||||
persistence is ready and stop in reverse order after active runs drain. Extension HTTP
|
||||
routers are mounted after every host route; definite shadows and routes entering the
|
||||
host's authentication- or CSRF-exempt paths are rejected with attributed diagnostics,
|
||||
while unrelated routers continue to load. Because the host's public paths are a reserved
|
||||
prefix list that extensions cannot enter, **every contributed endpoint requires an
|
||||
authenticated session** — there is currently no way for an extension to expose an
|
||||
unauthenticated route, so inbound provider webhooks and public status endpoints are out of
|
||||
scope for this release. Router startup/shutdown hooks, custom lifespans,
|
||||
Mounts, and WebSocket routes are not accepted; lifetime resources belong in
|
||||
`ExtensionService`, and WebSocket contributions require a future host-owned
|
||||
authentication/Origin wrapper. Lifecycle and system-model callbacks use the Gateway's
|
||||
canonical notification loop, including subagents on isolated loops.
|
||||
Plugin order is deterministic, per-plugin configuration is passed to `install()`, and
|
||||
`required: true` makes load failure abort startup; otherwise failures are reported and
|
||||
skipped. `enabled: false` skips resolution and import. The manager preserves the extension's
|
||||
private `config` when toggling it and writes `name`, `package`, `use`, `enabled`, and
|
||||
`required` metadata for managed installs. Installs are recorded `required: false` so a
|
||||
later broken extension is reported rather than blocking Gateway startup; pass
|
||||
`extensions install <source> --required` when the package's absence should abort startup
|
||||
instead. Plugins load once when the Gateway app is
|
||||
constructed, so install, enable, disable, remove, and manual `plugins:` edits all require a
|
||||
Gateway restart. Because this imports Python code, `plugins:` is intentionally unavailable
|
||||
through the API-writable `extensions_config.json`.
|
||||
|
||||
Management commands bootstrap the checkout environment without the extension group via
|
||||
`uv run --frozen --no-group extensions`. Frozen mode lets `disable` and `remove` start even
|
||||
when an installed extension's remote source or managed snapshot has become unavailable,
|
||||
while a fresh checkout can still create the non-extension environment from the existing lock. The
|
||||
manager itself owns the subsequent locked dependency transaction.
|
||||
Mutations for one checkout are serialized through a process lock. The initial manager
|
||||
surface is create/remove rather than in-place upgrade: to change an installed source, save
|
||||
its private `plugins[].config`, remove it, reinstall the new pin, and restore that config.
|
||||
|
||||
Local-directory installs are copied into
|
||||
`backend/extensions/sources/<normalized-distribution>/`; this deployable snapshot, rather
|
||||
than the original directory, is recorded in the lock. Git metadata, virtual environments,
|
||||
bytecode caches, symbolic links, and likely credential files are not accepted as snapshot
|
||||
content. Review what you install anyway: filtering accidental files does not sandbox an
|
||||
extension, its build backend, or its runtime code.
|
||||
|
||||
Local `make dev`/`make start`, Docker development, and the production Gateway image all
|
||||
consume the same `backend/pyproject.toml` and `backend/uv.lock`. Local and Docker-dev
|
||||
launchers perform a locked sync before starting; the production image performs that sync
|
||||
during its build and includes managed local snapshots in the build context. Gateway runtime
|
||||
commands then use the already-created environment without resolving or installing packages.
|
||||
Local and Docker-development pre-start syncs may download missing locked artifacts. A
|
||||
production deployment instead downloads them only during the explicit install or image
|
||||
build; starting the resulting production Gateway container never resolves or installs
|
||||
extensions from the network. A local wheel or `file://` Git URL is rejected because it
|
||||
would not exist in the Docker build context; pass a source directory to create a managed
|
||||
snapshot instead. Because environment configuration (such as a `UV_FIND_LINKS` wheelhouse)
|
||||
can still resolve a plain package name to a local wheel, the manager audits every new lock
|
||||
before enabling the extension: any local reference the stock image build cannot reproduce
|
||||
rolls back the entire install or removal.
|
||||
Rebuild with `make up` after changing the managed extension set. See
|
||||
`config.example.yaml` and the
|
||||
[reference extension](examples/deerflow-extension-example/) for a complete example.
|
||||
|
||||
Gateway-generated follow-up suggestions now normalize both plain-string model output and block/list-style rich content before parsing the JSON array response, so provider-specific content wrappers do not silently drop suggestions.
|
||||
|
||||
|
||||
@ -90,6 +90,11 @@ When making code changes, you MUST update the relevant documentation:
|
||||
```bash
|
||||
make check # Check system requirements
|
||||
make install # Install all dependencies (frontend + backend)
|
||||
make extension-install SOURCE=... # Install and enable a trusted Python extension
|
||||
make extension-list # List configured Python extensions
|
||||
make extension-enable NAME=... # Enable an installed extension
|
||||
make extension-disable NAME=... # Disable an extension without uninstalling it
|
||||
make extension-remove NAME=... # Remove a managed extension
|
||||
make detect-thread-boundaries # Inventory backend executor/thread/event-loop boundaries
|
||||
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make start # Start production services locally
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
# Stage 3 (runtime): clean image without compiler toolchain for production
|
||||
|
||||
# UV source image (override for restricted networks that cannot reach ghcr.io)
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.7.20
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.1
|
||||
FROM ${UV_IMAGE} AS uv-source
|
||||
|
||||
# ── Stage 1: Builder ──────────────────────────────────────────────────────────
|
||||
@ -28,6 +28,7 @@ RUN if [ -n "${APT_MIRROR}" ]; then \
|
||||
# Install build tools + Node.js (build-essential needed for native Python extensions)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
git \
|
||||
build-essential \
|
||||
gnupg \
|
||||
ca-certificates \
|
||||
@ -70,7 +71,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
esac; \
|
||||
extras_flags="$extras_flags --extra $extra"; \
|
||||
done; \
|
||||
UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync --extra redis $extras_flags'
|
||||
UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync --locked --extra redis $extras_flags'
|
||||
|
||||
# UTF-8 locale prevents UnicodeEncodeError on Chinese/emoji content in minimal
|
||||
# containers where locale configuration may be missing and the default encoding is not UTF-8.
|
||||
@ -88,7 +89,7 @@ COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
EXPOSE 8001
|
||||
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
# ── Stage 3: Runtime ──────────────────────────────────────────────────────────
|
||||
# Clean image without build-essential — reduces size (~200 MB) and attack surface.
|
||||
|
||||
@ -3,11 +3,11 @@ DEER_FLOW_HOME := $(abspath $(DEER_FLOW_HOME))
|
||||
BACKEND_SANDBOX_HOME := $(abspath $(CURDIR)/sandbox)
|
||||
|
||||
install:
|
||||
uv sync
|
||||
uv sync --locked
|
||||
|
||||
dev:
|
||||
mkdir -p "$(DEER_FLOW_HOME)" "$(BACKEND_SANDBOX_HOME)"
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 DEER_FLOW_HOME="$(DEER_FLOW_HOME)" uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 \
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 DEER_FLOW_HOME="$(DEER_FLOW_HOME)" uv run --locked uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 \
|
||||
--reload \
|
||||
--reload-include='*.yaml' \
|
||||
--reload-include='.env' \
|
||||
@ -17,7 +17,7 @@ dev:
|
||||
--reload-exclude="$(DEER_FLOW_HOME)"
|
||||
|
||||
gateway:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run --locked uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
|
||||
test:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live" tests/ -v
|
||||
|
||||
@ -614,6 +614,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
||||
ExtensionLoadError,
|
||||
initialize_runtime_diagnostics,
|
||||
load_extensions,
|
||||
record_runtime_diagnostics,
|
||||
set_loaded_extensions,
|
||||
)
|
||||
|
||||
@ -738,6 +739,14 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
||||
"""
|
||||
return {"status": "healthy", "service": "deer-flow-gateway"}
|
||||
|
||||
# Extension routes are deliberately last: FastAPI/Starlette dispatches in
|
||||
# registration order, so every host route (including conditional routes
|
||||
# and /health) keeps precedence. Definite shadows are rejected with an
|
||||
# attributed diagnostic while unrelated extension routers still mount.
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
record_runtime_diagnostics(include_contributed_routers(app, loaded_extensions))
|
||||
|
||||
return app
|
||||
|
||||
|
||||
|
||||
@ -26,6 +26,7 @@ from app.gateway.auth_disabled import (
|
||||
)
|
||||
from app.gateway.authz import AuthContext, resolve_route_permissions
|
||||
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
|
||||
from app.gateway.request_path import get_request_route_path
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
|
||||
# Paths that never require authentication.
|
||||
@ -86,7 +87,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
||||
if _is_public(request.url.path):
|
||||
if _is_public(get_request_route_path(request)):
|
||||
return await call_next(request)
|
||||
|
||||
internal_user = None
|
||||
|
||||
@ -17,10 +17,13 @@ from starlette.types import ASGIApp
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.session_cookie_state import SESSION_COOKIE_ISSUED_STATE_ATTR, SESSION_COOKIE_MAX_AGE_STATE_ATTR, SESSION_COOKIE_SECURE_STATE_ATTR, SKIP_AUTH_CSRF_COOKIE_STATE_ATTR
|
||||
from app.gateway.auth_disabled import is_auth_disabled
|
||||
from app.gateway.request_path import get_request_route_path
|
||||
|
||||
CSRF_COOKIE_NAME = "csrf_token"
|
||||
CSRF_HEADER_NAME = "X-CSRF-Token"
|
||||
CSRF_TOKEN_LENGTH = 64 # bytes
|
||||
_CSRF_STATE_CHANGING_METHODS: frozenset[str] = frozenset({"POST", "PUT", "DELETE", "PATCH"})
|
||||
_CSRF_EXEMPT_EXACT_PATHS: frozenset[str] = frozenset({"/api/v1/auth/me"})
|
||||
|
||||
|
||||
def is_secure_request(request: Request) -> bool:
|
||||
@ -39,19 +42,20 @@ def should_check_csrf(request: Request) -> bool:
|
||||
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
|
||||
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
|
||||
"""
|
||||
if request.method not in ("POST", "PUT", "DELETE", "PATCH"):
|
||||
if request.method not in _CSRF_STATE_CHANGING_METHODS:
|
||||
return False
|
||||
|
||||
if is_auth_disabled():
|
||||
return False
|
||||
|
||||
path = request.url.path.rstrip("/")
|
||||
# Exempt /api/v1/auth/me endpoint
|
||||
if path == "/api/v1/auth/me":
|
||||
route_path = get_request_route_path(request)
|
||||
path = route_path.rstrip("/")
|
||||
# Exempt host-owned endpoints that implement their own request posture.
|
||||
if path in _CSRF_EXEMPT_EXACT_PATHS:
|
||||
return False
|
||||
# Inbound webhooks authenticate themselves via provider-specific signatures
|
||||
# (e.g. GitHub's X-Hub-Signature-256), not the CSRF double-submit cookie.
|
||||
if request.url.path.startswith("/api/webhooks/"):
|
||||
if route_path.startswith("/api/webhooks/"):
|
||||
return False
|
||||
return True
|
||||
|
||||
@ -71,7 +75,7 @@ def is_auth_endpoint(request: Request) -> bool:
|
||||
|
||||
Auth endpoints don't need CSRF validation on first call (no token).
|
||||
"""
|
||||
return request.url.path.rstrip("/") in _AUTH_EXEMPT_PATHS
|
||||
return get_request_route_path(request).rstrip("/") in _AUTH_EXEMPT_PATHS
|
||||
|
||||
|
||||
def _host_with_optional_port(hostname: str, port: int | None, scheme: str) -> str:
|
||||
|
||||
@ -22,7 +22,7 @@ import logging
|
||||
import os
|
||||
from collections.abc import AsyncGenerator, Callable
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
from typing import TYPE_CHECKING, TypeVar, cast
|
||||
from typing import TYPE_CHECKING, Any, TypeVar, cast
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from langgraph.types import Checkpointer
|
||||
@ -420,6 +420,9 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
|
||||
# Initialize persistence engine BEFORE checkpointer so that
|
||||
# auto-create-database logic runs first (postgres backend).
|
||||
# Own cleanup before initialization so partial startup and host
|
||||
# cancellation cannot strand an engine created along the way.
|
||||
stack.push_async_callback(close_engine)
|
||||
await init_engine_from_config(config.database)
|
||||
|
||||
app.state.checkpointer = await stack.enter_async_context(make_checkpointer(config))
|
||||
@ -439,6 +442,35 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
app.state.run_store = MemoryRunStore()
|
||||
app.state.feedback_repo = None
|
||||
|
||||
# Services are app-scoped. Capture this app's immutable extension set
|
||||
# once and close over the same object for teardown; the process-wide
|
||||
# singleton may be replaced by another app/test before shutdown.
|
||||
from deerflow.extensions import EMPTY_EXTENSIONS, record_runtime_diagnostics
|
||||
from deerflow.extensions.gateway import start_services, stop_services
|
||||
|
||||
extensions = getattr(app.state, "extensions", EMPTY_EXTENSIONS)
|
||||
attempted_services: list[tuple[str, Any]] = []
|
||||
|
||||
async def stop_extension_services() -> None:
|
||||
record_runtime_diagnostics(
|
||||
await stop_services(
|
||||
extensions,
|
||||
service_entries=attempted_services,
|
||||
)
|
||||
)
|
||||
|
||||
# Register cleanup before starting: start() can partially acquire
|
||||
# resources and then fail or be cancelled.
|
||||
stack.push_async_callback(stop_extension_services)
|
||||
record_runtime_diagnostics(
|
||||
await start_services(
|
||||
extensions,
|
||||
config,
|
||||
sf,
|
||||
attempted_services=attempted_services,
|
||||
)
|
||||
)
|
||||
|
||||
from deerflow.persistence.thread_meta import make_thread_store
|
||||
|
||||
app.state.thread_store = make_thread_store(sf, app.state.store)
|
||||
@ -544,7 +576,6 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
),
|
||||
),
|
||||
)
|
||||
await close_engine()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
9
backend/app/gateway/request_path.py
Normal file
9
backend/app/gateway/request_path.py
Normal file
@ -0,0 +1,9 @@
|
||||
"""Canonical request-path projection shared by routing security middleware."""
|
||||
|
||||
from starlette._utils import get_route_path
|
||||
from starlette.requests import Request
|
||||
|
||||
|
||||
def get_request_route_path(request: Request) -> str:
|
||||
"""Return the same root-path-adjusted value Starlette routes match."""
|
||||
return get_route_path(request.scope)
|
||||
@ -41,9 +41,13 @@ def install_newer_minor_api(registry: ExtensionRegistry, config: Mapping[str, An
|
||||
|
||||
|
||||
def install_partial_then_raise(registry: ExtensionRegistry, config: Mapping[str, Any]) -> None:
|
||||
"""Registers two contributors, then fails — exercises rollback."""
|
||||
registry.middlewares(_Contributor("partial"))
|
||||
registry.task_lifecycle(_Contributor("partial"))
|
||||
"""Register every contribution kind, then fail to exercise five-bucket rollback."""
|
||||
partial = _Contributor("partial")
|
||||
registry.middlewares(partial)
|
||||
registry.task_lifecycle(partial)
|
||||
registry.system_model_observer(partial)
|
||||
registry.service(partial)
|
||||
registry.routers((partial,))
|
||||
raise ValueError("boom")
|
||||
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
"""Public contracts for DeerFlow extensions.
|
||||
|
||||
This package MUST NOT import `deerflow`. Everything an extension needs to
|
||||
integrate lives here, so an extension depends on this package alone and can
|
||||
be released independently of the host.
|
||||
This package MUST NOT import `deerflow`. Every host contract an extension
|
||||
needs lives here, while framework imports remain direct extension dependencies;
|
||||
extensions can therefore be released independently of the host.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -10,6 +10,8 @@ from __future__ import annotations
|
||||
from deerflow_extension_api.contracts import (
|
||||
ExtensionInstall,
|
||||
ExtensionRegistry,
|
||||
ExtensionRuntimeDeps,
|
||||
ExtensionService,
|
||||
HostPolicySnapshot,
|
||||
MiddlewareContributor,
|
||||
SystemModelCallObserver,
|
||||
@ -33,11 +35,9 @@ from deerflow_extension_api.runtime_bridge import (
|
||||
)
|
||||
from deerflow_extension_api.state import ExtensionData
|
||||
|
||||
#: Contract version. Pre-1.0: the contract surface is observational only
|
||||
#: (contributors and observers), so minors may break and only patches promise
|
||||
#: to be additive. From 1.0 on, bump the major on any breaking change; see the
|
||||
#: spec's evolution rules for what counts as additive.
|
||||
API_VERSION = "0.1.1"
|
||||
#: Contract version. Before 1.0, minors may break and patches are additive.
|
||||
#: From 1.0 on, bump the major for breaking changes.
|
||||
API_VERSION = "0.1.2"
|
||||
|
||||
__all__ = [
|
||||
"API_VERSION",
|
||||
@ -47,6 +47,8 @@ __all__ = [
|
||||
"ExtensionData",
|
||||
"ExtensionInstall",
|
||||
"ExtensionRegistry",
|
||||
"ExtensionRuntimeDeps",
|
||||
"ExtensionService",
|
||||
"HostPolicySnapshot",
|
||||
"MiddlewareContributor",
|
||||
"MiddlewarePlacement",
|
||||
|
||||
@ -10,7 +10,7 @@ Compatibility rules enforced throughout this module:
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from enum import StrEnum
|
||||
from typing import TYPE_CHECKING, Any, Literal, Protocol, TypeVar, runtime_checkable
|
||||
|
||||
@ -152,6 +152,26 @@ class MiddlewareContributor(Protocol):
|
||||
return ()
|
||||
|
||||
|
||||
# --- Extension services ----------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExtensionRuntimeDeps:
|
||||
"""Host capabilities bound after Gateway infrastructure is ready."""
|
||||
|
||||
app_store: ExtensionData | None = None
|
||||
policy: HostPolicySnapshot = field(default_factory=HostPolicySnapshot)
|
||||
session_factory: Any | None = None
|
||||
|
||||
|
||||
class ExtensionService(Protocol):
|
||||
async def start(self, deps: ExtensionRuntimeDeps) -> None:
|
||||
return None
|
||||
|
||||
async def stop(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
# --- Registration surface ---------------------------------------------------
|
||||
|
||||
|
||||
@ -159,11 +179,10 @@ class MiddlewareContributor(Protocol):
|
||||
class ExtensionRegistry(Protocol):
|
||||
"""The write-only registration surface handed to ``install()``.
|
||||
|
||||
Structural and minimal on purpose. This first capability slice exposes
|
||||
middleware contribution only; later slices can add defaulted registration
|
||||
methods without breaking existing implementations. The host's concrete
|
||||
registry additionally carries host-only machinery (attribution, positional
|
||||
rollback, build) that is deliberately absent here.
|
||||
Structural and minimal on purpose. Every method has a default so additive
|
||||
contract releases remain compatible with older registry implementations.
|
||||
The host's concrete registry additionally carries host-only machinery
|
||||
(attribution, positional rollback, build) that is deliberately absent here.
|
||||
"""
|
||||
|
||||
def middlewares(self, contributor: MiddlewareContributor) -> None:
|
||||
@ -175,6 +194,18 @@ class ExtensionRegistry(Protocol):
|
||||
def system_model_observer(self, observer: SystemModelCallObserver) -> None:
|
||||
return None
|
||||
|
||||
def service(self, service: ExtensionService) -> None:
|
||||
return None
|
||||
|
||||
def routers(self, routers: Sequence[Any]) -> None:
|
||||
"""Register HTTP routers constructed eagerly during extension install.
|
||||
|
||||
Router types stay ``Any`` so this contract package has no FastAPI
|
||||
dependency. The host validates supported route shapes before mounting;
|
||||
runtime resources belong in a separately registered service.
|
||||
"""
|
||||
return None
|
||||
|
||||
|
||||
#: The install() entry point signature every extension exposes.
|
||||
ExtensionInstall = Callable[[ExtensionRegistry, Mapping[str, Any]], None]
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "deerflow-extension-api"
|
||||
version = "0.1.1"
|
||||
version = "0.1.2"
|
||||
description = "Public contracts for DeerFlow extensions"
|
||||
requires-python = ">=3.12"
|
||||
# Keep the contract package import-light and independent from the host. Public
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
### Python Extension System (Runtime Slice)
|
||||
### Python Extension System (Runtime and Distribution)
|
||||
|
||||
Third-party Python packages can expose an `install(registry, config)` function and be
|
||||
loaded, in deterministic order, from the startup-only top-level `plugins:` list in
|
||||
@ -7,12 +7,142 @@ through Gateway APIs, while importing Python entry points is an operator-control
|
||||
execution boundary. A plugin marked `required: true` fails Gateway construction when it
|
||||
cannot load; optional plugins fail open with attributed diagnostics.
|
||||
|
||||
The public package is `packages/extension-api/` and must never import `deerflow`. Its
|
||||
registry contract exposes exactly three contribution kinds: middleware contributors,
|
||||
task-lifecycle contributors, and system-model-call observers. Middleware contributions
|
||||
declare lead/subagent scope, stable order, and a semantic placement (`MODEL_LOGICAL`,
|
||||
`MODEL_PHYSICAL`, `TOOL_VISIBLE`, `TOOL_RAW`, or `STANDARD`) rather than a fragile list
|
||||
index. `extensions/stack.py` is the single final composition point; do not inject inside
|
||||
Packaged extensions use one PEP 621 entry point in the
|
||||
`deerflow.extensions` group, for example
|
||||
`example = "deerflow_extension_example:install"`. The operator CLI is dispatched from
|
||||
the existing `deerflow` console script to `extensions/cli.py` and exposes only these
|
||||
surfaces: `install SOURCE [--yes]`, `list`, `enable NAME`, `disable NAME`, and
|
||||
`remove NAME`. `NAME` resolves against the entry-point name, distribution name, or
|
||||
`module:install` value. The root `make extension-*` targets are convenience wrappers;
|
||||
because they execute from `backend/`, documentation should use absolute local source
|
||||
paths with `SOURCE=` unless backend-relative behavior is intentional.
|
||||
|
||||
`ExtensionManager` owns the package/config transaction. Install runs a controlled
|
||||
`uv add --project <backend> --group extensions --no-workspace --no-sync -- <source>`, updates the dedicated
|
||||
`[dependency-groups].extensions` list and `uv.lock`, discovers exactly one packaging entry
|
||||
point, and inserts or adopts one
|
||||
managed `plugins:` record with `name`, `package`, `use`, `enabled`, `required`, and
|
||||
private `config`. New records are written `required: false`, matching the loader default:
|
||||
`required: true` turns any later load failure — a broken wheel, a missing native library, a
|
||||
deleted snapshot — into a Gateway startup abort recoverable only through shell access, so
|
||||
it is an explicit `install --required` opt-in rather than the managed default. Adoption of
|
||||
an existing hand-written record preserves whatever `required` the operator already chose.
|
||||
Enable/disable changes only the host-level `enabled` flag and preserves
|
||||
private configuration. Remove runs `uv remove --group extensions`, removes the plugin
|
||||
record, and deletes its managed source snapshot. Install validates the selected config
|
||||
file before running any uv command, because `uv add`/`uv sync` execute the package's build
|
||||
backend: a config this manager could never write to must fail before that code runs, not
|
||||
afterwards through rollback.
|
||||
Failed install/remove operations restore `pyproject.toml` and `uv.lock` and resynchronize
|
||||
the restored environment; that second restore runs even when the recovery sync itself fails
|
||||
(a recovery sync without `--locked` writes a lock while resolving), and a failing recovery
|
||||
sync reports the original failure alongside it. The restore is deliberately not blanket:
|
||||
when recovery detects a concurrent external edit to the dependency files or the config it
|
||||
preserves that edit and raises instead, and `remove` leaves the plugin deactivated in that
|
||||
case rather than reviving a record whose package declaration may already be gone. A
|
||||
cancellation skips the recovery sync entirely — the declarations are already restored and
|
||||
the next locked startup sync reconciles the environment, whereas blocking an interrupt on a
|
||||
full dependency resolve invites a second interrupt that escapes the handler mid-transaction.
|
||||
Package mutation is deferred from environment mutation: after `uv add/remove`
|
||||
updates the declaration and lock, one `uv sync --locked --all-packages` preserves the same
|
||||
config-/environment-detected optional extras as normal startup. All three uv calls pin the
|
||||
backend project explicitly and discard UV environment overrides that could redirect the
|
||||
project, working directory, sync mode, lock policy, or target environment — including
|
||||
`UV_PYTHON`, which would swap the interpreter that then loads the extension entry point,
|
||||
and `UV_INSECURE_HOST`, which would remove the TLS validation the HTTPS-only source rule
|
||||
depends on; index, proxy, cache, and credential-provider settings remain available.
|
||||
The `--no-workspace` boundary requires uv 0.8.0 or newer. The stock Docker paths pin uv
|
||||
0.11.1, and the manager fails before mutation when the host uv is older.
|
||||
All install/remove/enable/disable mutations for a checkout hold the cross-process
|
||||
`.deer-flow/extension-manager.lock`; remove deactivates config before changing the package
|
||||
declaration, and rollback preserves a concurrent external config edit instead of replacing
|
||||
it. The MVP has no in-place upgrade: operators retain private config, remove the old
|
||||
package, install the new source pin, and restore that config.
|
||||
|
||||
Local-directory installs are snapshots, not editable links. The manager validates the
|
||||
source, derives the destination from the normalized distribution name, and copies it to
|
||||
`backend/extensions/sources/<distribution>/`. It ignores Git metadata, virtual
|
||||
environments, Python caches, and bytecode; rejects symbolic links, path-escaping
|
||||
distribution names, and likely credential files; and the root `.dockerignore` explicitly
|
||||
re-includes the entire managed tree so package READMEs, native modules, and assets reach
|
||||
the backend builder. These checks prevent common packaging accidents, not malicious
|
||||
code. Both Python build hooks and imported extension code execute with Gateway
|
||||
privileges, so the CLI requires confirmation (or explicit `--yes`) and accepts only
|
||||
trusted operator sources; source URLs containing embedded credentials are rejected.
|
||||
Remote direct references are limited to HTTPS, and remote Git sources must use public
|
||||
Git-over-HTTPS (with loopback HTTP accepted for local tooling). SSH Git URLs are rejected
|
||||
because the stock Docker builder does not forward host SSH credentials; relative paths and
|
||||
local wheels must use the managed directory snapshot path instead. Git's SCP-like shorthand
|
||||
(`git@host:org/repo.git`) carries no URL scheme, so it is detected before the scheme rules
|
||||
and reported with the same public-HTTPS correction rather than the local-path message.
|
||||
Local wheel and `file://` sources are rejected because they cannot be reproduced inside the
|
||||
Docker build context; local code must enter through the directory-snapshot path. Stock
|
||||
production builds support public package indexes and public HTTPS Git sources reachable by
|
||||
the builder; authenticated source configuration must not be embedded in the recorded URL.
|
||||
Source validation alone cannot catch environment-driven resolution (for example a
|
||||
`UV_FIND_LINKS` wheelhouse turning a plain package requirement into a local wheel
|
||||
reference), so after every `uv add/remove` the manager audits the new lock before syncing
|
||||
or enabling anything. Any local reference that the stock backend image build cannot
|
||||
reproduce — absolute paths, `file:` URLs, or relative paths outside the project root, its
|
||||
exact workspace members, and the managed `extensions/sources/` snapshots — fails the whole
|
||||
transaction and rolls back the dependency files, config, snapshot, and environment. A
|
||||
loopback URL recorded in the lock is warned about rather than rolled back: `127.0.0.1`
|
||||
inside the image builder is a different machine, so the reference is just as
|
||||
non-reproducible, but unlike an environment-driven wheelhouse resolution it is a source the
|
||||
operator typed deliberately. A private-network index is left alone entirely — a builder on
|
||||
that network can reach it. A
|
||||
config with duplicate top-level `plugins:` keys is rejected outright rather than managed
|
||||
against one block while the Gateway reads another.
|
||||
|
||||
The managed `plugins:` block is rewritten in place, and both of its boundaries come from
|
||||
the YAML parser rather than a key-shaped pattern. `AppConfig` allows extra top-level keys,
|
||||
so a neighbouring section may be named anything YAML accepts (`my.key`, `2fa`, `$schema`, a
|
||||
non-ASCII word); a pattern that fails to recognize the next key does not fail loudly, it
|
||||
reports "no next section" and the rewrite replaces that neighbour and its whole subtree.
|
||||
Trailing comments below a file-final block are preserved for the same reason — the manager
|
||||
appends `plugins:` at end of file, so that is the steady-state shape.
|
||||
|
||||
Dependency synchronization has one lock authority: the manager's `uv add/remove` calls
|
||||
are the only extension workflow allowed to update `backend/uv.lock`, and each mutation is
|
||||
followed by the local-source audit described above. The `extensions`
|
||||
group is included in `[tool.uv].default-groups` alongside `dev`. Root/backend install
|
||||
targets use `uv sync --locked`; direct backend `make dev`/`make gateway` use
|
||||
`uv run --locked`; the local full-stack launcher and Docker-dev entrypoint perform one
|
||||
locked sync and then launch with `uv run --no-sync`; the production Docker builder syncs
|
||||
the same copied backend project and lock, and both image runtime commands use
|
||||
`--no-sync`. Thus production may download locked remote artifacts while building an
|
||||
image, but production container startup never resolves or installs an extension from the
|
||||
network. Local and Docker-dev pre-start syncs may fetch missing locked artifacts.
|
||||
`docker/dev-entrypoint.sh` retries a failed sync once after recreating `.venv`, but keeps
|
||||
`--locked` on the retry: that repairs a broken virtualenv, not a stale lock. A second
|
||||
failure aborts with recovery instructions instead of starting uvicorn against an
|
||||
environment that does not match the lock, because startup must never silently resolve
|
||||
dependencies.
|
||||
That discipline assumes the uv writing the lock and the uv reading it stay compatible, so
|
||||
uv is pinned rather than floating: `backend/Dockerfile`'s `UV_IMAGE` is the single source of
|
||||
truth, both compose defaults repeat it, and every `astral-sh/setup-uv` step pins the same
|
||||
version so CI exercises the manager against the binary production actually runs. Otherwise a
|
||||
newer uv can bump `uv.lock`'s `revision` (or make `uv lock --check` disagree with a lock
|
||||
generated elsewhere) while CI stays green, and the pinned uv in the production image then
|
||||
fails on the committed lock. `backend/tests/test_ci_uv_version_pin.py` keeps the four
|
||||
locations in step, which makes a uv upgrade one deliberate, reviewable change.
|
||||
Rebuild the Gateway image after changing the managed set. Every install, enable, disable,
|
||||
remove, or config mutation also requires a Gateway restart because plugin loading is
|
||||
startup-only.
|
||||
The root management wrappers bootstrap the checkout environment without the extension group
|
||||
via `uv run --frozen --no-group extensions`, so a broken or disappeared extension source cannot
|
||||
trigger project validation before the operator can list, disable, or remove it, while a
|
||||
fresh checkout can still install the non-extension environment from the existing lock. After CLI
|
||||
entry, the manager owns the controlled locked sync.
|
||||
|
||||
The public package is `packages/extension-api/` and must never import `deerflow` or carry
|
||||
framework dependencies. Extensions declare any FastAPI, LangChain, or LangGraph imports
|
||||
themselves. Its registry contract exposes five contribution kinds: middleware
|
||||
contributors, task-lifecycle contributors, system-model-call observers, Gateway-lifetime
|
||||
services, and eager routers. Middleware contributions declare lead/subagent scope, stable
|
||||
order, and a semantic placement (`MODEL_LOGICAL`, `MODEL_PHYSICAL`, `TOOL_VISIBLE`,
|
||||
`TOOL_RAW`, or `STANDARD`) rather than a fragile list index. `extensions/stack.py` is the
|
||||
single final composition point; do not inject inside
|
||||
the shared base builder because the lead builder appends more middleware afterward.
|
||||
`extensions/ordering.py` owns host ordering invariants and validates the final composed
|
||||
stack. Nothing under `extensions/` may import `agents.middlewares` at module scope: the
|
||||
@ -32,8 +162,9 @@ capability, so a single-sided wrapper receives a pass-through counterpart; imple
|
||||
both sides when the extension must observe both synchronous and asynchronous execution
|
||||
paths.
|
||||
|
||||
Lead runs and subagents allocate an `ExtensionData` task store only when at least one of
|
||||
the three contribution kinds is registered. Middleware and system-call sites recover the
|
||||
Lead runs and subagents allocate an `ExtensionData` task store only when middleware,
|
||||
task-lifecycle, or system-model observation is registered; services and routers are
|
||||
app-scoped and do not allocate one. Middleware and system-call sites recover the
|
||||
live store through `EXTENSION_TASK_STORE_KEY` / `task_store_from_runtime()`; lifecycle
|
||||
contributors receive that same store directly. Each task resolves the immutable
|
||||
loaded-extension snapshot once and binds that same object through task-store allocation,
|
||||
@ -84,6 +215,47 @@ subagent's isolated loop, while synchronous system callbacks submit fire-and-for
|
||||
there. Shutdown stops accepting detached observations before the memory shutdown flush and
|
||||
resets the loop only after in-flight run/subagent drain ordering is complete.
|
||||
|
||||
Gateway services start in registration order after the persistence engine and session
|
||||
factory are ready. Each receives the same `ExtensionRuntimeDeps` snapshot containing the
|
||||
app store, projected host policy, and session factory. Start failures are attributed and
|
||||
fail open. The runtime captures `app.state.extensions` once, registers cleanup before the
|
||||
start batch, and stops the attempted service prefix in reverse order after run/subagent
|
||||
drain but before store, checkpointer, and engine teardown. Each stop has an independent
|
||||
bounded timeout; failures do not starve later cleanup. A service-originated
|
||||
`CancelledError` fails open, while a new cancellation of the host task still propagates
|
||||
through the exit stack. Runtime diagnostics must be appended through
|
||||
`record_runtime_diagnostics()` so `app.state.extension_diagnostics` remains the canonical
|
||||
live list.
|
||||
|
||||
Routers are constructed eagerly during `install()` and mounted only after all host routes,
|
||||
so host handlers always win. The Gateway rejects a contributed router atomically when an
|
||||
earlier host or extension route provably covers one of its paths for the same HTTP method.
|
||||
The conservative matcher proves common shadows through normalized parameter names,
|
||||
static-vs-dynamic matching, known built-in-converter containment, supported compound
|
||||
segments, full-segment `path` catch-alls, and `Mount` descendants reducible to those same
|
||||
rules. Relationships requiring general regex-language inclusion are allowed rather than
|
||||
guessed. Host WebSocket routes do not collide with contributed HTTP routes, but contributed
|
||||
WebSocket routes are rejected until the host can supply authentication and Origin checks.
|
||||
Because `include_router()` recompiles contributed routes, preflight projects the converter
|
||||
registry at include time. Nonstandard converters fail closed against reserved security
|
||||
paths but otherwise prove a shadow only when their normalized matchers are identical.
|
||||
Host authentication- and CSRF-exempt paths are reserved, and contributed Mounts, unsupported
|
||||
route items, startup/shutdown hooks, and custom router lifespans are rejected; lifetime
|
||||
resources must use `ExtensionService`. Auth and CSRF classify
|
||||
`get_request_route_path(request)`, the same root-path-adjusted ASGI path Starlette routes
|
||||
match; do not switch those security predicates back to reconstructed `request.url.path`.
|
||||
That helper delegates to the private `starlette._utils.get_route_path` on purpose. Its
|
||||
requirement is not "strip `root_path` correctly" but "return exactly what the router is
|
||||
matching on", so importing the dispatcher's own implementation keeps the two in lockstep by
|
||||
construction. Do not vendor a local copy: a private import that disappears fails loudly at
|
||||
startup, while a stale copy diverges silently at a security boundary. `starlette` is
|
||||
therefore a declared, bounded direct dependency so the bump is visible in review, and
|
||||
`tests/test_gateway_request_path.py` pins the agreement independently of the mechanism.
|
||||
Any preflight, conflict, or include failure rolls back the whole router without preventing
|
||||
later routers from mounting. Do not introduce a framework-bound `RouterContributor`
|
||||
contract: the public registry accepts `Sequence[Any]`
|
||||
to keep extension-api dependency-free.
|
||||
|
||||
The memory kind reaches those observers through a different shape, and the difference is
|
||||
deliberate rather than an oversight to be "aligned" away. DeerMem must stay vendorable and
|
||||
cannot import the extension API, so it reports through the `MemoryCallbacks.on_memory_llm_result`
|
||||
@ -96,7 +268,8 @@ The host hook wrapper around the callback stays at `Exception`: only the hook's
|
||||
are non-fatal, and an observability path must not swallow `SystemExit` / `KeyboardInterrupt`.
|
||||
|
||||
Gateway `create_app()` loads plugins once, stores the immutable registry on `app.state`
|
||||
and in the process-wide singleton, and installs one canonical live diagnostics list.
|
||||
and in the process-wide singleton, mounts contributed routers last, and installs one
|
||||
canonical live diagnostics list.
|
||||
Changing `plugins` requires a restart. Any future contribution kind must be added to the
|
||||
public contract and host runtime in the same slice; never accept a registration method
|
||||
that the current host silently ignores.
|
||||
|
||||
127
backend/packages/harness/deerflow/extensions/cli.py
Normal file
127
backend/packages/harness/deerflow/extensions/cli.py
Normal file
@ -0,0 +1,127 @@
|
||||
"""Command-line interface for installing and managing trusted extensions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from pathlib import Path
|
||||
|
||||
from deerflow.extensions.manager import ExtensionManager
|
||||
|
||||
_NAME_ENV = "DEER_FLOW_EXTENSION_NAME"
|
||||
_SOURCE_ENV = "DEER_FLOW_EXTENSION_SOURCE"
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="deerflow extensions",
|
||||
description="Install and manage trusted Python extensions for this DeerFlow checkout.",
|
||||
)
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
install = commands.add_parser("install", help="install an extension and enable it in config.yaml")
|
||||
install.add_argument("--source-env", action="store_true", help=argparse.SUPPRESS)
|
||||
install.add_argument("source", help="local directory, Python package requirement, or Git URL")
|
||||
install.add_argument(
|
||||
"--yes",
|
||||
action="store_true",
|
||||
help="acknowledge that installing an extension executes trusted third-party code",
|
||||
)
|
||||
install.add_argument(
|
||||
"--required",
|
||||
action="store_true",
|
||||
help="abort Gateway startup when this extension fails to load (default: report and skip)",
|
||||
)
|
||||
disable = commands.add_parser("disable", help="disable an extension without uninstalling it")
|
||||
disable.add_argument("--name-env", action="store_true", help=argparse.SUPPRESS)
|
||||
disable.add_argument("name", help="extension name, distribution, or module:install entry point")
|
||||
enable = commands.add_parser("enable", help="enable an installed extension")
|
||||
enable.add_argument("--name-env", action="store_true", help=argparse.SUPPRESS)
|
||||
enable.add_argument("name", help="extension name, distribution, or module:install entry point")
|
||||
commands.add_parser("list", help="list configured extensions")
|
||||
remove = commands.add_parser("remove", help="uninstall an extension and remove its config entry")
|
||||
remove.add_argument("--name-env", action="store_true", help=argparse.SUPPRESS)
|
||||
remove.add_argument("name", help="extension name, distribution, or module:install entry point")
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(list(argv) if argv is not None else None)
|
||||
try:
|
||||
root = find_project_root()
|
||||
configured_path = os.environ.get("DEER_FLOW_CONFIG_PATH")
|
||||
manager = ExtensionManager(root, config_path=configured_path)
|
||||
if args.command == "install":
|
||||
source = _source_argument(args)
|
||||
trusted = args.yes
|
||||
if not trusted:
|
||||
print("Warning: a Python extension executes code with Gateway privileges.")
|
||||
try:
|
||||
trusted = input("Install this trusted source? [y/N] ").strip().lower() in {"y", "yes"}
|
||||
except EOFError:
|
||||
trusted = False
|
||||
if not trusted:
|
||||
print("Extension installation cancelled.", file=sys.stderr)
|
||||
return 2
|
||||
installed = manager.install(source, yes=trusted, required=args.required)
|
||||
print(f"Installed and enabled {installed.name} ({installed.distribution}). Restart DeerFlow to load it.")
|
||||
return 0
|
||||
if args.command == "disable":
|
||||
name = manager.set_enabled(_name_argument(args), enabled=False)
|
||||
print(f"Disabled {name}. Restart DeerFlow to apply the change.")
|
||||
return 0
|
||||
if args.command == "enable":
|
||||
name = manager.set_enabled(_name_argument(args), enabled=True)
|
||||
print(f"Enabled {name}. Restart DeerFlow to apply the change.")
|
||||
return 0
|
||||
if args.command == "list":
|
||||
configured = manager.list_configured()
|
||||
print("NAME\tSTATE\tPACKAGE\tENTRY POINT")
|
||||
for extension in configured:
|
||||
state = "enabled" if extension.enabled else "disabled"
|
||||
print(f"{extension.name}\t{state}\t{extension.distribution}\t{extension.use}")
|
||||
return 0
|
||||
if args.command == "remove":
|
||||
name = manager.remove(_name_argument(args))
|
||||
print(f"Removed {name}. Restart DeerFlow to apply the change.")
|
||||
return 0
|
||||
except (OSError, RuntimeError, ValueError, subprocess.CalledProcessError) as exc:
|
||||
print(f"extension command failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
raise AssertionError(f"unhandled extension command: {args.command}")
|
||||
|
||||
|
||||
def _name_argument(args: argparse.Namespace) -> str:
|
||||
if not args.name_env:
|
||||
return args.name
|
||||
name = os.environ.get(_NAME_ENV)
|
||||
if name is None or not name.strip():
|
||||
raise ValueError(f"{_NAME_ENV} must contain an extension name")
|
||||
return name
|
||||
|
||||
|
||||
def _source_argument(args: argparse.Namespace) -> str:
|
||||
if not args.source_env:
|
||||
return args.source
|
||||
source = os.environ.get(_SOURCE_ENV)
|
||||
if source is None or not source.strip():
|
||||
raise ValueError(f"{_SOURCE_ENV} must contain an extension source")
|
||||
return source
|
||||
|
||||
|
||||
def find_project_root() -> Path:
|
||||
configured = os.environ.get("DEER_FLOW_PROJECT_ROOT")
|
||||
if configured:
|
||||
candidate = Path(configured).expanduser().resolve()
|
||||
if (candidate / "backend" / "pyproject.toml").is_file():
|
||||
return candidate
|
||||
raise FileNotFoundError(f"DEER_FLOW_PROJECT_ROOT is not a DeerFlow checkout: {candidate}")
|
||||
|
||||
candidates = (Path.cwd(), *Path.cwd().parents)
|
||||
for candidate in candidates:
|
||||
candidate = candidate.resolve()
|
||||
if (candidate / "backend" / "pyproject.toml").is_file():
|
||||
return candidate
|
||||
raise FileNotFoundError("could not find a DeerFlow checkout; set DEER_FLOW_PROJECT_ROOT")
|
||||
637
backend/packages/harness/deerflow/extensions/gateway.py
Normal file
637
backend/packages/harness/deerflow/extensions/gateway.py
Normal file
@ -0,0 +1,637 @@
|
||||
"""Gateway-side plumbing for app-scoped extension contributions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from deerflow_extension_api import ExtensionRuntimeDeps
|
||||
|
||||
from deerflow.extensions.loader import Diagnostic
|
||||
from deerflow.extensions.policy import project_host_policy
|
||||
from deerflow.extensions.registry import LoadedExtensions
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_STOP_TIMEOUT_SECONDS = 30.0
|
||||
|
||||
_PATH_PARAMETER_GROUP = re.compile(r"\(\?P<[^>]+>")
|
||||
_PATH_PARAMETER = re.compile(r"{([a-zA-Z_][a-zA-Z0-9_]*)(?::([a-zA-Z_][a-zA-Z0-9_]*))?}")
|
||||
_RouteMethods = frozenset[str] | None
|
||||
_RouteScopes = frozenset[str]
|
||||
_HOST_PUBLIC_PATH_PREFIXES = (
|
||||
"/health",
|
||||
"/docs",
|
||||
"/redoc",
|
||||
"/openapi.json",
|
||||
"/api/v1/auth/oauth/",
|
||||
"/api/v1/auth/callback/",
|
||||
"/api/webhooks/",
|
||||
)
|
||||
_HOST_PUBLIC_EXACT_PATHS = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/setup-status",
|
||||
"/api/v1/auth/initialize",
|
||||
"/api/v1/auth/providers",
|
||||
}
|
||||
)
|
||||
_HOST_CSRF_EXEMPT_EXACT_PATHS = frozenset({"/api/v1/auth/me"})
|
||||
_CSRF_STATE_CHANGING_METHODS = frozenset({"POST", "PUT", "DELETE", "PATCH"})
|
||||
_STANDARD_CONVERTOR_REGEXES = {
|
||||
"str": "[^/]+",
|
||||
"path": ".*",
|
||||
"int": "[0-9]+",
|
||||
"float": r"[0-9]+(\.[0-9]+)?",
|
||||
"uuid": "[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _RouteClaim:
|
||||
path: str
|
||||
matcher: str
|
||||
methods: _RouteMethods
|
||||
scopes: _RouteScopes
|
||||
mount_prefix: str | None = None
|
||||
standard_convertors: bool = True
|
||||
|
||||
|
||||
_RouteOwners = list[tuple[_RouteClaim, str]]
|
||||
|
||||
|
||||
def _cancellation_count() -> int:
|
||||
task = asyncio.current_task()
|
||||
return task.cancelling() if task is not None else 0
|
||||
|
||||
|
||||
def _route_path_matcher(route: Any) -> str | None:
|
||||
path = getattr(route, "path", None)
|
||||
if path is None:
|
||||
return None
|
||||
pattern = getattr(getattr(route, "path_regex", None), "pattern", None)
|
||||
if not pattern:
|
||||
return path
|
||||
return _PATH_PARAMETER_GROUP.sub("(?:", pattern)
|
||||
|
||||
|
||||
def _route_methods(route: Any) -> _RouteMethods:
|
||||
methods = getattr(route, "methods", None)
|
||||
return frozenset(methods) if methods else None
|
||||
|
||||
|
||||
def _route_scopes(route: Any) -> _RouteScopes:
|
||||
from starlette.routing import Mount, Route, WebSocketRoute
|
||||
|
||||
if isinstance(route, WebSocketRoute):
|
||||
return frozenset({"websocket"})
|
||||
if isinstance(route, Route):
|
||||
return frozenset({"http"})
|
||||
if isinstance(route, Mount):
|
||||
return frozenset({"http", "websocket"})
|
||||
return frozenset({"http", "websocket"})
|
||||
|
||||
|
||||
def _route_claim(route: Any, *, recompile: bool = False) -> _RouteClaim | None:
|
||||
from starlette.routing import Mount, compile_path
|
||||
|
||||
path = getattr(route, "path", None)
|
||||
if recompile and path is not None:
|
||||
path_regex, _path_format, param_convertors = compile_path(path)
|
||||
matcher = _PATH_PARAMETER_GROUP.sub("(?:", path_regex.pattern)
|
||||
else:
|
||||
matcher = _route_path_matcher(route)
|
||||
param_convertors = getattr(route, "param_convertors", {})
|
||||
if path is None or matcher is None:
|
||||
return None
|
||||
is_mount = isinstance(route, Mount)
|
||||
return _RouteClaim(
|
||||
path=path,
|
||||
matcher=matcher,
|
||||
methods=_route_methods(route),
|
||||
scopes=_route_scopes(route),
|
||||
mount_prefix=path.rstrip("/") if is_mount else None,
|
||||
standard_convertors=_uses_standard_convertors(
|
||||
path,
|
||||
param_convertors,
|
||||
is_mount=is_mount,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _uses_standard_convertors(
|
||||
path: str,
|
||||
param_convertors: Any,
|
||||
*,
|
||||
is_mount: bool,
|
||||
) -> bool:
|
||||
for match in _PATH_PARAMETER.finditer(path):
|
||||
parameter_name = match.group(1)
|
||||
convertor_name = match.group(2) or "str"
|
||||
expected_regex = _STANDARD_CONVERTOR_REGEXES.get(convertor_name)
|
||||
actual_regex = getattr(param_convertors.get(parameter_name), "regex", None)
|
||||
if expected_regex is None or actual_regex != expected_regex:
|
||||
return False
|
||||
|
||||
if is_mount:
|
||||
return getattr(param_convertors.get("path"), "regex", None) == _STANDARD_CONVERTOR_REGEXES["path"]
|
||||
return True
|
||||
|
||||
|
||||
def _convertor_can_extend_prefix(convertor: str, prefix: str) -> bool:
|
||||
"""Return a proven built-in-convertor witness beginning with ``prefix``.
|
||||
|
||||
Public-route protection is a security boundary, so an unknown custom
|
||||
convertor fails closed when its value could begin inside a public prefix.
|
||||
Shadow detection remains separate and continues to allow relationships it
|
||||
cannot prove.
|
||||
"""
|
||||
if convertor == "path":
|
||||
return True
|
||||
if convertor == "str":
|
||||
return "/" not in prefix
|
||||
if convertor == "int":
|
||||
return all(character in "0123456789" for character in prefix)
|
||||
if convertor == "float":
|
||||
if not prefix:
|
||||
return True
|
||||
return bool(re.fullmatch(r"[0-9]+", prefix) or re.fullmatch(r"[0-9]+\.", prefix) or re.fullmatch(r"[0-9]+\.[0-9]+", prefix))
|
||||
if convertor == "uuid":
|
||||
groups = (8, 4, 4, 4, 12)
|
||||
for mask in range(1 << (len(groups) - 1)):
|
||||
shape = ""
|
||||
for index, width in enumerate(groups):
|
||||
shape += "h" * width
|
||||
if index < len(groups) - 1 and mask & (1 << index):
|
||||
shape += "-"
|
||||
if len(prefix) <= len(shape) and all((expected == "h" and character in "0123456789abcdefABCDEF") or character == expected for character, expected in zip(prefix, shape, strict=False)):
|
||||
return True
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _path_template_can_start_with(path: str, prefix: str) -> bool:
|
||||
"""Whether a built-in route template has a concrete path under ``prefix``."""
|
||||
from functools import cache
|
||||
|
||||
tokens: list[tuple[str, str]] = []
|
||||
cursor = 0
|
||||
for match in _PATH_PARAMETER.finditer(path):
|
||||
literal = path[cursor : match.start()]
|
||||
if literal:
|
||||
tokens.append(("literal", literal))
|
||||
tokens.append(("parameter", match.group(2) or "str"))
|
||||
cursor = match.end()
|
||||
trailing_literal = path[cursor:]
|
||||
if trailing_literal:
|
||||
tokens.append(("literal", trailing_literal))
|
||||
|
||||
@cache
|
||||
def can_match(token_index: int, prefix_index: int) -> bool:
|
||||
if prefix_index == len(prefix):
|
||||
return True
|
||||
if token_index == len(tokens):
|
||||
return False
|
||||
|
||||
kind, value = tokens[token_index]
|
||||
remaining = prefix[prefix_index:]
|
||||
if kind == "literal":
|
||||
if value.startswith(remaining):
|
||||
return True
|
||||
if remaining.startswith(value):
|
||||
return can_match(token_index + 1, prefix_index + len(value))
|
||||
return False
|
||||
|
||||
registered_regex = _STANDARD_CONVERTOR_REGEXES.get(value)
|
||||
if registered_regex is None:
|
||||
return False
|
||||
for end_index in range(prefix_index, len(prefix) + 1):
|
||||
concrete = prefix[prefix_index:end_index]
|
||||
if re.fullmatch(registered_regex, concrete) is not None and can_match(
|
||||
token_index + 1,
|
||||
end_index,
|
||||
):
|
||||
return True
|
||||
return _convertor_can_extend_prefix(value, remaining)
|
||||
|
||||
return can_match(0, 0)
|
||||
|
||||
|
||||
def _claim_can_enter_prefix(claim: _RouteClaim, prefix: str) -> bool:
|
||||
if claim.standard_convertors:
|
||||
return _path_template_can_start_with(claim.path, prefix)
|
||||
literal_prefix = claim.path.partition("{")[0]
|
||||
return claim.path.startswith(prefix) or prefix.startswith(literal_prefix)
|
||||
|
||||
|
||||
def _claim_can_enter_exact_path(claim: _RouteClaim, exact_path: str) -> bool:
|
||||
"""Whether a route can dispatch to ``exact_path`` plus trailing slashes."""
|
||||
if not claim.standard_convertors:
|
||||
return _claim_can_enter_prefix(claim, exact_path)
|
||||
|
||||
# Auth and CSRF normalize with rstrip("/"), so two or more trailing
|
||||
# slashes are just as exempt as one. For built-in convertors, a shortest
|
||||
# slash-only witness is bounded by the template's literal length because
|
||||
# ``path`` may be empty and every other built-in rejects slash.
|
||||
for slash_count in range(len(claim.path) + 2):
|
||||
if (
|
||||
re.fullmatch(
|
||||
claim.matcher,
|
||||
exact_path + "/" * slash_count,
|
||||
)
|
||||
is not None
|
||||
):
|
||||
return True
|
||||
|
||||
# Custom regex language inclusion is deliberately not guessed at a
|
||||
# security boundary. If its template can reach the exact-path prefix,
|
||||
# reject it fail-closed; shadow matching below remains fail-open.
|
||||
return False
|
||||
|
||||
|
||||
def _router_routes(router: Any) -> list[_RouteClaim]:
|
||||
from fastapi.routing import _DefaultLifespan
|
||||
from starlette.routing import Mount, Route, WebSocketRoute
|
||||
|
||||
if getattr(router, "on_startup", ()) or getattr(router, "on_shutdown", ()):
|
||||
raise TypeError("contributed router lifecycle hooks are not supported; register an ExtensionService instead")
|
||||
lifespan_context = getattr(router, "lifespan_context", None)
|
||||
if lifespan_context is not None and not isinstance(lifespan_context, _DefaultLifespan):
|
||||
raise TypeError("contributed router lifespan is not supported; register an ExtensionService instead")
|
||||
|
||||
claims: list[_RouteClaim] = []
|
||||
for route in getattr(router, "routes", []):
|
||||
if isinstance(route, Mount):
|
||||
raise TypeError("contributed router contains a Starlette Mount, which FastAPI.include_router() ignores")
|
||||
if isinstance(route, WebSocketRoute):
|
||||
raise TypeError("contributed WebSocket routes are not supported until the host can apply authentication and Origin checks")
|
||||
if not isinstance(route, Route):
|
||||
raise TypeError(f"contributed router contains an unsupported route item: {type(route).__name__}")
|
||||
# FastAPI.include_router() reconstructs every route from ``route.path``
|
||||
# using the converter registry at include time. Preflight must project
|
||||
# those same semantics, not the router object's older compiled regex.
|
||||
claim = _route_claim(route, recompile=True)
|
||||
if claim is not None:
|
||||
enters_public_exact_path = any(_claim_can_enter_exact_path(claim, public_path) for public_path in _HOST_PUBLIC_EXACT_PATHS)
|
||||
enters_csrf_exact_path = _methods_overlap(
|
||||
claim.methods,
|
||||
_CSRF_STATE_CHANGING_METHODS,
|
||||
) and any(_claim_can_enter_exact_path(claim, exempt_path) for exempt_path in _HOST_CSRF_EXEMPT_EXACT_PATHS)
|
||||
enters_public_prefix = any(_claim_can_enter_prefix(claim, public_prefix) for public_prefix in _HOST_PUBLIC_PATH_PREFIXES)
|
||||
if enters_public_prefix:
|
||||
raise TypeError(f"contributed route {claim.path} can enter a host public namespace")
|
||||
if enters_public_exact_path or enters_csrf_exact_path:
|
||||
raise TypeError(f"contributed route {claim.path} can enter a host-reserved exact path")
|
||||
claims.append(claim)
|
||||
return claims
|
||||
|
||||
|
||||
def _methods_overlap(left: _RouteMethods, right: _RouteMethods) -> bool:
|
||||
return left is None or right is None or not left.isdisjoint(right)
|
||||
|
||||
|
||||
def _dispatches_overlap(left: _RouteClaim, right: _RouteClaim) -> bool:
|
||||
shared_scopes = left.scopes & right.scopes
|
||||
if "websocket" in shared_scopes:
|
||||
return True
|
||||
return "http" in shared_scopes and _methods_overlap(left.methods, right.methods)
|
||||
|
||||
|
||||
def _path_shape(path: str) -> tuple[tuple[str, ...], tuple[str, ...]]:
|
||||
literals: list[str] = []
|
||||
convertors: list[str] = []
|
||||
cursor = 0
|
||||
for match in _PATH_PARAMETER.finditer(path):
|
||||
literals.append(path[cursor : match.start()])
|
||||
convertors.append(match.group(2) or "str")
|
||||
cursor = match.end()
|
||||
literals.append(path[cursor:])
|
||||
return tuple(literals), tuple(convertors)
|
||||
|
||||
|
||||
def _convertor_covers(owner: str, candidate: str) -> bool:
|
||||
if owner == candidate:
|
||||
return True
|
||||
if owner == "path":
|
||||
return candidate in {"int", "float", "uuid"}
|
||||
if owner == "str":
|
||||
return candidate in {"str", "int", "float", "uuid"}
|
||||
if owner == "float":
|
||||
return candidate in {"float", "int"}
|
||||
return False
|
||||
|
||||
|
||||
def _path_segments(path: str) -> tuple[tuple[str, str], ...] | None:
|
||||
if not path.startswith("/"):
|
||||
return None
|
||||
if path == "/":
|
||||
return ()
|
||||
|
||||
segments: list[tuple[str, str]] = []
|
||||
for segment in path[1:].split("/"):
|
||||
match = _PATH_PARAMETER.fullmatch(segment)
|
||||
if match is not None:
|
||||
segments.append(("parameter", match.group(2) or "str"))
|
||||
elif _PATH_PARAMETER.search(segment):
|
||||
segments.append(("compound", segment))
|
||||
else:
|
||||
segments.append(("literal", segment))
|
||||
return tuple(segments)
|
||||
|
||||
|
||||
def _static_segment_matches(convertor: str, value: str) -> bool:
|
||||
registered_regex = _STANDARD_CONVERTOR_REGEXES.get(convertor)
|
||||
return registered_regex is not None and re.fullmatch(registered_regex, value) is not None
|
||||
|
||||
|
||||
def _compound_segment_covers(owner: str, candidate: str) -> bool:
|
||||
owner_literals, owner_convertors = _path_shape(owner)
|
||||
candidate_literals, candidate_convertors = _path_shape(candidate)
|
||||
if (
|
||||
owner_literals == candidate_literals
|
||||
and len(owner_convertors) == len(candidate_convertors)
|
||||
and all(
|
||||
_convertor_covers(owner_convertor, candidate_convertor)
|
||||
for owner_convertor, candidate_convertor in zip(
|
||||
owner_convertors,
|
||||
candidate_convertors,
|
||||
strict=True,
|
||||
)
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
if len(owner_convertors) != 1 or owner_convertors[0] != "str":
|
||||
return False
|
||||
owner_prefix, owner_suffix = owner_literals
|
||||
minimum_candidate_length = sum(map(len, candidate_literals)) + sum(0 if convertor == "path" else 1 for convertor in candidate_convertors)
|
||||
return (
|
||||
candidate_literals[0].startswith(owner_prefix)
|
||||
and candidate_literals[-1].endswith(owner_suffix)
|
||||
and minimum_candidate_length > len(owner_prefix) + len(owner_suffix)
|
||||
and all(convertor in {"str", "int", "float", "uuid"} for convertor in candidate_convertors)
|
||||
)
|
||||
|
||||
|
||||
def _compound_is_ascii_digits(compound: str) -> bool:
|
||||
literals, convertors = _path_shape(compound)
|
||||
return all(character in "0123456789" for literal in literals for character in literal) and all(convertor == "int" for convertor in convertors)
|
||||
|
||||
|
||||
def _segment_excludes_newline(segment: tuple[str, str]) -> bool:
|
||||
kind, value = segment
|
||||
if kind == "literal":
|
||||
return "\n" not in value
|
||||
if kind == "parameter":
|
||||
return value in {"path", "int", "float", "uuid"}
|
||||
literals, convertors = _path_shape(value)
|
||||
return all("\n" not in literal for literal in literals) and all(convertor in {"path", "int", "float", "uuid"} for convertor in convertors)
|
||||
|
||||
|
||||
def _compound_segment_matches_static(compound: str, value: str) -> bool:
|
||||
pattern = ""
|
||||
cursor = 0
|
||||
for match in _PATH_PARAMETER.finditer(compound):
|
||||
pattern += re.escape(compound[cursor : match.start()])
|
||||
convertor = match.group(2) or "str"
|
||||
registered_regex = _STANDARD_CONVERTOR_REGEXES.get(convertor)
|
||||
if registered_regex is None:
|
||||
return False
|
||||
pattern += f"(?:{registered_regex})"
|
||||
cursor = match.end()
|
||||
pattern += re.escape(compound[cursor:])
|
||||
return re.fullmatch(pattern, value) is not None
|
||||
|
||||
|
||||
def _segment_covers(owner: tuple[str, str], candidate: tuple[str, str]) -> bool:
|
||||
owner_kind, owner_value = owner
|
||||
candidate_kind, candidate_value = candidate
|
||||
if owner_kind == "literal":
|
||||
return candidate_kind == "literal" and owner_value == candidate_value
|
||||
if owner_kind == "parameter" and candidate_kind == "compound":
|
||||
if owner_value == "path":
|
||||
return _segment_excludes_newline(candidate)
|
||||
if owner_value == "str":
|
||||
return all((match.group(2) or "str") in {"str", "int", "float", "uuid"} for match in _PATH_PARAMETER.finditer(candidate_value))
|
||||
if owner_value in {"int", "float"}:
|
||||
return _compound_is_ascii_digits(candidate_value)
|
||||
return False
|
||||
if owner_kind == "compound":
|
||||
if candidate_kind == "literal":
|
||||
return _compound_segment_matches_static(owner_value, candidate_value)
|
||||
if candidate_kind == "compound":
|
||||
return _compound_segment_covers(owner_value, candidate_value)
|
||||
return False
|
||||
if candidate_kind == "compound":
|
||||
return False
|
||||
if candidate_kind == "literal":
|
||||
return _static_segment_matches(owner_value, candidate_value)
|
||||
return _convertor_covers(owner_value, candidate_value)
|
||||
|
||||
|
||||
def _segmented_path_covers(owner_path: str, candidate_path: str) -> bool:
|
||||
owner = _path_segments(owner_path)
|
||||
candidate = _path_segments(candidate_path)
|
||||
if owner is None or candidate is None:
|
||||
return False
|
||||
|
||||
if owner and owner[-1] == ("parameter", "path"):
|
||||
prefix = owner[:-1]
|
||||
return (
|
||||
len(candidate) > len(prefix)
|
||||
and all(
|
||||
_segment_covers(owner_segment, candidate_segment)
|
||||
for owner_segment, candidate_segment in zip(
|
||||
prefix,
|
||||
candidate,
|
||||
strict=False,
|
||||
)
|
||||
)
|
||||
and all(_segment_excludes_newline(candidate_segment) for candidate_segment in candidate[len(prefix) :])
|
||||
)
|
||||
|
||||
return len(owner) == len(candidate) and all(_segment_covers(owner_segment, candidate_segment) for owner_segment, candidate_segment in zip(owner, candidate, strict=True))
|
||||
|
||||
|
||||
def _matcher_covers(owner: _RouteClaim, candidate: _RouteClaim) -> bool:
|
||||
"""Return whether every candidate path is consumed by an earlier owner."""
|
||||
if owner.matcher == candidate.matcher:
|
||||
return True
|
||||
|
||||
if not owner.standard_convertors or not candidate.standard_convertors:
|
||||
return False
|
||||
|
||||
owner_literals, owner_convertors = _path_shape(owner.path)
|
||||
candidate_literals, candidate_convertors = _path_shape(candidate.path)
|
||||
|
||||
if not candidate_convertors:
|
||||
return re.fullmatch(owner.matcher, candidate.path) is not None
|
||||
|
||||
if owner.mount_prefix is not None:
|
||||
if owner.mount_prefix == "":
|
||||
candidate_segments = _path_segments(candidate.path)
|
||||
return candidate_segments is not None and all(_segment_excludes_newline(segment) for segment in candidate_segments)
|
||||
return _segmented_path_covers(
|
||||
f"{owner.mount_prefix}/{{mount_path:path}}",
|
||||
candidate.path,
|
||||
)
|
||||
|
||||
if _segmented_path_covers(owner.path, candidate.path):
|
||||
return True
|
||||
|
||||
return (
|
||||
owner_literals == candidate_literals
|
||||
and len(owner_convertors) == len(candidate_convertors)
|
||||
and all(
|
||||
_convertor_covers(owner_convertor, candidate_convertor)
|
||||
for owner_convertor, candidate_convertor in zip(
|
||||
owner_convertors,
|
||||
candidate_convertors,
|
||||
strict=True,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _find_route_clash(
|
||||
routes: list[_RouteClaim],
|
||||
owners: _RouteOwners,
|
||||
candidate_holder: str,
|
||||
) -> tuple[str, str] | None:
|
||||
tentative_owners = list(owners)
|
||||
for route in routes:
|
||||
for owner, holder in tentative_owners:
|
||||
if _dispatches_overlap(route, owner) and _matcher_covers(owner, route):
|
||||
return route.path, holder
|
||||
tentative_owners.append((route, candidate_holder))
|
||||
return None
|
||||
|
||||
|
||||
def include_contributed_routers(app: Any, extensions: LoadedExtensions) -> list[Diagnostic]:
|
||||
"""Mount reachable routers in order and reject definite shadows atomically."""
|
||||
diagnostics: list[Diagnostic] = []
|
||||
if not extensions.routers:
|
||||
return diagnostics
|
||||
|
||||
mounted: list[str] = []
|
||||
owners: _RouteOwners = []
|
||||
for route in getattr(app, "routes", []):
|
||||
claim = _route_claim(route)
|
||||
if claim is not None:
|
||||
owners.append((claim, "host"))
|
||||
|
||||
for source, router in extensions.routers:
|
||||
try:
|
||||
routes = _router_routes(router)
|
||||
if not routes:
|
||||
raise TypeError(f"contributed router exposes no routes: {router!r}")
|
||||
clash = _find_route_clash(routes, owners, source)
|
||||
if clash is not None:
|
||||
path, holder = clash
|
||||
message = f"router path {path} is already served by {holder}; this router was not mounted"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.error("Extension %s: %s", source, message)
|
||||
continue
|
||||
app_routes = getattr(getattr(app, "router", None), "routes", None)
|
||||
route_mark = len(app_routes) if isinstance(app_routes, list) else None
|
||||
try:
|
||||
app.include_router(router)
|
||||
except BaseException:
|
||||
# FastAPI copies one route at a time. If a later copy fails,
|
||||
# remove every route added by this attempt before either
|
||||
# continuing fail-open or propagating a host-level exception.
|
||||
if route_mark is not None:
|
||||
del app_routes[route_mark:]
|
||||
raise
|
||||
for route in routes:
|
||||
owners.append((route, source))
|
||||
mounted.append(f"{source} -> {route.path}")
|
||||
except Exception as exc:
|
||||
message = f"router could not be mounted; continuing without it: {exc}"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.exception("Extension %s: %s", source, message)
|
||||
|
||||
if mounted:
|
||||
logger.info("Extension routers mounted: %s", "; ".join(mounted))
|
||||
return diagnostics
|
||||
|
||||
|
||||
async def start_services(
|
||||
extensions: LoadedExtensions,
|
||||
app_config: Any,
|
||||
session_factory: Any | None,
|
||||
*,
|
||||
attempted_services: list[tuple[str, Any]] | None = None,
|
||||
) -> list[Diagnostic]:
|
||||
"""Start extension services in registration order, failing open per item."""
|
||||
diagnostics: list[Diagnostic] = []
|
||||
if not extensions.services:
|
||||
return diagnostics
|
||||
|
||||
deps = ExtensionRuntimeDeps(
|
||||
app_store=extensions.app_store,
|
||||
policy=project_host_policy(app_config),
|
||||
session_factory=session_factory,
|
||||
)
|
||||
for entry in extensions.services:
|
||||
source, service = entry
|
||||
if attempted_services is not None:
|
||||
# Record before awaiting start(): a service may acquire resources
|
||||
# and then fail or be cancelled, so it still owns stop().
|
||||
attempted_services.append(entry)
|
||||
cancellation_count = _cancellation_count()
|
||||
try:
|
||||
await service.start(deps)
|
||||
except asyncio.CancelledError:
|
||||
if _cancellation_count() > cancellation_count:
|
||||
raise
|
||||
message = "service start() raised CancelledError; continuing without it"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.exception("Extension %s: %s", source, message)
|
||||
except Exception as exc:
|
||||
message = f"service start() failed; continuing without it: {exc}"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.exception("Extension %s: %s", source, message)
|
||||
return diagnostics
|
||||
|
||||
|
||||
async def stop_services(
|
||||
extensions: LoadedExtensions,
|
||||
timeout_seconds: float = DEFAULT_STOP_TIMEOUT_SECONDS,
|
||||
*,
|
||||
service_entries: tuple[tuple[str, Any], ...] | list[tuple[str, Any]] | None = None,
|
||||
) -> list[Diagnostic]:
|
||||
"""Stop services in reverse order with an independent budget per item."""
|
||||
diagnostics: list[Diagnostic] = []
|
||||
entries = extensions.services if service_entries is None else service_entries
|
||||
for source, service in reversed(entries):
|
||||
cancellation_count = _cancellation_count()
|
||||
timeout = asyncio.timeout(timeout_seconds)
|
||||
try:
|
||||
async with timeout:
|
||||
await service.stop()
|
||||
except TimeoutError as exc:
|
||||
if timeout.expired():
|
||||
message = f"service stop() timed out after {timeout_seconds}s; continuing shutdown"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.error("Extension %s: %s", source, message)
|
||||
else:
|
||||
message = f"service stop() failed; continuing shutdown: {exc}"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.exception("Extension %s: %s", source, message)
|
||||
except asyncio.CancelledError:
|
||||
if _cancellation_count() > cancellation_count:
|
||||
raise
|
||||
message = "service stop() raised CancelledError; continuing shutdown"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.exception("Extension %s: %s", source, message)
|
||||
except Exception as exc:
|
||||
message = f"service stop() failed; continuing shutdown: {exc}"
|
||||
diagnostics.append(Diagnostic.error(source, message))
|
||||
logger.exception("Extension %s: %s", source, message)
|
||||
return diagnostics
|
||||
@ -29,6 +29,18 @@ class ExtensionSpec(BaseModel):
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
enabled: bool = Field(
|
||||
default=True,
|
||||
description="When false, skip the extension without resolving or importing it",
|
||||
)
|
||||
name: str | None = Field(
|
||||
default=None,
|
||||
description="Stable operator-facing name recorded by the extension manager",
|
||||
)
|
||||
package: str | None = Field(
|
||||
default=None,
|
||||
description="Installed Python distribution recorded by the extension manager",
|
||||
)
|
||||
use: str = Field(description="Entry point path, e.g. 'my_extension:install'")
|
||||
config: dict[str, Any] = Field(
|
||||
default_factory=dict,
|
||||
@ -85,8 +97,8 @@ def _parse_version(version: object) -> tuple[int, ...] | None:
|
||||
def _compatible(declared: str, current: str) -> bool:
|
||||
"""One-directional, with the semver window for the contract's life stage.
|
||||
|
||||
Pre-1.0 the contract surface is observational only and minors may break,
|
||||
so the window is same major.minor with patches additive: host >= declared.
|
||||
Pre-1.0 minors may break, so the window is same major.minor with patches
|
||||
additive: host >= declared.
|
||||
From 1.0 on contracts only grow within a major, so a newer host stays
|
||||
compatible with older extensions while an extension written against a
|
||||
newer minor is refused — it would reach for contract additions the host
|
||||
@ -130,6 +142,9 @@ def load_extensions(specs: Sequence[ExtensionSpec]) -> tuple[LoadedExtensions, l
|
||||
loaded_sources: list[str] = []
|
||||
|
||||
for spec in specs:
|
||||
if not spec.enabled:
|
||||
continue
|
||||
|
||||
try:
|
||||
install = resolve_variable(spec.use)
|
||||
except Exception as exc:
|
||||
|
||||
1002
backend/packages/harness/deerflow/extensions/manager.py
Normal file
1002
backend/packages/harness/deerflow/extensions/manager.py
Normal file
File diff suppressed because it is too large
Load Diff
@ -7,13 +7,14 @@ runtime projection.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from collections.abc import Iterator, Sequence
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from deerflow_extension_api import (
|
||||
ExtensionData,
|
||||
ExtensionService,
|
||||
MiddlewareContributor,
|
||||
SystemModelCallObserver,
|
||||
TaskLifecycleContributor,
|
||||
@ -35,6 +36,8 @@ class LoadedExtensions:
|
||||
middleware_contributors: tuple[tuple[str, MiddlewareContributor], ...] = ()
|
||||
task_lifecycle: tuple[tuple[str, TaskLifecycleContributor], ...] = ()
|
||||
system_model_observers: tuple[tuple[str, SystemModelCallObserver], ...] = ()
|
||||
services: tuple[tuple[str, ExtensionService], ...] = ()
|
||||
routers: tuple[tuple[str, Any], ...] = ()
|
||||
|
||||
# Precomputed attributes, not methods: hook sites read one attribute to
|
||||
# short-circuit, so the zero-extension path constructs nothing.
|
||||
@ -57,6 +60,8 @@ class ExtensionRegistry(ExtensionRegistryContract):
|
||||
self._middlewares: list[_Entry] = []
|
||||
self._task_lifecycle: list[_Entry] = []
|
||||
self._system_model_observers: list[_Entry] = []
|
||||
self._services: list[_Entry] = []
|
||||
self._routers: list[_Entry] = []
|
||||
self._current_source: str | None = None
|
||||
|
||||
@contextmanager
|
||||
@ -83,6 +88,13 @@ class ExtensionRegistry(ExtensionRegistryContract):
|
||||
def system_model_observer(self, observer: SystemModelCallObserver) -> None:
|
||||
self._system_model_observers.append((self._source(), observer))
|
||||
|
||||
def service(self, service: ExtensionService) -> None:
|
||||
self._services.append((self._source(), service))
|
||||
|
||||
def routers(self, routers: Sequence[Any]) -> None:
|
||||
source = self._source()
|
||||
self._routers.extend((source, router) for router in routers)
|
||||
|
||||
def discard(self, source: str) -> None:
|
||||
"""Remove every entry registered by ``source``.
|
||||
|
||||
@ -100,18 +112,22 @@ class ExtensionRegistry(ExtensionRegistryContract):
|
||||
self._middlewares,
|
||||
self._task_lifecycle,
|
||||
self._system_model_observers,
|
||||
self._services,
|
||||
self._routers,
|
||||
):
|
||||
bucket[:] = [entry for entry in bucket if entry[0] != source]
|
||||
|
||||
def mark(self) -> tuple[int, int, int]:
|
||||
def mark(self) -> tuple[int, int, int, int, int]:
|
||||
"""Snapshot bucket lengths so one install() can be undone positionally."""
|
||||
return (
|
||||
len(self._middlewares),
|
||||
len(self._task_lifecycle),
|
||||
len(self._system_model_observers),
|
||||
len(self._services),
|
||||
len(self._routers),
|
||||
)
|
||||
|
||||
def rollback_to(self, mark: tuple[int, int, int]) -> None:
|
||||
def rollback_to(self, mark: tuple[int, int, int, int, int]) -> None:
|
||||
"""Undo every registration made since ``mark``.
|
||||
|
||||
Positional rather than source-keyed: two specs may legitimately share
|
||||
@ -123,6 +139,8 @@ class ExtensionRegistry(ExtensionRegistryContract):
|
||||
self._middlewares,
|
||||
self._task_lifecycle,
|
||||
self._system_model_observers,
|
||||
self._services,
|
||||
self._routers,
|
||||
),
|
||||
mark,
|
||||
strict=True,
|
||||
@ -135,6 +153,8 @@ class ExtensionRegistry(ExtensionRegistryContract):
|
||||
middleware_contributors=tuple(self._middlewares),
|
||||
task_lifecycle=tuple(self._task_lifecycle),
|
||||
system_model_observers=tuple(self._system_model_observers),
|
||||
services=tuple(self._services),
|
||||
routers=tuple(self._routers),
|
||||
has_middleware_contributors=bool(self._middlewares),
|
||||
has_task_lifecycle=bool(self._task_lifecycle),
|
||||
has_system_model_observers=bool(self._system_model_observers),
|
||||
|
||||
@ -49,6 +49,7 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="deerflow",
|
||||
description="DeerFlow terminal workbench — a TUI over the embedded DeerFlow harness.",
|
||||
epilog="Extension management: deerflow extensions --help",
|
||||
add_help=True,
|
||||
)
|
||||
parser.add_argument("message", nargs="*", help="initial prompt for the TUI, or message in --cli mode")
|
||||
@ -204,6 +205,7 @@ deerflow — DeerFlow terminal workbench
|
||||
deerflow --json "question" stream newline-delimited JSON events
|
||||
deerflow --recursion-limit N --print "question"
|
||||
set the headless agent-loop super-step limit
|
||||
deerflow extensions --help install and manage trusted Python extensions
|
||||
echo "question" | deerflow --print
|
||||
"""
|
||||
|
||||
@ -222,6 +224,10 @@ def _run_overrides(plan: LaunchPlan) -> dict[str, int]:
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
argv = list(sys.argv[1:] if argv is None else argv)
|
||||
if argv and argv[0] == "extensions":
|
||||
from deerflow.extensions.cli import main as extensions_main
|
||||
|
||||
return extensions_main(argv[1:])
|
||||
plan = plan_launch(
|
||||
argv,
|
||||
stdin_isatty=sys.stdin.isatty(),
|
||||
|
||||
@ -11,7 +11,7 @@ dependencies = [
|
||||
# the contract version it implements, extensions declare ranges. A range
|
||||
# here would let pip resolve a newer contract package than this harness
|
||||
# implements, making newer extensions look supported at runtime.
|
||||
"deerflow-extension-api==0.1.1",
|
||||
"deerflow-extension-api==0.1.2",
|
||||
"dotenv>=0.9.9",
|
||||
"exa-py>=1.0.0",
|
||||
"httpx>=0.28.0",
|
||||
@ -31,6 +31,7 @@ dependencies = [
|
||||
"langgraph-runtime-inmem>=0.28.0",
|
||||
"markdownify>=1.2.2",
|
||||
"markitdown[all,xlsx]>=0.0.1a2",
|
||||
"packaging>=24.2",
|
||||
"pydantic>=2.12.5",
|
||||
"pyyaml>=6.0.3",
|
||||
"readabilipy>=0.3.0",
|
||||
|
||||
@ -10,6 +10,16 @@ dependencies = [
|
||||
"httpx>=0.28.0",
|
||||
"python-multipart>=0.0.31",
|
||||
"sse-starlette>=2.1.0",
|
||||
# Direct dependency on purpose, even though FastAPI already pulls it in:
|
||||
# app/gateway/request_path.py imports the private
|
||||
# `starlette._utils.get_route_path` so the auth and CSRF predicates
|
||||
# classify the exact string Starlette's router matches on. A private
|
||||
# import is the safest option here because it fails loudly (ImportError at
|
||||
# startup) instead of silently drifting from the dispatcher at a security
|
||||
# boundary -- but it does mean a Starlette bump is a security-relevant
|
||||
# change. Declaring and bounding it here makes that bump visible in the
|
||||
# diff; tests/test_gateway_request_path.py pins the agreement itself.
|
||||
"starlette>=1.3.1,<2",
|
||||
"uvicorn[standard]>=0.34.0",
|
||||
"lark-oapi>=1.4.0",
|
||||
"slack-sdk>=3.33.0",
|
||||
@ -34,6 +44,10 @@ browser = ["deerflow-harness[browser]"]
|
||||
memory-zh = ["deerflow-harness[memory-zh]"]
|
||||
|
||||
[dependency-groups]
|
||||
# Managed extension packages are added here by `deerflow extensions install`.
|
||||
# Keeping them separate from development tooling lets every startup mode sync
|
||||
# the same locked runtime set without promoting those packages to core deps.
|
||||
extensions = []
|
||||
dev = [
|
||||
"blockbuster>=1.5.26,<1.6",
|
||||
"hypothesis>=6.100,<7",
|
||||
@ -64,6 +78,7 @@ markers = [
|
||||
|
||||
[tool.uv]
|
||||
index-url = "https://pypi.org/simple"
|
||||
default-groups = ["dev", "extensions"]
|
||||
# langgraph-sdk 0.4.2 (pulled in by langgraph 1.2.9 for DeltaChannel) pins
|
||||
# `websockets<16,>=14`, silently downgrading websockets 16.0 -> 15.0.1. The
|
||||
# pin is not grounded in any API incompatibility: websockets 16's only
|
||||
|
||||
@ -216,6 +216,58 @@ def test_public_auth_path_no_cookie(client):
|
||||
assert res.status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"encoded_path",
|
||||
[
|
||||
"/api/v1/auth/setup-sta%0Atus",
|
||||
"/api/v1/auth/setup-sta%0Dtus",
|
||||
"/api/v1/auth/setup-sta%09tus",
|
||||
"/api/v1/auth/setup-status%23private",
|
||||
"/api/v1/auth/setup-status%3Fprivate",
|
||||
],
|
||||
)
|
||||
def test_url_reconstruction_cannot_turn_a_protected_route_path_public(
|
||||
monkeypatch,
|
||||
encoded_path,
|
||||
):
|
||||
from fastapi import FastAPI
|
||||
|
||||
monkeypatch.setenv("DEER_FLOW_AUTH_DISABLED", "")
|
||||
app = FastAPI()
|
||||
app.add_middleware(AuthMiddleware)
|
||||
|
||||
@app.get("/api/v1/auth/setup-sta{gap}tus")
|
||||
async def control_gap(gap: str):
|
||||
return {"gap": gap}
|
||||
|
||||
@app.get("/api/v1/auth/setup-status{suffix}")
|
||||
async def delimiter_suffix(suffix: str):
|
||||
return {"suffix": suffix}
|
||||
|
||||
response = TestClient(app).get(encoded_path)
|
||||
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_auth_uses_the_same_root_path_projection_as_the_router(monkeypatch):
|
||||
from fastapi import FastAPI
|
||||
|
||||
monkeypatch.setenv("DEER_FLOW_AUTH_DISABLED", "")
|
||||
child = FastAPI()
|
||||
child.add_middleware(AuthMiddleware)
|
||||
|
||||
@child.get("/health")
|
||||
async def health():
|
||||
return {"ok": True}
|
||||
|
||||
parent = FastAPI()
|
||||
parent.mount("/prefix", child)
|
||||
|
||||
response = TestClient(parent).get("/prefix/health")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
def test_protected_auth_path_no_cookie(client):
|
||||
"""/auth/me requires cookie even though it's under /api/v1/auth/."""
|
||||
res = client.get("/api/v1/auth/me")
|
||||
|
||||
@ -70,6 +70,7 @@ class _FakeRequest:
|
||||
|
||||
def __init__(self, path: str, method: str = "POST"):
|
||||
self.method = method
|
||||
self.scope = {"path": path, "root_path": ""}
|
||||
|
||||
class _URL:
|
||||
def __init__(self, p):
|
||||
|
||||
111
backend/tests/test_ci_uv_version_pin.py
Normal file
111
backend/tests/test_ci_uv_version_pin.py
Normal file
@ -0,0 +1,111 @@
|
||||
"""Regression test pinning CI's uv binary to the version production ships.
|
||||
|
||||
``ExtensionManager`` is not a consumer of uv the build tool -- it is a program
|
||||
whose whole job is driving ``uv`` as a subprocess, depending on its CLI
|
||||
behavior (``--no-workspace``, ``--no-sync``, what ``uv add`` writes into
|
||||
``[dependency-groups] extensions``) and on the ``uv.lock`` serialization
|
||||
format. uv is therefore closer to a runtime dependency with a contract than to
|
||||
incidental tooling.
|
||||
|
||||
``backend/Dockerfile`` pins that binary to an exact version, but every
|
||||
``astral-sh/setup-uv`` step used to install whatever was latest at run time, so
|
||||
CI exercised the manager against a uv that is not the uv production runs. The
|
||||
sharpest failure that allows: a newer uv bumps ``uv.lock``'s ``revision``,
|
||||
CI stays green because the same uv reads back what it wrote, and the pinned uv
|
||||
in the production image then cannot read the committed lock. ``uv lock
|
||||
--check`` is version-sensitive for the same reason -- it verifies the lock is
|
||||
what *this* uv would produce, and two versions can emit equivalent but
|
||||
non-identical output.
|
||||
|
||||
Pinning is only half of it; without this test the two pins drift apart again
|
||||
the next time someone bumps one of them. Upgrading uv should be one explicit
|
||||
change that touches the Dockerfile, the workflows, and this test together.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
BACKEND_DOCKERFILE = REPO_ROOT / "backend" / "Dockerfile"
|
||||
WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows"
|
||||
COMPOSE_PATHS = (
|
||||
REPO_ROOT / "docker" / "docker-compose.yaml",
|
||||
REPO_ROOT / "docker" / "docker-compose-dev.yaml",
|
||||
)
|
||||
|
||||
_UV_IMAGE_REFERENCE = re.compile(r"ghcr\.io/astral-sh/uv:(?P<version>\d+\.\d+\.\d+)")
|
||||
_SETUP_UV_ACTION = re.compile(r"^astral-sh/setup-uv@(?P<ref>[^\s]+)$")
|
||||
|
||||
|
||||
def _pinned_uv_version() -> str:
|
||||
"""The single source of truth: the uv image the backend image builds from."""
|
||||
match = _UV_IMAGE_REFERENCE.search(BACKEND_DOCKERFILE.read_text(encoding="utf-8"))
|
||||
assert match is not None, f"{BACKEND_DOCKERFILE} no longer pins a ghcr.io/astral-sh/uv version"
|
||||
return match.group("version")
|
||||
|
||||
|
||||
def _workflow_paths() -> list[Path]:
|
||||
return sorted(path for path in WORKFLOWS_DIR.iterdir() if path.suffix in {".yml", ".yaml"})
|
||||
|
||||
|
||||
def _setup_uv_steps() -> list[tuple[str, str, dict]]:
|
||||
"""Return (workflow name, action ref, step mapping) for each setup-uv step."""
|
||||
steps: list[tuple[str, str, dict]] = []
|
||||
for path in _workflow_paths():
|
||||
workflow = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
||||
for job in (workflow.get("jobs") or {}).values():
|
||||
if not isinstance(job, dict):
|
||||
continue
|
||||
for step in job.get("steps") or []:
|
||||
if not isinstance(step, dict):
|
||||
continue
|
||||
uses = step.get("uses")
|
||||
if not isinstance(uses, str):
|
||||
continue
|
||||
match = _SETUP_UV_ACTION.match(uses.strip())
|
||||
if match is not None:
|
||||
steps.append((path.name, match.group("ref"), step))
|
||||
return steps
|
||||
|
||||
|
||||
def test_the_repository_still_has_setup_uv_steps_to_check():
|
||||
"""Guard against the other assertions silently passing on an empty list."""
|
||||
assert _setup_uv_steps(), "no astral-sh/setup-uv steps found; this test needs updating"
|
||||
|
||||
|
||||
def test_every_setup_uv_step_pins_the_uv_version_production_ships():
|
||||
expected = _pinned_uv_version()
|
||||
|
||||
unpinned: list[str] = []
|
||||
mismatched: list[str] = []
|
||||
for workflow_name, _ref, step in _setup_uv_steps():
|
||||
with_block = step.get("with")
|
||||
version = with_block.get("version") if isinstance(with_block, dict) else None
|
||||
if version is None:
|
||||
unpinned.append(f"{workflow_name}: {step.get('name', '<unnamed step>')}")
|
||||
elif str(version) != expected:
|
||||
mismatched.append(f"{workflow_name}: {version!r} != {expected!r}")
|
||||
|
||||
assert not unpinned, f"setup-uv steps install whatever uv is latest, so CI would not exercise the uv production ships ({expected}): {unpinned}"
|
||||
assert not mismatched, f"setup-uv steps pin a uv other than the one backend/Dockerfile ships ({expected}): {mismatched}"
|
||||
|
||||
|
||||
def test_every_setup_uv_step_uses_the_same_action_version():
|
||||
refs = {ref for _workflow_name, ref, _step in _setup_uv_steps()}
|
||||
|
||||
assert len(refs) == 1, f"astral-sh/setup-uv is referenced at mixed action versions, so the steps do not share caching or input behavior: {sorted(refs)}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("compose_path", COMPOSE_PATHS, ids=lambda path: path.name)
|
||||
def test_compose_uv_image_default_matches_the_backend_dockerfile(compose_path: Path):
|
||||
"""The compose override defaults must not drift from the image they build."""
|
||||
expected = _pinned_uv_version()
|
||||
versions = set(_UV_IMAGE_REFERENCE.findall(compose_path.read_text(encoding="utf-8")))
|
||||
|
||||
assert versions, f"{compose_path.name} no longer references a pinned uv image"
|
||||
assert versions == {expected}, f"{compose_path.name} defaults to uv {sorted(versions)} while backend/Dockerfile ships {expected!r}"
|
||||
@ -22,9 +22,50 @@ def _make_app() -> FastAPI:
|
||||
async def protected_mutation():
|
||||
return {"ok": True}
|
||||
|
||||
@app.post("/api/v1/auth/log{gap}in/local")
|
||||
async def control_gap(gap: str):
|
||||
return {"gap": gap}
|
||||
|
||||
@app.post("/api/v1/auth/me{suffix}")
|
||||
async def delimiter_suffix(suffix: str):
|
||||
return {"suffix": suffix}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
def test_url_reconstruction_cannot_create_a_csrf_exemption():
|
||||
client = TestClient(_make_app(), base_url="https://deerflow.example")
|
||||
|
||||
for encoded_path in (
|
||||
"/api/v1/auth/log%0Ain/local",
|
||||
"/api/v1/auth/log%0Din/local",
|
||||
"/api/v1/auth/log%09in/local",
|
||||
"/api/v1/auth/me%23private",
|
||||
"/api/v1/auth/me%3Fprivate",
|
||||
):
|
||||
response = client.post(encoded_path)
|
||||
assert response.status_code == 403, encoded_path
|
||||
|
||||
|
||||
def test_csrf_uses_the_same_root_path_projection_as_the_router():
|
||||
child = FastAPI()
|
||||
child.add_middleware(CSRFMiddleware)
|
||||
|
||||
@child.post("/api/v1/auth/login/local")
|
||||
async def login_local():
|
||||
return {"ok": True}
|
||||
|
||||
parent = FastAPI()
|
||||
parent.mount("/prefix", child)
|
||||
|
||||
response = TestClient(
|
||||
parent,
|
||||
base_url="https://deerflow.example",
|
||||
).post("/prefix/api/v1/auth/login/local")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
def test_auth_post_rejects_cross_origin_browser_request():
|
||||
"""CSRF-exempt auth routes must not accept hostile browser origins.
|
||||
|
||||
|
||||
@ -49,6 +49,7 @@ def test_backend_dockerfile_expands_multiple_uv_extras(tmp_path):
|
||||
|
||||
assert capture.read_text(encoding="utf-8").splitlines() == [
|
||||
"sync",
|
||||
"--locked",
|
||||
"--extra",
|
||||
"redis",
|
||||
"--extra",
|
||||
|
||||
@ -18,14 +18,26 @@ REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
ENTRYPOINT = REPO_ROOT / "docker" / "dev-entrypoint.sh"
|
||||
|
||||
|
||||
def _run(uv_extras: str | None) -> subprocess.CompletedProcess[str]:
|
||||
"""Invoke `dev-entrypoint.sh --print-extras` with UV_EXTRAS set."""
|
||||
def _run(
|
||||
uv_extras: str | None,
|
||||
*,
|
||||
config_path: Path | None = None,
|
||||
stream_bridge_redis_url: str | None = None,
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
"""Invoke the entrypoint's public extras-resolution dry run."""
|
||||
env = os.environ.copy()
|
||||
env.pop("UV_EXTRAS", None)
|
||||
env.pop("DEER_FLOW_CONFIG_PATH", None)
|
||||
env.pop("DEER_FLOW_STREAM_BRIDGE_REDIS_URL", None)
|
||||
if uv_extras is not None:
|
||||
env["UV_EXTRAS"] = uv_extras
|
||||
if config_path is not None:
|
||||
env["DEER_FLOW_CONFIG_PATH"] = str(config_path)
|
||||
if stream_bridge_redis_url is not None:
|
||||
env["DEER_FLOW_STREAM_BRIDGE_REDIS_URL"] = stream_bridge_redis_url
|
||||
return subprocess.run(
|
||||
["sh", str(ENTRYPOINT), "--print-extras"],
|
||||
cwd=ENTRYPOINT.parent,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
@ -54,12 +66,32 @@ def test_entrypoint_excludes_runtime_state_from_uvicorn_reload():
|
||||
assert "--reload-exclude=/app/backend/.deer-flow" in content
|
||||
|
||||
|
||||
def test_failed_sync_recreates_a_clean_virtual_environment():
|
||||
content = ENTRYPOINT.read_text(encoding="utf-8")
|
||||
|
||||
assert "uv venv --clear .venv" in content
|
||||
assert "uv venv --allow-existing .venv" not in content
|
||||
|
||||
|
||||
def test_no_uv_extras_yields_empty_flags():
|
||||
proc = _run(None)
|
||||
assert proc.returncode == 0
|
||||
assert proc.stdout.strip() == ""
|
||||
|
||||
|
||||
def test_no_explicit_extras_uses_the_runtime_selected_config(tmp_path: Path):
|
||||
config_path = tmp_path / "deployment.yaml"
|
||||
config_path.write_text(
|
||||
"database:\n backend: postgres\ntools:\n - name: browser_navigate\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
proc = _run(None, config_path=config_path)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert proc.stdout.strip() == "--extra browser --extra postgres"
|
||||
|
||||
|
||||
def test_single_extra():
|
||||
proc = _run("postgres")
|
||||
assert proc.returncode == 0
|
||||
@ -84,6 +116,26 @@ def test_multi_extra_mixed_separators():
|
||||
assert proc.stdout.strip() == "--extra postgres --extra ollama"
|
||||
|
||||
|
||||
def test_explicit_extras_override_config_and_are_deduplicated(tmp_path: Path):
|
||||
config_path = tmp_path / "deployment.yaml"
|
||||
config_path.write_text("database:\n backend: postgres\n", encoding="utf-8")
|
||||
|
||||
proc = _run("redis,redis browser redis", config_path=config_path)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert proc.stdout.strip() == "--extra redis --extra browser"
|
||||
|
||||
|
||||
def test_explicit_extras_keep_runtime_required_redis_without_duplicates():
|
||||
proc = _run(
|
||||
"postgres,postgres",
|
||||
stream_bridge_redis_url="redis://redis:6379/0",
|
||||
)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert proc.stdout.strip() == "--extra postgres --extra redis"
|
||||
|
||||
|
||||
def test_empty_string_yields_empty_flags():
|
||||
proc = _run("")
|
||||
assert proc.returncode == 0
|
||||
@ -114,3 +166,97 @@ def test_underscores_and_hyphens_in_name_are_allowed():
|
||||
proc = _run("post_gres,post-gres")
|
||||
assert proc.returncode == 0
|
||||
assert proc.stdout.strip() == "--extra post_gres --extra post-gres"
|
||||
|
||||
|
||||
# ── Dependency-sync failure branch ──────────────────────────────────────────
|
||||
#
|
||||
# The self-heal retry reuses `--locked`, so a lock that genuinely disagrees with
|
||||
# the environment fails the same way twice. `set -e` already stops the script
|
||||
# there -- these tests pin that the handoff to uvicorn is never reached, and
|
||||
# that the operator is told what to do instead of reading a bare uv traceback.
|
||||
#
|
||||
# `/app/backend` only exists inside the container, so the sync block is sliced
|
||||
# out of the real script and run against a stub `uv`. The block is read from
|
||||
# the file rather than duplicated here: editing the script changes what runs.
|
||||
|
||||
_SYNC_BLOCK_START = "# ── Sync dependencies (with self-heal) ──"
|
||||
_SYNC_BLOCK_END = "# ── Hand off to uvicorn ──"
|
||||
|
||||
_STUB_UV_ALWAYS_FAILS_SYNC = """#!/bin/sh
|
||||
# `uv venv` succeeds so the retry is actually reached; every sync fails.
|
||||
case "$1" in
|
||||
sync) exit 1 ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
"""
|
||||
|
||||
_STUB_UV_SUCCEEDS = """#!/bin/sh
|
||||
exit 0
|
||||
"""
|
||||
|
||||
_STUB_UV_FAILS_THEN_SUCCEEDS = """#!/bin/sh
|
||||
case "$1" in
|
||||
sync)
|
||||
if [ -f "$STUB_UV_STATE/first_sync_done" ]; then exit 0; fi
|
||||
: > "$STUB_UV_STATE/first_sync_done"
|
||||
exit 1
|
||||
;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
"""
|
||||
|
||||
|
||||
def _sync_block() -> str:
|
||||
content = ENTRYPOINT.read_text(encoding="utf-8")
|
||||
start = content.index(_SYNC_BLOCK_START)
|
||||
end = content.index(_SYNC_BLOCK_END)
|
||||
return content[start:end]
|
||||
|
||||
|
||||
def _run_sync_block(tmp_path: Path, stub_uv: str) -> subprocess.CompletedProcess[str]:
|
||||
"""Execute the script's real sync block with a stubbed `uv` on PATH."""
|
||||
bin_dir = tmp_path / "bin"
|
||||
bin_dir.mkdir()
|
||||
uv_stub = bin_dir / "uv"
|
||||
uv_stub.write_text(stub_uv, encoding="utf-8")
|
||||
uv_stub.chmod(0o755)
|
||||
|
||||
state_dir = tmp_path / "state"
|
||||
state_dir.mkdir()
|
||||
|
||||
# `cd` is shadowed because /app/backend does not exist outside the
|
||||
# container; everything else in the block runs verbatim.
|
||||
script = f'set -e\ncd() {{ :; }}\nEXTRAS_FLAGS=""\n{_sync_block()}\necho "REACHED_HANDOFF"\n'
|
||||
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = f"{bin_dir}{os.pathsep}{env['PATH']}"
|
||||
env["STUB_UV_STATE"] = str(state_dir)
|
||||
return subprocess.run(["sh", "-c", script], capture_output=True, text=True, check=False, env=env, cwd=tmp_path)
|
||||
|
||||
|
||||
def test_successful_sync_reaches_the_uvicorn_handoff(tmp_path: Path):
|
||||
proc = _run_sync_block(tmp_path, _STUB_UV_SUCCEEDS)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert "REACHED_HANDOFF" in proc.stdout
|
||||
|
||||
|
||||
def test_self_heal_retry_still_reaches_the_handoff(tmp_path: Path):
|
||||
proc = _run_sync_block(tmp_path, _STUB_UV_FAILS_THEN_SUCCEEDS)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert "recreating .venv" in proc.stdout
|
||||
assert "REACHED_HANDOFF" in proc.stdout
|
||||
|
||||
|
||||
def test_failed_retry_aborts_before_starting_uvicorn(tmp_path: Path):
|
||||
proc = _run_sync_block(tmp_path, _STUB_UV_ALWAYS_FAILS_SYNC)
|
||||
|
||||
assert proc.returncode != 0, "startup continued past an unsatisfied lock"
|
||||
assert "REACHED_HANDOFF" not in proc.stdout, "uvicorn would have been started against a stale or missing environment"
|
||||
|
||||
|
||||
def test_failed_retry_tells_the_operator_how_to_recover(tmp_path: Path):
|
||||
proc = _run_sync_block(tmp_path, _STUB_UV_ALWAYS_FAILS_SYNC)
|
||||
|
||||
assert "make install" in proc.stderr, f"no recovery guidance on stderr: {proc.stderr!r}"
|
||||
|
||||
@ -21,6 +21,8 @@ from deerflow_extension_api import (
|
||||
ExtensionData,
|
||||
ExtensionInstall,
|
||||
ExtensionRegistry,
|
||||
ExtensionRuntimeDeps,
|
||||
ExtensionService,
|
||||
HostPolicySnapshot,
|
||||
MiddlewareContributor,
|
||||
MiddlewarePlacement,
|
||||
@ -63,6 +65,7 @@ def test_middleware_placement_defaults():
|
||||
"cls",
|
||||
[
|
||||
HostPolicySnapshot,
|
||||
ExtensionRuntimeDeps,
|
||||
AgentBuildContext,
|
||||
TaskInfo,
|
||||
SystemModelRequest,
|
||||
@ -77,7 +80,7 @@ def test_every_dataclass_is_frozen(cls):
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"cls",
|
||||
[HostPolicySnapshot, TaskInfo, SystemModelRequest, SystemModelResult],
|
||||
[HostPolicySnapshot, ExtensionRuntimeDeps, TaskInfo, SystemModelRequest, SystemModelResult],
|
||||
)
|
||||
def test_additive_dataclasses_are_constructible_with_required_fields_only(cls):
|
||||
"""Fields added later must carry defaults, or old extensions break on upgrade.
|
||||
@ -114,6 +117,7 @@ def test_agent_build_context_optional_fields_keep_their_defaults():
|
||||
"protocol",
|
||||
[
|
||||
ExtensionRegistry,
|
||||
ExtensionService,
|
||||
MiddlewareContributor,
|
||||
TaskLifecycleContributor,
|
||||
SystemModelCallObserver,
|
||||
@ -195,16 +199,18 @@ def test_system_model_request_normalizes_messages_into_an_immutable_sequence():
|
||||
assert SystemModelRequest(messages=("already", "a", "tuple")).messages == ("already", "a", "tuple")
|
||||
|
||||
|
||||
def test_future_contribution_points_are_not_advertised_before_the_host_supports_them():
|
||||
"""A merged slice must not silently accept registrations it cannot run."""
|
||||
def test_gateway_contribution_points_are_part_of_the_public_surface():
|
||||
import deerflow_extension_api
|
||||
|
||||
for name in (
|
||||
"ExtensionRuntimeDeps",
|
||||
"ExtensionService",
|
||||
):
|
||||
assert name not in deerflow_extension_api.__all__
|
||||
assert not hasattr(deerflow_extension_api, name)
|
||||
assert name in deerflow_extension_api.__all__
|
||||
assert hasattr(deerflow_extension_api, name)
|
||||
assert callable(ExtensionRegistry.service)
|
||||
assert callable(ExtensionRegistry.routers)
|
||||
assert not hasattr(deerflow_extension_api, "RouterContributor")
|
||||
|
||||
|
||||
def test_task_store_from_runtime_reads_the_host_key():
|
||||
@ -263,6 +269,15 @@ def test_distribution_marks_the_contract_package_as_typed():
|
||||
assert marker.is_file()
|
||||
|
||||
|
||||
def test_contract_package_keeps_runtime_dependencies_empty():
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
pyproject = Path(__file__).parent.parent / "packages" / "extension-api" / "pyproject.toml"
|
||||
|
||||
assert tomllib.loads(pyproject.read_text())["project"]["dependencies"] == []
|
||||
|
||||
|
||||
def test_harness_pins_the_contract_package_exactly():
|
||||
"""The version contract (extension-system design): the host pins the
|
||||
contract package exactly, extensions use ranges. A range here would let an
|
||||
@ -289,5 +304,17 @@ def test_runtime_api_version_matches_the_installed_contract_package():
|
||||
"""Every additive contract slice bumps both gates together."""
|
||||
from importlib.metadata import version
|
||||
|
||||
assert API_VERSION == "0.1.1"
|
||||
assert API_VERSION == "0.1.2"
|
||||
assert API_VERSION == version("deerflow-extension-api")
|
||||
|
||||
|
||||
def test_extension_service_contract_is_public_and_defaults_to_noop():
|
||||
class _Bare:
|
||||
pass
|
||||
|
||||
deps = ExtensionRuntimeDeps()
|
||||
|
||||
assert deps.app_store is None
|
||||
assert deps.session_factory is None
|
||||
assert asyncio.run(ExtensionService.start(_Bare(), deps)) is None
|
||||
assert asyncio.run(ExtensionService.stop(_Bare())) is None
|
||||
|
||||
@ -85,6 +85,53 @@ def test_create_app_exposes_one_canonical_live_diagnostics_list(monkeypatch):
|
||||
]
|
||||
|
||||
|
||||
def test_create_app_mounts_extension_routers_after_all_host_routes(monkeypatch):
|
||||
from fastapi import APIRouter
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import deerflow.extensions as extensions_module
|
||||
|
||||
conflict = APIRouter()
|
||||
good = APIRouter()
|
||||
|
||||
@conflict.get("/health")
|
||||
async def shadow_health():
|
||||
return {"status": "extension"}
|
||||
|
||||
@good.get("/api/extension-test/ping")
|
||||
async def ping():
|
||||
return {"ok": True}
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("router:install"):
|
||||
registry.routers((conflict, good))
|
||||
loaded = registry.build()
|
||||
monkeypatch.setattr(
|
||||
extensions_module,
|
||||
"load_extensions",
|
||||
lambda plugins: (loaded, []),
|
||||
)
|
||||
|
||||
from app.gateway.app import create_app
|
||||
|
||||
app = create_app()
|
||||
paths = [getattr(route, "path", None) for route in app.routes]
|
||||
|
||||
assert paths.count("/health") == 1
|
||||
assert "/api/extension-test/ping" in paths
|
||||
assert len(app.state.extension_diagnostics) == 1
|
||||
assert app.state.extension_diagnostics == extensions_module.get_runtime_diagnostics()
|
||||
assert app.state.extension_diagnostics[0].source == "router:install"
|
||||
assert "host" in app.state.extension_diagnostics[0].message
|
||||
|
||||
client = TestClient(app)
|
||||
assert client.get("/health").json() == {
|
||||
"status": "healthy",
|
||||
"service": "deer-flow-gateway",
|
||||
}
|
||||
assert client.get("/api/extension-test/ping").status_code == 401
|
||||
|
||||
|
||||
def test_create_app_fails_open_when_extension_loading_raises_unexpectedly(monkeypatch):
|
||||
import deerflow.extensions as extensions_module
|
||||
|
||||
|
||||
298
backend/tests/test_extension_dependency_sync.py
Normal file
298
backend/tests/test_extension_dependency_sync.py
Normal file
@ -0,0 +1,298 @@
|
||||
"""Contracts for installing declared Python extensions in every startup mode."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import tomllib
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
BACKEND_ROOT = REPO_ROOT / "backend"
|
||||
|
||||
|
||||
def _make_recipe(path: Path, target: str) -> str:
|
||||
content = path.read_text(encoding="utf-8")
|
||||
match = re.search(rf"^{re.escape(target)}:[^\n]*\n(?P<recipe>(?:\t[^\n]*\n)+)", content, re.MULTILINE)
|
||||
assert match is not None, f"missing {target!r} target in {path}"
|
||||
return match.group("recipe")
|
||||
|
||||
|
||||
def test_extensions_dependency_group_is_part_of_the_default_sync() -> None:
|
||||
project = tomllib.loads((BACKEND_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
|
||||
|
||||
assert isinstance(project["dependency-groups"]["extensions"], list)
|
||||
assert set(project["tool"]["uv"]["default-groups"]) == {"dev", "extensions"}
|
||||
|
||||
|
||||
def test_backend_make_targets_never_mutate_the_extension_lock() -> None:
|
||||
makefile = BACKEND_ROOT / "Makefile"
|
||||
|
||||
assert "uv sync --locked" in _make_recipe(makefile, "install")
|
||||
for target in ("dev", "gateway"):
|
||||
recipe = _make_recipe(makefile, target)
|
||||
assert "uv run --locked uvicorn" in recipe, target
|
||||
assert "uv sync" not in recipe, target
|
||||
|
||||
|
||||
def test_root_local_startup_syncs_the_locked_backend_before_runtime() -> None:
|
||||
root_install = _make_recipe(REPO_ROOT / "Makefile", "install")
|
||||
serve = (REPO_ROOT / "scripts" / "serve.sh").read_text(encoding="utf-8")
|
||||
|
||||
assert "cd backend && uv sync --locked" in root_install
|
||||
sync_at = serve.find("uv sync --locked")
|
||||
runtime_at = serve.find("uv run --no-sync uvicorn app.gateway.app:app")
|
||||
assert sync_at != -1
|
||||
assert runtime_at > sync_at
|
||||
|
||||
|
||||
def test_root_makefile_exposes_extension_management_commands() -> None:
|
||||
makefile = REPO_ROOT / "Makefile"
|
||||
|
||||
install = _make_recipe(makefile, "extension-install")
|
||||
assert "deerflow extensions install" in install
|
||||
assert "--source-env __deerflow_extension_source__" in install
|
||||
assert "DEER_FLOW_EXTENSION_SOURCE" not in install
|
||||
assert "$(SOURCE)" not in install
|
||||
assert "uv run --frozen --no-group extensions" in install
|
||||
assert "--yes" not in install
|
||||
|
||||
for target, command in (
|
||||
("extension-list", "deerflow extensions list"),
|
||||
("extension-enable", "deerflow extensions enable"),
|
||||
("extension-disable", "deerflow extensions disable"),
|
||||
("extension-remove", "deerflow extensions remove"),
|
||||
):
|
||||
recipe = _make_recipe(makefile, target)
|
||||
assert command in recipe
|
||||
assert "uv run --frozen --no-group extensions" in recipe
|
||||
|
||||
|
||||
def test_extension_management_bootstrap_does_not_resolve_a_broken_extension_source() -> None:
|
||||
makefile = REPO_ROOT / "Makefile"
|
||||
|
||||
for target in (
|
||||
"extension-install",
|
||||
"extension-list",
|
||||
"extension-enable",
|
||||
"extension-disable",
|
||||
"extension-remove",
|
||||
):
|
||||
recipe = _make_recipe(makefile, target)
|
||||
assert "uv run --frozen --no-group extensions" in recipe, target
|
||||
assert "--locked" not in recipe, target
|
||||
|
||||
|
||||
def test_frozen_management_bootstrap_installs_core_without_reading_a_missing_extension(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
project = tmp_path / "backend"
|
||||
wheels = tmp_path / "wheels"
|
||||
project.mkdir()
|
||||
wheels.mkdir()
|
||||
|
||||
def _write_wheel(distribution: str, module: str) -> Path:
|
||||
wheel = wheels / f"{distribution.replace('-', '_')}-1.0.0-py3-none-any.whl"
|
||||
dist_info = f"{distribution.replace('-', '_')}-1.0.0.dist-info"
|
||||
records = {
|
||||
f"{module}/__init__.py": "VALUE = 'installed'\n",
|
||||
f"{dist_info}/METADATA": f"Metadata-Version: 2.1\nName: {distribution}\nVersion: 1.0.0\n",
|
||||
f"{dist_info}/WHEEL": "Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: py3-none-any\n",
|
||||
}
|
||||
records[f"{dist_info}/RECORD"] = "".join(f"{name},,\n" for name in (*records, f"{dist_info}/RECORD"))
|
||||
with zipfile.ZipFile(wheel, "w") as archive:
|
||||
for name, content in records.items():
|
||||
archive.writestr(name, content)
|
||||
return wheel
|
||||
|
||||
core_wheel = _write_wheel("bootstrap-core", "bootstrap_core")
|
||||
missing_extension = _write_wheel("broken-extension", "broken_extension")
|
||||
(project / "pyproject.toml").write_text(
|
||||
f"""\
|
||||
[project]
|
||||
name = "bootstrap-host"
|
||||
version = "0.0.0"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = ["bootstrap-core @ {core_wheel.as_uri()}"]
|
||||
|
||||
[dependency-groups]
|
||||
extensions = ["broken-extension @ {missing_extension.as_uri()}"]
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
environment = os.environ.copy()
|
||||
environment["UV_CACHE_DIR"] = str(tmp_path / "uv-cache")
|
||||
subprocess.run(["uv", "lock"], cwd=project, env=environment, check=True, capture_output=True)
|
||||
missing_extension.unlink()
|
||||
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"uv",
|
||||
"run",
|
||||
"--frozen",
|
||||
"--no-group",
|
||||
"extensions",
|
||||
"python",
|
||||
"-c",
|
||||
"import bootstrap_core; assert bootstrap_core.VALUE == 'installed'",
|
||||
],
|
||||
cwd=project,
|
||||
env=environment,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert completed.returncode == 0, completed.stderr
|
||||
|
||||
|
||||
def test_root_extension_shortcuts_are_cross_platform_and_keep_trust_confirmation() -> None:
|
||||
makefile = REPO_ROOT / "Makefile"
|
||||
|
||||
for target in (
|
||||
"extension-install",
|
||||
"extension-enable",
|
||||
"extension-disable",
|
||||
"extension-remove",
|
||||
):
|
||||
recipe = _make_recipe(makefile, target)
|
||||
assert "test -n" not in recipe, target
|
||||
assert "usage: make" in recipe, target
|
||||
|
||||
assert "--yes" not in _make_recipe(makefile, "extension-install")
|
||||
|
||||
|
||||
def test_root_extension_shortcuts_reject_ambient_environment_arguments() -> None:
|
||||
environment = os.environ.copy()
|
||||
|
||||
for target, variable in (("extension-install", "SOURCE"), ("extension-enable", "NAME")):
|
||||
environment[variable] = "ambient-value"
|
||||
result = subprocess.run(
|
||||
["make", "--no-print-directory", "-n", target],
|
||||
cwd=REPO_ROOT,
|
||||
env=environment,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode == 2, target
|
||||
assert f"usage: make {target}" in result.stderr, target
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("target", "variable", "env_option"),
|
||||
[
|
||||
("extension-install", "SOURCE", "--source-env __deerflow_extension_source__"),
|
||||
("extension-enable", "NAME", "--name-env __deerflow_extension_name__"),
|
||||
("extension-disable", "NAME", "--name-env __deerflow_extension_name__"),
|
||||
("extension-remove", "NAME", "--name-env __deerflow_extension_name__"),
|
||||
],
|
||||
)
|
||||
def test_root_extension_shortcuts_keep_command_line_arguments_out_of_the_shell_recipe(
|
||||
target: str,
|
||||
variable: str,
|
||||
env_option: str,
|
||||
) -> None:
|
||||
marker = "EXTENSION_WRAPPER_INJECTION"
|
||||
malicious_value = f'"; printf {marker}; #$(shell printf {marker})'
|
||||
|
||||
result = subprocess.run(
|
||||
["make", "--no-print-directory", "-n", target, f"{variable}={malicious_value}"],
|
||||
cwd=REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert marker not in result.stdout
|
||||
assert env_option in result.stdout
|
||||
assert "$DEER_FLOW_EXTENSION_" not in result.stdout
|
||||
assert "%DEER_FLOW_EXTENSION_" not in result.stdout
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("target", "variable", "env_option"),
|
||||
[
|
||||
("extension-install", "SOURCE", "--source-env __deerflow_extension_source__"),
|
||||
("extension-enable", "NAME", "--name-env __deerflow_extension_name__"),
|
||||
("extension-disable", "NAME", "--name-env __deerflow_extension_name__"),
|
||||
("extension-remove", "NAME", "--name-env __deerflow_extension_name__"),
|
||||
],
|
||||
)
|
||||
def test_root_extension_shortcuts_keep_values_out_of_the_cmd_recipe_on_windows(
|
||||
target: str,
|
||||
variable: str,
|
||||
env_option: str,
|
||||
) -> None:
|
||||
marker = "EXTENSION_WRAPPER_INJECTION"
|
||||
malicious_value = f'"; printf {marker}; #$(shell printf {marker})'
|
||||
|
||||
result = subprocess.run(
|
||||
[
|
||||
"make",
|
||||
"--no-print-directory",
|
||||
"-n",
|
||||
"OS=Windows_NT",
|
||||
target,
|
||||
f"{variable}={malicious_value}",
|
||||
],
|
||||
cwd=REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert marker not in result.stdout
|
||||
assert env_option in result.stdout
|
||||
assert "$DEER_FLOW_EXTENSION_" not in result.stdout
|
||||
assert "%DEER_FLOW_EXTENSION_" not in result.stdout
|
||||
|
||||
|
||||
def test_docker_dev_entrypoint_syncs_the_lock_before_runtime() -> None:
|
||||
entrypoint = (REPO_ROOT / "docker" / "dev-entrypoint.sh").read_text(encoding="utf-8")
|
||||
|
||||
sync_at = entrypoint.find("uv sync --locked --all-packages")
|
||||
runtime_at = entrypoint.find("uv run --no-sync uvicorn app.gateway.app:app")
|
||||
assert sync_at != -1
|
||||
assert runtime_at > sync_at
|
||||
|
||||
|
||||
def test_docker_image_builds_from_the_lock_and_never_syncs_at_runtime() -> None:
|
||||
dockerfile = (BACKEND_ROOT / "Dockerfile").read_text(encoding="utf-8")
|
||||
production_compose = (REPO_ROOT / "docker" / "docker-compose.yaml").read_text(encoding="utf-8")
|
||||
|
||||
assert "uv sync --locked --extra redis" in dockerfile
|
||||
assert "ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.1" in dockerfile
|
||||
assert dockerfile.count("uv run --no-sync uvicorn app.gateway.app:app") == 2
|
||||
assert "uv run --no-sync uvicorn app.gateway.app:app" in production_compose
|
||||
for compose_name in ("docker-compose.yaml", "docker-compose-dev.yaml"):
|
||||
compose = (REPO_ROOT / "docker" / compose_name).read_text(encoding="utf-8")
|
||||
assert "ghcr.io/astral-sh/uv:0.11.1" in compose
|
||||
|
||||
|
||||
def test_docker_context_keeps_every_managed_extension_artifact() -> None:
|
||||
dockerignore = (REPO_ROOT / ".dockerignore").read_text(encoding="utf-8")
|
||||
|
||||
reinclusion = "!backend/extensions/sources/**"
|
||||
assert reinclusion in dockerignore
|
||||
assert dockerignore.rfind(reinclusion) > dockerignore.rfind("*.md")
|
||||
assert dockerignore.rfind(reinclusion) > dockerignore.rfind("assets/")
|
||||
assert dockerignore.rfind(reinclusion) > dockerignore.rfind("*.so")
|
||||
|
||||
|
||||
def test_docker_builder_can_resolve_locked_git_extensions() -> None:
|
||||
dockerfile = (BACKEND_ROOT / "Dockerfile").read_text(encoding="utf-8")
|
||||
|
||||
builder_packages = re.search(
|
||||
r"apt-get install -y \\\n(?P<packages>.*?) && mkdir -p /etc/apt/keyrings",
|
||||
dockerfile,
|
||||
re.DOTALL,
|
||||
)
|
||||
assert builder_packages is not None
|
||||
assert re.search(r"^\s*git\s+\\$", builder_packages.group("packages"), re.MULTILINE)
|
||||
985
backend/tests/test_extension_gateway_wiring.py
Normal file
985
backend/tests/test_extension_gateway_wiring.py
Normal file
@ -0,0 +1,985 @@
|
||||
"""Gateway binding tests for app-scoped extension contributions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.extensions.registry import ExtensionRegistry
|
||||
|
||||
|
||||
class _Service:
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
events: list[str],
|
||||
*,
|
||||
fail_start: bool = False,
|
||||
fail_stop: bool = False,
|
||||
) -> None:
|
||||
self.name = name
|
||||
self.events = events
|
||||
self.fail_start = fail_start
|
||||
self.fail_stop = fail_stop
|
||||
self.deps = None
|
||||
|
||||
async def start(self, deps) -> None:
|
||||
self.events.append(f"start:{self.name}")
|
||||
self.deps = deps
|
||||
if self.fail_start:
|
||||
raise RuntimeError(f"{self.name} failed")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self.events.append(f"stop:{self.name}")
|
||||
if self.fail_stop:
|
||||
raise RuntimeError(f"{self.name} failed")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_services_start_in_order_with_narrow_deps_and_fail_open():
|
||||
from deerflow.extensions.gateway import start_services
|
||||
|
||||
events: list[str] = []
|
||||
first = _Service("first", events, fail_start=True)
|
||||
second = _Service("second", events)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(first)
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.service(second)
|
||||
extensions = registry.build()
|
||||
session_factory = object()
|
||||
config = SimpleNamespace(
|
||||
token_budget=SimpleNamespace(enabled=False, max_tokens=999),
|
||||
subagents=SimpleNamespace(max_total_per_run=4),
|
||||
)
|
||||
|
||||
diagnostics = await start_services(extensions, config, session_factory)
|
||||
|
||||
assert events == ["start:first", "start:second"]
|
||||
assert first.deps is second.deps
|
||||
assert second.deps.app_store is extensions.app_store
|
||||
assert second.deps.session_factory is session_factory
|
||||
assert second.deps.policy.token_budget_enabled is False
|
||||
assert second.deps.policy.max_total_tokens is None
|
||||
assert second.deps.policy.max_subagents_per_run == 4
|
||||
assert [(diagnostic.source, diagnostic.level) for diagnostic in diagnostics] == [("first:install", "error")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_originated_cancelled_error_does_not_abort_start_batch():
|
||||
from deerflow.extensions.gateway import start_services
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
class _CancelsItself(_Service):
|
||||
async def start(self, deps) -> None:
|
||||
self.events.append(f"start:{self.name}")
|
||||
raise asyncio.CancelledError()
|
||||
|
||||
first = _CancelsItself("first", events)
|
||||
second = _Service("second", events)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(first)
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.service(second)
|
||||
|
||||
diagnostics = await start_services(registry.build(), SimpleNamespace(), None)
|
||||
|
||||
assert events == ["start:first", "start:second"]
|
||||
assert len(diagnostics) == 1
|
||||
assert diagnostics[0].source == "first:install"
|
||||
assert "CancelledError" in diagnostics[0].message
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_services_stop_in_reverse_order_and_fail_open():
|
||||
from deerflow.extensions.gateway import stop_services
|
||||
|
||||
events: list[str] = []
|
||||
first = _Service("first", events)
|
||||
second = _Service("second", events, fail_stop=True)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(first)
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.service(second)
|
||||
|
||||
diagnostics = await stop_services(registry.build())
|
||||
|
||||
assert events == ["stop:second", "stop:first"]
|
||||
assert [(diagnostic.source, diagnostic.level) for diagnostic in diagnostics] == [("second:install", "error")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_originated_cancelled_error_does_not_abort_stop_batch():
|
||||
from deerflow.extensions.gateway import stop_services
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
class _CancelsItself(_Service):
|
||||
async def stop(self) -> None:
|
||||
self.events.append(f"stop:{self.name}")
|
||||
raise asyncio.CancelledError()
|
||||
|
||||
first = _Service("first", events)
|
||||
second = _CancelsItself("second", events)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(first)
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.service(second)
|
||||
|
||||
diagnostics = await stop_services(registry.build())
|
||||
|
||||
assert events == ["stop:second", "stop:first"]
|
||||
assert len(diagnostics) == 1
|
||||
assert diagnostics[0].source == "second:install"
|
||||
assert "CancelledError" in diagnostics[0].message
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_each_service_stop_has_its_own_timeout_budget():
|
||||
from deerflow.extensions.gateway import stop_services
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
class _HangingService(_Service):
|
||||
async def stop(self) -> None:
|
||||
self.events.append(f"stop:{self.name}")
|
||||
await asyncio.Event().wait()
|
||||
|
||||
first = _Service("first", events)
|
||||
second = _HangingService("second", events)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(first)
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.service(second)
|
||||
|
||||
diagnostics = await stop_services(registry.build(), timeout_seconds=0.01)
|
||||
|
||||
assert events == ["stop:second", "stop:first"]
|
||||
assert len(diagnostics) == 1
|
||||
assert diagnostics[0].source == "second:install"
|
||||
assert "timed out" in diagnostics[0].message
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_originated_timeout_error_is_reported_as_failure_not_budget_expiry():
|
||||
from deerflow.extensions.gateway import stop_services
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
class _RaisesTimeout(_Service):
|
||||
async def stop(self) -> None:
|
||||
self.events.append(f"stop:{self.name}")
|
||||
raise TimeoutError("extension deadline")
|
||||
|
||||
first = _Service("first", events)
|
||||
second = _RaisesTimeout("second", events)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(first)
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.service(second)
|
||||
|
||||
diagnostics = await stop_services(registry.build(), timeout_seconds=1.0)
|
||||
|
||||
assert events == ["stop:second", "stop:first"]
|
||||
assert diagnostics[0].source == "second:install"
|
||||
assert "failed" in diagnostics[0].message
|
||||
assert "timed out" not in diagnostics[0].message
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("owner_path", "owner_protocol", "candidate_path", "candidate_protocol", "rejected"),
|
||||
[
|
||||
("/exact", "GET", "/exact", "GET", True),
|
||||
("/items/{item_id}", "GET", "/items/{id}", "GET", True),
|
||||
("/items/{item_id}", "GET", "/items/new", "GET", True),
|
||||
("/items/{item_id}", "GET", "/items/prefix-{id}", "GET", True),
|
||||
("/pre{tenant}", "GET", "/prefoo{id}", "GET", True),
|
||||
("/items/new", "GET", "/items/{id}", "GET", False),
|
||||
("/records/{value}", "GET", "/records/{id:int}", "GET", True),
|
||||
("/records/{value:int}", "GET", "/records/0{id:int}", "GET", True),
|
||||
("/records/{value:int}", "GET", "/records/new", "GET", False),
|
||||
("/files/{rest:path}", "GET", "/files/{id:int}", "GET", True),
|
||||
("/files/{rest:path}", "GET", "/files/{id}", "GET", False),
|
||||
("/x/{rest:path}/tail", "GET", "/x/a/{id}/tail", "GET", False),
|
||||
("/items/{item_id}", "GET", "/items/{id}", "POST", False),
|
||||
("/live/{item_id}", "WS", "/live/{id}", "GET", False),
|
||||
],
|
||||
ids=[
|
||||
"exact",
|
||||
"renamed-parameter",
|
||||
"dynamic-shadows-static",
|
||||
"dynamic-shadows-compound",
|
||||
"compound-trailing-str-shadows-narrower-compound",
|
||||
"static-does-not-shadow-dynamic",
|
||||
"str-covers-int",
|
||||
"int-shadows-digit-compound",
|
||||
"int-does-not-cover-static",
|
||||
"path-covers-descendant",
|
||||
"path-does-not-cover-newline-capable-str",
|
||||
"nonterminal-path-does-not-cover-newline-capable-str",
|
||||
"disjoint-http-methods",
|
||||
"websocket-does-not-shadow-http",
|
||||
],
|
||||
)
|
||||
def test_router_conflicts_follow_starlette_dispatch_order(
|
||||
owner_path,
|
||||
owner_protocol,
|
||||
candidate_path,
|
||||
candidate_protocol,
|
||||
rejected,
|
||||
):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
app = FastAPI()
|
||||
if owner_protocol == "WS":
|
||||
app.add_api_websocket_route(owner_path, endpoint)
|
||||
else:
|
||||
app.add_api_route(owner_path, endpoint, methods=[owner_protocol])
|
||||
|
||||
router = APIRouter()
|
||||
if candidate_protocol == "WS":
|
||||
router.add_api_websocket_route(candidate_path, endpoint)
|
||||
else:
|
||||
router.add_api_route(candidate_path, endpoint, methods=[candidate_protocol])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("candidate:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert bool(diagnostics) is rejected
|
||||
if rejected:
|
||||
assert diagnostics[0].source == "candidate:install"
|
||||
assert "host" in diagnostics[0].message
|
||||
assert candidate_path in diagnostics[0].message
|
||||
else:
|
||||
assert any(getattr(route, "path", None) == candidate_path for route in app.routes)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("convertor_name", ["flip", "int"])
|
||||
def test_re_registered_convertor_does_not_create_a_false_shadow(
|
||||
monkeypatch,
|
||||
convertor_name,
|
||||
):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
||||
from starlette.routing import Match
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
class DigitsConvertor(Convertor[str]):
|
||||
regex = "[0-9]+"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
class LettersConvertor(Convertor[str]):
|
||||
regex = "[A-Z]+"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
async def endpoint(value: str):
|
||||
return {"value": value}
|
||||
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, convertor_name, DigitsConvertor())
|
||||
route_path = f"/owned/{{value:{convertor_name}}}"
|
||||
app = FastAPI()
|
||||
app.add_api_route(route_path, endpoint, methods=["GET"])
|
||||
owner = app.routes[-1]
|
||||
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, convertor_name, LettersConvertor())
|
||||
router = APIRouter()
|
||||
router.add_api_route(route_path, endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("candidate:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert diagnostics == []
|
||||
candidate = app.routes[-1]
|
||||
scope = {
|
||||
"type": "http",
|
||||
"path": "/owned/A",
|
||||
"method": "GET",
|
||||
"root_path": "",
|
||||
}
|
||||
assert owner.matches(scope)[0] is Match.NONE
|
||||
assert candidate.matches(scope)[0] is Match.FULL
|
||||
|
||||
|
||||
def test_router_claim_uses_converter_semantics_at_include_time(monkeypatch):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
||||
from starlette.routing import Match
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
class DigitsConvertor(Convertor[str]):
|
||||
regex = "[0-9]+"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
class LettersConvertor(Convertor[str]):
|
||||
regex = "[A-Z]+"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
async def endpoint(value: str):
|
||||
return {"value": value}
|
||||
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, "flip", DigitsConvertor())
|
||||
route_path = "/owned/{value:flip}"
|
||||
app = FastAPI()
|
||||
app.add_api_route(route_path, endpoint, methods=["GET"])
|
||||
owner = app.routes[-1]
|
||||
router = APIRouter()
|
||||
router.add_api_route(route_path, endpoint, methods=["GET"])
|
||||
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, "flip", LettersConvertor())
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("candidate:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert diagnostics == []
|
||||
candidate = app.routes[-1]
|
||||
scope = {
|
||||
"type": "http",
|
||||
"path": "/owned/A",
|
||||
"method": "GET",
|
||||
"root_path": "",
|
||||
}
|
||||
assert owner.matches(scope)[0] is Match.NONE
|
||||
assert candidate.matches(scope)[0] is Match.FULL
|
||||
|
||||
|
||||
def test_recompiled_converter_cannot_enter_a_public_namespace(monkeypatch):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
class PublicPathConvertor(Convertor[str]):
|
||||
regex = r"webhooks/.+"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
async def endpoint(value: str):
|
||||
return {"value": value}
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_route("/api/{value:int}", endpoint, methods=["GET"])
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, "int", PublicPathConvertor())
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("candidate:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["candidate:install"]
|
||||
assert "public namespace" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", None) == "/api/{value:int}" for route in app.routes)
|
||||
|
||||
|
||||
def test_host_mount_claims_descendant_http_paths():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
app = FastAPI()
|
||||
app.mount("/assets", FastAPI())
|
||||
router = APIRouter()
|
||||
|
||||
@router.get("/assets/{name:int}")
|
||||
async def asset(name: int):
|
||||
return {"name": name}
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("assets:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert len(diagnostics) == 1
|
||||
assert diagnostics[0].source == "assets:install"
|
||||
assert "host" in diagnostics[0].message
|
||||
|
||||
|
||||
def test_dynamic_host_mount_claims_matching_descendants():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
app = FastAPI()
|
||||
app.mount("/pre{tenant}", FastAPI())
|
||||
router = APIRouter()
|
||||
|
||||
@router.get("/prefoo/{item_id:int}")
|
||||
async def item(item_id: int):
|
||||
return {"item_id": item_id}
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("mount:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["mount:install"]
|
||||
assert "host" in diagnostics[0].message
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mount_path", "candidate_path", "witness"),
|
||||
[
|
||||
("/assets", "/assets/{name}", "/assets/a\nb"),
|
||||
("/pre{tenant}", "/prefoo{id}/{child}", "/prefoo1/a\nb"),
|
||||
],
|
||||
)
|
||||
def test_host_mount_does_not_claim_newline_capable_str_descendants(
|
||||
mount_path,
|
||||
candidate_path,
|
||||
witness,
|
||||
):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from starlette.routing import Match, Mount
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
app = FastAPI()
|
||||
app.mount(mount_path, FastAPI())
|
||||
host_mount = next(route for route in app.routes if isinstance(route, Mount) and route.path == mount_path)
|
||||
router = APIRouter()
|
||||
router.add_api_route(candidate_path, endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("mount:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert diagnostics == []
|
||||
candidate_route = next(route for route in app.routes if getattr(route, "path", None) == candidate_path)
|
||||
scope = {
|
||||
"type": "http",
|
||||
"path": witness,
|
||||
"method": "GET",
|
||||
"root_path": "",
|
||||
}
|
||||
assert host_mount.matches(scope)[0] is Match.NONE
|
||||
assert candidate_route.matches(scope)[0] is Match.FULL
|
||||
|
||||
|
||||
def test_contributed_websocket_route_is_rejected_until_host_auth_wraps_it():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def websocket_endpoint(websocket):
|
||||
await websocket.close()
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_websocket_route("/extension-ws", websocket_endpoint)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("websocket:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["websocket:install"]
|
||||
assert "WebSocket" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", None) == "/extension-ws" for route in app.routes)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[
|
||||
"/health-extension",
|
||||
"/docs-private",
|
||||
"/redoc-private",
|
||||
"/api/webhooks/extension",
|
||||
"/api/{rest:path}",
|
||||
"/api/{section}/extension",
|
||||
],
|
||||
)
|
||||
def test_extension_routes_cannot_enter_host_public_namespaces(path):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_route(path, endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("public:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["public:install"]
|
||||
assert "public namespace" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", None) == path for route in app.routes)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[
|
||||
"/api",
|
||||
"/heal",
|
||||
"/api/{item_id:int}",
|
||||
"/api/{item_id}",
|
||||
],
|
||||
)
|
||||
def test_extension_routes_that_cannot_enter_a_public_namespace_are_allowed(path):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_route(path, endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("private:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert diagnostics == []
|
||||
assert any(getattr(route, "path", None) == path for route in app.routes)
|
||||
|
||||
|
||||
def test_unknown_convertor_near_a_public_namespace_fails_closed(monkeypatch):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
class UppercaseConvertor(Convertor[str]):
|
||||
regex = "[A-Z]+"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, "uppercase", UppercaseConvertor())
|
||||
router = APIRouter()
|
||||
router.add_api_route(
|
||||
"/api/{value:uppercase}",
|
||||
lambda: {"ok": True},
|
||||
methods=["GET"],
|
||||
)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("custom-public:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(
|
||||
FastAPI(docs_url=None, redoc_url=None, openapi_url=None),
|
||||
registry.build(),
|
||||
)
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["custom-public:install"]
|
||||
assert "public namespace" in diagnostics[0].message
|
||||
|
||||
|
||||
def test_private_custom_convertor_with_named_backreference_is_allowed(monkeypatch):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
class DoubledLetterConvertor(Convertor[str]):
|
||||
regex = r"(?P<char>[A-Z])(?P=char)"
|
||||
|
||||
def convert(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
def to_string(self, value: str) -> str:
|
||||
return value
|
||||
|
||||
monkeypatch.setitem(CONVERTOR_TYPES, "doubled", DoubledLetterConvertor())
|
||||
router = APIRouter()
|
||||
router.add_api_route(
|
||||
"/private/{value:doubled}",
|
||||
lambda value: {"value": value},
|
||||
methods=["GET"],
|
||||
)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("custom-private:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert diagnostics == []
|
||||
assert any(getattr(route, "path", None) == "/private/{value:doubled}" for route in app.routes)
|
||||
|
||||
|
||||
def test_extension_public_paths_track_auth_middleware_public_paths():
|
||||
from app.gateway.auth_middleware import (
|
||||
_PUBLIC_EXACT_PATHS,
|
||||
_PUBLIC_PATH_PREFIXES,
|
||||
_is_public,
|
||||
)
|
||||
from deerflow.extensions.gateway import (
|
||||
_HOST_PUBLIC_EXACT_PATHS,
|
||||
_HOST_PUBLIC_PATH_PREFIXES,
|
||||
)
|
||||
|
||||
assert _HOST_PUBLIC_PATH_PREFIXES == _PUBLIC_PATH_PREFIXES
|
||||
assert _HOST_PUBLIC_EXACT_PATHS == _PUBLIC_EXACT_PATHS
|
||||
assert all(_is_public(f"{path}//") for path in _HOST_PUBLIC_EXACT_PATHS)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("host_path", "host_method", "candidate_path", "candidate_method"),
|
||||
[
|
||||
(
|
||||
"/api/v1/auth/login/local",
|
||||
"POST",
|
||||
"/api/v1/auth/login/local",
|
||||
"GET",
|
||||
),
|
||||
(
|
||||
"/api/v1/auth/login/local",
|
||||
"POST",
|
||||
"/api/v1/auth/login/local/",
|
||||
"GET",
|
||||
),
|
||||
(
|
||||
"/api/v1/auth/login/local",
|
||||
"POST",
|
||||
"/api/v1/auth/login/local//",
|
||||
"GET",
|
||||
),
|
||||
("/api/v1/auth/me", "GET", "/api/v1/auth/me", "POST"),
|
||||
("/api/v1/auth/me", "GET", "/api/v1/auth/me/", "POST"),
|
||||
],
|
||||
)
|
||||
def test_extension_routes_cannot_claim_reserved_exact_paths_with_a_disjoint_method(
|
||||
host_path,
|
||||
host_method,
|
||||
candidate_path,
|
||||
candidate_method,
|
||||
):
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
app = FastAPI()
|
||||
app.add_api_route(host_path, endpoint, methods=[host_method])
|
||||
router = APIRouter()
|
||||
router.add_api_route(candidate_path, endpoint, methods=[candidate_method])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("public-exact:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["public-exact:install"]
|
||||
assert "reserved" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", None) == candidate_path and getattr(route, "methods", set()) == {candidate_method} for route in app.routes)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[
|
||||
"/api/v1/auth/me",
|
||||
"/api/v1/auth/me/",
|
||||
"/api/v1/auth/me//",
|
||||
],
|
||||
)
|
||||
def test_extension_csrf_reserved_exact_paths_track_csrf_exemption(
|
||||
monkeypatch,
|
||||
path,
|
||||
):
|
||||
from starlette.requests import Request
|
||||
|
||||
from app.gateway import csrf_middleware
|
||||
from deerflow.extensions.gateway import (
|
||||
_CSRF_STATE_CHANGING_METHODS,
|
||||
_HOST_CSRF_EXEMPT_EXACT_PATHS,
|
||||
_HOST_PUBLIC_EXACT_PATHS,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(csrf_middleware, "is_auth_disabled", lambda: False)
|
||||
request = Request(
|
||||
{
|
||||
"type": "http",
|
||||
"method": "POST",
|
||||
"scheme": "http",
|
||||
"path": path,
|
||||
"raw_path": path.encode(),
|
||||
"query_string": b"",
|
||||
"headers": [],
|
||||
"server": ("testserver", 80),
|
||||
}
|
||||
)
|
||||
|
||||
assert path.rstrip("/") in _HOST_CSRF_EXEMPT_EXACT_PATHS
|
||||
assert _HOST_CSRF_EXEMPT_EXACT_PATHS == csrf_middleware._CSRF_EXEMPT_EXACT_PATHS
|
||||
assert _CSRF_STATE_CHANGING_METHODS == csrf_middleware._CSRF_STATE_CHANGING_METHODS
|
||||
assert csrf_middleware._AUTH_EXEMPT_PATHS <= _HOST_PUBLIC_EXACT_PATHS
|
||||
assert csrf_middleware.should_check_csrf(request) is False
|
||||
|
||||
|
||||
def test_safe_method_at_csrf_exempt_exact_path_is_not_reserved():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_route(
|
||||
"/api/v1/auth/me",
|
||||
lambda: {"ok": True},
|
||||
methods=["GET"],
|
||||
)
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("safe-csrf:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert diagnostics == []
|
||||
assert any(getattr(route, "path", None) == "/api/v1/auth/me" for route in app.routes)
|
||||
|
||||
|
||||
def test_router_with_one_conflict_is_rejected_atomically_and_names_first_owner():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
first = APIRouter()
|
||||
first.add_api_route("/shared", endpoint, methods=["GET"])
|
||||
second = APIRouter()
|
||||
second.add_api_route("/would-have-been-reachable", endpoint, methods=["GET"])
|
||||
second.add_api_route("/shared", endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.routers((first,))
|
||||
with registry.attributed_to("second:install"):
|
||||
registry.routers((second,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
paths = [getattr(route, "path", None) for route in app.routes]
|
||||
|
||||
assert "/shared" in paths
|
||||
assert "/would-have-been-reachable" not in paths
|
||||
assert len(diagnostics) == 1
|
||||
assert diagnostics[0].source == "second:install"
|
||||
assert "first:install" in diagnostics[0].message
|
||||
|
||||
|
||||
def test_router_is_rejected_when_its_own_earlier_route_shadows_a_later_one():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_route("/same/{value}", endpoint, methods=["GET"])
|
||||
router.add_api_route("/same/fixed", endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("self-shadow:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["self-shadow:install"]
|
||||
assert "self-shadow:install" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", "").startswith("/same/") for route in app.routes)
|
||||
|
||||
|
||||
def test_contributed_mount_is_rejected_but_does_not_starve_later_router():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
bad = APIRouter()
|
||||
bad.mount("/nested", FastAPI())
|
||||
good = APIRouter()
|
||||
|
||||
@good.get("/extension-good")
|
||||
async def extension_good():
|
||||
return {"ok": True}
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("bad:install"):
|
||||
registry.routers((bad,))
|
||||
with registry.attributed_to("good:install"):
|
||||
registry.routers((good,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["bad:install"]
|
||||
assert "Mount" in diagnostics[0].message
|
||||
assert any(getattr(route, "path", None) == "/extension-good" for route in app.routes)
|
||||
|
||||
|
||||
def test_router_with_unsupported_route_item_is_rejected_atomically():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
router = APIRouter()
|
||||
router.add_api_route("/otherwise-valid", endpoint, methods=["GET"])
|
||||
router.routes.append(object())
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("unsupported:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["unsupported:install"]
|
||||
assert "unsupported route" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", None) == "/otherwise-valid" for route in app.routes)
|
||||
|
||||
|
||||
def test_include_router_failure_rolls_back_partial_routes_before_continuing():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
broken = APIRouter()
|
||||
broken.add_api_route("/partial", endpoint, methods=["GET"])
|
||||
broken.add_api_route("/explodes", endpoint, methods=["GET"])
|
||||
broken.routes[-1].endpoint = None
|
||||
later = APIRouter()
|
||||
later.add_api_route("/partial", endpoint, methods=["GET"])
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("broken:install"):
|
||||
registry.routers((broken,))
|
||||
with registry.attributed_to("later:install"):
|
||||
registry.routers((later,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
partial_routes = [route for route in app.routes if getattr(route, "path", None) == "/partial"]
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["broken:install"]
|
||||
assert len(partial_routes) == 1
|
||||
|
||||
|
||||
def test_router_lifecycle_hooks_are_rejected_in_favor_of_extension_service():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
async def startup_hook():
|
||||
raise RuntimeError("must never be installed")
|
||||
|
||||
bad = APIRouter()
|
||||
bad.add_api_route("/has-lifecycle", endpoint, methods=["GET"])
|
||||
bad.add_event_handler("startup", startup_hook)
|
||||
good = APIRouter()
|
||||
good.add_api_route("/after-lifecycle", endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("lifecycle:install"):
|
||||
registry.routers((bad,))
|
||||
with registry.attributed_to("good:install"):
|
||||
registry.routers((good,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
paths = [getattr(route, "path", None) for route in app.routes]
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["lifecycle:install"]
|
||||
assert "ExtensionService" in diagnostics[0].message
|
||||
assert "/has-lifecycle" not in paths
|
||||
assert "/after-lifecycle" in paths
|
||||
assert startup_hook not in app.router.on_startup
|
||||
|
||||
|
||||
def test_router_custom_lifespan_is_rejected_in_favor_of_extension_service():
|
||||
from fastapi import APIRouter, FastAPI
|
||||
|
||||
from deerflow.extensions.gateway import include_contributed_routers
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_app):
|
||||
yield
|
||||
|
||||
async def endpoint():
|
||||
return {"ok": True}
|
||||
|
||||
router = APIRouter(lifespan=lifespan)
|
||||
router.add_api_route("/custom-lifespan", endpoint, methods=["GET"])
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("lifespan:install"):
|
||||
registry.routers((router,))
|
||||
|
||||
app = FastAPI()
|
||||
diagnostics = include_contributed_routers(app, registry.build())
|
||||
|
||||
assert [diagnostic.source for diagnostic in diagnostics] == ["lifespan:install"]
|
||||
assert "ExtensionService" in diagnostics[0].message
|
||||
assert not any(getattr(route, "path", None) == "/custom-lifespan" for route in app.routes)
|
||||
@ -28,6 +28,30 @@ def test_no_specs_yields_empty_result():
|
||||
assert loaded.has_middleware_contributors is False
|
||||
|
||||
|
||||
def test_host_disabled_required_extension_is_skipped_before_resolution(monkeypatch):
|
||||
def _must_not_resolve(path: str):
|
||||
raise AssertionError(f"disabled extension was resolved: {path}")
|
||||
|
||||
monkeypatch.setattr("deerflow.extensions.loader.resolve_variable", _must_not_resolve)
|
||||
|
||||
loaded, diagnostics = load_extensions([ExtensionSpec(use="missing_extension:install", enabled=False, required=True)])
|
||||
|
||||
assert diagnostics == []
|
||||
assert loaded.has_middleware_contributors is False
|
||||
|
||||
|
||||
def test_manager_metadata_is_accepted_without_reaching_the_install_hook():
|
||||
spec = ExtensionSpec(
|
||||
name="demo",
|
||||
package="deerflow-extension-demo",
|
||||
use=f"{_FIXTURE}:install_ok",
|
||||
enabled=False,
|
||||
)
|
||||
|
||||
assert spec.name == "demo"
|
||||
assert spec.package == "deerflow-extension-demo"
|
||||
|
||||
|
||||
def test_successful_install_registers_and_attributes():
|
||||
spec = ExtensionSpec(use=f"{_FIXTURE}:install_ok")
|
||||
loaded, diagnostics = load_extensions([spec])
|
||||
@ -88,6 +112,9 @@ def test_install_failure_rolls_back_partial_registration():
|
||||
assert sources == {f"{_FIXTURE}:install_ok"}
|
||||
assert len(loaded.middleware_contributors) == 1, "rollback must clear every partial registration"
|
||||
assert loaded.task_lifecycle == (), "rollback must clear partial lifecycle registrations too"
|
||||
assert loaded.system_model_observers == ()
|
||||
assert loaded.services == ()
|
||||
assert loaded.routers == ()
|
||||
|
||||
|
||||
def test_rollback_does_not_remove_a_different_specs_registrations_sharing_the_same_use():
|
||||
|
||||
2302
backend/tests/test_extension_manager.py
Normal file
2302
backend/tests/test_extension_manager.py
Normal file
File diff suppressed because it is too large
Load Diff
@ -80,6 +80,34 @@ def test_system_model_observers_are_attributed_and_require_task_storage():
|
||||
assert loaded.needs_task_store is True
|
||||
|
||||
|
||||
def test_services_are_attributed_without_allocating_task_storage():
|
||||
registry = ExtensionRegistry()
|
||||
service = _Contributor("service")
|
||||
with registry.attributed_to("service_ext:install"):
|
||||
registry.service(service)
|
||||
|
||||
loaded = registry.build()
|
||||
|
||||
assert loaded.services == (("service_ext:install", service),)
|
||||
assert loaded.needs_task_store is False
|
||||
|
||||
|
||||
def test_routers_are_flattened_in_registration_order_without_task_storage():
|
||||
registry = ExtensionRegistry()
|
||||
first = _Contributor("first-router")
|
||||
second = _Contributor("second-router")
|
||||
with registry.attributed_to("router_ext:install"):
|
||||
registry.routers((first, second))
|
||||
|
||||
loaded = registry.build()
|
||||
|
||||
assert loaded.routers == (
|
||||
("router_ext:install", first),
|
||||
("router_ext:install", second),
|
||||
)
|
||||
assert loaded.needs_task_store is False
|
||||
|
||||
|
||||
def test_rollback_restores_all_registration_buckets_positionally():
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("keep:install"):
|
||||
@ -89,6 +117,8 @@ def test_rollback_restores_all_registration_buckets_positionally():
|
||||
registry.middlewares(_Contributor("drop-middleware"))
|
||||
registry.task_lifecycle(_Contributor("drop-lifecycle"))
|
||||
registry.system_model_observer(_Contributor("drop-observer"))
|
||||
registry.service(_Contributor("drop-service"))
|
||||
registry.routers((_Contributor("drop-router"),))
|
||||
|
||||
registry.rollback_to(mark)
|
||||
loaded = registry.build()
|
||||
@ -96,6 +126,8 @@ def test_rollback_restores_all_registration_buckets_positionally():
|
||||
assert [contributor.tag for _, contributor in loaded.middleware_contributors] == ["keep"]
|
||||
assert loaded.task_lifecycle == ()
|
||||
assert loaded.system_model_observers == ()
|
||||
assert loaded.services == ()
|
||||
assert loaded.routers == ()
|
||||
|
||||
|
||||
def test_registration_order_is_preserved():
|
||||
@ -120,11 +152,15 @@ def test_discard_removes_every_entry_of_one_source():
|
||||
with registry.attributed_to("bad:install"):
|
||||
registry.middlewares(drop)
|
||||
registry.system_model_observer(drop)
|
||||
registry.service(drop)
|
||||
registry.routers((drop,))
|
||||
registry.discard("bad:install")
|
||||
loaded = registry.build()
|
||||
assert loaded.middleware_contributors == (("good:install", keep),)
|
||||
assert loaded.task_lifecycle == (("good:install", keep),)
|
||||
assert loaded.system_model_observers == ()
|
||||
assert loaded.services == ()
|
||||
assert loaded.routers == ()
|
||||
|
||||
|
||||
def test_registering_outside_attributed_to_raises():
|
||||
|
||||
203
backend/tests/test_gateway_extension_service_lifecycle.py
Normal file
203
backend/tests/test_gateway_extension_service_lifecycle.py
Normal file
@ -0,0 +1,203 @@
|
||||
"""Regression tests for Gateway ownership of extension services."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
|
||||
from deerflow.extensions import (
|
||||
get_runtime_diagnostics,
|
||||
initialize_runtime_diagnostics,
|
||||
reset_runtime_diagnostics,
|
||||
)
|
||||
from deerflow.extensions.registry import ExtensionRegistry
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_runtime_diagnostics():
|
||||
reset_runtime_diagnostics()
|
||||
yield
|
||||
reset_runtime_diagnostics()
|
||||
|
||||
|
||||
def _database_config() -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
backend="memory",
|
||||
checkpoint_channel_mode="full",
|
||||
checkpoint_delta=SimpleNamespace(snapshot_frequency=10),
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _resource(value):
|
||||
yield value
|
||||
|
||||
|
||||
def _patch_runtime_resources(monkeypatch, events: list[str]) -> None:
|
||||
async def init_engine(_database) -> None:
|
||||
return None
|
||||
|
||||
async def close_engine() -> None:
|
||||
events.append("engine_close")
|
||||
|
||||
monkeypatch.setattr("deerflow.runtime.make_stream_bridge", lambda _config: _resource(object()))
|
||||
monkeypatch.setattr("deerflow.runtime.make_store", lambda _config: _resource(object()))
|
||||
monkeypatch.setattr("deerflow.runtime.checkpointer.async_provider.make_checkpointer", lambda _config: _resource(object()))
|
||||
monkeypatch.setattr("deerflow.persistence.engine.init_engine_from_config", init_engine)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.close_engine", close_engine)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", lambda: None)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_runtime_owns_engine_cleanup_before_initialization(monkeypatch):
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
async def fail_engine_init(_database) -> None:
|
||||
events.append("engine_init")
|
||||
raise RuntimeError("schema bootstrap failed")
|
||||
|
||||
async def close_engine() -> None:
|
||||
events.append("engine_close")
|
||||
|
||||
monkeypatch.setattr("deerflow.runtime.make_stream_bridge", lambda _config: _resource(object()))
|
||||
monkeypatch.setattr("deerflow.persistence.engine.init_engine_from_config", fail_engine_init)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.close_engine", close_engine)
|
||||
|
||||
with pytest.raises(RuntimeError, match="schema bootstrap failed"):
|
||||
async with langgraph_runtime(
|
||||
FastAPI(),
|
||||
SimpleNamespace(database=_database_config()),
|
||||
):
|
||||
pytest.fail("runtime must not yield")
|
||||
|
||||
assert events == ["engine_init", "engine_close"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_later_startup_failure_stops_same_snapshot_and_appends_diagnostics(monkeypatch):
|
||||
import deerflow.extensions as extensions_module
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
class _Service:
|
||||
def __init__(self, name: str, *, fail_start: bool = False, fail_stop: bool = False) -> None:
|
||||
self.name = name
|
||||
self.fail_start = fail_start
|
||||
self.fail_stop = fail_stop
|
||||
|
||||
async def start(self, _deps) -> None:
|
||||
events.append(f"start:{self.name}")
|
||||
if self.fail_start:
|
||||
raise RuntimeError("start exploded")
|
||||
|
||||
async def stop(self) -> None:
|
||||
events.append(f"stop:{self.name}")
|
||||
if self.fail_stop:
|
||||
raise RuntimeError("stop exploded")
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("bad-start:install"):
|
||||
registry.service(_Service("bad-start", fail_start=True))
|
||||
with registry.attributed_to("bad-stop:install"):
|
||||
registry.service(_Service("bad-stop", fail_stop=True))
|
||||
snapshot = registry.build()
|
||||
|
||||
other_registry = ExtensionRegistry()
|
||||
with other_registry.attributed_to("other:install"):
|
||||
other_registry.service(_Service("other"))
|
||||
monkeypatch.setattr(extensions_module, "_loaded", other_registry.build())
|
||||
|
||||
_patch_runtime_resources(monkeypatch, events)
|
||||
monkeypatch.setattr(
|
||||
"deerflow.persistence.thread_meta.make_thread_store",
|
||||
lambda _sf, _store: (_ for _ in ()).throw(RuntimeError("thread store failed")),
|
||||
)
|
||||
|
||||
app = FastAPI()
|
||||
app.state.extensions = snapshot
|
||||
live_diagnostics = initialize_runtime_diagnostics([])
|
||||
app.state.extension_diagnostics = live_diagnostics
|
||||
|
||||
with pytest.raises(RuntimeError, match="thread store failed"):
|
||||
async with langgraph_runtime(
|
||||
app,
|
||||
SimpleNamespace(database=_database_config()),
|
||||
):
|
||||
pytest.fail("runtime must not yield")
|
||||
|
||||
assert events == [
|
||||
"start:bad-start",
|
||||
"start:bad-stop",
|
||||
"stop:bad-stop",
|
||||
"stop:bad-start",
|
||||
"engine_close",
|
||||
]
|
||||
assert all("other" not in event for event in events)
|
||||
assert app.state.extension_diagnostics is live_diagnostics
|
||||
assert app.state.extension_diagnostics == get_runtime_diagnostics()
|
||||
assert [diagnostic.source for diagnostic in live_diagnostics] == [
|
||||
"bad-start:install",
|
||||
"bad-stop:install",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancellation_during_service_start_propagates_after_cleanup(monkeypatch):
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
|
||||
events: list[str] = []
|
||||
blocking_start_entered = asyncio.Event()
|
||||
|
||||
class _Service:
|
||||
def __init__(self, name: str, *, block: bool = False) -> None:
|
||||
self.name = name
|
||||
self.block = block
|
||||
|
||||
async def start(self, _deps) -> None:
|
||||
events.append(f"start:{self.name}")
|
||||
if self.block:
|
||||
blocking_start_entered.set()
|
||||
await asyncio.Event().wait()
|
||||
|
||||
async def stop(self) -> None:
|
||||
events.append(f"stop:{self.name}")
|
||||
|
||||
registry = ExtensionRegistry()
|
||||
with registry.attributed_to("first:install"):
|
||||
registry.service(_Service("first"))
|
||||
with registry.attributed_to("blocking:install"):
|
||||
registry.service(_Service("blocking", block=True))
|
||||
with registry.attributed_to("never-started:install"):
|
||||
registry.service(_Service("never-started"))
|
||||
|
||||
_patch_runtime_resources(monkeypatch, events)
|
||||
app = FastAPI()
|
||||
app.state.extensions = registry.build()
|
||||
|
||||
async def run_runtime() -> None:
|
||||
async with langgraph_runtime(
|
||||
app,
|
||||
SimpleNamespace(database=_database_config()),
|
||||
):
|
||||
pytest.fail("runtime must not yield while service start is blocked")
|
||||
|
||||
task = asyncio.create_task(run_runtime())
|
||||
await asyncio.wait_for(blocking_start_entered.wait(), timeout=1.0)
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
assert events == [
|
||||
"start:first",
|
||||
"start:blocking",
|
||||
"stop:blocking",
|
||||
"stop:first",
|
||||
"engine_close",
|
||||
]
|
||||
180
backend/tests/test_gateway_request_path.py
Normal file
180
backend/tests/test_gateway_request_path.py
Normal file
@ -0,0 +1,180 @@
|
||||
"""Conformance tests for the shared request-path projection.
|
||||
|
||||
``app.gateway.request_path.get_request_route_path()`` exists to answer one
|
||||
question: *which path string is Starlette's router matching right now?* Auth
|
||||
and CSRF classify that value, so the security predicates and the dispatcher
|
||||
must agree on it exactly. When they disagree, a route mounted under a
|
||||
public-looking prefix can be classified public while the router dispatches to
|
||||
a protected handler.
|
||||
|
||||
These tests pin the agreement itself rather than the mechanism that produces
|
||||
it, so they stay meaningful whether the projection keeps delegating to
|
||||
Starlette or is ever reimplemented. Nothing here asserts *how* the value is
|
||||
computed -- only that middleware and router see the same string.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from starlette.requests import Request
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from app.gateway.auth_middleware import AuthMiddleware
|
||||
from app.gateway.csrf_middleware import CSRFMiddleware, is_auth_endpoint, should_check_csrf
|
||||
from app.gateway.request_path import get_request_route_path
|
||||
from deerflow.config.authorization_config import AuthorizationConfig
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _default_route_authorization_config(monkeypatch):
|
||||
"""Keep minimal middleware apps independent of a repository config.yaml."""
|
||||
monkeypatch.setattr(
|
||||
"app.gateway.authz._get_route_authorization_config",
|
||||
lambda: AuthorizationConfig(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _auth_enabled(monkeypatch):
|
||||
"""Every case here is about the enabled-auth path."""
|
||||
monkeypatch.delenv("DEER_FLOW_AUTH_DISABLED", raising=False)
|
||||
|
||||
|
||||
def _request(path: str, root_path: str = "", method: str = "GET") -> Request:
|
||||
return Request(
|
||||
{
|
||||
"type": "http",
|
||||
"method": method,
|
||||
"path": path,
|
||||
"root_path": root_path,
|
||||
"query_string": b"",
|
||||
"headers": [],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ── Projection edge cases ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "root_path", "expected"),
|
||||
[
|
||||
# No mount: the raw ASGI path is already what the router matches.
|
||||
("/api/models", "", "/api/models"),
|
||||
# Mounted: root_path is stripped on the segment boundary.
|
||||
("/prefix/api/models", "/prefix", "/api/models"),
|
||||
# Nested mounts accumulate into root_path; strip all of it.
|
||||
("/outer/inner/health", "/outer/inner", "/health"),
|
||||
# The mount itself was requested; the router matches the empty path.
|
||||
("/prefix", "/prefix", ""),
|
||||
# root_path is not a prefix at all -- never strip.
|
||||
("/other/models", "/prefix", "/other/models"),
|
||||
# Prefix collides mid-segment. Stripping here would produce "foo",
|
||||
# a string the router would never match.
|
||||
("/apifoo/models", "/api", "/apifoo/models"),
|
||||
("/apifoo", "/api", "/apifoo"),
|
||||
],
|
||||
)
|
||||
def test_projection_strips_root_path_only_on_segment_boundaries(path: str, root_path: str, expected: str):
|
||||
assert get_request_route_path(_request(path, root_path)) == expected
|
||||
|
||||
|
||||
# ── Agreement with the router ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_projection_matches_the_path_the_router_dispatched_on():
|
||||
"""The middleware-visible string equals the route's declared path."""
|
||||
seen: dict[str, str] = {}
|
||||
|
||||
child = FastAPI()
|
||||
|
||||
@child.get("/health")
|
||||
async def health(request: Request):
|
||||
seen["projection"] = get_request_route_path(request)
|
||||
return {"ok": True}
|
||||
|
||||
middle = FastAPI()
|
||||
middle.mount("/inner", child)
|
||||
parent = FastAPI()
|
||||
parent.mount("/outer", middle)
|
||||
|
||||
assert TestClient(parent).get("/outer/inner/health").status_code == 200
|
||||
# The router matched the declared "/health", not the wire path.
|
||||
assert seen["projection"] == "/health"
|
||||
|
||||
|
||||
def test_public_route_stays_public_under_nested_mounts():
|
||||
"""Availability direction: a mounted /health must not start 401-ing."""
|
||||
child = FastAPI()
|
||||
child.add_middleware(AuthMiddleware)
|
||||
|
||||
@child.get("/health")
|
||||
async def health():
|
||||
return {"ok": True}
|
||||
|
||||
middle = FastAPI()
|
||||
middle.mount("/inner", child)
|
||||
parent = FastAPI()
|
||||
parent.mount("/outer", middle)
|
||||
|
||||
assert TestClient(parent).get("/outer/inner/health").status_code == 200
|
||||
|
||||
|
||||
# ── The bypass these predicates exist to prevent ─────────────────────────────
|
||||
|
||||
|
||||
def test_mounting_under_a_public_prefix_does_not_expose_protected_routes():
|
||||
"""Security direction: the mount prefix must not leak into classification.
|
||||
|
||||
Classifying the raw wire path would see "/health/api/models", match the
|
||||
"/health" public prefix, and skip authentication entirely -- while the
|
||||
router dispatches to the protected "/api/models" handler.
|
||||
"""
|
||||
child = FastAPI()
|
||||
child.add_middleware(AuthMiddleware)
|
||||
|
||||
@child.get("/api/models")
|
||||
async def models():
|
||||
return {"models": []}
|
||||
|
||||
parent = FastAPI()
|
||||
parent.mount("/health", child)
|
||||
|
||||
assert TestClient(parent).get("/health/api/models").status_code == 401
|
||||
|
||||
|
||||
def test_csrf_is_enforced_for_routes_mounted_under_the_webhook_prefix():
|
||||
"""CSRF's webhook exemption keys off the projection, not the wire path."""
|
||||
child = FastAPI()
|
||||
child.add_middleware(CSRFMiddleware)
|
||||
|
||||
@child.post("/action")
|
||||
async def action():
|
||||
return {"ok": True}
|
||||
|
||||
parent = FastAPI()
|
||||
parent.mount("/api/webhooks", child)
|
||||
|
||||
response = TestClient(parent).post("/api/webhooks/action")
|
||||
|
||||
assert response.status_code == 403
|
||||
assert "CSRF token missing" in response.json()["detail"]
|
||||
|
||||
|
||||
# ── CSRF predicates read the same projection ─────────────────────────────────
|
||||
|
||||
|
||||
def test_csrf_exemptions_follow_the_projection():
|
||||
# Genuine host webhook: no mount, exempt.
|
||||
assert should_check_csrf(_request("/api/webhooks/github", method="POST")) is False
|
||||
# Same wire path, but the router is matching "/github" inside a mount --
|
||||
# not the host's webhook namespace, so CSRF still applies.
|
||||
assert should_check_csrf(_request("/api/webhooks/github", "/api/webhooks", method="POST")) is True
|
||||
|
||||
|
||||
def test_auth_endpoint_detection_follows_the_projection():
|
||||
assert is_auth_endpoint(_request("/api/v1/auth/login/local", method="POST")) is True
|
||||
# A mount whose prefix is itself the auth namespace: the router matches
|
||||
# "/local", which is not the host's exempt endpoint.
|
||||
assert is_auth_endpoint(_request("/api/v1/auth/login/local", "/api/v1/auth/login", method="POST")) is False
|
||||
# The mount point itself was requested; the router matches the empty path.
|
||||
assert is_auth_endpoint(_request("/api/v1/auth/login/local", "/api/v1/auth/login/local", method="POST")) is False
|
||||
@ -148,7 +148,7 @@ async def test_shutdown_is_noop_without_inflight_runs():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeypatch):
|
||||
"""The wiring order lock for #3373: drain in-flight runs, THEN close the pool.
|
||||
"""Drain runs before services, then close runtime resources in stack order.
|
||||
|
||||
Patches every ``langgraph_runtime`` collaborator down to trivial stand-ins so
|
||||
only the bootstrap/teardown ordering runs. The checkpointer probe records when
|
||||
@ -158,6 +158,7 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from deerflow.extensions.registry import ExtensionRegistry
|
||||
|
||||
events: list[str] = []
|
||||
|
||||
@ -170,17 +171,27 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_stream_bridge(_config):
|
||||
yield object()
|
||||
try:
|
||||
yield object()
|
||||
finally:
|
||||
events.append("stream_bridge_closed")
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_store(_config):
|
||||
yield object()
|
||||
try:
|
||||
yield object()
|
||||
finally:
|
||||
events.append("store_closed")
|
||||
|
||||
async def fake_init_engine(_db):
|
||||
events.append("engine_initialized")
|
||||
|
||||
def fake_session_factory():
|
||||
events.append("session_factory_resolved")
|
||||
return None
|
||||
|
||||
async def fake_close_engine():
|
||||
return None
|
||||
events.append("engine_closed")
|
||||
|
||||
async def spy_shutdown(self, *, timeout): # noqa: ANN001
|
||||
events.append("runs_drained")
|
||||
@ -197,7 +208,7 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
|
||||
monkeypatch.setattr("deerflow.runtime.make_store", fake_store)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.init_engine_from_config", fake_init_engine)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.close_engine", fake_close_engine)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", lambda: None)
|
||||
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", fake_session_factory)
|
||||
monkeypatch.setattr("deerflow.runtime.events.store.make_run_event_store", lambda _cfg: object())
|
||||
monkeypatch.setattr("deerflow.persistence.thread_meta.make_thread_store", lambda _sf, _store: object())
|
||||
monkeypatch.setattr(RunManager, "shutdown", spy_shutdown, raising=False)
|
||||
@ -205,16 +216,36 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp
|
||||
monkeypatch.setattr("deerflow.extensions.notify.reset_extension_notify_loop", spy_reset_extension_notify_loop)
|
||||
|
||||
app = FastAPI()
|
||||
registry = ExtensionRegistry()
|
||||
|
||||
class _Service:
|
||||
async def start(self, _deps):
|
||||
events.append("service_started")
|
||||
|
||||
async def stop(self):
|
||||
events.append("service_stopped")
|
||||
|
||||
with registry.attributed_to("service:install"):
|
||||
registry.service(_Service())
|
||||
app.state.extensions = registry.build()
|
||||
startup_config = SimpleNamespace(database=SimpleNamespace(backend="memory", checkpoint_channel_mode="full", checkpoint_delta=SimpleNamespace(snapshot_frequency=10)), run_events=None)
|
||||
|
||||
async with langgraph_runtime(app, startup_config):
|
||||
pass
|
||||
|
||||
assert "runs_drained" in events, "langgraph_runtime never drained in-flight runs on shutdown"
|
||||
assert "service_started" in events
|
||||
assert "service_stopped" in events
|
||||
assert "checkpointer_closed" in events
|
||||
assert events.index("runs_drained") < events.index("checkpointer_closed"), f"runs must be drained before the checkpointer pool is closed; got order {events}"
|
||||
assert events.index("engine_initialized") < events.index("session_factory_resolved")
|
||||
assert events.index("session_factory_resolved") < events.index("service_started")
|
||||
assert events.index("runs_drained") < events.index("service_stopped")
|
||||
assert events.index("service_stopped") < events.index("store_closed")
|
||||
assert events.index("store_closed") < events.index("checkpointer_closed")
|
||||
assert events.index("checkpointer_closed") < events.index("engine_closed")
|
||||
assert events.index("engine_closed") < events.index("stream_bridge_closed")
|
||||
assert events[0] == "extension_loop_set"
|
||||
assert events.index("checkpointer_closed") < events.index("extension_loop_reset"), f"extension loop reset must be the final runtime teardown; got order {events}"
|
||||
assert events.index("stream_bridge_closed") < events.index("extension_loop_reset"), f"extension loop reset must be the final runtime teardown; got order {events}"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@ -105,7 +105,10 @@ def test_backend_make_dev_gateway_reload_excludes_runtime_state_with_absolute_di
|
||||
assert "DEER_FLOW_HOME := $(abspath $(DEER_FLOW_HOME))" in makefile
|
||||
assert "BACKEND_SANDBOX_HOME := $(abspath $(CURDIR)/sandbox)" in makefile
|
||||
assert 'mkdir -p "$(DEER_FLOW_HOME)" "$(BACKEND_SANDBOX_HOME)"' in makefile
|
||||
assert 'DEER_FLOW_HOME="$(DEER_FLOW_HOME)" uv run uvicorn' in makefile
|
||||
# The launch line may carry runtime-only uv flags (`--locked` pins the
|
||||
# extension lock); what this guards is that DEER_FLOW_HOME is exported on it,
|
||||
# so the reload-excludes below resolve to the same absolute directories.
|
||||
assert re.search(r'DEER_FLOW_HOME="\$\(DEER_FLOW_HOME\)" uv run(?: --(?:locked|no-sync))? uvicorn', makefile)
|
||||
assert '--reload-exclude="$(DEER_FLOW_HOME)"' in makefile
|
||||
assert '--reload-exclude="$(BACKEND_SANDBOX_HOME)"' in makefile
|
||||
|
||||
|
||||
@ -2,13 +2,17 @@
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.tui.cli import LaunchPlan, plan_launch
|
||||
from deerflow.tui.cli import LaunchPlan, build_parser, plan_launch
|
||||
|
||||
|
||||
def plan(argv, *, stdin_tty=True, stdout_tty=True, env=None):
|
||||
return plan_launch(argv, stdin_isatty=stdin_tty, stdout_isatty=stdout_tty, env=env or {})
|
||||
|
||||
|
||||
def test_top_level_help_points_to_extension_management():
|
||||
assert "deerflow extensions --help" in build_parser().format_help()
|
||||
|
||||
|
||||
def test_bare_command_on_tty_launches_tui():
|
||||
p = plan([])
|
||||
assert p.mode == "tui"
|
||||
|
||||
@ -157,14 +157,15 @@ def test_sandbox_mkdir_precedes_uvicorn_launch(name):
|
||||
|
||||
``_mkdir_dirs`` only proves the mkdir is present somewhere; this pins script
|
||||
order so a future edit can't move (or guard) the mkdir below the launch and
|
||||
silently reintroduce the #3454 crash on a fresh checkout. ``uv run uvicorn``
|
||||
matches the launch but not serve.sh's ``stop_all`` kill line.
|
||||
silently reintroduce the #3454 crash on a fresh checkout. The ``uv run``
|
||||
matcher allows runtime-only flags while still excluding serve.sh's
|
||||
``stop_all`` kill line.
|
||||
"""
|
||||
lines = LAUNCHERS[name].read_text(encoding="utf-8").splitlines()
|
||||
launch_idx = next((i for i, ln in enumerate(lines) if "uv run uvicorn" in ln), None)
|
||||
launch_idx = next((i for i, ln in enumerate(lines) if re.search(r"\buv run(?: --(?:no-sync|locked))? uvicorn\b", ln)), None)
|
||||
mkdir_idx = next((i for i, ln in enumerate(lines) if re.search(r"\bmkdir\b", ln) and "sandbox" in ln.lower()), None)
|
||||
|
||||
assert launch_idx is not None, f"{name}: could not locate the 'uv run uvicorn' launch line"
|
||||
assert launch_idx is not None, f"{name}: could not locate the uvicorn launch line"
|
||||
assert mkdir_idx is not None, f"{name}: could not locate the sandbox mkdir line"
|
||||
assert mkdir_idx < launch_idx, f"{name}: sandbox mkdir (line {mkdir_idx + 1}) must precede uvicorn launch (line {launch_idx + 1})"
|
||||
|
||||
|
||||
11
backend/uv.lock
generated
11
backend/uv.lock
generated
@ -821,6 +821,7 @@ dependencies = [
|
||||
{ name = "python-telegram-bot" },
|
||||
{ name = "slack-sdk" },
|
||||
{ name = "sse-starlette" },
|
||||
{ name = "starlette" },
|
||||
{ name = "uvicorn", extra = ["standard"] },
|
||||
{ name = "wecom-aibot-python-sdk" },
|
||||
]
|
||||
@ -886,6 +887,7 @@ requires-dist = [
|
||||
{ name = "python-telegram-bot", specifier = ">=21.0" },
|
||||
{ name = "slack-sdk", specifier = ">=3.33.0" },
|
||||
{ name = "sse-starlette", specifier = ">=2.1.0" },
|
||||
{ name = "starlette", specifier = ">=1.3.1,<2" },
|
||||
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.34.0" },
|
||||
{ name = "wecom-aibot-python-sdk", specifier = ">=0.1.6" },
|
||||
]
|
||||
@ -904,10 +906,11 @@ dev = [
|
||||
{ name = "ruff", specifier = ">=0.14.11" },
|
||||
{ name = "textual", specifier = ">=0.80" },
|
||||
]
|
||||
extensions = []
|
||||
|
||||
[[package]]
|
||||
name = "deerflow-extension-api"
|
||||
version = "0.1.1"
|
||||
version = "0.1.2"
|
||||
source = { editable = "packages/extension-api" }
|
||||
|
||||
[[package]]
|
||||
@ -946,6 +949,7 @@ dependencies = [
|
||||
{ name = "langgraph-sdk" },
|
||||
{ name = "markdownify" },
|
||||
{ name = "markitdown", extra = ["all", "xlsx"] },
|
||||
{ name = "packaging" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "pyyaml" },
|
||||
{ name = "readabilipy" },
|
||||
@ -1028,6 +1032,7 @@ requires-dist = [
|
||||
{ name = "markdownify", specifier = ">=1.2.2" },
|
||||
{ name = "markitdown", extras = ["all", "xlsx"], specifier = ">=0.0.1a2" },
|
||||
{ name = "monocle-apptrace", marker = "extra == 'monocle'", specifier = ">=0.8.8" },
|
||||
{ name = "packaging", specifier = ">=24.2" },
|
||||
{ name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" },
|
||||
{ name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.3.3" },
|
||||
{ name = "psycopg-pool", marker = "extra == 'postgres'", specifier = ">=3.3.0" },
|
||||
@ -4514,8 +4519,8 @@ name = "standard-aifc"
|
||||
version = "3.13.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "audioop-lts" },
|
||||
{ name = "standard-chunk" },
|
||||
{ name = "audioop-lts", marker = "python_full_version >= '3.13'" },
|
||||
{ name = "standard-chunk", marker = "python_full_version >= '3.13'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c4/53/6050dc3dde1671eb3db592c13b55a8005e5040131f7509cef0215212cb84/standard_aifc-3.13.0.tar.gz", hash = "sha256:64e249c7cb4b3daf2fdba4e95721f811bde8bdfc43ad9f936589b7bb2fae2e43", size = 15240, upload-time = "2024-10-30T16:01:31.772Z" }
|
||||
wheels = [
|
||||
|
||||
@ -41,7 +41,8 @@ logging:
|
||||
# lead-only vs subagent-only configuration is not expressible yet. Treat these
|
||||
# files as trusted operator config because middleware classes execute code.
|
||||
# Uncomment this block to define middlewares in config.yaml. Leaving it commented
|
||||
# lets extensions_config.json remain the source of truth for extension packages.
|
||||
# lets extensions_config.json remain the source of truth for this legacy
|
||||
# config-declared middleware list. Packaged plugins use the `plugins:` block below.
|
||||
# extensions:
|
||||
# middlewares:
|
||||
# - my_company.deerflow_middlewares:DomainGuardMiddleware
|
||||
@ -2531,8 +2532,27 @@ authorization:
|
||||
# entry names an install entry point as "module.path:install"; the `config`
|
||||
# block is handed to that package verbatim and validated by the package itself.
|
||||
#
|
||||
# Loading is explicit on purpose: an installed package does nothing until it is
|
||||
# listed here, and the list order is what fixes middleware ordering.
|
||||
# Prefer the extension manager over editing this list by hand:
|
||||
# make extension-install SOURCE="deerflow-extension-acme==1.2.3"
|
||||
# make extension-install SOURCE="git+https://github.com/acme/deerflow-extension-acme.git@<commit>"
|
||||
# make extension-install SOURCE="/absolute/path/to/local-extension"
|
||||
# make extension-list
|
||||
# make extension-enable NAME=acme
|
||||
# make extension-disable NAME=acme
|
||||
# make extension-remove NAME=acme
|
||||
# The installer updates backend/pyproject.toml, backend/uv.lock, and this block
|
||||
# together. It also asks the operator to confirm that the source is trusted;
|
||||
# extensions and their build systems execute with Gateway privileges.
|
||||
#
|
||||
# Every install, enable, disable, remove, or manual edit requires a Gateway
|
||||
# restart. Development launchers may fetch missing locked artifacts before
|
||||
# handoff; a built production Gateway never installs plugins during startup.
|
||||
# Local directories are copied to backend/extensions/sources as deployable
|
||||
# snapshots; use a pinned version or public HTTPS Git commit for reproducible
|
||||
# remote sources. SSH Git URLs are rejected because the stock Docker builder
|
||||
# does not forward host SSH credentials.
|
||||
# Loading remains explicit: a package installed outside the manager does nothing
|
||||
# until it is listed here, and list order fixes contribution ordering.
|
||||
#
|
||||
# This is deliberately separate from the `extensions:` block above (MCP servers,
|
||||
# skills, config-declared middlewares). That one is backed by
|
||||
@ -2540,20 +2560,16 @@ authorization:
|
||||
# endpoint; a list that causes code to be imported must stay in this
|
||||
# operator-controlled file only.
|
||||
#
|
||||
# This initial extension-system slice accepts middleware contributors. Their
|
||||
# install functions register semantically placed middlewares for the lead and/or
|
||||
# subagent stack; the examples below intentionally cover only that surface.
|
||||
# Plugins can contribute semantically placed middleware, task-lifecycle hooks,
|
||||
# system-model observers, Gateway-lifetime services, and eager FastAPI HTTP routers.
|
||||
# See examples/deerflow-extension-example for a package that exercises all five.
|
||||
#
|
||||
# plugins:
|
||||
# - use: acme_request_observer:install
|
||||
# - name: example # PEP 621 entry-point name
|
||||
# package: deerflow-extension-example # Python distribution managed by uv
|
||||
# use: deerflow_extension_example:install
|
||||
# enabled: true # false skips import and registration
|
||||
# required: false # true makes load failure abort Gateway startup
|
||||
# # (install --required opts in)
|
||||
# config:
|
||||
# enabled: true
|
||||
# placement: model_logical
|
||||
# - use: acme_tool_observer:install
|
||||
# config:
|
||||
# enabled: true
|
||||
# placement: tool_raw
|
||||
# # `required: true` turns a load failure into a startup failure. Use it for
|
||||
# # packages whose absence changes behaviour rather than just observability.
|
||||
# - use: acme_required_middleware:install
|
||||
# required: true
|
||||
# label: example # extension-private values, if any
|
||||
|
||||
@ -5,13 +5,17 @@
|
||||
# (PR #2767, addressing review on Issue #2754).
|
||||
#
|
||||
# Responsibilities:
|
||||
# 1. Resolve `--extra X` flags from UV_EXTRAS (comma- or whitespace-separated,
|
||||
# mirroring scripts/detect_uv_extras.py for parity with local `make dev`).
|
||||
# 1. Resolve `--extra X` flags through scripts/detect_uv_extras.py, using the
|
||||
# selected config plus explicit UV_EXTRAS and runtime-required backends.
|
||||
# 2. Validate each extra against [A-Za-z][A-Za-z0-9_-]* so a stray shell
|
||||
# metacharacter in `.env` cannot reach `uv sync`.
|
||||
# 3. `uv sync --all-packages` so workspace member extras (deerflow-harness's
|
||||
# 3. `uv sync --locked --all-packages` so the declared extension group and
|
||||
# workspace member extras (deerflow-harness's
|
||||
# postgres extra in particular) are installed — see PR #2584.
|
||||
# 4. Self-heal: if the first sync fails, recreate .venv and retry once.
|
||||
# 4. Self-heal: if the first sync fails, recreate .venv and retry once. The
|
||||
# retry stays `--locked`, so it repairs a broken .venv but not a stale
|
||||
# lock; a second failure aborts with recovery instructions rather than
|
||||
# starting uvicorn against an environment that does not match the lock.
|
||||
# 5. Hand off to uvicorn with reload, replacing this shell so uvicorn becomes
|
||||
# PID 1 inside the container.
|
||||
#
|
||||
@ -38,22 +42,78 @@ fi
|
||||
# ── Resolve extras ──────────────────────────────────────────────────────────
|
||||
|
||||
EXTRAS_FLAGS=""
|
||||
EXTRA_NAMES=""
|
||||
set -f
|
||||
|
||||
append_extra() {
|
||||
extra_name="$1"
|
||||
case "$extra_name" in
|
||||
[!A-Za-z]* | *[!A-Za-z0-9_-]*)
|
||||
echo "[startup] UV_EXTRAS entry '$extra_name' is invalid (must match [A-Za-z][A-Za-z0-9_-]*) — aborting" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
case " $EXTRA_NAMES " in
|
||||
*" $extra_name "*) return ;;
|
||||
esac
|
||||
EXTRA_NAMES="$EXTRA_NAMES $extra_name"
|
||||
EXTRAS_FLAGS="$EXTRAS_FLAGS --extra $extra_name"
|
||||
}
|
||||
|
||||
# Validate explicit input before the detector normalizes it. The shared
|
||||
# detector deliberately drops invalid names with a warning, while container
|
||||
# startup fails closed so malformed .env input cannot be silently ignored.
|
||||
if [ -n "${UV_EXTRAS:-}" ]; then
|
||||
# Normalize comma → space, then split on whitespace via the unquoted `for`.
|
||||
for raw in $(printf '%s' "$UV_EXTRAS" | tr ',' ' '); do
|
||||
[ -z "$raw" ] && continue
|
||||
# Reject anything that does not look like an identifier.
|
||||
# Two patterns: leading non-letter, or any non-[A-Za-z0-9_-] character.
|
||||
case "$raw" in
|
||||
[!A-Za-z]* | *[!A-Za-z0-9_-]*)
|
||||
echo "[startup] UV_EXTRAS entry '$raw' is invalid (must match [A-Za-z][A-Za-z0-9_-]*) — aborting" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
EXTRAS_FLAGS="$EXTRAS_FLAGS --extra $raw"
|
||||
[ -z "$raw" ] || append_extra "$raw"
|
||||
done
|
||||
fi
|
||||
|
||||
# Docker dev mounts the host checkout at /app/project while
|
||||
# DEER_FLOW_PROJECT_ROOT points at /app for runtime path translation. Prefer
|
||||
# both locations, then the checkout-relative path used by direct invocations.
|
||||
ENTRYPOINT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
DETECTOR_PATH=""
|
||||
for candidate in \
|
||||
"${DEER_FLOW_PROJECT_ROOT:+$DEER_FLOW_PROJECT_ROOT/scripts/detect_uv_extras.py}" \
|
||||
/app/project/scripts/detect_uv_extras.py \
|
||||
"$ENTRYPOINT_DIR/../scripts/detect_uv_extras.py"
|
||||
do
|
||||
if [ -n "$candidate" ] && [ -f "$candidate" ]; then
|
||||
DETECTOR_PATH="$candidate"
|
||||
break
|
||||
fi
|
||||
done
|
||||
if [ -z "$DETECTOR_PATH" ]; then
|
||||
echo "[startup] scripts/detect_uv_extras.py is unavailable" >&2
|
||||
exit 1
|
||||
fi
|
||||
if command -v python3 >/dev/null 2>&1; then
|
||||
DETECTOR_PYTHON=python3
|
||||
elif command -v python >/dev/null 2>&1; then
|
||||
DETECTOR_PYTHON=python
|
||||
else
|
||||
echo "[startup] Python is required to resolve optional dependencies" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! DETECTED_FLAGS=$("$DETECTOR_PYTHON" "$DETECTOR_PATH"); then
|
||||
echo "[startup] detect_uv_extras.py failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The detector emits only validated `--extra NAME` pairs. Parse that small
|
||||
# interface instead of evaluating shell text, and validate again at the final
|
||||
# shell boundary before any value can reach uv.
|
||||
set -- $DETECTED_FLAGS
|
||||
while [ "$#" -gt 0 ]; do
|
||||
if [ "$1" != "--extra" ] || [ "$#" -lt 2 ]; then
|
||||
echo "[startup] detect_uv_extras.py returned invalid output" >&2
|
||||
exit 1
|
||||
fi
|
||||
append_extra "$2"
|
||||
shift 2
|
||||
done
|
||||
|
||||
if [ "$PRINT_EXTRAS_ONLY" = "1" ]; then
|
||||
# Trim leading space for tidier output, then exit.
|
||||
printf '%s\n' "${EXTRAS_FLAGS# }"
|
||||
@ -78,22 +138,31 @@ mkdir -p "$DEER_FLOW_HOME" /app/backend/.deer-flow /app/backend/sandbox
|
||||
cd /app/backend
|
||||
|
||||
# `--all-packages` propagates extras into workspace members (PR #2584).
|
||||
# `--extra redis` is always installed because docker-compose-dev defaults the
|
||||
# stream bridge to Redis (DEER_FLOW_STREAM_BRIDGE_REDIS_URL); redis is an
|
||||
# optional extra elsewhere. It is kept out of EXTRAS_FLAGS so the --print-extras
|
||||
# contract (UV_EXTRAS-derived flags only) stays unchanged.
|
||||
# docker-compose-dev's default DEER_FLOW_STREAM_BRIDGE_REDIS_URL is translated
|
||||
# to `--extra redis` by the shared detector, alongside config and UV_EXTRAS.
|
||||
# `$EXTRAS_FLAGS` intentionally unquoted so each `--extra X` becomes its own arg.
|
||||
# shellcheck disable=SC2086 # word-splitting is intentional here
|
||||
if ! uv sync --all-packages --extra redis $EXTRAS_FLAGS; then
|
||||
if ! uv sync --locked --all-packages $EXTRAS_FLAGS; then
|
||||
echo "[startup] uv sync failed; recreating .venv and retrying once"
|
||||
uv venv --allow-existing .venv
|
||||
uv venv --clear .venv
|
||||
# The retry keeps `--locked` on purpose: it repairs a corrupt or partial
|
||||
# .venv, not a lock that disagrees with pyproject.toml. Startup must never
|
||||
# silently resolve dependencies, so a second failure is fatal rather than
|
||||
# something uvicorn limps past and reports later as an import error.
|
||||
# `set -e` would already stop here; abort explicitly so the operator gets
|
||||
# the fix instead of a bare uv exit code.
|
||||
# shellcheck disable=SC2086
|
||||
uv sync --all-packages --extra redis $EXTRAS_FLAGS
|
||||
if ! uv sync --locked --all-packages $EXTRAS_FLAGS; then
|
||||
echo "[startup] uv sync --locked failed again after recreating .venv." >&2
|
||||
echo "[startup] backend/uv.lock does not match backend/pyproject.toml, or a locked artifact is unreachable." >&2
|
||||
echo "[startup] Run 'make install' on the host to refresh the lock, then restart this container." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Hand off to uvicorn ─────────────────────────────────────────────────────
|
||||
|
||||
PYTHONPATH=. exec uv run uvicorn app.gateway.app:app \
|
||||
PYTHONPATH=. exec uv run --no-sync uvicorn app.gateway.app:app \
|
||||
--host 0.0.0.0 --port 8001 \
|
||||
--reload \
|
||||
--reload-include='*.yaml' \
|
||||
|
||||
@ -153,7 +153,7 @@ services:
|
||||
# cache_from disabled - requires manual setup: mkdir -p /tmp/docker-cache-gateway
|
||||
args:
|
||||
APT_MIRROR: ${APT_MIRROR:-}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.11.1}
|
||||
UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple}
|
||||
NPM_REGISTRY: ${NPM_REGISTRY:-}
|
||||
LARK_CLI_NPM_VERSION: ${LARK_CLI_NPM_VERSION:-1.0.65}
|
||||
|
||||
@ -90,7 +90,7 @@ services:
|
||||
dockerfile: backend/Dockerfile
|
||||
args:
|
||||
APT_MIRROR: ${APT_MIRROR:-}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.11.1}
|
||||
UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple}
|
||||
UV_EXTRAS: ${UV_EXTRAS:-}
|
||||
NPM_REGISTRY: ${NPM_REGISTRY:-}
|
||||
@ -102,7 +102,7 @@ services:
|
||||
# reconnect, but run cancel, request dedup, and per-worker IM channel
|
||||
# services remain worker-local. Override GATEWAY_WORKERS only when the
|
||||
# Redis stream bridge is enabled and those limitations are acceptable.
|
||||
command: sh -c "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1}"
|
||||
command: sh -c "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1}"
|
||||
volumes:
|
||||
- ${DEER_FLOW_CONFIG_PATH}:/app/backend/config.yaml:ro
|
||||
- ${DEER_FLOW_EXTENSIONS_CONFIG_PATH}:/app/backend/extensions_config.json:ro
|
||||
|
||||
149
examples/deerflow-extension-example/README.md
Normal file
149
examples/deerflow-extension-example/README.md
Normal file
@ -0,0 +1,149 @@
|
||||
# DeerFlow extension example
|
||||
|
||||
This directory is a compact, standalone Python package showing all five DeerFlow
|
||||
extension contribution kinds. It depends on the public
|
||||
`deerflow-extension-api` contract and never imports `deerflow.*` or `app.*`.
|
||||
|
||||
The contract package intentionally has no framework dependencies. An extension
|
||||
must therefore declare every framework it imports itself; this example explicitly
|
||||
depends on FastAPI, LangChain, and LangGraph in `pyproject.toml`.
|
||||
|
||||
## What it demonstrates
|
||||
|
||||
| Contribution | Example behavior |
|
||||
| --- | --- |
|
||||
| Middleware | Counts tool calls through one `TOOL_VISIBLE` middleware for lead agents and subagents |
|
||||
| Task lifecycle | Creates task-scoped stats on start and folds them into app scope on stop |
|
||||
| System-model observer | Counts DeerFlow-owned model calls, including failures |
|
||||
| Service | Binds `ExtensionRuntimeDeps` only while the Gateway is running |
|
||||
| Router | Eagerly declares `GET /api/extension-example/stats` during `install()` |
|
||||
|
||||
The middleware reads task scope only through `task_store_from_runtime()`. It
|
||||
passes through unchanged when no task store exists. The router and service use
|
||||
the same `ExampleService` object: its FastAPI dependency returns `503` before
|
||||
`start()`, after `stop()`, or when no app store was bound. This keeps the route
|
||||
topology stable while runtime capabilities arrive later.
|
||||
|
||||
## Run the package tests
|
||||
|
||||
`deerflow-extension-api` is currently sourced from this checkout. Install it
|
||||
first, then install this independent package:
|
||||
|
||||
```bash
|
||||
cd examples/deerflow-extension-example
|
||||
uv venv --python 3.12
|
||||
uv pip install -e ../../backend/packages/extension-api
|
||||
uv pip install -e ".[dev]"
|
||||
uv run --no-project pytest -q
|
||||
uv run --no-project ruff check .
|
||||
uv run --no-project ruff format --check .
|
||||
```
|
||||
|
||||
The tests use only the public contract plus this package's declared dependencies;
|
||||
the DeerFlow harness and Gateway application are not imported.
|
||||
|
||||
## Install and load it in DeerFlow
|
||||
|
||||
From the DeerFlow checkout root, install this directory through the extension
|
||||
manager. Use an absolute path because the Make wrapper invokes the manager from
|
||||
`backend/`:
|
||||
|
||||
```bash
|
||||
make extension-install SOURCE="$PWD/examples/deerflow-extension-example"
|
||||
make extension-list
|
||||
```
|
||||
|
||||
After the trust prompt is accepted, the manager:
|
||||
|
||||
- copies a deployable snapshot to
|
||||
`backend/extensions/sources/deerflow-extension-example/`;
|
||||
- adds that snapshot to `backend/pyproject.toml`'s `extensions` dependency
|
||||
group and updates `backend/uv.lock`;
|
||||
- installs the locked environment; and
|
||||
- adds and enables this startup-only entry in the selected `config.yaml`.
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
- name: example
|
||||
package: deerflow-extension-example
|
||||
use: deerflow_extension_example:install
|
||||
enabled: true
|
||||
required: false
|
||||
config: {}
|
||||
```
|
||||
|
||||
Start or restart DeerFlow after installation:
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
The Gateway imports extensions only while constructing the application. Install,
|
||||
enable, disable, remove, and manual `plugins:` changes therefore take effect only
|
||||
after a restart. These commands manage the example afterward:
|
||||
|
||||
```bash
|
||||
make extension-disable NAME=example
|
||||
make extension-enable NAME=example
|
||||
make extension-remove NAME=example
|
||||
```
|
||||
|
||||
The manager can also install a PyPI requirement or a pinned public HTTPS Git
|
||||
URL. SSH Git URLs are rejected because the stock Docker builder does not
|
||||
forward host SSH credentials. The direct CLI surface, run from `backend/`, is:
|
||||
|
||||
```text
|
||||
uv run --frozen --no-group extensions deerflow extensions install <source> [--yes] [--required]
|
||||
uv run --frozen --no-group extensions deerflow extensions list
|
||||
uv run --frozen --no-group extensions deerflow extensions enable <name>
|
||||
uv run --frozen --no-group extensions deerflow extensions disable <name>
|
||||
uv run --frozen --no-group extensions deerflow extensions remove <name>
|
||||
```
|
||||
|
||||
`--yes` is intended only for automation that has already reviewed and trusted
|
||||
the source: extension build hooks and runtime code execute with Gateway
|
||||
privileges. `--required` records `required: true`, which turns any later load
|
||||
failure into a Gateway startup abort; leave it off unless the application is
|
||||
wrong without this extension.
|
||||
|
||||
The local snapshot is included in Docker builds. Local `make dev`, Docker dev,
|
||||
and the production Gateway image all consume the same `backend/uv.lock`.
|
||||
Development launchers may download missing locked artifacts before handing off
|
||||
to the Gateway; a built production container does not. Rebuild the production
|
||||
image with `make up` after changing the installed set.
|
||||
|
||||
After one or more runs, request the extension route:
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:2026/api/extension-example/stats
|
||||
```
|
||||
|
||||
The response contains aggregated task outcomes, tool-call counts, system-model
|
||||
call counts, the app scope id, and a small projection of the host policy. The
|
||||
route passes through the Gateway's normal authentication middleware; use an
|
||||
authenticated browser session when authentication is enabled.
|
||||
|
||||
## Packaging entry point
|
||||
|
||||
Managed packages expose exactly one standard PEP 621 entry point in the
|
||||
`deerflow.extensions` group. This example declares:
|
||||
|
||||
```toml
|
||||
[project.entry-points."deerflow.extensions"]
|
||||
example = "deerflow_extension_example:install"
|
||||
```
|
||||
|
||||
The entry-point name (`example`) is the stable operator-facing name accepted by
|
||||
`enable`, `disable`, and `remove`; those commands also accept the distribution
|
||||
name or the `module:install` value.
|
||||
|
||||
## Package layout
|
||||
|
||||
```text
|
||||
deerflow_extension_example/
|
||||
├── __init__.py # version-stamped install() entry point
|
||||
└── plugin.py # state plus all five small contribution implementations
|
||||
tests/
|
||||
├── test_entry_point.py
|
||||
└── test_plugin.py
|
||||
```
|
||||
@ -0,0 +1,35 @@
|
||||
"""A compact, standalone DeerFlow extension exercising every contribution kind."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from deerflow_extension_api import ExtensionInstall, ExtensionRegistry, extension
|
||||
|
||||
from deerflow_extension_example.plugin import (
|
||||
ExampleMiddlewareContributor,
|
||||
ExampleService,
|
||||
ExampleSystemObserver,
|
||||
ExampleTaskLifecycle,
|
||||
build_router,
|
||||
)
|
||||
|
||||
__all__ = ["install"]
|
||||
|
||||
|
||||
@extension(api="0.1.2", name="example")
|
||||
def install(registry: ExtensionRegistry, config: Mapping[str, Any]) -> None:
|
||||
"""Register one example of each supported contribution kind."""
|
||||
if config.get("enabled", True) is False:
|
||||
return
|
||||
|
||||
service = ExampleService()
|
||||
registry.middlewares(ExampleMiddlewareContributor())
|
||||
registry.task_lifecycle(ExampleTaskLifecycle())
|
||||
registry.system_model_observer(ExampleSystemObserver())
|
||||
registry.service(service)
|
||||
registry.routers((build_router(service),))
|
||||
|
||||
|
||||
_entry_point: ExtensionInstall = install
|
||||
@ -0,0 +1,176 @@
|
||||
"""The example's five deliberately small contribution implementations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from threading import Lock
|
||||
from typing import Any
|
||||
|
||||
from deerflow_extension_api import (
|
||||
AgentBuildContext,
|
||||
AgentScope,
|
||||
ExtensionData,
|
||||
ExtensionRuntimeDeps,
|
||||
MiddlewarePlacement,
|
||||
Placement,
|
||||
SystemModelRequest,
|
||||
SystemModelResult,
|
||||
SystemOperationKind,
|
||||
TaskInfo,
|
||||
TaskOutcome,
|
||||
task_store_from_runtime,
|
||||
)
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langgraph.prebuilt.tool_node import ToolCallRequest
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExampleStats:
|
||||
"""Small extension-owned value used in both app and task stores."""
|
||||
|
||||
tool_calls: int = 0
|
||||
tasks: dict[str, int] = field(default_factory=dict)
|
||||
system_model_calls: dict[str, dict[str, int]] = field(default_factory=dict)
|
||||
_lock: Lock = field(default_factory=Lock, repr=False, compare=False)
|
||||
|
||||
def note_tool_call(self) -> None:
|
||||
with self._lock:
|
||||
self.tool_calls += 1
|
||||
|
||||
def task_tool_calls(self) -> int:
|
||||
with self._lock:
|
||||
return self.tool_calls
|
||||
|
||||
def absorb_task(self, tool_calls: int, outcome: TaskOutcome) -> None:
|
||||
with self._lock:
|
||||
self.tool_calls += tool_calls
|
||||
key = outcome.value
|
||||
self.tasks[key] = self.tasks.get(key, 0) + 1
|
||||
|
||||
def note_system_call(self, kind: SystemOperationKind, *, failed: bool) -> None:
|
||||
with self._lock:
|
||||
entry = self.system_model_calls.setdefault(
|
||||
kind.value,
|
||||
{"calls": 0, "errors": 0},
|
||||
)
|
||||
entry["calls"] += 1
|
||||
if failed:
|
||||
entry["errors"] += 1
|
||||
|
||||
def snapshot(self) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
return {
|
||||
"tasks": dict(self.tasks),
|
||||
"tool_calls": self.tool_calls,
|
||||
"system_model_calls": {kind: dict(counts) for kind, counts in self.system_model_calls.items()},
|
||||
}
|
||||
|
||||
|
||||
def _stats(store: ExtensionData) -> ExampleStats:
|
||||
return store.get_or_init(ExampleStats, ExampleStats)
|
||||
|
||||
|
||||
class ExampleMiddleware(AgentMiddleware):
|
||||
async def awrap_tool_call(
|
||||
self,
|
||||
request: ToolCallRequest,
|
||||
handler: Callable[[ToolCallRequest], Awaitable[Any]],
|
||||
) -> Any:
|
||||
task_store = task_store_from_runtime(getattr(request, "runtime", None))
|
||||
task_stats = task_store.get(ExampleStats) if task_store is not None else None
|
||||
if task_stats is not None:
|
||||
task_stats.note_tool_call()
|
||||
return await handler(request)
|
||||
|
||||
|
||||
class ExampleMiddlewareContributor:
|
||||
def contribute_middlewares(
|
||||
self,
|
||||
app_store: ExtensionData,
|
||||
ctx: AgentBuildContext,
|
||||
) -> Sequence[MiddlewarePlacement]:
|
||||
return (
|
||||
MiddlewarePlacement(
|
||||
ExampleMiddleware(),
|
||||
Placement.TOOL_VISIBLE,
|
||||
AgentScope.BOTH,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ExampleTaskLifecycle:
|
||||
async def on_task_start(
|
||||
self,
|
||||
app_store: ExtensionData,
|
||||
task_store: ExtensionData,
|
||||
info: TaskInfo,
|
||||
) -> None:
|
||||
task_store.set(ExampleStats())
|
||||
|
||||
async def on_task_stop(
|
||||
self,
|
||||
app_store: ExtensionData,
|
||||
task_store: ExtensionData,
|
||||
info: TaskInfo,
|
||||
outcome: TaskOutcome,
|
||||
) -> None:
|
||||
task_stats = task_store.remove(ExampleStats)
|
||||
_stats(app_store).absorb_task(
|
||||
task_stats.task_tool_calls() if task_stats is not None else 0,
|
||||
outcome,
|
||||
)
|
||||
|
||||
|
||||
class ExampleSystemObserver:
|
||||
async def on_system_model_call(
|
||||
self,
|
||||
app_store: ExtensionData,
|
||||
task_store: ExtensionData,
|
||||
kind: SystemOperationKind,
|
||||
request: SystemModelRequest,
|
||||
result: SystemModelResult,
|
||||
) -> None:
|
||||
_stats(app_store).note_system_call(kind, failed=result.error is not None)
|
||||
|
||||
|
||||
class ExampleService:
|
||||
def __init__(self) -> None:
|
||||
self._deps: ExtensionRuntimeDeps | None = None
|
||||
|
||||
async def start(self, deps: ExtensionRuntimeDeps) -> None:
|
||||
self._deps = deps
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._deps = None
|
||||
|
||||
async def require_deps(self) -> ExtensionRuntimeDeps:
|
||||
deps = self._deps
|
||||
if deps is None or deps.app_store is None:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="extension-example is not running",
|
||||
)
|
||||
return deps
|
||||
|
||||
|
||||
def build_router(service: ExampleService) -> APIRouter:
|
||||
"""Build paths during registration, before runtime dependencies exist."""
|
||||
router = APIRouter(prefix="/api/extension-example", tags=["extension-example"])
|
||||
|
||||
@router.get("/stats")
|
||||
async def read_stats(
|
||||
deps: ExtensionRuntimeDeps = Depends(service.require_deps),
|
||||
) -> dict[str, Any]:
|
||||
assert deps.app_store is not None
|
||||
return {
|
||||
"scope_id": deps.app_store.scope_id,
|
||||
"session_factory_available": deps.session_factory is not None,
|
||||
"host_policy": {
|
||||
"max_subagents_per_run": deps.policy.max_subagents_per_run,
|
||||
},
|
||||
**_stats(deps.app_store).snapshot(),
|
||||
}
|
||||
|
||||
return router
|
||||
43
examples/deerflow-extension-example/pyproject.toml
Normal file
43
examples/deerflow-extension-example/pyproject.toml
Normal file
@ -0,0 +1,43 @@
|
||||
[project]
|
||||
name = "deerflow-extension-example"
|
||||
version = "0.1.0"
|
||||
description = "A compact standalone example covering every DeerFlow extension contribution kind"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"deerflow-extension-api>=0.1.2,<0.2",
|
||||
"fastapi>=0.115.0,<1",
|
||||
"langchain>=1.3,<2",
|
||||
"langgraph>=1.2.9,<1.3",
|
||||
]
|
||||
|
||||
[project.entry-points."deerflow.extensions"]
|
||||
example = "deerflow_extension_example:install"
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"httpx>=0.28,<1",
|
||||
"pytest>=9,<10",
|
||||
"ruff>=0.14,<1",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["deerflow_extension_example"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 240
|
||||
target-version = "py312"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "UP"]
|
||||
|
||||
[tool.ruff.lint.isort]
|
||||
known-first-party = ["deerflow_extension_example"]
|
||||
|
||||
[tool.ruff.format]
|
||||
quote-style = "double"
|
||||
indent-style = "space"
|
||||
@ -0,0 +1,11 @@
|
||||
from importlib.metadata import distribution
|
||||
|
||||
|
||||
def test_installed_distribution_exposes_deerflow_extension_entry_point() -> None:
|
||||
entry_points = [entry_point for entry_point in distribution("deerflow-extension-example").entry_points if entry_point.group == "deerflow.extensions"]
|
||||
|
||||
assert [(entry_point.name, entry_point.value) for entry_point in entry_points] == [("example", "deerflow_extension_example:install")]
|
||||
|
||||
install = entry_points[0].load()
|
||||
assert install.__deerflow_api__ == "0.1.2"
|
||||
assert install.__deerflow_name__ == "example"
|
||||
165
examples/deerflow-extension-example/tests/test_plugin.py
Normal file
165
examples/deerflow-extension-example/tests/test_plugin.py
Normal file
@ -0,0 +1,165 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from deerflow_extension_api import (
|
||||
EXTENSION_TASK_STORE_KEY,
|
||||
AgentBuildContext,
|
||||
AgentScope,
|
||||
ExtensionData,
|
||||
ExtensionRegistry,
|
||||
ExtensionRuntimeDeps,
|
||||
HostPolicySnapshot,
|
||||
SystemModelRequest,
|
||||
SystemModelResult,
|
||||
SystemOperationKind,
|
||||
TaskInfo,
|
||||
TaskOutcome,
|
||||
)
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from deerflow_extension_example import install
|
||||
|
||||
|
||||
class FakeRegistry:
|
||||
def __init__(self) -> None:
|
||||
self.middleware_contributors: list[Any] = []
|
||||
self.task_lifecycle_contributors: list[Any] = []
|
||||
self.system_model_observers: list[Any] = []
|
||||
self.services: list[Any] = []
|
||||
self.contributed_routers: list[Any] = []
|
||||
|
||||
def middlewares(self, contributor: Any) -> None:
|
||||
self.middleware_contributors.append(contributor)
|
||||
|
||||
def task_lifecycle(self, contributor: Any) -> None:
|
||||
self.task_lifecycle_contributors.append(contributor)
|
||||
|
||||
def system_model_observer(self, observer: Any) -> None:
|
||||
self.system_model_observers.append(observer)
|
||||
|
||||
def service(self, service: Any) -> None:
|
||||
self.services.append(service)
|
||||
|
||||
def routers(self, routers: Any) -> None:
|
||||
self.contributed_routers.extend(routers)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FakeRuntime:
|
||||
context: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FakeToolRequest:
|
||||
runtime: FakeRuntime
|
||||
|
||||
|
||||
def test_install_registers_all_five_contribution_kinds() -> None:
|
||||
registry = FakeRegistry()
|
||||
|
||||
install(registry, {})
|
||||
|
||||
assert isinstance(registry, ExtensionRegistry)
|
||||
assert len(registry.middleware_contributors) == 1
|
||||
assert len(registry.task_lifecycle_contributors) == 1
|
||||
assert len(registry.system_model_observers) == 1
|
||||
assert len(registry.services) == 1
|
||||
assert len(registry.contributed_routers) == 1
|
||||
assert [route.path for route in registry.contributed_routers[0].routes] == ["/api/extension-example/stats"]
|
||||
assert install.__deerflow_api__ == "0.1.2"
|
||||
assert install.__deerflow_name__ == "example"
|
||||
|
||||
|
||||
def test_disabled_extension_registers_nothing() -> None:
|
||||
registry = FakeRegistry()
|
||||
|
||||
install(registry, {"enabled": False})
|
||||
|
||||
assert registry.middleware_contributors == []
|
||||
assert registry.task_lifecycle_contributors == []
|
||||
assert registry.system_model_observers == []
|
||||
assert registry.services == []
|
||||
assert registry.contributed_routers == []
|
||||
|
||||
|
||||
def test_registered_contributions_publish_one_shared_stats_snapshot() -> None:
|
||||
registry = FakeRegistry()
|
||||
install(registry, {})
|
||||
app_store = ExtensionData("app")
|
||||
task_store = ExtensionData("task-1")
|
||||
task = TaskInfo(
|
||||
task_id="task-1",
|
||||
run_id="run-1",
|
||||
thread_id="thread-1",
|
||||
kind="lead",
|
||||
)
|
||||
|
||||
async def exercise_contributions() -> tuple[int, int, dict[str, Any], int]:
|
||||
lifecycle = registry.task_lifecycle_contributors[0]
|
||||
await lifecycle.on_task_start(app_store, task_store, task)
|
||||
placement = registry.middleware_contributors[0].contribute_middlewares(
|
||||
app_store,
|
||||
AgentBuildContext(scope=AgentScope.LEAD),
|
||||
)[0]
|
||||
|
||||
async def tool_handler(_request: object) -> str:
|
||||
return "tool-result"
|
||||
|
||||
request = FakeToolRequest(runtime=FakeRuntime(context={EXTENSION_TASK_STORE_KEY: task_store}))
|
||||
assert await placement.middleware.awrap_tool_call(request, tool_handler) == "tool-result"
|
||||
|
||||
await registry.system_model_observers[0].on_system_model_call(
|
||||
app_store,
|
||||
task_store,
|
||||
SystemOperationKind.TITLE,
|
||||
SystemModelRequest(messages="title prompt"),
|
||||
SystemModelResult(error=RuntimeError("provider unavailable")),
|
||||
)
|
||||
await lifecycle.on_task_stop(
|
||||
app_store,
|
||||
task_store,
|
||||
task,
|
||||
TaskOutcome.COMPLETED,
|
||||
)
|
||||
|
||||
app = FastAPI()
|
||||
app.include_router(registry.contributed_routers[0])
|
||||
async with AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
) as client:
|
||||
before_start = await client.get("/api/extension-example/stats")
|
||||
await registry.services[0].start(
|
||||
ExtensionRuntimeDeps(
|
||||
app_store=app_store,
|
||||
policy=HostPolicySnapshot(max_subagents_per_run=6),
|
||||
session_factory=object(),
|
||||
)
|
||||
)
|
||||
response = await client.get("/api/extension-example/stats")
|
||||
await registry.services[0].stop()
|
||||
after_stop = await client.get("/api/extension-example/stats")
|
||||
return (
|
||||
before_start.status_code,
|
||||
response.status_code,
|
||||
response.json(),
|
||||
after_stop.status_code,
|
||||
)
|
||||
|
||||
before_start, status_code, body, after_stop = asyncio.run(exercise_contributions())
|
||||
|
||||
assert before_start == 503
|
||||
assert status_code == 200
|
||||
assert after_stop == 503
|
||||
assert body == {
|
||||
"scope_id": "app",
|
||||
"session_factory_available": True,
|
||||
"host_policy": {"max_subagents_per_run": 6},
|
||||
"tasks": {"completed": 1},
|
||||
"tool_calls": 1,
|
||||
"system_model_calls": {"title": {"calls": 1, "errors": 1}},
|
||||
}
|
||||
@ -390,7 +390,7 @@ if ! $SKIP_INSTALL; then
|
||||
# `--all-packages` propagates extras into workspace members (deerflow-harness
|
||||
# in particular). Required for postgres extras — see PR #2584.
|
||||
# Intentionally unquoted to splat multiple `--extra X` pairs.
|
||||
(cd backend && uv sync --quiet --all-packages $UV_EXTRAS_FLAGS) || { echo "✗ Backend dependency install failed"; exit 1; }
|
||||
(cd backend && uv sync --locked --quiet --all-packages $UV_EXTRAS_FLAGS) || { echo "✗ Backend dependency install failed"; exit 1; }
|
||||
(cd frontend && "$DEERFLOW_PNPM_PYTHON" "$DEERFLOW_PNPM_RUNNER" install --silent) || { echo "✗ Frontend dependency install failed"; exit 1; }
|
||||
echo "✓ Dependencies synced"
|
||||
else
|
||||
@ -464,7 +464,7 @@ mkdir -p temp/client_body_temp temp/proxy_temp temp/fastcgi_temp temp/uwsgi_temp
|
||||
|
||||
# 1. Gateway API
|
||||
run_service "Gateway" \
|
||||
"cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 $GATEWAY_EXTRA_FLAGS > ../logs/gateway.log 2>&1" \
|
||||
"cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 $GATEWAY_EXTRA_FLAGS > ../logs/gateway.log 2>&1" \
|
||||
8001 30
|
||||
|
||||
# 2. Frontend
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user