mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-11 14:38:38 +00:00
* feat(extensions): add gateway services and routers * feat(extensions): add standalone reference extension * fix(extensions): harden contributed gateway routes * docs(extensions): document gateway contribution points * feat(extensions): add operator CLI for packaged extension management Add `deerflow extensions install/list/enable/disable/remove` plus the root `make extension-*` wrappers, backed by an `ExtensionManager` that owns one transaction over backend/pyproject.toml, backend/uv.lock, the managed source snapshot, the uv environment, and the `plugins:` block in config.yaml. Install accepts a package requirement, a public HTTPS Git URL, or a local directory. Local directories are copied to backend/extensions/sources/ as deployable snapshots rather than editable installs, and the root .dockerignore re-includes that tree so snapshots reach the backend builder. Remote sources are HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock Docker builder cannot reproduce them. Because environment configuration can still resolve a plain package name to a local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed by an audit of the new lock: any local reference the stock image build cannot reproduce rolls back the whole transaction. A config carrying duplicate top-level `plugins:` keys is rejected outright rather than managed against one block while the Gateway reads another. Dependency synchronization now has one lock authority. The `extensions` dependency group joins [tool.uv].default-groups, every startup path syncs the same lock with --locked and launches with --no-sync, and the Docker images move to uv 0.11.1 for the --no-workspace boundary the manager needs. Loader gains `enabled`, `name` and `package` fields so a disabled extension is skipped before resolution and import. Co-authored-by: Codex <codex@openai.com> * fix(extensions): stop the managed plugins rewrite from destroying config Two data-safety defects in the managed `plugins:` block writer. The "next top-level key" boundary was a regex matching only `[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a config may legally carry any top-level key, and a key the pattern cannot recognize did not fail loudly — it read as "no next section", and the rewrite replaced that neighbour and its entire subtree with the managed block. `my.key`, `2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a plain `extension-enable`/`disable`. Both boundaries now come from the YAML parser's node marks, so key shape is irrelevant. The file-final branch never consulted the trailing-comment scan the has-next-key branch used, so any comment below the block was dropped. Since the manager appends `plugins:` at end of file, that is the steady-state shape for most installs: an operator note below the block was destroyed on the next toggle. Separately, every managed install wrote `required: true` while the loader defaults to false. That turned any later load failure — broken wheel, missing native library, deleted snapshot — into a Gateway startup abort recoverable only with shell access. New records are now written `required: false`, with an explicit `install --required` opt-in; adopting an existing hand-written record still preserves the operator's own choice. * fix(extensions): harden the manager transaction and correct its docs Follow-up hardening on the extension package manager. Security posture, which the docs already claimed: - Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps the interpreter that the entry-point probe then imports and calls, and every later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS validation the HTTPS-only source rule depends on. Neither is an index, proxy, cache or credential-provider setting, so neither was covered by the carve-out. - Recognize run-together and all-caps secret query parameters (`accesstoken`, `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires on case transitions, so only the separated spellings were caught. Short generic words stay boundary-anchored, so `?keyword=` remains installable. - Validate the config before running any uv command. `uv add`/`uv sync` execute the package's build backend, so a config the manager could never write to must fail before that code runs rather than afterwards through rollback. Transaction integrity: - Run the second dependency-file restore from a `finally`. The recovery sync runs without `--locked` when the checkout had no lock, so uv writes one while resolving; if that sync then failed, the restore was skipped and the operator kept a lock file they never had. A failing recovery sync now also reports the original failure instead of replacing it. - Skip the recovery sync on cancellation. Answering Ctrl-C with a full dependency resolve invites a second interrupt that escapes the handler and strands the checkout mid-transaction; the declarations are already restored and the next locked startup sync reconciles the environment. - Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so contention surfaced as `Permission denied` rather than serializing. - Locate the entry-point probe's JSON payload instead of parsing stdout's first line, so a `sitecustomize`/`.pth` banner cannot roll back a good install. - Warn when the lock records a loopback source. `127.0.0.1` inside the image builder is a different machine, but unlike an environment-driven wheelhouse resolution this is a source the operator typed deliberately, so it is reported rather than rolled back. Private-network indexes are untouched: a builder on that network can reach them. Docs: the blanket claim that failed operations restore the config file was wrong — the conflict branches deliberately preserve a concurrent external edit and leave `remove` deactivated. Document that, the `required: false` default, the config preflight, the interrupt behaviour, and where the plugins-block boundaries come from. * test(gateway): pin the request-path projection agreement `get_request_route_path()` imports the private `starlette._utils.get_route_path` so the auth and CSRF predicates classify the exact string Starlette's router matches on. Its requirement is not "strip root_path correctly" but "return what the dispatcher is matching", so delegating to the router's own implementation keeps the two in lockstep by construction. Keep the private import rather than vendoring a copy: an import that disappears fails loudly at startup, while a stale copy diverges silently at a security boundary. Cover the property directly instead of the mechanism, so the tests survive a future reimplementation: - projection edge cases, including the segment-boundary guard that keeps root_path="/api" from slicing "/apifoo/models" into a string the router would never match - agreement with the router under nested mounts - the two bypasses these predicates exist to prevent: a protected route mounted under the "/health" public prefix must still 401, and a POST mounted under "/api/webhooks" must still require a CSRF token Both are verified to fail when the projection is reverted to `request.url.path` (9/13 red) and when a plausible vendored copy omits the boundary guard (the 2 boundary cases red). Declare starlette as a bounded direct dependency so a bump — which is security-relevant here — shows up in review rather than arriving silently through FastAPI. * ci: pin uv to the version production ships ExtensionManager is not a consumer of uv the build tool -- it is a program whose whole job is driving `uv` as a subprocess, depending on its CLI behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into `[dependency-groups] extensions`) and on the `uv.lock` serialization format. uv is closer to a runtime dependency with a contract than to incidental tooling. backend/Dockerfile pins that binary to 0.11.1, but all eight astral-sh/setup-uv steps installed whatever was latest at run time, so CI exercised the manager against a uv that is not the uv production runs. The sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI stays green because the same uv reads back what it wrote, and the pinned uv in the production image cannot read the committed lock. `uv lock --check` is version-sensitive for the same reason -- it verifies the lock is what *this* uv would produce, and two versions can emit equivalent but non-identical output. Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so the steps share input and caching behavior. Pinning alone drifts apart again on the next bump, so add a constraint test in the style of test_compose_default_bind_host.py: the Dockerfile's UV_IMAGE tag is the single source of truth, and both compose defaults plus every setup-uv step must match it. Verified to fail when a pin drifts, when a step omits `version`, and -- the real scenario -- when the Dockerfile is bumped alone, which lights up the workflows and both compose files at once. * fix(gateway): state the extension route auth limit and abort a failed dev sync Two scoped review follow-ups. README: contributed routers cannot enter the host's reserved public prefixes, which makes every extension endpoint session-authenticated -- there is no way to expose an unauthenticated route. The rejection rule was documented but its consequence was not, so inbound provider webhooks and public status endpoints read as merely undocumented rather than out of scope for this release. docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e` already stopped the script there -- uvicorn was not being started against a stale environment -- but it exited on a bare uv exit code with no indication of what to do. Abort explicitly with the cause and the fix. Tests slice the sync block out of the real script and run it against a stub uv, so they exercise the shipped code rather than a copy of it (/app/backend only exists inside the container). They cover the success path, the retry that recovers, the abort, and the guidance. Verified against the pre-fix script: only the guidance case goes red, confirming the abort itself was already correct. * fix(extensions): point Git SSH shorthand at the HTTPS correction Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git` reached the scheme rules looking like a bare path and was rejected with "local path references are not deployable; pass a local directory so DeerFlow can snapshot it". The operator asked for a remote source, so that guidance points at the wrong fix. Detect the shorthand ahead of the scheme rules and report the public-HTTPS correction instead. The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose `host` reads as a URL scheme and produced the generic HTTPS message. Both spellings now share one message, as does the PEP 508 named form. * docs: keep the root extension summary within its new budget #4799 split the depth out of the module guides and added a size gate; the root file's job is now orientation, and this branch had pushed it 192 bytes past the soft limit. The manager transaction, source rules, and lock discipline are already stated in full in the extensions guide, so the root keeps the one-line orientation and points there instead of restating them. --------- Co-authored-by: Codex <codex@openai.com>
986 lines
32 KiB
Python
986 lines
32 KiB
Python
"""Gateway binding tests for app-scoped extension contributions."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
from contextlib import asynccontextmanager
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from deerflow.extensions.registry import ExtensionRegistry
|
|
|
|
|
|
class _Service:
|
|
def __init__(
|
|
self,
|
|
name: str,
|
|
events: list[str],
|
|
*,
|
|
fail_start: bool = False,
|
|
fail_stop: bool = False,
|
|
) -> None:
|
|
self.name = name
|
|
self.events = events
|
|
self.fail_start = fail_start
|
|
self.fail_stop = fail_stop
|
|
self.deps = None
|
|
|
|
async def start(self, deps) -> None:
|
|
self.events.append(f"start:{self.name}")
|
|
self.deps = deps
|
|
if self.fail_start:
|
|
raise RuntimeError(f"{self.name} failed")
|
|
|
|
async def stop(self) -> None:
|
|
self.events.append(f"stop:{self.name}")
|
|
if self.fail_stop:
|
|
raise RuntimeError(f"{self.name} failed")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_services_start_in_order_with_narrow_deps_and_fail_open():
|
|
from deerflow.extensions.gateway import start_services
|
|
|
|
events: list[str] = []
|
|
first = _Service("first", events, fail_start=True)
|
|
second = _Service("second", events)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.service(first)
|
|
with registry.attributed_to("second:install"):
|
|
registry.service(second)
|
|
extensions = registry.build()
|
|
session_factory = object()
|
|
config = SimpleNamespace(
|
|
token_budget=SimpleNamespace(enabled=False, max_tokens=999),
|
|
subagents=SimpleNamespace(max_total_per_run=4),
|
|
)
|
|
|
|
diagnostics = await start_services(extensions, config, session_factory)
|
|
|
|
assert events == ["start:first", "start:second"]
|
|
assert first.deps is second.deps
|
|
assert second.deps.app_store is extensions.app_store
|
|
assert second.deps.session_factory is session_factory
|
|
assert second.deps.policy.token_budget_enabled is False
|
|
assert second.deps.policy.max_total_tokens is None
|
|
assert second.deps.policy.max_subagents_per_run == 4
|
|
assert [(diagnostic.source, diagnostic.level) for diagnostic in diagnostics] == [("first:install", "error")]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_service_originated_cancelled_error_does_not_abort_start_batch():
|
|
from deerflow.extensions.gateway import start_services
|
|
|
|
events: list[str] = []
|
|
|
|
class _CancelsItself(_Service):
|
|
async def start(self, deps) -> None:
|
|
self.events.append(f"start:{self.name}")
|
|
raise asyncio.CancelledError()
|
|
|
|
first = _CancelsItself("first", events)
|
|
second = _Service("second", events)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.service(first)
|
|
with registry.attributed_to("second:install"):
|
|
registry.service(second)
|
|
|
|
diagnostics = await start_services(registry.build(), SimpleNamespace(), None)
|
|
|
|
assert events == ["start:first", "start:second"]
|
|
assert len(diagnostics) == 1
|
|
assert diagnostics[0].source == "first:install"
|
|
assert "CancelledError" in diagnostics[0].message
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_services_stop_in_reverse_order_and_fail_open():
|
|
from deerflow.extensions.gateway import stop_services
|
|
|
|
events: list[str] = []
|
|
first = _Service("first", events)
|
|
second = _Service("second", events, fail_stop=True)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.service(first)
|
|
with registry.attributed_to("second:install"):
|
|
registry.service(second)
|
|
|
|
diagnostics = await stop_services(registry.build())
|
|
|
|
assert events == ["stop:second", "stop:first"]
|
|
assert [(diagnostic.source, diagnostic.level) for diagnostic in diagnostics] == [("second:install", "error")]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_service_originated_cancelled_error_does_not_abort_stop_batch():
|
|
from deerflow.extensions.gateway import stop_services
|
|
|
|
events: list[str] = []
|
|
|
|
class _CancelsItself(_Service):
|
|
async def stop(self) -> None:
|
|
self.events.append(f"stop:{self.name}")
|
|
raise asyncio.CancelledError()
|
|
|
|
first = _Service("first", events)
|
|
second = _CancelsItself("second", events)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.service(first)
|
|
with registry.attributed_to("second:install"):
|
|
registry.service(second)
|
|
|
|
diagnostics = await stop_services(registry.build())
|
|
|
|
assert events == ["stop:second", "stop:first"]
|
|
assert len(diagnostics) == 1
|
|
assert diagnostics[0].source == "second:install"
|
|
assert "CancelledError" in diagnostics[0].message
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_each_service_stop_has_its_own_timeout_budget():
|
|
from deerflow.extensions.gateway import stop_services
|
|
|
|
events: list[str] = []
|
|
|
|
class _HangingService(_Service):
|
|
async def stop(self) -> None:
|
|
self.events.append(f"stop:{self.name}")
|
|
await asyncio.Event().wait()
|
|
|
|
first = _Service("first", events)
|
|
second = _HangingService("second", events)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.service(first)
|
|
with registry.attributed_to("second:install"):
|
|
registry.service(second)
|
|
|
|
diagnostics = await stop_services(registry.build(), timeout_seconds=0.01)
|
|
|
|
assert events == ["stop:second", "stop:first"]
|
|
assert len(diagnostics) == 1
|
|
assert diagnostics[0].source == "second:install"
|
|
assert "timed out" in diagnostics[0].message
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_service_originated_timeout_error_is_reported_as_failure_not_budget_expiry():
|
|
from deerflow.extensions.gateway import stop_services
|
|
|
|
events: list[str] = []
|
|
|
|
class _RaisesTimeout(_Service):
|
|
async def stop(self) -> None:
|
|
self.events.append(f"stop:{self.name}")
|
|
raise TimeoutError("extension deadline")
|
|
|
|
first = _Service("first", events)
|
|
second = _RaisesTimeout("second", events)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.service(first)
|
|
with registry.attributed_to("second:install"):
|
|
registry.service(second)
|
|
|
|
diagnostics = await stop_services(registry.build(), timeout_seconds=1.0)
|
|
|
|
assert events == ["stop:second", "stop:first"]
|
|
assert diagnostics[0].source == "second:install"
|
|
assert "failed" in diagnostics[0].message
|
|
assert "timed out" not in diagnostics[0].message
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("owner_path", "owner_protocol", "candidate_path", "candidate_protocol", "rejected"),
|
|
[
|
|
("/exact", "GET", "/exact", "GET", True),
|
|
("/items/{item_id}", "GET", "/items/{id}", "GET", True),
|
|
("/items/{item_id}", "GET", "/items/new", "GET", True),
|
|
("/items/{item_id}", "GET", "/items/prefix-{id}", "GET", True),
|
|
("/pre{tenant}", "GET", "/prefoo{id}", "GET", True),
|
|
("/items/new", "GET", "/items/{id}", "GET", False),
|
|
("/records/{value}", "GET", "/records/{id:int}", "GET", True),
|
|
("/records/{value:int}", "GET", "/records/0{id:int}", "GET", True),
|
|
("/records/{value:int}", "GET", "/records/new", "GET", False),
|
|
("/files/{rest:path}", "GET", "/files/{id:int}", "GET", True),
|
|
("/files/{rest:path}", "GET", "/files/{id}", "GET", False),
|
|
("/x/{rest:path}/tail", "GET", "/x/a/{id}/tail", "GET", False),
|
|
("/items/{item_id}", "GET", "/items/{id}", "POST", False),
|
|
("/live/{item_id}", "WS", "/live/{id}", "GET", False),
|
|
],
|
|
ids=[
|
|
"exact",
|
|
"renamed-parameter",
|
|
"dynamic-shadows-static",
|
|
"dynamic-shadows-compound",
|
|
"compound-trailing-str-shadows-narrower-compound",
|
|
"static-does-not-shadow-dynamic",
|
|
"str-covers-int",
|
|
"int-shadows-digit-compound",
|
|
"int-does-not-cover-static",
|
|
"path-covers-descendant",
|
|
"path-does-not-cover-newline-capable-str",
|
|
"nonterminal-path-does-not-cover-newline-capable-str",
|
|
"disjoint-http-methods",
|
|
"websocket-does-not-shadow-http",
|
|
],
|
|
)
|
|
def test_router_conflicts_follow_starlette_dispatch_order(
|
|
owner_path,
|
|
owner_protocol,
|
|
candidate_path,
|
|
candidate_protocol,
|
|
rejected,
|
|
):
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
app = FastAPI()
|
|
if owner_protocol == "WS":
|
|
app.add_api_websocket_route(owner_path, endpoint)
|
|
else:
|
|
app.add_api_route(owner_path, endpoint, methods=[owner_protocol])
|
|
|
|
router = APIRouter()
|
|
if candidate_protocol == "WS":
|
|
router.add_api_websocket_route(candidate_path, endpoint)
|
|
else:
|
|
router.add_api_route(candidate_path, endpoint, methods=[candidate_protocol])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("candidate:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert bool(diagnostics) is rejected
|
|
if rejected:
|
|
assert diagnostics[0].source == "candidate:install"
|
|
assert "host" in diagnostics[0].message
|
|
assert candidate_path in diagnostics[0].message
|
|
else:
|
|
assert any(getattr(route, "path", None) == candidate_path for route in app.routes)
|
|
|
|
|
|
@pytest.mark.parametrize("convertor_name", ["flip", "int"])
|
|
def test_re_registered_convertor_does_not_create_a_false_shadow(
|
|
monkeypatch,
|
|
convertor_name,
|
|
):
|
|
from fastapi import APIRouter, FastAPI
|
|
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
|
from starlette.routing import Match
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
class DigitsConvertor(Convertor[str]):
|
|
regex = "[0-9]+"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
class LettersConvertor(Convertor[str]):
|
|
regex = "[A-Z]+"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
async def endpoint(value: str):
|
|
return {"value": value}
|
|
|
|
monkeypatch.setitem(CONVERTOR_TYPES, convertor_name, DigitsConvertor())
|
|
route_path = f"/owned/{{value:{convertor_name}}}"
|
|
app = FastAPI()
|
|
app.add_api_route(route_path, endpoint, methods=["GET"])
|
|
owner = app.routes[-1]
|
|
|
|
monkeypatch.setitem(CONVERTOR_TYPES, convertor_name, LettersConvertor())
|
|
router = APIRouter()
|
|
router.add_api_route(route_path, endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("candidate:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert diagnostics == []
|
|
candidate = app.routes[-1]
|
|
scope = {
|
|
"type": "http",
|
|
"path": "/owned/A",
|
|
"method": "GET",
|
|
"root_path": "",
|
|
}
|
|
assert owner.matches(scope)[0] is Match.NONE
|
|
assert candidate.matches(scope)[0] is Match.FULL
|
|
|
|
|
|
def test_router_claim_uses_converter_semantics_at_include_time(monkeypatch):
|
|
from fastapi import APIRouter, FastAPI
|
|
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
|
from starlette.routing import Match
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
class DigitsConvertor(Convertor[str]):
|
|
regex = "[0-9]+"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
class LettersConvertor(Convertor[str]):
|
|
regex = "[A-Z]+"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
async def endpoint(value: str):
|
|
return {"value": value}
|
|
|
|
monkeypatch.setitem(CONVERTOR_TYPES, "flip", DigitsConvertor())
|
|
route_path = "/owned/{value:flip}"
|
|
app = FastAPI()
|
|
app.add_api_route(route_path, endpoint, methods=["GET"])
|
|
owner = app.routes[-1]
|
|
router = APIRouter()
|
|
router.add_api_route(route_path, endpoint, methods=["GET"])
|
|
|
|
monkeypatch.setitem(CONVERTOR_TYPES, "flip", LettersConvertor())
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("candidate:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert diagnostics == []
|
|
candidate = app.routes[-1]
|
|
scope = {
|
|
"type": "http",
|
|
"path": "/owned/A",
|
|
"method": "GET",
|
|
"root_path": "",
|
|
}
|
|
assert owner.matches(scope)[0] is Match.NONE
|
|
assert candidate.matches(scope)[0] is Match.FULL
|
|
|
|
|
|
def test_recompiled_converter_cannot_enter_a_public_namespace(monkeypatch):
|
|
from fastapi import APIRouter, FastAPI
|
|
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
class PublicPathConvertor(Convertor[str]):
|
|
regex = r"webhooks/.+"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
async def endpoint(value: str):
|
|
return {"value": value}
|
|
|
|
router = APIRouter()
|
|
router.add_api_route("/api/{value:int}", endpoint, methods=["GET"])
|
|
monkeypatch.setitem(CONVERTOR_TYPES, "int", PublicPathConvertor())
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("candidate:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["candidate:install"]
|
|
assert "public namespace" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", None) == "/api/{value:int}" for route in app.routes)
|
|
|
|
|
|
def test_host_mount_claims_descendant_http_paths():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
app = FastAPI()
|
|
app.mount("/assets", FastAPI())
|
|
router = APIRouter()
|
|
|
|
@router.get("/assets/{name:int}")
|
|
async def asset(name: int):
|
|
return {"name": name}
|
|
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("assets:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert len(diagnostics) == 1
|
|
assert diagnostics[0].source == "assets:install"
|
|
assert "host" in diagnostics[0].message
|
|
|
|
|
|
def test_dynamic_host_mount_claims_matching_descendants():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
app = FastAPI()
|
|
app.mount("/pre{tenant}", FastAPI())
|
|
router = APIRouter()
|
|
|
|
@router.get("/prefoo/{item_id:int}")
|
|
async def item(item_id: int):
|
|
return {"item_id": item_id}
|
|
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("mount:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["mount:install"]
|
|
assert "host" in diagnostics[0].message
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("mount_path", "candidate_path", "witness"),
|
|
[
|
|
("/assets", "/assets/{name}", "/assets/a\nb"),
|
|
("/pre{tenant}", "/prefoo{id}/{child}", "/prefoo1/a\nb"),
|
|
],
|
|
)
|
|
def test_host_mount_does_not_claim_newline_capable_str_descendants(
|
|
mount_path,
|
|
candidate_path,
|
|
witness,
|
|
):
|
|
from fastapi import APIRouter, FastAPI
|
|
from starlette.routing import Match, Mount
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
app = FastAPI()
|
|
app.mount(mount_path, FastAPI())
|
|
host_mount = next(route for route in app.routes if isinstance(route, Mount) and route.path == mount_path)
|
|
router = APIRouter()
|
|
router.add_api_route(candidate_path, endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("mount:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert diagnostics == []
|
|
candidate_route = next(route for route in app.routes if getattr(route, "path", None) == candidate_path)
|
|
scope = {
|
|
"type": "http",
|
|
"path": witness,
|
|
"method": "GET",
|
|
"root_path": "",
|
|
}
|
|
assert host_mount.matches(scope)[0] is Match.NONE
|
|
assert candidate_route.matches(scope)[0] is Match.FULL
|
|
|
|
|
|
def test_contributed_websocket_route_is_rejected_until_host_auth_wraps_it():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def websocket_endpoint(websocket):
|
|
await websocket.close()
|
|
|
|
router = APIRouter()
|
|
router.add_api_websocket_route("/extension-ws", websocket_endpoint)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("websocket:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["websocket:install"]
|
|
assert "WebSocket" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", None) == "/extension-ws" for route in app.routes)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"path",
|
|
[
|
|
"/health-extension",
|
|
"/docs-private",
|
|
"/redoc-private",
|
|
"/api/webhooks/extension",
|
|
"/api/{rest:path}",
|
|
"/api/{section}/extension",
|
|
],
|
|
)
|
|
def test_extension_routes_cannot_enter_host_public_namespaces(path):
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
router = APIRouter()
|
|
router.add_api_route(path, endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("public:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["public:install"]
|
|
assert "public namespace" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", None) == path for route in app.routes)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"path",
|
|
[
|
|
"/api",
|
|
"/heal",
|
|
"/api/{item_id:int}",
|
|
"/api/{item_id}",
|
|
],
|
|
)
|
|
def test_extension_routes_that_cannot_enter_a_public_namespace_are_allowed(path):
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
router = APIRouter()
|
|
router.add_api_route(path, endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("private:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert diagnostics == []
|
|
assert any(getattr(route, "path", None) == path for route in app.routes)
|
|
|
|
|
|
def test_unknown_convertor_near_a_public_namespace_fails_closed(monkeypatch):
|
|
from fastapi import APIRouter, FastAPI
|
|
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
class UppercaseConvertor(Convertor[str]):
|
|
regex = "[A-Z]+"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
monkeypatch.setitem(CONVERTOR_TYPES, "uppercase", UppercaseConvertor())
|
|
router = APIRouter()
|
|
router.add_api_route(
|
|
"/api/{value:uppercase}",
|
|
lambda: {"ok": True},
|
|
methods=["GET"],
|
|
)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("custom-public:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(
|
|
FastAPI(docs_url=None, redoc_url=None, openapi_url=None),
|
|
registry.build(),
|
|
)
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["custom-public:install"]
|
|
assert "public namespace" in diagnostics[0].message
|
|
|
|
|
|
def test_private_custom_convertor_with_named_backreference_is_allowed(monkeypatch):
|
|
from fastapi import APIRouter, FastAPI
|
|
from starlette.convertors import CONVERTOR_TYPES, Convertor
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
class DoubledLetterConvertor(Convertor[str]):
|
|
regex = r"(?P<char>[A-Z])(?P=char)"
|
|
|
|
def convert(self, value: str) -> str:
|
|
return value
|
|
|
|
def to_string(self, value: str) -> str:
|
|
return value
|
|
|
|
monkeypatch.setitem(CONVERTOR_TYPES, "doubled", DoubledLetterConvertor())
|
|
router = APIRouter()
|
|
router.add_api_route(
|
|
"/private/{value:doubled}",
|
|
lambda value: {"value": value},
|
|
methods=["GET"],
|
|
)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("custom-private:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert diagnostics == []
|
|
assert any(getattr(route, "path", None) == "/private/{value:doubled}" for route in app.routes)
|
|
|
|
|
|
def test_extension_public_paths_track_auth_middleware_public_paths():
|
|
from app.gateway.auth_middleware import (
|
|
_PUBLIC_EXACT_PATHS,
|
|
_PUBLIC_PATH_PREFIXES,
|
|
_is_public,
|
|
)
|
|
from deerflow.extensions.gateway import (
|
|
_HOST_PUBLIC_EXACT_PATHS,
|
|
_HOST_PUBLIC_PATH_PREFIXES,
|
|
)
|
|
|
|
assert _HOST_PUBLIC_PATH_PREFIXES == _PUBLIC_PATH_PREFIXES
|
|
assert _HOST_PUBLIC_EXACT_PATHS == _PUBLIC_EXACT_PATHS
|
|
assert all(_is_public(f"{path}//") for path in _HOST_PUBLIC_EXACT_PATHS)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("host_path", "host_method", "candidate_path", "candidate_method"),
|
|
[
|
|
(
|
|
"/api/v1/auth/login/local",
|
|
"POST",
|
|
"/api/v1/auth/login/local",
|
|
"GET",
|
|
),
|
|
(
|
|
"/api/v1/auth/login/local",
|
|
"POST",
|
|
"/api/v1/auth/login/local/",
|
|
"GET",
|
|
),
|
|
(
|
|
"/api/v1/auth/login/local",
|
|
"POST",
|
|
"/api/v1/auth/login/local//",
|
|
"GET",
|
|
),
|
|
("/api/v1/auth/me", "GET", "/api/v1/auth/me", "POST"),
|
|
("/api/v1/auth/me", "GET", "/api/v1/auth/me/", "POST"),
|
|
],
|
|
)
|
|
def test_extension_routes_cannot_claim_reserved_exact_paths_with_a_disjoint_method(
|
|
host_path,
|
|
host_method,
|
|
candidate_path,
|
|
candidate_method,
|
|
):
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
app = FastAPI()
|
|
app.add_api_route(host_path, endpoint, methods=[host_method])
|
|
router = APIRouter()
|
|
router.add_api_route(candidate_path, endpoint, methods=[candidate_method])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("public-exact:install"):
|
|
registry.routers((router,))
|
|
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["public-exact:install"]
|
|
assert "reserved" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", None) == candidate_path and getattr(route, "methods", set()) == {candidate_method} for route in app.routes)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"path",
|
|
[
|
|
"/api/v1/auth/me",
|
|
"/api/v1/auth/me/",
|
|
"/api/v1/auth/me//",
|
|
],
|
|
)
|
|
def test_extension_csrf_reserved_exact_paths_track_csrf_exemption(
|
|
monkeypatch,
|
|
path,
|
|
):
|
|
from starlette.requests import Request
|
|
|
|
from app.gateway import csrf_middleware
|
|
from deerflow.extensions.gateway import (
|
|
_CSRF_STATE_CHANGING_METHODS,
|
|
_HOST_CSRF_EXEMPT_EXACT_PATHS,
|
|
_HOST_PUBLIC_EXACT_PATHS,
|
|
)
|
|
|
|
monkeypatch.setattr(csrf_middleware, "is_auth_disabled", lambda: False)
|
|
request = Request(
|
|
{
|
|
"type": "http",
|
|
"method": "POST",
|
|
"scheme": "http",
|
|
"path": path,
|
|
"raw_path": path.encode(),
|
|
"query_string": b"",
|
|
"headers": [],
|
|
"server": ("testserver", 80),
|
|
}
|
|
)
|
|
|
|
assert path.rstrip("/") in _HOST_CSRF_EXEMPT_EXACT_PATHS
|
|
assert _HOST_CSRF_EXEMPT_EXACT_PATHS == csrf_middleware._CSRF_EXEMPT_EXACT_PATHS
|
|
assert _CSRF_STATE_CHANGING_METHODS == csrf_middleware._CSRF_STATE_CHANGING_METHODS
|
|
assert csrf_middleware._AUTH_EXEMPT_PATHS <= _HOST_PUBLIC_EXACT_PATHS
|
|
assert csrf_middleware.should_check_csrf(request) is False
|
|
|
|
|
|
def test_safe_method_at_csrf_exempt_exact_path_is_not_reserved():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
router = APIRouter()
|
|
router.add_api_route(
|
|
"/api/v1/auth/me",
|
|
lambda: {"ok": True},
|
|
methods=["GET"],
|
|
)
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("safe-csrf:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert diagnostics == []
|
|
assert any(getattr(route, "path", None) == "/api/v1/auth/me" for route in app.routes)
|
|
|
|
|
|
def test_router_with_one_conflict_is_rejected_atomically_and_names_first_owner():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
first = APIRouter()
|
|
first.add_api_route("/shared", endpoint, methods=["GET"])
|
|
second = APIRouter()
|
|
second.add_api_route("/would-have-been-reachable", endpoint, methods=["GET"])
|
|
second.add_api_route("/shared", endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("first:install"):
|
|
registry.routers((first,))
|
|
with registry.attributed_to("second:install"):
|
|
registry.routers((second,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
paths = [getattr(route, "path", None) for route in app.routes]
|
|
|
|
assert "/shared" in paths
|
|
assert "/would-have-been-reachable" not in paths
|
|
assert len(diagnostics) == 1
|
|
assert diagnostics[0].source == "second:install"
|
|
assert "first:install" in diagnostics[0].message
|
|
|
|
|
|
def test_router_is_rejected_when_its_own_earlier_route_shadows_a_later_one():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
router = APIRouter()
|
|
router.add_api_route("/same/{value}", endpoint, methods=["GET"])
|
|
router.add_api_route("/same/fixed", endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("self-shadow:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["self-shadow:install"]
|
|
assert "self-shadow:install" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", "").startswith("/same/") for route in app.routes)
|
|
|
|
|
|
def test_contributed_mount_is_rejected_but_does_not_starve_later_router():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
bad = APIRouter()
|
|
bad.mount("/nested", FastAPI())
|
|
good = APIRouter()
|
|
|
|
@good.get("/extension-good")
|
|
async def extension_good():
|
|
return {"ok": True}
|
|
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("bad:install"):
|
|
registry.routers((bad,))
|
|
with registry.attributed_to("good:install"):
|
|
registry.routers((good,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["bad:install"]
|
|
assert "Mount" in diagnostics[0].message
|
|
assert any(getattr(route, "path", None) == "/extension-good" for route in app.routes)
|
|
|
|
|
|
def test_router_with_unsupported_route_item_is_rejected_atomically():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
router = APIRouter()
|
|
router.add_api_route("/otherwise-valid", endpoint, methods=["GET"])
|
|
router.routes.append(object())
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("unsupported:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["unsupported:install"]
|
|
assert "unsupported route" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", None) == "/otherwise-valid" for route in app.routes)
|
|
|
|
|
|
def test_include_router_failure_rolls_back_partial_routes_before_continuing():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
broken = APIRouter()
|
|
broken.add_api_route("/partial", endpoint, methods=["GET"])
|
|
broken.add_api_route("/explodes", endpoint, methods=["GET"])
|
|
broken.routes[-1].endpoint = None
|
|
later = APIRouter()
|
|
later.add_api_route("/partial", endpoint, methods=["GET"])
|
|
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("broken:install"):
|
|
registry.routers((broken,))
|
|
with registry.attributed_to("later:install"):
|
|
registry.routers((later,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
partial_routes = [route for route in app.routes if getattr(route, "path", None) == "/partial"]
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["broken:install"]
|
|
assert len(partial_routes) == 1
|
|
|
|
|
|
def test_router_lifecycle_hooks_are_rejected_in_favor_of_extension_service():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
async def startup_hook():
|
|
raise RuntimeError("must never be installed")
|
|
|
|
bad = APIRouter()
|
|
bad.add_api_route("/has-lifecycle", endpoint, methods=["GET"])
|
|
bad.add_event_handler("startup", startup_hook)
|
|
good = APIRouter()
|
|
good.add_api_route("/after-lifecycle", endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("lifecycle:install"):
|
|
registry.routers((bad,))
|
|
with registry.attributed_to("good:install"):
|
|
registry.routers((good,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
paths = [getattr(route, "path", None) for route in app.routes]
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["lifecycle:install"]
|
|
assert "ExtensionService" in diagnostics[0].message
|
|
assert "/has-lifecycle" not in paths
|
|
assert "/after-lifecycle" in paths
|
|
assert startup_hook not in app.router.on_startup
|
|
|
|
|
|
def test_router_custom_lifespan_is_rejected_in_favor_of_extension_service():
|
|
from fastapi import APIRouter, FastAPI
|
|
|
|
from deerflow.extensions.gateway import include_contributed_routers
|
|
|
|
@asynccontextmanager
|
|
async def lifespan(_app):
|
|
yield
|
|
|
|
async def endpoint():
|
|
return {"ok": True}
|
|
|
|
router = APIRouter(lifespan=lifespan)
|
|
router.add_api_route("/custom-lifespan", endpoint, methods=["GET"])
|
|
registry = ExtensionRegistry()
|
|
with registry.attributed_to("lifespan:install"):
|
|
registry.routers((router,))
|
|
|
|
app = FastAPI()
|
|
diagnostics = include_contributed_routers(app, registry.build())
|
|
|
|
assert [diagnostic.source for diagnostic in diagnostics] == ["lifespan:install"]
|
|
assert "ExtensionService" in diagnostics[0].message
|
|
assert not any(getattr(route, "path", None) == "/custom-lifespan" for route in app.routes)
|