Nan Gao 1f792d0f4b
feat(extensions): add middleware plugin foundation (#4636)
* feat(extensions): add middleware plugin foundation

* fix(extensions): stop config resolution from masking extension loading

`create_app()` resolved the configured plugin list inside the fail-open
guard around `load_extensions()`. CI has no `config.yaml` (gitignored and
never generated by the workflow), so `get_app_config()` raised
`FileNotFoundError` there and was swallowed as an extension failure --
`load_extensions()` never ran at all, and the four `create_app()` tests in
`test_extension_app_loading.py` passed locally but failed on every runner.

Resolve the plugin list before the guard. Only an absent `config.yaml` is
tolerated, mirroring `_resolve_trace_enabled_for_app_construction()`:
`create_app()` runs at import time, and lifespan still performs strict
config loading before serving. A `config.yaml` that exists but fails to
parse or validate now propagates instead of being reported as an extension
failure -- reporting it as the latter silently dropped a `required: true`
extension rather than failing the boot.

Make the tests config-independent with an autouse `stub_app_config`
fixture, following the existing pattern in `test_gateway_lifespan_shutdown.py`,
and cover both new branches of the config-resolution boundary.

* fix(extensions): bind the run's extension snapshot through subagent delegation

The lead-agent path resolves one immutable loaded-extension snapshot per run
and binds it through task-store allocation and graph construction, but the
subagent path re-read the process-wide singleton at execution time. In
production both are the same object, yet a `set_loaded_extensions()` between
the lead run's start and a subagent's execution (test teardown, a future
hot-reload path) would let one run mix two extension generations — exactly what
the documented invariant exists to prevent.

The graph-build binding is a ContextVar scoped to synchronous construction, so
it has already exited by the time a tool delegates; the snapshot has to travel
through runtime context instead. The run worker publishes it under the
host-internal `EXTENSION_SNAPSHOT_CONTEXT_KEY` (written after the caller merge,
popped when the run has none, so a caller-supplied value is never
authoritative), `task_tool` reads it back through the type-checking
`resolve_run_extensions()`, and `SubagentExecutor` binds it at construction.

Callers outside the Gateway run path — embedded `DeerFlowClient`, standalone
LangGraph Server — install no snapshot and keep the existing
`get_loaded_extensions()` fallback.

* refactor(extensions): defer the ordering table by call, not by a lying tuple

`CORE_ORDERING_CONSTRAINTS` was a `tuple` subclass that overrode only
`__iter__` and resolved into a class-level `_resolved` side channel. A tuple
cannot populate its own storage after construction, so the instance stayed the
empty tuple it was built as: `len()` was 0, `bool()` was False, `in` was always
False, indexing raised, slicing and `reversed()` came back empty, and it
compared unequal to the plain tuples tests substitute for it — all while
iteration yielded the real constraints. Only `assert_ordering` consumed it, and
only by iterating, so the split went unnoticed.

The sibling `_AnchorTable(dict)` uses the same idea soundly because dict is
mutable: `self.update()` fills the real storage, making every inherited
operation correct. That trick does not survive the port to an immutable type.

Replace it with `core_ordering_constraints()`, matching how `stack.py` defers
the same kind of table via `_anchors()`. The deferral is kept — it is about
dependency direction, not just cycles: `extensions/` is the layer the
middleware layer calls into, so a module-scope `agents.middlewares` import here
points the dependency backwards and closes a cycle as soon as any middleware
imports something under `extensions/` at module level. Resolution stays at
`assert_ordering` time, which already runs inside the middleware builder.

Tests pin both halves: the returned value is a plain tuple whose len/bool/
membership/indexing/reversal/equality agree with iteration, and a subprocess
probe asserts importing `extensions.ordering` does not load the middleware
layer while calling the function does.
2026-08-04 22:33:26 +08:00

159 lines
5.1 KiB
Python

"""DeerFlow's extension mechanism (host side).
The public contracts live in the separate `deerflow-extension-api` package;
this module implements loading, registration, middleware injection and the
hook-site plumbing.
"""
from __future__ import annotations
import threading
from collections.abc import Iterator, Mapping
from contextlib import contextmanager
from contextvars import ContextVar
from typing import Any
from deerflow.extensions.loader import (
Diagnostic,
ExtensionLoadError,
ExtensionSpec,
load_extensions,
)
from deerflow.extensions.registry import EMPTY_EXTENSIONS, ExtensionRegistry, LoadedExtensions
#: Runtime-context key carrying the run's immutable extension snapshot.
#:
#: The graph-build binding below is a ContextVar scoped to synchronous agent
#: construction, so it is long gone by the time a tool delegates work. Runtime
#: context is how the run reaches that later code. The double-underscore prefix
#: marks it as host-internal: the Gateway strips caller-supplied ``__`` keys,
#: and this snapshot is never part of the public extension contract.
EXTENSION_SNAPSHOT_CONTEXT_KEY = "__deerflow_extension_snapshot"
_loaded: LoadedExtensions = EMPTY_EXTENSIONS
_agent_build_extensions: ContextVar[LoadedExtensions | None] = ContextVar(
"deerflow_agent_build_extensions",
default=None,
)
def get_loaded_extensions() -> LoadedExtensions:
"""Return the process-wide loaded extensions.
Mirrors the existing `get_app_config()` convention so call sites can take
an explicit override parameter and fall back to this.
"""
return _loaded
def get_agent_build_extensions() -> LoadedExtensions:
"""Return the run-bound snapshot while an agent graph is being built."""
return _agent_build_extensions.get() or get_loaded_extensions()
@contextmanager
def bind_agent_build_extensions(loaded: LoadedExtensions) -> Iterator[None]:
"""Bind one immutable extension snapshot to synchronous graph assembly."""
token = _agent_build_extensions.set(loaded)
try:
yield
finally:
_agent_build_extensions.reset(token)
def resolve_run_extensions(context: Any | None) -> LoadedExtensions | None:
"""Return the run's extension snapshot from *context*, or ``None``.
Runtime context is caller-mergeable, so the value is type-checked rather
than trusted. ``None`` means "this caller installed no snapshot" (embedded
client, standalone LangGraph Server) and leaves consumers on their existing
``get_loaded_extensions()`` fallback.
"""
if not isinstance(context, Mapping):
return None
snapshot = context.get(EXTENSION_SNAPSHOT_CONTEXT_KEY)
return snapshot if isinstance(snapshot, LoadedExtensions) else None
def set_loaded_extensions(loaded: LoadedExtensions) -> None:
global _loaded
_loaded = loaded
def reset_loaded_extensions() -> None:
"""Reset to a FRESH empty set. Used by tests to prevent singleton leaks.
Builds a new instance rather than reusing EMPTY_EXTENSIONS: that singleton
owns a mutable ExtensionData app_store, so resetting to it would carry any
write made while "empty" across every later reset and across the process.
"""
global _loaded
_loaded = ExtensionRegistry().build()
_runtime_diagnostics: list[Diagnostic] = []
_runtime_diagnostics_lock = threading.RLock()
_MAX_RUNTIME_DIAGNOSTICS = 1000
def _trim_runtime_diagnostics() -> None:
overflow = len(_runtime_diagnostics) - _MAX_RUNTIME_DIAGNOSTICS
if overflow > 0:
del _runtime_diagnostics[:overflow]
def initialize_runtime_diagnostics(diagnostics: list[Diagnostic]) -> list[Diagnostic]:
"""Install and return the live diagnostic list for the current host."""
with _runtime_diagnostics_lock:
_runtime_diagnostics.clear()
_runtime_diagnostics.extend(diagnostics)
_trim_runtime_diagnostics()
return _runtime_diagnostics
def record_runtime_diagnostic(diagnostic: Diagnostic) -> None:
"""Collect one diagnostic in the canonical process sink."""
with _runtime_diagnostics_lock:
_runtime_diagnostics.append(diagnostic)
_trim_runtime_diagnostics()
def record_runtime_diagnostics(diagnostics: list[Diagnostic]) -> None:
"""Collect a diagnostic batch in the canonical process sink."""
with _runtime_diagnostics_lock:
_runtime_diagnostics.extend(diagnostics)
_trim_runtime_diagnostics()
def get_runtime_diagnostics() -> list[Diagnostic]:
with _runtime_diagnostics_lock:
return list(_runtime_diagnostics)
def reset_runtime_diagnostics() -> None:
with _runtime_diagnostics_lock:
_runtime_diagnostics.clear()
__all__ = [
"EMPTY_EXTENSIONS",
"EXTENSION_SNAPSHOT_CONTEXT_KEY",
"Diagnostic",
"ExtensionLoadError",
"ExtensionRegistry",
"ExtensionSpec",
"LoadedExtensions",
"bind_agent_build_extensions",
"get_agent_build_extensions",
"get_loaded_extensions",
"get_runtime_diagnostics",
"initialize_runtime_diagnostics",
"load_extensions",
"record_runtime_diagnostic",
"record_runtime_diagnostics",
"reset_loaded_extensions",
"reset_runtime_diagnostics",
"resolve_run_extensions",
"set_loaded_extensions",
]