deer-flow/backend/tests/test_extension_config.py
Nan Gao 7389331e65
feat(extensions): observe task lifecycle and system model calls (#4684)
* feat(extensions): observe task lifecycle and system model calls

PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.

Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).

`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.

Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.

Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.

System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.

The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.

Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.

Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.

* fix(extensions): decide notification fail-open by origin, observe cancellation

`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.

`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.

DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.

* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown

Review response on #4684:

- The memory observation bridge caught BaseException, which would swallow
  a teardown signal raised while dispatching; it now catches Exception,
  matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
  hook-failure path and logged an asyncio-internal traceback; it now logs
  a warning like the pre-hook budget skip, while a TimeoutError a
  contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
  so log lines said "task goal/title/..."; they now carry the task
  scope id alongside the kind.
2026-08-11 16:33:22 +08:00

147 lines
5.0 KiB
Python

"""Tests for extension configuration and the process-wide singleton."""
from __future__ import annotations
import pytest
from pydantic import ValidationError
from deerflow.config.app_config import AppConfig
from deerflow.config.reload_boundary import STARTUP_ONLY_FIELDS
from deerflow.extensions import (
EMPTY_EXTENSIONS,
ExtensionRegistry,
get_loaded_extensions,
reset_loaded_extensions,
set_loaded_extensions,
)
@pytest.fixture(autouse=True)
def _reset_singleton():
reset_loaded_extensions()
yield
reset_loaded_extensions()
class _Marker:
"""Sentinel written into an app_store to detect state leaking across resets."""
def __init__(self, tag: str = "") -> None:
self.tag = tag
# AppConfig.sandbox has no default (see app_config.py's
# `_drop_null_config_sections`: "Required sections without a default
# (sandbox) intentionally still error when null"), so every AppConfig
# construction below supplies it, matching the pattern already used in
# test_app_config_reload.py.
_SANDBOX = {"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}}
def test_app_config_defaults_to_no_plugins():
assert AppConfig.model_validate(_SANDBOX).plugins == []
def test_app_config_parses_plugin_entries():
config = AppConfig.model_validate(
{
**_SANDBOX,
"plugins": [
{"use": "acme_observability:install", "config": {"enabled": True}},
{"use": "acme_policy:install", "required": True},
],
}
)
assert [e.use for e in config.plugins] == ["acme_observability:install", "acme_policy:install"]
assert config.plugins[0].config == {"enabled": True}
assert config.plugins[0].required is False
assert config.plugins[1].required is True
def test_plugin_entries_reject_unknown_fields_instead_of_weakening_required():
with pytest.raises(ValidationError, match="require"):
AppConfig.model_validate(
{
**_SANDBOX,
"plugins": [
{
"use": "acme_policy:install",
"require": True,
}
],
}
)
def test_new_field_does_not_disturb_the_existing_extensions_field():
"""AppConfig.extensions is a pre-existing, unrelated field (MCP servers,
skills, config-declared middlewares) backed by extensions_config.json.
The plugin list is deliberately a separate top-level key: that file is
writable through an HTTP endpoint, and a code-loading list must not be."""
config = AppConfig.model_validate({**_SANDBOX, "plugins": [{"use": "a:install"}]})
assert config.plugins[0].use == "a:install"
assert hasattr(config.extensions, "mcp_servers")
assert hasattr(config.extensions, "middlewares")
def test_plugins_is_registered_as_startup_only():
"""Plugins load once in create_app(); a config.yaml edit needs a restart.
Registering here is what surfaces that to operators."""
assert "plugins" in STARTUP_ONLY_FIELDS
assert "restart" in STARTUP_ONLY_FIELDS["plugins"].lower()
def test_singleton_defaults_to_empty():
loaded = get_loaded_extensions()
assert loaded.has_middleware_contributors is False
assert loaded.has_task_lifecycle is False
assert loaded.has_system_model_observers is False
assert loaded.needs_task_store is False
def test_singleton_roundtrips():
loaded = ExtensionRegistry().build()
set_loaded_extensions(loaded)
assert get_loaded_extensions() is loaded
def test_reset_gives_a_fresh_instance():
"""Reset must not hand back a shared object. EMPTY_EXTENSIONS owns a
mutable app_store, so resetting to it would carry writes forward."""
populated = ExtensionRegistry().build()
set_loaded_extensions(populated)
reset_loaded_extensions()
after = get_loaded_extensions()
assert after is not populated
assert after is not EMPTY_EXTENSIONS
assert after.has_middleware_contributors is False
assert after.has_task_lifecycle is False
assert after.has_system_model_observers is False
def test_reset_does_not_leak_app_store_writes():
"""The regression the fresh-build reset exists to prevent."""
reset_loaded_extensions()
get_loaded_extensions().app_store.set(_Marker("dirty"))
reset_loaded_extensions()
assert get_loaded_extensions().app_store.get(_Marker) is None
def test_runtime_diagnostics_are_bounded_without_replacing_the_live_list(monkeypatch):
import deerflow.extensions as extensions_module
monkeypatch.setattr(extensions_module, "_MAX_RUNTIME_DIAGNOSTICS", 3)
extensions_module.reset_runtime_diagnostics()
live = extensions_module.initialize_runtime_diagnostics([])
try:
for index in range(5):
extensions_module.record_runtime_diagnostic(extensions_module.Diagnostic.error("demo:install", f"error-{index}"))
assert [diagnostic.message for diagnostic in live] == [
"error-2",
"error-3",
"error-4",
]
finally:
extensions_module.reset_runtime_diagnostics()