7 Commits

Author SHA1 Message Date
Bas Cancrinus
3751eb8daf
feat(extensions): pass a task's final status, error, goal verdict and run metadata to on_task_stop (#8304)
* feat(extensions): pass a task's final status, error, goal verdict and run metadata to on_task_stop

A lead run's final status is decided after the graph has ended: a model
that fails after retries ends the graph normally, so callback-based
tracing reports the run as OK. on_task_stop runs after the status is
persisted but only received completed | failed | aborted.

Additive, so existing contributors keep working:
- TaskInfo.metadata: a read-only copy of the run's metadata (scheduled
  runs carry scheduled_task_id, scheduled_task_run_id, scheduled_trigger);
  empty for subagents.
- TaskInfo.stop: a new TaskStopDetail (status, error, goal_verdict), set
  only on the info passed to on_task_stop.
- deerflow-extension-api 0.2.13.

Refs #8303

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(subagents): compare stop identity without the stop detail; assert its status

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extensions): keep TaskInfo identity and serialization; flag unconfirmed stop status

Review follow-up for #8304:
- Exclude TaskInfo.stop and .metadata from equality and hashing
  (compare=False), so the stop info finds state keyed by the start info.
- Replace MappingProxyType with ReadOnlyDict, a dict subclass that rejects
  mutation and freezes nested mappings (lists become tuples). asdict(),
  deepcopy, pickle and json.dumps work again; observers sharing one
  snapshot can no longer change nested values for the next observer.
- TaskStopDetail.persisted: False when the run store did not confirm the
  terminal write or the worker lost ownership, so a staged status is not
  reported as final.
- Document waiting_for_input as a resumable pause (outcome failed) and the
  persisted flag (en/zh observers and reference).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
2026-11-17 19:57:40 +08:00
Nan Gao
1dd6d05627
fix(extensions): hand compaction observers the live task store and name the task in CompactionEvent (#7831)
* feat(extension-api): add task identity fields to CompactionEvent (0.2.10)

CompactionEvent carried the transform, its hashes and its counts, but
nothing that names the task, run or thread the compaction happened in.
An observer that wants to relate a compaction to the run that produced it
had to match hashes across sessions, which is unreliable.

Append three optional fields, task_id, run_id and thread_id (str | None,
default None). Existing keyword and positional construction is unchanged,
so already-released extensions keep working. The host fills them in the
next commit.

Bump deerflow-extension-api to 0.2.10 and the harness's exact pin with it.

* fix(extensions): hand compaction observers the live task store and its identity

The summarization middleware already recovers the live task store from
the runtime to observe its summary system call, and the registry
allocates that store precisely because a compaction observer is
registered. notify_context_compacted() then discarded it and handed every
observer a detached store, with a docstring claiming there was no live
task.

- The lead run worker and the subagent executor now seed every task store
  they allocate with the task's TaskInfo, not only when a lifecycle
  contributor is registered; the lifecycle notification reuses the seeded
  object. A subagent without a parent run_id still gets no TaskInfo.
- compact_state() and acompact_state() pass the store from
  task_store_from_runtime() to _record_compaction().
- notify_context_compacted() takes an optional task_store. A live store
  goes to the observers and fills the event's task_id/run_id/thread_id
  from its TaskInfo; without one (manual /compact runs outside any run)
  observers still receive a detached store and the ids stay None.

For a background subagent the store keeps its public scope id (the
provider tool_call_id) while TaskInfo.task_id stays the execution id, as
lifecycle contributors already see it; the event follows TaskInfo.

* docs(extensions): document compaction event identity and seeded TaskInfo

The extensions guide and the developer manual still said compaction
observers always receive a detached store because no task is live at the
call site. Describe what the host does now: lead runs and subagents with a
parent run seed every task store with TaskInfo, compaction observers get
the live store, and CompactionEvent's task_id/run_id/thread_id come from
it; only manual /compact falls back to a detached store with None ids.
Adds the 0.2.10 row to the contract version table (PR number pending).

* docs(changelog): note compaction observers receive the live task identity

* docs(extensions): list compaction observers among task-store triggers

* docs: reference #7831 for the compaction event identity change

* docs(extensions): backfill contract history rows 0.2.5 to 0.2.9

The version table jumped from 0.2.4 to 0.2.10 although 0.2.5-0.2.9
shipped. Add those rows, attributing names merged without a bump to
the first release that carried them (#5796 in 0.2.5, #6920 and #5645
in 0.2.8, #7729 in 0.2.10). The page intro now says it documents
0.2.4 plus the 0.2.10 CompactionEvent fields, and points to the
history for the rest.
2026-11-03 15:06:52 +08:00
NanPan
e1430c832c
fix(extensions): make notify dispatch admission atomic with shutdown (#7383)
* fix(extensions): fence notify admission against shutdown

* test(extensions): cover notify admission shutdown race

* docs(extensions): record atomic notification admission

* style(tests): format extension notify shutdown regression

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-24 11:36:06 +08:00
sxh
595000ac89
fix(extensions): classify a spent notification budget without a clock recheck (#6469)
The extension notify loop told a spent shared budget apart from a
TimeoutError a contributor raised on its own by re-reading loop.time()
after the await and comparing it with the deadline. The event loop runs a
scheduled timeout up to one clock_resolution early (BaseEventLoop._run_once
compares each handle's _when against now + clock_resolution), so on a coarse
clock -- Windows' ~15.6 ms against budgets as small as 20 ms -- that read can
still land strictly before the deadline. Budget exhaustion was then logged as
a phantom hook failure (ERROR plus an asyncio-internal traceback), and the
same lag let a later contributor run on a budget that had already run out.

Classify from asyncio.timeout's own expiry (expired()) instead of a second
clock read, and latch budget_spent once the budget is gone so the skip of
later contributors is deterministic too.

Fixes #6453

Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
2026-10-08 10:22:48 +08:00
NanPan
a78ff4817f
fix(extensions): drain pending notify dispatches on shutdown (#6357)
* fix(extensions): drain pending notify dispatches on shutdown

* fix(gateway): await extension notify drain before reset

* test(extensions): retain pending notify dispatches on shutdown

* fix(extensions): bound and close notify shutdown drain

* test(extensions): cover bounded notify shutdown drain
2026-10-06 20:35:54 +08:00
Nan Gao
13f0a7f263
feat(extensions): let an out-of-tree extension observe what the agent did (#4863)
* feat(extensions): let an out-of-tree extension observe what the agent did

DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:

  * The middleware chain injects and rewrites a lot of context — date
    reminders, recalled memory, compaction summaries, durable-context data,
    image payloads, activated skill bodies. Downstream, none of it is
    attributable: at the model-call boundary an injected HumanMessage is
    indistinguishable from the user's own, and anything wanting to tell them
    apart has to pattern-match prompt wording, which breaks on the next copy
    edit.

  * Two runs of "the same agent" are only comparable if the chain enforced the
    same limits, prompts and thresholds. Recovering that from outside means
    reading private attributes and guessing which of them change behaviour — a
    guess that rots silently as middlewares gain fields.

  * The lead-agent factory resolves a model after runtime overrides, renders a
    prompt, filters tools through authorization and composes a stack, all
    inside one synchronous call, and none of it survives: a middleware sees its
    neighbours but not the prompt, the run worker sees a graph but not what
    went into it.

  * Summarization is destructive by design. N messages leave the context and
    one summary enters it; afterwards only the summary exists, so "which
    messages became this?" is not reconstructible.

This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.

Message provenance
  Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
  messages they inject or rewrite. Stamping is unconditional — a fact whose
  presence depends on whether an observer is installed is not a fact — and the
  keys are server-owned, so provenance cannot be forged from a request.

Middleware self-description
  Twelve middlewares declare their own behaviour-affecting parameters through
  a duck-typed `release_policy_parameters()`. Long text is hashed rather than
  embedded: a declaration is an identity, not a copy of the prompt.

Agent assembly descriptor
  `assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
  answers "did anything about this agent change between these two runs?".
  `make_lead_agent()` keeps its graph-only signature — it is the LangGraph
  Server ABI declared in langgraph.json. Tools and skills are sorted before
  hashing because their assembly order is incidental; middlewares are not,
  because stack order decides what wraps what. Host build identity is reported
  but excluded from the fingerprint, so a redeploy does not invalidate every
  agent's identity.

Context compaction observation
  Summarization emits the content hashes of the messages it is about to remove
  joined to the summary that replaced them. Content is the only identity
  available at that seam: the summary does not become a message, and what later
  projects it into a request renders it bounded and escaped rather than
  verbatim.

Neutral policy, transform and MCP-source facts
  Guardrail decisions are published to runtime context under a `__`-prefixed
  key; result-rewriting middlewares append a declared, ordered transform trail;
  MCP tools carry their credential-free logical origin.

Extension route identity
  Contributed routes are session-authenticated and cannot opt out, but
  "logged in" and "administrator" are different questions. Extensions get a
  neutral projection of the caller rather than the host's auth context, and
  `require_admin` fails closed when identity cannot be determined.

Extension-owned tables
  An extension that persists data owns its own MetaData and migration chain, so
  its tables are absent from Base.metadata and `alembic revision --autogenerate`
  proposes dropping them. Extensions declare a table prefix, which is rejected
  at registration if it would shadow a host table.

The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.

uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.

* fix(backend): sort gateway service imports
2026-08-23 09:57:12 +08:00
Nan Gao
7389331e65
feat(extensions): observe task lifecycle and system model calls (#4684)
* feat(extensions): observe task lifecycle and system model calls

PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.

Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).

`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.

Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.

Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.

System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.

The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.

Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.

Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.

* fix(extensions): decide notification fail-open by origin, observe cancellation

`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.

`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.

DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.

* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown

Review response on #4684:

- The memory observation bridge caught BaseException, which would swallow
  a teardown signal raised while dispatching; it now catches Exception,
  matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
  hook-failure path and logged an asyncio-internal traceback; it now logs
  a warning like the pre-hook budget skip, while a TimeoutError a
  contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
  so log lines said "task goal/title/..."; they now carry the task
  scope id alongside the kind.
2026-08-11 16:33:22 +08:00