Refresh the locked lark-oapi from 1.5.5 to 1.7.3 without touching the
declared >=1.4.0 constraint, which already admits the new release.
Deployments syncing from the lockfile move off the 1.6.x line, where a
server-side protocol drift reportedly left long-connection clients
connected but receiving no event pushes. Keeping the constraint as-is
avoids shutting out downstream consumers pinned to older but working
releases (1.5.5's WS event routing, including the callback processor map
used by register_p2_card_action_trigger, is verified intact in source).
- Refresh uv.lock only: lark-oapi 1.5.5 -> 1.7.3, specifier unchanged
- Feishu channel, channel-connections, and thread-binding suites pass
on 1.7.3 (111 passed, 1 skipped); ruff clean
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): exclude thinking blocks from Claude cache breakpoints
* fix(models): declare Claude test SDK and clarify caching docs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add host-bound Agent run control
* fix(extensions): preserve internal launches and scope run control
* docs(extensions): use a plugin-local run operation key
* chore(version):update the version to 2.2.0-dev
* chore(doc):Updated the CHANGLOG with the 2.1.0 release version
* chore(doc): Updated the CHANGLOG for the latest changes
* chore(doc): Added the release SOP document
* feat(authz): add plugin resource targets, provenance, action and management enforcement
PR1 of the extension-tool-authorization series
(docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin
authorization plumbing, no new author-facing declaration.
- `authz/plugin_targets.py` is the single encoder/validator for composite
targets (`namespace/part`), with one validating composer behind three
kind-checked constructors; invalid input raises instead of degrading.
- `authz/plugin_authz.py` is the harness decision layer (`plugin_action`,
`plugin_management`, plus the batch filters the page slice will consume):
disabled is a no-op, an explicit deny raises, and provider errors follow
`fail_closed` like the sandbox gate.
- `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke`
after the action resolves and before the body is streamed.
- `deerflow_extension_api.auth` publishes `require_plugin_management` /
`arequire_plugin_management` with two `app.state` resolver keys; the
Gateway installs both. Contract version 0.2.3 -> 0.2.4.
- `tools/tool_provenance.py` records host-written plugin provenance for
display (assembly descriptor, Layer-2 context later); no decision reads a
label.
- Plugin providers use a `(config signature, loop key)` cache so a
loop-affine provider is never handed to another loop or to a sync caller.
- RBAC gains `plugin_actions` / `plugin_management` aliases;
`config_version` 47 -> 48 across the three mirrors.
Tests: new plugin target/provenance/action/management suites plus a
blocking-IO anchor whose offloads were reverted to prove red->green.
* fix(authz): decide plugin resources from the request's config snapshot
Review round 1 on PR #5842 (willem-bd).
P1 — the decision layer no longer reloads the configuration. The six
`authz/plugin_authz.py` functions take a required `app_config` snapshot (the
same request-scoped config the Gateway used to resolve the provider), and
`resolve_/aresolve_plugin_authorization` return it as a third element, so
`enabled`/`fail_closed` can never come from a second read that drifted from
the one that produced the provider. A snapshot the caller could not read at
all is *unavailable*, not disabled: it fails closed as
`authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates
an absent `config.yaml` (no gate, today's behavior) from a config that exists
but cannot be read or validated (deny) instead of swallowing both through
`safe_app_config`.
P2 — `_validated_decision` requires `type(decision.allow) is bool` on both
the sync `authorize` and async `aauthorize` paths, so a provider returning
`AuthzDecision(allow="false")` is a malformed verdict that follows
`fail_closed` instead of being read as an allow.
Both fixes are covered by regression tests that were verified red against the
pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases
across fail_closed/fail_open and sync/async), plus tests pinning that an
absent config and `authorization.enabled: false` keep today's behavior.
* fix(authz): treat a request-time config absence as unavailable
Review round 2 on PR #5842 (willem-bd).
P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to
`None` ("no gate"). A Gateway cannot have started without a readable config
(`app/gateway/app.py` reads it through this same accessor in `lifespan`), so
absence *during a request* means the configuration this process was running on
became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is
unavailable, not disabled: the error propagates,
`resolve_/aresolve_plugin_authorization` raise
`_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher
returns 403 and the management resolvers answer `False`. The dead
`(None, None, None)` allow path is gone, the resolvers return a non-optional
`AppConfig` as their third element, and callers key "authorization disabled" on
`provider is None` alone. This matches the rule the decision layer already
documented (`authz.config_unavailable`) and the public docs.
P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the
code to `authz.denied` instead of raising `TypeError` outside the guarded
provider path, which turned a denial into an unhandled error. It is deliberately
not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict
is a valid denial, and treating it as malformed would flip it to an allow under
`fail_closed: false`.
Both fixes are covered by regression tests verified red against the pre-fix
behavior (P1: resolver True vs False, async `PermissionError` not raised, route
200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at
`plugin_authz.py:113`). The action-route tests in
`tests/test_plugin_contributions.py` now install a readable
`authorization.enabled: false` config instead of relying on a config-less
process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the
new rule.
* fix(authz): apply the running policy when the plugin config is unreadable
Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd).
The first attempt read the P1 finding as "absence is never disabled" and failed
every config read closed unconditionally. That is too broad for this repo: a
config-less host is deliberately treated as having no policy to apply
(`_get_route_authorization_config` falls back to a disabled config,
`sandbox_authz.safe_app_config` documents the same no-op for "CI runners and
direct-call tests"), and the rule turned the config-less e2e plugin host
(`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three
`bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally:
`POST /api/plugins/community.bookmarks/actions/save` against that fixture host
returned 403 before, 200 after this change.
The rule is now "has this process loaded a configuration?", not "was the last
read successful?":
- No loaded config (`peek_loaded_app_config()`, new read-only accessor on
`deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the
gate stays a no-op — unchanged for CI runners, direct-call tests and hosts
that mount only the plugins router.
- A loaded config whose file disappears or cannot be read: *unavailable*, never
*disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the
failure flag of the configuration this process runs on (authorization
disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by
default), so an enabled Gateway answers 403/False in the window instead of
letting protected plugin actions and management routes through.
- A config file that exists but cannot be read or validated right now: fail
closed when nothing is loaded (round 1's rule), otherwise the loaded policy
decides.
`peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it
reflects the last successfully loaded config, so a hot-reloaded policy is what
applies during the window, and there is no lifecycle hook to keep in sync.
Evidence: the three no-config/lost-config branches in
`tests/test_plugin_management_guard.py` and
`tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests
unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in
`tests/test_app_config_reload.py`, and the fixture-host probe above. Docs
(`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log
state the final rule; the log records the superseded variant.
* fix(authz): keep a denial whose reasons raise while being read
* fix(authz): intersect plugin batch filters with the candidates
The plugin_page/plugin_management batch helpers returned the provider's
`filter_resources` answer verbatim, so a custom provider could hand the
projections a target that was never asked about ("*", another namespace,
a phantom page). Every other filter call site re-projects from its own
candidate list; do the same here, state the intersection on the provider
contract, and pin it with a regression test that fails on the old
pass-through.
* docs(plugins): map a management denial to 403 in the route example
The contributed-route example called `arequire_plugin_management` and let
the documented `PermissionError` escape. The Gateway installs no
`PermissionError` handler, so a route copied from the example answered 500
on a normal policy denial instead of 403. Both example routes now translate
the denial, and the surrounding bullets say why the translation is the
route's job.
* docs(extensions): point the 0.2.4 contract row at the PR that ships it
Every other row in the version-history table links the pull request that
shipped the version; 0.2.4 linked the authorization tracking issue instead.
Both the English and Chinese tables now link this PR.
* chore(checkpoint): bump to checkpoint 4.2.0 and drop the stale saver patch
langgraph-checkpoint 4.2.0 fixes the write dropped after a full -> delta
migration (upstream langchain-ai/langgraph#8526) and keeps its own
InMemorySaver override, so the compatibility patch in checkpoint_patches.py
could neither detect the fix nor stand down: it read langgraph's version
while InMemorySaver ships in langgraph-checkpoint, and it would have kept
shadowing the upstream implementation on 4.2.0. The postgres release 3.1.2
locates plain-value delta seeds (upstream #8535).
- floor langgraph-checkpoint>=4.2.0,<5.0 in the harness and raise the postgres
extra to >=3.1.2,<3.2; langgraph's own >=4.1.0 would otherwise let a
resolver reintroduce the bug now that the patch is gone
- delete the patch, its version guard, and the three implementation-assertion
tests; test_full_to_delta_migration_replays_on_same_thread stays as the gate
and fails on 4.1.1 without the patch (verified by a one-off unpatch run)
- langgraph stays 1.2.9 and langgraph-checkpoint-sqlite stays 3.1.1
Verification: make lint; delta checkpointer/cache/worker-resume suites green
with the migration gate passing on memory and sqlite (postgres params skip
without TEST_POSTGRES_URI). The full make test run before the packaging-pin
fix below was 13 failed / 18523 passed; 12 of those failures reproduce on main
with the same environment (git stash baseline, identical failure sets), 7 of
them because the local config.yaml runs checkpoint_channel_mode: delta. The
remaining one, the postgres-extra string pin in tests/test_checkpointer.py, is
updated here.
Upstream status for the surrounding gaps: cache parity across off/cold/warm
states, postgres pagination (langchain-ai/langgraph#8448) and parallel-superstep
replay order (#8382) are still open upstream, as is the abandoned-branch fork
fix (#8548) that the delta resume linearization in runtime/runs/worker.py works
around, so that path stays.
* docs(changelog): keep checkpoint upgrade in Unreleased
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): expose incremental run evidence reader
* fix(extensions): address run evidence review feedback
* docs(extensions): clarify run deletion reconciliation
* docs(migrations): align current head documentation
* fix(extensions): isolate run evidence event reads
* test: avoid pinning run change migration to latest head
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): detect the ollama extra from configured models
make dev synced without --extra ollama, uninstalling langchain-ollama from a working setup. Same failure #2754 hit with postgres; ollama predates the detector added in #2767 and never got a rule.
* fix(deps): declare the ollama extra on backend, pin model use: matching
Review follow-ups.
`ollama` was the only extra in the detector's map that the root `backend`
project did not declare, so any consumer syncing without
`--all-packages` failed outright:
$ cd backend && uv sync --locked --extra ollama
error: Extra `ollama` is not defined in the project's
`optional-dependencies` table
`serve.sh` and `docker/dev-entrypoint.sh` both pass `--all-packages` and
were unaffected, but `backend/Dockerfile` does not, and
`config.example.yaml` documents `UV_EXTRAS` as an image build-arg — so
`UV_EXTRAS=ollama docker compose build` would have hard-failed on a
value this branch makes first-class. Declaring
`ollama = ["deerflow-harness[ollama]"]` alongside the other delegating
extras closes that, and `uv.lock` is regenerated to match.
The `use:` match also accepted any nesting depth inside `models:`, so a
`use` in a sub-mapping was read as the model's provider:
models:
- name: doubao
use: deerflow.models.patched_deepseek:PatchedChatDeepSeek
when_thinking_enabled:
use: langchain_ollama:ChatOllama
That yielded `--extra ollama` despite the model's own provider pointing
elsewhere, and `when_thinking_enabled` appears fifteen times in
config.example.yaml, so the shape is common rather than contrived. Pin
matching to the list item's own key indent, mirroring how
`section_value()` pins `child_indent` and documents deeper nesting as
ignored on purpose. Covered by a regression test that fails on the
looser parser.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(deps): drop unrelated standard-aifc marker churn from uv.lock
Review follow-up. Regenerating the lock with a newer local uv (0.11.19)
also rewrote the `standard-aifc` entry, adding
`python_full_version >= '3.13'` markers to its `audioop-lts` and
`standard-chunk` dependencies. Unrelated to this change, so restore
upstream's lines and keep the lock diff to the `ollama` extra.
`uv lock --check` passes with these lines under both the CI- and
Dockerfile-pinned uv 0.11.1 and uv 0.11.19.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(scripts): detect ollama in unindented model lists and after nested sequences
Review follow-ups on the models parser.
`yaml.safe_dump` — used by the setup wizard (`make setup`) and
scripts/config-upgrade.sh — writes list items unindented:
models:
- name: qwen3-local
use: langchain_ollama:ChatOllama
The parser treated any column-0 line as the end of the `models:` section,
so the first model ended it and nothing was detected. That is the layout
new users get from the recommended setup path, so `make dev` still synced
without `--extra ollama`. Model entries are now recognised before the
section-end test, the same ordering `tools_include_name()` already uses
for the unindented tools list (#4367).
Separately, every sequence item reset the key indent, including items
inside a model option. With `stop:` / `- END` before `use:`, the key indent
jumped to the nested item's and the model's own `use` was skipped, so
detection depended on key order within the model.
The first sequence item under `models:` now fixes the model-list indent;
only items at that indent start a model and set where its keys sit.
Nested sequence items and deeper mappings are skipped without moving it.
Regression tests cover the real setup-wizard output via
`build_minimal_config()`, a hand-written unindented list, and `use:`
after a nested `stop:` list — all three fail on the previous parser — plus
a guard that a `- use:` nested inside a model option is still ignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(web-fetch): resolve relative URLs in extracted Markdown
Pass the request URL through Jina and Browserless extraction and resolve
link/image destinations before Readability removes document base tags.
Preserve the optional legacy API and fallback text behavior.
Fixes#5307
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(web-fetch): address provider and base URL review feedback
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(web-fetch): preserve HTML source when resolving destinations
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(agents): make the injected current-date timezone configurable
## Why
The date reminder injected into the lead and subagent prompts (DynamicContextMiddleware / SubagentDateContextMiddleware) was formatted with the server's local wall clock. DeerFlow containers default to UTC, so a user in Asia/Shanghai chatting in the 00:00-08:00 window was told that 'today' is the previous day - the model then reasons, plans, and date-stamps against the wrong day.
## What changed
- _format_current_date() now reads the optional DEER_FLOW_DATE_TIMEZONE env var (IANA name, e.g. Asia/Shanghai) and renders the date in that zone.
- Unset = unchanged server-local behavior; invalid names log a warning and fall back to server-local.
- Documented the knob in config.example.yaml, the module docstring, and the DynamicContext entry in agents/middlewares/AGENTS.md.
## Surface area
- [x] Agents / LangGraph - prompt-layer date context only; message shape and midnight-update behavior unchanged
- [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies
- [x] Default behavior change (opt-in via env var - no behavior change unless set)
## Bug fix verification
- New tests: test_format_current_date_honors_configured_timezone (UTC 20:30 -> 2026-09-03 in Asia/Shanghai), test_format_current_date_defaults_to_server_local_without_env, test_format_current_date_invalid_timezone_falls_back.
- Existing mocked-datetime tests pass unchanged (no env -> datetime.now() path).
## Validation
- cd backend && python -m pytest tests/test_dynamic_context_middleware.py: 31 passed.
- blocking_io/test_dynamic_context_middleware.py: 2 pre-existing abefore_agent failures reproduce identically on clean main (blockbuster os.listdir detection on this host); the other 2 pass.
- ruff format + ruff check clean.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
* fix(agents): avoid passing tz to datetime.now when no timezone is configured
CI (backend-unit-tests shard 2) failed in test_tool_error_handling_middleware.py::test_subagent_chain_injects_date_without_memory_and_coalesces_for_strict_provider because its _FrozenDateTime.now() subclass override accepts no arguments, while _format_current_date() called datetime.now(None) even when DEER_FLOW_DATE_TIMEZONE was unset.
- _format_current_date() now calls datetime.now() with no arguments unless a timezone is actually configured, preserving the exact legacy call shape for every datetime-subclass test fake.
- The configured-zone path still calls datetime.now(tz) and converts via astimezone(tz).
- Updated the no-env unit test to assert datetime.now() is called without arguments.
Validation: python -m pytest tests/test_dynamic_context_middleware.py + the previously failing strict-provider test: 32 passed. ruff clean.
* fix(agents): declare the effective current-date timezone in the assembly descriptor
## Why
Maintainer review on the DEER_FLOW_DATE_TIMEZONE change (#5154): the knob is
prompt-affecting, yet both DynamicContextMiddleware and SubagentDateContextMiddleware
were invisible to the agent assembly descriptor - describe_middleware() fell back to
{"probed": true} for unset, UTC, and Asia/Shanghai alike, so deployments that inject
different dates shared one assembly fingerprint and release observers could not
distinguish or audit the behavior change.
## What changed
- Both middlewares now implement release_policy_parameters() -> dict[str, object],
declaring {"current_date_timezone": <name>} as required by the module's middleware
self-description contract.
- The declared value is the normalized effective zone: a configured, valid
DEER_FLOW_DATE_TIMEZONE is reported by its IANA key (ZoneInfo.key); otherwise the
server-local zone is resolved to its IANA key when the platform exposes one and to
its tzname label otherwise (fixed-offset hosts), with "UTC" as the final fallback.
- Added both middlewares to _MIDDLEWARE_DECLARATIONS in
backend/tests/test_middleware_release_policy.py so the existence check and the
construct-and-canonical-hash check cover them.
## Verification
- New tests: test_date_middlewares_declare_configured_timezone (Asia/Shanghai),
test_date_middlewares_declare_utc_timezone, plus resolved-server-local assertions
for the unset and invalid-env paths; both middlewares agree in every case.
- cd backend && python -m pytest tests/test_dynamic_context_middleware.py
tests/test_middleware_release_policy.py: 70 passed.
- Regression spot-check: tests/test_agent_assembly_descriptor.py,
tests/test_tool_error_handling_middleware.py, tests/test_system_message_coalescing_middleware.py:
102 passed.
- ruff check + ruff format clean.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
* fix(agents): stabilize the declared date timezone and simplify the formatting path
## Why
Follow-up review on #5154 (willem-bd). The release-policy declaration added in
884cec4b resolved the observability gap but pinned far less identity than its
docstrings claimed, and the formatting path carried a production no-op.
## What changed
- The declared label is now stable and unambiguous: a configured, valid
DEER_FLOW_DATE_TIMEZONE is reported by its IANA key; without one, the
server-local zone is resolved to a real IANA key from the TZ env var or the
/etc/localtime symlink (Linux/macOS); when no key is recoverable (Windows,
stripped containers) the declaration falls back to a stable
`server-local(+-HH:MM)` sentinel carrying the current UTC offset. It never
reports a bare abbreviation - datetime.now().astimezone() yields only a
fixed-offset timezone whose tzname (e.g. CST, EST/EDT, CET/CEST) is
ambiguous or DST-churns, which the assembly descriptor docstring says must
not happen.
- Dropped the redundant astimezone(tz) in _format_current_date():
datetime.now(tz) already returns the instant expressed in tz. The
configured-zone test now fakes datetime.now(tz) semantics (the fixed instant
converted into the requested zone) instead of relying on that conversion.
- Documented why the knob is an env var, not a config-schema field: it is read
at runtime by both date-context middlewares so an operator can point a
container at another zone without mounting a config.yaml (module docstring +
config.example.yaml note).
- AGENTS.md: fixed the glued DynamicContext sentence (missing separator).
- Added tzdata>=2025.1 to the harness runtime dependencies (with uv.lock) so
ZoneInfo works on stripped containers / Windows without an OS zone database.
## Verification
- New tests: test_server_local_timezone_name_reads_tz_env,
test_effective_timezone_sentinel_uses_offset_when_local_zone_is_not_resolvable;
reworked test_format_current_date_honors_configured_timezone to exercise the
real datetime.now(tz) path.
- cd backend && python -m pytest tests/test_dynamic_context_middleware.py
tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py
tests/test_tool_error_handling_middleware.py: 140 passed.
- ruff check + ruff format clean.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
* fix(agents): offload subagent date injection off the event loop
## Why
Follow-up review on #5154 (willem-bd, P2): SubagentDateContextMiddleware.abefore_agent()
called _inject() directly, so enabling DEER_FLOW_DATE_TIMEZONE could synchronously
read the OS timezone database (or the tzdata wheel) on a cold cache - filesystem
work on the async subagent execution path whenever no assembly observer resolved the
zone first.
## What changed
- SubagentDateContextMiddleware.abefore_agent() now offloads the injection via
asyncio.to_thread with the same bounded timeout DynamicContextMiddleware uses
(issue #3402); on timeout it logs and skips the date update for that run instead
of blocking the loop.
- Narrowed the exception handling in _date_timezone() and the TZ-env branch of
_server_local_timezone_name() to configuration-shaped failures
(ZoneInfoNotFoundError / ValueError / OSError). Previously a blanket
`except Exception` also swallowed BlockingError raised by the blocking-I/O
regression gate, mislabeling a loop-blocking call as an invalid timezone and
silently degrading to server-local - which made the new regression anchor
useless. Other exceptions now propagate.
## Verification
- New blocking-I/O regression anchor
(backend/tests/blocking_io/test_subagent_date_context_middleware.py): drives a
real create_agent graph under the strict Blockbuster gate with the knob enabled
and asserts the date reminder is injected. Verified it fails (BlockingError) when
the offload is reverted and passes with it in place.
- python -m pytest tests/blocking_io/test_subagent_date_context_middleware.py:
1 passed. The two pre-existing os.listdir failures in
tests/blocking_io/test_dynamic_context_middleware.py reproduce unchanged on this
host (same as clean main).
- python -m pytest tests/test_dynamic_context_middleware.py
tests/test_middleware_release_policy.py tests/test_tool_error_handling_middleware.py
tests/test_agent_assembly_descriptor.py: 139 passed; the single
ToolReceiptMiddleware-ordering failure reproduces with the change stashed
(local extensions registry, unrelated to this PR).
- ruff check + ruff format clean.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
* fix(agents): read the direct /etc/localtime symlink target for the zone key
## Why
Follow-up review on #5154 (willem-bd, P2): on macOS, /etc/localtime commonly
points to /var/db/timezone/zoneinfo/<zone>, but Path.resolve() follows that
directory's own symlink and yields a versioned path such as
/private/var/db/timezone/tz/2026c.1.0/zoneinfo/Asia/Shanghai, which matched no
configured prefix. The server-local resolution then returned None and the
assembly descriptor fell back to a server-local(+HH:MM) sentinel even though
the IANA key was available - conflating zones that share an offset and making
DST-based fingerprints unstable.
## What changed
- _server_local_timezone_name() now reads the direct symlink target via
os.readlink("/etc/localtime") instead of Path.resolve(), so macOS' unversioned
zoneinfo path is seen as-is and its IANA key is preserved.
- The zone key is taken from whatever follows the last "/zoneinfo/" segment,
which also handles Apple's canonical versioned path when a direct target
already carries it, and relative targets are normalized against /etc.
- Removed the now-unused Path import and the fixed zoneinfo prefix tuple.
## Verification
- New tests: test_server_local_timezone_name_reads_direct_macos_symlink_target,
test_server_local_timezone_name_reads_apple_versioned_symlink_target, and
test_server_local_timezone_name_normalizes_relative_symlink_target.
- python -m pytest tests/test_dynamic_context_middleware.py
tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py:
105 passed (75 after re-running the first two on the merged main). The blocking
subagent anchor still passes; the two pre-existing os.listdir blocking failures
on this host are unchanged.
- ruff check + ruff format clean.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(ci): split backend unit tests into parallel CI shards
Split the single offline backend `make test` job into four GitHub Actions
matrix shards (SPLITS=4, GROUP=1..4) via pytest-split, so the ~12k-test suite
runs in parallel instead of in one 15-minute job. Each shard runs on its own
runner with its own Postgres/Redis services; fail-fast: false lets a failing
shard report its owned tests without cancelling its peers.
`make test` stays the canonical full-suite entry point; CI now calls the new
`make test-shard SPLITS=4 GROUP=N`. tests/blocking_io remains owned solely by
the dedicated blocking-I/O workflow (excluded via --ignore), extending #5105.
Fixes#5088
* test(ci): make backend test shards duration-aware and pin the contract
Make `make test-shard` an explicit least_duration split that READS
backend/.test_durations (read-only for shards, so concurrent CI jobs never
race writes on it), and add `make test-shard-durations` to regenerate that
file from the full offline suite. Update the CI unit-test workflow contract to
call `make test-shard SPLITS=4 GROUP=<n>` and assert the shard command carries
--splits 4, --group 2, -m "not live", --ignore=tests/blocking_io and
--splitting-algorithm least_duration. Verified on the real 13,140-test normal
suite that the four shards are pairwise disjoint and their union equals the
unsplit suite.
Refs #5088
* test(ci): fail fast when the duration baseline is missing
`make test-shard` now requires backend/.test_durations and exits with a clear
error instead of letting pytest-split silently degrade to an even (count-based)
split. Harden the CI contract test to pin `--durations-path=.test_durations` and
to assert the repo ships the committed duration baseline.
Refs #5088
* docs: trim backend/AGENTS.md within guidance budget
* test(ci): add backend test duration baseline
Add the duration baseline generated by a full offline backend run on a
GitHub-hosted ubuntu-latest runner (the same runner type the shards use), so
`make test-shard` balances the four matrix shards by real wall-clock cost.
Refs #5088
* test(ci): make the duration writer honor DURATIONS_FILE
`test-shard-durations` now writes `--durations-path=$(DURATIONS_FILE)` instead of a
hard-coded .test_durations, so the reader and writer stay consistent when the
path is overridden.
Refs #5088
* test(ci): address sharding review feedback
* test: isolate subagent execution capacity state
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deps): depend on renamed tenki package instead of tenki-sandbox
tenki-sandbox has been removed from PyPI and republished as tenki. Its old wheel URL still resolves, so existing lockfiles keep installing and the breakage is invisible to anyone with a warm lock; any fresh resolution fails with 'tenki-sandbox was not found in the package registry'.
tenki 1.0.2 still ships the tenki_sandbox module, so the imports in community/tenki/provider.py and sandbox.py are unchanged.
Fixes#5081
* fix(tenki): point install guidance at the renamed distribution
The rename to `tenki` left the user-facing remediation still naming the
removed package. `_import_client` raised "pip install tenki-sandbox" on the
missing-extra path — the exact instruction this change proves now 404s on
PyPI, handed to the user at the exact moment they need it to work.
Update that message and the remaining `tenki-sandbox` references in the
provider, sandbox adapter, README, sandbox AGENTS.md and the test docstring.
The imported module stays `tenki_sandbox`, so the distribution and module
names now differ; each mention says so rather than just swapping the string.
No behavior change beyond the error text.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tenki): migrate the provider to the 1.x workspace-only API
Renaming the dependency was not enough. tenki 1.0.2 keeps the tenki_sandbox
module name but not its contract: Client.create dropped project_id and has no
**kwargs to absorb it, and IdentityWorkspace no longer carries `projects`
(the attribute is gone from the package entirely). Both configuration paths
therefore failed before a sandbox could be created — explicit project scope
raised TypeError, and automatic scope raised AttributeError walking
workspace.projects.
Scope is now the workspace alone. _resolve_scope returns a single workspace id,
auto-selecting when the account has exactly one, and project_id is gone from
create_kwargs and from the documented config surface.
A stale project_id in config.yaml warns rather than fails. SandboxConfig is
extra="allow", so simply not reading the key would leave it scoping nothing
with no signal; it also used to short-circuit the identity lookup, so operators
with more than one workspace need to know they must now set workspace_id.
The suite passed against the broken provider because the fake client took
**kwargs and swallowed the project_id the real SDK rejects. The double now
mirrors 1.0.2 — keyword-only, no **kwargs — so an unexpected argument is a
TypeError in tests exactly as it is against the SDK. Reintroducing the old
create call fails 20 tests; before this change it failed none.
Verified against the exact locked wheels: every other kwarg the provider
passes (name, workspace_id, sticky, wait, max_duration, image, cpu_cores,
memory_mb, env) and every SDK surface it touches (who_am_i, Identity.workspaces,
wait_ready, exec, close, the fs API, the four terminal exception classes) is
unchanged in 1.0.2.
Reported by willem-bd in review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(config): drop sandbox.project_id from the Tenki example
The canonical example still documented project_id as a supported optional key
after the provider stopped honouring it, so an operator following it could set
the key, get no scope from it, and hit a workspace-resolution failure with
nothing in the example to explain why.
Replaced with a migration note rather than a silent deletion: someone upgrading
already has the key in their config.yaml and needs to know it is inert now and
that workspace_id is what scopes a sandbox on Tenki 1.x.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Aniket Wagh <aniketwaghh@users.noreply.github.com>
* feat(extensions): let an out-of-tree extension observe what the agent did
DeerFlow's extension system can contribute middleware, services and routes,
but an extension cannot answer basic questions about a run without reaching
into host internals. Several of the facts it would need are destroyed by the
operations that produce them:
* The middleware chain injects and rewrites a lot of context — date
reminders, recalled memory, compaction summaries, durable-context data,
image payloads, activated skill bodies. Downstream, none of it is
attributable: at the model-call boundary an injected HumanMessage is
indistinguishable from the user's own, and anything wanting to tell them
apart has to pattern-match prompt wording, which breaks on the next copy
edit.
* Two runs of "the same agent" are only comparable if the chain enforced the
same limits, prompts and thresholds. Recovering that from outside means
reading private attributes and guessing which of them change behaviour — a
guess that rots silently as middlewares gain fields.
* The lead-agent factory resolves a model after runtime overrides, renders a
prompt, filters tools through authorization and composes a stack, all
inside one synchronous call, and none of it survives: a middleware sees its
neighbours but not the prompt, the run worker sees a graph but not what
went into it.
* Summarization is destructive by design. N messages leave the context and
one summary enters it; afterwards only the summary exists, so "which
messages became this?" is not reconstructible.
This adds seven neutral facilities so those facts are recorded where they are
still true, and releases the contract package as 0.2.0.
Message provenance
Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the
messages they inject or rewrite. Stamping is unconditional — a fact whose
presence depends on whether an observer is installed is not a fact — and the
keys are server-owned, so provenance cannot be forged from a request.
Middleware self-description
Twelve middlewares declare their own behaviour-affecting parameters through
a duck-typed `release_policy_parameters()`. Long text is hashed rather than
embedded: a declaration is an identity, not a copy of the prompt.
Agent assembly descriptor
`assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint
answers "did anything about this agent change between these two runs?".
`make_lead_agent()` keeps its graph-only signature — it is the LangGraph
Server ABI declared in langgraph.json. Tools and skills are sorted before
hashing because their assembly order is incidental; middlewares are not,
because stack order decides what wraps what. Host build identity is reported
but excluded from the fingerprint, so a redeploy does not invalidate every
agent's identity.
Context compaction observation
Summarization emits the content hashes of the messages it is about to remove
joined to the summary that replaced them. Content is the only identity
available at that seam: the summary does not become a message, and what later
projects it into a request renders it bounded and escaped rather than
verbatim.
Neutral policy, transform and MCP-source facts
Guardrail decisions are published to runtime context under a `__`-prefixed
key; result-rewriting middlewares append a declared, ordered transform trail;
MCP tools carry their credential-free logical origin.
Extension route identity
Contributed routes are session-authenticated and cannot opt out, but
"logged in" and "administrator" are different questions. Extensions get a
neutral projection of the caller rather than the host's auth context, and
`require_admin` fails closed when identity cannot be determined.
Extension-owned tables
An extension that persists data owns its own MetaData and migration chain, so
its tables are absent from Base.metadata and `alembic revision --autogenerate`
proposes dropping them. Extensions declare a table prefix, which is rejected
at registration if it would shadow a host table.
The contract package stays dependency-free and imports no host code; every new
Protocol method has a default so later additions remain additive. The loader's
pre-1.0 rule requires an exact major.minor match, so extensions written against
0.1 are now refused at startup with an actionable install hint rather than
loading into a host that implements a different surface.
uv.lock records the contract package's new version, so `uv sync --locked` still
resolves on a fresh checkout.
* fix(backend): sort gateway service imports
* feat(extensions): add gateway services and routers
* feat(extensions): add standalone reference extension
* fix(extensions): harden contributed gateway routes
* docs(extensions): document gateway contribution points
* feat(extensions): add operator CLI for packaged extension management
Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.
Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.
Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.
Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.
Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.
Co-authored-by: Codex <codex@openai.com>
* fix(extensions): stop the managed plugins rewrite from destroying config
Two data-safety defects in the managed `plugins:` block writer.
The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.
The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.
Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.
* fix(extensions): harden the manager transaction and correct its docs
Follow-up hardening on the extension package manager.
Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
`UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
the interpreter that the entry-point probe then imports and calls, and every
later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
validation the HTTPS-only source rule depends on. Neither is an index, proxy,
cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
`ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
on case transitions, so only the separated spellings were caught. Short
generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
the package's build backend, so a config the manager could never write to must
fail before that code runs rather than afterwards through rollback.
Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
runs without `--locked` when the checkout had no lock, so uv writes one while
resolving; if that sync then failed, the restore was skipped and the operator
kept a lock file they never had. A failing recovery sync now also reports the
original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
dependency resolve invites a second interrupt that escapes the handler and
strands the checkout mid-transaction; the declarations are already restored
and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
builder is a different machine, but unlike an environment-driven wheelhouse
resolution this is a source the operator typed deliberately, so it is reported
rather than rolled back. Private-network indexes are untouched: a builder on
that network can reach them.
Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.
* test(gateway): pin the request-path projection agreement
`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.
Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:
- projection edge cases, including the segment-boundary guard that keeps
root_path="/api" from slicing "/apifoo/models" into a string the router
would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
mounted under the "/health" public prefix must still 401, and a POST
mounted under "/api/webhooks" must still require a CSRF token
Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).
Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.
* ci: pin uv to the version production ships
ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.
backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.
Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.
Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.
* fix(gateway): state the extension route auth limit and abort a failed dev sync
Two scoped review follow-ups.
README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.
docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.
Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.
* fix(extensions): point Git SSH shorthand at the HTTPS correction
Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.
The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.
* docs: keep the root extension summary within its new budget
#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.
---------
Co-authored-by: Codex <codex@openai.com>
* feat(extensions): observe task lifecycle and system model calls
PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.
Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).
`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.
Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.
Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.
System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.
The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.
Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.
Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.
* fix(extensions): decide notification fail-open by origin, observe cancellation
`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.
`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.
DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.
* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown
Review response on #4684:
- The memory observation bridge caught BaseException, which would swallow
a teardown signal raised while dispatching; it now catches Exception,
matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
hook-failure path and logged an asyncio-internal traceback; it now logs
a warning like the pre-hook budget skip, while a TimeoutError a
contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
so log lines said "task goal/title/..."; they now carry the task
scope id alongside the kind.
* feat(channels): add Buzz (Nostr) channel connector
Adds a Buzz (https://github.com/block/buzz) channel so DeerFlow can join a
Nostr-relay workspace as a member: it answers @mentions in channels, replies
to DMs, and streams answers by editing one message in place.
* app/channels/buzz_nostr.py — pure NIP-01 helpers: canonical event ids,
BIP-340 signing/verification, chat/edit/auth builders, relay frames.
* app/channels/buzz.py — BuzzChannel: one NIP-42-authenticated websocket,
channel discovery (kind 39000) with one subscription per channel, live
membership tracking (44100/44101), per-channel replay watermarks, and
replies posted once then edited in place (kind 40003).
* app/channels/buzz_run_policy.py — same-thread serialization, mirroring
the Feishu precedent.
Inbound is gated in order: signature verification, self-drop, /connect
bind-and-return, pubkey allowlist, then mention / DM / mention-free /
thread-follow. Off by default; needs the new optional `buzz` extra
(coincurve, lazily imported), which detect_uv_extras resolves from
channels.buzz.enabled the same way it already handles channels.discord.
Two relay behaviours drove the design and are worth knowing when reviewing:
a global {"kinds":[9]} subscription receives nothing from buzz-relay and a
multi-value "#h" filter receives nothing either, so one REQ per channel is
required; and a single global `since` cursor skips quiet channels, so
watermarks are per channel.
Signed-off-by: Ajay R <ajayr@formbuddy.com>
* fix(channels): only publish assistant messages from the IM stream
`_accumulate_stream_text` decided what streamed `messages-tuple` payloads
become displayable text by rejecting ONLY payloads whose `type` contained
"tool", so it published everything else. DeerFlow writes hidden model
context into the messages channel as ordinary messages -- memory recall and
the rewritten user turn as hidden HumanMessages (DynamicContextMiddleware),
the `<durable_context_data>` block as another (DurableContextMiddleware) --
and LangGraph fans those state writes out on the messages stream, so they
reached every streaming IM channel as the assistant's reply.
Proved live on a Buzz relay: the connector published a `<memory>` fact block
and, in another run, a verbatim echo of the user's own inbound message.
Affects Feishu, Telegram, WeCom and Buzz; worst on Buzz, where each update
is an immutable public Nostr event that a corrective edit cannot unpublish.
Invert the filter to an allowlist of assistant message types. Two new pure
helpers keep it testable:
- `_stream_payload_type` resolves the type from both shapes the function
already handles: the `model_dump()` shape the gateway emits, and
LangChain's `to_json()` constructor shape whose own `type` is the literal
"constructor" and whose class name is the tail of the `id` path.
- `_is_assistant_stream_type` matches "ai"/"assistant" by PREFIX, not
substring -- ordinary words contain "ai" ("chain", "domain"), and a
substring test would admit a foreign type name by accident.
The bare-`str` branch is removed: an untyped payload cannot be attributed to
the assistant, nothing in DeerFlow produces one (serialize_messages_tuple
always emits `[message_dict, metadata]`), and a runtime that emitted raw text
deltas would emit hidden context the same way. Per-message-id buffering and
merging are unchanged.
Tests pin both directions, including multi-chunk merging across one message
id, so the allowlist cannot silently kill streaming, plus an end-to-end
`_handle_streaming_chat` test asserting the live payload never reaches an
outbound message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ajay R <ajayr@formbuddy.com>
* chore(helm): bump config_version to 33 in chart values and README
config.example.yaml moved to 33 for the buzz channel block; the chart's
embedded config example and its README copy track it (config_version only
drives the outdated-config warning, per scripts/check_config_version.sh).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ajay R <ajayr@formbuddy.com>
---------
Signed-off-by: Ajay R <ajayr@formbuddy.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
