3 Commits

Author	SHA1	Message	Date
Wenchao An	1b76ab9060	feat: add opt-in task notes and compacted history recall (#5382) ... * feat: add opt-in task notes and compacted history recall * fix: validate task continuity state and preserve user answers Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence. * fix: align Helm config version with task continuity schema * fix: preserve mixed task history and declare continuity policies * fix: recover malformed history and evict archives atomically	2026-09-12 21:01:46 +08:00
Michael	eebe909ebd	fix(agents): make the injected current-date timezone configurable (#5154) ... * fix(agents): make the injected current-date timezone configurable ## Why The date reminder injected into the lead and subagent prompts (DynamicContextMiddleware / SubagentDateContextMiddleware) was formatted with the server's local wall clock. DeerFlow containers default to UTC, so a user in Asia/Shanghai chatting in the 00:00-08:00 window was told that 'today' is the previous day - the model then reasons, plans, and date-stamps against the wrong day. ## What changed - _format_current_date() now reads the optional DEER_FLOW_DATE_TIMEZONE env var (IANA name, e.g. Asia/Shanghai) and renders the date in that zone. - Unset = unchanged server-local behavior; invalid names log a warning and fall back to server-local. - Documented the knob in config.example.yaml, the module docstring, and the DynamicContext entry in agents/middlewares/AGENTS.md. ## Surface area - [x] Agents / LangGraph - prompt-layer date context only; message shape and midnight-update behavior unchanged - [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies - [x] Default behavior change (opt-in via env var - no behavior change unless set) ## Bug fix verification - New tests: test_format_current_date_honors_configured_timezone (UTC 20:30 -> 2026-09-03 in Asia/Shanghai), test_format_current_date_defaults_to_server_local_without_env, test_format_current_date_invalid_timezone_falls_back. - Existing mocked-datetime tests pass unchanged (no env -> datetime.now() path). ## Validation - cd backend && python -m pytest tests/test_dynamic_context_middleware.py: 31 passed. - blocking_io/test_dynamic_context_middleware.py: 2 pre-existing abefore_agent failures reproduce identically on clean main (blockbuster os.listdir detection on this host); the other 2 pass. - ruff format + ruff check clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): avoid passing tz to datetime.now when no timezone is configured CI (backend-unit-tests shard 2) failed in test_tool_error_handling_middleware.py::test_subagent_chain_injects_date_without_memory_and_coalesces_for_strict_provider because its _FrozenDateTime.now() subclass override accepts no arguments, while _format_current_date() called datetime.now(None) even when DEER_FLOW_DATE_TIMEZONE was unset. - _format_current_date() now calls datetime.now() with no arguments unless a timezone is actually configured, preserving the exact legacy call shape for every datetime-subclass test fake. - The configured-zone path still calls datetime.now(tz) and converts via astimezone(tz). - Updated the no-env unit test to assert datetime.now() is called without arguments. Validation: python -m pytest tests/test_dynamic_context_middleware.py + the previously failing strict-provider test: 32 passed. ruff clean. * fix(agents): declare the effective current-date timezone in the assembly descriptor ## Why Maintainer review on the DEER_FLOW_DATE_TIMEZONE change (#5154): the knob is prompt-affecting, yet both DynamicContextMiddleware and SubagentDateContextMiddleware were invisible to the agent assembly descriptor - describe_middleware() fell back to {"probed": true} for unset, UTC, and Asia/Shanghai alike, so deployments that inject different dates shared one assembly fingerprint and release observers could not distinguish or audit the behavior change. ## What changed - Both middlewares now implement release_policy_parameters() -> dict[str, object], declaring {"current_date_timezone": <name>} as required by the module's middleware self-description contract. - The declared value is the normalized effective zone: a configured, valid DEER_FLOW_DATE_TIMEZONE is reported by its IANA key (ZoneInfo.key); otherwise the server-local zone is resolved to its IANA key when the platform exposes one and to its tzname label otherwise (fixed-offset hosts), with "UTC" as the final fallback. - Added both middlewares to _MIDDLEWARE_DECLARATIONS in backend/tests/test_middleware_release_policy.py so the existence check and the construct-and-canonical-hash check cover them. ## Verification - New tests: test_date_middlewares_declare_configured_timezone (Asia/Shanghai), test_date_middlewares_declare_utc_timezone, plus resolved-server-local assertions for the unset and invalid-env paths; both middlewares agree in every case. - cd backend && python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py: 70 passed. - Regression spot-check: tests/test_agent_assembly_descriptor.py, tests/test_tool_error_handling_middleware.py, tests/test_system_message_coalescing_middleware.py: 102 passed. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): stabilize the declared date timezone and simplify the formatting path ## Why Follow-up review on #5154 (willem-bd). The release-policy declaration added in 884cec4b resolved the observability gap but pinned far less identity than its docstrings claimed, and the formatting path carried a production no-op. ## What changed - The declared label is now stable and unambiguous: a configured, valid DEER_FLOW_DATE_TIMEZONE is reported by its IANA key; without one, the server-local zone is resolved to a real IANA key from the TZ env var or the /etc/localtime symlink (Linux/macOS); when no key is recoverable (Windows, stripped containers) the declaration falls back to a stable `server-local(+-HH:MM)` sentinel carrying the current UTC offset. It never reports a bare abbreviation - datetime.now().astimezone() yields only a fixed-offset timezone whose tzname (e.g. CST, EST/EDT, CET/CEST) is ambiguous or DST-churns, which the assembly descriptor docstring says must not happen. - Dropped the redundant astimezone(tz) in _format_current_date(): datetime.now(tz) already returns the instant expressed in tz. The configured-zone test now fakes datetime.now(tz) semantics (the fixed instant converted into the requested zone) instead of relying on that conversion. - Documented why the knob is an env var, not a config-schema field: it is read at runtime by both date-context middlewares so an operator can point a container at another zone without mounting a config.yaml (module docstring + config.example.yaml note). - AGENTS.md: fixed the glued DynamicContext sentence (missing separator). - Added tzdata>=2025.1 to the harness runtime dependencies (with uv.lock) so ZoneInfo works on stripped containers / Windows without an OS zone database. ## Verification - New tests: test_server_local_timezone_name_reads_tz_env, test_effective_timezone_sentinel_uses_offset_when_local_zone_is_not_resolvable; reworked test_format_current_date_honors_configured_timezone to exercise the real datetime.now(tz) path. - cd backend && python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py tests/test_tool_error_handling_middleware.py: 140 passed. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): offload subagent date injection off the event loop ## Why Follow-up review on #5154 (willem-bd, P2): SubagentDateContextMiddleware.abefore_agent() called _inject() directly, so enabling DEER_FLOW_DATE_TIMEZONE could synchronously read the OS timezone database (or the tzdata wheel) on a cold cache - filesystem work on the async subagent execution path whenever no assembly observer resolved the zone first. ## What changed - SubagentDateContextMiddleware.abefore_agent() now offloads the injection via asyncio.to_thread with the same bounded timeout DynamicContextMiddleware uses (issue #3402); on timeout it logs and skips the date update for that run instead of blocking the loop. - Narrowed the exception handling in _date_timezone() and the TZ-env branch of _server_local_timezone_name() to configuration-shaped failures (ZoneInfoNotFoundError / ValueError / OSError). Previously a blanket `except Exception` also swallowed BlockingError raised by the blocking-I/O regression gate, mislabeling a loop-blocking call as an invalid timezone and silently degrading to server-local - which made the new regression anchor useless. Other exceptions now propagate. ## Verification - New blocking-I/O regression anchor (backend/tests/blocking_io/test_subagent_date_context_middleware.py): drives a real create_agent graph under the strict Blockbuster gate with the knob enabled and asserts the date reminder is injected. Verified it fails (BlockingError) when the offload is reverted and passes with it in place. - python -m pytest tests/blocking_io/test_subagent_date_context_middleware.py: 1 passed. The two pre-existing os.listdir failures in tests/blocking_io/test_dynamic_context_middleware.py reproduce unchanged on this host (same as clean main). - python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_tool_error_handling_middleware.py tests/test_agent_assembly_descriptor.py: 139 passed; the single ToolReceiptMiddleware-ordering failure reproduces with the change stashed (local extensions registry, unrelated to this PR). - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): read the direct /etc/localtime symlink target for the zone key ## Why Follow-up review on #5154 (willem-bd, P2): on macOS, /etc/localtime commonly points to /var/db/timezone/zoneinfo/<zone>, but Path.resolve() follows that directory's own symlink and yields a versioned path such as /private/var/db/timezone/tz/2026c.1.0/zoneinfo/Asia/Shanghai, which matched no configured prefix. The server-local resolution then returned None and the assembly descriptor fell back to a server-local(+HH:MM) sentinel even though the IANA key was available - conflating zones that share an offset and making DST-based fingerprints unstable. ## What changed - _server_local_timezone_name() now reads the direct symlink target via os.readlink("/etc/localtime") instead of Path.resolve(), so macOS' unversioned zoneinfo path is seen as-is and its IANA key is preserved. - The zone key is taken from whatever follows the last "/zoneinfo/" segment, which also handles Apple's canonical versioned path when a direct target already carries it, and relative targets are normalized against /etc. - Removed the now-unused Path import and the fixed zoneinfo prefix tuple. ## Verification - New tests: test_server_local_timezone_name_reads_direct_macos_symlink_target, test_server_local_timezone_name_reads_apple_versioned_symlink_target, and test_server_local_timezone_name_normalizes_relative_symlink_target. - python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py: 105 passed (75 after re-running the first two on the merged main). The blocking subagent anchor still passes; the two pre-existing os.listdir blocking failures on this host are unchanged. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-04 23:39:31 +08:00
Nan Gao	13f0a7f263	feat(extensions): let an out-of-tree extension observe what the agent did (#4863) ... * feat(extensions): let an out-of-tree extension observe what the agent did DeerFlow's extension system can contribute middleware, services and routes, but an extension cannot answer basic questions about a run without reaching into host internals. Several of the facts it would need are destroyed by the operations that produce them: * The middleware chain injects and rewrites a lot of context — date reminders, recalled memory, compaction summaries, durable-context data, image payloads, activated skill bodies. Downstream, none of it is attributable: at the model-call boundary an injected HumanMessage is indistinguishable from the user's own, and anything wanting to tell them apart has to pattern-match prompt wording, which breaks on the next copy edit. * Two runs of "the same agent" are only comparable if the chain enforced the same limits, prompts and thresholds. Recovering that from outside means reading private attributes and guessing which of them change behaviour — a guess that rots silently as middlewares gain fields. * The lead-agent factory resolves a model after runtime overrides, renders a prompt, filters tools through authorization and composes a stack, all inside one synchronous call, and none of it survives: a middleware sees its neighbours but not the prompt, the run worker sees a graph but not what went into it. * Summarization is destructive by design. N messages leave the context and one summary enters it; afterwards only the summary exists, so "which messages became this?" is not reconstructible. This adds seven neutral facilities so those facts are recorded where they are still true, and releases the contract package as 0.2.0. Message provenance Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the messages they inject or rewrite. Stamping is unconditional — a fact whose presence depends on whether an observer is installed is not a fact — and the keys are server-owned, so provenance cannot be forged from a request. Middleware self-description Twelve middlewares declare their own behaviour-affecting parameters through a duck-typed `release_policy_parameters()`. Long text is hashed rather than embedded: a declaration is an identity, not a copy of the prompt. Agent assembly descriptor `assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint answers "did anything about this agent change between these two runs?". `make_lead_agent()` keeps its graph-only signature — it is the LangGraph Server ABI declared in langgraph.json. Tools and skills are sorted before hashing because their assembly order is incidental; middlewares are not, because stack order decides what wraps what. Host build identity is reported but excluded from the fingerprint, so a redeploy does not invalidate every agent's identity. Context compaction observation Summarization emits the content hashes of the messages it is about to remove joined to the summary that replaced them. Content is the only identity available at that seam: the summary does not become a message, and what later projects it into a request renders it bounded and escaped rather than verbatim. Neutral policy, transform and MCP-source facts Guardrail decisions are published to runtime context under a `__`-prefixed key; result-rewriting middlewares append a declared, ordered transform trail; MCP tools carry their credential-free logical origin. Extension route identity Contributed routes are session-authenticated and cannot opt out, but "logged in" and "administrator" are different questions. Extensions get a neutral projection of the caller rather than the host's auth context, and `require_admin` fails closed when identity cannot be determined. Extension-owned tables An extension that persists data owns its own MetaData and migration chain, so its tables are absent from Base.metadata and `alembic revision --autogenerate` proposes dropping them. Extensions declare a table prefix, which is rejected at registration if it would shadow a host table. The contract package stays dependency-free and imports no host code; every new Protocol method has a default so later additions remain additive. The loader's pre-1.0 rule requires an exact major.minor match, so extensions written against 0.1 are now refused at startup with an actionable install hint rather than loading into a host that implements a different surface. uv.lock records the contract package's new version, so `uv sync --locked` still resolves on a fresh checkout. * fix(backend): sort gateway service imports	2026-08-23 09:57:12 +08:00