mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-10 21:10:34 +00:00
841 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Wenchao An
|
ca23703ef0
|
fix(subagents): preserve actionable acceptance gaps after compaction (#5287)
* fix: preserve actionable subagent acceptance gaps Distinguish completed execution from acceptance in delegation guidance. Retain bounded unmet and unverified criteria after compaction and guide the lead to address remaining work within its budget. * docs: keep acceptance guidance within instruction budget |
||
|
Daoyuan Li
|
97c6883f42
|
fix(sandbox): use native AIO file append (#5278) | ||
|
Sami Belhareth
|
f8f6cde23f
|
fix: preserve assistant/tool history in compaction summaries (#5248)
* fix: preserve bounded assistant and tool input during compaction * fix: retain recent fallback summary input and clarify budget * fix: preserve recent content in mixed-history summary fallback * docs: trim middleware guidance to pass size check --------- Co-authored-by: Sami Belhareth <6599699+belharethsami@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Shxiao
|
48bbea6df3
|
fix(workspace-changes): record symlink targets without the verbatim prefix (#5250)
* fix(workspace-changes): record symlink targets without the verbatim prefix os.readlink on Windows reports absolute targets in extended-length form (\?\C:\... or \?\UNC\server\share). The scanner stored that raw spelling, so workspace-change events showed \?\-prefixed targets that do not match ordinary Windows paths. Strip the prefix when recording; POSIX readlink output is unchanged. Skills projection/review readlink sites are untouched — they have no user-facing contract pinned on the spelling. * fix(workspace-changes): gate symlink target normalization to Windows Review follow-up on #5250: - Gate _normalize_symlink_target on os.name == "nt". readlink(2) on POSIX returns the literal string the link was created with, and backslash is a valid filename byte on Linux, so a target that starts with the extended-length prefix there must be recorded verbatim. The docstring's POSIX claim is now provably true. - Commit the unit checks the PR body previously described as ad-hoc: drive and UNC prefix stripping, relative and plain POSIX targets, mid-string prefix left verbatim, and an off-Windows identity case, so the new branch has real coverage on every platform instead of relying on a Windows host with symlink privilege. * test(workspace-changes): force Windows platform in prefix-strip unit tests The os.name gate added in the previous commit makes _normalize_symlink_target a verbatim identity off-Windows, so the two prefix-strip assertions failed on the ubuntu-only unit CI. Force os.name to "nt" via monkeypatch in both, mirroring the off-Windows identity test, so every case pins exactly one platform's contract and the suite is green on every host. * fix(workspace-changes): strip only extended drive-letter prefixes Review follow-up on #5250: the catch-all branch also stripped the extended-length prefix from volume-GUID targets (\?\Volume{...}\...), leaving a relative-looking path that loses the target's namespace. Restrict the branch to extended drive-letter paths (letter, colon, separator) and keep every other \?\ namespace form verbatim; add the volume-GUID regression plus degenerate-prefix cases. |
||
|
Yusuf Gürdoğan
|
23bd76046a
|
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider Add a community provider backed by Sofya (https://sofya.co). Its search endpoint returns the content of the result pages, not only their snippets, and its fetch endpoint returns a page as markdown. Both are plain JSON over HTTP, so this needs no extra Python package (uses httpx, already a dependency). Changes: - backend/packages/harness/deerflow/community/sofya/__init__.py - backend/packages/harness/deerflow/community/sofya/tools.py Implements web_search_tool and web_fetch_tool using httpx. API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY env var. Follows the same interface and output shape as the existing ddg_search and serper providers, including the max_results parameter with config override and the structured "No results found" error. - backend/tests/test_sofya_tools.py Unit tests covering API key resolution, config overrides, result mapping, time range, HTTP errors, empty results, and fetch failures. - config.example.yaml: add commented-out Sofya web_search and web_fetch examples alongside the other providers - .env.example: add SOFYA_API_KEY placeholder - backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and the environment variables * fix(sofya): honor caller max_results, validate search_depth, join time_range contract test - Caller-supplied max_results now wins; config is used only when the argument is omitted, matching GroundRoute. - search_depth is clamped to basic/snippets; an unsupported value logs a warning and falls back to basic. - Sofya added to the shared time_range schema contract test. * fix(sofya): cap per-result content so a search stays inline An unbounded search payload (up to 20 read pages) crossed the tool output budget middleware's externalize_min_chars threshold, which replaces the result list with a file reference. Cap each result's content at contents_max_characters (default 2000, 0 disables), matching Exa's config key. Five capped results stay under the 12000 char threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): list Sofya in the recency contract, coerce non-string content _clip subscripted its input, so a non-string content or description from the API raised TypeError instead of degrading. Coerce to text first, the way _sofya_post and _response_results guard the shapes around it. Also add Sofya to the Web Search Recency section in backend/AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog web_fetch sliced its content the same way web_search did before the last push: a truthy non-string from the API passed the falsiness guard and then raised TypeError. Reuse _clip, keeping the `or ""` so empty content still reports "No content found". Also add sofya to the community provider inventory in packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs The ZH web_fetch tab list named five providers where EN names seven. Both tabs mirror their EN counterparts, so the two locales list the same web_fetch providers again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
wutongyuonce
|
e5d23943ce
|
fix(sandbox): stop E2B append from overwriting on read failure (#5261)
* fix(sandbox): stop E2B append from overwriting on read failure E2B has no native append, so write_file(append=True) read-modify-writes. Treat only FileNotFoundException/FileNotFoundError as an empty file; any other pre-read error must abort so a timeout cannot replace the original contents with just the new fragment. * fix(sandbox): distinguish E2B append pre-read refusal in logs A non-not-found pre-read error now logs as a refused overwrite instead of a write failure. Tests pin the successful read-modify-write path, including a bytes pre-image, so dropping `existing` cannot go green. |
||
|
Shxiao
|
cbd6621d52
|
fix(mcp): resolve drive-qualified paths in file reference rewriting (#5242)
* fix(mcp): resolve drive-qualified paths in file reference rewriting
urlparse reads a Windows drive prefix ("C:/...") as the URI scheme, so
_local_path_from_uri() returned None for every drive-qualified path and
MCP file references were never rewritten to /mnt/user-data/... virtual
paths on Windows hosts. file:// URIs were parsed with urlparse().path
alone, which also drops the drive qualifier.
- resolve file URIs through url2pathname so the /C:/... form keeps its
drive, and treat single-letter schemes as bare drive paths;
- match drive-qualified absolute paths in the free-text reference regex;
- build test URIs with Path.as_uri() and anchor absolute-path fixtures
at tmp_path so expectations are host-portable, and cover the
drive-prefix scheme quirk explicitly.
* fix(mcp): decode file URIs once and guard Windows path rejection
Review follow-up on #5242:
- url2pathname already percent-decodes on both platforms, so the extra
unquote() wrapper decoded references twice and broke filenames that
contain a literal '%'. Pass parsed.path straight through.
- On Windows, url2pathname raises OSError for paths containing a raw
'|' (e.g. file:///C:/tmp/a|b.png); catch it so one odd URI cannot
abort the whole best-effort rewrite pass.
- The relative-reference regex alternative now accepts backslash
separators, which is what Windows servers print for relative paths.
- Add Windows-only regressions driving the backslash free-text form
and a file:///C:/ URI end to end, plus the OSError rejection.
* fix(mcp): resolve file://C:/… URIs with a drive-qualified authority
Review follow-up on #5242 (two-slash Windows drive form):
- Some Windows tools emit file://C:/… without the third slash, which
puts the drive in the URI authority. Consult parsed.netloc: rebuild
the /C:/… URL path for a drive-qualified authority, keep the current
handling for empty and localhost authorities, and reject any other
host instead of silently treating its path as local.
- Extend the free-text regex so the two-slash form matches as one token
instead of the previous stray e://… mid-token match.
- Cover the two-slash form at the _local_path_from_uri unit, through
_rewrite_local_paths_in_text, and add a portable case asserting that
a remote-host file URI is ignored.
* fix(mcp): anchor the drive-qualified text alternative with a lookbehind
Review follow-up on #5242:
- [A-Za-z]:[\/] could steal a token at an earlier scan position:
for file:/tmp/… (single-slash form per RFC 8089 / Java File.toURI())
the match became e:/tmp/…, which resolves as a bare drive path and
left the reference unrewritten where /tmp/… was rewritten before.
Anchor the alternative with (?<![\w.-]) so word:/… shapes fall
through to the earlier alternatives.
- Add the missing coverage for the relative alternative's backslash
support (temp\page.yml through _rewrite_local_paths_in_text) and
a portable regression pinning the file:/… tokenization.
|
||
|
PeaceMaker-best
|
e7c059d8d4
|
fix(agents): prioritize loop hard stops across tool batches (#5245)
* fix(agents): prioritize loop hard stops across tool batches Scan an admitted multi-tool response completely before selecting a soft warning, so any configured hard limit can reject the whole batch. Preserve warning priority and sliding-window accounting. Add counter-level plus sync and async compiled-agent regressions proving rejected tools are not executed. Fixes bytedance/deer-flow#5243. AI-assisted implementation and tests. * fix(agents): rearm loop warnings after cross-tool eviction When another tool evicts an older tool below its frequency warning threshold, clear the older suppression mark so a later burst can warn again. Add the cross-tool sliding-window regression from the final boundary review. AI-assisted implementation and tests. * test(agents): cover override-aware loop warning rearm Cache the default frequency thresholds for sliding-window eviction and verify that an evicted tool uses its configured override when warning eligibility is rearmed. Document that simultaneous frequency warnings preserve legacy first-crossing selection while hard stops remain batch-severity-first. Addresses review on #5245. AI-assisted implementation and tests. --------- Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
99367100fb
|
fix(persistence): preserve rollback across the incarnation migration (#5219)
* fix(persistence): tolerate thread incarnation migration * docs(persistence): pin forward revision contract --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> |
||
|
Jun
|
d1f1c49dcd
|
fix(workspace-changes): avoid draining metadata scans on cancellation (#5234)
* fix(workspace-changes): keep metadata cancellation responsive * test(workspace-changes): cover metadata cancellation latency * docs(workspace-changes): document cancellation ownership * style(workspace-changes): format cancellation regressions * docs(workspace-changes): remove unapproved nested guidance * fix(workspace-changes): log only real cancellation drains * style(workspace-changes): apply repository ruff format * docs(harness): record workspace scan cancellation ownership * docs: compact harness guidance below inherited size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wu Shuwen
|
364dad06aa
|
docs: update middleware contribution examples (#4945)
* docs: update middleware contribution examples * docs: clarify middleware registration paths * docs: clarify middleware injection scope * docs: clarify middleware state updates * docs: clarify middleware state updates * docs: clarify middleware pipeline placement * docs: complete middleware order guidance * docs: align middleware guard conditions * docs(middleware): name runtime sanitization order * docs: pin middleware runtime order * docs: clarify middleware assembly paths * docs: clarify middleware anchor scope |
||
|
早上肚子疼
|
383263bd34
|
fix(llm): release owned recovery probe on cancellation (#5197)
* fix(llm): release owned recovery probe on cancellation * docs: keep middleware guidance within chain budget --------- Co-authored-by: zaoshangduziteng <309590849+zaoshangduziteng@users.noreply.github.com> |
||
|
Beautyl0ve
|
3bccd1474f
|
fix(client): scope embedded agent reuse by effective user (#5206)
Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> |
||
|
Ryker_Feng
|
98b8e4657e
|
feat(chats): add archive and restore (#5236)
* feat(chats): add archive and restore * test(chats): observe archive search requests in pagination e2e * docs(gateway): move thread lifecycle details out of inherited guidance * docs(chats): add concise archive and restore RFC * docs(chats): move archive RFC discussion to issue 5237 |
||
|
PeaceMaker-best
|
e3df6ea4a8
|
feat(channels): select custom agents per conversation (#5168)
* feat(channels): select custom agents per conversation Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(channels): reserve agent slash command across clients Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(tui): hide reserved slash commands from skills Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(channels): preserve selected agent across clients --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
Jun
|
d7afdbf9a3
|
fix(workspace-changes): drain snapshot scan before cancellation cleanup (#5232)
* fix(workspace-changes): drain cancelled snapshot scans * test(workspace-changes): cover cancellation during snapshot scan * fix(workspace-changes): consume drained scan outcome * test(workspace-changes): keep scan cancellation regression focused * test(workspace-changes): pin cleanup ownership under recancel |
||
|
hataa
|
aec7d73890
|
feat(knowledge): add read-only LightRAG retrieval, fixes #5208 (#5209) | ||
|
RongJie G
|
a4ff4b0b3b
|
fix(journal): dedup llm.ai.response persistence on re-fired on_llm_end (#5187)
* fix(journal): dedup llm.ai.response persistence on re-fired on_llm_end LangChain may deliver on_llm_end more than once for the same run_id. RunJournal already dedups token accounting and the run summary (_record_message_summary) on that premise via _counted_message_llm_run_ids, but the durable llm.ai.response self._put() call was left unguarded. The event store is append-only and count_messages/list_messages read raw rows without read-time dedup, so a replayed callback persists a second llm.ai.response row for one logical response while the run's own message_count counts it once. This inflates count_messages, duplicates a message in list_messages pagination, and leaves the durable feed inconsistent with the run summary. Gate the persistence + summary block by the existing per-run_id guard so a replayed callback is a no-op, keeping the durable message feed and the run summary in agreement. Distinct run_ids are unaffected. Adds regression tests: a re-fired callback for one run_id persists exactly one row (red on main), and distinct run_ids each still persist a message. * fix(journal): preserve canonical response on late usage * fix(journal): preserve late usage while deduplicating responses * fix(journal): keep first callback response canonical * fix(journal): snapshot canonical response summaries --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> |
||
|
Hao Zhe
|
ec274bdedb
|
fix(memory): enforce backend read failure policy (#4726)
* fix(memory): enforce backend read failure policy * fix(memory): harden failure policy handling * fix(memory): narrow strict read handling * fix(memory): keep timeout handling off saturated executor * fix(memory): preserve legacy fail-closed timeouts |
||
|
spud
|
2e85901876
|
fix(lark): enforce private ACLs on Windows credential tree (#5141)
* fix(lark): enforce private ACLs on Windows credential tree On Windows, posix chmod(0o700/0o600) does not map to NTFS ACLs, so the secret-bearing Lark CLI credential tree was not actually owner-restricted and existing trees were not repaired. Branch the permission application by platform: - POSIX: directories 0o700, files 0o600 (behavior unchanged). - Windows: disable inherited ACLs, grant the Gateway process user Full Control (resolved via its SID from whoami /user /fo csv /nh so it is locale-independent), and remove broad non-administrative grants (Everyone, Authenticated Users, Users). Fail closed on identity or icacls failures so a tree is never left accessible silently. Existing-tree handling is covered by asserting every entry in the tree is repaired, and the Windows command contract is covered by mocked tests run in CI. * fix(lark): harden Windows credential tree against TOCTOU and hard-link races This replaces the path-based Windows hardening (lstat -> SetFileSecurityW(path) -> iterdir) with a handle-relative walker, so validation, the ACL update, and traversal are bound to the opened object rather than a re-resolved pathname. Every credential object is opened no-follow; children are enumerated with GetFileInformationByHandleEx(FileFullDirectoryInfo) and opened/created relative to an already-open parent handle (NtOpenFile/NtCreateFile with OBJECT_ATTRIBUTES.RootDirectory), so a pathname swap cannot redirect the walk. Credential directories are opened exclusively (share=0): SetSecurityInfo therefore does not propagate the final owner-only OI|CI DACL into as-yet-unvalidated children, and the namespace is locked for the duration of the walk (concurrent child rename/replacement and hard-link insertion fail with sharing violations). Any file with nNumberOfLinks != 1 is rejected before its security descriptor is touched, so an NTFS hard link to an external file cannot change that file owner/DACL. POSIX keeps the lstat-before-descent walk. Tests: native regressions for exclusive no-propagation, late hard-link insertion being blocked, mid-walk junction swap being blocked, static hard-link rejection, and both real NTFS junction rejections. Mock seams updated for the handle-relative API, and Windows portability fixes make the suite green on Windows except the known #5116 sandbox-runtime executable-bit failures. * test(lark): keep the credential-tree symlink assertion portable The credential-tree symlink rejection is a ValueError; POSIX reports a symlink while the Windows handle-relative walker reports a reparse point. Use a platform-dependent regex so the test passes on Linux/macOS and Windows. * fix(lark): close remaining credential-tree hardening gaps Review follow-up for the handle-relative credential-tree walker: - Stage the transaction snapshot under the owner-only root, copying only config/ and data/. - Serialize ensure() per-user across threads and processes with a dedicated lock. - Make the walker iterative so deep trees cannot hit the recursion limit. - Re-reject a symlinked POSIX root before mkdir; drop the over-strict ancestor-chain check. - Soften the SetSecurityInfo failure claim; add regressions for each and carry os.SEEK_END in the os stub. * fix(lark): anchor hardening lock under trusted base and keep POSIX untouched Follow-up refinements to the credential-tree hardening: - The per-user hardening lock file now lives directly under the trusted base_dir instead of the unverified per-user chain, so it is never written through an ancestor that has not yet passed reparse validation. - ensure() takes the hardening lock only on the Windows branch; POSIX keeps the original contract, so no new lock-file side effect. - Strengthen the ancestor-junction regression (lock not written to the external target) and fix two test docstrings to match the parent-first order and the no-prior-broadening failure claim. * fix(lark): anchor credential-operation lock under trusted base on Windows The per-user credential lock (_lark_credential_lock) created its advisory lock file under the unverified per-user chain (users/<id>/integrations/.lark-cli.credentials.lock) before ensure() validated the ancestor chain. On Windows it is now anchored directly under the trusted paths.base_dir (mirroring the hardening lock), so a junction at integrations can no longer cause the credential lock to be written into an external target before reparse validation. POSIX keeps the original location unchanged. Tests: - Public-flow regression (start_lark_config -> credential lock -> ensure) uses an empty sentinel lock file to prove the old credential-lock path is never opened/written. - CLI-write re-harden tests restore the POSIX outcome assertion (file tightened to 0600). |
||
|
RongJie G
|
cd2633725b
|
fix(runtime): finish terminal signaling after hook cancellation (#5191)
* fix(runtime): finish terminal signaling after hook cancellation * fix(runtime): shield task-stop observer fan-out --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> |
||
|
Coder-xiaosuo
|
27b2b67680
|
fix(models): restore usage_metadata in MindIE tool-mode simulated streaming (#5195)
In tool-enabled requests MindIEChatModel._astream falls back to awaiting the full _agenerate response and re-emitting it as simulated AIMessageChunks. The full response carries usage_metadata, but none of the simulated chunks copied it, so chunk aggregation (add_ai_message_chunks) produced a final message with usage_metadata=None. Token usage therefore vanished from token accounting, run stats, persistence and the UI for every tool-enabled streamed turn. Mirror OpenAI's terminal-usage-frame convention: attach msg.usage_metadata to exactly the last simulated chunk (the trailing tool-call chunk when present, else the last text chunk / the single tool-only chunk) so the aggregated message carries it exactly once. add_usage() is per-chunk additive, so attaching usage to every chunk would multiply the totals. Scope: MindIEChatModel only; other providers keep native streaming and ainvoke/non-tool astream were already correct. Tests: regression guard asserting exactly one carrier chunk equals the last one and that merged usage equals the original across all three simulated-stream branches, plus chain-level tests driving the public astream() wrapper and asserting the persisted model_dump() shape. Closes #5192 |
||
|
PeaceMaker-best
|
3c36217a51
|
feat(observability): persist deferred tool promotions (#5183)
* feat(observability): persist deferred tool promotions Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(ci): trim agent guidance chain Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
Aari
|
0f7d8709d3
|
feat(sandbox): add controlled egress with approvals (#5152)
* feat(sandbox): add controlled egress approvals * Apply batched suggestions from code review * fix(sandbox): harden restricted network policy * fix(sandbox): harden denied egress handling * fix(sandbox): isolate network proxy sidecar * chore: retry sandbox image smoke * fix(sandbox): close remaining network policy gaps * fix(sandbox): harden relay token rejection * fix(sandbox): fence incompatible policy replacement * fix(sandbox): replace containers across network modes * fix(sandbox): close remaining lifecycle gaps --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Michael
|
eebe909ebd
|
fix(agents): make the injected current-date timezone configurable (#5154)
* fix(agents): make the injected current-date timezone configurable ## Why The date reminder injected into the lead and subagent prompts (DynamicContextMiddleware / SubagentDateContextMiddleware) was formatted with the server's local wall clock. DeerFlow containers default to UTC, so a user in Asia/Shanghai chatting in the 00:00-08:00 window was told that 'today' is the previous day - the model then reasons, plans, and date-stamps against the wrong day. ## What changed - _format_current_date() now reads the optional DEER_FLOW_DATE_TIMEZONE env var (IANA name, e.g. Asia/Shanghai) and renders the date in that zone. - Unset = unchanged server-local behavior; invalid names log a warning and fall back to server-local. - Documented the knob in config.example.yaml, the module docstring, and the DynamicContext entry in agents/middlewares/AGENTS.md. ## Surface area - [x] Agents / LangGraph - prompt-layer date context only; message shape and midnight-update behavior unchanged - [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies - [x] Default behavior change (opt-in via env var - no behavior change unless set) ## Bug fix verification - New tests: test_format_current_date_honors_configured_timezone (UTC 20:30 -> 2026-09-03 in Asia/Shanghai), test_format_current_date_defaults_to_server_local_without_env, test_format_current_date_invalid_timezone_falls_back. - Existing mocked-datetime tests pass unchanged (no env -> datetime.now() path). ## Validation - cd backend && python -m pytest tests/test_dynamic_context_middleware.py: 31 passed. - blocking_io/test_dynamic_context_middleware.py: 2 pre-existing abefore_agent failures reproduce identically on clean main (blockbuster os.listdir detection on this host); the other 2 pass. - ruff format + ruff check clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): avoid passing tz to datetime.now when no timezone is configured CI (backend-unit-tests shard 2) failed in test_tool_error_handling_middleware.py::test_subagent_chain_injects_date_without_memory_and_coalesces_for_strict_provider because its _FrozenDateTime.now() subclass override accepts no arguments, while _format_current_date() called datetime.now(None) even when DEER_FLOW_DATE_TIMEZONE was unset. - _format_current_date() now calls datetime.now() with no arguments unless a timezone is actually configured, preserving the exact legacy call shape for every datetime-subclass test fake. - The configured-zone path still calls datetime.now(tz) and converts via astimezone(tz). - Updated the no-env unit test to assert datetime.now() is called without arguments. Validation: python -m pytest tests/test_dynamic_context_middleware.py + the previously failing strict-provider test: 32 passed. ruff clean. * fix(agents): declare the effective current-date timezone in the assembly descriptor ## Why Maintainer review on the DEER_FLOW_DATE_TIMEZONE change (#5154): the knob is prompt-affecting, yet both DynamicContextMiddleware and SubagentDateContextMiddleware were invisible to the agent assembly descriptor - describe_middleware() fell back to {"probed": true} for unset, UTC, and Asia/Shanghai alike, so deployments that inject different dates shared one assembly fingerprint and release observers could not distinguish or audit the behavior change. ## What changed - Both middlewares now implement release_policy_parameters() -> dict[str, object], declaring {"current_date_timezone": <name>} as required by the module's middleware self-description contract. - The declared value is the normalized effective zone: a configured, valid DEER_FLOW_DATE_TIMEZONE is reported by its IANA key (ZoneInfo.key); otherwise the server-local zone is resolved to its IANA key when the platform exposes one and to its tzname label otherwise (fixed-offset hosts), with "UTC" as the final fallback. - Added both middlewares to _MIDDLEWARE_DECLARATIONS in backend/tests/test_middleware_release_policy.py so the existence check and the construct-and-canonical-hash check cover them. ## Verification - New tests: test_date_middlewares_declare_configured_timezone (Asia/Shanghai), test_date_middlewares_declare_utc_timezone, plus resolved-server-local assertions for the unset and invalid-env paths; both middlewares agree in every case. - cd backend && python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py: 70 passed. - Regression spot-check: tests/test_agent_assembly_descriptor.py, tests/test_tool_error_handling_middleware.py, tests/test_system_message_coalescing_middleware.py: 102 passed. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): stabilize the declared date timezone and simplify the formatting path ## Why Follow-up review on #5154 (willem-bd). The release-policy declaration added in 884cec4b resolved the observability gap but pinned far less identity than its docstrings claimed, and the formatting path carried a production no-op. ## What changed - The declared label is now stable and unambiguous: a configured, valid DEER_FLOW_DATE_TIMEZONE is reported by its IANA key; without one, the server-local zone is resolved to a real IANA key from the TZ env var or the /etc/localtime symlink (Linux/macOS); when no key is recoverable (Windows, stripped containers) the declaration falls back to a stable `server-local(+-HH:MM)` sentinel carrying the current UTC offset. It never reports a bare abbreviation - datetime.now().astimezone() yields only a fixed-offset timezone whose tzname (e.g. CST, EST/EDT, CET/CEST) is ambiguous or DST-churns, which the assembly descriptor docstring says must not happen. - Dropped the redundant astimezone(tz) in _format_current_date(): datetime.now(tz) already returns the instant expressed in tz. The configured-zone test now fakes datetime.now(tz) semantics (the fixed instant converted into the requested zone) instead of relying on that conversion. - Documented why the knob is an env var, not a config-schema field: it is read at runtime by both date-context middlewares so an operator can point a container at another zone without mounting a config.yaml (module docstring + config.example.yaml note). - AGENTS.md: fixed the glued DynamicContext sentence (missing separator). - Added tzdata>=2025.1 to the harness runtime dependencies (with uv.lock) so ZoneInfo works on stripped containers / Windows without an OS zone database. ## Verification - New tests: test_server_local_timezone_name_reads_tz_env, test_effective_timezone_sentinel_uses_offset_when_local_zone_is_not_resolvable; reworked test_format_current_date_honors_configured_timezone to exercise the real datetime.now(tz) path. - cd backend && python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py tests/test_tool_error_handling_middleware.py: 140 passed. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): offload subagent date injection off the event loop ## Why Follow-up review on #5154 (willem-bd, P2): SubagentDateContextMiddleware.abefore_agent() called _inject() directly, so enabling DEER_FLOW_DATE_TIMEZONE could synchronously read the OS timezone database (or the tzdata wheel) on a cold cache - filesystem work on the async subagent execution path whenever no assembly observer resolved the zone first. ## What changed - SubagentDateContextMiddleware.abefore_agent() now offloads the injection via asyncio.to_thread with the same bounded timeout DynamicContextMiddleware uses (issue #3402); on timeout it logs and skips the date update for that run instead of blocking the loop. - Narrowed the exception handling in _date_timezone() and the TZ-env branch of _server_local_timezone_name() to configuration-shaped failures (ZoneInfoNotFoundError / ValueError / OSError). Previously a blanket `except Exception` also swallowed BlockingError raised by the blocking-I/O regression gate, mislabeling a loop-blocking call as an invalid timezone and silently degrading to server-local - which made the new regression anchor useless. Other exceptions now propagate. ## Verification - New blocking-I/O regression anchor (backend/tests/blocking_io/test_subagent_date_context_middleware.py): drives a real create_agent graph under the strict Blockbuster gate with the knob enabled and asserts the date reminder is injected. Verified it fails (BlockingError) when the offload is reverted and passes with it in place. - python -m pytest tests/blocking_io/test_subagent_date_context_middleware.py: 1 passed. The two pre-existing os.listdir failures in tests/blocking_io/test_dynamic_context_middleware.py reproduce unchanged on this host (same as clean main). - python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_tool_error_handling_middleware.py tests/test_agent_assembly_descriptor.py: 139 passed; the single ToolReceiptMiddleware-ordering failure reproduces with the change stashed (local extensions registry, unrelated to this PR). - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. * fix(agents): read the direct /etc/localtime symlink target for the zone key ## Why Follow-up review on #5154 (willem-bd, P2): on macOS, /etc/localtime commonly points to /var/db/timezone/zoneinfo/<zone>, but Path.resolve() follows that directory's own symlink and yields a versioned path such as /private/var/db/timezone/tz/2026c.1.0/zoneinfo/Asia/Shanghai, which matched no configured prefix. The server-local resolution then returned None and the assembly descriptor fell back to a server-local(+HH:MM) sentinel even though the IANA key was available - conflating zones that share an offset and making DST-based fingerprints unstable. ## What changed - _server_local_timezone_name() now reads the direct symlink target via os.readlink("/etc/localtime") instead of Path.resolve(), so macOS' unversioned zoneinfo path is seen as-is and its IANA key is preserved. - The zone key is taken from whatever follows the last "/zoneinfo/" segment, which also handles Apple's canonical versioned path when a direct target already carries it, and relative targets are normalized against /etc. - Removed the now-unused Path import and the fixed zoneinfo prefix tuple. ## Verification - New tests: test_server_local_timezone_name_reads_direct_macos_symlink_target, test_server_local_timezone_name_reads_apple_versioned_symlink_target, and test_server_local_timezone_name_normalizes_relative_symlink_target. - python -m pytest tests/test_dynamic_context_middleware.py tests/test_middleware_release_policy.py tests/test_agent_assembly_descriptor.py: 105 passed (75 after re-running the first two on the merged main). The blocking subagent anchor still passes; the two pre-existing os.listdir blocking failures on this host are unchanged. - ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis, implementation, and regression tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jun
|
e21245fd5b
|
fix(runtime): add waiter-safe keyed lock reclamation (#5176)
* fix(runtime): reclaim idle keyed locks safely Replace the per-loop thread lock registries with a waiter-aware keyed lock table. Count holders and queued waiters before acquisition so idle entries can be reclaimed without allowing a late caller to bypass an existing waiter. Add regression coverage for runtime call-site reclamation, goal/checkpoint domain independence, queued-waiter ordering, cancellation cleanup, high-cardinality key reclamation, and cross-event-loop isolation. Fixes #5171 * style(runtime): format keyed lock helper |
||
|
哈基米
|
dbe11dc798
|
fix(mcp): keep ToolRuntime injection for sync-wrapped MCP tools (#5164)
* fix(mcp): keep ToolRuntime injection for sync-wrapped MCP tools make_sync_tool_wrapper attached an annotation-less wrapper to tool.func, which made LangGraph's ToolNode stop detecting the coroutine's "runtime" parameter (_get_all_injected_args falls back to func first and its type hints are empty). Every MCP tool in a sync agent caller then ran with runtime=None: resolve_runtime_user_id fell through to the default user, and the background-submit wrapper lost run_id/tool_call_id on the TaskSubmitRequest, so completion notifications launched under the default lead agent instead of the thread's agent. Wrap the generator and both sync_wrapper variants with functools.wraps so get_type_hints still sees the original annotations. Adds a regression test that drives a func-patched pooled MCP tool through a real ToolNode and asserts the ToolRuntime is injected with the thread's user context. It fails on main (runtime=None) and passes with the fix. * docs(mcp): record sync-wrapper annotation contract; extend regression coverage Address review feedback on #5164: - Expand the Notes block in make_sync_tool_wrapper to state the functools.wraps contract (copies __name__/__qualname__/__doc__/__annotations__/__dict__ and sets __wrapped__) and why that is what keeps get_type_hints resolving string annotations from callers like mcp/tools.py and skill_manage_tool.py. Drop the no-op wraps on the inner run_coroutine so the wrap surface stays minimal. - Rename the regression test to test_func_patched_mcp_tool_keeps_toolnode_runtime_injection. - Add test_sync_wrapped_builtin_tools_still_resolve_runtime to pin that the built-in tools (which carry runtime as a pydantic schema field) keep resolving runtime after their func is wrapped by make_sync_tool_wrapper, so a future wrapper refactor cannot silently regress per-user resolution for them. |
||
|
PeaceMaker-best
|
fb28ed0122
|
feat(subagents): enable historical upload discovery (#5170)
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
goloisme
|
683d146a30
|
fix(mcp): MCP cache re-initialization broken by cross-loop asyncio.Lock (#5062)
* Fixes #5060: P1 snapshot config before loading, P2 RLock for sync path P1: Config changes during initialization can permanently cache stale tools. - Snapshot _config_path and _config_signature BEFORE await get_mcp_tools() - Compare AFTER get_mcp_tools() completes using _current_config_state() - If config changed during loading, discard stale result and retry - Prevents publishing old tools with new signature, which would make _is_cache_stale() permanently return False P2: Module-level asyncio.Lock still fails across event loops after real contention. - _init_lock = threading.RLock() for sync path (reentrant, prevents races) - _async_init_lock = asyncio.Lock() for async init serialization - reset_mcp_tools_cache() now acquires _init_lock for serialization Also fixes test P3: removed duplicated test bodies that leaked state between tests. * fix(mcp): make cache initialization cross-loop safe - Replace the module-level asyncio.Lock with thread-safe generation claiming - Snapshot config state before/after MCP loading and discard stale results - Keep reset state changes short and non-blocking for async endpoints - Add regression coverage for contended cross-loop init, config rewrites during load, and reset while init is in flight * fix: release MCP init claim on cancellation Release the in-flight generation claim from a cancellation-safe finally block so cancelling the task that owns initialization does not strand future callers. Add regression coverage for cancelling the owner and then reinitializing successfully. * fix(mcp): retire session pool before cache reset release Prevent a concurrent MCP cache initializer from publishing tool wrappers bound to the session-pool singleton that reset_mcp_tools_cache() is already retiring. Add regression coverage for that interleaving. * fix(mcp): retire session pool on stale cache invalidation * fix(mcp): retire pool on init discard --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
PeaceMaker-best
|
6022bdf5ae
|
perf(frontend): avoid redundant chat state snapshots (#5159)
* perf(frontend): avoid redundant chat state snapshots Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(streaming): preserve incremental chat semantics Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
Jun
|
83cb6767b3
|
fix(sandbox): add FOWNER for AIO 1.11 startup (#5163)
* fix(sandbox): add FOWNER for AIO 1.11 startup * test(sandbox): cover FOWNER startup capability * docs(sandbox): document FOWNER capability * test(sandbox): pin FOWNER regression smoke * ci(sandbox): allow pinning FOWNER smoke image * style(sandbox): format FOWNER smoke test --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
luo jiyin
|
e5977320a0
|
feat(sandbox): surface structured mount upload result on E2B sandbox (#4884)
* feat(e2b-sandbox): make mount upload deadline configurable Replace the hardcoded 120-second mount upload deadline with a configurable `mount_upload_deadline_seconds` key read from SandboxConfig (extra=allow). The value is validated: zero and negative inputs are clamped to 1 second. Omitting the key preserves the existing 120-second default. This addresses the follow-up from PR #4842 review: operators with large mounts or slow networks can now size the deadline to their deployment without changing code. * fix(e2b-sandbox): address review feedback on configurable deadline - Remove import-time default capture from _mount_deadline_reason() and _MountUploadBudget.deadline_seconds to prevent silent drift. - Add warning log when mount_upload_deadline_seconds is clamped to 1 (was silent before). - Update AGENTS.md E2B Mount Uploads section: deadline is now configurable, not fixed 120. - Add mount_upload_deadline_seconds to YAML examples in provider docstring and __init__.py. - Add config-path test that exercises SandboxConfig -> _load_config -> _apply_mounts end-to-end. * feat(e2b-sandbox): surface structured mount upload result on sandbox Introduce MountUploadResult dataclass and attach it to E2BSandbox.mount_upload_result after creation. This makes mount truncation observable in code without re-parsing Gateway logs. _apply_mounts() now returns MountUploadResult with truncated, reason, and upload totals. _create_sandbox() captures the result, stores it on the sandbox instance, and records it in a provider-level map so the result survives warm-pool reclaim and reconnect. MountUploadResult.truncated is True only when the upload pass was stopped early by a resource limit (deadline, file count cap, or byte budget). Individual mount failures (missing host path, SDK errors) are logged but do NOT set truncated. Tests cover: success totals, deadline truncation, file-count truncation, byte-budget truncation, non-limit failure not reported as truncation, missing host path not reported as truncation, create→sandbox wiring, and create→release→warm-pool→acquire result preservation. * fix(e2b-sandbox-provider): fix _mount_results lifecycle leak and review findings - Add _forget_mount_result() helper and call it at all terminal sandbox paths: _reuse_in_process_sandbox dead-evict, _reclaim_warm_pool_sandbox reconnect/dead/bootstrap/ownership/shutdown failure branches, _forget_local_sandbox, _kill_and_close. Prevents unbounded dict growth over a long-running Gateway process. - Make MountUploadResult @dataclass(frozen=True) to prevent silent mutation of the shared reference between provider map and sandbox attribute. - Move _mount_results insert under self._lock in _create_sandbox to match the read discipline in _register_connected_sandbox. - Guard _resolve_mount_upload_deadline against None (YAML explicit null) to avoid int(None) TypeError. - Add 5 regression tests covering each bypass path and the frozen invariant. * fix(e2b-sandbox-provider): add _forget_mount_result to _evict_oldest_warm branches Add _forget_mount_result() calls to all four terminal exit paths in the E2B _evict_oldest_warm override (reconnect failure, already-gone, kill failure, kill success). The peer-owned path already cleans up via _forget_local_sandbox. Add test_evict_oldest_warm_cleans_mount_result to pin the kill-success branch. * docs: reduce agent guidance size --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Michael
|
85ffb66d6e
|
fix(subagents): stamp UTC-aware datetimes on SubagentResult lifecycle (#5153)
## Why DeerFlow declares one timestamp convention in deerflow/utils/time.py: every lifecycle timestamp is UTC (now_iso / datetime.now(UTC)). SubagentResult writers in subagents/executor.py still used naive datetime.now(), so on any non-UTC host the in-memory lifecycle metadata (started_at / completed_at) was local wall-clock time. The sibling durable-batch path (subagents/batch_service.py) already stamps datetime.now(UTC), so the same run model carried two different conventions depending on which path wrote it. ## What changed - Added an executor-local _utcnow() helper that stamps datetime.now(UTC). - SubagentResult.completed_at default in try_set_terminal(), result.started_at in _aexecute(), and the started_at default in _aexecute_admitted() now route through _utcnow(). - Explicit caller-supplied timestamps (completed_at=...) still pass through unchanged. - Added regression tests asserting the default writers produce UTC-aware datetimes. ## Surface area - [x] Backend runtime (deerflow.subagents.executor) - internal dataclass lifecycle metadata, no wire format change - [ ] Frontend UI / Backend API / Sandbox / Skills / Dependencies / Default behavior change ## Bug fix verification - New tests: tests/test_subagent_executor.py::test_timestamp_writers_stamp_utc_aware_datetimes and test_utcnow_helper_returns_utc_aware_datetime encode the convention. - Updated BlockingDateTime.now() in the terminal-publication-order test to mirror datetime.now's optional tz argument. ## Validation - cd backend && python -m pytest tests/test_subagent_executor.py: 136 passed; 2 pre-existing TestBashExecutionHarvest failures reproduce identically on clean main (Windows sandbox env), unrelated to this change. - ruff format + ruff check clean on both changed files. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** analysis of the timestamp conventions, implementation, and regression tests authored with AI assistance; change reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output. |
||
|
Tu Naichao
|
ae82f426bf
|
fix(summarization): stop fraction triggers from crashing the agent build (#4901)
* fix(summarization): resolve fraction triggers from declared context_window, degrade instead of crashing the agent build A fraction trigger/keep clause requires profile["max_input_tokens"], which any third-party OpenAI-compatible model lacks, so SummarizationMiddleware construction raised ValueError out of create_summarization_middleware and failed the whole agent build (#3103). - factory: translate a declared model context_window into the langchain profile (metadata-only, never reaches the provider payload); explicit caller/override profiles win - summarization factory: drop unusable fraction trigger clauses (absolute clauses survive), fall a fraction keep back to the messages default, and disable compaction with an actionable warning only when no usable trigger clause remains — the agent build never dies from summarization config - docs: config.example.yaml, ModelConfig.context_window, summarization.md * refactor(summarization): share the default keep constant with the fraction fallback The fraction-keep degradation fallback hardcoded ("messages", 20), duplicating SummarizationConfig.keep's default_factory literal. Move the value to a shared DEFAULT_KEEP constant so the two cannot drift apart. * fix(summarization): keep trigger-null + fraction-keep constructing after degradation A trigger of None with a fraction keep hit the all-clauses-dropped branch (has_usable_trigger=False) and disabled compaction, and the accompanying warning claimed configured triggers were all fraction-based when none were configured. Only report nothing-usable when trigger clauses actually existed; trigger:null keeps constructing the never-firing middleware with the degraded keep, matching its behavior outside the degradation path. * fix(summarization): address review — keep manual compaction, validate ContextSize, pin wiring Review follow-ups on #4901: - When every configured trigger is a dropped fraction clause, keep constructing the never-firing middleware (trigger=None) instead of returning None: manual /compact runs with force=True and never consults trigger clauses, so it must keep working for a profile-less model rather than reporting 'compaction is disabled'. The warning now says auto-compaction will not fire while manual compaction remains. - ContextSize gains a config-load validator: fraction values must be in (0,1] (a percent-style 80 instead of 0.8 previously produced a threshold the context could never reach — a silently inert trigger), absolute values must be positive. - New un-monkeypatched integration test pins the shipped wiring (context_window declared -> real factory attaches profile -> fraction clause survives -> middleware constructs), which the stubbed middleware-side tests and kwarg-capturing factory-side tests each stopped short of. - Docs (summarization.md + config.example.yaml) clarify that the fraction resolves against the summary/anchor model's context_window (summarization.model_name when set, else the run model), including the mismatch caveat for a larger-window summary model. * fix(summarization): reject non-finite ContextSize values at config load YAML .nan / .inf pass pydantic's float parsing, and nan <= 0 is False, so the positivity check alone let them through as dead thresholds (count >= nan is always False) — the same silent-inert-trigger class the range validator was added to close. Guard with math.isfinite first, consistent with the existing non-finite guards on mem0 timeout_seconds and poll_after_seconds. * fix(summarization): merge context_window into inferred profile, require whole message counts - construct the model first, then merge max_input_tokens into the provider-inferred langchain profile: passing profile= to the constructor replaced the whole inferred metadata (tool_calling, structured_output, io capabilities, output limits) with the single key. An explicitly configured profile is still never clobbered. - reject non-integral ContextSize values for type=messages at config load: langchain slices the message list with them, so a float index raised TypeError mid-compaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Otavio Rodrigues Santana
|
27cb73659d
|
fix(auth): correct OAuth conflict error message + validate multi-worker Postgres claim with real concurrency benchmark (#5026)
* fix(auth): correct OAuth uniqueness error and index parity on Postgres
create_user() caught any IntegrityError on commit and always reported it
as a duplicate email. The email pre-check already rules out a real email
collision in the common case, so any IntegrityError reaching that handler
is actually idx_users_oauth_identity firing instead -- confirmed against
both backends: SQLite reports "UNIQUE constraint failed:
users.oauth_provider, users.oauth_id", Postgres reports a
UniqueViolationError naming the same index. The caller saw "Email already
registered" for an OAuth account conflict, which is wrong and would send
API consumers debugging the wrong field.
Distinguish the two cases via a substring check on the driver error text
(both backends name the oauth columns) and raise an accurate message for
each.
Also add postgresql_where to the same index, alongside the existing
sqlite_where. This is not a correctness fix -- verified empirically that
Postgres already enforces the same practical uniqueness without it
(NULL is never equal to NULL in either backends unique index, so real
duplicate (provider, id) pairs are already rejected and NULL/NULL rows
are already unconstrained). postgresql_where makes the index genuinely
partial on Postgres too, matching the stated intent in the surrounding
comment and keeping the index smaller as the common case (plain-password
accounts, both columns NULL) accumulates.
* test(bench): add multi-process SQLite vs Postgres concurrency benchmark
CONFIGURATION.md documents that multi-worker deployments must use Postgres
because "SQLite silently ignores row-level locks", but nothing in the repo
exercised that claim against real separate worker processes -- the existing
checkpoint benchmarks (scripts/benchmark/checkpoint/) measure single-process
read/write latency, and the existing Postgres tests
(test_pg_schema_integration.py, test_multi_worker_postgres_gate.py) cover
schema placement and config validation, not throughput or lock behavior
under concurrent load.
run_concurrency_bench.py spawns N real OS processes (subprocess.Popen, not
asyncio tasks or threads within one process) against the shared users
table, mixing reads (get_user_by_email) and writes (create_user) at a
configurable ratio, and reports throughput, error counts by exception
type, and p50/p95/p99/max latency per run.
Measured locally (2/4/8/16 workers, 100 ops/worker, 70/30 read/write):
SQLite completed all operations with zero errors at every worker count
(busy_timeout absorbs contention rather than raising), but total
throughput stayed flat around 28-34 ops/s regardless of worker count, and
p99 latency grew from ~400ms at 2 workers to ~5.9s at 16, with a 22s max.
Postgres throughput scaled with worker count (41 to 66 ops/s) and p99
stayed under 500ms at every worker count tested. Raw JSON output from
both runs is available on request; exact numbers will vary by machine and
are not asserted in the test suite.
test_bench_concurrency.py unit-tests the pure aggregation logic
(percentile math, error grouping, crashed-worker handling) the same way
test_bench_checkpoint_channels.py does for the existing benchmarks --
fast, no DB required, not the full multi-process sweep in CI.
* fix(auth): inspect the driver exception for OAuth conflict detection
str(exc) embeds the full failed INSERT statement, whose column list
always names oauth_provider/oauth_id, so a substring check on it
misclassified every commit-time IntegrityError on the users table as
an OAuth conflict (reproduced on SQLite: a duplicate primary key with
a different email raised "OAuth account already linked: None/None").
_is_oauth_identity_violation now inspects exc.orig instead: constraint_name
on Postgres, both violated column names present (not a bare "oauth"
substring) on SQLite.
Also ships the alembic revision idx_users_oauth_identity's postgresql_where
predicate needed: 0001_baseline created it as a full index on Postgres,
and ORM metadata changes only affect fresh create_all databases, never an
already-versioned deployment.
Addresses review feedback from willem-bd.
* fix(bench): run the concurrency benchmark in an isolated schema and derive paths from the checkout
--pg-url accepted an arbitrary database URL while the code pinned
postgres_schema="public" and unconditionally ran DELETE FROM users --
against any non-disposable database that permanently destroyed every
auth account. Each run now generates a unique throwaway schema
(bench_<uuid>), points both the seeder and every worker subprocess at
it via postgres_schema, and drops only that schema (DROP SCHEMA ...
CASCADE) once the full worker-count sweep finishes.
Also stopped hard-coding /opt/deer-flow/backend as the checkout path
and .venv/bin/python3 as the interpreter: BACKEND_DIR is now derived
from Path(__file__), and workers are spawned with sys.executable (the
orchestrator's own interpreter) instead, so the documented
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py
command works from any checkout.
Addresses review feedback from willem-bd.
* fix: shorten oauth-index revision id, repin migration-head assertions, fix bench read/write mix
- 0017_users_oauth_identity_partial_pg (36 chars) exceeded
alembic_version.version_num's VARCHAR(32) limit, which would fail
stamping/upgrading on both fresh and existing Postgres deployments.
Renamed to 0017_oauth_identity_pg_partial (30 chars).
- Repinned every test asserting 0016_subagent_batches as the migration
head (test_persistence_bootstrap[.py|_concurrency.py|_regression.py],
test_migration_0004/0007/0015) to the new 0017 revision id.
- worker.py's `(i % 100) < int(read_ratio * 100)` assumed n_ops >= 100;
at the documented default (50 ops/worker, 0.7 read ratio) it produced
either all-reads or all-writes, never the claimed mixed workload.
Replaced with read_count()/is_read_op(), which distribute an exact
round(n_ops * read_ratio) reads evenly across the sequence via modular
spacing, and added test_bench_worker.py covering the default values
plus small op counts.
* fix(bench): establish a real physical connection before timing ops
async with sf(): pass entered an empty AsyncSession without checking out
a physical connection -- SQLAlchemy stays lazy until the first statement
executes. That pushed connection-establishment cost onto each worker's
first timed operation instead of conn_time_s, and at 16 workers those 16
cold first-ops (1% of a 1600-op sample) could skew the reported p99.
Execute a real `SELECT 1` before starting the timer instead.
Verified with a real end-to-end run (uv sync + sqlite backend, 2
workers/10 ops, 0 errors) plus the full auth/bench/migration-bootstrap
suites (135 tests) and ruff check/format, all clean.
* fix(bench): synchronize workers before timing, fix percentile off-by-one
Two remaining measurement issues from review:
- run_workers() started the wall clock before spawning any worker, so
throughput/wall_time absorbed N processes' staggered Python-startup and
connection-establishment cost, and early workers could run ahead of ones
still starting. Workers now print READY right before their timed loop
and block on stdin for a GO signal; the orchestrator waits for every
READY, then starts the timer and releases all workers together.
- summarize()'s pct() used int(len(latencies) * p) directly as a
zero-based index -- a one-based-rank-as-index bug that put p95 and p99
at the same slot (the max) for any 20-or-fewer-sample run, and for the
documented 100-sample default. Now delegates to
checkpoint_bench_common.percentile(), the already-correct nearest-rank
implementation used elsewhere in the same benchmark family, instead of
a second, broken one.
Verified: 14/14 unit tests pass (2 new pinned-value regression tests for
the percentile bug, using the reviewer's own 20-sample repro), ruff
clean, and a real 2/4-worker SQLite multi-process smoke run completes
with distinct p95/p99/max latencies and no hang.
* fix(bench): absolute SQLite bench path, surface crash diagnostics, exit nonzero on failure; share OAuth index constant + cover Postgres branch
Three more findings from review at 5fd25a7:
- seed_baseline() cleaned an absolute .deer-flow/bench_data path but
handed DatabaseConfig a relative one, which resolves against the
CALLER's CWD -- not BACKEND_DIR. Invoking the documented command from
anywhere other than backend/ silently pointed the seeder and the
(cwd=BACKEND_DIR) workers at two different directories: workers crashed
with 'unable to open database file' while the run still printed a
well-formed summary and exited 0. Both seed_baseline() and worker.py's
make_session_factory() now use the same absolute path.
- Crashed workers' stderr was captured then discarded, and main() always
exited 0 -- an all-crashed sweep was indistinguishable from a real
(uneventful) measurement to anything checking the exit code or
--out. run_workers() now prints each crash immediately and tags it with
the real worker_id (previously always None); summarize() exposes
crashed_worker_errors alongside the existing crashed_workers count;
main() exits 1 via the new summary_indicates_failure() whenever any
sweep crashed or fell short of expected_total_ops.
- idx_users_oauth_identity was hardcoded separately in the ORM Index and
in _is_oauth_identity_violation's Postgres branch, with no test to
catch drift, and that branch had zero non-skipped coverage (its only
guard needs a live Postgres CI never configures). Exported
OAUTH_IDENTITY_INDEX_NAME from user/model.py as the shared source of
truth (migrations intentionally keep their own frozen literal, matching
every other revision in that package) and added stub-exception unit
tests pinning both the asyncpg constraint_name path and the sqlite
message-substring path, positive and negative.
Verified: 107 passed locally (auth + bench-unit suites), ruff clean, and
two real reproductions -- invoking run_concurrency_bench.py from a
scratch directory outside backend/ (the reviewer's exact repro) now
completes 8/8 ops with crashed_workers: 0 instead of crashing, and the
new crashed_worker_errors/exit-code logic is exercised directly by the
new unit tests against the real summarize()/summary_indicates_failure().
* fix(auth): attribute create_user IntegrityErrors to the right constraint
Two coupled review findings on the classification helpers:
P3 (fall-through) -- after ruling out the OAuth-identity index, create_user
raised "Email already registered: {email}" for every remaining
IntegrityError, including the duplicate-primary-key case the new
regression test exercises, whose address is not registered. Added
_is_email_violation() so the email message is used only for an actual
users.email collision that raced past the pre-check; anything else (in
practice a duplicate id) now raises a neutral
"User already exists (constraint: <name>)".
P2 (unreachable asyncpg branch) -- exc.orig is not the asyncpg error.
SQLAlchemy's asyncpg dialect re-raises its own DBAPI IntegrityError
(pgcode/sqlstate only) 'from' the real asyncpg error, so constraint_name
lives on exc.orig.__cause__. getattr(exc.orig, "constraint_name", None)
was always None on Postgres; the helpers only worked there by accident,
matching asyncpg's DETAIL line in the message fallback. Added
_driver_constraint_name() which walks orig then orig.__cause__, and the
stub tests now model that real shape (orig wrapper + __cause__) instead of
a constraint_name that no driver puts on orig directly.
Tests: 76 passed. New coverage for the email-race path, both new helpers
on each backend, the neutral fallback message, and the cause-chain walk.
* fix(bench): match app SQLite PRAGMAs in workers; fail a sweep on any op error
Two review follow-ups:
- worker.py opened its SQLite engine with only connect_args timeout=30.
synchronous and foreign_keys are per-connection PRAGMAs, so workers ran
at SQLite's synchronous=FULL / foreign_keys=OFF while a real Gateway
worker runs synchronous=NORMAL (persistence/engine.py::_enable_sqlite_wal)
-- an extra fsync per commit on the measured 30%-write path, overstating
SQLite's cost in the direction that flatters the "use Postgres"
conclusion. Added a connect listener applying the same four PRAGMAs, with
a test asserting synchronous/foreign_keys/journal_mode on a real worker
connection.
- summary_indicates_failure() only looked at crashes and the completed vs
expected op counts, so a sweep where every op completed but raised
(e.g. writes hitting OperationalError) passed as a clean measurement:
completed_ops == expected, 0 crashes. Added an "errors > 0" clause; the
error breakdown stays in the JSON, only the exit code changes. Test added.
test_bench_concurrency.py + test_bench_worker.py green (20), plus a real
2-worker sqlite smoke run (6/6 ops, 0 errors, exit 0).
* fix(auth): match the real email index name; only claim "exists" for uniqueness
Review follow-ups on the classification helpers:
- email is mapped_column(unique=True, index=True), which SQLAlchemy and
0001_baseline realise as a single UNIQUE INDEX (ix_users_email), not a
named UNIQUE constraint. _is_email_violation compared the driver
constraint name against "users_email_key", which Postgres never emits,
so that arm was dead on Postgres (SQLite matched via the message). Fixed
to ix_users_email.
- the residual IntegrityError fallback raised "User already exists" for
every remaining IntegrityError -- a NOT NULL / CHECK / foreign-key
violation is not a "user already exists" condition and is not part of
create_user's ValueError contract. Added _is_uniqueness_violation
(sqlstate 23505, or the SQLite "UNIQUE/PRIMARY KEY constraint failed"
message); only that raises the "already exists" ValueError, everything
else propagates unchanged.
- documented scripts/benchmark/concurrency/ in backend/AGENTS.md alongside
the other benchmark family.
Tests: 78 auth + 20 bench-unit pass, ruff clean. New coverage for
_is_uniqueness_violation on both backends and for a non-uniqueness
IntegrityError propagating out of create_user.
* fix(bench): don't pre-close worker stdin (breaks communicate); require --pg-url for postgres
* fix(bench): ruff format; time throughput on the op phase, not teardown
- lint-backend: ruff format the files touched in this PR.
- Throughput window (P2): the orchestrator sampled its wall clock after
every worker's communicate() returned, so it also covered each worker's
engine.dispose(), result serialization and stdout transfer. Each worker
now times just its operation phase (GO -> last op) and reports
ops_elapsed_s; summarize() uses max(ops_elapsed_s) over the workers -- all
released by the same GO -- as the throughput window (ops_window_s).
- Exercise migration 0018 (P2): test_user_oauth_partial_index.py goes
through bootstrap create_all(), which builds the partial index from ORM
metadata and never runs 0018.upgrade(). New Postgres-gated
test_migration_0018_oauth_identity_pg_partial.py alembic-upgrades to 0017
(full index), then 0018 (asserts the predicate appears), then downgrades
(asserts the full index is restored) and re-upgrades.
* docs(middlewares): tighten SandboxAudit and Clarification entries in AGENTS.md
PR #5134 grew agents/middlewares/AGENTS.md ~1.8 KB, pushing the effective
AGENTS.md chain for that directory over the 96 KiB hard limit once this
branch also documents scripts/benchmark/concurrency/ in backend/AGENTS.md.
Condense the two longest middleware entries (SandboxAuditMiddleware,
ClarificationMiddleware) without dropping any identifier, example, issue
reference, ordering constraint, or documented gap; chain back to ~96.8 KiB.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Ishaan Potle
|
8ee3c83508
|
fix(browser): keep references to detached live-frame tasks (#5155)
BrowserSession scheduled three coroutines with a bare
asyncio.ensure_future(), so nothing held a reference to the resulting
tasks. The event loop only keeps weak references, so such a task can be
garbage collected before it finishes.
For the two live-frame schedulers the consequence is worse than losing
the task. Each sets a *_pending guard before scheduling and clears it in
a finally block:
self._settle_live_frames_pending = True
asyncio.ensure_future(self._settle_live_frames())
If the task is collected, the finally never runs, the guard stays True
forever, and every later _schedule_settle_live_frames()/
_schedule_input_live_frame() call returns early — silently stopping live
frame refresh for that session with no error.
Add _spawn_background(), which retains the task in a set and discards it
on completion, and route the three call sites through it. This matches
the pattern already used in task_tool, session_pool, notify and others.
Add regression tests asserting the task is retained across a gc.collect()
and released once it completes.
|
||
|
SPEC
|
822c7bca4b
|
fix(memory): cancel buffered extraction when agent is deleted or cleared (#5123)
* fix(memory): cancel buffered extraction when agent is deleted or cleared * fix(memory): cancel buffered work before agent delete Address review: cancel before/after delete to close the rmtree race, scope user_id=None cancels to the legacy root only, and import memory helpers at module scope. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * fix(memory): close remaining cancel races from review Post-clear cancel, legacy-only all_agents scope, always cancel even when memory is disabled, and fold cancel+delete into one offloaded thread. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * docs(memory): align cancel_by_agent None-scope with legacy root Document that user_id=None cancels only the legacy no-user bucket, matching clear/storage semantics, not the whole process-local queue. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * test(memory): fix cancel_by_agent docstring regression assertion Signed-off-by: SPEC <zt1y17@soton.ac.uk> * fix(memory): address final cancel review nits Type the delete helper with AgentStore, replace docstring pinning with a kwargs mapping test, and document scoped cancel + residual window in AGENTS.md. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * fix(memory): resolve agent store inside delete worker thread get_agent_store() does blocking config/FS work; keep it off the event loop so test_delete_agent_does_not_block_event_loop and backend-blocking-io CI pass. Signed-off-by: SPEC <zt1y17@soton.ac.uk> --------- Signed-off-by: SPEC <zt1y17@soton.ac.uk> |
||
|
Willem Jiang
|
037658b0ee
|
fix(ci):reduce the size of AGENTS.md in sandbox (#5146) | ||
|
gus
|
47f43f79f4
|
fix: improve local environment detection guidance (#5111)
Co-authored-by: angus-guo <217034332+angus-guo@users.noreply.github.com> |
||
|
Aari
|
9e0fbd60fa
|
fix(sandbox): isolate concurrent subagent shell sessions (#5134)
* fix(sandbox): isolate concurrent subagent shell sessions * fix(sandbox): make execution acquire idempotent * fix(sandbox): close execution lifecycle gaps * fix(sandbox): serialize retained client lifecycle * fix(sandbox): close remaining client lifecycle gaps * fix(sandbox): unwind failed client lookup * fix(sandbox): protect internal lease identities * fix(sandbox): make cancellation reconciliation durable * fix(sandbox): fence cancelled workers and IM uploads |
||
|
hataa
|
08b27aef73
|
feat(auth): make login rate-limit parameters configurable, fixes #5108 (#5110)
* feat(auth): make login rate-limit parameters configurable, fixes #5108 Add auth.local.max_login_attempts (default 5) and auth.local.lockout_seconds (default 300) so operators can tune the per-IP login throttle: raise the ceiling for shared-egress-IP offices behind proxies/NAT, or tighten it for stricter posture. Policy is live-read per call (matching the _local_registration_enabled precedent), so a config reload applies without a Gateway restart; raising the threshold mid-lockout immediately unblocks affected IPs. Review feedback addressed (willem-bd): - Only FileNotFoundError falls back to the hardcoded defaults; a malformed config propagates, mirroring _local_registration_enabled, so an operator who tightened the policy never silently gets the more permissive defaults. - _check_rate_limit looks up the record before resolving the policy, so a clean IP pays zero config reads (get_app_config re-hashes config.yaml per call and login_local is an unauthenticated async endpoint). Bumps config_version to 39 in config.example.yaml and the Helm chart (values.yaml + README example) so the chart drift check stays green. * fix(auth): reject max_login_attempts=1 and honor live lockout_seconds for active lockouts * fix(auth): close live-policy state gaps in login throttle (resurrection, count reset, broken-config verification) * fix(auth): commit evaluated lockout duration on decreases too, preventing raise-resurrection * test(auth): pin broken-config fail-closed sequence through the login route * fix(auth): sweep expired locks by stored sentence and keep policy reads off the event loop * fix(auth): re-read throttle record after the policy-resolution yield point --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
nicochow
|
bbcfd368bf
|
fix(sandbox): scrub SSH_AUTH_SOCK from the sandbox subprocess env (#5145)
SSH_AUTH_SOCK points at the host's ssh-agent socket. A sandbox subprocess that inherits it can sign and authenticate with every key the agent holds (git push, ssh logins) without reading any key file -- the same credential-pointer leak class as the *_ASKPASS helpers the env policy already scrubs deliberately. No wildcard pattern fits (*AUTH* would strip benign names), so add an exact entry to _BLOCKED_EXACT_NAMES. A skill that genuinely needs the agent socket can still declare it via required-secrets: injected values win over the blocklist by design. Co-authored-by: zhouyujie <zhouyujie@keep.com> |
||
|
qian
|
30788c79ff
|
fix(title): ignore upload context in conversation titles (#4729)
* fix(title): ignore upload context in conversation titles * fix(title): cover attachment-only conversations * fix(title): skip model for attachment-only messages * fix(title): handle whitespace-only user content --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
wutongyuonce
|
fb722770e4
|
fix(agents): do not hide invalid config with file fallback (#4952)
* fix(agents): do not hide invalid config with file fallback * test(agents): cover invalid on-disk config fallback * fix(agents): resolve stores off the event loop * fix(agents): distinguish missing nested config from main config * fix(agents): reject missing explicit config path * test(agents): isolate config fallback test * test(agents): isolate router blocking IO coverage * test(agents): pin malformed config.yaml parse-error propagation An unparseable config.yaml used to be swallowed by the broad except Exception and silently downgrade to FileAgentStore. The narrowed except FileNotFoundError already propagates yaml.ParserError/ScannerError; pin that contract with a real on-disk config instead of monkeypatched get_app_config. |
||
|
AoHanBei
|
9b32b5d841
|
feat(observability): persist loop detection events (#5127)
* feat(observability): persist loop detection events * fix(observability): persist subagent loop events * fix(observability): narrow subagent loop event bridge * fix(observability): attribute subagent loop events * fix(tests): isolate subagent executor imports |
||
|
Jeremy Schoemaker
|
6b4f803354
|
fix(sandbox): preserve trailing whitespace in filenames from list_dir and glob in remote providers (#4980)
* fix(sandbox): stop stripping filenames when parsing find output in remote providers The list_dir and glob parsers in the e2b, OpenSandbox, AIO, Tenki, and BoxLite providers called .strip() on every line of find output. A filename that legitimately ends (or begins) in whitespace was corrupted, so the listed path never resolved on any follow-up file API call, and the remote providers diverged from LocalSandbox, which preserves such names via pathlib. splitlines() already removes the line terminators, so filter empty lines only and keep each entry verbatim. Same class of bug as the e2b _sync_outputs_to_host fix (#4861), applied to the search parsers. Adds a trailing-space regression test per provider at the seam each suite already uses. * fix(sandbox): split find output on \n only, and rename the tenki test Review follow-ups from willem-bd: - aio_sandbox.list_dir used str.splitlines(), which also breaks records on \v, \f, \x1c-\x1e and \x85 - all legal inside a Linux filename, and all contrary to this PR's own rule that the newline is the only delimiter. find emits \n and nothing else, so split("\n") is the correct parse. - Renamed test_search_preserves_trailing_space_in_filename to test_list_dir_and_glob_preserve_trailing_space_in_filename, matching the sibling tests in test_opensandbox_provider.py and test_boxlite_provider.py. The body covers list_dir and glob; it never touches grep. |
||
|
Madan kumar
|
df68b59149
|
fix(skills): reject a blank SKILL.md description at the write gate (#4867)
* fix(skills): reject a blank SKILL.md description at the write gate
_validate_skill_frontmatter only applied its description rules when the
value was truthy, so a blank or whitespace-only description passed the
gate. The loader rejects it, so the file was written to disk and then
disappeared from every consumer.
On the PUT edit endpoint that meant the write committed first and the
response was a 404 -- the previously working skill was gone with no
rollback. Same shape via the .skill install path and the agent-facing
skill_manage tool, which reported success for a skill that never loads.
Empty names were already rejected; description was the only field where
the write gate and the loader disagreed.
* test(skills): pin rollback rejection of a blank-description history entry
Add a regression test for the rollback path: restoring a history entry
whose stored content has an empty description must return 400
("Description cannot be empty") and leave the on-disk SKILL.md untouched,
rather than the previous destructive path that wrote the unloadable
content and then 404'd. On main this returns 404, so the test also pins
the intended status-code change on this path.
|
||
|
Ricky-7-Yan
|
c56c7293f8
|
test(checkpoint): measure postgres storage growth (#5051) | ||
|
rayhpeng
|
cd35363a05
|
fix(history): early user messages vanish or jump mid-run when pagination and context compaction overlap (#4696)
* fix(history): stop dropping user messages that fall outside the loaded page window Two independent paths made a user's own message disappear from a long thread (#4666, #4508, #4363). Both are reproduced by a real two-round run: once the thread passes the 50-row `/messages/page` window AND context compaction fires, the two sources of truth stop overlapping at the head. 1. Middleware-answered tool results never reached the event store. A middleware that short-circuits a tool call (e.g. ReadBeforeWriteMiddleware's blocked write) returns a user-visible ToolMessage, but LangChain never emits `on_tool_end`, so RunJournal never persisted it — the user saw it during the run and it vanished on reload. RunJournal already reconciles final-output tool messages, but only for an `ask_clarification` allowlist. The allowlist is removed; scope stays bounded by the three conditions that actually matter (visible, this run's lead agent, not already persisted), so subagent results still stay in their own step feed. 2. mergeMessages discarded the checkpoint prefix before the first shared anchor. #4065 correctly established that a summarization-rescued early message must not be appended to the tail, and suppressed it instead. That suppression is what deletes the message when the first history page no longer reaches back to it. It is now woven in before the first shared anchor — the one position both the checkpoint and seq-sorted history agree on — so #4065's invariant (never the tail) still holds. A collapsed unloaded gap is recoverable by paging; a dropped message is not. Verified against real captured payloads from the reproducing run: the first user message returns to the transcript. Its exact position is still approximate — after compaction the live window carries too few anchors to place it precisely, which only seq-based ordering can close. Backend: 10809 passed (baseline 10808; same 15 pre-existing failures in browser/crawler community tools). Frontend: 986 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(events): look up a persisted message's seq by identity Groundwork for placing checkpoint messages in the seq-ordered thread feed (#4666). A checkpoint carries no seq of its own and loses messages to summarization, so once the feed's 50-row page window no longer reaches back to a surviving old message, a client has nothing to place it by. The seq already exists in run_events keyed by the message id — this exposes it without paging the whole feed. `message_identity` is the backend half of the identity rule the frontend applies in `hooks.ts::messageIdentity`: a ToolMessage is keyed by `tool_call_id`, and DynamicContextMiddleware's `X` / `X__user` human copies collapse to one identity. The two halves must stay in sync — a mismatch is silent, degrading placement rather than raising. `get_message_seqs` is implemented for all three stores. Misses are absent from the result rather than an error, so callers degrade to their own placement rule; the earliest seq wins when one identity resolves to several rows, so a re-persisted message keeps the position it first occupied. The DB store decodes rows in Python because `content` is a TEXT column holding a JSON string, not a JSON column — the identity fields cannot be projected in SQL. Nothing consumes this yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(runtime): carry each persisted message's feed seq on values frames Attaches `additional_kwargs.deerflow_seq` to messages in a root `values` frame that the thread feed already holds, so a client can place a message the checkpoint kept but its loaded history page window no longer reaches (#4666). Nothing is written back to the checkpoint: the seq is added when the frame is serialized and belongs to that frame only. Cost is bounded to frames introducing identities the run has not resolved yet. Messages this run produces are not in the feed while streaming, so they are looked up once, recorded as misses, and never retried — in a real run the only frame that pays for a query is the one where compaction brings older messages back into view. Measured on a reproducing two-round run: 1 lookup across 25 values frames. The stamper is built once per run rather than per `_stream_once`, or a goal continuation would discard the resolved seqs. Subgraph frames are not stamped: a subagent's snapshot is not part of this thread's feed ordering. A lookup failure logs and leaves the frame unstamped rather than failing it — placement is an enhancement and clients fall back to their own ordering rule. Frontend does not read the field yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(gateway): strip the server-owned message seq from untrusted input `deerflow_seq` is display metadata the Gateway attaches when it serializes a values frame. A client replaying messages (regenerate / edit-and-rerun) would otherwise write it into the checkpoint, where it becomes wrong the moment the thread is forked — a branch re-seeds its feed and reassigns seq (#4380). Joins the existing server-owned key set, so it follows the same trusted-internal rule as the dynamic-context and view-image markers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): place a checkpoint message by its feed seq, not its nearest anchor Completes #4666. Weaving a compaction-rescued message before the first shared anchor keeps it in the transcript, but not in the right place: after compaction the live window carries too few anchors, and the nearest one can sit deep inside the loaded page window — measured at row 25 of 50 on a reproducing run, which is why the first user turn rendered mid-transcript instead of at the head. Both sides now carry the backend's thread-global seq. `buildVisibleHistoryMessages` copies each row's `seq` onto the message (same shape as the existing `run_id`), and the Gateway stamps it onto `values` frame messages it has already persisted. A live message whose seq is below the loaded window's lower bound is placed ahead of everything on screen rather than before the nearest anchor. A message with no seq — still streaming, so not in the feed yet — keeps the weaving path, since the tail is already its correct position. Verified against the captured payloads of the reproducing run: the first user message goes from absent, to #13 (behind the second question), to #0. Frontend: 988 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): place a pre-window checkpoint message even when no anchor is shared Also #4666. Placing a compaction-rescued message by its feed seq was gated on reaching a shared anchor, because the split ran inside the anchor walk. When the loaded page and the live checkpoint share no identity at all, that walk never runs and the message fell through to `[...canonical, ...live]` — appended after the entire window, the one arrangement #4065 proved wrong, with its seq known the whole time. That is not a corner case. Open an old, already-summarized conversation and send a message: the page on screen is the newest rows from before that turn, while the checkpoint holds the rescued first user turn plus steps of the new run that are not in the feed yet. On a reproducing run the two sides shared zero anchors and the user's own first question rendered at row 50 of 50 — the reported "first message jumps to the bottom". Split `beforeWindow` out of `live` before walking anchors, walk `liveInWindow`, and use it for the no-anchor branch as well, so a message routed ahead of the window is not re-appended at the tail by dedup. Measured on captured payloads of a reproducing run (real gateway, real compaction), first user message position: no shared anchor: row 50 -> row 0, seq order monotonic again shared anchors: row 0 -> row 0 (unchanged) paged to the top: row 0 -> row 0 (unchanged) Regression test verified red-green: reverting the fix fails it with the message rendered after the window. Frontend: 989 passed, eslint + tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(gateway): stamp the message feed seq on checkpoint reads, not only on stream frames Completes #4666. `_MessageSeqStamper` sits on the streaming publish path, so a client that joins a live run learns where a summarization-rescued turn belongs while a client that merely opens the conversation does not — and opening is the common case. `GET /threads/{id}/state` and `POST /threads/{id}/history` returned the checkpoint with no seq at all, so the merge fell back to the nearest shared anchor, which after summarization sits deep inside the loaded page. Reproduced in a browser against a real gateway, on a thread that had already compacted: the user's first question rendered at row 320 of 389, behind the newest question instead of at the head. Both reads showed 0 of 13 messages carrying a seq. That is the reported symptom, still present after the streaming fix. Add `stamp_messages_with_seq`, the request-scoped counterpart of the stamper: everything a checkpoint still holds is already persisted, so one batched lookup resolves the whole list and there is nothing to retry later. Resolve the store through `_optional_run_event_store` rather than `get_run_event_store`, because seq is placement metadata — a deployment without a feed must still be able to read a thread. After the fix, on the same thread in the same browser: 13 of 13 messages carry a seq and the first question renders at the head, ahead of the newest one. Backend: ruff clean, 326 passed across the touched suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(harness): move the injected-user-id suffix helpers to utils.messages to break an import cycle message_identity imported strip_injected_user_message_id_suffix from the dynamic-context middleware, closing a cycle (middleware -> deerflow.runtime -> worker -> events -> middleware) that only stayed hidden while an earlier import happened to break it. Define INJECTED_USER_MESSAGE_ID_SUFFIX and the strip helper in deerflow.utils.messages and re-export them from the middleware so existing importers keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(docs): improve formatting and clarity in AGENTS.md and message-merge.test.ts * perf(events): stop the seq scan once every wanted identity is resolved Rows past the last wanted seq can only be re-persisted copies that already lose the earliest-seq-wins tiebreak, so all three stores now break out of the scan (and the db store out of its per-row JSON decoding) once found covers wanted. Matters most for /state and /history reads of long threads, where this lookup runs with no run cache and a typically tiny wanted set. Raised by review on #4696. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(events): share the seq-stamping expression between the two stampers The walrus-plus-merge expression was duplicated verbatim between stamp_messages_with_seq and _MessageSeqStamper.stamp — two counterparts of one rule where silent divergence is the likely failure mode if only one side is edited. Both now call attach_message_seq next to MESSAGE_SEQ_KEY in message_identity.py. The trailing isinstance(message, Mapping) guard was unreachable (a non-Mapping entry already got identity = None) and is gone with the extraction. Raised by review on #4696. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(events): seq stamping survives launch paths without user context The db store's get_message_seqs defaults to user_id=AUTO, which raises when no user is in the contextvar — the first strict-AUTO read ever called from the worker context. On a launch path that never inherits the auth context (e.g. a null-owner scheduled task), stamp()'s except clause swallowed that into a per-frame warning and silently disabled seq stamping for exactly the background runs that need it. The stamper now soft-resolves the user id once at build time — the same rule as the worker's write paths beside it (unset -> no filter) — and passes it explicitly. jsonl/memory stores gain the same user_id kwarg the base list_messages contract already carries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(events): SQL-prefilter the message seq lookup's candidate rows get_message_seqs scanned and JSON-decoded every message row of the thread: the early exit never fires when a wanted identity is absent from the feed (a message still streaming, or checkpoint-only), and /state / /history reads want the newest messages, so the ascending scan traversed essentially the whole feed — with the content column carrying full tool outputs, that is heavy I/O plus N JSON parses on exactly the long threads this lookup exists for. A LIKE prefilter now keeps that cost in SQL: only rows containing a wanted raw id as a substring are fetched and decoded. False positives are re-checked by message_identity; LIKE wildcards are escaped; an id json.dumps would escape (breaking the verbatim-substring guarantee) falls the whole set back to the full scan rather than silently missing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): sink runtime mechanism docs below the gateway guidance budget Merging main pushed backend/app/gateway/AGENTS.md past its 40KB soft budget (main had left 81 bytes of headroom). Per the nearest-file rule, move the mechanism detail of the message-seq stamping and run-delivery receipt sections — both owned by runtime/ code — into packages/harness/deerflow/runtime/AGENTS.md, leaving the gateway file the REST-surface summary and a pointer. The seq section also documents the stamper's build-time soft user-id resolution and the db store's SQL prefilter from the review follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): sink durable-MCP task detail below the backend guidance budget Merging main pushed backend/AGENTS.md past its 24KB module soft budget (main itself is at 24762 after #4848 — this branch adds zero net bytes to the file). Per the nearest-file rule, move the two durable-MCP task runtime bullets' mechanism detail into packages/harness/deerflow/mcp/AGENTS.md, leaving summaries and pointers; this also restores ~2KB of headroom so the next merge does not trip the same wire. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(events): re-ask a message-seq miss once the feed advances The run-scoped stamper cached lookup misses for the whole run. A message this run produces reaches a values frame before RunJournal flushes it, so its first lookup legitimately misses — and the journal persists it moments later, giving it a feed seq the stamper never asks for again. A long run that afterwards rolls past the history page and compacts then carries that message unstamped, back to the approximate anchor placement this stamper exists to replace (#4666). A transient store error had the same permanent effect, since the except clause degrades to an empty result. A miss is now provisional while a hit stays final: RunJournal counts its successful event-store writes as `feed_generation`, and the stamper re-asks a missed identity only once that counter moves. Retrying is therefore bounded by feed writes rather than by frames — the per-frame query the run-scoped cache was built to avoid — and a failed lookup costs one generation instead of the run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Ryker_Feng
|
8d8ca506ba
|
feat(artifacts): download run files as zip (#5117)
* feat(artifacts): download run files as zip * fix(artifacts): address archive review feedback * fix(artifacts): gate unavailable archive downloads * fix(artifacts): verify archive availability * fix(artifacts): harden archive consistency * fix(artifacts): reject archive path aliases |