mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-02 00:20:40 +00:00
89 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
llb555
|
794be98704
|
fix(client): keep a decided batch outcome through capture cleanup (#7517) | ||
|
XuRuuuy
|
c9d3e48a4a
|
fix(uploads): never truncate an existing upload on same-name collision (#7388)
* fix(uploads): never truncate an existing upload on same-name collision Same-name uploads silently destroyed the earlier file on disk: the shared write path (open_upload_file_no_symlink) opened the destination with O_WRONLY|O_CREAT (no O_EXCL) and ftruncate(0), so re-uploading report.pdf opened and emptied the report.pdf already stored for the thread, and two concurrent same-name uploads could race on one inode. The Gateway ingestion pipeline already refuses overwrites (staged no-overwrite link commit), but the embedded client upload path (copy_upload_file_no_symlink) and the channel ingress paths (write_upload_file_no_symlink) still went through the truncating open. Open every upload destination with O_EXCL. On FileExistsError, claim the next free _N suffix against the directory's current contents via the existing claim_unique_filename helper and retry — the same deterministic behavior the Gateway pipeline documents (name_1.ext). The existing file is never opened, so its bytes survive intact under any future naming policy. Embedded-client and channel inbound-file responses now report the name the bytes actually landed under instead of the requested name. * test(uploads): cover concurrent same-name writers * fix(uploads): skip known copy collisions before staging --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
SumEn
|
3ccc876493
|
fix(client): preserve existing Markdown when publishing upload companions (#7054)
* fix(client): preserve existing Markdown when publishing upload companions * test(client): explain running-loop conversion coverage * ci: upgrade Playwright to 1.62.1 for Ubuntu 26.04 Backport the already-merged upstream #7219 fix to unblock the older #7054 replay workflow, which still locks Playwright 1.59.1. (cherry picked from commit e790e7e6bab37c6e61d61f36c65f15d79a8655f8) --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_Feng
|
d1d0958bea
|
feat(skills): add guarded evolution host APIs (#5645)
* feat(skills): add guarded evolution host APIs * docs(agents): avoid growing saturated guidance chains * fix(skills): address mutation host review findings * fix(skills): order name fence before projection lock * fix(skills): preserve composite toggle lock order * fix(skills): address catalog and pagination review feedback * docs: keep subagent guidance within merged instruction budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
haotianqi666
|
9c74b1529b
|
fix(client): accumulate incremental streaming token usage (#6939)
* fix(client): accumulate incremental streaming token usage * test(client): cover OpenAI and vLLM stream usage * fix(client): clamp completed usage reconciliation deltas --------- Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
BaiHeng Peng
|
55b1f1cfba
|
fix(client): preserve text appended before the first state snapshot (#6910)
* fix(client): preserve text appended before the first state snapshot * test(client): exercise first-snapshot reconciliation with stable IDs --------- Co-authored-by: BaiHeng Peng <191328900+Aldenshile@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
lilei
|
b346b17fce
|
fix(client): convert the captured upload source (#6632)
* fix(client): convert the captured upload source * test(uploads): cover ACL sources and scope capture failures * fix(uploads): preserve concurrent uploads from one captured source Combine the embedded client's source capture with no-replace publication for first-time filenames and Markdown companions. Hold one private capture across collision retries, retain original descriptor metadata, and keep replacement behavior for names present when the call starts. Port concurrent-upload and staging-publication regressions from PR #6687. Extend source-edit/removal and real XLSX cases to both publication paths, and cover edits or removal during two filename collisions in ordinary and event-loop calls. Document capture, retry, publication, and cleanup contracts. Validation: related suites 469 passed, 24 skipped; full offline suite 28,788 passed, 488 skipped; blocking-I/O suite 245 passed, 1 skipped; Ruff, guidance, and diff checks passed. Nine race/collision cases failed on the #6632 baseline. Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com> * docs: keep agent guidance within CI budgets --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com> |
||
|
Willem Jiang
|
e5a72b5009
|
Revert "fix(uploads): convert the same source version the upload saved (#6621)" (#6763)
This reverts commit 9c071c7053379590b63618498fecd0a626ac95e6. |
||
|
Yufeng He
|
9c071c7053
|
fix(uploads): convert the same source version the upload saved (#6621)
* fix(uploads): convert the same source version the upload saved (#6600) DeerFlowClient.upload_files copied the caller's file into the uploads directory, then reopened the original path for Markdown conversion. A save landing between the two reads left the saved upload at version A while the companion registered version B, and resolve_companion still accepted the mismatched Markdown. Snapshot convertible sources into a private temp dir up front and run both the upload copy and the conversion from that one read. The snapshot keeps the #5611 invariant (conversion never reopens a name inside the sandbox-writable uploads dir), preserves metadata via copy2, and keeps the re-upload SameFileError rejection through an explicit same-file probe before the snapshot. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> * fix(uploads): pin the snapshot copy to one held source descriptor Review on #6621 found two gaps in the snapshot step: stat-by-name then copy2-by-name split identity from content across two path resolutions, and copy2 drops ACLs so a source readable only via ACL made the private snapshot unreadable. Open the source once, fstat it for the same-file rejection, and copyfileobj out of the held fd; restore the mtime from that same stat so the upload keeps the source's timestamps. Tests: the version-parity regression's injection now also fires when a future regression reads the caller's path during conversion, and a real macOS ACL case (mode 000 + read grant) uploads and converts cleanly. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> * test: catch upload conversion source regressions --------- Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
spud
|
75ce6f4920
|
fix(mcp): preserve server binding history across gateway workers (#6617)
* fix(mcp): preserve server binding history across gateway workers A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied. Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones. * fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call. Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer. * refactor(mcp): share stdio fingerprinting across lifecycle paths Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper. Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed. * docs(mcp): condense lifecycle guidance under the AGENTS chain limit The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract. * docs: trim inherited agent guidance below CI size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
787b44c74f
|
fix(client): count threads, not checkpoints, in list_threads (#6426)
* fix(client): count threads, not checkpoints, in list_threads checkpointer.list(None, limit=N) returns the newest N checkpoints across every thread, so one long conversation filled the limit and hid the rest: the TUI thread picker and --resume <title> only saw the latest one or two threads. List threads through their single step -1 root checkpoint instead, then read the latest checkpoint per returned thread. Add sort_by="updated_at" so --continue keeps resuming the most recently active thread rather than the most recently created one. * test(client): pin list_threads scan cost and namespace filtering; docs * docs(client): state list_threads cost as checkpoints loaded, not rows scanned * docs(changelog): reference #6426 * fix(client): list threads by checkpoint write order, including branches Discovering threads by their step -1 input checkpoint dropped Gateway branches, which are seeded through update_state (step 0), and ordering by checkpoint ts missed goal writes, which copy the previous ts under a fresh checkpoint id. Enumerate threads from the checkpoint primary-key index on SQLite and Postgres (unwrapping the delta-history cache), falling back to one list(None) walk for any other saver, and order by checkpoint_id, the key LangGraph itself uses for a thread's latest checkpoint. |
||
|
lilei
|
f840e843d3
|
fix: track converted upload ownership for outlines and listings (#6101)
* fix: track ownership of converted upload markdown * fix: invalidate overwritten companions and restore branch ownership * docs: explain legacy upload companion behavior on upgrade * docs: shorten upload companion guidance to fit instruction budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
d3e8e78f3f
|
feat(authz): middleware-declared tool path Layer 1 now covers tools declared by middlewares (#6105)
* feat(authz): extension tool authorization PR2 — middleware-declared tool path Layer 1 now covers tools declared by middlewares, which LangChain merges into the bound tool set after the host's explicit-list filter: - new agents/middlewares/tool_declarations.py: record the ordinary pass's verdict, collect declarations from the built stack, decide only the delta seeded by the first pass (a name denied for this build stays denied), narrow the stack build-locally on independent state-preserving copies (direct __dict__ write, refused up front for tools data descriptors, never mutating caller-owned instances), and verify the exact bound stack after the final normalization copy. - IsolatedMiddleware ships a state-preserving __copy__ so narrowing survives full/delta normalization and copies of copies. - TodoMiddleware degrades when write_todos is denied: no system-prompt injection, no completion/context-loss reminders, no jump_to. - Layer 2 forwards host-resolved tool provenance into AuthzRequest.context and binds the infrastructure exemption to the concrete host-created tool object instead of a name set (tool=None goes to the provider). - Wired into all four assembly paths (lead x2, native subagent with the decision offloaded via asyncio.to_thread, DeerFlowClient). Tests: new test_middleware_declared_tools.py (35) and test_layer2_tool_provenance.py (7); blocking-IO anchor pinning the subagent offload (teeth verified red->green); placement-guarantee and enforcement suites extended; executor test doubles converted for the async _create_agent. Behavior change called out in CHANGELOG: with authorization enabled and a tools policy that excludes write_todos, plan-mode builds no longer bind it (built-in RBAC default unchanged). * fix(authz): close residual gaps from review — narrow non-collectable declarations, dedupe candidates, warn on skipped pass - apply_declared_tool_view now removes non-BaseTool declarations from the bound view when authorization is enabled (they would otherwise bind via LangChain's create_tool auto-conversion with no provider decision); verify_declared_tool_view fails loudly if one survives normalization - decide_declared_tools dedupes never-submitted names so the provider sees each candidate once - subagent executor logs a warning when the provider is set but the seeded Layer-1 state is missing, instead of skipping the declaration pass silently - docs: middleware-declared tools must be BaseTool instances with usable names * fix(authz): fail loudly on non-list/tuple middleware tools declarations _iter_declared warned and returned () for a 'tools' attribute that is not a list/tuple, but LangChain's factory iterates the attribute as-is — so a set/generator/dict_values declaration bound with no Layer-1 decision. Raise DeclaredToolViewError instead, matching the module's fail-loud standard for every other un-narrowable shape. Disabled path remains a strict no-op. Docs updated to match. |
||
|
SuperSgdk
|
f74a290c42
|
fix(uploads): reject reserved staging filenames before batch ingestion (#6122)
* fix(uploads): reject filenames reserved for temporary staging Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace. * fix(uploads): report reserved names before HTTP batch ingestion Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation. * fix(uploads): reject Windows-style staging basenames on POSIX Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation. Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Fish
|
2000bffa8a
|
fix(harness): anchor DeerFlowClient agent-name validation with fullmatch (#6104) | ||
|
TAI-YUAN GUO
|
25671538af
|
fix(cli): exit non-zero when a headless run ends in an LLM error fallback (#6056)
* fix(cli): exit non-zero when a headless run ends in an LLM error fallback LLMErrorHandlingMiddleware turns provider failures (e.g. an expired credential returning 401) into an AIMessage flagged `deerflow_error_fallback` instead of raising. The Gateway run status and the subagent executor already treat that flag as a failure. #5963 gave `deerflow --print` / `--json` an error boundary for raised exceptions, but a flagged fallback never raises, so both modes still printed the fallback text and exited 0. Both headless modes now follow #5963's failure contract when the run's final AI message carries the flag: `--print` keeps the fallback text on stdout and adds an `Error:` line on stderr; `--json` appends the terminal `{"type": "error"}` record. Both exit 1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor(cli): share final-answer accumulation with the embedded client * docs: scope headless accumulator guidance to TUI --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
hataa
|
1cd3eb742b
|
feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3) (#4541)
* feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3) Phase 3 / Skills — the second of three resource-type PRs (Models, Skills, Sandbox). The RBAC provider already maps "skill" → config key "skills" (rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed. Layer 1 (assembly-time, mirrors Phase 1B): - filter_available_skills_by_authorization() in skill_filter.py filters the skill-name allowlist (set[str] | None) by the provider's "skill" policy. In lead agent, runs after _available_skill_names() so the catalog (describe_skill) and SkillActivationMiddleware share one filtered set. In subagent executor _load_skills(), filters config.skills before disk load. - filter_resources_by_authorization() in enforcement.py — generic version of filter_tools_by_authorization for any resource with a name attribute. Layer 2 (slash-activation): - No new middleware code needed. SkillActivationMiddleware._resolve_activation already checks "reference.name not in self._available_skills". Since Layer 1 filters the allowlist, denied skills cannot be slash-activated. authorization.enabled: false is a complete no-op. available_skills=None (no agent allowlist) + enabled: resolves all skill names from config and filters. 12 new tests + 243 existing authz/skills tests pass. * test(subagents): wait on the chained future instead of polling done() in caller-loop teardown test * feat(authz): enforce action-scoped skill:activate at runtime and reuse the catalog loader for filter candidates * feat(authz): arm skill:activate on the subagent chain, unify the client skill-surface user, and record the Layer 2 reversal in the design notes * chore(authz): align the subagent filter's fallback user with the chain's bucket convention * feat(authz): gate autonomous skill loading on skill:activate, close the bootstrap assembly leak, and reuse one provider per subagent assembly Review round 4 (willem-bd, CHANGES_REQUESTED): - [P1] Autonomous load path: describe_skill filters its match set on the action-scoped skill:activate decision (all-denied is indistinguishable from "no match"), and ToolErrorHandlingMiddleware stamps skill_context entries only when the action allows it — a denied SKILL.md read gets a skill_context_denied marker instead, so durable context, allowed-tools policy, and autonomous secret bindings never activate a denied skill. File visibility itself stays Layer 1 (sandbox-projection semantics). - [P1] Bootstrap assembly: build_middlewares/apply_prompt_template now receive the authorization-filtered set instead of the raw _BOOTSTRAP_SKILL_NAMES, and an empty skill_names index is preserved when deferred discovery is on rather than round-tripping through None into the legacy full-metadata prompt that re-advertises a denied skill (deferred-off keeps the legacy rendering; the filtered empty allowlist blanks it on denial). - [P2] SubagentExecutor resolves ResolvedSkillAuthorization once per assembly (_resolve_skill_authorization, memoized per executor) and shares it across the Layer 1 filter (authorization=), the activation middleware, and both new gates — uncached provider factories can no longer hand the layers different instances/policy snapshots. All gates share one helper (skill_filter.skill_activation_allowed) so slash, describe, and skill-file-load cannot drift; provider errors keep the configured fail-closed/fail-open policy. skill_authorization is threaded through _build_runtime_middlewares into both runtime builders. Tests: describe gating (deny/fail-closed/fail-open/disabled), read stamping (denied marker vs entry) + extract_skills skip without warning, bootstrap three-way param (allow/deny x deferred on/off), lead + subagent chain wiring for the stamp gate, and the distinct-instance provider factory repro. Each fix is mutation-verified (reverting it fails its test). Full A/B suite run: no new failures vs HEAD. * feat(authz): re-authorize persisted skill_context before reuse and route async paths through aauthorize Review round 5 (willem-bd): - [P1] Persisted entries: skill_context survives across runs, so both consumers now re-check the action-scoped skill:activate decision before reuse. SkillToolPolicyMiddleware._active_skills_for_paths skips a denied skill (same skip as a disabled/unallowlisted one; all-denied keeps the existing fail-closed-to-builtins semantics) and _in_context_secret_sources binds no secrets for a denied entry. skill_authorization is threaded into SkillToolPolicyMiddleware on both chains. - [P2] Async provider API: skill_activation_allowed_async mirrors the sync helper through await provider.aauthorize() (same pattern as authorize_sandbox_execution_async). Async hooks precompute an activation_decisions map on the event loop (slash target + persisted entry names, one batch of awaits) and hand it to the worker-thread handler, which consults the map instead of the sync API: SkillActivationMiddleware.awrap_model_call (activation + secret binding), ToolErrorHandlingMiddleware.awrap_tool_call (_amaybe_stamp awaits the decision before stamping), and SkillToolPolicyMiddleware's awrap_model_call/awrap_tool_call (entry names + slash source name). describe_skill becomes a StructuredTool with both a sync function (invoke -> authorize) and a coroutine (ainvoke -> aauthorize), so async agent execution never routes a loop-affine provider through the sync API while sync callers keep their semantics. Tests: allow-read-then-deny-next-run regressions for secret binding and allowed-tools (partial deny does not poison, all-denied fails closed to builtins), and an _AsyncOnlyProvider (sync authorize always fails, aauthorize works) driving four async-path assertions (activation, stamp allow/deny, describe, tool policy). All six fixes are mutation-verified; full A/B suite run against HEAD shows no new failures. * fix(authz): authorize skill activation by declared Skill.name, not directory basename ShenAC-SAC's P2: bundled skills may declare a name that differs from their directory (skills/public/vercel-deploy-claimable declares name: vercel-deploy), but the runtime gates derived authorization targets from the container path. - New deerflow/skills/container_registry.py: build_container_path_registry() + canonical_skill_name() — the one path-to-Skill.name resolution shared by the activation, tool-policy, and read-stamping middlewares. - ToolErrorHandlingMiddleware resolves the read path through the (user-scoped, via new user_id kwarg threaded from both runtime builders) registry before the skill:activate decision, sync and async (registry lookup in a worker thread, aauthorize() on the loop). RBAC allow by the declared name now activates; allow by the directory name does not. - SkillToolPolicyMiddleware async hooks canonicalize the policy paths first (three-phase: thread registry -> loop aauthorize -> thread filter reusing the registry; the cached per-model-step decision still short-circuits), so the decision map is keyed by the names _active_skills_for_paths checks — no more map miss falling back to the sync provider API from the thread. _entry_names removed: persisted entry["name"] is path-derived by construction; entries are resolved by path instead. - SkillActivationMiddleware collects slash names (already canonical) plus persisted entry *paths*, canonicalized off-loop via _canonical_names_for_paths(). Regressions (test_skills_authorization.py 45->50): sync stamp target, RBAC allow declared-name/deny directory-name both directions, async stamp via _AsyncOnlyProvider, composed async slash -> tool-policy keeps the skill's tools with zero sync fallback, persisted-entry secret binding survives the name mismatch. All three mutations (stamp target, policy map key, entry canonicalization) fail their tests. * fix(authz): close the sync-fallback class on skill activation paths willem-bd's two P2s on the canonical-name fix, plus a fourth instance of the same class found while sweeping the membership: - SkillToolPolicyMiddleware: a transient prepass registry failure is now preserved (_REGISTRY_LOAD_FAILED marker) — the worker-side filter applies its own fail-closed treatment instead of silently retrying storage, which recovered skills absent from the decision map and fell back to the synchronous provider API from the thread (fail-open retaining a denied skill's allowed-tools). - SkillActivationMiddleware: an empty entry-path list no longer pays a full skill-tree scan on ordinary authorization-enabled async model steps (previously a passive path with zero registry I/O). - 4th instance (reproduced with the PR fixtures, then fixed): the entry prepass failure on the secret-binding path let the consumer-side _load_skill_registry_by_path silently recover and resolve entry names outside the decision map — the same sync fallback, fail-open binding the aauthorize-denied skill's declared secret. The fix goes one step further than failure propagation: the prepass hands down the registry *snapshot* the decision map was keyed by, and entry sources resolve against that snapshot, so a map miss is impossible by construction (this also closes the narrower window where both loads succeed but disagree because storage changed mid-step). The slash source is deliberately unaffected: it does not consult the decision map (validated once at activation) and keeps its fresh per-call load. - Class membership swept: skill_activation_allowed() sync call sites are exactly four (stamping sync path and describe's sync implementation are legitimately sync); the only two async-reachable fallback points (the _activation_allowed helpers) are both construction-guarded now. Structural hardening (typed async-batch decision object where a map miss is fail-closed by construction) recorded in the plan notes as a follow-up, anchored at those two helpers. - _RegistryArg type alias replaces the degenerate dict|object|None union in the policy middleware signatures. Tests (test_skills_authorization.py 50->54): transient prepass failure -> builtins-only with exactly one storage attempt and zero provider calls; ordinary async step -> zero registry scans; two-call scenario -> entry binding zeroed while the slash binding survives; entries resolve against the prepass snapshot under a mid-step rename. Mutations verified red: marker reverted to None, over-broad slash kill, snapshot bypass. * fix(authz): correct secret-binding reference points and gate the rendered skill reminder Review round 9 (willem-bd) plus construction-audit follow-ups on the same class — every async divergence between two loads of the same data now has an explicit, pinned reference point: - Slash secret source resolves from a FRESH post-activation registry: the prepass snapshot is taken before the aauthorize awaits, so binding the slash source from it would activate NEW_KEY content while injecting OLD_KEY when a skill's declared secrets change in that window. The 'one scan per step' bonus that motivated sharing the snapshot is withdrawn — consistency wins. The two secret sources resolve independently: a transient fresh-load failure zeroes only the slash binding; entries keep binding from their own snapshot. - Same-skill dominance: when a skill is both the slash source (fresh) and a persisted entry (snapshot), only the activation-era declaration binds — unioning both eras injected a secret the skill no longer declares. Aligns with the tool-policy rule (slash activation dominates for the run); different-skill entries still stack additively. - The rendered 'Active skills' reminder (DurableContextMiddleware) no longer advertises skills the provider denies — surface consistency with describe_skill. Publication/consumption split: the activation middleware publishes per-step path-keyed skill:activate decisions into the run context (async: reused from the prepass snapshot, zero extra I/O; sync: one scan + authorize() only when entries exist — passive steps stay scan-free), the renderer consumes them and filters its render copy (state untouched; unpublished = permissive, matching the other run-context carriers). New carrier key listed in REDACTED_CONTEXT_KEYS with a redaction regression. - Wiring regressions on both chains for everything added above (user_id to the stamp gate, skill_authorization + token pairing + activation- before-durable ordering to the durable renderer): each mutation verified red (M4-M13 across this change set). - Tests 54->65: slash post-activation lookup, source independence, dominance, sync-chain sync-API path, render filter (async/sync/ disabled with load-count assertions), chain wiring x2, redaction listing. Plan notes record rounds 9-2..6 including the reference-point post-mortem and the audit axes exhausted. * fix(authz): anchor same-skill dominance on the slash identity, not binding results Review round 10 (willem-bd) plus a self-audit extension the new checklist items caught in the first fix itself: - R10 (P2): the same-skill exclusion was derived from successfully BOUND slash sources. When the prepass snapshots required-secrets [OLD_KEY] and the operator strips the declaration during the await window, the slash source binds nothing (required_secrets gate) — the exclusion set stayed empty and the stale entry view kept injecting OLD_KEY for a skill that now declares no secrets. The exclusion now derives from the AUTHENTICATED slash activation identity (run-context source path, owner-token checked): the identity's declared name resolves from the fresh registry first, then the entry snapshot, and joins the exclusion set regardless of binding success. - Self-audit (value->gate->downstream on the fix itself): the declared name can also change mid-run (same path, foo->bar). A name-anchored exclusion then holds only the new name while the snapshot entry resolves the same path to the old one — both eras bound again. The PATH is the stable identity shared by the activation and every earlier read: entries at the slash identity's path are now excluded before registry resolution (exclude_paths), independent of what either version calls the skill; the name-based set remains for cross-path same-name shadowing. Tests 65->67: no-secrets-era dominance (ACTIVE secrets None under the OLD_KEY-to-empty interleaving), path-anchored dominance across a mid-run rename (only the activation-era key binds). Mutations verified red: identity-derived exclusion disabled (M14), path exclusion disabled (M16). * test(authz): pass explicit config to lead middleware wiring test --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
295437692a
|
fix(agent,goal): stop coaching scripts when no bash tool is bound, and show the goal evaluator Human Input Card answers (#6082)
* fix(agent): drop script guidance from the lead prompt when no bash tool is bound
The workspace section told the agent how to write scripts and commands
("prefer relative paths", "avoid hardcoding /mnt/user-data/... inside
generated scripts") even when no tool could run them. With the default
LocalSandboxProvider host bash is off and the bash tool is removed, so
agents wrote helper scripts nothing could execute, then stalled or
computed by hand.
apply_prompt_template() takes bash_available (default True, so other
callers keep today's text). The lead agent (both assembly branches) and
the embedded client pass whether their authorized tools, bound or
deferred, include bash. Without it:
- the two script bullets become one line: no bash tool is bound, work
out results directly and write them with write_file;
- the subagent section no longer shows direct bash examples, and marks
the bash subagent unavailable, since it inherits the lead's tool groups;
- the ACP hint no longer suggests `bash cp`.
* fix(goal): show the goal evaluator the user's answer to a Human Input Card
The web UI sends a Human Input Card answer as a hidden human message
(hide_from_ui with a human_input_response) and shows it on the card.
format_visible_conversation() skipped every hidden message, so the
evaluator saw the question but never the answer. Following the user's
answer then read as the assistant guessing, and the goal stood down with
needs_user_input although the user had answered.
A hidden human message whose human_input_response passes
read_human_input_response() now adds a line
"User (Human Input Card answer): <value>" from the structured value; the
question is already on the card's own line. Other hidden messages, such
as goal continuations, stay out. Over the 12,000-char cap the latest
answer the cut drops is kept at the top next to the latest request, and
is never taken for the request itself. Without card answers the cut is
unchanged.
* fix(agent): keep the subagent examples and bash hint in line with the bound tools
Review follow-up for the lead prompt without bash.
- The last delegation example follows the tools. With a bash subagent on
offer it is unchanged. A lead with bash and no bash subagent keeps
"Run a routine test, build, or git command directly." without the Bash
subagent sentence. A lead without bash gets a file-work example instead.
- The unavailable-bash description no longer points at the sandbox
provider. Since #2253 it renders only when the bash subagent is
registered and the lead binds no bash, where the sandbox is not the cause.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
|
||
|
Hyeonsang Cho
|
af67738521
|
fix(agents): treat a null max_total_subagents override as unset (#6088)
* fix(agents): treat a null max_total_subagents override as unset An API caller sending "max_total_subagents": null in the run context failed that run with a TypeError: the key was present, so dict.get(key, default) returned None and building SubagentLimitMiddleware compared None with an int. Resolve the per-run delegation cap through one helper, effective_total_subagents_per_run, which treats None as "use subagents.max_total_per_run" and clamps to 1-50. The lead agent's middleware and assembly paths, DeerFlowClient, and the system prompt all use it, so the extension host policy and release policy now report the enforced cap instead of an out-of-range request. * docs(changelog): reference #6088 in the null max_total_subagents entry |
||
|
xbzz1018
|
e2c5027b0c
|
fix(tracing): tag graph runs with the effective model (#5902) | ||
|
xbzz1018
|
29dbce45a1
|
fix(client): exclude prior turns from resumed stream events (#5811)
* fix(client): exclude prior turns from embedded stream events * docs(client): tighten streaming guidance --------- Co-authored-by: xbzz1018 <186267668+xbzz1018@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
bcaf6db36a
|
feat(models): add declarative reasoning capability contract (#5780)
* feat(models): add declarative reasoning capability contract (#5073) Introduce an optional per-model `reasoning:` block beside the legacy `supports_thinking` / `supports_reasoning_effort` booleans. It declares thinking availability (unsupported | optional | required), the accepted effort values with aliases, default, and serialization path, the payload dialect, and the reasoning-history requirement. When present the booleans are derived from it and contradictory profiles fail at config load. `deerflow.models.reasoning` normalizes legacy and declared profiles into one immutable contract and resolves every caller's generic request against it. `create_chat_model` is the single enforcement point: required-thinking models never receive a synthesized disable payload, unsupported effort values never reach the provider, and the dialect synthesizes the on/off payload when no template exists. Legacy profiles keep the historical path byte-for-byte. The lead agent and subagent descriptor resolve the same policy first so run metadata reports effective values. `/api/models` and `DeerFlowClient` project the contract as `reasoning`; the frontend derives mode clamping and effort choices from it through `core/models/reasoning.ts`, and user preferences accept provider-specific effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to the contract, restoring its low/high/max effort control. * docs(models): move the reasoning contract guidance into models/AGENTS.md backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard limit for the middlewares, runtime, sandbox, and subagents guides, so the new section pushed scripts/check_agent_guidance.py over it. The models guide is the nearest file for the factory and the contract module, and its chain has ample room. * fix(models): address review on the reasoning contract - vLLM dialect: the enable path now mirrors whichever chat_template_kwargs switch the template declares, like the disable path already did, so a template using the older `thinking` key no longer also grows `enable_thinking`. - Validate operator-supplied effort values at `effort.path` inside `when_thinking_enabled`, `when_thinking_disabled`, and the `thinking` shortcut against the contract, not only the profile-level value; they are forwarded when the caller chooses nothing, so the "unknown values never reach the provider" guarantee now holds for them too. - Constrain `effort.path` to dotted identifiers and reject single-segment paths that would replace a whole mapping (`extra_body`, `thinking`, `model_kwargs`, `default_headers`, `default_query`). - Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the default also governs summarization, title generation, and subagent calls that never choose an effort; `max` stays selectable. Documented in the config example, the configuration manual, and the design note. * fix(models): close reasoning contract review gaps * fix(models): preserve native Ollama reasoning config * fix(models): keep native reasoning strings and project reasoning into the descriptor - `ModelConfig.reasoning` also accepts the level string ChatOllama takes for gpt-oss style models (`low` / `medium` / `high`); the factory forwards both the boolean and the string on the legacy path, so profiles that passed `reasoning: high` through before the contract field existed keep loading and keep sending it. - The assembly descriptor no longer excludes `reasoning` from `model_parameters`: the native ChatOllama value and a declared contract's dialect/history/effort change the request payload, so they now move the assembly fingerprint. - e2e: the sidecar spec pinned the pre-contract body where a model without effort control still received `reasoning_effort: minimal`; the preset is now dropped for such models (the Gateway discarded it anyway), so the assertion expects the key to be absent. |
||
|
RongJie G
|
e01314442c
|
fix(mcp): scope sessions and task access by thread incarnation (#5556)
* fix(mcp): scope sessions and task access by thread incarnation * fix(mcp): preserve thread incarnation in delegated subagents * fix(mcp): preserve incarnation in durable batches * fix(mcp): bind standalone graph lifecycle context * fix(studio): preserve implicit thread creation metadata --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
90f0866fa8
|
fix(uploads): stop deleting a converted companion we cannot prove we wrote (#5673)
* fix(uploads): stop deleting a converted companion we cannot prove we wrote Conversion names a document's Markdown companion after the document's stem and falls back to a _N suffix when that name is already taken, so the .md beside a document may belong to another document sharing the stem, or to the user. Delete removed it anyway: uploading a.docx and a.pdf produces a.md and a_1.md, and deleting a.pdf destroyed a.docx's companion while orphaning a.pdf's own. Delete now removes only the file it was asked to remove. The companion stays listed and can be deleted on its own. Orphans are the cost of not guessing; issue #5672 covers giving companions a provable owner, which is what a safe cleanup needs, along with the two related readers that still guess (the outline injected for a document and the agent's file listing). convertible_extensions loses its last use and is dropped from the signature and both call sites. The gateway router keeps importing CONVERTIBLE_EXTENSIONS for the ingestion bridge and now declares it in __all__, where that module documents its re-exports. * docs(changelog): note that delete keeps the converted markdown (#5673) |
||
|
Weng Qiang
|
60d5659d1d
|
fix(client): honor agent MCP plugin selections (#5630)
* fix(client): honor named-agent MCP plugin selections * fix(client): normalize MCP selection cache identity |
||
|
Hyeonsang Cho
|
29d285731b
|
fix(uploads): convert the bytes we wrote, not the name they landed under (#5611)
* fix(uploads): convert the bytes we wrote, not the name they landed under Document conversion re-opened the upload by name after it was already visible in the thread's uploads directory: the Gateway converted the committed file_path, and DeerFlowClient converted the copy it had just placed there. That directory is writable from local and AIO sandboxes, so a process watching it can replace the name with a symlink in the window between the upload landing and the converter opening it. The converter then reads whatever host file the link points at and writes that content back into the thread as the .md companion, which the sandbox can read. Reproduced end to end on both paths with a real xlsx: the companion came back holding the host file's rows. The Gateway now duplicates the descriptor of the staged file before the link-commit, copies those bytes into a private directory outside the uploads tree, and converts there. A descriptor cannot be redirected by replacing a name, so the conversion input is the content this request wrote. The client converts the caller's own source file instead of the copy in uploads; the source is the file the caller handed in, which the sandbox cannot reach. Both already wrote the companion without following a symlink, so only the read side changes. The uploads directory still receives exactly the same files. * docs(changelog): note upload conversion source fix (#5611) * fix(uploads): close the conversion descriptor when staging its copy fails Review follow-up. The private directory for the conversion copy was created before the try that owns the duplicated descriptor, so a failure there — a full or unwritable temporary filesystem — propagated without closing it. The upload's own cleanup only unlinks the committed name and releases the sandbox lease, so the descriptor stayed open for the life of the process and kept the unlinked staged bytes allocated with it; repeated failures accumulated both. Directory creation now happens inside that try, and the finally removes the directory only once it exists. * fix(uploads): keep the conversion descriptor owned across cancellation Review follow-up. run_file_io cannot interrupt its worker, so cancelling the await around os.dup only abandoned the result: the duplicate was created moments later with nothing left to close it, and it pinned the staged bytes of an upload whose name the cleanup had already unlinked. Cancellation after the duplication was just as leaky, because the commit-path handler caught Exception and CancelledError is not one. The duplication now runs as its own task, shielded from the caller's cancellation, and closes its own result when the caller is gone by the time the worker finishes. The commit path catches BaseException, closing the descriptor it already owns before re-raising. Both windows are pinned: one test stalls the duplication worker after it allocates and cancels ingestion, the other stalls the commit so the cancellation lands while the descriptor is owned. * fix(uploads): drain the conversion copy so its descriptor always closes Review follow-up. The copy worker owns the duplicated descriptor and closes it in its own finally, but a bare await let a cancellation cancel the executor job while it was still queued: the worker never ran, so that finally never ran either, and the enclosing scope had already handed ownership away and saw None. Draining also keeps a late worker from writing into a private directory this scope has since removed. The copy now goes through await_drained, the shield-and-drain helper the Gateway already uses for offloads that must not be abandoned mid-flight. Pinned by a test that holds the copy job queued, cancels ingestion, then releases it and requires the descriptor to come back closed. |
||
|
Hyeonsang Cho
|
2b6254f76d
|
fix(client): stop embedded uploads from writing through symlinks (#5578)
* fix(client): stop embedded uploads from writing through symlinks DeerFlowClient.upload_files copied each file with shutil.copy2 and let convert_file_to_markdown write the companion straight into the uploads directory. Local and AIO sandboxes can write to that directory, so a symlink planted at an upload name or at the companion's name was followed: the upload's bytes and the converted Markdown landed in whatever host file the link pointed to, and the call reported success. The Gateway refuses symlinked destinations and the IM channels write through write_upload_file_no_symlink; the embedded client never adopted either. Uploads now go through copy_upload_file_no_symlink, a new helper next to write_upload_file_no_symlink. It keeps copy2's content, permission bits and timestamps, so files stay readable to Docker sandboxes, but applies them to the descriptor opened with O_NOFOLLOW and opens the source first so a missing source cannot truncate an existing upload. As in the Gateway, a file with an unsafe destination is skipped and listed in skipped_files, success turns false, and the message says how many were skipped. The companion is converted inside a private temporary directory and then written with write_upload_file_no_symlink; one whose name is unsafe is left out like a failed conversion, and the original upload is kept. * docs(changelog): note embedded upload symlink fix (#5578) * fix(client): keep copy2's same-file guard and companion permissions Review follow-up. Two regressions in the previous commit. copy_upload_file_no_symlink opened the destination before comparing it with the source, and that open truncates. Passing a file that already sits in the thread's uploads directory therefore copied an emptied file over itself: the upload reported success with size 0 and the original bytes were gone, where copy2 raised SameFileError and left the file alone. The destination is now compared with the source through os.path.samestat before anything is opened, so identity — including a hardlink or another spelling of the same path — raises SameFileError as before. The Markdown companion was published with write_upload_file_no_symlink, which creates a new file as 0600 and ignores the converted file's mode. Under umask 022 the companion became 0600 while its own document stayed 0644, so a bind-mounted sandbox running as another uid could read the upload but not the Markdown the response advertises. It now goes through the same copy helper as the upload, which preserves the converter's permission bits. |
||
|
alanhuangyoo
|
d87a1c1193
|
fix(client): emit text a later node appends to an AI message already sent (#5479)
* fix(client): emit text a later node appends to an AI message already sent Loop detection, the token budget, safety termination, subagent limits and the terminal-response fallback rewrite the last AI message under the same id in their own after_model node. stream() skipped every id it had seen, so chat(), the TUI and --print kept the text from before the rewrite: no [FORCED STOP] notice, no fallback error, and token_usage_attribution added after the model node never arrived. Look again at a known id when a snapshot holds a different message object, emit only the text appended to what was already sent, and send new additional_kwargs as the existing metadata-only follow-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(client): pin that a non-extending replacement is not re-sent Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Beautyl0ve
|
80f13935c2
|
feat(agents): allow custom agents to disable memory (#5167)
* feat(agents): allow custom agents to disable memory Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> * fix(agents): honor memory opt-out during compaction Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> * fix(runtime): preserve agent binding across state rewrites * fix(client): apply named-agent memory policy * fix(agents): address memory policy review feedback Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> * fix(agents): address remaining memory opt-out reviews Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> --------- Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
alanhuangyoo
|
529b0e05ec
|
fix(client): emit streamed tool calls once with complete args (#5408)
* fix(client): emit streamed tool calls once with complete args
When a model streams a tool call as chunks (name and id first, then
argument fragments), DeerFlowClient.stream() emitted a tool_calls event
per chunk, each parsed from that fragment alone. Consumers got
`args={}` and then a call with no name or id, and never the complete
call, because the values snapshot skipped the already-streamed id.
Hold tool calls from AIMessageChunk.tool_call_chunks and emit them once
from the values snapshot, where the message is complete.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(client): pin additional_kwargs on the deferred tool_calls event
Also note why a tool-call chunk without a message id keeps the per-chunk
event.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Wenchao An
|
1b76ab9060
|
feat: add opt-in task notes and compacted history recall (#5382)
* feat: add opt-in task notes and compacted history recall * fix: validate task continuity state and preserve user answers Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence. * fix: align Helm config version with task continuity schema * fix: preserve mixed task history and declare continuity policies * fix: recover malformed history and evict archives atomically |
||
|
Hyeonsang Cho
|
bc4a33aba7
|
fix(skills): stop persisting resolved secrets when toggling skills (#5357)
Toggling a skill wrote resolved secrets into extensions_config.json. The Gateway skill toggle and DeerFlowClient.update_skill loaded the file with ExtensionsConfig.from_file(), which replaces every "$VAR" string with the environment value (and an unset variable with ""), then serialized that model back through to_file_dict(). A "$GITHUB_TOKEN" reference was persisted as the plaintext token, and an unset reference was erased for good. DeerFlowClient.update_mcp_config had the same flaw for every key other than mcpServers. Every writer now does a raw read-modify-write, the way the MCP router already did: read_raw_extensions_config reads the on-disk JSON, set_raw_skill_enabled changes only the target entry, and validate_raw_extensions_config checks the candidate the way the runtime will load it before the atomic write. When the file does not exist yet, the Gateway seeds it with the cached skill states only, never the resolved cached model. The MCP router's raw loader and candidate validation delegate to the same helpers, so the rule lives in one place, and to_file_dict() is removed so the unsafe serialization has no entry point left. Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Sami Belhareth
|
a3848ef155
|
fix: expose summary_text in embedded client values events (#5249)
* fix: expose context summary in embedded values events * test: cover summary values in mode-tagged streams --------- Co-authored-by: Sami Belhareth <6599699+belharethsami@users.noreply.github.com> |
||
|
Beautyl0ve
|
3bccd1474f
|
fix(client): scope embedded agent reuse by effective user (#5206)
Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> |
||
|
Aari
|
9e0fbd60fa
|
fix(sandbox): isolate concurrent subagent shell sessions (#5134)
* fix(sandbox): isolate concurrent subagent shell sessions * fix(sandbox): make execution acquire idempotent * fix(sandbox): close execution lifecycle gaps * fix(sandbox): serialize retained client lifecycle * fix(sandbox): close remaining client lifecycle gaps * fix(sandbox): unwind failed client lookup * fix(sandbox): protect internal lease identities * fix(sandbox): make cancellation reconciliation durable * fix(sandbox): fence cancelled workers and IM uploads |
||
|
Hyeonsang Cho
|
9146bfa03d
|
feature(gateway): issue request trace ids unconditionally (#5119)
* refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Aari
|
ff0a6768c2
|
feat(subagents): add unified capacity and durable batch execution (#4998)
* feat(subagents): add capacity controls and durable batches * fix(helm): sync subagent config schema version * fix(subagents): preserve batch history without worker * fix(subagents): support explicit factory runtimes * fix: address durable batch review findings |
||
|
Aari
|
236a068e77
|
fix(docker): let the Gateway write extensions_config.json in production (#4852)
* fix(docker): let the Gateway write extensions_config.json in production AGENTS.md states extensions_config.json may be edited at runtime through the Gateway API, and the Gateway implements that for the MCP enable switch, PUT/PATCH /api/mcp/config and the skill update route. Two properties of the production compose stack made every one of those writes fail: - the file was mounted read-only, and - Docker mounts it as its own mount point, so the temp-file-plus-rename in atomic_write_extensions_config hit EBUSY. Linux refuses rename() over a mount point whether or not the mount is writable, so making the mount read-write alone is not enough. Mount it read-write and fall back to an in-place overwrite on EBUSY only. The fallback is deliberately non-atomic and says so in a warning; it is reached only where the atomic route cannot work, and any other errno still propagates. config.yaml stays read-only: no API writes it. docker-compose-dev.yaml mounts the whole project directory, so the destination is an ordinary file there and this never surfaced in development. * test(docker): parse mount options instead of matching a :ro suffix Docker's short-syntax options segment is comma-separated, so a read-only mount can legally be spelled ":ro,z" or ":z,ro" — common with SELinux relabelling. Matching the raw string for a ":ro" suffix reads those as writable, which silently defeats the guard: the writability assertion would pass on a read-only mount, and the config.yaml assertion would fail on a correctly read-only one. Parse the options segment and test membership instead, and cover the parser with the spellings that broke the suffix check. * fix(config): harden mutable extensions config |
||
|
Aari
|
5ffaa09f5a
|
feat(memory): add hybrid fact eviction policy (#4789)
* feat(memory): add hybrid fact eviction policy * refactor(memory): simplify confirmation count update * fix(memory): clean up eviction audit metadata * fix(memory): harden eviction cleanup boundaries * fix(memory): address hybrid eviction review |
||
|
hataa
|
540940bac1
|
feat(authz): enforce model authorization at Gateway routes and runtime (#4063 Phase 3) (#4540)
* feat(authz): enforce model authorization at Gateway routes and runtime (#4063 Phase 3) Phase 3 / Models — the first of three resource-type PRs (Models, Skills, Sandbox). The RBAC provider already maps "model" → config key "models" (rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed. Gateway route layer (mirrors Phase 2A): - resolve_model_authorization() in authz.py returns (provider, principal), reusing _get_cached_route_provider and build_principal_from_context, including the INTERNAL_SYSTEM_ROLE → None pop for internal callers. - list_models filters via provider.filter_resources(principal, "model", names). - get_model checks provider.authorize("model", "use"). Deny → 403 (not 404, since the model exists but the role lacks permission). Runtime resolution layer (mirrors Phase 1B): - _authorize_model_name() in agent.py runs after _resolve_model_name. On deny, falls back to the first allowed model (RFC §9: graceful, not crash). All models denied + fail_closed → ValueError (matches existing contract). authorization.enabled: false is a complete no-op on both layers. Anonymous requests (user=None) bypass filtering. 18 new tests + 314 existing tests pass. * fix(authz): enforce model:use on the embedded DeerFlowClient path (Phase 3 follow-up) Round 4 review (willem-bd): _authorize_model_name only covered the Gateway runtime path (_make_lead_agent). The parallel lead-agent construction path DeerFlowClient._ensure_agent (client.py) filtered tools but not the model, so a library/embedded consumer with role-scoped model policies could run a model the role is denied model:use for. - Insert _authorize_model_name in _ensure_agent, mirroring _make_lead_agent. - Resolve None default to the first configured model before the gate so the implicit default (create_chat_model(name=None)) is also authorized. - Update test_authorization_filters_framework_tools_and_reuses_provider: the stub provider now returns an allow decision for model:use (checked during assembly) and patches resolve_authorization_provider in the agent namespace. - Add 3 DeerFlowClient._ensure_agent path tests (real-path fallback, None-default resolution, disabled no-op); 24 tests total. * docs(authz): document get_model provider-unavailable fail-open path + test zhfeng review (round 5): get_model's docstring only mentioned the deny→403 path, not the provider-resolution-error + fail-open path (which allows the request, mirroring list_models's documented fail-open semantics). The behavior itself is correct and symmetric with list_models, but it was undocumented and the _AuthorizationUnavailable path had no test coverage. - Extend get_model docstring to state the provider-error fail-closed/fail-open outcome, matching list_models's wording. - Add test_get_model_provider_unavailable_fail_closed_vs_open exercising the _AuthorizationUnavailable path (provider cannot be resolved at all), pinning fail-closed→403 / fail-open→200. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Vanzeren
|
095092418c
|
fix(gateway):unify thread id validation (#4589)
* fix(gateway): unify thread ID validation at the API boundary
Thread ID entry points accepted arbitrary strings while downstream
consumers (filesystem paths, Kubernetes Provisioner, JSONL event store)
each enforced different character restrictions, so invalid IDs were
persisted first and only failed later during sandbox/workspace init.
Centralize validation in deerflow.utils.thread_id (pattern
^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest,
scheduler dispatch, paths.py, JSONL store, embedded client, and align
the Provisioner pattern (pinned by a parity test). UUIDs are still
generated only when no ID is supplied; caller-supplied opaque IDs stay
supported.
Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy
cleanup escape hatch (filesystem cleanup guarded), read-only
client.get_thread stays unvalidated, and scheduler rows with legacy
invalid IDs record a failed dispatch instead of raising out of the
poll loop.
* docs: document canonical thread ID contract
README: caller-supplied thread IDs need not be UUIDs; the canonical
pattern and per-endpoint behavior. AGENTS.md: the shared
deerflow.utils.thread_id contract, its enforcement boundaries, and the
legacy-ID escape hatches.
* fix(gateway): close thread ID validation gaps at remaining entry points
Follow-up to the canonical thread ID contract: a full audit found the
uniform-422 coverage only reached about half of the thread_id surfaces.
- routers: 18 routes still took a bare thread_id: str — 13 in
thread_runs.py (including the five messages/events/workspace-changes
reads that returned 500 on the JSONL event store vs 404/empty on the
DB store), 4 read routes in threads.py, and the suggestions route
flagged in review. DELETE /api/threads/{id} keeps str as the declared
legacy-cleanup escape hatch.
- client: upload_files/delete_upload/list_uploads/get_artifact now
validate up front, fulfilling the RFC's 'all mutating entry points'
clause (get_thread stays unvalidated as the declared legacy read path).
- tui: the /resume literal-ref fallback validates against the canonical
contract and reports a descriptive error instead of failing deep in
the client.
- scripts/support_bundle.py: replace the drifted dot-allowing pattern
with a byte-identical copy of THREAD_ID_PATTERN (kept local so the
script still runs with a broken venv).
* test(gateway): guard the canonical thread ID contract against regressions
- test_thread_id_route_contract.py: static AST sweep asserting every
route handler with a thread_id parameter annotates ThreadId
(whitelist: the DELETE escape hatch), plus a runtime sweep hitting
all 44 thread_id routes with a non-canonical ID and asserting a 422
that names thread_id, plus a websocket upgrade-rejection case.
- test_thread_id_validation.py: client entry-point validation,
support_bundle pattern parity, and TUI literal-ref fallback tests.
- Align two tests that encoded the old contract (dotted IDs).
|
||
|
Xinmin Zeng
|
f2e832330e
|
fix(sandbox): enforce disabled skills in filesystem views (#4178)
* fix(sandbox): project enabled skills into sandbox views * fix(skills): keep projection mutations consistent * fix(skills): fail closed on projection errors * fix(skills): isolate per-scope failures during boot projection rebuild rebuild_all_skill_projections() propagated any exception from the public rebuild or from a single user's rebuild straight out of the gateway lifespan startup, uncaught. A single broken user directory (bad permissions, corrupted _skill_states.json, unreadable content) would therefore abort gateway boot for every user, not just that one - _rebuild_*_locked already fails closed internally (clears the view and re-raises), so the boot loop only needed to stop treating that re-raise as fatal. Each scope's rebuild now fails closed independently and boot continues; a scope left empty by a boot failure self-heals on the next sandbox acquire via ensure_skill_projections(). Also patches deerflow.skills.projection.rebuild_all_skill_projections in the memory-flush lifespan test fixture, matching the two sibling fixtures in the same file — this call is now on the lifespan startup path and the fixture's minimal SimpleNamespace config predates it. * test(skills): update authz test for the projection-aware public toggle _persist_shared_skill_state (introduced earlier in this branch) reads the shared extensions_config.json fresh from disk under the projection lock instead of through the cached get_extensions_config() singleton - that's the whole point of the fix (stale worker caches must not clobber another worker's concurrent update). The name no longer exists on the skills router module, so the test's monkeypatch of it started raising AttributeError instead of exercising the endpoint. The mock storage in this test isn't a real LocalSkillStorage instance, so _persist_shared_skill_state's projection-mutation branch is already skipped (nullcontext) and it falls back to a fresh ExtensionsConfig() for the nonexistent tmp config_path - no replacement monkeypatch needed. * fix(sandbox): make skill projection ensure best-effort in acquire acquire() called _ensure_skills_projection() directly, outside any try/except, in both LocalSandboxProvider and AioSandboxProvider. Every other skill-mount setup path in these providers has always caught exceptions and logged a warning rather than failing sandbox acquire outright (e.g. when config.yaml can't be resolved) - these two new call sites broke that contract, so any projection failure (including simply not having a config.yaml, as in CI's test environment) now failed acquire() itself instead of just leaving skill mounts off. _ensure_skills_projection now catches its own exceptions and returns None; both providers' callers already tolerate that (a None projection skips the skill-specific mounts, matching the existing degrade path) after making _append_public_skill_mapping and the custom/legacy mount block in LocalSandboxProvider explicitly None-safe. Caught by running the full suite with config.yaml removed, matching CI's environment - not caught locally because a real config.yaml was present, masking the failure. * fix(sandbox): make E2B skill projection mounts best-effort _skill_projection_mounts called ensure_skill_projections with no guard, unlike Local/AIO's _ensure_skills_projection. A raise propagated out of _apply_mounts before the configured-mounts loop ran, so a skills projection failure dropped the operator's own configured mounts too - only caught by create()'s outer warning, with nothing applied at all. Swallow here and return an empty mount list on failure, matching the Local/AIO pattern: still fail-closed for skills, but no longer widens the blast radius to unrelated configured mounts. Review feedback from PR #4178. * docs(skills): document projection trade-offs flagged in review - _update_tree_digest: note the metadata-only (not content) hashing trade-off and why runtime writes through this codebase are still covered regardless (rebuild-under-lock + rename always changes inode). - LocalSandboxProvider.acquire: note the acquire-time self-heal cost (cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift). - skill_projection_mutation: drop the no-op except-Exception-then-raise; a raise from the mutation already propagates past the yield with the view left cleared, no explicit re-raise needed. - provisioner README: spell out that hostPath skills volumes require the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node or shared storage), and that the custom/legacy volumes' hostPath type Directory (not DirectoryOrCreate) makes a violation of that assumption a visible Pod-creation failure instead of a silent empty mount. Review feedback from PR #4178. * fix(skills): lazily repair user projections * fix(skills): close projection review gaps * fix(skills): refresh user projection enable state * fix(skills): close projection review follow-ups * fix(skills): preserve state across projection writes --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Huixin615
|
133a82c6c2
|
fix: isolate MCP server toggles from invalid peer configs (#4577)
* fix: isolate MCP server toggle updates * fix: write extensions config atomically * fix: normalize MCP transport aliases |
||
|
Vanzeren
|
c48de5e70b
|
feat(checkpoint): make delta snapshot_frequency configurable (#4516)
* feat(checkpoint): make delta snapshot_frequency configurable * fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning Addresses review on #4516: the rename from the flat database.checkpoint_delta_snapshot_frequency key to nested database.checkpoint_delta.snapshot_frequency silently dropped the old value (pydantic extra="ignore"). Add a before-validator that maps the legacy key onto the nested one with a deprecation warning (nested key wins when both are set), plus a CHANGELOG breaking-change note covering the rename and the 1000 -> 10 default change. * fix(checkpoint): validate frozen snapshot frequency |
||
|
Vanzeren
|
e01173d8b2
|
bench(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency (#4467)
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency - Group benchmark scripts into per-family folders (checkpoint/, sandbox/) - Extract shared benchmark infrastructure into checkpoint_bench_common.py - Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen); freeze it in make_lead_agent and DeerFlowClient; key the state-schema adaptation cache by resolved frequency - New bench_production.py: per-case child processes run N ainvoke turns through the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver), then measure GET /state + POST /history through the real Gateway route stack in one event loop (httpx ASGITransport), cold/warm accessor-cache split, cross-mode digest gates - New summarize_production.py: delta/full ratios plus decision metrics (snapshot_write_spike, cache_effect_ms, checkpoint_write_share, auto-discovered history per-limit ratios) * fix(checkpoint): address production benchmark review |
||
|
ShitK
|
0f0955bf7b
|
fix(client): preserve ToolMessage artifacts (#4422)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
hataa
|
7857fa0cce
|
feat(authz): enforce tool authorization at assembly and runtime (#4370)
* feat(authz): enforce tool authorization at assembly and runtime * fix(middleware): guard deferred tool setup lookup (#4370) --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
lllyfff
|
01a89f2379
|
[feat] memory: pluggable MemoryManager interface for backend onboarding (#4326)
* refactor(memory): pluggable MemoryManager interface for backend onboarding
Optimize the MemoryManager interface layer so new backends (mem0/openviking)
onboard with less code and the contract stays stable as capabilities are
added. A minimal backend now implements only from_config + add + get_context
(verified by test_memory_manager_interface.py::_MinimalBackend onboarding via
the factory); the factory no longer knows a backend's private hooks.
- MemoryManager: ABC -> pydantic BaseModel; three-tier methods (tier-1
add/get_context abstract; tier-2 management defaults; tier-3 optional hooks
warm/reload/fact + on_pre_compress/on_turn_start). Dropped 3 self-serving
hooks. 6 hasattr probe sites -> direct call + try/except NotImplementedError.
- from_config classmethod: factory thins to resolve + inject storage_path +
collect host hooks + call from_config; DeerMem-specific hook consumption
moved from factory to DeerMem.from_config.
- Invariants: @model_validator (mode='tool' requires search via supports_search
ClassVar); DeerMemConfig storage_path-is-file check moved here from factory.
- Async: aadd/aget_context/asearch default to the sync path (speculative).
- Callbacks: MemoryCallbacks + LangfuseMemoryCallbacks; on_memory_llm_call
subsumes tracing_callback (same signature/timing/mutation); deleted the
tracing_callback field. DeerMem decoupled from langfuse (portability).
- noop keeps read-op empty overrides (avoids router 500s on the
disable-memory-via-noop path); only delete/export inherit the base raise.
Behavior preserved: 661 passed / 13 skipped. Docs: backends/README.md rewritten
(three-tier + from_config + callbacks); samples README updated; removed stale
private doc paths.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): 501 on unsupported read/manage endpoints + accurate warm log
Review follow-up on the three-tier MemoryManager refactor.
- Read/manage endpoints (GET /memory, /memory/export, /memory/status,
DELETE /memory, POST /memory/import) and the /memory/reload fallback now
catch NotImplementedError -> 501, matching the fact-CRUD endpoints. The
hasattr->try/except migration had skipped these: they were @abstractmethod
before (every backend implemented them, so they never raised), so once they
became tier-2 default-raise a minimal backend (only add + get_context) hit a
raw 500 -- there is no global NotImplementedError handler. get_memory is
shared via _get_memory_or_501 (covers /memory, export, status, reload
fallback). noop is unchanged: its read-op empty overrides never raise.
- warm() base default returns None (tri-state: True=warmed, False=failed,
None=nothing to warm) so the Gateway lifespan logs "skipping" for a
non-DeerMem backend (e.g. noop) instead of the inaccurate "warmed
successfully" it never earned. DeerMem.warm keeps True/False.
- Tests: 6 router 501 tests (read/manage + reload fallback) + 2 lifespan
warm-log tests (None->skipping, False->warning); conformance/pluggable
assert warm() is None.
705 passed / 13 skipped; lint clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): review follow-ups - search-flag consistency, client reload, backend_config purity
Address review feedback on the three-tier MemoryManager refactor:
- [Medium] supports_search/search drift: the invariant now requires the
supports_search ClassVar flag to MATCH whether search() is actually
overridden (type(self).search is not MemoryManager.search), so the flag
can't drift from the impl. Catches both directions at instantiation: a
backend that overrides search() but forgets supports_search=True (was a
misleading tool-mode rejection), and one that sets the flag without
overriding (was a runtime NotImplementedError on the first memory_search).
noop sets supports_search=True to match its search() override. Conformance
adds drift + consistent-backend tests.
- [Low] client.reload_memory fallback: wrap the get_memory fallback so a
minimal backend (only add + get_context) surfaces a clean NotImplementedError
("implements neither reload_memory nor get_memory") instead of an uncaught
propagation -- mirrors the router's 501. Test added.
- [Low] backend_config purity: DeerMem.from_config restores backend_config to
the pure data the host passed after model_post_init parses the injected hooks
into DeerMemConfig (self._config, PrivateAttr); the field stays serializable
(no callables/LLM) and matches the README ("host hooks NOT in backend_config").
Test asserts purity + hooks wired.
- [Low] CHANGELOG: breaking-change note that mode='tool' + non-search backend
now fails fast at startup (was silently empty) so operators recognize it on
upgrade.
- [Nit] .gitignore: drop the env-specific .tmp-pytest/ entry (--basetemp is
local-only, not make test/CI).
709 passed / 13 skipped; lint clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(changelog): correct memory tool-mode fail-fast note
The CHANGELOG entry said mode='tool' + a non-search backend "(e.g. noop)"
fails fast at startup, but noop overrides search() (returns []) and sets
supports_search=True (required by the consistency invariant), so noop IS
search-capable and noop+tool does NOT fail fast. The fail-fast only affects a
custom backend that onboards without overriding search(). Reworded to drop the
misleading noop example and state both shipping backends implement search().
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Vanzeren
|
42baed8c8c
|
feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel (#4292)
* feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel
Add a restart-required database.checkpoint_channel_mode ("full" default,
"delta") that stores the messages channel via LangGraph 1.2 DeltaChannel,
cutting checkpoint storage from O(n^2) to O(n) for append-only history.
Existing full checkpoints seed delta state transparently; no data migration.
- config: mode schema + freeze-on-first-use with
CheckpointModeReconfigurationError; mode marker persisted in checkpoint
metadata; unsafe delta->full downgrade rejected fail-closed with
CheckpointModeMismatchError (run-level error, failed state read)
- state: delta message state schema; CheckpointStateAccessor centralizes
materialized reads for all consumers (threads API, branches,
regeneration, compaction, state updates, memory, goal workers)
- runtime: raw writers (run durations, interrupted title, thread goal)
parent their checkpoints to the checkpoint they derive from, preserving
delta ancestry; rollback forks the pre-run lineage through a state
mutation graph with Overwrite restores; InMemorySaver delta-history
override delegates to the base walk (fixes dropped first write after
migration, also present upstream)
- tests: conformance suite over {memory, sqlite, postgres} covering
migration replay, stable message IDs, storage shape and writer
preservation; conftest fixture isolates the frozen mode between tests;
stale config fakes refreshed
- ci: backend unit tests gain a postgres service
* fix(checkpoint): close materialization gaps in goal flow, guard public factory
- Route goal-continuation message reads through CheckpointStateAccessor:
raw channel_values reads see the delta sentinel in delta mode, which
disabled goal continuation (stand_down=no_durable_end_of_turn) after
durable assistant turns. Raw tuples remain for tuple-only metadata
(checkpoint id, pending_writes).
- Reject checkpoint_channel_mode='delta' + checkpointer in
create_deerflow_agent at construction: factory-built persisted graphs
bypass mode-marker injection and the fail-closed gate, reproducing
silent mixed-mode state loss. Delta without persistence stays allowed.
- Import the postgres saver lazily (pytest.importorskip in the fixture)
so the documented default install collects the suite; add a CI job
running pytest --collect-only on uv sync --group dev without extras.
- Fix test_checkpointer fallback test to patch get_app_config at its
use site (provider module), making it deterministic when a local
config.yaml selects a persistent backend.
* fix(gateway): preserve extension-owned channels in state mutations, bump config version
- build_state_mutation_graph / build_checkpoint_state_mutation_accessor
accept an explicit state_schema; branch and POST /state now compile the
mutation graph from the thread's effective schema
(graph_state_schema on the assistant graph). The base-ThreadState
fallback silently discarded channels contributed by custom
AgentMiddleware.state_schema on branch (data loss) and returned a
false-success 200 on POST /state.
- POST /state validates values keys against the mutation graph's
channels and rejects unknown fields with 422 instead of ignoring
them; reducer detection covers extension channels
(BinaryOperatorAggregate or DeltaChannel) so Overwrite replace
semantics work for middleware reducers in both modes.
- Endpoint regression: custom AgentMiddleware.state_schema value
survives branch, updates through POST /state, and an unknown field
receives 422.
- config_version 26 -> 27 for the new database.checkpoint_channel_mode
(example, Helm chart values + README, support-bundle fixture), so
existing installs get the outdated-config warning and
make config-upgrade merges the field; covered by a test driving the
real example file and the real config-upgrade script.
* fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite
GET /threads/{id}/state now resolves the thread's assistant_id through a
single reusable boundary (thread metadata -> assistant_id -> effective
graph), so channels contributed by a custom AgentMiddleware.state_schema
are materialized instead of dropped by the default lead schema. POST
/state uses the same boundary instead of resolving the schema ad hoc.
Branch writes wrap every copied reducer channel in Overwrite (derived
from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel),
not just messages, so already-aggregated values are never re-merged.
Regression tests use a real AgentMiddleware.state_schema with a
non-identity reducer in both full and delta modes: GET /state returns the
extension value, POST /state replaces it, branch preserves it
byte-for-byte; the unknown-field 422 is a separate assertion.
* refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests
Address review round 4 on PR #4292:
- Push ahistory/history limit through Pregel into checkpointer.alist
(SQL LIMIT) instead of materializing all rows and breaking in Python
- Fold the read-side mode-compat gate onto the returned snapshot's
metadata; only writes keep the pre-write tuple fetch (fail-closed)
- Cache factory-built accessor graphs per (assistant_id, mode) with
factory-identity revalidation; state reads no longer build a lead
agent per request
- get_thread: one snapshot fetch + one raw pending_writes fetch on the
resolved checkpoint (post-checkpoint __error__ writes never surface
in snapshot.tasks; verified empirically)
- DeerFlowClient.get_thread: single checkpointer.list walk collects
pending_writes per checkpoint instead of N get_tuple calls
- InMemorySaver delta-history patch: stand-down when the upstream
override disappears, try/except guard, validated-version warning,
guard tests
- make_lead_agent mode precedence: first freeze is owned by app_config
(client-supplied configurable key ignored); once frozen, injected
key/app_config must match or fail closed
- Rollback: lock in non-message channel restoration via fork
inheritance with a dedicated reducer-channel test
- Add tests/test_threads_checkpoint_mode.py and
tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and
the PR validation section: lifecycle parity (memory + sqlite),
per-step blob-count storage guard, gateway endpoint parity
Counted-saver tests pin checkpoint round-trips for aget/ahistory so
these regressions cannot silently return.
* fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience
- threads router: map CheckpointModeMismatchError to 409 (with cause and
thread id) and CheckpointModeReconfigurationError to 503 across all state
endpoints instead of swallowing both into a generic 500
- gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and
assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary
through the real route stack
- rollback: compile the restore mutation graph with the thread's effective
state schema per the build_state_mutation_graph contract
- inheritance contract locks: rollback and manual compaction preserve
middleware-contributed channels via checkpoint fork cloning
- services: revalidate the accessor-graph cache against app_config identity
so config.yaml hot-reloads never serve a stale compiled graph
- services: degrade full-mode state reads to raw checkpointer reads when the
agent factory is unavailable (delta gate still applies; delta mode has no
fallback)
- deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently
downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility)
and bump the langchain lower bound to what the lockfile actually resolves
* fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread
- _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while
pregel's get_state_history treats config.checkpoint_id as the inclusive
start; fetch the anchor explicitly so both read paths paginate identically
- extend the degraded-path gateway test: GET /thread returns raw values, and
POST /history with before starts at the anchor checkpoint
* fix(gateway): preserve degraded checkpoint timestamps
* fix(gateway): harden degraded checkpoint access
* fix(gateway): resolve assistants for checkpoint reads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
ChenglongZ
|
ca16b64b26
|
feat(agent): support config-declared lead middlewares (#3964)
* feat(agent): support config-declared lead middlewares * fix(agent): preserve configured extension middlewares * fix(agent): address middleware extension review * fix(agent): tighten middleware extension docs and tests |