59 Commits

Author	SHA1	Message	Date
Wenchao An	1f437f86c6	feat(agents): persist default knowledge scopes for custom agents (#5579) ... * feat(agents): persist default knowledge scopes for custom agents * style: format agent knowledge guidance * fix(i18n): clarify default knowledge reset hint * fix(knowledge): preserve retries for initially unbound agents	2026-09-20 16:31:35 +08:00
zhangwei-way	b6503e9a35	feat(knowledge): add per-message RAGFlow retrieval scope (#5238) ... * feat(knowledge): integrate RAGFlow retrieval and management * test(knowledge): cover merged listing tool * feat(knowledge): add per-message retrieval scope * chore(docs): remove unrelated document * docs(knowledge): add interaction screenshots * feat(knowledge): simplify scope selector trigger * docs(knowledge): refresh selector screenshot * feat(knowledge): defer standalone management * docs(knowledge): show chat-only scope UI * fix(knowledge): honor scope on clarification replies * fix(knowledge): harden scoped replay validation * docs(knowledge): clarify replay scope precedence * fix(knowledge): keep provider settings on tools * fix(config): preserve tools-only knowledge settings * fix(knowledge): submit custom assistant identity * refactor(knowledge): trim PR scope changes * fix(knowledge): sanitize document scope display * feat(knowledge): enable scope selection in main chat * fix(knowledge): emphasize active scope icon without button frame * fix(knowledge): close context scrubbing and refresh e2e checks * fix(knowledge): preserve idempotent canonical retries * fix(knowledge): accept promptless conversation runs * style(knowledge): format backend regression tests * chore(knowledge): trim PR scope and fix frontend format * fix(knowledge): remove shared-scope notice * fix(knowledge): remove scope persistence notice * docs(knowledge): include main chat in catalog scope * fix(knowledge): preserve scope recovery and upgrades * fix(config): preserve LightRAG knowledge upgrades --------- Co-authored-by: foreleven <for-eleven@hotmail.com>	2026-09-18 16:59:31 +08:00
Zeren Wang	a58ab484a6	feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443) ... * feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md (issue #5160, tracker #5129) in the slice order of the spec's §16. Slices: - A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY admission pinning (both server-owned sets + worker hoist); latest-only request-scoped <project> block via DynamicContextMiddleware wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID prefix + marker + provenance, never persisted); journal audit fingerprints; Instructions tab. - B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with locked check-and-set; hash-qualified immutable shelf storage with Paths helpers; upload/list/content/delete-to-trash routes; project delete trashes the shelf in-transaction; request-scoped bounded <documents> index with honest count/shown + actionable overflow note; list_project_documents/read_project_document tools registered only on pinned runs; PAT allowlist + drift guards; blocking-IO anchors. - C: shared thread-upload ingestion service (uploads router refactored to parity); POST from-thread with provenance; attach-to-thread with lock-staged copy (archived source allowed); read-only thread-files view with per-group truncation reporting. - D: restore (restored/merged/not_found/no_target/content_missing; no file moves), purge (continuous row lock across unlink/delete/commit, retryable on FS errors), retention sweep (lazy + startup, 24h orphan guard, row-side reconciliation never deletes). - E: Documents tab (shelf + conversation-files browser, provenance, archived banner, content-missing rows), /workspace/trash route, sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks + specs. Review hardening folded in (10 rounds, all with tests): - force active shelf content (HTML/XML family) to download; nosniff on artifact + content responses; unified unsandboxed-iframe PDF preview (fixes the pre-existing Chromium sandbox blank in the artifact viewer) - scope document trash to the URL project under the document lock - atomic no-overwrite filename reservation for ALL ingestion (seeded claims + os.link commit with suffix retry; same-name re-upload now unique-names instead of replacing); hidden staging only, no visible placeholders; lease cleanup on setup failure - serialize conversion under the document lock with post-lock active revalidation; drain locked filesystem work on cancellation; preserve bytes when an insert's commit state is uncertain (including trashed rows) - original-integrity checks before serving text or cached conversions; content_missing surfaced in list responses (UI reads the flag, no 409-probe); downloads always serve original bytes - bounded streaming document reads with cached char counts; shelf limits declared in middleware release identity - thread-root confinement for from-thread sources; config fallback rejects fractional/infinite values; composer counts staged attachments; pending attachments persist until submission or removal; in-flight instruction/rename edits survive save refetches; shelf and trash pagination; conversation-file and thread-files pages stay subscribed to refetches Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md contracts, config.example.yaml projects block. Review follow-ups (head b4807477 → this revision): - The trash retention sweep is split so repeated lazy triggers stay bounded: the indexed expiry purge still runs on every trigger (GET /api/trash/documents, POST /api/trash/purge) while the O(all rows + all files) reconciliation is throttled to one run per user per 15 minutes (process-local, per-user window). The startup sweep now runs as a background task instead of blocking gateway readiness, and shutdown awaits it (bounded). - The export scrub (stripInternalMarkers) is fence- and indentation-aware like the render path, so a pasted, fenced <project>/<documents> snippet survives markdown export while real injected blocks (never fenced) are still removed. Fence regexes moved to a dependency-free leaf module to avoid the messages↔streamdown import cycle. - The artifact viewer's PDF iframe no longer carries an added title attribute (the upstream e2e contract locates it via :not([title])), and the upstream artifact-preview spec now pins the new contract: PDFs render unsandboxed, images keep sandbox="". * fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title - Shutdown cancelled only the shield around the background startup sweep, so an all-users reconciliation that outlived the 5s budget kept walking rows and files while the document repo and DB engine were disposed underneath it. The wait now lives in `_shutdown_startup_trash_sweep`, which cancels the task and drains it before worker exit: the shield keeps the wait bounded, the cancel makes it final (CancelledError lands at the sweep's next await, and `_run_startup_trash_sweep` only catches `Exception`, so nothing swallows it). - The browser-preview iframe lost `title={getFileName(filepath)}` in the previous fix round, leaving the PDF frame without an accessible name while its siblings keep theirs. Restore it (WCAG frame titles), assert it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]` instead of `iframe:not([title])`. * fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel `Task.cancel()` returns False when the sweep already finished inside the window between the deadline firing and the cancel, so the shutdown log claimed a cancellation that never happened. Branch on that outcome: the warning stays for a real cancel, a late finish is logged at info, and both paths still reap the task before worker exit. * fix(projects): round-4 review — make Empty trash delete what it confirms `POST /api/trash/purge` only ran the retention sweep, and the sweep's candidate selection is age-gated, so a freshly trashed document survived "Empty trash" even though the confirmation promises that every listed document is permanently deleted. With one trashed row the route answered `{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps expired rows before listing, so the visible rows were normally ineligible for the action by construction. Empty trash now drives `purge_all_trashed`: the caller's trashed rows (`list_all_trashed`, no age filter) each go through the same guarded, row-locked `purge` as the single-document delete — bytes first, then the row, in one transaction — so a row restored mid-flight is skipped instead of force-deleted, and an unlink failure rolls that row back and answers 500 with a retryable message. Retention expiry stays where it was: the sweep's `purge_candidates` is now the only age-gated selection, and the lazy retention sweep still runs on the listing and at startup. Tests: the router suite replaces the retention-gated expectation with the reviewer's repro (fresh row purged, bytes unlinked, shelf and other users' trash untouched, a failing unlink stays retryable and 500); a blocking-I/O anchor drives the new entry point through the offload; the mocked e2e covers the action end to end; a new real-backend spec performs it against the real gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE and the phase-2 design docs (en+zh) state the age-independent contract.	2026-09-16 18:46:18 +08:00
PeaceMaker-best	14c9d44440	feat(runtime): persist tool-progress phase transitions (#5214) ... * feat(runtime): persist tool-progress phase transitions Record bounded warn, block, and recover decisions for lead and task subagent runs while preserving event-loop isolation, fail-open behavior, and concurrent transition order. * fix(runtime): trust server-owned tool progress attribution * fix(runtime): centralize trusted audit attribution * fix(runtime): preserve complete tool progress audit state * docs: trim tool progress guidance to pass size check * fix(runtime): fence subagent audit recorder loop * docs(readme): sync tool-progress event coverage Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: 嗜鵼 <hy2010hy2010@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-15 08:45:08 +08:00
Hyeonsang Cho	bbced51022	fix(gateway): close two input-sanitization bypasses (#5375) ... * fix(gateway): reject forged framework-injection markers in run input `is_genuine_user_message` treats `hide_from_ui` and a human `name="summary"` as proof the framework authored a message, and `InputSanitizationMiddleware` skips those — escaping a real reminder's blocks would corrupt trusted context. Neither marker was server-owned, so an external caller could set either one and place a raw `<system-reminder>` outside the user-input boundary markers, which the lead-agent system prompt declares trusted internal framework data. The `hide_from_ui` variant is also filtered out of the thread UI, so the forgery was invisible where it landed. Both markers are now stripped from untrusted input, on the run path and on the thread-state mutation route that writes straight into a checkpoint. Framework injection happens inside the graph and never crosses this boundary, so nothing the framework does is affected, and `trusted_internal` callers (IM channels, the MCP task-notification launcher) keep writing hidden messages. HumanInputCard replies are the one legitimate external `hide_from_ui`: the frontend sends it alongside a `human_input_response` payload, so a message carrying a valid one keeps the marker. That buys no bypass — the predicate already classifies those as genuine, so they stay sanitized. The name check reuses the predicate's own `_SUMMARY_MESSAGE_NAME` rather than a fourth copy of the literal, and matches by `isinstance` exactly as the predicate does: `HumanMessageChunk` is a `HumanMessage` whose `type` is not `"human"`, so a type-based check would leave that subclass's marker settable. `name` is only reserved on human messages — on a ToolMessage it is the tool's own name. Three tests in test_gateway_services.py and test_message_provenance.py asserted that a caller-supplied `hide_from_ui` survives. That assumption was the bypass; they now pin the opposite, with a genuinely caller-owned key kept alongside to prove the stripper is surgical. * fix(agents): sanitize every genuine user message, not only the newest The input guardrail scanned backwards for the first genuine user message and returned, so only the newest turn was ever sanitized. The transformation is request-scoped (`wrap_model_call`, never written to state), so thread state keeps the raw text: once a newer turn arrived, the previous turn's payload was replayed to the model verbatim, outside the boundary markers the lead-agent prompt declares trusted framework data. The guardrail therefore held for exactly one model call. Reaching it needed no forged metadata and no crafted request body — type the payload in one turn, then send anything at all in the next. A single request carrying two user messages did it in one shot, since every message but the last was skipped. `_process_request` now walks the whole list and `_sanitize_message` owns the per-message work; every existing branch (the `original_user_content` split for upload turns, the multimodal rfind fallback, the metadata repair) is unchanged. Framework-injected messages stay excluded by `is_genuine_user_message`. Unexpected errors now fail open per message rather than per request. Iterating history widened the old blast radius: one unprocessable row would have dropped sanitization for the whole request, handing an attacker the newest turn by crafting an older one. `GraphBubbleUp` still propagates. Side effect worth noting: each turn's rendering is now stable across model calls. Previously a turn was wrapped on its own call and unwrapped on the next, changing the prompt prefix behind the newest turn and defeating prompt caching. test_only_processes_last_user_message pinned the old scope; it now pins that every turn is processed, and keeps driving the `wrap_model_call` entry point. * docs: record the message-metadata trust boundary and sanitization scope `agents/middlewares/AGENTS.md` owns the depth for InputSanitizationMiddleware and documented only the `original_user_content` half of its trust boundary. Left alone it would teach an agent that `hide_from_ui` is caller-owned and that the guardrail covers one turn — and the usual failure mode is an agent "restoring" the behaviour it believes was lost. The entry now carries both markers, the HumanInputCard exception, the whole-history scope, and the per-message fail-open rule. The note lives only there. The root and `backend/AGENTS.md` layers are orientation that points at the module guides owning the depth, and `backend/AGENTS.md` is inherited by every backend chain — prose added there inflates more than twenty of them, and `scripts/check_agent_guidance.py` shows the middlewares chain has about a kilobyte of room against its hard limit. CHANGELOG.md and CHANGELOG_zh.md record it under Security, continuing the existing prompt-injection lineage. * fix(gateway): mark caller-hidden messages instead of stripping the marker Review follow-up. Stripping a caller-supplied `hide_from_ui` closed the bypass but broke three frontend senders that use the marker purely to keep a context message out of the transcript: the quoted conversation context (`buildHiddenConversationQuoteMessage`), the sidecar context prompt (`buildHiddenSidecarContextMessage`), and the agent save command. None carries a `human_input_response`, so the HumanInputCard carve-out did not cover them, and nothing else hides them — `_is_branch_visible_message` and the frontend's `isHiddenFromUIMessage` both key solely on `hide_from_ui`, and no backend reads `conversation_quote_context` or `sidecar_context`. All three would have rendered as user-visible chat bubbles. The marker plays two roles and only one of them is a vulnerability. The security requirement is that a caller-supplied marker cannot skip sanitization, not that it cannot hide a message. So the roles are separated instead of the marker being removed: the Gateway keeps it and stamps the server-owned `UNTRUSTED_INPUT_KEY`, and the guardrail now asks `requires_input_sanitization` — the mark, else the genuine-user test. Hidden stays hidden; untrusted content is sanitized either way. The reserved `summary` name is handled the same way and no longer rewritten. Marking rather than removing is also the safer shape in general: `hide_from_ui` is read for presentation, journal persistence, memory filtering and IM outbound as well, and this boundary should not silently change any of them. `is_genuine_user_message` is deliberately left alone. `ToolReceiptMiddleware` uses it for turn-boundary detection, where a caller's hidden context message must keep counting as not user-authored; widening it there would move the ledger's turn window. `requires_input_sanitization` sits beside it in `message_utils` so the two questions can be compared. The three tests that asserted a caller-supplied `hide_from_ui` is removed now assert it survives and carries the mark — which restores the original intent of the two provenance cases, whose comment already read "caller-owned keys must survive". * docs(gateway): rewrite normalize_input's docstring around the mark Review follow-up. The paragraph still described the pre-4a3344f6 strip model and contradicted both the implementation and the middlewares AGENTS.md paragraph updated in that same commit: it called `hide_from_ui` server-owned, said carrying it skips sanitization entirely, and repeated the premise this branch disproved — that HumanInputCard replies are the only legitimate external use. It now describes what the code does: the markers stay caller-owned and are preserved because three frontend senders rely on `hide_from_ui` for hiding alone, the message is stamped with `untrusted_input` instead, and `requires_input_sanitization` sanitizes it anyway. `untrusted_input` joins the server-owned inventory in the preceding paragraph, which is what makes the stamp unforgeable and unclearable. The three surrounding docstrings now also say that these functions mark as well as strip; `_strip_external_message_metadata` had advertised only the removal, leaving a reader no way to find the stamp from the name. * fix(gateway): mark state writes whose message omits additional_kwargs Review follow-up. The state-route half of the fix missed the most natural request shape. `_strip_external_metadata_from_message_like` returned early when `additional_kwargs` was absent or not a dict — there was nothing to strip — and that early return also skipped the mark. A `POST /threads/{id}/state` body of `{"values": {"messages": [{"role": "user", "name": "summary", "content": "<system-reminder>…</system-reminder>"}]}}` therefore reached the checkpoint unmarked. The messages reducer's `convert_to_messages` then supplies `additional_kwargs={}`, so at model-call time `requires_input_sanitization` fell back to `is_genuine_user_message`, which a `summary` name fails, and the forged tag reached the model raw and outside the boundary markers. A missing or non-dict `additional_kwargs` is now treated as empty for both the strip and the mark. The identity return is kept for the case where nothing changes, so an ordinary key-omitted message does not gain an empty dict just by passing through. The run path was never affected: `normalize_input` coerces to BaseMessage first, which always carries the dict. Every existing state-write test supplied an `additional_kwargs` dict, which is why this shape slipped through; the regression now covers it at the route and end to end through the reducer into the guardrail. While checking the neighbouring shapes, `_skips_input_guardrail` keyed off key presence where `is_genuine_user_message` keys off truthiness, so `hide_from_ui: False` — already covered without a mark — would have been stamped. It now mirrors the predicate exactly, as its docstring claimed. * docs(middlewares): compress the sanitization note to fit the guidance chain main's growth left the middlewares AGENTS.md chain 84 bytes under its hard limit, and the fuller wording did not fit. The load-bearing facts stay — the markers are marked rather than stripped, and the scan covers every turn — since those are the two an agent editing this middleware could otherwise get wrong. The full model lives in the normalize_input, _mark_untrusted_framework_markers and requires_input_sanitization docstrings. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-14 09:54:42 +08:00
Hyeonsang Cho	6f81daefff	fix(runtime): resolve omitted run owners before idempotent reuse (#5401) ... * fix(runtime): resolve omitted run owners before idempotent reuse HTTP run admissions do not pass a user_id. The SQL run store stamps the request user from the contextvar onto the row, but RunManager kept None on its process-local RunRecord, so the two disagreed about who owns the run. A keyed retry that reached a peer worker, or the owning worker after cleanup() released its local record, hydrated the stamped row, compared its owner with None, and raised "Run idempotency key resolved to a different thread or user", which start_run surfaced as a 500. MemoryRunStore stored None on both sides and never hit the check. The same mismatch hid HTTP runs from owner-scoped history reads, which filter local records by the current user, and skipped the worker's MCP background_tasks projection, which only runs for records with an owner. create() and _admit_thread_operation() now resolve an omitted owner from the current user before building the record, and keep None when no user is in context instead of falling back to the default bucket, so the local record and every store agree on the owner. HTTP runs now receive the background_tasks projection, so the replay golden's values frames gain that key. * docs(changelog): reference #5401 in the keyed retry owner fix entry * test(runtime): close the SQL engine when peer-reuse test setup fails Review follow-up on #5401: the sql case of the two-worker start_run test initialized the engine above the try whose finally calls close_engine(). init_engine() assigns the module-global engine and session factory before bootstrapping the schema, so a failure there skipped the teardown and left a stale engine for later tests in the same process. Store setup and the RunManager workers now live inside the try, so the teardown runs whether setup or the test body fails.	2026-09-13 20:28:32 +08:00
Hyeonsang Cho	28a81452ce	fix(runtime): stop idempotent reuse from blocking the thread on a peer worker (#5393) ... * fix(runtime): stop idempotent reuse from blocking the thread on a peer worker When an idempotent run admission lands on a worker that does not own the run, RunManager hydrates the stored row and returns it as the reused record. It also registered that row in the worker's local run map, but only the owning worker's task lifecycle finalizes and cleans up local records, so the copy kept its admission-time pending/running status for the life of the process. On that worker every later reject-strategy admission for the thread returned 409 until a restart, run reads kept reporting the stale status, orphan reconciliation skipped the run as locally live if the owner crashed, and a cancel took the local-owner path and marked the owner's still-running row interrupted. Return the hydrated row as a detached store-only handle instead of registering it. A local record for the key is already returned before the store insert, so the removed lookup of an existing local record was unreachable. get(), cancel() and reconciliation now read the durable row on the peer, matching the documented non-owner contract. * test(runtime): pin keyed retries of a terminal reused run on the SQL store Review follow-up on #5393: the post-cleanup release relied on a keyed retry resolving through the terminal row's idempotency conflict, but only MemoryRunStore pinned that path, and no test retried on the owner after its local record was cleaned up. The SQL repository test now retries the key on the peer and on the owner once the run is terminal and cleaned up, asserting both get the same run back as a store-only reused handle before the keyless follow-up is admitted.	2026-09-13 18:16:22 +08:00
Xuehao Xu	56540fab01	fix(gateway): make recursion limit configurable (#5390) ... * fix(gateway): make recursion limit configurable * docs: keep backend guidance within inherited size budget * fix(gateway): address recursion limit review feedback * docs(gateway): clarify recursion default scope --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-13 17:08:33 +08:00
Zhengcy05	81f2015fe6	fix(runtime): keep agent construction off event loop (#5217) ... * fix(runtime): keep agent construction off event loop * fix: - offload checkpoint state accessor graph construction to a worker thread - update test * import AsyncKeyedLockTable * update Agents.md * fix: update test * fix: preserve single-flight builds after cancellation --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 21:09:05 +08:00
Shxiao	572744975d	fix(tools): run tool assembly off-loop at async entry points (#5224) ... * fix(tools): run tool assembly off-loop at async entry points get_available_tools() may block on MCP cache initialization while it is called on async agent-assembly paths (task_tool, durable batch execution), stalling the calling event loop for the full discovery duration. Dispatch the (unchanged, synchronous) assembly call to a worker thread via asyncio.to_thread at the two async entry points so the loop keeps processing requests, SSE frames, cancellations, and timers. Fixes #5172 * fix(tools): offload lead-agent assembly off-loop and pin with blocking-io anchors Review follow-up for #5224: - run_agent now dispatches agent_factory(...) through asyncio.to_thread, so lead-agent assembly (including both get_available_tools call sites in _assemble_lead_agent) runs off the event loop — the Gateway headline scenario from issue #5172. - _ensure_sync_invocable_tool takes a double-checked threading.Lock, making the in-place tool.func wrap on the shared tool singletons explicitly single-shot now that assembly can run concurrently on worker threads. - Add backend/tests/blocking_io/test_tool_assembly_offloop.py: blocking-probe anchors for task_tool and SubagentBatchService._execute_item under the strict Blockbuster gate, plus a meta-check proving the gate trips on the exact syscall class (ExtensionsConfig.from_file on the loop). Verified the anchor goes red when the offload is flattened back to a plain call. * fix(gateway): build checkpoint state accessor off-loop; anchor run_agent offload Review follow-up for #5224: - Add abuild_checkpoint_state_accessor (asyncio.to_thread around the unchanged sync builder) and switch every async call site to it: the stateless_wait route, thread_runs, both threads call sites, and the build_thread_checkpoint_state_accessor boundary. The agent-factory assembly re-enters get_available_tools() and may block on MCP cache initialization; repeat calls hit _state_accessor_graph_cache and only pay the thread hop. - Add a third blocking-io anchor driving the real run_agent with minimal RunManager/bridge stubs; the factory performs a real production blocking read (ExtensionsConfig.from_file()) and the test asserts assembly never runs on the main thread. Verified the anchor goes red when the run_agent offload is flattened back to a plain call. - Adapt the test_threads_router checkpoint-builder patch sites to the new async name. * refactor(tools): carry assembly offloads on a dedicated bounded pool Review follow-up for #5224: - Add utils/assembly_io.py: a dedicated ThreadPoolExecutor (default 8 workers, DEER_FLOW_ASSEMBLY_WORKERS-overridable, mirroring utils/file_io.py and tools/sync.py) with run_assembly(), which copies contextvars explicitly. A hung stdio MCP server parks its worker for the full MCP timeout; carrying assembly hops on the loop's default executor would let a few parked assemblies queue every other to_thread/run_in_executor(None, ...) caller behind them. - Switch all four offloads (run_agent, task_tool, batch _execute_item, abuild_checkpoint_state_accessor) to run_assembly(). - State the cold-path behavior in the accessor docstring: the graph cache validates factory identity, so non-identity-stable factories may duplicate lead-agent assembly across concurrent readers (MCP discovery stays process-wide single-flight); the pool bounds the duplicates. - Add a fourth blocking-io anchor driving build_thread_checkpoint_state_ accessor with a per-resolution fresh factory (always a cache miss) and the real production blocking read; enumerate all four offloads in the gate's module docstring. Verified the anchor goes red when abuild_checkpoint_state_accessor is flattened back to a plain call. * fix(subagents): revalidate batch item before launch; make assembly pool observable Review follow-up for #5224: - _execute_item() revalidates the durable state right after assembly and before executor.execute_async(): renew_item_lease() returns valid=False when cancel_batch() terminalized the item or the lease was lost while assembly was parked, and the launch is skipped (the canceller already finalized the item). Previously the launch was unconditional and the poll loop's cancellation checks only started after execution began. - Regression test driving the real SQLite repository: a blocking assembly probe parks _execute_item, cancel_batch() lands, and the launch is skipped with the item staying cancelled. Verified the test goes red when the revalidation is removed. - run_assembly() tracks pending assemblies and logs a throttled WARNING once the pending count exceeds the worker count, so assembly starvation (workers parked on a hung MCP server) is distinguishable from idle. - The run_agent blocking-io anchor now binds a sentinel extension snapshot via ctx.extensions and asserts the factory observed it through get_agent_build_extensions(), pinning run_assembly()'s ContextVar propagation. Verified red when ctx.run is dropped. - Document the assembly pool in backend/AGENTS.md. * fix(utils): decrement the assembly pending count on the pool thread The pending-assembly counter behind the starvation warning decremented from the asyncio future's done callback, which never fires once the submitting loop is closed while its worker is still running: the count ratcheted up permanently and eventually fired the starvation warning with no starvation behind it (reproduced at 97dc9bec by review). Decrement instead from the dispatched work item: run_assembly() wraps func so a finally drops the count under the pending lock on the pool thread, and the done callback is gone. Pin the counter with tests/test_assembly_io.py: a healthy call returns the count to zero, and an abandoned loop (stopped while the worker is parked) does not wedge it — the abandoned case goes red against the old done-callback decrement. * docs(utils): fix the pending-counter comment after the decrement move The comment still described the removed done-callback decrement, contradicting _work()'s own comment; state the actual mechanism (increment on the loop before dispatch, decrement from the dispatched work item's finally on a pool thread). * test(gateway): retarget checkpoint-accessor stubs to the services seam thread_runs and runs now call abuild_checkpoint_state_accessor, so the upstream wait-reader, regenerate-prepare, and idempotency tests must stub the sync builder where abuild resolves it (app.gateway.services); stubbing the removed router re-exports fails with AttributeError at setup. The async seam semantics are unchanged: run_assembly invokes the stubbed sync builder off-loop and propagates its return values and exceptions. Move the agent/tool assembly off-load note from backend/AGENTS.md to deerflow/utils/AGENTS.md (next to assembly_io.py) so the effective instruction chain for agents/middlewares no longer grows past the AG002 hard limit. * fix(runtime): serialize same-key accessor assembly and release queued-cancel slots Address the three review follow-ups on the assembly off-load: - assembly_io: a job cancelled while still queued never runs its work item, so the dispatched finally never fired and _pending_assemblies stayed elevated until a false starvation warning. Exactly-once cleanup now rides the concurrent future's cancelled() state — cancel() only succeeds before the executor starts the item, so cancelled() is true precisely when the finally will never run — plus a submit-failure release; the one-worker queued-cancellation case is pinned red/green. - services: overlapping cold readers sharing one cache key could both run full agent assembly. _state_accessor_graph now serializes per key through a thread-side KeyedLockTable (pool threads, no running loop) and re-validates factory/app-config identity under the lock, so the factory runs exactly once while identity changes still rebuild. Cache dict access is lock-guarded now that construction runs off-loop. - guidance inventory: register deerflow/utils/AGENTS.md in EXPECTED_GUIDANCE_PATHS so test_repository_has_the_approved_scoped_ guidance_shape matches the relocated assembly note (CI shard 4). * test(keyed-lock): pin KeyedLockTable reclamation and waiter bypass directly Thread-side counterparts of the async table's own tests: overlapping hold() calls serialize (a late arrival joins the live entry instead of creating a second lock that bypasses a queued waiter), the last check-in pops the entry, and many unique keys leave the registry empty. Both regressions verified red — popping unconditionally trips the late-arrival test, never reclaiming trips the many-keys test. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 07:41:24 +08:00
Hyeonsang Cho	556975f284	fix(gateway): gate github_token and disable_clarification on internal callers (#5338) ... * fix(gateway): gate github_token and disable_clarification on internal callers `non_interactive` is honored only for internally-authenticated callers because it strips `ask_clarification` from the lead-agent toolset. The two sibling run-context keys reproduced that effect without the gate. `merge_run_context_overrides` forwarded `_CONTEXT_RUNTIME_ONLY_KEYS` regardless of `internal`, and `strip_internal_context_keys` scrubbed only `_CONTEXT_INTERNAL_CALLER_KEYS` -- so any session or PAT caller could set `disable_clarification` through `body.context`, or through the free-form `body.config` that `build_run_config` copies verbatim. That is not a milder flag than `non_interactive`: ClarificationMiddleware answers every clarification -- `risk_confirmation` included -- with "proceed without asking" instead of interrupting, and SandboxMiddleware reads the two keys as the same non-interactive signal. `github_token` rode the same path into `runtime.context`, where the bash tool exports it as `GH_TOKEN`/`GITHUB_TOKEN`, and a copy smuggled through `body.config['configurable']` reached the checkpoint store the context-only rule exists to avoid. Both keys are produced server-side by the channel run policies, which reach the Gateway over the internally-authenticated request channel, so gate them the same way: forward them only when `internal=True`, and scrub the union `_INTERNAL_ONLY_CONTEXT_KEYS` from both config sections for every other caller. Destination stays an orthogonal axis -- `_CONTEXT_RUNTIME_ONLY_KEYS` still land in `context` alone, never in checkpoint-persisted `configurable`. Regression coverage in tests/test_gateway_services.py pins both smuggling surfaces and replays the real start_run assembly order for a session caller and for an internal one, so the GitHub channel keeps carrying its minted token. * docs(changelog): record the internal-only run-context key gate (#5338) * docs(agents): keep the run-context note inside the AGENTS.md budgets The AG002 inherited-chain check failed at this head. The new backend section and the root scheduled-task sentence added 993 B to the root and backend guidance both the sandbox and middlewares chains inherit, pushing sandbox 6 B over the 98304 B hard limit and growing the middlewares chain, which main already exceeds by 155 B. An already-over chain is only tolerated while it does not grow, so the shared ancestors had to come back to their base size. Condensed the new material and removed prose the root file was duplicating: - The trust-boundary section keeps both gated surfaces, both helpers, the trust-vs-destination split, and the disable_clarification note in half the space. - The root scheduled-task bullet names all three internal-only keys and both smuggling surfaces while staying under its previous size. - Dropped the root `scheduler.recursion_limit` bullet, which restated backend/AGENTS.md:18 almost verbatim; its one unique fact (a YAML edit needs no Gateway restart) moved to that bullet. - Deduplicated the nginx routing sentence, which already deferred to the backend routing table, and tightened the waiver note's sequencing tail. Root and backend guidance now sit 50 B under their combined base size, so the sandbox chain returns to 97310 B and the middlewares chain no longer grows. Every file stays under its AG001 soft budget.	2026-09-11 11:33:52 +08:00
Aari	9e0fbd60fa	fix(sandbox): isolate concurrent subagent shell sessions (#5134) ... * fix(sandbox): isolate concurrent subagent shell sessions * fix(sandbox): make execution acquire idempotent * fix(sandbox): close execution lifecycle gaps * fix(sandbox): serialize retained client lifecycle * fix(sandbox): close remaining client lifecycle gaps * fix(sandbox): unwind failed client lookup * fix(sandbox): protect internal lease identities * fix(sandbox): make cancellation reconciliation durable * fix(sandbox): fence cancelled workers and IM uploads	2026-09-02 21:05:23 +08:00
rayhpeng	cd35363a05	fix(history): early user messages vanish or jump mid-run when pagination and context compaction overlap (#4696) ... * fix(history): stop dropping user messages that fall outside the loaded page window Two independent paths made a user's own message disappear from a long thread (#4666, #4508, #4363). Both are reproduced by a real two-round run: once the thread passes the 50-row `/messages/page` window AND context compaction fires, the two sources of truth stop overlapping at the head. 1. Middleware-answered tool results never reached the event store. A middleware that short-circuits a tool call (e.g. ReadBeforeWriteMiddleware's blocked write) returns a user-visible ToolMessage, but LangChain never emits `on_tool_end`, so RunJournal never persisted it — the user saw it during the run and it vanished on reload. RunJournal already reconciles final-output tool messages, but only for an `ask_clarification` allowlist. The allowlist is removed; scope stays bounded by the three conditions that actually matter (visible, this run's lead agent, not already persisted), so subagent results still stay in their own step feed. 2. mergeMessages discarded the checkpoint prefix before the first shared anchor. #4065 correctly established that a summarization-rescued early message must not be appended to the tail, and suppressed it instead. That suppression is what deletes the message when the first history page no longer reaches back to it. It is now woven in before the first shared anchor — the one position both the checkpoint and seq-sorted history agree on — so #4065's invariant (never the tail) still holds. A collapsed unloaded gap is recoverable by paging; a dropped message is not. Verified against real captured payloads from the reproducing run: the first user message returns to the transcript. Its exact position is still approximate — after compaction the live window carries too few anchors to place it precisely, which only seq-based ordering can close. Backend: 10809 passed (baseline 10808; same 15 pre-existing failures in browser/crawler community tools). Frontend: 986 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(events): look up a persisted message's seq by identity Groundwork for placing checkpoint messages in the seq-ordered thread feed (#4666). A checkpoint carries no seq of its own and loses messages to summarization, so once the feed's 50-row page window no longer reaches back to a surviving old message, a client has nothing to place it by. The seq already exists in run_events keyed by the message id — this exposes it without paging the whole feed. `message_identity` is the backend half of the identity rule the frontend applies in `hooks.ts::messageIdentity`: a ToolMessage is keyed by `tool_call_id`, and DynamicContextMiddleware's `X` / `X__user` human copies collapse to one identity. The two halves must stay in sync — a mismatch is silent, degrading placement rather than raising. `get_message_seqs` is implemented for all three stores. Misses are absent from the result rather than an error, so callers degrade to their own placement rule; the earliest seq wins when one identity resolves to several rows, so a re-persisted message keeps the position it first occupied. The DB store decodes rows in Python because `content` is a TEXT column holding a JSON string, not a JSON column — the identity fields cannot be projected in SQL. Nothing consumes this yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(runtime): carry each persisted message's feed seq on values frames Attaches `additional_kwargs.deerflow_seq` to messages in a root `values` frame that the thread feed already holds, so a client can place a message the checkpoint kept but its loaded history page window no longer reaches (#4666). Nothing is written back to the checkpoint: the seq is added when the frame is serialized and belongs to that frame only. Cost is bounded to frames introducing identities the run has not resolved yet. Messages this run produces are not in the feed while streaming, so they are looked up once, recorded as misses, and never retried — in a real run the only frame that pays for a query is the one where compaction brings older messages back into view. Measured on a reproducing two-round run: 1 lookup across 25 values frames. The stamper is built once per run rather than per `_stream_once`, or a goal continuation would discard the resolved seqs. Subgraph frames are not stamped: a subagent's snapshot is not part of this thread's feed ordering. A lookup failure logs and leaves the frame unstamped rather than failing it — placement is an enhancement and clients fall back to their own ordering rule. Frontend does not read the field yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(gateway): strip the server-owned message seq from untrusted input `deerflow_seq` is display metadata the Gateway attaches when it serializes a values frame. A client replaying messages (regenerate / edit-and-rerun) would otherwise write it into the checkpoint, where it becomes wrong the moment the thread is forked — a branch re-seeds its feed and reassigns seq (#4380). Joins the existing server-owned key set, so it follows the same trusted-internal rule as the dynamic-context and view-image markers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): place a checkpoint message by its feed seq, not its nearest anchor Completes #4666. Weaving a compaction-rescued message before the first shared anchor keeps it in the transcript, but not in the right place: after compaction the live window carries too few anchors, and the nearest one can sit deep inside the loaded page window — measured at row 25 of 50 on a reproducing run, which is why the first user turn rendered mid-transcript instead of at the head. Both sides now carry the backend's thread-global seq. `buildVisibleHistoryMessages` copies each row's `seq` onto the message (same shape as the existing `run_id`), and the Gateway stamps it onto `values` frame messages it has already persisted. A live message whose seq is below the loaded window's lower bound is placed ahead of everything on screen rather than before the nearest anchor. A message with no seq — still streaming, so not in the feed yet — keeps the weaving path, since the tail is already its correct position. Verified against the captured payloads of the reproducing run: the first user message goes from absent, to #13 (behind the second question), to #0. Frontend: 988 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): place a pre-window checkpoint message even when no anchor is shared Also #4666. Placing a compaction-rescued message by its feed seq was gated on reaching a shared anchor, because the split ran inside the anchor walk. When the loaded page and the live checkpoint share no identity at all, that walk never runs and the message fell through to `[...canonical, ...live]` — appended after the entire window, the one arrangement #4065 proved wrong, with its seq known the whole time. That is not a corner case. Open an old, already-summarized conversation and send a message: the page on screen is the newest rows from before that turn, while the checkpoint holds the rescued first user turn plus steps of the new run that are not in the feed yet. On a reproducing run the two sides shared zero anchors and the user's own first question rendered at row 50 of 50 — the reported "first message jumps to the bottom". Split `beforeWindow` out of `live` before walking anchors, walk `liveInWindow`, and use it for the no-anchor branch as well, so a message routed ahead of the window is not re-appended at the tail by dedup. Measured on captured payloads of a reproducing run (real gateway, real compaction), first user message position: no shared anchor: row 50 -> row 0, seq order monotonic again shared anchors: row 0 -> row 0 (unchanged) paged to the top: row 0 -> row 0 (unchanged) Regression test verified red-green: reverting the fix fails it with the message rendered after the window. Frontend: 989 passed, eslint + tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(gateway): stamp the message feed seq on checkpoint reads, not only on stream frames Completes #4666. `_MessageSeqStamper` sits on the streaming publish path, so a client that joins a live run learns where a summarization-rescued turn belongs while a client that merely opens the conversation does not — and opening is the common case. `GET /threads/{id}/state` and `POST /threads/{id}/history` returned the checkpoint with no seq at all, so the merge fell back to the nearest shared anchor, which after summarization sits deep inside the loaded page. Reproduced in a browser against a real gateway, on a thread that had already compacted: the user's first question rendered at row 320 of 389, behind the newest question instead of at the head. Both reads showed 0 of 13 messages carrying a seq. That is the reported symptom, still present after the streaming fix. Add `stamp_messages_with_seq`, the request-scoped counterpart of the stamper: everything a checkpoint still holds is already persisted, so one batched lookup resolves the whole list and there is nothing to retry later. Resolve the store through `_optional_run_event_store` rather than `get_run_event_store`, because seq is placement metadata — a deployment without a feed must still be able to read a thread. After the fix, on the same thread in the same browser: 13 of 13 messages carry a seq and the first question renders at the head, ahead of the newest one. Backend: ruff clean, 326 passed across the touched suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(harness): move the injected-user-id suffix helpers to utils.messages to break an import cycle message_identity imported strip_injected_user_message_id_suffix from the dynamic-context middleware, closing a cycle (middleware -> deerflow.runtime -> worker -> events -> middleware) that only stayed hidden while an earlier import happened to break it. Define INJECTED_USER_MESSAGE_ID_SUFFIX and the strip helper in deerflow.utils.messages and re-export them from the middleware so existing importers keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(docs): improve formatting and clarity in AGENTS.md and message-merge.test.ts * perf(events): stop the seq scan once every wanted identity is resolved Rows past the last wanted seq can only be re-persisted copies that already lose the earliest-seq-wins tiebreak, so all three stores now break out of the scan (and the db store out of its per-row JSON decoding) once found covers wanted. Matters most for /state and /history reads of long threads, where this lookup runs with no run cache and a typically tiny wanted set. Raised by review on #4696. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(events): share the seq-stamping expression between the two stampers The walrus-plus-merge expression was duplicated verbatim between stamp_messages_with_seq and _MessageSeqStamper.stamp — two counterparts of one rule where silent divergence is the likely failure mode if only one side is edited. Both now call attach_message_seq next to MESSAGE_SEQ_KEY in message_identity.py. The trailing isinstance(message, Mapping) guard was unreachable (a non-Mapping entry already got identity = None) and is gone with the extraction. Raised by review on #4696. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(events): seq stamping survives launch paths without user context The db store's get_message_seqs defaults to user_id=AUTO, which raises when no user is in the contextvar — the first strict-AUTO read ever called from the worker context. On a launch path that never inherits the auth context (e.g. a null-owner scheduled task), stamp()'s except clause swallowed that into a per-frame warning and silently disabled seq stamping for exactly the background runs that need it. The stamper now soft-resolves the user id once at build time — the same rule as the worker's write paths beside it (unset -> no filter) — and passes it explicitly. jsonl/memory stores gain the same user_id kwarg the base list_messages contract already carries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(events): SQL-prefilter the message seq lookup's candidate rows get_message_seqs scanned and JSON-decoded every message row of the thread: the early exit never fires when a wanted identity is absent from the feed (a message still streaming, or checkpoint-only), and /state / /history reads want the newest messages, so the ascending scan traversed essentially the whole feed — with the content column carrying full tool outputs, that is heavy I/O plus N JSON parses on exactly the long threads this lookup exists for. A LIKE prefilter now keeps that cost in SQL: only rows containing a wanted raw id as a substring are fetched and decoded. False positives are re-checked by message_identity; LIKE wildcards are escaped; an id json.dumps would escape (breaking the verbatim-substring guarantee) falls the whole set back to the full scan rather than silently missing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): sink runtime mechanism docs below the gateway guidance budget Merging main pushed backend/app/gateway/AGENTS.md past its 40KB soft budget (main had left 81 bytes of headroom). Per the nearest-file rule, move the mechanism detail of the message-seq stamping and run-delivery receipt sections — both owned by runtime/ code — into packages/harness/deerflow/runtime/AGENTS.md, leaving the gateway file the REST-surface summary and a pointer. The seq section also documents the stamper's build-time soft user-id resolution and the db store's SQL prefilter from the review follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): sink durable-MCP task detail below the backend guidance budget Merging main pushed backend/AGENTS.md past its 24KB module soft budget (main itself is at 24762 after #4848 — this branch adds zero net bytes to the file). Per the nearest-file rule, move the two durable-MCP task runtime bullets' mechanism detail into packages/harness/deerflow/mcp/AGENTS.md, leaving summaries and pointers; this also restores ~2KB of headroom so the next merge does not trip the same wire. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(events): re-ask a message-seq miss once the feed advances The run-scoped stamper cached lookup misses for the whole run. A message this run produces reaches a values frame before RunJournal flushes it, so its first lookup legitimately misses — and the journal persists it moments later, giving it a feed seq the stamper never asks for again. A long run that afterwards rolls past the history page and compacts then carries that message unstamped, back to the approximate anchor placement this stamper exists to replace (#4666). A transient store error had the same permanent effect, since the except clause degrades to an empty result. A miss is now provisional while a hit stays final: RunJournal counts its successful event-store writes as `feed_generation`, and the stamper re-asks a missed identity only once that counter moves. Retrying is therefore bounded by feed writes rather than by frames — the per-frame query the run-scoped cache was built to avoid — and a failed lookup costs one generation instead of the run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-09-01 22:04:17 +08:00
Hyeonsang Cho	9146bfa03d	feature(gateway): issue request trace ids unconditionally (#5119) ... * refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-01 16:49:39 +08:00
Zeren Wang	a06a6fed7e	feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2) (#5109) ... * feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2) PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a subagent completes, so objectively checkable requirements can never be silently passed by a self-report. - subagents/acceptance_checks.py: deterministic leaf families — file:<path> exists|non-empty and file_written:<path> read through read_current_file_content scoped to the shared thread workspace; the read uses the sandbox-native virtual path form (the local read validator and provider mount tables resolve /mnt/user-data/... paths, not host paths); the scope decision canonicalizes with realpath on the local sandbox so workspace symlinks cannot escape into uploads; a remote provider's "Error: ..." return string is normalized to a failed check (provider-typed via is_local_sandbox); a UnicodeDecodeError marks a binary deliverable as existing and non-empty; out-of-scope paths degrade to UNVERIFIED. tests_passed:<command> anchors to a matching recorded bash execution with status=success and a test-summary shape; matching is shell-structure aware with control-flow attribution (span must end at the last segment with provable execution), negating-option values are ineligible evidence and a target negated anywhere in the command degrades the match, extra flags must be selection-preserving, extra positionals widen only after a path-scoped criterion, truncated commands degrade via command_truncated, the summary shape is read only from output attributable to the matched segment (preceding segments provably silent by invocation form), and pass shapes require a nonzero passed count. Criterion text is neutralized with neutralize_untrusted_tags before storage/rendering. Anything else renders UNVERIFIED, never silently passed. - executor: accumulate bounded bash command/output evidence per streamed chunk (merged by tool_call_id, newest-capped) so subagent summarization compacting earlier messages cannot erase a recorded execution; the recorded status is the actual shell exit status parsed from the output's exit marker (signed codes included; the remote Command exited with code N form is accepted only as the whole trimmed output), falling back to deerflow_tool_meta only when no marker exists. - sandbox providers: e2b/opensandbox/tenki/boxlite append the LocalSandbox-style "Exit Code: N" marker on nonzero exit even with non-empty output; aio propagates the SDK's structured exit_code on both exec paths the same way; local timeouts append Exit Code: 124; and _truncate_bash_output always preserves a trailing exit marker (signed included) inside its budget, with a 32-char floor raising any smaller configured limit, so the actual shell outcome always survives in the output text. - task_tool: run the checklist offloaded (asyncio.to_thread) on the completed branch, failure-isolated; stamp the verdict into result metadata and render the per-criterion section into the model-visible result text. - status contract: additive subagent_acceptance_verdict transport with read-side structural validation. - delegation ledger: entry carries the verdict and renders a compact acceptance segment; gateway strips caller-forged verdicts from both ledger entries and message metadata, like the citation verdict. - blocking-IO anchor pins the offload (teeth proven red->green); leaf read errors catch only OSError/SandboxError so unexpected errors reach the task-tool-level isolation instead of being mislabeled. * fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4) - negating options: overlap with a matched criterion target is now checked by path/nodeid prefix, not exact token equality — excluding a sub-path of the criterion's selection (pytest tests --deselect tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds - output attribution: any redirection token in the matched final segment makes the recorded tail non-attributable (> / >> / 2> are word characters to the parser, so redirection was invisible to the matcher) - silent-source allowlist narrowed from any *activate suffix to the */bin/activate shape - status_contract docstring: restore the shared-fixture sentence and note subagent_acceptance_verdict is deliberately outside the fixture - executor: update_bash_executions publishes [] (stream carried no bash-family calls) instead of collapsing it into None, mirroring update_tool_receipts * fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4) - tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error" and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass shape when the exit status is swallowed (|| true) or absent; zero-error counts stay clean. - file leaves: bound the deliverable read — a "wc -c" shell size probe answers files above 50k bytes without loading ~2x their size, honoring the host-bash kill switch and falling back to the full read on any non-integer rendering, so verdicts never get less sound. - executor: record the exit marker text as status_marker on harvested bash evidence; the leaf detail now reports the marker actually seen instead of asserting a failure indistinguishable from the command's own trailing text. - extend the blocking-IO anchor to drive the probe branch inside the offload; teeth re-verified red->green. * fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4) - file leaves: never read unbounded — size is established first (os.stat on the validated local host path, so the host-bash-disabled configuration needs no shell; a guarded wc -c on remote providers that renders missing/unreadable in its own words). Above the 50k cap the leaf answers from the size alone, at/below it the full read runs, and an unestablishable size degrades to UNVERIFIED instead of an unlimited fallback read. - output attribution: source/. prefixes are never provably silent — a crafted */bin/activate path shape says nothing about what the script prints, so sourced segments can no longer lend a passing summary. - executable identity: an explicitly path-spelled criterion now requires the same normalized executable path; the basename rule stays only for deliberately bare criterion commands. * fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4) - remote probe no longer runs in the sandbox's persistent shell: a fresh env -i /bin/sh with absolute-path stat/realpath (poisoned functions, aliases, PATH, exported functions, IFS, locale cannot steer it), plus a marker env routing AIO onto a fresh per-call bash.exec session. - metadata-only: stat never opens content, so a FIFO deliverable cannot block the parent for the provider's idle timeout; non-regular files (fifo/dir/symlink) degrade to UNVERIFIED. - containment canonicalized against the literal mount root: a final-component symlink or a swapped parent directory (root included) cannot redirect the check outside shared storage; unprovable layouts degrade to UNVERIFIED. * fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4) Literal-root equality made every remote file leaf permanently UNVERIFIED on e2b and Tenki, which realize /mnt/user-data as a symlink to the home dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink). Containment now compares the file's realpath against the mount root's realpath — exactly what the provider's own read path resolves, so probe and read-back stay consistent; final-component symlinks stay rejected by the non-dereferencing stat, and an intermediate dir-link escape under a sane root still lands ESCAPED. The inner script is a module constant and the suite now executes the composed probe for real against on-disk layouts (real dir, symlinked prefix, final symlink, fifo, missing, dir-link escape), which the canned-output stub could not see. * fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4) - matching: a criterion with no positional selection target (bare pytest, make test) stands for the runner's default selection, so ANY negating option (--ignore/--deselect/...) makes the recorded run a different selection — unprovable. The overlap guard only sees consumed criterion tokens, which a bare criterion does not have; scoped criteria keep the unrelated-exclusion behavior. - attribution: cd is no longer blanket-silent — CDPATH makes cd print the resolved (subagent-chosen) destination and the pass shapes match as substrings, so one mkdir 'all tests passed' plus an export minted a pass for any quiet command. A cd argument or CDPATH= value (export or leading assignment) carrying any summary shape makes the segment non-silent; shape-free cd dir wrappers keep matching. - docs: _truncate_bash_output states the effective 32-char floor (the guarantee previously read as an unconditional max_chars bound). * fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4) Self-audit in the shape of the last review rounds — channels the matcher classified as accounted-for that can change what runs, narrow the selection, or lend the summary text: - env assignments are no longer blanket-stripped: only an allowlist of inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a matched span, and a non-allowlisted assignment in any preceding segment (pure-assignment or export NAME=) is state pollution — PATH redirects the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code, PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs, BASH_ENV runs arbitrary shell startup. All degrade to unprovable. - runtime expansions: any span token carrying /$( )/backticks, any negating-option value carrying an expansion or glob (unknown excluded set), and any extra executed token carrying glob metacharacters (crafted option-looking filenames narrow invisibly) are unprovable. Criterion-side globs stay self-consistent (literal match). - cd: an argument carrying a runtime expansion or glob is non-silent (unknown destination, unknown print); CDPATH= assignments are now handled as state pollution at the match layer, subsuming the value-shape special case. * fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4) - tests_passed: on a persistent-shell provider (new Sandbox.persistent_shell_sessions capability, set by AioSandbox) every leaf degrades to UNVERIFIED — any earlier call in the shared session could have mutated the state the clean-looking run executed in, and only a fresh controlled session (RFC section 6 verifier) can prove otherwise. The flag is read from the provider registry without acquiring a sandbox. - env assignments: the allowlist is gone — no variable is provably inert across repositories (CI/DEBUG are routinely read by tests). The span's assignment prefix must equal the criterion's exactly (values included, order-insensitive); any assignment or export NAME= in a preceding segment is state pollution. - scoping: positional targets are now read by option arity, so a path embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never counts as a selection target and an extra positional after such a criterion narrows the default selection it denotes. * fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4) * fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4) * fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4) * fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4) * fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4) * fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4) * fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4) * fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4) * fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4)	2026-09-01 16:13:41 +08:00
Janlay	45adb8fbb5	perf(runtime): bound gateway memory after terminal runs (#5112) ... * fix(runtime): clean up terminal run records * perf(sandbox): bound local path caches * perf(runtime): release terminal run cycles * fix(runtime): address terminal cleanup review * fix(runtime): clean up after end publish failure * fix(runtime): guard terminal cleanup from cancellation * fix(runtime): discard fenced journal buffers * fix(runtime): harden abort and teardown paths	2026-09-01 10:56:43 +08:00
Zeren Wang	3b592c2053	feat(harness): subagent receipt citation verification (#5076) ... * feat(harness): subagent receipt citation verification - add receipt citation verification core - harvest subagent tool receipts at terminal status - transport subagent receipts and citation verdict via status contract - verify subagent report citations at task write-back - render citation verdicts in the delegation ledger * fix(gateway): strip forged receipt verdicts from the delegations channel normalize_input() and the checkpoint-state mutation sanitizer only stripped server-owned metadata from message-shaped values, so an external caller could submit a delegation entry carrying a forged receipt_verdict that render_delegation_ledger would present as runtime-owned execution evidence. Strip receipt_verdict from caller-supplied delegation entries on both the run-creation and thread-state mutation paths, with regression coverage for each. * fix(harness): close silent-pass gaps in the zero-citation heuristic The action-claim detector missed the most common completion verbs (fixed/added/tested/changed/...) and had no CJK coverage at all, so reports like 'I fixed the bug and added tests.' or '我已经创建了文件并运行了测试。' were treated as claim-free: citation_resolved=True with no ledger warning. Broaden the verb lists and add a language-independent safety net: when the run harvested receipts and a nontrivial (>=240 char) report cites none of them, flag it UNVERIFIED. Short claim-free confirmations remain a vacuous pass.	2026-08-29 16:56:56 +08:00
Aari	645ca08f16	fix(scheduler): enqueue busy scheduled task runs (#4918) ... * fix(frontend): clarify reuse-thread scheduling behavior * fix(scheduler): enqueue overlapping scheduled runs * fix(scheduler): preserve queue lease fencing * fix(scheduler): close queue concurrency races * fix(scheduler): harden queue timeout bookkeeping * fix(scheduler): preserve manual failure schedule --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-24 15:38:32 +08:00
Aleksandr Sapronov	613b90b0e6	feat(scheduler): make scheduled-run recursion_limit configurable (#4848) ... Adds scheduler.recursion_limit to config.yaml (default 1000, clamped by max_recursion_limit) so scheduled background runs can use a different recursion limit than the web UI. The value is read at dispatch time, so a YAML edit applies to the next scheduled run without a Gateway restart. Also logs a warning when the resolver falls back to the default or clamps the configured value.	2026-08-24 07:09:08 +08:00
Zeren Wang	4e35f0d1d4	feat(harness): deterministic tool receipts with model-visible ledger (RFC #4651, layer 1) (#4659) ... * feat(harness): add deterministic tool receipts with model-visible ledger Stamp an immutable per-call fact record (tool name, status, args/output hashes, byte count, timestamp) onto every tool result via a new ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN) into the model context so subagent reports can cite executed actions. - tool_receipt.py: receipt core (make/extract/render), newest-first budget eviction, ids derived from the append-only message stream - ToolReceiptMiddleware: stamps ToolMessages directly or inside Command-wrapped results; hidden ledger injection mirrors DurableContextMiddleware; sits between ToolProgress and ToolErrorHandling with a build-time ordering guard - config: new verification section (receipts on, judge off), config version 32 -> 33 with example/helm/docs updates * feat(harness): split receipt rendering from stamping; address PR review Review fixes (PR #4659): - output_sha256 now uses sort_keys=True for structured content, matching the order-invariant args fingerprint - stamping failures log at warning (silent ledger gaps would corrupt citations); tool execution remains never blocked - _insert_after_leading_system_messages extracted to shared public message_utils.insert_after_leading_system_messages; both middlewares depend on it instead of a private cross-module helper - code comments in English RFC #4651 revision-2 alignment: - receipts_render_mode config ('always' | 'delegation_only'): subagent chains always render the ledger (citations are produced there); the lead chain renders only while processing subagent results, removing the always-on token tax from ordinary turns - receipts gain bounded args_preview/output_preview (<=200 chars, tail for output) so later typed claim bindings (tests_passed) can anchor to a specific recorded execution * docs(harness): state receipt freshness caveat and vocabulary layering in module docstring * merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews - backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the verification.* hot-reload mention into config/AGENTS.md - config.example.yaml + helm values/README: config_version 33 -> 34 so existing v33 configs get the outdated-config prompt (review: willem-bd) - tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads them; re-add with the Layer 2 claim-binding consumer (review: willem-bd) * docs(harness): cover receipt id renumbering after compaction in module docstring Positional display ids are stable only while history is append-only; compaction drops ToolMessages and the survivors renumber, so Layer 2 citation verification must resolve [rN] against the ledger as of the citing turn (review: willem-bd, doc-only). * chore(config): bump config_version to 35 main reached 34 via #4780 without the verification section; publishing the new schema at the same number would silently skip the outdated-config prompt for configs synced from main in that window (review: willem-bd). * fix(skills): restore errno import dropped upstream in #4830 upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed the import, so the PR merge ref fails lint-backend (F821). * fix(harness): harden tool receipts against forgery and turn-scope delegation_only Address willem-bd's pre-merge review on #4659: 1. Untrusted receipt metadata: the gateway now strips the server-owned deerflow_tool_receipt key from external input messages; stamping always overwrites any tool-supplied value instead of preserving it; and extract_tool_receipts validates persisted receipt shapes (required typed fields, unknown keys ignored) so malformed entries are skipped instead of crashing render or passing as runtime-stamped evidence. 2. delegation_only no longer sticks on: _should_render now scopes the subagent_status scan to the current turn (messages after the latest genuine user message), so an old completed delegation stops rendering the ledger on later ordinary turns. The genuine-user predicate moves to message_utils.is_genuine_user_message, shared with input sanitization. * fix(harness): stamp receipts outside short-circuiting tool middlewares Address willem-bd's review on #4659: ToolReceiptMiddleware was registered inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which can return a ToolMessage without invoking its handler — blocked calls (e.g. a read-before-write-denied write_file) never got a receipt, silently gapping the ledger on a default-enabled path. SandboxAudit additionally rebuilds medium-risk results, dropping an inner stamp. ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the runtime tail. Normal results still carry deerflow_tool_meta (stamped by ToolErrorHandling on the inner return path); short-circuit messages self-stamp meta or fall back to message.status. The new invariant is declared as ordering constraints in deerflow.extensions.ordering, with composed-chain regression tests for a blocked write and a warn-rebuilt bash result.	2026-08-23 15:43:37 +08:00
ajayr	7e95bef2e7	feat(mcp): per-user credential injection for shared MCP servers (#4868) ... * feat(mcp): per-user credential injection for shared MCP servers A single HTTP/SSE MCP server entry can now serve several users, each authenticated to the remote service with their own credential. A server opts in with a user_auth block mapping user ids to credential header values ($ENV_VAR references supported): "user_auth": { "header": "Authorization", "users": { "<user-id>": "$SERVICE_TOKEN_ALICE" } } The built-in user-scoped auth interceptor resolves the authenticated runtime user on every tool call (request runtime -> ambient LangGraph runtime -> auth config -> request-scoped user ContextVar) and rewrites the configured header via request.override(), the same per-call mechanism as the OAuth interceptor. It registers after OAuth in the shared assembly so its per-user value wins the header when a server declares both. The entry's static headers are used only for startup tool discovery. Fail-closed by default: an unmapped user - including the anonymous default-user fallback - or a credential whose env reference resolved empty gets an actionable ToolException instead of another user's credential; on_missing: "passthrough" opts out per server. Combined with the existing per-(user, thread) MCP session scoping this gives credential isolation on shared servers. Gateway API: user_auth.users values are masked in GET responses, and PUT round-trips preserve stored credentials for masked values (same contract as env/headers/oauth secrets); a masked value for a user id not already stored is rejected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): address review — preserve stored user_auth sub-fields on partial PUT, warn-and-skip stdio, allow extras Review findings from #4868: 1. A partial user_auth payload (e.g. {"enabled": false}) merged to users={} and default on_missing, irreversibly wiping stored credentials on PUT. The merge is now sub-field-aware via model_fields_set — omitted sub-fields carry the stored values, an explicitly sent users map still replaces (so full-round-trip removal works), masked values still swap back for stored credentials. 2. user_auth on a stdio server was a silent no-op: rewritten headers go to call meta, never a transport header, while deny errors still fired. The interceptor builder now warns and skips non-sse/http servers, matching the tool_call_timeout transport-mismatch convention. 3. McpUserScopedAuthConfigResponse now allows extra keys like the harness-side model, and extras survive masking and merge, matching the server-level model_extra handling. Adds four regression tests (partial-PUT preservation, explicit-map replacement, extras round-trip, stdio warn-and-skip). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): reject blank user_auth.header at the gateway A blank header passed the gateway response model, was persisted, then failed the harness-side ExtensionsConfig validator on reload — the PUT returned 500 after the write and every later config load/startup failed until the file was hand-edited. Mirror the harness non-blank validator on McpUserScopedAuthConfigResponse so the PUT fails with 422 before anything is written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: ruff format extensions_config.py Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): harden the trust chain and masking around user-scoped credential selection Address review round 4: - Scrub client-supplied user_id from run context/configurable for external callers in inject_authenticated_user_context, before every early return, and restamp only from request.state.user. Now that user_id selects which user's credential user-scoped MCP auth injects, a forged value must not survive any future path that skips the restamp. Internal callers (IM channels, scheduler) keep supplying end-user identity as before (PR #3294 contract). Regression tests pin both the scrub and that a forged body.context.user_id can never resolve as another user through merge + inject ordering. - Include the caller's own resolved user id in the fail-closed deny message so operators can copy the exact users key (it differs by deployment path), and document the key formats in the mcp.mdx doc. - Mask sensitive extra keys inside user_auth on GET like server-level extras, and swap masked sentinels back for stored values on PUT via _merge_extra_value_preserving_masked. - Extract the interceptor wrap loop into compose_tool_interceptors and pin the security property functionally: an OAuth interceptor that actually sets Authorization loses the final header to the per-user credential through the same composition the session-pool path uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-23 15:16:04 +08:00
Nan Gao	13f0a7f263	feat(extensions): let an out-of-tree extension observe what the agent did (#4863) ... * feat(extensions): let an out-of-tree extension observe what the agent did DeerFlow's extension system can contribute middleware, services and routes, but an extension cannot answer basic questions about a run without reaching into host internals. Several of the facts it would need are destroyed by the operations that produce them: * The middleware chain injects and rewrites a lot of context — date reminders, recalled memory, compaction summaries, durable-context data, image payloads, activated skill bodies. Downstream, none of it is attributable: at the model-call boundary an injected HumanMessage is indistinguishable from the user's own, and anything wanting to tell them apart has to pattern-match prompt wording, which breaks on the next copy edit. * Two runs of "the same agent" are only comparable if the chain enforced the same limits, prompts and thresholds. Recovering that from outside means reading private attributes and guessing which of them change behaviour — a guess that rots silently as middlewares gain fields. * The lead-agent factory resolves a model after runtime overrides, renders a prompt, filters tools through authorization and composes a stack, all inside one synchronous call, and none of it survives: a middleware sees its neighbours but not the prompt, the run worker sees a graph but not what went into it. * Summarization is destructive by design. N messages leave the context and one summary enters it; afterwards only the summary exists, so "which messages became this?" is not reconstructible. This adds seven neutral facilities so those facts are recorded where they are still true, and releases the contract package as 0.2.0. Message provenance Producers stamp `deerflow_content_kind` / `deerflow_producer_kind` onto the messages they inject or rewrite. Stamping is unconditional — a fact whose presence depends on whether an observer is installed is not a fact — and the keys are server-owned, so provenance cannot be forged from a request. Middleware self-description Twelve middlewares declare their own behaviour-affecting parameters through a duck-typed `release_policy_parameters()`. Long text is hashed rather than embedded: a declaration is an identity, not a copy of the prompt. Agent assembly descriptor `assemble_lead_agent()` returns the graph plus a descriptor whose fingerprint answers "did anything about this agent change between these two runs?". `make_lead_agent()` keeps its graph-only signature — it is the LangGraph Server ABI declared in langgraph.json. Tools and skills are sorted before hashing because their assembly order is incidental; middlewares are not, because stack order decides what wraps what. Host build identity is reported but excluded from the fingerprint, so a redeploy does not invalidate every agent's identity. Context compaction observation Summarization emits the content hashes of the messages it is about to remove joined to the summary that replaced them. Content is the only identity available at that seam: the summary does not become a message, and what later projects it into a request renders it bounded and escaped rather than verbatim. Neutral policy, transform and MCP-source facts Guardrail decisions are published to runtime context under a `__`-prefixed key; result-rewriting middlewares append a declared, ordered transform trail; MCP tools carry their credential-free logical origin. Extension route identity Contributed routes are session-authenticated and cannot opt out, but "logged in" and "administrator" are different questions. Extensions get a neutral projection of the caller rather than the host's auth context, and `require_admin` fails closed when identity cannot be determined. Extension-owned tables An extension that persists data owns its own MetaData and migration chain, so its tables are absent from Base.metadata and `alembic revision --autogenerate` proposes dropping them. Extensions declare a table prefix, which is rejected at registration if it would shadow a host table. The contract package stays dependency-free and imports no host code; every new Protocol method has a default so later additions remain additive. The loader's pre-1.0 rule requires an exact major.minor match, so extensions written against 0.1 are now refused at startup with an actionable install hint rather than loading into a host that implements a different surface. uv.lock records the contract package's new version, so `uv sync --locked` still resolves on a fresh checkout. * fix(backend): sort gateway service imports	2026-08-23 09:57:12 +08:00
Aari	5ffc2d3e27	feat(mcp): complete durable task notifications and chat UI (#4833) ... * feat(mcp): add reliable task notifications and cancellation * feat(mcp): add background task chat UI * fix(mcp): hide and sanitize task notification prompts * fix(mcp): sanitize projected task names * fix(mcp): harden task notifications and details * fix(mcp): harden task lifecycle recovery * fix(mcp): gate task UI and isolate cancellations * test: scope plain-text response locator * fix(mcp): align task notification boundaries * fix(mcp): bound task delivery retries * fix background task notification races	2026-08-22 16:53:32 +08:00
Vanzeren	095092418c	fix(gateway):unify thread id validation (#4589) ... * fix(gateway): unify thread ID validation at the API boundary Thread ID entry points accepted arbitrary strings while downstream consumers (filesystem paths, Kubernetes Provisioner, JSONL event store) each enforced different character restrictions, so invalid IDs were persisted first and only failed later during sandbox/workspace init. Centralize validation in deerflow.utils.thread_id (pattern ^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest, scheduler dispatch, paths.py, JSONL store, embedded client, and align the Provisioner pattern (pinned by a parity test). UUIDs are still generated only when no ID is supplied; caller-supplied opaque IDs stay supported. Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy cleanup escape hatch (filesystem cleanup guarded), read-only client.get_thread stays unvalidated, and scheduler rows with legacy invalid IDs record a failed dispatch instead of raising out of the poll loop. * docs: document canonical thread ID contract README: caller-supplied thread IDs need not be UUIDs; the canonical pattern and per-endpoint behavior. AGENTS.md: the shared deerflow.utils.thread_id contract, its enforcement boundaries, and the legacy-ID escape hatches. * fix(gateway): close thread ID validation gaps at remaining entry points Follow-up to the canonical thread ID contract: a full audit found the uniform-422 coverage only reached about half of the thread_id surfaces. - routers: 18 routes still took a bare thread_id: str — 13 in thread_runs.py (including the five messages/events/workspace-changes reads that returned 500 on the JSONL event store vs 404/empty on the DB store), 4 read routes in threads.py, and the suggestions route flagged in review. DELETE /api/threads/{id} keeps str as the declared legacy-cleanup escape hatch. - client: upload_files/delete_upload/list_uploads/get_artifact now validate up front, fulfilling the RFC's 'all mutating entry points' clause (get_thread stays unvalidated as the declared legacy read path). - tui: the /resume literal-ref fallback validates against the canonical contract and reports a descriptive error instead of failing deep in the client. - scripts/support_bundle.py: replace the drifted dot-allowing pattern with a byte-identical copy of THREAD_ID_PATTERN (kept local so the script still runs with a broken venv). * test(gateway): guard the canonical thread ID contract against regressions - test_thread_id_route_contract.py: static AST sweep asserting every route handler with a thread_id parameter annotates ThreadId (whitelist: the DELETE escape hatch), plus a runtime sweep hitting all 44 thread_id routes with a non-canonical ID and asserting a 422 that names thread_id, plus a websocket upgrade-rejection case. - test_thread_id_validation.py: client entry-point validation, support_bundle pattern parity, and TUI literal-ref fallback tests. - Align two tests that encoded the old contract (dotted IDs).	2026-08-01 19:42:44 +08:00
Vanzeren	80848837b7	feat(gateway): seed checkpoint history (#4590) ... * feat(runtime): seed empty run-event feed from checkpoint history Threads created before the journaled run-event model hold their history only in the LangGraph checkpoint. Before the first journaled run, backfill an empty run-event message feed from the existing checkpoint head so legacy history receives earlier thread-global seq numbers and remains visible in the unified feed. Threads with no checkpoint or an already populated feed skip the path. The seed guard resolves the user explicitly instead of relying on the store's AUTO default, which raises without a user contextvar (scheduler launch path on the DB event store). * docs: document checkpoint history seeding in thread runs Before the first journaled run, an empty run-event message feed is seeded from an existing checkpoint head so legacy checkpoint-only history stays visible with earlier thread-global sequence numbers. * fix(gateway): make checkpoint-history seed guard thread-scoped The emptiness guard filtered by the current user whenever one was in context, answering "does this user have any messages?" rather than "has this thread's feed ever been journaled?". Seed rows stamped with a different principal (NULL for ownerless seeds, or another user on a shared NULL-owner thread) were invisible to the guard, so each new principal re-seeded a duplicate history. Pass user_id=None unconditionally; None also opts out of AUTO resolution, so the ownerless scheduler path still cannot raise. Adds a DbRunEventStore-backed regression test (the MemoryRunEventStore tests cannot catch this — the memory store ignores user_id) proving the ownerless-seed -> authenticated-run sequence seeds exactly once.	2026-07-31 17:25:07 +08:00
Vanzeren	c48de5e70b	feat(checkpoint): make delta snapshot_frequency configurable (#4516) ... * feat(checkpoint): make delta snapshot_frequency configurable * fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning Addresses review on #4516: the rename from the flat database.checkpoint_delta_snapshot_frequency key to nested database.checkpoint_delta.snapshot_frequency silently dropped the old value (pydantic extra="ignore"). Add a before-validator that maps the legacy key onto the nested one with a deprecation warning (nested key wins when both are set), plus a CHANGELOG breaking-change note covering the rename and the 1000 -> 10 default change. * fix(checkpoint): validate frozen snapshot frequency	2026-07-28 23:21:23 +08:00
阿泽	ea74367502	fix(runtime): honor LangGraph Server identity for user-scoped data (#4538) ... * fix(runtime): honor LangGraph Server identity for user-scoped data * fix(runtime): scope custom agent SOUL by resolved user	2026-07-28 22:59:14 +08:00
MiaoRuidx	8a78c264b7	fix(runtime): cancel runs across live gateway workers (#4500) ... * docs(runtime): design cross-worker cancellation * fix(runtime): cancel runs across gateway workers * fix(runtime): harden cross-worker cancellation races 让取消请求与 owner 终态写入通过持久化 CAS 决定先后，保证首次取消 action 在不同 worker 路由下保持一致。\n\n将 heartbeat 收敛为续租后仅发送本地中止信号，并补齐完成竞态与路由重试的回归用例。 * docs(runtime): drop implementation plan from PR 移除仅用于实现过程的跨 worker 取消设计记录，保留 README 和 backend/AGENTS.md 中面向最终行为的文档。 * fix(runtime): preserve local cancel fallback * test(runtime): adapt worker run manager fakes * docs(runtime): fix run cancel migration registry --------- Co-authored-by: MiaoRuidx <12540796+MiaoRuidx@users.noreply.github.com>	2026-07-28 21:45:20 +08:00
ShitK	b1984cf4ab	fix(security): reject legacy MCP credentials in run metadata (#4448) ... * docs: design run metadata secret admission * docs: refine run metadata secret boundaries * docs: plan run metadata secret fix * fix(security): centralize legacy run metadata policy * fix(security): reject secrets at run admission * fix(security): hide legacy secrets from history APIs * docs(security): migrate MCP credentials to secret context * fix(security): redact legacy runnable config metadata * fix(security): reject legacy config metadata credentials * fix(security): hide legacy secrets from run kwargs * docs(security): clarify config redaction boundary * docs: keep issue 4416 planning local	2026-07-28 21:31:23 +08:00
Huixin615	1cd5dea336	fix(streaming): signal replay history gaps (#4426) ... * fix(streaming): signal replay history gaps * fix(streaming): guard initial Redis replay window * fix(frontend): align inactive gap recovery --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-27 07:13:06 +08:00
MiaoRuidx	735f67a5b2	fix: guard pending run startup cancellation (#4450) ... * fix: guard pending run startup cancellation * fix(run): address startup review feedback * fix(run): narrow start_run store contract --------- Co-authored-by: MiaoRuidx <12540796+MiaoRuidx@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-25 23:50:21 +08:00
ShitK	a4ede80deb	fix(runtime): reject unsupported run options and stream modes (#4430) ... * fix(runtime): reject unsupported run options * fix(runtime): align SDK run compatibility * fix(frontend): avoid unsupported events stream mode --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-24 19:24:24 +08:00
March7	ce4a6d4c3d	fix(backend): remove transient image context after model calls (#4267) ... * fix(backend): discard transient image context * fix(backend): protect client image context ids * docs(backend): clarify image checkpoint lifecycle	2026-07-22 08:41:50 +08:00
Vanzeren	42baed8c8c	feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel (#4292) ... * feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel Add a restart-required database.checkpoint_channel_mode ("full" default, "delta") that stores the messages channel via LangGraph 1.2 DeltaChannel, cutting checkpoint storage from O(n^2) to O(n) for append-only history. Existing full checkpoints seed delta state transparently; no data migration. - config: mode schema + freeze-on-first-use with CheckpointModeReconfigurationError; mode marker persisted in checkpoint metadata; unsafe delta->full downgrade rejected fail-closed with CheckpointModeMismatchError (run-level error, failed state read) - state: delta message state schema; CheckpointStateAccessor centralizes materialized reads for all consumers (threads API, branches, regeneration, compaction, state updates, memory, goal workers) - runtime: raw writers (run durations, interrupted title, thread goal) parent their checkpoints to the checkpoint they derive from, preserving delta ancestry; rollback forks the pre-run lineage through a state mutation graph with Overwrite restores; InMemorySaver delta-history override delegates to the base walk (fixes dropped first write after migration, also present upstream) - tests: conformance suite over {memory, sqlite, postgres} covering migration replay, stable message IDs, storage shape and writer preservation; conftest fixture isolates the frozen mode between tests; stale config fakes refreshed - ci: backend unit tests gain a postgres service * fix(checkpoint): close materialization gaps in goal flow, guard public factory - Route goal-continuation message reads through CheckpointStateAccessor: raw channel_values reads see the delta sentinel in delta mode, which disabled goal continuation (stand_down=no_durable_end_of_turn) after durable assistant turns. Raw tuples remain for tuple-only metadata (checkpoint id, pending_writes). - Reject checkpoint_channel_mode='delta' + checkpointer in create_deerflow_agent at construction: factory-built persisted graphs bypass mode-marker injection and the fail-closed gate, reproducing silent mixed-mode state loss. Delta without persistence stays allowed. - Import the postgres saver lazily (pytest.importorskip in the fixture) so the documented default install collects the suite; add a CI job running pytest --collect-only on uv sync --group dev without extras. - Fix test_checkpointer fallback test to patch get_app_config at its use site (provider module), making it deterministic when a local config.yaml selects a persistent backend. * fix(gateway): preserve extension-owned channels in state mutations, bump config version - build_state_mutation_graph / build_checkpoint_state_mutation_accessor accept an explicit state_schema; branch and POST /state now compile the mutation graph from the thread's effective schema (graph_state_schema on the assistant graph). The base-ThreadState fallback silently discarded channels contributed by custom AgentMiddleware.state_schema on branch (data loss) and returned a false-success 200 on POST /state. - POST /state validates values keys against the mutation graph's channels and rejects unknown fields with 422 instead of ignoring them; reducer detection covers extension channels (BinaryOperatorAggregate or DeltaChannel) so Overwrite replace semantics work for middleware reducers in both modes. - Endpoint regression: custom AgentMiddleware.state_schema value survives branch, updates through POST /state, and an unknown field receives 422. - config_version 26 -> 27 for the new database.checkpoint_channel_mode (example, Helm chart values + README, support-bundle fixture), so existing installs get the outdated-config warning and make config-upgrade merges the field; covered by a test driving the real example file and the real config-upgrade script. * fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite GET /threads/{id}/state now resolves the thread's assistant_id through a single reusable boundary (thread metadata -> assistant_id -> effective graph), so channels contributed by a custom AgentMiddleware.state_schema are materialized instead of dropped by the default lead schema. POST /state uses the same boundary instead of resolving the schema ad hoc. Branch writes wrap every copied reducer channel in Overwrite (derived from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel), not just messages, so already-aggregated values are never re-merged. Regression tests use a real AgentMiddleware.state_schema with a non-identity reducer in both full and delta modes: GET /state returns the extension value, POST /state replaces it, branch preserves it byte-for-byte; the unknown-field 422 is a separate assertion. * refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests Address review round 4 on PR #4292: - Push ahistory/history limit through Pregel into checkpointer.alist (SQL LIMIT) instead of materializing all rows and breaking in Python - Fold the read-side mode-compat gate onto the returned snapshot's metadata; only writes keep the pre-write tuple fetch (fail-closed) - Cache factory-built accessor graphs per (assistant_id, mode) with factory-identity revalidation; state reads no longer build a lead agent per request - get_thread: one snapshot fetch + one raw pending_writes fetch on the resolved checkpoint (post-checkpoint __error__ writes never surface in snapshot.tasks; verified empirically) - DeerFlowClient.get_thread: single checkpointer.list walk collects pending_writes per checkpoint instead of N get_tuple calls - InMemorySaver delta-history patch: stand-down when the upstream override disappears, try/except guard, validated-version warning, guard tests - make_lead_agent mode precedence: first freeze is owned by app_config (client-supplied configurable key ignored); once frozen, injected key/app_config must match or fail closed - Rollback: lock in non-message channel restoration via fork inheritance with a dedicated reducer-channel test - Add tests/test_threads_checkpoint_mode.py and tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and the PR validation section: lifecycle parity (memory + sqlite), per-step blob-count storage guard, gateway endpoint parity Counted-saver tests pin checkpoint round-trips for aget/ahistory so these regressions cannot silently return. * fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience - threads router: map CheckpointModeMismatchError to 409 (with cause and thread id) and CheckpointModeReconfigurationError to 503 across all state endpoints instead of swallowing both into a generic 500 - gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary through the real route stack - rollback: compile the restore mutation graph with the thread's effective state schema per the build_state_mutation_graph contract - inheritance contract locks: rollback and manual compaction preserve middleware-contributed channels via checkpoint fork cloning - services: revalidate the accessor-graph cache against app_config identity so config.yaml hot-reloads never serve a stale compiled graph - services: degrade full-mode state reads to raw checkpointer reads when the agent factory is unavailable (delta gate still applies; delta mode has no fallback) - deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility) and bump the langchain lower bound to what the lockfile actually resolves * fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread - _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while pregel's get_state_history treats config.checkpoint_id as the inclusive start; fetch the anchor explicitly so both read paths paginate identically - extend the degraded-path gateway test: GET /thread returns raw values, and POST /history with before starts at the anchor checkpoint * fix(gateway): preserve degraded checkpoint timestamps * fix(gateway): harden degraded checkpoint access * fix(gateway): resolve assistants for checkpoint reads --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-22 08:33:29 +08:00
hataa	10890e10a8	feat(authz): propagate trusted authorization principal context (#4203)	2026-07-17 14:49:51 +08:00
AochenShen99	94a34f382d	feat(context): record effective memory identity per run (#3556) ... * feat(context): record effective memory identity per run * fix(context): address memory identity review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-16 09:39:09 +08:00
AnoobFeng	446fa03801	fix(context): resolve context compress bug (#4065) ... * fix(runtime): persist original human input outside model sanitization * refactor(history): load thread messages by global event sequence * fix(frontend): make summarization rescue a transient history bridge * fix(frontend): old message not append tail 1. add identity anchor 2. add bridgeOrder * fix(frontend): lint error fix * fix: address review feedback and harden pagination coverage - defer transient history ref writes until after render commit - cover large middleware-only history scans - verify infinite-query refetch recalculates page cursors - document AI event types and anchor-weaving differences * fix: harden message pagination and enrichment - append unmatched live tails after canonical history - warn and stop when pagination has_more lacks a cursor - deep-copy restored UI messages to isolate model-facing content - log invalid event sequence and non-advancing cursor errors - pass user_id explicitly through event-store history queries - cover middleware-only AI runs across memory, JSONL, and DB stores * fix: address pagination review feedback * fix(frontend): checkpoint has unknow redener content, optimize the anchor policy * fix(frontend): unit test issue missed previously, remove the TanStack cache trimming * fix(gateway): harden message history queries and provenance - reject externally forged original_user_content metadata - validate provenance metadata in upload and sanitization middleware - make run lookups fail closed by default - batch feedback queries by run ID - align memory message filtering with persistent stores	2026-07-14 10:43:13 +08:00
DanielWalnut	4e209827f3	feat(agent): Add subagent total delegation cap (#4115) ... * fix subagent total delegation cap * fix embedded subagent run cap context * fix subagent cap config consistency * fix resumed subagent run cap boundary * fix legacy resume subagent boundary * address subagent cap review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-13 19:26:07 +08:00
sqsge	927b833ef4	fix(guardrails): propagate internal owner attribution to guardrail context (#3839) ... * fix guardrail attribution for internal owner runs * fix guardrail owner attribution mismatch	2026-07-07 08:05:26 +08:00
AnoobFeng	47b0f604f4	feat(frontend):enhance the ask_clarification interaction with visualized card (#3956) ... * feat(frontend): add structured human input cards for ask_clarification Implement a reusable Human Input Card flow for ask_clarification while keeping the existing text fallback for older clients and IM channels. Backend: - Add structured ToolMessage.artifact.human_input payloads for clarification requests. - Preserve ToolMessage.content as the readable Markdown/text fallback. - Normalize clarification options from native lists, JSON strings, plain strings, mixed scalar values, None, and missing options. - Derive input_mode as choice_with_other when options exist, otherwise free_text. - Keep disable_clarification non-interactive behavior as a plain ToolMessage with no human_input artifact. - Cover artifact persistence and Gateway message metadata preservation in tests. Frontend: - Add human input protocol types, runtime guards, extractors, response builders, and thread-state helpers. - Add reusable HumanInputCard with option buttons, free-text input, pending, read-only, disabled, and answered states. - Render structured clarification cards from artifact.human_input, with Markdown fallback for malformed or legacy tool messages. - Preserve line breaks in structured question/context/option text. - Hide submitted clarification bridge messages from the chat UI via additional_kwargs.hide_from_ui. - Send structured human_input_response metadata through the fourth sendMessage options argument, preserving run context in the third argument. - Wire submissions for normal chats, custom agent chats, agent bootstrap chats, and sidecar chats. - Derive answered state from raw thread.messages so hidden replies still update the original card. - Clear pending state when the hidden reply arrives, dispatch is dropped, or a later async stream failure appears on thread.error. * perf(frontend): optimize HumanInputCard UI interactions - Support Enter key to submit text input (Shift+Enter for newline) - Render question and context fields as Markdown instead of plain text - Replace deprecated FormEventHandler type with structural typing * test(frontend): add unit test cover optimize HumanInputCard UI interactions * feat(frontend): disabled chatbox when has new human-input-card * fix(style): lint error fix * fix: sanitize hidden human input replies - Preserve IME composition safety for human input card Enter submits - Treat hidden human input responses as genuine user messages for sanitization - Keep hidden card replies in memory filtering while excluding malformed/internal hidden messages - Add regression coverage for card IME handling and hidden reply sanitization * fix: tighten human input response validation - Reject empty hidden human input response values - Remove invalid list ARIA role from human input card options - Add backend coverage for empty response payloads --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-06 22:34:41 +08:00
Zheng Feng	dcb2e687d5	feat(channels): add GitHub as a webhook-driven channel (#3754) ... * feat(channels): add GitHub event-driven agents (#3754) Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation. * fix(llm-middleware): classify bare IndexError as transient Upstream chat providers occasionally return 200 OK with an empty generations list (observed against Volces "coding" on ark.cn-beijing.volces.com). When that happens, langchain_core.language_models.chat_models.ainvoke raises ``IndexError: list index out of range`` at ``llm_result.generations[0][0].message`` and kills the run. Treat a bare IndexError reaching the middleware as a transient upstream-payload glitch and route it through the existing retry/backoff path instead of failing the whole agent run. The retry budget and backoff schedule are unchanged. Adds three regression tests covering the classifier and both the recover-on-retry and exhausted-retries paths. * fix(runtime): ignore stale LLM fallback markers from prior runs When a run on a thread ends with the LLM-error-handling middleware emitting a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError empty-generations classification fix lands), that message is persisted to the thread's checkpoint as part of the messages channel. LangGraph replays the full message history in `stream_mode="values"` chunks, so every subsequent run on the same thread re-streams the stale fallback marker — and the worker's chunk scanner faithfully picks it up, flipping `RunStatus.success` to `RunStatus.error` for runs that themselves had no LLM failure at all. Snapshot the set of pre-existing message ids from the pre-run checkpoint and thread it through `_extract_llm_error_fallback_message` / `_try_extract_from_message` as a filter. Markers on history messages are ignored; markers on fresh messages produced during this run still trip the error path. Falls back to an empty set when the checkpointer is absent or the snapshot can't be captured, preserving the prior behavior on first-run / no-state paths. Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`) plus an integration test exercising the full `run_agent` path with a stale history checkpointer. * fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed the langgraph_sdk default 300s read deadline. The manager's runs.wait call kept an HTTP stream open for the entire run lifetime, so the long run blew up with httpx.ReadTimeout and the outer except branch then released the dedupe key and emitted a false 'internal error' outbound. The GitHub channel's outbound send is log-only by design: agents post to the issue/PR via the gh CLI in the sandbox when they choose to comment or create a PR. There is nothing for the manager to ferry back, so the long-poll was pure overhead. This change adds ChannelRunPolicy.fire_and_forget (default False) and sets it True for the github channel. When fire_and_forget is True, _handle_chat dispatches via client.runs.create (short POST, returns once the run is pending) instead of client.runs.wait, and skips the response-extraction + outbound-publish block. ConflictError on a busy thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on the busy case is preserved for any future non-github fire-and-forget channel. Other (non-github) channels are unchanged: their policy defaults fire_and_forget=False and they continue to dispatch via runs.wait. Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget: - Default ChannelRunPolicy.fire_and_forget is False. - The github policy registers fire_and_forget=True. - github inbound calls runs.create, not runs.wait, with the right kwargs. - github inbound publishes no outbound on success. - ConflictError from runs.create still emits THREAD_BUSY_MESSAGE. - Non-github channels (slack) still dispatch via runs.wait. * test(lead-agent): accept user_id kwarg in skill-policy test stubs The two GitHub-channel tests added in #3754 stubbed _load_enabled_skills_for_tool_policy with a lambda that only accepted `available_skills` and `app_config`, but the real function (and its call site in agent.py) also passes `user_id`. This raised TypeError on every run, failing backend-unit-tests. Add `user_id=None` to match the three sibling stubs in the same file. * refactor(gateway): disambiguate context-key set names The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS shared a confusable "CONTEXT_ONLY" token in different orders, and the first broke the _CONTEXT_<X>_KEYS pattern of its sibling _CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit: _CONTEXT_INTERNAL_CALLER_KEYS - WHO: internal callers (scheduler) only _CONTEXT_RUNTIME_ONLY_KEYS - WHERE: runtime context only, never configurable Pure rename, no behavior change.	2026-07-04 22:56:24 +08:00
Xinmin Zeng	4fc08b4f15	feat: add scheduled tasks MVP (#3898) ... * feat: add scheduled tasks MVP * fix: harden scheduled task execution semantics * feat(scheduled-tasks): preset-driven schedule form with timezone and live preview Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom) plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local for one-time tasks, a timezone selector defaulting to the browser timezone, and a live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers). * feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin Localize the rest of the scheduled-tasks page (filters, detail pane, actions, edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four built-in recipe templates (GitHub Trending, news digest, issue triage, weekly report) exposed as a chip row that pre-fills title + prompt + schedule. Pin Playwright locale to en-US so E2E selectors stay stable against i18n. No backend change, no new deps. * fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test Merge with main surfaced three CI failures: - 0003_scheduled_tasks create_table collided with legacy test seeds that build from full metadata; guard with inspector.has_table so the revision no-ops when the table already exists (0004/0005 are already idempotent via _helpers.py). - persistence bootstrap concurrency/regression tests pinned HEAD to main's 0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable. - once-task router test used a fixed past run_at and tripped the must-be-in-the-future validation; use a future date. * address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences - fetchThreadScheduledTasks now checks response.ok like the other fetchers. - trigger endpoint returns 502 (not 409) when dispatch fails outright, so clients can distinguish a real conflict from a server-side failure. - E2E mock normalizes scheduled-task objects with context_mode/last_thread_id and nullable thread_id, matching the backend contract the UI renders against. - Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its revision id (file was renamed in spirit already; filename now follows). - CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence after the Scheduler notes so the sections render correctly. * fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review * fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review - defer a once task's terminal status to the run-completion hook; the task stays running until the real outcome, and a startup sweep cancels once tasks orphaned by a crash (launch-time 'completed' could stick forever) - record interrupted runs as a distinct 'interrupted' run status with a readable message; an interrupted once task ends 'cancelled', not 'failed' - enforce overlap_policy=skip for fresh_thread_per_run via an active-run pre-check (same-thread ConflictError can never fire across fresh threads) - protect terminal run statuses from the late launch-path 'running' write - honor context.non_interactive only for internally-authenticated callers; arbitrary clients can no longer strip ask_clarification - fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the offset at the resolved instant (once tasks fired an hour late around spring-forward and the create->edit round-trip diverged) - drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway API error helper between channels and scheduled-tasks frontends * fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics - scrub internal-only context keys (non_interactive) from the assembled run config for non-internal callers: gating body.context alone left the same key smuggle-able through the free-form body.config copied verbatim by build_run_config - guard update_after_launch with protect_terminal so the launch bookkeeping write cannot clobber a once task already finalized by a fast-failing run's completion hook (parent-row sibling of the run-row guard) - reject a manual trigger while the task has an active run (409) instead of launching a duplicate concurrent run on fresh_thread_per_run - re-arm a terminal once task to enabled when PATCH pushes run_at into the future; previously the endpoint returned 200 with a next_run_at that could never be claimed - make max_concurrent_runs a real global cap: each poll claims only into the remaining budget of active (queued/running) scheduled runs - paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the thread filter of /threads/{id}/scheduled-tasks into SQL - stamp context.user_id on scheduler-launched runs, matching IM channels, so user-scoped guardrail providers see the owning user --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-04 21:51:57 +08:00
Xinmin Zeng	09988caf95	feat(skills): request-scoped secrets for skills (closes #3861) (#3871) ... * feat(sandbox): per-call env injection + platform-secret scrubbing for skills Add an env parameter to Sandbox.execute_command (abstract + local + AIO) so request-scoped secrets can be injected into skill subprocesses, and scrub platform credentials (*KEY*/*SECRET*/*TOKEN*/*PASSWORD*/*CREDENTIAL*) from the inherited environment by default so scoped injection is not security theatre. LocalSandbox always passes an explicit scrubbed env; AioSandbox routes env-bearing commands through bash.exec(env=) on a fresh session and leaves the legacy persistent-shell path unchanged. Part of #3861. BEHAVIOR CHANGE: execute_command no longer inherits the full os.environ; Windows encoding tests updated to assert the scrubbed dict. * feat(skills): parse required-secrets frontmatter declaration Add SecretRequirement and Skill.required_secrets, and parse the required-secrets SKILL.md frontmatter field (a string list or {name, optional} mappings), dropping malformed entries with a warning so one bad declaration does not invalidate the skill. The declared name is both the context.secrets key and the env var injected at activation. Part of #3861. * feat(runtime): request-scoped secret carrier (context.secrets) Add SECRETS_CONTEXT_KEY + extract_request_secrets, centralising the context.secrets carrier contract. The existing context passthrough (build_run_config -> _build_runtime_context) already carries the sub-key to runtime.context without mirroring it into configurable; characterization tests lock that behaviour. Part of #3861. * feat(skills): inject declared secrets at slash-activation into bash env Binding point A: when a skill is slash-activated, SkillActivationMiddleware resolves its declared required-secrets against the request's context.secrets and writes the per-run injection set to runtime.context. The bash tool forwards that set to execute_command(env=). A skill cannot harvest a host platform credential (is_host_platform_secret guard, cf. GHSA-rhgp-j443-p4rf), and injected values are redacted from bash output (mask_secret_values) so an echoed secret never re-enters the prompt/trace. Part of #3861. * test(skills): lock the five secret leak surfaces + add trace redaction helper Regression tests assert the secret value is absent from all five surfaces: prompt (activation message), checkpoint (graph state vs context separation), audit (journal records names only), trace (metadata builder never copies context; never mirrored to configurable), and stdout (mask_secret_values). Add redact_secret_context_keys as a defensive helper for any context serialization. Part of #3861. * docs(backend): document request-scoped secrets for skills Add Request-Scoped Secrets subsection (Skills) + env policy note (Sandbox) and the execute_command(env=) signature change, per the doc-sync policy. Part of #3861. * fix(skills): close gaps found by end-to-end verification of request-scoped secrets Real-gateway e2e + independent review of #3861 surfaced three defects, now fixed: 1. Slash activation never fired in the live chain. InputSanitizationMiddleware wraps user input in BEGIN/END markers before SkillActivationMiddleware sees it, and the original text was only preserved when an upload or IM channel set it. For a plain text message the slash command became undetectable, so no secret was ever resolved. Fix: the sanitizer now setdefaults the pre-wrap text into ORIGINAL_USER_CONTENT_KEY (additive; sanitization behaviour unchanged), so slash activation works for all messages. Pre-existing latent bug surfaced here. 2. The raw request config (with context.secrets) was persisted to runs.kwargs_json and echoed by the run API (RunResponse.kwargs). Fix: redact_config_secrets() strips secret-bearing context keys from the persisted/echoed copy in start_run; the live config that drives the run keeps them. build_run_config now also sets configurable.thread_id on the context path (the checkpointer requires it). 3. Connection-string credentials (DATABASE_URL, REDIS_URL, SENTRY_DSN, GH_PAT, ...) were not scrubbed from the inherited sandbox env. Fix: env_policy adds a *DSN* pattern plus an explicit connection-string denylist (no blanket *URL* — benign service URLs stay readable). Verified end-to-end via a real gateway run (real LLM + skill activation + bash): the secret reaches the sandbox subprocess and appears in NONE of prompt, trace, checkpoint, audit, stdout, runs.kwargs_json, or the run API. Part of #3861. * docs(backend): document the env scrub, persistence redaction, and sanitizer interaction Sync the Request-Scoped Secrets section with the verification-driven fixes: inherited-env scrub (incl. connection-string denylist), run-record/run-API redaction as the 6th sealed leak surface, and the sanitizer preserving original content so slash activation fires. Part of #3861. * fix(skills): inject caller secret over scrubbed host value; drop redundant host-name guard A real-world demo (a skill calling a third-party cloud API with a request-scoped key) exposed that the is_host_platform_secret guard was both wrong and harmful: it refused to inject a caller-supplied secret whenever a same-named variable existed in the Gateway env — which is exactly the #3861 use case (a per-user key overriding a shared platform key). The guard was also redundant: build_sandbox_env already scrubs secret-looking names from the inherited env before injection, so a skill can never read a host credential — it only ever receives the caller's value. Remove the guard; the injected (caller) value simply wins over the scrubbed host value. Verified end-to-end: the agent called the real cloud API successfully with the caller's key, the host's same-named key was scrubbed and never used, and the caller's key leaked to none of the surfaces. Part of #3861. * fix(skills): address review on request-scoped secrets (#3861) Review fixes from PR #3871: - E2BSandbox.execute_command now accepts env/timeout and routes them to commands.run(envs=, timeout=). The bash tool passes env= unconditionally, so the prior signature (command only) raised TypeError on every e2b bash call and broke e2b deployments entirely. env=None stays backward-compatible. - SkillActivationMiddleware clears the active-secret set before resolving each activation, so a later skill in the same run never inherits an earlier skill's injection set (the #3861 contract: a skill only receives what the caller supplied AND that skill declared). - AioSandbox env path uses a dedicated _DEFAULT_HARD_TIMEOUT — bash.exec exposes no idle/no-change timeout, so the prior reuse of the legacy idle constant conflated wall-clock vs idle semantics. The env path also retries on the ErrorObservation signature now, sharing the legacy persistent-shell recovery contract. - mask_secret_values skips values below a minimum length floor so a short declared secret (e.g. "42") cannot shred unrelated bytes (exit codes, timestamps, sizes) of tool output. The secret is still injected into the subprocess; only the output mask skips it. session_id reuse on the env path is intentionally NOT added: a shared session could let request-scoped secrets ride the session env into later commands, which the SDK does not contractually forbid. The fresh-session choice matches the LocalSandbox model (each call is a fresh subprocess); the trade-off (consecutive env-bearing calls do not share cwd/venv/exports) is documented on _execute_with_env.	2026-07-03 07:51:22 +08:00
Yi Deng	4e6248f013	fix(gateway): clamp client-supplied recursion_limit to prevent runaway runs (#3903) ... build_run_config() copied every top-level request key (except configurable/context) verbatim into the LangGraph RunnableConfig, including recursion_limit. The server default of 100 was fully overridable by the caller with no upper bound, so a request like {"config": {"recursion_limit": 100000000}} could make a single run execute effectively unbounded LangGraph super-steps (each >= 1 LLM call), enabling runaway API cost / DoS. Validate the client value server-side and clamp it into a safe range: - valid positive ints are capped at a configurable ceiling (AppConfig.max_recursion_limit, default 1000 to match the existing frontend/public-skill default so legitimate deep runs are unaffected) - invalid/non-positive/bool/None values fall back to the 100 server default - applied on both the configurable and the LangGraph >= 0.6.0 context paths - WARNING logged on clamp for observability Add unit tests (including a configurable-ceiling case), expose max_recursion_limit in config.example.yaml (config_version 16), and document the ceiling/fallback in backend/docs/API.md. Co-authored-by: DengY11 <DengY11@users.noreply.github.com>	2026-07-02 11:38:21 +08:00
Huixin615	1ac9c9edee	fix: preserve langgraph resume command (#3732)	2026-06-23 22:22:07 +08:00
Vibhu Dixit	caf54938db	fix(gateway): propagate thread_id for LangGraph context API (#3735)	2026-06-23 16:18:43 +08:00
Huixin615	ca9428d0cd	feat: support regenerating latest answer (#3637) ... Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-06-20 18:59:42 +08:00
Huixin615	f43aa78107	fix(agents): sync agent_name across context/configurable and reject empty soul (#3549) (#3553) ... * fix(agents): sync agent_name across context/configurable and reject empty soul (#3549) Two independent issues caused custom agent creation to silently fail: 1. build_run_config only wrote agent_name into one container (configurable or context), so setup_agent — which reads ToolRuntime.context exclusively since LangGraph >=1.1.9 — saw agent_name=None and wrote SOUL.md to the global base_dir instead of users/{user_id}/agents/{name}/. Mirror the dual-write pattern already used by merge_run_context_overrides and naming.py so both containers always carry the same value. 2. setup_agent persisted whatever soul string it received, including empty or whitespace-only content, and still reported success. The frontend then surfaced an unusable agent and the global default SOUL.md could be silently overwritten with empty content. Reject empty soul before any filesystem operation so the model can retry. Tests: - test_gateway_services.py: dual-write regressions for both configurable and context entry paths, explicit-agent-name precedence on both sides, and a shape-parity test against merge_run_context_overrides. - test_setup_agent_tool.py: empty/whitespace soul rejection, plus no-overwrite guarantees for existing global and per-agent SOUL.md. * Update services.py	2026-06-14 10:40:16 +08:00
DanielWalnut	aa015462a7	feat(im): Add user-owned IM channel connections (#3487) ... * Add user-owned IM channel connections * Fix dev startup and channel connect popup * Use async channel connect flow * Harden dev service daemon startup * Support local IM channel connections * Align IM connections with local channels * Fix safe user id digest algorithm * Address Copilot IM channel feedback * Address IM channel review comments * Support all integrated IM channel connections * Format additional channel connection tests * Keep unavailable channel connect buttons clickable * Fix IM channel provider icons * Add runtime setup for enabled IM channels * Guard global shortcut key handling * Keep configured IM channels editable * Avoid password autofill for channel secrets * Make channel threads visible to connection owners * Persist IM runtime config locally * Allow disconnecting runtime IM channels * Route no-auth channel sessions to local user * Use default user for auth-disabled local mode * Show IM channel source on threads * Prefill IM channel runtime config * Reflect IM channel runtime health * Ignore Feishu message read events * Ignore Feishu non-content message events * Let setup wizard enable IM channels * Fix frontend formatting after merge * Stabilize backend tests without local config * Isolate channel runtime config tests * Address channel connection review comments * Use sha256 user buckets with legacy migration * Ensure runtime IM channels are ready after restart * Persist disconnected IM channel state * Address channel connection review comments * Address channel connection review findings Frontend connect flow: - Open the runtime-config dialog only when a provider still needs credentials; configured providers go straight to the connect flow, so the binding-code/deep-link path is reachable from the UI again. - After saving credentials, continue into the connect flow when a user binding is still required (multi-user mode) instead of stopping at a "Connected" toast. - Extract shared provider-state helpers to core/channels/provider-state and add unit + e2e coverage for the direct-connect and configure-then-connect paths. Provider status semantics: - Report connection_status from the user's newest connection row; with no binding it is not_connected, except in auth-disabled local mode where a configured running channel is effectively connected. Concurrency and event-loop correctness: - Offload ChannelRuntimeConfigStore construction and writes, channel service construction, and Slack connection replies to threads; add a tests/blocking_io/ anchor for the runtime-config handlers. - Consume binding codes with a conditional UPDATE so a code can only be used once under concurrent workers; retry upsert_connection as an update when a concurrent insert wins the unique constraint. - Serialize ensure_channel_ready per channel so concurrent provider polls cannot double-start a channel worker. Config and migration hardening: - Stop mutating the get_app_config()-cached Telegram provider config; the runtime store now owns the UI-entered bot username. - Register channel_connections in STARTUP_ONLY_FIELDS with the standardized startup-only Field description. - Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name so another user's same-prefix bucket can never be migrated. - Remove the unused Telegram process_webhook_update path and document src/core/channels in the frontend docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address PR review comments on authz scoping and channel runtime Security (review feedback from ShenAC-SAC): - Scope internal-token callers to the connection owner carried in X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright, in both require_permission(owner_check=True) and the stateless run endpoints. Internal callers keep access to their own and shared/legacy threads, and may claim a default-owned channel thread for its real owner, but a leaked internal token no longer grants cross-user thread access. - Require admin privileges for POST/DELETE /api/channels/{provider}/ runtime-config: runtime credentials and channel workers are instance-wide shared state (same model as the MCP config API). Read-only provider listing stays available to all users. Performance (review feedback from willem-bd): - Skip the redundant thread channel-metadata PATCH after the first successful backfill per thread. - Reuse the per-connection Slack WebClient until its token changes instead of constructing one per outbound message. - Reconcile channel readiness for all providers concurrently in GET /api/channels/providers. Also resolve the code-quality unused-import flag in the blocking-io anchor by pre-importing the channel service via importlib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix prettier formatting in provider-state test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reconcile UI runtime channel config with config reload on restart Main now reloads a channel's config.yaml entry on restart_channel() (#3514, issue #3497). Adapt the user-owned connection flow to coexist: - configure_channel() restarts with reload_config=False — the caller just supplied the authoritative config (browser-entered credentials that are never written to config.yaml), so a file reload must not clobber it with the stale on-disk entry. - _load_channel_config() re-applies the UI runtime-store overlay used at startup, so an operator-triggered restart keeps browser-entered credentials for channels without a config.yaml entry and does not resurrect a channel disconnected from the UI. - Offload the reload's disk IO (config.yaml + runtime store) with asyncio.to_thread, matching the blocking-IO policy on this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 15:24:58 +08:00
zhongli-sz	3ae82dc663	fix(mcp): add auth interceptor with channel user_id and keep header propagation to mcp tools (#3294) ... * 修复channel中的user_id传递到interceptor中的bug, mcp可通过header传递user_id到mcp工具 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(channel,mcp,gateway): normalize channel user_id and add regression tests Normalize external channel user ids into filesystem-safe runtime context while preserving raw channel_user_id, and document gateway user_id propagation semantics. Add regression coverage for channel user_id context mapping, gateway user_id precedence/internal-role behavior, and MCP interceptor header forwarding via meta.headers. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auth,mcp): harden user id normalization and header handling Increase sanitized user-id digest suffix to 16 hex chars, replace internal system role magic string with a shared constant, and harden MCP header forwarding with Mapping type checks. Add regression tests for empty channel user_id handling, unsupported header types, and updated digest length behavior. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: zhongli <335302680@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-03 15:48:19 +08:00