mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-30 15:10:41 +00:00
103 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
XuRuuuy
|
a9c8f79dfe
|
fix(gateway): resolve the agent factory off the event loop (#6206)
* fix(gateway): resolve the agent factory off the event loop start_run() called resolve_agent_factory() inline, which lazily imports the full lead-agent assembly stack (middlewares, authz, MCP, jsonschema; measured at ~8s cold on Windows, multi-second on slower filesystems). start_run is the single choke point every run-creation path flows through, so the first run after Gateway startup blocked the event loop for the whole import, stalling every other request, stream and channel on the same loop - the loop-stall family issue #5172 fixed for agent construction itself (the checkpoint-state path already resolves through the assembly pool). Resolve the factory via asyncio.to_thread: the first call pays one worker-thread hop for the import, later runs hit the cached module. Regression test pins the contract with a probe module whose module-level __getattr__ records the thread that performed the from-import. * fix(gateway): isolate factory resolution in the assembly pool Move import regressions into the strict blocking-I/O suite and cover default-executor isolation, request context, and failure-before-admission ordering. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
b9ee8af328
|
fix(skills): keep custom-skill edit and storage lookups off the event loop (#6332)
* fix(skills): keep custom-skill edit checks off the event loop update_custom_skill built the user-scoped storage, ran the editability probes, validated the frontmatter through a temporary directory and read the content being replaced on the Gateway loop; only the write and history append had been offloaded. Run the pre-scan checks in one worker and the replaced-content read in another, matching rollback_custom_skill, and build delete_custom_skill's storage off the loop as well. * fix(skills): build skill storage off the loop in install and skill_manage The same sweep that moved the custom-skill edit checks off the loop finds two more storage lookups on it: the archive-install helper behind both install routes, and the agent's skill_manage tool, whose lookup resolves and stats config.yaml on every call. Offload both, and pin that the caller's user context still reaches the storage lookup now that the edit route resolves it inside a worker. * docs(changelog): reference #6332 for the skill storage off-loop fix --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
c2f31e440e
|
fix(knowledge): load the catalog's agent config off the event loop (#6313)
* fix(knowledge): load the catalog's agent config off the event loop Both retrieval-catalog handlers called load_agent_config() directly from async code. The agent store is synchronous: file IO on the file backend, a sync SQLAlchemy round trip on the db backend, so a slow disk or database stalled every other Gateway request. Offload the load with asyncio.to_thread, matching agents.py, scheduled_tasks.py and the run-admission path in services.py. The strict Blockbuster anchors drive both handlers end to end against real file and SQLite agent stores, plus the unknown-agent 404 path. * docs(changelog): reference #6313 for the knowledge catalog fix * Added the Chinese changelog entry back --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
siye566
|
d8ca2c9358
|
test(community): make URL offload regressions DNS-independent (#6314)
* test(community): make URL offload regressions DNS-independent * test(community): reject unsupported DNS fixture families --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
0223d949bf
|
fix(threads): remove a deleted thread's files off the event loop (#6319)
* fix(threads): remove a deleted thread's files off the event loop
DELETE /api/threads/{id} called shutil.rmtree over the thread directory
directly on the event loop, so deleting a thread with a large workspace
stalled every other Gateway request and live SSE stream until the whole
tree was gone. Branch copying in the same router already offloads its
filesystem work through run_file_io; deletion did not.
The removal now runs on the file-IO pool. It is wrapped in await_drained
rather than bare-awaited: cancelling the await abandons the worker, not
the removal, so the durable delete reservation would otherwise release
while files are still being deleted, and a newly admitted run could
write into a directory that is about to disappear.
* docs(changelog): reference #6319 in the thread delete off-loop entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Hyeonsang Cho
|
52046aa82e
|
fix(projects): convert shelf documents outside the SQLite write lock (#6305)
* fix(projects): convert shelf documents outside the SQLite write lock Lazy conversion ran pymupdf/markitdown inside the BEGIN IMMEDIATE transaction that serializes the derived/converted.md publish against trash and purge. On SQLite that lock is database-wide, so every unrelated writer (run status, thread metadata, the scheduler) waited for the whole conversion and failed with "database is locked" past busy_timeout (30s). Conversion now writes into .staging/ outside any transaction. The document row lock is taken only to revalidate the live row and atomically rename the output into place (publish_under_live_lock, renamed from convert_under_live_lock); a row trashed or purged meanwhile discards the staged output, so nothing is published after purge. A read-only probe skips converting rows that are already gone, a failed conversion is still revalidated so a purged row reports content_missing, and the conversion is drained on cancellation so its staging file is always removed. * docs(changelog): reference #6305 in the shelf conversion lock entry * fix(projects): share one conversion across concurrent first reads A first-read conversion holds a bounded file-IO pool thread for its whole duration. With the conversion moved out of the row lock, concurrent first reads of one document each converted on their own thread, so a burst of reads could starve unrelated offloaded file work. Route the conversion through a single-flight keyed by (user, document, project): the first reader leads and the rest wait on its result without holding a pool thread. The map holds concurrent.futures.Future values under a threading.Lock because the read tool also runs on the isolated subagent loop. Followers wait through asyncio.shield, so a cancelled follower cannot cancel the shared future, and a leader that is cancelled or raises hands the work to a follower instead of failing it. |
||
|
xiaodu55
|
a0b7679edf
|
test(channels): skip the POSIX-bound mode assertions on Windows hosts (#6257)
The channel runtime config store's atomic 0o600-temp + replace + chmod cannot land a POSIX mode on Windows, where chmod is a no-op approximation. Skip the two POSIX-bound tests and guard the trailing mode assertion of the chmod-failure test, keeping its platform- independent log-behavior coverage everywhere. Fixes #6256 Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com> |
||
|
Guo-Yixin
|
68014de157
|
fix(artifacts): invalidate digests after external file replacement (#6195)
* fix(artifacts): invalidate digests after external file replacement * docs(gateway): keep artifact guidance within the module budget * fix(artifacts): validate oversized ranges by file identity --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Xuehao Xu
|
9b17bef630
|
feat(channel): add QQ WebSocket channel (#6081)
* feat: add QQ WebSocket channel * fix: address QQ channel review feedback * docs: trim inherited agent guidance below CI limit * fix: align QQ gateway and passive reply limits with protocol * docs: keep merged channel guidance within CI budget |
||
|
xiaodu55
|
6176b057a2
|
fix(mcp): drain the config read-modify-write across cancellation (#6210)
* fix(mcp): drain the config read-modify-write across cancellation Route the five MCP config endpoints through the canonical run_drained_write helper and fold the tools-cache reset into the drained unit, so a cancelled caller cannot detach from the worker holding the extensions-config locks or skip the cache invalidation behind a settled write. Caller-facing HTTPException contracts re-raise unlogged. Fixes #6209 * fix(mcp): type the drain helper against the worker signature and unlist drain adopters - _run_drained_mcp_apply now mirrors the canonical helper's [**P, T] Callable[P, T] typing so each call site's argument list is checked; - AGENTS.md stops enumerating the adopted routers, so the line stops needing an edit on every new adopter. --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com> |
||
|
Hyeonsang Cho
|
bb4f7ac5e2
|
fix(browser): pin browser connections to the SSRF-vetted addresses (#6201)
* fix(browser): pin browser connections to the SSRF-vetted addresses The navigate screen and the per-request guard resolve a hostname to vet it, but Chromium resolved the name again to open the connection. A rebinding DNS server could answer both checks with a public address and the connection with a private or cloud-metadata one, and the browser followed it. Launch each private browser through a per-session loopback SOCKS5 proxy. Over SOCKS5 Chromium sends the hostname itself, so the proxy resolves it exactly once through the session's egress resolver and connects only to the addresses that resolver vetted. resolve_browser_egress applies the same allow_private_addresses policy as validate_browser_url, on top of a new url_safety.resolve_public_addresses helper that validate_public_http_url now shares. Loopback is proxied explicitly with bypass "<-loopback>". The tools, navigate_and_capture and the Live WebSocket all pass the resolver, because a thread's first caller fixes its launch options. CDP-attached Chrome is unchanged. Delegated fetch services (crawl4ai, Browserless, fastcrw) still resolve on their own side, which the Gateway cannot pin. * docs(changelog): reference #6201 in the browser egress pinning entry * fix(browser): bound egress proxy faults and close, and record the WebRTC gap Review follow-ups for the pinned browser egress proxy: - A resolver exception other than ValueError escaped the connection handler as an unretrieved task error. Log it with its traceback and reply SOCKS5 general failure (0x01), so a resolver fault stays distinguishable from a policy refusal (0x02). - Bound the resolve with its own timeout. The handshake timeout did not cover it, so a wedged getaddrinfo held the half-open tunnel until Chromium gave up. A timeout replies host unreachable. - Bound close(): Server.wait_closed() also waits for a connection whose handler registered after the writer sweep, which could stall session close and eviction for the 15 s handshake. A handler that starts after close now drops its connection without opening a tunnel. - Docs said a launched browser connects "solely" through the proxy. WebRTC UDP does not traverse a SOCKS proxy, so AGENTS.md, README (en/zh) and both changelogs now scope the guarantee to TCP and name the gap. |
||
|
Linyao Du
|
616c6e2e36
|
fix(uploads): reject Windows staging aliases before batch ingestion (#6149)
* fix(uploads): reject Windows staging aliases before batch ingestion * fix(uploads): cover shelf names and case-insensitive staging aliases * test(uploads): isolate legacy alias filesystem cases |
||
|
Hyeonsang Cho
|
457aa3a492
|
fix(community): resolve SSRF-guarded hostnames off the event loop (#6140)
* test(blocking-io): catch socket.getaddrinfo on the event loop Blockbuster wraps socket.socket methods but not the module-level resolver, so a synchronous hostname lookup on the loop passes the strict gate. The SSRF guard in deerflow.community.url_safety resolves with socket.getaddrinfo from async web tools, and the anchors added next stay green against that unfixed code without this rule. Add socket.getaddrinfo to _PROJECT_BLOCKING_RULES, scoped to app.* and deerflow.* callers. loop.getaddrinfo and asyncio.to_thread resolve in a worker thread and are not flagged. The full make test-blocking-io suite stays green with the rule active (187 passed). * fix(community): resolve SSRF-guarded hostnames off the event loop validate_public_http_url resolves hostnames with blocking socket.getaddrinfo. The crawl4ai and Browserless web_fetch, web_capture, browser_navigate, the Gateway browser navigate route, and the Live stream's navigate input and seed all called it directly from async code, so a slow DNS answer for a model- or user-chosen URL stalled the Gateway event loop for the whole lookup. Run the guard through asyncio.to_thread at each of those call sites, the same way personal_network already offloads it. The screening result is unchanged. fastcrw's web_fetch is a sync tool and personal MCP validation already runs in a worker, so neither needed a change. * docs(changelog): reference #6140 in the SSRF guard event-loop entry * fix(browser): run the Playwright request guard off the shared browser loop BrowserSession._install_request_guard screens every redirect hop, iframe and subresource by calling the SSRF guard inside the async route callback. The guard resolves hostnames with blocking socket.getaddrinfo, and that callback runs on the process-wide Playwright loop, which also pumps Live frames and input for every browser session, so one slow lookup stalled all of them. Run the guard through asyncio.to_thread there as well, and pin it with a strict-gate anchor that drives the real route handler. Also note in the url_safety docstring that the screen is check-time only: a caller that re-resolves at connect time is not pinned to the vetted IPs. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
d3e8e78f3f
|
feat(authz): middleware-declared tool path Layer 1 now covers tools declared by middlewares (#6105)
* feat(authz): extension tool authorization PR2 — middleware-declared tool path Layer 1 now covers tools declared by middlewares, which LangChain merges into the bound tool set after the host's explicit-list filter: - new agents/middlewares/tool_declarations.py: record the ordinary pass's verdict, collect declarations from the built stack, decide only the delta seeded by the first pass (a name denied for this build stays denied), narrow the stack build-locally on independent state-preserving copies (direct __dict__ write, refused up front for tools data descriptors, never mutating caller-owned instances), and verify the exact bound stack after the final normalization copy. - IsolatedMiddleware ships a state-preserving __copy__ so narrowing survives full/delta normalization and copies of copies. - TodoMiddleware degrades when write_todos is denied: no system-prompt injection, no completion/context-loss reminders, no jump_to. - Layer 2 forwards host-resolved tool provenance into AuthzRequest.context and binds the infrastructure exemption to the concrete host-created tool object instead of a name set (tool=None goes to the provider). - Wired into all four assembly paths (lead x2, native subagent with the decision offloaded via asyncio.to_thread, DeerFlowClient). Tests: new test_middleware_declared_tools.py (35) and test_layer2_tool_provenance.py (7); blocking-IO anchor pinning the subagent offload (teeth verified red->green); placement-guarantee and enforcement suites extended; executor test doubles converted for the async _create_agent. Behavior change called out in CHANGELOG: with authorization enabled and a tools policy that excludes write_todos, plan-mode builds no longer bind it (built-in RBAC default unchanged). * fix(authz): close residual gaps from review — narrow non-collectable declarations, dedupe candidates, warn on skipped pass - apply_declared_tool_view now removes non-BaseTool declarations from the bound view when authorization is enabled (they would otherwise bind via LangChain's create_tool auto-conversion with no provider decision); verify_declared_tool_view fails loudly if one survives normalization - decide_declared_tools dedupes never-submitted names so the provider sees each candidate once - subagent executor logs a warning when the provider is set but the seeded Layer-1 state is missing, instead of skipping the declaration pass silently - docs: middleware-declared tools must be BaseTool instances with usable names * fix(authz): fail loudly on non-list/tuple middleware tools declarations _iter_declared warned and returned () for a 'tools' attribute that is not a list/tuple, but LangChain's factory iterates the attribute as-is — so a set/generator/dict_values declaration bound with no Layer-1 decision. Raise DeclaredToolViewError instead, matching the module's fail-loud standard for every other un-narrowable shape. Disabled path remains a strict no-op. Docs updated to match. |
||
|
xiaodu55
|
5b0ef628e2
|
fix(skills): drain custom-skill mutation tails across cancellation (#6078)
* fix(skills): drain custom-skill mutation tails across cancellation * fix(skills): log drained skill mutation failures and drain state/install tails Address review feedback on #6078: record cancelled-then-failed mutation tails inside the drained unit (exception type only, per the managed-models precedent), and extend the drained tail to the remaining write-to-cache shapes in update_skill and the skill install path. * test(skills): pin the lock contract under the drained cancellation timing The cancelled update_skill caller now drains its mutation tail, so CancelledError surfaces only after the tail settles. Start the MCP writer without awaiting the cancelled task first, assert the section stays owned while the drained tail is in flight, then await both and assert the final order. --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com> |
||
|
SuperSgdk
|
f74a290c42
|
fix(uploads): reject reserved staging filenames before batch ingestion (#6122)
* fix(uploads): reject filenames reserved for temporary staging Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace. * fix(uploads): report reserved names before HTTP batch ingestion Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation. * fix(uploads): reject Windows-style staging basenames on POSIX Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation. Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
fe22e3899a
|
fix(channels): keep blocking SDK teardown off the event loop in stop() (#6134)
* fix(channels): keep blocking SDK teardown off the event loop in stop()
SlackChannel.stop() called SocketModeClient.close() inline. close() joins
the SDK's message-processor thread (~0.7s every call, measured) and waits
for in-flight listeners, whose blocking Web API calls can run up to the
WebClient timeout, so every run, stream and channel on the Gateway stalled
during shutdown and POST /api/channels/{name}/restart.
Feishu and DingTalk had the same shape: they joined their SDK threads
inline with a 5s timeout, and since those threads only return on a fatal
error, the join normally froze the loop for the full 5s.
The teardown now runs through asyncio.to_thread. Slack detaches the client
first so a still-queued connect skips it, and shields and tracks the close
task so a cancelled stop() leaves it running and a retried stop() awaits
the same close instead of closing twice.
tests/blocking_io/test_channel_stop_offloop.py anchors all three stop()
paths to the strict Blockbuster gate, driving a real slack-sdk client.
* docs(changelog): reference #6134
* fix(channels): join the Discord client thread off the event loop in stop()
DiscordChannel.stop() still joined its client thread inline with a 10s
timeout. The thread usually exits right after the cross-loop client close,
but a timed-out close or a slow _run_client() drain keeps it alive for the
whole timeout, stalling the shared Gateway loop the same way the Slack,
Feishu and DingTalk teardown did. Run the join through asyncio.to_thread so
stop() matches the channels/AGENTS.md teardown rule, and add Discord to the
strict Blockbuster anchor.
|
||
|
ZJPex
|
1129ee4e8a
|
feat: add bounded JSON syntax acceptance criteria (#5947)
* feat: add bounded JSON syntax acceptance criteria * fix(subagents): authorize bounded local JSON acceptance reads * docs: keep JSON acceptance guidance within chain limit * docs: use English for JSON acceptance comments and guidance * fix(subagents): keep unauthorized JSON probes unverified |
||
|
Hyeonsang Cho
|
b72753c226
|
fix(gateway): load skills off the event loop in the listing and detail routes (#5945)
* fix(gateway): load skills off the event loop in the listing and detail routes
GET /api/skills, GET /api/skills/custom and GET /api/skills/{name} called
load_skills() inline. Resolving the caller's storage builds absolute
paths, and load_skills walks every public and custom skill directory and
parses each SKILL.md, so the work grows with the number of installed
skills. #5747 moved the same call off the loop for get_custom_skill and
documented why; these three routes were missed, and the strict
Blockbuster gate raises BlockingError on each of them.
Offload the load through one shared _load_user_skills helper (also used
by update_skill's reload), and add a gate module covering the three
routes including get_skill's 404 branch.
* docs(changelog): reference #5945 in the skills listing blocking I/O entries
|
||
|
Hyeonsang Cho
|
5b8b69bd45
|
fix(gateway): keep /api/user-profile filesystem work off the event loop (#5935)
* fix(gateway): keep /api/user-profile filesystem work off the event loop GET and PUT /api/user-profile resolved the per-user USER.md path (which builds absolute paths on every call), stat'ed, read, created the user bucket and wrote the file inline on the event loop, while every other handler in the custom-agent router offloads that work with asyncio.to_thread. The strict Blockbuster gate raises BlockingError on both; in production a slow disk stalls every other request on the worker. Offload the whole resolve-stat-read and resolve-mkdir-write sequences, capturing the effective user id on the loop first like the sibling handlers. The blocking-IO gate test for this router now covers both profile routes (missing profile, write, read back). * docs(changelog): reference #5935 in the user-profile blocking I/O entries |
||
|
李靖
|
10bd065615
|
feat(agents): persist tool artifacts as resolvable handles (#4676) (#4929)
* feat(agents): persist tool artifacts as resolvable handles (#4676) MCP and sandbox tools return file paths, URLs, and task ids that are lost when context compaction summarizes the conversation. This adds a per-thread artifact handle registry so the model can reference artifacts by short deterministic handles (art_xxxxxxxx) that survive compaction and resolve to real references at the tool-call boundary. - ThreadState.tool_artifacts: reducer channel capped at 100 entries, same-handle latest-wins, survives summarization compaction - artifact_registry: deterministic handle derivation from (thread_id, tool_call_id, call_index); extraction precedence structured_content -> file/image blocks -> conservative text scan; HTML-escaped model-context rendering with char budget - ArtifactCaptureMiddleware: before_model hook captures lightweight metadata (never content) and tracks handle consumption - ArtifactResolutionMiddleware: rewrites handles in tool-call args to real references before execution; unknown handles untouched - DurableContextMiddleware: projects the registry into the hidden durable-context data block ([available]/[consumed] status) - config: new tool_artifacts section (all switches default on) - frontend: artifact badges on tool-call steps keyed by tool_call_id Fixes #4676 * fix(agents): harden artifact registry after review (#4676) Review findings on #4929: - capture/consumption updates emitted in the same before_model call are now concatenated instead of dict-merged; dict-merging clobbered the shared tool_artifacts key and permanently dropped fresh captures - handles embed a per-result ordinal so multiple references inside one tool result never collide (two paths in one text block previously collapsed into one handle) - structured_content extraction pulls concrete refs from known keys (file/path/url/task_id/..., string or list-of-string values) instead of storing a truncated JSON preview, falls back to an untruncated whole-object data entry, and no longer skips file/image content blocks - plain-string tool results (e.g. bash/write_file output) are scanned for sandbox paths / remote URLs too, gated by detect_refs_in_text - wire max_entries through configure_tool_artifact_max_entries at assembly (clamped 10..1000) and gate detect_refs_in_text scans - resolution requires both enabled and resolve_handles_in_args, matching its docs - add real ToolCallRequest.override coverage and the spec 14.4 integration cycle (capture -> inject -> resolve -> consumption) * fix(agents): address second-round artifact registry review (#4676) Follow-up findings on #4929: - drop generic `output`/`outputs` from structured ref keys: prose under those keys no longer becomes a bogus file entry (falls back to the whole-object data entry; path-shaped refs are still caught by the free-text scan) - strip closing quotes, brackets, and backticks from detected refs so markdown- and JSON-wrapped paths resolve to clean real references - remove the process-wide cap setter: the reducer enforces only an absolute ceiling (1000) and ArtifactCaptureMiddleware applies the configured max_entries per agent when emitting updates, so agents with different configs in one process never share or flap a cap - skip extraction for tool messages whose tool_call_id already has registry entries and memoize known-empty results (bounded FIFO), so steady-state capture cost drops to the new message tail instead of a full-history rescan per model call - correct the inverted ordering claim: capture is position-independent in the wrap chain because it is a before_model hook reading state messages; fixed the assembly comment, AGENTS.md item 15, and the ordering test docstring CI parity verified locally: ruff check + format --check, full offline suite, blocking-io suite, collect-only, uv lock --check, check_agent_guidance, frontend format/lint/typecheck/build/tests. * fix(agents): sliding-window cap and URI gates for artifact registry (#4676) Third-round review findings on #4929: - max_entries is now a sliding retention window: fresh captures are always registered, and when the projection exceeds the configured cap the middleware appends a trailing {"op": "trim_to", "keep": N} directive that the reducer applies by evicting the oldest entries. The previous remaining-budget emission froze the registry at the cap, permanently dropping post-compaction artifacts — the exact failure mode #4676 targets — and silently disabled the round-2 rescan optimizations at the point threads grew longest - _track_consumption memoizes settled tool-call args in a bounded FIFO alongside the existing empty-result memo, so historical AIMessage args are not regex-rescanned every model call - file/image content-block URLs must be http(s) or path-shaped; data:/blob: URIs (MCP embedded resources carrying base64 payloads) can no longer enter thread state, tool args, or crowd out the render budget; protocol-relative //host forms are rejected as well - structured-content ref values go through the same URI-shape gate as content blocks, closing the embedded-payload hole via known keys such as url/urls - tool_artifacts.enabled now gates the whole middleware including consumption tracking via an early return in before_model Docs updated to describe the sliding window, URI gates, enabled scope, and both bounded memos. * fix(agents): final evictions and pending-aware consumption for artifact registry (#4676) Fourth-round review findings on #4929: - the sliding window interacted with the seen_call_ids skip to resurrect evicted entries every round: once an entry was trimmed, its tool_call_id left the seen set while its ToolMessage stayed in context, so the next model call re-extracted it, re-registered it as newest, and evicted something else — a never-converging rotation. The empty-result memo is now a unified _processed_results memo that records every extraction regardless of outcome, making evictions final; steady state over a saturated thread is quiescent (pinned by a multi-round test asserting no updates and zero extractions) - the consumption scan built its handle map from pre-capture state, so a call referencing a handle captured in the same before_model call missed, got settled as quiet, and its consumption mark was lost forever. _track_consumption now receives the pending capture entries and resolves against state + pending; a call settles as quiet only after every referenced handle resolved, so transient misses retry next round instead of permanently suppressing the mark AGENTS.md entry 15 updated to describe both memos and the eviction-is-final semantics. * fix(artifacts): address occurrence identity, PII and URL review findings * fix(artifacts): enforce resolved policy checks and bounded registry handling --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Bxl
|
41e8e75f56
|
fix: balance temporary sandbox leases for uploads (#3261)
* fix: balance temporary sandbox leases for uploads * fix: address sandbox upload review feedback --------- Co-authored-by: Bxl <188532127+BXL1015@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yc2bgr8
|
cdb5028d88
|
feat(community): offload image search to a thread via asyncio.to_thread (#5788)
* feat(community): offload image search to a thread via asyncio.to_thread
Make image_search_tool async and run the blocking ddgs call in a worker
thread with asyncio.to_thread, continuing the TODO.md planned feature
('Replace sync requests with httpx.AsyncClient in community tools').
The ddgs SDK exposes no async API, so instead of swapping the HTTP
client the blocking DDGS.images() call is offloaded to a thread — the
same pattern the infoquest/jina_ai tools use for readability
extraction. Callers see no change in behavior.
Tests invoke the async tool via ainvoke under @pytest.mark.anyio.
* fix(image-search): offload config lookup with DDGS search
* test(image-search): pin DDGS search off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
haochen
|
fc9fb2dea2
|
test(lark): make blocking I/O CLI fixture work on Windows (#5826)
* test(lark): make blocking I/O CLI fixture work on Windows * docs(tests): document portable Lark CLI fixtures --------- Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com> |
||
|
yang rui
|
c814943215
|
feat(extensions): grant bounded host model invocation (#5796)
* feat(extensions): grant bounded host model invocation * fix(extensions): retain model budgets and isolate schema validation * fix(extensions): normalize provider-origin model cancellation --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
82671596f3
|
feat(authz): add plugin resource targets, provenance, action and management enforcement (#5842)
* feat(authz): add plugin resource targets, provenance, action and management enforcement
PR1 of the extension-tool-authorization series
(docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin
authorization plumbing, no new author-facing declaration.
- `authz/plugin_targets.py` is the single encoder/validator for composite
targets (`namespace/part`), with one validating composer behind three
kind-checked constructors; invalid input raises instead of degrading.
- `authz/plugin_authz.py` is the harness decision layer (`plugin_action`,
`plugin_management`, plus the batch filters the page slice will consume):
disabled is a no-op, an explicit deny raises, and provider errors follow
`fail_closed` like the sandbox gate.
- `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke`
after the action resolves and before the body is streamed.
- `deerflow_extension_api.auth` publishes `require_plugin_management` /
`arequire_plugin_management` with two `app.state` resolver keys; the
Gateway installs both. Contract version 0.2.3 -> 0.2.4.
- `tools/tool_provenance.py` records host-written plugin provenance for
display (assembly descriptor, Layer-2 context later); no decision reads a
label.
- Plugin providers use a `(config signature, loop key)` cache so a
loop-affine provider is never handed to another loop or to a sync caller.
- RBAC gains `plugin_actions` / `plugin_management` aliases;
`config_version` 47 -> 48 across the three mirrors.
Tests: new plugin target/provenance/action/management suites plus a
blocking-IO anchor whose offloads were reverted to prove red->green.
* fix(authz): decide plugin resources from the request's config snapshot
Review round 1 on PR #5842 (willem-bd).
P1 — the decision layer no longer reloads the configuration. The six
`authz/plugin_authz.py` functions take a required `app_config` snapshot (the
same request-scoped config the Gateway used to resolve the provider), and
`resolve_/aresolve_plugin_authorization` return it as a third element, so
`enabled`/`fail_closed` can never come from a second read that drifted from
the one that produced the provider. A snapshot the caller could not read at
all is *unavailable*, not disabled: it fails closed as
`authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates
an absent `config.yaml` (no gate, today's behavior) from a config that exists
but cannot be read or validated (deny) instead of swallowing both through
`safe_app_config`.
P2 — `_validated_decision` requires `type(decision.allow) is bool` on both
the sync `authorize` and async `aauthorize` paths, so a provider returning
`AuthzDecision(allow="false")` is a malformed verdict that follows
`fail_closed` instead of being read as an allow.
Both fixes are covered by regression tests that were verified red against the
pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases
across fail_closed/fail_open and sync/async), plus tests pinning that an
absent config and `authorization.enabled: false` keep today's behavior.
* fix(authz): treat a request-time config absence as unavailable
Review round 2 on PR #5842 (willem-bd).
P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to
`None` ("no gate"). A Gateway cannot have started without a readable config
(`app/gateway/app.py` reads it through this same accessor in `lifespan`), so
absence *during a request* means the configuration this process was running on
became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is
unavailable, not disabled: the error propagates,
`resolve_/aresolve_plugin_authorization` raise
`_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher
returns 403 and the management resolvers answer `False`. The dead
`(None, None, None)` allow path is gone, the resolvers return a non-optional
`AppConfig` as their third element, and callers key "authorization disabled" on
`provider is None` alone. This matches the rule the decision layer already
documented (`authz.config_unavailable`) and the public docs.
P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the
code to `authz.denied` instead of raising `TypeError` outside the guarded
provider path, which turned a denial into an unhandled error. It is deliberately
not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict
is a valid denial, and treating it as malformed would flip it to an allow under
`fail_closed: false`.
Both fixes are covered by regression tests verified red against the pre-fix
behavior (P1: resolver True vs False, async `PermissionError` not raised, route
200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at
`plugin_authz.py:113`). The action-route tests in
`tests/test_plugin_contributions.py` now install a readable
`authorization.enabled: false` config instead of relying on a config-less
process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the
new rule.
* fix(authz): apply the running policy when the plugin config is unreadable
Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd).
The first attempt read the P1 finding as "absence is never disabled" and failed
every config read closed unconditionally. That is too broad for this repo: a
config-less host is deliberately treated as having no policy to apply
(`_get_route_authorization_config` falls back to a disabled config,
`sandbox_authz.safe_app_config` documents the same no-op for "CI runners and
direct-call tests"), and the rule turned the config-less e2e plugin host
(`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three
`bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally:
`POST /api/plugins/community.bookmarks/actions/save` against that fixture host
returned 403 before, 200 after this change.
The rule is now "has this process loaded a configuration?", not "was the last
read successful?":
- No loaded config (`peek_loaded_app_config()`, new read-only accessor on
`deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the
gate stays a no-op — unchanged for CI runners, direct-call tests and hosts
that mount only the plugins router.
- A loaded config whose file disappears or cannot be read: *unavailable*, never
*disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the
failure flag of the configuration this process runs on (authorization
disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by
default), so an enabled Gateway answers 403/False in the window instead of
letting protected plugin actions and management routes through.
- A config file that exists but cannot be read or validated right now: fail
closed when nothing is loaded (round 1's rule), otherwise the loaded policy
decides.
`peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it
reflects the last successfully loaded config, so a hot-reloaded policy is what
applies during the window, and there is no lifecycle hook to keep in sync.
Evidence: the three no-config/lost-config branches in
`tests/test_plugin_management_guard.py` and
`tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests
unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in
`tests/test_app_config_reload.py`, and the fixture-host probe above. Docs
(`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log
state the final rule; the log records the superseded variant.
* fix(authz): keep a denial whose reasons raise while being read
* fix(authz): intersect plugin batch filters with the candidates
The plugin_page/plugin_management batch helpers returned the provider's
`filter_resources` answer verbatim, so a custom provider could hand the
projections a target that was never asked about ("*", another namespace,
a phantom page). Every other filter call site re-projects from its own
candidate list; do the same here, state the intersection on the provider
contract, and pin it with a regression test that fails on the old
pass-through.
* docs(plugins): map a management denial to 403 in the route example
The contributed-route example called `arequire_plugin_management` and let
the documented `PermissionError` escape. The Gateway installs no
`PermissionError` handler, so a route copied from the example answered 500
on a normal policy denial instead of 403. Both example routes now translate
the denial, and the surrounding bullets say why the translation is the
route's job.
* docs(extensions): point the 0.2.4 contract row at the PR that ships it
Every other row in the version-history table links the pull request that
shipped the version; 0.2.4 linked the authorization tracking issue instead.
Both the English and Chinese tables now link this PR.
|
||
|
yetuge
|
4a92cb579a
|
fix(uploads): keep document-conversion stat and write-back off the event loop (#5830)
* fix(uploads): keep document-conversion stat and write-back off the event loop convert_file_to_markdown offloads the conversion itself for files above 1 MB (documented in its docstring), but the two bookend filesystem calls still ran on the event loop: the size stat before the threshold check and the write of the converted markdown afterwards — the latter a potentially multi-megabyte write in the per-upload ingestion path. Route both through run_file_io (the dedicated file IO pool that also keeps the caller's context), matching the off-the-loop fixes already applied to the sibling upload/document paths. Add a tests/blocking_io anchor: with the converter patched to a 2 MB payload it fails on the previous head (Blockbuster catches the blocking os.stat first; the broad except swallows it into a None return) and passes with this change. * fix(uploads): address review — drain conversion write-back, make run_file_io re-entrant Review follow-up on the off-loop conversion stat/write PR: 1. The write-back now goes through await_drained(): a plain run_file_io await cancels immediately while the worker keeps writing, so the ingestion scope could remove its private directory over an in-flight write and leave a large hidden partial file. The drain delivers the cancellation only after the worker settled, and the handler removes the partial output before re-raising. 2. run_file_io is now re-entrant: a nested offload issued from code already running inside a file-io worker (project-document conversion runs asyncio.run(convert_file_to_markdown(...)) inside a run_file_io worker) executes inline instead of submitting to the same bounded executor, which deadlocked with a single configured worker. Both fixes carry deterministic regressions: the cancellation test pins the drain ordering (cancel issued before the worker is released) and the partial-file cleanup; the re-entrancy test pins inline execution by thread identity. Both fail on the previous head. Full blocking_io + project-document + conversion suites show an identical failure set to the previous head (pre-existing Windows platform failures only). * fix(uploads): keep cancellation authoritative when conversion cleanup fails If the post-cancellation unlink of the staging output raised, the new exception escaped the CancelledError handler, was swallowed by the broad except as an ordinary conversion failure (None return), and left both the caller's cancellation and the partial file behind. Catch and log cleanup failures inside the handler and always re-raise the captured cancellation; a second cancellation still passes through untouched (CancelledError is not an Exception). Red/green: tests/blocking_io/test_file_conversion_offloop.py gains a regression where unlink fails while the write-back is in flight — on the reviewed head the task settled on None (DID NOT RAISE); with the change the CancelledError propagates and the failure is logged. Signed-off-by: yetuge <2219677952@qq.com> --------- Signed-off-by: yetuge <2219677952@qq.com> |
||
|
Zeren Wang
|
577c39643b
|
feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate (#5712)
* feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate Adds TypeSafeGuardrailProvider, a network guardrail provider that asks one `noul` question per probed tool call and denies calls whose Jev risk probability reaches `threshold`. It reuses the existing GuardrailProvider slot: no middleware, config-schema or dependency changes. Semantics that are load-bearing: - state is strict JSON (no `default=`, `allow_nan=False`, string keys only); unserializable arguments, lone surrogates among them, and argument text above `max_state_chars` are denied locally with zero network requests, never truncated and sent. - the async path cancels in-flight requests through `asyncio.timeout`; the sync path checks the deadline after the response headers arrive, around the body read and before accepting the result, and drops late results. No total return-time bound is promised on the sync path. - clients are created per evaluation inside a context manager, and the injected `transport_factory` is called once per network-needing evaluation: httpx closes an injected transport with its client, so a shared instance would be dead on the second call. - denial reason messages carry probability, threshold, served model version and state digest, so the threshold comparison can be replayed from the run journal; local denials record no probability or model version, and allow decisions stay unpersisted (the audit path is unchanged). - provider errors carry no response body or arguments: malformed responses report the field and the offending type only, so a response echoing tool arguments cannot reach middleware logs or the evaluation report. The online evaluation harness (`backend/scripts/eval_typesafe_risk_gate.py` plus its labeled case set) is the threshold/tools evidence that must precede enabling interception. It reports four mutually exclusive populations (uncached network, cache hits, unprobed tools, local refusals), separates model classification from local refusals and fail-closed errors, measures latency from the caller entering `aevaluate` through cleanup, discloses censored samples, probes DNS/TCP/TLS separately, and flags unprobed tools as a coverage gap rather than a pass. Real responses are never written to disk. Cache-pass samples are classified from `decision.metadata['cached']` and counted once, warming calls are retained as an auxiliary group with their failures named in the report and JSON, and an optional connection-probe diagnostic can never discard completed evaluation results. Tests: the provider suite over MockTransport plus the middleware integration, a blocking-IO anchor that drives `aevaluate` against a real loopback HTTP server with a meta-check proving the sync path on the loop trips the gate, and harness cases covering the reporting defects. Every fix above was verified to fail its new tests when reverted. Docs: GUARDRAILS.md gains Option 4 (privacy, local-refusal trade-off, deadline semantics, audit boundary, coverage/bypass risk, Allowlist->RBAC migration, lone-surrogate local refusals and the no-response-body property of error messages) and config.example.yaml gains a smoke-scope example. Config: the new example block bumps `config_version` 46 -> 47, with the Helm chart's embedded version and README example kept in step as `scripts/check_config_version.sh` (chart.yaml CI) requires. * feat(guardrails): whitelist tool calls inside the TypeSafe risk gate Static allow rules now live in the provider instead of waiting on a composite provider. `whitelist` mirrors `AllowlistProvider.allowed_tools`: omitted means "not enforced", `[]` refuses every tool, and a list refuses every tool outside it locally (`typesafe.tool_not_whitelisted`) before state construction, caching and networking, independently of `fail_closed`. Whitelisted tools are still probed and still denied when their risk probability reaches `threshold`, and the whitelist is checked before the probe scope so an unlisted tool cannot inherit an allow from being unprobed. Deny rules stay with `authorization.*`. The policy version moves to 1.1.0, the evaluation script reports whitelist refusals as their own population, and `GUARDRAILS.md` / `config.example.yaml` document the allow-half migration. * fix(guardrails): address TypeSafe risk gate review findings - Status errors report only the numeric status. The HTTP status line is server-controlled like the body, and this message reaches middleware logs and the evaluation report, so a hostile endpoint could echo the state (raw tool arguments) through it. - Expired cache entries are removed with `pop(key, None)`. Sync evaluation runs on executor threads when a turn produces parallel tool calls, so a second caller can remove the same expired entry; a bare `del` raised KeyError, which the middleware reports as a provider error and, under fail_closed, turns into a spurious denial. - The evaluation harness states its acceptance gates inline (no missed dangerous call, system false-block <= 5%, uncached network p95 <= 1s) instead of citing a design spec that is not in the repository, and reports the whitelist-refusal population alongside the other four. - `--tools` / `--whitelist` strip whitespace and drop empty entries, so `--tools "bash, write_file"` no longer probes only bash while treating `" write_file"` as an unknown tool name. - The case set's `padding` note says top-level argument fields only, matching what the runner implements. Each fix was verified by running its new test against the pre-fix code. * refactor(guardrails): name the TypeSafe permission list allowed_tools Align the key with `AllowlistProvider.allowed_tools` so an allowlist migration copies it verbatim instead of translating `allowed_tools` into `whitelist`. The companion names follow: reason code `typesafe.tool_not_allowed`, metadata `tool_not_allowed`, harness population `not_allowed`, and the `--allowed-tools` flag. Semantics are unchanged: omitted = not enforced, `[]` = refuse every tool, tools outside the list are refused locally before any state, request or cache entry, and listed tools still face the risk gate. * fix(guardrails): record only a bounded served model token; fail the eval exit code Two findings from the re-review at d3f0b072. `model` is response content, and `_model_decision` interpolated it verbatim into the reason message -- which reaches the denied ToolMessage, the run journal, middleware logs and the evaluation report on disk, and which the cache retains for replay. A malformed or hostile endpoint can put an echoed tool argument, injected text or a multi-megabyte string there. The token is now recorded as itself only when it matches a conservative shape; anything else is recorded as `unrecorded:sha256:<digest>`, which keeps the verdict and a stable provenance record without echoing the value. Rejecting the response instead would turn an odd version string into a fail-closed denial of every call the endpoint answers. The digest path encodes with `surrogatepass` so an escaped lone surrogate -- a value `json.loads` happily produces -- cannot raise where this must not fail. `scripts/eval_typesafe_risk_gate.py` returned 0 even when a gate failed, so an operator scripting `... && enable` got a green light on a failing evaluation. `main()` now returns 1 when any gate in `report["score"]["gates"]` fails, after the JSON report is written, and prints the failed gate names. Tests: `test_an_unrecordable_served_model_is_digested_instead_of_echoed` (an echoed argument, a space, a lone surrogate and an oversized value, plus the cached replay) and `test_main_exit_code_carries_the_gate_verdict` (both directions). Both fail against the pre-fix source. docs/GUARDRAILS.md audit boundary updated to match. 113 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py. * fix(evals): fail the risk-gate run when a labeled population is uncovered Review finding on dcd38cc3. The new exit code made `_score`'s vacuous passes load-bearing: unprobed calls are excluded from the score, so `risky_misses_zero` is true over an empty risky population. `--tools read_file` excludes all 18 risky cases in the bundled set, so the run printed every gate as PASS and exited 0 after evaluating no risky operation at all. `_score` now gates coverage itself: `risky_coverage_present` and `safe_coverage_present` fail when the probe scope leaves that labeled population with nothing evaluated, so a scope that cannot substantiate the verdict no longer reports one. The safe side had the same vacuity -- `system_false_block_within_target` passes trivially with zero safe cases. Verified end to end against a loopback mock endpoint with the bundled case set: `--tools read_file` exits 1 with `GATE risky_coverage_present: FAIL` while the other four gates still PASS, where dcd38cc3 exits 0 on the same command. Tests: `test_main_fails_when_the_tool_scope_leaves_no_risky_case` and `test_main_fails_when_no_safe_case_is_in_scope` (both fail against dcd38cc3), `test_bundled_case_set_keeps_both_labels_the_coverage_gates_require`, and the exit-code test now runs with both populations present. 116 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py. |
||
|
lihongyuan99
|
54c65d8ef6
|
fix(sandbox): resolve the async acquire lock path off the event loop (#5760)
* fix(sandbox): resolve the async acquire lock path off the event loop _discover_or_create_with_lock_async offloads every step of the acquire — ensure_thread_dirs, the lock file, cache re-checks, backend discovery and registration — except the one line that builds the lock path itself, which resolves through Paths.base_dir and therefore runs a syscall on the loop. The sibling anchor documented that as "a pre-existing blocking call in this coroutine" and stubbed the path layer to get past it. With the resolution in a worker thread the stub is no longer needed, so this removes it and pins the behaviour with an anchor of its own. * test(sandbox): word the offload anchor against the state after the stub removal * test(sandbox): pin that the gate still catches an inline thread_dir call The lock-path anchor only asserts execution reached the lock file, so it would stay green if the gate stopped instrumenting the syscalls behind Path.resolve() and the inline call were restored. Add a meta-check next to the store probe's: call thread_dir() directly on the event loop under the same isolation and assert BlockingError. |
||
|
lihongyuan99
|
c10068fd27
|
fix(channels): resolve the Buzz seen-event store path off the loop (#5759)
_start_channel built the default seen-event store path inline, and
Paths.base_dir resolves through realpath. Channel startup reaches that
wiring from POST /api/channels/{name}/restart, so every restart ran a
filesystem syscall on the Gateway event loop; the rest of the same
function already keeps its disk IO in a worker thread.
The existing wiring test stubs get_paths with a plain-string base_dir,
so it cannot see the syscall. The new anchor keeps the real Paths object
and isolates it with DEER_FLOW_HOME instead.
|
||
|
xiaodu55
|
736b5a4216
|
fix(gateway): keep custom-skill rollback filesystem IO off the event loop (#5751)
* fix(gateway): keep custom-skill rollback filesystem IO off the event loop Closes #5747. Three blocking sites remained on the Gateway event loop in the rollback route and its shared response reader, all offloaded through asyncio.to_thread following the #5729 pattern: - storage.validate_skill_markdown_content() creates a temp dir and writes SKILL.md into it (site 1) - the post-scan read of the file being replaced (site 2, named by #5729's test docstring) - _read_custom_skill_response(): load_skills() walks every skill directory and read_custom_skill() opens SKILL.md — O(installed skills) loop time paid per accepted rollback and per GET /skills/custom/{name} (site 3) A blocking_io regression drives the full accepted-rollback path (past the security scan) under the strict gate; the two pre-existing cases only covered the pre-scan branches. * fix(gateway): keep the skill element type in the response-parts closure Review nit on #5751: tuple[object, ...] erased the element type for the _skill_to_response call; Skill is already imported in this module. |
||
|
lihongyuan99
|
fff7aa6a18
|
fix(skills): rollback reads its history file off the event loop (#5729)
* fix(skills): rollback reads its history off the event loop rollback_custom_skill constructed the user-scoped storage, probed whether the skill exists and parsed custom/.history/<name>.jsonl inline, while the sibling get_custom_skill_history two lines above already offloads exactly those three steps and documents them as "blocking filesystem IO that must stay off the event loop". The history file grows one entry per edit carrying full previous and new content, so a rollback request parsed the whole edit history on the Gateway loop; the strict Blockbuster gate catches the construction as a blocking os.getcwd() through SkillsConfig.get_skills_path() -> project_root(). Offload the three steps through the same worker-thread closure and anchor both pre-scan branches under tests/blocking_io/. * test(skills): address rollback anchor review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
spud
|
8f08df9b48
|
fix(sandbox): fence ambiguous AIO session creation ownership (#5711)
* fix(sandbox): quarantine ambiguous shell session creation * fix(sandbox): quarantine ambiguous bash session creation * fix(sandbox): recycle sandboxes with ambiguous session creates * fix(sandbox): keep quarantined release teardown non-raising * docs(sandbox): keep guidance within instruction budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
lihongyuan99
|
53352287a7
|
fix(skills): install for a user keeps its custom-dir setup off the event loop (#5650)
* fix(skills): install for a user keeps its custom-dir setup off the event loop UserScopedSkillStorage.ainstall_skill_from_archive re-implemented the install pipeline and created the per-user custom directory inline, so the await behind POST /api/skills/install blocked the Gateway loop on os.mkdir. The base class offloads every filesystem phase around its LLM scan — and says so — but the override's setup step sat above that comment and escaped it, and the existing blocking-IO anchor drove only the host-scoped class. Offload the mkdir through the same worker thread and anchor the override itself under the strict Blockbuster gate. * test(blocking-io): pin per-user staging of supporting skill files Review follow-up on #5650: the user-scoped anchor asserted only the staged SKILL.md, so a regression that mis-stages the references/ tree under the per-user root would pass both anchors. Assert the supporting file against the per-user layout as well, mirroring the host-scoped twin. |
||
|
YxinMiracle
|
b8ab097ca2
|
fix(models): support official DeepSeek managed profiles (#5718)
Co-authored-by: YxinMiracle <“939157765@qq.com”> |
||
|
Xuehao Xu
|
8ef58eaa90
|
feat(models): manage shared models from Settings (#5596)
* feat(models): add admin UI for shared model management * docs(gateway): keep model guidance within size budget |
||
|
Wenchao An
|
42334f26d7
|
feat(capabilities): unify catalog, plugin configuration and agent selection (#5497)
* feat(capabilities): unify catalog, plugin configuration and agent selection * fix(capabilities): address review isolation, validation and demo issues * fix(capabilities): preserve concurrent selections and guide launcher repair |
||
|
wd_pan
|
cc27730348
|
feat(memory): add opt-in relevance-aware retrieval ranking (#5251)
* feat(memory): add opt-in relevance-aware retrieval ranking Add a deterministic, network-free lexical relevance strategy for DeerMem (issue #4495): memory_search ranks every fact in scope by idf-weighted token overlap combined with confidence, with optional greedy-MMR diversity against near-duplicate facts; prompt injection ranks facts against the current-turn query threaded from DynamicContextMiddleware through the new optional `query` keyword on MemoryManager.get_context/aget_context. Defaults preserve the legacy confidence-only behavior exactly; no prompt, storage-format, or vector/embedding-dependency changes. Refs #4495 Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): bound relevance retrieval and apply review feedback Bound tokenization and index shared stems, preserve mixed CJK tokens, warm jieba, and align missing confidence with legacy injection. Cache MMR token sets and stop selection at result or injection budgets. Document retrieval-adapter precedence and add regression coverage. Refs #4495. Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): preserve backend compatibility and normalize relevance Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): omit absent query hints and share injection IDF Signed-off-by: pwd11 <fvdsrc@163.com> * test(memory): retain timeout mock until injection worker exits Signed-off-by: pwd11 <fvdsrc@163.com> * docs(agents): drop root guidance compaction Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): validate token prefixes and preserve upload queries --------- Signed-off-by: pwd11 <fvdsrc@163.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Onefly
|
d8d110c637
|
fix(sandbox): prevent AIO subagent session eviction (#5178)
* fix(sandbox): prevent AIO subagent session eviction * fix(sandbox): address PR 5178 review issues * fix(sandbox): handle transient session and metadata failures * fix(sandbox): fence capacity upgrades and validate reused limits * docs(sandbox): restore list indentation and trim guidance * fix(ci): stabilize Buzz persistence test and trim sandbox guidance --------- Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
a58ab484a6
|
feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443)
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md (issue #5160, tracker #5129) in the slice order of the spec's §16. Slices: - A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY admission pinning (both server-owned sets + worker hoist); latest-only request-scoped <project> block via DynamicContextMiddleware wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID prefix + marker + provenance, never persisted); journal audit fingerprints; Instructions tab. - B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with locked check-and-set; hash-qualified immutable shelf storage with Paths helpers; upload/list/content/delete-to-trash routes; project delete trashes the shelf in-transaction; request-scoped bounded <documents> index with honest count/shown + actionable overflow note; list_project_documents/read_project_document tools registered only on pinned runs; PAT allowlist + drift guards; blocking-IO anchors. - C: shared thread-upload ingestion service (uploads router refactored to parity); POST from-thread with provenance; attach-to-thread with lock-staged copy (archived source allowed); read-only thread-files view with per-group truncation reporting. - D: restore (restored/merged/not_found/no_target/content_missing; no file moves), purge (continuous row lock across unlink/delete/commit, retryable on FS errors), retention sweep (lazy + startup, 24h orphan guard, row-side reconciliation never deletes). - E: Documents tab (shelf + conversation-files browser, provenance, archived banner, content-missing rows), /workspace/trash route, sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks + specs. Review hardening folded in (10 rounds, all with tests): - force active shelf content (HTML/XML family) to download; nosniff on artifact + content responses; unified unsandboxed-iframe PDF preview (fixes the pre-existing Chromium sandbox blank in the artifact viewer) - scope document trash to the URL project under the document lock - atomic no-overwrite filename reservation for ALL ingestion (seeded claims + os.link commit with suffix retry; same-name re-upload now unique-names instead of replacing); hidden staging only, no visible placeholders; lease cleanup on setup failure - serialize conversion under the document lock with post-lock active revalidation; drain locked filesystem work on cancellation; preserve bytes when an insert's commit state is uncertain (including trashed rows) - original-integrity checks before serving text or cached conversions; content_missing surfaced in list responses (UI reads the flag, no 409-probe); downloads always serve original bytes - bounded streaming document reads with cached char counts; shelf limits declared in middleware release identity - thread-root confinement for from-thread sources; config fallback rejects fractional/infinite values; composer counts staged attachments; pending attachments persist until submission or removal; in-flight instruction/rename edits survive save refetches; shelf and trash pagination; conversation-file and thread-files pages stay subscribed to refetches Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md contracts, config.example.yaml projects block. Review follow-ups (head b4807477 → this revision): - The trash retention sweep is split so repeated lazy triggers stay bounded: the indexed expiry purge still runs on every trigger (GET /api/trash/documents, POST /api/trash/purge) while the O(all rows + all files) reconciliation is throttled to one run per user per 15 minutes (process-local, per-user window). The startup sweep now runs as a background task instead of blocking gateway readiness, and shutdown awaits it (bounded). - The export scrub (stripInternalMarkers) is fence- and indentation-aware like the render path, so a pasted, fenced <project>/<documents> snippet survives markdown export while real injected blocks (never fenced) are still removed. Fence regexes moved to a dependency-free leaf module to avoid the messages↔streamdown import cycle. - The artifact viewer's PDF iframe no longer carries an added title attribute (the upstream e2e contract locates it via :not([title])), and the upstream artifact-preview spec now pins the new contract: PDFs render unsandboxed, images keep sandbox="". * fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title - Shutdown cancelled only the shield around the background startup sweep, so an all-users reconciliation that outlived the 5s budget kept walking rows and files while the document repo and DB engine were disposed underneath it. The wait now lives in `_shutdown_startup_trash_sweep`, which cancels the task and drains it before worker exit: the shield keeps the wait bounded, the cancel makes it final (CancelledError lands at the sweep's next await, and `_run_startup_trash_sweep` only catches `Exception`, so nothing swallows it). - The browser-preview iframe lost `title={getFileName(filepath)}` in the previous fix round, leaving the PDF frame without an accessible name while its siblings keep theirs. Restore it (WCAG frame titles), assert it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]` instead of `iframe:not([title])`. * fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel `Task.cancel()` returns False when the sweep already finished inside the window between the deadline firing and the cancel, so the shutdown log claimed a cancellation that never happened. Branch on that outcome: the warning stays for a real cancel, a late finish is logged at info, and both paths still reap the task before worker exit. * fix(projects): round-4 review — make Empty trash delete what it confirms `POST /api/trash/purge` only ran the retention sweep, and the sweep's candidate selection is age-gated, so a freshly trashed document survived "Empty trash" even though the confirmation promises that every listed document is permanently deleted. With one trashed row the route answered `{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps expired rows before listing, so the visible rows were normally ineligible for the action by construction. Empty trash now drives `purge_all_trashed`: the caller's trashed rows (`list_all_trashed`, no age filter) each go through the same guarded, row-locked `purge` as the single-document delete — bytes first, then the row, in one transaction — so a row restored mid-flight is skipped instead of force-deleted, and an unlink failure rolls that row back and answers 500 with a retryable message. Retention expiry stays where it was: the sweep's `purge_candidates` is now the only age-gated selection, and the lazy retention sweep still runs on the listing and at startup. Tests: the router suite replaces the retention-gated expectation with the reviewer's repro (fresh row purged, bytes unlinked, shelf and other users' trash untouched, a failing unlink stays retryable and 500); a blocking-I/O anchor drives the new entry point through the offload; the mocked e2e covers the action end to end; a new real-backend spec performs it against the real gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE and the phase-2 design docs (en+zh) state the age-independent contract. |
||
|
hataa
|
26800d1245
|
fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 (#5225)
* fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 * fix(channels): address WeCom APPID, decompression, and log-sanitization review findings (#5223) Round-4 review follow-ups on the inbound-media download cap: - The COS bucket numeric suffix is the owner's Tencent Cloud APPID and bucket names are user-chosen, so any Tencent Cloud account could register a matching ww-aibot-img-* bucket and pass the shape gate. The built-in rule now admits only the APPID observed in Tencent's published aibot callback examples (1258476243), across regions; any other account (including a future WeCom rotation) goes through channels.wecom.allowed_media_hosts. - aiter_bytes() transparently decodes Content-Encoding, and the decoder allocates the full decompressed body before the byte cap sees a chunk (an ~8 KB gzip wire chunk decoding to 8 MiB reproduces it). Both URL readers now send Accept-Encoding: identity, refuse a response with a residual Content-Encoding before reading, and iterate aiter_raw(). - httpx.HTTPStatusError formats the signed URL (path + query credentials) into its message, so _ingest_inbound_files' reader-failure branch logs a sanitized summary (class + status) instead of logger.exception, and the WeChat extract paths catch httpx.HTTPError so the polling loop's per-message logger.exception can never render a media URL. Every change ships with a red/green regression: the reviewer's 403 mock-transport repro asserted against caplog.text (fully formatted logs), the reviewer's different-APPID bucket host, and gzip bombs driven through real httpx mock transports in both readers. Docs (channels AGENTS.md, README, config.example.yaml) updated for the APPID pinning and encoding gate. * docs(channels): document why the inbound-media cap is 50 MB, not WeCom's 100 MB ceiling * fix(logging): redact URLs in httpx request logs down to scheme + host, fixes #5223 httpx emits 'HTTP Request: GET <full URL>' at the Gateway's INFO level before any response handling runs, so even successful signed-media downloads leaked their credentials. HttpxUrlQueryRedactionFilter (installed by configure_logging) rewrites those records in place — path and query become /<redacted>, method/status/duration observability is preserved — which also keeps Telegram's token-bearing Bot API paths out of the logs. Reader-level regression tests run at production INFO level with a real MockTransport, success paths included. * fix(logging): blank userinfo credentials in httpx request-log redaction * fix(logging): redact authority-only URLs and cover urllib3 redirect logs Two follow-ups from the review plus one extrapolation of the same class: - rest is now optional in _URL_REDACT_RE, so an authority-only URL (scheme://user:pass@host, no path) is rewritten too — userinfo had nowhere else to hide and previously passed through verbatim. A bare credential-free origin still passes through unchanged. - Renamed to UrlRedactionFilter / install_url_log_redaction and attached to the urllib3 logger as well: urllib3 logs 'Redirecting <url> -> <url>' at INFO with full URLs on both sides, the same leak class on a different library logger. No gateway path today both uses requests and redirects a signed URL, but the class stays closed instead of dormant. - Unit tests now build records with the real httpx 0.28.1 format string ('HTTP Request: %s %s "%s %d %s"', 5 args) and httpx.URL args, per the nit, instead of a synthetic shape httpx never emits. * fix(logging): install URL redaction at handler level so propagated records are covered A logging.Filter on a logger only runs for records emitted through that exact logger — child loggers neither inherit it nor trigger it on propagation — so the previous attachment to the bare urllib3 logger was dead code: urllib3 emits Redirecting via urllib3.poolmanager at INFO and urllib3.connectionpool at DEBUG. The filter is now attached to every root handler (mirroring _install_trace_filter, which already iterates root handlers; handler-level filters see propagated records) in addition to the httpx logger (httpx emits via the bare name, and emission-point coverage survives handlers added later). The wiring is pinned by tests that emit through the real urllib3 child loggers — a mutation removing the handler-level install turns them red. Comments, docstrings, and AGENTS.md now state the actual emitter names and levels. * fix(logging): redact urllib3 DEBUG request lines, whose split shape evaded the URL regex urllib3's per-request line (connectionpool.py:545 on 2.7.0) renders as `scheme://host:port "METHOD /path?query HTTP/x.x" status len` — the authority ends at a space so _URL_REDACT_RE's bare-origin early return applies, and the quoted origin-form target has no scheme, so neither half was rewritten. UrlRedactionFilter now runs a dedicated request-line shape first (collapsing the target to /<redacted>, keeping scheme+host+method+ version), then the absolute-URL pass. Regressions pin the exact format string both at unit level and through the real urllib3.connectionpool DEBUG emit path; AGENTS.md wording now names both covered DEBUG shapes. * fix(logging): redact urllib3 retry lines and linearize scheme scanning Closes the two open review threads on the inbound-media log hardening: Retry/redirect targets: urllib3 logs the request target with no scheme in five shapes the generic absolute-URL pass cannot see - `Retry: <target>` (connectionpool.py:954 DEBUG), `Incremented Retry for (url='<target>')` (util/retry.py:545 DEBUG, absolute on the redirect path), `Retrying (...) after connection broken by '<err>': <target>` (connectionpool.py:869 WARNING, above the INFO root), and origin-form halves of both Redirecting emitters (poolmanager.py:500 INFO / connectionpool.py:922 DEBUG). Each gets a rewrite anchored to the exact urllib3 format, collapsing the target to /<redacted>; the generic pass's rest now stops at quote characters so a quoted URL keeps its closing punctuation (previously the absolute-form increment line was mangled), and the request-line method class accepts any case. The emitter enumeration in channels AGENTS.md is closed against the installed urllib3 2.7.0 source. Quadratic scanning: both scheme-bearing patterns start with a character class, so re.sub retried every suffix of a long token - 64K paths cost ~1.8s and URL-free 64K error bodies ~3.1s per record, synchronously in every root handler. The two passes are now driven from literal "://" occurrences: _scheme_starts walks back over the scheme charset to each run's first letter and the pattern is attempted only there, reproducing re.sub's leftmost-non-overlapping result in linear time (256K path: 5.6ms; worst adversarial shapes <= 28ms). Long-input regressions pin the URL-bearing and URL-free cases with mutation-verified bounds, plus nested-scheme and digit-headed-run equivalence cases. Validation: tests/test_logging_config.py 12/12; scheme-pass equivalence against the old re.sub pipeline verified by two independent 30k+ case fuzz runs; full-suite A/B against HEAD shows zero tests that pass on HEAD and fail with this diff. * fix(logging): boundary-aware quote stops and whole-message Redirecting anchor Two follow-ups on the urllib3 redaction shapes: Embedded quotes: `rest` treated ANY quote as a closing mark, so a URL with an apostrophe in the path kept everything after it verbatim (`https://h/path'quoted'?token=Q` rendered the credential suffix in full) while the class docstring claimed path/query/fragment are replaced. A quote now closes `rest` only at a boundary - followed by whitespace, a closing parenthesis, or end of string - so urllib3's Incremented Retry (url='...') scaffolding keeps its ') closer while an embedded quote stays consumed. The increment line's url capture gets the same rule narrowed to its fixed ')' closer. Redirecting anchoring: the origin-half pass matched `(? <=-> )/path` as a substring, and an `-> /path` arrow is not urllib3-owned shape - the sandbox provider's actionable mount error (`sandbox.mounts entry <host> -> /mnt/knowledge ignored: ...`) had its container path rewritten to /<redacted>, failing test_setup_path_mappings_logs_actionable_error_for_missing_host_path on CI (backend-unit-tests shard 3). The pass is now anchored to the whole `Redirecting <t> -> <t>` message, which is exactly urllib3's record; origin slots collapse, absolute slots stay for the generic pass. Regression tests pin the embedded-quote shapes and the sandbox error's byte-for-byte passthrough; both mutations verified red. Validation: tests/test_logging_config.py 14/14; the CI-failing sandbox test green locally; every test file asserting redaction/arrow log content passes (attachments, support bundle, run metadata, skill secrets, ragflow, skillscan, sandbox provider); full offline backend suite 14084 passed / 164 failed with the failure set matching this machine's documented Windows-environment baseline (NTFS chmod/symlink, docker/lark/langfuse absences) - no failure involves redaction output. * fix(logging): redact redirects with spaced locations * fix(logging): grammar-complete Redirecting anchor; neutral WeChat guard labels Round-13 P3 (Redirecting anchor strictness): the whole-message anchor kept the ^Redirecting prefix (the urllib3-owned literal that stops the sandbox false positive) but required BOTH slots whitespace-free, so a Location header with an interior space voided the pass and leaked the origin-form request target in the first slot - redirect_location is the raw header string and interior spaces are legal field syntax. The tail is now loose (\S.*$) and the first slot gets the same grammar treatment (\S.*?): the recursive urlopen frame passes the previous raw Location as its url, so t1 can carry interior spaces too, lazy-split at the first arrow the way the line is constructed. A space-carrying slot collapses whole when it starts with /; the sandbox mount error keeps passing through untouched. Round-14 nit (None conflation): _download_cdn_bytes returns None for two reasons (in-flight cap abort, Content-Encoding refusal) but both image and file callers labeled it "exceeds size limit (N bytes)" - contradicting the accurate encoding line right above it, and reporting the plaintext limit for a ciphertext-cap decision. Callers now log a neutral "skipped by download guard" line (the manager reader callers' shape); the accurate reason stays inside the download function. The same sweep also logs _stage_downloaded_file's silent None (no state dir configured), which made an attachment vanish with no log line at all. Also anchors the emitter-enumeration closure to its urllib3 version: the closure reopens if an upgrade changes these format strings, so the comment now says so explicitly. Validation: logging 15/15 and attachments 60/62 (the two pre-existing Windows symlink-privilege failures documented in the PR body); three mutations verified red (old wording, strict t1, silent staging None); ruff clean. Full offline suite run before push (per round-11 lesson). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zhengcy05
|
81f2015fe6
|
fix(runtime): keep agent construction off event loop (#5217)
* fix(runtime): keep agent construction off event loop * fix: - offload checkpoint state accessor graph construction to a worker thread - update test * import AsyncKeyedLockTable * update Agents.md * fix: update test * fix: preserve single-flight builds after cancellation --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Shxiao
|
a06688ad82
|
test(skills): skip the skill-export capture suite where fd-based walking is unavailable (#5372)
* test(skills): skip the skill-export capture suite where fd-based walking is unavailable export._capture intentionally rejects every export with 422 skill_export_unsupported on platforms without os.O_NOFOLLOW and dir_fd directory walking (Windows), so the 40 capture-dependent tests in test_skill_export.py and test_gateway_skill_export.py fail there. Add a requires_safe_capture mark mirroring the production feature guard and apply it to those tests; the 21 tests that only exercise the zip installer, export lock, or router plumbing keep running. * test(skills): share requires_safe_capture and cover the blocking-io export suite Move the mirror of export._capture's platform guard into tests/support/skill_export_platform.py so the per-file copies cannot drift, and apply it to blocking_io/test_skill_export.py::test_export_worker_and_response_are_off_loop, which drives the real exporter and hits the same 422 skill_export_unsupported guard on Windows (the blocking-io suite is excluded from the default make test run). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
bc4a33aba7
|
fix(skills): stop persisting resolved secrets when toggling skills (#5357)
Toggling a skill wrote resolved secrets into extensions_config.json. The Gateway skill toggle and DeerFlowClient.update_skill loaded the file with ExtensionsConfig.from_file(), which replaces every "$VAR" string with the environment value (and an unset variable with ""), then serialized that model back through to_file_dict(). A "$GITHUB_TOKEN" reference was persisted as the plaintext token, and an unset reference was erased for good. DeerFlowClient.update_mcp_config had the same flaw for every key other than mcpServers. Every writer now does a raw read-modify-write, the way the MCP router already did: read_raw_extensions_config reads the on-disk JSON, set_raw_skill_enabled changes only the target entry, and validate_raw_extensions_config checks the candidate the way the runtime will load it before the atomic write. When the file does not exist yet, the Gateway seeds it with the cached skill states only, never the resolved cached model. The MCP router's raw loader and candidate validation delegate to the same helpers, so the rule lives in one place, and to_file_dict() is removed so the unsafe serialization has no entry point left. Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Shxiao
|
572744975d
|
fix(tools): run tool assembly off-loop at async entry points (#5224)
* fix(tools): run tool assembly off-loop at async entry points get_available_tools() may block on MCP cache initialization while it is called on async agent-assembly paths (task_tool, durable batch execution), stalling the calling event loop for the full discovery duration. Dispatch the (unchanged, synchronous) assembly call to a worker thread via asyncio.to_thread at the two async entry points so the loop keeps processing requests, SSE frames, cancellations, and timers. Fixes #5172 * fix(tools): offload lead-agent assembly off-loop and pin with blocking-io anchors Review follow-up for #5224: - run_agent now dispatches agent_factory(...) through asyncio.to_thread, so lead-agent assembly (including both get_available_tools call sites in _assemble_lead_agent) runs off the event loop — the Gateway headline scenario from issue #5172. - _ensure_sync_invocable_tool takes a double-checked threading.Lock, making the in-place tool.func wrap on the shared tool singletons explicitly single-shot now that assembly can run concurrently on worker threads. - Add backend/tests/blocking_io/test_tool_assembly_offloop.py: blocking-probe anchors for task_tool and SubagentBatchService._execute_item under the strict Blockbuster gate, plus a meta-check proving the gate trips on the exact syscall class (ExtensionsConfig.from_file on the loop). Verified the anchor goes red when the offload is flattened back to a plain call. * fix(gateway): build checkpoint state accessor off-loop; anchor run_agent offload Review follow-up for #5224: - Add abuild_checkpoint_state_accessor (asyncio.to_thread around the unchanged sync builder) and switch every async call site to it: the stateless_wait route, thread_runs, both threads call sites, and the build_thread_checkpoint_state_accessor boundary. The agent-factory assembly re-enters get_available_tools() and may block on MCP cache initialization; repeat calls hit _state_accessor_graph_cache and only pay the thread hop. - Add a third blocking-io anchor driving the real run_agent with minimal RunManager/bridge stubs; the factory performs a real production blocking read (ExtensionsConfig.from_file()) and the test asserts assembly never runs on the main thread. Verified the anchor goes red when the run_agent offload is flattened back to a plain call. - Adapt the test_threads_router checkpoint-builder patch sites to the new async name. * refactor(tools): carry assembly offloads on a dedicated bounded pool Review follow-up for #5224: - Add utils/assembly_io.py: a dedicated ThreadPoolExecutor (default 8 workers, DEER_FLOW_ASSEMBLY_WORKERS-overridable, mirroring utils/file_io.py and tools/sync.py) with run_assembly(), which copies contextvars explicitly. A hung stdio MCP server parks its worker for the full MCP timeout; carrying assembly hops on the loop's default executor would let a few parked assemblies queue every other to_thread/run_in_executor(None, ...) caller behind them. - Switch all four offloads (run_agent, task_tool, batch _execute_item, abuild_checkpoint_state_accessor) to run_assembly(). - State the cold-path behavior in the accessor docstring: the graph cache validates factory identity, so non-identity-stable factories may duplicate lead-agent assembly across concurrent readers (MCP discovery stays process-wide single-flight); the pool bounds the duplicates. - Add a fourth blocking-io anchor driving build_thread_checkpoint_state_ accessor with a per-resolution fresh factory (always a cache miss) and the real production blocking read; enumerate all four offloads in the gate's module docstring. Verified the anchor goes red when abuild_checkpoint_state_accessor is flattened back to a plain call. * fix(subagents): revalidate batch item before launch; make assembly pool observable Review follow-up for #5224: - _execute_item() revalidates the durable state right after assembly and before executor.execute_async(): renew_item_lease() returns valid=False when cancel_batch() terminalized the item or the lease was lost while assembly was parked, and the launch is skipped (the canceller already finalized the item). Previously the launch was unconditional and the poll loop's cancellation checks only started after execution began. - Regression test driving the real SQLite repository: a blocking assembly probe parks _execute_item, cancel_batch() lands, and the launch is skipped with the item staying cancelled. Verified the test goes red when the revalidation is removed. - run_assembly() tracks pending assemblies and logs a throttled WARNING once the pending count exceeds the worker count, so assembly starvation (workers parked on a hung MCP server) is distinguishable from idle. - The run_agent blocking-io anchor now binds a sentinel extension snapshot via ctx.extensions and asserts the factory observed it through get_agent_build_extensions(), pinning run_assembly()'s ContextVar propagation. Verified red when ctx.run is dropped. - Document the assembly pool in backend/AGENTS.md. * fix(utils): decrement the assembly pending count on the pool thread The pending-assembly counter behind the starvation warning decremented from the asyncio future's done callback, which never fires once the submitting loop is closed while its worker is still running: the count ratcheted up permanently and eventually fired the starvation warning with no starvation behind it (reproduced at 97dc9bec by review). Decrement instead from the dispatched work item: run_assembly() wraps func so a finally drops the count under the pending lock on the pool thread, and the done callback is gone. Pin the counter with tests/test_assembly_io.py: a healthy call returns the count to zero, and an abandoned loop (stopped while the worker is parked) does not wedge it — the abandoned case goes red against the old done-callback decrement. * docs(utils): fix the pending-counter comment after the decrement move The comment still described the removed done-callback decrement, contradicting _work()'s own comment; state the actual mechanism (increment on the loop before dispatch, decrement from the dispatched work item's finally on a pool thread). * test(gateway): retarget checkpoint-accessor stubs to the services seam thread_runs and runs now call abuild_checkpoint_state_accessor, so the upstream wait-reader, regenerate-prepare, and idempotency tests must stub the sync builder where abuild resolves it (app.gateway.services); stubbing the removed router re-exports fails with AttributeError at setup. The async seam semantics are unchanged: run_assembly invokes the stubbed sync builder off-loop and propagates its return values and exceptions. Move the agent/tool assembly off-load note from backend/AGENTS.md to deerflow/utils/AGENTS.md (next to assembly_io.py) so the effective instruction chain for agents/middlewares no longer grows past the AG002 hard limit. * fix(runtime): serialize same-key accessor assembly and release queued-cancel slots Address the three review follow-ups on the assembly off-load: - assembly_io: a job cancelled while still queued never runs its work item, so the dispatched finally never fired and _pending_assemblies stayed elevated until a false starvation warning. Exactly-once cleanup now rides the concurrent future's cancelled() state — cancel() only succeeds before the executor starts the item, so cancelled() is true precisely when the finally will never run — plus a submit-failure release; the one-worker queued-cancellation case is pinned red/green. - services: overlapping cold readers sharing one cache key could both run full agent assembly. _state_accessor_graph now serializes per key through a thread-side KeyedLockTable (pool threads, no running loop) and re-validates factory/app-config identity under the lock, so the factory runs exactly once while identity changes still rebuild. Cache dict access is lock-guarded now that construction runs off-loop. - guidance inventory: register deerflow/utils/AGENTS.md in EXPECTED_GUIDANCE_PATHS so test_repository_has_the_approved_scoped_ guidance_shape matches the relocated assembly note (CI shard 4). * test(keyed-lock): pin KeyedLockTable reclamation and waiter bypass directly Thread-side counterparts of the async table's own tests: overlapping hold() calls serialize (a late arrival joins the live entry instead of creating a second lock that bypasses a queued waiter), the last check-in pops the entry, and many unique keys leave the registry empty. Both regressions verified red — popping unconditionally trips the late-arrival test, never reclaiming trips the many-keys test. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_Feng
|
f52818fe5e
|
feat(skills): export custom skill packages with revision-bound preview (#5332)
* feat(skills): export custom skill packages with revision preview * docs(gateway): keep export guidance within size budget * ci: retry checks after transient uv setup download failure * docs: focus skill export agent guidance on maintenance invariants * fix(skills): handle export disconnects and bound archive transfers * docs(gateway): remove redundant export guidance to fit merged budget * fix(skills): reset export idle deadline after transfer progress |
||
|
Wenchao An
|
0b3dadbc9b
|
feat(subagents): add acceptance checks to durable batch items (#5289)
* feat(subagents): check and persist durable batch acceptance Carry optional per-item criteria into native subagents, reuse the deterministic checker, and expose separate verdicts through item queries and exports. Preserve execution and retry semantics, renew leases during checks, and migrate existing batch rows with nullable acceptance fields. * fix(subagents): align batch acceptance normalization and sandbox admission * test(auth): include project permissions in the full-stack contract |
||
|
RongJie G
|
99367100fb
|
fix(persistence): preserve rollback across the incarnation migration (#5219)
* fix(persistence): tolerate thread incarnation migration * docs(persistence): pin forward revision contract --------- Co-authored-by: CorgiBoyG <CorgiBoyG@users.noreply.github.com> |
||
|
Jun
|
d1f1c49dcd
|
fix(workspace-changes): avoid draining metadata scans on cancellation (#5234)
* fix(workspace-changes): keep metadata cancellation responsive * test(workspace-changes): cover metadata cancellation latency * docs(workspace-changes): document cancellation ownership * style(workspace-changes): format cancellation regressions * docs(workspace-changes): remove unapproved nested guidance * fix(workspace-changes): log only real cancellation drains * style(workspace-changes): apply repository ruff format * docs(harness): record workspace scan cancellation ownership * docs: compact harness guidance below inherited size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jun
|
d7afdbf9a3
|
fix(workspace-changes): drain snapshot scan before cancellation cleanup (#5232)
* fix(workspace-changes): drain cancelled snapshot scans * test(workspace-changes): cover cancellation during snapshot scan * fix(workspace-changes): consume drained scan outcome * test(workspace-changes): keep scan cancellation regression focused * test(workspace-changes): pin cleanup ownership under recancel |