diff --git a/CHANGELOG.md b/CHANGELOG.md index 50bdcade42..44d65fd3f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1397,6 +1397,12 @@ As of 2026-10-21, the milestone includes **829 merged pull requests**. ### Fixed +- **sandbox:** The AIO provider's async acquire no longer checks the thread's + skill directory on the event loop. Picking the sandbox ID ran a filesystem + `stat` and path resolution inline on every thread acquire, which stalls the + loop when `DEER_FLOW_HOME` is on slow or network storage. It now runs in a + worker like the rest of acquire. ([#7507]) + - **threads:** `POST /api/threads` now rejects an `assistant_id` that cannot name an agent with `422`. It used to store the value, and every later read of the thread (`GET` thread, state, history) answered @@ -13084,3 +13090,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#7410]: https://github.com/bytedance/deer-flow/pull/7410 [#7445]: https://github.com/bytedance/deer-flow/pull/7445 [#7449]: https://github.com/bytedance/deer-flow/pull/7449 +[#7507]: https://github.com/bytedance/deer-flow/pull/7507 diff --git a/CHANGELOG_zh.md b/CHANGELOG_zh.md index 0228124e5b..315bbb7d06 100644 --- a/CHANGELOG_zh.md +++ b/CHANGELOG_zh.md @@ -1112,6 +1112,11 @@ ### 修复 +- **沙箱：** AIO 提供方的异步 acquire 不再在事件循环上检查线程的技能目录。此前每次 + 线程 acquire 选择沙箱 ID 时，都会在事件循环上直接执行文件系统 `stat` 和路径解析； + `DEER_FLOW_HOME` 位于较慢或网络存储上时会阻塞事件循环。现在这一步与 acquire 的 + 其他步骤一样在工作线程中执行。([#7507]) + - **线程：** `POST /api/threads` 现在会以 `422` 拒绝无法对应任何智能体的 `assistant_id`。此前该值会被直接保存，之后读取该线程（`GET` 线程、状态、历史） 都会返回 `500 Internal Server Error`，直到删除线程。修复前已这样保存的线程 @@ -10445,3 +10450,4 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子 [#7410]: https://github.com/bytedance/deer-flow/pull/7410 [#7445]: https://github.com/bytedance/deer-flow/pull/7445 [#7449]: https://github.com/bytedance/deer-flow/pull/7449 +[#7507]: https://github.com/bytedance/deer-flow/pull/7507 diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py index aba3e2763d..b12875ad99 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py @@ -2319,8 +2319,15 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): if cached_id is not None: return cached_id - # Deterministic ID for thread-specific, random for anonymous - sandbox_id = self._sandbox_id_for_thread(thread_id, user_id) + # Deterministic ID for thread-specific, random for anonymous. Deriving + # it checks the thread's skills_view on disk through Paths.base_dir, so + # it runs in a worker like the layers around it. + sandbox_id = await _run_started_acquire_worker( + self._acquire_worker_executor, + self._sandbox_id_for_thread, + thread_id, + user_id, + ) if thread_id: key = self._thread_key(thread_id, user_id) with self._lock: diff --git a/backend/packages/harness/deerflow/sandbox/AGENTS.md b/backend/packages/harness/deerflow/sandbox/AGENTS.md index f067361b17..a80dcb036f 100644 --- a/backend/packages/harness/deerflow/sandbox/AGENTS.md +++ b/backend/packages/harness/deerflow/sandbox/AGENTS.md @@ -29,7 +29,7 @@ cancellation before propagating it. **Interface**: `Sandbox`: `execute_command(command, env=None)`, additive `execute_command_in_scope(..., scope_id=...)` / `release_command_scope(scope_id)`, `read_file`, `write_file`, `list_dir`, `glob`, `grep`. Scoped hooks default to pass-through without server-side sessions, preserving third-party subclasses. `grep` accepts a text file or directory tree. Per-call `env` injects secrets: `LocalSandbox` merges into the subprocess environment; `AioSandbox` uses fresh `bash.exec(env=...)` sessions. `list_dir`: missing path → `FileNotFoundError`; command/client failure → `OSError`, never `[]` (`ls_tool`: `(empty)`). Remote `glob`/`grep` share it via `sandbox/remote_search.py`: missing root → `FileNotFoundError`, failed search → `OSError`; only a genuine no-match returns `[]`. The parser takes the command's `limit` and reports `truncated` when output passed it, which providers return after Python-side filtering; tools call an empty truncated result incomplete. Remote `grep(glob=...)` scopes like `glob()` (root-relative `path_matches`), never by basename alone. Remotes use `sandbox/remote_list_dir.py`: capture `find`'s status, not `| head`'s (`sh -lc` lacks `pipefail`); missing binary → `OSError`; truncation SIGPIPE → success. **Bounded downloads**: `sandbox/download.py` owns `DEFAULT_DOWNLOAD_LIMIT` (100 MiB) and `MAX_IMAGE_BYTES` (20 MiB), shared by providers and vision readers. `download_file(path, max_bytes=...)` enforces the smaller of the caller and provider limits during reading. Streaming readers close on every exit and discard the first overflowing chunk without consuming later chunks; SDK chunk sizing still determines over-read. Local/BoxLite readers probe one extra byte. Oversize is `OSError(EFBIG)`, never a successful prefix. Explicit E2B limits require streaming; custom providers must implement the keyword for vision callers, which fail closed without an unbounded retry. AIO holds its transfer lock through stream cleanup so `close()` cannot close the shared HTTP client mid-read. Keep cancellation draining before lease release. Tests: `test_image_download_limit.py` and image cancellation tests. -**Provider Pattern**: `SandboxProvider`: `acquire`, `acquire_async`, `get`, `get_scoped`, `release`. `get_scoped` checks the identity-scoped cache without blocking; a miss uses canonical `acquire(user_id, thread_id)`, never a checkpoint id. AIO reads active identity maps. Async hooks keep Docker creation/discovery, cross-process locking, readiness checks, and release off-loop. Set `supports_agent_skill_isolation=True` only if every tool enforces explicit lead Agent policy: prepared thread-root bind mounts plus upload `sync_agent_skills`. Host-backed providers report false when enabled shells bypass path mappings; middleware rejects them before explicit-policy acquire. +**Provider Pattern**: `SandboxProvider`: `acquire`, `acquire_async`, `get`, `get_scoped`, `release`. `get_scoped` checks the identity-scoped cache without blocking; a miss uses canonical `acquire(user_id, thread_id)`, never a checkpoint id. AIO reads active identity maps. Async hooks keep Docker creation/discovery, sandbox ID derivation, cross-process locking, readiness checks, and release off-loop. Set `supports_agent_skill_isolation=True` only if every tool enforces explicit lead Agent policy: prepared thread-root bind mounts plus upload `sync_agent_skills`. Host-backed providers report false when enabled shells bypass path mappings; middleware rejects them before explicit-policy acquire. **Shared components** (RFC #4741): remote IDs use `derive_sandbox_scope_token` (`sandbox/identity.py`); preserve its keyword-only SHA-256/16-hex contract to avoid orphaning containers. `AcquireSerializer` (`sandbox/acquire_serialization.py`) serializes selected acquire/release transitions with a bounded, refcounted per-key `threading.Lock` table and dedicated bounded executor (no event-loop/default-executor blocking). Workers own cancellation cleanup without waiting for cancelled tasks to resume; provider `shutdown()`/`reset()` calls idempotent `close()`. Keys: AIO `(user_id, thread_id)`, E2B `(user_id, thread_id, skills_root)`, BoxLite/Tenki/OpenSandbox derived id. Random-UUID `thread_id=None` acquires bypass serialization. **Execution leases** (`sandbox/lease.py`, #5128): cross-instance ownership selects which Gateway may reap a container; process-local `SandboxLeaseManager` tracks a client's concurrent lead, subagent, Gateway-request, and channel-upload users. Runs have ephemeral owners. Restore persisted sandboxes only after `get_scoped(user_id, thread_id)` succeeds; otherwise canonical `acquire` replaces the id. Last holder performs pending `SandboxProvider.release`. External run/state boundaries reject server-owned `sandbox`. Outer lifecycle fences repeat idempotent release after each graph/tool/request batch; per-tool terminal `Command` wrappers never release while siblings run. Fork-restored children and upload syncs hold without requesting release: server-created forks may borrow the parent client; ordinary checkpoint ids stay identity-scoped. Serialize persisted lookup and retention by `(user_id, thread_id)`; stale or mismatched bindings reacquire, and a post-acquire miss rolls back then raises. Provider I/O runs outside the metadata lock. Repeated cancellation cannot interrupt acquire/rollback/release reconciliation or let `to_thread` work outlive its execution/request holder; failures are logged without replacing the original cancellation. Lease/scope context IDs are server-owned: Gateway/worker scrub caller values; only internal subagents assign task IDs. Managers key by provider object identity (not hash/equality), so unhashable custom providers work. Subagent owners double as `sandbox_command_scope_id`: AIO gives each scope an ordered persistent shell, replaces it after `ErrorObservation` or a missing-session 404, and cleans it on lease release. Revalidate registry identity after each scope-lock wait so queued commands cannot resurrect released sessions. Env-bearing commands use fresh `bash.exec` sessions so secrets do not persist. **Authorization gate** (`sandbox:execute`, RFC #4063 Phase 3): every sandbox-backed tool call passes through the gate in `deerflow/authz/sandbox_authz.py` - a binary `authorize(principal, "sandbox", "execute", target="*")` check before either reusing a persisted sandbox id or calling `provider.acquire`. Rechecking reuse is required because authorization config and user roles can change while the sandbox remains cached. Sync tool invocations call `authorize_sandbox_execution`; async tool invocations await `authorize_sandbox_execution_async` exactly once. A task-local `ContextVar` scopes that single decision across the complete composed tool invocation, including `ReadBeforeWriteMiddleware`'s pre-write inspection, tool body, and post-read mark; the value is copied into `asyncio.to_thread` workers. Authorization denial is converted to the normal error `ToolMessage` at the composed middleware boundary and is explicitly excluded from the gate's generic fail-open handlers. Async config loading and provider class discovery/import are offloaded before `aauthorize()` so reused sandbox calls do not hash config files or import custom modules on the event loop; provider construction remains on the running event loop because async providers may initialize loop-affine clients. The gate lives at the single tool initialization entry point (`ensure_sandbox_initialized` / `ensure_sandbox_initialized_async` in `tools.py`), while `SandboxMiddleware.before_agent` / `abefore_agent` apply the matching sync/async check to eager acquisition. Deny raises `SandboxAuthorizationError` (`sandbox/exceptions.py`), which propagates out of ordinary tool execution as a friendly error `ToolMessage` ("sandbox execution is not permitted for your role") - the eager path catches it and skips acquisition instead, deferring the deny to the first sandbox-touching tool call so both paths share the same semantics. Provider errors (authorization calls and provider resolution) follow `authorization.fail_closed` / `fail_open`; no readable `config.yaml` or `authorization.enabled: false` makes the gate a no-op (`safe_app_config` tolerates missing config). Gateway upload/artifact sync calls `try_acquire_sandbox_for_request` (`app/gateway/authz.py`), which gates, returns a request lease, and skips sync on deny while preserving the primary operation. Callers release after their last sandbox operation; artifacts request normal parking, uploads do not. Tests: `tests/test_sandbox_authorization.py` and `tests/blocking_io/test_sandbox_authorization.py`. diff --git a/backend/tests/blocking_io/test_aio_sandbox_get.py b/backend/tests/blocking_io/test_aio_sandbox_get.py index 40ecd126af..2e5403406a 100644 --- a/backend/tests/blocking_io/test_aio_sandbox_get.py +++ b/backend/tests/blocking_io/test_aio_sandbox_get.py @@ -19,6 +19,7 @@ If ownership work is put back on this path, this test fails. from __future__ import annotations +import asyncio import threading from pathlib import Path from unittest.mock import MagicMock @@ -199,3 +200,66 @@ async def test_blocking_probe_thread_dir_actually_trips_the_gate(tmp_path, monke with pytest.raises(BlockingError): get_paths().thread_dir("aio-sandbox-lock-wait") + + +@pytest.mark.parametrize("skills_view", [False, True], ids=["default-id", "policy-scoped-id"]) +async def test_async_acquire_derives_sandbox_id_off_the_loop(tmp_path, monkeypatch, skills_view) -> None: + """Deriving the sandbox ID checks the thread's skills_view on disk. + + ``_sandbox_id_for_thread`` stats ``skills_view`` through ``Paths.base_dir`` + to pick the policy-scoped or the historic deterministic ID. Every layer + around it in ``_acquire_internal_async`` already runs in a worker; the ID + derivation ran inline and blocked the loop on each thread acquire. + """ + from concurrent.futures import ThreadPoolExecutor + + from deerflow.config.paths import get_paths + + provider = _make_provider(tmp_path) + provider._acquire_worker_executor = ThreadPoolExecutor(thread_name_prefix="aio-sandbox-owned-worker") + monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path)) + monkeypatch.setattr("deerflow.config.paths._paths", None) + # Already offloaded with asyncio.to_thread; stubbed so the anchor does not + # build a real skills projection. + monkeypatch.setattr(provider, "_ensure_skills_projection", lambda user_id: None) + if skills_view: + await asyncio.to_thread(lambda: get_paths().thread_skills_view_dir("t-id", user_id="u1").mkdir(parents=True)) + expected_id = await asyncio.to_thread(provider._sandbox_id_for_thread, "t-id", "u1") + + reclaimed: list[str] = [] + + class _ReachedWarmPool(Exception): + pass + + def _stop_at_warm_pool(thread_id, sandbox_id, *, user_id=None): + reclaimed.append(sandbox_id) + raise _ReachedWarmPool + + monkeypatch.setattr(provider, "_reclaim_warm_pool_sandbox", _stop_at_warm_pool) + + try: + # Red before the fix: the gate raises BlockingError from the inline + # ``_sandbox_id_for_thread`` before the warm-pool layer is reached. + with pytest.raises(_ReachedWarmPool): + await provider._acquire_internal_async("t-id", user_id="u1") + finally: + provider._acquire_worker_executor.shutdown(wait=True) + + assert reclaimed == [expected_id] + + +async def test_blocking_probe_sandbox_id_actually_trips_the_gate(tmp_path, monkeypatch) -> None: + """Meta-check: deriving a thread's sandbox ID inline on the loop is blocking. + + Keeps the anchor above honest: if the gate stopped seeing the syscalls + behind ``thread_skills_view_dir(...).exists()``, the anchor would pass with + the inline call restored. + """ + from blockbuster import BlockingError + + provider = _make_provider(tmp_path) + monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path)) + monkeypatch.setattr("deerflow.config.paths._paths", None) + + with pytest.raises(BlockingError): + provider._sandbox_id_for_thread("t-id", "u1")