diff --git a/README.md b/README.md index 45df84a8c..a4d953ec9 100644 --- a/README.md +++ b/README.md @@ -853,6 +853,8 @@ When you install `.skill` archives through the Gateway, DeerFlow accepts standar Disabling a skill also removes it from the sandbox filesystem view, so shell commands and structured file tools follow the same enabled state. Local, Docker/AIO, hostPath provisioner, and newly created E2B sandboxes source `/mnt/skills` from enabled-only projections that update when public, custom, legacy, or managed integration skills are toggled, edited, created, deleted, or installed. Structured `read_file` calls (including line ranges and read-before-write checks) use the sandbox provider's mount mapping, so the user identity captured when the sandbox was acquired remains authoritative. Managed integration packages remain shared, while their projected filesystem visibility follows each user's enabled state. Multi-worker Gateways re-read on-disk enable state while rebuilding user projections, so a toggle handled by one worker is honored by another worker's next sandbox acquire. Existing E2B sandboxes retain their creation-time snapshot until they are recreated. PVC-backed provisioner skills keep their configured PVC snapshot/layout for now; dynamic PVC materialization is tracked separately. +For `LocalSandboxProvider`, this is a managed tool-path boundary rather than host filesystem isolation. Explicit per-Agent skill policies are accepted only while host bash is disabled (the default), because a host subprocess can address canonical paths without using the provider's virtual-path mappings. Use Docker/AIO, the Kubernetes provisioner, or E2B when the filesystem boundary must remain enforceable alongside shell access. + Managed integrations install shared read-only skill packs without mixing them into custom skills. The Lark/Feishu CLI integration is available under `Settings → Integrations → Lark / Feishu CLI`; an administrator installs or @@ -1205,6 +1207,12 @@ The Hash counts remote VMs and in-flight creates, repairs interrupted creates from E2B metadata, grace-protects stale inventory omissions, and blocks new creates while Redis or initial inventory is unavailable. Run Redis with persistence, non-evicting memory, and HA. +E2B snapshots `skills.container_path` when the provider starts and includes the +canonical root in its thread identity, warm-pool seed, and remote metadata. A +VM created for a different root is never adopted; reconciliation reaps it after +the configured grace period once no live peer owns it. Restart the Gateway after +changing the root. + E2B acquisition uses a bounded executor. Waiting acquisitions do not use the default asyncio executor. diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index 314d3deff..58a1ef5b2 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -544,10 +544,13 @@ provider in `config.yaml`. Notes specific to `E2BSandboxProvider`: - Each DeerFlow thread is bound to its E2B sandbox via metadata - (`deer_flow_user`, `deer_flow_thread`). Startup and periodic reconciliation - probe every bounded candidate, adopt one healthy canonical sandbox, and reap - duplicates after a grace period. Provider-tagged entries without a complete - user/thread identity are reaped only after the orphan TTL. + (`deer_flow_user`, `deer_flow_thread`, `deer_flow_skills_root`). Startup and + periodic reconciliation probe every bounded candidate, adopt one healthy + canonical sandbox, and reap duplicates after a grace period. A sandbox whose + skills root differs from the provider's startup snapshot is never adopted and + is reaped after the same grace period once no live peer owns it. + Provider-tagged entries without a complete user/thread identity are reaped + only after the orphan TTL. - Ownership leases prevent one gateway from adopting or destroying a sandbox another live gateway is responsible for. The default in-memory store is safe only for one gateway process. Multi-worker/load-balanced deployments must use @@ -752,6 +755,15 @@ skills: container_path: /mnt/skills ``` +For the AIO provider (including the Kubernetes provisioner) and E2B, +`skills.container_path` is captured when the provider starts and must be one +canonical absolute, non-root POSIX path. Do not use redundant separators, +`.`/`..`, or a path that contains or sits below DeerFlow's reserved mounts +(`/mnt/user-data`, `/mnt/acp-workspace`, or `/mnt/integrations/lark-cli`). +Restart the Gateway after changing it so sandbox identities and mounts use the +same root. E2B also records the root in remote metadata and refuses to adopt a +VM created for another root. + **How Skills Work**: - Skills are stored in `deer-flow/skills/{public,custom}/` - Each skill has a `SKILL.md` file with metadata @@ -777,6 +789,12 @@ This field is a discovery and activation allowlist; it does not activate every l The same semantics apply to `subagents.agents..skills` and `subagents.custom_agents..skills`: omitted or `null` exposes all enabled skills, `[]` exposes none, and a list limits discovery and activation. A passive subagent skill never removes baseline tools; its `allowed-tools` declaration becomes active only after slash activation or a completed `SKILL.md` read. +`LocalSandboxProvider` enforces this filesystem view through its managed virtual +path mappings only. Explicit per-Agent skill policies therefore fail closed when +`sandbox.allow_host_bash` is enabled, because host subprocesses can bypass those +mappings. Keep host bash disabled (the default), or use AIO/provisioner/E2B when +shell access and filesystem isolation are both required. + ### Title Generation Automatic conversation title generation: diff --git a/backend/packages/harness/deerflow/AGENTS.md b/backend/packages/harness/deerflow/AGENTS.md index 79e49ea7d..71acd9905 100644 --- a/backend/packages/harness/deerflow/AGENTS.md +++ b/backend/packages/harness/deerflow/AGENTS.md @@ -97,6 +97,13 @@ does not interrupt active filesystem or E2B SDK calls. The provider checks mount limits before upload. It rechecks each opened file descriptor against its preflight size before SDK upload. +For policy-scoped turns, clearing the four managed remote skill categories and +uploading their prepared projection is one per-user/thread/skills-root critical +section, shared with acquire and release. The provider snapshots that canonical +root at startup and carries it through warm-pool identity and E2B metadata; a VM +from another root is never adopted. A second policy sync cannot reset the remote +tree until the first upload pass has completed. + An invalid mount does not block later mounts. Each successful upload logs its source, destination, file count, byte count, and elapsed time. diff --git a/backend/packages/harness/deerflow/agents/lead_agent/agent.py b/backend/packages/harness/deerflow/agents/lead_agent/agent.py index b72c5423e..64937fd25 100644 --- a/backend/packages/harness/deerflow/agents/lead_agent/agent.py +++ b/backend/packages/harness/deerflow/agents/lead_agent/agent.py @@ -461,6 +461,7 @@ def build_middlewares( custom_middlewares: list[AgentMiddleware] | None = None, *, available_skills: set[str] | None = None, + owns_agent_skill_projection: bool = True, app_config: AppConfig | None = None, deferred_setup=None, mcp_routing_middleware: AgentMiddleware | None = None, @@ -481,6 +482,9 @@ def build_middlewares( model_name: Resolved runtime model name; gates vision-only middleware. agent_name: If provided, MemoryMiddleware will use per-agent memory storage. custom_middlewares: Optional list of custom middlewares to inject into the chain. + owns_agent_skill_projection: Whether this lead middleware chain owns the + thread's physical skill projection. Prompt-only bootstrap agents do + not; their narrow skill set must not replace the thread view. app_config: Explicit AppConfig; falls back to ``get_app_config()`` when omitted. deferred_setup: Optional deferred-MCP-tool setup that attaches ``DeferredToolFilterMiddleware`` when ``tool_search`` is enabled. @@ -506,6 +510,10 @@ def build_middlewares( "app_config": resolved_app_config, "lazy_init": True, } + if available_skills is not None: + runtime_middleware_kwargs["available_skills"] = available_skills + if not owns_agent_skill_projection: + runtime_middleware_kwargs["owns_agent_skill_projection"] = False if authorization_provider is not None: runtime_middleware_kwargs["authorization_provider"] = authorization_provider if authorization_provider is not None and deferred_setup is not None: @@ -1025,6 +1033,7 @@ def _assemble_lead_agent(config: RunnableConfig, *, app_config: AppConfig) -> Le model_name=model_name, agent_name=agent_name, available_skills=set(_BOOTSTRAP_SKILL_NAMES), + owns_agent_skill_projection=False, app_config=resolved_app_config, deferred_setup=setup, mcp_routing_middleware=mcp_routing_middleware, diff --git a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md index 20b8a0bda..426f9f3be 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md @@ -49,7 +49,11 @@ it to that middleware's declaration in the same change. output wording. 4. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves identity via `resolve_runtime_user_id(runtime)`, including Gateway runtime context and standalone LangGraph Server auth, then falls back to the request ContextVar / `"default"` 5. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation (lead agent only); upload existence checks use the same runtime-resolved user bucket as thread-data creation -6. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state +6. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state. The + lead runtime normally owns the thread's physical Agent-skill projection; + delegated subagents and the prompt-only bootstrap agent are non-owners, so + their narrower discovery allowlists never rebuild the shared thread view or + force eager sandbox acquisition. 7. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request 8. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later stages run 9. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md). diff --git a/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py index b64f84ebf..92a1cd8e2 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py +++ b/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py @@ -167,6 +167,8 @@ def _build_runtime_middlewares( receipts_render_mode: str = "delegation_only", authorization_provider=None, authorization_infrastructure_tool_names: frozenset[str] = frozenset(), + available_skills: set[str] | None = None, + owns_agent_skill_projection: bool = True, ) -> list[AgentMiddleware]: """Build shared base middlewares for agent execution.""" from deerflow.agents.middlewares.input_sanitization_middleware import InputSanitizationMiddleware @@ -199,7 +201,13 @@ def _build_runtime_middlewares( from deerflow.agents.middlewares.uploads_middleware import UploadsMiddleware thread_hooks.append(UploadsMiddleware()) - thread_hooks.append(SandboxMiddleware(lazy_init=lazy_init)) + thread_hooks.append( + SandboxMiddleware( + lazy_init=lazy_init, + available_skills=available_skills, + owns_agent_skill_projection=owns_agent_skill_projection, + ) + ) # Layer 3 — post-processing append-only middlewares. tail: list[AgentMiddleware] = [] @@ -313,6 +321,8 @@ def build_lead_runtime_middlewares( lazy_init: bool = True, authorization_provider=None, deferred_setup: "DeferredToolSetup | None" = None, + available_skills: set[str] | None = None, + owns_agent_skill_projection: bool = True, ) -> list[AgentMiddleware]: """Middlewares shared by lead agent runtime before lead-only middlewares.""" return _build_runtime_middlewares( @@ -324,6 +334,8 @@ def build_lead_runtime_middlewares( # results (default "delegation_only"); stamping stays always-on. receipts_render_mode=app_config.verification.receipts_render_mode, authorization_provider=authorization_provider, + available_skills=available_skills, + owns_agent_skill_projection=owns_agent_skill_projection, authorization_infrastructure_tool_names=(frozenset({deferred_setup.tool_search_tool.name}) if authorization_provider is not None and deferred_setup is not None and deferred_setup.tool_search_tool is not None else frozenset()), ) @@ -361,6 +373,7 @@ def build_subagent_runtime_middlewares( receipts_render_mode="always", authorization_provider=authorization_provider, authorization_infrastructure_tool_names=(frozenset({deferred_setup.tool_search_tool.name}) if authorization_provider is not None and deferred_setup is not None and deferred_setup.tool_search_tool is not None else frozenset()), + owns_agent_skill_projection=False, ) # Enabled/configured skills are discoverable metadata, not automatically diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py index 237c944b6..817668e10 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py @@ -13,6 +13,7 @@ The provider itself handles: import asyncio import atexit import contextlib +import hashlib import logging import os import signal @@ -37,6 +38,7 @@ from deerflow.community.warm_pool_lifecycle import ( ) from deerflow.config import get_app_config from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths, join_host_path +from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH from deerflow.integrations.lark_cli import INTEGRATION_ID as LARK_CLI_INTEGRATION_ID from deerflow.integrations.lark_cli import LARK_CLI_SANDBOX_CONFIG_DIR, LARK_CLI_SANDBOX_DATA_DIR, LARK_CLI_SANDBOX_LOCKS_DIR, LARK_CLI_SANDBOX_RUNTIME_DIR, ensure_lark_cli_credential_tree, lark_skills_installed from deerflow.runtime.user_context import get_effective_user_id @@ -44,6 +46,7 @@ from deerflow.sandbox.acquire_serialization import AcquireSerializer from deerflow.sandbox.identity import derive_sandbox_scope_token from deerflow.sandbox.sandbox import Sandbox from deerflow.sandbox.sandbox_provider import SandboxProvider +from deerflow.skills.types import SkillCategory from .aio_sandbox import AioSandbox from .backend import SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT, SandboxBackend, wait_for_sandbox_ready, wait_for_sandbox_ready_async @@ -57,7 +60,7 @@ from .ownership import ( make_sandbox_ownership_store, resolve_ownership_config, ) -from .remote_backend import RemoteSandboxBackend +from .remote_backend import RemoteSandboxBackend, _normalize_skills_container_path from .sandbox_info import SandboxInfo logger = logging.getLogger(__name__) @@ -143,6 +146,8 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): API_KEY: $MY_API_KEY """ + supports_agent_skill_isolation = True + # How long `_held_teardown_lease` waits for its heartbeat thread to exit # before deferring the final lease release to that (still-running) thread. # The store's socket timeout bounds each operation, but context exit can @@ -265,6 +270,13 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): idle_timeout = getattr(sandbox_config, "idle_timeout", None) replicas = getattr(sandbox_config, "replicas", None) + configured_skills_path = getattr( + getattr(config, "skills", None), + "container_path", + None, + ) + if not isinstance(configured_skills_path, str): + configured_skills_path = DEFAULT_SKILLS_CONTAINER_PATH return { "image": sandbox_config.image or DEFAULT_IMAGE, @@ -283,6 +295,9 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): # provisioner URL for dynamic pod management (e.g. http://provisioner:8002) "provisioner_url": getattr(sandbox_config, "provisioner_url", None) or "", "provisioner_api_key": getattr(sandbox_config, "provisioner_api_key", None) or "", + "skills_container_path": _normalize_skills_container_path( + configured_skills_path, + ), } @staticmethod @@ -723,6 +738,32 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): """ return derive_sandbox_scope_token(user_id=user_id, thread_id=thread_id) + @staticmethod + def _thread_skill_projection_active(thread_id: str, user_id: str) -> bool: + return get_paths().thread_skills_view_dir(thread_id, user_id=user_id).exists() + + @staticmethod + def _policy_scoped_sandbox_id( + thread_id: str, + user_id: str, + skills_container_path: str, + ) -> str: + """Return a root-aware domain-separated identity for a policy sandbox.""" + normalized_root = _normalize_skills_container_path(skills_container_path) + seed = b"agent-skills-v2\0" + user_id.encode() + b"\0" + thread_id.encode() + b"\0" + normalized_root.encode() + return hashlib.sha256(seed).hexdigest()[:16] + + @staticmethod + def _custom_root_sandbox_id( + thread_id: str, + user_id: str, + skills_container_path: str, + ) -> str: + """Return an identity for a shared-view sandbox at a custom root.""" + normalized_root = _normalize_skills_container_path(skills_container_path) + seed = b"skills-root-v1\0" + user_id.encode() + b"\0" + thread_id.encode() + b"\0" + normalized_root.encode() + return hashlib.sha256(seed).hexdigest()[:16] + def _assert_active_identity_available_locked( self, sandbox_id: str, @@ -758,17 +799,37 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): def _get_extra_mounts(self, thread_id: str | None, *, user_id: str | None = None) -> list[tuple[str, str, bool]]: """Collect all extra mounts for a sandbox (thread-specific + skills).""" mounts: list[tuple[str, str, bool]] = [] + skills_container_path = self._configured_skills_container_path() if thread_id: mounts.extend(self._get_thread_mounts(thread_id, user_id=user_id)) logger.info(f"Adding thread mounts for thread {thread_id}: {mounts}") - skills_mounts = self._get_skills_mounts(user_id=user_id) + skills_mounts = self._get_skills_mounts( + thread_id, + user_id=user_id, + skills_container_path=skills_container_path, + ) if skills_mounts: mounts.extend(skills_mounts) logger.info(f"Adding skills mounts: {skills_mounts}") - user_skill_mounts = self._get_user_skill_mounts(user_id=user_id) + effective_user_id = self._effective_acquire_user_id(user_id) + thread_projection_active = bool( + thread_id + and self._thread_skill_projection_active( + thread_id, + effective_user_id, + ) + ) + user_skill_mounts = ( + [] + if thread_projection_active + else self._get_user_skill_mounts( + user_id=user_id, + skills_container_path=skills_container_path, + ) + ) if user_skill_mounts: mounts.extend(user_skill_mounts) logger.info(f"Adding user skill mounts: {user_skill_mounts}") @@ -780,6 +841,34 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): return self._dedupe_mounts_by_container_path(mounts) + def _local_config_mount_exclusion_root( + self, + thread_id: str | None, + *, + user_id: str, + ) -> str | None: + """Return the skills subtree owned by a policy-scoped local sandbox.""" + if not isinstance(self._backend, LocalContainerBackend) or not thread_id: + return None + if not self._thread_skill_projection_active(thread_id, user_id): + return None + return self._configured_skills_container_path() + + def _configured_skills_container_path(self) -> str: + """Return the provider-startup skills root used by IDs and mounts.""" + # A few mount-helper callers intentionally construct an uninitialized + # provider. Production instances always use the startup snapshot, while + # that narrow compatibility path loads the same validated value lazily. + config = getattr(self, "_config", None) + if not isinstance(config, dict): + config = self._load_config() + return _normalize_skills_container_path( + config.get( + "skills_container_path", + DEFAULT_SKILLS_CONTAINER_PATH, + ) + ) + @staticmethod def _dedupe_mounts_by_container_path(mounts: list[tuple[str, str, bool]]) -> list[tuple[str, str, bool]]: """Keep the first mount for each container path. @@ -825,7 +914,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): ] @staticmethod - def _get_skills_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]: + def _get_skills_mounts( + thread_id: str | None = None, + *, + user_id: str | None = None, + skills_container_path: str | None = None, + ) -> list[tuple[str, str, bool]]: """Get skills directory mount configurations for three-way skills layout. Mirrors ``LocalSandboxProvider._build_thread_path_mappings`` for AIO @@ -841,12 +935,30 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): mounts: list[tuple[str, str, bool]] = [] try: config = get_app_config() - container_path = config.skills.container_path + container_path = _normalize_skills_container_path(skills_container_path or config.skills.container_path) effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id) - AioSandboxProvider._ensure_skills_projection(effective_user_id) paths = get_paths() host_base_dir = str(paths.host_base_dir) + if thread_id and AioSandboxProvider._thread_skill_projection_active( + thread_id, + effective_user_id, + ): + host_root = paths.host_thread_skills_view_dir( + thread_id, + user_id=effective_user_id, + ) + return [ + ( + join_host_path(host_root, category.value), + f"{container_path}/{category.value}", + True, + ) + for category in SkillCategory + ] + + AioSandboxProvider._ensure_skills_projection(effective_user_id) + # 1. Public skills: global, read-only — static, shared by all threads mounts.append( ( @@ -907,7 +1019,11 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): return None @staticmethod - def _get_user_skill_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]: + def _get_user_skill_mounts( + *, + user_id: str | None = None, + skills_container_path: str | None = None, + ) -> list[tuple[str, str, bool]]: """Mount enabled managed integration skills into AIO sandboxes. Per-user custom skills are already mounted by ``_get_skills_mounts``. @@ -917,7 +1033,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): try: config = get_app_config() paths = get_paths() - skills_container_path = config.skills.container_path + resolved_skills_container_path = _normalize_skills_container_path(skills_container_path or config.skills.container_path) effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id) AioSandboxProvider._ensure_skills_projection(effective_user_id) return [ @@ -929,7 +1045,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): "skills_view", "integrations", ), - f"{skills_container_path}/integrations", + f"{resolved_skills_container_path}/integrations", True, ), ] @@ -1275,7 +1391,26 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): def _sandbox_id_for_thread(self, thread_id: str | None, user_id: str | None) -> str: """Return deterministic IDs for thread sandboxes and random IDs otherwise.""" - return self._deterministic_sandbox_id(thread_id, self._effective_acquire_user_id(user_id)) if thread_id else str(uuid.uuid4())[:8] + if not thread_id: + return str(uuid.uuid4())[:8] + effective_user_id = self._effective_acquire_user_id(user_id) + skills_container_path = self._configured_skills_container_path() + if self._thread_skill_projection_active(thread_id, effective_user_id): + return self._policy_scoped_sandbox_id( + thread_id, + effective_user_id, + skills_container_path, + ) + # Preserve the historic deterministic ID for the default root while + # preventing a custom-root Pod/container from being reused after the + # configured mount destination changes. + if skills_container_path != DEFAULT_SKILLS_CONTAINER_PATH: + return self._custom_root_sandbox_id( + thread_id, + effective_user_id, + skills_container_path, + ) + return self._deterministic_sandbox_id(thread_id, effective_user_id) def _reuse_in_process_sandbox(self, thread_id: str | None, *, user_id: str | None = None, post_lock: bool = False) -> str | None: """Reuse an active in-process sandbox for a thread if one is still tracked.""" @@ -1284,22 +1419,44 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): effective_user_id = self._effective_acquire_user_id(user_id) key = self._thread_key(thread_id, effective_user_id) + root_scoped_identity = ( + self._thread_skill_projection_active( + thread_id, + effective_user_id, + ) + or self._configured_skills_container_path() != DEFAULT_SKILLS_CONTAINER_PATH + ) + expected_id = self._sandbox_id_for_thread(thread_id, effective_user_id) + stale_id: str | None = None with self._lock: if key not in self._thread_sandboxes: return None existing_id = self._thread_sandboxes[key] - if self._being_torn_down_locally(existing_id): + if root_scoped_identity and existing_id != expected_id: + stale_id = existing_id + elif self._being_torn_down_locally(existing_id): # A reaper thread in this process is stopping this container. # Same answer as a peer's `del:` lease: cold-start instead. logger.info("Cached sandbox %s is being destroyed by this instance; not reusing it", existing_id) return None - if existing_id in self._sandboxes: + elif existing_id in self._sandboxes: info = self._sandbox_infos.get(existing_id) else: del self._thread_sandboxes[key] return None + if stale_id is not None: + logger.info( + "Replacing sandbox %s with expected identity %s for user/thread %s/%s", + stale_id, + expected_id, + effective_user_id, + thread_id, + ) + self.destroy(stale_id) + return None + alive = self._check_tracked_sandbox_alive(existing_id, info) if info is not None else True if alive is False: self._drop_unhealthy_sandbox( @@ -1975,6 +2132,10 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): extra_mounts = self._get_extra_mounts(thread_id, user_id=effective_user_id) provision_lark_cli_runtime = self._lark_integration_active(effective_user_id) provision_lark_cli_broker = self._lark_broker_active(effective_user_id) + config_mount_exclusion_root = self._local_config_mount_exclusion_root( + thread_id, + user_id=effective_user_id, + ) # Enforce replicas: only warm-pool containers count toward eviction budget. # Active sandboxes are in use by live threads and must not be forcibly stopped. @@ -1983,6 +2144,11 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): evicted = self._evict_oldest_warm() self._log_replicas_soft_cap(replicas, sandbox_id, evicted) + create_kwargs = {} + if config_mount_exclusion_root is not None: + create_kwargs["config_mount_exclusion_root"] = config_mount_exclusion_root + if isinstance(self._backend, RemoteSandboxBackend): + create_kwargs["skills_container_path"] = self._configured_skills_container_path() info = self._backend.create( thread_id, sandbox_id, @@ -1990,6 +2156,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): user_id=effective_user_id, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, + **create_kwargs, ) # Wait for sandbox to be ready @@ -2009,6 +2176,11 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): extra_mounts = await asyncio.to_thread(self._get_extra_mounts, thread_id, user_id=effective_user_id) provision_lark_cli_runtime = await asyncio.to_thread(self._lark_integration_active, effective_user_id) provision_lark_cli_broker = await asyncio.to_thread(self._lark_broker_active, effective_user_id) + config_mount_exclusion_root = await asyncio.to_thread( + self._local_config_mount_exclusion_root, + thread_id, + user_id=effective_user_id, + ) # Enforce replicas: only warm-pool containers count toward eviction budget. # Active sandboxes are in use by live threads and must not be forcibly stopped. @@ -2017,6 +2189,11 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): evicted = await asyncio.to_thread(self._evict_oldest_warm) self._log_replicas_soft_cap(replicas, sandbox_id, evicted) + create_kwargs = {} + if config_mount_exclusion_root is not None: + create_kwargs["config_mount_exclusion_root"] = config_mount_exclusion_root + if isinstance(self._backend, RemoteSandboxBackend): + create_kwargs["skills_container_path"] = self._configured_skills_container_path() info = await asyncio.to_thread( self._backend.create, thread_id, @@ -2025,6 +2202,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): user_id=effective_user_id, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, + **create_kwargs, ) # Wait for sandbox to be ready without blocking the event loop. diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py b/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py index 434b1b082..6c63a248b 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py @@ -11,6 +11,7 @@ import ipaddress import json import logging import os +import posixpath import shlex import socket import subprocess @@ -522,6 +523,7 @@ class LocalContainerBackend(SandboxBackend): sandbox_id: str, extra_mounts: list[tuple[str, str, bool]] | None = None, *, + config_mount_exclusion_root: str | None = None, user_id: str | None = None, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, @@ -532,6 +534,10 @@ class LocalContainerBackend(SandboxBackend): thread_id: Thread ID for which the sandbox is being created. Useful for backends that want to organize sandboxes by thread. sandbox_id: Deterministic sandbox identifier (used in container name). extra_mounts: Additional volume mounts as (host_path, container_path, read_only) tuples. + config_mount_exclusion_root: Exclude config-level mounts at or + below this container path. Policy-scoped skill projections use + this to prevent a nested operator mount from overlaying an + excluded skill back into the restricted view. user_id: User bucket already reflected in extra_mounts. Accepted for interface compatibility with remote backends. provision_lark_cli_runtime: Ignored — the local backend provisions the @@ -559,7 +565,12 @@ class LocalContainerBackend(SandboxBackend): for _attempt in range(10): port = get_free_port(start_port=_next_start) try: - container_id = self._start_container(container_name, port, extra_mounts) + container_id = self._start_container( + container_name, + port, + extra_mounts, + config_mount_exclusion_root=config_mount_exclusion_root, + ) break except RuntimeError as exc: release_port(port) @@ -788,6 +799,8 @@ class LocalContainerBackend(SandboxBackend): container_name: str, port: int, extra_mounts: list[tuple[str, str, bool]] | None = None, + *, + config_mount_exclusion_root: str | None = None, ) -> str: """Start a new container. @@ -795,6 +808,8 @@ class LocalContainerBackend(SandboxBackend): container_name: Name for the container. port: Host port to map to container port 8080. extra_mounts: Additional volume mounts. + config_mount_exclusion_root: Config-level mounts at or below this + container root are omitted for this container only. Returns: The container ID. @@ -957,8 +972,21 @@ class LocalContainerBackend(SandboxBackend): for key, value in self._environment.items(): cmd.extend(["-e", f"{key}={value}"]) - # Config-level volume mounts + # Config-level volume mounts. A policy-scoped skills view owns its + # complete container subtree; keeping a more-specific config mount + # would let Docker overlay an excluded skill inside that view. + exclusion_root = None + if config_mount_exclusion_root is not None: + exclusion_root = posixpath.normpath(config_mount_exclusion_root.rstrip("/") or "/") + for mount in self._config_mounts: + mount_path = posixpath.normpath(str(mount.container_path).rstrip("/") or "/") + if exclusion_root is not None and (mount_path == exclusion_root or mount_path.startswith(exclusion_root.rstrip("/") + "/")): + logger.info( + "Skipping config mount inside policy-scoped skills root: %s", + mount.container_path, + ) + continue cmd.extend( _format_container_mount( self._runtime, diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py b/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py index 8fdee1929..1ed3e8b01 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py @@ -18,9 +18,12 @@ Architecture: from __future__ import annotations import logging +import posixpath +from pathlib import PurePosixPath import requests +from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH from deerflow.runtime.user_context import get_effective_user_id from deerflow.skills.storage import user_should_see_legacy_skills @@ -31,22 +34,57 @@ logger = logging.getLogger(__name__) _PROVISIONER_EXTRA_MOUNT_PATHS = { "/mnt/acp-workspace", - "/mnt/skills/custom", - "/mnt/skills/integrations", "/mnt/integrations/lark-cli/config", "/mnt/integrations/lark-cli/config/locks", "/mnt/integrations/lark-cli/data", "/mnt/integrations/lark-cli/runtime", } +_MANAGED_SKILL_CATEGORY_NAMES = ( + "public", + "custom", + "legacy", + "integrations", +) +_RESERVED_SANDBOX_MOUNT_PATHS = ( + "/mnt/user-data", + "/mnt/acp-workspace", + "/mnt/integrations/lark-cli", +) _LARK_CLI_RUNTIME_CONTAINER_PATH = "/mnt/integrations/lark-cli/runtime" _LARK_CLI_CONFIG_CONTAINER_PATH = "/mnt/integrations/lark-cli/config" _LARK_CLI_DATA_CONTAINER_PATH = "/mnt/integrations/lark-cli/data" +def _normalize_skills_container_path(container_path: str) -> str: + """Return a canonical skills root that cannot overlap platform mounts.""" + candidate = container_path + if not candidate or not candidate.startswith("/") or candidate.startswith("//"): + raise ValueError("The skills container path must be an absolute non-root path") + + normalized = posixpath.normpath(candidate) + if normalized != candidate: + raise ValueError("The skills container path must not contain redundant separators, '.' or '..'") + + root = PurePosixPath(normalized) + for reserved_path in _RESERVED_SANDBOX_MOUNT_PATHS: + reserved = PurePosixPath(reserved_path) + if root == reserved or root.is_relative_to(reserved) or reserved.is_relative_to(root): + raise ValueError(f"The skills container path {normalized!r} overlaps reserved sandbox path {reserved_path!r}") + return normalized + + +def _managed_skill_category_mount_paths( + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> set[str]: + root = _normalize_skills_container_path(skills_container_path) + return {posixpath.join(root, category) for category in _MANAGED_SKILL_CATEGORY_NAMES} + + def _provisioner_extra_mounts_payload( extra_mounts: list[tuple[str, str, bool]] | None, *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, ) -> list[dict[str, object]]: @@ -67,6 +105,7 @@ def _provisioner_extra_mounts_payload( available for the provisioner to place; the runtime entry is dropped in both modes. """ + allowed_paths = _PROVISIONER_EXTRA_MOUNT_PATHS | _managed_skill_category_mount_paths(skills_container_path) if not extra_mounts: return [] @@ -74,7 +113,7 @@ def _provisioner_extra_mounts_payload( payload: list[dict[str, object]] = [] for host_path, container_path, read_only in extra_mounts: - if container_path not in _PROVISIONER_EXTRA_MOUNT_PATHS: + if container_path not in allowed_paths: continue if drop_runtime and container_path == _LARK_CLI_RUNTIME_CONTAINER_PATH: continue @@ -130,6 +169,7 @@ class RemoteSandboxBackend(SandboxBackend): extra_mounts: list[tuple[str, str, bool]] | None = None, *, user_id: str | None = None, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, ) -> SandboxInfo: @@ -143,6 +183,7 @@ class RemoteSandboxBackend(SandboxBackend): sandbox_id, extra_mounts, user_id=user_id, + skills_container_path=skills_container_path, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, ) @@ -216,22 +257,26 @@ class RemoteSandboxBackend(SandboxBackend): extra_mounts: list[tuple[str, str, bool]] | None = None, *, user_id: str | None = None, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, ) -> SandboxInfo: """POST /api/sandboxes → create Pod + Service.""" effective_user_id = user_id or get_effective_user_id() include_legacy_skills = user_should_see_legacy_skills(effective_user_id) + normalized_skills_container_path = _normalize_skills_container_path(skills_container_path) payload = { "sandbox_id": sandbox_id, "thread_id": thread_id, "user_id": effective_user_id, "include_legacy_skills": include_legacy_skills, + "skills_container_path": normalized_skills_container_path, "provision_lark_cli_runtime": provision_lark_cli_runtime, "provision_lark_cli_broker": provision_lark_cli_broker, } provisioner_extra_mounts = _provisioner_extra_mounts_payload( extra_mounts, + skills_container_path=normalized_skills_container_path, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, ) diff --git a/backend/packages/harness/deerflow/community/e2b_sandbox/e2b_sandbox_provider.py b/backend/packages/harness/deerflow/community/e2b_sandbox/e2b_sandbox_provider.py index 7dbcdb805..ec4979dc1 100644 --- a/backend/packages/harness/deerflow/community/e2b_sandbox/e2b_sandbox_provider.py +++ b/backend/packages/harness/deerflow/community/e2b_sandbox/e2b_sandbox_provider.py @@ -37,9 +37,11 @@ from __future__ import annotations import asyncio import atexit +import hashlib import json import logging import os +import posixpath import shlex import signal import threading @@ -51,8 +53,8 @@ from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass from decimal import Decimal, InvalidOperation from functools import partial -from pathlib import Path -from typing import Any +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING, Any from e2b import SandboxQuery from e2b_code_interpreter import Sandbox as E2BClientSandbox @@ -81,6 +83,9 @@ from .capacity import ( ) from .e2b_sandbox import DEFAULT_E2B_HOME_DIR, E2BSandbox, _is_sandbox_gone_error +if TYPE_CHECKING: + from deerflow.skills.projection import SkillProjectionPaths + logger = logging.getLogger(__name__) @@ -112,11 +117,76 @@ _MAX_MOUNT_PASS_FILES = 2000 # Deadline checks stop preflight work and new writes. Active SDK writes finish. _MOUNT_PASS_DEADLINE_SECONDS = 120 +# Recursive skill projection replacement must never target an operating-system +# tree. The configured E2B home is handled separately: an isolated descendant +# such as /home/user/skills is supported, while the home directory itself and +# every ancestor remain protected. +_E2B_PROTECTED_SYSTEM_TREES = frozenset( + PurePosixPath(path) + for path in ( + "/bin", + "/boot", + "/dev", + "/etc", + "/home", + "/lib", + "/lib32", + "/lib64", + "/libx32", + "/lost+found", + "/media", + "/opt", + "/proc", + "/root", + "/run", + "/sbin", + "/snap", + "/srv", + "/sys", + "/tmp", + "/usr", + "/var", + ) +) + def _mount_deadline_reason(deadline_seconds: int) -> str: return f"time budget {deadline_seconds}s" +def _validate_skills_reset_root(container_path: str, *, home_dir: str) -> str: + """Return a canonical E2B root that is safe for recursive managed resets.""" + candidate = container_path.rstrip("/") + if not candidate or not candidate.startswith("/") or candidate.startswith("//"): + raise ValueError("The skills container path is not a safe E2B skills reset target: it must be an absolute non-root path") + + normalized = posixpath.normpath(candidate) + if normalized != candidate: + raise ValueError("The skills container path is not a safe E2B skills reset target: it must not contain redundant separators, '.' or '..'") + + root = PurePosixPath(normalized) + protected_roots = { + PurePosixPath("/mnt/user-data"), + PurePosixPath("/mnt/acp-workspace"), + } + normalized_home = posixpath.normpath(home_dir.rstrip("/") or DEFAULT_E2B_HOME_DIR) + home_root: PurePosixPath | None = None + if normalized_home.startswith("/") and not normalized_home.startswith("//"): + home_root = PurePosixPath(normalized_home) + protected_roots.add(home_root) + + for protected in protected_roots: + if protected == root or protected.is_relative_to(root): + raise ValueError(f"The skills container path is not a safe E2B skills reset target: {normalized!r} equals or contains protected path {str(protected)!r}") + + is_isolated_home_subtree = home_root is not None and root != home_root and root.is_relative_to(home_root) + if not is_isolated_home_subtree: + for protected in _E2B_PROTECTED_SYSTEM_TREES: + if root == protected or root.is_relative_to(protected): + raise ValueError(f"The skills container path is not a safe E2B skills reset target: {normalized!r} is inside protected operating-system tree {str(protected)!r}") + return normalized + + class _MountPassLimitExceeded(Exception): """Stop the current mount upload pass at its aggregate resource limit.""" @@ -148,6 +218,7 @@ META_KEY_GATEWAY = "deer_flow_gateway" META_KEY_CREATED_AT = "deer_flow_created_at" META_KEY_CAPACITY_LEDGER = "deer_flow_capacity_ledger" META_KEY_CAPACITY_RESERVATION = "deer_flow_capacity_reservation" +META_KEY_SKILLS_ROOT = "deer_flow_skills_root" META_VAL_PROVIDER = "e2b_sandbox_provider" E2B_EXTRA_CONFIG_KEYS = frozenset( { @@ -187,6 +258,7 @@ class E2BSandboxProvider(SandboxProvider): # remote backend in AioSandboxProvider sets the same flag). uses_thread_data_mounts = False needs_upload_permission_adjustment = True + supports_agent_skill_isolation = True # ── Construction & config ──────────────────────────────────────────── @@ -194,11 +266,13 @@ class E2BSandboxProvider(SandboxProvider): self._lock = threading.Lock() # Active sandboxes, keyed by DeerFlow-side sandbox id (== e2b id). self._sandboxes: dict[str, E2BSandbox] = {} - # (user_id, thread_id) -> sandbox id for fast in-process lookup. - self._thread_sandboxes: dict[tuple[str, str], str] = {} - # Per-(user,thread) serializer for acquire() and release() state + # (user_id, thread_id, skills_root) -> sandbox id for fast in-process + # lookup. The provider snapshots the root at startup, but keeping it in + # the key makes the identity boundary explicit and fail-safe. + self._thread_sandboxes: dict[tuple[str, str, str], str] = {} + # Per-(user,thread,skills_root) serializer for acquire() and release() state # transitions without holding the provider-wide lock across remote IO. - self._acquire_serializer: AcquireSerializer[tuple[str, str]] = AcquireSerializer(thread_name_prefix="e2b-sandbox-lock-wait") + self._acquire_serializer: AcquireSerializer[tuple[str, str, str]] = AcquireSerializer(thread_name_prefix="e2b-sandbox-lock-wait") # Warm pool: released sandboxes whose remote micro-VM is still alive. # ``OrderedDict`` maintains insertion / move_to_end order for LRU. self._warm_pool: OrderedDict[str, tuple[str, float]] = OrderedDict() @@ -255,7 +329,8 @@ class E2BSandboxProvider(SandboxProvider): def _load_config(self) -> dict[str, Any]: """Read e2b options off ``SandboxConfig`` (``extra="allow"``).""" - sandbox_config = get_app_config().sandbox + app_config = get_app_config() + sandbox_config = app_config.sandbox unknown_keys = sorted(set(getattr(sandbox_config, "model_extra", None) or {}) - E2B_EXTRA_CONFIG_KEYS) if unknown_keys: logger.warning( @@ -295,11 +370,18 @@ class E2BSandboxProvider(SandboxProvider): logger.warning("E2BSandboxProvider: overflow_policy is 'burst' but burst_limit is 0; falling back to 'reject'") overflow_policy = "reject" + home_dir = _opt("home_dir") or DEFAULT_E2B_HOME_DIR + skills_container_path = _validate_skills_reset_root( + app_config.skills.container_path, + home_dir=home_dir, + ) + return { "api_key": api_key, "template": _opt("template") or _opt("image") or DEFAULT_TEMPLATE, "domain": _opt("domain"), - "home_dir": _opt("home_dir") or DEFAULT_E2B_HOME_DIR, + "home_dir": home_dir, + "skills_container_path": skills_container_path, "idle_timeout": idle_timeout, "replicas": replicas, "overflow_policy": overflow_policy, @@ -308,7 +390,7 @@ class E2BSandboxProvider(SandboxProvider): "mounts": _opt("mounts") or [], "environment": self._resolve_env_vars(_opt("environment") or {}), "ownership": _opt("ownership"), - "stream_bridge": getattr(get_app_config(), "stream_bridge", None), + "stream_bridge": getattr(app_config, "stream_bridge", None), "reconciliation_interval_seconds": max( 1.0, float(_opt("reconciliation_interval_seconds", DEFAULT_RECONCILIATION_INTERVAL_SECONDS)), @@ -378,18 +460,18 @@ class E2BSandboxProvider(SandboxProvider): def _effective_acquire_user_id(user_id: str | None) -> str: return user_id or get_effective_user_id() - @staticmethod - def _thread_key(thread_id: str, user_id: str) -> tuple[str, str]: - return (user_id, thread_id) + def _thread_key(self, thread_id: str, user_id: str) -> tuple[str, str, str]: + return (user_id, thread_id, self._config["skills_container_path"]) - @staticmethod - def _stable_seed(thread_id: str, user_id: str) -> str: - """Warm-pool lookup seed derived from user/thread scope. + def _stable_seed(self, thread_id: str, user_id: str) -> str: + """Warm-pool lookup seed derived from user/thread/root scope. For E2B this value is the warm-pool lookup seed, not the provider-issued remote id (RFC #4741 §6). """ - return derive_sandbox_scope_token(user_id=user_id, thread_id=thread_id) + base_scope = derive_sandbox_scope_token(user_id=user_id, thread_id=thread_id) + skills_root = self._config["skills_container_path"] + return hashlib.sha256(f"{base_scope}\0{skills_root}".encode()).hexdigest()[:16] def _metadata_matches_capacity_ledger( self, @@ -617,13 +699,18 @@ class E2BSandboxProvider(SandboxProvider): META_KEY_PROVIDER: META_VAL_PROVIDER, META_KEY_USER: user_id, META_KEY_THREAD: thread_id, + META_KEY_SKILLS_ROOT: self._config["skills_container_path"], } ) candidates = sorted( ( (sandbox_id, metadata) for entry in entries - if (sandbox_id := self._entry_id(entry)) and (metadata := self._entry_metadata(entry)).get(META_KEY_USER) == user_id and metadata.get(META_KEY_THREAD) == thread_id and self._metadata_matches_capacity_ledger(metadata) + if (sandbox_id := self._entry_id(entry)) + and (metadata := self._entry_metadata(entry)).get(META_KEY_USER) == user_id + and metadata.get(META_KEY_THREAD) == thread_id + and metadata.get(META_KEY_SKILLS_ROOT) == self._config["skills_container_path"] + and self._metadata_matches_capacity_ledger(metadata) ), key=lambda item: (item[1].get(META_KEY_CREATED_AT, ""), item[0]), ) @@ -1075,6 +1162,7 @@ class E2BSandboxProvider(SandboxProvider): META_KEY_PROVIDER: META_VAL_PROVIDER, META_KEY_GATEWAY: self._owner_id, META_KEY_CREATED_AT: str(time.time()), + META_KEY_SKILLS_ROOT: self._config["skills_container_path"], } if self._deployment_capacity is not None: metadata[META_KEY_CAPACITY_LEDGER] = self._deployment_capacity.key @@ -1170,7 +1258,7 @@ class E2BSandboxProvider(SandboxProvider): # One-shot mount uploads. e2b has no host bind-mount, so we copy # files from ``host_path`` into ``container_path`` at sandbox start. try: - self._apply_mounts(client, user_id=user_id) + self._apply_mounts(client, user_id=user_id, thread_id=thread_id) except Exception as e: logger.warning("Failed to apply some mounts to e2b sandbox %s: %s", sandbox_id, e) @@ -1448,7 +1536,7 @@ class E2BSandboxProvider(SandboxProvider): ) groups: dict[tuple[str, str], list[tuple[str, dict[str, Any]]]] = {} - orphans: list[tuple[str, dict[str, Any]]] = [] + stale_entries: list[tuple[str, dict[str, Any], float]] = [] present_ids: set[str] = set() for entry in entries: @@ -1459,15 +1547,34 @@ class E2BSandboxProvider(SandboxProvider): metadata = self._entry_metadata(entry) user_id = metadata.get(META_KEY_USER) thread_id = metadata.get(META_KEY_THREAD) - if isinstance(user_id, str) and user_id and isinstance(thread_id, str) and thread_id: + skills_root = metadata.get(META_KEY_SKILLS_ROOT) + has_thread_identity = isinstance(user_id, str) and user_id and isinstance(thread_id, str) and thread_id + if has_thread_identity and skills_root == self._config["skills_container_path"]: groups.setdefault((user_id, thread_id), []).append((sandbox_id, metadata)) + elif has_thread_identity: + # A VM from an older root must never be adopted by this + # provider. Reap it after the shorter duplicate grace once no + # peer still owns it, so it does not strand deployment capacity. + stale_entries.append( + ( + sandbox_id, + metadata, + float(self._config["reconciliation_grace_seconds"]), + ) + ) else: - orphans.append((sandbox_id, metadata)) + stale_entries.append( + ( + sandbox_id, + metadata, + float(self._config["reconciliation_orphan_ttl_seconds"]), + ) + ) for (user_id, thread_id), candidates in groups.items(): candidates.sort(key=lambda item: (item[1].get(META_KEY_CREATED_AT, ""), item[0])) with self._lock: - local_id = self._thread_sandboxes.get((user_id, thread_id)) + local_id = self._thread_sandboxes.get(self._thread_key(thread_id, user_id)) if local_id: candidates.sort(key=lambda item: item[0] != local_id) @@ -1557,7 +1664,7 @@ class E2BSandboxProvider(SandboxProvider): self._release_ownership(sandbox_id) self._safe_close_client(client) - for sandbox_id, metadata in orphans: + for sandbox_id, metadata, minimum_age in stale_entries: if time.monotonic() >= deadline: stats.budget_exhausted = True break @@ -1567,7 +1674,7 @@ class E2BSandboxProvider(SandboxProvider): age = time.time() - float(created_at) if created_at is not None else observed_at - first_seen except (TypeError, ValueError): age = observed_at - first_seen - if age < float(self._config["reconciliation_orphan_ttl_seconds"]): + if age < minimum_age: stats.deferred += 1 continue if not self._claim_ownership(sandbox_id, for_destroy=True): @@ -1777,9 +1884,10 @@ class E2BSandboxProvider(SandboxProvider): f"if [ ! -e /mnt/acp-workspace ] || [ -L /mnt/acp-workspace ]; then " f" sudo ln -sfn {shlex.quote(home_dir)}/acp-workspace /mnt/acp-workspace; " f"fi; " - # /mnt/skills is left alone here; the optional ``mounts`` config - # uploads its content via _apply_mounts and creates the directory - # on demand. We only ensure that /mnt itself is traversable. + # The configured skills root is left alone here; the optional + # ``mounts`` config uploads its content via _apply_mounts and + # creates the directory on demand. We only ensure that /mnt itself + # is traversable for the default layout. f"sudo chmod a+rx /mnt 2>/dev/null || true; " f"echo BOOTSTRAP_OK" ) @@ -1795,7 +1903,11 @@ class E2BSandboxProvider(SandboxProvider): if exit_code not in (0, None) or "BOOTSTRAP_OK" not in stdout: raise RuntimeError(f"e2b bootstrap script failed with exit code {exit_code}; stderr={stderr.strip()}") - def _skill_projection_mounts(self, user_id: str) -> list[tuple[Path, str, bool]]: + def _skill_projection_mounts( + self, + user_id: str, + thread_id: str | None = None, + ) -> list[tuple[Path, str, bool]]: """Best-effort: a projection failure must not drop configured mounts too. Unlike Local/AIO's ``_ensure_skills_projection``, this used to raise @@ -1805,14 +1917,28 @@ class E2BSandboxProvider(SandboxProvider): no mounts applied at all). Swallowing here keeps the two mount sources independent, matching the other two providers. """ + from deerflow.config.paths import get_paths from deerflow.skills.projection import ensure_skill_projections from deerflow.skills.storage import get_or_new_user_skill_storage try: + # The middleware performs a strict, fail-closed upload after acquire + # for a policy-scoped thread. Do not first upload the shared view and + # briefly hydrate the VM with skills outside the Agent allowlist. + if ( + thread_id + and get_paths() + .thread_skills_view_dir( + thread_id, + user_id=user_id, + ) + .exists() + ): + return [] config = get_app_config() storage = get_or_new_user_skill_storage(user_id, app_config=config) projection = ensure_skill_projections(storage) - container_root = config.skills.container_path.rstrip("/") + container_root = self._config["skills_container_path"] return [ (projection.public, f"{container_root}/public", True), (projection.custom, f"{container_root}/custom", True), @@ -1823,7 +1949,13 @@ class E2BSandboxProvider(SandboxProvider): logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True) return [] - def _apply_mounts(self, client: E2BClientSandbox, *, user_id: str | None = None) -> None: + def _apply_mounts( + self, + client: E2BClientSandbox, + *, + user_id: str | None = None, + thread_id: str | None = None, + ) -> None: started_at = time.monotonic() deadline_seconds = self._config.get("mount_upload_deadline_seconds", _MOUNT_PASS_DEADLINE_SECONDS) budget = _MountUploadBudget( @@ -1844,9 +1976,9 @@ class E2BSandboxProvider(SandboxProvider): ) effective_user_id = user_id or get_effective_user_id() - projection_mounts = self._skill_projection_mounts(effective_user_id) + projection_mounts = self._skill_projection_mounts(effective_user_id, thread_id) if thread_id is not None else self._skill_projection_mounts(effective_user_id) configured_mounts = self._config.get("mounts") or [] - skills_root = get_app_config().skills.container_path.rstrip("/") + skills_root = self._config["skills_container_path"] mounts: list[tuple[Path, str, bool]] = list(projection_mounts) for mount in configured_mounts: @@ -1893,6 +2025,95 @@ class E2BSandboxProvider(SandboxProvider): except Exception as e: logger.warning("Failed to upload mount %s -> %s: %s", host_path, container_path, e) + def sync_agent_skills( + self, + sandbox_id: str, + *, + thread_id: str, + user_id: str, + projection: SkillProjectionPaths, + ) -> None: + """Atomically rebuild E2B's managed skills for one thread scope. + + The remote reset and all four category uploads form one critical + section. Reuse the acquire/release lock for the same user/thread so a + concurrent policy cannot wipe or repopulate the sandbox mid-upload. + """ + with self._acquire_serializer.hold(self._thread_key(thread_id, user_id)): + self._sync_agent_skills_locked( + sandbox_id, + projection=projection, + ) + + def _sync_agent_skills_locked( + self, + sandbox_id: str, + *, + projection: SkillProjectionPaths, + ) -> None: + with self._lock: + sandbox = self._sandboxes.get(sandbox_id) + if sandbox is None: + raise RuntimeError(f"E2B sandbox {sandbox_id} is not available for skill synchronization") + + skills_root = _validate_skills_reset_root( + self._config["skills_container_path"], + home_dir=sandbox.home_dir, + ) + + # A sandbox-visible marker cannot prove integrity: the sandbox user can + # modify the marker and replace category directories between turns. + # Rebuild on every policy sync so the no-follow root check and managed + # directory replacement always run before the sandbox is handed out. + marker_path = f"{skills_root}/.deerflow-projection-signature" + + category_paths = [ + f"{skills_root}/public", + f"{skills_root}/custom", + f"{skills_root}/legacy", + f"{skills_root}/integrations", + ] + quoted_root = shlex.quote(skills_root) + quoted_categories = " ".join(shlex.quote(path) for path in category_paths) + quoted_managed_paths = " ".join(shlex.quote(path) for path in (*category_paths, marker_path)) + reset_script = ( + f"set -e; if [ -L {quoted_root} ]; then echo 'Refusing symlinked skills root' >&2; exit 2; fi; " + f"sudo rm -rf -- {quoted_managed_paths}; " + f"sudo mkdir -p -- {quoted_categories}; " + f'sudo chown "$(id -u):$(id -g)" -- {quoted_root} {quoted_categories}; ' + "echo SKILLS_RESET_OK" + ) + result = sandbox.client.commands.run(reset_script) + stdout = getattr(result, "stdout", "") or "" + stderr = getattr(result, "stderr", "") or "" + exit_code = getattr(result, "exit_code", 0) + if exit_code not in (0, None) or "SKILLS_RESET_OK" not in stdout: + raise RuntimeError(f"Failed to reset E2B skill projection (exit_code={exit_code}, stderr={stderr.strip()})") + + started_at = time.monotonic() + deadline_seconds = self._config.get("mount_upload_deadline_seconds", _MOUNT_PASS_DEADLINE_SECONDS) + budget = _MountUploadBudget( + deadline=started_at + deadline_seconds, + deadline_seconds=deadline_seconds, + ) + for source, destination in zip( + ( + projection.public, + projection.custom, + projection.legacy, + projection.integrations, + ), + category_paths, + strict=True, + ): + self._upload_tree( + sandbox.client, + source, + destination, + True, + budget=budget, + ) + # ── Output mirroring ──────────────────────────────────────────────── _SYNC_BACK_SUBDIRS = ("outputs", "workspace") _SYNC_MANIFEST_NAME = ".e2b-output-sync.json" @@ -2370,7 +2591,7 @@ class E2BSandboxProvider(SandboxProvider): self._release_internal(sandbox_id) return - user_id, thread_id = thread_key + user_id, thread_id, _skills_root = thread_key with self._acquire_serializer.hold(self._thread_key(thread_id, user_id)): self._release_internal(sandbox_id) @@ -2385,7 +2606,7 @@ class E2BSandboxProvider(SandboxProvider): """ sandbox: E2BSandbox | None = None seed: str | None = None - removed_keys: list[tuple[str, str]] = [] + removed_keys: list[tuple[str, str, str]] = [] transition_slot_held = False with self._lock: @@ -2398,7 +2619,7 @@ class E2BSandboxProvider(SandboxProvider): for key in removed_keys: self._thread_sandboxes.pop(key, None) if removed_keys: - user_id, thread_id = removed_keys[0] + user_id, thread_id, _skills_root = removed_keys[0] seed = self._stable_seed(thread_id, user_id) # E2BSandbox.close() clears its client reference. Keep this reference @@ -2416,7 +2637,7 @@ class E2BSandboxProvider(SandboxProvider): sync_failed_due_to_dead_vm = False if seed is not None and removed_keys: - user_id_sync, thread_id_sync = removed_keys[0] + user_id_sync, thread_id_sync, _skills_root = removed_keys[0] try: self._sync_outputs_to_host(sandbox, thread_id=thread_id_sync, user_id=user_id_sync) except Exception as e: # pragma: no cover - defensive diff --git a/backend/packages/harness/deerflow/config/AGENTS.md b/backend/packages/harness/deerflow/config/AGENTS.md index dbe6715a1..991677779 100644 --- a/backend/packages/harness/deerflow/config/AGENTS.md +++ b/backend/packages/harness/deerflow/config/AGENTS.md @@ -10,7 +10,7 @@ Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** direc **Config Hot-Reload Boundary**: Gateway dependencies route through `get_app_config()` on every request, so per-run fields like `models[*].max_tokens`, `summarization.*`, `title.*`, `memory.*`, `subagents.*`, `verification.*`, `tools[*]`, and the agent system prompt pick up `config.yaml` edits on the next message. `AppConfig` is intentionally **not** cached on `app.state` — `lifespan()` keeps a local `startup_config` variable for one-shot bootstrap work and passes it to `langgraph_runtime(app, startup_config)`. -Infrastructure fields are **restart-required**. The authoritative list lives in `packages/harness/deerflow/config/reload_boundary.py::STARTUP_ONLY_FIELDS` and is mirrored by the standardised `"startup-only:"` prefix on the corresponding `Field(description=...)` in `AppConfig`, so IDE hover on those fields surfaces the reason inline (no need to context-switch into this table). Currently registered: `plugins`, `database`, `checkpointer`, `run_events`, `stream_bridge`, `sandbox`, `log_level`, `logging`, `channels`, `channel_connections`, `scheduler`, `mcp_tasks`, `subagent_runtime`, `subagent_batches`, `run_ownership`. Adding a new restart-required field requires updating the registry; drift is pinned by `tests/test_reload_boundary.py`. `scheduler.recursion_limit` is the exception inside that section: it is read from `get_app_config()` at each scheduled dispatch, so a YAML edit applies to the next run without restarting the poller. +Infrastructure fields are **restart-required**. The authoritative list lives in `packages/harness/deerflow/config/reload_boundary.py::STARTUP_ONLY_FIELDS` and is mirrored by the standardised `"startup-only:"` prefix on the corresponding `Field(description=...)` in `AppConfig` or an explicitly registered nested config model, so IDE hover on those fields surfaces the reason inline (no need to context-switch into this table). Currently registered: `plugins`, `database`, `checkpointer`, `run_events`, `agent_storage`, `stream_bridge`, `sandbox`, `skills.container_path`, `log_level`, `logging`, `channels`, `channel_connections`, `scheduler`, `mcp_tasks`, `subagent_runtime`, `subagent_batches`, `run_ownership`, `dedupe_storage`. Adding a new restart-required field requires updating the registry; drift is pinned by `tests/test_reload_boundary.py`. `scheduler.recursion_limit` is the exception inside that section: it is read from `get_app_config()` at each scheduled dispatch, so a YAML edit applies to the next run without restarting the poller. **Persistence backend resolution**: the unified `database` section selects the Gateway's LangGraph checkpointer, LangGraph Store, and DeerFlow SQL repositories. @@ -53,7 +53,7 @@ Extensions are optional only in the fallback *search* mode (priority 3-4 above): - `tools[]` - Tool configs with `use` variable path and `group` - `tool_groups[]` - Logical groupings for tools - `sandbox.use` - Sandbox provider class path -- `skills.path` / `skills.container_path` - Host and container paths to skills directory +- `skills.path` / `skills.container_path` - Host and container paths to skills directory. AIO and E2B snapshot the container path at provider startup. Their local/remote backends and the Kubernetes provisioner require one canonical absolute non-root path outside reserved platform mounts; custom roots participate in deterministic sandbox identity, and E2B records the root in remote metadata. - `skills.deferred_discovery` - When `true`, replaces the full-metadata `` prompt block with a compact `` (names only) and registers the `describe_skill` tool so the agent fetches metadata on demand. Defaults to `false` (legacy full-metadata injection) - `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path) - `summarization` - Context summarization (enabled, trigger conditions, keep policy) diff --git a/backend/packages/harness/deerflow/config/paths.py b/backend/packages/harness/deerflow/config/paths.py index af5abafd3..f793db6c1 100644 --- a/backend/packages/harness/deerflow/config/paths.py +++ b/backend/packages/harness/deerflow/config/paths.py @@ -295,6 +295,18 @@ class Paths: """Enabled managed integration skills exposed to one user's sandboxes.""" return self.user_skills_view_dir(user_id) / "integrations" + def thread_skills_view_dir(self, thread_id: str, *, user_id: str) -> Path: + """Sandbox-visible skill projection scoped to one user/thread. + + The directory lives below the thread root so ordinary thread deletion + also removes its policy projection. + """ + return self.thread_dir(thread_id, user_id=user_id) / "skills_view" + + def host_thread_skills_view_dir(self, thread_id: str, *, user_id: str) -> str: + """Host path for a thread-scoped skill projection.""" + return _join_host_path(self.host_thread_dir(thread_id, user_id=user_id), "skills_view") + def thread_dir(self, thread_id: str, *, user_id: str | None = None) -> Path: """ Host path for a thread's data. diff --git a/backend/packages/harness/deerflow/config/reload_boundary.py b/backend/packages/harness/deerflow/config/reload_boundary.py index 31488d063..0d1e72e05 100644 --- a/backend/packages/harness/deerflow/config/reload_boundary.py +++ b/backend/packages/harness/deerflow/config/reload_boundary.py @@ -10,7 +10,7 @@ change at runtime. The registry covers two kinds of entries: -- Top-level ``AppConfig`` fields (``database``, ``checkpointer``, +- ``AppConfig`` fields and explicitly registered nested fields (``database``, ``checkpointer``, ``run_events``, ``stream_bridge``, ``sandbox``, ``log_level``). For these, :func:`format_field_description` produces the standardised ``"startup-only: ..."`` prefix that the matching Pydantic @@ -50,6 +50,10 @@ STARTUP_ONLY_FIELDS: dict[str, str] = { "agent_storage": ("langgraph_runtime() validates agent_storage.backend against database.backend once at startup, and the db backend's synchronous SQLAlchemy engine is process-cached on first use; switching backend needs a restart."), "stream_bridge": ("make_stream_bridge() constructs the stream-bridge singleton once during startup."), "sandbox": ("get_sandbox_provider() caches the provider singleton (``_default_sandbox_provider``); a different ``sandbox.use`` class path only takes effect on next process start."), + "skills.container_path": ( + "AioSandboxProvider and E2BSandboxProvider normalize and capture the skills mount root when their provider singleton starts; " + "sandbox identity, mounts, remote metadata, and skill synchronization must keep using that one root until the Gateway restarts." + ), "log_level": ( "apply_logging_level() runs only during app.py startup; it sets the deerflow/app logger levels and may lower root handler thresholds so configured messages can propagate. A freshly reloaded AppConfig does not retrigger it." ), @@ -97,9 +101,9 @@ def iter_startup_only_field_paths() -> Iterator[str]: def is_startup_only_field(field_path: str) -> bool: """Return ``True`` when *field_path* is registered as restart-required. - Accepts only top-level paths (``"database"``, ``"sandbox"`` etc.); - nested keys like ``"database.url"`` are not modelled here because the - boundary is per-section, not per-leaf. + Most entries are top-level paths (``"database"``, ``"sandbox"`` etc.). + A nested path is registered only when one leaf has a different reload + boundary from the rest of its section, such as ``skills.container_path``. """ return field_path in STARTUP_ONLY_FIELDS @@ -112,7 +116,8 @@ def format_field_description(field_path: str, *, field_doc: str | None = None) - side against the other. Args: - field_path: A registered top-level field path (e.g. ``"log_level"``). + field_path: A registered field path (e.g. ``"log_level"`` or + ``"skills.container_path"``). field_doc: Optional human-facing description for the field itself (allowed values, semantics, etc.). When supplied, it is appended after the ``startup-only:`` marker block separated by diff --git a/backend/packages/harness/deerflow/config/skills_config.py b/backend/packages/harness/deerflow/config/skills_config.py index d18cca884..38cbcbf0d 100644 --- a/backend/packages/harness/deerflow/config/skills_config.py +++ b/backend/packages/harness/deerflow/config/skills_config.py @@ -3,6 +3,7 @@ from pathlib import Path from pydantic import BaseModel, Field +from deerflow.config.reload_boundary import format_field_description from deerflow.config.runtime_paths import project_root, resolve_path from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH @@ -27,7 +28,10 @@ class SkillsConfig(BaseModel): ) container_path: str = Field( default=DEFAULT_SKILLS_CONTAINER_PATH, - description="Path where skills are mounted in the sandbox container", + description=format_field_description( + "skills.container_path", + field_doc="Path where skills are mounted in the sandbox container.", + ), ) deferred_discovery: bool = Field( default=False, diff --git a/backend/packages/harness/deerflow/sandbox/AGENTS.md b/backend/packages/harness/deerflow/sandbox/AGENTS.md index 1ce36c344..a7f0e92af 100644 --- a/backend/packages/harness/deerflow/sandbox/AGENTS.md +++ b/backend/packages/harness/deerflow/sandbox/AGENTS.md @@ -1,18 +1,33 @@ ### Sandbox System (`packages/harness/deerflow/sandbox/`) **Interface**: Abstract `Sandbox` with `execute_command(command, env=None)`, `read_file`, `write_file`, `list_dir`, `glob`, and `grep`. `grep` accepts either one text file or a directory tree. The optional `env` injects per-call environment variables (request-scoped secrets — see Request-Scoped Secrets below); `LocalSandbox` merges it into the host subprocess environment and `AioSandbox` routes env-bearing commands through the `bash.exec(env=...)` API on a fresh session. -**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop. -**Shared components** (RFC #4741): remote providers derive their deterministic sandbox id through `derive_sandbox_scope_token` (`sandbox/identity.py`, keyword-only; the sha256/16-hex derivation is a compatibility contract — changing it orphans existing containers), and serialize provider-selected acquire/release transitions through `AcquireSerializer` (`sandbox/acquire_serialization.py`): per-key `threading.Lock` table with holder/waiter refcount reclamation (no unbounded per-thread lock growth), a bounded dedicated executor so async waits never touch the event loop or the default executor, worker-owned cancellation cleanup that does not depend on a cancelled event loop task resuming, and idempotent `close()` called from provider `shutdown()`/`reset()`. AIO/E2B key by `(user_id, thread_id)`; BoxLite/Tenki/OpenSandbox key by the derived sandbox id. `thread_id=None` acquires (random uuid ids) never enter the serializer. +**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop. Providers that can enforce a lead Agent's explicit skill policy across the current Agent-accessible tool surface set `supports_agent_skill_isolation=True`; bind-mount providers observe the prepared thread roots directly, while upload providers implement `sync_agent_skills`. Host-backed providers must report the capability as false whenever an enabled shell can bypass their path mappings. The middleware fails closed before acquisition for an explicit policy on an unsupported provider. +**Shared components** (RFC #4741): remote providers derive their deterministic sandbox id through `derive_sandbox_scope_token` (`sandbox/identity.py`, keyword-only; the sha256/16-hex derivation is a compatibility contract — changing it orphans existing containers), and serialize provider-selected acquire/release transitions through `AcquireSerializer` (`sandbox/acquire_serialization.py`): per-key `threading.Lock` table with holder/waiter refcount reclamation (no unbounded per-thread lock growth), a bounded dedicated executor so async waits never touch the event loop or the default executor, worker-owned cancellation cleanup that does not depend on a cancelled event loop task resuming, and idempotent `close()` called from provider `shutdown()`/`reset()`. AIO keys by `(user_id, thread_id)`; E2B keys by `(user_id, thread_id, skills_root)`; BoxLite/Tenki/OpenSandbox key by the derived sandbox id. `thread_id=None` acquires (random uuid ids) never enter the serializer. **Authorization gate** (`sandbox:execute`, RFC #4063 Phase 3): every sandbox-backed tool call passes through the gate in `deerflow/authz/sandbox_authz.py` - a binary `authorize(principal, "sandbox", "execute", target="*")` check before either reusing a persisted sandbox id or calling `provider.acquire`. Rechecking reuse is required because authorization config and user roles can change while the sandbox remains cached. Sync tool invocations call `authorize_sandbox_execution`; async tool invocations await `authorize_sandbox_execution_async` exactly once. A task-local `ContextVar` scopes that single decision across the complete composed tool invocation, including `ReadBeforeWriteMiddleware`'s pre-write inspection, tool body, and post-read mark; the value is copied into `asyncio.to_thread` workers. Authorization denial is converted to the normal error `ToolMessage` at the composed middleware boundary and is explicitly excluded from the gate's generic fail-open handlers. Async config loading and provider class discovery/import are offloaded before `aauthorize()` so reused sandbox calls do not hash config files or import custom modules on the event loop; provider construction remains on the running event loop because async providers may initialize loop-affine clients. The gate lives at the single tool initialization entry point (`ensure_sandbox_initialized` / `ensure_sandbox_initialized_async` in `tools.py`), while `SandboxMiddleware.before_agent` / `abefore_agent` apply the matching sync/async check to eager acquisition. Deny raises `SandboxAuthorizationError` (`sandbox/exceptions.py`), which propagates out of ordinary tool execution as a friendly error `ToolMessage` ("sandbox execution is not permitted for your role") - the eager path catches it and skips acquisition instead, deferring the deny to the first sandbox-touching tool call so both paths share the same semantics. Provider errors (authorization calls and provider resolution) follow `authorization.fail_closed` / `fail_open`; no readable `config.yaml` or `authorization.enabled: false` makes the gate a no-op (`safe_app_config` tolerates missing config). Gateway auxiliary sync paths (uploads/artifacts routers) call `try_acquire_sandbox_for_request` (`app/gateway/authz.py`), which gates via `authorize_sandbox_for_request` and skips the sync on deny - the upload/artifact edit itself still succeeds. Tests: `tests/test_sandbox_authorization.py` and `tests/blocking_io/test_sandbox_authorization.py`. **Environment policy** (`sandbox/env_policy.py`): `execute_command` no longer inherits the full `os.environ`. `build_sandbox_env()` scrubs secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. `OPENAI_API_KEY`) never leak into skill subprocesses. Benign vars (`PATH`, `HOME`, `LANG`, `VIRTUAL_ENV`, ...) are preserved. **Implementations**: -- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Public, custom, legacy, and managed integration skill mappings point at stable enabled-only projection roots rather than raw skill directories. On Windows, Git Bash/MSYS argument-conversion exclusions are limited to safe non-root virtual path prefixes; do not restore a blanket conversion disable, because host-native CLI launchers need normal MSYS path conversion for their own installation paths. -- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). `reset()` closes the per-instance acquire serializer so replacing the singleton cannot retain its executor workers; full remote sandbox teardown remains `shutdown()`. `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. Local-container and hostPath-provisioner mounts use the same stable skill projection roots; PVC-backed skills remain governed by the operator-supplied PVC layout until PVC materialization is implemented. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support. +- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-user/thread `LocalSandbox` (id `local:{user_id}:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Shared runs use category mappings; a policy-scoped run replaces them with one `/mnt/skills` root mapping to the coherent thread view, so structured file tools resolve through one managed boundary. This is not a host filesystem security boundary: an enabled host `bash` subprocess can use canonical paths without `PathMapping`, so `supports_agent_skill_isolation` is dynamic and explicit Agent policies fail closed while host bash is enabled. On Windows, Git Bash/MSYS argument-conversion exclusions are limited to safe non-root virtual path prefixes; do not restore a blanket conversion disable, because host-native CLI launchers need normal MSYS path conversion for their own installation paths. +- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). `reset()` closes the per-instance acquire serializer so replacing the singleton cannot retain its executor workers; full remote sandbox teardown remains `shutdown()`. `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. An explicit Agent policy uses four thread projection category mounts and a distinct deterministic sandbox identity, preventing reuse of an older container created with shared mounts. `skills.container_path` is a provider-startup snapshot shared by mount construction, sandbox identity, the remote Gateway request, and provisioner validation; custom roots are identity-scoped so a container or Pod created for one destination cannot be reused after the root changes. The Gateway and provisioner independently require one canonical absolute root that does not overlap reserved platform mounts, and both derive the four category allowlist entries from that root. The provisioner accepts all four category overrides; when all are present it suppresses the default hostPath or skills-PVC mount. With `USERDATA_PVC_NAME`, the thread projection categories use subpaths on that shared data PVC. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support. - `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation. - New sandboxes receive a one-shot upload from the enabled-only public, custom, - legacy, and managed integration projections. Existing E2B VMs keep their - creation-time snapshot because E2B has no shared host mount. - Acquire and release share an `AcquireSerializer` hold keyed by `(user_id, thread_id)`. The serializer does + New unrestricted sandboxes receive a one-shot upload from the enabled-only + public, custom, legacy, and managed integration projections. For a thread + with an explicit Agent policy, creation skips that shared upload; after + acquire, `sync_agent_skills` clears only DeerFlow's four managed category + directories and signature before strictly uploading the signed thread + projection. It rejects non-canonical paths, protected mounts/homes, symlinked + roots, and standard operating-system trees before destructive work. E2B + rebuilds the managed categories on every policy sync; it deletes legacy + sandbox-visible signature markers instead of trusting them as proof that the + remote tree is intact. Reset plus all four uploads hold the same per-user/thread + and skills-root serializer used by acquire and release, so overlapping policy syncs cannot interleave + a wipe with another run's upload. Delegated subagents are non-owners of the lead's thread + projection: they reuse that filesystem view and never rebuild it from their + discovery/activation policy. + The provider snapshots `skills.container_path` at startup and carries it through + mount construction, the warm-pool seed, remote metadata, discovery, reconciliation, + and synchronization. A VM from a different root is never adopted and is reaped + after the duplicate grace period once no live peer owns it. Acquire and release + share an `AcquireSerializer` hold keyed by `(user_id, thread_id, skills_root)`. The serializer does not cover remote IO. `burst_limit` adds capacity only for the `burst` policy. The `wait` policy fails the turn after `acquire_timeout`. The runtime does not retry the turn automatically. E2B acquisition uses a bounded executor. @@ -73,7 +88,7 @@ **Virtual Path System**: - Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills` -- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`; raw skills stay under `deer-flow/skills/` and managed integration storage, while sandboxes read `backend/.deer-flow/skills_view/public/` and `backend/.deer-flow/users/{user_id}/skills_view/{custom,legacy,integrations}/` +- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`; raw skills stay under `deer-flow/skills/` and managed integration storage. Unrestricted sandboxes read `backend/.deer-flow/skills_view/public/` and `backend/.deer-flow/users/{user_id}/skills_view/{custom,legacy,integrations}/`; explicit lead Agent policies read `backend/.deer-flow/users/{user_id}/threads/{thread_id}/skills_view/{public,custom,legacy,integrations}/`. - Translation: `LocalSandboxProvider` builds per-thread `PathMapping`s at acquire time. Sandbox-backed readers resolve only `/mnt/user-data/...` in the tool layer; skills, ACP workspaces, and configured custom mounts stay virtual so the provider mount table remains the single source of acquire-time identity and visibility. Full reads, ranged reads, and read-before-write hashing share this path. `tools.py` keeps `replace_virtual_path()` / `replace_virtual_paths_in_command()` as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept `/mnt/user-data/...` natively. - Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread) diff --git a/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py b/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py index 36ef8d7f6..bed20181a 100644 --- a/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py +++ b/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py @@ -3,9 +3,11 @@ import threading from collections import OrderedDict from pathlib import Path +from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH from deerflow.sandbox.local.local_sandbox import LocalSandbox, PathMapping from deerflow.sandbox.sandbox import Sandbox from deerflow.sandbox.sandbox_provider import SandboxProvider +from deerflow.sandbox.security import is_host_bash_allowed logger = logging.getLogger(__name__) @@ -60,11 +62,27 @@ class LocalSandboxProvider(SandboxProvider): next ``acquire``; the evicted thread's next ``acquire`` rebuilds a fresh sandbox (losing only its ``_agent_written_paths`` reverse-resolve hint, which gracefully degrades read_file output). + + The managed ``/mnt/skills`` projection is a logical boundary, not a host + filesystem security boundary. When host bash is enabled, a subprocess can + address canonical host paths without going through ``PathMapping``. The + provider therefore advertises Agent skill isolation only while host bash + remains disabled. """ uses_thread_data_mounts = True needs_upload_permission_adjustment = False + @property + def supports_agent_skill_isolation(self) -> bool: + """Whether the current tool surface can enforce the managed view.""" + try: + return not is_host_bash_allowed() + except Exception: + # An unreadable config must not turn a host-process provider into + # an isolation boundary by accident. + return False + def __init__(self, max_cached_threads: int = DEFAULT_MAX_CACHED_THREAD_SANDBOXES): """Initialize the local sandbox provider with static path mappings. @@ -73,6 +91,7 @@ class LocalSandboxProvider(SandboxProvider): the LRU cache. When exceeded, the least-recently-used entry is evicted on the next ``acquire``. """ + self._skills_container_path = DEFAULT_SKILLS_CONTAINER_PATH self._path_mappings = self._setup_path_mappings() self._generic_sandbox: LocalSandbox | None = None self._thread_sandboxes: OrderedDict[tuple[str, str], LocalSandbox] = OrderedDict() @@ -101,6 +120,7 @@ class LocalSandboxProvider(SandboxProvider): config = get_app_config() container_path = config.skills.container_path + self._skills_container_path = container_path.rstrip("/") projection = self._ensure_skills_projection() # Public skills: global, read-only — static, shared by all threads @@ -132,6 +152,7 @@ class LocalSandboxProvider(SandboxProvider): # ``/mnt/skills/custom`` to the init-time user's directory. # Map custom mounts from sandbox config + _RESERVED_CONTAINER_PATHS = {container_path} _RESERVED_CONTAINER_PREFIXES = [ f"{container_path}/public", f"{container_path}/custom", @@ -163,7 +184,7 @@ class LocalSandboxProvider(SandboxProvider): continue # Reject mounts that conflict with reserved container paths - if any(container_path == p or container_path.startswith(p + "/") for p in _RESERVED_CONTAINER_PREFIXES): + if container_path in _RESERVED_CONTAINER_PATHS or any(container_path == p or container_path.startswith(p + "/") for p in _RESERVED_CONTAINER_PREFIXES): logger.warning( "Mount container_path conflicts with reserved prefix, skipping: %s", mount.container_path, @@ -217,7 +238,7 @@ class LocalSandboxProvider(SandboxProvider): return (user_id, thread_id) @staticmethod - def _ensure_skills_projection(user_id: str | None = None): + def _ensure_skills_projection(user_id: str | None = None, *, thread_id: str | None = None): """Best-effort: a projection failure must not fail sandbox acquire. Mirrors the surrounding skill-mount setup, which has always logged @@ -227,7 +248,11 @@ class LocalSandboxProvider(SandboxProvider): acquire once the underlying condition clears. """ from deerflow.config import get_app_config - from deerflow.skills.projection import ensure_skill_projections + from deerflow.skills.projection import ( + ensure_skill_projections, + get_thread_skill_projection_paths, + thread_skill_projection_exists, + ) from deerflow.skills.storage import get_or_new_skill_storage, get_or_new_user_skill_storage try: @@ -236,9 +261,17 @@ class LocalSandboxProvider(SandboxProvider): storage = get_or_new_skill_storage(app_config=config) else: storage = get_or_new_user_skill_storage(user_id, app_config=config) + if thread_id is not None and thread_skill_projection_exists(storage, thread_id): + return get_thread_skill_projection_paths(storage, thread_id) return ensure_skill_projections(storage) except Exception as exc: - logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True) + logger.warning( + "Could not ensure skills projection for user/thread %s/%s: %s", + user_id, + thread_id, + exc, + exc_info=True, + ) return None @staticmethod @@ -326,7 +359,7 @@ class LocalSandboxProvider(SandboxProvider): ), ] - # Per-user category mounts stay present for the sandbox lifetime. Their + # Category mounts stay present for the sandbox lifetime. Their # enabled-only contents change beneath these stable roots. try: config = get_app_config() @@ -334,30 +367,73 @@ class LocalSandboxProvider(SandboxProvider): projection = skill_projection if skill_projection is not None else LocalSandboxProvider._ensure_skills_projection(effective_user_id) if projection is not None: - mappings.extend( - [ - PathMapping( - container_path=f"{skills_container_path}/custom", - local_path=str(projection.custom), - read_only=True, - ), - PathMapping( - container_path=f"{skills_container_path}/legacy", - local_path=str(projection.legacy), - read_only=True, - ), - PathMapping( - container_path=f"{skills_container_path}/integrations", - local_path=str(projection.integrations), - read_only=True, - ), - ] + thread_projection_root = paths.thread_skills_view_dir( + thread_id, + user_id=effective_user_id, ) + if projection.public.parent == thread_projection_root: + mappings.append( + PathMapping( + container_path=skills_container_path, + local_path=str(thread_projection_root), + read_only=True, + ) + ) + else: + mappings.extend( + [ + PathMapping( + container_path=f"{skills_container_path}/public", + local_path=str(projection.public), + read_only=True, + ), + PathMapping( + container_path=f"{skills_container_path}/custom", + local_path=str(projection.custom), + read_only=True, + ), + PathMapping( + container_path=f"{skills_container_path}/legacy", + local_path=str(projection.legacy), + read_only=True, + ), + PathMapping( + container_path=f"{skills_container_path}/integrations", + local_path=str(projection.integrations), + read_only=True, + ), + ] + ) except Exception as exc: logger.warning("Could not setup per-thread skills projection mounts: %s", exc, exc_info=True) return mappings + def _without_managed_skill_mappings( + self, + mappings: list[PathMapping], + *, + policy_scoped: bool, + ) -> list[PathMapping]: + """Drop mappings that would overlap the selected managed skill view. + + Ordinary sandboxes retain operator-defined mounts elsewhere below the + configured skills root for backward compatibility. A policy-scoped + sandbox removes the entire subtree before installing its coherent root + mapping, so a nested custom mount cannot bypass the Agent allowlist. + """ + root = self._skills_container_path + managed_categories = tuple(f"{root}/{category}" for category in ("public", "custom", "legacy", "integrations")) + + def conflicts(mapping: PathMapping) -> bool: + path = mapping.container_path.rstrip("/") + if path == root: + return True + prefixes = (root,) if policy_scoped else managed_categories + return any(path == prefix or path.startswith(prefix + "/") for prefix in prefixes) + + return [mapping for mapping in mappings if not conflicts(mapping)] + def acquire(self, thread_id: str | None = None, *, user_id: str | None = None) -> str: """Return a sandbox id scoped to *thread_id* (or the generic singleton). @@ -390,23 +466,28 @@ class LocalSandboxProvider(SandboxProvider): # triggers a full rebuild (~400 ms measured locally) under the # cross-process projection lock, serializing concurrent acquires and # mutations for that user. Acceptable for an editing-frequency event. - skill_projection = self._ensure_skills_projection(effective_user_id) + skill_projection = self._ensure_skills_projection( + effective_user_id, + thread_id=thread_id, + ) key = self._thread_key(thread_id, effective_user_id) - # Fast path under lock. - with self._lock: - cached = self._thread_sandboxes.get(key) - if cached is not None: - # Mark as most-recently used so frequently-touched threads - # survive eviction. - self._thread_sandboxes.move_to_end(key) - if cached is not None: - return cached.id - # ``_build_thread_path_mappings`` touches the filesystem # (``ensure_thread_dirs``); release the lock during I/O. - new_mappings = list(self._path_mappings) - self._append_public_skill_mapping(new_mappings, skill_projection) + from deerflow.config.paths import get_paths + + policy_scoped = bool( + skill_projection is not None + and skill_projection.public.parent + == get_paths().thread_skills_view_dir( + thread_id, + user_id=effective_user_id, + ) + ) + new_mappings = self._without_managed_skill_mappings( + list(self._path_mappings), + policy_scoped=policy_scoped, + ) new_mappings += self._build_thread_path_mappings( thread_id, user_id=effective_user_id, @@ -414,11 +495,15 @@ class LocalSandboxProvider(SandboxProvider): ) with self._lock: - # Re-check after the lock-free I/O: another caller may have - # populated the cache while we were computing mappings. cached = self._thread_sandboxes.get(key) - if cached is None: - cached = LocalSandbox(self._sandbox_id_for_thread(thread_id, effective_user_id), path_mappings=new_mappings) + if cached is None or cached.path_mappings != new_mappings: + replacement = LocalSandbox( + self._sandbox_id_for_thread(thread_id, effective_user_id), + path_mappings=new_mappings, + ) + if cached is not None: + replacement._agent_written_paths.update(cached._agent_written_paths) + cached = replacement self._thread_sandboxes[key] = cached self._evict_until_within_cap_locked() else: diff --git a/backend/packages/harness/deerflow/sandbox/middleware.py b/backend/packages/harness/deerflow/sandbox/middleware.py index ed7e8e758..f04257375 100644 --- a/backend/packages/harness/deerflow/sandbox/middleware.py +++ b/backend/packages/harness/deerflow/sandbox/middleware.py @@ -9,7 +9,7 @@ from langchain.agents.middleware import AgentMiddleware from langchain_core.messages import ToolMessage from langgraph.prebuilt.tool_node import ToolCallRequest from langgraph.runtime import Runtime -from langgraph.types import Command +from langgraph.types import Command, Overwrite from deerflow.agents.thread_state import SandboxStateField, ThreadDataState from deerflow.authz.sandbox_authz import ( @@ -20,7 +20,7 @@ from deerflow.authz.sandbox_authz import ( ) from deerflow.runtime.user_context import resolve_runtime_user_id from deerflow.sandbox import get_sandbox_provider -from deerflow.sandbox.exceptions import SandboxAuthorizationError +from deerflow.sandbox.exceptions import SandboxAuthorizationError, SandboxRuntimeError from deerflow.sandbox.overwrite import unwrap_sandbox logger = logging.getLogger(__name__) @@ -46,16 +46,67 @@ class SandboxMiddleware(AgentMiddleware[SandboxMiddlewareState]): state_schema = SandboxMiddlewareState - def __init__(self, lazy_init: bool = True): + def __init__( + self, + lazy_init: bool = True, + *, + available_skills: set[str] | None = None, + owns_agent_skill_projection: bool = True, + ): """Initialize sandbox middleware. Args: lazy_init: If True, defer sandbox acquisition until first tool call. If False, acquire sandbox eagerly in before_agent(). Default is True for optimal performance. + owns_agent_skill_projection: Whether this middleware may create or + rebuild the thread's physical skill projection. Delegated + subagents share the lead thread sandbox and must preserve the + lead-owned view instead of applying their discovery policy to it. """ super().__init__() self._lazy_init = lazy_init + self._available_skills = set(available_skills) if available_skills is not None else None + self._owns_agent_skill_projection = owns_agent_skill_projection + + def _prepare_agent_skill_projection(self, thread_id: str, *, user_id: str): + """Build the run's physical skill view before any sandbox is reused.""" + if not self._owns_agent_skill_projection: + # Subagents inherit the lead's thread id and sandbox state. Their + # skill lists scope discovery/activation only; rebuilding here + # would widen or narrow the shared filesystem for every concurrent + # agent using this sandbox. + return None + + from deerflow.config.paths import get_paths + + # Preserve the zero-copy shared view for ordinary threads. A thread + # that previously used a restricted Agent keeps its stable mount root; + # an unrestricted run repopulates that root with all enabled skills. + if self._available_skills is None and not get_paths().thread_skills_view_dir(thread_id, user_id=user_id).exists(): + return None + + provider = get_sandbox_provider() + if not provider.supports_agent_skill_isolation: + if self._available_skills is not None: + raise SandboxRuntimeError(f"Sandbox provider {provider.__class__.__name__} cannot enforce per-Agent skill filesystem isolation") + # The thread projection may have been created under a different + # provider. An unrestricted run does not need that policy view and + # may safely use this provider's ordinary shared skill behavior. + return None + + from deerflow.config import get_app_config + from deerflow.skills.projection import ensure_thread_skill_projection + from deerflow.skills.storage import get_or_new_user_skill_storage + + app_config = get_app_config() + storage = get_or_new_user_skill_storage(user_id, app_config=app_config) + return ensure_thread_skill_projection(storage, thread_id, self._available_skills) + + @staticmethod + def _require_projection_support(provider, projection) -> None: + if projection is not None and not provider.supports_agent_skill_isolation: + raise SandboxRuntimeError(f"Sandbox provider {provider.__class__.__name__} cannot enforce per-Agent skill filesystem isolation") def _acquire_sandbox(self, thread_id: str, *, user_id: str) -> str: provider = get_sandbox_provider() @@ -74,62 +125,109 @@ class SandboxMiddleware(AgentMiddleware[SandboxMiddlewareState]): @override def before_agent(self, state: SandboxMiddlewareState, runtime: Runtime) -> dict | None: - # Skip acquisition if lazy_init is enabled - if self._lazy_init: + thread_id = (runtime.context or {}).get("thread_id") + if thread_id is None: + return super().before_agent(state, runtime) + user_id = resolve_runtime_user_id(runtime) + projection = self._prepare_agent_skill_projection(thread_id, user_id=user_id) + + # Preserve lazy initialization for threads that use the shared view. + # A policy-scoped view is acquired eagerly so an old shared-view + # sandbox cannot survive into this run through checkpoint state. + if self._lazy_init and projection is None: return super().before_agent(state, runtime) - # Eager initialization (original behavior) - if "sandbox" not in state or state["sandbox"] is None: - thread_id = (runtime.context or {}).get("thread_id") - if thread_id is None: - return super().before_agent(state, runtime) + existing_sandbox_id = self._read_sandbox_id_from_state(state) + if existing_sandbox_id is None or projection is not None: # Phase 3: enforce sandbox:execute authorization before acquiring # (eager path). On deny, skip the eager acquisition instead of # raising: an exception here is outside any tool call, so it would # surface as a run-level graph error rather than the RFC §9 - # friendly ToolMessage. Skipping defers to the lazy gate inside - # ``ensure_sandbox_initialized``, which denies per-tool with the - # friendly message on the first sandbox-touching tool call. + # friendly ToolMessage. Shared-view runs skip and defer to the lazy + # gate inside ``ensure_sandbox_initialized``. Policy-scoped runs + # abort here because retaining an older checkpointed sandbox would + # bypass the new filesystem view. try: authorize_sandbox_execution( context=runtime.context or {}, app_config=safe_app_config(), ) except SandboxAuthorizationError: + if projection is not None: + # An explicit skill policy cannot leave a checkpointed, + # previously shared sandbox reusable by downstream tools. + # Abort this run before the model can reach that state. + raise logger.info("Sandbox execution denied for this role; skipping eager sandbox acquisition (thread_id=%s)", thread_id) return None - sandbox_id = self._acquire_sandbox(thread_id, user_id=resolve_runtime_user_id(runtime)) + provider = get_sandbox_provider() + self._require_projection_support(provider, projection) + sandbox_id = self._acquire_sandbox(thread_id, user_id=user_id) + if projection is not None: + provider.sync_agent_skills( + sandbox_id, + thread_id=thread_id, + user_id=user_id, + projection=projection, + ) logger.info(f"Assigned sandbox {sandbox_id} to thread {thread_id}") + if existing_sandbox_id == sandbox_id: + return super().before_agent(state, runtime) + if existing_sandbox_id is not None: + return { + "sandbox": Overwrite({"sandbox_id": sandbox_id}), + } return {"sandbox": {"sandbox_id": sandbox_id}} return super().before_agent(state, runtime) @override async def abefore_agent(self, state: SandboxMiddlewareState, runtime: Runtime) -> dict | None: - # Skip acquisition if lazy_init is enabled - if self._lazy_init: + thread_id = (runtime.context or {}).get("thread_id") + if thread_id is None: + return await super().abefore_agent(state, runtime) + user_id = resolve_runtime_user_id(runtime) + projection = await asyncio.to_thread( + self._prepare_agent_skill_projection, + thread_id, + user_id=user_id, + ) + + if self._lazy_init and projection is None: return await super().abefore_agent(state, runtime) - # Eager initialization (original behavior), but use the async provider - # hook so blocking sandbox startup/polling runs outside the event loop. - if "sandbox" not in state or state["sandbox"] is None: - thread_id = (runtime.context or {}).get("thread_id") - if thread_id is None: - return await super().abefore_agent(state, runtime) + existing_sandbox_id = self._read_sandbox_id_from_state(state) + if existing_sandbox_id is None or projection is not None: # Phase 3: enforce sandbox:execute authorization before acquiring # (eager path, async counterpart of the gate in before_agent). On - # deny, skip the eager acquisition — the lazy gate inside - # ``ensure_sandbox_initialized`` denies per-tool with the RFC §9 - # friendly message on the first sandbox-touching tool call. + # deny, shared-view runs skip and defer to the lazy tool gate; + # policy-scoped runs abort before an older sandbox can be reused. try: await authorize_sandbox_execution_async( context=runtime.context or {}, app_config=await safe_app_config_async(), ) except SandboxAuthorizationError: + if projection is not None: + raise logger.info("Sandbox execution denied for this role; skipping eager sandbox acquisition (thread_id=%s)", thread_id) return None - sandbox_id = await self._acquire_sandbox_async(thread_id, user_id=resolve_runtime_user_id(runtime)) + provider = get_sandbox_provider() + self._require_projection_support(provider, projection) + sandbox_id = await self._acquire_sandbox_async(thread_id, user_id=user_id) + if projection is not None: + await provider.sync_agent_skills_async( + sandbox_id, + thread_id=thread_id, + user_id=user_id, + projection=projection, + ) logger.info(f"Assigned sandbox {sandbox_id} to thread {thread_id}") + if existing_sandbox_id == sandbox_id: + return await super().abefore_agent(state, runtime) + if existing_sandbox_id is not None: + return { + "sandbox": Overwrite({"sandbox_id": sandbox_id}), + } return {"sandbox": {"sandbox_id": sandbox_id}} return await super().abefore_agent(state, runtime) @@ -198,7 +296,7 @@ class SandboxMiddleware(AgentMiddleware[SandboxMiddlewareState]): def _read_sandbox_id_from_state(state: object) -> str | None: if not isinstance(state, dict): return None - sandbox_state = state.get("sandbox") + sandbox_state, _ = unwrap_sandbox(state.get("sandbox")) if not isinstance(sandbox_state, dict): return None sandbox_id = sandbox_state.get("sandbox_id") diff --git a/backend/packages/harness/deerflow/sandbox/sandbox_provider.py b/backend/packages/harness/deerflow/sandbox/sandbox_provider.py index 58ee72604..80c18d4f1 100644 --- a/backend/packages/harness/deerflow/sandbox/sandbox_provider.py +++ b/backend/packages/harness/deerflow/sandbox/sandbox_provider.py @@ -1,17 +1,25 @@ import asyncio import threading from abc import ABC, abstractmethod +from typing import TYPE_CHECKING from deerflow.config import get_app_config from deerflow.reflection import resolve_class from deerflow.sandbox.sandbox import Sandbox +if TYPE_CHECKING: + from deerflow.skills.projection import SkillProjectionPaths + class SandboxProvider(ABC): """Abstract base class for sandbox providers""" uses_thread_data_mounts: bool = False needs_upload_permission_adjustment: bool = True + # Capability for enforcing a lead Agent's physical skill view across the + # provider's current Agent-accessible tool surface. Host-backed providers + # must return False whenever shell access can bypass managed path mappings. + supports_agent_skill_isolation: bool = False @abstractmethod def acquire(self, thread_id: str | None = None, *, user_id: str | None = None) -> str: @@ -32,6 +40,37 @@ class SandboxProvider(ABC): """ return await asyncio.to_thread(self.acquire, thread_id, user_id=user_id) + def sync_agent_skills( + self, + sandbox_id: str, + *, + thread_id: str, + user_id: str, + projection: "SkillProjectionPaths", + ) -> None: + """Synchronize a prepared thread skill projection into a sandbox. + + Bind-mount providers observe the stable projection roots directly and + use this no-op implementation. Upload-based providers override it. + """ + + async def sync_agent_skills_async( + self, + sandbox_id: str, + *, + thread_id: str, + user_id: str, + projection: "SkillProjectionPaths", + ) -> None: + """Async wrapper for upload-based skill synchronization.""" + await asyncio.to_thread( + self.sync_agent_skills, + sandbox_id, + thread_id=thread_id, + user_id=user_id, + projection=projection, + ) + @abstractmethod def get(self, sandbox_id: str) -> Sandbox | None: """Get a sandbox environment by ID. diff --git a/backend/packages/harness/deerflow/skills/AGENTS.md b/backend/packages/harness/deerflow/skills/AGENTS.md index bbb39118c..5eb8f10d3 100644 --- a/backend/packages/harness/deerflow/skills/AGENTS.md +++ b/backend/packages/harness/deerflow/skills/AGENTS.md @@ -4,8 +4,8 @@ - **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools as a spec-compatible string or YAML list, argument-hint, required-secrets). Exact portable spellings such as `Bash`, `WebFetch`, `WebSearch`, `Glob`, `Grep`, `Read`, `Write`, and `Edit` map to `bash`, `web_fetch`, `web_search`, `glob`, `grep`, `read_file`, `write_file`, and `str_replace`; lowercase or otherwise unknown scalar names and YAML-list entries preserve their exact runtime spelling. Argument-scoped entries remain literal and inactive because the tool policy does not inspect arguments; the scalar tokenizer keeps spaces, quotes, and escaped parentheses inside patterns intact. - **Loading**: `load_skills()` recursively scans public, per-user custom, global integration, and legacy custom locations for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json plus per-user skill state for non-public categories; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. A custom skill directory may be a one-level symlink to an external directory for compatibility with operator-managed skill trees; activation still rejects a symlinked `SKILL.md` or deeper path escape. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json. - **External reload**: `POST /api/skills/reload` is an admin-only, process-local invalidation hook for trusted MinIO/NFS/CSI writes. `SkillStorage` instances do not cache a catalog — `load_skills()` scans on every call — so the route clears all `(app_config, user_id)` entries and the rendered prompt-section LRU, then waits up to the shared refresh timeout for the existing off-loop single-flight refresh. Each invalidation receives a generation-bound result handle; a successful scan atomically replaces the global enabled-skills cache, while a loader-level failure propagates to the HTTP waiter and preserves the last-known-good global cache. Per-user/config scans capture the refresh version and cannot repopulate shared caches if invalidation occurs while they are loading. A timed-out HTTP wait fails generically while the daemon refresh worker continues. Subsequent runs rescan after a successful reload; active runs keep their existing snapshot. Each Uvicorn worker/Kubernetes Pod must be targeted separately. Direct mount writes bypass install/edit validation, SkillScan, and history, so mounted roots are an operator-controlled trust boundary. -- **Tool policy**: Agent `allowed-tools` declarations apply dynamically only to slash-activated skills and skills captured in `ThreadState.skill_context` through configured `read_file` loads; passive enabled skills and custom-agent/subagent skill allowlists remain discoverable without clamping the baseline toolset. Subagents render only skill discovery metadata at startup and reuse the same adjacent `SkillActivationMiddleware` + `SkillToolPolicyMiddleware` pair as the lead; their configured `skills` field limits discovery and activation instead of eagerly loading bodies or unioning policies. Slash policy is dominant for its run, preventing subsequently read skills from widening explicit authority; autonomous captured skills use the existing union only when no slash source exists. `tool_search` and `describe_skill` stay available as framework discovery infrastructure, while every discovered or promoted business tool still requires active-policy permission for schema visibility and execution; `task`, `list_background_tasks`, and `cancel_background_task` likewise require explicit declarations. Each active model call intentionally reloads the full live registry so enable/disable changes, frontmatter edits, and custom/public name-shadow winners take effect without a stale TTL or unsafe direct-path cache; all tool calls produced by that model step reuse the resulting source-and-path-signed decision. Registry failures and all-invalid active sets fail closed, while stale individual paths are skipped when another valid skill remains. This is best-effort behavioral scoping, not a hard security boundary: alternate loading paths are not captured and bounded autonomous context may evict entries. -- **Sandbox projection**: `skills/projection.py` materializes enabled-only trees at `{base_dir}/skills_view/public` and `{base_dir}/users/{user_id}/skills_view/{custom,legacy,integrations}`. It copies files into the view (`_copy_into_view`) so a sandbox write cannot mutate the canonical skill inode; the operational trade-off is an O(total bytes) I/O and per-user storage multiplier across rebuilds, prioritized for write isolation over zero-copy hardlinks. Steady-state freshness checks combine source and view metadata tree digests, so in-sandbox view tampering is detected and repaired on the next acquire. Storage writes, archive installs, deletes, and toggles rebuild under a cross-process lock; Gateway boot ensures only the shared public view, while each user view is repaired lazily on first sandbox acquire. Managed integration packages are global, but their projected category is per-user because enabled state is isolated. Rebuilds stage a complete tree and reconcile it with per-file atomic replacement, so unrelated enabled skills remain continuously visible; disable/delete paths remove only the affected package before mutating to preserve fail-closed behavior. User projection rebuilds re-read global enable state from disk instead of the process singleton, so a toggle handled by another Gateway worker is reflected on the next acquire. Gateway public-skill toggles take the public projection lock before the shared `extensions_config_write_lock`, re-read an existing config from disk, persist the full model shape, and rebuild before responding; keep this as one worker-owned critical section so MCP writes cannot interleave and request cancellation cannot release either lock while the worker still runs. The shared public steady-state signature check runs without the global projection lock; stale/error paths take the lock and re-check before rebuilding or clearing. User-scope checks remain serialized per user. Category root inodes remain stable so live bind mounts observe content changes without sandbox recreation. Projection failures clear the affected view before raising. +- **Tool policy**: Agent `allowed-tools` declarations apply dynamically only to slash-activated skills and skills captured in `ThreadState.skill_context` through configured `read_file` loads; passive enabled skills and skill allowlists do not clamp the baseline tool set. A lead custom Agent's explicit `skills` list is additionally enforced at the sandbox filesystem layer (see Sandbox projection); subagent skill lists still scope discovery and activation only because concurrently delegated subagents share the lead thread sandbox. Subagents render only skill discovery metadata at startup and reuse the same adjacent `SkillActivationMiddleware` + `SkillToolPolicyMiddleware` pair as the lead; their configured `skills` field limits discovery and activation instead of eagerly loading bodies or unioning policies. Slash policy is dominant for its run, preventing subsequently read skills from widening explicit authority; autonomous captured skills use the existing union only when no slash source exists. `tool_search` and `describe_skill` stay available as framework discovery infrastructure, while every discovered or promoted business tool still requires active-policy permission for schema visibility and execution; `task`, `list_background_tasks`, and `cancel_background_task` likewise require explicit declarations. Each active model call intentionally reloads the full live registry so enable/disable changes, frontmatter edits, and custom/public name-shadow winners take effect without a stale TTL or unsafe direct-path cache; all tool calls produced by that model step reuse the resulting source-and-path-signed decision. Registry failures and all-invalid active sets fail closed, while stale individual paths are skipped when another valid skill remains. The dynamic `allowed-tools` policy remains best-effort behavioral scoping: alternate loading paths are not captured and bounded autonomous context may evict entries. +- **Sandbox projection**: `skills/projection.py` materializes enabled-only shared trees at `{base_dir}/skills_view/public` and `{base_dir}/users/{user_id}/skills_view/{custom,legacy,integrations}`. A lead Agent with an explicit `skills` allowlist (including `[]`) gets the intersection of enabled public/user-visible skills and that allowlist at `{base_dir}/users/{user_id}/threads/{thread_id}/skills_view/{public,custom,legacy,integrations}`. `skills=None` keeps the shared zero-copy mount until a thread has used an explicit policy; later unrestricted runs repopulate the same stable thread root with all enabled skills. Rebuilds sign source state, view state, and normalized policy in a manifest, revoke every old category before adding the new policy, stage copies in temporary directories, and atomically replace files. Category root inodes stay stable for live bind mounts; concurrent readers can briefly see fewer skills during a policy change, never a skill revoked by the new policy. Policy-scoped copies reject absolute symlinks and relative symlinks that resolve outside their own skill package, preventing a permitted package from linking back to an omitted source. It copies files into the view (`_copy_into_view`) so a sandbox write cannot mutate the canonical skill inode; the operational trade-off is an O(total bytes) I/O and per-user/thread storage multiplier across rebuilds, prioritized for write isolation over zero-copy hardlinks. Steady-state freshness checks combine source and view metadata tree digests, so in-sandbox view tampering is detected and repaired on the next acquire. Storage writes, archive installs, deletes, and toggles rebuild shared scopes under a cross-process lock; Gateway boot ensures only the shared public view, user views are repaired lazily on acquire, and Agent thread views are recomputed before sandbox reuse. Managed integration packages are global, but their projected category is per-user because enabled state is isolated. User projection rebuilds re-read global enable state from disk instead of the process singleton, so a toggle handled by another Gateway worker is reflected on the next acquire. Gateway public-skill toggles take the public projection lock before the shared `extensions_config_write_lock`, re-read an existing config from disk, persist the full model shape, and rebuild before responding; keep this as one worker-owned critical section so MCP writes cannot interleave and request cancellation cannot release either lock while the worker still runs. The shared public steady-state signature check runs without the global projection lock; stale/error paths take the lock and re-check before rebuilding or clearing. User and thread scope checks are serialized per scope. Projection failures clear the affected view before raising. - **Injection (legacy / default)**: Enabled skills are listed in the agent system prompt with full metadata and container paths (`` block). Controlled by `skills.deferred_discovery: false` (default). - **Deferred discovery** (`skills.deferred_discovery: true`): Skills are listed by name only in a compact `` block, keeping the system prompt prefix-cache friendly. The agent calls the `describe_skill` tool at runtime to fetch full metadata for skills it wants to use, then loads the SKILL.md via `read_file`. Two new modules support this path: - `skills/catalog.py` — `SkillCatalog` (immutable, searchable; query forms: `select:a,b`, `+prefix`, free-text regex); `select:` returns all requested skills without a result cap; other modes cap at `MAX_RESULTS=5`. diff --git a/backend/packages/harness/deerflow/skills/projection.py b/backend/packages/harness/deerflow/skills/projection.py index 1960c8c2a..f33a6558d 100644 --- a/backend/packages/harness/deerflow/skills/projection.py +++ b/backend/packages/harness/deerflow/skills/projection.py @@ -34,6 +34,7 @@ _locks_guard = threading.Lock() _process_locks: dict[Path, threading.RLock] = {} _MANIFEST_VERSION = 1 _MAX_REBUILD_ATTEMPTS = 2 +_THREAD_PROJECTION_POLICY_VERSION = 1 @dataclass(frozen=True) @@ -66,6 +67,29 @@ def get_skill_projection_paths(storage: SkillStorage) -> SkillProjectionPaths: ) +def get_thread_skill_projection_paths(storage: SkillStorage, thread_id: str) -> SkillProjectionPaths: + """Return stable category roots for one user's thread policy view.""" + from deerflow.config.paths import get_paths + + paths = getattr(storage, "_paths", None) or get_paths() + user_id = getattr(storage, "user_id", None) + if user_id is None: + raise ValueError("Thread skill projections require user-scoped skill storage") + root = paths.thread_skills_view_dir(thread_id, user_id=user_id) + return SkillProjectionPaths( + public=root / SkillCategory.PUBLIC.value, + custom=root / SkillCategory.CUSTOM.value, + legacy=root / SkillCategory.LEGACY.value, + integrations=root / SkillCategory.INTEGRATION.value, + ) + + +def thread_skill_projection_exists(storage: SkillStorage, thread_id: str) -> bool: + """Whether this thread has crossed into a stable policy-scoped view.""" + paths = get_thread_skill_projection_paths(storage, thread_id) + return paths.public.parent.exists() + + def _lock_for(path: Path) -> threading.RLock: resolved = path.resolve() with _locks_guard: @@ -103,7 +127,42 @@ def _copy_into_view(source: str, target: str, *, follow_symlinks: bool = True) - return target -def _stage_skill(source: Path, target: Path, nested_skill_roots: set[Path]) -> None: +def _validate_projected_skill_symlinks(source: Path) -> None: + """Reject links that would escape a policy-scoped skill package. + + Relative links to support files inside the same package remain useful and + resolve inside the copied projection. Absolute links would keep pointing at + the canonical host tree, and links outside the package could expose an + omitted skill, so both fail closed before any live view is changed. + """ + package_root = source.resolve(strict=True) + for current_root, dir_names, file_names in os.walk(source, followlinks=False): + current = Path(current_root) + for name in (*dir_names, *file_names): + path = current / name + if not path.is_symlink(): + continue + raw_target = Path(os.readlink(path)) + if raw_target.is_absolute(): + raise ValueError(f"Policy-scoped skill contains an absolute symlink: {path}") + try: + resolved_target = path.resolve(strict=True) + except (OSError, RuntimeError) as exc: + raise ValueError(f"Policy-scoped skill contains an invalid symlink: {path}") from exc + if not resolved_target.is_relative_to(package_root): + raise ValueError(f"Policy-scoped skill symlink escapes its package: {path}") + + +def _stage_skill( + source: Path, + target: Path, + nested_skill_roots: set[Path], + *, + enforce_symlink_boundary: bool = False, +) -> None: + if enforce_symlink_boundary: + _validate_projected_skill_symlinks(source) + def _exclude_nested_skills(current: str, names: list[str]) -> list[str]: relative_root = Path(current).relative_to(source) return [name for name in names if relative_root / name in nested_skill_roots] @@ -190,14 +249,25 @@ def _sync_staged_category(root: Path, staging: Path) -> None: (staging / relative_path).replace(target) -def _replace_category(root: Path, desired: dict[Path, Skill], skill_boundaries: set[Path]) -> None: +def _replace_category( + root: Path, + desired: dict[Path, Skill], + skill_boundaries: set[Path], + *, + enforce_symlink_boundary: bool = False, +) -> None: """Reconcile entries beneath a stable category root without blanking it.""" root.mkdir(parents=True, exist_ok=True) with tempfile.TemporaryDirectory(prefix=f".{root.name}.projection-", dir=root.parent) as staging_dir: staging = Path(staging_dir) for relative_path, skill in desired.items(): nested_roots = {boundary.relative_to(relative_path) for boundary in skill_boundaries if boundary != relative_path and boundary.is_relative_to(relative_path)} - _stage_skill(skill.skill_dir, staging / relative_path, nested_roots) + _stage_skill( + skill.skill_dir, + staging / relative_path, + nested_roots, + enforce_symlink_boundary=enforce_symlink_boundary, + ) _sync_staged_category(root, staging) @@ -343,7 +413,9 @@ def _view_signature(paths: SkillProjectionPaths, scope: str) -> str: digest = hashlib.sha256() if scope == "public": _update_tree_digest(digest, paths.public, "public_view") - elif scope == "user": + elif scope in {"user", "thread"}: + if scope == "thread": + _update_tree_digest(digest, paths.public, "public_view") _update_tree_digest(digest, paths.custom, "custom_view") _update_tree_digest(digest, paths.legacy, "legacy_view") _update_tree_digest(digest, paths.integrations, "integrations_view") @@ -352,6 +424,37 @@ def _view_signature(paths: SkillProjectionPaths, scope: str) -> str: return digest.hexdigest() +def _thread_source_signature( + storage: SkillStorage, + allowed_skills: set[str] | None, +) -> str: + """Sign every input that changes a thread's effective skill view.""" + digest = hashlib.sha256() + digest.update(f"policy-version:{_THREAD_PROJECTION_POLICY_VERSION}\0".encode()) + digest.update(f"public:{_source_signature(storage, 'public')}\0".encode()) + digest.update(f"user:{_source_signature(storage, 'user')}\0".encode()) + policy = None if allowed_skills is None else sorted(allowed_skills) + digest.update(json.dumps(policy, separators=(",", ":"), ensure_ascii=True).encode()) + return digest.hexdigest() + + +def _thread_projection_is_fresh( + storage: SkillStorage, + paths: SkillProjectionPaths, + allowed_skills: set[str] | None, +) -> bool: + scope_root = paths.public.parent + if not all(path.is_dir() for path in (paths.public, paths.custom, paths.legacy, paths.integrations)): + return False + manifest_before = _read_manifest(scope_root) + if manifest_before is None or manifest_before.get("version") != _MANIFEST_VERSION: + return False + source_sig = _thread_source_signature(storage, allowed_skills) + view_sig = _view_signature(paths, "thread") + manifest_after = _read_manifest(scope_root) + return manifest_before == manifest_after and manifest_before.get("source_signature") == source_sig and manifest_before.get("view_signature") == view_sig + + def _load_public_skills(storage: SkillStorage, *, enabled_only: bool) -> list[Skill]: from deerflow.config.extensions_config import ExtensionsConfig @@ -443,6 +546,109 @@ def _rebuild_user_locked(storage: SkillStorage, paths: SkillProjectionPaths) -> raise +def _rebuild_thread_locked( + storage: SkillStorage, + paths: SkillProjectionPaths, + allowed_skills: set[str] | None, +) -> None: + """Rebuild one thread's effective view, removing old authority first. + + Category roots stay inode-stable for live bind mounts. Clearing all four + before repopulating is deliberately fail-closed: concurrent readers may + briefly observe fewer allowed skills, but never a skill removed by the new + policy. + """ + scope_root = paths.public.parent + category_roots = (paths.public, paths.custom, paths.legacy, paths.integrations) + try: + for _attempt in range(_MAX_REBUILD_ATTEMPTS): + before = _thread_source_signature(storage, allowed_skills) + # User-scoped loading performs the same name-shadow resolution as + # prompt discovery. Use that effective catalog for every category; + # independently loading public skills would expose both copies when + # a custom or integration skill shadows a built-in skill by name. + all_effective_skills = storage.load_skills(enabled_only=False) + enabled_skills = [skill for skill in all_effective_skills if skill.enabled] + if allowed_skills is not None: + enabled_skills = [skill for skill in enabled_skills if skill.name in allowed_skills] + + # Revoke the previous policy before exposing any part of the new + # one. The enclosing cross-process lock serializes competing runs. + _clear_projection_scope(scope_root, *category_roots) + _replace_category( + paths.public, + _by_relative_path(enabled_skills, SkillCategory.PUBLIC), + _category_boundaries(all_effective_skills, SkillCategory.PUBLIC), + enforce_symlink_boundary=True, + ) + _replace_category( + paths.custom, + _by_relative_path(enabled_skills, SkillCategory.CUSTOM), + _category_boundaries(all_effective_skills, SkillCategory.CUSTOM), + enforce_symlink_boundary=True, + ) + _replace_category( + paths.legacy, + _by_relative_path(enabled_skills, SkillCategory.LEGACY), + _category_boundaries(all_effective_skills, SkillCategory.LEGACY), + enforce_symlink_boundary=True, + ) + _replace_category( + paths.integrations, + _by_relative_path(enabled_skills, SkillCategory.INTEGRATION), + _category_boundaries(all_effective_skills, SkillCategory.INTEGRATION), + enforce_symlink_boundary=True, + ) + after = _thread_source_signature(storage, allowed_skills) + if before == after: + _write_manifest(scope_root, after, _view_signature(paths, "thread")) + return + raise RuntimeError("Skills changed repeatedly while rebuilding the thread sandbox projection") + except Exception: + _clear_projection_scope(scope_root, *category_roots) + raise + + +def ensure_thread_skill_projection( + storage: SkillStorage, + thread_id: str, + allowed_skills: set[str] | None, +) -> SkillProjectionPaths | None: + """Ensure the filesystem view for one run's effective Agent skill policy. + + ``None`` keeps the existing shared projection for threads that have never + needed policy isolation. Once a thread has a scoped view, later unrestricted + runs rebuild that same stable mount with all enabled skills so switching + agents cannot leave the thread accidentally restricted. + """ + paths = get_thread_skill_projection_paths(storage, thread_id) + scope_root = paths.public.parent + if allowed_skills is None and not scope_root.exists(): + return None + + try: + fresh = _thread_projection_is_fresh(storage, paths, allowed_skills) + except Exception: + fresh = False + if fresh: + return paths + + with _projection_lock(scope_root): + try: + if not _thread_projection_is_fresh(storage, paths, allowed_skills): + _rebuild_thread_locked(storage, paths, allowed_skills) + except Exception: + _clear_projection_scope( + scope_root, + paths.public, + paths.custom, + paths.legacy, + paths.integrations, + ) + raise + return paths + + def rebuild_skill_projections( storage: SkillStorage, *, diff --git a/backend/tests/test_aio_sandbox_local_backend.py b/backend/tests/test_aio_sandbox_local_backend.py index 4394b776a..05b93c7c4 100644 --- a/backend/tests/test_aio_sandbox_local_backend.py +++ b/backend/tests/test_aio_sandbox_local_backend.py @@ -128,6 +128,61 @@ def test_start_container_logs_redacted_env_values(monkeypatch, caplog): assert "visible-value" not in log_output +def test_start_container_filters_nested_config_mounts_for_policy_scoped_skills( + monkeypatch, +): + backend = LocalContainerBackend( + image="sandbox:latest", + base_port=8080, + container_prefix="sandbox", + config_mounts=[ + SimpleNamespace( + host_path="/host/excluded-skill", + container_path="/mnt/skills/public/excluded-skill", + read_only=True, + ), + SimpleNamespace( + host_path="/host/unrelated", + container_path="/mnt/unrelated", + read_only=True, + ), + SimpleNamespace( + host_path="/host/sibling", + container_path="/mnt/skills-extra", + read_only=True, + ), + ], + environment={}, + ) + monkeypatch.setattr(backend, "_runtime", "docker") + captured_cmd: list[str] = [] + + def fake_run(cmd, **kwargs): + captured_cmd.extend(cmd) + return SimpleNamespace(stdout="container-id\n", stderr="", returncode=0) + + monkeypatch.setattr("subprocess.run", fake_run) + + backend._start_container( + "sandbox-test", + 18080, + extra_mounts=[ + ( + "/host/thread-view/public", + "/mnt/skills/public", + True, + ) + ], + config_mount_exclusion_root="/mnt/skills", + ) + + command = " ".join(captured_cmd) + assert "/host/excluded-skill" not in command + assert "/host/unrelated" in command + assert "/host/sibling" in command + assert "/host/thread-view/public" in command + + def _capture_start_container_command(monkeypatch, backend: LocalContainerBackend, runtime: str = "docker") -> list[str]: monkeypatch.setattr(backend, "_runtime", runtime) captured_cmd: list[str] = [] @@ -812,7 +867,11 @@ def test_discover_brackets_ipv6_sandbox_host_for_url(monkeypatch, sandbox_host): def test_create_brackets_ipv6_sandbox_host_for_url(monkeypatch, sandbox_host): backend = _backend_for_inspect_tests() monkeypatch.setenv("DEER_FLOW_SANDBOX_HOST", sandbox_host) - monkeypatch.setattr(backend, "_start_container", lambda name, port, mounts=None: "container-id") + monkeypatch.setattr( + backend, + "_start_container", + lambda name, port, mounts=None, **_kwargs: "container-id", + ) monkeypatch.setattr("deerflow.community.aio_sandbox.local_backend.get_free_port", lambda start_port=None: 18082) info = backend.create(thread_id="t", sandbox_id="sbx-ipv6") diff --git a/backend/tests/test_aio_sandbox_provider.py b/backend/tests/test_aio_sandbox_provider.py index b70238cde..cb537cd26 100644 --- a/backend/tests/test_aio_sandbox_provider.py +++ b/backend/tests/test_aio_sandbox_provider.py @@ -43,6 +43,23 @@ def test_load_config_preserves_thread_data_mounts_override(sandbox_overrides, ex provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider) assert provider._load_config()["thread_data_mounts"] is expected + assert provider._load_config()["skills_container_path"] == "/mnt/skills" + + +def test_load_config_snapshots_custom_skills_container_path(monkeypatch): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + sandbox_config = SandboxConfig( + use="deerflow.community.aio_sandbox:AioSandboxProvider", + ) + app_config = SimpleNamespace( + sandbox=sandbox_config, + stream_bridge=None, + skills=SimpleNamespace(container_path="/custom-skills"), + ) + monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config) + provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider) + + assert provider._load_config()["skills_container_path"] == "/custom-skills" @pytest.mark.parametrize( @@ -256,6 +273,7 @@ def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_pat monkeypatch.setattr(remote_backend, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False) provider = _make_provider(tmp_path) + provider._config["skills_container_path"] = config.skills.container_path mounts = provider._get_extra_mounts("thread-1", user_id="alice") container_paths = [container for _host, container, _read_only in mounts] @@ -279,7 +297,9 @@ def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_pat assert len(validated_paths) == len(set(validated_paths)) assert set(validated_paths) == { "/mnt/acp-workspace", + "/mnt/skills/public", "/mnt/skills/custom", + "/mnt/skills/legacy", "/mnt/skills/integrations", lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR, lark_cli.LARK_CLI_SANDBOX_LOCKS_DIR, @@ -288,6 +308,337 @@ def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_pat } +def test_thread_skill_projection_mounts_all_categories( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + paths = Paths(base_dir=tmp_path / "home") + projection_root = paths.thread_skills_view_dir( + "thread-policy", + user_id="alice", + ) + for category in ("public", "custom", "legacy", "integrations"): + (projection_root / category).mkdir(parents=True, exist_ok=True) + config = SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")) + monkeypatch.setattr(aio_mod, "get_app_config", lambda: config) + monkeypatch.setattr(aio_mod, "get_paths", lambda: paths) + + mounts = aio_mod.AioSandboxProvider._get_skills_mounts( + "thread-policy", + user_id="alice", + ) + + assert {container_path: host_path for host_path, container_path, _ in mounts} == {f"/mnt/skills/{category}": str(projection_root / category) for category in ("public", "custom", "legacy", "integrations")} + assert all(read_only for _host, _container, read_only in mounts) + + +def test_thread_skill_projection_uses_distinct_sandbox_identity( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + paths = Paths(base_dir=tmp_path / "home") + monkeypatch.setattr(aio_mod, "get_paths", lambda: paths) + monkeypatch.setattr( + aio_mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")), + ) + provider = _make_provider(tmp_path) + + shared_id = provider._sandbox_id_for_thread("thread-policy", "alice") + paths.thread_skills_view_dir( + "thread-policy", + user_id="alice", + ).mkdir(parents=True) + policy_id = provider._sandbox_id_for_thread("thread-policy", "alice") + + assert policy_id != shared_id + assert policy_id == provider._sandbox_id_for_thread("thread-policy", "alice") + assert policy_id != provider._deterministic_sandbox_id( + "thread-policy:agent-skills-v1", + "alice", + ) + + +def test_policy_scoped_sandbox_identity_changes_with_skills_container_root( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + paths = Paths(base_dir=tmp_path / "home") + paths.thread_skills_view_dir( + "thread-policy", + user_id="alice", + ).mkdir(parents=True) + monkeypatch.setattr(aio_mod, "get_paths", lambda: paths) + config = SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")) + monkeypatch.setattr(aio_mod, "get_app_config", lambda: config) + provider = _make_provider(tmp_path) + + default_root_id = provider._sandbox_id_for_thread( + "thread-policy", + "alice", + ) + config.skills.container_path = "/custom-skills" + provider._config["skills_container_path"] = config.skills.container_path + custom_root_id = provider._sandbox_id_for_thread( + "thread-policy", + "alice", + ) + + assert custom_root_id != default_root_id + + +def test_shared_sandbox_identity_changes_when_custom_skills_root_changes( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + paths = Paths(base_dir=tmp_path / "home") + monkeypatch.setattr(aio_mod, "get_paths", lambda: paths) + config = SimpleNamespace(skills=SimpleNamespace(container_path="/custom-skills-a")) + monkeypatch.setattr(aio_mod, "get_app_config", lambda: config) + provider = _make_provider(tmp_path) + provider._config["skills_container_path"] = config.skills.container_path + + first_root_id = provider._sandbox_id_for_thread( + "thread-shared", + "alice", + ) + config.skills.container_path = "/custom-skills-b" + provider._config["skills_container_path"] = config.skills.container_path + second_root_id = provider._sandbox_id_for_thread( + "thread-shared", + "alice", + ) + + assert second_root_id != first_root_id + + +def test_cached_sandbox_is_replaced_when_expected_identity_changes( + tmp_path, + monkeypatch, +): + provider = _make_provider(tmp_path) + provider._config["skills_container_path"] = "/custom-skills" + provider._thread_sandboxes = {("alice", "thread-shared"): "stale-root-id"} + provider._sandboxes = {"stale-root-id": object()} + provider._sandbox_infos = {} + monkeypatch.setattr( + provider, + "_sandbox_id_for_thread", + lambda *_args, **_kwargs: "new-root-id", + ) + destroy = MagicMock() + monkeypatch.setattr(provider, "destroy", destroy) + + assert ( + provider._reuse_in_process_sandbox( + "thread-shared", + user_id="alice", + ) + is None + ) + destroy.assert_called_once_with("stale-root-id") + + +def test_policy_scoped_create_excludes_local_config_mounts_below_skills_root( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + paths = Paths(base_dir=tmp_path / "home") + paths.thread_skills_view_dir( + "thread-policy", + user_id="alice", + ).mkdir(parents=True) + monkeypatch.setattr(aio_mod, "get_paths", lambda: paths) + monkeypatch.setattr( + aio_mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")), + ) + + provider = _make_provider(tmp_path) + provider._config = { + "replicas": 3, + "skills_container_path": "/mnt/skills", + } + provider._thread_locks = {} + provider._warm_pool = {} + provider._sandbox_infos = {} + provider._thread_sandboxes = {} + provider._last_activity = {} + provider._lock = aio_mod.threading.Lock() + captured: dict = {} + + backend = object.__new__(aio_mod.LocalContainerBackend) + + def _create(thread_id, sandbox_id, **kwargs): + captured.update(kwargs) + return aio_mod.SandboxInfo( + sandbox_id=sandbox_id, + sandbox_url="http://sandbox", + ) + + backend.create = _create + provider._backend = backend + monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda *_a, **_k: True) + monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: []) + monkeypatch.setattr( + aio_mod.AioSandboxProvider, + "_lark_integration_active", + staticmethod(lambda user_id=None: False), + ) + monkeypatch.setattr( + aio_mod.AioSandboxProvider, + "_lark_broker_active", + staticmethod(lambda user_id=None: False), + ) + monkeypatch.setattr( + provider, + "_register_created_sandbox", + lambda *a, **k: "sandbox-policy", + ) + + provider._create_sandbox( + "thread-policy", + "sandbox-policy", + user_id="alice", + ) + + assert captured["config_mount_exclusion_root"] == "/mnt/skills" + + +def test_remote_create_forwards_configured_skills_container_path( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + provider = _make_provider(tmp_path) + provider._config = { + "replicas": 3, + "skills_container_path": "/custom-skills", + } + provider._thread_locks = {} + provider._warm_pool = {} + provider._sandbox_infos = {} + provider._thread_sandboxes = {} + provider._last_activity = {} + provider._lock = aio_mod.threading.Lock() + captured: dict = {} + + backend = aio_mod.RemoteSandboxBackend("http://provisioner:8002") + + def _create(thread_id, sandbox_id, **kwargs): + captured.update(kwargs) + return aio_mod.SandboxInfo( + sandbox_id=sandbox_id, + sandbox_url="http://sandbox", + ) + + backend.create = _create + provider._backend = backend + monkeypatch.setattr( + aio_mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/custom-skills")), + ) + monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda *_a, **_k: True) + monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: []) + monkeypatch.setattr( + aio_mod.AioSandboxProvider, + "_lark_integration_active", + staticmethod(lambda user_id=None: False), + ) + monkeypatch.setattr( + aio_mod.AioSandboxProvider, + "_lark_broker_active", + staticmethod(lambda user_id=None: False), + ) + monkeypatch.setattr( + provider, + "_register_created_sandbox", + lambda *a, **k: "sandbox-custom-root", + ) + + provider._create_sandbox( + "thread-custom-root", + "sandbox-custom-root", + user_id="alice", + ) + + assert captured["skills_container_path"] == "/custom-skills" + + +@pytest.mark.anyio +async def test_remote_create_async_forwards_configured_skills_container_path( + tmp_path, + monkeypatch, +): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + provider = _make_provider(tmp_path) + provider._config = { + "replicas": 3, + "skills_container_path": "/custom-skills", + } + provider._thread_locks = {} + provider._warm_pool = {} + provider._sandbox_infos = {} + provider._thread_sandboxes = {} + provider._last_activity = {} + provider._lock = aio_mod.threading.Lock() + captured: dict = {} + + backend = aio_mod.RemoteSandboxBackend("http://provisioner:8002") + + def _create(thread_id, sandbox_id, **kwargs): + captured.update(kwargs) + return aio_mod.SandboxInfo( + sandbox_id=sandbox_id, + sandbox_url="http://sandbox", + ) + + backend.create = _create + provider._backend = backend + monkeypatch.setattr( + aio_mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/custom-skills")), + ) + + async def _ready(*_args, **_kwargs): + return True + + monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready_async", _ready) + monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: []) + monkeypatch.setattr( + aio_mod.AioSandboxProvider, + "_lark_integration_active", + staticmethod(lambda user_id=None: False), + ) + monkeypatch.setattr( + aio_mod.AioSandboxProvider, + "_lark_broker_active", + staticmethod(lambda user_id=None: False), + ) + monkeypatch.setattr( + provider, + "_register_created_sandbox", + lambda *a, **k: "sandbox-custom-root", + ) + + await provider._create_sandbox_async( + "thread-custom-root", + "sandbox-custom-root", + user_id="alice", + ) + + assert captured["skills_container_path"] == "/custom-skills" + + def test_join_host_path_preserves_windows_drive_letter_style(): base = r"C:\Users\demo\deer-flow\backend\.deer-flow" @@ -583,6 +934,7 @@ def test_remote_backend_create_forwards_effective_user_id(monkeypatch): "thread_id": "thread-42", "user_id": "user-7", "include_legacy_skills": True, + "skills_container_path": "/mnt/skills", "provision_lark_cli_runtime": False, "provision_lark_cli_broker": False, } diff --git a/backend/tests/test_e2b_sandbox_provider.py b/backend/tests/test_e2b_sandbox_provider.py index 7b205e57a..4bf84fad6 100644 --- a/backend/tests/test_e2b_sandbox_provider.py +++ b/backend/tests/test_e2b_sandbox_provider.py @@ -3,6 +3,7 @@ from __future__ import annotations import asyncio +import hashlib import importlib import json import os @@ -244,7 +245,15 @@ class FakeOwnershipStore: return None -def _make_provider(*, replicas: int = 3, idle_timeout: int = 1800, overflow_policy: str = "wait", acquire_timeout: int = 30, burst_limit: int = 0) -> Any: +def _make_provider( + *, + replicas: int = 3, + idle_timeout: int = 1800, + overflow_policy: str = "wait", + acquire_timeout: int = 30, + burst_limit: int = 0, + skills_container_path: str = "/mnt/skills", +) -> Any: """Build a ``E2BSandboxProvider`` instance bypassing ``__init__``.""" mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") provider = mod.E2BSandboxProvider.__new__(mod.E2BSandboxProvider) @@ -280,6 +289,7 @@ def _make_provider(*, replicas: int = 3, idle_timeout: int = 1800, overflow_poli "template": "code-interpreter-v1", "domain": None, "home_dir": "/home/user", + "skills_container_path": skills_container_path, "idle_timeout": idle_timeout, "replicas": replicas, "overflow_policy": overflow_policy, @@ -366,6 +376,471 @@ def test_apply_mounts_uploads_only_enabled_skill_projection(monkeypatch, tmp_pat assert "/mnt/skills/integrations/lark-cli/disabled-integration/SKILL.md" not in uploaded_paths +def test_policy_scoped_thread_skips_shared_projection_during_create( + monkeypatch, + tmp_path, +): + paths = Paths(base_dir=tmp_path) + paths.thread_skills_view_dir("thread-1", user_id="user-1").mkdir(parents=True) + monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: paths) + + provider = _make_provider() + + assert provider._skill_projection_mounts("user-1", "thread-1") == [] + + +def test_sync_agent_skills_rebuilds_managed_remote_tree_despite_matching_legacy_marker( + monkeypatch, + tmp_path, +): + from deerflow.skills.projection import SkillProjectionPaths + + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + root = tmp_path / "skills-view" + projection = SkillProjectionPaths( + public=root / "public", + custom=root / "custom", + legacy=root / "legacy", + integrations=root / "integrations", + ) + for category in ( + projection.public, + projection.custom, + projection.legacy, + projection.integrations, + ): + category.mkdir(parents=True, exist_ok=True) + _write_skill(projection.public, "allowed-skill") + manifest_path = root / ".projection-manifest.json" + manifest_path.write_text( + json.dumps( + { + "version": 1, + "source_signature": "source-a", + "view_signature": "view-a", + } + ), + encoding="utf-8", + ) + legacy_signature = hashlib.sha256(manifest_path.read_bytes()).hexdigest() + monkeypatch.setattr( + mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")), + ) + + files = FakeFilesAPI( + { + "/mnt/skills/public/excluded-skill/SKILL.md": b"excluded", + "/mnt/skills/.deerflow-projection-signature": legacy_signature.encode(), + "/mnt/skills/unmanaged.txt": b"keep", + } + ) + + def reset_remote_tree(command: str): + assert "sudo rm -rf -- /mnt/skills;" not in command + assert "sudo chown -R" not in command + assert "if [ -L /mnt/skills ]" in command + managed_paths = ( + "/mnt/skills/public", + "/mnt/skills/custom", + "/mnt/skills/legacy", + "/mnt/skills/integrations", + "/mnt/skills/.deerflow-projection-signature", + ) + for managed_path in managed_paths: + assert managed_path in command + for path in list(files.store): + if any(path == managed_path or path.startswith(f"{managed_path}/") for managed_path in managed_paths): + files.store.pop(path) + return SimpleNamespace( + stdout="SKILLS_RESET_OK\n", + stderr="", + exit_code=0, + ) + + chmod_ok = SimpleNamespace(stdout="", stderr="", exit_code=0) + commands = FakeCommandsAPI([reset_remote_tree, chmod_ok, reset_remote_tree, chmod_ok]) + client = FakeClient(sandbox_id="sandbox-1", commands=commands, files=files) + provider = _make_provider() + provider._sandboxes["sandbox-1"] = mod.E2BSandbox( + id="sandbox-1", + client=client, + home_dir="/home/user", + ) + + provider.sync_agent_skills( + "sandbox-1", + thread_id="thread-1", + user_id="user-1", + projection=projection, + ) + + assert "/mnt/skills/public/excluded-skill/SKILL.md" not in files.store + assert files.store["/mnt/skills/public/allowed-skill/SKILL.md"].startswith(b"---") + assert files.store["/mnt/skills/unmanaged.txt"] == b"keep" + assert "/mnt/skills/.deerflow-projection-signature" not in files.store + assert files.read_calls == [] + first_command_count = len(commands.calls) + first_write_count = len(files.write_calls) + + provider.sync_agent_skills( + "sandbox-1", + thread_id="thread-1", + user_id="user-1", + projection=projection, + ) + + assert len(commands.calls) == first_command_count * 2 + assert len(files.write_calls) == first_write_count * 2 + + +def test_sync_agent_skills_serializes_reset_and_upload_for_same_thread( + monkeypatch, + tmp_path, +): + from deerflow.skills.projection import SkillProjectionPaths + + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + projections: list[SkillProjectionPaths] = [] + for name in ("policy-a", "policy-b"): + root = tmp_path / name + projection = SkillProjectionPaths( + public=root / "public", + custom=root / "custom", + legacy=root / "legacy", + integrations=root / "integrations", + ) + for category in ( + projection.public, + projection.custom, + projection.legacy, + projection.integrations, + ): + category.mkdir(parents=True) + projections.append(projection) + + monkeypatch.setattr( + mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")), + ) + + first_upload_started = threading.Event() + allow_first_upload_to_finish = threading.Event() + second_sync_started = threading.Event() + second_reset_started = threading.Event() + reset_count = 0 + reset_count_lock = threading.Lock() + + def reset_remote_tree(_command: str): + nonlocal reset_count + with reset_count_lock: + reset_count += 1 + current_reset = reset_count + if current_reset == 2: + second_reset_started.set() + return SimpleNamespace(stdout="SKILLS_RESET_OK\n", stderr="", exit_code=0) + + commands = FakeCommandsAPI([reset_remote_tree, reset_remote_tree]) + client = FakeClient(sandbox_id="sandbox-1", commands=commands) + provider = _make_provider() + provider._sandboxes["sandbox-1"] = mod.E2BSandbox( + id="sandbox-1", + client=client, + home_dir="/home/user", + ) + + first_projection_root = projections[0].public.parent + + def blocking_upload(_client, source, _destination, _read_only, *, budget): + del budget + if source.parent == first_projection_root and not first_upload_started.is_set(): + first_upload_started.set() + assert allow_first_upload_to_finish.wait(timeout=5) + + monkeypatch.setattr(provider, "_upload_tree", blocking_upload) + errors: list[BaseException] = [] + + def sync( + projection: SkillProjectionPaths, + *, + started: threading.Event | None = None, + ) -> None: + try: + if started is not None: + started.set() + provider.sync_agent_skills( + "sandbox-1", + thread_id="thread-1", + user_id="user-1", + projection=projection, + ) + except BaseException as exc: # pragma: no cover - surfaced below + errors.append(exc) + + first = threading.Thread(target=sync, args=(projections[0],)) + second = threading.Thread( + target=sync, + args=(projections[1],), + kwargs={"started": second_sync_started}, + ) + first.start() + assert first_upload_started.wait(timeout=5) + second.start() + try: + assert second_sync_started.wait(timeout=5) + assert not second_reset_started.wait(timeout=0.2) + finally: + allow_first_upload_to_finish.set() + first.join(timeout=5) + second.join(timeout=5) + + assert not first.is_alive() + assert not second.is_alive() + assert errors == [] + assert second_reset_started.is_set() + + +@pytest.mark.parametrize( + "container_path", + [ + "skills", + "/", + "//mnt/skills", + "/mnt//skills", + "/mnt/skills/.", + "/mnt/skills/..", + "/mnt", + "/mnt/user-data", + "/mnt/acp-workspace", + "/home", + "/home/user", + "/bin", + "/boot", + "/dev", + "/etc", + "/etc/deerflow-skills", + "/lib", + "/lib32", + "/lib64", + "/libx32", + "/lost+found", + "/media", + "/opt", + "/proc", + "/root", + "/run", + "/sbin", + "/snap", + "/srv", + "/sys", + "/tmp", + "/usr", + "/usr/local/deerflow-skills", + "/var", + "/var/lib/deerflow-skills", + ], +) +def test_sync_agent_skills_rejects_unsafe_reset_roots_before_remote_access( + tmp_path, + container_path, +): + from deerflow.skills.projection import SkillProjectionPaths + + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + root = tmp_path / "skills-view" + projection = SkillProjectionPaths( + public=root / "public", + custom=root / "custom", + legacy=root / "legacy", + integrations=root / "integrations", + ) + client = FakeClient(sandbox_id="sandbox-1") + provider = _make_provider() + provider._config["skills_container_path"] = container_path + provider._sandboxes["sandbox-1"] = mod.E2BSandbox( + id="sandbox-1", + client=client, + home_dir="/home/user", + ) + + with pytest.raises(ValueError, match="safe E2B skills reset target"): + provider.sync_agent_skills( + "sandbox-1", + thread_id="thread-1", + user_id="user-1", + projection=projection, + ) + + assert client.files.read_calls == [] + assert client.commands.calls == [] + + +@pytest.mark.parametrize( + ("container_path", "expected"), + [ + ("/mnt/skills", "/mnt/skills"), + ("/mnt/skills/", "/mnt/skills"), + ("/home/user/skills", "/home/user/skills"), + ("/custom-skills", "/custom-skills"), + ("/custom/skills", "/custom/skills"), + ], +) +def test_validate_skills_reset_root_accepts_isolated_directories( + container_path, + expected, +): + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + + assert mod._validate_skills_reset_root(container_path, home_dir="/home/user") == expected + + +@pytest.mark.parametrize( + ("home_dir", "container_path"), + [ + ("/opt/e2b-home", "/opt/e2b-home/skills"), + ("/tmp/e2b-home", "/tmp/e2b-home/skills"), + ], +) +def test_validate_skills_reset_root_accepts_isolated_custom_home_subtree( + home_dir, + container_path, +): + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + + assert ( + mod._validate_skills_reset_root( + container_path, + home_dir=home_dir, + ) + == container_path + ) + + +def test_sync_agent_skills_rejects_symlinked_remote_root_before_deleting( + monkeypatch, + tmp_path, +): + from deerflow.skills.projection import SkillProjectionPaths + + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + root = tmp_path / "skills-view" + projection = SkillProjectionPaths( + public=root / "public", + custom=root / "custom", + legacy=root / "legacy", + integrations=root / "integrations", + ) + for category in ( + projection.public, + projection.custom, + projection.legacy, + projection.integrations, + ): + category.mkdir(parents=True, exist_ok=True) + manifest_path = root / ".projection-manifest.json" + manifest_path.write_text("{}", encoding="utf-8") + legacy_signature = hashlib.sha256(manifest_path.read_bytes()).hexdigest() + monkeypatch.setattr( + mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")), + ) + + def reject_symlinked_root(command: str): + assert "if [ -L /mnt/skills ]" in command + return SimpleNamespace( + stdout="", + stderr="Refusing symlinked skills root\n", + exit_code=2, + ) + + client = FakeClient( + sandbox_id="sandbox-1", + commands=FakeCommandsAPI([reject_symlinked_root]), + files=FakeFilesAPI( + { + "/mnt/skills/.deerflow-projection-signature": legacy_signature.encode(), + } + ), + ) + provider = _make_provider() + provider._sandboxes["sandbox-1"] = mod.E2BSandbox( + id="sandbox-1", + client=client, + home_dir="/home/user", + ) + monkeypatch.setattr( + provider, + "_upload_tree", + lambda *_args, **_kwargs: (_ for _ in ()).throw(AssertionError("upload must not start after a rejected reset")), + ) + + with pytest.raises(RuntimeError, match="Failed to reset E2B skill projection"): + provider.sync_agent_skills( + "sandbox-1", + thread_id="thread-1", + user_id="user-1", + projection=projection, + ) + + assert client.files.read_calls == [] + assert client.files.write_calls == [] + + +def test_sync_agent_skills_leaves_no_signature_after_upload_failure( + monkeypatch, + tmp_path, +): + from deerflow.skills.projection import SkillProjectionPaths + + mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") + root = tmp_path / "skills-view" + projection = SkillProjectionPaths( + public=root / "public", + custom=root / "custom", + legacy=root / "legacy", + integrations=root / "integrations", + ) + for category in ( + projection.public, + projection.custom, + projection.legacy, + projection.integrations, + ): + category.mkdir(parents=True, exist_ok=True) + (root / ".projection-manifest.json").write_text("{}", encoding="utf-8") + monkeypatch.setattr( + mod, + "get_app_config", + lambda: SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills")), + ) + commands = FakeCommandsAPI([SimpleNamespace(stdout="SKILLS_RESET_OK\n", stderr="", exit_code=0)]) + client = FakeClient(sandbox_id="sandbox-1", commands=commands) + provider = _make_provider() + provider._sandboxes["sandbox-1"] = mod.E2BSandbox( + id="sandbox-1", + client=client, + home_dir="/home/user", + ) + monkeypatch.setattr( + provider, + "_upload_tree", + lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("upload failed")), + ) + + with pytest.raises(RuntimeError, match="upload failed"): + provider.sync_agent_skills( + "sandbox-1", + thread_id="thread-1", + user_id="user-1", + projection=projection, + ) + + assert "/mnt/skills/.deerflow-projection-signature" not in client.files.store + + def test_upload_tree_streams_file_contents(tmp_path): source = tmp_path / "large.bin" source.write_bytes(b"mount content") @@ -831,6 +1306,7 @@ def test_load_config_clamps_invalid_mount_upload_deadline(monkeypatch, caplog, r mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") class FakeConfig: + skills = SimpleNamespace(container_path="/mnt/skills") sandbox = SimpleNamespace( model_extra={"mount_upload_deadline_seconds": raw}, api_key="test-key", @@ -1046,9 +1522,9 @@ def _make_sandbox(client: FakeClient, *, sandbox_id: str | None = None) -> Any: ) -def test_thread_key_returns_user_thread_tuple(): +def test_thread_key_includes_the_provider_skills_root(): p = _make_provider() - assert p._thread_key("t1", "u1") == ("u1", "t1") + assert p._thread_key("t1", "u1") == ("u1", "t1", "/mnt/skills") def test_sandbox_id_falls_back_when_client_id_is_none(): @@ -1060,6 +1536,7 @@ def test_sandbox_id_falls_back_when_client_id_is_none(): def test_stable_seed_is_deterministic_and_user_scoped(): p = _make_provider() + custom_root = _make_provider(skills_container_path="/custom-skills") s_a = p._stable_seed("t1", "u1") s_b = p._stable_seed("t1", "u1") s_other_user = p._stable_seed("t1", "u2") @@ -1067,6 +1544,7 @@ def test_stable_seed_is_deterministic_and_user_scoped(): assert s_a == s_b assert s_a != s_other_user assert s_a != s_other_thread + assert s_a != custom_root._stable_seed("t1", "u1") def test_is_sandbox_gone_error_matches_known_signatures(): @@ -1285,7 +1763,7 @@ def test_refresh_owned_leases_reclaims_lapsed_lease(): client = FakeClient(sandbox_id="sb-lapsed") sandbox = _make_sandbox(client, sandbox_id="sb-lapsed") p._sandboxes["sb-lapsed"] = sandbox - p._thread_sandboxes[("u1", "t1")] = "sb-lapsed" + p._thread_sandboxes[p._thread_key("t1", "u1")] = "sb-lapsed" p._owned_sandbox_ids.add("sb-lapsed") p._refresh_owned_leases() @@ -1300,7 +1778,8 @@ def test_refresh_owned_leases_forgets_peer_owned_sandbox(): client = FakeClient(sandbox_id="sb-lost") sandbox = _make_sandbox(client, sandbox_id="sb-lost") p._sandboxes["sb-lost"] = sandbox - p._thread_sandboxes[("u1", "t1")] = "sb-lost" + key = p._thread_key("t1", "u1") + p._thread_sandboxes[key] = "sb-lost" p._owned_sandbox_ids.add("sb-lost") p._ownership = FakeOwnershipStore( {"sb-lost": ("owner-peer", "own")}, @@ -1310,7 +1789,7 @@ def test_refresh_owned_leases_forgets_peer_owned_sandbox(): p._refresh_owned_leases() assert p.get("sb-lost") is None - assert ("u1", "t1") not in p._thread_sandboxes + assert key not in p._thread_sandboxes assert "sb-lost" not in p._owned_sandbox_ids assert client.closed is True @@ -1320,7 +1799,7 @@ def test_reuse_in_process_sandbox_returns_cached_id_on_healthy_reuse(): client = FakeClient() sb = _make_sandbox(client, sandbox_id="sb-1") p._sandboxes["sb-1"] = sb - p._thread_sandboxes[("u1", "t1")] = "sb-1" + p._thread_sandboxes[p._thread_key("t1", "u1")] = "sb-1" sid = p._reuse_in_process_sandbox("t1", user_id="u1") assert sid == "sb-1" @@ -1333,12 +1812,13 @@ def test_reuse_in_process_sandbox_evicts_dead_sandbox(): sb = _make_sandbox(client, sandbox_id="sb-dead") sb._dead = True p._sandboxes["sb-dead"] = sb - p._thread_sandboxes[("u1", "t1")] = "sb-dead" + key = p._thread_key("t1", "u1") + p._thread_sandboxes[key] = "sb-dead" sid = p._reuse_in_process_sandbox("t1", user_id="u1") assert sid is None assert "sb-dead" not in p._sandboxes - assert ("u1", "t1") not in p._thread_sandboxes + assert key not in p._thread_sandboxes def test_reuse_in_process_sandbox_evicts_when_ping_fails(): @@ -1346,7 +1826,7 @@ def test_reuse_in_process_sandbox_evicts_when_ping_fails(): client = FakeClient(commands=FakeCommandsAPI([FakeCommandsAPI.GONE])) sb = _make_sandbox(client, sandbox_id="sb-stale") p._sandboxes["sb-stale"] = sb - p._thread_sandboxes[("u1", "t1")] = "sb-stale" + p._thread_sandboxes[p._thread_key("t1", "u1")] = "sb-stale" sid = p._reuse_in_process_sandbox("t1", user_id="u1") assert sid is None @@ -1356,10 +1836,11 @@ def test_reuse_in_process_sandbox_evicts_when_ping_fails(): def test_reuse_in_process_sandbox_cleans_dangling_mapping(): p = _make_provider() - p._thread_sandboxes[("u1", "t1")] = "ghost" + key = p._thread_key("t1", "u1") + p._thread_sandboxes[key] = "ghost" sid = p._reuse_in_process_sandbox("t1", user_id="u1") assert sid is None - assert ("u1", "t1") not in p._thread_sandboxes + assert key not in p._thread_sandboxes def test_reuse_in_process_sandbox_returns_none_when_no_mapping(): @@ -1376,7 +1857,7 @@ def test_reclaim_warm_pool_sandbox_happy_path(monkeypatch): sid = p._reclaim_warm_pool_sandbox("t1", user_id="u1") assert sid == "sb-warm" assert "sb-warm" in p._sandboxes - assert p._thread_sandboxes[("u1", "t1")] == "sb-warm" + assert p._thread_sandboxes[p._thread_key("t1", "u1")] == "sb-warm" assert "sb-warm" not in p._warm_pool assert [c[0] for c in fake_cls.connect_calls] == ["sb-warm"] @@ -1465,17 +1946,42 @@ class _FakePaginator: return page -def _info(sandbox_id: str, user_id: str, thread_id: str): +def _info( + sandbox_id: str, + user_id: str, + thread_id: str, + *, + skills_container_path: str = "/mnt/skills", +): return SimpleNamespace( sandbox_id=sandbox_id, metadata={ "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": user_id, "deer_flow_thread": thread_id, + "deer_flow_skills_root": skills_container_path, }, ) +def test_discover_remote_sandbox_rejects_a_different_skills_root(monkeypatch): + p = _make_provider(skills_container_path="/custom-skills") + fake_cls = _install_fake_sdk(monkeypatch, p) + fake_cls.list_return = [_info("sb-old-root", "u1", "t1")] + + assert p._discover_remote_sandbox("t1", user_id="u1") is None + assert fake_cls.connect_calls == [] + + +def test_create_metadata_records_the_snapshotted_skills_root(monkeypatch): + provider = _make_provider(skills_container_path="/custom-skills") + fake_cls = _install_fake_sdk(monkeypatch, provider) + + provider.acquire("t1", user_id="u1") + + assert fake_cls.create_calls[0]["metadata"]["deer_flow_skills_root"] == "/custom-skills" + + def test_discover_remote_sandbox_walks_paginator(monkeypatch): p = _make_provider() fake_cls = _install_fake_sdk(monkeypatch, p) @@ -1488,7 +1994,7 @@ def test_discover_remote_sandbox_walks_paginator(monkeypatch): sid = p._discover_remote_sandbox("t1", user_id="u1") assert sid == "sb-match" - assert p._thread_sandboxes[("u1", "t1")] == "sb-match" + assert p._thread_sandboxes[p._thread_key("t1", "u1")] == "sb-match" def test_discover_remote_sandbox_accepts_legacy_list(monkeypatch): @@ -1511,7 +2017,7 @@ def test_discover_remote_sandbox_skips_dead_candidate(monkeypatch): fake_cls.connect_factory = lambda _sid, **_kw: client assert p._discover_remote_sandbox("t1", user_id="u1") is None - assert ("u1", "t1") not in p._thread_sandboxes + assert p._thread_key("t1", "u1") not in p._thread_sandboxes assert client.closed is True @@ -1531,7 +2037,7 @@ def test_discover_remote_sandbox_tries_later_candidate_when_first_is_dead(monkey assert p._discover_remote_sandbox("t1", user_id="u1") == "sb-b-live" assert dead.closed is True - assert p._thread_sandboxes[("u1", "t1")] == "sb-b-live" + assert p._thread_sandboxes[p._thread_key("t1", "u1")] == "sb-b-live" assert "sb-b-live" in p._owned_sandbox_ids @@ -1677,7 +2183,7 @@ def test_reconcile_adopts_canonical_after_restart_loses_local_state(monkeypatch) stats = p._reconcile_remote_sandboxes(now=100.0) assert stats.adopted == 1 - assert p._thread_sandboxes[("u1", "t1")] == "sb-existing" + assert p._thread_sandboxes[p._thread_key("t1", "u1")] == "sb-existing" assert "sb-existing" in p._owned_sandbox_ids @@ -1748,6 +2254,33 @@ def test_reconcile_kills_metadata_orphan_only_after_ttl(monkeypatch): assert client.killed is True +def test_reconcile_never_adopts_an_old_skills_root_and_reaps_it_after_grace( + monkeypatch, +): + provider = _make_provider(skills_container_path="/custom-skills") + fake_cls = _install_fake_sdk(monkeypatch, provider) + fake_cls.list_return = [ + _info( + "sb-old-root", + "u1", + "t1", + skills_container_path="/mnt/skills", + ) + ] + client = FakeClient(sandbox_id="sb-old-root") + fake_cls.connect_factory = lambda _sid, **_kw: client + provider._config["reconciliation_grace_seconds"] = 5.0 + + first = provider._reconcile_remote_sandboxes(now=100.0) + second = provider._reconcile_remote_sandboxes(now=106.0) + + assert first.adopted == 0 + assert first.deferred == 1 + assert provider._thread_key("t1", "u1") not in provider._thread_sandboxes + assert second.killed == 1 + assert client.killed is True + + def test_discover_remote_sandbox_discards_candidate_when_bootstrap_fails(monkeypatch): p = _make_provider() fake_cls = _install_fake_sdk(monkeypatch, p) @@ -1766,7 +2299,7 @@ def test_discover_remote_sandbox_discards_candidate_when_bootstrap_fails(monkeyp assert p._discover_remote_sandbox("t1", user_id="u1") is None assert client.killed is True assert client.closed is True - assert ("u1", "t1") not in p._thread_sandboxes + assert p._thread_key("t1", "u1") not in p._thread_sandboxes def test_discovery_claims_ownership_before_bootstrap_cleanup(monkeypatch): @@ -1900,7 +2433,14 @@ def test_e2b_config_accepts_documented_reconciliation_fields(monkeypatch, caplog reconciliation_max_seconds=15, ) provider = mod.E2BSandboxProvider.__new__(mod.E2BSandboxProvider) - monkeypatch.setattr(mod, "get_app_config", lambda: SimpleNamespace(sandbox=config)) + monkeypatch.setattr( + mod, + "get_app_config", + lambda: SimpleNamespace( + sandbox=config, + skills=SimpleNamespace(container_path="/mnt/skills"), + ), + ) with caplog.at_level("WARNING"): provider._load_config() @@ -1916,7 +2456,14 @@ def test_e2b_config_warns_about_unknown_fields(monkeypatch, caplog): overflo_policy="reject", ) provider = mod.E2BSandboxProvider.__new__(mod.E2BSandboxProvider) - monkeypatch.setattr(mod, "get_app_config", lambda: SimpleNamespace(sandbox=config)) + monkeypatch.setattr( + mod, + "get_app_config", + lambda: SimpleNamespace( + sandbox=config, + skills=SimpleNamespace(container_path="/mnt/skills"), + ), + ) with caplog.at_level("WARNING"): provider._load_config() @@ -2108,13 +2655,14 @@ def test_release_dead_sandbox_skips_warm_pool(monkeypatch): sb = _make_sandbox(client, sandbox_id="sb-dead") sb._dead = True p._sandboxes["sb-dead"] = sb - p._thread_sandboxes[("u1", "t1")] = "sb-dead" + key = p._thread_key("t1", "u1") + p._thread_sandboxes[key] = "sb-dead" p.release("sb-dead") assert "sb-dead" not in p._warm_pool, "dead sandbox must not be parked" assert "sb-dead" not in p._sandboxes - assert ("u1", "t1") not in p._thread_sandboxes + assert key not in p._thread_sandboxes assert client.killed is True, "release of dead sandbox must kill the remote VM" @@ -2125,7 +2673,7 @@ def test_release_healthy_sandbox_parks_in_warm_pool(monkeypatch, tmp_path): client = FakeClient(commands=cmds) sb = _make_sandbox(client, sandbox_id="sb-warm-1") p._sandboxes["sb-warm-1"] = sb - p._thread_sandboxes[("u1", "t1")] = "sb-warm-1" + p._thread_sandboxes[p._thread_key("t1", "u1")] = "sb-warm-1" p.release("sb-warm-1") @@ -2142,7 +2690,7 @@ def test_acquire_waits_for_same_thread_release_transition(monkeypatch): client = FakeClient(sandbox_id="sb-release-race") sandbox = _make_sandbox(client) provider._sandboxes[sandbox.id] = sandbox - provider._thread_sandboxes[("user-1", "thread-1")] = sandbox.id + provider._thread_sandboxes[provider._thread_key("thread-1", "user-1")] = sandbox.id sync_started = threading.Event() allow_sync_to_finish = threading.Event() @@ -2190,7 +2738,7 @@ def test_release_skips_warm_pool_when_sync_reveals_dead_vm(monkeypatch, tmp_path client = FakeClient(commands=FakeCommandsAPI([FakeCommandsAPI.GONE])) sb = _make_sandbox(client, sandbox_id="sb-died-during-sync") p._sandboxes["sb-died-during-sync"] = sb - p._thread_sandboxes[("u1", "t1")] = "sb-died-during-sync" + p._thread_sandboxes[p._thread_key("t1", "u1")] = "sb-died-during-sync" p.release("sb-died-during-sync") @@ -2987,10 +3535,19 @@ def test_discovery_uses_sdk_query_and_tracks_without_reserving(monkeypatch) -> N "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": "user-a", "deer_flow_thread": "thread-a", + "deer_flow_skills_root": "/mnt/skills", "deer_flow_capacity_ledger": store.key, }, ) - expected_query = {key: entry.metadata[key] for key in ("deer_flow_provider", "deer_flow_user", "deer_flow_thread")} + expected_query = { + key: entry.metadata[key] + for key in ( + "deer_flow_provider", + "deer_flow_user", + "deer_flow_thread", + "deer_flow_skills_root", + ) + } sdk.list_return = SimpleNamespace( has_next=False, next_items=lambda: [entry] if sdk.list_calls[-1]["query"].metadata == expected_query else [], @@ -4134,6 +4691,7 @@ def test_discovery_reports_busy_capacity_without_killing_remote_vm(monkeypatch): "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": "u2", "deer_flow_thread": "t2", + "deer_flow_skills_root": "/mnt/skills", }, ) ] @@ -4159,6 +4717,7 @@ def test_discovery_reports_shutdown_without_killing_remote_vm(monkeypatch, caplo "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": "u1", "deer_flow_thread": "t1", + "deer_flow_skills_root": "/mnt/skills", }, ) ] @@ -4196,6 +4755,7 @@ def test_discovery_bootstrap_kill_failure_retains_reserved_slot(monkeypatch): "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": "u1", "deer_flow_thread": "t1", + "deer_flow_skills_root": "/mnt/skills", }, ) ] @@ -4230,6 +4790,7 @@ def test_shutdown_does_not_retry_kill_for_unowned_discovery_vm(monkeypatch): "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": "u1", "deer_flow_thread": "t1", + "deer_flow_skills_root": "/mnt/skills", }, ) ] @@ -4274,6 +4835,7 @@ def test_shutdown_during_discovery_does_not_kill_unowned_vm(monkeypatch): "deer_flow_provider": "e2b_sandbox_provider", "deer_flow_user": "u1", "deer_flow_thread": "t1", + "deer_flow_skills_root": "/mnt/skills", }, ) ] @@ -4321,5 +4883,10 @@ def test_shutdown_during_discovery_does_not_kill_unowned_vm(monkeypatch): def test_stable_seed_matches_shared_identity(): from deerflow.sandbox.identity import derive_sandbox_scope_token - mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider") - assert mod.E2BSandboxProvider._stable_seed("t-1", "u-1") == derive_sandbox_scope_token(user_id="u-1", thread_id="t-1") + provider = _make_provider(skills_container_path="/custom-skills") + base_scope = derive_sandbox_scope_token(user_id="u-1", thread_id="t-1") + expected = hashlib.sha256( + f"{base_scope}\0/custom-skills".encode(), + ).hexdigest()[:16] + + assert provider._stable_seed("t-1", "u-1") == expected diff --git a/backend/tests/test_lead_agent_model_resolution.py b/backend/tests/test_lead_agent_model_resolution.py index db35e7284..e80a36c50 100644 --- a/backend/tests/test_lead_agent_model_resolution.py +++ b/backend/tests/test_lead_agent_model_resolution.py @@ -205,6 +205,8 @@ def test_make_lead_agent_scopes_bootstrap_middlewares_to_custom_agent(monkeypatc assert len(middleware_calls) == 1 assert middleware_calls[0]["agent_name"] == "game" + assert middleware_calls[0]["available_skills"] == {"bootstrap"} + assert middleware_calls[0]["owns_agent_skill_projection"] is False assert len(prompt_calls) == 1 diff --git a/backend/tests/test_local_sandbox_provider_mounts.py b/backend/tests/test_local_sandbox_provider_mounts.py index c41d37bf8..557b31d0d 100644 --- a/backend/tests/test_local_sandbox_provider_mounts.py +++ b/backend/tests/test_local_sandbox_provider_mounts.py @@ -565,6 +565,21 @@ class TestMultipleMounts: class TestLocalSandboxProviderMounts: + def test_skill_isolation_capability_fails_closed_when_host_bash_is_enabled(self): + provider = LocalSandboxProvider.__new__(LocalSandboxProvider) + + with patch( + "deerflow.sandbox.local.local_sandbox_provider.is_host_bash_allowed", + return_value=False, + ): + assert provider.supports_agent_skill_isolation is True + + with patch( + "deerflow.sandbox.local.local_sandbox_provider.is_host_bash_allowed", + return_value=True, + ): + assert provider.supports_agent_skill_isolation is False + def test_thread_mappings_mount_per_user_integration_projections(self, tmp_path): from deerflow.config.paths import Paths diff --git a/backend/tests/test_provisioner_mount_contract.py b/backend/tests/test_provisioner_mount_contract.py index d074a3d89..24dee8116 100644 --- a/backend/tests/test_provisioner_mount_contract.py +++ b/backend/tests/test_provisioner_mount_contract.py @@ -1,4 +1,4 @@ -"""Keep Gateway/provisioner extra-mount allowlists in lockstep.""" +"""Keep Gateway/provisioner fixed and dynamic mount contracts in lockstep.""" from __future__ import annotations @@ -22,8 +22,15 @@ def test_gateway_and_provisioner_extra_mount_contracts_match() -> None: gateway_paths = _literal_assignment(gateway_path, "_PROVISIONER_EXTRA_MOUNT_PATHS") provisioner_paths = _literal_assignment(provisioner_path, "ALLOWED_EXTRA_MOUNT_PATHS") + gateway_categories = _literal_assignment(gateway_path, "_MANAGED_SKILL_CATEGORY_NAMES") + provisioner_categories = _literal_assignment(provisioner_path, "MANAGED_SKILL_CATEGORY_NAMES") + gateway_reserved = _literal_assignment(gateway_path, "_RESERVED_SANDBOX_MOUNT_PATHS") + provisioner_reserved = _literal_assignment(provisioner_path, "RESERVED_SANDBOX_MOUNT_PATHS") assert gateway_paths == provisioner_paths + assert gateway_categories == provisioner_categories + assert gateway_reserved == provisioner_reserved + assert _literal_assignment(provisioner_path, "DEFAULT_SKILLS_CONTAINER_PATH") == "/mnt/skills" assert "/mnt/integrations/lark-cli/runtime" in gateway_paths assert "/mnt/integrations/lark-cli/config/locks" in gateway_paths assert _literal_assignment(provisioner_path, "MAX_EXTRA_MOUNTS") == 10 diff --git a/backend/tests/test_provisioner_pvc_volumes.py b/backend/tests/test_provisioner_pvc_volumes.py index 43b8d7663..ae7c1b7ca 100644 --- a/backend/tests/test_provisioner_pvc_volumes.py +++ b/backend/tests/test_provisioner_pvc_volumes.py @@ -2,6 +2,21 @@ import pytest + +def _thread_skill_mounts( + provisioner_module, + skills_container_path="/mnt/skills", +): + return [ + provisioner_module.ExtraMount( + host_path=f"/state/users/alice/threads/thread-1/skills_view/{category}", + container_path=f"{skills_container_path}/{category}", + read_only=True, + ) + for category in ("public", "custom", "legacy", "integrations") + ] + + # ── _build_volumes ───────────────────────────────────────────────────── @@ -176,6 +191,72 @@ class TestBuildVolumes: assert exc_info.value.status_code == 400 + def test_thread_skill_mounts_replace_hostpath_skill_volumes( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + + volumes = provisioner_module._build_volumes( + "thread-1", + user_id="alice", + extra_mounts=_thread_skill_mounts(provisioner_module), + ) + + names = {volume.name for volume in volumes} + assert not {"skills-public", "skills-custom", "skills-legacy"} & names + assert names == {"user-data", "extra-0", "extra-1", "extra-2", "extra-3"} + + def test_custom_root_thread_mounts_replace_hostpath_skill_volumes( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + skills_root = "/custom-skills" + + volumes = provisioner_module._build_volumes( + "thread-1", + user_id="alice", + extra_mounts=_thread_skill_mounts( + provisioner_module, + skills_root, + ), + skills_container_path=skills_root, + ) + + names = {volume.name for volume in volumes} + assert not {"skills-public", "skills-custom", "skills-legacy"} & names + assert names == { + "user-data", + "extra-0", + "extra-1", + "extra-2", + "extra-3", + } + + def test_thread_skill_mounts_replace_skills_pvc_with_userdata_pvc_categories( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "skills-pvc" + provisioner_module.USERDATA_PVC_NAME = "userdata-pvc" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + + volumes = provisioner_module._build_volumes( + "thread-1", + user_id="alice", + extra_mounts=_thread_skill_mounts(provisioner_module), + ) + + assert all(volume.name != "skills" for volume in volumes) + extra_volumes = [volume for volume in volumes if volume.name.startswith("extra-")] + assert len(extra_volumes) == 4 + assert all(volume.persistent_volume_claim.claim_name == "userdata-pvc" for volume in extra_volumes) + # ── _build_volume_mounts ─────────────────────────────────────────────── @@ -348,6 +429,119 @@ class TestBuildVolumeMounts: assert exc_info.value.status_code == 400 + def test_thread_skill_category_mounts_are_unique_in_hostpath_mode( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + + mounts = provisioner_module._build_volume_mounts( + "thread-1", + user_id="alice", + extra_mounts=_thread_skill_mounts(provisioner_module), + ) + + mount_paths = [mount.mount_path for mount in mounts] + assert len(mount_paths) == len(set(mount_paths)) + assert set(mount_paths) == { + "/mnt/user-data", + "/mnt/skills/public", + "/mnt/skills/custom", + "/mnt/skills/legacy", + "/mnt/skills/integrations", + } + + def test_thread_skill_category_mounts_use_userdata_pvc_subpaths( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "skills-pvc" + provisioner_module.USERDATA_PVC_NAME = "userdata-pvc" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + + mounts = provisioner_module._build_volume_mounts( + "thread-1", + user_id="alice", + extra_mounts=_thread_skill_mounts(provisioner_module), + ) + + skill_mounts = [mount for mount in mounts if mount.mount_path.startswith("/mnt/skills/")] + assert len(skill_mounts) == 4 + assert all(mount.name != "skills" for mount in mounts) + assert {mount.sub_path for mount in skill_mounts} == {f"deer-flow/users/alice/threads/thread-1/skills_view/{category}" for category in ("public", "custom", "legacy", "integrations")} + + @pytest.mark.parametrize("use_userdata_pvc", [False, True]) + def test_custom_root_thread_skill_mounts_replace_every_default_path( + self, + provisioner_module, + use_userdata_pvc, + ): + provisioner_module.SKILLS_PVC_NAME = "skills-pvc" if use_userdata_pvc else "" + provisioner_module.USERDATA_PVC_NAME = "userdata-pvc" if use_userdata_pvc else "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + skills_root = "/custom-skills" + + mounts = provisioner_module._build_volume_mounts( + "thread-1", + user_id="alice", + extra_mounts=_thread_skill_mounts( + provisioner_module, + skills_root, + ), + skills_container_path=skills_root, + ) + + mount_paths = {mount.mount_path for mount in mounts} + assert not any(path.startswith("/mnt/skills") for path in mount_paths) + assert {f"{skills_root}/{category}" for category in ("public", "custom", "legacy", "integrations")} <= mount_paths + + def test_custom_root_is_used_for_unrestricted_skills_mounts( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + + mounts = provisioner_module._build_volume_mounts( + "thread-1", + skills_container_path="/custom-skills", + ) + + assert {mount.mount_path for mount in mounts if mount.name.startswith("skills-")} == { + "/custom-skills/public", + "/custom-skills/custom", + "/custom-skills/legacy", + } + + @pytest.mark.parametrize( + "skills_root", + [ + "/", + "relative-skills", + "//custom-skills", + "/custom//skills", + "/custom/../skills", + "/mnt", + "/mnt/user-data/skills", + "/mnt/acp-workspace/skills", + "/mnt/integrations/lark-cli/skills", + ], + ) + def test_rejects_unsafe_skills_container_roots( + self, + provisioner_module, + skills_root, + ): + with pytest.raises(provisioner_module.HTTPException) as exc_info: + provisioner_module._build_volume_mounts( + "thread-1", + skills_container_path=skills_root, + ) + + assert exc_info.value.status_code == 400 + # ── _build_pod integration ───────────────────────────────────────────── @@ -669,6 +863,41 @@ class TestLarkCliBrokerSidecar: env = {e.name: e.value for e in (sandbox.env or [])} assert env.get("DEERFLOW_LARK_BROKER_URL") == provisioner_module.LARK_BROKER_URL + def test_custom_skills_root_is_compatible_with_broker_credentials( + self, + provisioner_module, + ): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + provisioner_module.LARK_CLI_BROKER_IMAGE = "deer-flow/lark-cli-broker:v1.0.65" + skills_root = "/custom-skills" + + pod = provisioner_module._build_pod( + "sandbox-1", + "thread-1", + user_id="alice", + extra_mounts=[ + *_thread_skill_mounts( + provisioner_module, + skills_root, + ), + *self._credential_mounts(provisioner_module), + ], + skills_container_path=skills_root, + provision_lark_cli_broker=True, + ) + + sandbox_mount_paths = {mount.mount_path for mount in pod.spec.containers[0].volume_mounts} + assert not any(path.startswith("/mnt/skills") for path in sandbox_mount_paths) + assert {f"{skills_root}/{category}" for category in ("public", "custom", "legacy", "integrations")} <= sandbox_mount_paths + sidecar = next(container for container in pod.spec.containers if container.name == "lark-cli-broker") + assert {mount.mount_path for mount in sidecar.volume_mounts} == { + provisioner_module.LARK_BROKER_SIDECAR_CONFIG_PATH, + provisioner_module.LARK_BROKER_SIDECAR_LOCKS_PATH, + provisioner_module.LARK_BROKER_SIDECAR_DATA_PATH, + } + def test_broker_supersedes_init_container(self, provisioner_module): """Both images set + both flags on → broker wins (shim init, sidecar).""" provisioner_module.SKILLS_PVC_NAME = "" diff --git a/backend/tests/test_provisioner_request_threading.py b/backend/tests/test_provisioner_request_threading.py index c0e74ea6f..33c03bfeb 100644 --- a/backend/tests/test_provisioner_request_threading.py +++ b/backend/tests/test_provisioner_request_threading.py @@ -42,6 +42,14 @@ def test_provisioner_accepts_canonical_thread_ids(provisioner_module, thread_id: assert request.thread_id == thread_id +def test_provisioner_request_defaults_skills_container_path(provisioner_module) -> None: + request = provisioner_module.CreateSandboxRequest( + sandbox_id="sandbox-validation", + ) + + assert request.skills_container_path == "/mnt/skills" + + class _RecordingCoreV1: def __init__( self, @@ -232,6 +240,41 @@ def test_create_sandbox_route_builds_expected_skills_mount_layout( assert mount_names == expected_mount_names +def test_create_sandbox_route_threads_custom_skills_root_into_pod( + monkeypatch: pytest.MonkeyPatch, + provisioner_module, +) -> None: + fake_core_v1 = _RecordingCoreV1( + event_loop_thread_id=-1, + ready_after_service_reads={"sandbox-custom-skills": 1}, + ) + monkeypatch.setattr(provisioner_module, "core_v1", fake_core_v1) + categories = ("public", "custom", "legacy", "integrations") + + response = provisioner_module.create_sandbox( + provisioner_module.CreateSandboxRequest( + sandbox_id="sandbox-custom-skills", + thread_id="thread-1", + user_id="alice", + skills_container_path="/custom-skills", + extra_mounts=[ + provisioner_module.ExtraMount( + host_path=(f"/.deer-flow/users/alice/threads/thread-1/skills_view/{category}"), + container_path=f"/custom-skills/{category}", + read_only=True, + ) + for category in categories + ], + ) + ) + + assert response.status == "Running" + pod = fake_core_v1.created_pod_specs["sandbox-custom-skills"] + mount_paths = {mount.mount_path for mount in pod.spec.containers[0].volume_mounts} + assert not any(path.startswith("/mnt/skills") for path in mount_paths) + assert {f"/custom-skills/{category}" for category in categories} <= mount_paths + + def test_create_sandbox_retries_transient_service_read_errors(monkeypatch: pytest.MonkeyPatch, provisioner_module) -> None: fake_core_v1 = _RecordingCoreV1( event_loop_thread_id=-1, diff --git a/backend/tests/test_reload_boundary.py b/backend/tests/test_reload_boundary.py index 639ec39f5..fca782d37 100644 --- a/backend/tests/test_reload_boundary.py +++ b/backend/tests/test_reload_boundary.py @@ -20,6 +20,7 @@ from deerflow.config.reload_boundary import ( is_startup_only_field, iter_startup_only_field_paths, ) +from deerflow.config.skills_config import SkillsConfig def test_registry_has_a_reason_for_every_field(): @@ -115,6 +116,15 @@ def test_appconfig_schema_marks_registered_fields_with_prefix(): assert description.startswith(STARTUP_ONLY_PREFIX), f"AppConfig.{field_path} should have Field(description=) starting with {STARTUP_ONLY_PREFIX!r}, got {description!r}" +def test_skills_container_path_is_registered_as_startup_only(): + """AIO and E2B snapshot the mount root when their provider starts.""" + field_path = "skills.container_path" + assert field_path in STARTUP_ONLY_FIELDS + description = SkillsConfig.model_fields["container_path"].description or "" + assert description.startswith(STARTUP_ONLY_PREFIX) + assert STARTUP_ONLY_FIELDS[field_path] in description + + def test_no_appconfig_field_uses_prefix_without_registration(): """Reverse drift check: if a future schema edit adds the ``startup-only:`` prefix to a new field, the registry must list it. diff --git a/backend/tests/test_remote_sandbox_backend.py b/backend/tests/test_remote_sandbox_backend.py index 1c3bb0eb5..df7e4154a 100644 --- a/backend/tests/test_remote_sandbox_backend.py +++ b/backend/tests/test_remote_sandbox_backend.py @@ -34,6 +34,31 @@ class _StubResponse: return self._payload +@pytest.mark.parametrize( + "container_path", + [ + "/", + "relative-skills", + "//custom-skills", + "/custom//skills", + "/custom/../skills", + "/mnt", + "/mnt/user-data/skills", + "/mnt/acp-workspace/skills", + "/mnt/integrations/lark-cli/skills", + ], +) +def test_skills_container_path_rejects_unsafe_or_noncanonical_roots( + container_path, +): + with pytest.raises(ValueError): + remote_backend_mod._normalize_skills_container_path(container_path) + + +def test_skills_container_path_accepts_isolated_custom_root(): + assert remote_backend_mod._normalize_skills_container_path("/custom-skills") == "/custom-skills" + + def test_list_running_delegates_to_provisioner_list(monkeypatch): backend = RemoteSandboxBackend("http://provisioner:8002") sandbox_info = SandboxInfo(sandbox_id="test-id", sandbox_url="http://localhost:8080") @@ -165,11 +190,21 @@ def test_create_delegates_to_provisioner_create(monkeypatch, expected_user_id): backend = RemoteSandboxBackend("http://provisioner:8002") expected = SandboxInfo(sandbox_id="abc123", sandbox_url="http://k3s:31001") - def mock_create(thread_id: str, sandbox_id: str, extra_mounts=None, *, user_id=None, provision_lark_cli_runtime=False, provision_lark_cli_broker=False): + def mock_create( + thread_id: str, + sandbox_id: str, + extra_mounts=None, + *, + user_id=None, + skills_container_path="/mnt/skills", + provision_lark_cli_runtime=False, + provision_lark_cli_broker=False, + ): assert thread_id == "thread-1" assert sandbox_id == "abc123" assert extra_mounts == [("/host", "/container", False)] assert user_id == expected_user_id + assert skills_container_path == "/mnt/skills" assert provision_lark_cli_runtime is True assert provision_lark_cli_broker is False return expected @@ -197,6 +232,7 @@ def test_provisioner_create_returns_sandbox_info(monkeypatch): "thread_id": "thread-1", "user_id": "test-user-autouse", "include_legacy_skills": True, + "skills_container_path": "/mnt/skills", "provision_lark_cli_runtime": False, "provision_lark_cli_broker": False, } @@ -247,6 +283,48 @@ def test_provisioner_create_forwards_supported_extra_mounts(monkeypatch): ) +def test_provisioner_create_forwards_custom_skills_root_and_all_policy_mounts( + monkeypatch, +): + backend = RemoteSandboxBackend("http://provisioner:8002") + monkeypatch.setattr( + remote_backend_mod, + "user_should_see_legacy_skills", + lambda user_id: False, + ) + captured: dict = {} + + def mock_post(url: str, json: dict, timeout: int, headers=None): + captured.update(json) + return _StubResponse( + payload={ + "sandbox_id": "abc123", + "sandbox_url": "http://k3s:31001", + } + ) + + monkeypatch.setattr(requests, "post", mock_post) + categories = ("public", "custom", "legacy", "integrations") + + backend._provisioner_create( + "thread-1", + "abc123", + extra_mounts=[ + ( + f"/state/users/alice/threads/thread-1/skills_view/{category}", + f"/custom-skills/{category}", + True, + ) + for category in categories + ], + user_id="alice", + skills_container_path="/custom-skills", + ) + + assert captured["skills_container_path"] == "/custom-skills" + assert {mount["container_path"] for mount in captured["extra_mounts"]} == {f"/custom-skills/{category}" for category in categories} + + def test_provisioner_create_strips_runtime_mount_when_init_container_enabled(monkeypatch): backend = RemoteSandboxBackend("http://provisioner:8002") monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda user_id: False) @@ -318,6 +396,7 @@ def test_provisioner_create_accepts_anonymous_thread_id(monkeypatch): "thread_id": None, "user_id": "test-user-autouse", "include_legacy_skills": False, + "skills_container_path": "/mnt/skills", "provision_lark_cli_runtime": False, "provision_lark_cli_broker": False, } diff --git a/backend/tests/test_sandbox_middleware.py b/backend/tests/test_sandbox_middleware.py index ac6818b8a..6e009c4e0 100644 --- a/backend/tests/test_sandbox_middleware.py +++ b/backend/tests/test_sandbox_middleware.py @@ -12,6 +12,7 @@ from langgraph.runtime import Runtime from langgraph.types import Command, Overwrite from deerflow.agents.thread_state import ThreadState +from deerflow.sandbox.exceptions import SandboxAuthorizationError, SandboxRuntimeError from deerflow.sandbox.middleware import SandboxMiddleware, SandboxMiddlewareState from deerflow.sandbox.sandbox import Sandbox from deerflow.sandbox.sandbox_provider import SandboxProvider, reset_sandbox_provider, set_sandbox_provider @@ -36,6 +37,24 @@ class _SyncProvider(SandboxProvider): return None +class _AgentSkillSyncProvider(_SyncProvider): + supports_agent_skill_isolation = True + + def __init__(self) -> None: + super().__init__() + self.skill_syncs: list[tuple[str, str, str, object]] = [] + + def sync_agent_skills( + self, + sandbox_id: str, + *, + thread_id: str, + user_id: str, + projection, + ) -> None: + self.skill_syncs.append((sandbox_id, thread_id, user_id, projection)) + + class _SandboxStub(Sandbox): def execute_command( self, @@ -151,6 +170,129 @@ async def test_abefore_agent_uses_async_provider_acquire() -> None: assert provider.user_ids == ["owner-2"] +def test_explicit_skill_policy_eagerly_acquires_and_syncs_existing_thread( + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider = _AgentSkillSyncProvider() + projection = object() + middleware = SandboxMiddleware(lazy_init=True, available_skills=set()) + monkeypatch.setattr( + middleware, + "_prepare_agent_skill_projection", + lambda *_args, **_kwargs: projection, + ) + set_sandbox_provider(provider) + try: + result = middleware.before_agent( + {"sandbox": {"sandbox_id": "shared-view-sandbox"}}, + Runtime(context={"thread_id": "thread-policy", "user_id": "owner-policy"}), + ) + finally: + reset_sandbox_provider() + + assert result is not None + assert isinstance(result["sandbox"], Overwrite) + assert result["sandbox"].value == {"sandbox_id": "sync-sandbox"} + assert provider.thread_ids == ["thread-policy"] + assert provider.user_ids == ["owner-policy"] + assert provider.skill_syncs == [("sync-sandbox", "thread-policy", "owner-policy", projection)] + + +def test_explicit_skill_policy_fails_closed_for_unsupported_provider( + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider = _SyncProvider() + middleware = SandboxMiddleware(lazy_init=True, available_skills={"allowed"}) + monkeypatch.setattr( + middleware, + "_prepare_agent_skill_projection", + lambda *_args, **_kwargs: object(), + ) + set_sandbox_provider(provider) + try: + with pytest.raises( + SandboxRuntimeError, + match="cannot enforce per-Agent skill filesystem isolation", + ): + middleware.before_agent( + {}, + Runtime(context={"thread_id": "thread-policy", "user_id": "owner-policy"}), + ) + finally: + reset_sandbox_provider() + + assert provider.thread_ids == [] + + +def test_non_owner_skill_policy_preserves_lazy_init_without_projection_or_acquire( + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider = _SyncProvider() + middleware = SandboxMiddleware( + lazy_init=True, + available_skills={"bootstrap"}, + owns_agent_skill_projection=False, + ) + prepare_calls: list[tuple[str, str]] = [] + original_prepare = middleware._prepare_agent_skill_projection + + def _prepare(thread_id: str, *, user_id: str): + prepare_calls.append((thread_id, user_id)) + return original_prepare(thread_id, user_id=user_id) + + monkeypatch.setattr(middleware, "_prepare_agent_skill_projection", _prepare) + set_sandbox_provider(provider) + try: + result = middleware.before_agent( + {}, + Runtime( + context={ + "thread_id": "thread-bootstrap", + "user_id": "owner-bootstrap", + } + ), + ) + finally: + reset_sandbox_provider() + + assert result is None + assert prepare_calls == [("thread-bootstrap", "owner-bootstrap")] + assert provider.thread_ids == [] + + +def test_explicit_skill_policy_does_not_reuse_checkpointed_sandbox_after_auth_denial( + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider = _AgentSkillSyncProvider() + middleware = SandboxMiddleware(lazy_init=True, available_skills=set()) + monkeypatch.setattr( + middleware, + "_prepare_agent_skill_projection", + lambda *_args, **_kwargs: object(), + ) + monkeypatch.setattr( + "deerflow.sandbox.middleware.authorize_sandbox_execution", + lambda **_kwargs: (_ for _ in ()).throw(SandboxAuthorizationError("denied")), + ) + set_sandbox_provider(provider) + try: + with pytest.raises(SandboxAuthorizationError, match="denied"): + middleware.before_agent( + {"sandbox": {"sandbox_id": "shared-view-sandbox"}}, + Runtime( + context={ + "thread_id": "thread-policy", + "user_id": "owner-policy", + } + ), + ) + finally: + reset_sandbox_provider() + + assert provider.thread_ids == [] + assert provider.skill_syncs == [] + + @pytest.mark.anyio @pytest.mark.parametrize( ("middleware", "state", "runtime"), diff --git a/backend/tests/test_skill_projection.py b/backend/tests/test_skill_projection.py index c4da764b1..5e7d068e5 100644 --- a/backend/tests/test_skill_projection.py +++ b/backend/tests/test_skill_projection.py @@ -14,7 +14,14 @@ import pytest from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig from deerflow.config.paths import Paths -from deerflow.skills.projection import ensure_public_skill_projection, ensure_skill_projections, rebuild_skill_projections, skill_projection_mutation +from deerflow.sandbox.middleware import SandboxMiddleware +from deerflow.skills.projection import ( + ensure_public_skill_projection, + ensure_skill_projections, + ensure_thread_skill_projection, + rebuild_skill_projections, + skill_projection_mutation, +) from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage @@ -227,6 +234,242 @@ def test_managed_integration_projection_is_filtered_per_user(projection_env) -> assert (bob_projection.integrations / "lark-cli" / "lark-doc" / "SKILL.md").is_file() +def test_thread_projection_enforces_agent_allowlist_across_skill_categories(projection_env) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "public-allowed") + _write_skill(env.skills_root / "public", "public-denied") + env.storage.write_custom_skill("custom-allowed", "SKILL.md", _skill_content("custom-allowed")) + env.storage.write_custom_skill("custom-denied", "SKILL.md", _skill_content("custom-denied")) + integration_root = env.paths.integration_skills_dir() / "provider" + _write_skill(integration_root, "integration-allowed") + _write_skill(integration_root, "integration-denied") + + projected = ensure_thread_skill_projection( + env.storage, + "thread-1", + {"public-allowed", "custom-allowed", "integration-allowed"}, + ) + + assert projected is not None + assert (projected.public / "public-allowed" / "SKILL.md").is_file() + assert not (projected.public / "public-denied").exists() + assert (projected.custom / "custom-allowed" / "SKILL.md").is_file() + assert not (projected.custom / "custom-denied").exists() + assert (projected.integrations / "provider" / "integration-allowed" / "SKILL.md").is_file() + assert not (projected.integrations / "provider" / "integration-denied").exists() + + +def test_thread_projection_rejects_symlink_escape_to_omitted_skill( + projection_env, +) -> None: + env = projection_env + allowed_file = _write_skill(env.skills_root / "public", "public-allowed") + _write_skill(env.skills_root / "public", "public-denied", "DENIED_MARKER") + leak = allowed_file.parent / "leak.md" + try: + leak.symlink_to(Path("..") / "public-denied" / "SKILL.md") + except OSError as exc: + if getattr(exc, "winerror", None) == 1314: + pytest.skip("Windows symlink creation requires SeCreateSymbolicLinkPrivilege") + raise + + with pytest.raises(ValueError, match="symlink escapes its package"): + ensure_thread_skill_projection( + env.storage, + "thread-symlink-escape", + {"public-allowed"}, + ) + + root = env.paths.thread_skills_view_dir( + "thread-symlink-escape", + user_id="alice", + ) + assert all(list((root / category).iterdir()) == [] for category in ("public", "custom", "legacy", "integrations")) + assert not (root / ".projection-manifest.json").exists() + + +def test_thread_projection_exposes_only_effective_same_name_winner( + projection_env, +) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "shadowed-skill", "public copy") + env.storage.write_custom_skill( + "shadowed-skill", + "SKILL.md", + _skill_content("shadowed-skill", "custom copy"), + ) + + projected = ensure_thread_skill_projection( + env.storage, + "thread-shadowing", + {"shadowed-skill"}, + ) + + assert projected is not None + assert not (projected.public / "shadowed-skill").exists() + custom_file = projected.custom / "shadowed-skill" / "SKILL.md" + assert custom_file.is_file() + assert "custom copy" in custom_file.read_text(encoding="utf-8") + + +def test_thread_projection_enforces_allowlist_for_legacy_skills( + projection_env, +) -> None: + env = projection_env + _write_skill(env.skills_root / "custom", "legacy-allowed") + _write_skill(env.skills_root / "custom", "legacy-denied") + + projected = ensure_thread_skill_projection( + env.storage, + "thread-legacy", + {"legacy-allowed"}, + ) + + assert projected is not None + assert (projected.legacy / "legacy-allowed" / "SKILL.md").is_file() + assert not (projected.legacy / "legacy-denied").exists() + + +def test_thread_projection_intersects_agent_allowlist_with_deployment_state( + projection_env, +) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "deployment-disabled") + env.extensions.skills["deployment-disabled"] = SkillStateConfig(enabled=False) + + projected = ensure_thread_skill_projection( + env.storage, + "thread-disabled", + {"deployment-disabled"}, + ) + + assert projected is not None + assert not (projected.public / "deployment-disabled").exists() + + +def test_empty_agent_allowlist_projects_no_business_skills(projection_env) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "public-skill") + env.storage.write_custom_skill("custom-skill", "SKILL.md", _skill_content("custom-skill")) + + projected = ensure_thread_skill_projection(env.storage, "thread-empty", set()) + + assert projected is not None + for category in (projected.public, projected.custom, projected.legacy, projected.integrations): + assert list(category.iterdir()) == [] + + +def test_unrestricted_thread_without_policy_keeps_shared_projection( + projection_env, +) -> None: + env = projection_env + + projected = ensure_thread_skill_projection( + env.storage, + "thread-unrestricted", + None, + ) + + assert projected is None + assert not env.paths.thread_skills_view_dir( + "thread-unrestricted", + user_id="alice", + ).exists() + + +def test_unrestricted_run_reuses_existing_thread_mount_with_full_enabled_view(projection_env) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "alpha") + _write_skill(env.skills_root / "public", "beta") + + projected = ensure_thread_skill_projection(env.storage, "thread-switch", {"alpha"}) + assert projected is not None + category_inodes = {category: getattr(projected, category).stat().st_ino for category in ("public", "custom", "legacy", "integrations")} + assert (projected.public / "alpha" / "SKILL.md").is_file() + assert not (projected.public / "beta").exists() + + unrestricted = ensure_thread_skill_projection(env.storage, "thread-switch", None) + + assert unrestricted == projected + assert (projected.public / "alpha" / "SKILL.md").is_file() + assert (projected.public / "beta" / "SKILL.md").is_file() + assert {category: getattr(projected, category).stat().st_ino for category in ("public", "custom", "legacy", "integrations")} == category_inodes + + +def test_subagent_non_owner_preserves_restricted_lead_projection( + projection_env, + monkeypatch, +) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "allowed") + _write_skill(env.skills_root / "public", "omitted") + provider = SimpleNamespace(supports_agent_skill_isolation=True) + monkeypatch.setattr( + "deerflow.sandbox.middleware.get_sandbox_provider", + lambda: provider, + ) + monkeypatch.setattr("deerflow.config.get_app_config", lambda: env.config) + monkeypatch.setattr( + "deerflow.skills.storage.get_or_new_user_skill_storage", + lambda *_args, **_kwargs: env.storage, + ) + + lead = SandboxMiddleware(available_skills={"allowed"}) + projected = lead._prepare_agent_skill_projection( + "thread-delegation", + user_id="alice", + ) + assert projected is not None + assert (projected.public / "allowed" / "SKILL.md").is_file() + assert not (projected.public / "omitted").exists() + + subagent = SandboxMiddleware( + available_skills=None, + owns_agent_skill_projection=False, + ) + assert ( + subagent._prepare_agent_skill_projection( + "thread-delegation", + user_id="alice", + ) + is None + ) + assert (projected.public / "allowed" / "SKILL.md").is_file() + assert not (projected.public / "omitted").exists() + + +def test_thread_projection_revokes_removed_skill_before_repopulation(projection_env, monkeypatch) -> None: + env = projection_env + _write_skill(env.skills_root / "public", "alpha") + _write_skill(env.skills_root / "public", "beta") + projected = ensure_thread_skill_projection(env.storage, "thread-update", {"alpha", "beta"}) + assert projected is not None + + from deerflow.skills import projection as projection_module + + real_stage_skill = projection_module._stage_skill + staging_started = Event() + release_staging = Event() + + def _delayed_stage_skill(*args, **kwargs): + if not staging_started.is_set(): + staging_started.set() + assert release_staging.wait(timeout=5) + return real_stage_skill(*args, **kwargs) + + monkeypatch.setattr(projection_module, "_stage_skill", _delayed_stage_skill) + with ThreadPoolExecutor(max_workers=1) as executor: + rebuild = executor.submit(ensure_thread_skill_projection, env.storage, "thread-update", {"alpha"}) + assert staging_started.wait(timeout=5) + beta_was_revoked = not (projected.public / "beta").exists() + release_staging.set() + rebuild.result(timeout=5) + + assert beta_was_revoked + assert (projected.public / "alpha" / "SKILL.md").is_file() + assert not (projected.public / "beta").exists() + + def test_disabling_custom_skill_hides_only_target_while_rebuilding(projection_env, monkeypatch) -> None: env = projection_env env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha")) @@ -505,7 +748,10 @@ def test_rebuild_failure_clears_old_projection(projection_env, monkeypatch) -> N replacement = source.with_suffix(".replacement") replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8") replacement.replace(source) - monkeypatch.setattr("deerflow.skills.projection._stage_skill", lambda *_args: (_ for _ in ()).throw(OSError("disk full"))) + monkeypatch.setattr( + "deerflow.skills.projection._stage_skill", + lambda *_args, **_kwargs: (_ for _ in ()).throw(OSError("disk full")), + ) with pytest.raises(OSError, match="disk full"): ensure_skill_projections(env.storage) diff --git a/backend/tests/test_three_way_skills_mount_e2e.py b/backend/tests/test_three_way_skills_mount_e2e.py index a873c5f6c..091fc4001 100644 --- a/backend/tests/test_three_way_skills_mount_e2e.py +++ b/backend/tests/test_three_way_skills_mount_e2e.py @@ -21,7 +21,10 @@ from deerflow.config.paths import Paths from deerflow.sandbox.local.local_sandbox import PathMapping from deerflow.sandbox.local.local_sandbox_provider import LocalSandboxProvider from deerflow.sandbox.tools import read_file_tool -from deerflow.skills.projection import rebuild_skill_projections +from deerflow.skills.projection import ( + ensure_thread_skill_projection, + rebuild_skill_projections, +) from deerflow.skills.storage import reset_user_skill_storage from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage from deerflow.skills.types import SKILL_MD_FILE, Skill, SkillCategory @@ -375,6 +378,146 @@ class TestThreeWayMountEndToEnd: assert "SECRET_PROCEDURE" not in structured_disabled_again assert "disabled" in structured_disabled_again.lower() + def test_local_agent_allowlist_is_enforced_by_every_filesystem_path( + self, + tmp_path, + ): + skills_root = tmp_path / "skills" + _write_skill(skills_root / "public", "allowed-skill", "ALLOWED_MARKER") + _write_skill(skills_root / "public", "excluded-skill", "EXCLUDED_MARKER") + (skills_root / "custom").mkdir(parents=True) + paths = Paths(base_dir=tmp_path) + cfg = _build_config(skills_root) + extensions = ExtensionsConfig() + + with ( + patch("deerflow.config.get_app_config", return_value=cfg), + patch("deerflow.config.paths.get_paths", return_value=paths), + patch( + "deerflow.config.extensions_config.ExtensionsConfig.from_file", + return_value=extensions, + ), + patch( + "deerflow.config.extensions_config.get_extensions_config", + return_value=extensions, + ), + ): + storage = UserScopedSkillStorage( + "user-1", + host_path=str(skills_root), + app_config=cfg, + ) + projection = ensure_thread_skill_projection( + storage, + "thread-policy", + {"allowed-skill"}, + ) + assert projection is not None + provider = LocalSandboxProvider() + sandbox_id = provider.acquire("thread-policy", user_id="user-1") + sandbox = provider.get(sandbox_id) + assert sandbox is not None + + mappings = {mapping.container_path: mapping for mapping in sandbox.path_mappings} + assert set(path for path in mappings if path.startswith("/mnt/skills")) == {"/mnt/skills"} + assert Path(mappings["/mnt/skills"].local_path) == projection.public.parent + + listing = "\n".join(sandbox.list_dir("/mnt/skills", max_depth=4)) + assert "allowed-skill" in listing + assert "excluded-skill" not in listing + assert "EXCLUDED_MARKER" not in sandbox.execute_command("find /mnt/skills -name SKILL.md -print -exec cat {} \\;") + + excluded_path = "/mnt/skills/public/excluded-skill/SKILL.md" + with pytest.raises(FileNotFoundError): + sandbox.read_file(excluded_path) + globbed, _ = sandbox.glob("/mnt/skills", "**/SKILL.md") + assert any("allowed-skill" in path for path in globbed) + assert all("excluded-skill" not in path for path in globbed) + grepped, _ = sandbox.grep( + "/mnt/skills", + "EXCLUDED_MARKER", + literal=True, + ) + assert grepped == [] + + absolute_read = sandbox.execute_command(f"cat {excluded_path}") + relative_read = sandbox.execute_command("cd /mnt/skills/public && cat excluded-skill/SKILL.md") + python_read = sandbox.execute_command("python3 -c \"from pathlib import Path; print(Path('/mnt/skills/public/excluded-skill/SKILL.md').read_text())\"") + symlink_read = sandbox.execute_command("ln -s /mnt/skills/public/excluded-skill /mnt/skills/public/excluded-link && cat /mnt/skills/public/excluded-link/SKILL.md") + for result in ( + absolute_read, + relative_read, + python_read, + symlink_read, + ): + assert "EXCLUDED_MARKER" not in result + assert "Exit Code:" in result + + def test_local_empty_agent_allowlist_exposes_no_business_skill( + self, + tmp_path, + ): + skills_root = tmp_path / "skills" + _write_skill(skills_root / "public", "public-skill", "PUBLIC_MARKER") + (skills_root / "custom").mkdir(parents=True) + paths = Paths(base_dir=tmp_path) + cfg = _build_config(skills_root) + extensions = ExtensionsConfig() + + with ( + patch("deerflow.config.get_app_config", return_value=cfg), + patch("deerflow.config.paths.get_paths", return_value=paths), + patch( + "deerflow.config.extensions_config.ExtensionsConfig.from_file", + return_value=extensions, + ), + patch( + "deerflow.config.extensions_config.get_extensions_config", + return_value=extensions, + ), + ): + storage = UserScopedSkillStorage( + "user-1", + host_path=str(skills_root), + app_config=cfg, + ) + storage.write_custom_skill( + "custom-skill", + "SKILL.md", + "---\nname: custom-skill\ndescription: CUSTOM_MARKER\n---\n", + ) + projection = ensure_thread_skill_projection( + storage, + "thread-empty-policy", + set(), + ) + assert projection is not None + provider = LocalSandboxProvider() + sandbox_id = provider.acquire( + "thread-empty-policy", + user_id="user-1", + ) + sandbox = provider.get(sandbox_id) + assert sandbox is not None + + listing = "\n".join(sandbox.list_dir("/mnt/skills", max_depth=4)) + assert "public-skill" not in listing + assert "custom-skill" not in listing + shell_listing = sandbox.execute_command("ls -R /mnt/skills") + assert "public-skill" not in shell_listing + assert "custom-skill" not in shell_listing + assert "MARKER" not in sandbox.execute_command("find /mnt/skills -name SKILL.md -print -exec cat {} \\;") + with pytest.raises(FileNotFoundError): + sandbox.read_file("/mnt/skills/public/public-skill/SKILL.md") + globbed, _ = sandbox.glob("/mnt/skills", "**/SKILL.md") + assert globbed == [] + grepped, _ = sandbox.grep( + "/mnt/skills", + "MARKER", + literal=True, + ) + assert grepped == [] + # ── AioSandboxProvider ────────────────────────────────────────────── def test_aio_public_skill_mount(self, skills_fs, aio_mod): diff --git a/backend/tests/test_tool_error_handling_middleware.py b/backend/tests/test_tool_error_handling_middleware.py index d38aae278..5f44206ee 100644 --- a/backend/tests/test_tool_error_handling_middleware.py +++ b/backend/tests/test_tool_error_handling_middleware.py @@ -203,6 +203,20 @@ def test_build_subagent_runtime_middlewares_threads_app_config_to_llm_middleware assert policy_idx < durable_idx < date_idx == len(middlewares) - 2 +def test_subagent_runtime_sandbox_does_not_own_lead_skill_projection() -> None: + from deerflow.extensions.registry import ExtensionRegistry + from deerflow.sandbox.middleware import SandboxMiddleware + + middlewares = build_subagent_runtime_middlewares( + app_config=_make_app_config(), + available_skills={"allowed"}, + extensions=ExtensionRegistry().build(), + ) + + sandbox_middleware = next(middleware for middleware in middlewares if isinstance(middleware, SandboxMiddleware)) + assert sandbox_middleware._owns_agent_skill_projection is False + + def test_tool_progress_middleware_is_outer_relative_to_error_handling(monkeypatch: pytest.MonkeyPatch): # ToolProgressMiddleware must have a lower index than ToolErrorHandlingMiddleware # so that the framework's "first in list = outermost" rule makes it outer. @@ -289,6 +303,63 @@ def test_lead_runtime_middlewares_thread_app_config_to_tool_error_handling(monke assert tool_middleware._app_config is app_config +def test_lead_runtime_middlewares_pass_agent_skills_to_sandbox( + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setitem( + sys.modules, + "deerflow.agents.middlewares.input_sanitization_middleware", + _module( + "deerflow.agents.middlewares.input_sanitization_middleware", + InputSanitizationMiddleware=object, + neutralize_untrusted_tags=lambda value: value, + ), + ) + app_config = _make_app_config() + _stub_runtime_middleware_imports(monkeypatch) + + middlewares = build_lead_runtime_middlewares( + app_config=app_config, + available_skills={"allowed-skill"}, + ) + + sandbox_middleware = next(middleware for middleware in middlewares if getattr(middleware, "kwargs", {}).get("available_skills") == {"allowed-skill"}) + assert sandbox_middleware.kwargs == { + "lazy_init": True, + "available_skills": {"allowed-skill"}, + "owns_agent_skill_projection": True, + } + + +def test_lead_runtime_middlewares_can_delegate_skill_projection_ownership( + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setitem( + sys.modules, + "deerflow.agents.middlewares.input_sanitization_middleware", + _module( + "deerflow.agents.middlewares.input_sanitization_middleware", + InputSanitizationMiddleware=object, + neutralize_untrusted_tags=lambda value: value, + ), + ) + app_config = _make_app_config() + _stub_runtime_middleware_imports(monkeypatch) + + middlewares = build_lead_runtime_middlewares( + app_config=app_config, + available_skills={"bootstrap"}, + owns_agent_skill_projection=False, + ) + + sandbox_middleware = next(middleware for middleware in middlewares if getattr(middleware, "kwargs", {}).get("available_skills") == {"bootstrap"}) + assert sandbox_middleware.kwargs == { + "lazy_init": True, + "available_skills": {"bootstrap"}, + "owns_agent_skill_projection": False, + } + + def test_build_lead_runtime_middlewares_orders_thread_data_before_uploads(): """ThreadDataMiddleware must run before UploadsMiddleware so the uploads directory is guaranteed to exist when UploadsMiddleware scans it under diff --git a/config.example.yaml b/config.example.yaml index db36aa393..97783fe2b 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -1688,6 +1688,10 @@ skills: # Path where skills are mounted in the sandbox container # This is used by the agent to access skills in both local and Docker sandbox + # AIO/provisioner and E2B modes require a canonical absolute non-root path + # that does not overlap reserved platform mounts. Providers snapshot this + # path for identity, mounts, metadata, and synchronization, so restart the + # Gateway after changing it. # Default: /mnt/skills container_path: /mnt/skills diff --git a/docker/provisioner/README.md b/docker/provisioner/README.md index fd156f808..87d2814dc 100644 --- a/docker/provisioner/README.md +++ b/docker/provisioner/README.md @@ -20,12 +20,13 @@ The **Sandbox Provisioner** is a FastAPI service that dynamically manages sandbo ### How It Works -1. **Backend Request**: When the backend needs to execute code, it sends a `POST /api/sandboxes` request with a `sandbox_id`, `thread_id`, and optional `user_id`. +1. **Backend Request**: When the backend needs to execute code, it sends a `POST /api/sandboxes` request with a `sandbox_id`, `thread_id`, optional `user_id`, and the configured `skills_container_path` (default: `/mnt/skills`). 2. **Pod Creation**: The provisioner creates a dedicated Pod in the `deer-flow` namespace with: - The sandbox container image (all-in-one-sandbox) - HostPath volumes mounted for: - - `/mnt/skills/{public,custom,legacy}` → Read-only enabled-only skill projections + - `{skills_container_path}/{public,custom,legacy}` → Default read-only skill projections + - `{skills_container_path}/integrations` → Optional read-only managed-integration projection supplied by the Gateway - `/mnt/user-data` → Read-write access to thread-specific data - Resource limits (CPU, memory, ephemeral storage) - Readiness/liveness probes @@ -208,7 +209,7 @@ PYTHONPATH=. python scripts/migrate_user_isolation.py --user-id This moves legacy `threads/{thread_id}/user-data` data under `users//threads/{thread_id}/user-data`, which matches the new provisioner PVC subPath when the gateway base directory is mounted at `deer-flow/` on the PVC. Use `default` as the target user only when the legacy data should remain in the default no-auth user namespace. Run the migration while no gateway or sandbox Pods are writing to those paths. -In hostPath mode, the gateway materializes enabled-only views under `skills_view/public` and `users/{user_id}/skills_view/{custom,legacy}` beneath `DEER_FLOW_HOST_BASE_DIR`; the provisioner mounts those stable directories. When skills are materialized per thread on the same PVC, set `SKILLS_PVC_NAME` to that PVC and configure `SKILLS_PVC_SUBPATH_TEMPLATE=deer-flow/users/{user_id}/threads/{thread_id}/skills`. Leaving the template empty preserves the legacy behavior of mounting the skills PVC root at `/mnt/skills`. The gateway does not yet populate that PVC layout dynamically, so PVC-backed skills do not receive hostPath projection updates. +In hostPath mode, the gateway materializes enabled-only views under `skills_view/public` and `users/{user_id}/skills_view/{custom,legacy}` beneath `DEER_FLOW_HOST_BASE_DIR`; the provisioner mounts those stable directories. A lead Agent with an explicit skills policy supplies all four category mounts from `users/{user_id}/threads/{thread_id}/skills_view`; those overrides replace the default hostPath or root skills-PVC mount. When `USERDATA_PVC_NAME` is configured, the provisioner mounts these thread categories from that PVC using `deer-flow/users/{user_id}/threads/{thread_id}/skills_view/{category}` subpaths. For unrestricted threads, operators can still set `SKILLS_PVC_NAME` and optionally configure `SKILLS_PVC_SUBPATH_TEMPLATE`; leaving the template empty mounts the skills PVC root unchanged. The gateway does not populate the unrestricted `SKILLS_PVC_SUBPATH_TEMPLATE` layout dynamically. **hostPath skills volumes require the gateway and the K8s node to see the same `DEER_FLOW_HOST_BASE_DIR`** (single-node deployment, or NFS/shared storage mounted at that path on every node). The gateway writes the projection there before every sandbox acquire, so as long as that path is shared, the directory the provisioner mounts always exists by the time the Pod is scheduled — even a boot-time rebuild failure for one user self-heals on their next acquire, before the provisioner is called. `skills-custom` and `skills-legacy` use hostPath type `Directory` (not `DirectoryOrCreate`): if the shared-storage assumption is violated — the gateway wrote to a different node than the one the Pod lands on — Pod creation now fails visibly instead of silently mounting an empty directory. Use `SKILLS_PVC_NAME` instead of hostPath for genuinely multi-node clusters without shared storage. diff --git a/docker/provisioner/app.py b/docker/provisioner/app.py index 071e6d8b2..f642d4150 100644 --- a/docker/provisioner/app.py +++ b/docker/provisioner/app.py @@ -36,6 +36,7 @@ import re import secrets import time from contextlib import asynccontextmanager +from pathlib import PurePosixPath import urllib3 from fastapi import FastAPI, HTTPException, Request, Response @@ -107,16 +108,26 @@ if SANDBOX_SERVICE_TYPE not in {"NodePort", "ClusterIP"}: SAFE_THREAD_ID_PATTERN = r"^[A-Za-z0-9_-]{1,64}$" SAFE_USER_ID_PATTERN = r"^[A-Za-z0-9_\-]+$" DEFAULT_USER_ID = "default" +DEFAULT_SKILLS_CONTAINER_PATH = "/mnt/skills" MAX_EXTRA_MOUNTS = 10 ALLOWED_EXTRA_MOUNT_PATHS = { "/mnt/acp-workspace", - "/mnt/skills/custom", - "/mnt/skills/integrations", "/mnt/integrations/lark-cli/config", "/mnt/integrations/lark-cli/config/locks", "/mnt/integrations/lark-cli/data", "/mnt/integrations/lark-cli/runtime", } +MANAGED_SKILL_CATEGORY_NAMES = ( + "public", + "custom", + "legacy", + "integrations", +) +RESERVED_SANDBOX_MOUNT_PATHS = ( + "/mnt/user-data", + "/mnt/acp-workspace", + "/mnt/integrations/lark-cli", +) # Path to the kubeconfig *inside* the provisioner container. # Typically the host's ~/.kube/config is mounted here. @@ -171,16 +182,57 @@ def _is_path_under_base(path: str, base: str) -> bool: return False -def _normalize_extra_mount_container_path(container_path: str) -> str: +def _normalize_skills_container_path(container_path: str) -> str: + """Return a canonical skills root that cannot overlap platform mounts.""" + if not container_path or not container_path.startswith("/") or container_path.startswith("//"): + raise HTTPException(status_code=400, detail="The skills container path must be an absolute non-root path") + + normalized = posixpath.normpath(container_path) + if normalized == "/" or normalized != container_path: + raise HTTPException( + status_code=400, + detail="The skills container path must not be root or contain redundant separators, '.' or '..'", + ) + + root = PurePosixPath(normalized) + for reserved_path in RESERVED_SANDBOX_MOUNT_PATHS: + reserved = PurePosixPath(reserved_path) + if root == reserved or root.is_relative_to(reserved) or reserved.is_relative_to(root): + raise HTTPException( + status_code=400, + detail=f"The skills container path {normalized!r} overlaps reserved sandbox path {reserved_path!r}", + ) + return normalized + + +def _managed_skill_category_mount_paths( + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> set[str]: + root = _normalize_skills_container_path(skills_container_path) + return {posixpath.join(root, category) for category in MANAGED_SKILL_CATEGORY_NAMES} + + +def _normalize_extra_mount_container_path( + container_path: str, + *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> str: normalized = posixpath.normpath(container_path) if not normalized.startswith("/"): raise HTTPException(status_code=400, detail=f"Extra mount path must be absolute: {container_path}") - if normalized not in ALLOWED_EXTRA_MOUNT_PATHS: + allowed_paths = ALLOWED_EXTRA_MOUNT_PATHS | _managed_skill_category_mount_paths( + skills_container_path + ) + if normalized not in allowed_paths: raise HTTPException(status_code=400, detail=f"Unsupported extra mount path: {container_path}") return normalized -def _validated_extra_mounts(extra_mounts: list["ExtraMount"] | None) -> list["ExtraMount"]: +def _validated_extra_mounts( + extra_mounts: list["ExtraMount"] | None, + *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> list["ExtraMount"]: """Validate extra mounts before converting them into K8s hostPath/PVC mounts.""" if not extra_mounts: return [] @@ -197,7 +249,10 @@ def _validated_extra_mounts(extra_mounts: list["ExtraMount"] | None) -> list["Ex if not _is_path_under_base(host_path, host_base_dir): raise HTTPException(status_code=400, detail=f"Extra mount host path is outside DeerFlow state: {mount.host_path}") - container_path = _normalize_extra_mount_container_path(mount.container_path) + container_path = _normalize_extra_mount_container_path( + mount.container_path, + skills_container_path=skills_container_path, + ) if container_path in seen_container_paths: raise HTTPException(status_code=400, detail=f"Duplicate extra mount path: {container_path}") seen_container_paths.add(container_path) @@ -259,14 +314,21 @@ def _runtime_provided_extra_mounts( return [mount for mount in extra_mounts if posixpath.normpath(mount.container_path) not in dropped] -def _lark_broker_credential_mounts(extra_mounts: list["ExtraMount"] | None) -> dict[str, "ExtraMount"]: +def _lark_broker_credential_mounts( + extra_mounts: list["ExtraMount"] | None, + *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> dict[str, "ExtraMount"]: """Extract the config/locks/data mounts the broker sidecar needs. Keyed by container path so the caller can wire each into the sidecar's fixed ``/var/lark/{config,config/locks,data}`` paths. """ result: dict[str, ExtraMount] = {} - for mount in _validated_extra_mounts(extra_mounts): + for mount in _validated_extra_mounts( + extra_mounts, + skills_container_path=skills_container_path, + ): normalized = posixpath.normpath(mount.container_path) if normalized in ( LARK_CLI_CONFIG_CONTAINER_PATH, @@ -409,6 +471,8 @@ class CreateSandboxRequest(BaseModel): user_id: str = Field(default=DEFAULT_USER_ID, pattern=SAFE_USER_ID_PATTERN) extra_mounts: list[ExtraMount] = Field(default_factory=list) include_legacy_skills: bool = False + # Sent explicitly by new Gateways; the default keeps old Gateways compatible. + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH # When true (and LARK_CLI_INIT_IMAGE is configured), provision the sandbox # lark-cli runtime via an init container + emptyDir instead of a runtime # hostPath/PVC extra mount. @@ -447,9 +511,18 @@ def _sandbox_url(sandbox_id: str, node_port: int | None = None) -> str: return f"http://{NODE_HOST}:{node_port}" -def _build_extra_volumes(extra_mounts: list[ExtraMount] | None = None) -> list[k8s_client.V1Volume]: +def _build_extra_volumes( + extra_mounts: list[ExtraMount] | None = None, + *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> list[k8s_client.V1Volume]: volumes: list[k8s_client.V1Volume] = [] - for index, mount in enumerate(_validated_extra_mounts(extra_mounts)): + for index, mount in enumerate( + _validated_extra_mounts( + extra_mounts, + skills_container_path=skills_container_path, + ) + ): if USERDATA_PVC_NAME: volumes.append( k8s_client.V1Volume( @@ -473,9 +546,18 @@ def _build_extra_volumes(extra_mounts: list[ExtraMount] | None = None) -> list[k return volumes -def _build_extra_volume_mounts(extra_mounts: list[ExtraMount] | None = None) -> list[k8s_client.V1VolumeMount]: +def _build_extra_volume_mounts( + extra_mounts: list[ExtraMount] | None = None, + *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, +) -> list[k8s_client.V1VolumeMount]: mounts: list[k8s_client.V1VolumeMount] = [] - for index, mount in enumerate(_validated_extra_mounts(extra_mounts)): + for index, mount in enumerate( + _validated_extra_mounts( + extra_mounts, + skills_container_path=skills_container_path, + ) + ): volume_mount = k8s_client.V1VolumeMount( name=_extra_mount_volume_name(index), mount_path=mount.container_path, @@ -493,25 +575,42 @@ def _build_volumes( *, include_legacy_skills: bool = False, extra_mounts: list[ExtraMount] | None = None, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, ) -> list[k8s_client.V1Volume]: """Build volume list: PVC when configured, otherwise hostPath. Skills are split into public, per-user custom, and legacy (global-custom) - volumes so that ``/mnt/skills/{public,custom,legacy}/`` paths resolve + volumes so that ``/{public,custom,legacy}/`` paths resolve correctly inside the sandbox — matching the hostPath layout produced by ``LocalSandboxProvider`` and ``AioSandboxProvider``. """ volumes: list[k8s_client.V1Volume] = [] del include_legacy_skills # retained for request compatibility + skills_root = _normalize_skills_container_path(skills_container_path) + managed_skill_paths = _managed_skill_category_mount_paths(skills_root) + validated_extra_mounts = _validated_extra_mounts( + extra_mounts, + skills_container_path=skills_root, + ) + skill_overrides = { + posixpath.normpath(mount.container_path) + for mount in validated_extra_mounts + if posixpath.normpath(mount.container_path) + in managed_skill_paths + } + all_skill_categories_overridden = managed_skill_paths <= skill_overrides # ── Skills volumes ──────────────────────────────────────────────── - if SKILLS_PVC_NAME: - # PVC mode: three-way subPath not yet supported; fall back to - # single-volume mount for backward compatibility. - logger.warning("SKILLS_PVC_NAME is set — three-way skills layout is not supported in PVC mode yet; falling back to single /mnt/skills mount") + if SKILLS_PVC_NAME and not all_skill_categories_overridden: + # An unrestricted thread keeps the operator-provided skills PVC root. + logger.warning( + "SKILLS_PVC_NAME is set — three-way skills layout is not supported in PVC mode yet; " + "falling back to single %s mount", + skills_root, + ) volumes.append( k8s_client.V1Volume( name="skills", @@ -521,18 +620,19 @@ def _build_volumes( ), ) ) - else: + elif not SKILLS_PVC_NAME: # hostPath mode: three-way layout public_path = join_host_path(DEER_FLOW_HOST_BASE_DIR, "skills_view", "public") - volumes.append( - k8s_client.V1Volume( - name="skills-public", - host_path=k8s_client.V1HostPathVolumeSource( - path=public_path, - type="Directory", - ), + if posixpath.join(skills_root, "public") not in skill_overrides: + volumes.append( + k8s_client.V1Volume( + name="skills-public", + host_path=k8s_client.V1HostPathVolumeSource( + path=public_path, + type="Directory", + ), + ) ) - ) user_custom_path = join_host_path( DEER_FLOW_HOST_BASE_DIR, @@ -541,28 +641,30 @@ def _build_volumes( "skills_view", "custom", ) - volumes.append( - k8s_client.V1Volume( - name="skills-custom", - host_path=k8s_client.V1HostPathVolumeSource( - path=user_custom_path, - type="Directory", - ), + if posixpath.join(skills_root, "custom") not in skill_overrides: + volumes.append( + k8s_client.V1Volume( + name="skills-custom", + host_path=k8s_client.V1HostPathVolumeSource( + path=user_custom_path, + type="Directory", + ), + ) ) - ) legacy_path = join_host_path( DEER_FLOW_HOST_BASE_DIR, "users", user_id, "skills_view", "legacy" ) - volumes.append( - k8s_client.V1Volume( - name="skills-legacy", - host_path=k8s_client.V1HostPathVolumeSource( - path=legacy_path, - type="Directory", - ), + if posixpath.join(skills_root, "legacy") not in skill_overrides: + volumes.append( + k8s_client.V1Volume( + name="skills-legacy", + host_path=k8s_client.V1HostPathVolumeSource( + path=legacy_path, + type="Directory", + ), + ) ) - ) # ── User-data volume ────────────────────────────────────────────── @@ -586,10 +688,11 @@ def _build_volumes( volumes.extend( _build_extra_volumes( _runtime_provided_extra_mounts( - extra_mounts, + validated_extra_mounts, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, - ) + ), + skills_container_path=skills_root, ) ) # The runtime emptyDir is shared by the init container (writer) and the @@ -603,7 +706,10 @@ def _build_volumes( ) # Pattern B: config/locks/data volumes go to the broker sidecar only. if _lark_cli_broker_enabled(provision_lark_cli_broker): - credential_mounts = _lark_broker_credential_mounts(extra_mounts) + credential_mounts = _lark_broker_credential_mounts( + validated_extra_mounts, + skills_container_path=skills_root, + ) for container_path, volume_name in ( (LARK_CLI_CONFIG_CONTAINER_PATH, LARK_BROKER_CONFIG_VOLUME_NAME), (LARK_CLI_LOCKS_CONTAINER_PATH, LARK_BROKER_LOCKS_VOLUME_NAME), @@ -640,23 +746,37 @@ def _build_volume_mounts( *, include_legacy_skills: bool = False, extra_mounts: list[ExtraMount] | None = None, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, ) -> list[k8s_client.V1VolumeMount]: """Build volume mount list, mirroring three-way skills layout. - Skills are mounted to ``/mnt/skills/{public,custom,legacy}/`` so that + Skills are mounted to ``/{public,custom,legacy}/`` so that category-aware ``Skill.get_container_path()`` paths resolve correctly. - PVC mode falls back to a single ``/mnt/skills`` mount and can optionally + PVC mode falls back to a single ```` mount and can optionally scope that mount with ``SKILLS_PVC_SUBPATH_TEMPLATE``. """ mounts: list[k8s_client.V1VolumeMount] = [] del include_legacy_skills # retained for request compatibility + skills_root = _normalize_skills_container_path(skills_container_path) + managed_skill_paths = _managed_skill_category_mount_paths(skills_root) + validated_extra_mounts = _validated_extra_mounts( + extra_mounts, + skills_container_path=skills_root, + ) + skill_overrides = { + posixpath.normpath(mount.container_path) + for mount in validated_extra_mounts + if posixpath.normpath(mount.container_path) + in managed_skill_paths + } + all_skill_categories_overridden = managed_skill_paths <= skill_overrides - if SKILLS_PVC_NAME: + if SKILLS_PVC_NAME and not all_skill_categories_overridden: skills_mount = k8s_client.V1VolumeMount( name="skills", - mount_path="/mnt/skills", + mount_path=skills_root, read_only=True, ) if SKILLS_PVC_SUBPATH_TEMPLATE: @@ -665,26 +785,25 @@ def _build_volume_mounts( thread_id=thread_id, ) mounts.append(skills_mount) - else: - mounts.extend( - [ - k8s_client.V1VolumeMount( - name="skills-public", - mount_path="/mnt/skills/public", - read_only=True, - ), - k8s_client.V1VolumeMount( - name="skills-custom", - mount_path="/mnt/skills/custom", - read_only=True, - ), - k8s_client.V1VolumeMount( - name="skills-legacy", - mount_path="/mnt/skills/legacy", - read_only=True, - ), - ] - ) + elif not SKILLS_PVC_NAME: + default_skill_mounts = [ + k8s_client.V1VolumeMount( + name="skills-public", + mount_path=posixpath.join(skills_root, "public"), + read_only=True, + ), + k8s_client.V1VolumeMount( + name="skills-custom", + mount_path=posixpath.join(skills_root, "custom"), + read_only=True, + ), + k8s_client.V1VolumeMount( + name="skills-legacy", + mount_path=posixpath.join(skills_root, "legacy"), + read_only=True, + ), + ] + mounts.extend(mount for mount in default_skill_mounts if mount.mount_path not in skill_overrides) userdata_mount = k8s_client.V1VolumeMount( name="user-data", @@ -697,10 +816,11 @@ def _build_volume_mounts( mounts.extend( _build_extra_volume_mounts( _runtime_provided_extra_mounts( - extra_mounts, + validated_extra_mounts, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, - ) + ), + skills_container_path=skills_root, ) ) # Sandbox reads the runtime dir (real binary in Pattern A, shim in Pattern B). @@ -766,6 +886,8 @@ def _build_lark_cli_init_containers( def _build_lark_cli_broker_sidecars( provision_lark_cli_broker: bool, extra_mounts: list[ExtraMount] | None, + *, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, ) -> list[k8s_client.V1Container]: """Broker sidecar that holds lark-cli + the per-user credentials (Pattern B). @@ -776,7 +898,10 @@ def _build_lark_cli_broker_sidecars( """ if not _lark_cli_broker_enabled(provision_lark_cli_broker): return [] - credential_mounts = _lark_broker_credential_mounts(extra_mounts) + credential_mounts = _lark_broker_credential_mounts( + extra_mounts, + skills_container_path=skills_container_path, + ) volume_mounts: list[k8s_client.V1VolumeMount] = [] for container_path, volume_name, sidecar_path in ( (LARK_CLI_CONFIG_CONTAINER_PATH, LARK_BROKER_CONFIG_VOLUME_NAME, LARK_BROKER_SIDECAR_CONFIG_PATH), @@ -830,6 +955,7 @@ def _build_pod( *, include_legacy_skills: bool = False, extra_mounts: list[ExtraMount] | None = None, + skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH, provision_lark_cli_runtime: bool = False, provision_lark_cli_broker: bool = False, ) -> k8s_client.V1Pod: @@ -903,6 +1029,7 @@ def _build_pod( user_id=user_id, include_legacy_skills=include_legacy_skills, extra_mounts=extra_mounts, + skills_container_path=skills_container_path, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, ), @@ -911,7 +1038,11 @@ def _build_pod( allow_privilege_escalation=True, ), ), - *_build_lark_cli_broker_sidecars(provision_lark_cli_broker, extra_mounts), + *_build_lark_cli_broker_sidecars( + provision_lark_cli_broker, + extra_mounts, + skills_container_path=skills_container_path, + ), ], init_containers=init_containers, volumes=_build_volumes( @@ -919,6 +1050,7 @@ def _build_pod( user_id=user_id, include_legacy_skills=include_legacy_skills, extra_mounts=extra_mounts, + skills_container_path=skills_container_path, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, ), @@ -1032,15 +1164,19 @@ def create_sandbox(req: CreateSandboxRequest): thread_id = req.thread_id or sandbox_id user_id = req.user_id include_legacy_skills = req.include_legacy_skills + skills_container_path = _normalize_skills_container_path( + req.skills_container_path + ) provision_lark_cli_runtime = req.provision_lark_cli_runtime provision_lark_cli_broker = req.provision_lark_cli_broker logger.info( - "Received request to create sandbox '%s' for thread '%s' user '%s' include_legacy_skills=%s provision_lark_cli_runtime=%s provision_lark_cli_broker=%s", + "Received request to create sandbox '%s' for thread '%s' user '%s' include_legacy_skills=%s skills_container_path=%s provision_lark_cli_runtime=%s provision_lark_cli_broker=%s", sandbox_id, thread_id, user_id, include_legacy_skills, + skills_container_path, _lark_cli_runtime_enabled(provision_lark_cli_runtime), _lark_cli_broker_enabled(provision_lark_cli_broker), ) @@ -1064,6 +1200,7 @@ def create_sandbox(req: CreateSandboxRequest): user_id=user_id, include_legacy_skills=include_legacy_skills, extra_mounts=req.extra_mounts, + skills_container_path=skills_container_path, provision_lark_cli_runtime=provision_lark_cli_runtime, provision_lark_cli_broker=provision_lark_cli_broker, ), diff --git a/frontend/src/content/en/harness/skills.mdx b/frontend/src/content/en/harness/skills.mdx index b47ed3540..0ac29004d 100644 --- a/frontend/src/content/en/harness/skills.mdx +++ b/frontend/src/content/en/harness/skills.mdx @@ -133,6 +133,16 @@ skills: - **Empty list `[]`**: the agent has no skills. - **Named list**: the agent loads only those specific skills. +For a lead custom Agent, an explicit list is enforced in both discovery and the +sandbox filesystem. `/mnt/skills` contains only the intersection of globally +enabled, user-visible, and Agent-allowed skills. AIO and E2B enforce that view +for shell commands and file tools. Local enforces it through managed virtual +paths only while host bash is disabled (the default); its host filesystem is +not a security boundary, and enabling host bash makes an explicit Agent policy +fail closed. A different sandbox provider must advertise the same contract or +an explicit policy fails closed before the run acquires a sandbox. Agent +configuration changes take effect on the thread's next run. + ## Skill evolution DeerFlow includes an optional **skill evolution** feature that allows the agent to autonomously create and improve skills in the `skills/custom/` directory: diff --git a/frontend/src/content/zh/harness/skills.mdx b/frontend/src/content/zh/harness/skills.mdx index 56d112ee1..8cda27409 100644 --- a/frontend/src/content/zh/harness/skills.mdx +++ b/frontend/src/content/zh/harness/skills.mdx @@ -127,6 +127,13 @@ skills: - **空列表 `[]`**：Agent 没有技能。 - **命名列表**：Agent 只加载那些特定技能。 +对于作为主 Agent 运行的自定义 Agent，显式列表会同时约束技能发现和沙箱文件系统。 +`/mnt/skills` 只包含“全局已启用、当前用户可见、Agent 已允许”三者的交集。AIO 和 +E2B 会对 shell 命令及文件工具强制执行该视图。Local 仅在 host bash 保持默认禁用时， +通过受管理的虚拟路径执行该视图；主机文件系统并不是安全边界，启用 host bash 后， +显式 Agent 策略会失败关闭。其他 sandbox provider 若未声明支持，显式策略会在获取 +沙箱前失败关闭。Agent 配置变更会在该线程的下一次运行生效。 + ## 技能进化 DeerFlow 包含一个可选的**技能进化**功能，允许 Agent 在 `skills/custom/` 目录中自主创建和改进技能：