fix(sandbox): enforce disabled skills in filesystem views (#4178)

* fix(sandbox): project enabled skills into sandbox views

* fix(skills): keep projection mutations consistent

* fix(skills): fail closed on projection errors

* fix(skills): isolate per-scope failures during boot projection rebuild

rebuild_all_skill_projections() propagated any exception from the public
rebuild or from a single user's rebuild straight out of the gateway
lifespan startup, uncaught. A single broken user directory (bad
permissions, corrupted _skill_states.json, unreadable content) would
therefore abort gateway boot for every user, not just that one -
_rebuild_*_locked already fails closed internally (clears the view and
re-raises), so the boot loop only needed to stop treating that re-raise
as fatal.

Each scope's rebuild now fails closed independently and boot continues;
a scope left empty by a boot failure self-heals on the next sandbox
acquire via ensure_skill_projections().

Also patches deerflow.skills.projection.rebuild_all_skill_projections in
the memory-flush lifespan test fixture, matching the two sibling
fixtures in the same file — this call is now on the lifespan startup
path and the fixture's minimal SimpleNamespace config predates it.

* test(skills): update authz test for the projection-aware public toggle

_persist_shared_skill_state (introduced earlier in this branch) reads
the shared extensions_config.json fresh from disk under the projection
lock instead of through the cached get_extensions_config() singleton -
that's the whole point of the fix (stale worker caches must not clobber
another worker's concurrent update). The name no longer exists on the
skills router module, so the test's monkeypatch of it started raising
AttributeError instead of exercising the endpoint.

The mock storage in this test isn't a real LocalSkillStorage instance,
so _persist_shared_skill_state's projection-mutation branch is already
skipped (nullcontext) and it falls back to a fresh ExtensionsConfig()
for the nonexistent tmp config_path - no replacement monkeypatch needed.

* fix(sandbox): make skill projection ensure best-effort in acquire

acquire() called _ensure_skills_projection() directly, outside any
try/except, in both LocalSandboxProvider and AioSandboxProvider. Every
other skill-mount setup path in these providers has always caught
exceptions and logged a warning rather than failing sandbox acquire
outright (e.g. when config.yaml can't be resolved) - these two new call
sites broke that contract, so any projection failure (including simply
not having a config.yaml, as in CI's test environment) now failed
acquire() itself instead of just leaving skill mounts off.

_ensure_skills_projection now catches its own exceptions and returns
None; both providers' callers already tolerate that (a None projection
skips the skill-specific mounts, matching the existing degrade path)
after making _append_public_skill_mapping and the custom/legacy mount
block in LocalSandboxProvider explicitly None-safe.

Caught by running the full suite with config.yaml removed, matching
CI's environment - not caught locally because a real config.yaml was
present, masking the failure.

* fix(sandbox): make E2B skill projection mounts best-effort

_skill_projection_mounts called ensure_skill_projections with no guard,
unlike Local/AIO's _ensure_skills_projection. A raise propagated out of
_apply_mounts before the configured-mounts loop ran, so a skills
projection failure dropped the operator's own configured mounts too -
only caught by create()'s outer warning, with nothing applied at all.

Swallow here and return an empty mount list on failure, matching the
Local/AIO pattern: still fail-closed for skills, but no longer widens
the blast radius to unrelated configured mounts.

Review feedback from PR #4178.

* docs(skills): document projection trade-offs flagged in review

- _update_tree_digest: note the metadata-only (not content) hashing
  trade-off and why runtime writes through this codebase are still
  covered regardless (rebuild-under-lock + rename always changes inode).
- LocalSandboxProvider.acquire: note the acquire-time self-heal cost
  (cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift).
- skill_projection_mutation: drop the no-op except-Exception-then-raise;
  a raise from the mutation already propagates past the yield with the
  view left cleared, no explicit re-raise needed.
- provisioner README: spell out that hostPath skills volumes require
  the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node
  or shared storage), and that the custom/legacy volumes' hostPath type
  Directory (not DirectoryOrCreate) makes a violation of that assumption
  a visible Pod-creation failure instead of a silent empty mount.

Review feedback from PR #4178.

* fix(skills): lazily repair user projections

* fix(skills): close projection review gaps

* fix(skills): refresh user projection enable state

* fix(skills): close projection review follow-ups

* fix(skills): preserve state across projection writes

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Xinmin Zeng 2026-07-31 17:55:24 +08:00 committed by GitHub
parent 486b51eb51
commit f2e832330e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
35 changed files with 2010 additions and 264 deletions

View File

@ -12,6 +12,18 @@ This section accumulates work toward the **2.1.0** milestone
### ⚠ Breaking changes
- **skills:** Sandboxes now reserve `/mnt/skills` for managed enabled-only
projections. `DEER_FLOW_HOST_SKILLS_PATH` and `SKILLS_HOST_PATH` are no longer
used; Docker/AIO and hostPath deployments derive projection paths from
`DEER_FLOW_HOST_BASE_DIR`. E2B operator mounts targeting `/mnt/skills` or any
child path are skipped with a warning so they cannot shadow the managed
projection; move extra E2B content to a different container path. User
projections re-read global enable state from disk so toggles propagate across
Gateway workers on the next sandbox acquire. Existing E2B sandboxes retain
their creation-time snapshot until they are recreated. PVC-backed provisioner
deployments still mount the operator-supplied PVC snapshot directly, so
disabled-skill filesystem isolation does not apply in PVC mode until dynamic
PVC materialization is implemented. ([#4178])
- **sandbox:** E2B now enforces `sandbox.replicas` as a process-local capacity
limit. The default `wait` policy waits for `acquire_timeout`, then fails the
agent turn. DeerFlow does not retry the turn automatically. Use `burst` with
@ -1264,6 +1276,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
[#4170]: https://github.com/bytedance/deer-flow/pull/4170
[#4171]: https://github.com/bytedance/deer-flow/pull/4171
[#4174]: https://github.com/bytedance/deer-flow/pull/4174
[#4178]: https://github.com/bytedance/deer-flow/pull/4178
[#4181]: https://github.com/bytedance/deer-flow/pull/4181
[#4187]: https://github.com/bytedance/deer-flow/pull/4187
[#4188]: https://github.com/bytedance/deer-flow/pull/4188

View File

@ -718,6 +718,8 @@ An enabled skill's `allowed-tools` policy applies only after that skill is expli
When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills.
Disabling a skill also removes it from the sandbox filesystem view, so shell commands and structured file tools follow the same enabled state. Local, Docker/AIO, hostPath provisioner, and newly created E2B sandboxes source `/mnt/skills` from enabled-only projections that update when public, custom, legacy, or managed integration skills are toggled, edited, created, deleted, or installed. Managed integration packages remain shared, while their projected filesystem visibility follows each user's enabled state. Multi-worker Gateways re-read on-disk enable state while rebuilding user projections, so a toggle handled by one worker is honored by another worker's next sandbox acquire. Existing E2B sandboxes retain their creation-time snapshot until they are recreated. PVC-backed provisioner skills keep their configured PVC snapshot/layout for now; dynamic PVC materialization is tracked separately.
Managed integrations install shared read-only skill packs without mixing them
into custom skills. The Lark/Feishu CLI integration is available under
`Settings → Integrations → Lark / Feishu CLI`; an administrator installs or

View File

@ -608,9 +608,12 @@ that cannot tell sibling branches apart.
**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop.
**Environment policy** (`sandbox/env_policy.py`): `execute_command` no longer inherits the full `os.environ`. `build_sandbox_env()` scrubs secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. `OPENAI_API_KEY`) never leak into skill subprocesses. Benign vars (`PATH`, `HOME`, `LANG`, `VIRTUAL_ENV`, ...) are preserved.
**Implementations**:
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Legacy global-custom mounts are gated by the same user-scoped skill discovery rule used for prompt/list visibility; providers must not infer visibility from raw directory presence alone.
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. Legacy global-custom mounts follow the same shared visibility helper as local and remote providers. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support.
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Public, custom, legacy, and managed integration skill mappings point at stable enabled-only projection roots rather than raw skill directories.
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. Local-container and hostPath-provisioner mounts use the same stable skill projection roots; PVC-backed skills remain governed by the operator-supplied PVC layout until PVC materialization is implemented. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support.
- `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation.
New sandboxes receive a one-shot upload from the enabled-only public, custom,
legacy, and managed integration projections. Existing E2B VMs keep their
creation-time snapshot because E2B has no shared host mount.
Acquire and release share a per-user and thread lock. The provider lock does
not cover remote IO. `burst_limit` adds capacity only for the `burst` policy.
The `wait` policy fails the turn after `acquire_timeout`. The runtime does not
@ -672,7 +675,7 @@ that cannot tell sibling branches apart.
**Virtual Path System**:
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`; raw skills stay under `deer-flow/skills/` and managed integration storage, while sandboxes read `backend/.deer-flow/skills_view/public/` and `backend/.deer-flow/users/{user_id}/skills_view/{custom,legacy,integrations}/`
- Translation: `LocalSandboxProvider` builds per-thread `PathMapping`s for the user-data prefixes at acquire time; `tools.py` keeps `replace_virtual_path()` / `replace_virtual_paths_in_command()` as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept `/mnt/user-data/...` natively.
- Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread)
@ -766,6 +769,7 @@ E2B output sync records remote file versions and actual host file metadata in a
- **Loading**: `load_skills()` recursively scans public, per-user custom, global integration, and legacy custom locations for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json plus per-user skill state for non-public categories; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json.
- **External reload**: `POST /api/skills/reload` is an admin-only, process-local invalidation hook for trusted MinIO/NFS/CSI writes. `SkillStorage` instances do not cache a catalog — `load_skills()` scans on every call — so the route clears all `(app_config, user_id)` entries and the rendered prompt-section LRU, then waits up to the shared refresh timeout for the existing off-loop single-flight refresh. Each invalidation receives a generation-bound result handle; a successful scan atomically replaces the global enabled-skills cache, while a loader-level failure propagates to the HTTP waiter and preserves the last-known-good global cache. Per-user/config scans capture the refresh version and cannot repopulate shared caches if invalidation occurs while they are loading. A timed-out HTTP wait fails generically while the daemon refresh worker continues. Subsequent runs rescan after a successful reload; active runs keep their existing snapshot. Each Uvicorn worker/Kubernetes Pod must be targeted separately. Direct mount writes bypass install/edit validation, SkillScan, and history, so mounted roots are an operator-controlled trust boundary.
- **Tool policy**: Agent `allowed-tools` declarations apply dynamically only to slash-activated skills and skills captured in `ThreadState.skill_context` through configured `read_file` loads; passive enabled skills and custom-agent/subagent skill allowlists remain discoverable without clamping the baseline toolset. Subagents render only skill discovery metadata at startup and reuse the same adjacent `SkillActivationMiddleware` + `SkillToolPolicyMiddleware` pair as the lead; their configured `skills` field limits discovery and activation instead of eagerly loading bodies or unioning policies. Slash policy is dominant for its run, preventing subsequently read skills from widening explicit authority; autonomous captured skills use the existing union only when no slash source exists. `tool_search` and `describe_skill` stay available as framework discovery infrastructure, while every discovered or promoted business tool still requires active-policy permission for schema visibility and execution; `task` likewise requires an explicit declaration. Each active model call intentionally reloads the full live registry so enable/disable changes, frontmatter edits, and custom/public name-shadow winners take effect without a stale TTL or unsafe direct-path cache; all tool calls produced by that model step reuse the resulting source-and-path-signed decision. Registry failures and all-invalid active sets fail closed, while stale individual paths are skipped when another valid skill remains. This is best-effort behavioral scoping, not a hard security boundary: alternate loading paths are not captured and bounded autonomous context may evict entries.
- **Sandbox projection**: `skills/projection.py` materializes enabled-only trees at `{base_dir}/skills_view/public` and `{base_dir}/users/{user_id}/skills_view/{custom,legacy,integrations}`. It hardlinks files when possible and falls back to copies across filesystems. Storage writes, archive installs, deletes, and toggles rebuild under a cross-process lock; Gateway boot ensures only the shared public view, while each user view is repaired lazily on first sandbox acquire. Managed integration packages are global, but their projected category is per-user because enabled state is isolated. Rebuilds stage a complete tree and reconcile it with per-file atomic replacement, so unrelated enabled skills remain continuously visible; disable/delete paths remove only the affected package before mutating to preserve fail-closed behavior. User projection rebuilds re-read global enable state from disk instead of the process singleton, so a toggle handled by another Gateway worker is reflected on the next acquire. Gateway public-skill toggles take the public projection lock before the shared `extensions_config_write_lock`, re-read an existing config from disk, persist the full model shape, and rebuild before responding; keep this as one worker-owned critical section so MCP writes cannot interleave and request cancellation cannot release either lock while the worker still runs. The shared public steady-state signature check runs without the global projection lock; stale/error paths take the lock and re-check before rebuilding or clearing. User-scope checks remain serialized per user. Category root inodes remain stable so live bind mounts observe content changes without sandbox recreation. Projection failures clear the affected view before raising.
- **Injection (legacy / default)**: Enabled skills are listed in the agent system prompt with full metadata and container paths (`<available_skills>` block). Controlled by `skills.deferred_discovery: false` (default).
- **Deferred discovery** (`skills.deferred_discovery: true`): Skills are listed by name only in a compact `<skill_index>` block, keeping the system prompt prefix-cache friendly. The agent calls the `describe_skill` tool at runtime to fetch full metadata for skills it wants to use, then loads the SKILL.md via `read_file`. Two new modules support this path:
- `skills/catalog.py``SkillCatalog` (immutable, searchable; query forms: `select:a,b`, `+prefix`, free-text regex); `select:` returns all requested skills without a result cap; other modes cap at `MAX_RESULTS=5`.

View File

@ -210,6 +210,12 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
config = get_gateway_config()
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
from deerflow.skills.projection import ensure_public_skill_projection
public_projection_ready = await asyncio.to_thread(ensure_public_skill_projection, app_config=startup_config)
if public_projection_ready:
logger.info("Ensured the public skill projection; user projections repair lazily on sandbox acquire")
# Agent observability (Monocle). Off by default; enabled with
# MONOCLE_TRACING. Initialized here at startup — not at import time — so a
# plain `import deerflow.agents` never installs a process-global tracer.

View File

@ -273,8 +273,9 @@ async def update_custom_skill(skill_name: str, body: CustomSkillUpdateRequest, r
if scan.decision == "block":
raise HTTPException(status_code=400, detail=f"Security scan blocked the edit: {scan.reason}")
prev_content = storage.read_custom_skill(skill_name)
storage.write_custom_skill(skill_name, SKILL_MD_FILE, body.content)
storage.append_history(
await asyncio.to_thread(storage.write_custom_skill, skill_name, SKILL_MD_FILE, body.content)
await asyncio.to_thread(
storage.append_history,
skill_name,
{
"action": "human_edit",
@ -305,7 +306,8 @@ async def delete_custom_skill(skill_name: str, request: Request, config: AppConf
try:
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
storage = _get_user_skill_storage(config)
storage.delete_custom_skill(
await asyncio.to_thread(
storage.delete_custom_skill,
skill_name,
history_meta={
"action": "human_delete",
@ -384,10 +386,10 @@ async def rollback_custom_skill(skill_name: str, body: SkillRollbackRequest, req
"scanner": {"decision": scan.decision, "reason": scan.reason, "static_findings": static_findings},
}
if scan.decision == "block":
storage.append_history(skill_name, history_entry)
await asyncio.to_thread(storage.append_history, skill_name, history_entry)
raise HTTPException(status_code=400, detail=f"Rollback blocked by security scanner: {scan.reason}")
storage.write_custom_skill(skill_name, SKILL_MD_FILE, target_content)
storage.append_history(skill_name, history_entry)
await asyncio.to_thread(storage.write_custom_skill, skill_name, SKILL_MD_FILE, target_content)
await asyncio.to_thread(storage.append_history, skill_name, history_entry)
await refresh_user_skills_system_prompt_cache_async(get_effective_user_id())
return await _read_custom_skill_response(skill_name, config)
except HTTPException:
@ -426,35 +428,47 @@ async def get_skill(skill_name: str, config: AppConfig = Depends(get_config)) ->
raise HTTPException(status_code=500, detail=f"Failed to get skill: {str(e)}")
def _write_extensions_skill_state(skill_name: str, enabled: bool) -> None:
def _write_extensions_skill_state(
storage: SkillStorage,
skill_name: str,
enabled: bool,
*,
rebuild_public_projection: bool,
) -> None:
"""Read-modify-write a skill's enabled state in the shared extensions_config.json.
Blocking filesystem IO: always call this via ``asyncio.to_thread``. It takes
``extensions_config_write_lock`` itself, so that this router and the MCP
router (which performs the same RMW on the same file) cannot interleave and
drop each other's change. The lock is held by the worker rather than by the
awaiting task, so cancelling the request cannot release it mid-write.
the public projection lock before ``extensions_config_write_lock``. The first
keeps the enabled-only view synchronized across workers; the second prevents
this router and the MCP router from interleaving writes to the shared file.
Both locks are held by the worker, so request cancellation cannot release
either lock while the write or projection rebuild is still running.
"""
with extensions_config_write_lock:
config_path = ExtensionsConfig.resolve_config_path()
if config_path is None:
config_path = Path.cwd().parent / "extensions_config.json"
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
from contextlib import nullcontext
# Work on a deep copy rather than the cached singleton: mutating the
# singleton in place would publish the new state to readers before it is
# durable on disk, and leave it applied even if the write below fails.
# to_file_dict() serializes the full extensions_config.json shape (all
# top-level keys), so no field is dropped from the file.
extensions_config = get_extensions_config().model_copy(deep=True)
extensions_config.skills[skill_name] = SkillStateConfig(enabled=enabled)
from deerflow.skills.projection import skill_projection_mutation
from deerflow.skills.storage.local_skill_storage import LocalSkillStorage
config_data = extensions_config.to_file_dict()
removal_names = (skill_name,) if not enabled else ()
projection_update = skill_projection_mutation(storage, "public", remove_names=removal_names) if rebuild_public_projection and isinstance(storage, LocalSkillStorage) else nullcontext()
with projection_update:
with extensions_config_write_lock:
config_path = ExtensionsConfig.resolve_config_path()
if config_path is None:
config_path = Path.cwd().parent / "extensions_config.json"
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
atomic_write_extensions_config(config_path, config_data)
# The projection lock is cross-process, but the singleton cache is
# not. Existing files are therefore re-read under the lock; a new
# file starts from a deep snapshot of the cached defaults.
extensions_config = ExtensionsConfig.from_file(config_path) if config_path.exists() else get_extensions_config().model_copy(deep=True)
extensions_config.skills[skill_name] = SkillStateConfig(enabled=enabled)
logger.info(f"Skills configuration updated and saved to: {config_path}")
reload_extensions_config()
config_data = extensions_config.to_file_dict()
atomic_write_extensions_config(config_path, config_data)
logger.info(f"Skills configuration updated and saved to: {config_path}")
reload_extensions_config()
@router.put(
@ -488,10 +502,13 @@ async def update_skill(skill_name: str, body: SkillUpdateRequest, request: Reque
# CUSTOM / LEGACY skills → per-user _skill_states.json (isolated state)
# so that two users with same-named custom skills can toggle independently.
if skill.category == SkillCategory.PUBLIC:
# Shared-file RMW. The worker takes extensions_config_write_lock for
# the whole read→write window, so it stays serialized against the MCP
# router even if this request is cancelled mid-write.
await asyncio.to_thread(_write_extensions_skill_state, skill_name, body.enabled)
await asyncio.to_thread(
_write_extensions_skill_state,
storage,
skill_name,
body.enabled,
rebuild_public_projection=True,
)
else:
# CUSTOM / LEGACY: write per-user state
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
@ -500,8 +517,15 @@ async def update_skill(skill_name: str, body: SkillUpdateRequest, request: Reque
await asyncio.to_thread(storage.set_skill_enabled_state, skill_name, body.enabled)
else:
# Fallback for non-user-scoped storage (unlikely in practice):
# same shared-file RMW as the PUBLIC branch, same lock.
await asyncio.to_thread(_write_extensions_skill_state, skill_name, body.enabled)
# same shared-file RMW as the PUBLIC branch, without a public
# projection rebuild for this non-public skill.
await asyncio.to_thread(
_write_extensions_skill_state,
storage,
skill_name,
body.enabled,
rebuild_public_projection=False,
)
# PUBLIC skill enabled state lives in the global extensions_config.json
# and affects every user, so the prompt cache for ALL users must be

View File

@ -1281,13 +1281,19 @@ class DeerFlowClient:
if config_path is None:
raise FileNotFoundError("Cannot locate extensions_config.json. Set DEER_FLOW_EXTENSIONS_CONFIG_PATH or ensure it exists in the project root.")
extensions_config = get_extensions_config()
extensions_config.skills[name] = SkillStateConfig(enabled=enabled)
from deerflow.skills.projection import skill_projection_mutation
config_data = extensions_config.to_file_dict()
removal_names = (name,) if not enabled else ()
with skill_projection_mutation(storage, "public", remove_names=removal_names):
# The projection lock is cross-process, but the singleton cache
# is not. Reload from disk under the lock before this RMW.
extensions_config = ExtensionsConfig.from_file(config_path)
extensions_config.skills[name] = SkillStateConfig(enabled=enabled)
self._atomic_write_json(config_path, config_data)
reload_extensions_config()
config_data = extensions_config.to_file_dict()
self._atomic_write_json(config_path, config_data)
reload_extensions_config()
else:
# CUSTOM / LEGACY: write per-user state
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage

View File

@ -44,7 +44,6 @@ from deerflow.integrations.lark_cli import LARK_CLI_SANDBOX_CONFIG_DIR, LARK_CLI
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.sandbox.sandbox import Sandbox
from deerflow.sandbox.sandbox_provider import SandboxProvider
from deerflow.skills.storage import user_should_see_legacy_skills
from .aio_sandbox import AioSandbox
from .backend import SandboxBackend, wait_for_sandbox_ready, wait_for_sandbox_ready_async
@ -868,41 +867,34 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
that ``Skill.get_container_path()`` category-aware paths resolve
correctly inside the sandbox.
Mount sources use ``DEER_FLOW_HOST_SKILLS_PATH`` and
``DEER_FLOW_HOST_BASE_DIR`` when running inside Docker (DooD) so the
host Docker daemon can resolve the paths.
Mount sources use ``DEER_FLOW_HOST_BASE_DIR`` when running inside
Docker (DooD) so the host Docker daemon can resolve the projection
paths.
"""
mounts: list[tuple[str, str, bool]] = []
try:
config = get_app_config()
skills_path = config.skills.get_skills_path()
container_path = config.skills.container_path
# When running inside Docker with DooD, use host-side skills path.
host_skills_root = os.environ.get("DEER_FLOW_HOST_SKILLS_PATH") or str(skills_path)
effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
AioSandboxProvider._ensure_skills_projection(effective_user_id)
paths = get_paths()
host_base_dir = str(paths.host_base_dir)
# 1. Public skills: global, read-only — static, shared by all threads
public_skills_path = skills_path / "public"
if public_skills_path.exists():
mounts.append(
(
join_host_path(host_skills_root, "public"),
f"{container_path}/public",
True,
)
mounts.append(
(
join_host_path(host_base_dir, "skills_view", "public"),
f"{container_path}/public",
True,
)
)
# 2. Per-user custom skills: read-only, per-thread/per-user
effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
paths = get_paths()
user_custom_path = paths.user_custom_skills_dir(effective_user_id)
user_custom_path.mkdir(parents=True, exist_ok=True)
host_user_custom = join_host_path(
str(paths.host_base_dir),
host_base_dir,
"users",
effective_user_id,
"skills",
"skills_view",
"custom",
)
mounts.append(
@ -913,38 +905,66 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
)
)
# 3. Legacy (pre-migration global-custom) skills: only mount for
# users who have no per-user custom skills yet, mirroring
# ``UserScopedSkillStorage._iter_skill_files`` visibility rule.
legacy_skills_path = skills_path / "custom"
if user_should_see_legacy_skills(effective_user_id, host_path=str(skills_path)) and legacy_skills_path.exists():
mounts.append(
(
join_host_path(host_skills_root, "custom"),
f"{container_path}/legacy",
True,
)
# 3. Legacy visibility is encoded by projection contents. Keep the
# mount stable even when the directory is empty so a later state
# change is visible without recreating the sandbox.
mounts.append(
(
join_host_path(host_base_dir, "users", effective_user_id, "skills_view", "legacy"),
f"{container_path}/legacy",
True,
)
)
except Exception as e:
logger.warning("Could not setup skills mounts: %s", e)
return mounts
@staticmethod
def _ensure_skills_projection(user_id: str):
"""Best-effort: a projection failure must not fail sandbox acquire.
Called directly (for its side effect) from ``_acquire_internal`` /
``_acquire_internal_async`` outside any try/except, as well as from
within ``_get_skills_mounts``'s own guarded block — swallowing here
keeps both call sites safe without duplicating the guard.
"""
from deerflow.skills.projection import ensure_skill_projections
from deerflow.skills.storage import get_or_new_user_skill_storage
try:
storage = get_or_new_user_skill_storage(user_id, app_config=get_app_config())
return ensure_skill_projections(storage)
except Exception as exc:
logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
return None
@staticmethod
def _get_user_skill_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]:
"""Mount managed integration skills into AIO sandboxes.
"""Mount enabled managed integration skills into AIO sandboxes.
Per-user custom skills are already mounted by ``_get_skills_mounts``.
This helper adds the shared integration skill root so sandbox paths match
the skill registry without duplicating ``/mnt/skills/custom``.
Integration packages are shared, but their enabled state is per-user, so
this helper mounts the user's projection instead of the raw shared root.
"""
try:
config = get_app_config()
paths = get_paths()
skills_container_path = config.skills.container_path
paths.integration_skills_dir().mkdir(parents=True, exist_ok=True)
effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
AioSandboxProvider._ensure_skills_projection(effective_user_id)
return [
(paths.host_integration_skills_dir(), f"{skills_container_path}/integrations", True),
(
join_host_path(
str(paths.host_base_dir),
"users",
effective_user_id,
"skills_view",
"integrations",
),
f"{skills_container_path}/integrations",
True,
),
]
except Exception as e:
logger.warning(f"Could not setup user skill mounts: {e}")
@ -1810,6 +1830,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
sandbox_id is deterministic from thread_id so no shared state file
is needed any process can derive the same container name)
"""
self._ensure_skills_projection(user_id)
cached_id = self._reuse_in_process_sandbox(thread_id, user_id=user_id)
if cached_id is not None:
return cached_id
@ -1837,6 +1858,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
async def _acquire_internal_async(self, thread_id: str | None, *, user_id: str) -> str:
"""Async counterpart to ``_acquire_internal``."""
await asyncio.to_thread(self._ensure_skills_projection, user_id)
cached_id = await asyncio.to_thread(self._reuse_in_process_sandbox, thread_id, user_id=user_id)
if cached_id is not None:
return cached_id

View File

@ -1092,7 +1092,7 @@ class E2BSandboxProvider(SandboxProvider):
# One-shot mount uploads. e2b has no host bind-mount, so we copy
# files from ``host_path`` into ``container_path`` at sandbox start.
try:
self._apply_mounts(client)
self._apply_mounts(client, user_id=user_id)
except Exception as e:
logger.warning("Failed to apply some mounts to e2b sandbox %s: %s", sandbox_id, e)
@ -1717,11 +1717,42 @@ class E2BSandboxProvider(SandboxProvider):
if exit_code not in (0, None) or "BOOTSTRAP_OK" not in stdout:
raise RuntimeError(f"e2b bootstrap script failed with exit code {exit_code}; stderr={stderr.strip()}")
def _apply_mounts(self, client: E2BClientSandbox) -> None:
mounts = self._config.get("mounts") or []
if not mounts:
return
for mount in mounts:
def _skill_projection_mounts(self, user_id: str) -> list[tuple[Path, str, bool]]:
"""Best-effort: a projection failure must not drop configured mounts too.
Unlike Local/AIO's ``_ensure_skills_projection``, this used to raise
straight out of ``_apply_mounts`` before the configured-mounts loop
ran, so a projection hiccup dropped the operator's own mounts as
collateral damage (only caught by ``create()``'s outer warning, with
no mounts applied at all). Swallowing here keeps the two mount
sources independent, matching the other two providers.
"""
from deerflow.skills.projection import ensure_skill_projections
from deerflow.skills.storage import get_or_new_user_skill_storage
try:
config = get_app_config()
storage = get_or_new_user_skill_storage(user_id, app_config=config)
projection = ensure_skill_projections(storage)
container_root = config.skills.container_path.rstrip("/")
return [
(projection.public, f"{container_root}/public", True),
(projection.custom, f"{container_root}/custom", True),
(projection.legacy, f"{container_root}/legacy", True),
(projection.integrations, f"{container_root}/integrations", True),
]
except Exception as exc:
logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
return []
def _apply_mounts(self, client: E2BClientSandbox, *, user_id: str | None = None) -> None:
effective_user_id = user_id or get_effective_user_id()
projection_mounts = self._skill_projection_mounts(effective_user_id)
configured_mounts = self._config.get("mounts") or []
skills_root = get_app_config().skills.container_path.rstrip("/")
mounts: list[tuple[Path, str, bool]] = list(projection_mounts)
for mount in configured_mounts:
try:
host_path = Path(getattr(mount, "host_path", "") or "")
container_path = (getattr(mount, "container_path", "") or "").rstrip("/")
@ -1731,6 +1762,12 @@ class E2BSandboxProvider(SandboxProvider):
container_path = (mount.get("container_path", "") or "").rstrip("/")
read_only = bool(mount.get("read_only", False))
if container_path == skills_root or container_path.startswith(skills_root + "/"):
logger.warning("Skipping e2b mount that conflicts with managed skills projection: %s", container_path)
continue
mounts.append((host_path, container_path, read_only))
for host_path, container_path, read_only in mounts:
if not host_path.exists():
logger.warning("Skipping e2b mount: host_path %s does not exist", host_path)
continue

View File

@ -258,6 +258,32 @@ class Paths:
"""
return self.base_dir / "integrations" / "skills"
@property
def skills_view_dir(self) -> Path:
"""Global sandbox-visible skills projection: ``{base_dir}/skills_view/``."""
return self.base_dir / "skills_view"
@property
def public_skills_view_dir(self) -> Path:
"""Enabled public skills exposed to sandboxes."""
return self.skills_view_dir / "public"
def user_skills_view_dir(self, user_id: str) -> Path:
"""Per-user sandbox-visible skills projection root."""
return self.user_dir(user_id) / "skills_view"
def user_custom_skills_view_dir(self, user_id: str) -> Path:
"""Enabled custom skills exposed to one user's sandboxes."""
return self.user_skills_view_dir(user_id) / "custom"
def user_legacy_skills_view_dir(self, user_id: str) -> Path:
"""Enabled legacy skills exposed to one user's sandboxes."""
return self.user_skills_view_dir(user_id) / "legacy"
def user_integration_skills_view_dir(self, user_id: str) -> Path:
"""Enabled managed integration skills exposed to one user's sandboxes."""
return self.user_skills_view_dir(user_id) / "integrations"
def thread_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
"""
Host path for a thread's data.

View File

@ -6,7 +6,6 @@ from pathlib import Path
from deerflow.sandbox.local.local_sandbox import LocalSandbox, PathMapping
from deerflow.sandbox.sandbox import Sandbox
from deerflow.sandbox.sandbox_provider import SandboxProvider
from deerflow.skills.storage import user_should_see_legacy_skills
logger = logging.getLogger(__name__)
@ -101,11 +100,11 @@ class LocalSandboxProvider(SandboxProvider):
from deerflow.config import get_app_config
config = get_app_config()
skills_path = config.skills.get_skills_path()
container_path = config.skills.container_path
projection = self._ensure_skills_projection()
# Public skills: global, read-only — static, shared by all threads
public_skills_path = skills_path / "public"
public_skills_path = projection.public
if public_skills_path.exists():
mappings.append(
PathMapping(
@ -217,6 +216,52 @@ class LocalSandboxProvider(SandboxProvider):
def _thread_key(thread_id: str, user_id: str) -> tuple[str, str]:
return (user_id, thread_id)
@staticmethod
def _ensure_skills_projection(user_id: str | None = None):
"""Best-effort: a projection failure must not fail sandbox acquire.
Mirrors the surrounding skill-mount setup, which has always logged
and continued rather than failing the whole acquire (e.g. missing
config.yaml in a test double). Callers see ``None`` and skip the
skill mounts for this acquire; the projection self-heals on a later
acquire once the underlying condition clears.
"""
from deerflow.config import get_app_config
from deerflow.skills.projection import ensure_skill_projections
from deerflow.skills.storage import get_or_new_skill_storage, get_or_new_user_skill_storage
try:
config = get_app_config()
if user_id is None:
storage = get_or_new_skill_storage(app_config=config)
else:
storage = get_or_new_user_skill_storage(user_id, app_config=config)
return ensure_skill_projections(storage)
except Exception as exc:
logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
return None
@staticmethod
def _append_public_skill_mapping(mappings: list[PathMapping], projection) -> None:
if projection is None:
return
try:
from deerflow.config import get_app_config
container_path = get_app_config().skills.container_path.rstrip("/")
public_container_path = f"{container_path}/public"
if any(mapping.container_path.rstrip("/") == public_container_path for mapping in mappings):
return
mappings.append(
PathMapping(
container_path=public_container_path,
local_path=str(projection.public),
read_only=True,
)
)
except Exception as exc:
logger.warning("Could not append public skill mapping: %s", exc, exc_info=True)
@staticmethod
def _sandbox_id_for_thread(thread_id: str, user_id: str) -> str:
return f"local:{user_id}:{thread_id}"
@ -232,7 +277,7 @@ class LocalSandboxProvider(SandboxProvider):
return (user_id, thread_id)
@staticmethod
def _build_thread_path_mappings(thread_id: str, *, user_id: str | None = None) -> list[PathMapping]:
def _build_thread_path_mappings(thread_id: str, *, user_id: str | None = None, skill_projection=None) -> list[PathMapping]:
"""Build per-thread path mappings for /mnt/user-data, /mnt/acp-workspace,
and /mnt/skills/custom.
@ -281,56 +326,35 @@ class LocalSandboxProvider(SandboxProvider):
),
]
# Per-user custom skills mount (read-only). This must be per-thread
# because ``/mnt/skills/custom`` resolves to different host directories
# for different users.
# Per-user category mounts stay present for the sandbox lifetime. Their
# enabled-only contents change beneath these stable roots.
try:
config = get_app_config()
skills_container_path = config.skills.container_path
user_custom_path = paths.user_custom_skills_dir(effective_user_id)
integrations_path = paths.integration_skills_dir()
user_custom_path.mkdir(parents=True, exist_ok=True)
integrations_path.mkdir(parents=True, exist_ok=True)
projection = skill_projection if skill_projection is not None else LocalSandboxProvider._ensure_skills_projection(effective_user_id)
mappings.append(
PathMapping(
container_path=f"{skills_container_path}/custom",
local_path=str(user_custom_path),
read_only=True,
)
)
mappings.append(
PathMapping(
container_path=f"{skills_container_path}/integrations",
local_path=str(integrations_path),
read_only=True,
)
)
except Exception as exc:
logger.warning("Could not setup per-thread custom skills mount: %s", exc, exc_info=True)
# Legacy (pre-migration global-custom) skills: only mount for users
# who have no per-user custom skills yet, mirroring the
# ``UserScopedSkillStorage._iter_skill_files`` visibility rule. Users
# with their own per-user custom skills cannot see LEGACY in the
# listing/prompt and must not be able to read it via the sandbox
# either — otherwise the listing layer and the sandbox layer disagree
# about visibility, and the sandbox layer is the more permissive one.
try:
config = get_app_config()
skills_container_path = config.skills.container_path
user_custom_path = paths.user_custom_skills_dir(effective_user_id)
legacy_skills_path = config.skills.get_skills_path() / "custom"
if user_should_see_legacy_skills(effective_user_id, host_path=str(config.skills.get_skills_path())) and legacy_skills_path.exists():
mappings.append(
PathMapping(
container_path=f"{skills_container_path}/legacy",
local_path=str(legacy_skills_path),
read_only=True,
)
if projection is not None:
mappings.extend(
[
PathMapping(
container_path=f"{skills_container_path}/custom",
local_path=str(projection.custom),
read_only=True,
),
PathMapping(
container_path=f"{skills_container_path}/legacy",
local_path=str(projection.legacy),
read_only=True,
),
PathMapping(
container_path=f"{skills_container_path}/integrations",
local_path=str(projection.integrations),
read_only=True,
),
]
)
except Exception as exc:
logger.warning("Could not setup per-thread legacy skills mount: %s", exc, exc_info=True)
logger.warning("Could not setup per-thread skills projection mounts: %s", exc, exc_info=True)
return mappings
@ -350,13 +374,23 @@ class LocalSandboxProvider(SandboxProvider):
global _singleton
if thread_id is None:
skill_projection = self._ensure_skills_projection()
with self._lock:
if self._generic_sandbox is None:
self._generic_sandbox = LocalSandbox("local", path_mappings=list(self._path_mappings))
mappings = list(self._path_mappings)
self._append_public_skill_mapping(mappings, skill_projection)
self._generic_sandbox = LocalSandbox("local", path_mappings=mappings)
_singleton = self._generic_sandbox
return self._generic_sandbox.id
effective_user_id = self._effective_acquire_user_id(user_id)
# Runs on every acquire, including cache hits, to self-heal drift —
# cheap (~3-4 ms metadata walk) when the manifest is fresh. If another
# worker mutated this user's skills since the last check, this
# triggers a full rebuild (~400 ms measured locally) under the
# cross-process projection lock, serializing concurrent acquires and
# mutations for that user. Acceptable for an editing-frequency event.
skill_projection = self._ensure_skills_projection(effective_user_id)
key = self._thread_key(thread_id, effective_user_id)
# Fast path under lock.
@ -366,11 +400,18 @@ class LocalSandboxProvider(SandboxProvider):
# Mark as most-recently used so frequently-touched threads
# survive eviction.
self._thread_sandboxes.move_to_end(key)
return cached.id
if cached is not None:
return cached.id
# ``_build_thread_path_mappings`` touches the filesystem
# (``ensure_thread_dirs``); release the lock during I/O.
new_mappings = list(self._path_mappings) + self._build_thread_path_mappings(thread_id, user_id=effective_user_id)
new_mappings = list(self._path_mappings)
self._append_public_skill_mapping(new_mappings, skill_projection)
new_mappings += self._build_thread_path_mappings(
thread_id,
user_id=effective_user_id,
skill_projection=skill_projection,
)
with self._lock:
# Re-check after the lock-free I/O: another caller may have

View File

@ -0,0 +1,552 @@
"""Materialize enabled-only skill trees for sandbox filesystem exposure."""
from __future__ import annotations
import errno
import hashlib
import json
import logging
import os
import shutil
import tempfile
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING
from deerflow.skills.parser import parse_skill_file
from deerflow.skills.types import SKILL_MD_FILE, Skill, SkillCategory
if TYPE_CHECKING:
from deerflow.skills.storage.skill_storage import SkillStorage
logger = logging.getLogger(__name__)
try:
import fcntl
except ImportError: # pragma: no cover - Windows
fcntl = None # type: ignore[assignment]
import msvcrt
_locks_guard = threading.Lock()
_process_locks: dict[Path, threading.RLock] = {}
_MANIFEST_VERSION = 1
_MAX_REBUILD_ATTEMPTS = 2
@dataclass(frozen=True)
class SkillProjectionPaths:
"""Stable category roots mounted or uploaded by sandbox providers."""
public: Path
custom: Path
legacy: Path
integrations: Path
def get_skill_projection_paths(storage: SkillStorage) -> SkillProjectionPaths:
from deerflow.config.paths import get_paths
paths = getattr(storage, "_paths", None) or get_paths()
user_id = getattr(storage, "user_id", None)
if user_id is None:
return SkillProjectionPaths(
public=paths.public_skills_view_dir,
custom=paths.skills_view_dir / "custom",
legacy=paths.skills_view_dir / "legacy",
integrations=paths.skills_view_dir / "integrations",
)
return SkillProjectionPaths(
public=paths.public_skills_view_dir,
custom=paths.user_custom_skills_view_dir(user_id),
legacy=paths.user_legacy_skills_view_dir(user_id),
integrations=paths.user_integration_skills_view_dir(user_id),
)
def _lock_for(path: Path) -> threading.RLock:
resolved = path.resolve()
with _locks_guard:
return _process_locks.setdefault(resolved, threading.RLock())
@contextmanager
def _projection_lock(root: Path) -> Iterator[None]:
"""Serialize projection replacement in-process and across POSIX workers."""
lock_path = root.parent / f".{root.name}.projection.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
process_lock = _lock_for(lock_path)
with process_lock, lock_path.open("a", encoding="utf-8") as lock_file:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_EX)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_LOCK, 1)
try:
yield
finally:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_UN)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
def _link_or_copy(source: str, target: str, *, follow_symlinks: bool = True) -> str:
# Hardlinks share the source inode and provide no write isolation. Any
# read-only guarantee must come from the consuming sandbox or mount.
try:
os.link(source, target, follow_symlinks=follow_symlinks)
except OSError as exc:
if exc.errno not in {errno.EXDEV, errno.EPERM, errno.EACCES, errno.ENOTSUP}:
raise
shutil.copy2(source, target, follow_symlinks=follow_symlinks)
return target
def _stage_skill(source: Path, target: Path, nested_skill_roots: set[Path]) -> None:
def _exclude_nested_skills(current: str, names: list[str]) -> list[str]:
relative_root = Path(current).relative_to(source)
return [name for name in names if relative_root / name in nested_skill_roots]
shutil.copytree(
source,
target,
copy_function=_link_or_copy,
symlinks=True,
ignore=_exclude_nested_skills,
dirs_exist_ok=True,
)
def _path_kind(path: Path) -> str:
if path.is_symlink():
return "symlink"
if path.is_dir():
return "directory"
return "file"
def _tree_entries(root: Path) -> dict[Path, str]:
entries: dict[Path, str] = {}
for current_root, dir_names, file_names in os.walk(root, followlinks=False):
current = Path(current_root)
for name in dir_names:
path = current / name
entries[path.relative_to(root)] = _path_kind(path)
dir_names[:] = [name for name in dir_names if not (current / name).is_symlink()]
for name in file_names:
path = current / name
entries[path.relative_to(root)] = _path_kind(path)
return entries
def _remove_projection_entry(path: Path) -> None:
if path.is_dir() and not path.is_symlink():
shutil.rmtree(path)
else:
path.unlink(missing_ok=True)
def _validate_projection_relative_path(relative_path: Path) -> None:
if relative_path.is_absolute() or not relative_path.parts or any(part in {"", ".", ".."} for part in relative_path.parts):
raise ValueError("Projection removal path must identify a package within its category root")
def _remove_projection_relative(root: Path, relative_path: Path) -> None:
"""Remove a projected package without following a drifted namespace symlink."""
current = root
for part in relative_path.parts:
current /= part
if current.is_symlink():
current.unlink()
return
_remove_projection_entry(current)
def _sync_staged_category(root: Path, staging: Path) -> None:
desired = _tree_entries(staging)
live = _tree_entries(root)
for relative_path, live_kind in sorted(live.items(), key=lambda item: len(item[0].parts), reverse=True):
if desired.get(relative_path) != live_kind:
_remove_projection_entry(root / relative_path)
for relative_path, kind in sorted(desired.items(), key=lambda item: len(item[0].parts)):
if kind == "directory":
(root / relative_path).mkdir(parents=True, exist_ok=True)
for relative_path, kind in desired.items():
if kind == "directory":
continue
target = root / relative_path
target.parent.mkdir(parents=True, exist_ok=True)
(staging / relative_path).replace(target)
def _replace_category(root: Path, desired: dict[Path, Skill], skill_boundaries: set[Path]) -> None:
"""Reconcile entries beneath a stable category root without blanking it."""
root.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(prefix=f".{root.name}.projection-", dir=root.parent) as staging_dir:
staging = Path(staging_dir)
for relative_path, skill in desired.items():
nested_roots = {boundary.relative_to(relative_path) for boundary in skill_boundaries if boundary != relative_path and boundary.is_relative_to(relative_path)}
_stage_skill(skill.skill_dir, staging / relative_path, nested_roots)
_sync_staged_category(root, staging)
def _clear_category(root: Path) -> None:
root.mkdir(parents=True, exist_ok=True)
for path in root.iterdir():
if path.is_dir() and not path.is_symlink():
shutil.rmtree(path)
else:
path.unlink()
def _clear_projection_scope(scope_root: Path, *category_roots: Path) -> None:
for category_root in category_roots:
_clear_category(category_root)
_manifest_path(scope_root).unlink(missing_ok=True)
def _update_tree_digest(digest, root: Path, label: str) -> None:
"""Hash directory metadata (inode/mode/size/mtime), not file contents.
Trade-off: fast enough to run on every sandbox acquire (O(files), no
reads), but an external edit that preserves inode+size+mtime unlikely,
not zero-probability is invisible to this signature and leaves the
projection stale until the next explicit rebuild. Runtime writes through
this codebase are covered regardless: the mutation path rebuilds under
lock, and atomic-rename always changes the inode.
"""
digest.update(f"root:{label}\0".encode())
if not root.exists():
digest.update(b"absent\0")
return
stack = [(root, Path("."))]
while stack:
current, relative_root = stack.pop()
with os.scandir(current) as entries:
ordered = sorted(entries, key=lambda entry: entry.name)
child_dirs: list[tuple[Path, Path]] = []
for entry in ordered:
relative = relative_root / entry.name
metadata = entry.stat(follow_symlinks=False)
if entry.is_symlink():
kind = "link"
elif entry.is_dir(follow_symlinks=False):
kind = "dir"
child_dirs.append((Path(entry.path), relative))
else:
kind = "file"
digest.update((f"{label}:{relative.as_posix()}:{kind}:{metadata.st_ino}:{metadata.st_mode}:{metadata.st_size}:{metadata.st_mtime_ns}\0").encode())
stack.extend(reversed(child_dirs))
def _extensions_state() -> dict:
from deerflow.config.extensions_config import ExtensionsConfig
config = ExtensionsConfig.from_file()
return {name: state.model_dump(mode="json") for name, state in config.skills.items()}
def _source_signature(storage: SkillStorage, scope: str) -> str:
digest = hashlib.sha256()
host_root = storage.get_skills_root_path()
if scope == "public":
_update_tree_digest(digest, host_root / SkillCategory.PUBLIC.value, "public")
state = {"extensions": _extensions_state()}
elif scope == "user":
user_custom_root = storage.get_user_custom_root()
integration_root = storage.get_user_integrations_root()
_update_tree_digest(digest, user_custom_root, "custom")
_update_tree_digest(digest, host_root / SkillCategory.CUSTOM.value, "legacy")
_update_tree_digest(digest, integration_root, "integrations")
# CUSTOM/LEGACY/INTEGRATION visibility is the intersection of the
# per-user state and the global extensions default, so both belong in
# this signature.
state = {
"extensions": _extensions_state(),
"user": storage._read_skill_states(),
}
else: # pragma: no cover - internal invariant
raise ValueError(f"Unknown skill projection scope: {scope}")
digest.update(json.dumps(state, sort_keys=True, separators=(",", ":")).encode())
return digest.hexdigest()
def _manifest_path(scope_root: Path) -> Path:
return scope_root / ".projection-manifest.json"
def _read_manifest(scope_root: Path) -> dict | None:
try:
value = json.loads(_manifest_path(scope_root).read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
return value if isinstance(value, dict) else None
def _write_manifest(scope_root: Path, source_signature: str) -> None:
scope_root.mkdir(parents=True, exist_ok=True)
target = _manifest_path(scope_root)
fd, temporary_name = tempfile.mkstemp(prefix=".projection-manifest-", suffix=".tmp", dir=scope_root)
temporary = Path(temporary_name)
try:
with os.fdopen(fd, "w", encoding="utf-8") as stream:
json.dump({"version": _MANIFEST_VERSION, "source_signature": source_signature}, stream, sort_keys=True)
temporary.replace(target)
except Exception:
temporary.unlink(missing_ok=True)
raise
def _load_public_skills(storage: SkillStorage, *, enabled_only: bool) -> list[Skill]:
from deerflow.config.extensions_config import ExtensionsConfig
public_root = storage.get_skills_root_path() / SkillCategory.PUBLIC.value
if not public_root.is_dir():
return []
extensions = ExtensionsConfig.from_file()
skills: list[Skill] = []
for current_root, dir_names, file_names in os.walk(public_root, followlinks=True):
dir_names[:] = sorted(name for name in dir_names if not name.startswith("."))
if SKILL_MD_FILE not in file_names:
continue
# Match the runtime loader: nested SKILL.md files inside a package are
# support data, not independently configurable skills.
dir_names.clear()
skill_file = Path(current_root) / SKILL_MD_FILE
skill = parse_skill_file(
skill_file,
category=SkillCategory.PUBLIC,
relative_path=skill_file.parent.relative_to(public_root),
)
if skill is None:
continue
enabled = extensions.is_skill_enabled(skill.name, SkillCategory.PUBLIC.value)
if not enabled_only or enabled:
skills.append(skill)
return skills
def _by_relative_path(skills: list[Skill], category: SkillCategory) -> dict[Path, Skill]:
return {skill.relative_path: skill for skill in skills if skill.category == category}
def _category_boundaries(skills: list[Skill], category: SkillCategory) -> set[Path]:
return {skill.relative_path for skill in skills if skill.category == category}
def _rebuild_public_locked(storage: SkillStorage, paths: SkillProjectionPaths) -> None:
scope_root = paths.public.parent
try:
for _attempt in range(_MAX_REBUILD_ATTEMPTS):
before = _source_signature(storage, "public")
all_public_skills = _load_public_skills(storage, enabled_only=False)
enabled_public_skills = _load_public_skills(storage, enabled_only=True)
_replace_category(
paths.public,
_by_relative_path(enabled_public_skills, SkillCategory.PUBLIC),
_category_boundaries(all_public_skills, SkillCategory.PUBLIC),
)
after = _source_signature(storage, "public")
if before == after:
_write_manifest(scope_root, after)
return
raise RuntimeError("Public skills changed repeatedly while rebuilding the sandbox projection")
except Exception:
_clear_projection_scope(scope_root, paths.public)
raise
def _rebuild_user_locked(storage: SkillStorage, paths: SkillProjectionPaths) -> None:
scope_root = paths.custom.parent
try:
for _attempt in range(_MAX_REBUILD_ATTEMPTS):
before = _source_signature(storage, "user")
all_user_skills = storage.load_skills(enabled_only=False)
enabled_user_skills = [skill for skill in all_user_skills if skill.enabled]
_replace_category(
paths.custom,
_by_relative_path(enabled_user_skills, SkillCategory.CUSTOM),
_category_boundaries(all_user_skills, SkillCategory.CUSTOM),
)
_replace_category(
paths.legacy,
_by_relative_path(enabled_user_skills, SkillCategory.LEGACY),
_category_boundaries(all_user_skills, SkillCategory.LEGACY),
)
_replace_category(
paths.integrations,
_by_relative_path(enabled_user_skills, SkillCategory.INTEGRATION),
_category_boundaries(all_user_skills, SkillCategory.INTEGRATION),
)
after = _source_signature(storage, "user")
if before == after:
_write_manifest(scope_root, after)
return
raise RuntimeError("User skills changed repeatedly while rebuilding the sandbox projection")
except Exception:
_clear_projection_scope(scope_root, paths.custom, paths.legacy, paths.integrations)
raise
def rebuild_skill_projections(
storage: SkillStorage,
*,
include_public: bool = True,
include_user: bool = True,
) -> SkillProjectionPaths:
"""Rebuild enabled-only projection scopes visible through ``storage``."""
paths = get_skill_projection_paths(storage)
user_id = getattr(storage, "user_id", None)
if include_public:
with _projection_lock(paths.public.parent):
_rebuild_public_locked(storage, paths)
if include_user and user_id is not None:
with _projection_lock(paths.custom.parent):
_rebuild_user_locked(storage, paths)
return paths
def _public_projection_is_fresh(storage: SkillStorage, paths: SkillProjectionPaths) -> bool:
if not paths.public.is_dir():
return False
manifest_before = _read_manifest(paths.public.parent)
if manifest_before is None or manifest_before.get("version") != _MANIFEST_VERSION:
return False
signature = _source_signature(storage, "public")
manifest_after = _read_manifest(paths.public.parent)
return manifest_before == manifest_after and manifest_before.get("source_signature") == signature
def ensure_skill_projections(storage: SkillStorage) -> SkillProjectionPaths:
"""Repair stale projection scopes, otherwise leave their inodes untouched."""
paths = get_skill_projection_paths(storage)
try:
public_is_fresh = _public_projection_is_fresh(storage, paths)
except Exception:
# Re-check under the mutation lock before failing closed. A concurrent
# writer may have exposed a transient source/manifest state.
public_is_fresh = False
if not public_is_fresh:
with _projection_lock(paths.public.parent):
try:
if not _public_projection_is_fresh(storage, paths):
_rebuild_public_locked(storage, paths)
except Exception:
_clear_projection_scope(paths.public.parent, paths.public)
raise
if getattr(storage, "user_id", None) is not None:
with _projection_lock(paths.custom.parent):
try:
manifest = _read_manifest(paths.custom.parent)
signature = _source_signature(storage, "user")
if not paths.custom.is_dir() or not paths.legacy.is_dir() or not paths.integrations.is_dir() or manifest is None or manifest.get("version") != _MANIFEST_VERSION or manifest.get("source_signature") != signature:
_rebuild_user_locked(storage, paths)
except Exception:
_clear_projection_scope(paths.custom.parent, paths.custom, paths.legacy, paths.integrations)
raise
return paths
@contextmanager
def skill_projection_mutation(
storage: SkillStorage,
scope: str,
*,
remove: tuple[tuple[SkillCategory, Path], ...] = (),
remove_names: tuple[str, ...] = (),
) -> Iterator[None]:
"""Hold a projection scope lock across a source/state mutation."""
if not isinstance(storage.get_skills_root_path(), Path):
# Lightweight unit-test doubles sometimes return MagicMock here. The
# SkillStorage contract requires a Path; real storage implementations
# therefore never take this compatibility branch.
yield
return
paths = get_skill_projection_paths(storage)
if scope == "public":
scope_root = paths.public.parent
category_roots = {SkillCategory.PUBLIC: paths.public}
def rebuild() -> None:
_rebuild_public_locked(storage, paths)
elif scope == "user":
scope_root = paths.custom.parent
category_roots = {
SkillCategory.CUSTOM: paths.custom,
SkillCategory.LEGACY: paths.legacy,
SkillCategory.INTEGRATION: paths.integrations,
}
def rebuild() -> None:
_rebuild_user_locked(storage, paths)
else:
raise ValueError(f"Unknown skill projection scope: {scope}")
removals: set[tuple[Path, Path]] = set()
for category, relative_path in remove:
root = category_roots.get(category)
if root is None:
raise ValueError(f"Skill category {category.value!r} does not belong to projection scope {scope!r}")
_validate_projection_relative_path(relative_path)
removals.add((root, relative_path))
with _projection_lock(scope_root):
if remove_names:
names = set(remove_names)
skills = _load_public_skills(storage, enabled_only=False) if scope == "public" else storage.load_skills(enabled_only=False)
for skill in skills:
root = category_roots.get(skill.category)
if skill.name not in names or root is None:
continue
_validate_projection_relative_path(skill.relative_path)
removals.add((root, skill.relative_path))
try:
_manifest_path(scope_root).unlink(missing_ok=True)
for root, relative_path in removals:
_remove_projection_relative(root, relative_path)
yield
rebuild()
except Exception:
_clear_projection_scope(scope_root, *category_roots.values())
raise
def ensure_public_skill_projection(*, app_config=None) -> bool:
"""Ensure the global public view during boot without scanning user data.
User projections are repaired lazily by sandbox acquire. Eagerly rebuilding
every historical user would make gateway readiness scale with tenant count,
while providing no additional safety before that user's next acquire.
"""
from deerflow.config import get_app_config
from deerflow.config.paths import get_paths
from deerflow.skills.storage import get_or_new_skill_storage
try:
config = app_config or get_app_config()
public_storage = get_or_new_skill_storage(app_config=config)
ensure_skill_projections(public_storage)
except Exception:
logger.warning("Failed to ensure the public skill projection during boot; clearing it until a sandbox acquire self-heals it", exc_info=True)
try:
paths = get_paths()
with _projection_lock(paths.public_skills_view_dir.parent):
_clear_projection_scope(paths.public_skills_view_dir.parent, paths.public_skills_view_dir)
except Exception:
logger.error("Failed to clear the public skill projection after a boot-time error", exc_info=True)
return False
return True

View File

@ -10,6 +10,7 @@ import os
import shutil
import tempfile
from collections.abc import Iterable
from contextlib import nullcontext
from datetime import UTC, datetime
from pathlib import Path
@ -107,8 +108,18 @@ class LocalSkillStorage(SkillStorage):
) as tmp_file:
tmp_file.write(content)
tmp_path = Path(tmp_file.name)
tmp_path.replace(target)
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
try:
with self._skill_projection_mutation():
tmp_path.replace(target)
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
except Exception:
tmp_path.unlink(missing_ok=True)
raise
def remove_custom_skill_file(self, name: str, relative_path: str) -> str:
removal = ((SkillCategory.CUSTOM, Path(name)),)
with self._skill_projection_mutation(remove=removal):
return super().remove_custom_skill_file(name, relative_path)
async def ainstall_skill_from_archive(self, archive_path: str | Path) -> dict:
from deerflow.skills.installer import _scan_skill_archive_contents_or_raise
@ -200,11 +211,12 @@ class LocalSkillStorage(SkillStorage):
"""Stage and move the validated skill into place (blocking; runs off the event loop)."""
from deerflow.skills.installer import _move_staged_skill_into_reserved_target
with tempfile.TemporaryDirectory(prefix=f".installing-{skill_name}-", dir=custom_dir) as staging_root:
staging_target = Path(staging_root) / skill_name
shutil.copytree(skill_dir, staging_target)
_move_staged_skill_into_reserved_target(staging_target, target)
make_skill_written_path_sandbox_readable(custom_dir, target)
with self._skill_projection_mutation():
with tempfile.TemporaryDirectory(prefix=f".installing-{skill_name}-", dir=custom_dir) as staging_root:
staging_target = Path(staging_root) / skill_name
shutil.copytree(skill_dir, staging_target)
_move_staged_skill_into_reserved_target(staging_target, target)
make_skill_written_path_sandbox_readable(custom_dir, target)
def delete_custom_skill(self, name: str, *, history_meta: dict | None = None) -> None:
self.validate_skill_name(name)
@ -222,8 +234,22 @@ class LocalSkillStorage(SkillStorage):
name,
e,
)
if target.exists():
shutil.rmtree(target)
removal = ((SkillCategory.CUSTOM, Path(name)),)
with self._skill_projection_mutation(remove=removal):
if target.exists():
shutil.rmtree(target)
def _skill_projection_mutation(
self,
*,
remove: tuple[tuple[SkillCategory, Path], ...] = (),
remove_names: tuple[str, ...] = (),
):
if getattr(self, "user_id", None) is None:
return nullcontext()
from deerflow.skills.projection import skill_projection_mutation
return skill_projection_mutation(self, "user", remove=remove, remove_names=remove_names)
def append_history(self, name: str, record: dict) -> None:
self.validate_skill_name(name)

View File

@ -155,6 +155,15 @@ class SkillStorage(ABC):
Origin: ``deerflow.skills.manager.atomic_write``.
"""
def remove_custom_skill_file(self, name: str, relative_path: str) -> str:
"""Remove a supporting file and return its previous text content."""
target = self.ensure_safe_support_path(name, relative_path)
if not target.exists():
raise FileNotFoundError(f"Supporting file '{relative_path}' not found for skill '{name}'.")
previous_content = target.read_text(encoding="utf-8")
target.unlink()
return previous_content
@abstractmethod
async def ainstall_skill_from_archive(self, archive_path: str | Path) -> dict:
"""Async install of a skill from a ``.skill`` ZIP archive.

View File

@ -82,6 +82,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
self._user_id = _validate_user_id(user_id)
paths = get_paths()
self._paths = paths
self._user_custom_root: Path = paths.user_custom_skills_dir(self._user_id)
self._integrations_root: Path = paths.integration_skills_dir()
self._user_skills_root: Path = paths.user_skills_dir(self._user_id)
@ -153,9 +154,11 @@ class UserScopedSkillStorage(LocalSkillStorage):
def set_skill_enabled_state(self, skill_name: str, enabled: bool) -> None:
"""Set the enabled state for a custom/legacy skill and persist."""
states = self._read_skill_states()
states[skill_name] = {"enabled": enabled}
self._write_skill_states(states)
removal_names = (skill_name,) if not enabled else ()
with self._skill_projection_mutation(remove_names=removal_names):
states = self._read_skill_states()
states[skill_name] = {"enabled": enabled}
self._write_skill_states(states)
# ------------------------------------------------------------------
# Path helpers — redirect custom skill paths to user directory
@ -204,10 +207,12 @@ class UserScopedSkillStorage(LocalSkillStorage):
# being silently re-enabled by an absent per-user entry, while still
# letting the per-user state override the global default when both
# are present. PUBLIC skill state remains governed solely by
# extensions_config (handled by ``super().load_skills`` above).
from deerflow.config.extensions_config import get_extensions_config
# extensions_config (handled by ``super().load_skills`` above). Re-read
# from disk here too so another worker's update cannot be masked by
# this process's singleton cache while rebuilding a user projection.
from deerflow.config.extensions_config import ExtensionsConfig
extensions_config = get_extensions_config()
extensions_config = ExtensionsConfig.from_file()
skills = [
dataclasses.replace(s, enabled=self.get_skill_enabled_state(s.name) and extensions_config.is_skill_enabled(s.name, s.category.value if hasattr(s.category, "value") else s.category))
if dataclasses.is_dataclass(s) and not isinstance(s, type) and (s.category.value if hasattr(s.category, "value") else s.category) != SkillCategory.PUBLIC.value
@ -369,8 +374,13 @@ class UserScopedSkillStorage(LocalSkillStorage):
) as tmp_file:
tmp_file.write(content)
tmp_path = Path(tmp_file.name)
tmp_path.replace(target)
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
try:
with self._skill_projection_mutation():
tmp_path.replace(target)
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
except Exception:
tmp_path.unlink(missing_ok=True)
raise
# ------------------------------------------------------------------
# Public helpers

View File

@ -242,11 +242,7 @@ async def _skill_manage_impl(
await _to_thread(skill_storage.ensure_custom_skill_is_editable, name)
if path is None:
raise ValueError("path is required for remove_file.")
target = await _to_thread(skill_storage.ensure_safe_support_path, name, path)
if not await _to_thread(target.exists):
raise FileNotFoundError(f"Supporting file '{path}' not found for skill '{name}'.")
prev_content = await _to_thread(target.read_text, encoding="utf-8")
await _to_thread(target.unlink)
prev_content = await _to_thread(skill_storage.remove_custom_skill_file, name, path)
await _to_thread(
skill_storage.append_history,
name,

View File

@ -119,6 +119,22 @@ async def test_update_skill_writes_from_snapshot_without_mutating_singleton(tmp_
assert "middlewares" in written
async def test_update_skill_persists_state_when_source_omits_skills(tmp_path: Path, monkeypatch) -> None:
config_path = tmp_path / "extensions_config.json"
await asyncio.to_thread(
config_path.write_text,
json.dumps({"mcpServers": {}, "middlewares": ["pkg:Middleware"]}),
encoding="utf-8",
)
_patch_config_infra(monkeypatch, config_path)
await update_skill("demo-skill", SkillUpdateRequest(enabled=False), _admin_request(), SimpleNamespace())
written = json.loads(await asyncio.to_thread(config_path.read_text, encoding="utf-8"))
assert written["skills"] == {"demo-skill": {"enabled": False}}
assert written["middlewares"] == ["pkg:Middleware"]
@pytest.mark.allow_blocking_io # gate-exempt: needs real worker-thread overlap to observe serialization
async def test_update_skill_serializes_concurrent_writes(tmp_path: Path, monkeypatch) -> None:
state_lock = threading.Lock()

View File

@ -219,8 +219,9 @@ def test_get_user_skill_mounts_mounts_only_global_integrations(tmp_path, monkeyp
assert set(alice) == {"/mnt/skills/integrations"}
assert set(bob) == {"/mnt/skills/integrations"}
assert alice["/mnt/skills/integrations"] == bob["/mnt/skills/integrations"]
assert alice["/mnt/skills/integrations"] == str(tmp_path / "home" / "integrations" / "skills")
assert alice["/mnt/skills/integrations"] != bob["/mnt/skills/integrations"]
assert alice["/mnt/skills/integrations"] == str(tmp_path / "home" / "users" / "alice" / "skills_view" / "integrations")
assert bob["/mnt/skills/integrations"] == str(tmp_path / "home" / "users" / "bob" / "skills_view" / "integrations")
def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_path, monkeypatch, provisioner_module):
@ -243,7 +244,7 @@ def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_pat
monkeypatch.setattr(aio_mod, "get_app_config", lambda: config)
monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=home))
monkeypatch.setattr(aio_mod, "get_effective_user_id", lambda: "default")
monkeypatch.setattr(aio_mod, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False)
monkeypatch.setattr(remote_backend, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False)
provider = _make_provider(tmp_path)
mounts = provider._get_extra_mounts("thread-1", user_id="alice")
@ -526,7 +527,10 @@ async def test_acquire_internal_async_offloads_cached_reuse_health_check(tmp_pat
sandbox_id = await provider._acquire_internal_async("thread-cached-async", user_id="default")
assert sandbox_id == "sandbox-cached-async"
assert to_thread_calls == [(provider._reuse_in_process_sandbox, ("thread-cached-async",))]
assert to_thread_calls == [
(provider._ensure_skills_projection, ("default",)),
(provider._reuse_in_process_sandbox, ("thread-cached-async",)),
]
def test_remote_backend_create_forwards_effective_user_id(monkeypatch):
@ -548,7 +552,7 @@ def test_remote_backend_create_forwards_effective_user_id(monkeypatch):
return _Response()
monkeypatch.setattr(remote_mod.requests, "post", _post)
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda user_id: True)
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda _user_id: True)
try:
backend.create("thread-42", "sandbox-42")
@ -585,7 +589,7 @@ def test_remote_backend_create_prefers_explicit_user_id(monkeypatch):
monkeypatch.setattr(remote_mod.requests, "post", _post)
monkeypatch.setattr(remote_mod, "get_effective_user_id", lambda: "default")
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda user_id: False)
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda _user_id: False)
backend.create("thread-42", "sandbox-42", user_id="ou-user")

View File

@ -1861,10 +1861,8 @@ class TestSkillsManagement:
skill = self._make_skill(enabled=True)
updated_skill = self._make_skill(enabled=False)
ext_config = ExtensionsConfig()
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
json.dump({}, f)
json.dump({"mcpServers": {}, "skills": {"untouched-skill": {"enabled": False}}}, f)
tmp_path = Path(f.name)
try:
@ -1881,12 +1879,37 @@ class TestSkillsManagement:
side_effect=[[skill], [skill], [updated_skill], [updated_skill]],
),
patch("deerflow.client.ExtensionsConfig.resolve_config_path", return_value=tmp_path),
patch("deerflow.client.get_extensions_config", return_value=ext_config),
patch("deerflow.client.reload_extensions_config"),
):
result = client.update_skill("test-skill", enabled=False)
assert result["enabled"] is False
assert client._agent is None # M2: agent invalidated
persisted = json.loads(tmp_path.read_text(encoding="utf-8"))
assert persisted["skills"]["untouched-skill"] == {"enabled": False}
finally:
tmp_path.unlink()
def test_update_skill_persists_state_when_source_omits_skills(self, client):
skill = self._make_skill(enabled=True)
updated_skill = self._make_skill(enabled=False)
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
json.dump({"mcpServers": {}}, f)
tmp_path = Path(f.name)
try:
with (
patch(
"deerflow.skills.storage.local_skill_storage.LocalSkillStorage.load_skills",
side_effect=[[skill], [skill], [updated_skill], [updated_skill]],
),
patch("deerflow.client.ExtensionsConfig.resolve_config_path", return_value=tmp_path),
patch("deerflow.client.reload_extensions_config"),
):
client.update_skill("test-skill", enabled=False)
persisted = json.loads(tmp_path.read_text(encoding="utf-8"))
assert persisted["skills"] == {"test-skill": {"enabled": False}}
finally:
tmp_path.unlink()

View File

@ -10,6 +10,7 @@ import threading
import time
from collections import OrderedDict
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from types import SimpleNamespace
from typing import Any
from unittest.mock import MagicMock
@ -313,6 +314,100 @@ def _install_fake_sdk(monkeypatch, provider) -> FakeSandboxClass:
return fake_cls
def _write_skill(root: Path, name: str) -> None:
target = root / name / "SKILL.md"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(f"---\nname: {name}\ndescription: test\n---\n", encoding="utf-8")
def test_apply_mounts_uploads_only_enabled_skill_projection(monkeypatch, tmp_path):
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
paths = Paths(base_dir=tmp_path)
skills_root = tmp_path / "skills"
_write_skill(skills_root / "public", "enabled-skill")
_write_skill(skills_root / "public", "disabled-skill")
(skills_root / "custom").mkdir()
_write_skill(paths.integration_skills_dir() / "lark-cli", "enabled-integration")
_write_skill(paths.integration_skills_dir() / "lark-cli", "disabled-integration")
user_skills_root = paths.user_skills_dir("user-1")
user_skills_root.mkdir(parents=True, exist_ok=True)
(user_skills_root / "_skill_states.json").write_text(
json.dumps({"disabled-integration": {"enabled": False}}),
encoding="utf-8",
)
extensions = ExtensionsConfig(skills={"disabled-skill": SkillStateConfig(enabled=False)})
config = SimpleNamespace(
skills=SimpleNamespace(
get_skills_path=lambda: skills_root,
container_path="/mnt/skills",
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
)
)
monkeypatch.setattr(mod, "get_app_config", lambda: config)
monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: paths)
monkeypatch.setattr("deerflow.config.extensions_config.ExtensionsConfig.from_file", lambda *_args, **_kwargs: extensions)
monkeypatch.setattr("deerflow.config.extensions_config.get_extensions_config", lambda: extensions)
provider = _make_provider()
client = FakeClient()
provider._apply_mounts(client, user_id="user-1")
uploaded_paths = {path for path, _content in client.files.write_calls}
assert "/mnt/skills/public/enabled-skill/SKILL.md" in uploaded_paths
assert "/mnt/skills/public/disabled-skill/SKILL.md" not in uploaded_paths
assert "/mnt/skills/integrations/lark-cli/enabled-integration/SKILL.md" in uploaded_paths
assert "/mnt/skills/integrations/lark-cli/disabled-integration/SKILL.md" not in uploaded_paths
def test_skill_projection_mounts_swallows_projection_failure(monkeypatch):
"""``_skill_projection_mounts`` must not raise — a projection failure used
to propagate out of ``_apply_mounts`` before the configured-mounts loop
ran, dropping the operator's own configured mounts as collateral (#4107
review)."""
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
config = SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills"))
monkeypatch.setattr(mod, "get_app_config", lambda: config)
monkeypatch.setattr(
"deerflow.skills.projection.ensure_skill_projections",
lambda storage: (_ for _ in ()).throw(RuntimeError("simulated projection failure")),
)
provider = _make_provider()
assert provider._skill_projection_mounts("user-1") == []
def test_apply_mounts_keeps_configured_mounts_when_projection_fails(monkeypatch, tmp_path):
"""End-to-end: a skills-projection failure must not drop the operator's
own configured mounts too the two mount sources are independent."""
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
host_dir = tmp_path / "operator-mount"
host_dir.mkdir()
(host_dir / "notes.txt").write_text("hello", encoding="utf-8")
config = SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills"))
monkeypatch.setattr(mod, "get_app_config", lambda: config)
monkeypatch.setattr(
"deerflow.skills.projection.ensure_skill_projections",
lambda storage: (_ for _ in ()).throw(RuntimeError("simulated projection failure")),
)
provider = _make_provider()
provider._config["mounts"] = [
SimpleNamespace(host_path=str(host_dir), container_path="/mnt/operator", read_only=True),
]
client = FakeClient()
provider._apply_mounts(client, user_id="user-1")
uploaded_paths = {path for path, _content in client.files.write_calls}
assert "/mnt/operator/notes.txt" in uploaded_paths
def _make_sandbox(client: FakeClient, *, sandbox_id: str | None = None) -> Any:
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox")
return mod.E2BSandbox(

View File

@ -52,6 +52,7 @@ async def _run_lifespan_with_hanging_stop() -> float:
patch("app.gateway.app.get_app_config", return_value=startup_config),
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
patch("deerflow.skills.projection.ensure_public_skill_projection"),
patch("app.gateway.app.auth.close_oidc_service", close_oidc_service),
patch("app.channels.service.start_channel_service", side_effect=fake_start),
patch("app.channels.service.stop_channel_service", side_effect=hang_forever),
@ -98,6 +99,7 @@ async def _run_lifespan_with_upload_staging_cleanup():
patch("app.gateway.app.get_app_config", return_value=startup_config),
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
patch("deerflow.skills.projection.ensure_public_skill_projection"),
patch("app.gateway.app.cleanup_stale_upload_staging_files", cleanup_upload_staging_files),
patch("app.gateway.app.auth.close_oidc_service", close_oidc_service),
patch("app.channels.service.start_channel_service", side_effect=fake_start),
@ -156,6 +158,7 @@ async def _run_lifespan_with_memory_flush(*, enabled: bool, flush_return: bool |
patch("app.gateway.app.get_app_config", return_value=startup_config),
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
patch("deerflow.skills.projection.ensure_public_skill_projection"),
patch("app.gateway.app.auth.close_oidc_service", close_oidc_service),
patch("app.channels.service.start_channel_service", side_effect=fake_start),
patch("app.channels.service.stop_channel_service", stop_channel_service),

View File

@ -544,7 +544,7 @@ class TestMultipleMounts:
class TestLocalSandboxProviderMounts:
def test_thread_mappings_mount_global_integrations_for_every_user(self, tmp_path):
def test_thread_mappings_mount_per_user_integration_projections(self, tmp_path):
from deerflow.config.paths import Paths
paths = Paths(base_dir=tmp_path / "home")
@ -568,10 +568,11 @@ class TestLocalSandboxProviderMounts:
alice_integrations = next(mapping for mapping in alice if mapping.container_path == "/mnt/skills/integrations")
bob_integrations = next(mapping for mapping in bob if mapping.container_path == "/mnt/skills/integrations")
expected = str(tmp_path / "home" / "integrations" / "skills")
assert alice_integrations.local_path == expected
assert bob_integrations.local_path == expected
assert alice_integrations.local_path == str(paths.user_integration_skills_view_dir("alice"))
assert bob_integrations.local_path == str(paths.user_integration_skills_view_dir("bob"))
assert alice_integrations.local_path != bob_integrations.local_path
assert alice_integrations.read_only is True
assert bob_integrations.read_only is True
def test_setup_path_mappings_uses_configured_skills_container_path_as_reserved_prefix(self, tmp_path):
skills_dir = tmp_path / "skills"

View File

@ -339,6 +339,7 @@ def test_gateway_lifespan_initializes_monocle():
patch("app.gateway.app.get_app_config", return_value=startup_config),
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
patch("deerflow.skills.projection.ensure_public_skill_projection"),
patch("app.gateway.app.setup_monocle_tracing_if_enabled", setup_spy),
patch("app.gateway.app.auth.close_oidc_service", AsyncMock()),
patch("app.channels.service.start_channel_service", side_effect=fake_start),
@ -381,6 +382,7 @@ def test_gateway_lifespan_survives_monocle_setup_failure(caplog):
patch("app.gateway.app.get_app_config", return_value=startup_config),
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
patch("deerflow.skills.projection.ensure_public_skill_projection"),
patch("app.gateway.app.setup_monocle_tracing_if_enabled", setup_spy),
patch("app.gateway.app.auth.close_oidc_service", AsyncMock()),
patch("app.channels.service.start_channel_service", side_effect=fake_start),

View File

@ -10,12 +10,12 @@ class TestBuildVolumes:
# ── hostPath mode (default) ────────────────────────────────────────
def test_hostpath_without_legacy_returns_three_volumes(self, provisioner_module):
"""hostPath mode omits legacy volume unless the backend requests it."""
def test_hostpath_uses_three_projection_volumes(self, provisioner_module):
"""hostPath mode always mounts stable public/custom/legacy views."""
provisioner_module.SKILLS_PVC_NAME = ""
provisioner_module.USERDATA_PVC_NAME = ""
volumes = provisioner_module._build_volumes("thread-1")
assert len(volumes) == 3
assert len(volumes) == 4
def test_hostpath_skills_public_volume(self, provisioner_module):
"""First skills volume mounts public/ subdirectory."""
@ -24,7 +24,7 @@ class TestBuildVolumes:
pub = volumes[0]
assert pub.name == "skills-public"
assert pub.host_path is not None
assert pub.host_path.path.endswith("/public")
assert pub.host_path.path.endswith("/skills_view/public")
assert pub.host_path.type == "Directory"
assert pub.persistent_volume_claim is None
@ -35,11 +35,11 @@ class TestBuildVolumes:
custom = volumes[1]
assert custom.name == "skills-custom"
assert custom.host_path is not None
assert "users/user-7/skills/custom" in custom.host_path.path
assert custom.host_path.type == "DirectoryOrCreate"
assert "users/user-7/skills_view/custom" in custom.host_path.path
assert custom.host_path.type == "Directory"
def test_hostpath_skills_legacy_volume(self, provisioner_module):
"""Legacy global-custom directory is mounted only when requested."""
"""Legacy projection is per-user and stable regardless of visibility."""
provisioner_module.SKILLS_PVC_NAME = ""
volumes = provisioner_module._build_volumes(
"thread-1",
@ -48,16 +48,17 @@ class TestBuildVolumes:
legacy = volumes[2]
assert legacy.name == "skills-legacy"
assert legacy.host_path is not None
assert legacy.host_path.path.endswith("/custom")
assert "users/default/skills_view/legacy" in legacy.host_path.path
assert legacy.host_path.type == "Directory"
def test_hostpath_without_legacy_has_no_legacy_volume(self, provisioner_module):
"""Fresh installs should not require a missing global legacy directory."""
def test_hostpath_without_legacy_flag_still_has_empty_capable_mount(self, provisioner_module):
"""Visibility changes update contents without recreating the Pod."""
provisioner_module.SKILLS_PVC_NAME = ""
volumes = provisioner_module._build_volumes("thread-1")
assert [volume.name for volume in volumes] == [
"skills-public",
"skills-custom",
"skills-legacy",
"user-data",
]
@ -129,8 +130,8 @@ class TestBuildVolumes:
volumes = provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts)
# skills-public + skills-custom + user-data (3 base) + 1 extra volume.
assert len(volumes) == 4
# Three skill projections + user-data (4 base) + 1 extra volume.
assert len(volumes) == 5
extra_vol = volumes[-1]
assert extra_vol.name == "extra-0"
assert extra_vol.host_path.path == "/state/users/alice/integrations/lark-cli/config"
@ -151,8 +152,8 @@ class TestBuildVolumes:
volumes = provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts)
# skills-public + skills-custom + user-data (3 base) + 1 extra volume.
assert len(volumes) == 4
# Three skill projections + user-data (4 base) + 1 extra volume.
assert len(volumes) == 5
extra_vol = volumes[-1]
assert extra_vol.name == "extra-0"
assert extra_vol.persistent_volume_claim is not None
@ -184,12 +185,12 @@ class TestBuildVolumeMounts:
# ── hostPath mode ──────────────────────────────────────────────────
def test_hostpath_without_legacy_returns_three_mounts(self, provisioner_module):
"""hostPath mode omits legacy mount unless the backend requests it."""
def test_hostpath_uses_three_projection_mounts(self, provisioner_module):
"""hostPath mode always mounts stable public/custom/legacy views."""
provisioner_module.SKILLS_PVC_NAME = ""
provisioner_module.USERDATA_PVC_NAME = ""
mounts = provisioner_module._build_volume_mounts("thread-1")
assert len(mounts) == 3
assert len(mounts) == 4
def test_hostpath_skills_public_mount(self, provisioner_module):
"""Public skills mount at /mnt/skills/public, read-only."""
@ -218,13 +219,14 @@ class TestBuildVolumeMounts:
assert mounts[2].mount_path == "/mnt/skills/legacy"
assert mounts[2].read_only is True
def test_hostpath_without_legacy_has_no_legacy_mount(self, provisioner_module):
"""Users with custom skills should not see hidden legacy content in the sandbox."""
def test_hostpath_without_legacy_flag_still_has_legacy_mount(self, provisioner_module):
"""Hidden legacy content is represented by an empty mounted view."""
provisioner_module.SKILLS_PVC_NAME = ""
mounts = provisioner_module._build_volume_mounts("thread-1")
assert [mount.name for mount in mounts] == [
"skills-public",
"skills-custom",
"skills-legacy",
"user-data",
]
@ -353,19 +355,19 @@ class TestBuildVolumeMounts:
class TestBuildPodVolumes:
"""Integration: _build_pod should wire volumes and mounts correctly."""
def test_pod_hostpath_without_legacy_has_three_volumes(self, provisioner_module):
"""hostPath Pod spec should omit legacy volume by default."""
def test_pod_hostpath_has_four_volumes(self, provisioner_module):
"""hostPath Pod spec includes all three stable projection categories."""
provisioner_module.SKILLS_PVC_NAME = ""
provisioner_module.USERDATA_PVC_NAME = ""
pod = provisioner_module._build_pod("sandbox-1", "thread-1")
assert len(pod.spec.volumes) == 3
assert len(pod.spec.volumes) == 4
def test_pod_hostpath_without_legacy_has_three_mounts(self, provisioner_module):
"""hostPath container should omit legacy mount by default."""
def test_pod_hostpath_has_four_mounts(self, provisioner_module):
"""hostPath container includes all three stable projection categories."""
provisioner_module.SKILLS_PVC_NAME = ""
provisioner_module.USERDATA_PVC_NAME = ""
pod = provisioner_module._build_pod("sandbox-1", "thread-1")
assert len(pod.spec.containers[0].volume_mounts) == 3
assert len(pod.spec.containers[0].volume_mounts) == 4
def test_pod_hostpath_with_legacy_has_four_volumes(self, provisioner_module):
"""Legacy volume should be present when the backend requests it."""
@ -438,8 +440,8 @@ class TestBuildPodVolumes:
extra_mounts=extra_mounts,
)
# skills-public + skills-custom + user-data (3 base) + 2 extra mounts.
assert len(pod.spec.volumes) == 5
# Three skill projections + user-data (4 base) + 2 extra mounts.
assert len(pod.spec.volumes) == 6
mount_paths = {mount.mount_path for mount in pod.spec.containers[0].volume_mounts}
assert "/mnt/integrations/lark-cli/config" in mount_paths
assert "/mnt/integrations/lark-cli/data" in mount_paths

View File

@ -167,7 +167,7 @@ async def test_sandbox_business_routes_run_k8s_client_off_event_loop_thread(
[
(
False,
["skills-public", "skills-custom", "user-data"],
["skills-public", "skills-custom", "skills-legacy", "user-data"],
),
(
True,

View File

@ -210,6 +210,48 @@ def test_skill_manage_rejects_support_path_traversal(monkeypatch, tmp_path):
)
def test_skill_manage_remove_file_updates_sandbox_projection_before_return(monkeypatch, tmp_path):
skills_root = tmp_path / "skills"
config = _make_config(skills_root)
monkeypatch.setattr("deerflow.config.get_app_config", lambda: config)
monkeypatch.setattr("deerflow.skills.security_scanner.get_app_config", lambda: config)
from deerflow.config.paths import Paths
monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: Paths(base_dir=tmp_path))
monkeypatch.setattr("deerflow.config.paths._paths", None)
async def _refresh(user_id: str):
return None
monkeypatch.setattr(skill_manage_module, "refresh_user_skills_system_prompt_cache_async", _refresh)
monkeypatch.setattr(skill_manage_module, "scan_skill_content", lambda *args, **kwargs: _async_result("allow", "ok"))
runtime = _make_runtime(user_id="default")
anyio.run(skill_manage_module.skill_manage_tool.coroutine, runtime, "create", "demo-skill", _skill_content("demo-skill"))
anyio.run(
skill_manage_module.skill_manage_tool.coroutine,
runtime,
"write_file",
"demo-skill",
"supporting content",
"references/guide.md",
)
projected_file = tmp_path / "users" / "default" / "skills_view" / "custom" / "demo-skill" / "references" / "guide.md"
assert projected_file.read_text(encoding="utf-8") == "supporting content"
result = anyio.run(
skill_manage_module.skill_manage_tool.coroutine,
runtime,
"remove_file",
"demo-skill",
None,
"references/guide.md",
)
assert result == "Removed 'references/guide.md' from custom skill 'demo-skill'."
assert not projected_file.exists()
def test_skill_manage_static_critical_blocks_create_before_llm(monkeypatch, tmp_path):
skills_root = tmp_path / "skills"
config = _make_config(skills_root)

View File

@ -0,0 +1,612 @@
"""Enabled-only filesystem projections exposed to sandbox providers."""
from __future__ import annotations
import errno
import shutil
import zipfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from threading import Barrier, Event
from types import SimpleNamespace
from unittest.mock import patch
import pytest
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig
from deerflow.config.paths import Paths
from deerflow.skills.projection import ensure_public_skill_projection, ensure_skill_projections, rebuild_skill_projections, skill_projection_mutation
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
def _skill_content(name: str, marker: str = "v1") -> str:
return f"---\nname: {name}\ndescription: {marker}\n---\n\n# {name}\n\n{marker}\n"
def _write_skill(root: Path, name: str, marker: str = "v1") -> Path:
target = root / name / "SKILL.md"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(_skill_content(name, marker), encoding="utf-8")
return target
@pytest.fixture
def projection_env(tmp_path: Path):
skills_root = tmp_path / "skills"
(skills_root / "public").mkdir(parents=True)
(skills_root / "custom").mkdir()
paths = Paths(base_dir=tmp_path)
config = SimpleNamespace(
skills=SimpleNamespace(
get_skills_path=lambda: skills_root,
container_path="/mnt/skills",
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
)
)
extensions = ExtensionsConfig()
with (
patch("deerflow.config.paths.get_paths", return_value=paths),
patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=extensions),
patch("deerflow.config.extensions_config.get_extensions_config", return_value=extensions),
):
storage = UserScopedSkillStorage("alice", host_path=str(skills_root), app_config=config)
yield SimpleNamespace(
root=tmp_path,
skills_root=skills_root,
paths=paths,
config=config,
extensions=extensions,
storage=storage,
)
def test_projection_contains_only_enabled_skills(projection_env) -> None:
env = projection_env
enabled = _write_skill(env.skills_root / "public", "enabled-skill")
_write_skill(env.skills_root / "public", "disabled-skill")
env.extensions.skills["disabled-skill"] = SkillStateConfig(enabled=False)
projected = rebuild_skill_projections(env.storage)
enabled_view = projected.public / "enabled-skill" / "SKILL.md"
assert enabled_view.read_text(encoding="utf-8") == enabled.read_text(encoding="utf-8")
assert not (projected.public / "disabled-skill").exists()
def test_projection_rebuild_removes_newly_disabled_skill(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "demo-skill")
projected = rebuild_skill_projections(env.storage)
assert (projected.public / "demo-skill" / "SKILL.md").is_file()
env.extensions.skills["demo-skill"] = SkillStateConfig(enabled=False)
rebuild_skill_projections(env.storage)
assert not (projected.public / "demo-skill").exists()
def test_nested_skill_frontmatter_is_supporting_data_inside_parent_package(projection_env) -> None:
env = projection_env
parent_root = env.skills_root / "public" / "parent-skill"
_write_skill(env.skills_root / "public", "parent-skill")
nested = _write_skill(parent_root / "fixtures", "nested-skill")
env.extensions.skills["nested-skill"] = SkillStateConfig(enabled=False)
projected = rebuild_skill_projections(env.storage)
nested_view = projected.public / nested.parent.relative_to(env.skills_root / "public")
assert (projected.public / "parent-skill" / "SKILL.md").is_file()
assert (nested_view / "SKILL.md").is_file()
def test_projection_falls_back_to_copy_when_hardlink_is_unavailable(projection_env, monkeypatch) -> None:
env = projection_env
source = _write_skill(env.skills_root / "public", "demo-skill")
def _cross_device_link(*_args, **_kwargs):
raise OSError(errno.EXDEV, "cross-device link")
monkeypatch.setattr("deerflow.skills.projection.os.link", _cross_device_link)
projected = rebuild_skill_projections(env.storage)
target = projected.public / "demo-skill" / "SKILL.md"
assert target.read_text(encoding="utf-8") == source.read_text(encoding="utf-8")
assert target.stat().st_ino != source.stat().st_ino
def test_atomic_custom_skill_rewrite_refreshes_projection(projection_env) -> None:
env = projection_env
env.storage.write_custom_skill("demo-skill", "SKILL.md", _skill_content("demo-skill", "before"))
projected = rebuild_skill_projections(env.storage)
target = projected.custom / "demo-skill" / "SKILL.md"
old_inode = target.stat().st_ino
env.storage.write_custom_skill("demo-skill", "SKILL.md", _skill_content("demo-skill", "after"))
assert "after" in target.read_text(encoding="utf-8")
assert target.stat().st_ino != old_inode
def test_custom_content_write_keeps_unrelated_skill_visible_during_rebuild(projection_env, monkeypatch) -> None:
env = projection_env
env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha"))
env.storage.write_custom_skill("beta", "SKILL.md", _skill_content("beta", "before"))
projected = rebuild_skill_projections(env.storage)
alpha_view = projected.custom / "alpha" / "SKILL.md"
from deerflow.skills import projection as projection_module
real_stage_skill = projection_module._stage_skill
staging_started = Event()
release_staging = Event()
def _delayed_stage_skill(*args, **kwargs):
if not staging_started.is_set():
staging_started.set()
assert release_staging.wait(timeout=5)
return real_stage_skill(*args, **kwargs)
monkeypatch.setattr(projection_module, "_stage_skill", _delayed_stage_skill)
with ThreadPoolExecutor(max_workers=1) as executor:
write_future = executor.submit(env.storage.write_custom_skill, "beta", "SKILL.md", _skill_content("beta", "after"))
assert staging_started.wait(timeout=5)
unrelated_skill_remained_visible = alpha_view.is_file()
release_staging.set()
write_future.result(timeout=5)
assert unrelated_skill_remained_visible
assert "after" in (projected.custom / "beta" / "SKILL.md").read_text(encoding="utf-8")
def test_per_user_toggle_removes_custom_skill_before_returning(projection_env) -> None:
env = projection_env
env.storage.write_custom_skill("demo-skill", "SKILL.md", _skill_content("demo-skill"))
projected = rebuild_skill_projections(env.storage)
assert (projected.custom / "demo-skill" / "SKILL.md").is_file()
env.storage.set_skill_enabled_state("demo-skill", False)
assert not (projected.custom / "demo-skill").exists()
def test_managed_integration_projection_is_filtered_per_user(projection_env) -> None:
env = projection_env
integration_root = env.paths.integration_skills_dir()
_write_skill(integration_root / "lark-cli", "lark-doc")
alice_projection = rebuild_skill_projections(env.storage)
alice_skill = alice_projection.integrations / "lark-cli" / "lark-doc" / "SKILL.md"
assert alice_skill.is_file()
env.storage.set_skill_enabled_state("lark-doc", False)
assert not alice_skill.exists()
bob_storage = UserScopedSkillStorage("bob", host_path=str(env.skills_root), app_config=env.config)
bob_projection = rebuild_skill_projections(bob_storage)
assert (bob_projection.integrations / "lark-cli" / "lark-doc" / "SKILL.md").is_file()
def test_disabling_custom_skill_hides_only_target_while_rebuilding(projection_env, monkeypatch) -> None:
env = projection_env
env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha"))
env.storage.write_custom_skill("beta", "SKILL.md", _skill_content("beta"))
projected = rebuild_skill_projections(env.storage)
from deerflow.skills import projection as projection_module
real_stage_skill = projection_module._stage_skill
staging_started = Event()
release_staging = Event()
def _delayed_stage_skill(*args, **kwargs):
if not staging_started.is_set():
staging_started.set()
assert release_staging.wait(timeout=5)
return real_stage_skill(*args, **kwargs)
monkeypatch.setattr(projection_module, "_stage_skill", _delayed_stage_skill)
with ThreadPoolExecutor(max_workers=1) as executor:
disable_future = executor.submit(env.storage.set_skill_enabled_state, "beta", False)
assert staging_started.wait(timeout=5)
alpha_remained_visible = (projected.custom / "alpha" / "SKILL.md").is_file()
beta_was_hidden = not (projected.custom / "beta").exists()
release_staging.set()
disable_future.result(timeout=5)
assert alpha_remained_visible
assert beta_was_hidden
def test_disabling_namespaced_skill_hides_its_real_projection_path(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "custom" / "team", "helper")
projected = rebuild_skill_projections(env.storage)
target = projected.legacy / "team" / "helper"
assert (target / "SKILL.md").is_file()
with skill_projection_mutation(env.storage, "user", remove_names=("helper",)):
env.storage._write_skill_states({"helper": {"enabled": False}})
assert not target.exists()
assert not target.exists()
def test_disabling_duplicate_namespaced_public_skills_hides_every_path(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "public" / "team-a", "helper")
_write_skill(env.skills_root / "public" / "team-b", "helper")
projected = rebuild_skill_projections(env.storage)
targets = [projected.public / team / "helper" for team in ("team-a", "team-b")]
assert all((target / "SKILL.md").is_file() for target in targets)
with skill_projection_mutation(env.storage, "public", remove_names=("helper",)):
env.extensions.skills["helper"] = SkillStateConfig(enabled=False)
assert all(not target.exists() for target in targets)
assert all(not target.exists() for target in targets)
def test_mutation_failure_clears_projection_scope(projection_env) -> None:
env = projection_env
env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha"))
projected = rebuild_skill_projections(env.storage)
manifest = projected.custom.parent / ".projection-manifest.json"
assert (projected.custom / "alpha" / "SKILL.md").is_file()
with pytest.raises(OSError, match="mutation failed"):
with skill_projection_mutation(env.storage, "user"):
raise OSError("mutation failed")
assert list(projected.custom.iterdir()) == []
assert list(projected.legacy.iterdir()) == []
assert list(projected.integrations.iterdir()) == []
assert not manifest.exists()
def test_targeted_removal_failure_clears_drifted_projection_scope(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "public" / "team", "helper")
projected = rebuild_skill_projections(env.storage)
manifest = projected.public.parent / ".projection-manifest.json"
namespace = projected.public / "team"
shutil.rmtree(namespace)
namespace.write_text("drifted file", encoding="utf-8")
with pytest.raises(NotADirectoryError):
with skill_projection_mutation(env.storage, "public", remove_names=("helper",)):
env.extensions.skills["helper"] = SkillStateConfig(enabled=False)
assert list(projected.public.iterdir()) == []
assert not manifest.exists()
def test_user_custom_skill_replaces_legacy_projection(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "custom", "legacy-skill")
projected = rebuild_skill_projections(env.storage)
assert (projected.legacy / "legacy-skill" / "SKILL.md").is_file()
env.storage.write_custom_skill("custom-skill", "SKILL.md", _skill_content("custom-skill"))
assert (projected.custom / "custom-skill" / "SKILL.md").is_file()
assert list(projected.legacy.iterdir()) == []
@pytest.mark.parametrize("category", ["custom", "legacy"])
def test_ensure_user_projection_uses_fresh_global_state_across_workers(projection_env, category: str) -> None:
env = projection_env
skill_name = f"{category}-skill"
source_root = env.storage.get_user_custom_root() if category == "custom" else env.skills_root / "custom"
_write_skill(source_root, skill_name)
projected = rebuild_skill_projections(env.storage)
target = getattr(projected, category) / skill_name / "SKILL.md"
manifest = projected.custom.parent / ".projection-manifest.json"
assert target.is_file()
assert manifest.is_file()
fresh_extensions = ExtensionsConfig()
fresh_extensions.skills[skill_name] = SkillStateConfig(enabled=False)
with patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=fresh_extensions):
ensure_skill_projections(env.storage)
assert not target.exists()
assert manifest.is_file()
rebuilt_manifest = manifest.read_text(encoding="utf-8")
# The rebuilt manifest must not mark the stale visible view as fresh.
ensure_skill_projections(env.storage)
assert not target.exists()
assert manifest.read_text(encoding="utf-8") == rebuilt_manifest
def test_ensure_repairs_direct_atomic_source_replacement(projection_env) -> None:
env = projection_env
source = _write_skill(env.skills_root / "public", "demo-skill", "before")
projected = rebuild_skill_projections(env.storage)
target = projected.public / "demo-skill" / "SKILL.md"
old_projected_inode = target.stat().st_ino
replacement = source.with_suffix(".replacement")
replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8")
replacement.replace(source)
ensure_skill_projections(env.storage)
assert "after" in target.read_text(encoding="utf-8")
assert target.stat().st_ino != old_projected_inode
def test_ensure_without_source_changes_keeps_projected_inode(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "demo-skill")
projected = rebuild_skill_projections(env.storage)
target = projected.public / "demo-skill" / "SKILL.md"
projected_inode = target.stat().st_ino
ensure_skill_projections(env.storage)
assert target.stat().st_ino == projected_inode
def test_ensure_steady_state_public_signature_checks_do_not_serialize(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "demo-skill")
rebuild_skill_projections(env.storage)
from deerflow.skills import projection as projection_module
real_source_signature = projection_module._source_signature
public_signatures = Barrier(2, timeout=2)
def _synchronized_source_signature(storage, scope):
if scope == "public":
public_signatures.wait()
return real_source_signature(storage, scope)
monkeypatch.setattr(projection_module, "_source_signature", _synchronized_source_signature)
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [executor.submit(ensure_skill_projections, env.storage) for _ in range(2)]
for future in futures:
future.result(timeout=5)
def test_unlocked_public_snapshot_detects_manifest_change_during_signature_scan(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "demo-skill")
projected = rebuild_skill_projections(env.storage)
manifest = projected.public.parent / ".projection-manifest.json"
from deerflow.skills import projection as projection_module
real_source_signature = projection_module._source_signature
signature_read = Event()
release_signature = Event()
def _delayed_source_signature(storage, scope):
signature = real_source_signature(storage, scope)
if scope == "public":
signature_read.set()
assert release_signature.wait(timeout=5)
return signature
monkeypatch.setattr(projection_module, "_source_signature", _delayed_source_signature)
with ThreadPoolExecutor(max_workers=1) as executor:
ensure_future = executor.submit(ensure_skill_projections, env.storage)
assert signature_read.wait(timeout=5)
manifest.unlink()
release_signature.set()
ensure_future.result(timeout=5)
assert manifest.is_file()
def test_concurrent_stale_public_ensure_rebuilds_once_after_lock_recheck(projection_env) -> None:
env = projection_env
source = _write_skill(env.skills_root / "public", "demo-skill", "before")
rebuild_skill_projections(env.storage)
replacement = source.with_suffix(".replacement")
replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8")
replacement.replace(source)
from deerflow.skills import projection as projection_module
with patch.object(projection_module, "_rebuild_public_locked", wraps=projection_module._rebuild_public_locked) as rebuild:
with ThreadPoolExecutor(max_workers=8) as executor:
list(executor.map(ensure_skill_projections, [env.storage] * 8))
assert rebuild.call_count == 1
def test_rebuild_keeps_category_root_inode_stable(projection_env) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "demo-skill")
projected = rebuild_skill_projections(env.storage)
root_inode = projected.public.stat().st_ino
env.extensions.skills["demo-skill"] = SkillStateConfig(enabled=False)
rebuild_skill_projections(env.storage)
assert projected.public.stat().st_ino == root_inode
def test_rebuild_failure_clears_old_projection(projection_env, monkeypatch) -> None:
env = projection_env
source = _write_skill(env.skills_root / "public", "demo-skill", "before")
projected = rebuild_skill_projections(env.storage)
assert (projected.public / "demo-skill" / "SKILL.md").is_file()
replacement = source.with_suffix(".replacement")
replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8")
replacement.replace(source)
monkeypatch.setattr("deerflow.skills.projection._stage_skill", lambda *_args: (_ for _ in ()).throw(OSError("disk full")))
with pytest.raises(OSError, match="disk full"):
ensure_skill_projections(env.storage)
assert list(projected.public.iterdir()) == []
def test_signature_failure_clears_old_projection_and_manifest(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "demo-skill")
projected = rebuild_skill_projections(env.storage)
manifest = projected.public.parent / ".projection-manifest.json"
assert (projected.public / "demo-skill" / "SKILL.md").is_file()
assert manifest.is_file()
monkeypatch.setattr(
"deerflow.skills.projection._source_signature",
lambda *_args, **_kwargs: (_ for _ in ()).throw(PermissionError("source metadata unavailable")),
)
with pytest.raises(PermissionError, match="source metadata unavailable"):
ensure_skill_projections(env.storage)
assert list(projected.public.iterdir()) == []
assert not manifest.exists()
def test_boot_ensures_public_projection_without_scanning_known_users(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "public-skill")
_write_skill(env.paths.user_custom_skills_dir("bob"), "custom-skill")
def _unexpected_user_storage(*_args, **_kwargs):
raise AssertionError("gateway boot must not enumerate user skill storage")
monkeypatch.setattr("deerflow.skills.storage.get_or_new_user_skill_storage", _unexpected_user_storage)
assert ensure_public_skill_projection(app_config=env.config) is True
assert (env.paths.public_skills_view_dir / "public-skill" / "SKILL.md").is_file()
assert not env.paths.user_custom_skills_view_dir("bob").exists()
bob_storage = UserScopedSkillStorage("bob", host_path=str(env.skills_root), app_config=env.config)
ensure_skill_projections(bob_storage)
assert (env.paths.user_custom_skills_view_dir("bob") / "custom-skill" / "SKILL.md").is_file()
def test_boot_public_projection_failure_is_fail_closed_without_aborting(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "public-skill", "before")
rebuild_skill_projections(env.storage, include_user=False)
monkeypatch.setattr(
"deerflow.skills.projection._source_signature",
lambda *_args, **_kwargs: (_ for _ in ()).throw(PermissionError("source unavailable")),
)
assert ensure_public_skill_projection(app_config=env.config) is False
assert list(env.paths.public_skills_view_dir.iterdir()) == []
def test_boot_public_storage_factory_failure_is_fail_closed_without_aborting(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "public-skill")
rebuild_skill_projections(env.storage, include_user=False)
monkeypatch.setattr(
"deerflow.skills.storage.get_or_new_skill_storage",
lambda **_kwargs: (_ for _ in ()).throw(RuntimeError("storage init failed")),
)
assert ensure_public_skill_projection(app_config=env.config) is False
assert list(env.paths.public_skills_view_dir.iterdir()) == []
def test_boot_factory_failure_cleanup_waits_for_concurrent_public_rebuild(projection_env, monkeypatch) -> None:
env = projection_env
_write_skill(env.skills_root / "public", "public-skill")
from deerflow.skills import projection as projection_module
before_manifest = Event()
release_rebuild = Event()
cleanup_finished = Event()
real_write_manifest = projection_module._write_manifest
def _delayed_write_manifest(scope_root, signature):
before_manifest.set()
assert release_rebuild.wait(timeout=5)
real_write_manifest(scope_root, signature)
monkeypatch.setattr(projection_module, "_write_manifest", _delayed_write_manifest)
monkeypatch.setattr(
"deerflow.skills.storage.get_or_new_skill_storage",
lambda **_kwargs: (_ for _ in ()).throw(RuntimeError("storage init failed")),
)
def _rebuild() -> None:
rebuild_skill_projections(env.storage, include_user=False)
def _fail_boot_ensure() -> None:
assert ensure_public_skill_projection(app_config=env.config) is False
cleanup_finished.set()
with ThreadPoolExecutor(max_workers=2) as executor:
rebuild_future = executor.submit(_rebuild)
assert before_manifest.wait(timeout=5)
cleanup_future = executor.submit(_fail_boot_ensure)
cleanup_waited_for_rebuild = not cleanup_finished.wait(timeout=0.2)
release_rebuild.set()
rebuild_future.result(timeout=5)
cleanup_future.result(timeout=5)
assert cleanup_waited_for_rebuild
assert list(env.paths.public_skills_view_dir.iterdir()) == []
assert not (env.paths.public_skills_view_dir.parent / ".projection-manifest.json").exists()
@pytest.mark.anyio
async def test_archive_install_is_projected_before_return(projection_env, monkeypatch, tmp_path) -> None:
from deerflow.skills.security_scanner import ScanResult
env = projection_env
archive = tmp_path / "archive-skill.skill"
with zipfile.ZipFile(archive, "w") as bundle:
bundle.writestr("archive-skill/SKILL.md", _skill_content("archive-skill"))
bundle.writestr("archive-skill/references/guide.md", "# Guide\n")
async def _allow_scan(*_args, **_kwargs):
return ScanResult(decision="allow", reason="test")
monkeypatch.setattr("deerflow.skills.installer.scan_skill_content", _allow_scan)
result = await env.storage.ainstall_skill_from_archive(archive)
projected = env.paths.user_custom_skills_view_dir("alice") / "archive-skill"
assert result["success"] is True
assert (projected / "SKILL.md").is_file()
assert (projected / "references" / "guide.md").read_text(encoding="utf-8") == "# Guide\n"
def test_concurrent_custom_skill_writes_do_not_lose_projected_entries(projection_env) -> None:
env = projection_env
names = [f"skill-{index}" for index in range(8)]
def _write(name: str) -> None:
env.storage.write_custom_skill(name, "SKILL.md", _skill_content(name))
with ThreadPoolExecutor(max_workers=4) as executor:
list(executor.map(_write, names))
projected_names = {path.name for path in env.paths.user_custom_skills_view_dir("alice").iterdir()}
assert projected_names == set(names)
def test_concurrent_custom_skill_toggles_do_not_lose_state(projection_env) -> None:
env = projection_env
names = ("skill-a", "skill-b")
for name in names:
env.storage.write_custom_skill(name, "SKILL.md", _skill_content(name))
with ThreadPoolExecutor(max_workers=2) as executor:
list(executor.map(lambda name: env.storage.set_skill_enabled_state(name, False), names))
assert env.storage._read_skill_states() == {
"skill-a": {"enabled": False},
"skill-b": {"enabled": False},
}
assert list(env.paths.user_custom_skills_view_dir("alice").iterdir()) == []

View File

@ -628,7 +628,6 @@ def test_update_skill_refreshes_prompt_cache_before_return(monkeypatch, tmp_path
mock_storage = _FakeUserScopedStorage()
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: mock_storage)
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "default")
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: SimpleNamespace(mcp_servers={}, skills={}))
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(lambda config_path=None: config_path))
monkeypatch.setattr("app.gateway.routers.skills.refresh_user_skills_system_prompt_cache_async", _refresh)
@ -705,7 +704,6 @@ def test_public_skill_toggle_clears_all_users_cache(monkeypatch, tmp_path):
return config_path
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(_resolve))
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: __import__("deerflow.config.extensions_config", fromlist=["ExtensionsConfig"]).ExtensionsConfig.from_file(config_path))
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
monkeypatch.setattr("app.gateway.routers.skills.clear_skills_system_prompt_cache", _clear)
monkeypatch.setattr("app.gateway.routers.skills.refresh_user_skills_system_prompt_cache_async", _refresh)
@ -725,6 +723,107 @@ def test_public_skill_toggle_clears_all_users_cache(monkeypatch, tmp_path):
assert persisted["skills"]["public-skill"]["enabled"] is False
def test_public_skill_toggle_creates_missing_extensions_config(monkeypatch, tmp_path):
backend_dir = tmp_path / "backend"
backend_dir.mkdir()
monkeypatch.chdir(backend_dir)
config_path = tmp_path / "extensions_config.json"
def _load_skills(*, enabled_only: bool):
enabled = True
if config_path.exists():
enabled = json.loads(config_path.read_text(encoding="utf-8"))["skills"]["public-skill"]["enabled"]
skill = _make_skill("public-skill", enabled=enabled)
return [] if enabled_only and not enabled else [skill]
storage = SimpleNamespace(load_skills=_load_skills)
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda _config: storage)
def _resolve_config_path(explicit_path=None):
if explicit_path is None:
return None
path = Path(explicit_path)
if not path.exists():
raise FileNotFoundError(path)
return path
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(_resolve_config_path))
monkeypatch.setattr(skills_router, "reload_extensions_config", lambda: None)
monkeypatch.setattr(skills_router, "clear_skills_system_prompt_cache", lambda: None)
with TestClient(_make_test_app(SimpleNamespace())) as client:
response = client.put("/api/skills/public-skill", json={"enabled": False})
assert response.status_code == 200, response.text
assert response.json()["enabled"] is False
assert json.loads(config_path.read_text(encoding="utf-8")) == {
"mcpServers": {},
"skills": {"public-skill": {"enabled": False}},
"middlewares": [],
}
def test_public_skill_toggle_rebuilds_projection_before_response(monkeypatch, tmp_path):
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, reset_extensions_config, set_extensions_config
from deerflow.config.paths import Paths
from deerflow.skills.projection import rebuild_skill_projections
skills_root = tmp_path / "skills"
skill_file = skills_root / "public" / "public-skill" / "SKILL.md"
skill_file.parent.mkdir(parents=True)
skill_file.write_text(_skill_content("public-skill"), encoding="utf-8")
(skills_root / "custom").mkdir()
config_path = tmp_path / "extensions_config.json"
config_path.write_text(
json.dumps(
{
"mcpServers": {},
"skills": {
"public-skill": {"enabled": True},
"untouched-skill": {"enabled": False},
},
}
),
encoding="utf-8",
)
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(config_path))
paths = Paths(base_dir=tmp_path)
config = SimpleNamespace(
skills=SimpleNamespace(
get_skills_path=lambda: skills_root,
container_path="/mnt/skills",
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
),
skill_evolution=SimpleNamespace(enabled=True, moderation_model_name=None),
)
monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: paths)
monkeypatch.setattr("deerflow.config.paths._paths", None)
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "default")
monkeypatch.setattr(skills_router, "clear_skills_system_prompt_cache", lambda: None)
# Simulate another worker having updated the file after this worker cached
# an older snapshot. The public toggle must reload from disk under the
# cross-process projection lock before its read-modify-write.
set_extensions_config(ExtensionsConfig(skills={"public-skill": SkillStateConfig(enabled=True)}))
storage = UserScopedSkillStorage("default", host_path=str(skills_root), app_config=config)
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda _config: storage)
projected = rebuild_skill_projections(storage)
assert (projected.public / "public-skill" / "SKILL.md").is_file()
try:
with TestClient(_make_test_app(config)) as client:
response = client.put("/api/skills/public-skill", json={"enabled": False})
assert response.status_code == 200, response.text
assert response.json()["enabled"] is False
assert not (projected.public / "public-skill").exists()
persisted = json.loads(config_path.read_text(encoding="utf-8"))
assert persisted["skills"]["untouched-skill"] == {"enabled": False}
finally:
reset_extensions_config()
class TestMultiUserSkillIsolation:
"""End-to-end integration tests verifying per-user skill isolation
through the HTTP router _get_user_skill_storage filesystem chain.
@ -955,7 +1054,6 @@ class TestMultiUserSkillIsolation:
)
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: alice_storage)
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "alice")
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: SimpleNamespace(mcp_servers={}, skills={}))
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
app = _make_test_app(config)
@ -1009,7 +1107,6 @@ class TestMultiUserSkillIsolation:
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: alice_storage)
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "alice")
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: SimpleNamespace(mcp_servers={}, skills={}))
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
monkeypatch.setattr("app.gateway.routers.skills.refresh_user_skills_system_prompt_cache_async", _noop_async)

View File

@ -125,6 +125,17 @@ def test_enable_toggle_allowed_for_admin(monkeypatch, tmp_path):
from deerflow.skills.types import Skill
config_path = tmp_path / "extensions_config.json"
config_path.write_text(
json.dumps(
{
"mcpServers": {},
"skills": {},
"middlewares": ["pkg:Middleware"],
"mcpInterceptors": ["pkg.interceptor:build"],
}
),
encoding="utf-8",
)
def _load_skills(*, enabled_only: bool):
return [
@ -141,21 +152,12 @@ def test_enable_toggle_allowed_for_admin(monkeypatch, tmp_path):
]
app = _make_app(system_role="admin")
# Not a real LocalSkillStorage instance, so _write_extensions_skill_state's
# projection-mutation branch is skipped (nullcontext) and it reads the
# config_path fresh via ExtensionsConfig.from_file.
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: SimpleNamespace(load_skills=_load_skills))
from deerflow.config.extensions_config import ExtensionsConfig
monkeypatch.setattr(
skills_router,
"get_extensions_config",
lambda: ExtensionsConfig(
mcp_servers={},
skills={},
middlewares=["pkg:Middleware"],
mcpInterceptors=["pkg.interceptor:build"],
),
)
monkeypatch.setattr(skills_router, "reload_extensions_config", lambda: None)
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(lambda: config_path))
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(lambda _config_path=None: config_path))
async def _refresh(_user_id: str):
return None

View File

@ -1,8 +1,8 @@
"""End-to-end tests for three-way skills mount across sandbox providers.
"""End-to-end tests for enabled-only skill mounts across sandbox providers.
Verifies that (a) public, (b) per-user custom, and (c) legacy global-custom
skills all resolve to correct container paths that the sandbox providers
actually mount covering ``LocalSandboxProvider`` and
Verifies that public, per-user custom, legacy global-custom, and managed
integration skills all resolve to correct container paths that the sandbox
providers actually mount covering ``LocalSandboxProvider`` and
``AioSandboxProvider`` (DooD / local-backend path).
Includes a full-pipeline test that exercises the actual path the model
@ -16,9 +16,12 @@ from unittest.mock import patch
import pytest
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig
from deerflow.config.paths import Paths
from deerflow.sandbox.local.local_sandbox import PathMapping
from deerflow.sandbox.local.local_sandbox_provider import LocalSandboxProvider
from deerflow.skills.projection import rebuild_skill_projections
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
from deerflow.skills.types import SKILL_MD_FILE, Skill, SkillCategory
_AIO_MODULE = "deerflow.community.aio_sandbox.aio_sandbox_provider"
@ -93,6 +96,32 @@ class TestThreeWayMountEndToEnd:
idx = _local_mounts(provider, "thread-1", user_id="user-1")
assert "/mnt/skills/public" in idx
assert idx["/mnt/skills/public"].read_only is True
assert Path(idx["/mnt/skills/public"].local_path) == paths.public_skills_view_dir
def test_local_acquire_recovers_public_mount_after_initial_projection_failure(self, skills_fs):
cfg = _build_config(skills_fs["root"])
paths = Paths(base_dir=skills_fs["users_dir"].parent)
projection = SimpleNamespace(
public=paths.public_skills_view_dir,
custom=paths.user_custom_skills_view_dir("user-1"),
legacy=paths.user_legacy_skills_view_dir("user-1"),
integrations=paths.user_integration_skills_view_dir("user-1"),
)
for root in (projection.public, projection.custom, projection.legacy, projection.integrations):
root.mkdir(parents=True, exist_ok=True)
with (
patch("deerflow.config.get_app_config", return_value=cfg),
patch("deerflow.config.paths.get_paths", return_value=paths),
patch.object(LocalSandboxProvider, "_ensure_skills_projection", side_effect=[OSError("transient"), projection]),
):
provider = LocalSandboxProvider()
sandbox_id = provider.acquire("thread-1", user_id="user-1")
sandbox = provider.get(sandbox_id)
mappings = {mapping.container_path: mapping for mapping in sandbox.path_mappings}
assert Path(mappings["/mnt/skills/public"].local_path) == projection.public
assert mappings["/mnt/skills/public"].read_only is True
def test_local_per_user_custom_skill_mounted(self, skills_fs):
cfg = _build_config(skills_fs["root"])
@ -101,7 +130,17 @@ class TestThreeWayMountEndToEnd:
provider = LocalSandboxProvider()
idx = _local_mounts(provider, "thread-1", user_id="user-1")
assert "/mnt/skills/custom" in idx
assert str(skills_fs["user_custom"]) in idx["/mnt/skills/custom"].local_path
assert Path(idx["/mnt/skills/custom"].local_path) == paths.user_custom_skills_view_dir("user-1")
def test_local_managed_integrations_use_per_user_projection(self, skills_fs):
cfg = _build_config(skills_fs["root"])
paths = Paths(base_dir=skills_fs["users_dir"].parent)
with patch("deerflow.config.get_app_config", return_value=cfg), patch("deerflow.config.paths.get_paths", return_value=paths):
provider = LocalSandboxProvider()
idx = _local_mounts(provider, "thread-1", user_id="user-1")
assert "/mnt/skills/integrations" in idx
assert Path(idx["/mnt/skills/integrations"].local_path) == paths.user_integration_skills_view_dir("user-1")
assert idx["/mnt/skills/integrations"].read_only is True
def test_local_legacy_mounted_for_user_without_custom(self, skills_fs):
cfg = _build_config(skills_fs["root"])
@ -110,7 +149,7 @@ class TestThreeWayMountEndToEnd:
provider = LocalSandboxProvider()
idx = _local_mounts(provider, "thread-1", user_id="noob")
assert "/mnt/skills/legacy" in idx
assert str(skills_fs["legacy_global"]) in idx["/mnt/skills/legacy"].local_path
assert Path(idx["/mnt/skills/legacy"].local_path) == paths.user_legacy_skills_view_dir("noob")
def test_local_legacy_not_mounted_when_user_has_custom(self, skills_fs):
cfg = _build_config(skills_fs["root"])
@ -118,7 +157,8 @@ class TestThreeWayMountEndToEnd:
with patch("deerflow.config.get_app_config", return_value=cfg), patch("deerflow.config.paths.get_paths", return_value=paths):
provider = LocalSandboxProvider()
idx = _local_mounts(provider, "thread-1", user_id="user-1")
assert "/mnt/skills/legacy" not in idx
assert "/mnt/skills/legacy" in idx
assert list(Path(idx["/mnt/skills/legacy"].local_path).iterdir()) == []
def test_local_legacy_still_mounted_when_user_has_only_non_skill_subdir(self, skills_fs):
(skills_fs["users_dir"] / "ghost" / "skills" / "custom" / "dangling-dir").mkdir(parents=True, exist_ok=True)
@ -204,6 +244,40 @@ class TestThreeWayMountEndToEnd:
assert cp == "/mnt/skills/legacy/leg-skill/SKILL.md"
assert "leg-skill" in sandbox_noob.read_file(cp)
def test_local_bash_observes_toggle_without_sandbox_recreation(self, tmp_path):
skills_root = tmp_path / "skills"
_write_skill(skills_root / "public", "secret-skill", "SECRET_PROCEDURE")
paths = Paths(base_dir=tmp_path)
cfg = _build_config(skills_root)
extensions = ExtensionsConfig(skills={"secret-skill": SkillStateConfig(enabled=False)})
with (
patch("deerflow.config.get_app_config", return_value=cfg),
patch("deerflow.config.paths.get_paths", return_value=paths),
patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=extensions),
patch("deerflow.config.extensions_config.get_extensions_config", return_value=extensions),
):
storage = UserScopedSkillStorage("user-1", host_path=str(skills_root), app_config=cfg)
rebuild_skill_projections(storage)
provider = LocalSandboxProvider()
sandbox_id = provider.acquire("thread-1", user_id="user-1")
sandbox = provider.get(sandbox_id)
assert sandbox is not None
disabled = sandbox.execute_command("cat /mnt/skills/public/secret-skill/SKILL.md")
assert "SECRET_PROCEDURE" not in disabled
extensions.skills["secret-skill"] = SkillStateConfig(enabled=True)
rebuild_skill_projections(storage)
enabled = sandbox.execute_command("cat /mnt/skills/public/secret-skill/SKILL.md")
assert "SECRET_PROCEDURE" in enabled
assert provider.acquire("thread-1", user_id="user-1") == sandbox_id
extensions.skills["secret-skill"] = SkillStateConfig(enabled=False)
rebuild_skill_projections(storage)
disabled_again = sandbox.execute_command("cat /mnt/skills/public/secret-skill/SKILL.md")
assert "SECRET_PROCEDURE" not in disabled_again
# ── AioSandboxProvider ──────────────────────────────────────────────
def test_aio_public_skill_mount(self, skills_fs, aio_mod):
@ -212,6 +286,8 @@ class TestThreeWayMountEndToEnd:
mounts = aio_mod.AioSandboxProvider._get_skills_mounts(user_id="user-1")
idx = {m[1]: m for m in mounts}
assert "/mnt/skills/public" in idx
host, _, _ = idx["/mnt/skills/public"]
assert "skills_view/public" in host.replace("\\", "/")
def test_aio_per_user_custom_skill_mount(self, skills_fs, aio_mod, monkeypatch):
cfg = _build_config(skills_fs["root"])
@ -222,7 +298,7 @@ class TestThreeWayMountEndToEnd:
idx = {m[1]: m for m in mounts}
assert "/mnt/skills/custom" in idx
host, _, _ = idx["/mnt/skills/custom"]
assert "users/user-1/skills/custom" in host.replace("\\", "/")
assert "users/user-1/skills_view/custom" in host.replace("\\", "/")
def test_aio_legacy_mounted_for_user_without_custom(self, skills_fs, aio_mod, monkeypatch):
cfg = _build_config(skills_fs["root"])
@ -240,7 +316,7 @@ class TestThreeWayMountEndToEnd:
with patch(_AIO_GET_CONFIG, return_value=cfg), patch("deerflow.config.paths.get_paths", return_value=paths):
mounts = aio_mod.AioSandboxProvider._get_skills_mounts(user_id="user-1")
idx = {m[1]: m for m in mounts}
assert "/mnt/skills/legacy" not in idx
assert "/mnt/skills/legacy" in idx
def test_aio_legacy_still_mounted_when_user_has_only_non_skill_subdir(self, skills_fs, aio_mod, monkeypatch):
(skills_fs["users_dir"] / "ghost" / "skills" / "custom" / "dangling-dir").mkdir(parents=True, exist_ok=True)
@ -286,7 +362,11 @@ class TestThreeWayMountEndToEnd:
assert "/mnt/skills/custom" in mount_entries
assert "dst=/mnt/skills/custom" in mount_entries["/mnt/skills/custom"]
assert "users/noob/skills/custom" in mount_entries["/mnt/skills/custom"]
assert "users/noob/skills_view/custom" in mount_entries["/mnt/skills/custom"]
assert "/mnt/skills/integrations" in mount_entries
assert "dst=/mnt/skills/integrations" in mount_entries["/mnt/skills/integrations"]
assert "users/noob/skills_view/integrations" in mount_entries["/mnt/skills/integrations"]
# noob has no per-user custom → legacy is mounted
assert "/mnt/skills/legacy" in mount_entries

View File

@ -539,11 +539,9 @@ class TestSkillLoadingRespectsGlobalDisable:
skills={"shared-skill": SimpleNamespace(enabled=False)},
is_skill_enabled=lambda name, _cat: not (name == "shared-skill"),
)
# The function inside ``load_skills`` does a
# function-local ``from deerflow.config.extensions_config
# import get_extensions_config``, so patch the
# extension_config module symbol.
with patch("deerflow.config.extensions_config.get_extensions_config", return_value=ext_cfg):
# User-scoped loading re-reads disk state so another
# worker's global disable is not hidden by a stale cache.
with patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=ext_cfg):
storage = get_or_new_user_skill_storage("alice", app_config=cfg)
loaded = storage.load_skills(enabled_only=False)
shared = [s for s in loaded if s.name == "shared-skill" and s.category == SkillCategory.LEGACY]

View File

@ -91,8 +91,6 @@ spec:
value: http://gateway:8001
- name: DEER_FLOW_HOST_BASE_DIR
value: /app/backend/.deer-flow
- name: DEER_FLOW_HOST_SKILLS_PATH
value: /app/skills
- name: GATEWAY_HOST
value: 0.0.0.0
- name: GATEWAY_PORT

View File

@ -57,7 +57,6 @@ services:
# On Docker Desktop/OrbStack, use your actual host paths like /Users/username/...
# Set these in your shell before running docker-compose:
# export DEER_FLOW_ROOT=/absolute/path/to/deer-flow
- SKILLS_HOST_PATH=${DEER_FLOW_ROOT}/skills
- THREADS_HOST_PATH=${DEER_FLOW_ROOT}/backend/.deer-flow/threads
# Per-user data base directory for user-scoped skill mounts
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_ROOT}/backend/.deer-flow
@ -201,7 +200,6 @@ services:
- DEER_FLOW_CHANNELS_GATEWAY_URL=${DEER_FLOW_CHANNELS_GATEWAY_URL:-http://gateway:8001}
- DEER_FLOW_INTERNAL_AUTH_TOKEN=${DEER_FLOW_INTERNAL_AUTH_TOKEN:-}
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_ROOT}/backend/.deer-flow
- DEER_FLOW_HOST_SKILLS_PATH=${DEER_FLOW_ROOT}/skills
- DEER_FLOW_SANDBOX_HOST=host.docker.internal
# Pass PROVISIONER_API_KEY into the gateway container so config.yaml can reference it
# as sandbox.provisioner_api_key: $PROVISIONER_API_KEY

View File

@ -127,7 +127,6 @@ services:
- DEER_FLOW_INTERNAL_AUTH_TOKEN=${DEER_FLOW_INTERNAL_AUTH_TOKEN}
# DooD path/network translation
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_HOME}
- DEER_FLOW_HOST_SKILLS_PATH=${DEER_FLOW_REPO_ROOT}/skills
- DEER_FLOW_SANDBOX_HOST=host.docker.internal
# Proxy values (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY) are inherited from ../.env via env_file.
# Only NO_PROXY is declared here so internal service hostnames are always exempt from the proxy.
@ -167,7 +166,6 @@ services:
# plaintext config/data are never mounted into the sandbox. Supersedes
# LARK_CLI_INIT_IMAGE when both are set. Empty ⇒ broker off.
- LARK_CLI_BROKER_IMAGE=${LARK_CLI_BROKER_IMAGE:-}
- SKILLS_HOST_PATH=${DEER_FLOW_REPO_ROOT}/skills
- THREADS_HOST_PATH=${DEER_FLOW_HOME}/threads
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_HOME}
- KUBECONFIG_PATH=/root/.kube/config

View File

@ -25,7 +25,7 @@ The **Sandbox Provisioner** is a FastAPI service that dynamically manages sandbo
2. **Pod Creation**: The provisioner creates a dedicated Pod in the `deer-flow` namespace with:
- The sandbox container image (all-in-one-sandbox)
- HostPath volumes mounted for:
- `/mnt/skills` → Read-only access to public skills
- `/mnt/skills/{public,custom,legacy}` → Read-only enabled-only skill projections
- `/mnt/user-data` → Read-write access to thread-specific data
- Resource limits (CPU, memory, ephemeral storage)
- Readiness/liveness probes
@ -153,8 +153,8 @@ The provisioner is configured via environment variables (set in [docker-compose-
| `SANDBOX_IMAGE` | `enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest` | AIO-compatible container image for sandbox Pods |
| `LARK_CLI_INIT_IMAGE` | empty (feature off) | Optional lark-cli init image (Pattern A). When set, sandbox Pods requesting the lark-cli runtime get an init container + shared `emptyDir` that provisions `lark-cli`, instead of a hostPath/PVC runtime mount. See [`docker/lark-cli-init`](../lark-cli-init/README.md) |
| `LARK_CLI_BROKER_IMAGE` | empty (feature off) | Optional lark-cli broker image (Pattern B, issue #4338). When set, sandbox Pods requesting the broker get a shim init container + a `lark-cli-broker` sidecar that holds the credentials; the plaintext `config`/`data` are mounted into the **sidecar only**, never the sandbox. Supersedes `LARK_CLI_INIT_IMAGE` when both are set. See [`docker/lark-cli-broker`](../lark-cli-broker/README.md) |
| `SKILLS_HOST_PATH` | - | **Host machine** path to skills directory (must be absolute) |
| `THREADS_HOST_PATH` | - | **Host machine** path to threads data directory (must be absolute) |
| `DEER_FLOW_HOST_BASE_DIR` | `/.deer-flow` | **Host machine** DeerFlow data root containing global and per-user `skills_view` projections |
| `SKILLS_PVC_NAME` | empty (use hostPath) | PVC name for skills volume; when set, sandbox Pods use PVC instead of hostPath |
| `SKILLS_PVC_SUBPATH_TEMPLATE` | empty | Optional `subPath` template for `SKILLS_PVC_NAME`. Supports `{user_id}` and `{thread_id}`. When empty, the skills PVC root is mounted unchanged |
| `USERDATA_PVC_NAME` | empty (use hostPath) | PVC name for user-data volume; when set, uses PVC with `subPath: deer-flow/users/{user_id}/threads/{thread_id}/user-data` |
@ -208,7 +208,9 @@ PYTHONPATH=. python scripts/migrate_user_isolation.py --user-id <target-user-id>
This moves legacy `threads/{thread_id}/user-data` data under `users/<target-user-id>/threads/{thread_id}/user-data`, which matches the new provisioner PVC subPath when the gateway base directory is mounted at `deer-flow/` on the PVC. Use `default` as the target user only when the legacy data should remain in the default no-auth user namespace. Run the migration while no gateway or sandbox Pods are writing to those paths.
When skills are materialized per thread on the same PVC, set `SKILLS_PVC_NAME` to that PVC and configure `SKILLS_PVC_SUBPATH_TEMPLATE=deer-flow/users/{user_id}/threads/{thread_id}/skills`. Leaving the template empty preserves the legacy behavior of mounting the skills PVC root at `/mnt/skills`.
In hostPath mode, the gateway materializes enabled-only views under `skills_view/public` and `users/{user_id}/skills_view/{custom,legacy}` beneath `DEER_FLOW_HOST_BASE_DIR`; the provisioner mounts those stable directories. When skills are materialized per thread on the same PVC, set `SKILLS_PVC_NAME` to that PVC and configure `SKILLS_PVC_SUBPATH_TEMPLATE=deer-flow/users/{user_id}/threads/{thread_id}/skills`. Leaving the template empty preserves the legacy behavior of mounting the skills PVC root at `/mnt/skills`. The gateway does not yet populate that PVC layout dynamically, so PVC-backed skills do not receive hostPath projection updates.
**hostPath skills volumes require the gateway and the K8s node to see the same `DEER_FLOW_HOST_BASE_DIR`** (single-node deployment, or NFS/shared storage mounted at that path on every node). The gateway writes the projection there before every sandbox acquire, so as long as that path is shared, the directory the provisioner mounts always exists by the time the Pod is scheduled — even a boot-time rebuild failure for one user self-heals on their next acquire, before the provisioner is called. `skills-custom` and `skills-legacy` use hostPath type `Directory` (not `DirectoryOrCreate`): if the shared-storage assumption is violated — the gateway wrote to a different node than the one the Pod lands on — Pod creation now fails visibly instead of silently mounting an empty directory. Use `SKILLS_PVC_NAME` instead of hostPath for genuinely multi-node clusters without shared storage.
### Important: K8S_API_SERVER Override
@ -248,7 +250,7 @@ kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
- Read Namespaces (to create `deer-flow` if missing)
4. **Host Paths**:
- The `SKILLS_HOST_PATH` and `THREADS_HOST_PATH` must be **absolute paths on the host machine**
- `DEER_FLOW_HOST_BASE_DIR` and `THREADS_HOST_PATH` must be **absolute paths on the host machine**
- These paths are mounted into sandbox Pods via K8s HostPath volumes
- The paths must exist and be readable by the K8s node
@ -350,7 +352,7 @@ docker exec deer-flow-gateway curl -s $SANDBOX_URL/v1/sandbox
**Cause**: HostPath volumes contain invalid paths (e.g., relative paths with `..`).
**Solution**:
- Use absolute paths for `SKILLS_HOST_PATH` and `THREADS_HOST_PATH`
- Use absolute paths for `DEER_FLOW_HOST_BASE_DIR` and `THREADS_HOST_PATH`
- Verify the paths exist on your host machine:
```bash
ls -la /path/to/skills

View File

@ -85,7 +85,6 @@ LARK_BROKER_SIDECAR_DATA_PATH = "/var/lark/data"
LARK_BROKER_CONFIG_VOLUME_NAME = "lark-cli-config"
LARK_BROKER_DATA_VOLUME_NAME = "lark-cli-data"
LARK_BROKER_URL = "http://127.0.0.1:8788"
SKILLS_HOST_PATH = os.environ.get("SKILLS_HOST_PATH", "/skills")
THREADS_HOST_PATH = os.environ.get("THREADS_HOST_PATH", "/.deer-flow/threads")
DEER_FLOW_HOST_BASE_DIR = os.environ.get("DEER_FLOW_HOST_BASE_DIR", "/.deer-flow")
SKILLS_PVC_NAME = os.environ.get("SKILLS_PVC_NAME", "")
@ -494,6 +493,7 @@ def _build_volumes(
``LocalSandboxProvider`` and ``AioSandboxProvider``.
"""
volumes: list[k8s_client.V1Volume] = []
del include_legacy_skills # retained for request compatibility
# ── Skills volumes ────────────────────────────────────────────────
@ -512,7 +512,7 @@ def _build_volumes(
)
else:
# hostPath mode: three-way layout
public_path = join_host_path(SKILLS_HOST_PATH, "public")
public_path = join_host_path(DEER_FLOW_HOST_BASE_DIR, "skills_view", "public")
volumes.append(
k8s_client.V1Volume(
name="skills-public",
@ -527,7 +527,7 @@ def _build_volumes(
DEER_FLOW_HOST_BASE_DIR,
"users",
user_id,
"skills",
"skills_view",
"custom",
)
volumes.append(
@ -535,22 +535,23 @@ def _build_volumes(
name="skills-custom",
host_path=k8s_client.V1HostPathVolumeSource(
path=user_custom_path,
type="DirectoryOrCreate",
type="Directory",
),
)
)
if include_legacy_skills:
legacy_path = join_host_path(SKILLS_HOST_PATH, "custom")
volumes.append(
k8s_client.V1Volume(
name="skills-legacy",
host_path=k8s_client.V1HostPathVolumeSource(
path=legacy_path,
type="Directory",
),
)
legacy_path = join_host_path(
DEER_FLOW_HOST_BASE_DIR, "users", user_id, "skills_view", "legacy"
)
volumes.append(
k8s_client.V1Volume(
name="skills-legacy",
host_path=k8s_client.V1HostPathVolumeSource(
path=legacy_path,
type="Directory",
),
)
)
# ── User-data volume ──────────────────────────────────────────────
@ -638,6 +639,7 @@ def _build_volume_mounts(
scope that mount with ``SKILLS_PVC_SUBPATH_TEMPLATE``.
"""
mounts: list[k8s_client.V1VolumeMount] = []
del include_legacy_skills # retained for request compatibility
if SKILLS_PVC_NAME:
skills_mount = k8s_client.V1VolumeMount(
@ -664,16 +666,13 @@ def _build_volume_mounts(
mount_path="/mnt/skills/custom",
read_only=True,
),
]
)
if include_legacy_skills:
mounts.append(
k8s_client.V1VolumeMount(
name="skills-legacy",
mount_path="/mnt/skills/legacy",
read_only=True,
)
)
),
]
)
userdata_mount = k8s_client.V1VolumeMount(
name="user-data",