mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-31 18:36:00 +00:00
fix(sandbox): enforce disabled skills in filesystem views (#4178)
* fix(sandbox): project enabled skills into sandbox views * fix(skills): keep projection mutations consistent * fix(skills): fail closed on projection errors * fix(skills): isolate per-scope failures during boot projection rebuild rebuild_all_skill_projections() propagated any exception from the public rebuild or from a single user's rebuild straight out of the gateway lifespan startup, uncaught. A single broken user directory (bad permissions, corrupted _skill_states.json, unreadable content) would therefore abort gateway boot for every user, not just that one - _rebuild_*_locked already fails closed internally (clears the view and re-raises), so the boot loop only needed to stop treating that re-raise as fatal. Each scope's rebuild now fails closed independently and boot continues; a scope left empty by a boot failure self-heals on the next sandbox acquire via ensure_skill_projections(). Also patches deerflow.skills.projection.rebuild_all_skill_projections in the memory-flush lifespan test fixture, matching the two sibling fixtures in the same file — this call is now on the lifespan startup path and the fixture's minimal SimpleNamespace config predates it. * test(skills): update authz test for the projection-aware public toggle _persist_shared_skill_state (introduced earlier in this branch) reads the shared extensions_config.json fresh from disk under the projection lock instead of through the cached get_extensions_config() singleton - that's the whole point of the fix (stale worker caches must not clobber another worker's concurrent update). The name no longer exists on the skills router module, so the test's monkeypatch of it started raising AttributeError instead of exercising the endpoint. The mock storage in this test isn't a real LocalSkillStorage instance, so _persist_shared_skill_state's projection-mutation branch is already skipped (nullcontext) and it falls back to a fresh ExtensionsConfig() for the nonexistent tmp config_path - no replacement monkeypatch needed. * fix(sandbox): make skill projection ensure best-effort in acquire acquire() called _ensure_skills_projection() directly, outside any try/except, in both LocalSandboxProvider and AioSandboxProvider. Every other skill-mount setup path in these providers has always caught exceptions and logged a warning rather than failing sandbox acquire outright (e.g. when config.yaml can't be resolved) - these two new call sites broke that contract, so any projection failure (including simply not having a config.yaml, as in CI's test environment) now failed acquire() itself instead of just leaving skill mounts off. _ensure_skills_projection now catches its own exceptions and returns None; both providers' callers already tolerate that (a None projection skips the skill-specific mounts, matching the existing degrade path) after making _append_public_skill_mapping and the custom/legacy mount block in LocalSandboxProvider explicitly None-safe. Caught by running the full suite with config.yaml removed, matching CI's environment - not caught locally because a real config.yaml was present, masking the failure. * fix(sandbox): make E2B skill projection mounts best-effort _skill_projection_mounts called ensure_skill_projections with no guard, unlike Local/AIO's _ensure_skills_projection. A raise propagated out of _apply_mounts before the configured-mounts loop ran, so a skills projection failure dropped the operator's own configured mounts too - only caught by create()'s outer warning, with nothing applied at all. Swallow here and return an empty mount list on failure, matching the Local/AIO pattern: still fail-closed for skills, but no longer widens the blast radius to unrelated configured mounts. Review feedback from PR #4178. * docs(skills): document projection trade-offs flagged in review - _update_tree_digest: note the metadata-only (not content) hashing trade-off and why runtime writes through this codebase are still covered regardless (rebuild-under-lock + rename always changes inode). - LocalSandboxProvider.acquire: note the acquire-time self-heal cost (cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift). - skill_projection_mutation: drop the no-op except-Exception-then-raise; a raise from the mutation already propagates past the yield with the view left cleared, no explicit re-raise needed. - provisioner README: spell out that hostPath skills volumes require the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node or shared storage), and that the custom/legacy volumes' hostPath type Directory (not DirectoryOrCreate) makes a violation of that assumption a visible Pod-creation failure instead of a silent empty mount. Review feedback from PR #4178. * fix(skills): lazily repair user projections * fix(skills): close projection review gaps * fix(skills): refresh user projection enable state * fix(skills): close projection review follow-ups * fix(skills): preserve state across projection writes --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
parent
486b51eb51
commit
f2e832330e
13
CHANGELOG.md
13
CHANGELOG.md
@ -12,6 +12,18 @@ This section accumulates work toward the **2.1.0** milestone
|
||||
|
||||
### ⚠ Breaking changes
|
||||
|
||||
- **skills:** Sandboxes now reserve `/mnt/skills` for managed enabled-only
|
||||
projections. `DEER_FLOW_HOST_SKILLS_PATH` and `SKILLS_HOST_PATH` are no longer
|
||||
used; Docker/AIO and hostPath deployments derive projection paths from
|
||||
`DEER_FLOW_HOST_BASE_DIR`. E2B operator mounts targeting `/mnt/skills` or any
|
||||
child path are skipped with a warning so they cannot shadow the managed
|
||||
projection; move extra E2B content to a different container path. User
|
||||
projections re-read global enable state from disk so toggles propagate across
|
||||
Gateway workers on the next sandbox acquire. Existing E2B sandboxes retain
|
||||
their creation-time snapshot until they are recreated. PVC-backed provisioner
|
||||
deployments still mount the operator-supplied PVC snapshot directly, so
|
||||
disabled-skill filesystem isolation does not apply in PVC mode until dynamic
|
||||
PVC materialization is implemented. ([#4178])
|
||||
- **sandbox:** E2B now enforces `sandbox.replicas` as a process-local capacity
|
||||
limit. The default `wait` policy waits for `acquire_timeout`, then fails the
|
||||
agent turn. DeerFlow does not retry the turn automatically. Use `burst` with
|
||||
@ -1264,6 +1276,7 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
||||
[#4170]: https://github.com/bytedance/deer-flow/pull/4170
|
||||
[#4171]: https://github.com/bytedance/deer-flow/pull/4171
|
||||
[#4174]: https://github.com/bytedance/deer-flow/pull/4174
|
||||
[#4178]: https://github.com/bytedance/deer-flow/pull/4178
|
||||
[#4181]: https://github.com/bytedance/deer-flow/pull/4181
|
||||
[#4187]: https://github.com/bytedance/deer-flow/pull/4187
|
||||
[#4188]: https://github.com/bytedance/deer-flow/pull/4188
|
||||
|
||||
@ -718,6 +718,8 @@ An enabled skill's `allowed-tools` policy applies only after that skill is expli
|
||||
|
||||
When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills.
|
||||
|
||||
Disabling a skill also removes it from the sandbox filesystem view, so shell commands and structured file tools follow the same enabled state. Local, Docker/AIO, hostPath provisioner, and newly created E2B sandboxes source `/mnt/skills` from enabled-only projections that update when public, custom, legacy, or managed integration skills are toggled, edited, created, deleted, or installed. Managed integration packages remain shared, while their projected filesystem visibility follows each user's enabled state. Multi-worker Gateways re-read on-disk enable state while rebuilding user projections, so a toggle handled by one worker is honored by another worker's next sandbox acquire. Existing E2B sandboxes retain their creation-time snapshot until they are recreated. PVC-backed provisioner skills keep their configured PVC snapshot/layout for now; dynamic PVC materialization is tracked separately.
|
||||
|
||||
Managed integrations install shared read-only skill packs without mixing them
|
||||
into custom skills. The Lark/Feishu CLI integration is available under
|
||||
`Settings → Integrations → Lark / Feishu CLI`; an administrator installs or
|
||||
|
||||
@ -608,9 +608,12 @@ that cannot tell sibling branches apart.
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop.
|
||||
**Environment policy** (`sandbox/env_policy.py`): `execute_command` no longer inherits the full `os.environ`. `build_sandbox_env()` scrubs secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. `OPENAI_API_KEY`) never leak into skill subprocesses. Benign vars (`PATH`, `HOME`, `LANG`, `VIRTUAL_ENV`, ...) are preserved.
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Legacy global-custom mounts are gated by the same user-scoped skill discovery rule used for prompt/list visibility; providers must not infer visibility from raw directory presence alone.
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. Legacy global-custom mounts follow the same shared visibility helper as local and remote providers. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support.
|
||||
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Public, custom, legacy, and managed integration skill mappings point at stable enabled-only projection roots rather than raw skill directories.
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. Local-container and hostPath-provisioner mounts use the same stable skill projection roots; PVC-backed skills remain governed by the operator-supplied PVC layout until PVC materialization is implemented. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support.
|
||||
- `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation.
|
||||
New sandboxes receive a one-shot upload from the enabled-only public, custom,
|
||||
legacy, and managed integration projections. Existing E2B VMs keep their
|
||||
creation-time snapshot because E2B has no shared host mount.
|
||||
Acquire and release share a per-user and thread lock. The provider lock does
|
||||
not cover remote IO. `burst_limit` adds capacity only for the `burst` policy.
|
||||
The `wait` policy fails the turn after `acquire_timeout`. The runtime does not
|
||||
@ -672,7 +675,7 @@ that cannot tell sibling branches apart.
|
||||
|
||||
**Virtual Path System**:
|
||||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`; raw skills stay under `deer-flow/skills/` and managed integration storage, while sandboxes read `backend/.deer-flow/skills_view/public/` and `backend/.deer-flow/users/{user_id}/skills_view/{custom,legacy,integrations}/`
|
||||
- Translation: `LocalSandboxProvider` builds per-thread `PathMapping`s for the user-data prefixes at acquire time; `tools.py` keeps `replace_virtual_path()` / `replace_virtual_paths_in_command()` as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept `/mnt/user-data/...` natively.
|
||||
- Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread)
|
||||
|
||||
@ -766,6 +769,7 @@ E2B output sync records remote file versions and actual host file metadata in a
|
||||
- **Loading**: `load_skills()` recursively scans public, per-user custom, global integration, and legacy custom locations for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json plus per-user skill state for non-public categories; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json.
|
||||
- **External reload**: `POST /api/skills/reload` is an admin-only, process-local invalidation hook for trusted MinIO/NFS/CSI writes. `SkillStorage` instances do not cache a catalog — `load_skills()` scans on every call — so the route clears all `(app_config, user_id)` entries and the rendered prompt-section LRU, then waits up to the shared refresh timeout for the existing off-loop single-flight refresh. Each invalidation receives a generation-bound result handle; a successful scan atomically replaces the global enabled-skills cache, while a loader-level failure propagates to the HTTP waiter and preserves the last-known-good global cache. Per-user/config scans capture the refresh version and cannot repopulate shared caches if invalidation occurs while they are loading. A timed-out HTTP wait fails generically while the daemon refresh worker continues. Subsequent runs rescan after a successful reload; active runs keep their existing snapshot. Each Uvicorn worker/Kubernetes Pod must be targeted separately. Direct mount writes bypass install/edit validation, SkillScan, and history, so mounted roots are an operator-controlled trust boundary.
|
||||
- **Tool policy**: Agent `allowed-tools` declarations apply dynamically only to slash-activated skills and skills captured in `ThreadState.skill_context` through configured `read_file` loads; passive enabled skills and custom-agent/subagent skill allowlists remain discoverable without clamping the baseline toolset. Subagents render only skill discovery metadata at startup and reuse the same adjacent `SkillActivationMiddleware` + `SkillToolPolicyMiddleware` pair as the lead; their configured `skills` field limits discovery and activation instead of eagerly loading bodies or unioning policies. Slash policy is dominant for its run, preventing subsequently read skills from widening explicit authority; autonomous captured skills use the existing union only when no slash source exists. `tool_search` and `describe_skill` stay available as framework discovery infrastructure, while every discovered or promoted business tool still requires active-policy permission for schema visibility and execution; `task` likewise requires an explicit declaration. Each active model call intentionally reloads the full live registry so enable/disable changes, frontmatter edits, and custom/public name-shadow winners take effect without a stale TTL or unsafe direct-path cache; all tool calls produced by that model step reuse the resulting source-and-path-signed decision. Registry failures and all-invalid active sets fail closed, while stale individual paths are skipped when another valid skill remains. This is best-effort behavioral scoping, not a hard security boundary: alternate loading paths are not captured and bounded autonomous context may evict entries.
|
||||
- **Sandbox projection**: `skills/projection.py` materializes enabled-only trees at `{base_dir}/skills_view/public` and `{base_dir}/users/{user_id}/skills_view/{custom,legacy,integrations}`. It hardlinks files when possible and falls back to copies across filesystems. Storage writes, archive installs, deletes, and toggles rebuild under a cross-process lock; Gateway boot ensures only the shared public view, while each user view is repaired lazily on first sandbox acquire. Managed integration packages are global, but their projected category is per-user because enabled state is isolated. Rebuilds stage a complete tree and reconcile it with per-file atomic replacement, so unrelated enabled skills remain continuously visible; disable/delete paths remove only the affected package before mutating to preserve fail-closed behavior. User projection rebuilds re-read global enable state from disk instead of the process singleton, so a toggle handled by another Gateway worker is reflected on the next acquire. Gateway public-skill toggles take the public projection lock before the shared `extensions_config_write_lock`, re-read an existing config from disk, persist the full model shape, and rebuild before responding; keep this as one worker-owned critical section so MCP writes cannot interleave and request cancellation cannot release either lock while the worker still runs. The shared public steady-state signature check runs without the global projection lock; stale/error paths take the lock and re-check before rebuilding or clearing. User-scope checks remain serialized per user. Category root inodes remain stable so live bind mounts observe content changes without sandbox recreation. Projection failures clear the affected view before raising.
|
||||
- **Injection (legacy / default)**: Enabled skills are listed in the agent system prompt with full metadata and container paths (`<available_skills>` block). Controlled by `skills.deferred_discovery: false` (default).
|
||||
- **Deferred discovery** (`skills.deferred_discovery: true`): Skills are listed by name only in a compact `<skill_index>` block, keeping the system prompt prefix-cache friendly. The agent calls the `describe_skill` tool at runtime to fetch full metadata for skills it wants to use, then loads the SKILL.md via `read_file`. Two new modules support this path:
|
||||
- `skills/catalog.py` — `SkillCatalog` (immutable, searchable; query forms: `select:a,b`, `+prefix`, free-text regex); `select:` returns all requested skills without a result cap; other modes cap at `MAX_RESULTS=5`.
|
||||
|
||||
@ -210,6 +210,12 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
config = get_gateway_config()
|
||||
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
|
||||
|
||||
from deerflow.skills.projection import ensure_public_skill_projection
|
||||
|
||||
public_projection_ready = await asyncio.to_thread(ensure_public_skill_projection, app_config=startup_config)
|
||||
if public_projection_ready:
|
||||
logger.info("Ensured the public skill projection; user projections repair lazily on sandbox acquire")
|
||||
|
||||
# Agent observability (Monocle). Off by default; enabled with
|
||||
# MONOCLE_TRACING. Initialized here at startup — not at import time — so a
|
||||
# plain `import deerflow.agents` never installs a process-global tracer.
|
||||
|
||||
@ -273,8 +273,9 @@ async def update_custom_skill(skill_name: str, body: CustomSkillUpdateRequest, r
|
||||
if scan.decision == "block":
|
||||
raise HTTPException(status_code=400, detail=f"Security scan blocked the edit: {scan.reason}")
|
||||
prev_content = storage.read_custom_skill(skill_name)
|
||||
storage.write_custom_skill(skill_name, SKILL_MD_FILE, body.content)
|
||||
storage.append_history(
|
||||
await asyncio.to_thread(storage.write_custom_skill, skill_name, SKILL_MD_FILE, body.content)
|
||||
await asyncio.to_thread(
|
||||
storage.append_history,
|
||||
skill_name,
|
||||
{
|
||||
"action": "human_edit",
|
||||
@ -305,7 +306,8 @@ async def delete_custom_skill(skill_name: str, request: Request, config: AppConf
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
storage = _get_user_skill_storage(config)
|
||||
storage.delete_custom_skill(
|
||||
await asyncio.to_thread(
|
||||
storage.delete_custom_skill,
|
||||
skill_name,
|
||||
history_meta={
|
||||
"action": "human_delete",
|
||||
@ -384,10 +386,10 @@ async def rollback_custom_skill(skill_name: str, body: SkillRollbackRequest, req
|
||||
"scanner": {"decision": scan.decision, "reason": scan.reason, "static_findings": static_findings},
|
||||
}
|
||||
if scan.decision == "block":
|
||||
storage.append_history(skill_name, history_entry)
|
||||
await asyncio.to_thread(storage.append_history, skill_name, history_entry)
|
||||
raise HTTPException(status_code=400, detail=f"Rollback blocked by security scanner: {scan.reason}")
|
||||
storage.write_custom_skill(skill_name, SKILL_MD_FILE, target_content)
|
||||
storage.append_history(skill_name, history_entry)
|
||||
await asyncio.to_thread(storage.write_custom_skill, skill_name, SKILL_MD_FILE, target_content)
|
||||
await asyncio.to_thread(storage.append_history, skill_name, history_entry)
|
||||
await refresh_user_skills_system_prompt_cache_async(get_effective_user_id())
|
||||
return await _read_custom_skill_response(skill_name, config)
|
||||
except HTTPException:
|
||||
@ -426,35 +428,47 @@ async def get_skill(skill_name: str, config: AppConfig = Depends(get_config)) ->
|
||||
raise HTTPException(status_code=500, detail=f"Failed to get skill: {str(e)}")
|
||||
|
||||
|
||||
def _write_extensions_skill_state(skill_name: str, enabled: bool) -> None:
|
||||
def _write_extensions_skill_state(
|
||||
storage: SkillStorage,
|
||||
skill_name: str,
|
||||
enabled: bool,
|
||||
*,
|
||||
rebuild_public_projection: bool,
|
||||
) -> None:
|
||||
"""Read-modify-write a skill's enabled state in the shared extensions_config.json.
|
||||
|
||||
Blocking filesystem IO: always call this via ``asyncio.to_thread``. It takes
|
||||
``extensions_config_write_lock`` itself, so that this router and the MCP
|
||||
router (which performs the same RMW on the same file) cannot interleave and
|
||||
drop each other's change. The lock is held by the worker rather than by the
|
||||
awaiting task, so cancelling the request cannot release it mid-write.
|
||||
the public projection lock before ``extensions_config_write_lock``. The first
|
||||
keeps the enabled-only view synchronized across workers; the second prevents
|
||||
this router and the MCP router from interleaving writes to the shared file.
|
||||
Both locks are held by the worker, so request cancellation cannot release
|
||||
either lock while the write or projection rebuild is still running.
|
||||
"""
|
||||
with extensions_config_write_lock:
|
||||
config_path = ExtensionsConfig.resolve_config_path()
|
||||
if config_path is None:
|
||||
config_path = Path.cwd().parent / "extensions_config.json"
|
||||
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
|
||||
from contextlib import nullcontext
|
||||
|
||||
# Work on a deep copy rather than the cached singleton: mutating the
|
||||
# singleton in place would publish the new state to readers before it is
|
||||
# durable on disk, and leave it applied even if the write below fails.
|
||||
# to_file_dict() serializes the full extensions_config.json shape (all
|
||||
# top-level keys), so no field is dropped from the file.
|
||||
extensions_config = get_extensions_config().model_copy(deep=True)
|
||||
extensions_config.skills[skill_name] = SkillStateConfig(enabled=enabled)
|
||||
from deerflow.skills.projection import skill_projection_mutation
|
||||
from deerflow.skills.storage.local_skill_storage import LocalSkillStorage
|
||||
|
||||
config_data = extensions_config.to_file_dict()
|
||||
removal_names = (skill_name,) if not enabled else ()
|
||||
projection_update = skill_projection_mutation(storage, "public", remove_names=removal_names) if rebuild_public_projection and isinstance(storage, LocalSkillStorage) else nullcontext()
|
||||
with projection_update:
|
||||
with extensions_config_write_lock:
|
||||
config_path = ExtensionsConfig.resolve_config_path()
|
||||
if config_path is None:
|
||||
config_path = Path.cwd().parent / "extensions_config.json"
|
||||
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
|
||||
|
||||
atomic_write_extensions_config(config_path, config_data)
|
||||
# The projection lock is cross-process, but the singleton cache is
|
||||
# not. Existing files are therefore re-read under the lock; a new
|
||||
# file starts from a deep snapshot of the cached defaults.
|
||||
extensions_config = ExtensionsConfig.from_file(config_path) if config_path.exists() else get_extensions_config().model_copy(deep=True)
|
||||
extensions_config.skills[skill_name] = SkillStateConfig(enabled=enabled)
|
||||
|
||||
logger.info(f"Skills configuration updated and saved to: {config_path}")
|
||||
reload_extensions_config()
|
||||
config_data = extensions_config.to_file_dict()
|
||||
atomic_write_extensions_config(config_path, config_data)
|
||||
|
||||
logger.info(f"Skills configuration updated and saved to: {config_path}")
|
||||
reload_extensions_config()
|
||||
|
||||
|
||||
@router.put(
|
||||
@ -488,10 +502,13 @@ async def update_skill(skill_name: str, body: SkillUpdateRequest, request: Reque
|
||||
# CUSTOM / LEGACY skills → per-user _skill_states.json (isolated state)
|
||||
# so that two users with same-named custom skills can toggle independently.
|
||||
if skill.category == SkillCategory.PUBLIC:
|
||||
# Shared-file RMW. The worker takes extensions_config_write_lock for
|
||||
# the whole read→write window, so it stays serialized against the MCP
|
||||
# router even if this request is cancelled mid-write.
|
||||
await asyncio.to_thread(_write_extensions_skill_state, skill_name, body.enabled)
|
||||
await asyncio.to_thread(
|
||||
_write_extensions_skill_state,
|
||||
storage,
|
||||
skill_name,
|
||||
body.enabled,
|
||||
rebuild_public_projection=True,
|
||||
)
|
||||
else:
|
||||
# CUSTOM / LEGACY: write per-user state
|
||||
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
|
||||
@ -500,8 +517,15 @@ async def update_skill(skill_name: str, body: SkillUpdateRequest, request: Reque
|
||||
await asyncio.to_thread(storage.set_skill_enabled_state, skill_name, body.enabled)
|
||||
else:
|
||||
# Fallback for non-user-scoped storage (unlikely in practice):
|
||||
# same shared-file RMW as the PUBLIC branch, same lock.
|
||||
await asyncio.to_thread(_write_extensions_skill_state, skill_name, body.enabled)
|
||||
# same shared-file RMW as the PUBLIC branch, without a public
|
||||
# projection rebuild for this non-public skill.
|
||||
await asyncio.to_thread(
|
||||
_write_extensions_skill_state,
|
||||
storage,
|
||||
skill_name,
|
||||
body.enabled,
|
||||
rebuild_public_projection=False,
|
||||
)
|
||||
|
||||
# PUBLIC skill enabled state lives in the global extensions_config.json
|
||||
# and affects every user, so the prompt cache for ALL users must be
|
||||
|
||||
@ -1281,13 +1281,19 @@ class DeerFlowClient:
|
||||
if config_path is None:
|
||||
raise FileNotFoundError("Cannot locate extensions_config.json. Set DEER_FLOW_EXTENSIONS_CONFIG_PATH or ensure it exists in the project root.")
|
||||
|
||||
extensions_config = get_extensions_config()
|
||||
extensions_config.skills[name] = SkillStateConfig(enabled=enabled)
|
||||
from deerflow.skills.projection import skill_projection_mutation
|
||||
|
||||
config_data = extensions_config.to_file_dict()
|
||||
removal_names = (name,) if not enabled else ()
|
||||
with skill_projection_mutation(storage, "public", remove_names=removal_names):
|
||||
# The projection lock is cross-process, but the singleton cache
|
||||
# is not. Reload from disk under the lock before this RMW.
|
||||
extensions_config = ExtensionsConfig.from_file(config_path)
|
||||
extensions_config.skills[name] = SkillStateConfig(enabled=enabled)
|
||||
|
||||
self._atomic_write_json(config_path, config_data)
|
||||
reload_extensions_config()
|
||||
config_data = extensions_config.to_file_dict()
|
||||
|
||||
self._atomic_write_json(config_path, config_data)
|
||||
reload_extensions_config()
|
||||
else:
|
||||
# CUSTOM / LEGACY: write per-user state
|
||||
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
|
||||
|
||||
@ -44,7 +44,6 @@ from deerflow.integrations.lark_cli import LARK_CLI_SANDBOX_CONFIG_DIR, LARK_CLI
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.sandbox.sandbox import Sandbox
|
||||
from deerflow.sandbox.sandbox_provider import SandboxProvider
|
||||
from deerflow.skills.storage import user_should_see_legacy_skills
|
||||
|
||||
from .aio_sandbox import AioSandbox
|
||||
from .backend import SandboxBackend, wait_for_sandbox_ready, wait_for_sandbox_ready_async
|
||||
@ -868,41 +867,34 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
||||
that ``Skill.get_container_path()`` category-aware paths resolve
|
||||
correctly inside the sandbox.
|
||||
|
||||
Mount sources use ``DEER_FLOW_HOST_SKILLS_PATH`` and
|
||||
``DEER_FLOW_HOST_BASE_DIR`` when running inside Docker (DooD) so the
|
||||
host Docker daemon can resolve the paths.
|
||||
Mount sources use ``DEER_FLOW_HOST_BASE_DIR`` when running inside
|
||||
Docker (DooD) so the host Docker daemon can resolve the projection
|
||||
paths.
|
||||
"""
|
||||
mounts: list[tuple[str, str, bool]] = []
|
||||
try:
|
||||
config = get_app_config()
|
||||
skills_path = config.skills.get_skills_path()
|
||||
container_path = config.skills.container_path
|
||||
|
||||
# When running inside Docker with DooD, use host-side skills path.
|
||||
host_skills_root = os.environ.get("DEER_FLOW_HOST_SKILLS_PATH") or str(skills_path)
|
||||
effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
|
||||
AioSandboxProvider._ensure_skills_projection(effective_user_id)
|
||||
paths = get_paths()
|
||||
host_base_dir = str(paths.host_base_dir)
|
||||
|
||||
# 1. Public skills: global, read-only — static, shared by all threads
|
||||
public_skills_path = skills_path / "public"
|
||||
if public_skills_path.exists():
|
||||
mounts.append(
|
||||
(
|
||||
join_host_path(host_skills_root, "public"),
|
||||
f"{container_path}/public",
|
||||
True,
|
||||
)
|
||||
mounts.append(
|
||||
(
|
||||
join_host_path(host_base_dir, "skills_view", "public"),
|
||||
f"{container_path}/public",
|
||||
True,
|
||||
)
|
||||
)
|
||||
|
||||
# 2. Per-user custom skills: read-only, per-thread/per-user
|
||||
effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
|
||||
paths = get_paths()
|
||||
user_custom_path = paths.user_custom_skills_dir(effective_user_id)
|
||||
user_custom_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
host_user_custom = join_host_path(
|
||||
str(paths.host_base_dir),
|
||||
host_base_dir,
|
||||
"users",
|
||||
effective_user_id,
|
||||
"skills",
|
||||
"skills_view",
|
||||
"custom",
|
||||
)
|
||||
mounts.append(
|
||||
@ -913,38 +905,66 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
||||
)
|
||||
)
|
||||
|
||||
# 3. Legacy (pre-migration global-custom) skills: only mount for
|
||||
# users who have no per-user custom skills yet, mirroring
|
||||
# ``UserScopedSkillStorage._iter_skill_files`` visibility rule.
|
||||
legacy_skills_path = skills_path / "custom"
|
||||
if user_should_see_legacy_skills(effective_user_id, host_path=str(skills_path)) and legacy_skills_path.exists():
|
||||
mounts.append(
|
||||
(
|
||||
join_host_path(host_skills_root, "custom"),
|
||||
f"{container_path}/legacy",
|
||||
True,
|
||||
)
|
||||
# 3. Legacy visibility is encoded by projection contents. Keep the
|
||||
# mount stable even when the directory is empty so a later state
|
||||
# change is visible without recreating the sandbox.
|
||||
mounts.append(
|
||||
(
|
||||
join_host_path(host_base_dir, "users", effective_user_id, "skills_view", "legacy"),
|
||||
f"{container_path}/legacy",
|
||||
True,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("Could not setup skills mounts: %s", e)
|
||||
|
||||
return mounts
|
||||
|
||||
@staticmethod
|
||||
def _ensure_skills_projection(user_id: str):
|
||||
"""Best-effort: a projection failure must not fail sandbox acquire.
|
||||
|
||||
Called directly (for its side effect) from ``_acquire_internal`` /
|
||||
``_acquire_internal_async`` outside any try/except, as well as from
|
||||
within ``_get_skills_mounts``'s own guarded block — swallowing here
|
||||
keeps both call sites safe without duplicating the guard.
|
||||
"""
|
||||
from deerflow.skills.projection import ensure_skill_projections
|
||||
from deerflow.skills.storage import get_or_new_user_skill_storage
|
||||
|
||||
try:
|
||||
storage = get_or_new_user_skill_storage(user_id, app_config=get_app_config())
|
||||
return ensure_skill_projections(storage)
|
||||
except Exception as exc:
|
||||
logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _get_user_skill_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]:
|
||||
"""Mount managed integration skills into AIO sandboxes.
|
||||
"""Mount enabled managed integration skills into AIO sandboxes.
|
||||
|
||||
Per-user custom skills are already mounted by ``_get_skills_mounts``.
|
||||
This helper adds the shared integration skill root so sandbox paths match
|
||||
the skill registry without duplicating ``/mnt/skills/custom``.
|
||||
Integration packages are shared, but their enabled state is per-user, so
|
||||
this helper mounts the user's projection instead of the raw shared root.
|
||||
"""
|
||||
try:
|
||||
config = get_app_config()
|
||||
paths = get_paths()
|
||||
skills_container_path = config.skills.container_path
|
||||
paths.integration_skills_dir().mkdir(parents=True, exist_ok=True)
|
||||
effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
|
||||
AioSandboxProvider._ensure_skills_projection(effective_user_id)
|
||||
return [
|
||||
(paths.host_integration_skills_dir(), f"{skills_container_path}/integrations", True),
|
||||
(
|
||||
join_host_path(
|
||||
str(paths.host_base_dir),
|
||||
"users",
|
||||
effective_user_id,
|
||||
"skills_view",
|
||||
"integrations",
|
||||
),
|
||||
f"{skills_container_path}/integrations",
|
||||
True,
|
||||
),
|
||||
]
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not setup user skill mounts: {e}")
|
||||
@ -1810,6 +1830,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
||||
sandbox_id is deterministic from thread_id so no shared state file
|
||||
is needed — any process can derive the same container name)
|
||||
"""
|
||||
self._ensure_skills_projection(user_id)
|
||||
cached_id = self._reuse_in_process_sandbox(thread_id, user_id=user_id)
|
||||
if cached_id is not None:
|
||||
return cached_id
|
||||
@ -1837,6 +1858,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
||||
|
||||
async def _acquire_internal_async(self, thread_id: str | None, *, user_id: str) -> str:
|
||||
"""Async counterpart to ``_acquire_internal``."""
|
||||
await asyncio.to_thread(self._ensure_skills_projection, user_id)
|
||||
cached_id = await asyncio.to_thread(self._reuse_in_process_sandbox, thread_id, user_id=user_id)
|
||||
if cached_id is not None:
|
||||
return cached_id
|
||||
|
||||
@ -1092,7 +1092,7 @@ class E2BSandboxProvider(SandboxProvider):
|
||||
# One-shot mount uploads. e2b has no host bind-mount, so we copy
|
||||
# files from ``host_path`` into ``container_path`` at sandbox start.
|
||||
try:
|
||||
self._apply_mounts(client)
|
||||
self._apply_mounts(client, user_id=user_id)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to apply some mounts to e2b sandbox %s: %s", sandbox_id, e)
|
||||
|
||||
@ -1717,11 +1717,42 @@ class E2BSandboxProvider(SandboxProvider):
|
||||
if exit_code not in (0, None) or "BOOTSTRAP_OK" not in stdout:
|
||||
raise RuntimeError(f"e2b bootstrap script failed with exit code {exit_code}; stderr={stderr.strip()}")
|
||||
|
||||
def _apply_mounts(self, client: E2BClientSandbox) -> None:
|
||||
mounts = self._config.get("mounts") or []
|
||||
if not mounts:
|
||||
return
|
||||
for mount in mounts:
|
||||
def _skill_projection_mounts(self, user_id: str) -> list[tuple[Path, str, bool]]:
|
||||
"""Best-effort: a projection failure must not drop configured mounts too.
|
||||
|
||||
Unlike Local/AIO's ``_ensure_skills_projection``, this used to raise
|
||||
straight out of ``_apply_mounts`` before the configured-mounts loop
|
||||
ran, so a projection hiccup dropped the operator's own mounts as
|
||||
collateral damage (only caught by ``create()``'s outer warning, with
|
||||
no mounts applied at all). Swallowing here keeps the two mount
|
||||
sources independent, matching the other two providers.
|
||||
"""
|
||||
from deerflow.skills.projection import ensure_skill_projections
|
||||
from deerflow.skills.storage import get_or_new_user_skill_storage
|
||||
|
||||
try:
|
||||
config = get_app_config()
|
||||
storage = get_or_new_user_skill_storage(user_id, app_config=config)
|
||||
projection = ensure_skill_projections(storage)
|
||||
container_root = config.skills.container_path.rstrip("/")
|
||||
return [
|
||||
(projection.public, f"{container_root}/public", True),
|
||||
(projection.custom, f"{container_root}/custom", True),
|
||||
(projection.legacy, f"{container_root}/legacy", True),
|
||||
(projection.integrations, f"{container_root}/integrations", True),
|
||||
]
|
||||
except Exception as exc:
|
||||
logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
|
||||
return []
|
||||
|
||||
def _apply_mounts(self, client: E2BClientSandbox, *, user_id: str | None = None) -> None:
|
||||
effective_user_id = user_id or get_effective_user_id()
|
||||
projection_mounts = self._skill_projection_mounts(effective_user_id)
|
||||
configured_mounts = self._config.get("mounts") or []
|
||||
skills_root = get_app_config().skills.container_path.rstrip("/")
|
||||
|
||||
mounts: list[tuple[Path, str, bool]] = list(projection_mounts)
|
||||
for mount in configured_mounts:
|
||||
try:
|
||||
host_path = Path(getattr(mount, "host_path", "") or "")
|
||||
container_path = (getattr(mount, "container_path", "") or "").rstrip("/")
|
||||
@ -1731,6 +1762,12 @@ class E2BSandboxProvider(SandboxProvider):
|
||||
container_path = (mount.get("container_path", "") or "").rstrip("/")
|
||||
read_only = bool(mount.get("read_only", False))
|
||||
|
||||
if container_path == skills_root or container_path.startswith(skills_root + "/"):
|
||||
logger.warning("Skipping e2b mount that conflicts with managed skills projection: %s", container_path)
|
||||
continue
|
||||
mounts.append((host_path, container_path, read_only))
|
||||
|
||||
for host_path, container_path, read_only in mounts:
|
||||
if not host_path.exists():
|
||||
logger.warning("Skipping e2b mount: host_path %s does not exist", host_path)
|
||||
continue
|
||||
|
||||
@ -258,6 +258,32 @@ class Paths:
|
||||
"""
|
||||
return self.base_dir / "integrations" / "skills"
|
||||
|
||||
@property
|
||||
def skills_view_dir(self) -> Path:
|
||||
"""Global sandbox-visible skills projection: ``{base_dir}/skills_view/``."""
|
||||
return self.base_dir / "skills_view"
|
||||
|
||||
@property
|
||||
def public_skills_view_dir(self) -> Path:
|
||||
"""Enabled public skills exposed to sandboxes."""
|
||||
return self.skills_view_dir / "public"
|
||||
|
||||
def user_skills_view_dir(self, user_id: str) -> Path:
|
||||
"""Per-user sandbox-visible skills projection root."""
|
||||
return self.user_dir(user_id) / "skills_view"
|
||||
|
||||
def user_custom_skills_view_dir(self, user_id: str) -> Path:
|
||||
"""Enabled custom skills exposed to one user's sandboxes."""
|
||||
return self.user_skills_view_dir(user_id) / "custom"
|
||||
|
||||
def user_legacy_skills_view_dir(self, user_id: str) -> Path:
|
||||
"""Enabled legacy skills exposed to one user's sandboxes."""
|
||||
return self.user_skills_view_dir(user_id) / "legacy"
|
||||
|
||||
def user_integration_skills_view_dir(self, user_id: str) -> Path:
|
||||
"""Enabled managed integration skills exposed to one user's sandboxes."""
|
||||
return self.user_skills_view_dir(user_id) / "integrations"
|
||||
|
||||
def thread_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
|
||||
"""
|
||||
Host path for a thread's data.
|
||||
|
||||
@ -6,7 +6,6 @@ from pathlib import Path
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox, PathMapping
|
||||
from deerflow.sandbox.sandbox import Sandbox
|
||||
from deerflow.sandbox.sandbox_provider import SandboxProvider
|
||||
from deerflow.skills.storage import user_should_see_legacy_skills
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -101,11 +100,11 @@ class LocalSandboxProvider(SandboxProvider):
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
config = get_app_config()
|
||||
skills_path = config.skills.get_skills_path()
|
||||
container_path = config.skills.container_path
|
||||
projection = self._ensure_skills_projection()
|
||||
|
||||
# Public skills: global, read-only — static, shared by all threads
|
||||
public_skills_path = skills_path / "public"
|
||||
public_skills_path = projection.public
|
||||
if public_skills_path.exists():
|
||||
mappings.append(
|
||||
PathMapping(
|
||||
@ -217,6 +216,52 @@ class LocalSandboxProvider(SandboxProvider):
|
||||
def _thread_key(thread_id: str, user_id: str) -> tuple[str, str]:
|
||||
return (user_id, thread_id)
|
||||
|
||||
@staticmethod
|
||||
def _ensure_skills_projection(user_id: str | None = None):
|
||||
"""Best-effort: a projection failure must not fail sandbox acquire.
|
||||
|
||||
Mirrors the surrounding skill-mount setup, which has always logged
|
||||
and continued rather than failing the whole acquire (e.g. missing
|
||||
config.yaml in a test double). Callers see ``None`` and skip the
|
||||
skill mounts for this acquire; the projection self-heals on a later
|
||||
acquire once the underlying condition clears.
|
||||
"""
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.skills.projection import ensure_skill_projections
|
||||
from deerflow.skills.storage import get_or_new_skill_storage, get_or_new_user_skill_storage
|
||||
|
||||
try:
|
||||
config = get_app_config()
|
||||
if user_id is None:
|
||||
storage = get_or_new_skill_storage(app_config=config)
|
||||
else:
|
||||
storage = get_or_new_user_skill_storage(user_id, app_config=config)
|
||||
return ensure_skill_projections(storage)
|
||||
except Exception as exc:
|
||||
logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _append_public_skill_mapping(mappings: list[PathMapping], projection) -> None:
|
||||
if projection is None:
|
||||
return
|
||||
try:
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
container_path = get_app_config().skills.container_path.rstrip("/")
|
||||
public_container_path = f"{container_path}/public"
|
||||
if any(mapping.container_path.rstrip("/") == public_container_path for mapping in mappings):
|
||||
return
|
||||
mappings.append(
|
||||
PathMapping(
|
||||
container_path=public_container_path,
|
||||
local_path=str(projection.public),
|
||||
read_only=True,
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Could not append public skill mapping: %s", exc, exc_info=True)
|
||||
|
||||
@staticmethod
|
||||
def _sandbox_id_for_thread(thread_id: str, user_id: str) -> str:
|
||||
return f"local:{user_id}:{thread_id}"
|
||||
@ -232,7 +277,7 @@ class LocalSandboxProvider(SandboxProvider):
|
||||
return (user_id, thread_id)
|
||||
|
||||
@staticmethod
|
||||
def _build_thread_path_mappings(thread_id: str, *, user_id: str | None = None) -> list[PathMapping]:
|
||||
def _build_thread_path_mappings(thread_id: str, *, user_id: str | None = None, skill_projection=None) -> list[PathMapping]:
|
||||
"""Build per-thread path mappings for /mnt/user-data, /mnt/acp-workspace,
|
||||
and /mnt/skills/custom.
|
||||
|
||||
@ -281,56 +326,35 @@ class LocalSandboxProvider(SandboxProvider):
|
||||
),
|
||||
]
|
||||
|
||||
# Per-user custom skills mount (read-only). This must be per-thread
|
||||
# because ``/mnt/skills/custom`` resolves to different host directories
|
||||
# for different users.
|
||||
# Per-user category mounts stay present for the sandbox lifetime. Their
|
||||
# enabled-only contents change beneath these stable roots.
|
||||
try:
|
||||
config = get_app_config()
|
||||
skills_container_path = config.skills.container_path
|
||||
user_custom_path = paths.user_custom_skills_dir(effective_user_id)
|
||||
integrations_path = paths.integration_skills_dir()
|
||||
user_custom_path.mkdir(parents=True, exist_ok=True)
|
||||
integrations_path.mkdir(parents=True, exist_ok=True)
|
||||
projection = skill_projection if skill_projection is not None else LocalSandboxProvider._ensure_skills_projection(effective_user_id)
|
||||
|
||||
mappings.append(
|
||||
PathMapping(
|
||||
container_path=f"{skills_container_path}/custom",
|
||||
local_path=str(user_custom_path),
|
||||
read_only=True,
|
||||
)
|
||||
)
|
||||
mappings.append(
|
||||
PathMapping(
|
||||
container_path=f"{skills_container_path}/integrations",
|
||||
local_path=str(integrations_path),
|
||||
read_only=True,
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Could not setup per-thread custom skills mount: %s", exc, exc_info=True)
|
||||
|
||||
# Legacy (pre-migration global-custom) skills: only mount for users
|
||||
# who have no per-user custom skills yet, mirroring the
|
||||
# ``UserScopedSkillStorage._iter_skill_files`` visibility rule. Users
|
||||
# with their own per-user custom skills cannot see LEGACY in the
|
||||
# listing/prompt and must not be able to read it via the sandbox
|
||||
# either — otherwise the listing layer and the sandbox layer disagree
|
||||
# about visibility, and the sandbox layer is the more permissive one.
|
||||
try:
|
||||
config = get_app_config()
|
||||
skills_container_path = config.skills.container_path
|
||||
user_custom_path = paths.user_custom_skills_dir(effective_user_id)
|
||||
legacy_skills_path = config.skills.get_skills_path() / "custom"
|
||||
if user_should_see_legacy_skills(effective_user_id, host_path=str(config.skills.get_skills_path())) and legacy_skills_path.exists():
|
||||
mappings.append(
|
||||
PathMapping(
|
||||
container_path=f"{skills_container_path}/legacy",
|
||||
local_path=str(legacy_skills_path),
|
||||
read_only=True,
|
||||
)
|
||||
if projection is not None:
|
||||
mappings.extend(
|
||||
[
|
||||
PathMapping(
|
||||
container_path=f"{skills_container_path}/custom",
|
||||
local_path=str(projection.custom),
|
||||
read_only=True,
|
||||
),
|
||||
PathMapping(
|
||||
container_path=f"{skills_container_path}/legacy",
|
||||
local_path=str(projection.legacy),
|
||||
read_only=True,
|
||||
),
|
||||
PathMapping(
|
||||
container_path=f"{skills_container_path}/integrations",
|
||||
local_path=str(projection.integrations),
|
||||
read_only=True,
|
||||
),
|
||||
]
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Could not setup per-thread legacy skills mount: %s", exc, exc_info=True)
|
||||
logger.warning("Could not setup per-thread skills projection mounts: %s", exc, exc_info=True)
|
||||
|
||||
return mappings
|
||||
|
||||
@ -350,13 +374,23 @@ class LocalSandboxProvider(SandboxProvider):
|
||||
global _singleton
|
||||
|
||||
if thread_id is None:
|
||||
skill_projection = self._ensure_skills_projection()
|
||||
with self._lock:
|
||||
if self._generic_sandbox is None:
|
||||
self._generic_sandbox = LocalSandbox("local", path_mappings=list(self._path_mappings))
|
||||
mappings = list(self._path_mappings)
|
||||
self._append_public_skill_mapping(mappings, skill_projection)
|
||||
self._generic_sandbox = LocalSandbox("local", path_mappings=mappings)
|
||||
_singleton = self._generic_sandbox
|
||||
return self._generic_sandbox.id
|
||||
|
||||
effective_user_id = self._effective_acquire_user_id(user_id)
|
||||
# Runs on every acquire, including cache hits, to self-heal drift —
|
||||
# cheap (~3-4 ms metadata walk) when the manifest is fresh. If another
|
||||
# worker mutated this user's skills since the last check, this
|
||||
# triggers a full rebuild (~400 ms measured locally) under the
|
||||
# cross-process projection lock, serializing concurrent acquires and
|
||||
# mutations for that user. Acceptable for an editing-frequency event.
|
||||
skill_projection = self._ensure_skills_projection(effective_user_id)
|
||||
key = self._thread_key(thread_id, effective_user_id)
|
||||
|
||||
# Fast path under lock.
|
||||
@ -366,11 +400,18 @@ class LocalSandboxProvider(SandboxProvider):
|
||||
# Mark as most-recently used so frequently-touched threads
|
||||
# survive eviction.
|
||||
self._thread_sandboxes.move_to_end(key)
|
||||
return cached.id
|
||||
if cached is not None:
|
||||
return cached.id
|
||||
|
||||
# ``_build_thread_path_mappings`` touches the filesystem
|
||||
# (``ensure_thread_dirs``); release the lock during I/O.
|
||||
new_mappings = list(self._path_mappings) + self._build_thread_path_mappings(thread_id, user_id=effective_user_id)
|
||||
new_mappings = list(self._path_mappings)
|
||||
self._append_public_skill_mapping(new_mappings, skill_projection)
|
||||
new_mappings += self._build_thread_path_mappings(
|
||||
thread_id,
|
||||
user_id=effective_user_id,
|
||||
skill_projection=skill_projection,
|
||||
)
|
||||
|
||||
with self._lock:
|
||||
# Re-check after the lock-free I/O: another caller may have
|
||||
|
||||
552
backend/packages/harness/deerflow/skills/projection.py
Normal file
552
backend/packages/harness/deerflow/skills/projection.py
Normal file
@ -0,0 +1,552 @@
|
||||
"""Materialize enabled-only skill trees for sandbox filesystem exposure."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import threading
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from deerflow.skills.parser import parse_skill_file
|
||||
from deerflow.skills.types import SKILL_MD_FILE, Skill, SkillCategory
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.skills.storage.skill_storage import SkillStorage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
import fcntl
|
||||
except ImportError: # pragma: no cover - Windows
|
||||
fcntl = None # type: ignore[assignment]
|
||||
import msvcrt
|
||||
|
||||
_locks_guard = threading.Lock()
|
||||
_process_locks: dict[Path, threading.RLock] = {}
|
||||
_MANIFEST_VERSION = 1
|
||||
_MAX_REBUILD_ATTEMPTS = 2
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SkillProjectionPaths:
|
||||
"""Stable category roots mounted or uploaded by sandbox providers."""
|
||||
|
||||
public: Path
|
||||
custom: Path
|
||||
legacy: Path
|
||||
integrations: Path
|
||||
|
||||
|
||||
def get_skill_projection_paths(storage: SkillStorage) -> SkillProjectionPaths:
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
paths = getattr(storage, "_paths", None) or get_paths()
|
||||
user_id = getattr(storage, "user_id", None)
|
||||
if user_id is None:
|
||||
return SkillProjectionPaths(
|
||||
public=paths.public_skills_view_dir,
|
||||
custom=paths.skills_view_dir / "custom",
|
||||
legacy=paths.skills_view_dir / "legacy",
|
||||
integrations=paths.skills_view_dir / "integrations",
|
||||
)
|
||||
return SkillProjectionPaths(
|
||||
public=paths.public_skills_view_dir,
|
||||
custom=paths.user_custom_skills_view_dir(user_id),
|
||||
legacy=paths.user_legacy_skills_view_dir(user_id),
|
||||
integrations=paths.user_integration_skills_view_dir(user_id),
|
||||
)
|
||||
|
||||
|
||||
def _lock_for(path: Path) -> threading.RLock:
|
||||
resolved = path.resolve()
|
||||
with _locks_guard:
|
||||
return _process_locks.setdefault(resolved, threading.RLock())
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _projection_lock(root: Path) -> Iterator[None]:
|
||||
"""Serialize projection replacement in-process and across POSIX workers."""
|
||||
lock_path = root.parent / f".{root.name}.projection.lock"
|
||||
lock_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
process_lock = _lock_for(lock_path)
|
||||
with process_lock, lock_path.open("a", encoding="utf-8") as lock_file:
|
||||
if fcntl is not None:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_EX)
|
||||
else: # pragma: no cover - Windows
|
||||
lock_file.seek(0)
|
||||
msvcrt.locking(lock_file.fileno(), msvcrt.LK_LOCK, 1)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
if fcntl is not None:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_UN)
|
||||
else: # pragma: no cover - Windows
|
||||
lock_file.seek(0)
|
||||
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
|
||||
|
||||
|
||||
def _link_or_copy(source: str, target: str, *, follow_symlinks: bool = True) -> str:
|
||||
# Hardlinks share the source inode and provide no write isolation. Any
|
||||
# read-only guarantee must come from the consuming sandbox or mount.
|
||||
try:
|
||||
os.link(source, target, follow_symlinks=follow_symlinks)
|
||||
except OSError as exc:
|
||||
if exc.errno not in {errno.EXDEV, errno.EPERM, errno.EACCES, errno.ENOTSUP}:
|
||||
raise
|
||||
shutil.copy2(source, target, follow_symlinks=follow_symlinks)
|
||||
return target
|
||||
|
||||
|
||||
def _stage_skill(source: Path, target: Path, nested_skill_roots: set[Path]) -> None:
|
||||
def _exclude_nested_skills(current: str, names: list[str]) -> list[str]:
|
||||
relative_root = Path(current).relative_to(source)
|
||||
return [name for name in names if relative_root / name in nested_skill_roots]
|
||||
|
||||
shutil.copytree(
|
||||
source,
|
||||
target,
|
||||
copy_function=_link_or_copy,
|
||||
symlinks=True,
|
||||
ignore=_exclude_nested_skills,
|
||||
dirs_exist_ok=True,
|
||||
)
|
||||
|
||||
|
||||
def _path_kind(path: Path) -> str:
|
||||
if path.is_symlink():
|
||||
return "symlink"
|
||||
if path.is_dir():
|
||||
return "directory"
|
||||
return "file"
|
||||
|
||||
|
||||
def _tree_entries(root: Path) -> dict[Path, str]:
|
||||
entries: dict[Path, str] = {}
|
||||
for current_root, dir_names, file_names in os.walk(root, followlinks=False):
|
||||
current = Path(current_root)
|
||||
for name in dir_names:
|
||||
path = current / name
|
||||
entries[path.relative_to(root)] = _path_kind(path)
|
||||
dir_names[:] = [name for name in dir_names if not (current / name).is_symlink()]
|
||||
for name in file_names:
|
||||
path = current / name
|
||||
entries[path.relative_to(root)] = _path_kind(path)
|
||||
return entries
|
||||
|
||||
|
||||
def _remove_projection_entry(path: Path) -> None:
|
||||
if path.is_dir() and not path.is_symlink():
|
||||
shutil.rmtree(path)
|
||||
else:
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _validate_projection_relative_path(relative_path: Path) -> None:
|
||||
if relative_path.is_absolute() or not relative_path.parts or any(part in {"", ".", ".."} for part in relative_path.parts):
|
||||
raise ValueError("Projection removal path must identify a package within its category root")
|
||||
|
||||
|
||||
def _remove_projection_relative(root: Path, relative_path: Path) -> None:
|
||||
"""Remove a projected package without following a drifted namespace symlink."""
|
||||
current = root
|
||||
for part in relative_path.parts:
|
||||
current /= part
|
||||
if current.is_symlink():
|
||||
current.unlink()
|
||||
return
|
||||
_remove_projection_entry(current)
|
||||
|
||||
|
||||
def _sync_staged_category(root: Path, staging: Path) -> None:
|
||||
desired = _tree_entries(staging)
|
||||
live = _tree_entries(root)
|
||||
|
||||
for relative_path, live_kind in sorted(live.items(), key=lambda item: len(item[0].parts), reverse=True):
|
||||
if desired.get(relative_path) != live_kind:
|
||||
_remove_projection_entry(root / relative_path)
|
||||
|
||||
for relative_path, kind in sorted(desired.items(), key=lambda item: len(item[0].parts)):
|
||||
if kind == "directory":
|
||||
(root / relative_path).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for relative_path, kind in desired.items():
|
||||
if kind == "directory":
|
||||
continue
|
||||
target = root / relative_path
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
(staging / relative_path).replace(target)
|
||||
|
||||
|
||||
def _replace_category(root: Path, desired: dict[Path, Skill], skill_boundaries: set[Path]) -> None:
|
||||
"""Reconcile entries beneath a stable category root without blanking it."""
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix=f".{root.name}.projection-", dir=root.parent) as staging_dir:
|
||||
staging = Path(staging_dir)
|
||||
for relative_path, skill in desired.items():
|
||||
nested_roots = {boundary.relative_to(relative_path) for boundary in skill_boundaries if boundary != relative_path and boundary.is_relative_to(relative_path)}
|
||||
_stage_skill(skill.skill_dir, staging / relative_path, nested_roots)
|
||||
_sync_staged_category(root, staging)
|
||||
|
||||
|
||||
def _clear_category(root: Path) -> None:
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
for path in root.iterdir():
|
||||
if path.is_dir() and not path.is_symlink():
|
||||
shutil.rmtree(path)
|
||||
else:
|
||||
path.unlink()
|
||||
|
||||
|
||||
def _clear_projection_scope(scope_root: Path, *category_roots: Path) -> None:
|
||||
for category_root in category_roots:
|
||||
_clear_category(category_root)
|
||||
_manifest_path(scope_root).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _update_tree_digest(digest, root: Path, label: str) -> None:
|
||||
"""Hash directory metadata (inode/mode/size/mtime), not file contents.
|
||||
|
||||
Trade-off: fast enough to run on every sandbox acquire (O(files), no
|
||||
reads), but an external edit that preserves inode+size+mtime — unlikely,
|
||||
not zero-probability — is invisible to this signature and leaves the
|
||||
projection stale until the next explicit rebuild. Runtime writes through
|
||||
this codebase are covered regardless: the mutation path rebuilds under
|
||||
lock, and atomic-rename always changes the inode.
|
||||
"""
|
||||
digest.update(f"root:{label}\0".encode())
|
||||
if not root.exists():
|
||||
digest.update(b"absent\0")
|
||||
return
|
||||
|
||||
stack = [(root, Path("."))]
|
||||
while stack:
|
||||
current, relative_root = stack.pop()
|
||||
with os.scandir(current) as entries:
|
||||
ordered = sorted(entries, key=lambda entry: entry.name)
|
||||
child_dirs: list[tuple[Path, Path]] = []
|
||||
for entry in ordered:
|
||||
relative = relative_root / entry.name
|
||||
metadata = entry.stat(follow_symlinks=False)
|
||||
if entry.is_symlink():
|
||||
kind = "link"
|
||||
elif entry.is_dir(follow_symlinks=False):
|
||||
kind = "dir"
|
||||
child_dirs.append((Path(entry.path), relative))
|
||||
else:
|
||||
kind = "file"
|
||||
digest.update((f"{label}:{relative.as_posix()}:{kind}:{metadata.st_ino}:{metadata.st_mode}:{metadata.st_size}:{metadata.st_mtime_ns}\0").encode())
|
||||
stack.extend(reversed(child_dirs))
|
||||
|
||||
|
||||
def _extensions_state() -> dict:
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
|
||||
config = ExtensionsConfig.from_file()
|
||||
return {name: state.model_dump(mode="json") for name, state in config.skills.items()}
|
||||
|
||||
|
||||
def _source_signature(storage: SkillStorage, scope: str) -> str:
|
||||
digest = hashlib.sha256()
|
||||
host_root = storage.get_skills_root_path()
|
||||
if scope == "public":
|
||||
_update_tree_digest(digest, host_root / SkillCategory.PUBLIC.value, "public")
|
||||
state = {"extensions": _extensions_state()}
|
||||
elif scope == "user":
|
||||
user_custom_root = storage.get_user_custom_root()
|
||||
integration_root = storage.get_user_integrations_root()
|
||||
_update_tree_digest(digest, user_custom_root, "custom")
|
||||
_update_tree_digest(digest, host_root / SkillCategory.CUSTOM.value, "legacy")
|
||||
_update_tree_digest(digest, integration_root, "integrations")
|
||||
# CUSTOM/LEGACY/INTEGRATION visibility is the intersection of the
|
||||
# per-user state and the global extensions default, so both belong in
|
||||
# this signature.
|
||||
state = {
|
||||
"extensions": _extensions_state(),
|
||||
"user": storage._read_skill_states(),
|
||||
}
|
||||
else: # pragma: no cover - internal invariant
|
||||
raise ValueError(f"Unknown skill projection scope: {scope}")
|
||||
digest.update(json.dumps(state, sort_keys=True, separators=(",", ":")).encode())
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _manifest_path(scope_root: Path) -> Path:
|
||||
return scope_root / ".projection-manifest.json"
|
||||
|
||||
|
||||
def _read_manifest(scope_root: Path) -> dict | None:
|
||||
try:
|
||||
value = json.loads(_manifest_path(scope_root).read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
return value if isinstance(value, dict) else None
|
||||
|
||||
|
||||
def _write_manifest(scope_root: Path, source_signature: str) -> None:
|
||||
scope_root.mkdir(parents=True, exist_ok=True)
|
||||
target = _manifest_path(scope_root)
|
||||
fd, temporary_name = tempfile.mkstemp(prefix=".projection-manifest-", suffix=".tmp", dir=scope_root)
|
||||
temporary = Path(temporary_name)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||
json.dump({"version": _MANIFEST_VERSION, "source_signature": source_signature}, stream, sort_keys=True)
|
||||
temporary.replace(target)
|
||||
except Exception:
|
||||
temporary.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
|
||||
def _load_public_skills(storage: SkillStorage, *, enabled_only: bool) -> list[Skill]:
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
|
||||
public_root = storage.get_skills_root_path() / SkillCategory.PUBLIC.value
|
||||
if not public_root.is_dir():
|
||||
return []
|
||||
extensions = ExtensionsConfig.from_file()
|
||||
skills: list[Skill] = []
|
||||
for current_root, dir_names, file_names in os.walk(public_root, followlinks=True):
|
||||
dir_names[:] = sorted(name for name in dir_names if not name.startswith("."))
|
||||
if SKILL_MD_FILE not in file_names:
|
||||
continue
|
||||
# Match the runtime loader: nested SKILL.md files inside a package are
|
||||
# support data, not independently configurable skills.
|
||||
dir_names.clear()
|
||||
skill_file = Path(current_root) / SKILL_MD_FILE
|
||||
skill = parse_skill_file(
|
||||
skill_file,
|
||||
category=SkillCategory.PUBLIC,
|
||||
relative_path=skill_file.parent.relative_to(public_root),
|
||||
)
|
||||
if skill is None:
|
||||
continue
|
||||
enabled = extensions.is_skill_enabled(skill.name, SkillCategory.PUBLIC.value)
|
||||
if not enabled_only or enabled:
|
||||
skills.append(skill)
|
||||
return skills
|
||||
|
||||
|
||||
def _by_relative_path(skills: list[Skill], category: SkillCategory) -> dict[Path, Skill]:
|
||||
return {skill.relative_path: skill for skill in skills if skill.category == category}
|
||||
|
||||
|
||||
def _category_boundaries(skills: list[Skill], category: SkillCategory) -> set[Path]:
|
||||
return {skill.relative_path for skill in skills if skill.category == category}
|
||||
|
||||
|
||||
def _rebuild_public_locked(storage: SkillStorage, paths: SkillProjectionPaths) -> None:
|
||||
scope_root = paths.public.parent
|
||||
try:
|
||||
for _attempt in range(_MAX_REBUILD_ATTEMPTS):
|
||||
before = _source_signature(storage, "public")
|
||||
all_public_skills = _load_public_skills(storage, enabled_only=False)
|
||||
enabled_public_skills = _load_public_skills(storage, enabled_only=True)
|
||||
_replace_category(
|
||||
paths.public,
|
||||
_by_relative_path(enabled_public_skills, SkillCategory.PUBLIC),
|
||||
_category_boundaries(all_public_skills, SkillCategory.PUBLIC),
|
||||
)
|
||||
after = _source_signature(storage, "public")
|
||||
if before == after:
|
||||
_write_manifest(scope_root, after)
|
||||
return
|
||||
raise RuntimeError("Public skills changed repeatedly while rebuilding the sandbox projection")
|
||||
except Exception:
|
||||
_clear_projection_scope(scope_root, paths.public)
|
||||
raise
|
||||
|
||||
|
||||
def _rebuild_user_locked(storage: SkillStorage, paths: SkillProjectionPaths) -> None:
|
||||
scope_root = paths.custom.parent
|
||||
try:
|
||||
for _attempt in range(_MAX_REBUILD_ATTEMPTS):
|
||||
before = _source_signature(storage, "user")
|
||||
all_user_skills = storage.load_skills(enabled_only=False)
|
||||
enabled_user_skills = [skill for skill in all_user_skills if skill.enabled]
|
||||
_replace_category(
|
||||
paths.custom,
|
||||
_by_relative_path(enabled_user_skills, SkillCategory.CUSTOM),
|
||||
_category_boundaries(all_user_skills, SkillCategory.CUSTOM),
|
||||
)
|
||||
_replace_category(
|
||||
paths.legacy,
|
||||
_by_relative_path(enabled_user_skills, SkillCategory.LEGACY),
|
||||
_category_boundaries(all_user_skills, SkillCategory.LEGACY),
|
||||
)
|
||||
_replace_category(
|
||||
paths.integrations,
|
||||
_by_relative_path(enabled_user_skills, SkillCategory.INTEGRATION),
|
||||
_category_boundaries(all_user_skills, SkillCategory.INTEGRATION),
|
||||
)
|
||||
after = _source_signature(storage, "user")
|
||||
if before == after:
|
||||
_write_manifest(scope_root, after)
|
||||
return
|
||||
raise RuntimeError("User skills changed repeatedly while rebuilding the sandbox projection")
|
||||
except Exception:
|
||||
_clear_projection_scope(scope_root, paths.custom, paths.legacy, paths.integrations)
|
||||
raise
|
||||
|
||||
|
||||
def rebuild_skill_projections(
|
||||
storage: SkillStorage,
|
||||
*,
|
||||
include_public: bool = True,
|
||||
include_user: bool = True,
|
||||
) -> SkillProjectionPaths:
|
||||
"""Rebuild enabled-only projection scopes visible through ``storage``."""
|
||||
paths = get_skill_projection_paths(storage)
|
||||
user_id = getattr(storage, "user_id", None)
|
||||
if include_public:
|
||||
with _projection_lock(paths.public.parent):
|
||||
_rebuild_public_locked(storage, paths)
|
||||
if include_user and user_id is not None:
|
||||
with _projection_lock(paths.custom.parent):
|
||||
_rebuild_user_locked(storage, paths)
|
||||
|
||||
return paths
|
||||
|
||||
|
||||
def _public_projection_is_fresh(storage: SkillStorage, paths: SkillProjectionPaths) -> bool:
|
||||
if not paths.public.is_dir():
|
||||
return False
|
||||
manifest_before = _read_manifest(paths.public.parent)
|
||||
if manifest_before is None or manifest_before.get("version") != _MANIFEST_VERSION:
|
||||
return False
|
||||
signature = _source_signature(storage, "public")
|
||||
manifest_after = _read_manifest(paths.public.parent)
|
||||
return manifest_before == manifest_after and manifest_before.get("source_signature") == signature
|
||||
|
||||
|
||||
def ensure_skill_projections(storage: SkillStorage) -> SkillProjectionPaths:
|
||||
"""Repair stale projection scopes, otherwise leave their inodes untouched."""
|
||||
paths = get_skill_projection_paths(storage)
|
||||
|
||||
try:
|
||||
public_is_fresh = _public_projection_is_fresh(storage, paths)
|
||||
except Exception:
|
||||
# Re-check under the mutation lock before failing closed. A concurrent
|
||||
# writer may have exposed a transient source/manifest state.
|
||||
public_is_fresh = False
|
||||
if not public_is_fresh:
|
||||
with _projection_lock(paths.public.parent):
|
||||
try:
|
||||
if not _public_projection_is_fresh(storage, paths):
|
||||
_rebuild_public_locked(storage, paths)
|
||||
except Exception:
|
||||
_clear_projection_scope(paths.public.parent, paths.public)
|
||||
raise
|
||||
|
||||
if getattr(storage, "user_id", None) is not None:
|
||||
with _projection_lock(paths.custom.parent):
|
||||
try:
|
||||
manifest = _read_manifest(paths.custom.parent)
|
||||
signature = _source_signature(storage, "user")
|
||||
if not paths.custom.is_dir() or not paths.legacy.is_dir() or not paths.integrations.is_dir() or manifest is None or manifest.get("version") != _MANIFEST_VERSION or manifest.get("source_signature") != signature:
|
||||
_rebuild_user_locked(storage, paths)
|
||||
except Exception:
|
||||
_clear_projection_scope(paths.custom.parent, paths.custom, paths.legacy, paths.integrations)
|
||||
raise
|
||||
return paths
|
||||
|
||||
|
||||
@contextmanager
|
||||
def skill_projection_mutation(
|
||||
storage: SkillStorage,
|
||||
scope: str,
|
||||
*,
|
||||
remove: tuple[tuple[SkillCategory, Path], ...] = (),
|
||||
remove_names: tuple[str, ...] = (),
|
||||
) -> Iterator[None]:
|
||||
"""Hold a projection scope lock across a source/state mutation."""
|
||||
if not isinstance(storage.get_skills_root_path(), Path):
|
||||
# Lightweight unit-test doubles sometimes return MagicMock here. The
|
||||
# SkillStorage contract requires a Path; real storage implementations
|
||||
# therefore never take this compatibility branch.
|
||||
yield
|
||||
return
|
||||
paths = get_skill_projection_paths(storage)
|
||||
if scope == "public":
|
||||
scope_root = paths.public.parent
|
||||
category_roots = {SkillCategory.PUBLIC: paths.public}
|
||||
|
||||
def rebuild() -> None:
|
||||
_rebuild_public_locked(storage, paths)
|
||||
|
||||
elif scope == "user":
|
||||
scope_root = paths.custom.parent
|
||||
category_roots = {
|
||||
SkillCategory.CUSTOM: paths.custom,
|
||||
SkillCategory.LEGACY: paths.legacy,
|
||||
SkillCategory.INTEGRATION: paths.integrations,
|
||||
}
|
||||
|
||||
def rebuild() -> None:
|
||||
_rebuild_user_locked(storage, paths)
|
||||
|
||||
else:
|
||||
raise ValueError(f"Unknown skill projection scope: {scope}")
|
||||
|
||||
removals: set[tuple[Path, Path]] = set()
|
||||
for category, relative_path in remove:
|
||||
root = category_roots.get(category)
|
||||
if root is None:
|
||||
raise ValueError(f"Skill category {category.value!r} does not belong to projection scope {scope!r}")
|
||||
_validate_projection_relative_path(relative_path)
|
||||
removals.add((root, relative_path))
|
||||
|
||||
with _projection_lock(scope_root):
|
||||
if remove_names:
|
||||
names = set(remove_names)
|
||||
skills = _load_public_skills(storage, enabled_only=False) if scope == "public" else storage.load_skills(enabled_only=False)
|
||||
for skill in skills:
|
||||
root = category_roots.get(skill.category)
|
||||
if skill.name not in names or root is None:
|
||||
continue
|
||||
_validate_projection_relative_path(skill.relative_path)
|
||||
removals.add((root, skill.relative_path))
|
||||
|
||||
try:
|
||||
_manifest_path(scope_root).unlink(missing_ok=True)
|
||||
for root, relative_path in removals:
|
||||
_remove_projection_relative(root, relative_path)
|
||||
yield
|
||||
rebuild()
|
||||
except Exception:
|
||||
_clear_projection_scope(scope_root, *category_roots.values())
|
||||
raise
|
||||
|
||||
|
||||
def ensure_public_skill_projection(*, app_config=None) -> bool:
|
||||
"""Ensure the global public view during boot without scanning user data.
|
||||
|
||||
User projections are repaired lazily by sandbox acquire. Eagerly rebuilding
|
||||
every historical user would make gateway readiness scale with tenant count,
|
||||
while providing no additional safety before that user's next acquire.
|
||||
"""
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.skills.storage import get_or_new_skill_storage
|
||||
|
||||
try:
|
||||
config = app_config or get_app_config()
|
||||
public_storage = get_or_new_skill_storage(app_config=config)
|
||||
ensure_skill_projections(public_storage)
|
||||
except Exception:
|
||||
logger.warning("Failed to ensure the public skill projection during boot; clearing it until a sandbox acquire self-heals it", exc_info=True)
|
||||
try:
|
||||
paths = get_paths()
|
||||
with _projection_lock(paths.public_skills_view_dir.parent):
|
||||
_clear_projection_scope(paths.public_skills_view_dir.parent, paths.public_skills_view_dir)
|
||||
except Exception:
|
||||
logger.error("Failed to clear the public skill projection after a boot-time error", exc_info=True)
|
||||
return False
|
||||
return True
|
||||
@ -10,6 +10,7 @@ import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from collections.abc import Iterable
|
||||
from contextlib import nullcontext
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
@ -107,8 +108,18 @@ class LocalSkillStorage(SkillStorage):
|
||||
) as tmp_file:
|
||||
tmp_file.write(content)
|
||||
tmp_path = Path(tmp_file.name)
|
||||
tmp_path.replace(target)
|
||||
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
|
||||
try:
|
||||
with self._skill_projection_mutation():
|
||||
tmp_path.replace(target)
|
||||
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
|
||||
except Exception:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
def remove_custom_skill_file(self, name: str, relative_path: str) -> str:
|
||||
removal = ((SkillCategory.CUSTOM, Path(name)),)
|
||||
with self._skill_projection_mutation(remove=removal):
|
||||
return super().remove_custom_skill_file(name, relative_path)
|
||||
|
||||
async def ainstall_skill_from_archive(self, archive_path: str | Path) -> dict:
|
||||
from deerflow.skills.installer import _scan_skill_archive_contents_or_raise
|
||||
@ -200,11 +211,12 @@ class LocalSkillStorage(SkillStorage):
|
||||
"""Stage and move the validated skill into place (blocking; runs off the event loop)."""
|
||||
from deerflow.skills.installer import _move_staged_skill_into_reserved_target
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix=f".installing-{skill_name}-", dir=custom_dir) as staging_root:
|
||||
staging_target = Path(staging_root) / skill_name
|
||||
shutil.copytree(skill_dir, staging_target)
|
||||
_move_staged_skill_into_reserved_target(staging_target, target)
|
||||
make_skill_written_path_sandbox_readable(custom_dir, target)
|
||||
with self._skill_projection_mutation():
|
||||
with tempfile.TemporaryDirectory(prefix=f".installing-{skill_name}-", dir=custom_dir) as staging_root:
|
||||
staging_target = Path(staging_root) / skill_name
|
||||
shutil.copytree(skill_dir, staging_target)
|
||||
_move_staged_skill_into_reserved_target(staging_target, target)
|
||||
make_skill_written_path_sandbox_readable(custom_dir, target)
|
||||
|
||||
def delete_custom_skill(self, name: str, *, history_meta: dict | None = None) -> None:
|
||||
self.validate_skill_name(name)
|
||||
@ -222,8 +234,22 @@ class LocalSkillStorage(SkillStorage):
|
||||
name,
|
||||
e,
|
||||
)
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
removal = ((SkillCategory.CUSTOM, Path(name)),)
|
||||
with self._skill_projection_mutation(remove=removal):
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
|
||||
def _skill_projection_mutation(
|
||||
self,
|
||||
*,
|
||||
remove: tuple[tuple[SkillCategory, Path], ...] = (),
|
||||
remove_names: tuple[str, ...] = (),
|
||||
):
|
||||
if getattr(self, "user_id", None) is None:
|
||||
return nullcontext()
|
||||
from deerflow.skills.projection import skill_projection_mutation
|
||||
|
||||
return skill_projection_mutation(self, "user", remove=remove, remove_names=remove_names)
|
||||
|
||||
def append_history(self, name: str, record: dict) -> None:
|
||||
self.validate_skill_name(name)
|
||||
|
||||
@ -155,6 +155,15 @@ class SkillStorage(ABC):
|
||||
Origin: ``deerflow.skills.manager.atomic_write``.
|
||||
"""
|
||||
|
||||
def remove_custom_skill_file(self, name: str, relative_path: str) -> str:
|
||||
"""Remove a supporting file and return its previous text content."""
|
||||
target = self.ensure_safe_support_path(name, relative_path)
|
||||
if not target.exists():
|
||||
raise FileNotFoundError(f"Supporting file '{relative_path}' not found for skill '{name}'.")
|
||||
previous_content = target.read_text(encoding="utf-8")
|
||||
target.unlink()
|
||||
return previous_content
|
||||
|
||||
@abstractmethod
|
||||
async def ainstall_skill_from_archive(self, archive_path: str | Path) -> dict:
|
||||
"""Async install of a skill from a ``.skill`` ZIP archive.
|
||||
|
||||
@ -82,6 +82,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
|
||||
|
||||
self._user_id = _validate_user_id(user_id)
|
||||
paths = get_paths()
|
||||
self._paths = paths
|
||||
self._user_custom_root: Path = paths.user_custom_skills_dir(self._user_id)
|
||||
self._integrations_root: Path = paths.integration_skills_dir()
|
||||
self._user_skills_root: Path = paths.user_skills_dir(self._user_id)
|
||||
@ -153,9 +154,11 @@ class UserScopedSkillStorage(LocalSkillStorage):
|
||||
|
||||
def set_skill_enabled_state(self, skill_name: str, enabled: bool) -> None:
|
||||
"""Set the enabled state for a custom/legacy skill and persist."""
|
||||
states = self._read_skill_states()
|
||||
states[skill_name] = {"enabled": enabled}
|
||||
self._write_skill_states(states)
|
||||
removal_names = (skill_name,) if not enabled else ()
|
||||
with self._skill_projection_mutation(remove_names=removal_names):
|
||||
states = self._read_skill_states()
|
||||
states[skill_name] = {"enabled": enabled}
|
||||
self._write_skill_states(states)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Path helpers — redirect custom skill paths to user directory
|
||||
@ -204,10 +207,12 @@ class UserScopedSkillStorage(LocalSkillStorage):
|
||||
# being silently re-enabled by an absent per-user entry, while still
|
||||
# letting the per-user state override the global default when both
|
||||
# are present. PUBLIC skill state remains governed solely by
|
||||
# extensions_config (handled by ``super().load_skills`` above).
|
||||
from deerflow.config.extensions_config import get_extensions_config
|
||||
# extensions_config (handled by ``super().load_skills`` above). Re-read
|
||||
# from disk here too so another worker's update cannot be masked by
|
||||
# this process's singleton cache while rebuilding a user projection.
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
|
||||
extensions_config = get_extensions_config()
|
||||
extensions_config = ExtensionsConfig.from_file()
|
||||
skills = [
|
||||
dataclasses.replace(s, enabled=self.get_skill_enabled_state(s.name) and extensions_config.is_skill_enabled(s.name, s.category.value if hasattr(s.category, "value") else s.category))
|
||||
if dataclasses.is_dataclass(s) and not isinstance(s, type) and (s.category.value if hasattr(s.category, "value") else s.category) != SkillCategory.PUBLIC.value
|
||||
@ -369,8 +374,13 @@ class UserScopedSkillStorage(LocalSkillStorage):
|
||||
) as tmp_file:
|
||||
tmp_file.write(content)
|
||||
tmp_path = Path(tmp_file.name)
|
||||
tmp_path.replace(target)
|
||||
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
|
||||
try:
|
||||
with self._skill_projection_mutation():
|
||||
tmp_path.replace(target)
|
||||
make_skill_written_path_sandbox_readable(self.get_custom_skill_dir(name), target)
|
||||
except Exception:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public helpers
|
||||
|
||||
@ -242,11 +242,7 @@ async def _skill_manage_impl(
|
||||
await _to_thread(skill_storage.ensure_custom_skill_is_editable, name)
|
||||
if path is None:
|
||||
raise ValueError("path is required for remove_file.")
|
||||
target = await _to_thread(skill_storage.ensure_safe_support_path, name, path)
|
||||
if not await _to_thread(target.exists):
|
||||
raise FileNotFoundError(f"Supporting file '{path}' not found for skill '{name}'.")
|
||||
prev_content = await _to_thread(target.read_text, encoding="utf-8")
|
||||
await _to_thread(target.unlink)
|
||||
prev_content = await _to_thread(skill_storage.remove_custom_skill_file, name, path)
|
||||
await _to_thread(
|
||||
skill_storage.append_history,
|
||||
name,
|
||||
|
||||
@ -119,6 +119,22 @@ async def test_update_skill_writes_from_snapshot_without_mutating_singleton(tmp_
|
||||
assert "middlewares" in written
|
||||
|
||||
|
||||
async def test_update_skill_persists_state_when_source_omits_skills(tmp_path: Path, monkeypatch) -> None:
|
||||
config_path = tmp_path / "extensions_config.json"
|
||||
await asyncio.to_thread(
|
||||
config_path.write_text,
|
||||
json.dumps({"mcpServers": {}, "middlewares": ["pkg:Middleware"]}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
_patch_config_infra(monkeypatch, config_path)
|
||||
|
||||
await update_skill("demo-skill", SkillUpdateRequest(enabled=False), _admin_request(), SimpleNamespace())
|
||||
|
||||
written = json.loads(await asyncio.to_thread(config_path.read_text, encoding="utf-8"))
|
||||
assert written["skills"] == {"demo-skill": {"enabled": False}}
|
||||
assert written["middlewares"] == ["pkg:Middleware"]
|
||||
|
||||
|
||||
@pytest.mark.allow_blocking_io # gate-exempt: needs real worker-thread overlap to observe serialization
|
||||
async def test_update_skill_serializes_concurrent_writes(tmp_path: Path, monkeypatch) -> None:
|
||||
state_lock = threading.Lock()
|
||||
|
||||
@ -219,8 +219,9 @@ def test_get_user_skill_mounts_mounts_only_global_integrations(tmp_path, monkeyp
|
||||
|
||||
assert set(alice) == {"/mnt/skills/integrations"}
|
||||
assert set(bob) == {"/mnt/skills/integrations"}
|
||||
assert alice["/mnt/skills/integrations"] == bob["/mnt/skills/integrations"]
|
||||
assert alice["/mnt/skills/integrations"] == str(tmp_path / "home" / "integrations" / "skills")
|
||||
assert alice["/mnt/skills/integrations"] != bob["/mnt/skills/integrations"]
|
||||
assert alice["/mnt/skills/integrations"] == str(tmp_path / "home" / "users" / "alice" / "skills_view" / "integrations")
|
||||
assert bob["/mnt/skills/integrations"] == str(tmp_path / "home" / "users" / "bob" / "skills_view" / "integrations")
|
||||
|
||||
|
||||
def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_path, monkeypatch, provisioner_module):
|
||||
@ -243,7 +244,7 @@ def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_pat
|
||||
monkeypatch.setattr(aio_mod, "get_app_config", lambda: config)
|
||||
monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=home))
|
||||
monkeypatch.setattr(aio_mod, "get_effective_user_id", lambda: "default")
|
||||
monkeypatch.setattr(aio_mod, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False)
|
||||
monkeypatch.setattr(remote_backend, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False)
|
||||
|
||||
provider = _make_provider(tmp_path)
|
||||
mounts = provider._get_extra_mounts("thread-1", user_id="alice")
|
||||
@ -526,7 +527,10 @@ async def test_acquire_internal_async_offloads_cached_reuse_health_check(tmp_pat
|
||||
sandbox_id = await provider._acquire_internal_async("thread-cached-async", user_id="default")
|
||||
|
||||
assert sandbox_id == "sandbox-cached-async"
|
||||
assert to_thread_calls == [(provider._reuse_in_process_sandbox, ("thread-cached-async",))]
|
||||
assert to_thread_calls == [
|
||||
(provider._ensure_skills_projection, ("default",)),
|
||||
(provider._reuse_in_process_sandbox, ("thread-cached-async",)),
|
||||
]
|
||||
|
||||
|
||||
def test_remote_backend_create_forwards_effective_user_id(monkeypatch):
|
||||
@ -548,7 +552,7 @@ def test_remote_backend_create_forwards_effective_user_id(monkeypatch):
|
||||
return _Response()
|
||||
|
||||
monkeypatch.setattr(remote_mod.requests, "post", _post)
|
||||
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda user_id: True)
|
||||
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda _user_id: True)
|
||||
|
||||
try:
|
||||
backend.create("thread-42", "sandbox-42")
|
||||
@ -585,7 +589,7 @@ def test_remote_backend_create_prefers_explicit_user_id(monkeypatch):
|
||||
|
||||
monkeypatch.setattr(remote_mod.requests, "post", _post)
|
||||
monkeypatch.setattr(remote_mod, "get_effective_user_id", lambda: "default")
|
||||
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda user_id: False)
|
||||
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda _user_id: False)
|
||||
|
||||
backend.create("thread-42", "sandbox-42", user_id="ou-user")
|
||||
|
||||
|
||||
@ -1861,10 +1861,8 @@ class TestSkillsManagement:
|
||||
skill = self._make_skill(enabled=True)
|
||||
updated_skill = self._make_skill(enabled=False)
|
||||
|
||||
ext_config = ExtensionsConfig()
|
||||
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
|
||||
json.dump({}, f)
|
||||
json.dump({"mcpServers": {}, "skills": {"untouched-skill": {"enabled": False}}}, f)
|
||||
tmp_path = Path(f.name)
|
||||
|
||||
try:
|
||||
@ -1881,12 +1879,37 @@ class TestSkillsManagement:
|
||||
side_effect=[[skill], [skill], [updated_skill], [updated_skill]],
|
||||
),
|
||||
patch("deerflow.client.ExtensionsConfig.resolve_config_path", return_value=tmp_path),
|
||||
patch("deerflow.client.get_extensions_config", return_value=ext_config),
|
||||
patch("deerflow.client.reload_extensions_config"),
|
||||
):
|
||||
result = client.update_skill("test-skill", enabled=False)
|
||||
assert result["enabled"] is False
|
||||
assert client._agent is None # M2: agent invalidated
|
||||
persisted = json.loads(tmp_path.read_text(encoding="utf-8"))
|
||||
assert persisted["skills"]["untouched-skill"] == {"enabled": False}
|
||||
finally:
|
||||
tmp_path.unlink()
|
||||
|
||||
def test_update_skill_persists_state_when_source_omits_skills(self, client):
|
||||
skill = self._make_skill(enabled=True)
|
||||
updated_skill = self._make_skill(enabled=False)
|
||||
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
|
||||
json.dump({"mcpServers": {}}, f)
|
||||
tmp_path = Path(f.name)
|
||||
|
||||
try:
|
||||
with (
|
||||
patch(
|
||||
"deerflow.skills.storage.local_skill_storage.LocalSkillStorage.load_skills",
|
||||
side_effect=[[skill], [skill], [updated_skill], [updated_skill]],
|
||||
),
|
||||
patch("deerflow.client.ExtensionsConfig.resolve_config_path", return_value=tmp_path),
|
||||
patch("deerflow.client.reload_extensions_config"),
|
||||
):
|
||||
client.update_skill("test-skill", enabled=False)
|
||||
|
||||
persisted = json.loads(tmp_path.read_text(encoding="utf-8"))
|
||||
assert persisted["skills"] == {"test-skill": {"enabled": False}}
|
||||
finally:
|
||||
tmp_path.unlink()
|
||||
|
||||
|
||||
@ -10,6 +10,7 @@ import threading
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
@ -313,6 +314,100 @@ def _install_fake_sdk(monkeypatch, provider) -> FakeSandboxClass:
|
||||
return fake_cls
|
||||
|
||||
|
||||
def _write_skill(root: Path, name: str) -> None:
|
||||
target = root / name / "SKILL.md"
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(f"---\nname: {name}\ndescription: test\n---\n", encoding="utf-8")
|
||||
|
||||
|
||||
def test_apply_mounts_uploads_only_enabled_skill_projection(monkeypatch, tmp_path):
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig
|
||||
|
||||
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
|
||||
paths = Paths(base_dir=tmp_path)
|
||||
skills_root = tmp_path / "skills"
|
||||
_write_skill(skills_root / "public", "enabled-skill")
|
||||
_write_skill(skills_root / "public", "disabled-skill")
|
||||
(skills_root / "custom").mkdir()
|
||||
_write_skill(paths.integration_skills_dir() / "lark-cli", "enabled-integration")
|
||||
_write_skill(paths.integration_skills_dir() / "lark-cli", "disabled-integration")
|
||||
user_skills_root = paths.user_skills_dir("user-1")
|
||||
user_skills_root.mkdir(parents=True, exist_ok=True)
|
||||
(user_skills_root / "_skill_states.json").write_text(
|
||||
json.dumps({"disabled-integration": {"enabled": False}}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
extensions = ExtensionsConfig(skills={"disabled-skill": SkillStateConfig(enabled=False)})
|
||||
config = SimpleNamespace(
|
||||
skills=SimpleNamespace(
|
||||
get_skills_path=lambda: skills_root,
|
||||
container_path="/mnt/skills",
|
||||
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr(mod, "get_app_config", lambda: config)
|
||||
monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: paths)
|
||||
monkeypatch.setattr("deerflow.config.extensions_config.ExtensionsConfig.from_file", lambda *_args, **_kwargs: extensions)
|
||||
monkeypatch.setattr("deerflow.config.extensions_config.get_extensions_config", lambda: extensions)
|
||||
|
||||
provider = _make_provider()
|
||||
client = FakeClient()
|
||||
provider._apply_mounts(client, user_id="user-1")
|
||||
|
||||
uploaded_paths = {path for path, _content in client.files.write_calls}
|
||||
assert "/mnt/skills/public/enabled-skill/SKILL.md" in uploaded_paths
|
||||
assert "/mnt/skills/public/disabled-skill/SKILL.md" not in uploaded_paths
|
||||
assert "/mnt/skills/integrations/lark-cli/enabled-integration/SKILL.md" in uploaded_paths
|
||||
assert "/mnt/skills/integrations/lark-cli/disabled-integration/SKILL.md" not in uploaded_paths
|
||||
|
||||
|
||||
def test_skill_projection_mounts_swallows_projection_failure(monkeypatch):
|
||||
"""``_skill_projection_mounts`` must not raise — a projection failure used
|
||||
to propagate out of ``_apply_mounts`` before the configured-mounts loop
|
||||
ran, dropping the operator's own configured mounts as collateral (#4107
|
||||
review)."""
|
||||
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
|
||||
|
||||
config = SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills"))
|
||||
monkeypatch.setattr(mod, "get_app_config", lambda: config)
|
||||
monkeypatch.setattr(
|
||||
"deerflow.skills.projection.ensure_skill_projections",
|
||||
lambda storage: (_ for _ in ()).throw(RuntimeError("simulated projection failure")),
|
||||
)
|
||||
|
||||
provider = _make_provider()
|
||||
|
||||
assert provider._skill_projection_mounts("user-1") == []
|
||||
|
||||
|
||||
def test_apply_mounts_keeps_configured_mounts_when_projection_fails(monkeypatch, tmp_path):
|
||||
"""End-to-end: a skills-projection failure must not drop the operator's
|
||||
own configured mounts too — the two mount sources are independent."""
|
||||
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
|
||||
|
||||
host_dir = tmp_path / "operator-mount"
|
||||
host_dir.mkdir()
|
||||
(host_dir / "notes.txt").write_text("hello", encoding="utf-8")
|
||||
|
||||
config = SimpleNamespace(skills=SimpleNamespace(container_path="/mnt/skills"))
|
||||
monkeypatch.setattr(mod, "get_app_config", lambda: config)
|
||||
monkeypatch.setattr(
|
||||
"deerflow.skills.projection.ensure_skill_projections",
|
||||
lambda storage: (_ for _ in ()).throw(RuntimeError("simulated projection failure")),
|
||||
)
|
||||
|
||||
provider = _make_provider()
|
||||
provider._config["mounts"] = [
|
||||
SimpleNamespace(host_path=str(host_dir), container_path="/mnt/operator", read_only=True),
|
||||
]
|
||||
|
||||
client = FakeClient()
|
||||
provider._apply_mounts(client, user_id="user-1")
|
||||
|
||||
uploaded_paths = {path for path, _content in client.files.write_calls}
|
||||
assert "/mnt/operator/notes.txt" in uploaded_paths
|
||||
|
||||
|
||||
def _make_sandbox(client: FakeClient, *, sandbox_id: str | None = None) -> Any:
|
||||
mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox")
|
||||
return mod.E2BSandbox(
|
||||
|
||||
@ -52,6 +52,7 @@ async def _run_lifespan_with_hanging_stop() -> float:
|
||||
patch("app.gateway.app.get_app_config", return_value=startup_config),
|
||||
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
|
||||
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
|
||||
patch("deerflow.skills.projection.ensure_public_skill_projection"),
|
||||
patch("app.gateway.app.auth.close_oidc_service", close_oidc_service),
|
||||
patch("app.channels.service.start_channel_service", side_effect=fake_start),
|
||||
patch("app.channels.service.stop_channel_service", side_effect=hang_forever),
|
||||
@ -98,6 +99,7 @@ async def _run_lifespan_with_upload_staging_cleanup():
|
||||
patch("app.gateway.app.get_app_config", return_value=startup_config),
|
||||
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
|
||||
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
|
||||
patch("deerflow.skills.projection.ensure_public_skill_projection"),
|
||||
patch("app.gateway.app.cleanup_stale_upload_staging_files", cleanup_upload_staging_files),
|
||||
patch("app.gateway.app.auth.close_oidc_service", close_oidc_service),
|
||||
patch("app.channels.service.start_channel_service", side_effect=fake_start),
|
||||
@ -156,6 +158,7 @@ async def _run_lifespan_with_memory_flush(*, enabled: bool, flush_return: bool |
|
||||
patch("app.gateway.app.get_app_config", return_value=startup_config),
|
||||
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
|
||||
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
|
||||
patch("deerflow.skills.projection.ensure_public_skill_projection"),
|
||||
patch("app.gateway.app.auth.close_oidc_service", close_oidc_service),
|
||||
patch("app.channels.service.start_channel_service", side_effect=fake_start),
|
||||
patch("app.channels.service.stop_channel_service", stop_channel_service),
|
||||
|
||||
@ -544,7 +544,7 @@ class TestMultipleMounts:
|
||||
|
||||
|
||||
class TestLocalSandboxProviderMounts:
|
||||
def test_thread_mappings_mount_global_integrations_for_every_user(self, tmp_path):
|
||||
def test_thread_mappings_mount_per_user_integration_projections(self, tmp_path):
|
||||
from deerflow.config.paths import Paths
|
||||
|
||||
paths = Paths(base_dir=tmp_path / "home")
|
||||
@ -568,10 +568,11 @@ class TestLocalSandboxProviderMounts:
|
||||
|
||||
alice_integrations = next(mapping for mapping in alice if mapping.container_path == "/mnt/skills/integrations")
|
||||
bob_integrations = next(mapping for mapping in bob if mapping.container_path == "/mnt/skills/integrations")
|
||||
expected = str(tmp_path / "home" / "integrations" / "skills")
|
||||
assert alice_integrations.local_path == expected
|
||||
assert bob_integrations.local_path == expected
|
||||
assert alice_integrations.local_path == str(paths.user_integration_skills_view_dir("alice"))
|
||||
assert bob_integrations.local_path == str(paths.user_integration_skills_view_dir("bob"))
|
||||
assert alice_integrations.local_path != bob_integrations.local_path
|
||||
assert alice_integrations.read_only is True
|
||||
assert bob_integrations.read_only is True
|
||||
|
||||
def test_setup_path_mappings_uses_configured_skills_container_path_as_reserved_prefix(self, tmp_path):
|
||||
skills_dir = tmp_path / "skills"
|
||||
|
||||
@ -339,6 +339,7 @@ def test_gateway_lifespan_initializes_monocle():
|
||||
patch("app.gateway.app.get_app_config", return_value=startup_config),
|
||||
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
|
||||
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
|
||||
patch("deerflow.skills.projection.ensure_public_skill_projection"),
|
||||
patch("app.gateway.app.setup_monocle_tracing_if_enabled", setup_spy),
|
||||
patch("app.gateway.app.auth.close_oidc_service", AsyncMock()),
|
||||
patch("app.channels.service.start_channel_service", side_effect=fake_start),
|
||||
@ -381,6 +382,7 @@ def test_gateway_lifespan_survives_monocle_setup_failure(caplog):
|
||||
patch("app.gateway.app.get_app_config", return_value=startup_config),
|
||||
patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
|
||||
patch("app.gateway.app.langgraph_runtime", _noop_langgraph_runtime),
|
||||
patch("deerflow.skills.projection.ensure_public_skill_projection"),
|
||||
patch("app.gateway.app.setup_monocle_tracing_if_enabled", setup_spy),
|
||||
patch("app.gateway.app.auth.close_oidc_service", AsyncMock()),
|
||||
patch("app.channels.service.start_channel_service", side_effect=fake_start),
|
||||
|
||||
@ -10,12 +10,12 @@ class TestBuildVolumes:
|
||||
|
||||
# ── hostPath mode (default) ────────────────────────────────────────
|
||||
|
||||
def test_hostpath_without_legacy_returns_three_volumes(self, provisioner_module):
|
||||
"""hostPath mode omits legacy volume unless the backend requests it."""
|
||||
def test_hostpath_uses_three_projection_volumes(self, provisioner_module):
|
||||
"""hostPath mode always mounts stable public/custom/legacy views."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
provisioner_module.USERDATA_PVC_NAME = ""
|
||||
volumes = provisioner_module._build_volumes("thread-1")
|
||||
assert len(volumes) == 3
|
||||
assert len(volumes) == 4
|
||||
|
||||
def test_hostpath_skills_public_volume(self, provisioner_module):
|
||||
"""First skills volume mounts public/ subdirectory."""
|
||||
@ -24,7 +24,7 @@ class TestBuildVolumes:
|
||||
pub = volumes[0]
|
||||
assert pub.name == "skills-public"
|
||||
assert pub.host_path is not None
|
||||
assert pub.host_path.path.endswith("/public")
|
||||
assert pub.host_path.path.endswith("/skills_view/public")
|
||||
assert pub.host_path.type == "Directory"
|
||||
assert pub.persistent_volume_claim is None
|
||||
|
||||
@ -35,11 +35,11 @@ class TestBuildVolumes:
|
||||
custom = volumes[1]
|
||||
assert custom.name == "skills-custom"
|
||||
assert custom.host_path is not None
|
||||
assert "users/user-7/skills/custom" in custom.host_path.path
|
||||
assert custom.host_path.type == "DirectoryOrCreate"
|
||||
assert "users/user-7/skills_view/custom" in custom.host_path.path
|
||||
assert custom.host_path.type == "Directory"
|
||||
|
||||
def test_hostpath_skills_legacy_volume(self, provisioner_module):
|
||||
"""Legacy global-custom directory is mounted only when requested."""
|
||||
"""Legacy projection is per-user and stable regardless of visibility."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
volumes = provisioner_module._build_volumes(
|
||||
"thread-1",
|
||||
@ -48,16 +48,17 @@ class TestBuildVolumes:
|
||||
legacy = volumes[2]
|
||||
assert legacy.name == "skills-legacy"
|
||||
assert legacy.host_path is not None
|
||||
assert legacy.host_path.path.endswith("/custom")
|
||||
assert "users/default/skills_view/legacy" in legacy.host_path.path
|
||||
assert legacy.host_path.type == "Directory"
|
||||
|
||||
def test_hostpath_without_legacy_has_no_legacy_volume(self, provisioner_module):
|
||||
"""Fresh installs should not require a missing global legacy directory."""
|
||||
def test_hostpath_without_legacy_flag_still_has_empty_capable_mount(self, provisioner_module):
|
||||
"""Visibility changes update contents without recreating the Pod."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
volumes = provisioner_module._build_volumes("thread-1")
|
||||
assert [volume.name for volume in volumes] == [
|
||||
"skills-public",
|
||||
"skills-custom",
|
||||
"skills-legacy",
|
||||
"user-data",
|
||||
]
|
||||
|
||||
@ -129,8 +130,8 @@ class TestBuildVolumes:
|
||||
|
||||
volumes = provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts)
|
||||
|
||||
# skills-public + skills-custom + user-data (3 base) + 1 extra volume.
|
||||
assert len(volumes) == 4
|
||||
# Three skill projections + user-data (4 base) + 1 extra volume.
|
||||
assert len(volumes) == 5
|
||||
extra_vol = volumes[-1]
|
||||
assert extra_vol.name == "extra-0"
|
||||
assert extra_vol.host_path.path == "/state/users/alice/integrations/lark-cli/config"
|
||||
@ -151,8 +152,8 @@ class TestBuildVolumes:
|
||||
|
||||
volumes = provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts)
|
||||
|
||||
# skills-public + skills-custom + user-data (3 base) + 1 extra volume.
|
||||
assert len(volumes) == 4
|
||||
# Three skill projections + user-data (4 base) + 1 extra volume.
|
||||
assert len(volumes) == 5
|
||||
extra_vol = volumes[-1]
|
||||
assert extra_vol.name == "extra-0"
|
||||
assert extra_vol.persistent_volume_claim is not None
|
||||
@ -184,12 +185,12 @@ class TestBuildVolumeMounts:
|
||||
|
||||
# ── hostPath mode ──────────────────────────────────────────────────
|
||||
|
||||
def test_hostpath_without_legacy_returns_three_mounts(self, provisioner_module):
|
||||
"""hostPath mode omits legacy mount unless the backend requests it."""
|
||||
def test_hostpath_uses_three_projection_mounts(self, provisioner_module):
|
||||
"""hostPath mode always mounts stable public/custom/legacy views."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
provisioner_module.USERDATA_PVC_NAME = ""
|
||||
mounts = provisioner_module._build_volume_mounts("thread-1")
|
||||
assert len(mounts) == 3
|
||||
assert len(mounts) == 4
|
||||
|
||||
def test_hostpath_skills_public_mount(self, provisioner_module):
|
||||
"""Public skills mount at /mnt/skills/public, read-only."""
|
||||
@ -218,13 +219,14 @@ class TestBuildVolumeMounts:
|
||||
assert mounts[2].mount_path == "/mnt/skills/legacy"
|
||||
assert mounts[2].read_only is True
|
||||
|
||||
def test_hostpath_without_legacy_has_no_legacy_mount(self, provisioner_module):
|
||||
"""Users with custom skills should not see hidden legacy content in the sandbox."""
|
||||
def test_hostpath_without_legacy_flag_still_has_legacy_mount(self, provisioner_module):
|
||||
"""Hidden legacy content is represented by an empty mounted view."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
mounts = provisioner_module._build_volume_mounts("thread-1")
|
||||
assert [mount.name for mount in mounts] == [
|
||||
"skills-public",
|
||||
"skills-custom",
|
||||
"skills-legacy",
|
||||
"user-data",
|
||||
]
|
||||
|
||||
@ -353,19 +355,19 @@ class TestBuildVolumeMounts:
|
||||
class TestBuildPodVolumes:
|
||||
"""Integration: _build_pod should wire volumes and mounts correctly."""
|
||||
|
||||
def test_pod_hostpath_without_legacy_has_three_volumes(self, provisioner_module):
|
||||
"""hostPath Pod spec should omit legacy volume by default."""
|
||||
def test_pod_hostpath_has_four_volumes(self, provisioner_module):
|
||||
"""hostPath Pod spec includes all three stable projection categories."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
provisioner_module.USERDATA_PVC_NAME = ""
|
||||
pod = provisioner_module._build_pod("sandbox-1", "thread-1")
|
||||
assert len(pod.spec.volumes) == 3
|
||||
assert len(pod.spec.volumes) == 4
|
||||
|
||||
def test_pod_hostpath_without_legacy_has_three_mounts(self, provisioner_module):
|
||||
"""hostPath container should omit legacy mount by default."""
|
||||
def test_pod_hostpath_has_four_mounts(self, provisioner_module):
|
||||
"""hostPath container includes all three stable projection categories."""
|
||||
provisioner_module.SKILLS_PVC_NAME = ""
|
||||
provisioner_module.USERDATA_PVC_NAME = ""
|
||||
pod = provisioner_module._build_pod("sandbox-1", "thread-1")
|
||||
assert len(pod.spec.containers[0].volume_mounts) == 3
|
||||
assert len(pod.spec.containers[0].volume_mounts) == 4
|
||||
|
||||
def test_pod_hostpath_with_legacy_has_four_volumes(self, provisioner_module):
|
||||
"""Legacy volume should be present when the backend requests it."""
|
||||
@ -438,8 +440,8 @@ class TestBuildPodVolumes:
|
||||
extra_mounts=extra_mounts,
|
||||
)
|
||||
|
||||
# skills-public + skills-custom + user-data (3 base) + 2 extra mounts.
|
||||
assert len(pod.spec.volumes) == 5
|
||||
# Three skill projections + user-data (4 base) + 2 extra mounts.
|
||||
assert len(pod.spec.volumes) == 6
|
||||
mount_paths = {mount.mount_path for mount in pod.spec.containers[0].volume_mounts}
|
||||
assert "/mnt/integrations/lark-cli/config" in mount_paths
|
||||
assert "/mnt/integrations/lark-cli/data" in mount_paths
|
||||
|
||||
@ -167,7 +167,7 @@ async def test_sandbox_business_routes_run_k8s_client_off_event_loop_thread(
|
||||
[
|
||||
(
|
||||
False,
|
||||
["skills-public", "skills-custom", "user-data"],
|
||||
["skills-public", "skills-custom", "skills-legacy", "user-data"],
|
||||
),
|
||||
(
|
||||
True,
|
||||
|
||||
@ -210,6 +210,48 @@ def test_skill_manage_rejects_support_path_traversal(monkeypatch, tmp_path):
|
||||
)
|
||||
|
||||
|
||||
def test_skill_manage_remove_file_updates_sandbox_projection_before_return(monkeypatch, tmp_path):
|
||||
skills_root = tmp_path / "skills"
|
||||
config = _make_config(skills_root)
|
||||
monkeypatch.setattr("deerflow.config.get_app_config", lambda: config)
|
||||
monkeypatch.setattr("deerflow.skills.security_scanner.get_app_config", lambda: config)
|
||||
from deerflow.config.paths import Paths
|
||||
|
||||
monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: Paths(base_dir=tmp_path))
|
||||
monkeypatch.setattr("deerflow.config.paths._paths", None)
|
||||
|
||||
async def _refresh(user_id: str):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(skill_manage_module, "refresh_user_skills_system_prompt_cache_async", _refresh)
|
||||
monkeypatch.setattr(skill_manage_module, "scan_skill_content", lambda *args, **kwargs: _async_result("allow", "ok"))
|
||||
|
||||
runtime = _make_runtime(user_id="default")
|
||||
anyio.run(skill_manage_module.skill_manage_tool.coroutine, runtime, "create", "demo-skill", _skill_content("demo-skill"))
|
||||
anyio.run(
|
||||
skill_manage_module.skill_manage_tool.coroutine,
|
||||
runtime,
|
||||
"write_file",
|
||||
"demo-skill",
|
||||
"supporting content",
|
||||
"references/guide.md",
|
||||
)
|
||||
projected_file = tmp_path / "users" / "default" / "skills_view" / "custom" / "demo-skill" / "references" / "guide.md"
|
||||
assert projected_file.read_text(encoding="utf-8") == "supporting content"
|
||||
|
||||
result = anyio.run(
|
||||
skill_manage_module.skill_manage_tool.coroutine,
|
||||
runtime,
|
||||
"remove_file",
|
||||
"demo-skill",
|
||||
None,
|
||||
"references/guide.md",
|
||||
)
|
||||
|
||||
assert result == "Removed 'references/guide.md' from custom skill 'demo-skill'."
|
||||
assert not projected_file.exists()
|
||||
|
||||
|
||||
def test_skill_manage_static_critical_blocks_create_before_llm(monkeypatch, tmp_path):
|
||||
skills_root = tmp_path / "skills"
|
||||
config = _make_config(skills_root)
|
||||
|
||||
612
backend/tests/test_skill_projection.py
Normal file
612
backend/tests/test_skill_projection.py
Normal file
@ -0,0 +1,612 @@
|
||||
"""Enabled-only filesystem projections exposed to sandbox providers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import shutil
|
||||
import zipfile
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from threading import Barrier, Event
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig
|
||||
from deerflow.config.paths import Paths
|
||||
from deerflow.skills.projection import ensure_public_skill_projection, ensure_skill_projections, rebuild_skill_projections, skill_projection_mutation
|
||||
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
|
||||
|
||||
|
||||
def _skill_content(name: str, marker: str = "v1") -> str:
|
||||
return f"---\nname: {name}\ndescription: {marker}\n---\n\n# {name}\n\n{marker}\n"
|
||||
|
||||
|
||||
def _write_skill(root: Path, name: str, marker: str = "v1") -> Path:
|
||||
target = root / name / "SKILL.md"
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(_skill_content(name, marker), encoding="utf-8")
|
||||
return target
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def projection_env(tmp_path: Path):
|
||||
skills_root = tmp_path / "skills"
|
||||
(skills_root / "public").mkdir(parents=True)
|
||||
(skills_root / "custom").mkdir()
|
||||
paths = Paths(base_dir=tmp_path)
|
||||
config = SimpleNamespace(
|
||||
skills=SimpleNamespace(
|
||||
get_skills_path=lambda: skills_root,
|
||||
container_path="/mnt/skills",
|
||||
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
|
||||
)
|
||||
)
|
||||
extensions = ExtensionsConfig()
|
||||
|
||||
with (
|
||||
patch("deerflow.config.paths.get_paths", return_value=paths),
|
||||
patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=extensions),
|
||||
patch("deerflow.config.extensions_config.get_extensions_config", return_value=extensions),
|
||||
):
|
||||
storage = UserScopedSkillStorage("alice", host_path=str(skills_root), app_config=config)
|
||||
yield SimpleNamespace(
|
||||
root=tmp_path,
|
||||
skills_root=skills_root,
|
||||
paths=paths,
|
||||
config=config,
|
||||
extensions=extensions,
|
||||
storage=storage,
|
||||
)
|
||||
|
||||
|
||||
def test_projection_contains_only_enabled_skills(projection_env) -> None:
|
||||
env = projection_env
|
||||
enabled = _write_skill(env.skills_root / "public", "enabled-skill")
|
||||
_write_skill(env.skills_root / "public", "disabled-skill")
|
||||
env.extensions.skills["disabled-skill"] = SkillStateConfig(enabled=False)
|
||||
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
|
||||
enabled_view = projected.public / "enabled-skill" / "SKILL.md"
|
||||
assert enabled_view.read_text(encoding="utf-8") == enabled.read_text(encoding="utf-8")
|
||||
assert not (projected.public / "disabled-skill").exists()
|
||||
|
||||
|
||||
def test_projection_rebuild_removes_newly_disabled_skill(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "demo-skill")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
assert (projected.public / "demo-skill" / "SKILL.md").is_file()
|
||||
|
||||
env.extensions.skills["demo-skill"] = SkillStateConfig(enabled=False)
|
||||
rebuild_skill_projections(env.storage)
|
||||
|
||||
assert not (projected.public / "demo-skill").exists()
|
||||
|
||||
|
||||
def test_nested_skill_frontmatter_is_supporting_data_inside_parent_package(projection_env) -> None:
|
||||
env = projection_env
|
||||
parent_root = env.skills_root / "public" / "parent-skill"
|
||||
_write_skill(env.skills_root / "public", "parent-skill")
|
||||
nested = _write_skill(parent_root / "fixtures", "nested-skill")
|
||||
env.extensions.skills["nested-skill"] = SkillStateConfig(enabled=False)
|
||||
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
nested_view = projected.public / nested.parent.relative_to(env.skills_root / "public")
|
||||
|
||||
assert (projected.public / "parent-skill" / "SKILL.md").is_file()
|
||||
assert (nested_view / "SKILL.md").is_file()
|
||||
|
||||
|
||||
def test_projection_falls_back_to_copy_when_hardlink_is_unavailable(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
source = _write_skill(env.skills_root / "public", "demo-skill")
|
||||
|
||||
def _cross_device_link(*_args, **_kwargs):
|
||||
raise OSError(errno.EXDEV, "cross-device link")
|
||||
|
||||
monkeypatch.setattr("deerflow.skills.projection.os.link", _cross_device_link)
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
target = projected.public / "demo-skill" / "SKILL.md"
|
||||
|
||||
assert target.read_text(encoding="utf-8") == source.read_text(encoding="utf-8")
|
||||
assert target.stat().st_ino != source.stat().st_ino
|
||||
|
||||
|
||||
def test_atomic_custom_skill_rewrite_refreshes_projection(projection_env) -> None:
|
||||
env = projection_env
|
||||
env.storage.write_custom_skill("demo-skill", "SKILL.md", _skill_content("demo-skill", "before"))
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
target = projected.custom / "demo-skill" / "SKILL.md"
|
||||
old_inode = target.stat().st_ino
|
||||
|
||||
env.storage.write_custom_skill("demo-skill", "SKILL.md", _skill_content("demo-skill", "after"))
|
||||
|
||||
assert "after" in target.read_text(encoding="utf-8")
|
||||
assert target.stat().st_ino != old_inode
|
||||
|
||||
|
||||
def test_custom_content_write_keeps_unrelated_skill_visible_during_rebuild(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha"))
|
||||
env.storage.write_custom_skill("beta", "SKILL.md", _skill_content("beta", "before"))
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
alpha_view = projected.custom / "alpha" / "SKILL.md"
|
||||
|
||||
from deerflow.skills import projection as projection_module
|
||||
|
||||
real_stage_skill = projection_module._stage_skill
|
||||
staging_started = Event()
|
||||
release_staging = Event()
|
||||
|
||||
def _delayed_stage_skill(*args, **kwargs):
|
||||
if not staging_started.is_set():
|
||||
staging_started.set()
|
||||
assert release_staging.wait(timeout=5)
|
||||
return real_stage_skill(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(projection_module, "_stage_skill", _delayed_stage_skill)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
write_future = executor.submit(env.storage.write_custom_skill, "beta", "SKILL.md", _skill_content("beta", "after"))
|
||||
assert staging_started.wait(timeout=5)
|
||||
unrelated_skill_remained_visible = alpha_view.is_file()
|
||||
release_staging.set()
|
||||
write_future.result(timeout=5)
|
||||
|
||||
assert unrelated_skill_remained_visible
|
||||
assert "after" in (projected.custom / "beta" / "SKILL.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_per_user_toggle_removes_custom_skill_before_returning(projection_env) -> None:
|
||||
env = projection_env
|
||||
env.storage.write_custom_skill("demo-skill", "SKILL.md", _skill_content("demo-skill"))
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
assert (projected.custom / "demo-skill" / "SKILL.md").is_file()
|
||||
|
||||
env.storage.set_skill_enabled_state("demo-skill", False)
|
||||
|
||||
assert not (projected.custom / "demo-skill").exists()
|
||||
|
||||
|
||||
def test_managed_integration_projection_is_filtered_per_user(projection_env) -> None:
|
||||
env = projection_env
|
||||
integration_root = env.paths.integration_skills_dir()
|
||||
_write_skill(integration_root / "lark-cli", "lark-doc")
|
||||
|
||||
alice_projection = rebuild_skill_projections(env.storage)
|
||||
alice_skill = alice_projection.integrations / "lark-cli" / "lark-doc" / "SKILL.md"
|
||||
assert alice_skill.is_file()
|
||||
|
||||
env.storage.set_skill_enabled_state("lark-doc", False)
|
||||
assert not alice_skill.exists()
|
||||
|
||||
bob_storage = UserScopedSkillStorage("bob", host_path=str(env.skills_root), app_config=env.config)
|
||||
bob_projection = rebuild_skill_projections(bob_storage)
|
||||
assert (bob_projection.integrations / "lark-cli" / "lark-doc" / "SKILL.md").is_file()
|
||||
|
||||
|
||||
def test_disabling_custom_skill_hides_only_target_while_rebuilding(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha"))
|
||||
env.storage.write_custom_skill("beta", "SKILL.md", _skill_content("beta"))
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
|
||||
from deerflow.skills import projection as projection_module
|
||||
|
||||
real_stage_skill = projection_module._stage_skill
|
||||
staging_started = Event()
|
||||
release_staging = Event()
|
||||
|
||||
def _delayed_stage_skill(*args, **kwargs):
|
||||
if not staging_started.is_set():
|
||||
staging_started.set()
|
||||
assert release_staging.wait(timeout=5)
|
||||
return real_stage_skill(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(projection_module, "_stage_skill", _delayed_stage_skill)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
disable_future = executor.submit(env.storage.set_skill_enabled_state, "beta", False)
|
||||
assert staging_started.wait(timeout=5)
|
||||
alpha_remained_visible = (projected.custom / "alpha" / "SKILL.md").is_file()
|
||||
beta_was_hidden = not (projected.custom / "beta").exists()
|
||||
release_staging.set()
|
||||
disable_future.result(timeout=5)
|
||||
|
||||
assert alpha_remained_visible
|
||||
assert beta_was_hidden
|
||||
|
||||
|
||||
def test_disabling_namespaced_skill_hides_its_real_projection_path(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "custom" / "team", "helper")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
target = projected.legacy / "team" / "helper"
|
||||
assert (target / "SKILL.md").is_file()
|
||||
|
||||
with skill_projection_mutation(env.storage, "user", remove_names=("helper",)):
|
||||
env.storage._write_skill_states({"helper": {"enabled": False}})
|
||||
assert not target.exists()
|
||||
|
||||
assert not target.exists()
|
||||
|
||||
|
||||
def test_disabling_duplicate_namespaced_public_skills_hides_every_path(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public" / "team-a", "helper")
|
||||
_write_skill(env.skills_root / "public" / "team-b", "helper")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
targets = [projected.public / team / "helper" for team in ("team-a", "team-b")]
|
||||
assert all((target / "SKILL.md").is_file() for target in targets)
|
||||
|
||||
with skill_projection_mutation(env.storage, "public", remove_names=("helper",)):
|
||||
env.extensions.skills["helper"] = SkillStateConfig(enabled=False)
|
||||
assert all(not target.exists() for target in targets)
|
||||
|
||||
assert all(not target.exists() for target in targets)
|
||||
|
||||
|
||||
def test_mutation_failure_clears_projection_scope(projection_env) -> None:
|
||||
env = projection_env
|
||||
env.storage.write_custom_skill("alpha", "SKILL.md", _skill_content("alpha"))
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
manifest = projected.custom.parent / ".projection-manifest.json"
|
||||
assert (projected.custom / "alpha" / "SKILL.md").is_file()
|
||||
|
||||
with pytest.raises(OSError, match="mutation failed"):
|
||||
with skill_projection_mutation(env.storage, "user"):
|
||||
raise OSError("mutation failed")
|
||||
|
||||
assert list(projected.custom.iterdir()) == []
|
||||
assert list(projected.legacy.iterdir()) == []
|
||||
assert list(projected.integrations.iterdir()) == []
|
||||
assert not manifest.exists()
|
||||
|
||||
|
||||
def test_targeted_removal_failure_clears_drifted_projection_scope(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public" / "team", "helper")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
manifest = projected.public.parent / ".projection-manifest.json"
|
||||
namespace = projected.public / "team"
|
||||
shutil.rmtree(namespace)
|
||||
namespace.write_text("drifted file", encoding="utf-8")
|
||||
|
||||
with pytest.raises(NotADirectoryError):
|
||||
with skill_projection_mutation(env.storage, "public", remove_names=("helper",)):
|
||||
env.extensions.skills["helper"] = SkillStateConfig(enabled=False)
|
||||
|
||||
assert list(projected.public.iterdir()) == []
|
||||
assert not manifest.exists()
|
||||
|
||||
|
||||
def test_user_custom_skill_replaces_legacy_projection(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "custom", "legacy-skill")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
assert (projected.legacy / "legacy-skill" / "SKILL.md").is_file()
|
||||
|
||||
env.storage.write_custom_skill("custom-skill", "SKILL.md", _skill_content("custom-skill"))
|
||||
|
||||
assert (projected.custom / "custom-skill" / "SKILL.md").is_file()
|
||||
assert list(projected.legacy.iterdir()) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("category", ["custom", "legacy"])
|
||||
def test_ensure_user_projection_uses_fresh_global_state_across_workers(projection_env, category: str) -> None:
|
||||
env = projection_env
|
||||
skill_name = f"{category}-skill"
|
||||
source_root = env.storage.get_user_custom_root() if category == "custom" else env.skills_root / "custom"
|
||||
_write_skill(source_root, skill_name)
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
target = getattr(projected, category) / skill_name / "SKILL.md"
|
||||
manifest = projected.custom.parent / ".projection-manifest.json"
|
||||
assert target.is_file()
|
||||
assert manifest.is_file()
|
||||
|
||||
fresh_extensions = ExtensionsConfig()
|
||||
fresh_extensions.skills[skill_name] = SkillStateConfig(enabled=False)
|
||||
with patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=fresh_extensions):
|
||||
ensure_skill_projections(env.storage)
|
||||
assert not target.exists()
|
||||
assert manifest.is_file()
|
||||
rebuilt_manifest = manifest.read_text(encoding="utf-8")
|
||||
|
||||
# The rebuilt manifest must not mark the stale visible view as fresh.
|
||||
ensure_skill_projections(env.storage)
|
||||
assert not target.exists()
|
||||
assert manifest.read_text(encoding="utf-8") == rebuilt_manifest
|
||||
|
||||
|
||||
def test_ensure_repairs_direct_atomic_source_replacement(projection_env) -> None:
|
||||
env = projection_env
|
||||
source = _write_skill(env.skills_root / "public", "demo-skill", "before")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
target = projected.public / "demo-skill" / "SKILL.md"
|
||||
old_projected_inode = target.stat().st_ino
|
||||
|
||||
replacement = source.with_suffix(".replacement")
|
||||
replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8")
|
||||
replacement.replace(source)
|
||||
ensure_skill_projections(env.storage)
|
||||
|
||||
assert "after" in target.read_text(encoding="utf-8")
|
||||
assert target.stat().st_ino != old_projected_inode
|
||||
|
||||
|
||||
def test_ensure_without_source_changes_keeps_projected_inode(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "demo-skill")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
target = projected.public / "demo-skill" / "SKILL.md"
|
||||
projected_inode = target.stat().st_ino
|
||||
|
||||
ensure_skill_projections(env.storage)
|
||||
|
||||
assert target.stat().st_ino == projected_inode
|
||||
|
||||
|
||||
def test_ensure_steady_state_public_signature_checks_do_not_serialize(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "demo-skill")
|
||||
rebuild_skill_projections(env.storage)
|
||||
|
||||
from deerflow.skills import projection as projection_module
|
||||
|
||||
real_source_signature = projection_module._source_signature
|
||||
public_signatures = Barrier(2, timeout=2)
|
||||
|
||||
def _synchronized_source_signature(storage, scope):
|
||||
if scope == "public":
|
||||
public_signatures.wait()
|
||||
return real_source_signature(storage, scope)
|
||||
|
||||
monkeypatch.setattr(projection_module, "_source_signature", _synchronized_source_signature)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
futures = [executor.submit(ensure_skill_projections, env.storage) for _ in range(2)]
|
||||
for future in futures:
|
||||
future.result(timeout=5)
|
||||
|
||||
|
||||
def test_unlocked_public_snapshot_detects_manifest_change_during_signature_scan(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "demo-skill")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
manifest = projected.public.parent / ".projection-manifest.json"
|
||||
|
||||
from deerflow.skills import projection as projection_module
|
||||
|
||||
real_source_signature = projection_module._source_signature
|
||||
signature_read = Event()
|
||||
release_signature = Event()
|
||||
|
||||
def _delayed_source_signature(storage, scope):
|
||||
signature = real_source_signature(storage, scope)
|
||||
if scope == "public":
|
||||
signature_read.set()
|
||||
assert release_signature.wait(timeout=5)
|
||||
return signature
|
||||
|
||||
monkeypatch.setattr(projection_module, "_source_signature", _delayed_source_signature)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
ensure_future = executor.submit(ensure_skill_projections, env.storage)
|
||||
assert signature_read.wait(timeout=5)
|
||||
manifest.unlink()
|
||||
release_signature.set()
|
||||
ensure_future.result(timeout=5)
|
||||
|
||||
assert manifest.is_file()
|
||||
|
||||
|
||||
def test_concurrent_stale_public_ensure_rebuilds_once_after_lock_recheck(projection_env) -> None:
|
||||
env = projection_env
|
||||
source = _write_skill(env.skills_root / "public", "demo-skill", "before")
|
||||
rebuild_skill_projections(env.storage)
|
||||
|
||||
replacement = source.with_suffix(".replacement")
|
||||
replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8")
|
||||
replacement.replace(source)
|
||||
|
||||
from deerflow.skills import projection as projection_module
|
||||
|
||||
with patch.object(projection_module, "_rebuild_public_locked", wraps=projection_module._rebuild_public_locked) as rebuild:
|
||||
with ThreadPoolExecutor(max_workers=8) as executor:
|
||||
list(executor.map(ensure_skill_projections, [env.storage] * 8))
|
||||
|
||||
assert rebuild.call_count == 1
|
||||
|
||||
|
||||
def test_rebuild_keeps_category_root_inode_stable(projection_env) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "demo-skill")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
root_inode = projected.public.stat().st_ino
|
||||
|
||||
env.extensions.skills["demo-skill"] = SkillStateConfig(enabled=False)
|
||||
rebuild_skill_projections(env.storage)
|
||||
|
||||
assert projected.public.stat().st_ino == root_inode
|
||||
|
||||
|
||||
def test_rebuild_failure_clears_old_projection(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
source = _write_skill(env.skills_root / "public", "demo-skill", "before")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
assert (projected.public / "demo-skill" / "SKILL.md").is_file()
|
||||
|
||||
replacement = source.with_suffix(".replacement")
|
||||
replacement.write_text(_skill_content("demo-skill", "after"), encoding="utf-8")
|
||||
replacement.replace(source)
|
||||
monkeypatch.setattr("deerflow.skills.projection._stage_skill", lambda *_args: (_ for _ in ()).throw(OSError("disk full")))
|
||||
|
||||
with pytest.raises(OSError, match="disk full"):
|
||||
ensure_skill_projections(env.storage)
|
||||
|
||||
assert list(projected.public.iterdir()) == []
|
||||
|
||||
|
||||
def test_signature_failure_clears_old_projection_and_manifest(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "demo-skill")
|
||||
projected = rebuild_skill_projections(env.storage)
|
||||
manifest = projected.public.parent / ".projection-manifest.json"
|
||||
assert (projected.public / "demo-skill" / "SKILL.md").is_file()
|
||||
assert manifest.is_file()
|
||||
|
||||
monkeypatch.setattr(
|
||||
"deerflow.skills.projection._source_signature",
|
||||
lambda *_args, **_kwargs: (_ for _ in ()).throw(PermissionError("source metadata unavailable")),
|
||||
)
|
||||
|
||||
with pytest.raises(PermissionError, match="source metadata unavailable"):
|
||||
ensure_skill_projections(env.storage)
|
||||
|
||||
assert list(projected.public.iterdir()) == []
|
||||
assert not manifest.exists()
|
||||
|
||||
|
||||
def test_boot_ensures_public_projection_without_scanning_known_users(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "public-skill")
|
||||
_write_skill(env.paths.user_custom_skills_dir("bob"), "custom-skill")
|
||||
|
||||
def _unexpected_user_storage(*_args, **_kwargs):
|
||||
raise AssertionError("gateway boot must not enumerate user skill storage")
|
||||
|
||||
monkeypatch.setattr("deerflow.skills.storage.get_or_new_user_skill_storage", _unexpected_user_storage)
|
||||
|
||||
assert ensure_public_skill_projection(app_config=env.config) is True
|
||||
|
||||
assert (env.paths.public_skills_view_dir / "public-skill" / "SKILL.md").is_file()
|
||||
assert not env.paths.user_custom_skills_view_dir("bob").exists()
|
||||
|
||||
bob_storage = UserScopedSkillStorage("bob", host_path=str(env.skills_root), app_config=env.config)
|
||||
ensure_skill_projections(bob_storage)
|
||||
assert (env.paths.user_custom_skills_view_dir("bob") / "custom-skill" / "SKILL.md").is_file()
|
||||
|
||||
|
||||
def test_boot_public_projection_failure_is_fail_closed_without_aborting(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "public-skill", "before")
|
||||
rebuild_skill_projections(env.storage, include_user=False)
|
||||
monkeypatch.setattr(
|
||||
"deerflow.skills.projection._source_signature",
|
||||
lambda *_args, **_kwargs: (_ for _ in ()).throw(PermissionError("source unavailable")),
|
||||
)
|
||||
|
||||
assert ensure_public_skill_projection(app_config=env.config) is False
|
||||
|
||||
assert list(env.paths.public_skills_view_dir.iterdir()) == []
|
||||
|
||||
|
||||
def test_boot_public_storage_factory_failure_is_fail_closed_without_aborting(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "public-skill")
|
||||
rebuild_skill_projections(env.storage, include_user=False)
|
||||
monkeypatch.setattr(
|
||||
"deerflow.skills.storage.get_or_new_skill_storage",
|
||||
lambda **_kwargs: (_ for _ in ()).throw(RuntimeError("storage init failed")),
|
||||
)
|
||||
|
||||
assert ensure_public_skill_projection(app_config=env.config) is False
|
||||
assert list(env.paths.public_skills_view_dir.iterdir()) == []
|
||||
|
||||
|
||||
def test_boot_factory_failure_cleanup_waits_for_concurrent_public_rebuild(projection_env, monkeypatch) -> None:
|
||||
env = projection_env
|
||||
_write_skill(env.skills_root / "public", "public-skill")
|
||||
from deerflow.skills import projection as projection_module
|
||||
|
||||
before_manifest = Event()
|
||||
release_rebuild = Event()
|
||||
cleanup_finished = Event()
|
||||
real_write_manifest = projection_module._write_manifest
|
||||
|
||||
def _delayed_write_manifest(scope_root, signature):
|
||||
before_manifest.set()
|
||||
assert release_rebuild.wait(timeout=5)
|
||||
real_write_manifest(scope_root, signature)
|
||||
|
||||
monkeypatch.setattr(projection_module, "_write_manifest", _delayed_write_manifest)
|
||||
monkeypatch.setattr(
|
||||
"deerflow.skills.storage.get_or_new_skill_storage",
|
||||
lambda **_kwargs: (_ for _ in ()).throw(RuntimeError("storage init failed")),
|
||||
)
|
||||
|
||||
def _rebuild() -> None:
|
||||
rebuild_skill_projections(env.storage, include_user=False)
|
||||
|
||||
def _fail_boot_ensure() -> None:
|
||||
assert ensure_public_skill_projection(app_config=env.config) is False
|
||||
cleanup_finished.set()
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
rebuild_future = executor.submit(_rebuild)
|
||||
assert before_manifest.wait(timeout=5)
|
||||
cleanup_future = executor.submit(_fail_boot_ensure)
|
||||
cleanup_waited_for_rebuild = not cleanup_finished.wait(timeout=0.2)
|
||||
release_rebuild.set()
|
||||
rebuild_future.result(timeout=5)
|
||||
cleanup_future.result(timeout=5)
|
||||
|
||||
assert cleanup_waited_for_rebuild
|
||||
assert list(env.paths.public_skills_view_dir.iterdir()) == []
|
||||
assert not (env.paths.public_skills_view_dir.parent / ".projection-manifest.json").exists()
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_archive_install_is_projected_before_return(projection_env, monkeypatch, tmp_path) -> None:
|
||||
from deerflow.skills.security_scanner import ScanResult
|
||||
|
||||
env = projection_env
|
||||
archive = tmp_path / "archive-skill.skill"
|
||||
with zipfile.ZipFile(archive, "w") as bundle:
|
||||
bundle.writestr("archive-skill/SKILL.md", _skill_content("archive-skill"))
|
||||
bundle.writestr("archive-skill/references/guide.md", "# Guide\n")
|
||||
|
||||
async def _allow_scan(*_args, **_kwargs):
|
||||
return ScanResult(decision="allow", reason="test")
|
||||
|
||||
monkeypatch.setattr("deerflow.skills.installer.scan_skill_content", _allow_scan)
|
||||
|
||||
result = await env.storage.ainstall_skill_from_archive(archive)
|
||||
|
||||
projected = env.paths.user_custom_skills_view_dir("alice") / "archive-skill"
|
||||
assert result["success"] is True
|
||||
assert (projected / "SKILL.md").is_file()
|
||||
assert (projected / "references" / "guide.md").read_text(encoding="utf-8") == "# Guide\n"
|
||||
|
||||
|
||||
def test_concurrent_custom_skill_writes_do_not_lose_projected_entries(projection_env) -> None:
|
||||
env = projection_env
|
||||
names = [f"skill-{index}" for index in range(8)]
|
||||
|
||||
def _write(name: str) -> None:
|
||||
env.storage.write_custom_skill(name, "SKILL.md", _skill_content(name))
|
||||
|
||||
with ThreadPoolExecutor(max_workers=4) as executor:
|
||||
list(executor.map(_write, names))
|
||||
|
||||
projected_names = {path.name for path in env.paths.user_custom_skills_view_dir("alice").iterdir()}
|
||||
assert projected_names == set(names)
|
||||
|
||||
|
||||
def test_concurrent_custom_skill_toggles_do_not_lose_state(projection_env) -> None:
|
||||
env = projection_env
|
||||
names = ("skill-a", "skill-b")
|
||||
for name in names:
|
||||
env.storage.write_custom_skill(name, "SKILL.md", _skill_content(name))
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
list(executor.map(lambda name: env.storage.set_skill_enabled_state(name, False), names))
|
||||
|
||||
assert env.storage._read_skill_states() == {
|
||||
"skill-a": {"enabled": False},
|
||||
"skill-b": {"enabled": False},
|
||||
}
|
||||
assert list(env.paths.user_custom_skills_view_dir("alice").iterdir()) == []
|
||||
@ -628,7 +628,6 @@ def test_update_skill_refreshes_prompt_cache_before_return(monkeypatch, tmp_path
|
||||
mock_storage = _FakeUserScopedStorage()
|
||||
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: mock_storage)
|
||||
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "default")
|
||||
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: SimpleNamespace(mcp_servers={}, skills={}))
|
||||
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
|
||||
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(lambda config_path=None: config_path))
|
||||
monkeypatch.setattr("app.gateway.routers.skills.refresh_user_skills_system_prompt_cache_async", _refresh)
|
||||
@ -705,7 +704,6 @@ def test_public_skill_toggle_clears_all_users_cache(monkeypatch, tmp_path):
|
||||
return config_path
|
||||
|
||||
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(_resolve))
|
||||
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: __import__("deerflow.config.extensions_config", fromlist=["ExtensionsConfig"]).ExtensionsConfig.from_file(config_path))
|
||||
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
|
||||
monkeypatch.setattr("app.gateway.routers.skills.clear_skills_system_prompt_cache", _clear)
|
||||
monkeypatch.setattr("app.gateway.routers.skills.refresh_user_skills_system_prompt_cache_async", _refresh)
|
||||
@ -725,6 +723,107 @@ def test_public_skill_toggle_clears_all_users_cache(monkeypatch, tmp_path):
|
||||
assert persisted["skills"]["public-skill"]["enabled"] is False
|
||||
|
||||
|
||||
def test_public_skill_toggle_creates_missing_extensions_config(monkeypatch, tmp_path):
|
||||
backend_dir = tmp_path / "backend"
|
||||
backend_dir.mkdir()
|
||||
monkeypatch.chdir(backend_dir)
|
||||
config_path = tmp_path / "extensions_config.json"
|
||||
|
||||
def _load_skills(*, enabled_only: bool):
|
||||
enabled = True
|
||||
if config_path.exists():
|
||||
enabled = json.loads(config_path.read_text(encoding="utf-8"))["skills"]["public-skill"]["enabled"]
|
||||
skill = _make_skill("public-skill", enabled=enabled)
|
||||
return [] if enabled_only and not enabled else [skill]
|
||||
|
||||
storage = SimpleNamespace(load_skills=_load_skills)
|
||||
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda _config: storage)
|
||||
|
||||
def _resolve_config_path(explicit_path=None):
|
||||
if explicit_path is None:
|
||||
return None
|
||||
path = Path(explicit_path)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(path)
|
||||
return path
|
||||
|
||||
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(_resolve_config_path))
|
||||
monkeypatch.setattr(skills_router, "reload_extensions_config", lambda: None)
|
||||
monkeypatch.setattr(skills_router, "clear_skills_system_prompt_cache", lambda: None)
|
||||
|
||||
with TestClient(_make_test_app(SimpleNamespace())) as client:
|
||||
response = client.put("/api/skills/public-skill", json={"enabled": False})
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["enabled"] is False
|
||||
assert json.loads(config_path.read_text(encoding="utf-8")) == {
|
||||
"mcpServers": {},
|
||||
"skills": {"public-skill": {"enabled": False}},
|
||||
"middlewares": [],
|
||||
}
|
||||
|
||||
|
||||
def test_public_skill_toggle_rebuilds_projection_before_response(monkeypatch, tmp_path):
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, reset_extensions_config, set_extensions_config
|
||||
from deerflow.config.paths import Paths
|
||||
from deerflow.skills.projection import rebuild_skill_projections
|
||||
|
||||
skills_root = tmp_path / "skills"
|
||||
skill_file = skills_root / "public" / "public-skill" / "SKILL.md"
|
||||
skill_file.parent.mkdir(parents=True)
|
||||
skill_file.write_text(_skill_content("public-skill"), encoding="utf-8")
|
||||
(skills_root / "custom").mkdir()
|
||||
config_path = tmp_path / "extensions_config.json"
|
||||
config_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"mcpServers": {},
|
||||
"skills": {
|
||||
"public-skill": {"enabled": True},
|
||||
"untouched-skill": {"enabled": False},
|
||||
},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(config_path))
|
||||
|
||||
paths = Paths(base_dir=tmp_path)
|
||||
config = SimpleNamespace(
|
||||
skills=SimpleNamespace(
|
||||
get_skills_path=lambda: skills_root,
|
||||
container_path="/mnt/skills",
|
||||
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
|
||||
),
|
||||
skill_evolution=SimpleNamespace(enabled=True, moderation_model_name=None),
|
||||
)
|
||||
monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: paths)
|
||||
monkeypatch.setattr("deerflow.config.paths._paths", None)
|
||||
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "default")
|
||||
monkeypatch.setattr(skills_router, "clear_skills_system_prompt_cache", lambda: None)
|
||||
# Simulate another worker having updated the file after this worker cached
|
||||
# an older snapshot. The public toggle must reload from disk under the
|
||||
# cross-process projection lock before its read-modify-write.
|
||||
set_extensions_config(ExtensionsConfig(skills={"public-skill": SkillStateConfig(enabled=True)}))
|
||||
|
||||
storage = UserScopedSkillStorage("default", host_path=str(skills_root), app_config=config)
|
||||
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda _config: storage)
|
||||
projected = rebuild_skill_projections(storage)
|
||||
assert (projected.public / "public-skill" / "SKILL.md").is_file()
|
||||
|
||||
try:
|
||||
with TestClient(_make_test_app(config)) as client:
|
||||
response = client.put("/api/skills/public-skill", json={"enabled": False})
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["enabled"] is False
|
||||
assert not (projected.public / "public-skill").exists()
|
||||
persisted = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
assert persisted["skills"]["untouched-skill"] == {"enabled": False}
|
||||
finally:
|
||||
reset_extensions_config()
|
||||
|
||||
|
||||
class TestMultiUserSkillIsolation:
|
||||
"""End-to-end integration tests verifying per-user skill isolation
|
||||
through the HTTP router → _get_user_skill_storage → filesystem chain.
|
||||
@ -955,7 +1054,6 @@ class TestMultiUserSkillIsolation:
|
||||
)
|
||||
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: alice_storage)
|
||||
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "alice")
|
||||
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: SimpleNamespace(mcp_servers={}, skills={}))
|
||||
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
|
||||
|
||||
app = _make_test_app(config)
|
||||
@ -1009,7 +1107,6 @@ class TestMultiUserSkillIsolation:
|
||||
|
||||
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: alice_storage)
|
||||
monkeypatch.setattr(skills_router, "get_effective_user_id", lambda: "alice")
|
||||
monkeypatch.setattr("app.gateway.routers.skills.get_extensions_config", lambda: SimpleNamespace(mcp_servers={}, skills={}))
|
||||
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda: None)
|
||||
monkeypatch.setattr("app.gateway.routers.skills.refresh_user_skills_system_prompt_cache_async", _noop_async)
|
||||
|
||||
|
||||
@ -125,6 +125,17 @@ def test_enable_toggle_allowed_for_admin(monkeypatch, tmp_path):
|
||||
from deerflow.skills.types import Skill
|
||||
|
||||
config_path = tmp_path / "extensions_config.json"
|
||||
config_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"mcpServers": {},
|
||||
"skills": {},
|
||||
"middlewares": ["pkg:Middleware"],
|
||||
"mcpInterceptors": ["pkg.interceptor:build"],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
def _load_skills(*, enabled_only: bool):
|
||||
return [
|
||||
@ -141,21 +152,12 @@ def test_enable_toggle_allowed_for_admin(monkeypatch, tmp_path):
|
||||
]
|
||||
|
||||
app = _make_app(system_role="admin")
|
||||
# Not a real LocalSkillStorage instance, so _write_extensions_skill_state's
|
||||
# projection-mutation branch is skipped (nullcontext) and it reads the
|
||||
# config_path fresh via ExtensionsConfig.from_file.
|
||||
monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda cfg: SimpleNamespace(load_skills=_load_skills))
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
|
||||
monkeypatch.setattr(
|
||||
skills_router,
|
||||
"get_extensions_config",
|
||||
lambda: ExtensionsConfig(
|
||||
mcp_servers={},
|
||||
skills={},
|
||||
middlewares=["pkg:Middleware"],
|
||||
mcpInterceptors=["pkg.interceptor:build"],
|
||||
),
|
||||
)
|
||||
monkeypatch.setattr(skills_router, "reload_extensions_config", lambda: None)
|
||||
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(lambda: config_path))
|
||||
monkeypatch.setattr(skills_router.ExtensionsConfig, "resolve_config_path", staticmethod(lambda _config_path=None: config_path))
|
||||
|
||||
async def _refresh(_user_id: str):
|
||||
return None
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
"""End-to-end tests for three-way skills mount across sandbox providers.
|
||||
"""End-to-end tests for enabled-only skill mounts across sandbox providers.
|
||||
|
||||
Verifies that (a) public, (b) per-user custom, and (c) legacy global-custom
|
||||
skills all resolve to correct container paths that the sandbox providers
|
||||
actually mount — covering ``LocalSandboxProvider`` and
|
||||
Verifies that public, per-user custom, legacy global-custom, and managed
|
||||
integration skills all resolve to correct container paths that the sandbox
|
||||
providers actually mount — covering ``LocalSandboxProvider`` and
|
||||
``AioSandboxProvider`` (DooD / local-backend path).
|
||||
|
||||
Includes a full-pipeline test that exercises the actual path the model
|
||||
@ -16,9 +16,12 @@ from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig
|
||||
from deerflow.config.paths import Paths
|
||||
from deerflow.sandbox.local.local_sandbox import PathMapping
|
||||
from deerflow.sandbox.local.local_sandbox_provider import LocalSandboxProvider
|
||||
from deerflow.skills.projection import rebuild_skill_projections
|
||||
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
|
||||
from deerflow.skills.types import SKILL_MD_FILE, Skill, SkillCategory
|
||||
|
||||
_AIO_MODULE = "deerflow.community.aio_sandbox.aio_sandbox_provider"
|
||||
@ -93,6 +96,32 @@ class TestThreeWayMountEndToEnd:
|
||||
idx = _local_mounts(provider, "thread-1", user_id="user-1")
|
||||
assert "/mnt/skills/public" in idx
|
||||
assert idx["/mnt/skills/public"].read_only is True
|
||||
assert Path(idx["/mnt/skills/public"].local_path) == paths.public_skills_view_dir
|
||||
|
||||
def test_local_acquire_recovers_public_mount_after_initial_projection_failure(self, skills_fs):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
paths = Paths(base_dir=skills_fs["users_dir"].parent)
|
||||
projection = SimpleNamespace(
|
||||
public=paths.public_skills_view_dir,
|
||||
custom=paths.user_custom_skills_view_dir("user-1"),
|
||||
legacy=paths.user_legacy_skills_view_dir("user-1"),
|
||||
integrations=paths.user_integration_skills_view_dir("user-1"),
|
||||
)
|
||||
for root in (projection.public, projection.custom, projection.legacy, projection.integrations):
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with (
|
||||
patch("deerflow.config.get_app_config", return_value=cfg),
|
||||
patch("deerflow.config.paths.get_paths", return_value=paths),
|
||||
patch.object(LocalSandboxProvider, "_ensure_skills_projection", side_effect=[OSError("transient"), projection]),
|
||||
):
|
||||
provider = LocalSandboxProvider()
|
||||
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||
sandbox = provider.get(sandbox_id)
|
||||
|
||||
mappings = {mapping.container_path: mapping for mapping in sandbox.path_mappings}
|
||||
assert Path(mappings["/mnt/skills/public"].local_path) == projection.public
|
||||
assert mappings["/mnt/skills/public"].read_only is True
|
||||
|
||||
def test_local_per_user_custom_skill_mounted(self, skills_fs):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
@ -101,7 +130,17 @@ class TestThreeWayMountEndToEnd:
|
||||
provider = LocalSandboxProvider()
|
||||
idx = _local_mounts(provider, "thread-1", user_id="user-1")
|
||||
assert "/mnt/skills/custom" in idx
|
||||
assert str(skills_fs["user_custom"]) in idx["/mnt/skills/custom"].local_path
|
||||
assert Path(idx["/mnt/skills/custom"].local_path) == paths.user_custom_skills_view_dir("user-1")
|
||||
|
||||
def test_local_managed_integrations_use_per_user_projection(self, skills_fs):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
paths = Paths(base_dir=skills_fs["users_dir"].parent)
|
||||
with patch("deerflow.config.get_app_config", return_value=cfg), patch("deerflow.config.paths.get_paths", return_value=paths):
|
||||
provider = LocalSandboxProvider()
|
||||
idx = _local_mounts(provider, "thread-1", user_id="user-1")
|
||||
assert "/mnt/skills/integrations" in idx
|
||||
assert Path(idx["/mnt/skills/integrations"].local_path) == paths.user_integration_skills_view_dir("user-1")
|
||||
assert idx["/mnt/skills/integrations"].read_only is True
|
||||
|
||||
def test_local_legacy_mounted_for_user_without_custom(self, skills_fs):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
@ -110,7 +149,7 @@ class TestThreeWayMountEndToEnd:
|
||||
provider = LocalSandboxProvider()
|
||||
idx = _local_mounts(provider, "thread-1", user_id="noob")
|
||||
assert "/mnt/skills/legacy" in idx
|
||||
assert str(skills_fs["legacy_global"]) in idx["/mnt/skills/legacy"].local_path
|
||||
assert Path(idx["/mnt/skills/legacy"].local_path) == paths.user_legacy_skills_view_dir("noob")
|
||||
|
||||
def test_local_legacy_not_mounted_when_user_has_custom(self, skills_fs):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
@ -118,7 +157,8 @@ class TestThreeWayMountEndToEnd:
|
||||
with patch("deerflow.config.get_app_config", return_value=cfg), patch("deerflow.config.paths.get_paths", return_value=paths):
|
||||
provider = LocalSandboxProvider()
|
||||
idx = _local_mounts(provider, "thread-1", user_id="user-1")
|
||||
assert "/mnt/skills/legacy" not in idx
|
||||
assert "/mnt/skills/legacy" in idx
|
||||
assert list(Path(idx["/mnt/skills/legacy"].local_path).iterdir()) == []
|
||||
|
||||
def test_local_legacy_still_mounted_when_user_has_only_non_skill_subdir(self, skills_fs):
|
||||
(skills_fs["users_dir"] / "ghost" / "skills" / "custom" / "dangling-dir").mkdir(parents=True, exist_ok=True)
|
||||
@ -204,6 +244,40 @@ class TestThreeWayMountEndToEnd:
|
||||
assert cp == "/mnt/skills/legacy/leg-skill/SKILL.md"
|
||||
assert "leg-skill" in sandbox_noob.read_file(cp)
|
||||
|
||||
def test_local_bash_observes_toggle_without_sandbox_recreation(self, tmp_path):
|
||||
skills_root = tmp_path / "skills"
|
||||
_write_skill(skills_root / "public", "secret-skill", "SECRET_PROCEDURE")
|
||||
paths = Paths(base_dir=tmp_path)
|
||||
cfg = _build_config(skills_root)
|
||||
extensions = ExtensionsConfig(skills={"secret-skill": SkillStateConfig(enabled=False)})
|
||||
|
||||
with (
|
||||
patch("deerflow.config.get_app_config", return_value=cfg),
|
||||
patch("deerflow.config.paths.get_paths", return_value=paths),
|
||||
patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=extensions),
|
||||
patch("deerflow.config.extensions_config.get_extensions_config", return_value=extensions),
|
||||
):
|
||||
storage = UserScopedSkillStorage("user-1", host_path=str(skills_root), app_config=cfg)
|
||||
rebuild_skill_projections(storage)
|
||||
provider = LocalSandboxProvider()
|
||||
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||
sandbox = provider.get(sandbox_id)
|
||||
assert sandbox is not None
|
||||
|
||||
disabled = sandbox.execute_command("cat /mnt/skills/public/secret-skill/SKILL.md")
|
||||
assert "SECRET_PROCEDURE" not in disabled
|
||||
|
||||
extensions.skills["secret-skill"] = SkillStateConfig(enabled=True)
|
||||
rebuild_skill_projections(storage)
|
||||
enabled = sandbox.execute_command("cat /mnt/skills/public/secret-skill/SKILL.md")
|
||||
assert "SECRET_PROCEDURE" in enabled
|
||||
assert provider.acquire("thread-1", user_id="user-1") == sandbox_id
|
||||
|
||||
extensions.skills["secret-skill"] = SkillStateConfig(enabled=False)
|
||||
rebuild_skill_projections(storage)
|
||||
disabled_again = sandbox.execute_command("cat /mnt/skills/public/secret-skill/SKILL.md")
|
||||
assert "SECRET_PROCEDURE" not in disabled_again
|
||||
|
||||
# ── AioSandboxProvider ──────────────────────────────────────────────
|
||||
|
||||
def test_aio_public_skill_mount(self, skills_fs, aio_mod):
|
||||
@ -212,6 +286,8 @@ class TestThreeWayMountEndToEnd:
|
||||
mounts = aio_mod.AioSandboxProvider._get_skills_mounts(user_id="user-1")
|
||||
idx = {m[1]: m for m in mounts}
|
||||
assert "/mnt/skills/public" in idx
|
||||
host, _, _ = idx["/mnt/skills/public"]
|
||||
assert "skills_view/public" in host.replace("\\", "/")
|
||||
|
||||
def test_aio_per_user_custom_skill_mount(self, skills_fs, aio_mod, monkeypatch):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
@ -222,7 +298,7 @@ class TestThreeWayMountEndToEnd:
|
||||
idx = {m[1]: m for m in mounts}
|
||||
assert "/mnt/skills/custom" in idx
|
||||
host, _, _ = idx["/mnt/skills/custom"]
|
||||
assert "users/user-1/skills/custom" in host.replace("\\", "/")
|
||||
assert "users/user-1/skills_view/custom" in host.replace("\\", "/")
|
||||
|
||||
def test_aio_legacy_mounted_for_user_without_custom(self, skills_fs, aio_mod, monkeypatch):
|
||||
cfg = _build_config(skills_fs["root"])
|
||||
@ -240,7 +316,7 @@ class TestThreeWayMountEndToEnd:
|
||||
with patch(_AIO_GET_CONFIG, return_value=cfg), patch("deerflow.config.paths.get_paths", return_value=paths):
|
||||
mounts = aio_mod.AioSandboxProvider._get_skills_mounts(user_id="user-1")
|
||||
idx = {m[1]: m for m in mounts}
|
||||
assert "/mnt/skills/legacy" not in idx
|
||||
assert "/mnt/skills/legacy" in idx
|
||||
|
||||
def test_aio_legacy_still_mounted_when_user_has_only_non_skill_subdir(self, skills_fs, aio_mod, monkeypatch):
|
||||
(skills_fs["users_dir"] / "ghost" / "skills" / "custom" / "dangling-dir").mkdir(parents=True, exist_ok=True)
|
||||
@ -286,7 +362,11 @@ class TestThreeWayMountEndToEnd:
|
||||
|
||||
assert "/mnt/skills/custom" in mount_entries
|
||||
assert "dst=/mnt/skills/custom" in mount_entries["/mnt/skills/custom"]
|
||||
assert "users/noob/skills/custom" in mount_entries["/mnt/skills/custom"]
|
||||
assert "users/noob/skills_view/custom" in mount_entries["/mnt/skills/custom"]
|
||||
|
||||
assert "/mnt/skills/integrations" in mount_entries
|
||||
assert "dst=/mnt/skills/integrations" in mount_entries["/mnt/skills/integrations"]
|
||||
assert "users/noob/skills_view/integrations" in mount_entries["/mnt/skills/integrations"]
|
||||
|
||||
# noob has no per-user custom → legacy is mounted
|
||||
assert "/mnt/skills/legacy" in mount_entries
|
||||
|
||||
@ -539,11 +539,9 @@ class TestSkillLoadingRespectsGlobalDisable:
|
||||
skills={"shared-skill": SimpleNamespace(enabled=False)},
|
||||
is_skill_enabled=lambda name, _cat: not (name == "shared-skill"),
|
||||
)
|
||||
# The function inside ``load_skills`` does a
|
||||
# function-local ``from deerflow.config.extensions_config
|
||||
# import get_extensions_config``, so patch the
|
||||
# extension_config module symbol.
|
||||
with patch("deerflow.config.extensions_config.get_extensions_config", return_value=ext_cfg):
|
||||
# User-scoped loading re-reads disk state so another
|
||||
# worker's global disable is not hidden by a stale cache.
|
||||
with patch("deerflow.config.extensions_config.ExtensionsConfig.from_file", return_value=ext_cfg):
|
||||
storage = get_or_new_user_skill_storage("alice", app_config=cfg)
|
||||
loaded = storage.load_skills(enabled_only=False)
|
||||
shared = [s for s in loaded if s.name == "shared-skill" and s.category == SkillCategory.LEGACY]
|
||||
|
||||
@ -91,8 +91,6 @@ spec:
|
||||
value: http://gateway:8001
|
||||
- name: DEER_FLOW_HOST_BASE_DIR
|
||||
value: /app/backend/.deer-flow
|
||||
- name: DEER_FLOW_HOST_SKILLS_PATH
|
||||
value: /app/skills
|
||||
- name: GATEWAY_HOST
|
||||
value: 0.0.0.0
|
||||
- name: GATEWAY_PORT
|
||||
|
||||
@ -57,7 +57,6 @@ services:
|
||||
# On Docker Desktop/OrbStack, use your actual host paths like /Users/username/...
|
||||
# Set these in your shell before running docker-compose:
|
||||
# export DEER_FLOW_ROOT=/absolute/path/to/deer-flow
|
||||
- SKILLS_HOST_PATH=${DEER_FLOW_ROOT}/skills
|
||||
- THREADS_HOST_PATH=${DEER_FLOW_ROOT}/backend/.deer-flow/threads
|
||||
# Per-user data base directory for user-scoped skill mounts
|
||||
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_ROOT}/backend/.deer-flow
|
||||
@ -201,7 +200,6 @@ services:
|
||||
- DEER_FLOW_CHANNELS_GATEWAY_URL=${DEER_FLOW_CHANNELS_GATEWAY_URL:-http://gateway:8001}
|
||||
- DEER_FLOW_INTERNAL_AUTH_TOKEN=${DEER_FLOW_INTERNAL_AUTH_TOKEN:-}
|
||||
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_ROOT}/backend/.deer-flow
|
||||
- DEER_FLOW_HOST_SKILLS_PATH=${DEER_FLOW_ROOT}/skills
|
||||
- DEER_FLOW_SANDBOX_HOST=host.docker.internal
|
||||
# Pass PROVISIONER_API_KEY into the gateway container so config.yaml can reference it
|
||||
# as sandbox.provisioner_api_key: $PROVISIONER_API_KEY
|
||||
|
||||
@ -127,7 +127,6 @@ services:
|
||||
- DEER_FLOW_INTERNAL_AUTH_TOKEN=${DEER_FLOW_INTERNAL_AUTH_TOKEN}
|
||||
# DooD path/network translation
|
||||
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_HOME}
|
||||
- DEER_FLOW_HOST_SKILLS_PATH=${DEER_FLOW_REPO_ROOT}/skills
|
||||
- DEER_FLOW_SANDBOX_HOST=host.docker.internal
|
||||
# Proxy values (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY) are inherited from ../.env via env_file.
|
||||
# Only NO_PROXY is declared here so internal service hostnames are always exempt from the proxy.
|
||||
@ -167,7 +166,6 @@ services:
|
||||
# plaintext config/data are never mounted into the sandbox. Supersedes
|
||||
# LARK_CLI_INIT_IMAGE when both are set. Empty ⇒ broker off.
|
||||
- LARK_CLI_BROKER_IMAGE=${LARK_CLI_BROKER_IMAGE:-}
|
||||
- SKILLS_HOST_PATH=${DEER_FLOW_REPO_ROOT}/skills
|
||||
- THREADS_HOST_PATH=${DEER_FLOW_HOME}/threads
|
||||
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_HOME}
|
||||
- KUBECONFIG_PATH=/root/.kube/config
|
||||
|
||||
@ -25,7 +25,7 @@ The **Sandbox Provisioner** is a FastAPI service that dynamically manages sandbo
|
||||
2. **Pod Creation**: The provisioner creates a dedicated Pod in the `deer-flow` namespace with:
|
||||
- The sandbox container image (all-in-one-sandbox)
|
||||
- HostPath volumes mounted for:
|
||||
- `/mnt/skills` → Read-only access to public skills
|
||||
- `/mnt/skills/{public,custom,legacy}` → Read-only enabled-only skill projections
|
||||
- `/mnt/user-data` → Read-write access to thread-specific data
|
||||
- Resource limits (CPU, memory, ephemeral storage)
|
||||
- Readiness/liveness probes
|
||||
@ -153,8 +153,8 @@ The provisioner is configured via environment variables (set in [docker-compose-
|
||||
| `SANDBOX_IMAGE` | `enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest` | AIO-compatible container image for sandbox Pods |
|
||||
| `LARK_CLI_INIT_IMAGE` | empty (feature off) | Optional lark-cli init image (Pattern A). When set, sandbox Pods requesting the lark-cli runtime get an init container + shared `emptyDir` that provisions `lark-cli`, instead of a hostPath/PVC runtime mount. See [`docker/lark-cli-init`](../lark-cli-init/README.md) |
|
||||
| `LARK_CLI_BROKER_IMAGE` | empty (feature off) | Optional lark-cli broker image (Pattern B, issue #4338). When set, sandbox Pods requesting the broker get a shim init container + a `lark-cli-broker` sidecar that holds the credentials; the plaintext `config`/`data` are mounted into the **sidecar only**, never the sandbox. Supersedes `LARK_CLI_INIT_IMAGE` when both are set. See [`docker/lark-cli-broker`](../lark-cli-broker/README.md) |
|
||||
| `SKILLS_HOST_PATH` | - | **Host machine** path to skills directory (must be absolute) |
|
||||
| `THREADS_HOST_PATH` | - | **Host machine** path to threads data directory (must be absolute) |
|
||||
| `DEER_FLOW_HOST_BASE_DIR` | `/.deer-flow` | **Host machine** DeerFlow data root containing global and per-user `skills_view` projections |
|
||||
| `SKILLS_PVC_NAME` | empty (use hostPath) | PVC name for skills volume; when set, sandbox Pods use PVC instead of hostPath |
|
||||
| `SKILLS_PVC_SUBPATH_TEMPLATE` | empty | Optional `subPath` template for `SKILLS_PVC_NAME`. Supports `{user_id}` and `{thread_id}`. When empty, the skills PVC root is mounted unchanged |
|
||||
| `USERDATA_PVC_NAME` | empty (use hostPath) | PVC name for user-data volume; when set, uses PVC with `subPath: deer-flow/users/{user_id}/threads/{thread_id}/user-data` |
|
||||
@ -208,7 +208,9 @@ PYTHONPATH=. python scripts/migrate_user_isolation.py --user-id <target-user-id>
|
||||
|
||||
This moves legacy `threads/{thread_id}/user-data` data under `users/<target-user-id>/threads/{thread_id}/user-data`, which matches the new provisioner PVC subPath when the gateway base directory is mounted at `deer-flow/` on the PVC. Use `default` as the target user only when the legacy data should remain in the default no-auth user namespace. Run the migration while no gateway or sandbox Pods are writing to those paths.
|
||||
|
||||
When skills are materialized per thread on the same PVC, set `SKILLS_PVC_NAME` to that PVC and configure `SKILLS_PVC_SUBPATH_TEMPLATE=deer-flow/users/{user_id}/threads/{thread_id}/skills`. Leaving the template empty preserves the legacy behavior of mounting the skills PVC root at `/mnt/skills`.
|
||||
In hostPath mode, the gateway materializes enabled-only views under `skills_view/public` and `users/{user_id}/skills_view/{custom,legacy}` beneath `DEER_FLOW_HOST_BASE_DIR`; the provisioner mounts those stable directories. When skills are materialized per thread on the same PVC, set `SKILLS_PVC_NAME` to that PVC and configure `SKILLS_PVC_SUBPATH_TEMPLATE=deer-flow/users/{user_id}/threads/{thread_id}/skills`. Leaving the template empty preserves the legacy behavior of mounting the skills PVC root at `/mnt/skills`. The gateway does not yet populate that PVC layout dynamically, so PVC-backed skills do not receive hostPath projection updates.
|
||||
|
||||
**hostPath skills volumes require the gateway and the K8s node to see the same `DEER_FLOW_HOST_BASE_DIR`** (single-node deployment, or NFS/shared storage mounted at that path on every node). The gateway writes the projection there before every sandbox acquire, so as long as that path is shared, the directory the provisioner mounts always exists by the time the Pod is scheduled — even a boot-time rebuild failure for one user self-heals on their next acquire, before the provisioner is called. `skills-custom` and `skills-legacy` use hostPath type `Directory` (not `DirectoryOrCreate`): if the shared-storage assumption is violated — the gateway wrote to a different node than the one the Pod lands on — Pod creation now fails visibly instead of silently mounting an empty directory. Use `SKILLS_PVC_NAME` instead of hostPath for genuinely multi-node clusters without shared storage.
|
||||
|
||||
### Important: K8S_API_SERVER Override
|
||||
|
||||
@ -248,7 +250,7 @@ kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
|
||||
- Read Namespaces (to create `deer-flow` if missing)
|
||||
|
||||
4. **Host Paths**:
|
||||
- The `SKILLS_HOST_PATH` and `THREADS_HOST_PATH` must be **absolute paths on the host machine**
|
||||
- `DEER_FLOW_HOST_BASE_DIR` and `THREADS_HOST_PATH` must be **absolute paths on the host machine**
|
||||
- These paths are mounted into sandbox Pods via K8s HostPath volumes
|
||||
- The paths must exist and be readable by the K8s node
|
||||
|
||||
@ -350,7 +352,7 @@ docker exec deer-flow-gateway curl -s $SANDBOX_URL/v1/sandbox
|
||||
**Cause**: HostPath volumes contain invalid paths (e.g., relative paths with `..`).
|
||||
|
||||
**Solution**:
|
||||
- Use absolute paths for `SKILLS_HOST_PATH` and `THREADS_HOST_PATH`
|
||||
- Use absolute paths for `DEER_FLOW_HOST_BASE_DIR` and `THREADS_HOST_PATH`
|
||||
- Verify the paths exist on your host machine:
|
||||
```bash
|
||||
ls -la /path/to/skills
|
||||
|
||||
@ -85,7 +85,6 @@ LARK_BROKER_SIDECAR_DATA_PATH = "/var/lark/data"
|
||||
LARK_BROKER_CONFIG_VOLUME_NAME = "lark-cli-config"
|
||||
LARK_BROKER_DATA_VOLUME_NAME = "lark-cli-data"
|
||||
LARK_BROKER_URL = "http://127.0.0.1:8788"
|
||||
SKILLS_HOST_PATH = os.environ.get("SKILLS_HOST_PATH", "/skills")
|
||||
THREADS_HOST_PATH = os.environ.get("THREADS_HOST_PATH", "/.deer-flow/threads")
|
||||
DEER_FLOW_HOST_BASE_DIR = os.environ.get("DEER_FLOW_HOST_BASE_DIR", "/.deer-flow")
|
||||
SKILLS_PVC_NAME = os.environ.get("SKILLS_PVC_NAME", "")
|
||||
@ -494,6 +493,7 @@ def _build_volumes(
|
||||
``LocalSandboxProvider`` and ``AioSandboxProvider``.
|
||||
"""
|
||||
volumes: list[k8s_client.V1Volume] = []
|
||||
del include_legacy_skills # retained for request compatibility
|
||||
|
||||
# ── Skills volumes ────────────────────────────────────────────────
|
||||
|
||||
@ -512,7 +512,7 @@ def _build_volumes(
|
||||
)
|
||||
else:
|
||||
# hostPath mode: three-way layout
|
||||
public_path = join_host_path(SKILLS_HOST_PATH, "public")
|
||||
public_path = join_host_path(DEER_FLOW_HOST_BASE_DIR, "skills_view", "public")
|
||||
volumes.append(
|
||||
k8s_client.V1Volume(
|
||||
name="skills-public",
|
||||
@ -527,7 +527,7 @@ def _build_volumes(
|
||||
DEER_FLOW_HOST_BASE_DIR,
|
||||
"users",
|
||||
user_id,
|
||||
"skills",
|
||||
"skills_view",
|
||||
"custom",
|
||||
)
|
||||
volumes.append(
|
||||
@ -535,22 +535,23 @@ def _build_volumes(
|
||||
name="skills-custom",
|
||||
host_path=k8s_client.V1HostPathVolumeSource(
|
||||
path=user_custom_path,
|
||||
type="DirectoryOrCreate",
|
||||
type="Directory",
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
if include_legacy_skills:
|
||||
legacy_path = join_host_path(SKILLS_HOST_PATH, "custom")
|
||||
volumes.append(
|
||||
k8s_client.V1Volume(
|
||||
name="skills-legacy",
|
||||
host_path=k8s_client.V1HostPathVolumeSource(
|
||||
path=legacy_path,
|
||||
type="Directory",
|
||||
),
|
||||
)
|
||||
legacy_path = join_host_path(
|
||||
DEER_FLOW_HOST_BASE_DIR, "users", user_id, "skills_view", "legacy"
|
||||
)
|
||||
volumes.append(
|
||||
k8s_client.V1Volume(
|
||||
name="skills-legacy",
|
||||
host_path=k8s_client.V1HostPathVolumeSource(
|
||||
path=legacy_path,
|
||||
type="Directory",
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
# ── User-data volume ──────────────────────────────────────────────
|
||||
|
||||
@ -638,6 +639,7 @@ def _build_volume_mounts(
|
||||
scope that mount with ``SKILLS_PVC_SUBPATH_TEMPLATE``.
|
||||
"""
|
||||
mounts: list[k8s_client.V1VolumeMount] = []
|
||||
del include_legacy_skills # retained for request compatibility
|
||||
|
||||
if SKILLS_PVC_NAME:
|
||||
skills_mount = k8s_client.V1VolumeMount(
|
||||
@ -664,16 +666,13 @@ def _build_volume_mounts(
|
||||
mount_path="/mnt/skills/custom",
|
||||
read_only=True,
|
||||
),
|
||||
]
|
||||
)
|
||||
if include_legacy_skills:
|
||||
mounts.append(
|
||||
k8s_client.V1VolumeMount(
|
||||
name="skills-legacy",
|
||||
mount_path="/mnt/skills/legacy",
|
||||
read_only=True,
|
||||
)
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
userdata_mount = k8s_client.V1VolumeMount(
|
||||
name="user-data",
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user