feat(skills): add guarded evolution host APIs (#5645)

* feat(skills): add guarded evolution host APIs

* docs(agents): avoid growing saturated guidance chains

* fix(skills): address mutation host review findings

* fix(skills): order name fence before projection lock

* fix(skills): preserve composite toggle lock order

* fix(skills): address catalog and pagination review feedback

* docs: keep subagent guidance within merged instruction budget

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Ryker_Feng 2026-10-18 21:17:02 +08:00 committed by GitHub
parent cd0166ae76
commit d1d0958bea
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
108 changed files with 8269 additions and 215 deletions

View File

@ -1915,6 +1915,24 @@ classifies the sanitized text a remote tool result shows the model, PII-redacted
`pii_redaction` is enabled, and adds an advisory warning through a lifecycle state
update; it imports only the extension contract.
Extensions may opt into owner-scoped completed-run evidence and safe updates to
existing user-custom skills through `host_access` in operator-controlled
`config.yaml`. These optional APIs stage an immutable candidate, check its exact
contents, and publish with version conflict detection and crash recovery. They
do **not** implement a learning/evolution plugin. Automatic changes are limited to
the body and description of `SKILL.md`; changing permissions, supporting files,
or enabled state is out of scope. The host contract requires `deerflow-extension-api==0.2.7`. See [Skill evolution host API](docs/skill-evolution-host.md)
for deployment requirements, grants, recovery, and the plugin integration flow.
Once an owner is enrolled, use the Gateway for skill management: standalone SDK
processes without its durable mutation runtime refuse enrolled skill access,
including custom-skill YAML diagnostics. Gateway diagnostics recover pending
publications before scanning.
Skill discovery preserves user scope when the caller uses the default configuration.
Admin operation lists return `next_cursor: null` at the end of pagination.
Keyed resume retries preserve the original run's evidence origin and agent identity.
This host slice follows up the [Skill Self-Evolution RFC](https://github.com/bytedance/deer-flow/issues/1865)
and the bilingual [Plugin Host APIs RFC](https://github.com/bytedance/deer-flow/issues/5539).
Full-stack contributions can additionally provide browser pages, conversation actions,
authenticated backend operations and model tools through the
[plugin APIs](docs/full-stack-plugins.md). The independent
@ -2371,6 +2389,8 @@ async with runtime:
Gateway run finalization also retains ownership through caller cancellation: it finishes terminal persistence, stream completion, and resource cleanup before propagating the interruption. Backend I/O must terminate or enforce its own timeout. Gateway shutdown includes workers still finalizing after their terminal status commits; an expired shutdown drain budget is logged and does not cancel their outstanding writes.
Terminal cleanup releases the run journal and its message-sequence callback even when diagnostic logs retain a checkpoint-inspection traceback.
The factory still does not load YAML or create SQL infrastructure: the caller supplies the config snapshot, repository, and lifecycle. Because it accepts a caller-owned `system_prompt`, direct integrations also own any model-visible wording about those limits; the default middleware enforces the runtime limits regardless. The factory does not mount the Gateway owner-scoped HTTP routes or Web UI, so direct applications must expose their own result API/UI if they need those surfaces. For ordinary delegation only, `SubagentRuntime(...)` needs no asynchronous startup.
Administrators can add, edit, disable, and delete reusable worker definitions from **Settings → Subagents**. Built-in and `config.yaml` definitions remain visible there as read-only entries. The default Lead Agent can use every enabled runtime sub-agent; each page-created Custom Agent can instead allow all, none, or a selected set. That selection is enforced both in the model-visible directory and by the server-side `task` tool. Managed definitions are deployment-wide in this version and follow `agent_storage.backend`: atomic files for a local deployment or the shared application database for multiple instances.

View File

@ -49,6 +49,7 @@ from app.gateway.routers import (
projects,
runs,
scheduled_tasks,
skill_mutations,
skills,
subagent_batches,
subagents,
@ -1330,6 +1331,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
# Skills API is mounted at /api/skills
app.include_router(skills.router)
app.include_router(skill_mutations.router)
# First-party integrations API is mounted at /api/integrations
app.include_router(integrations.router)

View File

@ -854,6 +854,14 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
from deerflow.extensions.gateway import start_services, stop_services
extensions = getattr(app.state, "extensions", EMPTY_EXTENSIONS)
from deerflow.extensions.host_capabilities import HostCapabilities
host_capabilities = HostCapabilities(config, app.state.run_store, app.state.run_event_store)
app.state.skill_mutation_host = host_capabilities
# Own partial startup and cancellation before acquiring any locks or
# worker resources. LIFO shutdown stops plugins before host recovery.
stack.push_async_callback(host_capabilities.close)
await host_capabilities.start()
attempted_services: list[tuple[str, Any]] = []
async def stop_extension_services() -> None:
@ -875,6 +883,7 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
config,
sf,
run_evidence_reader=app.state.run_evidence_reader,
host_capabilities=host_capabilities.bindings,
attempted_services=attempted_services,
)
)

View File

@ -16,6 +16,8 @@ create-only: collisions return 409, never overwrite or upsert. Ordinary creation
and import share `_persist_new_agent`, which uses `run_drained_write` so a
cancelled request still drains the owned persistence worker.
Skill mutation operation lists return `next_cursor: null` when `has_more` is false, including empty pages after a supplied `after_id`. The next cursor is the final returned operation only when a further page exists.
`GET /api/agent-tool-groups` projects only configured group names for the
custom-Agent settings allowlist. Never expose tool provider paths, credentials,
or raw tool configuration through this catalog.

View File

@ -0,0 +1,52 @@
"""Host-admin inspection and reconciliation, with no force-overwrite endpoint."""
from dataclasses import asdict
from typing import Annotated
from deerflow_extension_api import HostCapabilityError
from fastapi import APIRouter, HTTPException, Path, Query, Request
from app.gateway.deps import require_admin_user
router = APIRouter(prefix="/api/skill-mutations", tags=["skill-mutations"])
OperationId = Annotated[str, Path(pattern=r"^[a-f0-9]{32}$")]
OwnerId = Annotated[str, Path(min_length=1, max_length=128, pattern=r"^[A-Za-z0-9_-]+$")]
async def _invoke(request, method, *args, **kwargs):
await require_admin_user(request, detail="Admin privileges required to inspect or recover skill publications.")
host = getattr(request.app.state, "skill_mutation_host", None)
if host is None or host.recovery is None:
raise HTTPException(status_code=503, detail="UNAVAILABLE")
try:
return await host.workers.run(lambda _check: getattr(host.recovery, method)(*args, **kwargs))
except HostCapabilityError as exc:
status = {"NOT_FOUND_OR_FORBIDDEN": 404, "NOT_FOUND": 404, "NEEDS_REPAIR": 409, "REVISION_CONFLICT": 409}.get(exc.code, 503)
raise HTTPException(status_code=status, detail=exc.code) from None
except Exception:
# SQL parameters may contain package bytes. No raw exception response.
raise HTTPException(status_code=503, detail="UNAVAILABLE") from None
@router.get("/operations")
async def list_operations(request: Request, limit: Annotated[int, Query(ge=1, le=100)] = 50, after_id: Annotated[str | None, Query(pattern=r"^[a-f0-9]{32}$")] = None):
rows = await _invoke(request, "list_operations", limit=limit + 1, after_id=after_id)
items = rows[:limit]
has_more = len(rows) > limit
return {"items": [asdict(row) for row in items], "has_more": has_more, "next_cursor": items[-1].operation_id if has_more else None}
@router.get("/operations/{operation_id}")
async def get_operation(request: Request, operation_id: OperationId):
return asdict(await _invoke(request, "get_operation", operation_id))
@router.post("/operations/{operation_id}/recover")
async def recover_operation(request: Request, operation_id: OperationId):
return asdict(await _invoke(request, "recover_operation", operation_id))
@router.post("/owners/{owner_id}/recover")
async def recover_owner(request: Request, owner_id: OwnerId):
await _invoke(request, "recover_owner", owner_id)
return {"owner_id": owner_id, "reconciled": True}

View File

@ -910,12 +910,12 @@ async def rollback_custom_skill(skill_name: str, body: SkillRollbackRequest, req
await asyncio.to_thread(storage.validate_skill_markdown_content, skill_name, target_content)
static_findings = await _scan_static_skill_markdown_or_raise(skill_name, target_content, app_config=config)
scan = await scan_skill_content(target_content, executable=False, location=f"{skill_name}/{SKILL_MD_FILE}", app_config=config, static_findings=static_findings)
from deerflow.skills.mutations.guard import read_optional_text
def _read_current_content() -> str | None:
# Worker thread: the post-scan read of the file being replaced is
# blocking filesystem IO (#5747), same rule as the history read.
skill_file = storage.get_custom_skill_file(skill_name)
return skill_file.read_text(encoding="utf-8") if skill_file.exists() else None
# Called off-loop, inside the mutation critical section for writes.
# Preserve managed-owner readiness and projection-lock admission.
return read_optional_text(storage, storage.get_custom_skill_file(skill_name))
if scan.decision == "block":
# The blocked path overwrites nothing, so the entry is purely
@ -1021,12 +1021,12 @@ def _write_extensions_skill_state(
"""Read-modify-write a skill's enabled state in the shared extensions_config.json.
Blocking filesystem IO: always call this via ``asyncio.to_thread``. It takes
the public projection lock before the process-local and cross-process
extensions config locks. The first keeps the enabled-only view synchronized
across workers; the latter two prevent this router and the MCP router from
interleaving writes to the shared file. All locks are held by the worker, so
request cancellation cannot release them while the write or projection
rebuild is still running.
the same-name mutation fence before the public projection, process-local,
and cross-process extensions config locks. The projection lock keeps the
enabled-only view synchronized across workers; the latter two prevent this
router and the MCP router from interleaving writes to the shared file. All
locks are held by the worker, so request cancellation cannot release them
while the write or projection rebuild is still running.
"""
from contextlib import nullcontext
@ -1040,7 +1040,9 @@ def _write_extensions_skill_state(
config_path = Path.cwd().parent / "extensions_config.json"
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
with projection_update:
from deerflow.skills.mutations.guard import managed_global_state_write
with managed_global_state_write(storage, skill_name), projection_update:
with extensions_config_write_lock, extensions_config_file_lock(config_path):
# The projection lock is cross-process, but the singleton cache is
# not. Existing files are therefore re-read under the lock, raw, so

View File

@ -93,6 +93,7 @@ from deerflow.runtime.journal import build_checkpoint_history_seed_events
from deerflow.runtime.keyed_lock import KeyedLockTable
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, make_origin
from deerflow.runtime.runs.naming import resolve_root_run_name
from deerflow.runtime.runs.schemas import EVIDENCE_ORIGINS
from deerflow.runtime.secret_context import (
LegacyRunMetadataSecretError,
redact_config_secrets,
@ -1862,6 +1863,7 @@ async def start_run(
*,
idempotency_key: str | None = None,
require_existing_thread: bool = False,
evidence_origin: str | None = None,
scheduled_task_runtime: Mapping[str, Any] | None = None,
) -> RunRecord:
"""Create a RunRecord and launch the background agent task.
@ -1877,6 +1879,9 @@ async def start_run(
require_existing_thread : bool
Reject a missing thread instead of auto-creating metadata. Internal
notification runs use this so a deleted chat cannot be resurrected.
evidence_origin : str | None
Trusted Python entry-point provenance, never copied from request data.
Internal launches without an explicit origin remain unknown.
"""
# Cancel-capability gate. interrupt/rollback strategies terminate an already
# active run — runs:cancel capability, not runs:create — so a create-only
@ -1976,6 +1981,9 @@ async def start_run(
# so import-lock waiters cannot starve unrelated default-executor work.
# Keep this before admission so import failures cannot create a run.
agent_factory = await run_assembly(resolve_agent_factory, body.assistant_id)
admitted_origin = evidence_origin or ("unknown" if is_internal_caller else "interactive")
if admitted_origin not in EVIDENCE_ORIGINS:
raise ValueError("Unsupported host evidence origin")
command = getattr(body, "command", None)
if command and command.get("resume") is not None:
graph_input = Command(resume=command["resume"])
@ -2356,6 +2364,10 @@ async def start_run(
model_name=model_name,
user_id=owner_user_id,
idempotency_key=idempotency_key,
evidence_origin=admitted_origin,
# Canonical agent keys, not routing aliases or display names.
# Bootstrap and unresolved legacy identities stay read-only.
evidence_agent_id=(None if scope_runtime_config.get("is_bootstrap") else (_DEFAULT_ASSISTANT_ID if scope_assistant_id == _DEFAULT_ASSISTANT_ID else getattr(agent_config, "name", None))),
)
except RunIdempotencyUnsupported as exc:
raise HTTPException(status_code=503, detail=str(exc)) from exc
@ -2507,6 +2519,7 @@ async def launch_scheduled_thread_run(
thread_id,
request,
idempotency_key=idempotency_key,
evidence_origin="scheduled",
scheduled_task_runtime=scheduled_task_runtime,
)
return {"run_id": record.run_id, "thread_id": record.thread_id}

View File

@ -28,6 +28,7 @@ from deerflow_extension_api.compaction import (
CompactionEvent,
ContextCompactionObserver,
)
from deerflow_extension_api.completed_run_evidence import CompletedRunEvent, CompletedRunEventPage, CompletedRunEvidenceReader, CompletedRunSnapshot, EvidenceLimits
from deerflow_extension_api.contracts import (
ExtensionInstall,
ExtensionRegistry,
@ -44,6 +45,7 @@ from deerflow_extension_api.contracts import (
TaskOutcome,
extension,
)
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow_extension_api.model_invocation import (
ModelInvocationError,
ModelInvocationFailed,
@ -95,6 +97,7 @@ from deerflow_extension_api.runtime_bridge import (
task_store_from_runtime,
)
from deerflow_extension_api.settings import SettingsField
from deerflow_extension_api.skill_mutations import AssessmentRef, AssetRevision, BundleFile, HostCheckResult, MutationCapabilities, Operation, Proposal, ProposalBundle, SkillMutationService, SkillRevisionView
from deerflow_extension_api.state import ExtensionData
#: Contract version. Before 1.0, minors may break and patches are additive.
@ -126,8 +129,24 @@ __all__ = [
"ModelTool",
"PluginToolResult",
"PluginContribution",
"ToolContext",
"SettingsField",
"ToolContext",
"CompletedRunEvent",
"CompletedRunEventPage",
"CompletedRunEvidenceReader",
"CompletedRunSnapshot",
"EvidenceLimits",
"HostCapabilityError",
"AssessmentRef",
"AssetRevision",
"BundleFile",
"HostCheckResult",
"MutationCapabilities",
"Operation",
"Proposal",
"ProposalBundle",
"SkillMutationService",
"SkillRevisionView",
"API_VERSION",
"EXTENSION_PLUGIN_AUTHZ_RESOLVER_ASYNC_KEY",
"EXTENSION_PLUGIN_AUTHZ_RESOLVER_KEY",

View File

@ -0,0 +1,90 @@
"""Immutable, bounded evidence projections. References never confer authorization."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol
from deerflow_extension_api.run_evidence import RunPage
@dataclass(frozen=True)
class EvidenceLimits:
max_events: int = 200
max_bytes: int = 256 * 1024
def __post_init__(self) -> None:
if type(self.max_events) is not int or not 1 <= self.max_events <= 2000:
raise ValueError("max_events must be between 1 and 2000")
if type(self.max_bytes) is not int or not 1024 <= self.max_bytes <= 1024 * 1024:
raise ValueError("max_bytes must be between 1024 and 1048576, including the envelope")
@dataclass(frozen=True)
class CompletedRunSnapshot:
snapshot_ref: str = ""
evidence_revision: str = ""
retention_revision: int = 0
thread_id: str = ""
run_id: str = ""
owner_id: str = ""
agent_id: str | None = None
origin: str = "unknown"
status: str = ""
stop_reason: str | None = None
seal_state: str = "partial"
seal_error: str | None = None
upper_event_seq: int = 0
event_count: int = 0
coverage: str = "lead-journal-v1"
coverage_limits: tuple[str, ...] = (
"No complete background subagent, remote MCP, or external side-effect coverage.",
"Prior conversation and attachment bodies are excluded.",
"Skill loading and actual content hashes are unknown unless recorded.",
)
expires_at: str | None = None
skill_observations: tuple[str, ...] = ()
def __post_init__(self) -> None:
object.__setattr__(self, "coverage_limits", tuple(self.coverage_limits))
object.__setattr__(self, "skill_observations", tuple(self.skill_observations))
@dataclass(frozen=True)
class CompletedRunEvent:
"""JSON strings keep nested event data immutable and detached from the host.
``content_sha256`` hashes the persisted UTF-8 content, before any preview.
``content_bytes`` is its persisted byte count, not the JSON envelope size.
"""
event_id: str = ""
seq: int = 0
event_type: str = ""
category: str = ""
content_json: str = "null"
metadata_json: str = "{}"
content_bytes: int = 0
content_sha256: str = ""
truncated: bool = False
@dataclass(frozen=True)
class CompletedRunEventPage:
items: tuple[CompletedRunEvent, ...] = ()
next_cursor: str | None = None
has_more: bool = False
def __post_init__(self) -> None:
object.__setattr__(self, "items", tuple(self.items))
class CompletedRunEvidenceReader(Protocol):
async def list_changed_runs(self, *, cursor: str | None = None, limit: int = 200) -> RunPage:
raise NotImplementedError
async def get_snapshot(self, *, thread_id: str, run_id: str, limits: EvidenceLimits = EvidenceLimits()) -> CompletedRunSnapshot:
raise NotImplementedError
async def read_events(self, *, snapshot_ref: str, cursor: str | None = None, limits: EvidenceLimits = EvidenceLimits()) -> CompletedRunEventPage:
raise NotImplementedError

View File

@ -20,9 +20,11 @@ from deerflow_extension_api.state import ExtensionData
if TYPE_CHECKING: # pragma: no cover - typing only
from deerflow_extension_api.assembly import AgentAssemblyObserver
from deerflow_extension_api.compaction import ContextCompactionObserver
from deerflow_extension_api.completed_run_evidence import CompletedRunEvidenceReader
from deerflow_extension_api.model_invocation import ModelInvoker
from deerflow_extension_api.placement import AgentBuildContext, MiddlewarePlacement
from deerflow_extension_api.run_evidence import RunEvidenceReader
from deerflow_extension_api.skill_mutations import SkillMutationService
F = TypeVar("F", bound=Callable[..., Any])
@ -169,6 +171,8 @@ class ExtensionRuntimeDeps:
session_factory: Any | None = None
run_evidence_reader: RunEvidenceReader | None = None
model_invoker: ModelInvoker | None = None
completed_run_evidence: CompletedRunEvidenceReader | None = None
skill_mutations: SkillMutationService | None = None
class ExtensionService(Protocol):

View File

@ -0,0 +1,12 @@
"""Dependency-free errors shared by optional host capabilities."""
from __future__ import annotations
class HostCapabilityError(RuntimeError):
"""A stable machine-readable failure, optionally carrying retry guidance."""
def __init__(self, code: str, message: str = "", *, retry_after: float | None = None) -> None:
self.code = code
self.retry_after = retry_after
super().__init__(f"{code}: {message}" if message else code)

View File

@ -0,0 +1,161 @@
"""Host-owned, conditional updates of existing custom skills; no learning policy."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol
@dataclass(frozen=True)
class AssetRevision:
incarnation_id: str
mutation_seq: int
content_digest: str
@dataclass(frozen=True)
class SkillRevisionView:
target_ref: str
owner_id: str
name: str
revision: AssetRevision
content: str
enabled: bool
@dataclass(frozen=True)
class MutationCapabilities:
operations: tuple[str, ...]
protocol_version: str = "skill-mutations-v1"
max_main_bytes: int = 131072
max_package_bytes: int = 16777216
max_package_files: int = 256
max_sources: int = 20
max_pending_proposals: int = 20
max_model_scan_files: int = 32
max_model_scan_bytes: int = 1048576
max_model_scan_file_bytes: int = 131072
max_concurrent_scans: int = 4
max_owner_plugin_concurrent_scans: int = 2
max_owner_plugin_scans_per_hour: int = 20
check_timeout_seconds: int = 120
proposal_ttl_seconds: int = 86400
check_ttl_seconds: int = 900
rollback_ttl_seconds: int = 2592000
topology: str = "posix-single-host-local-filesystem"
def __post_init__(self):
object.__setattr__(self, "operations", tuple(self.operations))
@dataclass(frozen=True)
class Proposal:
proposal_id: str
owner_id: str
target_ref: str
name: str
base_revision: AssetRevision
candidate_hash: str
state: str
expires_at: str
source_refs: tuple[str, ...]
operation_id: str | None = None
def __post_init__(self):
object.__setattr__(self, "source_refs", tuple(self.source_refs))
@dataclass(frozen=True)
class BundleFile:
path: str
content: bytes
sha256: str
executable: bool
def __post_init__(self):
object.__setattr__(self, "content", bytes(self.content))
@dataclass(frozen=True)
class ProposalBundle:
proposal_id: str
baseline: tuple[BundleFile, ...]
candidate: tuple[BundleFile, ...]
base_revision: AssetRevision
candidate_hash: str
def __post_init__(self):
object.__setattr__(self, "baseline", tuple(self.baseline))
object.__setattr__(self, "candidate", tuple(self.candidate))
@dataclass(frozen=True)
class HostCheckResult:
proposal_id: str
decision: str
candidate_hash: str
base_revision: AssetRevision
policy_version: str
expires_at: str
reason_code: str
@dataclass(frozen=True)
class AssessmentRef:
candidate_hash: str
base_revision: AssetRevision
evaluator: str
evaluator_version: str
report_id: str
@dataclass(frozen=True)
class Operation:
operation_id: str
owner_id: str
name: str
publication: str
views: str
before_revision: AssetRevision
after_revision: AssetRevision
generation: int
proposal_id: str | None = None
reverts_operation_id: str | None = None
error_code: str | None = None
superseded_by_generation: int | None = None
class SkillMutationService(Protocol):
@property
def capabilities(self) -> MutationCapabilities:
raise NotImplementedError
async def read_skill(self, *, source_ref: str, name: str) -> SkillRevisionView:
raise NotImplementedError
async def stage(self, *, source_refs: tuple[str, ...], target_ref: str, expected_revision: AssetRevision, content: str, idempotency_key: str) -> Proposal:
raise NotImplementedError
async def get_proposal(self, *, proposal_id: str) -> Proposal:
raise NotImplementedError
async def read_proposal_bundle(self, *, proposal_id: str, max_bytes: int = 33554432) -> ProposalBundle:
raise NotImplementedError
async def check(self, *, proposal_id: str) -> HostCheckResult:
raise NotImplementedError
async def commit(self, *, proposal_id: str, idempotency_key: str, assessment_ref: AssessmentRef | None = None) -> Operation:
raise NotImplementedError
async def discard(self, *, proposal_id: str) -> Proposal:
raise NotImplementedError
async def revert(self, *, operation_id: str, expected_current_revision: AssetRevision, idempotency_key: str) -> Operation:
raise NotImplementedError
async def get_operation(self, *, operation_id: str) -> Operation:
raise NotImplementedError
async def find_operation(self, *, idempotency_key: str, method: str = "commit") -> Operation | None:
raise NotImplementedError

View File

@ -282,6 +282,14 @@ def get_enabled_skills_for_config(app_config: AppConfig | None = None, user_id:
to load public + user-level custom skills. Otherwise falls back to the
global storage (public + global custom fallback).
"""
from deerflow.skills.mutations.guard import owner_is_managed
if user_id is not None and (app_config is None or owner_is_managed(user_id)):
# Without an explicit config, resolve the user storage on each call.
# Enrolled owners also bypass the process-local LRU. The guarded loader
# checks durable readiness, so peer publications cannot leave a stale
# catalog or treat PREPARED bytes as a committed generation.
return list(get_or_new_user_skill_storage(user_id, app_config=app_config).load_skills(enabled_only=True))
if app_config is None:
return _get_enabled_skills()

View File

@ -279,6 +279,13 @@ class SkillActivationMiddleware(AgentMiddleware):
@staticmethod
def _read_skill_content(skill_file: Path, skills_root: Path, *, storage: SkillStorage | None = None) -> str:
from deerflow.skills.mutations.guard import managed_read
with managed_read(storage):
return SkillActivationMiddleware._read_skill_content_unlocked(skill_file, skills_root, storage=storage)
@staticmethod
def _read_skill_content_unlocked(skill_file: Path, skills_root: Path, *, storage: SkillStorage | None = None) -> str:
if skill_file.name != SKILL_MD_FILE:
raise ValueError(f"Expected {SKILL_MD_FILE}, got {skill_file.name}")
# Use the storage's path validation if available — UserScopedSkillStorage

View File

@ -1574,10 +1574,11 @@ class DeerFlowClient:
if config_path is None:
raise FileNotFoundError("Cannot locate extensions_config.json. Set DEER_FLOW_EXTENSIONS_CONFIG_PATH or ensure it exists in the project root.")
from deerflow.skills.mutations.guard import managed_global_state_write
from deerflow.skills.projection import skill_projection_mutation
removal_names = (name,) if not enabled else ()
with skill_projection_mutation(storage, "public", remove_names=removal_names):
with managed_global_state_write(storage, name), skill_projection_mutation(storage, "public", remove_names=removal_names):
with extensions_config_write_lock, extensions_config_file_lock(config_path):
# The projection lock is cross-process, but the singleton
# cache is not. Reload raw from disk under the config lock.
@ -1590,10 +1591,12 @@ class DeerFlowClient:
storage.set_skill_enabled_state(name, enabled)
else:
# Fallback for non-user-scoped storage (unlikely in practice)
from deerflow.skills.mutations.guard import managed_global_state_write
config_path = ExtensionsConfig.resolve_config_path()
if config_path is None:
raise FileNotFoundError("Cannot locate extensions_config.json. Set DEER_FLOW_EXTENSIONS_CONFIG_PATH or ensure it exists in the project root.")
with extensions_config_write_lock, extensions_config_file_lock(config_path):
with managed_global_state_write(storage, name), extensions_config_write_lock, extensions_config_file_lock(config_path):
self._write_skill_enabled_state(config_path, name, enabled)
# Invalidate the prompt cache for this caller (and for all users if

View File

@ -440,6 +440,12 @@ to delegation. Browser fields are allowlisted. Code is trusted, not sandboxed.
within 64 KiB and snapshot artifacts. Keep identity, policy/deadline and citation
gates. Test ToolNode and checkpoint replay. See `docs/full-stack-plugins.md`.
Evolution host grants (`host_access.evidence` / `skill_mutations`) compose with
`host_access.model_invocation` through `HostAccess`. Evolution grants require a
unique stable name and entry point; model-only grants retain their per-install
identity and permit repeated entry points. Keep all granted dependencies when
wrapping services with `ModelInvocationService`. The additive evolution contracts
ship in extension API 0.2.7, after 0.2.6's PluginToolResult contract.
Full Agent run control is an optional `deerflow_extension_api.AgentRuns` handle
on action/tool contexts and the request resolver. Gateway owns principal binding,
revocation and ordinary route admission in `app/gateway/extension_agent_runs.py`.

View File

@ -0,0 +1,322 @@
"""Owner-bound durable run evidence, with fixed boundaries and bounded reads."""
from __future__ import annotations
import asyncio
import base64
import hashlib
import json
from dataclasses import asdict, replace
from datetime import UTC, datetime
from time import monotonic
from uuid import uuid4
from deerflow_extension_api.completed_run_evidence import CompletedRunEvent, CompletedRunEventPage, CompletedRunSnapshot, EvidenceLimits
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow_extension_api.run_evidence import RunPage, RunStatusView
from sqlalchemy import LargeBinary, Text, and_, cast, func, or_, select, text
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.persistence.run.model import CompletedRunSnapshotRow, RunRow
from deerflow.persistence.run.sql import RunRepository
from deerflow.runtime.events.store.db import DbRunEventStore
from deerflow.runtime.secret_context import redact_metadata_secrets
from deerflow.utils.time import coerce_iso
def _json(value) -> str:
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
def _size(value) -> int:
return len(_json(asdict(value)).encode("utf-8"))
def _snapshot(payload: dict) -> CompletedRunSnapshot:
return CompletedRunSnapshot(**{**payload, "coverage_limits": tuple(payload["coverage_limits"]), "skill_observations": tuple(payload["skill_observations"])})
class HostCompletedRunEvidenceReader:
"""Bindings come from trusted installation grants, never call arguments.
This is a supported capability boundary for trusted in-process plugins,
not a sandbox against code holding a raw database session factory.
New events carry an ingestion digest. Legacy digest computation has a
cumulative per-page cap of 1 MiB, 64 SQL chunks, and 2 seconds (including
query waits). Above those caps we raise LIMIT_EXCEEDED, never invent a hash
or trust a digest supplied in event metadata.
"""
def __init__(self, run_store, event_store, *, plugin_id: str, owner_ids) -> None:
if not isinstance(plugin_id, str) or not plugin_id.strip():
raise ValueError("a stable plugin identity is required")
if not owner_ids or isinstance(owner_ids, (str, bytes)) or any(not isinstance(owner, str) or not owner.strip() or owner == "*" for owner in owner_ids):
raise ValueError("an explicit nonempty owner allowlist is required")
self._owners = frozenset(owner_ids)
self._scope = hashlib.sha256(_json([plugin_id, sorted(self._owners)]).encode()).hexdigest()
self._sf = run_store._sf if isinstance(run_store, RunRepository) and isinstance(event_store, DbRunEventStore) and run_store._sf is event_store._sf else None
@property
def scope_digest(self) -> str:
"""Stable host-internal scope binding used by transactional consumers."""
return self._scope
def _supported(self) -> None:
if self._sf is None:
raise HostCapabilityError("UNSUPPORTED", "completed evidence requires durable run and event stores on the same database")
def _cursor(self, kind: str, position, reference: str = "") -> str:
return base64.urlsafe_b64encode(_json([1, self._scope, kind, reference, position]).encode()).decode().rstrip("=")
def _position(self, cursor: str | None, kind: str, reference: str = ""):
if cursor is None:
return None
try:
if not isinstance(cursor, str) or len(cursor) > 2048:
raise ValueError
value = json.loads(base64.b64decode(cursor + "=" * (-len(cursor) % 4), altchars=b"-_", validate=True))
if not isinstance(value, list) or len(value) != 5 or value[:4] != [1, self._scope, kind, reference]:
raise ValueError
position = value[4]
if kind == "runs":
if not isinstance(position, list) or len(position) != 2 or type(position[0]) is not int or position[0] < 0 or not isinstance(position[1], str):
raise ValueError
elif type(position) is not int or position < 0:
raise ValueError
return position
except (ValueError, TypeError, UnicodeDecodeError) as exc:
raise HostCapabilityError("INVALID_CURSOR", "cursor does not match this plugin, scope, version, or snapshot") from exc
async def list_changed_runs(self, *, cursor: str | None = None, limit: int = 200) -> RunPage:
self._supported()
limit = EvidenceLimits(max_events=limit).max_events
seq, run_id = self._position(cursor, "runs") or (-1, "")
stmt = (
select(RunRow.thread_id, RunRow.run_id, RunRow.status, RunRow.created_at, RunRow.updated_at, RunRow.stop_reason, RunRow.change_seq)
.where(
RunRow.user_id.in_(self._owners),
RunRow.operation_kind == "run",
or_(RunRow.change_seq > seq, and_(RunRow.change_seq == seq, RunRow.run_id > run_id)),
)
.order_by(RunRow.change_seq, RunRow.run_id)
.limit(limit + 1)
)
async with self._sf() as session:
rows = (await session.execute(stmt)).all()
items = []
next_cursor = cursor
for row in rows[:limit]:
item = RunStatusView(thread_id=row.thread_id, run_id=row.run_id, status=row.status, created_at=coerce_iso(row.created_at), updated_at=coerce_iso(row.updated_at), stop_reason=row.stop_reason)
candidate_cursor = self._cursor("runs", [row.change_seq, row.run_id])
if _size(RunPage(tuple([*items, item]), candidate_cursor, True)) > 1024 * 1024:
break
items.append(item)
next_cursor = candidate_cursor
return RunPage(tuple(items), next_cursor, len(rows) > len(items))
async def _begin_read(self, session) -> None:
# sqlite's legacy transaction mode does not BEGIN on SELECT. Explicitly
# start a transaction so owner/revision checks and content share a snapshot.
if session.bind.dialect.name == "sqlite":
await session.execute(text("BEGIN"))
async def _run(self, session, thread_id: str, run_id: str, *, write: bool = False):
columns = [
RunRow.thread_id,
RunRow.run_id,
RunRow.user_id,
RunRow.assistant_id,
RunRow.status,
RunRow.stop_reason,
RunRow.created_at,
RunRow.evidence_origin,
RunRow.evidence_agent_id,
RunRow.evidence_seal_state,
RunRow.evidence_seal_error,
RunRow.evidence_revision,
RunRow.evidence_upper_seq,
RunRow.evidence_event_count,
RunRow.evidence_retention_revision,
RunRow.lease_expires_at,
]
stmt = select(*columns).where(RunRow.thread_id == thread_id, RunRow.run_id == run_id, RunRow.user_id.in_(self._owners), RunRow.operation_kind == "run").with_for_update(read=not write)
row = (await session.execute(stmt)).one_or_none()
if row is None:
raise HostCapabilityError("NOT_FOUND", "run or snapshot is unavailable in this scope")
return row
@staticmethod
def _seal_state(run) -> str:
seal = run.evidence_seal_state or "partial"
if seal == "pending" and run.status not in ("pending", "running"):
lease = run.lease_expires_at
if lease is None or lease.replace(tzinfo=UTC) <= datetime.now(UTC):
return "partial"
return seal
@classmethod
def revision_for_run(cls, run) -> str:
# Include lifecycle transitions before a seal exists. Otherwise a
# pending snapshot could hide the subsequent terminal/partial record.
return hashlib.sha256(_json([run.evidence_revision, run.run_id, coerce_iso(run.created_at), run.status, cls._seal_state(run)]).encode()).hexdigest()
@staticmethod
def _event_conditions(run):
return [RunEventRow.thread_id == run.thread_id, RunEventRow.run_id == run.run_id, or_(RunEventRow.user_id == run.user_id, RunEventRow.user_id.is_(None))]
async def get_snapshot(self, *, thread_id: str, run_id: str, limits: EvidenceLimits = EvidenceLimits()) -> CompletedRunSnapshot:
self._supported()
if not isinstance(limits, EvidenceLimits):
raise ValueError("limits must be EvidenceLimits")
async with self._sf() as session:
if session.bind.dialect.name == "sqlite":
await session.execute(text("BEGIN IMMEDIATE"))
run = await self._run(session, thread_id, run_id, write=True)
revision = self.revision_for_run(run)
existing = await session.scalar(
select(CompletedRunSnapshotRow).where(
CompletedRunSnapshotRow.run_id == run_id,
CompletedRunSnapshotRow.scope_digest == self._scope,
CompletedRunSnapshotRow.evidence_revision == revision,
CompletedRunSnapshotRow.retention_revision == run.evidence_retention_revision,
)
)
if existing is not None:
snap = _snapshot(existing.snapshot_json)
if _size(snap) > limits.max_bytes:
raise HostCapabilityError("LIMIT_EXCEEDED", "snapshot envelope exceeds byte budget")
return snap
seal = self._seal_state(run)
if seal == "sealed":
upper, count = run.evidence_upper_seq, run.evidence_event_count
if upper is None or count is None:
seal = "partial"
if seal != "sealed":
upper, count = (await session.execute(select(func.max(RunEventRow.seq), func.count()).where(*self._event_conditions(run)))).one()
snap = CompletedRunSnapshot(
snapshot_ref=uuid4().hex,
evidence_revision=revision,
retention_revision=run.evidence_retention_revision,
thread_id=thread_id,
run_id=run_id,
owner_id=run.user_id,
agent_id=run.evidence_agent_id,
origin=run.evidence_origin or "unknown",
status=run.status,
stop_reason=run.stop_reason,
seal_state=seal,
seal_error=run.evidence_seal_error,
upper_event_seq=upper or 0,
event_count=count or 0,
)
if _size(snap) > limits.max_bytes:
raise HostCapabilityError("LIMIT_EXCEEDED", "snapshot envelope exceeds byte budget")
session.add(CompletedRunSnapshotRow(snapshot_ref=snap.snapshot_ref, run_id=run_id, scope_digest=self._scope, evidence_revision=revision, retention_revision=snap.retention_revision, snapshot_json=asdict(snap)))
await session.commit()
return snap
@staticmethod
def _content_bytes(session):
if session.bind.dialect.name == "postgresql":
return func.convert_to(RunEventRow.content, "UTF8")
return cast(RunEventRow.content, LargeBinary)
async def _content_hash(self, session, event_id: int, size: int, budget: dict, raw: bytes | None = None) -> str:
chunks = (size + 16383) // 16384 if raw is None else 0
remaining_time = budget["deadline"] - monotonic()
if size > budget["bytes"] or chunks > budget["queries"] or remaining_time <= 0:
raise HostCapabilityError("LIMIT_EXCEEDED", "legacy event digest exceeds the per-page verification budget")
budget["bytes"] -= size
budget["queries"] -= chunks
if raw is not None:
return hashlib.sha256(raw).hexdigest()
digest = hashlib.sha256()
try:
async with asyncio.timeout(remaining_time):
for start in range(1, size + 1, 16384):
chunk = await session.scalar(select(func.substr(self._content_bytes(session), start, 16384)).where(RunEventRow.id == event_id))
digest.update(chunk or b"")
except TimeoutError as exc:
raise HostCapabilityError("LIMIT_EXCEEDED", "legacy event digest verification deadline exceeded") from exc
return digest.hexdigest()
async def _resolve_snapshot(self, session, snapshot_ref: str):
stored = await session.scalar(select(CompletedRunSnapshotRow).where(CompletedRunSnapshotRow.snapshot_ref == snapshot_ref, CompletedRunSnapshotRow.scope_digest == self._scope))
if stored is None:
raise HostCapabilityError("NOT_FOUND", "snapshot is unavailable in this scope")
snap = _snapshot(stored.snapshot_json)
run = await self._run(session, snap.thread_id, snap.run_id)
if run.user_id != snap.owner_id or run.evidence_retention_revision != snap.retention_revision or self.revision_for_run(run) != snap.evidence_revision:
raise HostCapabilityError("EVIDENCE_EXPIRED", "snapshot revision was replaced or its events were removed")
return snap, run
async def resolve_snapshot(self, snapshot_ref: str) -> CompletedRunSnapshot:
"""Internal host composition hook; authorization is checked anew per call."""
self._supported()
async with self._sf() as session:
await self._begin_read(session)
snap, _run = await self._resolve_snapshot(session, snapshot_ref)
return snap
async def read_events(self, *, snapshot_ref: str, cursor: str | None = None, limits: EvidenceLimits = EvidenceLimits()) -> CompletedRunEventPage:
self._supported()
if not isinstance(limits, EvidenceLimits):
raise ValueError("limits must be EvidenceLimits")
after = self._position(cursor, "events", snapshot_ref) or 0
legacy_budget = {"bytes": 1024 * 1024, "queries": 64, "deadline": monotonic() + 2}
async with self._sf() as session:
await self._begin_read(session)
snap, run = await self._resolve_snapshot(session, snapshot_ref)
conditions = [*self._event_conditions(run), RunEventRow.seq > after, RunEventRow.seq <= snap.upper_event_seq]
length = (lambda col: func.octet_length(col)) if session.bind.dialect.name == "postgresql" else (lambda col: func.length(cast(col, LargeBinary)))
metadata_text = cast(RunEventRow.event_metadata, Text)
# Metadata and body lengths are projected separately; ORM never
# hydrates all rows (or an entire giant event) for a bounded page.
rows = (
await session.execute(
select(RunEventRow.id, RunEventRow.seq, RunEventRow.event_type, RunEventRow.category, RunEventRow.content_sha256, length(RunEventRow.content).label("body_size"), length(metadata_text).label("meta_size"))
.where(*conditions)
.order_by(RunEventRow.seq)
.limit(limits.max_events + 1)
)
).all()
items = []
next_cursor = cursor
for row in rows[: limits.max_events]:
remaining = limits.max_bytes - _size(CompletedRunEventPage(tuple(items), next_cursor, True))
oversized = (row.body_size or 0) + (row.meta_size or 0) > max(0, remaining - 700) // 2
body_limit = 128 if oversized else limits.max_bytes
raw_bytes, raw_meta = (await session.execute(select(func.substr(self._content_bytes(session), 1, body_limit), func.substr(metadata_text, 1, 2 if oversized else limits.max_bytes)).where(RunEventRow.id == row.id))).one()
raw = (raw_bytes or b"").decode("utf-8", errors="ignore" if oversized else "strict")
metadata = {} if oversized else json.loads(raw_meta or "{}")
content = raw or ""
if not oversized and (metadata.get("content_is_json") or metadata.get("content_is_dict")):
try:
content = json.loads(content)
except ValueError:
pass
digest = row.content_sha256 or await self._content_hash(session, row.id, row.body_size or 0, legacy_budget, None if oversized else raw_bytes or b"")
item = CompletedRunEvent(
event_id=str(row.id),
seq=row.seq,
event_type=row.event_type,
category=row.category,
content_json=_json(content),
metadata_json=_json(redact_metadata_secrets(metadata)),
content_bytes=row.body_size or 0,
content_sha256=digest,
truncated=oversized,
)
candidate_cursor = self._cursor("events", row.seq, snapshot_ref)
candidate = CompletedRunEventPage(tuple([*items, item]), candidate_cursor, True)
if _size(candidate) > limits.max_bytes:
if items:
break
item = replace(item, content_json='""', metadata_json="{}", truncated=True)
candidate = CompletedRunEventPage((item,), candidate_cursor, True)
if _size(candidate) > limits.max_bytes:
raise HostCapabilityError("LIMIT_EXCEEDED", "event identity envelope exceeds byte budget")
items.append(item)
next_cursor = candidate_cursor
return CompletedRunEventPage(tuple(items), next_cursor, len(rows) > len(items))

View File

@ -579,6 +579,7 @@ async def start_services(
session_factory: Any | None,
*,
run_evidence_reader: Any | None = None,
host_capabilities: dict[str, tuple[Any | None, Any | None]] | None = None,
attempted_services: list[tuple[str, Any]] | None = None,
) -> list[Diagnostic]:
"""Start extension services in registration order, failing open per item."""
@ -586,14 +587,24 @@ async def start_services(
if not extensions.services:
return diagnostics
deps = ExtensionRuntimeDeps(
base_deps = ExtensionRuntimeDeps(
app_store=extensions.app_store,
policy=project_host_policy(app_config),
session_factory=session_factory,
run_evidence_reader=run_evidence_reader,
)
source_deps = {}
for entry in extensions.services:
source, service = entry
evidence, mutations = (host_capabilities or {}).get(source, (None, None))
if evidence is None and mutations is None:
deps = base_deps # Preserve the legacy shared-dependency identity.
else:
from dataclasses import replace
if source not in source_deps:
source_deps[source] = replace(base_deps, completed_run_evidence=evidence, skill_mutations=mutations)
deps = source_deps[source]
if attempted_services is not None:
# Record before awaiting start(): a service may acquire resources
# and then fail or be cancelled, so it still owns stop().

View File

@ -0,0 +1,91 @@
"""Restart-bound, host-owned grants; not a sandbox for trusted Python code."""
from __future__ import annotations
import hashlib
import json
import re
from collections import Counter
from dataclasses import dataclass
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator
from deerflow.extensions.model_access import ExtensionHostAccess
class EvidenceAccess(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
owners: tuple[str, ...] = ()
@field_validator("owners")
@classmethod
def exact_identifiers(cls, values):
if len(values) > 1000 or len(set(values)) != len(values):
raise ValueError("grant lists must be bounded and unique")
if any(not value or len(value) > 128 or value.strip() != value or "*" in value or any(ord(c) < 32 for c in value) for value in values):
raise ValueError("grant lists require exact nonempty identifiers, without wildcards")
return values
class SkillMutationAccess(EvidenceAccess):
operations: tuple[Literal["stage", "check", "commit", "revert"], ...] = ()
trigger_agents: tuple[str, ...] = ()
target_skills: tuple[str, ...] = ()
topology: Literal["single_host_local"] | None = Field(
default=None,
description="Operator assertion: all writers upgraded; one POSIX host and shared local disk, not NFS/CSI. Required for mutation service.",
)
@field_validator("trigger_agents", "target_skills")
@classmethod
def exact_targets(cls, values):
return cls.exact_identifiers(values)
@field_validator("target_skills")
@classmethod
def skill_names(cls, values):
if any(not re.fullmatch(r"[a-z0-9]+(?:-[a-z0-9]+)*", name) or len(name) > 64 for name in values):
raise ValueError("target_skills must contain canonical skill names")
return values
class HostAccess(ExtensionHostAccess):
model_config = ConfigDict(extra="forbid", frozen=True)
evidence: EvidenceAccess = Field(default_factory=EvidenceAccess)
skill_mutations: SkillMutationAccess = Field(default_factory=SkillMutationAccess)
@property
def granted(self):
return bool(self.evidence.owners or self.skill_mutations.owners or self.skill_mutations.operations)
@dataclass(frozen=True)
class BoundHostAccess:
plugin_id: str
access: HostAccess
def allows(self, operation: str, owner_id: str, name: str) -> bool:
grant = self.access.skill_mutations
return operation in grant.operations and owner_id in grant.owners and name in grant.target_skills
def bind_host_access(specs) -> dict[str, BoundHostAccess]:
"""Source attribution is host registry data, never a plugin-supplied ID.
Ungranted legacy duplicate entry points stay legal. Once a source is
granted, all of its enabled registrations must identify one installation.
Changing the name or entry point deliberately creates a new durable identity.
"""
enabled = [spec for spec in specs if spec.enabled]
names = Counter(spec.name for spec in enabled)
entries = Counter(spec.use for spec in enabled)
bindings = {}
for spec in enabled:
if not spec.host_access.granted:
continue
if not spec.name or names[spec.name] != 1 or entries[spec.use] != 1:
raise ValueError("host_access requires a unique explicit plugin name and entry point")
identity = json.dumps([spec.name, spec.use], ensure_ascii=True, separators=(",", ":"))
bindings[spec.use] = BoundHostAccess(hashlib.sha256(identity.encode()).hexdigest(), spec.host_access)
return bindings

View File

@ -0,0 +1,151 @@
"""Gateway-owned capability assembly and plugin-independent recovery lifetime."""
from __future__ import annotations
import asyncio
import logging
from deerflow_extension_api import HostCapabilityError
from sqlalchemy import select
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
from deerflow.extensions.host_access import bind_host_access
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOwnerRow
from deerflow.skills.mutations.guard import SkillMutationRuntime, configure_mutation_runtime
from deerflow.skills.mutations.repository import SkillMutationRepository
from deerflow.skills.mutations.topology import PublicationTopology, mutation_session_factory
from deerflow.skills.mutations.workers import MutationWorkers
from deerflow.utils.file_io import await_drained
logger = logging.getLogger(__name__)
class HostCapabilities:
"""Register close() before start(); extension services must stop before close().
Grants are restart-bound. Persisted enrollment is intentionally independent
of the current plugin list: uninstalling a plugin cannot turn off recovery
or let an old managed writer bypass an unresolved publication.
"""
def __init__(self, config, run_store, event_store, *, root=None, storage_factory=None, recovery_factory=None, service_factory=None):
self.config = config
self.run_store = run_store
self.event_store = event_store
self.root = root
self.storage_factory = storage_factory
self.recovery_factory = recovery_factory
self.service_factory = service_factory
self.bindings = {}
self.runtime = None
self.recovery = None
self.workers = None
self._engine = None
self._topology = None
self._maintenance = None
self._services = []
async def start(self):
access = bind_host_access(getattr(self.config, "plugins", ()))
mutation_bindings = {source: binding for source, binding in access.items() if binding.access.skill_mutations.owners}
for binding in mutation_bindings.values():
if binding.access.skill_mutations.topology != "single_host_local":
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY", "Skill mutations require an explicit single_host_local operator assertion")
if not set(binding.access.skill_mutations.owners).issubset(binding.access.evidence.owners):
raise HostCapabilityError("INVALID_GRANT", "Skill mutation owners must also have completed-evidence access")
owners = set()
sf = getattr(self.run_store, "_sf", None)
if sf is not None:
async with sf() as session:
owners.update(await session.scalars(select(SkillOwnerRow.owner_id)))
owners.update(await session.scalars(select(SkillAssetRow.owner_id).distinct()))
for binding in mutation_bindings.values():
owners.update(binding.access.skill_mutations.owners)
for source, binding in access.items():
evidence = None
if binding.access.evidence.owners:
evidence = HostCompletedRunEvidenceReader(self.run_store, self.event_store, plugin_id=binding.plugin_id, owner_ids=binding.access.evidence.owners)
if source in mutation_bindings:
# Refuse a working-looking automatic API on unsupported stores.
evidence._supported()
self.bindings[source] = (evidence, None)
if not owners:
return
if self.config.database.backend not in {"sqlite", "postgres"}:
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY")
self.workers = MutationWorkers()
def initialize(_check):
from deerflow.config.paths import get_paths
self._topology = PublicationTopology(self.config.database.backend, self.root or get_paths().base_dir)
self._topology.acquire()
self._engine, sessions = mutation_session_factory(self.config.database.app_sync_sqlalchemy_url, postgres_schema=self.config.database.postgres_schema)
self.runtime = SkillMutationRuntime(SkillMutationRepository(sessions), owners=frozenset(owners))
if self.storage_factory is None:
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
self.storage_factory = lambda owner: UserScopedSkillStorage(owner, app_config=self.config)
factory = self.recovery_factory
if factory is None:
from deerflow.skills.mutations.recovery import SkillMutationRecovery
factory = SkillMutationRecovery
self.recovery = factory(self.runtime, self.storage_factory)
configure_mutation_runtime(self.runtime)
self.recovery.recover_all()
self.recovery.collect_garbage()
await self.workers.run(initialize)
if mutation_bindings:
from deerflow.config import get_app_config
from deerflow.skills.mutations.scanner import CandidateScanner
factory = self.service_factory
if factory is None:
from deerflow.skills.mutations.service import HostSkillMutationService
factory = HostSkillMutationService
scanner = CandidateScanner(get_app_config)
for source, binding in mutation_bindings.items():
evidence, _ = self.bindings[source]
service = factory(binding, runtime=self.runtime, evidence=evidence, storage_factory=self.storage_factory, scanner=scanner, workers=self.workers, recovery=self.recovery)
self._services.append(service)
self.bindings[source] = (evidence, service)
self._maintenance = asyncio.create_task(self._maintain(), name="skill-mutation-recovery")
async def _maintain(self):
while True:
await asyncio.sleep(60)
try:
await self.workers.run(lambda _check: (self.recovery.recover_all(), self.recovery.collect_garbage()))
except Exception:
# No candidate bytes, model diagnostics, DB parameters or paths.
logger.warning("Skill mutation maintenance unavailable; owner readiness barriers remain active")
async def close(self):
async def drain():
if self._maintenance is not None:
self._maintenance.cancel()
await asyncio.gather(self._maintenance, return_exceptions=True)
self._maintenance = None
for service in self._services:
close = getattr(service, "close", None)
if close is not None:
await close()
if self.workers is not None:
await self.workers.close()
if self.runtime is not None:
configure_mutation_runtime(None)
if self._engine is not None:
await asyncio.to_thread(self._engine.dispose)
self._engine = None
if self._topology is not None:
await asyncio.to_thread(self._topology.close)
self._topology = None
await await_drained(drain())

View File

@ -14,9 +14,10 @@ from dataclasses import dataclass
from typing import Any, Literal
from deerflow_extension_api import API_VERSION
from pydantic import BaseModel, ConfigDict, Field
from pydantic import BaseModel, ConfigDict, Field, model_validator
from deerflow.extensions.model_access import ExtensionHostAccess, ModelInvocationScope
from deerflow.extensions.host_access import HostAccess, bind_host_access
from deerflow.extensions.model_access import ModelInvocationScope
from deerflow.extensions.registry import ExtensionRegistry, LoadedExtensions
from deerflow.persistence.migrations._env_filters import register_extension_table_prefix
from deerflow.reflection import resolve_variable
@ -44,11 +45,18 @@ class ExtensionSpec(BaseModel):
description="Installed Python distribution recorded by the extension manager",
)
use: str = Field(description="Entry point path, e.g. 'my_extension:install'")
host_access: ExtensionHostAccess = Field(default_factory=ExtensionHostAccess)
config: dict[str, Any] = Field(
default_factory=dict,
description="Extension-private configuration, passed to install() verbatim",
)
host_access: HostAccess = Field(default_factory=HostAccess)
@model_validator(mode="after")
def _validate_host_identity(self):
if self.host_access.granted and (not self.name or self.name.strip() != self.name or len(self.name) > 128):
raise ValueError("host_access requires an explicit stable plugin name")
return self
required: bool = Field(
default=False,
description="When true, a load failure aborts startup instead of being skipped",
@ -156,6 +164,8 @@ def load_extensions(specs: Sequence[ExtensionSpec]) -> tuple[LoadedExtensions, l
the Gateway still starts. `required: true` flips that to fail-closed for
extensions whose absence changes behaviour rather than just observability.
"""
# Validate attribution before importing any extension, even optional ones.
bind_host_access(specs)
registry = ExtensionRegistry()
diagnostics: list[Diagnostic] = []
loaded_sources: list[str] = []

View File

@ -29,11 +29,12 @@ The empty-DB path keeps using `create_all` because `Base.metadata` is the only a
`0028_parked_attempts` → `0029_scheduler_agent_tasks` →
`0030_notification_claim_tokens` → `0031_scheduled_streak_boundary` →
`0032_activity_and_task_events` → `0033_batch_result_artifact` → `0034_run_event_seq_watermark` →
`0035_login_throttle` → `0036_run_idempotency_request` → `0037_project_document_summaries` →
`0038_channel_thread_bindings` → `0039_user_disabled` →
`0040_scheduled_group_notice` → `0041_im_notice_preferences` →
`0042_channel_leases` → `0043_runtime_im_notices` →
`0044_channel_runtime_configs` (current head). The preference
`0035_login_throttle` → `0036_run_idempotency_request` →
`0037_project_document_summaries` → `0038_channel_thread_bindings` →
`0039_user_disabled` → `0040_scheduled_group_notice` →
`0041_im_notice_preferences` → `0042_channel_leases` →
`0043_runtime_im_notices` → `0044_channel_runtime_configs` →
`0045_completed_run_evidence` → `0046_skill_mutations` (current head). The preference
revision adds a separate owner/key table with a cascading users foreign key and
does not alter users; the project-documents revision adds a new owner-scoped
shelf table, and the MCP lease-token revision adds two nullable token columns to
@ -192,7 +193,6 @@ on installs that never enabled it. The convention is:
- `extensions/loader.py::load_extensions` — registers each spec's `table_prefix` with `register_extension_table_prefix()`
- Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter, including extension-owned tables), `tests/test_extension_loader.py::TestTablePrefixRegistration` (spec-to-filter wiring), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor), `tests/test_migration_0004_run_ownership_dedupe.py` + `tests/test_migration_0007_scheduled_run_active_dedupe.py` (dedupe-before-unique-index pre-steps), `tests/test_migration_0025_repair_run_change_seq.py` (issue #5516 skipped-revision heal)
`0041_im_notice_preferences` adds the owner/provider proactive-notification
boolean table without a users FK (local synthetic IM owners are supported).
Upgrade/drop are table-guarded; fresh bootstrap registers the ORM model.
@ -200,3 +200,6 @@ It follows 0040 without reparenting history, has no backfill and leaves the
forward-compatibility floor unchanged. Downgrade removes the saved choices;
reenabling a provider never reopens notifications canceled before downgrade.
Tests: `test_migration_0041_im_notice_preferences.py`.
- `0045_completed_run_evidence` adds nullable evidence sealing fields and completed-run snapshots after `0044_channel_runtime_configs`.
- `0046_skill_mutations` adds durable owner enrollment, asset revisions, proposals, publication operations and scan leases. The single-head pin lives in `tests/test_migration_0046_skill_mutations.py`.

View File

@ -0,0 +1,48 @@
"""Add explicit evidence seal and durable bounded snapshot references."""
import sqlalchemy as sa
from alembic import op
revision = "0045_completed_run_evidence"
down_revision = "0044_channel_runtime_configs"
branch_labels = None
depends_on = None
def upgrade() -> None:
from deerflow.persistence.migrations._helpers import safe_add_column
safe_add_column("run_events", sa.Column("content_sha256", sa.String(64), nullable=True))
for column in (
sa.Column("evidence_origin", sa.String(32), nullable=True),
sa.Column("evidence_agent_id", sa.String(128), nullable=True),
sa.Column("evidence_seal_state", sa.String(16), nullable=True),
sa.Column("evidence_seal_error", sa.String(512), nullable=True),
sa.Column("evidence_revision", sa.String(64), nullable=True),
sa.Column("evidence_upper_seq", sa.BigInteger(), nullable=True),
sa.Column("evidence_event_count", sa.BigInteger(), nullable=True),
sa.Column("evidence_retention_revision", sa.BigInteger(), nullable=False, server_default=sa.text("0")),
):
safe_add_column("runs", column)
if "completed_run_snapshots" not in sa.inspect(op.get_bind()).get_table_names():
op.create_table(
"completed_run_snapshots",
sa.Column("snapshot_ref", sa.String(64), primary_key=True),
sa.Column("run_id", sa.String(64), nullable=False),
sa.Column("scope_digest", sa.String(64), nullable=False),
sa.Column("evidence_revision", sa.String(64), nullable=False),
sa.Column("retention_revision", sa.BigInteger(), nullable=False),
sa.Column("snapshot_json", sa.JSON(), nullable=False),
sa.UniqueConstraint("run_id", "scope_digest", "evidence_revision", "retention_revision", name="uq_completed_run_snapshot_revision"),
)
op.create_index("ix_completed_run_snapshots_run_id", "completed_run_snapshots", ["run_id"])
def downgrade() -> None:
from deerflow.persistence.migrations._helpers import safe_drop_column
safe_drop_column("run_events", "content_sha256")
if "completed_run_snapshots" in sa.inspect(op.get_bind()).get_table_names():
op.drop_table("completed_run_snapshots")
for name in ("evidence_retention_revision", "evidence_event_count", "evidence_upper_seq", "evidence_revision", "evidence_seal_error", "evidence_seal_state", "evidence_agent_id", "evidence_origin"):
safe_drop_column("runs", name)

View File

@ -0,0 +1,161 @@
"""skill mutation publication journal
Revision ID: 0046_skill_mutations
Revises: 0045_completed_run_evidence
Create Date: 2026-09-21 11:28:24.923105
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "0046_skill_mutations"
down_revision: str | Sequence[str] | None = "0045_completed_run_evidence"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _create_table(name, *columns):
# Legacy databases may already contain tables from an older create_all.
inspector = sa.inspect(op.get_bind())
if not inspector.has_table(name):
op.create_table(name, *columns)
else:
existing = {column["name"] for column in inspector.get_columns(name)}
required = {column.name for column in columns if isinstance(column, sa.Column)}
if not required <= existing:
raise RuntimeError(f"Incompatible preexisting skill mutation table: {name}")
def _create_index(name, table, columns):
if name not in {index["name"] for index in sa.inspect(op.get_bind()).get_indexes(table)}:
op.create_index(name, table, columns, unique=False)
def upgrade() -> None:
"""Upgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
_create_table(
"skill_mutation_assets",
sa.Column("target_ref", sa.String(length=32), nullable=False),
sa.Column("owner_id", sa.String(length=36), nullable=False),
sa.Column("name", sa.String(length=64), nullable=False),
sa.Column("incarnation_id", sa.String(length=32), nullable=False),
sa.Column("mutation_seq", sa.Integer(), nullable=False),
sa.Column("content_digest", sa.String(length=64), nullable=False),
sa.Column("enabled", sa.Boolean(), nullable=False),
sa.Column("deleted", sa.Boolean(), nullable=False),
sa.Column("mutating", sa.Boolean(), nullable=False),
sa.Column("operation_id", sa.String(length=32), nullable=True),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("target_ref"),
sa.UniqueConstraint("owner_id", "name", name="uq_skill_asset_owner_name"),
)
_create_index("ix_skill_mutation_assets_owner_id", "skill_mutation_assets", ["owner_id"])
_create_table(
"skill_mutation_operations",
sa.Column("operation_id", sa.String(length=32), nullable=False),
sa.Column("plugin_id", sa.String(length=128), nullable=False),
sa.Column("owner_id", sa.String(length=36), nullable=False),
sa.Column("target_ref", sa.String(length=32), nullable=False),
sa.Column("name", sa.String(length=64), nullable=False),
sa.Column("method", sa.String(length=16), nullable=False),
sa.Column("idempotency_key", sa.String(length=128), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("proposal_id", sa.String(length=32), nullable=True),
sa.Column("reverts_operation_id", sa.String(length=32), nullable=True),
sa.Column("before_operation_id", sa.String(length=32), nullable=True),
sa.Column("before_revision", sa.JSON(), nullable=False),
sa.Column("after_revision", sa.JSON(), nullable=False),
sa.Column("before_blob", sa.LargeBinary(), nullable=True),
sa.Column("after_blob", sa.LargeBinary(), nullable=True),
sa.Column("generation", sa.Integer(), nullable=False),
sa.Column("publication", sa.String(length=16), nullable=False),
sa.Column("views", sa.String(length=16), nullable=False),
sa.Column("superseded_by_generation", sa.Integer(), nullable=True),
sa.Column("created_at", sa.Float(), nullable=False),
sa.Column("rollback_expires_at", sa.Float(), nullable=False),
sa.Column("assessment", sa.JSON(), nullable=True),
sa.Column("error_code", sa.String(length=64), nullable=True),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("operation_id"),
sa.UniqueConstraint("plugin_id", "method", "idempotency_key", name="uq_skill_operation_key"),
sa.UniqueConstraint("proposal_id"),
)
_create_index("ix_skill_mutation_operations_owner_id", "skill_mutation_operations", ["owner_id"])
_create_index("ix_skill_mutation_operations_publication", "skill_mutation_operations", ["publication"])
_create_table(
"skill_mutation_owners",
sa.Column("owner_id", sa.String(length=36), nullable=False),
sa.Column("generation", sa.Integer(), nullable=False),
sa.Column("deleting", sa.Boolean(), nullable=False),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("owner_id"),
)
_create_table(
"skill_mutation_proposals",
sa.Column("proposal_id", sa.String(length=32), nullable=False),
sa.Column("plugin_id", sa.String(length=128), nullable=False),
sa.Column("owner_id", sa.String(length=36), nullable=False),
sa.Column("target_ref", sa.String(length=32), nullable=False),
sa.Column("name", sa.String(length=64), nullable=False),
sa.Column("base_revision", sa.JSON(), nullable=False),
sa.Column("sources", sa.JSON(), nullable=False),
sa.Column("candidate_hash", sa.String(length=64), nullable=False),
sa.Column("package_blob", sa.LargeBinary(), nullable=True),
sa.Column("candidate_blob", sa.LargeBinary(), nullable=True),
sa.Column("state", sa.String(length=16), nullable=False),
sa.Column("expires_at", sa.Float(), nullable=False),
sa.Column("check_result", sa.JSON(), nullable=True),
sa.Column("idempotency_key", sa.String(length=128), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("operation_id", sa.String(length=32), nullable=True),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("proposal_id"),
sa.UniqueConstraint("plugin_id", "idempotency_key", name="uq_skill_proposal_key"),
)
_create_index("ix_skill_mutation_proposals_owner_id", "skill_mutation_proposals", ["owner_id"])
_create_table(
"skill_mutation_scan_attempts",
sa.Column("token", sa.String(length=32), nullable=False),
sa.Column("plugin_id", sa.String(length=128), nullable=False),
sa.Column("owner_id", sa.String(length=36), nullable=False),
sa.Column("subject", sa.String(length=80), nullable=False),
sa.Column("started_at", sa.Float(), nullable=False),
sa.Column("lease_until", sa.Float(), nullable=False),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("token"),
)
_create_index("ix_skill_mutation_scan_attempts_owner_id", "skill_mutation_scan_attempts", ["owner_id"])
_create_index("ix_skill_mutation_scan_attempts_plugin_id", "skill_mutation_scan_attempts", ["plugin_id"])
# ### end Alembic commands ###
def downgrade() -> None:
"""Downgrade schema."""
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table("skill_mutation_scan_attempts")
with op.batch_alter_table("skill_mutation_proposals", schema=None) as batch_op:
batch_op.drop_index(batch_op.f("ix_skill_mutation_proposals_owner_id"))
op.drop_table("skill_mutation_proposals")
op.drop_table("skill_mutation_owners")
with op.batch_alter_table("skill_mutation_operations", schema=None) as batch_op:
batch_op.drop_index(batch_op.f("ix_skill_mutation_operations_publication"))
batch_op.drop_index(batch_op.f("ix_skill_mutation_operations_owner_id"))
op.drop_table("skill_mutation_operations")
with op.batch_alter_table("skill_mutation_assets", schema=None) as batch_op:
batch_op.drop_index(batch_op.f("ix_skill_mutation_assets_owner_id"))
op.drop_table("skill_mutation_assets")
# ### end Alembic commands ###

View File

@ -36,6 +36,7 @@ from deerflow.persistence.run.model import RunChangeClockRow, RunRow
from deerflow.persistence.scheduled_task_events.model import ScheduledTaskEventRow
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOperationRow, SkillOwnerRow, SkillProposalRow
from deerflow.persistence.subagent_batches.model import SubagentBatchItemRow, SubagentBatchRow
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.thread_reads.model import ThreadReadMarkerRow, ThreadReadVersionRow
@ -64,6 +65,10 @@ __all__ = [
"RunEventThreadSeqRow",
"RunChangeClockRow",
"RunRow",
"SkillAssetRow",
"SkillOperationRow",
"SkillOwnerRow",
"SkillProposalRow",
"ScheduledTaskEventRow",
"ScheduledTaskRow",
"ScheduledTaskRunRow",

View File

@ -25,6 +25,9 @@ class RunEventRow(Base):
category: Mapped[str] = mapped_column(String(RUN_EVENT_CATEGORY_MAX_LENGTH), nullable=False)
# Category values and semantics are defined by runtime/events/catalog.py
content: Mapped[str] = mapped_column(Text, default="")
# Digest of the exact persisted UTF-8 content, after serialization/truncation.
# NULL denotes legacy ingestion; event metadata is never trusted as a digest.
content_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
event_metadata: Mapped[dict] = mapped_column(JSON, default=dict)
seq: Mapped[int] = mapped_column(nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))

View File

@ -3,8 +3,9 @@
from __future__ import annotations
from datetime import UTC, datetime
from uuid import uuid4
from sqlalchemy import JSON, BigInteger, DateTime, Index, Integer, String, Text, text
from sqlalchemy import JSON, BigInteger, DateTime, Index, Integer, String, Text, UniqueConstraint, text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@ -61,6 +62,14 @@ class RunRow(Base):
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
change_seq: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default=text("0"))
evidence_origin: Mapped[str | None] = mapped_column(String(32), nullable=True)
evidence_agent_id: Mapped[str | None] = mapped_column(String(128), nullable=True)
evidence_seal_state: Mapped[str | None] = mapped_column(String(16), nullable=True)
evidence_seal_error: Mapped[str | None] = mapped_column(String(512), nullable=True)
evidence_revision: Mapped[str | None] = mapped_column(String(64), nullable=True, default=lambda: uuid4().hex)
evidence_upper_seq: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
evidence_event_count: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
evidence_retention_revision: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default=text("0"))
# Server-owned origin kind of the run (``deerflow_origin``: schedule,
# im_channel, github, extension, mcp_notification). NULL means an
# interactive or legacy run; legacy rows are never backfilled.
@ -95,3 +104,18 @@ class RunChangeClockRow(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True)
value: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default=text("0"))
class CompletedRunSnapshotRow(Base):
"""Small durable boundaries only; never a copy of event contents."""
__tablename__ = "completed_run_snapshots"
snapshot_ref: Mapped[str] = mapped_column(String(64), primary_key=True)
run_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
scope_digest: Mapped[str] = mapped_column(String(64), nullable=False)
evidence_revision: Mapped[str] = mapped_column(String(64), nullable=False)
retention_revision: Mapped[int] = mapped_column(BigInteger, nullable=False)
snapshot_json: Mapped[dict] = mapped_column(JSON, nullable=False)
__table_args__ = (UniqueConstraint("run_id", "scope_digest", "evidence_revision", "retention_revision", name="uq_completed_run_snapshot_revision"),)

View File

@ -10,12 +10,13 @@ from __future__ import annotations
import json
from datetime import UTC, datetime, timedelta
from typing import Any
from uuid import uuid4
from sqlalchemy import and_, case, delete, func, or_, select, update
from sqlalchemy import and_, case, delete, func, or_, select, text, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
from deerflow.persistence.run.model import CompletedRunSnapshotRow, RunChangeClockRow, RunRow
from deerflow.runtime.run_origin import origin_kind_of
from deerflow.runtime.runs.store.base import (
LeaseRenewal,
@ -138,6 +139,8 @@ class RunRepository(RunStore):
owner_worker_id: str | None = None,
lease_expires_at: str | None = None,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
):
"""Insert or update a run row.
@ -152,6 +155,8 @@ class RunRepository(RunStore):
lease_dt = datetime.fromisoformat(lease_expires_at) if lease_expires_at else None
values = {
"thread_id": thread_id,
"evidence_origin": evidence_origin,
"evidence_agent_id": evidence_agent_id,
"assistant_id": assistant_id,
"user_id": resolved_user_id,
"model_name": self._normalize_model_name(model_name),
@ -175,12 +180,53 @@ class RunRepository(RunStore):
values["change_seq"] = await self._next_change_seq(session)
row = await session.get(RunRow, run_id)
if row is None:
session.add(RunRow(run_id=run_id, created_at=created, **values))
session.add(RunRow(run_id=run_id, created_at=created, evidence_seal_state="pending", **values))
else:
for key, value in values.items():
setattr(row, key, value)
await session.commit()
async def seal_evidence(self, run_id: str, *, owner_worker_id: str, receipt) -> bool:
"""Persist a completed drain and changed-run position in one fenced transaction."""
from deerflow.runtime.events.evidence import JournalSealReceipt
if not isinstance(receipt, JournalSealReceipt) or receipt.session_factory is not self._sf or receipt.run_id != run_id:
return False
async with self._sf() as session:
change_seq = await self._next_change_seq(session)
result = await session.execute(
update(RunRow)
.where(
RunRow.run_id == run_id,
RunRow.thread_id == receipt.thread_id,
RunRow.owner_worker_id == owner_worker_id,
or_(RunRow.lease_expires_at.is_(None), RunRow.lease_expires_at > datetime.now(UTC)),
RunRow.evidence_retention_revision == receipt.retention_revision,
RunRow.status.in_(("success", "error", "timeout", "interrupted")),
RunRow.evidence_seal_state == "pending",
)
.values(evidence_seal_state="sealed", evidence_revision=uuid4().hex, evidence_upper_seq=receipt.upper_event_seq, evidence_event_count=receipt.event_count, change_seq=change_seq)
)
await session.commit()
return result.rowcount != 0
async def mark_evidence_partial(self, run_id: str, *, owner_worker_id: str, error: str) -> bool:
"""Make a failed seal discoverable without changing business outcome."""
async with self._sf() as session:
change_seq = await self._next_change_seq(session)
result = await session.execute(
update(RunRow)
.where(
RunRow.run_id == run_id,
RunRow.owner_worker_id == owner_worker_id,
or_(RunRow.lease_expires_at.is_(None), RunRow.lease_expires_at > datetime.now(UTC)),
RunRow.evidence_seal_state == "pending",
)
.values(evidence_seal_state="partial", evidence_seal_error=error[:512], evidence_revision=uuid4().hex, change_seq=change_seq)
)
await session.commit()
return result.rowcount != 0
async def get(
self,
run_id,
@ -386,11 +432,14 @@ class RunRepository(RunStore):
):
resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.delete")
async with self._sf() as session:
row = await session.get(RunRow, run_id)
if session.bind.dialect.name == "sqlite":
await session.execute(text("BEGIN IMMEDIATE"))
row = await session.get(RunRow, run_id, with_for_update=True)
if row is None:
return
if resolved_user_id is not None and row.user_id != resolved_user_id:
return
await session.execute(delete(CompletedRunSnapshotRow).where(CompletedRunSnapshotRow.run_id == run_id))
await session.delete(row)
await session.commit()
@ -426,6 +475,14 @@ class RunRepository(RunStore):
conditions.append(RunRow.user_id == resolved_user_id)
async with self._sf() as session:
if session.bind.dialect.name == "sqlite":
await session.execute(text("BEGIN IMMEDIATE"))
# The run-row lock serializes snapshot creation and cleanup on PG;
# SQLite's immediate transaction supplies the equivalent fence.
# Acquire locks in a separate statement: READ COMMITTED must take a
# fresh snapshot after waiting for an in-flight snapshot creator.
await session.execute(select(RunRow.run_id).where(*conditions).with_for_update())
await session.execute(delete(CompletedRunSnapshotRow).where(CompletedRunSnapshotRow.run_id.in_(select(RunRow.run_id).where(*conditions))))
count = await session.scalar(select(func.count()).select_from(RunRow).where(*conditions)) or 0
if count:
await session.execute(delete(RunRow).where(*conditions))
@ -793,6 +850,8 @@ class RunRepository(RunStore):
values: dict[str, Any] = {
"status": "error",
"error": error,
"evidence_seal_state": "partial",
"evidence_seal_error": "Worker lease expired without a completed journal seal.",
"updated_at": datetime.now(UTC),
}
if stop_reason is not None:
@ -811,6 +870,47 @@ class RunRepository(RunStore):
await session.commit()
return result.rowcount != 0
async def recover_unsealed_evidence(
self,
*,
before: str | None = None,
grace_seconds: int = 10,
exclude_run_ids: tuple[str, ...] = (),
) -> int:
"""Publish crash-interrupted terminal seals without changing business status.
Clock-first locking matches other run mutations. The bounded scan and
conditional update run in one transaction, so a renewed lease wins.
"""
before_dt = datetime.fromisoformat(before) if isinstance(before, str) else before or datetime.now(UTC)
cutoff = datetime.now(UTC) - timedelta(seconds=grace_seconds)
predicates = (
RunRow.operation_kind == "run",
RunRow.status.in_(("success", "error", "timeout", "interrupted")),
RunRow.evidence_seal_state == "pending",
RunRow.created_at <= before_dt,
_lease_expired_or_null(RunRow.lease_expires_at, cutoff),
RunRow.run_id.not_in(exclude_run_ids),
)
async with self._sf() as session:
change_seq = await self._next_change_seq(session)
candidates = tuple((await session.scalars(select(RunRow.run_id).where(*predicates).order_by(RunRow.created_at, RunRow.run_id).limit(200))).all())
if not candidates:
await session.rollback()
return 0
result = await session.execute(
update(RunRow)
.where(RunRow.run_id.in_(candidates), *predicates)
.values(
evidence_seal_state="partial",
evidence_seal_error="Worker lease expired without a completed journal seal.",
evidence_revision=uuid4().hex,
change_seq=change_seq,
)
)
await session.commit()
return result.rowcount
async def list_inflight_with_expired_lease(
self,
*,
@ -854,6 +954,8 @@ class RunRepository(RunStore):
created_at: str | None = None,
grace_seconds: int = 10,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
) -> tuple[dict[str, Any], list[dict[str, Any]]]:
"""Atomically create a run with cross-process thread-uniqueness.
@ -879,6 +981,9 @@ class RunRepository(RunStore):
values = {
"thread_id": thread_id,
"evidence_origin": evidence_origin,
"evidence_agent_id": evidence_agent_id,
"evidence_seal_state": "pending",
"assistant_id": assistant_id,
"user_id": resolved_user_id,
"model_name": self._normalize_model_name(model_name),

View File

@ -0,0 +1 @@
"""Durable host state for managed Skill changes."""

View File

@ -0,0 +1,89 @@
"""Host-owned skill version and publication records (Alembic managed)."""
from __future__ import annotations
from sqlalchemy import JSON, Boolean, ForeignKey, Integer, LargeBinary, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
class SkillOwnerRow(Base):
__tablename__ = "skill_mutation_owners"
owner_id: Mapped[str] = mapped_column(String(36), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True)
generation: Mapped[int] = mapped_column(Integer, default=0)
deleting: Mapped[bool] = mapped_column(Boolean, default=False)
class SkillAssetRow(Base):
__tablename__ = "skill_mutation_assets"
target_ref: Mapped[str] = mapped_column(String(32), primary_key=True)
owner_id: Mapped[str] = mapped_column(String(36), ForeignKey("users.id", ondelete="CASCADE"), index=True)
name: Mapped[str] = mapped_column(String(64))
incarnation_id: Mapped[str] = mapped_column(String(32))
mutation_seq: Mapped[int] = mapped_column(Integer)
content_digest: Mapped[str] = mapped_column(String(64))
enabled: Mapped[bool] = mapped_column(Boolean)
deleted: Mapped[bool] = mapped_column(Boolean, default=False)
mutating: Mapped[bool] = mapped_column(Boolean, default=False)
operation_id: Mapped[str | None] = mapped_column(String(32), nullable=True)
__table_args__ = (UniqueConstraint("owner_id", "name", name="uq_skill_asset_owner_name"),)
class SkillProposalRow(Base):
__tablename__ = "skill_mutation_proposals"
proposal_id: Mapped[str] = mapped_column(String(32), primary_key=True)
plugin_id: Mapped[str] = mapped_column(String(128))
owner_id: Mapped[str] = mapped_column(String(36), ForeignKey("users.id", ondelete="CASCADE"), index=True)
target_ref: Mapped[str] = mapped_column(String(32))
name: Mapped[str] = mapped_column(String(64))
base_revision: Mapped[dict] = mapped_column(JSON)
sources: Mapped[list] = mapped_column(JSON)
candidate_hash: Mapped[str] = mapped_column(String(64))
package_blob: Mapped[bytes | None] = mapped_column(LargeBinary, nullable=True, deferred=True)
candidate_blob: Mapped[bytes | None] = mapped_column(LargeBinary, nullable=True, deferred=True)
state: Mapped[str] = mapped_column(String(16))
expires_at: Mapped[float]
check_result: Mapped[dict | None] = mapped_column(JSON, nullable=True)
idempotency_key: Mapped[str] = mapped_column(String(128))
request_hash: Mapped[str] = mapped_column(String(64))
operation_id: Mapped[str | None] = mapped_column(String(32), nullable=True)
__table_args__ = (UniqueConstraint("plugin_id", "idempotency_key", name="uq_skill_proposal_key"),)
class SkillOperationRow(Base):
__tablename__ = "skill_mutation_operations"
operation_id: Mapped[str] = mapped_column(String(32), primary_key=True)
plugin_id: Mapped[str] = mapped_column(String(128))
owner_id: Mapped[str] = mapped_column(String(36), ForeignKey("users.id", ondelete="CASCADE"), index=True)
target_ref: Mapped[str] = mapped_column(String(32))
name: Mapped[str] = mapped_column(String(64))
method: Mapped[str] = mapped_column(String(16))
idempotency_key: Mapped[str] = mapped_column(String(128))
request_hash: Mapped[str] = mapped_column(String(64))
proposal_id: Mapped[str | None] = mapped_column(String(32), nullable=True, unique=True)
reverts_operation_id: Mapped[str | None] = mapped_column(String(32), nullable=True)
before_operation_id: Mapped[str | None] = mapped_column(String(32), nullable=True)
before_revision: Mapped[dict] = mapped_column(JSON)
after_revision: Mapped[dict] = mapped_column(JSON)
before_blob: Mapped[bytes | None] = mapped_column(LargeBinary, nullable=True, deferred=True)
after_blob: Mapped[bytes | None] = mapped_column(LargeBinary, nullable=True, deferred=True)
generation: Mapped[int] = mapped_column(Integer)
publication: Mapped[str] = mapped_column(String(16), index=True)
views: Mapped[str] = mapped_column(String(16))
superseded_by_generation: Mapped[int | None] = mapped_column(Integer, nullable=True)
created_at: Mapped[float]
rollback_expires_at: Mapped[float]
assessment: Mapped[dict | None] = mapped_column(JSON, nullable=True)
error_code: Mapped[str | None] = mapped_column(String(64), nullable=True)
__table_args__ = (UniqueConstraint("plugin_id", "method", "idempotency_key", name="uq_skill_operation_key"),)
class SkillScanAttemptRow(Base):
__tablename__ = "skill_mutation_scan_attempts"
token: Mapped[str] = mapped_column(String(32), primary_key=True)
plugin_id: Mapped[str] = mapped_column(String(128), index=True)
owner_id: Mapped[str] = mapped_column(String(36), ForeignKey("users.id", ondelete="CASCADE"), index=True)
subject: Mapped[str] = mapped_column(String(80))
started_at: Mapped[float]
lease_until: Mapped[float]

View File

@ -219,6 +219,11 @@ Check all owners and categories; surviving rows retain the allocation floor.
**Terminal run cleanup:** Close streams, journals, and graph references even on cancellation. Details: `backend/docs/runtime-guidance-details.md`.
Clear both the terminal worker's journal and message-sequence stamper references;
the stamper's generation callback captures the journal. Diagnostic tracebacks may
retain finalizer frames, so config/callback removal alone is insufficient. Keep
the DEBUG checkpoint-inspection failure regression in `test_run_worker_rollback.py`.
**`RunManager._runs` holds only records this worker admitted.** A cross-worker idempotent reuse returns the `store_only` row from `_record_from_store()` unregistered: the peer never finalizes or `cleanup()`s it, so a registered copy stays `pending`/`running`, 409s later same-thread admissions, hides the owner's orphan from reconciliation, and sends a peer `cancel()` down the local-owner path. Pinned by `test_peer_idempotent_reuse_*` and `test_peer_cancel_of_reused_run_*` (`tests/test_multi_worker_run_ownership.py`) plus `tests/test_gateway_services.py::test_start_run_peer_idempotent_reuse_*`.
**A `RunRecord` owner matches its durable row.** HTTP admissions omit `user_id` and the SQL store stamps the ambient user, so `create()` and `_admit_thread_operation()` resolve an omitted owner from the contextvar, keeping `None` without one — never `get_effective_user_id()`'s `default` bucket. A `None` record made SQL keyed retries on a peer or after `cleanup()` return 500, hid HTTP runs from owner-scoped history, and skipped their MCP `background_tasks` projection. Pinned by `test_run_record_owner_matches_row_stamped_from_context`, `test_keyed_retry_without_explicit_user_*`, the `sql` case of `test_start_run_peer_idempotent_reuse_*`, `test_run_manager_*_admitted_without_explicit_user`, and `test_run_manager_keeps_omitted_owner_unset_without_user_context`.
@ -318,3 +323,15 @@ the admitted group keeps its records on success or completes rollback on failure
This is a store-local guarantee, not a change to RunJournal cancellation policy or
JSONL's single-process deployment constraint. Regression coverage is in
`tests/test_jsonl_event_store_cancellation.py`.
## Completed-evidence provenance
Gateway admission and RunManager validate host-owned evidence origin against the shared `EVIDENCE_ORIGINS` in `runtime/runs/schemas.py`. Preserve the string values and reject client metadata as provenance; only trusted Python entry points set the admission argument.
Idempotent resume reuse preserves both the private request identity and the original evidence origin/agent across worker and store boundaries. A retry must not relabel existing evidence. Memory/SQLite parity is covered in `tests/test_completed_evidence_admission.py`.
Completed-evidence finalization obeys the journal write-ownership contract:
unknown or cancelled write outcomes cannot yield a receipt or be replayed. Only
proven-not-committed batches may be retried. Normal close stops new evidence
producers and drains already admitted cross-thread callbacks inside its operation
lock before flushing and detaching; do not reacquire that lock through `flush()`.

View File

@ -0,0 +1,13 @@
"""Internal durable journal drain acknowledgement; not an extension write API."""
from dataclasses import dataclass
@dataclass(frozen=True)
class JournalSealReceipt:
run_id: str
thread_id: str
upper_event_seq: int
event_count: int
retention_revision: int
session_factory: object

View File

@ -7,6 +7,7 @@ at ``max_trace_content`` bytes to avoid bloating the database.
from __future__ import annotations
import asyncio
import hashlib
import json
import logging
import re
@ -14,10 +15,12 @@ import weakref
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import delete, func, or_, select, text
from sqlalchemy import delete, func, or_, select, text, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.models.run_event import RunEventRow, RunEventThreadSeqRow
from deerflow.persistence.run.model import CompletedRunSnapshotRow, RunRow
from deerflow.persistence.run.sql import RunRepository
from deerflow.runtime.events.message_identity import message_identity
from deerflow.runtime.events.store.base import RunEventStore
from deerflow.runtime.user_context import AUTO, _AutoSentinel, get_current_user, resolve_user_id
@ -232,6 +235,7 @@ class DbRunEventStore(RunEventStore):
event_type=event_type,
category=category,
content=db_content,
content_sha256=hashlib.sha256(db_content.encode("utf-8")).hexdigest(),
event_metadata=metadata,
seq=seq,
created_at=_parse_created_at(created_at),
@ -269,6 +273,7 @@ class DbRunEventStore(RunEventStore):
event_type=e["event_type"],
category=category,
content=db_content,
content_sha256=hashlib.sha256(db_content.encode("utf-8")).hexdigest(),
event_metadata=metadata,
seq=seq,
created_at=_parse_created_at(e.get("created_at")),
@ -324,6 +329,7 @@ class DbRunEventStore(RunEventStore):
event_type=event_type,
category=category,
content=db_content,
content_sha256=hashlib.sha256(db_content.encode("utf-8")).hexdigest(),
event_metadata=metadata,
seq=(max_seq or 0) + 1,
created_at=_parse_created_at(created_at),
@ -364,6 +370,45 @@ class DbRunEventStore(RunEventStore):
rows = list(result.scalars())
return [self._row_to_dict(r) for r in reversed(rows)]
async def evidence_receipt(self, thread_id: str, run_id: str):
"""Capture acknowledged durable bounds after the journal's producer drain."""
from deerflow.runtime.events.evidence import JournalSealReceipt
async with self._get_write_lock(thread_id):
async with self._sf() as session:
async with session.begin():
await self._acquire_thread_mutation_fence(session, thread_id)
run = (await session.execute(select(RunRow.user_id, RunRow.evidence_retention_revision).where(RunRow.run_id == run_id, RunRow.thread_id == thread_id))).one_or_none()
if run is None:
return None
bounds = await session.execute(
select(func.max(RunEventRow.seq), func.count()).where(
RunEventRow.thread_id == thread_id,
RunEventRow.run_id == run_id,
or_(RunEventRow.user_id == run.user_id, RunEventRow.user_id.is_(None)),
)
)
upper, count = bounds.one()
return JournalSealReceipt(run_id, thread_id, upper or 0, count, run.evidence_retention_revision, self._sf)
@staticmethod
async def _invalidate_evidence(session, thread_id: str, user_id: str | None, change_seq: int, run_id: str | None = None) -> None:
conditions = [RunRow.thread_id == thread_id]
if user_id is not None:
conditions.append(RunRow.user_id == user_id)
if run_id is not None:
conditions.append(RunRow.run_id == run_id)
await session.execute(
update(RunRow)
.where(*conditions)
.values(
evidence_retention_revision=RunRow.evidence_retention_revision + 1,
evidence_seal_state="partial",
change_seq=change_seq,
)
)
await session.execute(delete(CompletedRunSnapshotRow).where(CompletedRunSnapshotRow.run_id.in_(select(RunRow.run_id).where(*conditions))))
async def list_events(
self,
thread_id,
@ -547,7 +592,9 @@ class DbRunEventStore(RunEventStore):
async with self._get_write_lock(thread_id):
async with self._sf() as session:
async with session.begin():
change_seq = await RunRepository._next_change_seq(session)
await self._acquire_thread_mutation_fence(session, thread_id)
await self._invalidate_evidence(session, thread_id, resolved_user_id, change_seq)
count_conditions = [RunEventRow.thread_id == thread_id]
if resolved_user_id is not None:
count_conditions.append(RunEventRow.user_id == resolved_user_id)
@ -586,7 +633,9 @@ class DbRunEventStore(RunEventStore):
async with self._get_write_lock(thread_id):
async with self._sf() as session:
async with session.begin():
change_seq = await RunRepository._next_change_seq(session)
max_seq = await self._max_seq_for_thread(session, thread_id)
await self._invalidate_evidence(session, thread_id, resolved_user_id, change_seq, run_id)
count_conditions = [RunEventRow.thread_id == thread_id, RunEventRow.run_id == run_id]
if resolved_user_id is not None:
count_conditions.append(RunEventRow.user_id == resolved_user_id)

View File

@ -48,6 +48,7 @@ from deerflow.runtime.events.catalog import (
RUN_START_EVENT,
)
from deerflow.runtime.events.store.base import RunEventStore, RunEventStoreNotCommittedError
from deerflow.runtime.user_context import AUTO, _AutoSentinel
from deerflow.utils.llm_text import strip_leading_think_blocks
from deerflow.utils.messages import message_to_text, restore_original_human_message
@ -319,12 +320,19 @@ class RunJournal(BaseCallbackHandler):
flush_threshold: int = 20,
progress_reporter: Callable[[dict], Awaitable[None]] | None = None,
progress_flush_interval: float = 5.0,
user_id: str | None | _AutoSentinel = AUTO,
):
super().__init__()
from deerflow.runtime.events.store.db import DbRunEventStore
self.run_id = run_id
self.thread_id = thread_id
self._evidence_owner_id = user_id if isinstance(event_store, DbRunEventStore) else AUTO
self._store: RunEventStore | None = event_store
self._closed = False
self._accepting_evidence = True
self._producer_lock = threading.Lock()
self._pending_producer_callbacks = 0
self._track_tokens = track_token_usage
self._flush_threshold = flush_threshold
self._progress_reporter = progress_reporter
@ -570,7 +578,7 @@ class RunJournal(BaseCallbackHandler):
tags: list[str] | None = None,
**kwargs: Any,
) -> None:
if self._closed:
if self._closed or not self._accepting_evidence:
return
messages: list[AnyMessage] = []
@ -956,6 +964,7 @@ class RunJournal(BaseCallbackHandler):
return {
"thread_id": self.thread_id,
"run_id": self.run_id,
**({"user_id": self._evidence_owner_id} if self._evidence_owner_id is not AUTO else {}),
"event_type": event_type,
"category": category,
"content": content,
@ -1029,7 +1038,7 @@ class RunJournal(BaseCallbackHandler):
caller: str,
) -> None:
"""Queue one logical response and merge usage into its canonical callback."""
if self._closed:
if self._closed or not self._accepting_evidence:
return
has_usage = self._has_positive_usage(events)
@ -1067,8 +1076,8 @@ class RunJournal(BaseCallbackHandler):
# Some providers immediately re-fire on_llm_end with usage filled in.
# Defer an incomplete copy until the next event or flush.
def _put(self, *, event_type: str, category: str, content: str | dict = "", metadata: dict | None = None) -> None:
if self._closed:
def _put(self, *, event_type: str, category: str, content: str | dict = "", metadata: dict | None = None, _admitted: bool = False) -> None:
if self._closed or (not self._accepting_evidence and not _admitted):
return
self._commit_pending_llm_response()
self._buffer.append(self._make_event(event_type=event_type, category=category, content=content, metadata=metadata))
@ -1496,17 +1505,22 @@ class RunJournal(BaseCallbackHandler):
if owner_loop.is_closed() or not owner_loop.is_running():
logger.warning("Dropping cross-thread middleware event after run loop shutdown")
return
try:
owner_loop.call_soon_threadsafe(
partial(
self._put,
event_type=event_type,
category=MIDDLEWARE_EVENT_PATTERN.category,
content={"name": name, "hook": hook, "action": action, "changes": dict(changes)},
with self._producer_lock:
if not self._accepting_evidence:
return
self._pending_producer_callbacks += 1
try:
owner_loop.call_soon_threadsafe(
partial(
self._record_admitted_middleware,
event_type=event_type,
category=MIDDLEWARE_EVENT_PATTERN.category,
content={"name": name, "hook": hook, "action": action, "changes": dict(changes)},
)
)
)
except RuntimeError:
logger.warning("Dropping cross-thread middleware event after run loop shutdown")
except RuntimeError:
self._pending_producer_callbacks -= 1
logger.warning("Dropping cross-thread middleware event after run loop shutdown")
return
self._put(
@ -1515,6 +1529,13 @@ class RunJournal(BaseCallbackHandler):
content={"name": name, "hook": hook, "action": action, "changes": changes},
)
def _record_admitted_middleware(self, **kwargs) -> None:
try:
self._put(**kwargs, _admitted=True)
finally:
with self._producer_lock:
self._pending_producer_callbacks -= 1
def claim_tool_promotions(self, tool_names: Iterable[str]) -> list[str]:
"""Atomically claim names not yet reported by this run's lead agent."""
candidates = sorted(set(tool_names))
@ -1697,9 +1718,39 @@ class RunJournal(BaseCallbackHandler):
_batch, error = self._quarantined_batches[0]
raise RuntimeError(f"RunJournal flush refused: {len(_batch)} event(s) for run {self.run_id} have an unknown durable outcome") from error
async def _stop_evidence_producers(self) -> None:
with self._producer_lock:
self._accepting_evidence = False
while self._pending_producer_callbacks:
await asyncio.sleep(0)
async def _drain_evidence(self) -> None:
if self._closed:
return
await self._stop_evidence_producers()
await self.flush()
async def finish_evidence(self):
"""Stop declared producers and drain accepted work before capturing bounds.
This is independent of terminal business status and does not detach the
journal. Proven uncommitted writes remain available for explicit retry;
unknown outcomes stay quarantined and never yield a durable receipt.
Unsupported stores return no durable receipt.
"""
if self._closed:
return None
await self._drain_evidence()
from deerflow.runtime.events.store.db import DbRunEventStore
if isinstance(self._store, DbRunEventStore):
return await self._store.evidence_receipt(self.thread_id, self.run_id)
return None
def _detach_runtime_dependencies(self) -> None:
"""Drop every external or potentially cyclic run-scoped reference."""
self._closed = True
self._accepting_evidence = False
with self._skill_usage_lock:
self._skill_usages.clear()
self._store = None
@ -1746,6 +1797,7 @@ class RunJournal(BaseCallbackHandler):
raise RuntimeError(f"RunJournal for run {self.run_id} was force-closed; close cannot report success")
if self._closed:
return
await self._stop_evidence_producers()
await self._flush_locked()
self._detach_runtime_dependencies()
return

View File

@ -22,7 +22,7 @@ from deerflow.runtime.user_context import AUTO, _AutoSentinel, get_current_user,
from deerflow.utils.time import is_lease_expired
from deerflow.utils.time import now_iso as _now_iso
from .schemas import DisconnectMode, RunStatus, ThreadOperationKind
from .schemas import EVIDENCE_ORIGINS, DisconnectMode, RunStatus, ThreadOperationKind
from .store.base import (
EditReplayVisibility,
RunIdempotencyConflict,
@ -267,6 +267,8 @@ class RunRecord:
# True only on the caller that recovered an existing idempotent admission;
# that caller must not attach a second worker to the durable run.
idempotency_reused: bool = False
evidence_origin: str = "unknown"
evidence_agent_id: str | None = None
class RunStartOutcome(StrEnum):
@ -366,6 +368,10 @@ class RunManager:
"idempotency_key": record.idempotency_key,
"goal_verdict": record.goal_verdict,
}
if record.evidence_origin != "unknown":
payload["evidence_origin"] = record.evidence_origin
if record.evidence_agent_id is not None:
payload["evidence_agent_id"] = record.evidence_agent_id
if record.idempotency_request is not None:
payload["idempotency_request"] = record.idempotency_request
if record.user_id is not None:
@ -543,6 +549,8 @@ class RunManager:
last_ai_message=row.get("last_ai_message"),
first_human_message=row.get("first_human_message"),
owner_worker_id=row.get("owner_worker_id"),
evidence_origin=row.get("evidence_origin") or "unknown",
evidence_agent_id=row.get("evidence_agent_id"),
lease_expires_at=row.get("lease_expires_at"),
stop_reason=row.get("stop_reason"),
goal_verdict=row.get("goal_verdict"),
@ -656,6 +664,34 @@ class RunManager:
except Exception:
logger.warning("Failed to persist finalizing progress for %s", run_id, exc_info=True)
@staticmethod
def _validate_evidence_origin(origin: str) -> None:
if origin not in EVIDENCE_ORIGINS:
raise ValueError("invalid evidence origin")
async def seal_completed_evidence(self, run_id: str, receipt) -> bool:
"""Best-effort evidence metadata is independent of business completion."""
async with self._lock:
record = self._runs.get(run_id)
if record is None or record.ownership_lost:
return False
owner_worker_id = record.owner_worker_id
seal = getattr(self._store, "seal_evidence", None)
if seal is None or owner_worker_id is None:
return False
return await seal(run_id, owner_worker_id=owner_worker_id, receipt=receipt)
async def mark_evidence_partial(self, run_id: str, *, error: str) -> bool:
async with self._lock:
record = self._runs.get(run_id)
if record is None or record.ownership_lost:
return False
owner_worker_id = record.owner_worker_id
mark_partial = getattr(self._store, "mark_evidence_partial", None)
if mark_partial is None or owner_worker_id is None:
return False
return await mark_partial(run_id, owner_worker_id=owner_worker_id, error=error)
async def create(
self,
thread_id: str,
@ -666,6 +702,8 @@ class RunManager:
kwargs: dict | None = None,
multitask_strategy: str = "reject",
user_id: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
) -> RunRecord:
"""Create a new pending run and register it.
@ -678,6 +716,7 @@ class RunManager:
"""
from deerflow.runtime.assembly_evidence import EVIDENCE_KEY
self._validate_evidence_origin(evidence_origin)
run_id = str(uuid.uuid4())
now = _now_iso()
user_id = _resolve_record_user_id(user_id)
@ -696,6 +735,8 @@ class RunManager:
updated_at=now,
owner_worker_id=self._worker_id,
lease_expires_at=lease_expires_at,
evidence_origin=evidence_origin,
evidence_agent_id=evidence_agent_id,
)
async with self._lock:
self._runs[run_id] = record
@ -1613,6 +1654,8 @@ class RunManager:
model_name: str | None = None,
user_id: str | None = None,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
) -> RunRecord:
"""Atomically admit a normal agent run for a thread."""
@ -1627,6 +1670,8 @@ class RunManager:
model_name=model_name,
user_id=user_id,
idempotency_key=idempotency_key,
evidence_origin=evidence_origin,
evidence_agent_id=evidence_agent_id,
idempotency_request=idempotency_request,
)
@ -1688,6 +1733,8 @@ class RunManager:
model_name: str | None = None,
user_id: str | None = None,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
) -> RunRecord:
"""Atomically check for inflight runs and create a new one.
@ -1706,6 +1753,7 @@ class RunManager:
"""
from deerflow.runtime.assembly_evidence import EVIDENCE_KEY
self._validate_evidence_origin(evidence_origin)
run_id = str(uuid.uuid4())
now = _now_iso()
# Resolve before the idempotency checks below compare it with stored rows.
@ -1736,6 +1784,8 @@ class RunManager:
owner_worker_id=self._worker_id,
lease_expires_at=lease_expires_at,
idempotency_key=idempotency_key,
evidence_origin=evidence_origin,
evidence_agent_id=evidence_agent_id,
idempotency_request=idempotency_request,
)
@ -1809,6 +1859,10 @@ class RunManager:
}
if idempotency_key is not None:
create_kwargs["idempotency_key"] = idempotency_key
if evidence_origin != "unknown":
create_kwargs["evidence_origin"] = evidence_origin
if evidence_agent_id is not None:
create_kwargs["evidence_agent_id"] = evidence_agent_id
if idempotency_request is not None:
create_kwargs["idempotency_request"] = idempotency_request
try:
@ -1843,6 +1897,10 @@ class RunManager:
}
if idempotency_key is not None:
create_kwargs["idempotency_key"] = idempotency_key
if evidence_origin != "unknown":
create_kwargs["evidence_origin"] = evidence_origin
if evidence_agent_id is not None:
create_kwargs["evidence_agent_id"] = evidence_agent_id
if idempotency_request is not None:
create_kwargs["idempotency_request"] = idempotency_request
# Interrupt / rollback: store-side claim + insert in one
@ -2005,6 +2063,24 @@ class RunManager:
if self._store is None:
return []
grace_seconds = self._run_ownership_config.grace_seconds if self._run_ownership_config else 10
recover_evidence = getattr(self._store, "recover_unsealed_evidence", None)
if callable(recover_evidence):
try:
evidence_before = before or _now_iso()
while True:
async with self._lock:
local_finalizing = tuple(r.run_id for r in self._runs.values() if not r.ownership_lost and (r.finalizing or (r.task is not None and not r.task.done())))
recovered_count = await self._call_store_with_retry(
"recover_unsealed_evidence",
"*",
lambda: recover_evidence(before=evidence_before, grace_seconds=grace_seconds, exclude_run_ids=local_finalizing),
)
# Drain at startup too: single-worker mode has no heartbeat.
if not recovered_count:
break
except Exception:
# Evidence availability must not prevent normal run recovery.
logger.warning("Failed to recover orphaned completed-run evidence", exc_info=True)
try:
rows = await self._call_store_with_retry(
"list_inflight_with_expired_lease",

View File

@ -1,7 +1,10 @@
"""Run status and disconnect mode enums."""
"""Run status, disconnect modes and host-owned evidence provenance."""
from enum import StrEnum
# Shared by Gateway admission and RunManager; values remain wire-format strings.
EVIDENCE_ORIGINS = frozenset({"interactive", "scheduled", "extension_evaluation", "unknown"})
class ThreadOperationKind(StrEnum):
"""Kind of operation holding exclusive admission for a thread."""

View File

@ -159,6 +159,8 @@ class RunStore(abc.ABC):
owner_worker_id: str | None = None,
lease_expires_at: str | None = None,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
) -> None:
pass
@ -452,6 +454,8 @@ class RunStore(abc.ABC):
created_at: str | None = None,
grace_seconds: int = 10,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
) -> tuple[dict[str, Any], list[dict[str, Any]]]:
"""Atomically create an active thread operation with cross-process uniqueness.

View File

@ -69,6 +69,8 @@ class MemoryRunStore(RunStore):
owner_worker_id=None,
lease_expires_at=None,
idempotency_key=None,
evidence_origin="unknown",
evidence_agent_id=None,
idempotency_request=None,
):
now = datetime.now(UTC).isoformat()
@ -77,6 +79,8 @@ class MemoryRunStore(RunStore):
"run_id": run_id,
"thread_id": thread_id,
"assistant_id": assistant_id,
"evidence_origin": evidence_origin,
"evidence_agent_id": evidence_agent_id,
"user_id": user_id,
"model_name": model_name,
"status": status,
@ -505,6 +509,8 @@ class MemoryRunStore(RunStore):
created_at: str | None = None,
grace_seconds: int = 10,
idempotency_key: str | None = None,
evidence_origin: str = "unknown",
evidence_agent_id: str | None = None,
idempotency_request: dict[str, Any] | None = None,
) -> tuple[dict[str, Any], list[dict[str, Any]]]:
from deerflow.runtime.runs.manager import ConflictError
@ -583,6 +589,8 @@ class MemoryRunStore(RunStore):
"run_id": run_id,
"thread_id": thread_id,
"assistant_id": assistant_id,
"evidence_origin": evidence_origin,
"evidence_agent_id": evidence_agent_id,
"user_id": user_id,
"model_name": model_name,
"status": "pending",

View File

@ -988,6 +988,7 @@ async def run_agent(
accessor: CheckpointStateAccessor | None = None
rollback_point: RollbackPoint | None = None
journal = None
seq_stamper: _MessageSeqStamper | None = None
runtime_ctx: dict[str, Any] | None = None
runtime: Any | None = None
agent: Any | None = None
@ -1004,6 +1005,7 @@ async def run_agent(
# checkpoint failures / cancellation while waiting did not write an empty
# completion snapshot into RunStore.
persist_completion = False
evidence_execution_started = False
completion_data: dict[str, Any] | None = None
# Buffers subagent step events for batched persistence (#3779); assigned once
# streaming starts and flushed in the finally block. Pre-bound to None so the
@ -1072,6 +1074,7 @@ async def run_agent(
thread_id=thread_id,
event_store=event_store,
track_token_usage=getattr(run_events_config, "track_token_usage", True),
user_id=record.user_id,
progress_reporter=lambda snapshot: run_manager.update_run_progress(run_id, **snapshot),
)
@ -1567,6 +1570,7 @@ async def run_agent(
if stand_down_reason.startswith("thread_changed"):
final_goal_verdict.update(satisfied=False, blocker="missing_evidence", relied_on_assumption=False)
evidence_execution_started = True
await _stream_once(graph_input, initial_runnable_config)
while not record.abort_event.is_set() and not llm_error_fallback_message and (journal is None or not journal.had_llm_error_fallback):
continuation_input = await _prepare_goal_continuation_input(
@ -1691,6 +1695,7 @@ async def run_agent(
async def finalize() -> None:
nonlocal completion_hook_task
nonlocal journal, seq_stamper
nonlocal accessor, agent, agent_result, cancel_action, checkpoint_rollback_completed
nonlocal completion_data, deferred_finalization_interrupt, delivery_content, goal_completion, goal_evaluator_model
nonlocal graph_input, pre_run_workspace_snapshot, produced_output_paths, rollback_point, runtime
@ -1976,6 +1981,29 @@ async def run_agent(
"Extension task-stop notification interrupted for run %s; completing cleanup first",
run_id,
)
if not record.ownership_lost and journal is not None:
evidence_failure = None
try:
evidence_receipt = await journal.finish_evidence()
if evidence_receipt is not None:
if not evidence_execution_started:
evidence_failure = "Lead graph execution did not begin; producer coverage is partial."
elif not await run_manager.seal_completed_evidence(run_id, evidence_receipt):
evidence_failure = "The owning worker could not persist the journal seal."
except BaseException as exc:
evidence_failure = f"Journal seal failed ({type(exc).__name__})."
if not isinstance(exc, Exception):
deferred_finalization_interrupt = _defer_finalization_interrupt(deferred_finalization_interrupt, exc)
logger.warning("Failed to seal completed-run evidence for %s (non-fatal)", run_id, exc_info=True)
if evidence_failure is not None:
try:
await run_manager.mark_evidence_partial(run_id, error=evidence_failure)
except Exception:
logger.warning("Failed to persist partial evidence for %s (non-fatal)", run_id, exc_info=True)
except BaseException as exc:
deferred_finalization_interrupt = _defer_finalization_interrupt(deferred_finalization_interrupt, exc)
logger.warning("Partial evidence persistence interrupted for %s; completing cleanup first", run_id)
if record.finalizing:
await run_manager.set_finalizing(run_id, False)
@ -2015,6 +2043,10 @@ async def run_agent(
)
# Drop graph and per-run payload references before the terminal
# worker task itself becomes collectable.
# Diagnostic tracebacks can retain this frame. Release both
# the journal and the stamper callback that closes over it.
seq_stamper = None
journal = None
agent = None
agent_result = None
accessor = None

View File

@ -54,3 +54,28 @@ Lets a caller pass per-request, short-lived end-user credentials (e.g. an ERP to
- Build archives from captured bytes and require the preview's revision. Preserve file contents, empty directories, and normalized executable flags; import must not restore privileged permission bits.
- Keep traversal, YAML parsing, and archive construction bounded and cancellable. Reject unsupported filesystem operations rather than following links; report limits instead of truncating results. Preserve YAML preflight before object construction.
- Export includes raw saved files and is not a secret audit. Return relative paths and generic errors without leaking source content. See the [export API contract](../../../../docs/API.md#export-a-custom-skill) for response fields and limits.
### Evolution host checks and recovery
`mutations/service.py` shares one shielded check per proposal. A cancelled waiter
does not cancel the shared scan; a cancelled scan or timeout must propagate and
drain token-fenced lease cleanup, leaving no cached verdict. Keep the scan attempt
for hourly quota accounting. Cleanup must not erase a replacement worker's lease.
Single-operation recovery propagates an unresolved owner barrier as `NEEDS_REPAIR`
(HTTP 409); only the host-wide recovery sweep suppresses it to continue with other
owners. Publication and derived-view status remain separate.
`tests/blocking_io/test_skill_mutation_host.py` exercises the real host lifetime,
evidence pages, scanner, publication, revert and admin routes under Blockbuster;
only remote moderation is stubbed. Keep new async paths inside that gate.
Standalone SDK/storage processes have no publication runtime. `mutations/enrollment.py` checks the configured application DB on each access and refuses enrolled owners with `MUTATION_RUNTIME_REQUIRED`; global toggles check all persisted owners. Never cache negative enrollment or infer it from the plugin list. Reads and writes must keep these guards, and the local storage factory must retain its AppConfig so the probe uses the same DB as the Gateway. `tests/test_skill_sdk_enrollment.py` exercises fresh real SDK processes without injected runtime state.
- `get_enabled_skills_for_config` preserves user scope even when `app_config` is omitted; only calls without both config and user use the nonblocking global cache. Enrolled owners always bypass the local catalog LRU and pass the durable readiness guard.
Owned custom-skill YAML diagnostics use the same managed-read guard as catalog
reads: recover pending publications before scanning, and refuse standalone reads
for enrolled owners without a mutation runtime. Keep the scan off the event loop.
Projection readers and mutation writers share the reentrant owner lock. Both use
`utils.file_lock.retry_until_locked` for contention-only OS lock retries; preserve
deadlines, cancellation checks and the shared descriptor for nested reads.

View File

@ -366,12 +366,14 @@ def _open_directory_chain(path, budget):
def _capture(storage, name, snapshot, budget):
from deerflow.skills.mutations.guard import managed_read
if not isinstance(name, str) or len(name) > 64 or not _NAME.fullmatch(name):
raise SkillExportError(422, "skill_export_unsupported", "Invalid skill name.")
root = storage.get_custom_skill_dir(name)
budget.check()
try:
with skill_projection_read_lock(storage, timeout=LOCK_TIMEOUT_SECONDS, check=budget.check):
with skill_projection_read_lock(storage, timeout=LOCK_TIMEOUT_SECONDS, check=budget.check), managed_read(storage):
if not hasattr(os, "O_NOFOLLOW") or os.open not in os.supports_dir_fd or os.scandir not in os.supports_fd:
raise SkillExportError(422, "skill_export_unsupported", "This platform cannot safely capture skill files.")
parent_fd = _open_directory_chain(root.parent, budget)

View File

@ -0,0 +1 @@
"""Host-managed conditional updates of existing user custom skills."""

View File

@ -0,0 +1,148 @@
"""Bounded byte snapshots for managed custom packages (POSIX, no links)."""
from __future__ import annotations
import hashlib
import json
import os
import stat
from dataclasses import dataclass
from pathlib import Path
from deerflow.skills.mutations.validation import MAX_MAIN_BYTES
MAX_FILES = 256
MAX_PACKAGE_BYTES = 16 * 1024 * 1024
MAX_DEPTH = 24
@dataclass(frozen=True)
class PackageFile:
path: str
content: bytes
executable: bool
@property
def digest(self) -> str:
return hashlib.sha256(self.content).hexdigest()
@dataclass(frozen=True)
class PackageSnapshot:
files: tuple[PackageFile, ...]
@property
def main_content(self) -> str:
return next(item.content for item in self.files if item.path == "SKILL.md").decode("utf-8")
@property
def digest(self) -> str:
manifest = [(item.path, item.digest, item.executable) for item in self.files]
return hashlib.sha256(json.dumps(manifest, ensure_ascii=False, separators=(",", ":")).encode("utf-8")).hexdigest()
def with_main(self, content: str) -> PackageSnapshot:
return PackageSnapshot(tuple(PackageFile(item.path, content.encode("utf-8"), item.executable) if item.path == "SKILL.md" else item for item in self.files))
def capture_package(root: Path) -> PackageSnapshot:
"""Caller owns mutation guard. Never read outside the opened package tree."""
if os.name != "posix":
raise ValueError("UNSUPPORTED_ASSET")
files: list[PackageFile] = []
total = 0
entries = 0
flags = os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK
def walk(directory_fd: int, prefix: str = "", depth: int = 0) -> None:
nonlocal total, entries
if depth > MAX_DEPTH:
raise ValueError("QUOTA_EXCEEDED")
# Bound directory enumeration, including empty directories, before sort.
names = []
with os.scandir(directory_fd) as iterator:
for entry in iterator:
entries += 1
if entries > MAX_FILES * 2:
raise ValueError("QUOTA_EXCEEDED")
names.append(entry.name)
for name in sorted(names):
if "\\" in name or any(ord(c) < 32 for c in name):
raise ValueError("UNSUPPORTED_ASSET")
path = f"{prefix}/{name}" if prefix else name
if len(path.encode("utf-8")) > 1024:
raise ValueError("QUOTA_EXCEEDED")
info = os.stat(name, dir_fd=directory_fd, follow_symlinks=False)
if not stat.S_ISDIR(info.st_mode) and not stat.S_ISREG(info.st_mode):
raise ValueError("UNSUPPORTED_ASSET")
fd = os.open(name, flags | (os.O_DIRECTORY if stat.S_ISDIR(info.st_mode) else 0), dir_fd=directory_fd)
try:
opened = os.fstat(fd)
if (info.st_dev, info.st_ino, info.st_mode) != (opened.st_dev, opened.st_ino, opened.st_mode):
raise ValueError("REVISION_CONFLICT")
if stat.S_ISDIR(opened.st_mode):
walk(fd, path, depth + 1)
continue
if len(files) >= MAX_FILES:
raise ValueError("QUOTA_EXCEEDED")
limit = min(MAX_PACKAGE_BYTES - total, MAX_MAIN_BYTES if path == "SKILL.md" else MAX_PACKAGE_BYTES)
if opened.st_size > limit:
raise ValueError("QUOTA_EXCEEDED")
data = bytearray()
while True:
chunk = os.read(fd, min(65536, limit + 1 - len(data)))
if not chunk:
break
data.extend(chunk)
if len(data) > limit:
raise ValueError("QUOTA_EXCEEDED")
after = os.fstat(fd)
if (opened.st_size, opened.st_mtime_ns, opened.st_ctime_ns) != (after.st_size, after.st_mtime_ns, after.st_ctime_ns):
raise ValueError("REVISION_CONFLICT")
total += len(data)
files.append(PackageFile(path, bytes(data), bool(opened.st_mode & 0o111)))
finally:
os.close(fd)
try:
root_fd = open_directory_chain(root)
try:
walk(root_fd)
finally:
os.close(root_fd)
except OSError as exc:
raise ValueError("UNSUPPORTED_ASSET") from exc
result = PackageSnapshot(tuple(sorted(files, key=lambda item: item.path)))
try:
result.main_content
except (StopIteration, UnicodeError) as exc:
raise ValueError("UNSUPPORTED_ASSET") from exc
return result
def open_directory_chain(path: Path) -> int:
"""Return an anchored directory FD; reject links in every ancestor.
Publication also uses this helper: validating only the package's final
component would let a linked owner/custom root escape the owner guard.
Caller owns the returned descriptor.
"""
path = path.absolute()
if ".." in path.parts or len(path.parts) > 128 or len(os.fsencode(path)) > 4096:
raise ValueError("UNSUPPORTED_ASSET")
flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_NONBLOCK
fd = os.open(path.anchor, flags)
try:
for part in path.parts[1:]:
before = os.stat(part, dir_fd=fd, follow_symlinks=False)
if not stat.S_ISDIR(before.st_mode):
raise ValueError("UNSUPPORTED_ASSET")
child = os.open(part, flags, dir_fd=fd)
os.close(fd)
fd = child
after = os.fstat(fd)
if (before.st_dev, before.st_ino, before.st_mode) != (after.st_dev, after.st_ino, after.st_mode):
raise ValueError("REVISION_CONFLICT")
return fd
except BaseException:
os.close(fd)
raise

View File

@ -0,0 +1,89 @@
"""Detached database encodings. Blob storage never enters skill loading roots."""
from __future__ import annotations
import base64
import hashlib
import json
from datetime import UTC, datetime
from deerflow_extension_api.skill_mutations import AssetRevision, HostCheckResult, Operation, Proposal
from deerflow.skills.mutations.assets import MAX_FILES, MAX_PACKAGE_BYTES, PackageFile, PackageSnapshot
from deerflow.skills.mutations.validation import MAX_MAIN_BYTES
def fingerprint(value) -> str:
return hashlib.sha256(json.dumps(value, sort_keys=True, ensure_ascii=False, separators=(",", ":"), allow_nan=False).encode()).hexdigest()
def encode_package(package: PackageSnapshot) -> bytes:
return json.dumps([(item.path, base64.b64encode(item.content).decode("ascii"), item.executable) for item in package.files], separators=(",", ":")).encode()
def decode_package(blob: bytes) -> PackageSnapshot:
try:
if not isinstance(blob, bytes) or len(blob) > 24 * 1024 * 1024:
raise ValueError
rows = json.loads(blob)
if not isinstance(rows, list) or not 1 <= len(rows) <= MAX_FILES:
raise ValueError
files, total, previous = [], 0, ""
for item in rows:
if not isinstance(item, list) or len(item) != 3:
raise ValueError
path, content, executable = item
if not isinstance(path, str) or not path or len(path.encode()) > 1024 or path <= previous or "\\" in path or any(ord(c) < 32 for c in path) or any(part in {"", ".", ".."} for part in path.split("/")):
raise ValueError
if type(executable) is not bool or not isinstance(content, str):
raise ValueError
data = base64.b64decode(content, validate=True)
total += len(data)
if total > MAX_PACKAGE_BYTES or (path == "SKILL.md" and len(data) > MAX_MAIN_BYTES):
raise ValueError
files.append(PackageFile(path, data, executable))
previous = path
result = PackageSnapshot(tuple(files))
result.main_content
return result
except (ValueError, TypeError, StopIteration, UnicodeError, RecursionError) as exc:
raise ValueError("BLOB_INTEGRITY_ERROR") from exc
def operation_packages(row):
before, after = decode_package(row.before_blob), decode_package(row.after_blob)
if before.digest != row.before_revision.get("content_digest") or after.digest != row.after_revision.get("content_digest") or before.digest == after.digest:
raise ValueError("BLOB_INTEGRITY_ERROR")
return before, after
def timestamp(value: float) -> str:
return datetime.fromtimestamp(value, UTC).isoformat()
def proposal_view(row) -> Proposal:
return Proposal(
row.proposal_id, row.owner_id, row.target_ref, row.name, AssetRevision(**row.base_revision), row.candidate_hash, row.state, timestamp(row.expires_at), tuple(source["snapshot_ref"] for source in row.sources), row.operation_id
)
def operation_view(row) -> Operation:
return Operation(
row.operation_id,
row.owner_id,
row.name,
row.publication,
row.views,
AssetRevision(**row.before_revision),
AssetRevision(**row.after_revision),
row.generation,
row.proposal_id,
row.reverts_operation_id,
row.error_code,
row.superseded_by_generation,
)
def check_view(row) -> HostCheckResult:
result = row.check_result
return HostCheckResult(row.proposal_id, result["decision"], row.candidate_hash, AssetRevision(**row.base_revision), result["policy_version"], timestamp(result["expires_at"]), result["reason_code"])

View File

@ -0,0 +1,65 @@
"""Fail closed when an embedded process lacks the enrolled owner's runtime.
Enrollment belongs to the application database, not the current plugin list or
process. Do not cache negative lookups: a Gateway can enroll an owner after an
SDK client has already been constructed. Probes never create/migrate a database.
"""
from __future__ import annotations
from pathlib import Path
from urllib.parse import quote
from deerflow_extension_api.host_capabilities import HostCapabilityError
from sqlalchemy import create_engine, inspect, select, union_all
from sqlalchemy.engine import make_url
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy.pool import NullPool
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOwnerRow
def require_unenrolled(storage, *, global_scope: bool = False) -> None:
config = getattr(storage, "_app_config", None)
database = getattr(config, "database", None)
if database is None or database.backend == "memory":
return
owner_id = getattr(storage, "user_id", None)
if owner_id is None and not global_scope:
return
engine = None
try:
url = make_url(database.app_sync_sqlalchemy_url)
if database.backend == "sqlite":
path = Path(database.sqlite_path)
if not path.exists():
return
# URI read-only mode closes the exists/connect creation race and
# avoids journal/schema changes in an uninitialized SDK process.
url = url.set(database="file:" + quote(str(path), safe="/"), query={"mode": "ro", "uri": "true"})
args = {"timeout": 5}
else:
from deerflow.persistence.postgres_schema import build_psycopg_options
options = build_psycopg_options(database.postgres_schema) or ""
args = {"connect_timeout": 5, "options": f"{options} -c lock_timeout=5000 -c statement_timeout=5000".strip()}
# A short-lived connection has no process-wide SDK teardown requirement
# and cannot retain an enrollment snapshot across later skill calls.
engine = create_engine(url, connect_args=args, poolclass=NullPool)
with engine.connect() as connection:
schema = inspect(connection)
queries = []
for model in (SkillOwnerRow, SkillAssetRow):
if schema.has_table(model.__tablename__):
query = select(model.owner_id)
queries.append(query if global_scope else query.where(model.owner_id == owner_id))
enrolled = bool(queries) and connection.execute(union_all(*queries).limit(1)).first() is not None
if enrolled:
raise HostCapabilityError("MUTATION_RUNTIME_REQUIRED", "Use the Gateway for enrolled skill access; this process has no durable mutation runtime")
except (SQLAlchemyError, OSError) as exc:
# Unavailable enrollment is not proof that a write is unmanaged. Do not
# expose a DB URL or driver diagnostics in the public exception.
raise HostCapabilityError("UNAVAILABLE", "Cannot verify durable skill enrollment") from exc
finally:
if engine is not None:
engine.dispose()

View File

@ -0,0 +1,186 @@
"""One host-owned coordination point for managed storage readers and writers."""
from __future__ import annotations
import hashlib
from contextlib import ExitStack, contextmanager, nullcontext
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.skills.mutations.assets import capture_package
from deerflow.skills.mutations.enrollment import require_unenrolled
from deerflow.skills.mutations.repository import revision
_runtime: SkillMutationRuntime | None = None
def configure_mutation_runtime(runtime: SkillMutationRuntime | None) -> None:
"""Gateway startup/shutdown only. Recovery remains independent of plugins."""
global _runtime
_runtime = runtime
def runtime_for(storage) -> SkillMutationRuntime | None:
runtime = _runtime
if runtime is None:
require_unenrolled(storage)
return runtime if runtime is not None and getattr(storage, "user_id", None) in runtime.owners else None
def owner_is_managed(owner_id: str | None) -> bool:
return _runtime is not None and owner_id in _runtime.owners
@contextmanager
def managed_read(storage):
runtime = runtime_for(storage)
if runtime is None:
yield
return
from deerflow.skills.projection import skill_projection_read_lock
with skill_projection_read_lock(storage):
runtime.ensure_readable(storage)
yield
def managed_global_read(storage):
"""Only enrolled owners need global -> owner ordering around view rebuilds."""
if runtime_for(storage) is None:
return nullcontext()
from deerflow.skills.projection import _projection_lock, get_skill_projection_paths
return _projection_lock(get_skill_projection_paths(storage).public.parent, timeout=5.0)
def read_optional_text(storage, path):
"""Canonical reads for existing edit/rollback audit paths, inside readiness."""
from deerflow.skills.storage.skill_storage import read_text_or_none
with managed_read(storage):
return read_text_or_none(path) if path.exists() else None
@contextmanager
def managed_name_writes(storage, names, *, global_scope=False, timeout=None):
"""Serialize one Skill name across public state and enrolled owner writes."""
runtime = _runtime
owner_id = getattr(storage, "user_id", None)
if runtime is None:
require_unenrolled(storage, global_scope=global_scope)
if runtime is None or not names or (not global_scope and owner_id not in runtime.owners):
yield
return
from deerflow.skills.projection import _projection_lock, get_skill_projection_paths
lock_root = get_skill_projection_paths(storage).public.parent / ".skill-mutation-names"
with ExitStack() as stack:
for name in sorted(set(names)):
token = hashlib.sha256(name.encode("utf-8")).hexdigest()
stack.enter_context(_projection_lock(lock_root / token, timeout=timeout))
yield
@contextmanager
def managed_global_state_write(storage, name):
"""A global same-name enable toggle also changes each user's custom asset.
Acquire before config locks: name fence -> global projection -> sorted owner guards -> DB.
A name fence prevents concurrent same-name creation. Only owners that have
the affected custom asset are then reserved before changing the shared flag.
"""
runtime = _runtime
if runtime is None:
require_unenrolled(storage, global_scope=True)
yield
return
from deerflow.skills.projection import _projection_lock, get_skill_projection_paths, skill_projection_read_lock
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
with managed_name_writes(storage, (name,), global_scope=True, timeout=5.0), _projection_lock(get_skill_projection_paths(storage).public.parent, timeout=5.0), ExitStack() as stack:
storages = []
for owner in sorted(runtime.owners):
scoped = UserScopedSkillStorage(owner, host_path=str(storage.get_skills_root_path()))
if not scoped.get_custom_skill_file(name).exists():
continue
stack.enter_context(skill_projection_read_lock(scoped))
runtime.ensure_readable(scoped)
if scoped.get_custom_skill_file(name).exists():
storages.append(scoped)
for scoped in storages:
stack.enter_context(runtime.writer(scoped, (name,)))
yield
class SkillMutationRuntime:
def __init__(self, repository, *, owners: frozenset[str]):
self.repository = repository
self.owners = owners
self.recover_locked = None
def ensure_readable(self, storage):
"""Caller owns the owner guard; lazy recovery never rebuilds views."""
try:
self.repository.check_readable(storage.user_id)
except HostCapabilityError as exc:
if exc.code != "NEEDS_REPAIR" or self.recover_locked is None:
raise
self.recover_locked(storage)
self.repository.check_readable(storage.user_id)
@staticmethod
def state(storage, name):
root = storage.get_custom_skill_dir(name)
exists = root.exists()
if not exists:
return "", False, False
try:
digest = capture_package(root).digest
except ValueError:
# Old APIs retain their broader asset support. Such assets cannot
# enter the new automatic API; each legacy write invalidates CAS.
import uuid
digest = "unsupported-" + uuid.uuid4().hex
from deerflow.config.extensions_config import ExtensionsConfig
enabled = storage.get_skill_enabled_state(name) and ExtensionsConfig.from_file().is_skill_enabled(name, "custom")
return digest, enabled, True
def read_revision(self, storage, name):
from deerflow.skills.projection import skill_projection_read_lock
with skill_projection_read_lock(storage):
digest, enabled, exists = self.state(storage, name)
row = self.repository.observe(storage.user_id, name, digest, enabled, exists=exists)
if not exists:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
return revision(row)
@contextmanager
def writer(self, storage, names):
# The caller already owns the projection/user guard, on one worker.
self.ensure_readable(storage)
for name in names:
digest, enabled, exists = self.state(storage, name)
self.repository.observe(storage.user_id, name, digest, enabled, exists=exists, mark_mutating=True)
try:
yield
except BaseException:
# Keep the marker: the next guarded access must conservatively
# advance the version even if a failed writer restored old bytes.
raise
else:
for name in names:
digest, enabled, exists = self.state(storage, name)
self.repository.finish_writer(storage.user_id, name, digest, enabled, exists=exists)
@contextmanager
def managed_writer(storage, names):
runtime = runtime_for(storage)
if runtime is None:
yield
else:
with runtime.writer(storage, names):
yield

View File

@ -0,0 +1,103 @@
"""Bounded best-effort metadata mirror; the operation database is authoritative."""
import json
import os
import stat
from collections import defaultdict
from dataclasses import asdict
from deerflow.skills.mutations.assets import open_directory_chain
MAX_HISTORY_BYTES = 8 * 1024 * 1024
MAX_HISTORY_LINE_BYTES = 64 * 1024
def mirror_operations(storage, operations):
"""Caller owns the owner guard. Never copy retained Skill body blobs here.
Replay at most the latest 100 operations per owner. An oversized/malformed
history is left untouched, since bounded work cannot prove deduplication.
"""
grouped = defaultdict(list)
for operation in operations:
grouped[operation.name].append(operation)
if not grouped:
return
custom = open_directory_chain(storage.get_custom_skill_dir(next(iter(grouped))).parent)
try:
try:
os.mkdir(".history", 0o700, dir_fd=custom)
except FileExistsError:
pass
directory = os.open(".history", os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=custom)
finally:
os.close(custom)
try:
for name, items in grouped.items():
storage.validate_skill_name(name)
fd = os.open(name + ".jsonl", os.O_RDWR | os.O_APPEND | os.O_CREAT | os.O_NOFOLLOW | os.O_NONBLOCK, 0o600, dir_fd=directory)
try:
before = os.fstat(fd)
if not stat.S_ISREG(before.st_mode) or before.st_size > MAX_HISTORY_BYTES:
continue
seen, pending, total = set(), b"", 0
valid = True
while chunk := os.read(fd, min(65536, MAX_HISTORY_BYTES + 1 - total)):
total += len(chunk)
if total > MAX_HISTORY_BYTES:
valid = False
break
lines = (pending + chunk).split(b"\n")
pending = lines.pop()
for line in lines:
if len(line) > MAX_HISTORY_LINE_BYTES:
valid = False
break
if not line.strip():
continue
try:
record = json.loads(line)
if not isinstance(record, dict):
raise ValueError
identifier = record.get("operation_id")
if isinstance(identifier, str):
seen.add(identifier)
except (ValueError, UnicodeError):
valid = False
break
if not valid or len(pending) > MAX_HISTORY_LINE_BYTES:
valid = False
break
if not valid or pending.strip() or os.fstat(fd).st_size != before.st_size:
continue
records = []
for operation in reversed(items):
if operation.operation_id in seen:
continue
records.append(
json.dumps(
{
"operation_id": operation.operation_id,
"action": "host_revert" if operation.reverts_operation_id else "host_commit",
"publication": operation.publication,
"prev_revision": asdict(operation.before_revision),
"new_revision": asdict(operation.after_revision),
"scope": "main-file-only",
"file_path": "SKILL.md",
},
separators=(",", ":"),
).encode()
+ b"\n"
)
if total + sum(map(len, records)) > MAX_HISTORY_BYTES:
continue
for record in records:
if os.write(fd, record) != len(record):
raise OSError("incomplete history mirror")
if records:
os.fsync(fd)
os.fsync(directory)
finally:
os.close(fd)
finally:
os.close(directory)

View File

@ -0,0 +1,174 @@
"""One-file durable publication. The PREPARED journal precedes every replace."""
from __future__ import annotations
import os
import stat
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOperationRow, SkillOwnerRow
from deerflow.skills.mutations.assets import capture_package, open_directory_chain
from deerflow.skills.mutations.codec import operation_packages
from deerflow.skills.mutations.repository import supersede_views
def replace_main(root, content: bytes, *, operation_id: str) -> None:
"""Use the validated directory descriptor throughout; preserve mode bits."""
directory = open_directory_chain(root)
parent = None
created = False
temporary = ".host-mutation-" + operation_id
try:
parent = open_directory_chain(root.parent)
original = os.stat("SKILL.md", dir_fd=directory, follow_symlinks=False)
if not stat.S_ISREG(original.st_mode):
raise HostCapabilityError("UNSUPPORTED_ASSET")
fd = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=parent)
created = True
try:
offset = 0
while offset < len(content):
written = os.write(fd, content[offset:])
if written <= 0:
raise OSError("short publication write")
offset += written
# Preserve regular permissions, never propagate setuid/setgid/sticky.
os.fchmod(fd, stat.S_IMODE(original.st_mode) & 0o777)
os.fsync(fd)
finally:
os.close(fd)
os.replace(temporary, "SKILL.md", src_dir_fd=parent, dst_dir_fd=directory)
os.fsync(directory)
os.fsync(parent)
finally:
try:
if created:
try:
os.unlink(temporary, dir_fd=parent)
except FileNotFoundError:
pass
finally:
os.close(directory)
if parent is not None:
os.close(parent)
def assert_temporary_absent(root, operation_id):
parent = open_directory_chain(root.parent)
try:
try:
os.stat(".host-mutation-" + operation_id, dir_fd=parent, follow_symlinks=False)
except FileNotFoundError:
return
raise HostCapabilityError("UNSUPPORTED_ASSET")
finally:
os.close(parent)
def durable_current(root):
"""Recovery also needs successful barriers before claiming a disk outcome."""
directory = open_directory_chain(root)
try:
fd = os.open("SKILL.md", os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
try:
if not stat.S_ISREG(os.fstat(fd).st_mode):
raise OSError("unsupported main file")
os.fsync(fd)
finally:
os.close(fd)
os.fsync(directory)
finally:
os.close(directory)
parent = open_directory_chain(root.parent)
try:
os.fsync(parent)
finally:
os.close(parent)
def cleanup_temporary(root, operation_id):
parent = open_directory_chain(root.parent)
try:
try:
os.unlink(".host-mutation-" + operation_id, dir_fd=parent)
os.fsync(parent)
except FileNotFoundError:
pass
finally:
os.close(parent)
def finalize(session, operation, publication: str) -> None:
"""Apply reserved counters exactly once, while holding the owner guard."""
asset = session.get(SkillAssetRow, operation.target_ref)
owner = session.get(SkillOwnerRow, operation.owner_id)
if asset is None or owner is None or asset.operation_id != operation.operation_id:
operation.publication = "NEEDS_REPAIR"
operation.error_code = "ASSET_IDENTITY_CHANGED"
return
operation.publication = publication
if publication == "APPLIED":
after = operation.after_revision
asset.incarnation_id, asset.mutation_seq, asset.content_digest = after["incarnation_id"], after["mutation_seq"], after["content_digest"]
asset.mutating = False
owner.generation = operation.generation
supersede_views(session, operation.owner_id, owner.generation)
operation.error_code = None
elif publication == "ABORTED":
asset.mutating = False
asset.operation_id = operation.before_operation_id
operation.views = "SUPERSEDED"
operation.superseded_by_generation = owner.generation
operation.error_code = "PUBLICATION_ABORTED"
else:
operation.error_code = "DISK_STATE_UNKNOWN"
def resolve_prepared(repository, storage, operation_id: str) -> None:
"""Classify disk only. Recovery never replays a write."""
with repository.sessions.begin() as session:
operation = session.get(SkillOperationRow, operation_id)
if operation is None or operation.publication not in {"PREPARED", "NEEDS_REPAIR"}:
return
try:
operation_packages(operation)
except ValueError:
operation.publication = "NEEDS_REPAIR"
operation.error_code = "BLOB_INTEGRITY_ERROR"
return
try:
disk = capture_package(storage.get_custom_skill_dir(operation.name)).digest
except (OSError, ValueError):
disk = None
before, after = operation.before_revision["content_digest"], operation.after_revision["content_digest"]
publication = "ABORTED" if disk == before else "APPLIED" if disk == after else "NEEDS_REPAIR"
if publication != "NEEDS_REPAIR":
try:
durable_current(storage.get_custom_skill_dir(operation.name))
except (OSError, ValueError):
operation.error_code = "DURABILITY_UNCONFIRMED"
return
finalize(session, operation, publication)
if publication != "NEEDS_REPAIR":
cleanup_temporary(storage.get_custom_skill_dir(operation.name), operation_id)
def publish_prepared(repository, storage, operation_id: str) -> None:
with repository.sessions() as session:
operation = session.get(SkillOperationRow, operation_id)
try:
_, package = operation_packages(operation)
except ValueError:
resolve_prepared(repository, storage, operation_id)
return
name = operation.name
try:
replace_main(storage.get_custom_skill_dir(name), package.main_content.encode("utf-8"), operation_id=operation_id)
except Exception:
# A response failure after replace must never look like ordinary refusal.
# Journal classification distinguishes not-written, written and uncertain.
resolve_prepared(repository, storage, operation_id)
return
with repository.sessions.begin() as session:
finalize(session, session.get(SkillOperationRow, operation_id), "APPLIED")

View File

@ -0,0 +1,170 @@
"""Host-owned recovery, deletion fencing and retention independent of plugins."""
from __future__ import annotations
import time
from deerflow_extension_api.host_capabilities import HostCapabilityError
from sqlalchemy import and_, delete, exists, literal, or_, select, update
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOperationRow, SkillOwnerRow, SkillProposalRow, SkillScanAttemptRow
from deerflow.skills.mutations.codec import operation_view
from deerflow.skills.mutations.publication import resolve_prepared
from deerflow.skills.mutations.repository import supersede_views
from deerflow.skills.projection import skill_projection_read_lock
class SkillMutationRecovery:
def __init__(self, runtime, storage_factory, *, rebuild_views=None):
self.runtime, self.storage_factory = runtime, storage_factory
self.repository = runtime.repository
if rebuild_views is None:
from deerflow.skills.projection import rebuild_skill_projections
def rebuild_views(storage):
return rebuild_skill_projections(storage, include_public=False)
self.rebuild_views = rebuild_views
runtime.recover_locked = self.recover_locked
def get_operation(self, operation_id):
with self.repository.sessions() as session:
row = session.get(SkillOperationRow, operation_id)
if row is None:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
return operation_view(row)
def list_operations(self, *, limit=50, after_id=None):
if type(limit) is not int or not 1 <= limit <= 101 or (after_id is not None and (not isinstance(after_id, str) or len(after_id) > 32)):
raise HostCapabilityError("INVALID_REQUEST")
with self.repository.sessions() as session:
cursor = session.get(SkillOperationRow, after_id) if after_id else None
if after_id and cursor is None:
return ()
after = or_(SkillOperationRow.created_at > cursor.created_at, and_(SkillOperationRow.created_at == cursor.created_at, SkillOperationRow.operation_id > cursor.operation_id)) if cursor is not None else True
rows = session.scalars(
select(SkillOperationRow)
.where(
or_(SkillOperationRow.publication.in_(("PREPARED", "NEEDS_REPAIR")), (SkillOperationRow.publication == "APPLIED") & SkillOperationRow.views.in_(("PENDING", "ERROR"))),
after,
)
.order_by(SkillOperationRow.created_at, SkillOperationRow.operation_id)
.limit(limit)
).all()
return tuple(operation_view(row) for row in rows)
def recover_locked(self, storage):
"""Caller owns owner lock. No projection rebuild or plugin callbacks."""
with self.repository.sessions() as session:
pending = session.scalars(select(SkillOperationRow.operation_id).where(SkillOperationRow.owner_id == storage.user_id, SkillOperationRow.publication.in_(("PREPARED", "NEEDS_REPAIR")))).all()
for operation_id in pending:
resolve_prepared(self.repository, storage, operation_id)
with self.repository.sessions() as session:
uncertain = session.scalars(select(SkillAssetRow.name).where(SkillAssetRow.owner_id == storage.user_id, SkillAssetRow.mutating.is_(True))).all()
for name in uncertain:
digest, enabled, exists = self.runtime.state(storage, name)
self.repository.observe(storage.user_id, name, digest, enabled, exists=exists)
def refresh_views(self, owner_id):
storage = self.storage_factory(owner_id)
with skill_projection_read_lock(storage):
with self.repository.sessions.begin() as session:
owner = session.get(SkillOwnerRow, owner_id)
if owner is None or owner.deleting:
return
from deerflow.skills.mutations.history import mirror_operations
recent = session.scalars(
select(SkillOperationRow).where(SkillOperationRow.owner_id == owner_id, SkillOperationRow.publication == "APPLIED").order_by(SkillOperationRow.created_at.desc(), SkillOperationRow.operation_id).limit(100)
).all()
try:
mirror_operations(storage, [operation_view(row) for row in recent])
except Exception:
# Mirror availability never changes publication/view status.
pass
generation = owner.generation
supersede_views(session, owner_id, generation)
operations = session.scalars(select(SkillOperationRow.operation_id).where(SkillOperationRow.owner_id == owner_id, SkillOperationRow.publication == "APPLIED", SkillOperationRow.views.in_(("PENDING", "ERROR")))).all()
if not operations:
return
try:
self.rebuild_views(storage)
state = "READY"
except Exception:
state = "ERROR"
with skill_projection_read_lock(storage), self.repository.sessions.begin() as session:
owner = session.get(SkillOwnerRow, owner_id)
for operation_id in operations:
row = session.get(SkillOperationRow, operation_id)
if row is not None:
row.views = state if owner and owner.generation == row.generation == generation else "SUPERSEDED"
if row.views == "SUPERSEDED" and owner:
row.superseded_by_generation = owner.generation
def recover_owner(self, owner_id):
storage = self.storage_factory(owner_id)
with skill_projection_read_lock(storage):
self.recover_locked(storage)
self.repository.check_readable(owner_id)
self.refresh_views(owner_id)
def recover_operation(self, operation_id):
operation = self.get_operation(operation_id)
self.recover_owner(operation.owner_id)
return self.get_operation(operation_id)
def recover_all(self):
for owner_id in self._owners():
try:
self.recover_owner(owner_id)
except (HostCapabilityError, OSError, TimeoutError):
# Owner readiness remains closed; admin recovery stays reachable.
continue
def _owners(self):
after = ""
while True:
with self.repository.sessions() as session:
owners = session.scalars(select(SkillOwnerRow.owner_id).where(SkillOwnerRow.owner_id > after).order_by(SkillOwnerRow.owner_id).limit(100)).all()
if not owners:
return
yield from owners
after = owners[-1]
def quiesce_owner(self, owner_id):
# Accepted workers own this same lock through finalization. Taking it
# drains them before admission is fenced. Deletion need not keep blobs.
storage = self.storage_factory(owner_id)
with skill_projection_read_lock(storage):
self.recover_locked(storage)
with self.repository.sessions.begin() as session:
owner = session.get(SkillOwnerRow, owner_id)
if owner is None:
owner = self.repository.owner(session, owner_id)
owner.deleting = True
def collect_garbage(self):
now = time.time()
for owner_id in self._owners():
try:
self._collect_owner_garbage(owner_id, now)
except (HostCapabilityError, OSError, TimeoutError):
# Keep inaccessible owners' blobs intact for a later retry,
# without blocking startup or cleanup for unrelated owners.
continue
def _collect_owner_garbage(self, owner_id, now):
with skill_projection_read_lock(self.storage_factory(owner_id)), self.repository.sessions.begin() as session:
# Clear columns in SQL: retained idempotency history never causes
# unbounded blob hydration. Owner guard serializes new references.
live_operation = exists(select(SkillOperationRow.operation_id).where(SkillOperationRow.operation_id == SkillProposalRow.operation_id, SkillOperationRow.publication.in_(("PREPARED", "NEEDS_REPAIR"))))
live_scan = exists(select(SkillScanAttemptRow.token).where(SkillScanAttemptRow.subject == literal("check:") + SkillProposalRow.proposal_id, SkillScanAttemptRow.lease_until > now))
eligible = (SkillProposalRow.owner_id == owner_id, ~live_operation, ~live_scan)
session.execute(update(SkillProposalRow).where(*eligible, or_(SkillProposalRow.expires_at <= now, SkillProposalRow.state.in_(("DISCARDED", "PUBLISHED")))).values(package_blob=None, candidate_blob=None))
session.execute(update(SkillProposalRow).where(*eligible, SkillProposalRow.expires_at <= now, SkillProposalRow.state == "PENDING").values(state="EXPIRED"))
session.execute(
update(SkillOperationRow)
.where(SkillOperationRow.owner_id == owner_id, SkillOperationRow.rollback_expires_at <= now, SkillOperationRow.publication.not_in(("PREPARED", "NEEDS_REPAIR")))
.values(before_blob=None, after_blob=None)
)
session.execute(delete(SkillScanAttemptRow).where(SkillScanAttemptRow.owner_id == owner_id, SkillScanAttemptRow.started_at < now - 3600, SkillScanAttemptRow.lease_until <= now))

View File

@ -0,0 +1,127 @@
"""Short synchronous transactions, always acquired after the owner file guard."""
from __future__ import annotations
import uuid
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow_extension_api.skill_mutations import AssetRevision
from sqlalchemy import select, update
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOperationRow, SkillOwnerRow
from deerflow.persistence.user.model import UserRow
def revision(row: SkillAssetRow) -> AssetRevision:
return AssetRevision(row.incarnation_id, row.mutation_seq, row.content_digest)
def supersede_views(session, owner_id, generation):
session.execute(
update(SkillOperationRow)
.where(SkillOperationRow.owner_id == owner_id, SkillOperationRow.publication.in_(("APPLIED", "NO_CHANGE")), SkillOperationRow.generation < generation, SkillOperationRow.views != "SUPERSEDED")
.values(views="SUPERSEDED", superseded_by_generation=generation)
)
class SkillMutationRepository:
def __init__(self, sessions):
self.sessions = sessions
def owner(self, session, owner_id: str) -> SkillOwnerRow:
user = session.get(UserRow, owner_id)
if user is None:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
row = session.get(SkillOwnerRow, owner_id, with_for_update=True)
if row is None:
row = SkillOwnerRow(owner_id=owner_id, generation=0, deleting=False)
session.add(row)
session.flush()
if row.deleting:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
return row
@staticmethod
def asset(session, owner_id: str, name: str) -> SkillAssetRow | None:
return session.scalar(select(SkillAssetRow).where(SkillAssetRow.owner_id == owner_id, SkillAssetRow.name == name))
def observe(self, owner_id: str, name: str, digest: str, enabled: bool, *, exists: bool, mark_mutating: bool = False) -> SkillAssetRow:
with self.sessions.begin() as session:
owner = self.owner(session, owner_id)
row = self.asset(session, owner_id, name)
if row is not None and row.operation_id is not None:
operation = session.get(SkillOperationRow, row.operation_id)
if operation is not None and operation.publication in {"PREPARED", "NEEDS_REPAIR"}:
raise HostCapabilityError("NEEDS_REPAIR")
if row is None:
row = SkillAssetRow(
target_ref=uuid.uuid4().hex,
owner_id=owner_id,
name=name,
incarnation_id=uuid.uuid4().hex,
mutation_seq=1,
content_digest=digest,
enabled=enabled,
deleted=not exists,
mutating=False,
)
session.add(row)
owner.generation += 1
elif row.mutating or (row.content_digest, row.enabled, row.deleted) != (digest, enabled, not exists):
if row.deleted and exists:
row.incarnation_id = uuid.uuid4().hex
row.mutation_seq += 1
row.content_digest, row.enabled, row.deleted = digest, enabled, not exists
row.operation_id = None
owner.generation += 1
row.mutating = mark_mutating
supersede_views(session, owner_id, owner.generation)
session.flush()
return row
def finish_writer(self, owner_id: str, name: str, digest: str, enabled: bool, *, exists: bool) -> None:
with self.sessions.begin() as session:
owner = self.owner(session, owner_id)
row = self.asset(session, owner_id, name)
if row is None or not row.mutating:
raise HostCapabilityError("NEEDS_REPAIR")
if (row.content_digest, row.enabled, row.deleted) != (digest, enabled, not exists):
if row.deleted and exists:
row.incarnation_id = uuid.uuid4().hex
row.mutation_seq += 1
row.content_digest, row.enabled, row.deleted = digest, enabled, not exists
row.operation_id = None
owner.generation += 1
row.mutating = False
supersede_views(session, owner_id, owner.generation)
def generation(self, owner_id: str) -> int:
with self.sessions() as session:
row = session.get(SkillOwnerRow, owner_id)
return row.generation if row else 0
def check_readable(self, owner_id: str) -> None:
with self.sessions() as session:
if session.get(UserRow, owner_id) is None:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
owner = session.get(SkillOwnerRow, owner_id)
if owner is not None and owner.deleting:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
blocked = session.scalar(
select(SkillOperationRow.operation_id)
.where(
SkillOperationRow.owner_id == owner_id,
SkillOperationRow.publication.in_(("PREPARED", "NEEDS_REPAIR")),
)
.limit(1)
)
uncertain = session.scalar(
select(SkillAssetRow.target_ref)
.where(
SkillAssetRow.owner_id == owner_id,
SkillAssetRow.mutating.is_(True),
)
.limit(1)
)
if blocked is not None or uncertain is not None:
raise HostCapabilityError("NEEDS_REPAIR")

View File

@ -0,0 +1,38 @@
"""Cross-worker scan leases and hourly cost accounting, including reverts."""
import time
from deerflow_extension_api.host_capabilities import HostCapabilityError
from sqlalchemy import func, select, text
from deerflow.persistence.skill_mutations.model import SkillScanAttemptRow
def reserve_scan(session, *, plugin_id, owner_id, subject, token):
# Called before other statements in this short transaction. The global
# admission mutex guards count+insert on different owners and processes.
if session.bind.dialect.name == "sqlite":
session.execute(text("BEGIN IMMEDIATE"))
elif session.bind.dialect.name == "postgresql":
session.execute(text("SELECT pg_advisory_xact_lock(731129850451)"))
now = time.time()
def count(*conditions):
return session.scalar(select(func.count()).select_from(SkillScanAttemptRow).where(*conditions))
live = SkillScanAttemptRow.lease_until > now
plugin = SkillScanAttemptRow.plugin_id == plugin_id
owner = SkillScanAttemptRow.owner_id == owner_id
if count(live) >= 4 or count(live, plugin, owner) >= 2 or count(live, plugin, owner, SkillScanAttemptRow.subject == subject):
raise HostCapabilityError("BUSY", retry_after=1)
recent = SkillScanAttemptRow.started_at > now - 3600
if count(plugin, owner, recent) >= 20:
raise HostCapabilityError("QUOTA_EXCEEDED")
session.add(SkillScanAttemptRow(token=token, plugin_id=plugin_id, owner_id=owner_id, subject=subject, started_at=now, lease_until=now + 130))
def release_scan(repository, token):
with repository.sessions.begin() as session:
row = session.get(SkillScanAttemptRow, token)
if row is not None:
row.lease_until = 0

View File

@ -0,0 +1,111 @@
"""Fail-closed checks of exact detached packages, never model-visible activation."""
from __future__ import annotations
import asyncio
import hashlib
import json
import tempfile
from dataclasses import dataclass
from pathlib import Path
from deerflow.skills.mutations.assets import PackageSnapshot
from deerflow.skills.mutations.validation import MAX_MAIN_BYTES, PARSER_VERSION
from deerflow.skills.package_files import is_code_file
from deerflow.skills.security_scanner import scan_skill_content
from deerflow.skills.skillscan.orchestrator import scan_skill_dir
from deerflow.utils.file_io import await_drained
CHECK_POLICY_VERSION = "skill-mutation-check-v1"
MAX_MODEL_FILES = 32
MAX_MODEL_BYTES = 1024 * 1024
CHECK_TIMEOUT_SECONDS = 120
@dataclass(frozen=True)
class ScanVerdict:
decision: str
reason_code: str
policy_version: str
class CandidateScanner:
def __init__(self, config_provider):
self._config_provider = config_provider
@staticmethod
def _policy(config):
# Credentials never enter the returned value. Model configuration and
# explicit code-policy versions invalidate approvals after changes.
models = [model.model_dump(mode="json", exclude={"api_key"}) for model in config.models]
policy = [CHECK_POLICY_VERSION, PARSER_VERSION, config.skill_scan.enabled, config.skill_evolution.moderation_model_name, models]
return hashlib.sha256(json.dumps(policy, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
async def policy_version(self):
return await asyncio.to_thread(self.policy_version_sync)
def policy_version_sync(self):
"""Revalidate policy during short guarded publication admission."""
return self._policy(self._config_provider())
@staticmethod
def _prepare(package: PackageSnapshot):
text_files = []
total = 0
for item in package.files:
executable = item.executable or is_code_file(item.path, item.content[:4096])
try:
content = item.content.decode("utf-8")
except UnicodeError:
if executable or item.path == "SKILL.md":
return None, None
continue # Binary assets still receive the whole-package static scan.
if "\0" in content:
if executable or item.path == "SKILL.md":
return None, None
continue
total += len(item.content)
if len(item.content) > MAX_MAIN_BYTES or total > MAX_MODEL_BYTES or len(text_files) >= MAX_MODEL_FILES:
return None, None
text_files.append((item.path, content, executable))
with tempfile.TemporaryDirectory(prefix="deerflow-host-check-") as temporary:
root = Path(temporary)
for item in package.files:
target = root / item.path
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(item.content)
target.chmod(0o700 if item.executable else 0o600)
result = scan_skill_dir(root)
return result, text_files
async def scan(self, package: PackageSnapshot, name: str) -> ScanVerdict:
policy = "unavailable"
try:
config = await asyncio.to_thread(self._config_provider)
policy = self._policy(config)
if not config.skill_scan.enabled:
return ScanVerdict("unavailable", "SCANNER_DISABLED", policy)
# This capability never inherits the old manual path's fail-open flag.
config = config.model_copy(deep=True)
config.skill_evolution.security_fail_closed = True
async with asyncio.timeout(CHECK_TIMEOUT_SECONDS):
result, files = await await_drained(asyncio.to_thread(self._prepare, package))
if result is None:
return ScanVerdict("unavailable", "SCAN_LIMIT_EXCEEDED", policy)
if result["scanner_errors"]:
return ScanVerdict("unavailable", "SCANNER_INCOMPLETE", policy)
if result["blocked"]:
return ScanVerdict("reject", "STATIC_SCAN_REJECTED", policy)
for path, content, executable in files:
findings = [finding for finding in result["findings"] if finding["file"] == path]
scanned = await scan_skill_content(content, executable=executable, location=f"{name}/{path}", app_config=config, static_findings=findings, redact_diagnostics=True)
if not scanned.available:
return ScanVerdict("unavailable", "SCANNER_UNAVAILABLE", policy)
if scanned.decision == "block" or (executable and scanned.decision != "allow"):
return ScanVerdict("reject", "MODEL_SCAN_REJECTED", policy)
return ScanVerdict("allow", "CHECK_PASSED", policy)
except (asyncio.CancelledError, TimeoutError):
raise
except Exception:
# Error details can include model/provider content. Return codes only.
return ScanVerdict("unavailable", "SCANNER_UNAVAILABLE", policy)

View File

@ -0,0 +1,608 @@
"""Bound P0 host service: exact proposals, fail-closed checks and durable CAS."""
from __future__ import annotations
import asyncio
import json
import os
import time
import uuid
from contextlib import contextmanager
from dataclasses import asdict
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow_extension_api.skill_mutations import AssessmentRef, AssetRevision, BundleFile, MutationCapabilities, ProposalBundle, SkillRevisionView
from sqlalchemy import func, select, text
from sqlalchemy.exc import IntegrityError, OperationalError
from deerflow.persistence.run.model import CompletedRunSnapshotRow, RunRow
from deerflow.persistence.skill_mutations.model import SkillAssetRow, SkillOperationRow, SkillOwnerRow, SkillProposalRow
from deerflow.persistence.user.model import UserRow
from deerflow.skills.mutations.assets import MAX_PACKAGE_BYTES, capture_package
from deerflow.skills.mutations.codec import check_view, decode_package, encode_package, fingerprint, operation_packages, operation_view, proposal_view
from deerflow.skills.mutations.publication import assert_temporary_absent, publish_prepared
from deerflow.skills.mutations.recovery import SkillMutationRecovery
from deerflow.skills.mutations.repository import revision
from deerflow.skills.mutations.scan_budget import release_scan, reserve_scan
from deerflow.skills.mutations.validation import MAX_MAIN_BYTES, validate_candidate
from deerflow.skills.mutations.workers import MutationWorkers
from deerflow.skills.projection import skill_projection_read_lock
class HostSkillMutationService:
def __init__(self, binding, *, runtime, evidence, storage_factory, scanner, workers=None, recovery=None):
self.binding, self.runtime, self.evidence = binding, runtime, evidence
self.storage_factory, self.scanner = storage_factory, scanner
self.repository = runtime.repository
self._owns_workers = workers is None
self.workers = workers or MutationWorkers()
self.recovery = recovery or SkillMutationRecovery(runtime, storage_factory)
self._checks = {}
self._reverts = set()
self._closed = False
self._closing = False
@property
def capabilities(self):
return MutationCapabilities(operations=self.binding.access.skill_mutations.operations)
async def close(self):
self._closing = True
if self._checks or self._reverts:
from deerflow.utils.file_io import await_drained
await await_drained(asyncio.gather(*tuple(self._checks.values()), *tuple(self._reverts), return_exceptions=True))
self._closed = True
if self._owns_workers:
await self.workers.close()
def _authorize(self, owner_id, name, operation=None):
grant = self.binding.access.skill_mutations
allowed = self.binding.allows(operation, owner_id, name) if operation else bool(grant.operations) and owner_id in grant.owners and name in grant.target_skills
if not allowed or owner_id not in self.runtime.owners:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
if os.name != "posix" or grant.topology != "single_host_local":
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY")
def _owner_exists(self, session, owner_id):
owner = session.get(SkillOwnerRow, owner_id)
if session.get(UserRow, owner_id) is None or (owner is not None and owner.deleting):
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
def _row(self, session, model, identifier, operation=None):
row = session.get(model, identifier)
if row is None or row.plugin_id != self.binding.plugin_id:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
self._authorize(row.owner_id, row.name, operation)
self._owner_exists(session, row.owner_id)
return row
async def _work(self, work):
if self._closed:
raise HostCapabilityError("UNAVAILABLE")
def invoke(check):
try:
return work(check)
except TimeoutError as exc:
raise HostCapabilityError("BUSY", retry_after=1) from exc
except (OperationalError, IntegrityError) as exc:
raise HostCapabilityError("BUSY", retry_after=1) from exc
except ValueError as exc:
code = str(exc)
raise HostCapabilityError(code if code in {"UNSUPPORTED_ASSET", "REVISION_CONFLICT", "QUOTA_EXCEEDED", "INVALID_FRONTMATTER", "FRONTMATTER_CHANGED"} else "INVALID_REQUEST") from exc
return await self.workers.run(invoke)
@staticmethod
def _key(value):
if not isinstance(value, str) or not value or len(value) > 128 or any(ord(c) < 32 for c in value):
raise HostCapabilityError("INVALID_REQUEST")
@contextmanager
def _guard(self, owner_id, check):
storage = self.storage_factory(owner_id)
with skill_projection_read_lock(storage, check=check):
check()
self.recovery.recover_locked(storage)
self.repository.check_readable(owner_id)
yield storage
def _capture(self, storage, name):
package = capture_package(storage.get_custom_skill_dir(name))
_, enabled, exists = self.runtime.state(storage, name)
row = self.repository.observe(storage.user_id, name, package.digest, enabled, exists=exists)
if not exists:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
return row, package
async def _source(self, source_ref):
try:
snap = await self.evidence.resolve_snapshot(source_ref)
except HostCapabilityError as exc:
raise HostCapabilityError("SOURCE_STALE_OR_GONE") from exc
grant = self.binding.access.skill_mutations
if snap.owner_id not in grant.owners or snap.agent_id not in grant.trigger_agents or snap.seal_state != "sealed" or snap.origin not in {"interactive", "scheduled"} or snap.coverage != "lead-journal-v1":
raise HostCapabilityError("SOURCE_STALE_OR_GONE")
return asdict(snap)
def _sources_locked(self, session, sources, owner_id):
"""Retain run share locks until PREPARED commits; deletion takes UPDATE."""
for source in sorted(sources, key=lambda item: item["run_id"]):
stored = session.get(CompletedRunSnapshotRow, source["snapshot_ref"])
run = session.scalar(select(RunRow).where(RunRow.run_id == source["run_id"]).with_for_update(read=True))
if (
stored is None
or stored.scope_digest != self.evidence.scope_digest
or fingerprint(stored.snapshot_json) != fingerprint(source)
or run is None
or run.user_id != owner_id
or run.thread_id != source["thread_id"]
or run.operation_kind != "run"
or run.evidence_seal_state != "sealed"
or run.evidence_origin not in {"interactive", "scheduled"}
or run.evidence_agent_id not in self.binding.access.skill_mutations.trigger_agents
or run.evidence_origin != source["origin"]
or run.evidence_agent_id != source["agent_id"]
or source["owner_id"] != owner_id
or source.get("coverage") != "lead-journal-v1"
or run.evidence_retention_revision != source["retention_revision"]
or self.evidence.revision_for_run(run) != source["evidence_revision"]
):
raise HostCapabilityError("SOURCE_STALE_OR_GONE")
@staticmethod
def _begin_fence(session):
if session.bind.dialect.name == "sqlite":
session.execute(text("BEGIN IMMEDIATE"))
async def read_skill(self, *, source_ref, name):
source = await self._source(source_ref)
self._authorize(source["owner_id"], name)
def read(check):
with self._guard(source["owner_id"], check) as storage:
with self.repository.sessions() as session:
self._sources_locked(session, [source], source["owner_id"])
asset, package = self._capture(storage, name)
return SkillRevisionView(asset.target_ref, asset.owner_id, name, revision(asset), package.main_content, asset.enabled)
return await self._work(read)
async def stage(self, *, source_refs, target_ref, expected_revision, content, idempotency_key):
self._key(idempotency_key)
if not isinstance(source_refs, (tuple, list)) or not 1 <= len(source_refs) <= 20 or len(set(source_refs)) != len(source_refs):
raise HostCapabilityError("QUOTA_EXCEEDED")
if not isinstance(expected_revision, AssetRevision) or not isinstance(content, str):
raise HostCapabilityError("INVALID_REQUEST")
if len(content.encode("utf-8")) > MAX_MAIN_BYTES:
raise HostCapabilityError("QUOTA_EXCEEDED")
request = fingerprint([list(source_refs), target_ref, asdict(expected_revision), content])
def existing(_):
with self.repository.sessions() as session:
row = session.scalar(select(SkillProposalRow).where(SkillProposalRow.plugin_id == self.binding.plugin_id, SkillProposalRow.idempotency_key == idempotency_key))
if row is None:
return None
self._row(session, SkillProposalRow, row.proposal_id, "stage")
if row.request_hash != request:
raise HostCapabilityError("IDEMPOTENCY_CONFLICT")
return proposal_view(row)
cached = await self._work(existing)
if cached is not None:
return cached
sources = [await self._source(ref) for ref in source_refs]
owner_id = sources[0]["owner_id"]
if any(source["owner_id"] != owner_id for source in sources):
raise HostCapabilityError("SOURCE_STALE_OR_GONE")
def create(check):
with self.repository.sessions() as session:
asset = session.get(SkillAssetRow, target_ref)
if asset is None or asset.owner_id != owner_id:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN")
name = asset.name
self._authorize(owner_id, name, "stage")
with self._guard(owner_id, check) as storage:
cached = existing(check)
if cached:
return cached
asset, baseline = self._capture(storage, name)
if asset.target_ref != target_ref or revision(asset) != expected_revision:
raise HostCapabilityError("REVISION_CONFLICT")
if not asset.enabled:
raise HostCapabilityError("ASSET_DISABLED")
validate_candidate(baseline.main_content, content, name)
candidate = baseline.with_main(content)
if sum(len(item.content) for item in candidate.files) > MAX_PACKAGE_BYTES:
raise HostCapabilityError("QUOTA_EXCEEDED")
with self.repository.sessions.begin() as session:
self._begin_fence(session)
self.repository.owner(session, owner_id)
self._sources_locked(session, sources, owner_id)
count = session.scalar(
select(func.count())
.select_from(SkillProposalRow)
.where(SkillProposalRow.owner_id == owner_id, SkillProposalRow.plugin_id == self.binding.plugin_id, SkillProposalRow.state == "PENDING", SkillProposalRow.expires_at > time.time())
)
if count >= 20:
raise HostCapabilityError("QUOTA_EXCEEDED")
check()
row = SkillProposalRow(
proposal_id=uuid.uuid4().hex,
plugin_id=self.binding.plugin_id,
owner_id=owner_id,
target_ref=target_ref,
name=name,
base_revision=asdict(expected_revision),
sources=sources,
candidate_hash=candidate.digest,
package_blob=encode_package(baseline),
candidate_blob=encode_package(candidate),
state="PENDING",
expires_at=time.time() + 86400,
idempotency_key=idempotency_key,
request_hash=request,
)
session.add(row)
session.flush()
result = proposal_view(row)
return result
return await self._work(create)
async def get_proposal(self, *, proposal_id):
def read(_):
with self.repository.sessions() as session:
return proposal_view(self._row(session, SkillProposalRow, proposal_id))
return await self._work(read)
@staticmethod
def _pending(row):
if row.state != "PENDING" or row.expires_at <= time.time() or row.package_blob is None or row.candidate_blob is None:
raise HostCapabilityError("PROPOSAL_UNAVAILABLE")
async def read_proposal_bundle(self, *, proposal_id, max_bytes=33554432):
if type(max_bytes) is not int or not 1 <= max_bytes <= 33554432:
raise HostCapabilityError("INVALID_REQUEST")
def read(_):
with self.repository.sessions() as session:
row = self._row(session, SkillProposalRow, proposal_id)
if row.package_blob is None or row.candidate_blob is None or row.state == "DISCARDED" or row.expires_at <= time.time():
raise HostCapabilityError("PROPOSAL_UNAVAILABLE")
baseline, candidate = decode_package(row.package_blob), decode_package(row.candidate_blob)
if sum(len(item.content) for package in (baseline, candidate) for item in package.files) > max_bytes:
raise HostCapabilityError("QUOTA_EXCEEDED")
bundles = [tuple(BundleFile(item.path, item.content, item.digest, item.executable) for item in package.files) for package in (baseline, candidate)]
return ProposalBundle(row.proposal_id, *bundles, AssetRevision(**row.base_revision), row.candidate_hash)
return await self._work(read)
async def discard(self, *, proposal_id):
owner_id = (await self.get_proposal(proposal_id=proposal_id)).owner_id
def discard(check):
with skill_projection_read_lock(self.storage_factory(owner_id), check=check), self.repository.sessions.begin() as session:
row = self._row(session, SkillProposalRow, proposal_id, "stage")
if row.operation_id:
raise HostCapabilityError("ALREADY_PUBLISHED")
row.state = "DISCARDED"
return proposal_view(row)
return await self._work(discard)
async def check(self, *, proposal_id):
if self._closed or self._closing:
raise HostCapabilityError("UNAVAILABLE")
task = self._checks.get(proposal_id)
if task is None:
task = asyncio.create_task(self._check(proposal_id))
self._checks[proposal_id] = task
def finished(done):
self._checks.pop(proposal_id, None)
if not done.cancelled():
done.exception()
task.add_done_callback(finished)
return await asyncio.shield(task)
async def _check(self, proposal_id):
proposal = await self.get_proposal(proposal_id=proposal_id)
policy = await self.scanner.policy_version()
token = uuid.uuid4().hex
def reserve(check):
with self._guard(proposal.owner_id, check) as storage:
asset, _ = self._capture(storage, proposal.name)
with self.repository.sessions.begin() as session:
row = self._row(session, SkillProposalRow, proposal_id, "check")
self._pending(row)
if revision(asset) != AssetRevision(**row.base_revision):
raise HostCapabilityError("REVISION_CONFLICT")
if not asset.enabled:
raise HostCapabilityError("ASSET_DISABLED")
now = time.time()
previous = row.check_result or {}
if previous.get("policy_version") == policy and previous.get("expires_at", 0) > now:
return check_view(row), None
if previous.get("lease_until", 0) > now:
raise HostCapabilityError("BUSY", retry_after=1)
reserve_scan(session, plugin_id=self.binding.plugin_id, owner_id=row.owner_id, subject="check:" + proposal_id, token=token)
row.check_result = {"token": token, "lease_until": now + 130}
return None, decode_package(row.candidate_blob)
cached, package = await self._work(reserve)
if cached:
return cached
def finish(check):
with skill_projection_read_lock(self.storage_factory(proposal.owner_id), check=check), self.repository.sessions.begin() as session:
row = self._row(session, SkillProposalRow, proposal_id, "check")
self._pending(row)
if (row.check_result or {}).get("token") != token:
raise HostCapabilityError("CHECK_REQUIRED")
row.check_result = {"decision": verdict.decision, "reason_code": verdict.reason_code, "policy_version": verdict.policy_version, "expires_at": time.time() + 900}
return check_view(row)
def cleanup(_check):
try:
with skill_projection_read_lock(self.storage_factory(proposal.owner_id)), self.repository.sessions.begin() as session:
row = session.get(SkillProposalRow, proposal_id)
if row is not None and (row.check_result or {}).get("token") == token:
# An interrupted attempt is not an authoritative verdict.
# Never clear a newer worker's lease or completed result.
row.check_result = None
finally:
release_scan(self.repository, token)
try:
try:
async with asyncio.timeout(120):
verdict = await self.scanner.scan(package, proposal.name)
except TimeoutError:
raise
except Exception:
from deerflow.skills.mutations.scanner import ScanVerdict
verdict = ScanVerdict("unavailable", "SCANNER_UNAVAILABLE", policy)
return await self._work(finish)
finally:
from deerflow.utils.file_io import await_drained
await await_drained(self.workers.run(cleanup))
def _existing_operation(self, session, method, key, request):
row = session.scalar(select(SkillOperationRow).where(SkillOperationRow.plugin_id == self.binding.plugin_id, SkillOperationRow.method == method, SkillOperationRow.idempotency_key == key))
if row is not None:
self._row(session, SkillOperationRow, row.operation_id, method)
if row.request_hash != request:
raise HostCapabilityError("IDEMPOTENCY_CONFLICT")
return row
def _prepare(self, session, *, owner_id, asset, before, after, method, key, request, proposal=None, reverts=None, assessment=None):
owner = self.repository.owner(session, owner_id)
changed = before.digest != after.digest
before_revision = revision(asset)
after_revision = AssetRevision(before_revision.incarnation_id, before_revision.mutation_seq + int(changed), after.digest)
operation = SkillOperationRow(
operation_id=uuid.uuid4().hex,
plugin_id=self.binding.plugin_id,
owner_id=owner_id,
target_ref=asset.target_ref,
name=asset.name,
method=method,
idempotency_key=key,
request_hash=request,
proposal_id=proposal.proposal_id if proposal else None,
reverts_operation_id=reverts,
before_operation_id=asset.operation_id,
before_revision=asdict(before_revision),
after_revision=asdict(after_revision),
before_blob=encode_package(before),
after_blob=encode_package(after),
generation=owner.generation + int(changed),
publication="PREPARED" if changed else "NO_CHANGE",
views="PENDING" if changed else "READY",
created_at=time.time(),
rollback_expires_at=time.time() + 2592000,
assessment=assessment,
)
if changed:
assert_temporary_absent(self.storage_factory(owner_id).get_custom_skill_dir(asset.name), operation.operation_id)
session.add(operation)
if changed:
canonical = session.get(SkillAssetRow, asset.target_ref)
canonical.operation_id = operation.operation_id
canonical.mutating = True
if proposal:
proposal.operation_id, proposal.state = operation.operation_id, "PUBLISHED"
session.flush()
return operation.operation_id, changed, operation_view(operation)
def _publish_result(self, storage, operation_id, changed, prepared):
try:
if changed:
publish_prepared(self.repository, storage, operation_id)
return self.recovery.get_operation(operation_id)
except Exception:
# Admission is already durable. Even an unavailable database must
# return the accepted ID, never a refusal that invites republication.
return prepared
def _refresh_result(self, owner_id, operation):
try:
self.recovery.refresh_views(owner_id)
return self.recovery.get_operation(operation.operation_id)
except Exception:
return operation
@staticmethod
def _assessment_payload(assessment_ref):
if assessment_ref is None:
return None
try:
if not isinstance(assessment_ref, AssessmentRef) or not isinstance(assessment_ref.base_revision, AssetRevision):
raise TypeError
assessment = asdict(assessment_ref)
base = assessment["base_revision"]
if (
not isinstance(assessment["candidate_hash"], str)
or len(assessment["candidate_hash"]) != 64
or set(base) != {"incarnation_id", "mutation_seq", "content_digest"}
or not isinstance(base["incarnation_id"], str)
or not isinstance(base["content_digest"], str)
or type(base["mutation_seq"]) is not int
or any(not isinstance(assessment[field], str) or not 1 <= len(assessment[field]) <= 512 for field in ("evaluator", "evaluator_version", "report_id"))
or len(json.dumps(assessment, allow_nan=False).encode("utf-8")) > 4096
):
raise ValueError
return assessment
except Exception as exc:
raise HostCapabilityError("INVALID_ASSESSMENT") from exc
async def commit(self, *, proposal_id, idempotency_key, assessment_ref=None):
self._key(idempotency_key)
assessment = self._assessment_payload(assessment_ref)
request = fingerprint([proposal_id, assessment])
proposal = await self.get_proposal(proposal_id=proposal_id)
def commit(check):
with skill_projection_read_lock(self.storage_factory(proposal.owner_id), check=check):
storage = self.storage_factory(proposal.owner_id)
self.recovery.recover_locked(storage)
with self.repository.sessions() as session:
row = self._row(session, SkillProposalRow, proposal_id, "commit")
existing = self._existing_operation(session, "commit", idempotency_key, request)
if existing is not None:
return operation_view(existing)
if row.operation_id:
raise HostCapabilityError("PROPOSAL_ALREADY_COMMITTED")
self.repository.check_readable(proposal.owner_id)
asset, baseline = self._capture(storage, proposal.name)
with self.repository.sessions.begin() as session:
self._begin_fence(session)
row = self._row(session, SkillProposalRow, proposal_id, "commit")
self._pending(row)
self.repository.owner(session, row.owner_id)
self._sources_locked(session, row.sources, row.owner_id)
if asset.target_ref != row.target_ref or revision(asset) != AssetRevision(**row.base_revision):
raise HostCapabilityError("REVISION_CONFLICT")
if not asset.enabled:
raise HostCapabilityError("ASSET_DISABLED")
checked = row.check_result or {}
try:
policy = self.scanner.policy_version_sync()
except Exception as exc:
raise HostCapabilityError("CHECK_REQUIRED") from exc
if checked.get("decision") != "allow" or checked.get("expires_at", 0) <= time.time() or checked.get("policy_version") != policy:
raise HostCapabilityError("CHECK_REQUIRED")
if assessment and (assessment["candidate_hash"] != row.candidate_hash or assessment["base_revision"] != row.base_revision):
raise HostCapabilityError("INVALID_ASSESSMENT")
candidate = decode_package(row.candidate_blob)
if baseline.digest != decode_package(row.package_blob).digest or candidate.digest != row.candidate_hash:
raise HostCapabilityError("REVISION_CONFLICT")
validate_candidate(baseline.main_content, candidate.main_content, row.name)
check() # Last cancellation boundary: PREPARED is irrevocable admission.
operation_id, changed, prepared = self._prepare(session, owner_id=row.owner_id, asset=asset, before=baseline, after=candidate, method="commit", key=idempotency_key, request=request, proposal=row, assessment=assessment)
result = self._publish_result(storage, operation_id, changed, prepared)
return self._refresh_result(proposal.owner_id, result)
return await self._work(commit)
async def get_operation(self, *, operation_id):
def read(_):
with self.repository.sessions() as session:
return operation_view(self._row(session, SkillOperationRow, operation_id))
return await self._work(read)
async def find_operation(self, *, idempotency_key, method="commit"):
self._key(idempotency_key)
if method not in {"commit", "revert"}:
raise HostCapabilityError("INVALID_REQUEST")
def read(_):
with self.repository.sessions() as session:
row = session.scalar(select(SkillOperationRow).where(SkillOperationRow.plugin_id == self.binding.plugin_id, SkillOperationRow.method == method, SkillOperationRow.idempotency_key == idempotency_key))
return operation_view(self._row(session, SkillOperationRow, row.operation_id)) if row else None
return await self._work(read)
async def revert(self, *, operation_id, expected_current_revision, idempotency_key):
if self._closing or self._closed:
raise HostCapabilityError("UNAVAILABLE")
task = asyncio.create_task(self._revert(operation_id=operation_id, expected_current_revision=expected_current_revision, idempotency_key=idempotency_key))
self._reverts.add(task)
task.add_done_callback(self._reverts.discard)
return await task
async def _revert(self, *, operation_id, expected_current_revision, idempotency_key):
self._key(idempotency_key)
request = fingerprint([operation_id, asdict(expected_current_revision)])
def inspect(check):
with self.repository.sessions() as session:
original = self._row(session, SkillOperationRow, operation_id, "revert")
existing = self._existing_operation(session, "revert", idempotency_key, request)
if existing:
return operation_view(existing), None, None
if original.publication != "APPLIED" or original.before_blob is None or original.rollback_expires_at <= time.time():
raise HostCapabilityError("ROLLBACK_UNAVAILABLE")
try:
before, _ = operation_packages(original)
except ValueError as exc:
raise HostCapabilityError("ROLLBACK_UNAVAILABLE") from exc
return None, operation_view(original), before
existing, original, restored = await self._work(inspect)
if existing:
return existing
token = uuid.uuid4().hex
def reserve(check):
with self._guard(original.owner_id, check) as storage:
asset, _ = self._capture(storage, original.name)
if revision(asset) != expected_current_revision or revision(asset) != original.after_revision or asset.operation_id != operation_id:
raise HostCapabilityError("REVISION_CONFLICT")
with self.repository.sessions.begin() as session:
reserve_scan(session, plugin_id=self.binding.plugin_id, owner_id=original.owner_id, subject="revert:" + operation_id, token=token)
await self._work(reserve)
try:
async with asyncio.timeout(120):
verdict = await self.scanner.scan(restored, original.name)
except Exception as exc:
raise HostCapabilityError("CHECK_REQUIRED") from exc
finally:
await self.workers.run(lambda _: release_scan(self.repository, token))
if verdict.decision != "allow":
raise HostCapabilityError("CHECK_REQUIRED")
def revert(check):
with self._guard(original.owner_id, check) as storage:
asset, current = self._capture(storage, original.name)
with self.repository.sessions.begin() as session:
row = self._row(session, SkillOperationRow, operation_id, "revert")
existing = self._existing_operation(session, "revert", idempotency_key, request)
if existing:
return operation_view(existing)
if row.before_blob is None or row.rollback_expires_at <= time.time():
raise HostCapabilityError("ROLLBACK_UNAVAILABLE")
if revision(asset) != expected_current_revision or revision(asset) != AssetRevision(**row.after_revision) or asset.operation_id != row.operation_id:
raise HostCapabilityError("REVISION_CONFLICT")
if not asset.enabled:
raise HostCapabilityError("ASSET_DISABLED")
if verdict.policy_version != self.scanner.policy_version_sync():
raise HostCapabilityError("CHECK_REQUIRED")
if restored.digest != decode_package(row.before_blob).digest:
raise HostCapabilityError("ROLLBACK_UNAVAILABLE")
check()
new_id, changed, prepared = self._prepare(session, owner_id=row.owner_id, asset=asset, before=current, after=restored, method="revert", key=idempotency_key, request=request, reverts=row.operation_id)
result = self._publish_result(storage, new_id, changed, prepared)
return self._refresh_result(original.owner_id, result)
return await self._work(revert)

View File

@ -0,0 +1,81 @@
"""Publication topology admission and a short-transaction durable DB pool."""
from __future__ import annotations
import os
from pathlib import Path
from deerflow_extension_api.host_capabilities import HostCapabilityError
from sqlalchemy import create_engine, event
from sqlalchemy.engine import make_url
from sqlalchemy.orm import sessionmaker
class PublicationTopology:
"""Operator attests local single host; SQLite additionally enforces one host.
A file lock cannot prove that a mount is not NFS/CSI. That remains an explicit
deployment prerequisite, not a capability inferred from PostgreSQL alone.
"""
def __init__(self, backend: str, root: Path):
self.backend = backend
self.root = root
self._fd: int | None = None
def acquire(self):
if os.name != "posix" or self.backend not in {"sqlite", "postgres"}:
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY")
if self._fd is not None or self.backend == "postgres":
return
import fcntl
self.root.mkdir(parents=True, exist_ok=True)
fd = os.open(self.root / ".skill-mutations-gateway.lock", os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
try:
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError as exc:
os.close(fd)
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY", "SQLite publication requires one Gateway process") from exc
self._fd = fd
def close(self):
if self._fd is not None:
fd, self._fd = self._fd, None
os.close(fd)
def mutation_session_factory(url: str, *, postgres_schema: str = ""):
"""Same application DB, dedicated bounded pool and fsync-strict journal writes.
Do not change PRAGMAs on the shared agent-definition engine. FULL is needed
for PREPARED to be durable before replacing the canonical filesystem file.
"""
parsed = make_url(url)
if parsed.get_backend_name() == "sqlite":
if not parsed.database or parsed.database == ":memory:":
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY")
args = {"check_same_thread": False, "timeout": 5}
elif parsed.get_backend_name() == "postgresql":
from deerflow.config.postgres_schema import validate_postgres_schema
from deerflow.persistence.postgres_schema import build_psycopg_options
options = build_psycopg_options(validate_postgres_schema(postgres_schema)) or ""
args = {"connect_timeout": 5, "options": f"{options} -c lock_timeout=5000 -c statement_timeout=30000 -c synchronous_commit=on".strip()}
else:
raise HostCapabilityError("UNSUPPORTED_TOPOLOGY")
engine = create_engine(url, connect_args=args, pool_size=4, max_overflow=0, pool_timeout=5, pool_pre_ping=True)
if parsed.get_backend_name() == "sqlite":
@event.listens_for(engine, "connect")
def setup(connection, _record):
cursor = connection.cursor()
try:
cursor.execute("PRAGMA journal_mode=WAL")
cursor.execute("PRAGMA synchronous=FULL")
cursor.execute("PRAGMA foreign_keys=ON")
cursor.execute("PRAGMA busy_timeout=5000")
finally:
cursor.close()
return engine, sessionmaker(engine, expire_on_commit=False)

View File

@ -0,0 +1,73 @@
"""Strict P0 authority-preserving validation; never activate the candidate."""
from __future__ import annotations
import json
import yaml
from deerflow.skills.frontmatter import _FRONTMATTER_RE, ALLOWED_FRONTMATTER_PROPERTIES
from deerflow.skills.validation import validate_skill_frontmatter_text
MAX_MAIN_BYTES = 128 * 1024
PARSER_VERSION = "skill-mutation-frontmatter-v1"
class _StrictLoader(yaml.SafeLoader):
def construct_mapping(self, node, deep=False):
result = {}
for key_node, value_node in node.value:
key = self.construct_object(key_node, deep=deep)
if not isinstance(key, str) or key in result:
raise ValueError("INVALID_FRONTMATTER")
result[key] = self.construct_object(value_node, deep=deep)
return result
def _metadata(content: str) -> dict:
if not isinstance(content, str) or len(content.encode("utf-8")) > MAX_MAIN_BYTES:
raise ValueError("QUOTA_EXCEEDED")
match = _FRONTMATTER_RE.match(content)
if match is None:
raise ValueError("INVALID_FRONTMATTER")
source = match.group(1)
try:
depth = 0
for count, event in enumerate(yaml.parse(source, Loader=yaml.SafeLoader)):
if count > 4096 or isinstance(event, yaml.events.AliasEvent):
raise ValueError("INVALID_FRONTMATTER")
if isinstance(event, (yaml.events.MappingStartEvent, yaml.events.SequenceStartEvent)):
depth += 1
elif isinstance(event, (yaml.events.MappingEndEvent, yaml.events.SequenceEndEvent)):
depth -= 1
if depth > 24:
raise ValueError("INVALID_FRONTMATTER")
metadata = yaml.load(source, Loader=_StrictLoader)
if not isinstance(metadata, dict):
raise ValueError("INVALID_FRONTMATTER")
# Reject values without stable JSON/type semantics (dates, sets, NaN).
json.dumps(metadata, allow_nan=False)
return metadata
except (yaml.YAMLError, ValueError, TypeError, RecursionError, OverflowError) as exc:
raise ValueError("INVALID_FRONTMATTER") from exc
def _typed(value):
if isinstance(value, dict):
return (dict, tuple(sorted((key, _typed(item)) for key, item in value.items())))
if isinstance(value, list):
return (list, tuple(_typed(item) for item in value))
return (type(value), value)
def validate_candidate(baseline: str, candidate: str, name: str) -> None:
"""Only body and description can differ; unknown declarations stay intact."""
before, after = _metadata(baseline), _metadata(candidate)
if _typed({k: v for k, v in before.items() if k != "description"}) != _typed({k: v for k, v in after.items() if k != "description"}):
raise ValueError("FRONTMATTER_CHANGED")
# Reuse runtime/install field validation without discarding unknown values
# from the actual candidate: comparison above already preserves them.
known = {k: v for k, v in after.items() if k in ALLOWED_FRONTMATTER_PROPERTIES}
valid, _, declared_name = validate_skill_frontmatter_text("---\n" + yaml.safe_dump(known) + "---\n")
if not valid or declared_name != name:
raise ValueError("INVALID_FRONTMATTER")

View File

@ -0,0 +1,67 @@
"""Bounded off-loop work. Caller cancellation never releases a worker's locks."""
from __future__ import annotations
import asyncio
import threading
from concurrent.futures import ThreadPoolExecutor
from contextvars import copy_context
from deerflow_extension_api.host_capabilities import HostCapabilityError
class MutationWorkers:
def __init__(self, *, limit: int = 4):
self._slots = threading.BoundedSemaphore(limit)
self._executor = ThreadPoolExecutor(max_workers=limit, thread_name_prefix="skill-mutation")
self._tasks: set[asyncio.Future] = set()
self._closed = False
async def run(self, work):
if self._closed:
raise HostCapabilityError("UNAVAILABLE")
if not self._slots.acquire(blocking=False):
raise HostCapabilityError("BUSY", retry_after=1)
cancelled = threading.Event()
def check():
# Call during admission/lock retries, never after PREPARED.
if cancelled.is_set():
raise HostCapabilityError("CANCELLED")
def invoke():
try:
check()
return work(check)
finally:
self._slots.release()
context = copy_context()
try:
task = asyncio.get_running_loop().run_in_executor(self._executor, context.run, invoke)
except BaseException:
self._slots.release()
raise
self._tasks.add(task)
def finished(done):
self._tasks.discard(done)
if not done.cancelled():
done.exception() # Detached cancellations still consume failures.
task.add_done_callback(finished)
try:
return await asyncio.shield(task)
except asyncio.CancelledError:
cancelled.set()
raise
async def close(self):
from deerflow.utils.file_io import await_drained
self._closed = True
if self._tasks:
await await_drained(asyncio.gather(*tuple(self._tasks), return_exceptions=True))
# All accepted functions have completed. Signal idle threads to exit
# without making shutdown depend on the shared default executor.
self._executor.shutdown(wait=False, cancel_futures=True)

View File

@ -33,6 +33,7 @@ except ImportError: # pragma: no cover - Windows
_locks_guard = threading.Lock()
_process_locks: dict[Path, threading.RLock] = {}
_held_projection_locks = threading.local()
_MANIFEST_VERSION = 1
_MAX_REBUILD_ATTEMPTS = 2
_THREAD_PROJECTION_POLICY_VERSION = 1
@ -98,25 +99,59 @@ def _lock_for(path: Path) -> threading.RLock:
@contextmanager
def _projection_lock(root: Path) -> Iterator[None]:
def _projection_lock(root: Path, *, timeout: float | None = None, check=None) -> Iterator[None]:
"""Serialize projection replacement in-process and across POSIX workers."""
import time
lock_path = root.parent / f".{root.name}.projection.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
process_lock = _lock_for(lock_path)
with process_lock, lock_path.open("a", encoding="utf-8") as lock_file:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_EX)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_LOCK, 1)
try:
key = lock_path.resolve()
deadline = None if timeout is None else time.monotonic() + timeout
while not process_lock.acquire(timeout=0.05):
if check:
check()
if deadline is not None and time.monotonic() >= deadline:
raise TimeoutError("Skill projection lock timeout")
try:
held = getattr(_held_projection_locks, "paths", None)
if held is None:
held = _held_projection_locks.paths = set()
if key in held:
# Opening and flocking a second descriptor deadlocks even though
# the in-process RLock is reentrant. Nested readers share the fd.
yield
finally:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_UN)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
return
with lock_path.open("a", encoding="utf-8") as lock_file:
def acquire_region() -> None:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_NBLCK, 1)
retry_until_locked(
acquire_region,
interval=0.02,
deadline=deadline,
sleep=time.sleep,
monotonic=time.monotonic,
check=check,
timeout_message="Skill projection lock timeout",
)
held.add(key)
try:
yield
finally:
held.remove(key)
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_UN)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
finally:
process_lock.release()
def _copy_into_view(source: str, target: str, *, follow_symlinks: bool = True) -> str:
@ -614,6 +649,19 @@ def ensure_thread_skill_projection(
storage: SkillStorage,
thread_id: str,
allowed_skills: set[str] | None,
) -> SkillProjectionPaths | None:
from deerflow.skills.mutations.guard import managed_global_read, managed_read
# Global -> owner -> thread. No fresh-view fast path bypasses recovery.
with managed_global_read(storage):
with managed_read(storage):
return _ensure_thread_skill_projection(storage, thread_id, allowed_skills)
def _ensure_thread_skill_projection(
storage: SkillStorage,
thread_id: str,
allowed_skills: set[str] | None,
) -> SkillProjectionPaths | None:
"""Ensure the filesystem view for one run's effective Agent skill policy.
@ -655,6 +703,19 @@ def rebuild_skill_projections(
*,
include_public: bool = True,
include_user: bool = True,
) -> SkillProjectionPaths:
from deerflow.skills.mutations.guard import managed_global_read, managed_read
with managed_global_read(storage):
with managed_read(storage):
return _rebuild_skill_projections(storage, include_public=include_public, include_user=include_user)
def _rebuild_skill_projections(
storage: SkillStorage,
*,
include_public: bool = True,
include_user: bool = True,
) -> SkillProjectionPaths:
"""Rebuild enabled-only projection scopes visible through ``storage``."""
paths = get_skill_projection_paths(storage)
@ -682,6 +743,14 @@ def _public_projection_is_fresh(storage: SkillStorage, paths: SkillProjectionPat
def ensure_skill_projections(storage: SkillStorage) -> SkillProjectionPaths:
from deerflow.skills.mutations.guard import managed_global_read, managed_read
with managed_global_read(storage):
with managed_read(storage):
return _ensure_skill_projections(storage)
def _ensure_skill_projections(storage: SkillStorage) -> SkillProjectionPaths:
"""Repair stale projection scopes, otherwise leave their inodes untouched."""
paths = get_skill_projection_paths(storage)
@ -729,6 +798,7 @@ def skill_projection_mutation(
*,
remove: tuple[tuple[SkillCategory, Path], ...] = (),
remove_names: tuple[str, ...] = (),
mutation_names: tuple[str, ...] = (),
) -> Iterator[None]:
"""Hold a projection scope lock across a source/state mutation."""
if not isinstance(storage.get_skills_root_path(), Path):
@ -767,7 +837,9 @@ def skill_projection_mutation(
_validate_projection_relative_path(relative_path)
removals.add((root, relative_path))
with _projection_lock(scope_root):
from deerflow.skills.mutations.guard import managed_name_writes
with managed_name_writes(storage, mutation_names), _projection_lock(scope_root):
if remove_names:
names = set(remove_names)
skills = _load_public_skills(storage, enabled_only=False) if scope == "public" else storage.load_skills(enabled_only=False)
@ -782,7 +854,10 @@ def skill_projection_mutation(
_manifest_path(scope_root).unlink(missing_ok=True)
for root, relative_path in removals:
_remove_projection_relative(root, relative_path)
yield
from deerflow.skills.mutations.guard import managed_writer
with managed_writer(storage, mutation_names):
yield
rebuild()
except Exception:
_clear_projection_scope(scope_root, *category_roots.values())
@ -819,50 +894,6 @@ def ensure_public_skill_projection(*, app_config=None) -> bool:
@contextmanager
def skill_projection_read_lock(storage: SkillStorage, *, timeout: float = 5.0, check=None) -> Iterator[None]:
"""Bounded, non-mutating acquisition of the existing user projection lock."""
import time
root = (storage.get_skills_root_path() / "custom") if getattr(storage, "user_id", None) is None else get_skill_projection_paths(storage).custom.parent
lock_path = root.parent / f".{root.name}.projection.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
process_lock = _lock_for(lock_path)
deadline = time.monotonic() + timeout
acquired = False
try:
while not acquired:
if check:
check()
acquired = process_lock.acquire(timeout=min(0.05, max(0, deadline - time.monotonic())))
if not acquired and time.monotonic() >= deadline:
raise TimeoutError("Skill projection lock timeout")
with lock_path.open("a", encoding="utf-8") as lock_file:
locked = False
try:
def acquire_region() -> None:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_NBLCK, 1)
retry_until_locked(
acquire_region,
interval=0.02,
deadline=deadline,
sleep=time.sleep,
monotonic=time.monotonic,
check=check,
timeout_message="Skill projection lock timeout",
)
locked = True
yield
finally:
if locked:
if fcntl is not None:
fcntl.flock(lock_file, fcntl.LOCK_UN)
else: # pragma: no cover - Windows
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
finally:
if acquired:
process_lock.release()
with _projection_lock(root, timeout=timeout, check=check):
yield

View File

@ -24,6 +24,7 @@ logger = logging.getLogger(__name__)
class ScanResult:
decision: str
reason: str
available: bool = True
def _resolve_fail_closed(app_config: AppConfig | None) -> bool:
@ -101,6 +102,7 @@ async def scan_skill_content(
app_config: AppConfig | None = None,
static_findings: list[dict[str, Any]] | None = None,
attach_tracing: bool = True,
redact_diagnostics: bool = False,
) -> ScanResult:
"""Screen skill content before it is written to disk.
@ -162,15 +164,18 @@ async def scan_skill_content(
decision = str(parsed.get("decision", "")).lower()
if decision in {"allow", "warn", "block"}:
return ScanResult(decision, str(parsed.get("reason") or "No reason provided."))
logger.warning("Security scan produced unparseable output: %s", raw[:200])
if redact_diagnostics:
logger.warning("Security scan produced unparseable output")
else:
logger.warning("Security scan produced unparseable output: %s", raw[:200])
except Exception:
logger.warning("Skill security scan model call failed; applying configured fail-closed/fail-open policy", exc_info=True)
logger.warning("Skill security scan model call failed; applying configured fail-closed/fail-open policy", exc_info=not redact_diagnostics)
if model_responded:
return ScanResult("block", "Security scan produced unparseable output; manual review required.")
return ScanResult("block", "Security scan produced unparseable output; manual review required.", available=False)
if executable:
return ScanResult("block", "Security scan unavailable for executable content; manual review required.")
return ScanResult("block", "Security scan unavailable for executable content; manual review required.", available=False)
if _resolve_fail_closed(app_config):
return ScanResult("block", "Security scan unavailable for skill content; manual review required.")
return ScanResult("block", "Security scan unavailable for skill content; manual review required.", available=False)
logger.warning("Security scan unavailable; failing open for non-executable skill content at %s (manual review recommended)", location)
return ScanResult("warn", "Security scan unavailable for non-executable skill content; manual review recommended.")
return ScanResult("warn", "Security scan unavailable for non-executable skill content; manual review recommended.", available=False)

View File

@ -55,10 +55,12 @@ def get_or_new_skill_storage(**kwargs) -> SkillStorage:
from deerflow.config import get_app_config
from deerflow.config.skills_config import SkillsConfig
def _make_storage(skills_config: SkillsConfig, *, host_path: str | None = None, **kwargs) -> SkillStorage:
def _make_storage(skills_config: SkillsConfig, *, app_config=None, host_path: str | None = None, **kwargs) -> SkillStorage:
from deerflow.reflection import resolve_class
cls = resolve_class(skills_config.use, SkillStorage)
if issubclass(cls, LocalSkillStorage):
kwargs["app_config"] = app_config
return cls(
host_path=host_path if host_path is not None else str(skills_config.get_skills_path()),
container_path=skills_config.container_path,
@ -70,7 +72,7 @@ def get_or_new_skill_storage(**kwargs) -> SkillStorage:
if skills_path is not None:
if app_config is not None:
return _make_storage(app_config.skills, host_path=str(skills_path), **kwargs)
return _make_storage(app_config.skills, app_config=app_config, host_path=str(skills_path), **kwargs)
# No app_config: use a default SkillsConfig so we never need to read config.yaml
# when the caller has already supplied an explicit host path.
from deerflow.config.skills_config import SkillsConfig
@ -78,7 +80,7 @@ def get_or_new_skill_storage(**kwargs) -> SkillStorage:
return _make_storage(SkillsConfig(), host_path=str(skills_path), **kwargs)
if app_config is not None:
return _make_storage(app_config.skills, **kwargs)
return _make_storage(app_config.skills, app_config=app_config, **kwargs)
# If the singleton was manually injected (e.g. in tests) without a config
# identity (_default_skill_storage_config is None), skip get_app_config()
@ -96,7 +98,7 @@ def get_or_new_skill_storage(**kwargs) -> SkillStorage:
# orphaned instance from a losing racer could not be cleaned up.
with _skill_storage_lock:
if _default_skill_storage is None or _default_skill_storage_config is not app_config_now:
_default_skill_storage = _make_storage(app_config_now.skills, **kwargs)
_default_skill_storage = _make_storage(app_config_now.skills, app_config=app_config_now, **kwargs)
_default_skill_storage_config = app_config_now
return _default_skill_storage

View File

@ -91,12 +91,15 @@ class LocalSkillStorage(SkillStorage):
yield category, category_path, Path(current_root) / SKILL_MD_FILE
def read_custom_skill(self, name: str) -> str:
if not self.custom_skill_exists(name):
raise FileNotFoundError(f"Custom skill '{name}' not found.")
return (self.get_custom_skill_dir(name) / SKILL_MD_FILE).read_text(encoding="utf-8")
from deerflow.skills.mutations.guard import managed_read
with managed_read(self):
if not self.custom_skill_exists(name):
raise FileNotFoundError(f"Custom skill '{name}' not found.")
return (self.get_custom_skill_dir(name) / SKILL_MD_FILE).read_text(encoding="utf-8")
def write_custom_skill(self, name: str, relative_path: str, content: str) -> None:
with self._skill_projection_mutation():
with self._skill_projection_mutation(names=(name,)):
target = self.validate_relative_path(relative_path, self.get_custom_skill_dir(name))
target.parent.mkdir(parents=True, exist_ok=True)
tmp_path = None
@ -112,7 +115,7 @@ class LocalSkillStorage(SkillStorage):
def remove_custom_skill_file(self, name: str, relative_path: str) -> str | None:
removal = ((SkillCategory.CUSTOM, Path(name)),)
with self._skill_projection_mutation(remove=removal):
with self._skill_projection_mutation(remove=removal, names=(name,)):
return super().remove_custom_skill_file(name, relative_path)
async def ainstall_skill_from_archive(self, archive_path: str | Path) -> dict:
@ -205,7 +208,7 @@ class LocalSkillStorage(SkillStorage):
"""Stage and move the validated skill into place (blocking; runs off the event loop)."""
from deerflow.skills.installer import _move_staged_skill_into_reserved_target
with self._skill_projection_mutation():
with self._skill_projection_mutation(names=(skill_name,)):
with tempfile.TemporaryDirectory(prefix=f".installing-{skill_name}-", dir=custom_dir) as staging_root:
staging_target = Path(staging_root) / skill_name
shutil.copytree(skill_dir, staging_target)
@ -229,7 +232,7 @@ class LocalSkillStorage(SkillStorage):
e,
)
removal = ((SkillCategory.CUSTOM, Path(name)),)
with self._skill_projection_mutation(remove=removal):
with self._skill_projection_mutation(remove=removal, names=(name,)):
if target.is_symlink():
# An operator-linked package (see
# ``_is_external_skill_directory_symlink``): the skill is the
@ -244,6 +247,7 @@ class LocalSkillStorage(SkillStorage):
*,
remove: tuple[tuple[SkillCategory, Path], ...] = (),
remove_names: tuple[str, ...] = (),
names: tuple[str, ...] = (),
):
if getattr(self, "user_id", None) is None:
from deerflow.skills.projection import _projection_lock
@ -251,7 +255,7 @@ class LocalSkillStorage(SkillStorage):
return _projection_lock(self.get_skills_root_path() / "custom")
from deerflow.skills.projection import skill_projection_mutation
return skill_projection_mutation(self, "user", remove=remove, remove_names=remove_names)
return skill_projection_mutation(self, "user", remove=remove, remove_names=remove_names, mutation_names=names)
def append_history(self, name: str, record: dict) -> None:
self.validate_skill_name(name)

View File

@ -157,7 +157,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
def set_skill_enabled_state(self, skill_name: str, enabled: bool) -> None:
"""Set the enabled state for a custom/legacy skill and persist."""
removal_names = (skill_name,) if not enabled else ()
with self._skill_projection_mutation(remove_names=removal_names):
with self._skill_projection_mutation(remove_names=removal_names, names=(skill_name,)):
states = self._read_skill_states()
states[skill_name] = {"enabled": enabled}
self._write_skill_states(states)
@ -185,6 +185,12 @@ class UserScopedSkillStorage(LocalSkillStorage):
# ------------------------------------------------------------------
def load_skills(self, *, enabled_only: bool = False) -> list:
from deerflow.skills.mutations.guard import managed_read
with managed_read(self):
return self._load_skills_unlocked(enabled_only=enabled_only)
def _load_skills_unlocked(self, *, enabled_only: bool = False) -> list:
"""Discover all skills and merge enabled state per isolation scope.
Delegates skill discovery and PUBLIC enabled-state to
@ -279,30 +285,33 @@ class UserScopedSkillStorage(LocalSkillStorage):
diagnostics cannot establish ownership outside the caller's custom root.
Root ancestors must not redirect into another user's storage either.
"""
root = self._user_custom_root
expected = self._paths.base_dir.resolve() / "users" / self._user_id / "skills" / "custom"
if root.resolve() != expected or not root.is_dir():
return []
diagnostics: list[SkillLoadDiagnostic] = []
for current, dirs, files in walk_skill_directories(root):
package_dir = Path(current)
if not package_dir.resolve().is_relative_to(expected):
dirs.clear()
continue
dirs[:] = sorted(name for name in dirs if not name.startswith("."))
if SKILL_MD_FILE not in files:
continue
dirs.clear()
try:
skill_file = (package_dir / SKILL_MD_FILE).resolve(strict=True)
if not skill_file.is_relative_to(expected):
from deerflow.skills.mutations.guard import managed_read
with managed_read(self):
root = self._user_custom_root
expected = self._paths.base_dir.resolve() / "users" / self._user_id / "skills" / "custom"
if root.resolve() != expected or not root.is_dir():
return []
diagnostics: list[SkillLoadDiagnostic] = []
for current, dirs, files in walk_skill_directories(root):
package_dir = Path(current)
if not package_dir.resolve().is_relative_to(expected):
dirs.clear()
continue
# Preserve the logical SKILL.md name for owned internal links.
parse_skill_file(package_dir / SKILL_MD_FILE, SkillCategory.CUSTOM, package_dir.relative_to(root), diagnostics=diagnostics)
except (OSError, RuntimeError):
# A concurrently removed/retargeted package is not a YAML error.
continue
return diagnostics
dirs[:] = sorted(name for name in dirs if not name.startswith("."))
if SKILL_MD_FILE not in files:
continue
dirs.clear()
try:
skill_file = (package_dir / SKILL_MD_FILE).resolve(strict=True)
if not skill_file.is_relative_to(expected):
continue
# Preserve the logical SKILL.md name for owned internal links.
parse_skill_file(package_dir / SKILL_MD_FILE, SkillCategory.CUSTOM, package_dir.relative_to(root), diagnostics=diagnostics)
except (OSError, RuntimeError):
# A concurrently removed/retargeted package is not a YAML error.
continue
return diagnostics
def _iter_skill_files(self) -> Iterable[tuple[SkillCategory, Path, Path]]:
# 1. Public skills: always from global root
@ -399,7 +408,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
# ------------------------------------------------------------------
def write_custom_skill(self, name: str, relative_path: str, content: str) -> None:
with self._skill_projection_mutation():
with self._skill_projection_mutation(names=(name,)):
target = self.validate_relative_path(relative_path, self.get_custom_skill_dir(name))
target.parent.mkdir(parents=True, exist_ok=True)
tmp_path = None

View File

@ -164,4 +164,5 @@ truncating its tail. Keep `step_events.py` pure and unit-tested.
Loop-detection, deferred-tool-promotion, and tool-progress audit records cross this boundary through one narrow exception. `task_tool` captures the parent loop and passes a `_ParentLoopMiddlewareRecorderProxy` to `SubagentExecutor`, which installs separate context keys and schedules `record_middleware` on the journal's owner loop. The proxy also owns the execution-local atomic promotion claim that deduplicates parallel `tool_search` calls. Other middleware consumers never see `__run_journal`. Close the shared proxy once before `task` returns to fence late events and drain accepted appends before completion capture. Only the proxy crosses loops; do not broaden it or call the event store from a subagent loop. Durable batch subagents have no parent journal and do not use this bridge.
**Reverse direction of the loop boundary — deferred cleanup & final usage delivery (#5069)**: when a task-tool poller exits unexpectedly, the registry cleanup is pinned **to** the persistent subagent loop via the public `run_on_isolated_subagent_loop()` (executor) so it survives caller-loop teardown — `asyncio.run()` cancels caller-loop tasks on exit, so a caller-loop `asyncio.create_task` would be cancelled before running. The final usage report crosses the boundary the **other way**: `_schedule_deferred_subagent_cleanup` captures the parent run's loop at unwind time (alive in every path that continues the run — the polling-timeout branch returns normally, and a generic poller error becomes an error `ToolMessage`), and `_deliver_final_usage_report` hands the report back onto that loop with `call_soon_threadsafe`. `record_external_llm_usage_records` must never be invoked from the persistent loop or a worker thread (`to_thread`): the journal's accumulators are unlocked read-modify-write fields and `get_completion_data()` iterates `_tokens_by_model`, so a cross-thread write silently loses token updates or breaks iteration mid-run — calling `_report_subagent_usage` directly inside `_deferred_cleanup_subagent_task` would reintroduce exactly this race. The deferred cleaner captures only the resolved usage recorder (plus ids and the captured report loop) — never the whole `runtime`: the strongly-referenced cleanup task lives for up to the full poll budget, and through `runtime` it would pin the parent run's journal and event store for that entire window. When the captured parent loop is already closed (synchronous `asyncio.run` teardown), the report is dropped on purpose — the run has persisted its completion data and nothing reads the counters back — and logged at info with the execution id and unaccounted record count, because the registry entry is removed right after and those records exist nowhere else.
**Deferred cleanup and final usage delivery (#5069)**: Unexpected task-tool poller exit schedules registry cleanup on the persistent subagent loop via `run_on_isolated_subagent_loop()`, so caller-loop teardown cannot cancel it. Capture the parent report loop during unwind (polling timeout continues the run; generic poller errors become ToolMessages). `_deliver_final_usage_report` sends final usage back with `call_soon_threadsafe`.
Never call `record_external_llm_usage_records` from the persistent loop or `to_thread`: journal counters use unlocked read-modify-write operations, and completion iterates `_tokens_by_model`. Capture only the resolved usage recorder, IDs and report loop, not the whole `runtime`; cleanup can retain them for the full poll budget. If the parent loop has closed after synchronous `asyncio.run`, drop the report because completion is already persisted. Log the execution ID and unaccounted record count at info before removing the registry entry; those records have no other owner.

View File

@ -26,7 +26,7 @@ from deerflow.skills.security_static_scanner import (
enforce_static_scan,
)
from deerflow.skills.storage import get_or_new_user_skill_storage
from deerflow.skills.storage.skill_storage import SkillStorage, read_text_or_none
from deerflow.skills.storage.skill_storage import SkillStorage
from deerflow.skills.types import SKILL_MD_FILE
from deerflow.tools.sync import make_sync_tool_wrapper
from deerflow.tools.types import Runtime
@ -155,7 +155,10 @@ async def _scan_static_candidate_or_raise(name: str, updates: dict[str, str], sk
if skill_storage is None:
skill_dir.mkdir(parents=True)
else:
shutil.copytree(skill_storage.get_custom_skill_dir(name), skill_dir)
from deerflow.skills.mutations.guard import managed_read
with managed_read(skill_storage):
shutil.copytree(skill_storage.get_custom_skill_dir(name), skill_dir)
for relative_path, content in updates.items():
target = skill_dir / relative_path
target.parent.mkdir(parents=True, exist_ok=True)
@ -229,8 +232,7 @@ async def _skill_manage_impl(
static_findings = await _scan_static_candidate_or_raise(name, {SKILL_MD_FILE: content})
scan = await _scan_or_raise(content, executable=False, location=f"{name}/{SKILL_MD_FILE}", static_findings=static_findings)
scan["static_findings"] = static_findings
skill_file = skill_storage.get_custom_skill_file(name)
prev_content = await _to_thread(skill_file.read_text, encoding="utf-8")
prev_content = await _to_thread(skill_storage.read_custom_skill, name)
await _to_thread(skill_storage.write_custom_skill, name, SKILL_MD_FILE, content)
await _to_thread(
skill_storage.append_history,
@ -244,8 +246,7 @@ async def _skill_manage_impl(
await _to_thread(skill_storage.ensure_custom_skill_is_editable, name)
if find is None or replace is None:
raise ValueError("find and replace are required for patch.")
skill_file = skill_storage.get_custom_skill_file(name)
prev_content = await _to_thread(skill_file.read_text, encoding="utf-8")
prev_content = await _to_thread(skill_storage.read_custom_skill, name)
occurrences = prev_content.count(find)
if occurrences == 0:
raise ValueError("Patch target not found in SKILL.md.")
@ -289,10 +290,9 @@ async def _skill_manage_impl(
target = await _to_thread(skill_storage.ensure_safe_support_path, name, path)
if await _to_thread(target.is_dir):
raise ValueError(f"Supporting file path '{path}' is a directory, not a file.")
exists = await _to_thread(target.exists)
# A binary asset (e.g. from a .skill archive) has no previous *text*;
# the history record takes None rather than aborting the write.
prev_content = await _to_thread(read_text_or_none, target) if exists else None
from deerflow.skills.mutations.guard import read_optional_text
prev_content = await _to_thread(read_optional_text, skill_storage, target)
executable = "scripts/" in path or path.startswith("scripts/")
static_findings = await _scan_static_candidate_or_raise(name, {path: content}, skill_storage)
scan = await _scan_or_raise(content, executable=executable, location=f"{name}/{path}", static_findings=static_findings)

View File

@ -0,0 +1,122 @@
"""Strict runtime coverage of the durable host, scanner, evidence and admin API."""
import asyncio
import os
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
from fastapi import FastAPI, Request
from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from sqlalchemy.orm import Session
from app.gateway.routers.skill_mutations import router
from deerflow.config.app_config import AppConfig
from deerflow.config.paths import Paths
from deerflow.extensions.host_capabilities import HostCapabilities
from deerflow.persistence.base import Base
from deerflow.persistence.run import RunRepository
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.user.model import UserRow
from deerflow.runtime.events.store.db import DbRunEventStore
from deerflow.skills.security_scanner import ScanResult
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
pytestmark = pytest.mark.skipif(os.name != "posix", reason="Skill publication requires POSIX local storage")
CONTENT = "---\nname: example\ndescription: A sample skill\n---\nOriginal instructions.\n"
@pytest.fixture
def durable_host(tmp_path, monkeypatch):
"""Only seed data synchronously; all host lifetime work stays under the gate."""
monkeypatch.setattr("deerflow.config.paths._paths", Paths(base_dir=tmp_path / "home"))
config = AppConfig.model_validate(
{
"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"},
"database": {"backend": "sqlite", "sqlite_dir": str(tmp_path)},
"skill_scan": {"enabled": True},
"plugins": [
{
"use": "fixture:install",
"name": "fixture",
"host_access": {
"evidence": {"owners": ["owner"]},
"skill_mutations": {"owners": ["owner"], "operations": ["stage", "check", "commit", "revert"], "trigger_agents": ["lead_agent"], "target_skills": ["example"], "topology": "single_host_local"},
},
}
],
}
)
monkeypatch.setattr("deerflow.config.get_app_config", lambda: config)
# Replace only the remote moderation boundary, retaining the real scanner,
# package staging/static scan, publication, projection and SQLite paths.
model = AsyncMock(return_value=ScanResult("allow", "safe"))
monkeypatch.setattr("deerflow.skills.mutations.scanner.scan_skill_content", model)
storage = UserScopedSkillStorage("owner", host_path=str(tmp_path / "skills"), app_config=config)
root = storage.get_custom_skill_dir("example")
root.mkdir(parents=True)
(root / "SKILL.md").write_text(CONTENT, encoding="utf-8")
engine = create_engine(config.database.app_sync_sqlalchemy_url)
try:
Base.metadata.create_all(engine)
with Session(engine) as session, session.begin():
session.add(UserRow(id="owner", email="owner@example.test"))
session.add(RunRow(run_id="run", thread_id="thread", user_id="owner", status="success", evidence_seal_state="sealed", evidence_agent_id="lead_agent", evidence_origin="interactive", evidence_upper_seq=2, evidence_event_count=2))
finally:
engine.dispose()
return SimpleNamespace(config=config, storage=storage, model=model, root=tmp_path, database_url=config.database.app_sqlalchemy_url, skill_file=root / "SKILL.md")
@pytest.mark.asyncio
async def test_host_check_evidence_publication_revert_and_admin_are_off_loop(durable_host):
seed = durable_host
engine = create_async_engine(seed.database_url)
sf = async_sessionmaker(engine, expire_on_commit=False)
events = DbRunEventStore(sf)
host = HostCapabilities(seed.config, RunRepository(sf), events, root=seed.root, storage_factory=lambda _: seed.storage)
try:
await events.put_batch([dict(thread_id="thread", run_id="run", user_id="owner", event_type="test", category="message", content=text) for text in ("first", "second")])
await host.start()
evidence, mutations = host.bindings["fixture:install"]
assert (await evidence.list_changed_runs()).items[0].run_id == "run"
snapshot = await evidence.get_snapshot(thread_id="thread", run_id="run")
from deerflow_extension_api import EvidenceLimits
page = await evidence.read_events(snapshot_ref=snapshot.snapshot_ref, limits=EvidenceLimits(max_events=1))
assert len(page.items) == 1 and page.has_more
next_page = await evidence.read_events(snapshot_ref=snapshot.snapshot_ref, cursor=page.next_cursor, limits=EvidenceLimits(max_events=1))
assert len(next_page.items) == 1 and not next_page.has_more
assert next_page.items[0].seq > page.items[0].seq
skill = await mutations.read_skill(source_ref=snapshot.snapshot_ref, name="example")
proposal = await mutations.stage(source_refs=(snapshot.snapshot_ref,), target_ref=skill.target_ref, expected_revision=skill.revision, content=CONTENT.replace("Original", "Updated"), idempotency_key="stage")
result = await mutations.check(proposal_id=proposal.proposal_id)
assert result.decision == "allow"
seed.model.assert_awaited_once()
operation = await mutations.commit(proposal_id=proposal.proposal_id, idempotency_key="commit")
assert operation.publication == "APPLIED" and operation.views == "READY"
app = FastAPI()
app.state.skill_mutation_host = host
@app.middleware("http")
async def identity(request: Request, call_next):
request.state.user = SimpleNamespace(id="admin", system_role="admin")
return await call_next(request)
app.include_router(router)
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as client:
url = f"/api/skill-mutations/operations/{operation.operation_id}"
assert (await client.get("/api/skill-mutations/operations")).status_code == 200
assert (await client.get(url)).json()["publication"] == "APPLIED"
assert (await client.post(url + "/recover")).json()["views"] == "READY"
assert (await client.post("/api/skill-mutations/owners/owner/recover")).status_code == 200
reverted = await mutations.revert(operation_id=operation.operation_id, expected_current_revision=operation.after_revision, idempotency_key="revert")
assert reverted.publication == "APPLIED" and reverted.views == "READY"
content = await asyncio.to_thread(seed.skill_file.read_text, encoding="utf-8")
assert content == CONTENT
finally:
await host.close()
await engine.dispose()

View File

@ -0,0 +1,25 @@
"""External-consumer contract fixture: imports only the standalone public API."""
from deerflow_extension_api import ExtensionRuntimeDeps
async def exercise(deps: ExtensionRuntimeDeps, *, thread_id: str, run_id: str, name: str):
evidence, skills = deps.completed_run_evidence, deps.skill_mutations
assert evidence is not None and skills is not None
snapshot = await evidence.get_snapshot(thread_id=thread_id, run_id=run_id)
assert snapshot.seal_state == "sealed"
await evidence.read_events(snapshot_ref=snapshot.snapshot_ref)
skill = await skills.read_skill(source_ref=snapshot.snapshot_ref, name=name)
proposal = await skills.stage(source_refs=(snapshot.snapshot_ref,), target_ref=skill.target_ref, expected_revision=skill.revision, content=skill.content + "\nA fixed test improvement.\n", idempotency_key="fixture-stage")
bundle = await skills.read_proposal_bundle(proposal_id=proposal.proposal_id)
assert bundle.base_revision == skill.revision
result = await skills.check(proposal_id=proposal.proposal_id)
assert result.decision == "allow"
operation = await skills.commit(proposal_id=proposal.proposal_id, idempotency_key="fixture-commit")
assert operation.publication == "APPLIED"
assert await skills.get_operation(operation_id=operation.operation_id) == operation
assert await skills.find_operation(idempotency_key="fixture-commit") == operation
reverted = await skills.revert(operation_id=operation.operation_id, expected_current_revision=operation.after_revision, idempotency_key="fixture-revert")
assert reverted.publication == "APPLIED"
assert reverted.after_revision.mutation_seq > operation.after_revision.mutation_seq
return operation, reverted

View File

@ -0,0 +1,80 @@
"""Fresh process terminated by SIGKILL at a selected publication boundary."""
import asyncio
import json
import os
import signal
import stat
import sys
from types import SimpleNamespace
from deerflow.config import paths
from deerflow.config.paths import Paths
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
from deerflow.extensions.host_access import BoundHostAccess, HostAccess
from deerflow.skills.mutations import publication
from deerflow.skills.mutations import service as service_module
from deerflow.skills.mutations.guard import SkillMutationRuntime, configure_mutation_runtime
from deerflow.skills.mutations.recovery import SkillMutationRecovery
from deerflow.skills.mutations.repository import SkillMutationRepository
from deerflow.skills.mutations.topology import mutation_session_factory
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
def die():
os.kill(os.getpid(), signal.SIGKILL)
# Signal delivery is asynchronous: never return to a pre-rename fault
# injection site if the process is still briefly scheduled.
while True:
signal.pause()
async def main(args):
paths._paths = Paths(base_dir=args["base"])
engine, sessions = mutation_session_factory(args["database"])
storage = UserScopedSkillStorage("owner", host_path=args["skills"])
runtime = SkillMutationRuntime(SkillMutationRepository(sessions), owners=frozenset({"owner"}))
configure_mutation_runtime(runtime)
recovery = SkillMutationRecovery(runtime, lambda _: storage, rebuild_views=lambda _: None)
binding = BoundHostAccess("plugin", HostAccess.model_validate(args["grant"]))
evidence = SimpleNamespace(scope_digest="scope", revision_for_run=HostCompletedRunEvidenceReader.revision_for_run)
service = service_module.HostSkillMutationService(binding, runtime=runtime, evidence=evidence, storage_factory=lambda _: storage, scanner=SimpleNamespace(policy_version_sync=lambda: "policy-1"), recovery=recovery)
boundary = args["boundary"]
if boundary == "prepared":
service_module.publish_prepared = lambda *_: die()
elif boundary == "written":
write = os.write
def write_then_die(fd, content):
write(fd, content)
die()
publication.os.write = write_then_die
elif boundary in {"file_synced", "directory_synced"}:
fsync = os.fsync
def sync_then_die(fd):
fsync(fd)
mode = os.fstat(fd).st_mode
if (boundary == "file_synced" and stat.S_ISREG(mode)) or (boundary == "directory_synced" and stat.S_ISDIR(mode)):
die()
publication.os.fsync = sync_then_die
elif boundary == "renamed":
replace = os.replace
def replace_then_die(*args, **kwargs):
replace(*args, **kwargs)
die()
publication.os.replace = replace_then_die
elif boundary == "response_lost":
recovery.refresh_views = lambda _: die()
await service.commit(proposal_id=args["proposal"], idempotency_key="crash-commit")
await service.close()
engine.dispose()
raise AssertionError("fault injection did not fire")
if __name__ == "__main__":
asyncio.run(main(json.loads(sys.argv[1])))

View File

@ -9,6 +9,7 @@ import stat
import tempfile
import threading
import zipfile
from contextlib import contextmanager
from enum import Enum
from pathlib import Path
from types import SimpleNamespace
@ -31,6 +32,7 @@ from deerflow.agents.thread_state import DeltaThreadState, ThreadState
from deerflow.client import DeerFlowClient, StreamEvent
from deerflow.config.agents_config import AgentConfig
from deerflow.config.authorization_config import AuthorizationConfig, AuthorizationProviderConfig
from deerflow.config.database_config import DatabaseConfig
from deerflow.config.extensions_config import ExtensionsConfig, McpServerConfig
from deerflow.config.paths import Paths
from deerflow.config.subagent_runtime_config import SubagentRuntimeConfig
@ -61,8 +63,7 @@ def mock_app_config():
config.skills.deferred_discovery = False
config.skills.container_path = "/mnt/skills"
config.tool_search.enabled = False
config.database.checkpoint_channel_mode = "full"
config.database.checkpoint_delta.snapshot_frequency = 10
config.database = DatabaseConfig(backend="memory")
config.authorization = AuthorizationConfig(enabled=False)
return config
@ -2786,6 +2787,46 @@ class TestSkillsManagement:
finally:
tmp_path.unlink()
def test_update_public_skill_enters_managed_write_before_projection(self, client, tmp_path):
from deerflow.skills import projection
from deerflow.skills.mutations import guard
skill = self._make_skill(enabled=True)
updated_skill = self._make_skill(enabled=False)
config_file = tmp_path / "extensions_config.json"
config_file.write_text('{"skills": {}}', encoding="utf-8")
events = []
@contextmanager
def record_managed_write(*_args, **_kwargs):
events.append("managed-enter")
try:
yield
finally:
events.append("managed-exit")
@contextmanager
def record_projection(*_args, **_kwargs):
events.append("projection-enter")
try:
yield
finally:
events.append("projection-exit")
with (
patch(
"deerflow.skills.storage.local_skill_storage.LocalSkillStorage.load_skills",
side_effect=[[skill], [skill], [updated_skill], [updated_skill]],
),
patch("deerflow.client.ExtensionsConfig.resolve_config_path", return_value=config_file),
patch("deerflow.client.reload_extensions_config"),
patch.object(guard, "managed_global_state_write", record_managed_write),
patch.object(projection, "skill_projection_mutation", record_projection),
):
client.update_skill("test-skill", enabled=False)
assert events == ["managed-enter", "projection-enter", "projection-exit", "managed-exit"]
def test_update_skill_persists_state_when_source_omits_skills(self, client):
skill = self._make_skill(enabled=True)
updated_skill = self._make_skill(enabled=False)
@ -2833,6 +2874,7 @@ class TestSkillsManagement:
skill = self._make_skill(enabled=True)
skill.category = category
storage = MagicMock()
storage._app_config = client._app_config
storage.load_skills.side_effect = [[skill], [self._make_skill(enabled=False)]]
with (

View File

@ -0,0 +1,156 @@
"""Only host entry points may label the provenance used by automatic publication."""
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
import pytest
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore
from app.gateway.auth_disabled import AUTH_SOURCE_INTERNAL
from app.gateway.routers.thread_runs import RunCreateRequest
from app.gateway.services import launch_scheduled_thread_run, start_run
from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config
from deerflow.persistence.thread_meta.memory import MemoryThreadMetaStore
from deerflow.runtime.events.store.memory import MemoryRunEventStore
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.store.memory import MemoryRunStore
@pytest.fixture(autouse=True)
def app_config():
set_app_config(AppConfig.model_validate({"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}}))
yield
reset_app_config()
@pytest.mark.asyncio
@pytest.mark.parametrize(
"auth_source, host_origin, expected",
[(None, None, "interactive"), (AUTH_SOURCE_INTERNAL, None, "unknown")] + [(AUTH_SOURCE_INTERNAL, origin, origin) for origin in ("interactive", "scheduled", "extension_evaluation", "unknown")],
)
async def test_client_metadata_cannot_forge_completed_evidence_origin(auth_source, host_origin, expected):
manager = RunManager(store=MemoryRunStore())
request = SimpleNamespace(
headers={},
state=SimpleNamespace(auth_source=auth_source),
app=SimpleNamespace(
state=SimpleNamespace(
run_manager=manager,
stream_bridge=SimpleNamespace(),
checkpointer=InMemorySaver(),
store=InMemoryStore(),
run_event_store=MemoryRunEventStore(),
run_events_config=None,
thread_store=MemoryThreadMetaStore(InMemoryStore()),
)
),
)
body = RunCreateRequest(
input={"messages": [{"role": "user", "content": "hello"}]},
metadata={"evidence_origin": "scheduled", "scheduled_task_id": "forged"},
config={"context": {"evidence_origin": "interactive"}, "configurable": {"evidence_origin": "scheduled"}},
context={"evidence_origin": "interactive"},
)
with (
patch("app.gateway.services.resolve_agent_factory", return_value=object()),
patch("app.gateway.services.run_agent", new=AsyncMock()),
patch.object(manager, "create_or_reject", wraps=manager.create_or_reject) as admission,
):
record = await start_run(body, "evidence-origin-test", request, evidence_origin=host_origin)
await record.task
assert admission.call_args.kwargs["evidence_origin"] == expected
@pytest.mark.asyncio
async def test_scheduler_passes_origin_outside_request_metadata():
admitted = AsyncMock(return_value=SimpleNamespace(run_id="r1", thread_id="t1"))
with patch("app.gateway.services.start_run", new=admitted):
await launch_scheduled_thread_run(thread_id="t1", assistant_id="lead_agent", prompt="hello", app=SimpleNamespace(), owner_user_id="owner")
assert admitted.call_args.kwargs["evidence_origin"] == "scheduled"
@pytest.mark.asyncio
async def test_evidence_agent_is_effective_canonical_id_not_routing_or_display_name():
manager = RunManager(store=MemoryRunStore())
request = SimpleNamespace(
headers={},
state=SimpleNamespace(),
app=SimpleNamespace(
state=SimpleNamespace(
run_manager=manager,
stream_bridge=SimpleNamespace(),
checkpointer=InMemorySaver(),
store=InMemoryStore(),
run_event_store=MemoryRunEventStore(),
run_events_config=None,
thread_store=MemoryThreadMetaStore(InMemoryStore()),
)
),
)
from deerflow.config.agents_config import AgentConfig
agent = AgentConfig(name="researcher", display_name="My research helper")
body = RunCreateRequest(assistant_id="lead_agent", input={"messages": [{"role": "user", "content": "hello"}]}, context={"agent_name": "researcher"}, metadata={"evidence_agent_id": "forged"})
with (
patch("app.gateway.services.resolve_agent_factory", return_value=object()),
patch("app.gateway.services._load_scope_agent_config", new=AsyncMock(return_value=agent)),
patch("app.gateway.services.run_agent", new=AsyncMock()),
patch.object(manager, "create_or_reject", wraps=manager.create_or_reject) as admission,
):
record = await start_run(body, "effective-agent-test", request)
await record.task
assert admission.call_args.kwargs["evidence_agent_id"] == "researcher"
@pytest.mark.asyncio
@pytest.mark.parametrize("backend", ["memory", "sqlite"])
async def test_keyed_resume_peer_reuse_preserves_evidence_identity(tmp_path, backend):
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from deerflow.persistence.base import Base
from deerflow.persistence.run.sql import RunRepository
engine = None
if backend == "sqlite":
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'admission.db'}")
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.create_all)
store = RunRepository(async_sessionmaker(engine, expire_on_commit=False))
else:
store = MemoryRunStore()
identity = {"version": 1, "kind": "resume", "sha256": "a" * 64}
try:
owner = RunManager(store=store)
original = await owner.create_or_reject(
"resume-thread",
"lead_agent",
user_id="alice",
idempotency_key="resume-key",
idempotency_request=identity,
evidence_origin="extension_evaluation",
evidence_agent_id="researcher",
)
peer = RunManager(store=store)
reused = await peer.create_or_reject(
"resume-thread",
"lead_agent",
user_id="alice",
idempotency_key="resume-key",
idempotency_request=identity,
evidence_origin="interactive",
evidence_agent_id="different-agent",
)
assert reused.run_id == original.run_id
assert reused.idempotency_reused
assert reused.store_only
assert reused.idempotency_request == identity
assert reused.evidence_origin == "extension_evaluation"
assert reused.evidence_agent_id == "researcher"
persisted = await store.get(original.run_id, user_id="alice")
assert persisted["idempotency_request"] == identity
assert persisted["evidence_origin"] == "extension_evaluation"
assert persisted["evidence_agent_id"] == "researcher"
finally:
if engine is not None:
await engine.dispose()

View File

@ -0,0 +1,895 @@
from __future__ import annotations
import hashlib
import json
from dataclasses import FrozenInstanceError, asdict
from datetime import UTC, datetime, timedelta
import pytest
import pytest_asyncio
from sqlalchemy import event, select, update
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from deerflow.persistence.base import Base
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.persistence.run.model import CompletedRunSnapshotRow, RunRow
from deerflow.persistence.run.sql import RunRepository
from deerflow.runtime.events.store.db import DbRunEventStore
from deerflow.runtime.events.store.memory import MemoryRunEventStore
from deerflow.runtime.journal import RunJournal
@pytest_asyncio.fixture
async def evidence(tmp_path):
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'evidence.db'}")
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
sf = async_sessionmaker(engine, expire_on_commit=False)
events = DbRunEventStore(sf)
runs = RunRepository(sf)
reader = HostCompletedRunEvidenceReader(runs, events, plugin_id="test", owner_ids=frozenset({"alice"}))
yield sf, events, runs, reader
await engine.dispose()
async def put_run(runs, run_id="r", *, owner="alice", status="success", **kwargs):
await runs.put(run_id, thread_id=f"t-{run_id}", user_id=owner, assistant_id="routing-alias", evidence_agent_id="stable-agent", status=status, **kwargs)
async def put_event(events, run_id="r", content="hello", owner="alice"):
return await events.put_batch([dict(thread_id=f"t-{run_id}", run_id=run_id, user_id=owner, event_type="test", category="message", content=content)])
@pytest.mark.asyncio
async def test_source_fence_metadata_has_explicit_accessors(evidence):
sf, _events, runs, reader = evidence
await put_run(runs)
snapshot = await reader.get_snapshot(thread_id="t-r", run_id="r")
async with sf() as session:
run = await session.scalar(select(RunRow).where(RunRow.run_id == "r"))
assert len(reader.scope_digest) == 64
assert reader.revision_for_run(run) == snapshot.evidence_revision
@pytest.mark.asyncio
async def test_new_giant_digest_has_constant_read_queries(evidence):
sf, events, runs, reader = evidence
await put_run(runs)
content = "x" * (16 * 1024 * 1024)
await put_event(events, content=content)
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
queries = []
engine = sf.kw["bind"].sync_engine
def track(_conn, _cursor, statement, _parameters, _context, _many):
queries.append(statement)
event.listen(engine, "before_cursor_execute", track)
try:
page = await reader.read_events(snapshot_ref=snap.snapshot_ref)
finally:
event.remove(engine, "before_cursor_execute", track)
assert page.items[0].content_sha256 == hashlib.sha256(content.encode()).hexdigest()
assert len(queries) <= 6
@pytest.mark.asyncio
@pytest.mark.parametrize("sizes", [[1024 * 1024 + 1], [600 * 1024, 600 * 1024]])
async def test_legacy_digest_budget_is_cumulative_and_never_trusts_metadata(evidence, sizes):
from deerflow_extension_api.host_capabilities import HostCapabilityError
sf, events, runs, reader = evidence
await put_run(runs)
async with sf() as session:
session.add_all([RunEventRow(thread_id="t-r", run_id="r", user_id="alice", event_type="test", category="message", content="x" * size, event_metadata={"content_sha256": "fake"}, seq=i + 1) for i, size in enumerate(sizes)])
await session.commit()
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
queries = []
engine = sf.kw["bind"].sync_engine
def track(_conn, _cursor, statement, _parameters, _context, _many):
queries.append(statement)
event.listen(engine, "before_cursor_execute", track)
try:
with pytest.raises(HostCapabilityError, match="LIMIT_EXCEEDED"):
await reader.read_events(snapshot_ref=snap.snapshot_ref)
finally:
event.remove(engine, "before_cursor_execute", track)
assert len(queries) <= 72
@pytest.mark.asyncio
async def test_legacy_digest_within_budget_is_exact(evidence):
sf, events, runs, reader = evidence
await put_run(runs)
content = "中\x00" * 10000
async with sf() as session:
session.add(RunEventRow(thread_id="t-r", run_id="r", user_id="alice", event_type="test", category="message", content=content, seq=1))
await session.commit()
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
from deerflow_extension_api.completed_run_evidence import EvidenceLimits
page = await reader.read_events(snapshot_ref=snap.snapshot_ref, limits=EvidenceLimits(max_bytes=1024))
assert page.items[0].content_sha256 == hashlib.sha256(content.encode()).hexdigest()
@pytest.mark.asyncio
async def test_legacy_digest_deadline_interrupts_query_wait(evidence, monkeypatch):
import asyncio
from deerflow_extension_api.host_capabilities import HostCapabilityError
from sqlalchemy.ext.asyncio import AsyncSession
from deerflow.extensions import completed_run_evidence as module
sf, events, runs, reader = evidence
await put_run(runs)
async with sf() as session:
session.add(RunEventRow(thread_id="t-r", run_id="r", user_id="alice", event_type="test", category="message", content="x" * 200000, seq=1))
await session.commit()
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
clock = iter([0, 1.999])
monkeypatch.setattr(module, "monotonic", lambda: next(clock))
original = AsyncSession.scalar
async def delayed(session, statement, *args, **kwargs):
if "substr(" in str(statement):
await asyncio.sleep(0.02)
return await original(session, statement, *args, **kwargs)
monkeypatch.setattr(AsyncSession, "scalar", delayed)
with pytest.raises(HostCapabilityError, match="LIMIT_EXCEEDED.*deadline"):
await reader.read_events(snapshot_ref=snap.snapshot_ref)
@pytest.mark.asyncio
@pytest.mark.parametrize("method", ["put", "put_batch", "put_if_absent"])
async def test_ingestion_digest_is_of_persisted_truncated_serialization(evidence, method):
sf, events, runs, reader = evidence
payload = dict(thread_id="t-r", run_id="r", event_type="test", category="trace", content={"text": "中" * 20000})
if method == "put_batch":
await events.put_batch([payload])
else:
await getattr(events, method)(**payload)
async with sf() as session:
row = await session.scalar(select(RunEventRow))
assert getattr(row, "content_sha256", None) == hashlib.sha256(row.content.encode("utf-8")).hexdigest()
@pytest.mark.asyncio
async def test_thread_snapshot_cleanup_locks_runs_in_prior_statement(evidence):
sf, events, runs, reader = evidence
await put_run(runs)
await reader.get_snapshot(thread_id="t-r", run_id="r")
statements = []
engine = sf.kw["bind"].sync_engine
def track(_conn, _cursor, statement, _parameters, _context, _many):
statements.append(statement)
event.listen(engine, "before_cursor_execute", track)
try:
await runs.delete_by_thread("t-r", user_id="alice")
finally:
event.remove(engine, "before_cursor_execute", track)
lock_indices = [i for i, statement in enumerate(statements) if statement.startswith("SELECT") and "runs.run_id" in statement]
assert lock_indices, "run locks must be acquired in their own statement"
cleanup_index = next(i for i, statement in enumerate(statements) if statement.startswith("DELETE FROM completed_run_snapshots"))
assert lock_indices[0] < cleanup_index
@pytest.mark.asyncio
@pytest.mark.parametrize("operation", ["run", "thread", "run_events", "thread_events"])
async def test_snapshot_cleanup_is_owner_scoped(evidence, operation):
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
sf, events, runs, reader = evidence
await put_run(runs)
await runs.put("b", thread_id="t-r", user_id="bob", status="success")
bob = HostCompletedRunEvidenceReader(runs, events, plugin_id="test", owner_ids={"bob"})
alice_snapshot = await reader.get_snapshot(thread_id="t-r", run_id="r")
bob_snapshot = await bob.get_snapshot(thread_id="t-r", run_id="b")
async def remove(owner):
if operation == "run":
await runs.delete("r", user_id=owner)
elif operation == "thread":
await runs.delete_by_thread("t-r", user_id=owner)
elif operation == "run_events":
await events.delete_by_run("t-r", "r", user_id=owner)
else:
await events.delete_by_thread("t-r", user_id=owner)
await remove("mallory")
async with sf() as session:
assert set(await session.scalars(select(CompletedRunSnapshotRow.snapshot_ref))) == {alice_snapshot.snapshot_ref, bob_snapshot.snapshot_ref}
await remove("alice")
async with sf() as session:
assert set(await session.scalars(select(CompletedRunSnapshotRow.snapshot_ref))) == {bob_snapshot.snapshot_ref}
def test_new_dtos_detach_mutable_constructor_sequences():
from deerflow_extension_api.completed_run_evidence import CompletedRunEventPage, CompletedRunSnapshot
notes = ["limited"]
items = []
snapshot = CompletedRunSnapshot(coverage_limits=notes, skill_observations=notes)
page = CompletedRunEventPage(items=items)
notes.append("changed")
items.append("changed")
assert snapshot.coverage_limits == ("limited",)
assert snapshot.skill_observations == ("limited",)
assert page.items == ()
@pytest.mark.asyncio
async def test_scope_filters_before_pagination_and_cursor_binds_plugin(evidence):
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
sf, events, runs, reader = evidence
await put_run(runs, "other", owner="bob")
await put_run(runs, "a")
await put_run(runs, "b")
first = await reader.list_changed_runs(cursor=None, limit=1)
assert [r.run_id for r in first.items] == ["a"]
assert first.has_more
await runs.update_model_name("a", "changed")
second = await reader.list_changed_runs(cursor=first.next_cursor, limit=10)
assert [r.run_id for r in second.items] == ["b", "a"]
peer = HostCompletedRunEvidenceReader(runs, events, plugin_id="other", owner_ids={"alice"})
with pytest.raises(HostCapabilityError, match="INVALID_CURSOR"):
await peer.list_changed_runs(cursor=first.next_cursor, limit=1)
with pytest.raises(HostCapabilityError, match="NOT_FOUND"):
await reader.get_snapshot(thread_id="t-other", run_id="other")
@pytest.mark.parametrize("owners", [None, set(), {None}, {"*"}, {""}])
def test_explicit_owner_scope_is_required(owners):
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
with pytest.raises(ValueError):
HostCompletedRunEvidenceReader(None, MemoryRunEventStore(), plugin_id="test", owner_ids=owners)
@pytest.mark.asyncio
async def test_unsupported_is_explicit():
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.extensions.completed_run_evidence import HostCompletedRunEvidenceReader
reader = HostCompletedRunEvidenceReader(None, MemoryRunEventStore(), plugin_id="test", owner_ids={"alice"})
with pytest.raises(HostCapabilityError, match="UNSUPPORTED"):
await reader.list_changed_runs(cursor=None, limit=1)
@pytest.mark.asyncio
async def test_snapshot_fixed_boundaries_and_retention_invalidation(evidence):
from deerflow_extension_api.host_capabilities import HostCapabilityError
sf, events, runs, reader = evidence
await put_run(runs)
await put_event(events, content={"text": "first"})
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
assert snap.owner_id == "alice"
assert snap.agent_id == "stable-agent"
assert snap.origin == "unknown"
assert snap.status == "success"
assert snap.seal_state == "partial"
assert snap.coverage == "lead-journal-v1"
assert snap.coverage_limits
assert snap.event_count == 1
assert (await reader.get_snapshot(thread_id="t-r", run_id="r")).snapshot_ref == snap.snapshot_ref
late = await put_event(events, content="late")
page = await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=None)
assert len(page.items) == 1
assert not page.has_more
assert json.loads(page.items[0].content_json) == {"text": "first"}
with pytest.raises(FrozenInstanceError):
snap.owner_id = "bob"
await events.delete_by_run("t-r", "r", user_id="alice")
# Reopen the store to ensure deletion preserved the durable sequence floor.
replacement = await put_event(DbRunEventStore(sf), content="replacement")
assert replacement[0]["seq"] > late[0]["seq"]
with pytest.raises(HostCapabilityError, match="NOT_FOUND"):
await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=None)
@pytest.mark.asyncio
async def test_every_page_reauthorizes_current_owner(evidence):
from deerflow_extension_api.host_capabilities import HostCapabilityError
sf, events, runs, reader = evidence
await put_run(runs)
await put_event(events)
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
async with sf() as session:
await session.execute(update(RunRow).where(RunRow.run_id == "r").values(user_id="bob"))
await session.commit()
with pytest.raises(HostCapabilityError, match="NOT_FOUND"):
await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=None)
@pytest.mark.asyncio
async def test_internal_resolver_rechecks_current_retention_without_reading_events(evidence):
from deerflow_extension_api.host_capabilities import HostCapabilityError
sf, events, runs, reader = evidence
await put_run(runs)
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
assert await reader.resolve_snapshot(snap.snapshot_ref) == snap
await events.delete_by_run("t-r", "r", user_id="alice")
with pytest.raises(HostCapabilityError, match="NOT_FOUND"):
await reader.resolve_snapshot(snap.snapshot_ref)
@pytest.mark.asyncio
async def test_giant_event_preview_fits_envelope_and_advances(evidence):
from deerflow_extension_api.completed_run_evidence import EvidenceLimits
sf, events, runs, reader = evidence
await put_run(runs)
await put_event(events, content="巨大" * 100000)
await put_event(events, content="last")
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
limits = EvidenceLimits(max_events=1, max_bytes=4096)
first = await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=None, limits=limits)
assert first.has_more
assert first.items[0].truncated
assert first.items[0].content_bytes == 600000
assert len(first.items[0].content_sha256) == 64
assert len(json.dumps(asdict(first), ensure_ascii=False, separators=(",", ":")).encode()) <= 4096
second = await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=first.next_cursor, limits=limits)
assert json.loads(second.items[0].content_json) == "last"
assert not second.has_more
@pytest.mark.asyncio
async def test_content_hash_and_body_preserve_embedded_null(evidence):
import hashlib
sf, events, runs, reader = evidence
await put_run(runs)
content = "before\x00after"
await put_event(events, content=content)
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
page = await reader.read_events(snapshot_ref=snap.snapshot_ref)
assert json.loads(page.items[0].content_json) == content
assert page.items[0].content_sha256 == hashlib.sha256(content.encode()).hexdigest()
@pytest.mark.asyncio
async def test_seal_requires_drain_and_current_owner(evidence):
sf, events, runs, reader = evidence
await put_run(runs, owner_worker_id="worker", lease_expires_at=(datetime.now(UTC) + timedelta(minutes=5)).isoformat())
journal = RunJournal("r", "t-r", events, user_id="alice")
journal.record_middleware("test", name="test", hook="test", action="test", changes={})
receipt = await journal.finish_evidence()
assert not journal._closed
journal.record_middleware("late", name="late", hook="late", action="late", changes={})
assert await runs.seal_evidence("r", owner_worker_id="wrong", receipt=receipt) is False
assert await runs.seal_evidence("r", owner_worker_id="worker", receipt=receipt) is True
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
assert snap.seal_state == "sealed"
assert snap.event_count == 1
assert snap.upper_event_seq == 1
await journal.close()
@pytest.mark.asyncio
async def test_proven_not_committed_flush_never_issues_seal_and_close_can_retry():
from deerflow.runtime.events.store.base import RunEventStoreNotCommittedError
class FailingStore(MemoryRunEventStore):
fail = True
async def put_batch(self, events):
if self.fail:
raise RunEventStoreNotCommittedError("write failed")
return await super().put_batch(events)
store = FailingStore()
journal = RunJournal("r", "t", store)
journal.record_middleware("test", name="test", hook="test", action="test", changes={})
with pytest.raises(RuntimeError, match="write failed"):
await journal.finish_evidence()
assert not journal._closed
assert journal._buffer
store.fail = False
await journal.close()
assert journal._closed
assert len(await store.list_events("t", "r")) == 1
@pytest.mark.asyncio
@pytest.mark.parametrize("status", ["success", "error", "interrupted", "timeout"])
async def test_legacy_terminal_run_is_partial_even_with_delivery(evidence, status):
sf, events, runs, reader = evidence
await put_run(runs, status=status)
async with sf() as session:
await session.execute(update(RunRow).where(RunRow.run_id == "r").values(evidence_seal_state=None))
await session.commit()
await events.put(thread_id="t-r", run_id="r", event_type="run.delivery", category="outputs", content={})
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
assert snap.status == status
assert snap.seal_state == "partial"
@pytest.mark.asyncio
async def test_admission_origin_survives_hydration_and_ignores_metadata(evidence):
from deerflow.runtime.runs.manager import RunManager
sf, events, runs, reader = evidence
manager = RunManager(store=runs)
first = await manager.create_or_reject("t", "stable", user_id="alice", evidence_origin="interactive", metadata={"evidence_origin": "scheduled"})
row = await runs.get(first.run_id, user_id="alice")
assert row["evidence_origin"] == "interactive"
assert manager._record_from_store(row).evidence_origin == "interactive"
with pytest.raises(ValueError, match="origin"):
await manager.create("bad", user_id="alice", evidence_origin="untrusted")
@pytest.mark.asyncio
async def test_expired_lease_cannot_seal(evidence):
sf, events, runs, reader = evidence
await put_run(runs, owner_worker_id="worker", lease_expires_at=(datetime.now(UTC) - timedelta(seconds=1)).isoformat())
journal = RunJournal("r", "t-r", events)
receipt = await journal.finish_evidence()
assert not await runs.seal_evidence("r", owner_worker_id="worker", receipt=receipt)
assert (await reader.get_snapshot(thread_id="t-r", run_id="r")).seal_state != "sealed"
await journal.close()
@pytest.mark.asyncio
@pytest.mark.parametrize("status", ["success", "error", "interrupted", "timeout"])
async def test_terminal_unsealed_crash_recovery_advances_discovery(evidence, status):
from deerflow.runtime.runs.manager import RunManager
sf, events, runs, reader = evidence
expired = (datetime.now(UTC) - timedelta(minutes=5)).isoformat()
await put_run(runs, "crash", status=status, owner_worker_id="dead", lease_expires_at=expired)
await put_run(runs, "live", owner_worker_id="live", lease_expires_at=(datetime.now(UTC) + timedelta(minutes=5)).isoformat())
before = await reader.list_changed_runs()
await RunManager(store=runs).reconcile_orphaned_inflight_runs(error="owner died")
row = await runs.get("crash", user_id="alice")
assert row["status"] == status
assert row["evidence_seal_state"] == "partial"
assert row["evidence_seal_error"]
assert (await runs.get("live", user_id="alice"))["evidence_seal_state"] == "pending"
changes = await reader.list_changed_runs(cursor=before.next_cursor)
assert [item.run_id for item in changes.items] == ["crash"]
@pytest.mark.asyncio
async def test_startup_recovers_more_than_one_evidence_batch(evidence):
from deerflow.runtime.runs.manager import RunManager
sf, events, runs, reader = evidence
async with sf() as session:
session.add_all([RunRow(run_id=f"crash-{i}", thread_id=f"t-{i}", user_id="alice", status="success", evidence_seal_state="pending") for i in range(201)])
await session.commit()
await RunManager(store=runs).reconcile_orphaned_inflight_runs(error="owner died")
changes = await reader.list_changed_runs(limit=500)
assert len(changes.items) == 201
async with sf() as session:
from sqlalchemy import func, select
assert await session.scalar(select(func.count()).select_from(RunRow).where(RunRow.evidence_seal_state == "pending")) == 0
@pytest.mark.asyncio
async def test_terminal_unsealed_recovery_keeps_live_local_finalization(evidence):
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.schemas import RunStatus
sf, events, runs, reader = evidence
manager = RunManager(store=runs)
record = await manager.create("t-local", user_id="alice")
await manager.set_status(record.run_id, RunStatus.success)
await manager.set_finalizing(record.run_id, True)
await put_run(runs, "crashed-without-lease")
await manager.reconcile_orphaned_inflight_runs(error="owner died")
assert (await runs.get(record.run_id, user_id="alice"))["evidence_seal_state"] == "pending"
assert (await runs.get("crashed-without-lease", user_id="alice"))["evidence_seal_state"] == "partial"
@pytest.mark.asyncio
async def test_thread_delete_invalidates_even_empty_snapshot(evidence):
from deerflow_extension_api.host_capabilities import HostCapabilityError
sf, events, runs, reader = evidence
await put_run(runs)
snap = await reader.get_snapshot(thread_id="t-r", run_id="r")
await events.delete_by_thread("t-r", user_id="alice")
with pytest.raises(HostCapabilityError, match="NOT_FOUND"):
await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=None)
@pytest.mark.asyncio
@pytest.mark.parametrize("outcome", ["success", "error", "interrupted"])
async def test_worker_seals_after_completion_hook_before_end(evidence, outcome):
from types import SimpleNamespace
from unittest.mock import AsyncMock
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.worker import RunContext, run_agent
sf, events, runs, reader = evidence
manager = RunManager(store=runs, event_store=events)
record = await manager.create("worker-thread", "stable-agent", user_id="alice")
captured = []
class Agent:
async def astream(self, graph_input, config=None, **kwargs):
captured.append(config["context"]["__run_journal"])
if outcome == "error":
raise RuntimeError("agent failure")
if outcome == "interrupted":
record.abort_event.set()
yield {"messages": []}
async def completion_hook(record):
captured[0].record_middleware("completion", name="completion", hook="stop", action="done", changes={})
async def end(run_id):
snap = await reader.get_snapshot(thread_id=record.thread_id, run_id=run_id)
assert snap.seal_state == "sealed"
page = await reader.read_events(snapshot_ref=snap.snapshot_ref, cursor=None)
assert "middleware:completion" in {e.event_type for e in page.items}
bridge = SimpleNamespace(publish=AsyncMock(), publish_end=end, cleanup=AsyncMock())
await run_agent(bridge, manager, record, ctx=RunContext(checkpointer=None, event_store=events, on_run_completed=completion_hook), agent_factory=lambda **kwargs: Agent(), graph_input={}, config={})
assert record.status.value == outcome
assert captured[0]._closed
@pytest.mark.asyncio
async def test_seal_failure_does_not_change_business_success(evidence, monkeypatch):
from types import SimpleNamespace
from unittest.mock import AsyncMock
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.worker import RunContext, run_agent
sf, events, runs, reader = evidence
manager = RunManager(store=runs, event_store=events)
record = await manager.create("worker-thread", "stable-agent", user_id="alice")
failure = AsyncMock(side_effect=RuntimeError("seal database unavailable"))
monkeypatch.setattr(runs, "seal_evidence", failure)
class Agent:
async def astream(self, *args, **kwargs):
yield {"messages": []}
bridge = SimpleNamespace(publish=AsyncMock(), publish_end=AsyncMock(), cleanup=AsyncMock())
await run_agent(bridge, manager, record, ctx=RunContext(checkpointer=None, event_store=events), agent_factory=lambda **kwargs: Agent(), graph_input={}, config={})
failure.assert_awaited_once()
assert record.status.value == "success"
bridge.publish_end.assert_awaited_once()
row = await runs.get(record.run_id, user_id="alice")
assert row["evidence_seal_state"] == "partial"
assert row["evidence_seal_error"]
@pytest.mark.asyncio
async def test_seal_cancellation_is_deferred_until_after_end(evidence, monkeypatch):
import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.worker import RunContext, run_agent
sf, events, runs, reader = evidence
manager = RunManager(store=runs, event_store=events)
record = await manager.create("cancel-seal", user_id="alice")
class Agent:
async def astream(self, *args, **kwargs):
yield {"messages": []}
monkeypatch.setattr(RunJournal, "finish_evidence", AsyncMock(side_effect=asyncio.CancelledError()))
bridge = SimpleNamespace(publish=AsyncMock(), publish_end=AsyncMock(), cleanup=AsyncMock())
with pytest.raises(asyncio.CancelledError):
await run_agent(bridge, manager, record, ctx=RunContext(checkpointer=None, event_store=events), agent_factory=lambda **kwargs: Agent(), graph_input={}, config={})
assert record.status.value == "success"
assert not record.finalizing
bridge.publish_end.assert_awaited_once()
row = await runs.get(record.run_id, user_id="alice")
assert row["evidence_seal_state"] == "partial"
@pytest.mark.asyncio
async def test_host_cancellation_drains_durable_evidence_seal(evidence, monkeypatch):
import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.worker import RunContext, run_agent
_sf, events, runs, reader = evidence
manager = RunManager(store=runs, event_store=events)
record = await manager.create("cancel-durable-seal", user_id="alice")
entered, release = asyncio.Event(), asyncio.Event()
original_seal = runs.seal_evidence
journals = []
async def gated_seal(*args, **kwargs):
entered.set()
await release.wait()
return await original_seal(*args, **kwargs)
seal = AsyncMock(side_effect=gated_seal)
monkeypatch.setattr(runs, "seal_evidence", seal)
class Agent:
async def astream(self, graph_input, config=None, **kwargs):
journals.append(config["context"]["__run_journal"])
yield {"messages": []}
async def aget_state(self, _config):
return SimpleNamespace(values={"messages": []})
bridge = SimpleNamespace(publish=AsyncMock(), publish_end=AsyncMock(), cleanup=AsyncMock())
worker = asyncio.create_task(run_agent(bridge, manager, record, ctx=RunContext(checkpointer=None, event_store=events), agent_factory=lambda **kwargs: Agent(), graph_input={}, config={}))
record.task = worker
try:
await asyncio.wait_for(entered.wait(), 5)
worker.cancel("first host cancellation")
await asyncio.sleep(0)
for _ in range(3):
worker.cancel("repeated host cancellation")
await asyncio.sleep(0)
assert not worker.done()
assert not journals[0]._closed
bridge.publish_end.assert_not_awaited()
release.set()
done, _pending = await asyncio.wait({worker}, timeout=5)
assert done, "worker did not finish after the seal write was released"
with pytest.raises(asyncio.CancelledError, match="first host cancellation"):
await worker
seal.assert_awaited_once()
snapshot = await reader.get_snapshot(thread_id=record.thread_id, run_id=record.run_id)
assert snapshot.seal_state == "sealed"
assert snapshot.status == "success"
assert journals[0]._closed
assert not record.finalizing
bridge.publish_end.assert_awaited_once()
finally:
release.set()
await asyncio.gather(worker, return_exceptions=True)
@pytest.mark.asyncio
async def test_graph_assembly_failure_has_partial_coverage(evidence):
from types import SimpleNamespace
from unittest.mock import AsyncMock
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.worker import RunContext, run_agent
sf, events, runs, reader = evidence
manager = RunManager(store=runs, event_store=events)
record = await manager.create("preflight", user_id="alice")
def fail(**kwargs):
raise RuntimeError("assembly failed")
bridge = SimpleNamespace(publish=AsyncMock(), publish_end=AsyncMock(), cleanup=AsyncMock())
await run_agent(bridge, manager, record, ctx=RunContext(checkpointer=None, event_store=events), agent_factory=fail, graph_input={}, config={})
snap = await reader.get_snapshot(thread_id=record.thread_id, run_id=record.run_id)
assert snap.status == "error"
assert snap.seal_state == "partial"
assert "execution" in snap.seal_error
def test_migration_is_additive_current_chain():
from alembic.script import ScriptDirectory
from deerflow.persistence.bootstrap import _MIGRATIONS_DIR
script = ScriptDirectory(str(_MIGRATIONS_DIR))
rev = script.get_revision("0045_completed_run_evidence")
assert rev.down_revision == "0044_channel_runtime_configs"
@pytest.mark.asyncio
async def test_cancelled_background_flush_cannot_silently_seal_an_empty_tail(evidence):
import asyncio
sf, events, runs, reader = evidence
await put_run(runs, owner_worker_id="worker")
original = events.put_batch
entered = asyncio.Event()
async def blocked(batch):
entered.set()
await asyncio.Event().wait()
events.put_batch = blocked
journal = RunJournal("r", "t-r", events, user_id="alice", flush_threshold=1)
journal.record_middleware("test", name="test", hook="test", action="test", changes={})
await entered.wait()
pending = tuple(journal._owned_writes)
for task in pending:
task.cancel()
await asyncio.gather(*pending, return_exceptions=True)
events.put_batch = original
with pytest.raises(RuntimeError, match="unknown durable outcome"):
await journal.finish_evidence()
assert (await reader.get_snapshot(thread_id="t-r", run_id="r")).seal_state != "sealed"
assert not await events.list_events("t-r", "r", user_id="alice")
await journal.close(flush=False)
@pytest.mark.asyncio
async def test_cancelled_flush_before_first_execution_cannot_seal_an_empty_tail(evidence):
import asyncio
sf, events, runs, reader = evidence
await put_run(runs)
journal = RunJournal("r", "t-r", events, user_id="alice", flush_threshold=1)
journal.record_middleware("test", name="test", hook="test", action="test", changes={})
pending = tuple(journal._owned_writes)
for task in pending:
task.cancel()
await asyncio.gather(*pending, return_exceptions=True)
with pytest.raises(RuntimeError, match="unknown durable outcome"):
await journal.finish_evidence()
assert (await reader.get_snapshot(thread_id="t-r", run_id="r")).seal_state != "sealed"
assert not await events.list_events("t-r", "r", user_id="alice")
await journal.close(flush=False)
@pytest.mark.asyncio
async def test_already_admitted_cross_thread_callback_survives_stop(evidence):
import threading
sf, events, runs, reader = evidence
await put_run(runs)
journal = RunJournal("r", "t-r", events, user_id="alice")
thread = threading.Thread(target=lambda: journal.record_middleware("accepted", name="accepted", hook="test", action="test", changes={}))
thread.start()
thread.join()
receipt = await journal.finish_evidence()
assert receipt.event_count == 1
assert not journal._closed
journal.record_middleware("rejected", name="rejected", hook="test", action="test", changes={})
await journal.close()
assert len(await events.list_events("t-r", "r", user_id="alice")) == 1
@pytest.mark.asyncio
async def test_lifecycle_transition_invalidates_pending_snapshot(evidence):
from deerflow_extension_api.host_capabilities import HostCapabilityError
sf, events, runs, reader = evidence
await put_run(runs, status="running")
pending = await reader.get_snapshot(thread_id="t-r", run_id="r")
await runs.update_status("r", "error")
terminal = await reader.get_snapshot(thread_id="t-r", run_id="r")
assert terminal.status == "error"
assert terminal.seal_state == "partial"
assert terminal.evidence_revision != pending.evidence_revision
with pytest.raises(HostCapabilityError, match="EVIDENCE_EXPIRED"):
await reader.read_events(snapshot_ref=pending.snapshot_ref)
def test_migration_upgrades_real_sqlite_and_preserves_legacy_partial(tmp_path):
from alembic import command
from sqlalchemy import create_engine, inspect, text
from deerflow.persistence.bootstrap import _get_alembic_config
engine = create_engine(f"sqlite:///{tmp_path / 'migration.db'}")
with engine.begin() as conn:
Base.metadata.create_all(conn)
conn.execute(RunRow.__table__.insert().values(run_id="old", thread_id="thread", user_id="alice", status="success"))
conn.execute(text("INSERT INTO channel_runtime_configs VALUES ('telegram', 'synthetic-ciphertext', 0, 7, 1.0)"))
conn.execute(text("DROP TABLE completed_run_snapshots"))
conn.execute(RunEventRow.__table__.insert().values(thread_id="thread", run_id="old", user_id="alice", event_type="test", category="message", content="legacy", seq=1))
conn.execute(text("ALTER TABLE run_events DROP COLUMN content_sha256"))
for name in ("evidence_origin", "evidence_agent_id", "evidence_seal_state", "evidence_seal_error", "evidence_revision", "evidence_upper_seq", "evidence_event_count", "evidence_retention_revision"):
conn.execute(text(f"ALTER TABLE runs DROP COLUMN {name}"))
config = _get_alembic_config(engine)
config.set_main_option("sqlalchemy.url", f"sqlite+aiosqlite:///{tmp_path / 'migration.db'}")
command.stamp(config, "0044_channel_runtime_configs")
command.upgrade(config, "0045_completed_run_evidence")
with engine.connect() as conn:
row = conn.execute(text("SELECT evidence_seal_state, evidence_revision, evidence_retention_revision FROM runs WHERE run_id='old'")).one()
assert tuple(row) == (None, None, 0)
assert conn.scalar(text("SELECT content_sha256 FROM run_events WHERE run_id='old'")) is None
assert "completed_run_snapshots" in inspect(conn).get_table_names()
assert conn.execute(text("SELECT config_ciphertext, version FROM channel_runtime_configs WHERE provider='telegram'")).one() == ("synthetic-ciphertext", 7)
engine.dispose()
@pytest.mark.asyncio
@pytest.mark.parametrize("limit", [0, -1, 2001, True, 1.5])
async def test_changed_run_page_rejects_invalid_limits(evidence, limit):
_sf, _events, runs, reader = evidence
await put_run(runs)
with pytest.raises(ValueError, match="max_events"):
await reader.list_changed_runs(limit=limit)
@pytest.mark.asyncio
async def test_changed_run_page_uses_validated_limit(evidence, monkeypatch):
from deerflow_extension_api import EvidenceLimits
_sf, _events, runs, reader = evidence
for run_id in ("a", "b", "c"):
await put_run(runs, run_id)
def bounded_limits(*, max_events):
return EvidenceLimits(max_events=min(max_events, 1))
# Exercise a tighter host policy without changing the public hard limit.
monkeypatch.setattr("deerflow.extensions.completed_run_evidence.EvidenceLimits", bounded_limits)
first = await reader.list_changed_runs(limit=100)
assert [row.run_id for row in first.items] == ["a"]
assert first.has_more
second = await reader.list_changed_runs(cursor=first.next_cursor, limit=100)
assert [row.run_id for row in second.items] == ["b"]
@pytest.mark.asyncio
async def test_normal_close_stops_evidence_producers_before_draining(evidence):
import asyncio
sf, events, runs, reader = evidence
await put_run(runs)
entered, release = asyncio.Event(), asyncio.Event()
original = events.put_batch
async def blocked(batch):
entered.set()
await release.wait()
return await original(batch)
events.put_batch = blocked
journal = RunJournal("r", "t-r", events, user_id="alice")
journal.record_middleware("accepted", name="accepted", hook="test", action="test", changes={})
closing = asyncio.create_task(journal.close())
try:
await entered.wait()
journal.record_middleware("late", name="late", hook="test", action="test", changes={})
finally:
release.set()
await closing
rows = await events.list_events("t-r", "r", user_id="alice")
assert len(rows) == 1
assert rows[0]["event_type"] == "middleware:accepted"
@pytest.mark.asyncio
async def test_commit_then_error_never_replays_or_issues_evidence_receipt(evidence):
sf, events, runs, reader = evidence
await put_run(runs)
original = events.put_batch
calls = 0
async def commit_then_error(batch):
nonlocal calls
calls += 1
await original(batch)
raise OSError("acknowledgement lost")
events.put_batch = commit_then_error
journal = RunJournal("r", "t-r", events, user_id="alice")
journal.record_middleware("accepted", name="accepted", hook="test", action="test", changes={})
try:
with pytest.raises(OSError, match="acknowledgement lost"):
await journal.finish_evidence()
with pytest.raises(RuntimeError, match="unknown durable outcome"):
await journal.finish_evidence()
assert calls == 1
assert len(await events.list_events("t-r", "r", user_id="alice")) == 1
assert (await reader.get_snapshot(thread_id="t-r", run_id="r")).seal_state != "sealed"
finally:
await journal.close(flush=False)

View File

@ -1,5 +1,6 @@
import asyncio
import weakref
from types import SimpleNamespace
import pytest
@ -7,17 +8,17 @@ from deerflow.runtime.events.store.db import DbRunEventStore
class _PausedDeleteSession:
"""Fake session that pauses the deletion on its first aggregate read.
"""Pause deletion on its first scalar (the retention change clock).
The fixed deletion path opens the session, takes the thread mutation fence
(a no-op on this fake's dialect) and only then reads the expected count, so
pausing in ``scalar`` proves the caller already owns the thread's mutation
critical section.
The in-process thread lock must already be held before any database work.
The database advisory fence is a no-op for this fake's SQLite dialect.
"""
def __init__(self, scalar_started: asyncio.Event, allow_scalar: asyncio.Event) -> None:
self._scalar_started = scalar_started
self._allow_scalar = allow_scalar
# Deletion now allocates the evidence-retention change sequence too.
self.bind = SimpleNamespace(dialect=SimpleNamespace(name="sqlite"))
async def __aenter__(self):
return self
@ -94,7 +95,7 @@ async def test_delete_waiter_handoff_keeps_one_write_lock_generation():
# Delete now owns the same generation, and the queued writer is still
# waiting behind it.
await scalar_started.wait()
await asyncio.wait_for(scalar_started.wait(), timeout=5)
assert not waiter_entered.is_set()
allow_scalar.set()

View File

@ -0,0 +1,166 @@
from types import SimpleNamespace
import pytest
from pydantic import ValidationError
from deerflow.extensions.loader import ExtensionSpec, load_extensions
def granted(**overrides):
return ExtensionSpec(
use="example:install",
name="evolution",
host_access={
"evidence": {"owners": ["owner"]},
"skill_mutations": {"owners": ["owner"], "operations": ["stage", "check"], "trigger_agents": ["agent"], "target_skills": ["example"], **overrides},
},
)
def test_host_access_is_default_deny_and_not_private_config():
plain = ExtensionSpec(use="example:install")
assert plain.host_access.evidence.owners == ()
assert plain.host_access.skill_mutations.operations == ()
spec = granted()
assert spec.config == {}
assert spec.host_access.skill_mutations.operations == ("stage", "check")
@pytest.mark.parametrize("key,value", [("owners", ["*"]), ("operations", ["create"]), ("trigger_agents", [""]), ("target_skills", ["../bad"]), ("topology", "multi_node")])
def test_invalid_grants_rejected(key, value):
with pytest.raises(ValidationError):
granted(**{key: value})
def test_host_access_requires_stable_name():
with pytest.raises(ValidationError):
ExtensionSpec(use="example:install", host_access={"evidence": {"owners": ["owner"]}})
def test_granted_source_cannot_be_ambiguous_even_with_other_ungranted_entry(monkeypatch):
calls = []
monkeypatch.setattr("deerflow.extensions.loader.resolve_variable", lambda _: calls.append(1))
with pytest.raises(ValueError, match="unique"):
load_extensions([granted(), ExtensionSpec(use="example:install", name="other")])
assert calls == []
def test_granted_name_cannot_be_ambiguous(monkeypatch):
monkeypatch.setattr("deerflow.extensions.loader.resolve_variable", lambda _: None)
with pytest.raises(ValueError, match="unique"):
load_extensions([granted(), ExtensionSpec(use="other:install", name="evolution")])
def test_binding_changes_with_entry_point_and_empty_scope_means_none():
from deerflow.extensions.host_access import bind_host_access
first = bind_host_access([granted()])["example:install"]
replacement = granted().model_copy(update={"use": "replacement:install"})
second = bind_host_access([replacement])["replacement:install"]
assert first.plugin_id != second.plugin_id
assert not first.allows("commit", "owner", "example")
assert first.allows("stage", "owner", "example")
assert not first.allows("stage", "another", "example")
empty = granted(owners=[])
assert not bind_host_access([empty])["example:install"].allows("stage", "owner", "example")
@pytest.mark.asyncio
async def test_services_receive_independently_bound_capabilities():
from deerflow.extensions.gateway import start_services
from deerflow.extensions.registry import ExtensionRegistry
received = []
class Service:
async def start(self, deps):
received.append(deps)
registry = ExtensionRegistry()
for source in ("first", "second"):
with registry.attributed_to(source):
registry.service(Service())
first, second = object(), object()
await start_services(registry.build(), SimpleNamespace(), None, host_capabilities={"first": (first, None), "second": (None, second)})
assert received[0].completed_run_evidence is first
assert received[0].skill_mutations is None
assert received[1].completed_run_evidence is None
assert received[1].skill_mutations is second
def test_new_host_contracts_are_available_without_harness_imports():
import deerflow_extension_api as api
for name in (
"HostCapabilityError",
"CompletedRunSnapshot",
"CompletedRunEvent",
"CompletedRunEventPage",
"CompletedRunEvidenceReader",
"EvidenceLimits",
"AssetRevision",
"SkillRevisionView",
"MutationCapabilities",
"Proposal",
"ProposalBundle",
"BundleFile",
"HostCheckResult",
"AssessmentRef",
"Operation",
"SkillMutationService",
):
assert name in api.__all__
assert getattr(api, name).__module__.startswith("deerflow_extension_api.")
def test_mutation_bundles_and_capabilities_detach_mutable_constructor_values():
from deerflow_extension_api import AssetRevision, BundleFile, MutationCapabilities, Proposal, ProposalBundle
content = bytearray(b"original")
files = [BundleFile("SKILL.md", content, "digest", False)]
revision = AssetRevision("incarnation", 1, "digest")
bundle = ProposalBundle("p", files, files, revision, "hash")
ops = ["stage"]
caps = MutationCapabilities(ops)
sources = ["s"]
proposal = Proposal("p", "o", "t", "skill", revision, "hash", "STAGED", "expiry", sources)
content[:] = b"tampered"
files.clear()
ops.append("commit")
sources.clear()
assert bundle.baseline[0].content == b"original"
assert caps.operations == ("stage",)
assert proposal.source_refs == ("s",)
@pytest.mark.asyncio
async def test_model_and_evolution_grants_compose_in_same_service(monkeypatch):
from deerflow.extensions.gateway import start_services, stop_services
received = []
class Service:
async def start(self, deps):
received.append(deps)
async def stop(self):
pass
def install(registry, config):
registry.service(Service())
monkeypatch.setattr("deerflow.extensions.loader.resolve_variable", lambda _: install)
access = granted().host_access.model_dump()
access["model_invocation"] = {"roles": {"default": "host-model"}}
spec = ExtensionSpec(use="example:install", name="evolution", host_access=access)
loaded, diagnostics = load_extensions([spec])
assert not diagnostics
evidence, mutations = object(), object()
try:
diagnostics = await start_services(loaded, SimpleNamespace(), None, host_capabilities={spec.use: (evidence, mutations)})
assert not diagnostics
assert received[0].model_invoker is not None
assert received[0].completed_run_evidence is evidence
assert received[0].skill_mutations is mutations
finally:
await stop_services(loaded)

View File

@ -52,6 +52,46 @@ def _patch_runtime_resources(monkeypatch, events: list[str]) -> None:
monkeypatch.setattr("deerflow.persistence.engine.get_session_factory", lambda: None)
@pytest.mark.asyncio
async def test_host_capabilities_outlive_extension_stop_and_close_before_database(monkeypatch):
from app.gateway.deps import langgraph_runtime
events = []
_patch_runtime_resources(monkeypatch, events)
evidence, mutations = object(), object()
class Host:
def __init__(self, *args):
self.bindings = {"example:install": (evidence, mutations)}
async def start(self):
events.append("host_start")
async def close(self):
events.append("host_close")
class Service:
async def start(self, deps):
assert deps.completed_run_evidence is evidence
assert deps.skill_mutations is mutations
events.append("service_start")
async def stop(self):
events.append("service_stop")
monkeypatch.setattr("deerflow.extensions.host_capabilities.HostCapabilities", Host)
monkeypatch.setattr("deerflow.persistence.thread_meta.make_thread_store", lambda *_: (_ for _ in ()).throw(RuntimeError("later failure")))
registry = ExtensionRegistry()
with registry.attributed_to("example:install"):
registry.service(Service())
app = FastAPI()
app.state.extensions = registry.build()
with pytest.raises(RuntimeError, match="later failure"):
async with langgraph_runtime(app, SimpleNamespace(database=_database_config())):
pytest.fail("must not yield")
assert events == ["host_start", "service_start", "service_stop", "host_close", "engine_close"]
@pytest.mark.asyncio
async def test_runtime_owns_engine_cleanup_before_initialization(monkeypatch):
from app.gateway.deps import langgraph_runtime

View File

@ -276,6 +276,9 @@ async def test_sql_runtime_shares_run_repository_with_scheduler(monkeypatch):
async def noop(*_args, **_kwargs):
return None
# This test asserts repository identity, not SQL startup/recovery. Its
# session factory is intentionally an opaque sentinel, not a usable DB.
monkeypatch.setattr("deerflow.extensions.host_capabilities.HostCapabilities.start", noop)
monkeypatch.setattr(engine_module, "init_engine_from_config", noop)
monkeypatch.setattr(engine_module, "get_session_factory", lambda: session_factory)
monkeypatch.setattr(engine_module, "close_engine", noop)

View File

@ -3643,7 +3643,7 @@ def test_launch_scheduled_thread_run_marks_context_non_interactive(_stub_app_con
async def _scenario():
captured: dict[str, object] = {}
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, scheduled_task_runtime=None):
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, evidence_origin=None, scheduled_task_runtime=None):
captured["body"] = body
captured["thread_id"] = thread_id
captured["context"] = body.context
@ -3705,7 +3705,7 @@ def test_launch_scheduled_thread_run_uses_configured_recursion_limit(_stub_app_c
async def _scenario():
captured: dict[str, object] = {}
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, scheduled_task_runtime=None):
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, evidence_origin=None, scheduled_task_runtime=None):
assert idempotency_key is None
captured["config"] = body.config
return SimpleNamespace(run_id="run-1", thread_id=thread_id)
@ -3747,7 +3747,7 @@ def test_launch_scheduled_thread_run_recursion_limit_is_clamped_to_ceiling(_stub
async def _scenario():
captured: dict[str, object] = {}
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, scheduled_task_runtime=None):
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, evidence_origin=None, scheduled_task_runtime=None):
assert idempotency_key is None
captured["config"] = body.config
return SimpleNamespace(run_id="run-1", thread_id=thread_id)
@ -3780,7 +3780,7 @@ def test_launch_scheduled_thread_run_falls_back_when_config_unloadable(_stub_app
async def _scenario():
captured: dict[str, object] = {}
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, scheduled_task_runtime=None):
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, evidence_origin=None, scheduled_task_runtime=None):
assert idempotency_key is None
captured["config"] = body.config
return SimpleNamespace(run_id="run-1", thread_id=thread_id)
@ -5416,7 +5416,7 @@ async def _capture_scheduled_launch(**kwargs):
captured: dict[str, object] = {}
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, scheduled_task_runtime=None):
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, evidence_origin=None, scheduled_task_runtime=None):
captured["body"] = body
captured["idempotency_key"] = idempotency_key
return SimpleNamespace(run_id="run-1", thread_id=thread_id)

View File

@ -189,7 +189,7 @@ def test_skills_writer_preserves_the_lifecycle_ledger(
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(cfg))
monkeypatch.setattr("app.gateway.routers.skills.reload_extensions_config", lambda *a, **k: None)
skills_router._write_extensions_skill_state(MagicMock(), "demo", False, rebuild_public_projection=False)
skills_router._write_extensions_skill_state(MagicMock(_app_config=None), "demo", False, rebuild_public_projection=False)
written = json.loads(cfg.read_text(encoding="utf-8"))
assert written["mcpLifecycle"] == ledger # preserved byte-for-value, never recomputed

View File

@ -2,7 +2,7 @@
Adds the nullable per-claim token columns ``McpTaskRepository`` uses to fence
poll, cancel, and notification mutations to the exact claim generation. The
chain-head pin moved on to ``test_migration_0027_0028_notification_outbox``.
chain-head pin lives in ``test_migration_0046_skill_mutations``.
"""
from __future__ import annotations

View File

@ -23,7 +23,7 @@ OUTBOX = "0027_notification_deliveries"
PARKED = "0028_parked_attempts"
SCHEDULER = "0029_scheduler_agent_tasks"
CLAIM_TOKENS = "0030_notification_claim_tokens"
HEAD = "0044_channel_runtime_configs"
HEAD = bootstrap._get_head_revision()
PREVIOUS = "0026_mcp_task_lease_tokens"
TABLE = "notification_deliveries"

View File

@ -23,7 +23,6 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
REVISION = "0029_scheduler_agent_tasks"
CURRENT_HEAD = "0044_channel_runtime_configs"
NEXT = "0030_notification_claim_tokens"
PREVIOUS = "0028_parked_attempts"
TASK_FIELDS = {"origin_thread_id", "goal_objective", "max_runs", "end_at", "standing_notes"}
@ -33,7 +32,8 @@ pytestmark = pytest.mark.asyncio
async def test_0029_remains_in_the_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision(REVISION).down_revision == PREVIOUS
assert script.get_revision(NEXT).down_revision == REVISION
assert len(REVISION) <= 32
@ -75,7 +75,7 @@ async def test_0029_preserves_legacy_fields_and_downgrades_unmet(tmp_path, migra
assert all(columns[table][field]["nullable"] and columns[table][field]["default"] is None for field in fields)
# Compare with the ORM at the current head: later revisions add
# scheduled_tasks columns (0031) that the model already declares.
await asyncio.to_thread(command.upgrade, cfg, CURRENT_HEAD)
await asyncio.to_thread(command.upgrade, cfg, "head")
async with engine.connect() as conn:
def scheduler_schema_diff(sync):
@ -130,7 +130,7 @@ async def test_0029_preserves_legacy_fields_and_downgrades_unmet(tmp_path, migra
assert (await conn.execute(sa.text("SELECT run_count FROM scheduled_tasks WHERE id='legacy'"))).scalar_one() == 7
# The ORM model carries later scheduled_tasks columns (0031), so read
# it back at the current head.
await asyncio.to_thread(command.upgrade, cfg, CURRENT_HEAD)
await asyncio.to_thread(command.upgrade, cfg, "head")
assert (await repo.get("new", user_id="owner"))["origin_thread_id"] is None
finally:
if schema:

View File

@ -24,7 +24,6 @@ IDEMPOTENCY = "0036_run_idempotency_request"
DOCUMENT_SUMMARIES = "0037_project_document_summaries"
CHANNEL_BINDINGS = "0038_channel_thread_bindings"
USER_DISABLED = "0039_user_disabled"
CURRENT_HEAD = "0044_channel_runtime_configs"
PREVIOUS = "0031_scheduled_streak_boundary"
TABLES = {"thread_read_markers", "thread_read_versions", "scheduled_task_events"}
INDEXES = {
@ -37,7 +36,8 @@ pytestmark = pytest.mark.asyncio
async def test_0032_remains_in_the_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision(REVISION).down_revision == PREVIOUS
assert script.get_revision(NEXT).down_revision == REVISION
assert script.get_revision(AFTER_NEXT).down_revision == NEXT

View File

@ -24,14 +24,14 @@ IDEMPOTENCY = "0036_run_idempotency_request"
DOCUMENT_SUMMARIES = "0037_project_document_summaries"
CHANNEL_BINDINGS = "0038_channel_thread_bindings"
USER_DISABLED = "0039_user_disabled"
CURRENT_HEAD = "0044_channel_runtime_configs"
TABLE = "run_event_thread_seq"
pytestmark = pytest.mark.asyncio
async def test_0034_remains_in_the_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision(REVISION).down_revision == PREVIOUS
assert script.get_revision(LOGIN_THROTTLE).down_revision == REVISION
assert script.get_revision(IDEMPOTENCY).down_revision == LOGIN_THROTTLE

View File

@ -25,7 +25,6 @@ IDEMPOTENCY = "0036_run_idempotency_request"
DOCUMENT_SUMMARIES = "0037_project_document_summaries"
CHANNEL_BINDINGS = "0038_channel_thread_bindings"
USER_DISABLED = "0039_user_disabled"
CURRENT_HEAD = "0044_channel_runtime_configs"
PREVIOUS = "0034_run_event_seq_watermark"
TABLE = "login_throttle"
COLUMNS = {"ip", "fail_count", "locked_at", "lock_duration_seconds", "updated_at"}
@ -33,9 +32,10 @@ INDEX = "ix_login_throttle_updated_at"
pytestmark = pytest.mark.asyncio
async def test_0035_remains_in_the_single_head_chain_after_0034():
async def test_0035_remains_in_the_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision(REVISION).down_revision == PREVIOUS
assert script.get_revision(IDEMPOTENCY).down_revision == REVISION
assert script.get_revision(DOCUMENT_SUMMARIES).down_revision == IDEMPOTENCY
@ -186,6 +186,6 @@ async def test_bootstrap_provisions_the_table_on_an_empty_database_and_is_idempo
assert set(shape["columns"]) == COLUMNS
assert INDEX in shape["indexes"]
async with engine.connect() as conn:
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == CURRENT_HEAD
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == bootstrap._get_head_revision()
finally:
await engine.dispose()

View File

@ -24,20 +24,21 @@ GROUP_NOTICE = "0040_scheduled_group_notice"
IM_NOTICE_PREFERENCES = "0041_im_notice_preferences"
CHANNEL_LEASES = "0042_channel_leases"
RUNTIME_IM_NOTICES = "0043_runtime_im_notices"
CURRENT_HEAD = "0044_channel_runtime_configs"
CHANNEL_RUNTIME_CONFIGS = "0044_channel_runtime_configs"
pytestmark = pytest.mark.asyncio
async def test_0037_chains_after_0036_and_is_single_head():
async def test_0037_chains_after_0036_in_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision(CHANNEL_BINDINGS).down_revision == REVISION
assert script.get_revision(USER_DISABLED).down_revision == CHANNEL_BINDINGS
assert script.get_revision(GROUP_NOTICE).down_revision == USER_DISABLED
assert script.get_revision(IM_NOTICE_PREFERENCES).down_revision == GROUP_NOTICE
assert script.get_revision(CHANNEL_LEASES).down_revision == IM_NOTICE_PREFERENCES
assert script.get_revision(RUNTIME_IM_NOTICES).down_revision == CHANNEL_LEASES
assert script.get_revision(CURRENT_HEAD).down_revision == RUNTIME_IM_NOTICES
assert script.get_revision(CHANNEL_RUNTIME_CONFIGS).down_revision == RUNTIME_IM_NOTICES
assert script.get_revision(REVISION).down_revision == PREVIOUS
# alembic_version.version_num is VARCHAR(32).
assert len(REVISION) <= 32

View File

@ -22,7 +22,6 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
REVISION = "0038_channel_thread_bindings"
PREVIOUS = "0037_project_document_summaries"
CURRENT_HEAD = "0044_channel_runtime_configs"
TABLE = "channel_thread_bindings"
COLUMNS = {"key", "channel_name", "chat_id", "topic_id", "thread_id", "user_id", "created_at", "updated_at"}
INDEX = "ix_channel_thread_bindings_channel_chat"
@ -31,7 +30,8 @@ pytestmark = pytest.mark.asyncio
async def test_0038_remains_in_the_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision(REVISION).down_revision == PREVIOUS
# alembic_version.version_num is VARCHAR(32).
assert len(REVISION) <= 32
@ -185,6 +185,6 @@ async def test_bootstrap_provisions_the_table_on_an_empty_database_and_is_idempo
async with engine.connect() as conn:
# Bootstrap runs the chain to its head, which this test no longer
# owns once later revisions chain after 0038.
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == CURRENT_HEAD
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == bootstrap._get_head_revision()
finally:
await engine.dispose()

View File

@ -16,13 +16,13 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
REVISION = "0040_scheduled_group_notice"
PREVIOUS = "0039_user_disabled"
CURRENT_HEAD = "0044_channel_runtime_configs"
COLUMN = "notification_group_connection_id"
def test_group_notification_revision_has_one_head_after_0039():
def test_group_notification_revision_is_in_single_chain_after_0039():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision("0041_im_notice_preferences").down_revision == REVISION
assert script.get_revision(REVISION).down_revision == PREVIOUS
assert len(REVISION) <= 32

View File

@ -23,7 +23,6 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
REVISION = "0042_channel_leases"
PREVIOUS = "0041_im_notice_preferences"
NEXT = "0043_runtime_im_notices"
CURRENT_HEAD = "0044_channel_runtime_configs"
TABLE = "channel_leases"
COLUMNS = {"provider", "owner_id", "epoch", "lease_expires_at", "transport_running", "updated_at"}
pytestmark = pytest.mark.asyncio
@ -31,8 +30,9 @@ pytestmark = pytest.mark.asyncio
async def test_0042_remains_in_the_single_migration_chain():
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_heads() == [CURRENT_HEAD]
assert script.get_revision(CURRENT_HEAD).down_revision == NEXT
assert len(script.get_heads()) == 1
assert REVISION in {revision.revision for revision in script.walk_revisions()}
assert script.get_revision("0044_channel_runtime_configs").down_revision == NEXT
assert script.get_revision(NEXT).down_revision == REVISION
assert script.get_revision(REVISION).down_revision == PREVIOUS
# alembic_version.version_num is VARCHAR(32).
@ -158,6 +158,6 @@ async def test_bootstrap_provisions_the_table_on_an_empty_database(tmp_path):
await bootstrap.bootstrap_schema(engine, backend="sqlite")
assert set((await _shape(engine))["columns"]) == COLUMNS
async with engine.connect() as conn:
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == CURRENT_HEAD
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == bootstrap._get_head_revision()
finally:
await engine.dispose()

View File

@ -22,7 +22,7 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
REVISION = "0044_channel_runtime_configs"
PREVIOUS = "0043_runtime_im_notices"
CURRENT_HEAD = REVISION
CURRENT_HEAD = bootstrap._get_head_revision()
TABLE = "channel_runtime_configs"
COLUMNS = {"provider", "config_ciphertext", "disconnected", "version", "updated_at"}
pytestmark = pytest.mark.asyncio

View File

@ -0,0 +1,56 @@
"""Additive Skill host schema and current migration-chain head."""
import asyncio
import pytest
import sqlalchemy as sa
from alembic import command
from alembic.script import ScriptDirectory
from sqlalchemy.ext.asyncio import create_async_engine
from deerflow.persistence import bootstrap
from deerflow.persistence.run.model import RunRow
REVISION = "0046_skill_mutations"
TABLES = {"skill_mutation_assets", "skill_mutation_owners", "skill_mutation_proposals", "skill_mutation_operations", "skill_mutation_scan_attempts"}
def test_0046_is_the_chain_head():
assert bootstrap._get_head_revision() == REVISION
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
assert script.get_revision(REVISION).down_revision == "0045_completed_run_evidence"
assert script.get_revision("0045_completed_run_evidence").down_revision == "0044_channel_runtime_configs"
assert script.get_revision("0044_channel_runtime_configs").down_revision == "0043_runtime_im_notices"
@pytest.mark.asyncio
async def test_0046_additive_upgrade_and_nullable_recovery_metadata(tmp_path):
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'skill-host.db'}")
config = bootstrap._get_alembic_config(engine)
try:
await asyncio.to_thread(bootstrap._upgrade, config, "0045_completed_run_evidence")
# The historical create_all bootstrap uses today's metadata. Remove
# exactly the new, empty tables to represent a real 0045 installation.
async with engine.begin() as connection:
for table in sorted(TABLES):
await connection.execute(sa.text(f'DROP TABLE IF EXISTS "{table}"'))
await connection.execute(RunRow.__table__.insert().values(run_id="legacy", thread_id="thread", status="success"))
await connection.execute(sa.text("INSERT INTO channel_runtime_configs VALUES ('telegram', 'synthetic-ciphertext', 0, 7, 1.0)"))
await asyncio.to_thread(bootstrap._upgrade, config, REVISION)
async with engine.connect() as connection:
names = await connection.run_sync(lambda sync: sa.inspect(sync).get_table_names())
assert TABLES <= set(names)
columns = await connection.run_sync(lambda sync: sa.inspect(sync).get_columns("skill_mutation_operations"))
nullable = {column["name"]: column["nullable"] for column in columns}
assert nullable["before_operation_id"]
assert nullable["superseded_by_generation"]
assert await connection.scalar(sa.text("SELECT status FROM runs WHERE run_id='legacy'")) == "success"
assert (await connection.execute(sa.text("SELECT config_ciphertext, version FROM channel_runtime_configs WHERE provider='telegram'"))).one() == ("synthetic-ciphertext", 7)
await asyncio.to_thread(command.downgrade, config, "0045_completed_run_evidence")
async with engine.connect() as connection:
names = await connection.run_sync(lambda sync: sa.inspect(sync).get_table_names())
assert not TABLES.intersection(names)
assert await connection.scalar(sa.text("SELECT status FROM runs WHERE run_id='legacy'")) == "success"
assert (await connection.execute(sa.text("SELECT config_ciphertext, version FROM channel_runtime_configs WHERE provider='telegram'"))).one() == ("synthetic-ciphertext", 7)
finally:
await engine.dispose()

View File

@ -646,6 +646,7 @@ class TestDbRunEventStore:
class FakeSession:
def __init__(self):
self.dialect = postgresql.dialect()
self.bind = self
self.execute_calls = []
def get_bind(self):
@ -675,8 +676,13 @@ class TestDbRunEventStore:
assert count == 3
assert session.execute_calls
assert "pg_advisory_xact_lock" in str(session.execute_calls[0][0])
assert session.execute_calls[0][1] == {"thread_id": "thread-1"}
# The shared changed-run clock precedes every run-row mutation.
assert "run_change_clock" in str(session.execute_calls[0][0])
assert "pg_advisory_xact_lock" in str(session.execute_calls[1][0])
assert session.execute_calls[1][1] == {"thread_id": "thread-1"}
assert "UPDATE runs" in str(session.execute_calls[2][0])
assert "DELETE FROM completed_run_snapshots" in str(session.execute_calls[3][0])
assert "DELETE FROM run_events" in str(session.execute_calls[4][0])
@pytest.mark.anyio
async def test_delete_by_run_takes_postgres_advisory_lock(self):
@ -688,6 +694,7 @@ class TestDbRunEventStore:
class FakeSession:
def __init__(self):
self.dialect = postgresql.dialect()
self.bind = self
self.execute_calls = []
def get_bind(self):
@ -723,8 +730,12 @@ class TestDbRunEventStore:
assert count == 2
assert session.execute_calls
assert "pg_advisory_xact_lock" in str(session.execute_calls[0][0])
assert session.execute_calls[0][1] == {"thread_id": "thread-1"}
assert "run_change_clock" in str(session.execute_calls[0][0])
assert "pg_advisory_xact_lock" in str(session.execute_calls[1][0])
assert session.execute_calls[1][1] == {"thread_id": "thread-1"}
assert "UPDATE runs" in str(session.execute_calls[2][0])
assert "DELETE FROM completed_run_snapshots" in str(session.execute_calls[3][0])
assert "DELETE FROM run_events" in str(session.execute_calls[4][0])
assert sum("pg_advisory_xact_lock" in str(call[0]) for call in session.execute_calls) == 1
@pytest.mark.anyio

View File

@ -1451,7 +1451,8 @@ async def test_terminal_cycle_collection_is_coalesced_contextless_and_off_loop(m
@pytest.mark.anyio
async def test_run_agent_releases_terminal_runtime_callbacks(caplog):
@pytest.mark.parametrize("fail_checkpoint_inspection", [False, True])
async def test_run_agent_releases_terminal_runtime_callbacks(caplog, fail_checkpoint_inspection):
from deerflow.runtime.journal import RunJournal
caplog.set_level(logging.DEBUG, logger="deerflow.runtime.runs.worker")
@ -1473,8 +1474,10 @@ async def test_run_agent_releases_terminal_runtime_callbacks(caplog):
yield {"messages": []}
async def aget_state(self, _config):
# Finalization reads checkpoint messages. A missing-method traceback
# retained by DEBUG log capture would itself keep the journal alive.
# A real checkpoint failure can retain the worker frame through a
# diagnostic traceback, just as a missing mock method can.
if fail_checkpoint_inspection:
raise RuntimeError("checkpoint inspection unavailable")
return SimpleNamespace(values={"messages": []})
await run_agent(
@ -1492,7 +1495,10 @@ async def test_run_agent_releases_terminal_runtime_callbacks(caplog):
stream_config = captured["config"]
journal = captured["journal"]
assert "Failed to inspect trailing human_input_request" not in caplog.text
if fail_checkpoint_inspection:
assert any(record.exc_info and "Failed to inspect trailing human_input_request" in record.message for record in caplog.records)
else:
assert "Failed to inspect trailing human_input_request" not in caplog.text
assert "__pregel_runtime" not in stream_config["configurable"]
assert "__run_journal" not in stream_config["context"]
assert journal not in (stream_config.get("callbacks") or [])

View File

@ -0,0 +1,401 @@
"""Existing storage writers participate in the same version chain as plugins."""
import threading
from contextlib import contextmanager
import pytest
from sqlalchemy import create_engine, select
from sqlalchemy.orm import sessionmaker
from deerflow.config.paths import Paths
from deerflow.persistence.base import Base
from deerflow.skills.mutations.guard import SkillMutationRuntime, configure_mutation_runtime
from deerflow.skills.mutations.repository import SkillMutationRepository
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
CONTENT = "---\nname: example\ndescription: Before\n---\nA\n"
@pytest.fixture
def assets(tmp_path, monkeypatch):
from deerflow.persistence.user.model import UserRow
monkeypatch.setattr("deerflow.config.paths._paths", Paths(base_dir=tmp_path / "home"))
storage = UserScopedSkillStorage("owner", host_path=str(tmp_path / "skills"))
root = storage.get_custom_skill_dir("example")
root.mkdir(parents=True)
(root / "SKILL.md").write_text(CONTENT, encoding="utf-8")
engine = create_engine(f"sqlite:///{tmp_path / 'mutations.db'}")
Base.metadata.create_all(engine)
sessions = sessionmaker(engine, expire_on_commit=False)
with sessions.begin() as session:
session.add(UserRow(id="owner", email="owner@example.test"))
repository = SkillMutationRepository(sessions)
runtime = SkillMutationRuntime(repository, owners=frozenset({"owner"}))
configure_mutation_runtime(runtime)
# This suite checks canonical versions; view semantics have their own suite.
monkeypatch.setattr("deerflow.skills.projection._rebuild_user_locked", lambda *_: None)
yield storage, runtime, repository
configure_mutation_runtime(None)
engine.dispose()
def test_managed_aba_conflicts_even_when_digest_matches(assets):
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
storage.write_custom_skill("example", "SKILL.md", CONTENT.replace("\nA\n", "\nB\n"))
storage.write_custom_skill("example", "SKILL.md", CONTENT)
last = runtime.read_revision(storage, "example")
assert last.content_digest == first.content_digest
assert last.incarnation_id == first.incarnation_id
assert last.mutation_seq == first.mutation_seq + 2
def test_no_change_does_not_increment_revision(assets):
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
storage.write_custom_skill("example", "SKILL.md", CONTENT)
assert runtime.read_revision(storage, "example") == first
def test_delete_recreate_changes_incarnation_and_retains_sequence(assets):
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
storage.delete_custom_skill("example")
storage.write_custom_skill("example", "SKILL.md", CONTENT)
last = runtime.read_revision(storage, "example")
assert last.incarnation_id != first.incarnation_id
assert last.mutation_seq > first.mutation_seq
def test_support_file_and_toggle_changes_invalidate_prior_revision(assets):
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
storage.write_custom_skill("example", "references/help.txt", "help")
second = runtime.read_revision(storage, "example")
assert second.mutation_seq == first.mutation_seq + 1
storage.set_skill_enabled_state("example", False)
third = runtime.read_revision(storage, "example")
assert third.mutation_seq == second.mutation_seq + 1
assert third.content_digest == second.content_digest
def test_other_skill_does_not_conflict(assets):
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
storage.write_custom_skill("other", "SKILL.md", CONTENT.replace("example", "other"))
assert runtime.read_revision(storage, "example") == first
def test_observed_external_edit_advances_revision(assets):
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
storage.get_custom_skill_file("example").write_text(CONTENT + "external", encoding="utf-8")
assert runtime.read_revision(storage, "example").mutation_seq == first.mutation_seq + 1
def test_catalog_read_uses_owner_guard(assets, monkeypatch):
from contextlib import contextmanager
storage, runtime, _ = assets
entered = []
@contextmanager
def guard(_storage, **_kwargs):
entered.append(True)
yield
monkeypatch.setattr("deerflow.skills.projection.skill_projection_read_lock", guard)
storage.load_skills()
assert entered
def test_owner_prompt_cache_never_hides_other_worker_change(assets):
from deerflow.agents.lead_agent.prompt import get_enabled_skills_for_config
from deerflow.config.app_config import AppConfig
storage, _, _ = assets
config = AppConfig.model_validate({"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, "skills": {"path": str(storage.get_skills_root_path())}})
first = get_enabled_skills_for_config(config, "owner")
storage.write_custom_skill("example", "SKILL.md", CONTENT.replace("Before", "After"))
second = get_enabled_skills_for_config(config, "owner")
assert next(skill.description for skill in first if skill.name == "example") == "Before"
assert next(skill.description for skill in second if skill.name == "example") == "After"
def test_global_disable_participates_in_effective_revision(assets, monkeypatch):
from deerflow.config.extensions_config import ExtensionsConfig
storage, runtime, _ = assets
first = runtime.read_revision(storage, "example")
monkeypatch.setattr(ExtensionsConfig, "from_file", lambda *a, **kw: ExtensionsConfig(skills={"example": {"enabled": False}}))
assert runtime.state(storage, "example")[1] is False
assert runtime.read_revision(storage, "example").mutation_seq > first.mutation_seq
def test_deleted_owner_cannot_read_catalog(assets):
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.persistence.user.model import UserRow
storage, _, repository = assets
with repository.sessions.begin() as session:
session.delete(session.get(UserRow, "owner"))
with pytest.raises(HostCapabilityError, match="NOT_FOUND_OR_FORBIDDEN"):
storage.load_skills()
def test_uncertain_legacy_write_blocks_catalog_until_revalidated(assets):
from deerflow_extension_api.host_capabilities import HostCapabilityError
storage, runtime, repository = assets
digest, enabled, exists = runtime.state(storage, "example")
repository.observe("owner", "example", digest, enabled, exists=exists, mark_mutating=True)
with pytest.raises(HostCapabilityError, match="NEEDS_REPAIR"):
storage.load_skills()
runtime.read_revision(storage, "example")
assert storage.load_skills()
@pytest.mark.parametrize("entry", ["shared", "thread", "rebuild", "slash"])
def test_every_new_asset_acquire_rejects_unresolved_owner(assets, monkeypatch, entry):
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.skills.projection import ensure_skill_projections, ensure_thread_skill_projection, rebuild_skill_projections
storage, _, repository = assets
def blocked(_):
raise HostCapabilityError("NEEDS_REPAIR")
monkeypatch.setattr(repository, "check_readable", blocked)
# Even an otherwise fresh thread projection must honor the DB barrier.
monkeypatch.setattr("deerflow.skills.projection._thread_projection_is_fresh", lambda *_: True)
with pytest.raises(HostCapabilityError, match="NEEDS_REPAIR"):
if entry == "shared":
ensure_skill_projections(storage)
elif entry == "thread":
ensure_thread_skill_projection(storage, "thread", {"example"})
elif entry == "rebuild":
rebuild_skill_projections(storage, include_public=False)
else:
from deerflow.agents.middlewares.skill_activation_middleware import SkillActivationMiddleware
SkillActivationMiddleware._read_skill_content(storage.get_custom_skill_file("example"), storage.get_skills_root_path(), storage=storage)
def test_uncertain_owner_blocks_other_managed_writer_before_touching_source(assets):
from deerflow_extension_api.host_capabilities import HostCapabilityError
storage, runtime, repository = assets
digest, enabled, exists = runtime.state(storage, "example")
repository.observe("owner", "example", digest, enabled, exists=exists, mark_mutating=True)
with pytest.raises(HostCapabilityError, match="NEEDS_REPAIR"):
storage.write_custom_skill("other", "SKILL.md", CONTENT.replace("example", "other"))
assert not storage.get_custom_skill_file("other").exists()
def test_global_toggle_aba_invalidates_same_named_custom_asset(assets, monkeypatch, tmp_path):
from app.gateway.routers.skills import _write_extensions_skill_state
from deerflow.config.extensions_config import reload_extensions_config, reset_extensions_config
storage, runtime, _ = assets
config_file = tmp_path / "extensions.json"
config_file.write_text('{"skills": {}}', encoding="utf-8")
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(config_file))
reload_extensions_config()
first = runtime.read_revision(storage, "example")
_write_extensions_skill_state(storage, "example", False, rebuild_public_projection=False)
_write_extensions_skill_state(storage, "example", True, rebuild_public_projection=False)
last = runtime.read_revision(storage, "example")
assert last.content_digest == first.content_digest
assert last.mutation_seq == first.mutation_seq + 2
reset_extensions_config()
def test_global_toggle_does_not_lock_unrelated_owner(assets, monkeypatch, tmp_path):
from app.gateway.routers.skills import _write_extensions_skill_state
from deerflow.config.extensions_config import ExtensionsConfig, reload_extensions_config, reset_extensions_config
from deerflow.persistence.user.model import UserRow
from deerflow.skills.projection import skill_projection_read_lock
storage, runtime, repository = assets
with repository.sessions.begin() as session:
session.add(UserRow(id="unrelated", email="unrelated@example.test"))
runtime.owners = frozenset({"owner", "unrelated"})
config_file = tmp_path / "extensions.json"
config_file.write_text('{"skills": {}}', encoding="utf-8")
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(config_file))
reload_extensions_config()
@contextmanager
def reject_unrelated(scoped, *args, **kwargs):
if scoped.user_id == "unrelated":
raise TimeoutError("unrelated owner is busy")
with skill_projection_read_lock(scoped, *args, **kwargs):
yield
monkeypatch.setattr("deerflow.skills.projection.skill_projection_read_lock", reject_unrelated)
_write_extensions_skill_state(storage, "example", False, rebuild_public_projection=False)
assert ExtensionsConfig.from_file().is_skill_enabled("example", "public") is False
reset_extensions_config()
def test_global_toggle_acquires_name_fence_before_public_projection(assets, monkeypatch):
from deerflow.skills import projection
from deerflow.skills.mutations import guard
storage, _runtime, _repository = assets
events = []
@contextmanager
def record_name_fence(*_args, **_kwargs):
events.append("name-enter")
try:
yield
finally:
events.append("name-exit")
@contextmanager
def record_projection_lock(*_args, **_kwargs):
events.append("projection-enter")
try:
yield
finally:
events.append("projection-exit")
monkeypatch.setattr(guard, "managed_name_writes", record_name_fence)
monkeypatch.setattr(projection, "_projection_lock", record_projection_lock)
with guard.managed_global_state_write(storage, "missing"):
pass
assert events == ["name-enter", "projection-enter", "projection-exit", "name-exit"]
def test_gateway_global_toggle_enters_managed_write_before_projection(assets, monkeypatch, tmp_path):
from app.gateway.routers.skills import _write_extensions_skill_state
from deerflow.config.extensions_config import reload_extensions_config, reset_extensions_config
from deerflow.skills import projection
from deerflow.skills.mutations import guard
storage, _runtime, _repository = assets
config_file = tmp_path / "extensions.json"
config_file.write_text('{"skills": {}}', encoding="utf-8")
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(config_file))
reload_extensions_config()
events = []
@contextmanager
def record_managed_write(*_args, **_kwargs):
events.append("managed-enter")
try:
yield
finally:
events.append("managed-exit")
@contextmanager
def record_projection(*_args, **_kwargs):
events.append("projection-enter")
try:
yield
finally:
events.append("projection-exit")
monkeypatch.setattr(guard, "managed_global_state_write", record_managed_write)
monkeypatch.setattr(projection, "skill_projection_mutation", record_projection)
try:
_write_extensions_skill_state(storage, "example", False, rebuild_public_projection=True)
finally:
reset_extensions_config()
assert events == ["managed-enter", "projection-enter", "projection-exit", "managed-exit"]
def test_global_toggle_serializes_same_name_creation(assets, monkeypatch, tmp_path):
from app.gateway.routers import skills as skills_router
from deerflow.config.extensions_config import reload_extensions_config, reset_extensions_config
from deerflow.persistence.skill_mutations.model import SkillAssetRow
from deerflow.persistence.user.model import UserRow
storage, runtime, repository = assets
with repository.sessions.begin() as session:
session.add(UserRow(id="new-owner", email="new-owner@example.test"))
runtime.owners = frozenset({"owner", "new-owner"})
other = UserScopedSkillStorage("new-owner", host_path=str(storage.get_skills_root_path()))
config_file = tmp_path / "extensions.json"
config_file.write_text('{"skills": {}}', encoding="utf-8")
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(config_file))
reload_extensions_config()
entered, release, writer_done = threading.Event(), threading.Event(), threading.Event()
errors = []
write_config = skills_router.atomic_write_extensions_config
def pause_config_write(path, raw):
entered.set()
if not release.wait(2):
raise TimeoutError("test did not release config write")
write_config(path, raw)
def toggle():
try:
skills_router._write_extensions_skill_state(storage, "example", False, rebuild_public_projection=False)
except BaseException as exc:
errors.append(exc)
def create_same_name():
try:
other.write_custom_skill("example", "SKILL.md", CONTENT)
except BaseException as exc:
errors.append(exc)
finally:
writer_done.set()
monkeypatch.setattr(skills_router, "atomic_write_extensions_config", pause_config_write)
toggle_thread = threading.Thread(target=toggle)
writer_thread = threading.Thread(target=create_same_name)
toggle_thread.start()
assert entered.wait(2)
writer_thread.start()
try:
serialized = not writer_done.wait(0.2)
finally:
release.set()
toggle_thread.join(2)
writer_thread.join(2)
assert not toggle_thread.is_alive()
assert not writer_thread.is_alive()
assert serialized
assert not errors
with repository.sessions() as session:
asset = session.scalar(select(SkillAssetRow).where(SkillAssetRow.owner_id == "new-owner", SkillAssetRow.name == "example"))
assert asset.enabled is False
reset_extensions_config()
def test_export_refuses_uncertain_canonical_state(assets):
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.skills.export import export_manifest
storage, runtime, repository = assets
digest, enabled, exists = runtime.state(storage, "example")
repository.observe("owner", "example", digest, enabled, exists=exists, mark_mutating=True)
with pytest.raises(HostCapabilityError, match="NEEDS_REPAIR"):
export_manifest(storage, "example")
@pytest.mark.asyncio
async def test_tool_candidate_copy_refuses_uncertain_canonical_state(assets):
from deerflow_extension_api.host_capabilities import HostCapabilityError
from deerflow.tools.skill_manage_tool import _scan_static_candidate_or_raise
storage, runtime, repository = assets
digest, enabled, exists = runtime.state(storage, "example")
repository.observe("owner", "example", digest, enabled, exists=exists, mark_mutating=True)
with pytest.raises(HostCapabilityError, match="NEEDS_REPAIR"):
await _scan_static_candidate_or_raise("example", {"help.txt": "help"}, storage)

View File

@ -0,0 +1,83 @@
"""Real SQLite, host assembly, evidence, projection and public-only consumer."""
import importlib.util
from pathlib import Path
from types import SimpleNamespace
import pytest
from deerflow_extension_api import ExtensionData, ExtensionRuntimeDeps, HostPolicySnapshot
from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from deerflow.config.database_config import DatabaseConfig
from deerflow.config.paths import Paths
from deerflow.extensions.host_capabilities import HostCapabilities
from deerflow.extensions.loader import ExtensionSpec
from deerflow.persistence.base import Base
from deerflow.persistence.run import RunRepository
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.user.model import UserRow
from deerflow.runtime.events.store.db import DbRunEventStore
from deerflow.skills.mutations.scanner import CandidateScanner, ScanVerdict
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
@pytest.mark.asyncio
async def test_public_consumer_end_to_end_and_plugin_removed_restart(tmp_path, monkeypatch):
monkeypatch.setattr("deerflow.config.paths._paths", Paths(base_dir=tmp_path / "home"))
storage = UserScopedSkillStorage("owner", host_path=str(tmp_path / "skills"))
root = storage.get_custom_skill_dir("example")
root.mkdir(parents=True)
content = "---\nname: example\ndescription: A sample skill\n---\nOriginal body.\n"
(root / "SKILL.md").write_text(content, encoding="utf-8")
database = DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))
engine = create_engine(database.app_sync_sqlalchemy_url)
Base.metadata.create_all(engine)
engine.dispose()
async_engine = create_async_engine(database.app_sqlalchemy_url)
sf = async_sessionmaker(async_engine, expire_on_commit=False)
async with sf.begin() as session:
session.add(UserRow(id="owner", email="owner@example.test"))
session.add(RunRow(run_id="run", thread_id="thread", user_id="owner", status="success", evidence_seal_state="sealed", evidence_agent_id="lead_agent", evidence_origin="interactive", evidence_upper_seq=0, evidence_event_count=0))
async def scan(self, package, name):
assert name == "example"
assert package.main_content.startswith("---")
return ScanVerdict("allow", "CHECK_PASSED", "test-policy")
async def policy(self):
return "test-policy"
monkeypatch.setattr(CandidateScanner, "scan", scan)
monkeypatch.setattr(CandidateScanner, "policy_version", policy)
monkeypatch.setattr(CandidateScanner, "policy_version_sync", lambda _: "test-policy")
spec = ExtensionSpec(
use="fixture:install",
name="fixture",
host_access={
"evidence": {"owners": ["owner"]},
"skill_mutations": {"owners": ["owner"], "operations": ["stage", "check", "commit", "revert"], "trigger_agents": ["lead_agent"], "target_skills": ["example"], "topology": "single_host_local"},
},
)
config = SimpleNamespace(database=database, plugins=[spec])
host = HostCapabilities(config, RunRepository(sf), DbRunEventStore(sf), root=tmp_path, storage_factory=lambda _: storage)
try:
await host.start()
evidence, mutations = host.bindings[spec.use]
deps = ExtensionRuntimeDeps(app_store=ExtensionData("fixture"), policy=HostPolicySnapshot(), completed_run_evidence=evidence, skill_mutations=mutations)
module_spec = importlib.util.spec_from_file_location("external_consumer", Path(__file__).parent / "fixtures" / "skill_evolution_consumer.py")
consumer = importlib.util.module_from_spec(module_spec)
module_spec.loader.exec_module(consumer)
operation, reverted = await consumer.exercise(deps, thread_id="thread", run_id="run", name="example")
assert (operation.views, reverted.views) == ("READY", "READY")
assert (root / "SKILL.md").read_text(encoding="utf-8") == content
await host.close()
config.plugins = []
host = HostCapabilities(config, RunRepository(sf), DbRunEventStore(sf), root=tmp_path, storage_factory=lambda _: storage)
await host.start()
assert host.bindings == {}
assert "owner" in host.runtime.owners
assert host.recovery.get_operation(reverted.operation_id).publication == "APPLIED"
finally:
await host.close()
await async_engine.dispose()

View File

@ -319,7 +319,7 @@ def test_temp_creation_and_cleanup_within_mutation(tmp_path, monkeypatch, user_s
active = False
@contextmanager
def locked():
def locked(*, names=()):
nonlocal active
assert not root.exists()
active = True

View File

@ -0,0 +1,144 @@
"""Recovery is an authenticated host operation, never a plugin route."""
from dataclasses import dataclass
from types import SimpleNamespace
import httpx
import pytest
from fastapi import FastAPI, Request
from deerflow.skills.mutations.workers import MutationWorkers
@pytest.mark.asyncio
@pytest.mark.parametrize("role,source,status", [("user", None, 403), ("admin", "pat", 403), ("admin", None, 200)])
async def test_admin_recovery_auth_and_off_loop(role, source, status):
import threading
from app.gateway.routers.skill_mutations import router
loop_thread = threading.get_ident()
calls = []
@dataclass
class Result:
operation_id: str
publication: str = "APPLIED"
class Recovery:
def list_operations(self, *, limit, after_id):
assert threading.get_ident() != loop_thread
assert limit == 51 and after_id is None
calls.append("list")
return (Result("a" * 32),)
def get_operation(self, operation_id):
assert threading.get_ident() != loop_thread
calls.append("query")
return Result(operation_id)
def recover_operation(self, operation_id):
assert threading.get_ident() != loop_thread
calls.append("recover")
return Result(operation_id)
app = FastAPI()
workers = MutationWorkers()
app.state.skill_mutation_host = SimpleNamespace(workers=workers, recovery=Recovery())
@app.middleware("http")
async def identity(request: Request, call_next):
request.state.user = SimpleNamespace(id="admin", system_role=role)
request.state.auth_source = source
return await call_next(request)
app.include_router(router)
try:
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as client:
base = "/api/skill-mutations/operations/" + "a" * 32
listed = await client.get("/api/skill-mutations/operations")
assert listed.status_code == status
if status == 200:
assert listed.json()["items"][0]["operation_id"] == "a" * 32
assert (await client.get(base)).status_code == status
assert (await client.post(base + "/recover")).status_code == status
assert calls == (["list", "query", "recover"] if status == 200 else [])
finally:
await workers.close()
@pytest.mark.asyncio
async def test_admin_errors_do_not_return_database_or_candidate_content():
from deerflow_extension_api import HostCapabilityError
from app.gateway.routers.skill_mutations import router
class Recovery:
def get_operation(self, operation_id):
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN", "candidate content must not leak")
app = FastAPI()
workers = MutationWorkers()
app.state.skill_mutation_host = SimpleNamespace(workers=workers, recovery=Recovery())
@app.middleware("http")
async def identity(request: Request, call_next):
request.state.user = SimpleNamespace(id="admin", system_role="admin")
return await call_next(request)
app.include_router(router)
try:
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as client:
response = await client.get("/api/skill-mutations/operations/" + "a" * 32)
assert response.status_code == 404
assert "candidate content" not in response.text
finally:
await workers.close()
@pytest.mark.asyncio
@pytest.mark.parametrize("count,limit", [(0, 2), (1, 2), (2, 2), (3, 2), (3, 1)])
async def test_operation_pagination_terminates_with_null_cursor(count, limit):
from app.gateway.routers.skill_mutations import router
@dataclass
class Result:
operation_id: str
operations = [Result(f"{number:032x}") for number in range(1, count + 1)]
class Recovery:
def list_operations(self, *, limit, after_id):
return tuple(row for row in operations if after_id is None or row.operation_id > after_id)[:limit]
app = FastAPI()
workers = MutationWorkers()
app.state.skill_mutation_host = SimpleNamespace(workers=workers, recovery=Recovery())
@app.middleware("http")
async def identity(request: Request, call_next):
request.state.user = SimpleNamespace(id="admin", system_role="admin")
return await call_next(request)
app.include_router(router)
try:
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as client:
params = {"limit": limit}
seen = []
for _ in range(count + 1):
response = await client.get("/api/skill-mutations/operations", params=params)
assert response.status_code == 200
page = response.json()
seen.extend(row["operation_id"] for row in page["items"])
assert page["has_more"] == (page["next_cursor"] is not None)
if page["next_cursor"] is None:
break
params["after_id"] = page["next_cursor"]
else:
pytest.fail("Pagination failed to terminate")
assert seen == [row.operation_id for row in operations]
# A stale end cursor must not echo itself back forever.
empty = await client.get("/api/skill-mutations/operations", params={"after_id": "f" * 32, "limit": limit})
assert empty.json() == {"items": [], "has_more": False, "next_cursor": None}
finally:
await workers.close()

View File

@ -0,0 +1,20 @@
import threading
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
@pytest.mark.asyncio
async def test_gateway_catalog_never_takes_managed_guard_on_event_loop(monkeypatch):
from app.gateway.routers import skills
loop_thread = threading.get_ident()
def load_skills(**kwargs):
assert threading.get_ident() != loop_thread
return []
monkeypatch.setattr(skills, "_get_user_skill_storage", lambda _: SimpleNamespace(load_skills=load_skills))
monkeypatch.setattr(skills, "_filter_visible_skills", AsyncMock(return_value=[]))
assert (await skills.list_skills(SimpleNamespace(), SimpleNamespace())).skills == []

View File

@ -0,0 +1,66 @@
"""The same owner lock protects exports, old writers and new publication."""
import os
import subprocess
import sys
import threading
import pytest
from deerflow.skills.projection import _projection_lock, skill_projection_read_lock
from deerflow.skills.storage.local_skill_storage import LocalSkillStorage
def test_read_guard_is_reentrant_inside_existing_writer(tmp_path):
storage = LocalSkillStorage(host_path=str(tmp_path))
with _projection_lock(tmp_path / "custom"):
with skill_projection_read_lock(storage, timeout=0.05):
pass
def test_guard_wait_is_bounded_across_threads(tmp_path):
storage = LocalSkillStorage(host_path=str(tmp_path))
held, release = threading.Event(), threading.Event()
def holder():
with _projection_lock(tmp_path / "custom"):
held.set()
release.wait(5)
thread = threading.Thread(target=holder)
thread.start()
try:
assert held.wait(2)
with pytest.raises(TimeoutError):
with skill_projection_read_lock(storage, timeout=0.05):
pytest.fail("Concurrent writer lock was bypassed")
finally:
release.set()
thread.join(2)
@pytest.mark.skipif(os.name != "posix", reason="POSIX publication locks")
def test_another_process_cannot_bypass_owner_guard_and_kill_releases_it(tmp_path):
script = """
import sys
from pathlib import Path
from deerflow.skills.projection import _projection_lock
with _projection_lock(Path(sys.argv[1]) / 'custom'):
print('locked', flush=True)
sys.stdin.read()
"""
child = subprocess.Popen([sys.executable, "-c", script, str(tmp_path)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
try:
assert child.stdout.readline().strip() == "locked"
storage = LocalSkillStorage(host_path=str(tmp_path))
with pytest.raises(TimeoutError):
with skill_projection_read_lock(storage, timeout=0.05):
pytest.fail("process bypassed owner lock")
child.kill()
child.wait(timeout=5)
with skill_projection_read_lock(storage, timeout=0.5):
pass
finally:
if child.poll() is None:
child.kill()
child.communicate(timeout=5)

View File

@ -0,0 +1,312 @@
"""The host owns recovery even when every extension is disabled."""
import asyncio
import threading
from contextlib import contextmanager
from types import SimpleNamespace
import pytest
import pytest_asyncio
from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from sqlalchemy.orm import Session
from deerflow.config.database_config import DatabaseConfig
from deerflow.config.run_events_config import RunEventsConfig
from deerflow.extensions.host_access import HostAccess
from deerflow.extensions.loader import ExtensionSpec
from deerflow.persistence.base import Base
from deerflow.persistence.run import RunRepository
from deerflow.persistence.skill_mutations.model import SkillOwnerRow, SkillProposalRow
from deerflow.persistence.user.model import UserRow
from deerflow.runtime.events.store.db import DbRunEventStore
@pytest_asyncio.fixture
async def durable(tmp_path):
database = DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))
sync = create_engine(database.app_sync_sqlalchemy_url)
Base.metadata.create_all(sync)
with Session(sync) as session, session.begin():
session.add(UserRow(id="owner", email="owner@example.test"))
sync.dispose()
engine = create_async_engine(database.app_sqlalchemy_url)
sf = async_sessionmaker(engine, expire_on_commit=False)
yield database, sf
await engine.dispose()
def config(database, plugins=()):
return SimpleNamespace(database=database, run_events=RunEventsConfig(backend="db"), plugins=plugins)
def granted():
return ExtensionSpec(
use="example:install",
name="evolution",
host_access=HostAccess.model_validate(
{
"evidence": {"owners": ["owner"]},
"skill_mutations": {"owners": ["owner"], "operations": ["stage", "check", "commit", "revert"], "trigger_agents": ["lead_agent"], "target_skills": ["example"], "topology": "single_host_local"},
}
),
)
@pytest.mark.asyncio
async def test_default_has_no_mutation_runtime_or_worker_pool(durable, tmp_path):
from deerflow.extensions.host_capabilities import HostCapabilities
database, sf = durable
host = HostCapabilities(config(database), RunRepository(sf), DbRunEventStore(sf), root=tmp_path)
await host.start()
try:
assert host.bindings == {}
assert host.runtime is None
assert host.recovery is None
finally:
await host.close()
@pytest.mark.asyncio
async def test_disabled_plugin_still_recovers_enrolled_owners(durable, tmp_path):
from deerflow.extensions.host_capabilities import HostCapabilities
database, sf = durable
async with sf.begin() as session:
session.add(SkillOwnerRow(owner_id="owner", generation=4, deleting=False))
events = []
class Recovery:
def __init__(self, runtime, storage_factory):
assert runtime.owners == frozenset({"owner"})
events.append("constructed")
def recover_all(self):
events.append("recovered")
def collect_garbage(self):
events.append("collected")
host = HostCapabilities(config(database), RunRepository(sf), DbRunEventStore(sf), root=tmp_path, recovery_factory=Recovery)
await host.start()
try:
assert host.bindings == {}
assert events == ["constructed", "recovered", "collected"]
assert host.runtime.owners == frozenset({"owner"})
finally:
await host.close()
assert host.workers._closed
@pytest.mark.asyncio
async def test_source_binding_and_independent_evidence_scope(durable, tmp_path):
from deerflow.extensions.host_capabilities import HostCapabilities
database, sf = durable
seen = []
class Recovery:
def __init__(self, *args):
pass
def recover_all(self):
pass
def collect_garbage(self):
pass
def service(binding, **kwargs):
seen.append((binding, kwargs))
return object()
host = HostCapabilities(config(database, [granted()]), RunRepository(sf), DbRunEventStore(sf), root=tmp_path, recovery_factory=Recovery, service_factory=service)
await host.start()
try:
reader, mutations = host.bindings["example:install"]
assert reader is seen[0][1]["evidence"]
assert reader._owners == frozenset({"owner"})
assert mutations is not None
assert seen[0][0].plugin_id != "evolution"
finally:
await host.close()
@pytest.mark.asyncio
async def test_missing_topology_refuses_mutation_startup(durable, tmp_path):
from deerflow_extension_api import HostCapabilityError
from deerflow.extensions.host_capabilities import HostCapabilities
database, sf = durable
spec = granted()
spec = spec.model_copy(update={"host_access": spec.host_access.model_copy(update={"skill_mutations": spec.host_access.skill_mutations.model_copy(update={"topology": None})})})
host = HostCapabilities(config(database, [spec]), RunRepository(sf), DbRunEventStore(sf), root=tmp_path)
try:
with pytest.raises(HostCapabilityError, match="UNSUPPORTED_TOPOLOGY"):
await host.start()
finally:
await host.close()
@pytest.mark.asyncio
async def test_cancelled_startup_drains_worker_before_releasing_topology(durable, tmp_path):
from deerflow.extensions.host_capabilities import HostCapabilities
from deerflow.skills.mutations.topology import PublicationTopology
database, sf = durable
entered, release = threading.Event(), threading.Event()
class Recovery:
def __init__(self, *args):
pass
def recover_all(self):
entered.set()
assert release.wait(5)
def collect_garbage(self):
pass
host = HostCapabilities(config(database, [granted()]), RunRepository(sf), DbRunEventStore(sf), root=tmp_path, recovery_factory=Recovery)
task = asyncio.create_task(host.start())
assert await asyncio.to_thread(entered.wait, 5)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
closing = asyncio.create_task(host.close())
await asyncio.sleep(0)
assert not closing.done()
assert host._topology._fd is not None
release.set()
await closing
next_host = PublicationTopology("sqlite", tmp_path)
next_host.acquire()
next_host.close()
@pytest.mark.asyncio
async def test_read_only_evidence_reports_unsupported_without_mutation_pool(tmp_path):
from deerflow_extension_api import HostCapabilityError
from deerflow.extensions.host_capabilities import HostCapabilities
from deerflow.runtime.events.store.memory import MemoryRunEventStore
from deerflow.runtime.runs.store.memory import MemoryRunStore
spec = ExtensionSpec(use="example:install", name="reader", host_access=HostAccess.model_validate({"evidence": {"owners": ["owner"]}}))
host = HostCapabilities(config(DatabaseConfig(), [spec]), MemoryRunStore(), MemoryRunEventStore(), root=tmp_path)
await host.start()
try:
reader, mutations = host.bindings["example:install"]
assert mutations is None
assert host.workers is None
with pytest.raises(HostCapabilityError, match="UNSUPPORTED"):
await reader.list_changed_runs()
finally:
await host.close()
@pytest.mark.asyncio
async def test_mutation_owner_without_evidence_grant_is_rejected(durable, tmp_path):
from deerflow_extension_api import HostCapabilityError
from deerflow.extensions.host_capabilities import HostCapabilities
database, sf = durable
spec = granted()
spec.host_access = spec.host_access.model_copy(update={"evidence": HostAccess().evidence})
host = HostCapabilities(config(database, [spec]), RunRepository(sf), DbRunEventStore(sf), root=tmp_path)
try:
with pytest.raises(HostCapabilityError, match="INVALID_GRANT"):
await host.start()
assert host.workers is None
finally:
await host.close()
@pytest.mark.asyncio
@pytest.mark.parametrize("phase", ["startup", "maintenance"])
@pytest.mark.parametrize("failure", ["lock-directory", "lock-timeout"])
async def test_owner_gc_failure_does_not_block_host_or_healthy_owner(durable, tmp_path, monkeypatch, phase, failure):
from deerflow.config.paths import Paths
from deerflow.extensions.host_capabilities import HostCapabilities
from deerflow.skills.mutations import recovery as recovery_module
from deerflow.skills.projection import get_skill_projection_paths
from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
database, sf = durable
monkeypatch.setattr("deerflow.config.paths._paths", Paths(base_dir=tmp_path / "home"))
owners = ("a-broken", "z-healthy")
async with sf.begin() as session:
for owner_id in owners:
session.add(UserRow(id=owner_id, email=f"{owner_id}@example.test"))
session.add(SkillOwnerRow(owner_id=owner_id, generation=0, deleting=False))
def storage_factory(owner_id):
return UserScopedSkillStorage(owner_id, host_path=str(tmp_path / "skills"))
async def expire_proposals():
async with sf.begin() as session:
for owner_id in owners:
session.add(
SkillProposalRow(
proposal_id=owner_id,
plugin_id="removed-plugin",
owner_id=owner_id,
target_ref=owner_id,
name="example",
base_revision={},
sources=[],
candidate_hash="unused",
package_blob=b"baseline",
candidate_blob=b"candidate",
state="PENDING",
expires_at=0,
idempotency_key=owner_id,
request_hash="unused",
)
)
def break_owner_lock():
if failure == "lock-directory":
root = get_skill_projection_paths(storage_factory(owners[0])).custom.parent
lock_path = root.parent / f".{root.name}.projection.lock"
# During maintenance an earlier successful acquire made a file.
if lock_path.is_file():
lock_path.unlink()
lock_path.mkdir(parents=True)
else:
original_lock = recovery_module.skill_projection_read_lock
@contextmanager
def blocked_lock(storage, **kwargs):
if storage.user_id == owners[0]:
raise TimeoutError("owner lock unavailable")
with original_lock(storage, **kwargs):
yield
monkeypatch.setattr(recovery_module, "skill_projection_read_lock", blocked_lock)
# No enabled plugin: persisted host state alone must remain recoverable.
host = HostCapabilities(config(database), RunRepository(sf), DbRunEventStore(sf), root=tmp_path, storage_factory=storage_factory)
try:
if phase == "startup":
await expire_proposals()
break_owner_lock()
await host.start()
if phase == "maintenance":
await expire_proposals()
break_owner_lock()
await host.workers.run(lambda _: (host.recovery.recover_all(), host.recovery.collect_garbage()))
assert host.bindings == {}
assert host.recovery is not None
assert not host._maintenance.done()
with host.runtime.repository.sessions() as session:
broken = session.get(SkillProposalRow, owners[0])
healthy = session.get(SkillProposalRow, owners[1])
assert broken.state == "PENDING"
assert broken.package_blob == b"baseline" and broken.candidate_blob == b"candidate"
assert healthy.state == "EXPIRED"
assert healthy.package_blob is None and healthy.candidate_blob is None
finally:
await host.close()

View File

@ -0,0 +1,24 @@
"""Reproducible local capture measurements; not a model/provider benchmark."""
import statistics
import time
import pytest
from deerflow.skills.mutations.assets import capture_package
@pytest.mark.parametrize("files,size", [(4, 1024), (256, 65535)])
def test_capture_cost_at_typical_and_package_limit(tmp_path, files, size):
(tmp_path / "SKILL.md").write_text("---\nname: example\ndescription: Example\n---\nBody\n", encoding="utf-8")
for index in range(files - 1):
(tmp_path / f"resource-{index:03}.bin").write_bytes(b"x" * size)
timings, digests = [], set()
for _ in range(5):
start = time.perf_counter()
package = capture_package(tmp_path)
digests.add(package.digest)
timings.append((time.perf_counter() - start) * 1000)
assert len(digests) == 1
assert len(package.files) == files
print(f"capture+digest files={files} bytes={sum(len(item.content) for item in package.files)} median_ms={statistics.median(timings):.2f} max_ms={max(timings):.2f}")

View File

@ -0,0 +1,69 @@
"""Opt-in live PostgreSQL fence test; never substitute SQLite for PG locking."""
import asyncio
import os
import threading
import uuid
import pytest
from sqlalchemy import create_engine, select, text
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from deerflow.config.database_config import DatabaseConfig
from deerflow.persistence.base import Base
from deerflow.persistence.run import RunRepository
from deerflow.persistence.run.model import CompletedRunSnapshotRow, RunRow
from deerflow.skills.mutations.topology import mutation_session_factory
pytestmark = pytest.mark.skipif(not os.environ.get("DEERFLOW_TEST_POSTGRES_URL"), reason="requires an explicit live DEERFLOW_TEST_POSTGRES_URL")
@pytest.mark.asyncio
async def test_run_deletion_waits_for_publication_source_share_lock():
schema = "skill_evolution_test_" + uuid.uuid4().hex
config = DatabaseConfig(backend="postgres", postgres_url=os.environ["DEERFLOW_TEST_POSTGRES_URL"], postgres_schema=schema)
admin = create_engine(config.app_sync_sqlalchemy_url)
with admin.begin() as connection:
connection.execute(text(f'CREATE SCHEMA "{schema}"'))
sync_engine, sessions = mutation_session_factory(config.app_sync_sqlalchemy_url, postgres_schema=schema)
async_engine = create_async_engine(config.app_sqlalchemy_url, connect_args={"server_settings": {"search_path": schema}})
sf = async_sessionmaker(async_engine, expire_on_commit=False)
release, acquired = threading.Event(), threading.Event()
worker = None
deleting = None
try:
Base.metadata.create_all(sync_engine)
with sessions.begin() as session:
session.add(RunRow(run_id="run", thread_id="thread", user_id="owner", status="success"))
session.flush()
session.add(CompletedRunSnapshotRow(snapshot_ref="snapshot", run_id="run", scope_digest="scope", evidence_revision="revision", retention_revision=0, snapshot_json={}))
def publication_fence():
with sessions.begin() as session:
session.scalar(select(RunRow).where(RunRow.run_id == "run").with_for_update(read=True))
acquired.set()
assert release.wait(5)
worker = asyncio.create_task(asyncio.to_thread(publication_fence))
assert await asyncio.to_thread(acquired.wait, 5)
deleting = asyncio.create_task(RunRepository(sf).delete("run", user_id="owner"))
await asyncio.sleep(0.2)
assert not deleting.done(), "source deletion must wait for the accepted publication fence"
release.set()
await asyncio.wait_for(deleting, 5)
await worker
async with sf() as session:
assert await session.get(RunRow, "run") is None
assert await session.get(CompletedRunSnapshotRow, "snapshot") is None
finally:
release.set()
if worker is not None:
await worker
if deleting is not None:
await asyncio.gather(deleting, return_exceptions=True)
await async_engine.dispose()
sync_engine.dispose()
# Exact, randomly-created test schema only; never a configured schema.
with admin.begin() as connection:
connection.execute(text(f'DROP SCHEMA "{schema}" CASCADE'))
admin.dispose()

View File

@ -0,0 +1,50 @@
"""Real SIGKILL verifies journal/file ordering, not exception-only simulation."""
import asyncio
import json
import os
import signal
import sys
from pathlib import Path
import pytest
from test_skill_mutations import CONTENT, NEW, host, stage # noqa: F401 -- shared real-SQLite fixture
@pytest.mark.skipif(os.name != "posix", reason="P0 publication supports POSIX only")
@pytest.mark.asyncio
@pytest.mark.parametrize(
"boundary,expected",
[
("prepared", "ABORTED"),
("written", "ABORTED"),
("file_synced", "ABORTED"),
("renamed", "APPLIED"),
("directory_synced", "APPLIED"),
("response_lost", "APPLIED"),
],
)
async def test_sigkill_recovers_without_replay_or_double_version(host, boundary, expected): # noqa: F811 -- injected fixture
proposal = await stage(host)
await host.service.check(proposal_id=proposal.proposal_id)
args = {
"base": str(host.storage._paths.base_dir),
"skills": str(host.storage.get_skills_root_path()),
"database": str(host.sessions.kw["bind"].url),
"grant": host.binding.access.model_dump(mode="json"),
"proposal": proposal.proposal_id,
"boundary": boundary,
}
child = await asyncio.create_subprocess_exec(sys.executable, str(Path(__file__).parent / "fixtures" / "skill_mutation_crash_worker.py"), json.dumps(args), stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
_, stderr = await asyncio.wait_for(child.communicate(), timeout=30)
assert child.returncode == -signal.SIGKILL, stderr.decode()
await asyncio.to_thread(host.recovery.recover_owner, "owner")
operation = await host.service.find_operation(idempotency_key="crash-commit")
assert operation.publication == expected
expected_content = NEW if expected == "APPLIED" else CONTENT
assert host.storage.get_custom_skill_file("example").read_text(encoding="utf-8") == expected_content
first_revision = host.runtime.read_revision(host.storage, "example")
await asyncio.to_thread(host.recovery.recover_owner, "owner")
assert host.runtime.read_revision(host.storage, "example") == first_revision
assert first_revision == (operation.after_revision if expected == "APPLIED" else operation.before_revision)
assert not list(host.storage.get_custom_skill_dir("example").parent.glob(".host-mutation-*"))

View File

@ -0,0 +1,72 @@
from unittest.mock import AsyncMock
import pytest
from deerflow.config.app_config import AppConfig
from deerflow.skills.mutations.assets import PackageFile, PackageSnapshot
from deerflow.skills.security_scanner import ScanResult
CONTENT = b"---\nname: example\ndescription: Fine\n---\nHelpful instructions.\n"
def configuration(enabled=True):
return AppConfig.model_validate({"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, "skill_scan": {"enabled": enabled}, "skill_evolution": {"security_fail_closed": False}})
@pytest.mark.asyncio
async def test_host_check_never_treats_disabled_scanner_as_allow(monkeypatch):
from deerflow.skills.mutations.scanner import CandidateScanner
model = AsyncMock(return_value=ScanResult("allow", "fine"))
monkeypatch.setattr("deerflow.skills.mutations.scanner.scan_skill_content", model)
result = await CandidateScanner(lambda: configuration(False)).scan(PackageSnapshot((PackageFile("SKILL.md", CONTENT, False),)), "example")
assert result.decision == "unavailable"
model.assert_not_called()
@pytest.mark.asyncio
async def test_host_scan_covers_support_files_and_forces_fail_closed(monkeypatch):
from deerflow.skills.mutations.scanner import CandidateScanner
model = AsyncMock(return_value=ScanResult("allow", "fine"))
monkeypatch.setattr("deerflow.skills.mutations.scanner.scan_skill_content", model)
scanner = CandidateScanner(configuration)
package = PackageSnapshot((PackageFile("SKILL.md", CONTENT, False), PackageFile("references/help.md", b"Supporting advice", False)))
result = await scanner.scan(package, "example")
assert result.decision == "allow"
assert result.policy_version == await scanner.policy_version()
assert model.await_count == 2
assert all(call.kwargs["app_config"].skill_evolution.security_fail_closed for call in model.await_args_list)
@pytest.mark.asyncio
async def test_static_block_prevents_model_scan(monkeypatch):
from deerflow.skills.mutations.scanner import CandidateScanner
model = AsyncMock(return_value=ScanResult("allow", "fine"))
monkeypatch.setattr("deerflow.skills.mutations.scanner.scan_skill_content", model)
package = PackageSnapshot((PackageFile("SKILL.md", CONTENT, False), PackageFile("scripts/run.py", b"eval(input())", True)))
result = await CandidateScanner(configuration).scan(package, "example")
assert result.decision == "reject"
model.assert_not_called()
@pytest.mark.asyncio
async def test_scanner_resource_limit_is_not_partial_approval(monkeypatch):
from deerflow.skills.mutations.scanner import CandidateScanner
model = AsyncMock(return_value=ScanResult("allow", "fine"))
monkeypatch.setattr("deerflow.skills.mutations.scanner.scan_skill_content", model)
package = PackageSnapshot((PackageFile("SKILL.md", CONTENT, False), PackageFile("notes.txt", b"x" * (128 * 1024 + 1), False)))
result = await CandidateScanner(configuration).scan(package, "example")
assert result.decision == "unavailable"
model.assert_not_called()
@pytest.mark.asyncio
async def test_host_scan_timeout_propagates_without_authoritative_verdict(monkeypatch):
from deerflow.skills.mutations.scanner import CandidateScanner
monkeypatch.setattr("deerflow.skills.mutations.scanner.CHECK_TIMEOUT_SECONDS", 0)
with pytest.raises(TimeoutError):
await CandidateScanner(configuration).scan(PackageSnapshot((PackageFile("SKILL.md", CONTENT, False),)), "example")

Some files were not shown because too many files have changed in this diff Show More