feat(plugins): add evidence-gated skill self-evolution example (#7984)

* feat(plugins): add evidence-gated skill self-evolution example

Demonstrates the #7884 roadmap deliverable: a proposed change, the regression
case it fixes, the old and new results, and a rollback to a known-good version,
driven only through the public SkillMutationService contract merged in #5645.

The example stages a candidate against a sealed completed-run evidence anchor,
evaluates recorded JSON regression cases against both frozen bundles with a
deterministic static evaluator (never executing package content), and commits
only a "justified" verdict with an AssessmentRef binding the report to the
exact candidate hash and base revision. Unjustified or failed proposals are
discarded; revert stays append-only. Ships an HTTP walkthrough surface, an
offline lifecycle test against a contract-faithful fake host, and a README
section proposing a concrete evidence standard for the automatic-publication
discussion.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(skill-evolution): resolve request models and honor APPLIED publication

The walkthrough routes had never been exercised through FastAPI. With
`from __future__ import annotations`, request models defined inside
`build_router` are unresolvable forward references, so FastAPI read `body`
as a query parameter and every POST answered 422. Move the models to module
scope and cover the routes through TestClient.

A committed operation reports `publication == "APPLIED"`; `READY` is a
`views` state. The committed check listed READY, so a successful apply never
flipped `record.committed` and the ALREADY_COMMITTED guard never fired. Fix
the check, mirror the real literals in the lifecycle fake host, and assert
both the flag and the second-apply refusal.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(skill-evolution): address review feedback and register the example catalog

- Bind walkthrough routes to the authenticated principal: owner checks on propose/apply/revert, owner-scoped listing, and non-disclosing 404s.
- Map unknown skills, proposals and operations to 404 for both NOT_FOUND and the real host's NOT_FOUND_OR_FORBIDDEN; map broken case files to 400.
- Ignore URL paths and trim sentence punctuation when extracting in-package references.
- Validate config.enabled's type instead of treating any non-False value as on.
- Soften the README claim about report persistence and add CHANGELOG entries.
- Register the example in the frontend extension catalog, fixing frontend-unit-tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(skill-evolution): narrow case-file error mapping and cover run visibility

- Label ValueError as CASE_INVALID only on POST /proposals, the route that actually loads case files; apply and revert now surface host bugs as 500 instead of pointing at the case directory.
- Cover the request-scoped run-evidence reader the real Gateway installs: hidden anchor returns 403 SOURCE_NOT_VISIBLE before the host is touched, a denied resolver returns 403, and a visible anchor still proceeds.
- Add regressions pinning host ValueError on apply/revert to 500.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* docs(changelog): drop contributor changelog entry per policy

Contributor PRs no longer modify CHANGELOG.md or CHANGELOG_zh.md (#8202);
maintainers add the entry at merge time.

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
This commit is contained in:
Loveca 2026-11-16 16:04:42 +08:00 committed by GitHub
parent 4b98e46388
commit db02b77837
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
17 changed files with 2068 additions and 3 deletions

View File

@ -185,7 +185,7 @@ def test_bundled_documentation_snapshot_is_current():
script = runpy.run_path(str(root / "scripts/sync_extension_documentation.py"))
assert script["DESTINATION"].read_text(encoding="utf-8") == script["render"]()
pages = json.loads(script["render"]())
assert len(pages) == 16
assert len(pages) == 17
assert pages["deerflow-extension-office-preview"]["id"] == "office-preview"

View File

@ -12,8 +12,9 @@ independent Python package using the public `deerflow-extension-api` contract.
Additional examples cover [Office file previews](deerflow-extension-office-preview/README.md),
[text-list classification](deerflow-extension-jev-classify/README.md),
[context pruning](deerflow-extension-jev-context/README.md),
[advisory screening of fetched content](deerflow-extension-jev-screening/README.md), and
[sectioned long-form report writing](deerflow-extension-report-sections/README.md).
[advisory screening of fetched content](deerflow-extension-jev-screening/README.md),
[sectioned long-form report writing](deerflow-extension-report-sections/README.md), and
[evidence-gated skill self-evolution](deerflow-extension-skill-evolution/README.md).
## Describe the extension

View File

@ -0,0 +1,194 @@
# Skill self-evolution example — evidence-gated revisions
An opt-in Python plugin that demonstrates the roadmap deliverable from
[#7884](https://github.com/bytedance/deer-flow/issues/7884) ("Skills,
extensions, and reproducible agents"): a **proposed change**, the **regression
case it fixes**, the **old and new results**, and a **rollback to a known-good
version** — driven entirely through the host `SkillMutationService` contract
merged in [#5645](https://github.com/bytedance/deer-flow/pull/5645).
It uses extension API 0.2.11 with no host code changes. The host owns storage,
security scanning, revision identity and authorization; this package owns only
one policy: **a revision commits only when a recorded regression case
demonstrates it fixes something**.
## Why evidence-gated
With `skill_evolution.enabled`, the agent can already rewrite its own skills
through `skill_manage`. What is missing is a visible answer to "was that change
an improvement?". This example attaches one to every commit:
```
propose ──> stage() freeze baseline + candidate bundles (host-side)
evaluate ─> recorded cases run static checks against BOTH bundles
apply ────> check() + commit(assessment_ref) host security gate, then commit
with the verdict attached
rollback ─> revert() append-only return to the known-good revision
```
The `AssessmentRef` carried into `commit` binds the evidence to the exact
proposal (`candidate_hash` + `base_revision`); the host rejects a mismatched
assessment with `INVALID_ASSESSMENT`. Afterwards every committed `Operation`
answers: what changed, which recorded cases justified it, and how to undo it.
## Recorded regression cases
Cases are JSON files in `deerflow_extension_skill_evolution/cases/` (override
with `config.cases_dir`). One case per file:
```json
{
"id": "converter-arg",
"description": "SKILL.md must instruct the converter with --format json",
"checks": [
{ "kind": "text_contains", "path": "SKILL.md", "needle": "--format json" },
{ "kind": "text_not_contains", "path": "SKILL.md", "needle": "--fmt " }
]
}
````
Check kinds: `file_exists`, `text_contains`, `text_not_contains`,
`referenced_files_exist` (bare `scripts/…` / `assets/…` tokens in a text file
must exist in the package), and `scripts_parse` (every `scripts/*.py` must pass
`ast.parse`).
**The evaluator never executes package content.** The candidate is an
agent-proposed revision that has not passed the host security `check` yet, so
the only safe operations are structural (presence, text, parse-only). A
malformed case file raises and stops the evaluation rather than silently
narrowing it.
### Verdicts
| Verdict | Meaning | Service behavior |
| --- | --- | --- |
| `justified` | every case passes on the candidate and at least one fails on the baseline | eligible for `apply` |
| `unjustified` | every case passes on both sides — nothing recorded was broken | proposal discarded, `apply` refuses |
| `failed` | some case still fails on the candidate (or none recorded) | proposal discarded, `apply` refuses |
## Install and configure
From the DeerFlow checkout root:
```sh
make extension-install SOURCE=examples/deerflow-extension-skill-evolution
```
The mutation capability requires an explicit operator grant in `config.yaml`
(restart required). Prerequisites are the same as any `skill_mutations`
consumer: durable database backend, `run_events.backend: db`, and a single
POSIX host with local disk:
```yaml
plugins:
- name: skill-evolution
package: deerflow-extension-skill-evolution
use: deerflow_extension_skill_evolution:install
enabled: true
host_access:
evidence:
owners: ["your-user-id"]
skill_mutations:
owners: ["your-user-id"] # subset of evidence.owners
operations: [stage, check, commit, revert]
trigger_agents: [lead_agent]
target_skills: [example-skill] # canonical names this plugin may revise
topology: single_host_local
config:
cases_dir: null # optional: default bundled cases
```
Without the grant the service fails to start with a `NOT_GRANTED` diagnostic;
nothing else in the deployment changes.
## HTTP surface
All routes are under `/api/extension-skill-evolution`:
| Route | Purpose |
| --- | --- |
| `GET /health` | readiness, evaluator identity, recorded case count |
| `POST /proposals` | resolve the sealed run evidence, stage a candidate, evaluate, return the verdict + report |
| `GET /proposals` · `GET /proposals/{id}` | list / inspect proposals and their reports |
| `POST /proposals/{id}/apply` | host `check`, then commit with the `AssessmentRef` attached |
| `POST /operations/{id}/revert` | roll back a committed revision |
`POST /proposals` takes `{skill_name, candidate_content, thread_id, run_id}`.
The run must be completed and sealed — it is the evidence anchor the host
requires for staging (`source_refs`), so every proposal records *which run
motivated the change*.
**Every route binds the caller.** The capability handles injected into the
service are scoped to the extension by the operator's grant, not to the
requesting user, so the routes resolve the host principal
(`deerflow_extension_api.auth.resolve_principal`) and refuse to act on a skill
owned by anyone else: `propose`, `apply` and `revert` answer 401 without a
principal and 403 (`FORBIDDEN`) across owners, and `GET /proposals` lists only
the caller's own proposals (another owner's proposal reads as 404, without
disclosing that it exists). When the host exposes a request-scoped run-evidence
reader, the evidence anchor must also be visible to the caller; hosts without
one fall back to the owner check.
## Offline walkthrough (tests)
The package ships the demonstration as tests; no Gateway, model, or network:
```sh
cd examples/deerflow-extension-skill-evolution
uv run --no-project --python 3.12 --with pytest --with pytest-asyncio \
--with ../../backend/packages/extension-api \
--with-editable . python -m pytest tests/ -q
```
`tests/test_lifecycle.py` drives the full loop against a fake host that
implements only the public contract: propose a fix for a broken converter
argument → verdict `justified` → apply → the committed operation carries the
report id → revert → the skill is byte-identical to the known-good baseline.
It also pins the refusal paths: `unjustified` and `failed` proposals are
discarded and cannot commit, a denied host `check` blocks the commit, and
double-commit is rejected.
## Scope and boundaries
- **This is an example, not a production engine.** The evaluator is a
deterministic static checker; it has no live-model runs, metrics, or signal
collection.
- [#6954](https://github.com/bytedance/deer-flow/discussions/6954) proposes a
distributable evolution plugin package with its own snapshot/versioning
design; this example stops at the merged #5645 contract and should be
revisited against that RFC if it lands.
- [#7066](https://github.com/bytedance/deer-flow/issues/7066) (eval-gated
evolution) owns the broader signal → attribution → candidate → verification
loop; the minimal evaluator here is deliberately smaller than that scope.
- The `skill_manage` direct-write path is unchanged. This plugin is additive
and opt-in.
## A concrete evidence standard (input for the roadmap discussion)
The roadmap asks what evidence should be required before enabling automatic
publication. Rather than propose a standard in the abstract, this example makes
one concrete and reviewable:
1. **Every proposal names its motivating run.** Staging requires sealed
completed-run evidence (`source_refs`); the host rejects stale or unsealed
anchors.
2. **Evidence is recorded, not narrated.** A case is a versioned JSON file with
deterministic checks; the verdict is computed from the frozen baseline and
candidate bundles, not from the proposal text.
3. **The verdict travels with the commit.** `AssessmentRef` pins
`evaluator`, `evaluator_version`, and a content-addressed `report_id`
(SHA-256 over the canonical report JSON) to the exact `candidate_hash` and
`base_revision`. The report bytes themselves are held in this service's
in-memory cache, not persisted; the id lets anyone holding the report — or
recomputing it from the same cases and frozen bundles — verify that it is
the one this commit cites.
4. **Fail-closed by default.** `unjustified` and `failed` proposals are
discarded before they can commit; a host `check` denial blocks the commit;
an assessment mismatch is rejected by the host itself.
5. **Rollback is as auditable as the change.** `revert` appends a new
operation referencing the one it undoes.
Automatic publication would additionally need: case authorship provenance (who
recorded the case, and was it derived from the same run that motivated the
change), a minimum case count, and a policy for `unjustified` proposals that
keep recurring. Those are deliberately out of scope here.

View File

@ -0,0 +1,44 @@
"""Evidence-gated skill self-evolution example.
Demonstrates the roadmap deliverable: a proposed change, the regression case it
fixes, the old and new results, and a rollback to a known-good version — all
through the host ``SkillMutationService`` contract merged in #5645.
Install via ``config.yaml -> plugins:`` with a ``host_access.skill_mutations``
grant (see README.md). Without the grant the service fails to start and the
Gateway reports a diagnostic; nothing else changes.
"""
from __future__ import annotations
from collections.abc import Mapping
from pathlib import Path
from typing import Any
from deerflow_extension_api import ExtensionInstall, ExtensionRegistry, extension
from deerflow_extension_skill_evolution.service import EvolutionService, build_router
__all__ = ["install"]
_DEFAULT_CASES_DIR = Path(__file__).resolve().parent / "cases"
@extension(api="0.2.11", name="skill-evolution")
def install(registry: ExtensionRegistry, config: Mapping[str, Any]) -> None:
"""Register the evolution service and its HTTP walkthrough surface."""
# ``is False`` alone would silently keep the plugin on for ``enabled: 0`` or
# ``enabled: "false"``; a mistyped off switch must fail the deployment.
enabled = config.get("enabled", True)
if type(enabled) is not bool:
raise ValueError(f"skill-evolution: config.enabled must be a boolean, got {enabled!r}")
if not enabled:
return
cases_dir = config.get("cases_dir")
service = EvolutionService(cases_dir=Path(cases_dir) if cases_dir else _DEFAULT_CASES_DIR)
registry.service(service)
registry.routers((build_router(service),))
_entry_point: ExtensionInstall = install

View File

@ -0,0 +1,8 @@
{
"id": "converter-arg",
"description": "SKILL.md must instruct the converter with --format json, not the removed --fmt alias",
"checks": [
{ "kind": "text_contains", "path": "SKILL.md", "needle": "--format json" },
{ "kind": "text_not_contains", "path": "SKILL.md", "needle": "--fmt " }
]
}

View File

@ -0,0 +1,51 @@
{
"schema_version": 1,
"id": "catalog/skillEvolution",
"locales": {
"zh-CN": {
"title": "证据驱动的技能进化",
"description": "在回归用例与宿主校验通过后才提议技能修改,对比新旧结果,并可回滚到已确认的版本。",
"usage": [
{
"title": "安装并授权",
"description": "通过 make extension-install 安装示例,在 config.yaml 中配置 skill mutation 与 evidence 授权后重启 Gateway。"
},
{
"title": "记录回归用例",
"description": "在包的 cases 目录添加 JSON 用例,用 file_exists、text_contains、text_not_contains 等静态检查描述这次修改要修复什么。"
},
{
"title": "提出、评估、应用与回滚",
"description": "调用 /api/extension-skill-evolution/proposals 提交候选版本,查看 justified、unjustified 或 failed 结论;通过后 apply,必要时 revert 到已确认版本。"
}
],
"requirements": [
"需要支持 skill_mutations 的持久化数据库、run_events.backend: db,以及单实例 POSIX 本地磁盘。",
"需要在 config.yaml 中显式授予 skill mutation 与 evidence 访问权限。"
]
},
"en-US": {
"title": "Evidence-gated skill evolution",
"description": "Propose skill edits only after a regression case and host checks pass, compare old and new results, and roll back to a known-good revision.",
"usage": [
{
"title": "Install and grant access",
"description": "Install the example with make extension-install, configure the skill mutation and evidence grants in config.yaml, then restart Gateway."
},
{
"title": "Record a regression case",
"description": "Add a JSON case under the package's cases directory using static checks such as file_exists, text_contains and text_not_contains to state what the change fixes."
},
{
"title": "Propose, evaluate, apply and roll back",
"description": "Submit a candidate to /api/extension-skill-evolution/proposals, inspect the justified, unjustified or failed verdict, apply an accepted revision, and revert when needed."
}
],
"requirements": [
"Requires a durable database backend, run_events.backend: db, and a single POSIX host with local disk.",
"Requires explicit skill mutation and evidence grants in config.yaml."
]
}
},
"docs_url": "https://github.com/bytedance/deer-flow/tree/main/examples/deerflow-extension-skill-evolution#readme"
}

View File

@ -0,0 +1,312 @@
"""Deterministic static evaluation of proposed skill revisions.
This evaluator never executes package content. A candidate bundle is an
agent-proposed revision that has not passed the host security ``check`` yet,
so the only safe operations are structural: file presence, decoded-text
assertions, reference integrity, and ``ast.parse`` (parse only, no import,
no exec) for bundled Python scripts.
Recorded regression cases describe the defect a proposal claims to fix. The
verdict compares both sides of the proposal:
- ``justified`` — every case passes on the candidate and at least one fails
on the baseline: the change demonstrably fixes a recorded defect.
- ``unjustified`` — every case passes on both sides: nothing recorded was
broken, so the proposal carries no evidence-backed reason to land.
- ``failed`` — some case still fails on the candidate (or no cases were
recorded at all): the fix does not hold, so the proposal must not commit.
The report is deterministic — same inputs, same bytes, same ``report_id`` —
which is what lets it be referenced from a commit's ``AssessmentRef`` and
re-verified later.
"""
from __future__ import annotations
import ast
import hashlib
import json
import re
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Literal
EVALUATOR_NAME = "skill-evolution-example-static"
EVALUATOR_VERSION = "0.1.0"
Decision = Literal["justified", "unjustified", "failed"]
#: ``Package`` is the evaluator's view of one skill revision: POSIX-relative
#: file path → raw bytes. The host bundle (``ProposalBundle.baseline`` /
#: ``.candidate``) converts with ``{f.path: f.content for f in files}``.
Package = Mapping[str, bytes]
# Bare in-package references inside SKILL.md prose. A reference must appear as
# a standalone token: the lookbehind rejects a preceding word character, path
# separator, dot, colon or dash, so a URL such as
# ``https://example.com/scripts/run.py`` is not mistaken for an in-package file.
# Trailing sentence punctuation is trimmed after the match (see ``_references``).
# Only the two conventional skill subdirectories are scanned.
_REFERENCE_RE = re.compile(r"(?<![\w/.:-])((?:scripts|assets)/[A-Za-z0-9_./-]+)")
_CASE_ID_RE = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*")
_MAX_DETAIL_CHARS = 300
_CHECK_KINDS = frozenset(
{
"file_exists",
"text_contains",
"text_not_contains",
"referenced_files_exist",
"scripts_parse",
}
)
def _reject_unsafe_path(value: str, field: str) -> str:
"""Validate a case-declared package path before it is ever used as a key."""
if not value or len(value) > 1024 or "\\" in value or value.startswith("/"):
raise ValueError(f"{field} must be a relative POSIX path: {value!r}")
if any(part in {"", ".", ".."} for part in value.split("/")):
raise ValueError(f"{field} must not contain empty or dot segments: {value!r}")
return value
@dataclass(frozen=True)
class Check:
"""One static assertion against a skill package."""
kind: str
path: str | None = None
needle: str | None = None
@classmethod
def from_json(cls, raw: Any) -> Check:
if not isinstance(raw, dict):
raise ValueError("check must be an object")
kind = raw.get("kind")
if not isinstance(kind, str) or kind not in _CHECK_KINDS:
raise ValueError(f"unknown check kind: {kind!r}")
path = raw.get("path")
if path is not None:
if not isinstance(path, str):
raise ValueError(f"{kind}: path must be a string")
_reject_unsafe_path(path, f"{kind}: path")
needle = raw.get("needle")
if needle is not None and (not isinstance(needle, str) or not needle):
raise ValueError(f"{kind}: needle must be a nonempty string")
if kind in {"file_exists", "text_contains", "text_not_contains", "referenced_files_exist"} and path is None:
raise ValueError(f"{kind} requires a path")
if kind in {"text_contains", "text_not_contains"} and needle is None:
raise ValueError(f"{kind} requires a needle")
return cls(kind=kind, path=path, needle=needle)
@dataclass(frozen=True)
class Case:
"""A recorded regression case: all of its checks must pass on a fixed package."""
case_id: str
description: str
checks: tuple[Check, ...]
@classmethod
def from_json(cls, raw: Any, *, origin: str) -> Case:
if not isinstance(raw, dict):
raise ValueError(f"{origin}: case must be an object")
case_id = raw.get("id")
if not isinstance(case_id, str) or not _CASE_ID_RE.fullmatch(case_id) or len(case_id) > 64:
raise ValueError(f"{origin}: id must be kebab-case and at most 64 characters")
description = raw.get("description")
if not isinstance(description, str) or not description.strip():
raise ValueError(f"{origin}: description must be a nonempty string")
checks_raw = raw.get("checks")
if not isinstance(checks_raw, list) or not checks_raw:
raise ValueError(f"{origin}: checks must be a nonempty list")
return cls(
case_id=case_id,
description=description.strip(),
checks=tuple(Check.from_json(item) for item in checks_raw),
)
def load_cases_dir(directory: str | Path) -> tuple[Case, ...]:
"""Load recorded cases from ``*.json`` files, ordered by file name.
Malformed files raise ``ValueError`` naming the file: a broken case must
stop the evaluation rather than silently narrow it.
"""
root = Path(directory)
if not root.is_dir():
raise ValueError(f"cases directory not found: {root}")
cases: list[Case] = []
for file in sorted(root.glob("*.json")):
try:
raw = json.loads(file.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
raise ValueError(f"{file.name}: unreadable case file ({exc})") from exc
cases.append(Case.from_json(raw, origin=file.name))
return tuple(cases)
@dataclass(frozen=True)
class CheckOutcome:
kind: str
passed: bool
detail: str
@dataclass(frozen=True)
class CaseOutcome:
case_id: str
description: str
baseline: tuple[CheckOutcome, ...]
candidate: tuple[CheckOutcome, ...]
@property
def baseline_passed(self) -> bool:
return all(outcome.passed for outcome in self.baseline)
@property
def candidate_passed(self) -> bool:
return all(outcome.passed for outcome in self.candidate)
def _references(text: str) -> list[str]:
"""Extract bare in-package file references from prose.
``scripts/run.py.`` at the end of a sentence is the file ``scripts/run.py``,
not a file whose name ends in a period, so trailing ``.``/``/``/``-`` are
stripped before the token is looked up in the package.
"""
return [token for raw in _REFERENCE_RE.findall(text) if (token := raw.rstrip("./-"))]
def _decode(package: Package, path: str) -> tuple[str | None, str]:
content = package.get(path)
if content is None:
return None, f"missing {path}"
try:
return content.decode("utf-8"), ""
except UnicodeDecodeError:
return None, f"{path} is not valid UTF-8"
def _clip(detail: str) -> str:
return detail if len(detail) <= _MAX_DETAIL_CHARS else detail[: _MAX_DETAIL_CHARS - 1] + "…"
def run_check(check: Check, package: Package) -> CheckOutcome:
"""Run one static check against one package revision. Never executes code."""
if check.kind == "file_exists":
assert check.path is not None
passed = check.path in package
return CheckOutcome(check.kind, passed, "" if passed else f"missing {check.path}")
if check.kind == "text_contains":
assert check.path is not None and check.needle is not None
text, detail = _decode(package, check.path)
if text is None:
return CheckOutcome(check.kind, False, detail)
passed = check.needle in text
return CheckOutcome(check.kind, passed, "" if passed else _clip(f"{check.path} does not contain {check.needle!r}"))
if check.kind == "text_not_contains":
assert check.path is not None and check.needle is not None
text, detail = _decode(package, check.path)
if text is None:
return CheckOutcome(check.kind, False, detail)
passed = check.needle not in text
return CheckOutcome(check.kind, passed, "" if passed else _clip(f"{check.path} still contains {check.needle!r}"))
if check.kind == "referenced_files_exist":
assert check.path is not None
text, detail = _decode(package, check.path)
if text is None:
return CheckOutcome(check.kind, False, detail)
missing = sorted({token for token in _references(text) if token not in package})
passed = not missing
return CheckOutcome(check.kind, passed, "" if passed else _clip(f"{check.path} references missing files: {', '.join(missing)}"))
# scripts_parse: parse-only syntax gate over every bundled Python script.
failures: list[str] = []
for path in sorted(key for key in package if key.startswith("scripts/") and key.endswith(".py")):
text, detail = _decode(package, path)
if text is None:
failures.append(detail)
continue
try:
ast.parse(text, filename=path)
except SyntaxError as exc:
failures.append(f"{path}: {exc.msg} (line {exc.lineno})")
passed = not failures
return CheckOutcome("scripts_parse", passed, "" if passed else _clip("; ".join(failures)))
def evaluate_case(case: Case, baseline: Package, candidate: Package) -> CaseOutcome:
return CaseOutcome(
case_id=case.case_id,
description=case.description,
baseline=tuple(run_check(check, baseline) for check in case.checks),
candidate=tuple(run_check(check, candidate) for check in case.checks),
)
@dataclass(frozen=True)
class Evaluation:
decision: Decision
reasons: tuple[str, ...]
cases: tuple[CaseOutcome, ...]
def report(self, *, candidate_hash: str) -> dict[str, Any]:
"""Deterministic JSON-able report; identical inputs produce identical bytes."""
return {
"evaluator": EVALUATOR_NAME,
"evaluator_version": EVALUATOR_VERSION,
"decision": self.decision,
"reasons": list(self.reasons),
"candidate_hash": candidate_hash,
"cases": [
{
"id": outcome.case_id,
"description": outcome.description,
"baseline_passed": outcome.baseline_passed,
"candidate_passed": outcome.candidate_passed,
"baseline": [{"kind": item.kind, "passed": item.passed, "detail": item.detail} for item in outcome.baseline],
"candidate": [{"kind": item.kind, "passed": item.passed, "detail": item.detail} for item in outcome.candidate],
}
for outcome in self.cases
],
}
def evaluate(cases: Sequence[Case], baseline: Package, candidate: Package) -> Evaluation:
"""Compare recorded cases across both sides of a proposal."""
if not cases:
return Evaluation(decision="failed", reasons=("no regression cases recorded; nothing justifies a change",), cases=())
outcomes = tuple(evaluate_case(case, baseline, candidate) for case in cases)
candidate_failures = [outcome.case_id for outcome in outcomes if not outcome.candidate_passed]
baseline_failures = [outcome.case_id for outcome in outcomes if not outcome.baseline_passed]
if candidate_failures:
return Evaluation(
decision="failed",
reasons=(f"cases still failing on the candidate: {', '.join(candidate_failures)}",),
cases=outcomes,
)
if not baseline_failures:
return Evaluation(
decision="unjustified",
reasons=("every recorded case already passed on the baseline; the proposal fixes nothing on record",),
cases=outcomes,
)
return Evaluation(
decision="justified",
reasons=(f"candidate fixes recorded failures: {', '.join(baseline_failures)}",),
cases=outcomes,
)
def report_id(report: Mapping[str, Any]) -> str:
"""Content-addressed identifier for a report, safe to store in an AssessmentRef."""
canonical = json.dumps(dict(report), sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
return hashlib.sha256(canonical).hexdigest()

View File

@ -0,0 +1,463 @@
"""Evidence-gated skill evolution orchestration over the host mutation contract.
This service is the demonstration the roadmap asks for: it drives the merged
``SkillMutationService`` (#5645) through *propose → evaluate → apply → rollback*
and refuses to commit a revision that carries no recorded, passing evidence.
It depends only on the public ``deerflow-extension-api`` contract — never on
harness internals — so it is the same code path a third-party integrator would
write. The host owns storage, security scanning, revision identity and
authorization; this service owns only the *policy* of "commit only when a
recorded regression case justifies the change".
"""
from __future__ import annotations
import uuid
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from deerflow_extension_api import (
AssessmentRef,
CompletedRunEvidenceReader,
ExtensionPrincipal,
ExtensionRuntimeDeps,
HostCapabilityError,
SkillMutationService,
resolve_principal,
resolve_run_evidence_reader,
)
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from deerflow_extension_skill_evolution.evaluator import (
EVALUATOR_NAME,
EVALUATOR_VERSION,
Case,
Evaluation,
evaluate,
load_cases_dir,
report_id,
)
@dataclass
class ProposalRecord:
"""The example's own view of one in-flight proposal and its evaluation."""
proposal_id: str
name: str
owner_id: str
candidate_hash: str
decision: str
reasons: tuple[str, ...]
report_id: str
report: dict[str, Any]
operation_id: str | None = None
committed: bool = False
class SkillEvolutionError(RuntimeError):
"""A named, caller-facing failure distinct from host capability errors."""
def __init__(self, code: str, message: str = "") -> None:
self.code = code
super().__init__(f"{code}: {message}" if message else code)
class EvolutionService:
"""Orchestrates evidence-gated skill revisions through the host contract.
Lifecycle: ``start(deps)`` captures the host capability handles the Gateway
injects (``skill_mutations`` + ``completed_run_evidence``); ``stop()`` drops
them. Reports are held in memory for the life of the instance — the durable
evidence is the ``assessment`` the host freezes onto each committed
``Operation``, not this cache.
"""
def __init__(self, *, cases_dir: str | Path | None = None, cases: Sequence[Case] | None = None) -> None:
self._mutations: SkillMutationService | None = None
self._evidence: CompletedRunEvidenceReader | None = None
self._cases_dir = Path(cases_dir) if cases_dir is not None else None
self._static_cases: tuple[Case, ...] | None = tuple(cases) if cases is not None else None
self._proposals: dict[str, ProposalRecord] = {}
# -- ExtensionService protocol -----------------------------------------
async def start(self, deps: ExtensionRuntimeDeps) -> None:
self._mutations = deps.skill_mutations
self._evidence = deps.completed_run_evidence
if self._mutations is None:
raise SkillEvolutionError(
"NOT_GRANTED",
"skill_mutations capability missing; grant host_access.skill_mutations in config.yaml plugins",
)
async def stop(self) -> None:
self._mutations = None
self._evidence = None
# -- introspection ------------------------------------------------------
@property
def ready(self) -> bool:
return self._mutations is not None
def cases(self) -> tuple[Case, ...]:
"""Recorded regression cases: explicit set, else loaded from cases_dir."""
if self._static_cases is not None:
return self._static_cases
if self._cases_dir is None:
return ()
return load_cases_dir(self._cases_dir)
def get_proposal(self, proposal_id: str) -> ProposalRecord:
record = self._proposals.get(proposal_id)
if record is None:
raise SkillEvolutionError("NO_SUCH_PROPOSAL", proposal_id)
return record
def list_proposals(self) -> list[ProposalRecord]:
return list(self._proposals.values())
# -- core orchestration -------------------------------------------------
async def resolve_source(self, *, thread_id: str, run_id: str) -> str:
"""Turn a completed run into the sealed evidence ref ``stage`` requires."""
if self._evidence is None:
raise SkillEvolutionError("NO_EVIDENCE", "completed_run_evidence capability not granted")
snapshot = await self._evidence.get_snapshot(thread_id=thread_id, run_id=run_id)
if not snapshot.snapshot_ref:
raise SkillEvolutionError("SOURCE_UNSEALED", f"run {run_id} has no sealed evidence snapshot")
return snapshot.snapshot_ref
async def propose(
self,
*,
skill_name: str,
candidate_content: str,
source_ref: str,
idempotency_key: str | None = None,
cases: Sequence[Case] | None = None,
expected_owner_id: str | None = None,
) -> ProposalRecord:
"""Stage a candidate, evaluate it against recorded cases, and record the verdict.
Staging freezes both bundles host-side; evaluation runs entirely on the
returned ``ProposalBundle`` bytes, so the verdict is about exactly the
revision that would commit. Nothing is written to the live skill here.
``expected_owner_id`` binds the operation to a caller: when given, a
skill owned by anyone else is refused *before* staging, so a contributed
route can never act on another user's skill.
"""
mutations = self._require_mutations()
recorded = tuple(cases) if cases is not None else self.cases()
view = await mutations.read_skill(source_ref=source_ref, name=skill_name)
self._check_owner(view.owner_id, expected_owner_id)
proposal = await mutations.stage(
source_refs=(source_ref,),
target_ref=view.target_ref,
expected_revision=view.revision,
content=candidate_content,
idempotency_key=idempotency_key or uuid.uuid4().hex,
)
bundle = await mutations.read_proposal_bundle(proposal_id=proposal.proposal_id)
baseline = {item.path: item.content for item in bundle.baseline}
candidate = {item.path: item.content for item in bundle.candidate}
evaluation: Evaluation = evaluate(recorded, baseline, candidate)
report = evaluation.report(candidate_hash=proposal.candidate_hash)
rid = report_id(report)
record = ProposalRecord(
proposal_id=proposal.proposal_id,
name=skill_name,
owner_id=proposal.owner_id,
candidate_hash=proposal.candidate_hash,
decision=evaluation.decision,
reasons=evaluation.reasons,
report_id=rid,
report=report,
)
self._proposals[proposal.proposal_id] = record
# An unjustified or failed proposal must not linger pending; discard it
# so the host's pending-proposal quota is not consumed by dead weight.
if evaluation.decision != "justified":
await mutations.discard(proposal_id=proposal.proposal_id)
return record
async def apply(self, *, proposal_id: str, idempotency_key: str | None = None, expected_owner_id: str | None = None) -> Any:
"""Commit a justified proposal with its evidence attached, after host check.
Refuses unless the recorded evaluation was ``justified``. The host
``check`` (security scan / moderation) runs next and must allow; only
then does ``commit`` land, carrying an ``AssessmentRef`` whose
``candidate_hash`` and ``base_revision`` the host re-validates against
the proposal — the evidence is cryptographically bound to the change.
"""
mutations = self._require_mutations()
record = self.get_proposal(proposal_id)
self._check_owner(record.owner_id, expected_owner_id)
if record.committed:
raise SkillEvolutionError("ALREADY_COMMITTED", proposal_id)
if record.decision != "justified":
raise SkillEvolutionError(
"EVIDENCE_REQUIRED",
f"proposal evaluated '{record.decision}': {'; '.join(record.reasons)}",
)
proposal = await mutations.get_proposal(proposal_id=proposal_id)
check = await mutations.check(proposal_id=proposal_id)
if check.decision != "allow":
raise SkillEvolutionError("HOST_CHECK_DENIED", f"{check.reason_code} (decision={check.decision})")
assessment = AssessmentRef(
candidate_hash=proposal.candidate_hash,
base_revision=proposal.base_revision,
evaluator=EVALUATOR_NAME,
evaluator_version=EVALUATOR_VERSION,
report_id=record.report_id,
)
operation = await mutations.commit(
proposal_id=proposal_id,
idempotency_key=idempotency_key or uuid.uuid4().hex,
assessment_ref=assessment,
)
record.operation_id = operation.operation_id
# ``publication`` is the operation's publication state (PREPARED /
# APPLIED / NO_CHANGE / ABORTED / NEEDS_REPAIR), not its ``views``
# state (PENDING / READY). A commit that the host accepted is either
# still publishing (PREPARED) or already landed (APPLIED).
record.committed = operation.publication in {"PREPARED", "APPLIED", "NO_CHANGE"}
return operation
async def rollback(self, *, operation_id: str, idempotency_key: str | None = None, expected_owner_id: str | None = None) -> Any:
"""Revert a committed revision to its known-good predecessor.
The revert is itself an audited ``Operation`` (history stays append-only),
so a rollback is as traceable as the change it undoes.
"""
mutations = self._require_mutations()
operation = await mutations.get_operation(operation_id=operation_id)
self._check_owner(operation.owner_id, expected_owner_id)
return await mutations.revert(
operation_id=operation_id,
expected_current_revision=operation.after_revision,
idempotency_key=idempotency_key or uuid.uuid4().hex,
)
@staticmethod
def _check_owner(owner_id: str, expected_owner_id: str | None) -> None:
"""Refuse to act on a skill owned by someone other than the caller.
Omitting ``expected_owner_id`` keeps the trusted in-process path (tests,
a host service) usable; every user-facing route passes it.
"""
if expected_owner_id is not None and owner_id != expected_owner_id:
raise SkillEvolutionError("FORBIDDEN", "this skill belongs to another user")
def _require_mutations(self) -> SkillMutationService:
if self._mutations is None:
raise SkillEvolutionError("NOT_STARTED", "service.start(deps) has not bound the host capability")
return self._mutations
# Request models live at module scope: this file uses ``from __future__ import
# annotations``, and FastAPI cannot resolve a string forward reference to a
# class that only exists inside ``build_router``'s local namespace.
class ProposeRequest(BaseModel):
model_config = {"extra": "forbid"}
skill_name: str = Field(min_length=1, max_length=64)
candidate_content: str = Field(min_length=1)
thread_id: str = Field(min_length=1)
run_id: str = Field(min_length=1)
idempotency_key: str | None = None
class ApplyRequest(BaseModel):
model_config = {"extra": "forbid"}
idempotency_key: str | None = None
class RevertRequest(BaseModel):
model_config = {"extra": "forbid"}
idempotency_key: str | None = None
def build_router(service: EvolutionService):
"""Expose the closed loop as a small HTTP surface for the README walkthrough.
Every route that touches a skill binds the caller: the host's request
principal must own the skill, and a proposal's owner is re-checked on
``apply`` and ``revert``. The capability handles injected into the service
are operator-scoped, not caller-scoped, so without this binding any
authenticated session user could drive the loop against another owner's
committed skill (see ``backend/packages/harness/deerflow/extensions/AGENTS.md``).
"""
router = APIRouter(prefix="/api/extension-skill-evolution", tags=["skill-evolution"])
def _principal(request: Request) -> ExtensionPrincipal:
principal = resolve_principal(request)
if principal is None:
raise HTTPException(status_code=401, detail="authentication required")
return principal
async def _require_visible_run(request: Request, thread_id: str, run_id: str) -> None:
"""Confirm the evidence anchor is visible to the caller, when the host says.
``skill_mutations`` / ``completed_run_evidence`` in ``ExtensionRuntimeDeps``
are allowlist-scoped for the extension, not scoped to the requesting user.
A host that offers the request-scoped reader answers visibility here;
hosts without it (``None``) fall back to the owner check alone.
"""
try:
reader = resolve_run_evidence_reader(request)
if reader is None:
return
status = await reader.get_run_status(thread_id=thread_id, run_id=run_id)
except NotImplementedError:
return
if status is None:
raise SkillEvolutionError("SOURCE_NOT_VISIBLE", f"run {run_id} is not visible to this caller")
def _guard(coro, *, case_errors: bool = False):
"""Translate named example/host errors into HTTP statuses.
``case_errors`` is set only on routes that load the operator case files
(``POST /proposals``; ``/health`` handles them itself). Elsewhere a
``ValueError`` is a host or example bug and must surface as a 500 — not
as a misleading ``CASE_INVALID`` pointing at the case directory.
"""
async def run():
try:
return await coro
except SkillEvolutionError as exc:
status = _STATUS_BY_CODE.get(exc.code, 400)
raise HTTPException(status_code=status, detail=str(exc)) from exc
except PermissionError as exc:
# Raised by the host's request-scoped resolvers on denial.
raise HTTPException(status_code=403, detail=str(exc)) from exc
except HostCapabilityError as exc:
# A missing run raises ``NOT_FOUND``; the skill-mutation host
# raises ``NOT_FOUND_OR_FORBIDDEN`` for an unknown skill,
# proposal or operation. Both read as 404 (never disclosing
# which), not as a conflict; everything else is a 409.
status = 404 if exc.code in {"NOT_FOUND", "NOT_FOUND_OR_FORBIDDEN"} else 409
raise HTTPException(status_code=status, detail=str(exc)) from exc
except ValueError as exc:
# A broken operator-supplied case file fails closed
# (`load_cases_dir` raises), but as a 400 diagnostic rather
# than an unhandled 500 — only where cases are actually read.
if not case_errors:
raise
raise HTTPException(status_code=400, detail=f"CASE_INVALID: {exc}") from exc
return run()
@router.get("/health")
async def health() -> dict[str, Any]:
try:
case_count = len(service.cases())
except ValueError as exc:
# A broken case file must not turn readiness into a 500 traceback.
raise HTTPException(status_code=400, detail=f"CASE_INVALID: {exc}") from exc
return {"ready": service.ready, "evaluator": EVALUATOR_NAME, "evaluator_version": EVALUATOR_VERSION, "cases": case_count}
@router.post("/proposals")
async def create_proposal(body: ProposeRequest, request: Request) -> dict[str, Any]:
principal = _principal(request)
async def work():
await _require_visible_run(request, body.thread_id, body.run_id)
source_ref = await service.resolve_source(thread_id=body.thread_id, run_id=body.run_id)
record = await service.propose(
skill_name=body.skill_name,
candidate_content=body.candidate_content,
source_ref=source_ref,
idempotency_key=body.idempotency_key,
expected_owner_id=principal.user_id,
)
return {
"proposal_id": record.proposal_id,
"decision": record.decision,
"reasons": list(record.reasons),
"report_id": record.report_id,
"report": record.report,
}
return await _guard(work(), case_errors=True)
@router.get("/proposals")
async def list_proposals(request: Request) -> dict[str, Any]:
principal = _principal(request)
return {
"proposals": [
{
"proposal_id": record.proposal_id,
"name": record.name,
"decision": record.decision,
"committed": record.committed,
"operation_id": record.operation_id,
"report_id": record.report_id,
}
for record in service.list_proposals()
if record.owner_id == principal.user_id
]
}
@router.get("/proposals/{proposal_id}")
async def read_proposal(proposal_id: str, request: Request) -> dict[str, Any]:
principal = _principal(request)
try:
record = service.get_proposal(proposal_id)
except SkillEvolutionError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
# Do not disclose another owner's proposal (or even that it exists).
if record.owner_id != principal.user_id:
raise HTTPException(status_code=404, detail=f"NO_SUCH_PROPOSAL: {proposal_id}")
return {"proposal_id": record.proposal_id, "decision": record.decision, "reasons": list(record.reasons), "committed": record.committed, "operation_id": record.operation_id, "report": record.report}
@router.post("/proposals/{proposal_id}/apply")
async def apply_proposal(proposal_id: str, body: ApplyRequest, request: Request) -> dict[str, Any]:
principal = _principal(request)
async def work():
operation = await service.apply(proposal_id=proposal_id, idempotency_key=body.idempotency_key, expected_owner_id=principal.user_id)
return {
"operation_id": operation.operation_id,
"publication": operation.publication,
"before_revision": operation.before_revision.content_digest,
"after_revision": operation.after_revision.content_digest,
}
return await _guard(work())
@router.post("/operations/{operation_id}/revert")
async def revert_operation(operation_id: str, body: RevertRequest, request: Request) -> dict[str, Any]:
principal = _principal(request)
async def work():
operation = await service.rollback(operation_id=operation_id, idempotency_key=body.idempotency_key, expected_owner_id=principal.user_id)
return {"operation_id": operation.operation_id, "publication": operation.publication, "reverts": operation.reverts_operation_id}
return await _guard(work())
return router
#: Caller-facing failure codes mapped to HTTP statuses by ``build_router``.
_STATUS_BY_CODE = {
"EVIDENCE_REQUIRED": 409,
"HOST_CHECK_DENIED": 409,
"ALREADY_COMMITTED": 409,
"FORBIDDEN": 403,
"SOURCE_NOT_VISIBLE": 403,
# Only the in-memory proposal cache knows ids this process never staged.
"NO_SUCH_PROPOSAL": 404,
}

View File

@ -0,0 +1,44 @@
[project]
name = "deerflow-extension-skill-evolution"
version = "0.1.0"
description = "Example: evidence-gated skill self-evolution through the host SkillMutationService contract"
readme = "README.md"
requires-python = ">=3.12"
license = "MIT"
dependencies = [
"deerflow-extension-api>=0.2.11,<0.3",
"fastapi>=0.115.0,<1",
"pydantic>=2.12,<3",
]
[project.entry-points."deerflow.extensions"]
skill-evolution = "deerflow_extension_skill_evolution:install"
[project.optional-dependencies]
dev = [
"httpx>=0.28,<1",
"pytest>=9,<10",
"pytest-asyncio>=1.3,<2",
"ruff>=0.14,<1",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["deerflow_extension_skill_evolution"]
[tool.ruff]
line-length = 240
target-version = "py312"
[tool.ruff.lint]
select = ["E", "F", "I", "UP"]
[tool.ruff.lint.isort]
known-first-party = ["deerflow_extension_skill_evolution"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"

View File

@ -0,0 +1,201 @@
"""Offline unit tests for the static evaluator: no host, no package content executed."""
import json
import pytest
from deerflow_extension_skill_evolution.evaluator import (
EVALUATOR_NAME,
EVALUATOR_VERSION,
Case,
Check,
evaluate,
load_cases_dir,
report_id,
run_check,
)
def pkg(**files: str) -> dict[str, bytes]:
return {name: content.encode("utf-8") for name, content in files.items()}
MAIN_OK = "---\nname: example-skill\ndescription: demo\n---\n\nRun `scripts/run.py --format json` to convert.\n"
MAIN_BAD_ARG = MAIN_OK.replace("--format json", "--fmt json")
SCRIPT_OK = "import json\nprint(json.dumps({'ok': True}))\n"
SCRIPT_BROKEN = "def f(:\n"
# --- check kinds -----------------------------------------------------------
def test_file_exists_both_ways():
package = pkg(**{"SKILL.md": MAIN_OK})
assert run_check(Check("file_exists", path="SKILL.md"), package).passed
missing = run_check(Check("file_exists", path="scripts/run.py"), package)
assert not missing.passed and "missing scripts/run.py" in missing.detail
def test_text_contains_and_not_contains():
package = pkg(**{"SKILL.md": MAIN_OK})
assert run_check(Check("text_contains", path="SKILL.md", needle="--format json"), package).passed
failed = run_check(Check("text_contains", path="SKILL.md", needle="--format yaml"), package)
assert not failed.passed and "--format yaml" in failed.detail
assert run_check(Check("text_not_contains", path="SKILL.md", needle="--fmt"), package).passed
still = run_check(Check("text_not_contains", path="SKILL.md", needle="--format"), package)
assert not still.passed
def test_text_checks_report_missing_file_and_bad_encoding():
assert not run_check(Check("text_contains", path="SKILL.md", needle="x"), {}).passed
binary = {"SKILL.md": b"\xff\xfe not utf-8"}
outcome = run_check(Check("text_contains", path="SKILL.md", needle="x"), binary)
assert not outcome.passed and "not valid UTF-8" in outcome.detail
def test_referenced_files_exist_flags_dangling_script_reference():
baseline = pkg(**{"SKILL.md": MAIN_OK})
outcome = run_check(Check("referenced_files_exist", path="SKILL.md"), baseline)
assert not outcome.passed and "scripts/run.py" in outcome.detail
fixed = pkg(**{"SKILL.md": MAIN_OK, "scripts/run.py": SCRIPT_OK})
assert run_check(Check("referenced_files_exist", path="SKILL.md"), fixed).passed
def test_referenced_files_exist_ignores_urls_and_trims_sentence_punctuation():
# A URL path is not an in-package reference; a bare token still is.
prose = pkg(**{"SKILL.md": "Use the scripts/run helper. See https://example.com/scripts/run.py docs.\n"})
outcome = run_check(Check("referenced_files_exist", path="SKILL.md"), prose)
assert not outcome.passed
assert "scripts/run" in outcome.detail
assert "example.com" not in outcome.detail
# A file named at the end of a sentence is the file, not "run.py.".
fixed = pkg(**{"SKILL.md": "Run scripts/run.py.\n", "scripts/run.py": SCRIPT_OK})
assert run_check(Check("referenced_files_exist", path="SKILL.md"), fixed).passed
def test_scripts_parse_is_syntax_only_and_never_executes():
# A script that would print if executed; parse must not run it.
hostile = pkg(**{"scripts/run.py": "print('EXECUTED')\n"})
assert run_check(Check("scripts_parse"), hostile).passed
broken = pkg(**{"scripts/run.py": SCRIPT_BROKEN})
outcome = run_check(Check("scripts_parse"), broken)
assert not outcome.passed and "SyntaxError" not in outcome.detail and "line 1" in outcome.detail
# Non-UTF8 scripts are reported, not crashed on.
binary = {"scripts/run.py": b"\xff\xfe"}
assert not run_check(Check("scripts_parse"), binary).passed
# No scripts at all passes vacuously.
assert run_check(Check("scripts_parse"), pkg(**{"SKILL.md": MAIN_OK})).passed
# --- validation ------------------------------------------------------------
@pytest.mark.parametrize(
"path",
["", "/abs/path", "..\\win", "a/../b", "./x", "a//b", "x/" + "y" * 1100],
)
def test_case_path_validation_rejects_unsafe_paths(path):
with pytest.raises(ValueError):
Check.from_json({"kind": "file_exists", "path": path})
def test_check_from_json_requires_kind_specific_fields():
with pytest.raises(ValueError):
Check.from_json({"kind": "nope"})
with pytest.raises(ValueError):
Check.from_json({"kind": "file_exists"}) # missing path
with pytest.raises(ValueError):
Check.from_json({"kind": "text_contains", "path": "SKILL.md"}) # missing needle
with pytest.raises(ValueError):
Check.from_json({"kind": "text_contains", "path": "SKILL.md", "needle": ""})
def test_case_from_json_validation():
good = {"id": "arg-order", "description": "converter arg is wrong", "checks": [{"kind": "text_contains", "path": "SKILL.md", "needle": "--format json"}]}
case = Case.from_json(good, origin="case.json")
assert case.case_id == "arg-order" and len(case.checks) == 1
for bad in (
{"id": "Upper", "description": "d", "checks": [{"kind": "scripts_parse"}]},
{"id": "ok", "description": " ", "checks": [{"kind": "scripts_parse"}]},
{"id": "ok", "description": "d", "checks": []},
{"id": "x" * 65, "description": "d", "checks": [{"kind": "scripts_parse"}]},
"not-an-object",
):
with pytest.raises(ValueError):
Case.from_json(bad, origin="case.json")
def test_load_cases_dir_orders_by_name_and_reports_broken_files(tmp_path):
(tmp_path / "01-a.json").write_text(json.dumps({"id": "a", "description": "d", "checks": [{"kind": "scripts_parse"}]}), encoding="utf-8")
(tmp_path / "02-b.json").write_text(json.dumps({"id": "b", "description": "d", "checks": [{"kind": "scripts_parse"}]}), encoding="utf-8")
(tmp_path / "ignored.txt").write_text("x", encoding="utf-8")
cases = load_cases_dir(tmp_path)
assert [case.case_id for case in cases] == ["a", "b"]
(tmp_path / "03-broken.json").write_text("{not json", encoding="utf-8")
with pytest.raises(ValueError, match="03-broken.json"):
load_cases_dir(tmp_path)
with pytest.raises(ValueError, match="cases directory not found"):
load_cases_dir(tmp_path / "missing")
# --- decisions -------------------------------------------------------------
ARG_CASE = Case(
case_id="converter-arg",
description="SKILL.md must instruct --format json, not --fmt",
checks=(Check("text_contains", path="SKILL.md", needle="--format json"),),
)
PARSE_CASE = Case(case_id="scripts-parse", description="bundled scripts must parse", checks=(Check("scripts_parse"),))
def test_justified_when_candidate_fixes_a_recorded_baseline_failure():
baseline = pkg(**{"SKILL.md": MAIN_BAD_ARG, "scripts/run.py": SCRIPT_OK})
candidate = pkg(**{"SKILL.md": MAIN_OK, "scripts/run.py": SCRIPT_OK})
evaluation = evaluate([ARG_CASE, PARSE_CASE], baseline, candidate)
assert evaluation.decision == "justified"
assert "converter-arg" in evaluation.reasons[0]
def test_unjustified_when_baseline_already_passes_everything():
package = pkg(**{"SKILL.md": MAIN_OK, "scripts/run.py": SCRIPT_OK})
evaluation = evaluate([ARG_CASE], package, package)
assert evaluation.decision == "unjustified"
def test_failed_when_candidate_still_fails():
baseline = pkg(**{"SKILL.md": MAIN_BAD_ARG})
candidate = pkg(**{"SKILL.md": MAIN_BAD_ARG.replace("--fmt", "--format"), "scripts/run.py": SCRIPT_BROKEN})
evaluation = evaluate([ARG_CASE, PARSE_CASE], baseline, candidate)
assert evaluation.decision == "failed" and "scripts-parse" in evaluation.reasons[0]
def test_failed_when_no_cases_recorded():
evaluation = evaluate([], pkg(), pkg())
assert evaluation.decision == "failed" and "no regression cases" in evaluation.reasons[0]
# --- report determinism ----------------------------------------------------
def test_report_is_deterministic_and_report_id_is_content_addressed():
baseline = pkg(**{"SKILL.md": MAIN_BAD_ARG})
candidate = pkg(**{"SKILL.md": MAIN_OK})
first = evaluate([ARG_CASE], baseline, candidate).report(candidate_hash="a" * 64)
second = evaluate([ARG_CASE], baseline, candidate).report(candidate_hash="a" * 64)
assert json.dumps(first, sort_keys=True) == json.dumps(second, sort_keys=True)
assert report_id(first) == report_id(second)
assert first["evaluator"] == EVALUATOR_NAME and first["evaluator_version"] == EVALUATOR_VERSION
assert first["decision"] == "justified"
assert first["cases"][0]["baseline_passed"] is False and first["cases"][0]["candidate_passed"] is True
# A different candidate hash changes the report identity.
other = evaluate([ARG_CASE], baseline, candidate).report(candidate_hash="b" * 64)
assert report_id(other) != report_id(first)
def test_detail_is_clipped_to_bound():
huge = pkg(**{"SKILL.md": "x" * 5000})
outcome = run_check(Check("text_contains", path="SKILL.md", needle="y" * 5000), huge)
assert len(outcome.detail) <= 300

View File

@ -0,0 +1,352 @@
"""End-to-end orchestration test against a fake host implementing only the public contract.
CI installs this package with the extension-api alone (no harness), so the fake
host here mirrors what ``HostSkillMutationService`` guarantees: content-addressed
revisions, baseline/candidate bundles, an allow-only check gate, commit that
re-validates the assessment against the proposal, and an append-only revert.
The point is that the example needs nothing beyond the contract.
"""
import hashlib
import json
from dataclasses import dataclass, replace
from typing import Any
import pytest
from deerflow_extension_api import (
AssessmentRef,
AssetRevision,
BundleFile,
CompletedRunEventPage,
CompletedRunSnapshot,
EvidenceLimits,
ExtensionRuntimeDeps,
HostCapabilityError,
HostCheckResult,
MutationCapabilities,
Operation,
Proposal,
ProposalBundle,
SkillRevisionView,
)
from deerflow_extension_api.run_evidence import RunPage
from deerflow_extension_skill_evolution import install
from deerflow_extension_skill_evolution.evaluator import Case, Check
from deerflow_extension_skill_evolution.service import EvolutionService, SkillEvolutionError
SKILL = "example-skill"
OWNER = "owner"
BASE_MAIN = "---\nname: example-skill\ndescription: demo\n---\n\nRun `scripts/run.py --fmt json`.\n"
FIXED_MAIN = BASE_MAIN.replace("--fmt json", "--format json")
SCRIPT = "import json\nprint(json.dumps({'ok': True}))\n"
ARG_CASE = Case(
case_id="converter-arg",
description="SKILL.md must instruct --format json, not --fmt",
checks=(Check("text_contains", path="SKILL.md", needle="--format json"),),
)
def _digest(files: dict[str, bytes]) -> str:
canonical = json.dumps({k: hashlib.sha256(v).hexdigest() for k, v in sorted(files.items())}, sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(canonical).hexdigest()
def _bundle(files: dict[str, bytes]) -> tuple[BundleFile, ...]:
return tuple(BundleFile(path=p, content=c, sha256=hashlib.sha256(c).hexdigest(), executable=p.startswith("scripts/") and p.endswith(".py")) for p, c in sorted(files.items()))
@dataclass
class _OpRow:
operation_id: str
publication: str
before_revision: AssetRevision
after_revision: AssetRevision
assessment: dict[str, Any] | None
reverts_operation_id: str | None = None
class FakeMutationService:
"""A contract-faithful in-memory host for the mutation lifecycle."""
def __init__(self, *, files: dict[str, bytes] | None = None, check_decision: str = "allow") -> None:
self._files = files if files is not None else {"SKILL.md": BASE_MAIN.encode(), "scripts/run.py": SCRIPT.encode()}
self._check_decision = check_decision
self._seq = 0
self._proposals: dict[str, Proposal] = {}
self._bundles: dict[str, tuple[tuple[BundleFile, ...], tuple[BundleFile, ...]]] = {}
self._operations: dict[str, _OpRow] = {}
self._committed: set[str] = set()
self._discarded: set[str] = set()
@property
def capabilities(self) -> MutationCapabilities:
return MutationCapabilities(operations=("stage", "check", "commit", "revert"))
def _current_revision(self) -> AssetRevision:
return AssetRevision(incarnation_id="inc-1", mutation_seq=self._seq, content_digest=_digest(self._files))
async def read_skill(self, *, source_ref: str, name: str) -> SkillRevisionView:
if name != SKILL:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN", name)
rev = self._current_revision()
return SkillRevisionView(target_ref=f"asset:{name}", owner_id=OWNER, name=name, revision=rev, content=self._files["SKILL.md"].decode(), enabled=True)
async def stage(self, *, source_refs: tuple[str, ...], target_ref: str, expected_revision: AssetRevision, content: str, idempotency_key: str) -> Proposal:
if expected_revision != self._current_revision():
raise HostCapabilityError("REVISION_CONFLICT")
candidate = {**self._files, "SKILL.md": content.encode()}
pid = f"prop-{len(self._proposals) + 1}"
base = self._current_revision()
proposal = Proposal(
proposal_id=pid,
owner_id=OWNER,
target_ref=target_ref,
name=SKILL,
base_revision=base,
candidate_hash=_digest(candidate),
state="STAGED",
expires_at="2999-01-01T00:00:00Z",
source_refs=tuple(source_refs),
)
self._proposals[pid] = proposal
self._bundles[pid] = (_bundle(self._files), _bundle(candidate))
return proposal
async def get_proposal(self, *, proposal_id: str) -> Proposal:
proposal = self._proposals.get(proposal_id)
if proposal is None:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN", proposal_id)
return proposal
async def read_proposal_bundle(self, *, proposal_id: str, max_bytes: int = 33554432) -> ProposalBundle:
proposal = await self.get_proposal(proposal_id=proposal_id)
baseline, candidate = self._bundles[proposal_id]
return ProposalBundle(proposal_id=proposal_id, baseline=baseline, candidate=candidate, base_revision=proposal.base_revision, candidate_hash=proposal.candidate_hash)
async def discard(self, *, proposal_id: str) -> Proposal:
await self.get_proposal(proposal_id=proposal_id)
self._discarded.add(proposal_id)
return replace(self._proposals[proposal_id], state="DISCARDED")
async def check(self, *, proposal_id: str) -> HostCheckResult:
proposal = await self.get_proposal(proposal_id=proposal_id)
return HostCheckResult(
proposal_id=proposal_id,
decision=self._check_decision,
candidate_hash=proposal.candidate_hash,
base_revision=proposal.base_revision,
policy_version="policy-1",
expires_at="2999-01-01T00:00:00Z",
reason_code="CHECK_PASSED" if self._check_decision == "allow" else "CHECK_BLOCKED",
)
async def commit(self, *, proposal_id: str, idempotency_key: str, assessment_ref: AssessmentRef | None = None) -> Operation:
proposal = await self.get_proposal(proposal_id=proposal_id)
if proposal_id in self._committed:
raise HostCapabilityError("PROPOSAL_ALREADY_COMMITTED")
if proposal_id in self._discarded:
raise HostCapabilityError("PROPOSAL_DISCARDED")
# The host re-validates the assessment against the exact proposal.
if assessment_ref is not None and (assessment_ref.candidate_hash != proposal.candidate_hash or assessment_ref.base_revision != proposal.base_revision):
raise HostCapabilityError("INVALID_ASSESSMENT")
before = self._current_revision()
# Recover the candidate bytes from the staged bundle.
baseline_files, candidate_files = self._bundles[proposal_id]
del baseline_files
self._files = {f.path: f.content for f in candidate_files}
changed = _digest(self._files) != before.content_digest
self._seq += int(changed)
after = AssetRevision(before.incarnation_id, before.mutation_seq + int(changed), _digest(self._files))
op_id = f"op-{len(self._operations) + 1}"
assessment = None
if assessment_ref is not None:
assessment = {"candidate_hash": assessment_ref.candidate_hash, "evaluator": assessment_ref.evaluator, "report_id": assessment_ref.report_id}
# ``publication``, not ``views``: a committed change is APPLIED (or
# NO_CHANGE when the candidate is identical to the baseline). The real
# host never reports READY here -- that is a ``views`` state.
self._operations[op_id] = _OpRow(operation_id=op_id, publication="APPLIED" if changed else "NO_CHANGE", before_revision=before, after_revision=after, assessment=assessment)
self._committed.add(proposal_id)
return self._operation_view(op_id)
async def revert(self, *, operation_id: str, expected_current_revision: AssetRevision, idempotency_key: str) -> Operation:
row = self._operations.get(operation_id)
if row is None:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN", operation_id)
if expected_current_revision != self._current_revision():
raise HostCapabilityError("REVISION_CONFLICT")
# Revert to the revision that operation replaced.
self._files = {f.path: f.content for f in self._bundles_of_operation(operation_id)}
self._seq += 1
after = AssetRevision("inc-1", self._seq, _digest(self._files))
new_id = f"op-{len(self._operations) + 1}"
self._operations[new_id] = _OpRow(operation_id=new_id, publication="APPLIED", before_revision=expected_current_revision, after_revision=after, assessment=None, reverts_operation_id=operation_id)
return self._operation_view(new_id)
def _bundles_of_operation(self, operation_id: str) -> tuple[BundleFile, ...]:
# The pre-image of a commit is the baseline bundle it staged from.
for pid, row in self._operations.items():
if row.operation_id == operation_id:
for staged_pid, proposal in self._proposals.items():
baseline, _candidate = self._bundles[staged_pid]
return baseline
return _bundle(self._files)
async def get_operation(self, *, operation_id: str) -> Operation:
if operation_id not in self._operations:
raise HostCapabilityError("NOT_FOUND_OR_FORBIDDEN", operation_id)
return self._operation_view(operation_id)
async def find_operation(self, *, idempotency_key: str, method: str = "commit") -> Operation | None:
return None
def _operation_view(self, op_id: str) -> Operation:
row = self._operations[op_id]
return Operation(
operation_id=row.operation_id,
owner_id=OWNER,
name=SKILL,
publication=row.publication,
views="READY",
before_revision=row.before_revision,
after_revision=row.after_revision,
generation=row.after_revision.mutation_seq,
reverts_operation_id=row.reverts_operation_id,
)
class FakeEvidence:
"""Minimal sealed-run evidence source keyed by (thread_id, run_id)."""
def __init__(self, snapshot_ref: str = "source") -> None:
self._ref = snapshot_ref
async def list_changed_runs(self, *, cursor: str | None = None, limit: int = 200) -> RunPage:
return RunPage(items=())
async def get_snapshot(self, *, thread_id: str, run_id: str, limits: EvidenceLimits = EvidenceLimits()) -> CompletedRunSnapshot:
return CompletedRunSnapshot(snapshot_ref=self._ref, run_id=run_id, thread_id=thread_id, owner_id=OWNER, agent_id="lead_agent", origin="interactive", status="success", seal_state="sealed", coverage="lead-journal-v1")
async def read_events(self, *, snapshot_ref: str, cursor: str | None = None, limits: EvidenceLimits = EvidenceLimits()) -> CompletedRunEventPage:
return CompletedRunEventPage()
@pytest.fixture
def service(tmp_path):
cases_dir = tmp_path / "cases"
cases_dir.mkdir()
(cases_dir / "01-arg.json").write_text(
json.dumps({"id": "converter-arg", "description": ARG_CASE.description, "checks": [{"kind": "text_contains", "path": "SKILL.md", "needle": "--format json"}]}),
encoding="utf-8",
)
return EvolutionService(cases_dir=cases_dir)
async def _start(service: EvolutionService, mutations: FakeMutationService) -> None:
deps = ExtensionRuntimeDeps(skill_mutations=mutations, completed_run_evidence=FakeEvidence())
await service.start(deps)
async def test_full_loop_propose_apply_rollback(service):
mutations = FakeMutationService()
await _start(service, mutations)
source = await service.resolve_source(thread_id="t", run_id="r")
record = await service.propose(skill_name=SKILL, candidate_content=FIXED_MAIN, source_ref=source)
assert record.decision == "justified"
assert record.candidate_hash and len(record.report_id) == 64
operation = await service.apply(proposal_id=record.proposal_id)
assert operation.publication == "APPLIED"
assert record.committed is True
assert operation.before_revision.content_digest != operation.after_revision.content_digest
# The committed operation carries the evaluator's assessment.
assert mutations._operations[operation.operation_id].assessment["report_id"] == record.report_id
revert = await service.rollback(operation_id=operation.operation_id)
assert revert.reverts_operation_id == operation.operation_id
assert mutations._files["SKILL.md"].decode() == BASE_MAIN # back to known-good
async def test_apply_refuses_without_justifying_evidence(service):
mutations = FakeMutationService()
await _start(service, mutations)
source = await service.resolve_source(thread_id="t", run_id="r")
# A recorded case the baseline already satisfies: the change fixes nothing
# on record, so the proposal is unjustified and must not commit.
passing_on_both = Case(case_id="scripts-parse", description="bundled scripts parse", checks=(Check("scripts_parse"),))
record = await service.propose(skill_name=SKILL, candidate_content=FIXED_MAIN, source_ref=source, cases=[passing_on_both])
assert record.decision == "unjustified"
assert record.proposal_id in mutations._discarded # dead proposal cleaned up
with pytest.raises(SkillEvolutionError) as exc:
await service.apply(proposal_id=record.proposal_id)
assert exc.value.code == "EVIDENCE_REQUIRED"
async def test_apply_refuses_when_candidate_still_fails(service):
mutations = FakeMutationService()
await _start(service, mutations)
source = await service.resolve_source(thread_id="t", run_id="r")
broken = FIXED_MAIN.replace("--format json", "--format xml")
record = await service.propose(skill_name=SKILL, candidate_content=broken, source_ref=source)
assert record.decision == "failed"
with pytest.raises(SkillEvolutionError) as exc:
await service.apply(proposal_id=record.proposal_id)
assert exc.value.code == "EVIDENCE_REQUIRED"
async def test_apply_blocks_when_host_check_denies(service):
mutations = FakeMutationService(check_decision="block")
await _start(service, mutations)
source = await service.resolve_source(thread_id="t", run_id="r")
record = await service.propose(skill_name=SKILL, candidate_content=FIXED_MAIN, source_ref=source)
assert record.decision == "justified"
with pytest.raises(SkillEvolutionError) as exc:
await service.apply(proposal_id=record.proposal_id)
assert exc.value.code == "HOST_CHECK_DENIED"
async def test_apply_rejects_double_commit(service):
mutations = FakeMutationService()
await _start(service, mutations)
source = await service.resolve_source(thread_id="t", run_id="r")
record = await service.propose(skill_name=SKILL, candidate_content=FIXED_MAIN, source_ref=source)
await service.apply(proposal_id=record.proposal_id)
# A successful apply must flip the record's committed flag; otherwise the
# ALREADY_COMMITTED guard below never fires against a real host.
assert record.committed is True
with pytest.raises(SkillEvolutionError) as exc:
await service.apply(proposal_id=record.proposal_id)
assert exc.value.code == "ALREADY_COMMITTED"
async def test_start_requires_the_grant(service):
with pytest.raises(SkillEvolutionError) as exc:
await service.start(ExtensionRuntimeDeps(skill_mutations=None))
assert exc.value.code == "NOT_GRANTED"
@pytest.mark.parametrize("value", [0, 1, "false", "true", None])
def test_install_rejects_a_non_boolean_enabled_switch(value):
"""A mistyped off switch must fail the deployment instead of staying on."""
with pytest.raises(ValueError, match="config.enabled must be a boolean"):
install(None, {"enabled": value})
def test_install_accepts_an_explicit_off_switch():
install(None, {"enabled": False})
async def test_propose_before_start_raises(service):
with pytest.raises(SkillEvolutionError) as exc:
await service.propose(skill_name=SKILL, candidate_content=FIXED_MAIN, source_ref="source")
assert exc.value.code == "NOT_STARTED"
async def test_resolve_source_requires_evidence_grant(service):
await service.start(ExtensionRuntimeDeps(skill_mutations=FakeMutationService(), completed_run_evidence=None))
with pytest.raises(SkillEvolutionError) as exc:
await service.resolve_source(thread_id="t", run_id="r")
assert exc.value.code == "NO_EVIDENCE"

View File

@ -0,0 +1,330 @@
"""HTTP-surface tests for the walkthrough routes.
The lifecycle tests drive ``EvolutionService`` directly, so they cannot see the
FastAPI plumbing at all. That gap is real: with ``from __future__ import
annotations``, a request model defined inside ``build_router`` becomes an
unresolvable forward reference, FastAPI silently treats ``body`` as a query
parameter, and every POST answers 422 while every service-level test stays
green. These tests go through ``TestClient`` so the request models are actually
resolved, validated, and rejected the way a caller experiences them.
They also pin caller binding: the host capability handles the service holds are
operator-scoped, so the routes — not the host — must refuse to act on another
user's skill.
"""
import asyncio
import json
from deerflow_extension_api import ExtensionPrincipal, ExtensionRuntimeDeps, HostCapabilityError
from deerflow_extension_api.auth import EXTENSION_PRINCIPAL_RESOLVER_KEY
from deerflow_extension_api.run_evidence import RUN_EVIDENCE_READER_RESOLVER_KEY, RunStatusView
from fastapi import FastAPI
from fastapi.testclient import TestClient
from test_lifecycle import OWNER, FakeEvidence, FakeMutationService
from deerflow_extension_skill_evolution.service import EvolutionService, build_router
SKILL = "example-skill"
PREFIX = "/api/extension-skill-evolution"
BASE_MAIN = "---\nname: example-skill\ndescription: demo\n---\n\nRun `scripts/run.py --fmt json`.\n"
FIXED_MAIN = BASE_MAIN.replace("--fmt json", "--format json")
CASE_JSON = json.dumps({"id": "converter-arg", "description": "SKILL.md must instruct --format json", "checks": [{"kind": "text_contains", "path": "SKILL.md", "needle": "--format json"}]})
def _as(client: TestClient, user_id: str | None) -> None:
"""Switch the request principal this client resolves (``None`` = anonymous)."""
def resolver(request):
return None if user_id is None else ExtensionPrincipal(user_id=user_id)
setattr(client.app.state, EXTENSION_PRINCIPAL_RESOLVER_KEY, resolver)
class _MissingEvidence(FakeEvidence):
"""A host whose sealed-run reader answers ``NOT_FOUND`` for every anchor."""
async def get_snapshot(self, **kwargs):
raise HostCapabilityError("NOT_FOUND", "run is unavailable in this scope")
class _ValueErrorOnCheck(FakeMutationService):
"""A host invariant bug raised from ``check`` on the apply route."""
async def check(self, *, proposal_id: str):
raise ValueError("host invariant failed")
class _ValueErrorOnGetOperation(FakeMutationService):
"""A host invariant bug raised before ``revert`` can run."""
async def get_operation(self, *, operation_id: str):
raise ValueError("host invariant failed")
class _InvisibleRunReader:
"""A request-scoped reader for which the anchor run does not exist."""
async def get_run_status(self, *, thread_id: str, run_id: str):
return None
class _VisibleRunReader:
"""A request-scoped reader that confirms the anchor run is visible."""
async def get_run_status(self, *, thread_id: str, run_id: str):
return RunStatusView(thread_id=thread_id, run_id=run_id, status="success")
def _client(
tmp_path,
mutations: FakeMutationService | None = None,
*,
user_id: str | None = OWNER,
case_text: str = CASE_JSON,
evidence: FakeEvidence | None = None,
raise_server_exceptions: bool = True,
) -> tuple[TestClient, FakeMutationService]:
cases_dir = tmp_path / "cases"
cases_dir.mkdir()
(cases_dir / "01-arg.json").write_text(case_text, encoding="utf-8")
service = EvolutionService(cases_dir=cases_dir)
host = mutations if mutations is not None else FakeMutationService()
asyncio.run(service.start(ExtensionRuntimeDeps(skill_mutations=host, completed_run_evidence=evidence if evidence is not None else FakeEvidence())))
app = FastAPI()
app.include_router(build_router(service))
client = TestClient(app, raise_server_exceptions=raise_server_exceptions)
_as(client, user_id)
return client, host
def _propose(client: TestClient, **overrides):
body = {"skill_name": SKILL, "candidate_content": FIXED_MAIN, "thread_id": "t", "run_id": "r"} | overrides
return client.post(f"{PREFIX}/proposals", json=body)
def test_proposals_route_accepts_a_json_body(tmp_path):
"""A documented request body must reach the service, not become a query param."""
client, _host = _client(tmp_path)
response = _propose(client)
assert response.status_code == 200, response.text
payload = response.json()
assert payload["decision"] == "justified"
assert len(payload["report_id"]) == 64
def test_proposals_route_rejects_an_incomplete_body(tmp_path):
client, _host = _client(tmp_path)
response = client.post(f"{PREFIX}/proposals", json={"skill_name": SKILL, "candidate_content": FIXED_MAIN})
assert response.status_code == 422
def test_proposals_route_forbids_unknown_fields(tmp_path):
client, _host = _client(tmp_path)
response = _propose(client, unexpected="value")
assert response.status_code == 422
def test_apply_route_marks_the_proposal_committed_and_refuses_a_second_apply(tmp_path):
client, _host = _client(tmp_path)
proposal_id = _propose(client).json()["proposal_id"]
applied = client.post(f"{PREFIX}/proposals/{proposal_id}/apply", json={})
assert applied.status_code == 200, applied.text
assert applied.json()["publication"] == "APPLIED"
listing = client.get(f"{PREFIX}/proposals").json()["proposals"]
committed = {item["proposal_id"]: item["committed"] for item in listing}
assert committed[proposal_id] is True
again = client.post(f"{PREFIX}/proposals/{proposal_id}/apply", json={})
assert again.status_code == 409
assert again.json()["detail"].startswith("ALREADY_COMMITTED")
def test_apply_route_returns_409_for_an_unevidenced_change(tmp_path):
client, _host = _client(tmp_path)
broken = FIXED_MAIN.replace("--format json", "--format xml")
proposal = _propose(client, candidate_content=broken).json()
assert proposal["decision"] == "failed"
response = client.post(f"{PREFIX}/proposals/{proposal['proposal_id']}/apply", json={})
assert response.status_code == 409
assert response.json()["detail"].startswith("EVIDENCE_REQUIRED")
def test_revert_route_is_append_only(tmp_path):
client, host = _client(tmp_path)
proposal_id = _propose(client).json()["proposal_id"]
operation_id = client.post(f"{PREFIX}/proposals/{proposal_id}/apply", json={}).json()["operation_id"]
reverted = client.post(f"{PREFIX}/operations/{operation_id}/revert", json={})
assert reverted.status_code == 200, reverted.text
assert reverted.json()["reverts"] == operation_id
assert host._files["SKILL.md"].decode() == BASE_MAIN
def test_unknown_proposal_returns_404(tmp_path):
client, _host = _client(tmp_path)
response = client.get(f"{PREFIX}/proposals/does-not-exist")
assert response.status_code == 404
def test_unknown_operation_returns_404(tmp_path):
"""``NOT_FOUND_OR_FORBIDDEN`` from the real host is a 404, not a conflict."""
client, _host = _client(tmp_path)
response = client.post(f"{PREFIX}/operations/does-not-exist/revert", json={})
assert response.status_code == 404
def test_apply_unknown_proposal_returns_404(tmp_path):
"""A proposal this service never staged is absent, not a conflict."""
client, _host = _client(tmp_path)
response = client.post(f"{PREFIX}/proposals/does-not-exist/apply", json={})
assert response.status_code == 404
def test_unknown_evidence_run_returns_404(tmp_path):
"""``NOT_FOUND`` from the sealed-run reader is a 404 as well."""
client, _host = _client(tmp_path, evidence=_MissingEvidence())
response = _propose(client)
assert response.status_code == 404
def test_health_route_reports_the_evaluator(tmp_path):
client, _host = _client(tmp_path)
payload = client.get(f"{PREFIX}/health").json()
assert payload["ready"] is True
assert payload["evaluator"]
assert payload["evaluator_version"]
assert payload["cases"] == 1
def test_routes_require_an_authenticated_principal(tmp_path):
client, _host = _client(tmp_path, user_id=None)
assert client.get(f"{PREFIX}/proposals").status_code == 401
assert _propose(client).status_code == 401
def test_propose_refuses_a_skill_owned_by_someone_else(tmp_path):
"""The injected host capability is allowlist-scoped, so the route binds the owner."""
client, _host = _client(tmp_path, user_id="someone-else")
response = _propose(client)
assert response.status_code == 403
assert response.json()["detail"].startswith("FORBIDDEN")
def test_proposals_are_scoped_to_their_owner(tmp_path):
client, _host = _client(tmp_path)
proposal_id = _propose(client).json()["proposal_id"]
_as(client, "someone-else")
assert client.get(f"{PREFIX}/proposals").json()["proposals"] == []
# Another owner's proposal is not disclosed — not even its existence.
assert client.get(f"{PREFIX}/proposals/{proposal_id}").status_code == 404
applied = client.post(f"{PREFIX}/proposals/{proposal_id}/apply", json={})
assert applied.status_code == 403
assert applied.json()["detail"].startswith("FORBIDDEN")
def test_revert_is_scoped_to_the_operation_owner(tmp_path):
client, _host = _client(tmp_path)
proposal_id = _propose(client).json()["proposal_id"]
operation_id = client.post(f"{PREFIX}/proposals/{proposal_id}/apply", json={}).json()["operation_id"]
_as(client, "someone-else")
response = client.post(f"{PREFIX}/operations/{operation_id}/revert", json={})
assert response.status_code == 403
assert response.json()["detail"].startswith("FORBIDDEN")
def test_a_broken_case_file_is_a_400_diagnostic(tmp_path):
"""A malformed operator case file fails closed, but not as a 500 traceback."""
client, _host = _client(tmp_path, case_text="{ not json")
health = client.get(f"{PREFIX}/health")
proposed = _propose(client)
assert health.status_code == 400
assert health.json()["detail"].startswith("CASE_INVALID")
assert proposed.status_code == 400
assert proposed.json()["detail"].startswith("CASE_INVALID")
def test_apply_does_not_label_a_host_value_error_as_case_invalid(tmp_path):
"""Apply never reads the case files, so a host bug must surface as a 500."""
client, _host = _client(tmp_path, mutations=_ValueErrorOnCheck(), raise_server_exceptions=False)
proposal_id = _propose(client).json()["proposal_id"]
response = client.post(f"{PREFIX}/proposals/{proposal_id}/apply", json={})
assert response.status_code == 500
def test_revert_does_not_label_a_host_value_error_as_case_invalid(tmp_path):
"""Revert never reads the case files either."""
client, _host = _client(tmp_path, mutations=_ValueErrorOnGetOperation(), raise_server_exceptions=False)
response = client.post(f"{PREFIX}/operations/does-not-matter/revert", json={})
assert response.status_code == 500
def test_hidden_run_anchor_is_403_before_the_host_is_touched(tmp_path):
"""The request-scoped reader is the production path; None means not visible."""
client, host = _client(tmp_path)
setattr(client.app.state, RUN_EVIDENCE_READER_RESOLVER_KEY, lambda request: _InvisibleRunReader())
response = _propose(client)
assert response.status_code == 403
assert response.json()["detail"].startswith("SOURCE_NOT_VISIBLE")
assert host._proposals == {}
def test_denied_run_reader_is_403(tmp_path):
"""The Gateway's resolver raises PermissionError without ``runs:read``."""
client, _host = _client(tmp_path)
def denied(request):
raise PermissionError("run evidence requires runs:read")
setattr(client.app.state, RUN_EVIDENCE_READER_RESOLVER_KEY, denied)
response = _propose(client)
assert response.status_code == 403
def test_visible_run_anchor_still_reaches_the_service(tmp_path):
"""A visible anchor must not be blocked by the new reader check."""
client, _host = _client(tmp_path)
setattr(client.app.state, RUN_EVIDENCE_READER_RESOLVER_KEY, lambda request: _VisibleRunReader())
response = _propose(client)
assert response.status_code == 200

View File

@ -583,6 +583,57 @@
},
"docs_url": "https://github.com/bytedance/deer-flow/tree/main/examples/deerflow-extension-retrieval#readme"
},
"deerflow-extension-skill-evolution": {
"schema_version": 1,
"id": "catalog/skillEvolution",
"locales": {
"zh-CN": {
"title": "证据驱动的技能进化",
"description": "在回归用例与宿主校验通过后才提议技能修改,对比新旧结果,并可回滚到已确认的版本。",
"usage": [
{
"title": "安装并授权",
"description": "通过 make extension-install 安装示例,在 config.yaml 中配置 skill mutation 与 evidence 授权后重启 Gateway。"
},
{
"title": "记录回归用例",
"description": "在包的 cases 目录添加 JSON 用例,用 file_exists、text_contains、text_not_contains 等静态检查描述这次修改要修复什么。"
},
{
"title": "提出、评估、应用与回滚",
"description": "调用 /api/extension-skill-evolution/proposals 提交候选版本,查看 justified、unjustified 或 failed 结论;通过后 apply,必要时 revert 到已确认版本。"
}
],
"requirements": [
"需要支持 skill_mutations 的持久化数据库、run_events.backend: db,以及单实例 POSIX 本地磁盘。",
"需要在 config.yaml 中显式授予 skill mutation 与 evidence 访问权限。"
]
},
"en-US": {
"title": "Evidence-gated skill evolution",
"description": "Propose skill edits only after a regression case and host checks pass, compare old and new results, and roll back to a known-good revision.",
"usage": [
{
"title": "Install and grant access",
"description": "Install the example with make extension-install, configure the skill mutation and evidence grants in config.yaml, then restart Gateway."
},
{
"title": "Record a regression case",
"description": "Add a JSON case under the package's cases directory using static checks such as file_exists, text_contains and text_not_contains to state what the change fixes."
},
{
"title": "Propose, evaluate, apply and roll back",
"description": "Submit a candidate to /api/extension-skill-evolution/proposals, inspect the justified, unjustified or failed verdict, apply an accepted revision, and revert when needed."
}
],
"requirements": [
"Requires a durable database backend, run_events.backend: db, and a single POSIX host with local disk.",
"Requires explicit skill mutation and evidence grants in config.yaml."
]
}
},
"docs_url": "https://github.com/bytedance/deer-flow/tree/main/examples/deerflow-extension-skill-evolution#readme"
},
"deerflow-extension-weknora": {
"schema_version": 1,
"id": "catalog/weknora",

View File

@ -78,6 +78,11 @@ export const extensionCatalog = [
package: "deerflow-extension-knowledge-center",
namespace: "knowledge-center",
},
{
key: "skillEvolution",
package: "deerflow-extension-skill-evolution",
namespace: null,
},
{ key: "lightrag", package: "deerflow-extension-lightrag", namespace: null },
{ key: "weknora", package: "deerflow-extension-weknora", namespace: null },
{ key: "example", package: "deerflow-extension-example", namespace: null },

View File

@ -200,6 +200,10 @@ export const enUS: Translations = {
"deerflow-extension-knowledge-center",
"en-US",
),
skillEvolution: bundledCopy(
"deerflow-extension-skill-evolution",
"en-US",
),
lightrag: bundledCopy("deerflow-extension-lightrag", "en-US"),
weknora: bundledCopy("deerflow-extension-weknora", "en-US"),
example: bundledCopy("deerflow-extension-example", "en-US"),

View File

@ -153,6 +153,7 @@ export interface Translations {
| "knowledgeManagement"
| "knowledgePlatform"
| "knowledgeCenter"
| "skillEvolution"
| "weknora"
| "lightrag"
| "example",

View File

@ -188,6 +188,10 @@ export const zhCN: Translations = {
"deerflow-extension-knowledge-center",
"zh-CN",
),
skillEvolution: bundledCopy(
"deerflow-extension-skill-evolution",
"zh-CN",
),
lightrag: bundledCopy("deerflow-extension-lightrag", "zh-CN"),
weknora: bundledCopy("deerflow-extension-weknora", "zh-CN"),
example: bundledCopy("deerflow-extension-example", "zh-CN"),