* feat(plugins): add evidence-gated skill self-evolution example Demonstrates the #7884 roadmap deliverable: a proposed change, the regression case it fixes, the old and new results, and a rollback to a known-good version, driven only through the public SkillMutationService contract merged in #5645. The example stages a candidate against a sealed completed-run evidence anchor, evaluates recorded JSON regression cases against both frozen bundles with a deterministic static evaluator (never executing package content), and commits only a "justified" verdict with an AssessmentRef binding the report to the exact candidate hash and base revision. Unjustified or failed proposals are discarded; revert stays append-only. Ships an HTTP walkthrough surface, an offline lifecycle test against a contract-faithful fake host, and a README section proposing a concrete evidence standard for the automatic-publication discussion. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(skill-evolution): resolve request models and honor APPLIED publication The walkthrough routes had never been exercised through FastAPI. With `from __future__ import annotations`, request models defined inside `build_router` are unresolvable forward references, so FastAPI read `body` as a query parameter and every POST answered 422. Move the models to module scope and cover the routes through TestClient. A committed operation reports `publication == "APPLIED"`; `READY` is a `views` state. The committed check listed READY, so a successful apply never flipped `record.committed` and the ALREADY_COMMITTED guard never fired. Fix the check, mirror the real literals in the lifecycle fake host, and assert both the flag and the second-apply refusal. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(skill-evolution): address review feedback and register the example catalog - Bind walkthrough routes to the authenticated principal: owner checks on propose/apply/revert, owner-scoped listing, and non-disclosing 404s. - Map unknown skills, proposals and operations to 404 for both NOT_FOUND and the real host's NOT_FOUND_OR_FORBIDDEN; map broken case files to 400. - Ignore URL paths and trim sentence punctuation when extracting in-package references. - Validate config.enabled's type instead of treating any non-False value as on. - Soften the README claim about report persistence and add CHANGELOG entries. - Register the example in the frontend extension catalog, fixing frontend-unit-tests. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(skill-evolution): narrow case-file error mapping and cover run visibility - Label ValueError as CASE_INVALID only on POST /proposals, the route that actually loads case files; apply and revert now surface host bugs as 500 instead of pointing at the case directory. - Cover the request-scoped run-evidence reader the real Gateway installs: hidden anchor returns 403 SOURCE_NOT_VISIBLE before the host is touched, a denied resolver returns 403, and a visible anchor still proceeds. - Add regressions pinning host ValueError on apply/revert to 500. Co-Authored-By: Claude Code <noreply@anthropic.com> * docs(changelog): drop contributor changelog entry per policy Contributor PRs no longer modify CHANGELOG.md or CHANGELOG_zh.md (#8202); maintainers add the entry at merge time. --------- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: Nan Gao <thunderggnn@gmail.com>
Skill self-evolution example — evidence-gated revisions
An opt-in Python plugin that demonstrates the roadmap deliverable from
#7884 ("Skills,
extensions, and reproducible agents"): a proposed change, the regression
case it fixes, the old and new results, and a rollback to a known-good
version — driven entirely through the host SkillMutationService contract
merged in #5645.
It uses extension API 0.2.11 with no host code changes. The host owns storage, security scanning, revision identity and authorization; this package owns only one policy: a revision commits only when a recorded regression case demonstrates it fixes something.
Why evidence-gated
With skill_evolution.enabled, the agent can already rewrite its own skills
through skill_manage. What is missing is a visible answer to "was that change
an improvement?". This example attaches one to every commit:
propose ──> stage() freeze baseline + candidate bundles (host-side)
evaluate ─> recorded cases run static checks against BOTH bundles
apply ────> check() + commit(assessment_ref) host security gate, then commit
with the verdict attached
rollback ─> revert() append-only return to the known-good revision
The AssessmentRef carried into commit binds the evidence to the exact
proposal (candidate_hash + base_revision); the host rejects a mismatched
assessment with INVALID_ASSESSMENT. Afterwards every committed Operation
answers: what changed, which recorded cases justified it, and how to undo it.
Recorded regression cases
Cases are JSON files in deerflow_extension_skill_evolution/cases/ (override
with config.cases_dir). One case per file:
{
"id": "converter-arg",
"description": "SKILL.md must instruct the converter with --format json",
"checks": [
{ "kind": "text_contains", "path": "SKILL.md", "needle": "--format json" },
{ "kind": "text_not_contains", "path": "SKILL.md", "needle": "--fmt " }
]
}
Check kinds: file_exists, text_contains, text_not_contains,
referenced_files_exist (bare scripts/… / assets/… tokens in a text file
must exist in the package), and scripts_parse (every scripts/*.py must pass
ast.parse).
The evaluator never executes package content. The candidate is an
agent-proposed revision that has not passed the host security check yet, so
the only safe operations are structural (presence, text, parse-only). A
malformed case file raises and stops the evaluation rather than silently
narrowing it.
Verdicts
| Verdict | Meaning | Service behavior |
|---|---|---|
justified |
every case passes on the candidate and at least one fails on the baseline | eligible for apply |
unjustified |
every case passes on both sides — nothing recorded was broken | proposal discarded, apply refuses |
failed |
some case still fails on the candidate (or none recorded) | proposal discarded, apply refuses |
Install and configure
From the DeerFlow checkout root:
make extension-install SOURCE=examples/deerflow-extension-skill-evolution
The mutation capability requires an explicit operator grant in config.yaml
(restart required). Prerequisites are the same as any skill_mutations
consumer: durable database backend, run_events.backend: db, and a single
POSIX host with local disk:
plugins:
- name: skill-evolution
package: deerflow-extension-skill-evolution
use: deerflow_extension_skill_evolution:install
enabled: true
host_access:
evidence:
owners: ["your-user-id"]
skill_mutations:
owners: ["your-user-id"] # subset of evidence.owners
operations: [stage, check, commit, revert]
trigger_agents: [lead_agent]
target_skills: [example-skill] # canonical names this plugin may revise
topology: single_host_local
config:
cases_dir: null # optional: default bundled cases
Without the grant the service fails to start with a NOT_GRANTED diagnostic;
nothing else in the deployment changes.
HTTP surface
All routes are under /api/extension-skill-evolution:
| Route | Purpose |
|---|---|
GET /health |
readiness, evaluator identity, recorded case count |
POST /proposals |
resolve the sealed run evidence, stage a candidate, evaluate, return the verdict + report |
GET /proposals · GET /proposals/{id} |
list / inspect proposals and their reports |
POST /proposals/{id}/apply |
host check, then commit with the AssessmentRef attached |
POST /operations/{id}/revert |
roll back a committed revision |
POST /proposals takes {skill_name, candidate_content, thread_id, run_id}.
The run must be completed and sealed — it is the evidence anchor the host
requires for staging (source_refs), so every proposal records which run
motivated the change.
Every route binds the caller. The capability handles injected into the
service are scoped to the extension by the operator's grant, not to the
requesting user, so the routes resolve the host principal
(deerflow_extension_api.auth.resolve_principal) and refuse to act on a skill
owned by anyone else: propose, apply and revert answer 401 without a
principal and 403 (FORBIDDEN) across owners, and GET /proposals lists only
the caller's own proposals (another owner's proposal reads as 404, without
disclosing that it exists). When the host exposes a request-scoped run-evidence
reader, the evidence anchor must also be visible to the caller; hosts without
one fall back to the owner check.
Offline walkthrough (tests)
The package ships the demonstration as tests; no Gateway, model, or network:
cd examples/deerflow-extension-skill-evolution
uv run --no-project --python 3.12 --with pytest --with pytest-asyncio \
--with ../../backend/packages/extension-api \
--with-editable . python -m pytest tests/ -q
tests/test_lifecycle.py drives the full loop against a fake host that
implements only the public contract: propose a fix for a broken converter
argument → verdict justified → apply → the committed operation carries the
report id → revert → the skill is byte-identical to the known-good baseline.
It also pins the refusal paths: unjustified and failed proposals are
discarded and cannot commit, a denied host check blocks the commit, and
double-commit is rejected.
Scope and boundaries
- This is an example, not a production engine. The evaluator is a deterministic static checker; it has no live-model runs, metrics, or signal collection.
- #6954 proposes a distributable evolution plugin package with its own snapshot/versioning design; this example stops at the merged #5645 contract and should be revisited against that RFC if it lands.
- #7066 (eval-gated evolution) owns the broader signal → attribution → candidate → verification loop; the minimal evaluator here is deliberately smaller than that scope.
- The
skill_managedirect-write path is unchanged. This plugin is additive and opt-in.
A concrete evidence standard (input for the roadmap discussion)
The roadmap asks what evidence should be required before enabling automatic publication. Rather than propose a standard in the abstract, this example makes one concrete and reviewable:
- Every proposal names its motivating run. Staging requires sealed
completed-run evidence (
source_refs); the host rejects stale or unsealed anchors. - Evidence is recorded, not narrated. A case is a versioned JSON file with deterministic checks; the verdict is computed from the frozen baseline and candidate bundles, not from the proposal text.
- The verdict travels with the commit.
AssessmentRefpinsevaluator,evaluator_version, and a content-addressedreport_id(SHA-256 over the canonical report JSON) to the exactcandidate_hashandbase_revision. The report bytes themselves are held in this service's in-memory cache, not persisted; the id lets anyone holding the report — or recomputing it from the same cases and frozen bundles — verify that it is the one this commit cites. - Fail-closed by default.
unjustifiedandfailedproposals are discarded before they can commit; a hostcheckdenial blocks the commit; an assessment mismatch is rejected by the host itself. - Rollback is as auditable as the change.
revertappends a new operation referencing the one it undoes.
Automatic publication would additionally need: case authorship provenance (who
recorded the case, and was it derived from the same run that motivated the
change), a minimum case count, and a policy for unjustified proposals that
keep recurring. Those are deliberately out of scope here.