Loveca db02b77837
feat(plugins): add evidence-gated skill self-evolution example (#7984)
* feat(plugins): add evidence-gated skill self-evolution example

Demonstrates the #7884 roadmap deliverable: a proposed change, the regression
case it fixes, the old and new results, and a rollback to a known-good version,
driven only through the public SkillMutationService contract merged in #5645.

The example stages a candidate against a sealed completed-run evidence anchor,
evaluates recorded JSON regression cases against both frozen bundles with a
deterministic static evaluator (never executing package content), and commits
only a "justified" verdict with an AssessmentRef binding the report to the
exact candidate hash and base revision. Unjustified or failed proposals are
discarded; revert stays append-only. Ships an HTTP walkthrough surface, an
offline lifecycle test against a contract-faithful fake host, and a README
section proposing a concrete evidence standard for the automatic-publication
discussion.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(skill-evolution): resolve request models and honor APPLIED publication

The walkthrough routes had never been exercised through FastAPI. With
`from __future__ import annotations`, request models defined inside
`build_router` are unresolvable forward references, so FastAPI read `body`
as a query parameter and every POST answered 422. Move the models to module
scope and cover the routes through TestClient.

A committed operation reports `publication == "APPLIED"`; `READY` is a
`views` state. The committed check listed READY, so a successful apply never
flipped `record.committed` and the ALREADY_COMMITTED guard never fired. Fix
the check, mirror the real literals in the lifecycle fake host, and assert
both the flag and the second-apply refusal.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(skill-evolution): address review feedback and register the example catalog

- Bind walkthrough routes to the authenticated principal: owner checks on propose/apply/revert, owner-scoped listing, and non-disclosing 404s.
- Map unknown skills, proposals and operations to 404 for both NOT_FOUND and the real host's NOT_FOUND_OR_FORBIDDEN; map broken case files to 400.
- Ignore URL paths and trim sentence punctuation when extracting in-package references.
- Validate config.enabled's type instead of treating any non-False value as on.
- Soften the README claim about report persistence and add CHANGELOG entries.
- Register the example in the frontend extension catalog, fixing frontend-unit-tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(skill-evolution): narrow case-file error mapping and cover run visibility

- Label ValueError as CASE_INVALID only on POST /proposals, the route that actually loads case files; apply and revert now surface host bugs as 500 instead of pointing at the case directory.
- Cover the request-scoped run-evidence reader the real Gateway installs: hidden anchor returns 403 SOURCE_NOT_VISIBLE before the host is touched, a denied resolver returns 403, and a visible anchor still proceeds.
- Add regressions pinning host ValueError on apply/revert to 500.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* docs(changelog): drop contributor changelog entry per policy

Contributor PRs no longer modify CHANGELOG.md or CHANGELOG_zh.md (#8202);
maintainers add the entry at merge time.

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
2026-11-16 16:04:42 +08:00

8.9 KiB

Skill self-evolution example — evidence-gated revisions

An opt-in Python plugin that demonstrates the roadmap deliverable from #7884 ("Skills, extensions, and reproducible agents"): a proposed change, the regression case it fixes, the old and new results, and a rollback to a known-good version — driven entirely through the host SkillMutationService contract merged in #5645.

It uses extension API 0.2.11 with no host code changes. The host owns storage, security scanning, revision identity and authorization; this package owns only one policy: a revision commits only when a recorded regression case demonstrates it fixes something.

Why evidence-gated

With skill_evolution.enabled, the agent can already rewrite its own skills through skill_manage. What is missing is a visible answer to "was that change an improvement?". This example attaches one to every commit:

propose ──> stage()          freeze baseline + candidate bundles (host-side)
evaluate ─> recorded cases    run static checks against BOTH bundles
apply ────> check() + commit(assessment_ref)   host security gate, then commit
                                                  with the verdict attached
rollback ─> revert()          append-only return to the known-good revision

The AssessmentRef carried into commit binds the evidence to the exact proposal (candidate_hash + base_revision); the host rejects a mismatched assessment with INVALID_ASSESSMENT. Afterwards every committed Operation answers: what changed, which recorded cases justified it, and how to undo it.

Recorded regression cases

Cases are JSON files in deerflow_extension_skill_evolution/cases/ (override with config.cases_dir). One case per file:

{
  "id": "converter-arg",
  "description": "SKILL.md must instruct the converter with --format json",
  "checks": [
    { "kind": "text_contains", "path": "SKILL.md", "needle": "--format json" },
    { "kind": "text_not_contains", "path": "SKILL.md", "needle": "--fmt " }
  ]
}

Check kinds: file_exists, text_contains, text_not_contains, referenced_files_exist (bare scripts/… / assets/… tokens in a text file must exist in the package), and scripts_parse (every scripts/*.py must pass ast.parse).

The evaluator never executes package content. The candidate is an agent-proposed revision that has not passed the host security check yet, so the only safe operations are structural (presence, text, parse-only). A malformed case file raises and stops the evaluation rather than silently narrowing it.

Verdicts

Verdict Meaning Service behavior
justified every case passes on the candidate and at least one fails on the baseline eligible for apply
unjustified every case passes on both sides — nothing recorded was broken proposal discarded, apply refuses
failed some case still fails on the candidate (or none recorded) proposal discarded, apply refuses

Install and configure

From the DeerFlow checkout root:

make extension-install SOURCE=examples/deerflow-extension-skill-evolution

The mutation capability requires an explicit operator grant in config.yaml (restart required). Prerequisites are the same as any skill_mutations consumer: durable database backend, run_events.backend: db, and a single POSIX host with local disk:

plugins:
  - name: skill-evolution
    package: deerflow-extension-skill-evolution
    use: deerflow_extension_skill_evolution:install
    enabled: true
    host_access:
      evidence:
        owners: ["your-user-id"]
      skill_mutations:
        owners: ["your-user-id"]          # subset of evidence.owners
        operations: [stage, check, commit, revert]
        trigger_agents: [lead_agent]
        target_skills: [example-skill]    # canonical names this plugin may revise
        topology: single_host_local
    config:
      cases_dir: null                     # optional: default bundled cases

Without the grant the service fails to start with a NOT_GRANTED diagnostic; nothing else in the deployment changes.

HTTP surface

All routes are under /api/extension-skill-evolution:

Route Purpose
GET /health readiness, evaluator identity, recorded case count
POST /proposals resolve the sealed run evidence, stage a candidate, evaluate, return the verdict + report
GET /proposals · GET /proposals/{id} list / inspect proposals and their reports
POST /proposals/{id}/apply host check, then commit with the AssessmentRef attached
POST /operations/{id}/revert roll back a committed revision

POST /proposals takes {skill_name, candidate_content, thread_id, run_id}. The run must be completed and sealed — it is the evidence anchor the host requires for staging (source_refs), so every proposal records which run motivated the change.

Every route binds the caller. The capability handles injected into the service are scoped to the extension by the operator's grant, not to the requesting user, so the routes resolve the host principal (deerflow_extension_api.auth.resolve_principal) and refuse to act on a skill owned by anyone else: propose, apply and revert answer 401 without a principal and 403 (FORBIDDEN) across owners, and GET /proposals lists only the caller's own proposals (another owner's proposal reads as 404, without disclosing that it exists). When the host exposes a request-scoped run-evidence reader, the evidence anchor must also be visible to the caller; hosts without one fall back to the owner check.

Offline walkthrough (tests)

The package ships the demonstration as tests; no Gateway, model, or network:

cd examples/deerflow-extension-skill-evolution
uv run --no-project --python 3.12 --with pytest --with pytest-asyncio \
    --with ../../backend/packages/extension-api \
    --with-editable . python -m pytest tests/ -q

tests/test_lifecycle.py drives the full loop against a fake host that implements only the public contract: propose a fix for a broken converter argument → verdict justified → apply → the committed operation carries the report id → revert → the skill is byte-identical to the known-good baseline. It also pins the refusal paths: unjustified and failed proposals are discarded and cannot commit, a denied host check blocks the commit, and double-commit is rejected.

Scope and boundaries

  • This is an example, not a production engine. The evaluator is a deterministic static checker; it has no live-model runs, metrics, or signal collection.
  • #6954 proposes a distributable evolution plugin package with its own snapshot/versioning design; this example stops at the merged #5645 contract and should be revisited against that RFC if it lands.
  • #7066 (eval-gated evolution) owns the broader signal → attribution → candidate → verification loop; the minimal evaluator here is deliberately smaller than that scope.
  • The skill_manage direct-write path is unchanged. This plugin is additive and opt-in.

A concrete evidence standard (input for the roadmap discussion)

The roadmap asks what evidence should be required before enabling automatic publication. Rather than propose a standard in the abstract, this example makes one concrete and reviewable:

  1. Every proposal names its motivating run. Staging requires sealed completed-run evidence (source_refs); the host rejects stale or unsealed anchors.
  2. Evidence is recorded, not narrated. A case is a versioned JSON file with deterministic checks; the verdict is computed from the frozen baseline and candidate bundles, not from the proposal text.
  3. The verdict travels with the commit. AssessmentRef pins evaluator, evaluator_version, and a content-addressed report_id (SHA-256 over the canonical report JSON) to the exact candidate_hash and base_revision. The report bytes themselves are held in this service's in-memory cache, not persisted; the id lets anyone holding the report — or recomputing it from the same cases and frozen bundles — verify that it is the one this commit cites.
  4. Fail-closed by default. unjustified and failed proposals are discarded before they can commit; a host check denial blocks the commit; an assessment mismatch is rejected by the host itself.
  5. Rollback is as auditable as the change. revert appends a new operation referencing the one it undoes.

Automatic publication would additionally need: case authorship provenance (who recorded the case, and was it derived from the same run that motivated the change), a minimum case count, and a policy for unjustified proposals that keep recurring. Those are deliberately out of scope here.