mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-16 16:31:18 +00:00
* feat(plugins): add evidence-gated skill self-evolution example Demonstrates the #7884 roadmap deliverable: a proposed change, the regression case it fixes, the old and new results, and a rollback to a known-good version, driven only through the public SkillMutationService contract merged in #5645. The example stages a candidate against a sealed completed-run evidence anchor, evaluates recorded JSON regression cases against both frozen bundles with a deterministic static evaluator (never executing package content), and commits only a "justified" verdict with an AssessmentRef binding the report to the exact candidate hash and base revision. Unjustified or failed proposals are discarded; revert stays append-only. Ships an HTTP walkthrough surface, an offline lifecycle test against a contract-faithful fake host, and a README section proposing a concrete evidence standard for the automatic-publication discussion. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(skill-evolution): resolve request models and honor APPLIED publication The walkthrough routes had never been exercised through FastAPI. With `from __future__ import annotations`, request models defined inside `build_router` are unresolvable forward references, so FastAPI read `body` as a query parameter and every POST answered 422. Move the models to module scope and cover the routes through TestClient. A committed operation reports `publication == "APPLIED"`; `READY` is a `views` state. The committed check listed READY, so a successful apply never flipped `record.committed` and the ALREADY_COMMITTED guard never fired. Fix the check, mirror the real literals in the lifecycle fake host, and assert both the flag and the second-apply refusal. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(skill-evolution): address review feedback and register the example catalog - Bind walkthrough routes to the authenticated principal: owner checks on propose/apply/revert, owner-scoped listing, and non-disclosing 404s. - Map unknown skills, proposals and operations to 404 for both NOT_FOUND and the real host's NOT_FOUND_OR_FORBIDDEN; map broken case files to 400. - Ignore URL paths and trim sentence punctuation when extracting in-package references. - Validate config.enabled's type instead of treating any non-False value as on. - Soften the README claim about report persistence and add CHANGELOG entries. - Register the example in the frontend extension catalog, fixing frontend-unit-tests. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(skill-evolution): narrow case-file error mapping and cover run visibility - Label ValueError as CASE_INVALID only on POST /proposals, the route that actually loads case files; apply and revert now surface host bugs as 500 instead of pointing at the case directory. - Cover the request-scoped run-evidence reader the real Gateway installs: hidden anchor returns 403 SOURCE_NOT_VISIBLE before the host is touched, a denied resolver returns 403, and a visible anchor still proceeds. - Add regressions pinning host ValueError on apply/revert to 500. Co-Authored-By: Claude Code <noreply@anthropic.com> * docs(changelog): drop contributor changelog entry per policy Contributor PRs no longer modify CHANGELOG.md or CHANGELOG_zh.md (#8202); maintainers add the entry at merge time. --------- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: Nan Gao <thunderggnn@gmail.com>
195 lines
8.9 KiB
Markdown
195 lines
8.9 KiB
Markdown
# Skill self-evolution example — evidence-gated revisions
|
|
|
|
An opt-in Python plugin that demonstrates the roadmap deliverable from
|
|
[#7884](https://github.com/bytedance/deer-flow/issues/7884) ("Skills,
|
|
extensions, and reproducible agents"): a **proposed change**, the **regression
|
|
case it fixes**, the **old and new results**, and a **rollback to a known-good
|
|
version** — driven entirely through the host `SkillMutationService` contract
|
|
merged in [#5645](https://github.com/bytedance/deer-flow/pull/5645).
|
|
|
|
It uses extension API 0.2.11 with no host code changes. The host owns storage,
|
|
security scanning, revision identity and authorization; this package owns only
|
|
one policy: **a revision commits only when a recorded regression case
|
|
demonstrates it fixes something**.
|
|
|
|
## Why evidence-gated
|
|
|
|
With `skill_evolution.enabled`, the agent can already rewrite its own skills
|
|
through `skill_manage`. What is missing is a visible answer to "was that change
|
|
an improvement?". This example attaches one to every commit:
|
|
|
|
```
|
|
propose ──> stage() freeze baseline + candidate bundles (host-side)
|
|
evaluate ─> recorded cases run static checks against BOTH bundles
|
|
apply ────> check() + commit(assessment_ref) host security gate, then commit
|
|
with the verdict attached
|
|
rollback ─> revert() append-only return to the known-good revision
|
|
```
|
|
|
|
The `AssessmentRef` carried into `commit` binds the evidence to the exact
|
|
proposal (`candidate_hash` + `base_revision`); the host rejects a mismatched
|
|
assessment with `INVALID_ASSESSMENT`. Afterwards every committed `Operation`
|
|
answers: what changed, which recorded cases justified it, and how to undo it.
|
|
|
|
## Recorded regression cases
|
|
|
|
Cases are JSON files in `deerflow_extension_skill_evolution/cases/` (override
|
|
with `config.cases_dir`). One case per file:
|
|
|
|
```json
|
|
{
|
|
"id": "converter-arg",
|
|
"description": "SKILL.md must instruct the converter with --format json",
|
|
"checks": [
|
|
{ "kind": "text_contains", "path": "SKILL.md", "needle": "--format json" },
|
|
{ "kind": "text_not_contains", "path": "SKILL.md", "needle": "--fmt " }
|
|
]
|
|
}
|
|
````
|
|
|
|
Check kinds: `file_exists`, `text_contains`, `text_not_contains`,
|
|
`referenced_files_exist` (bare `scripts/…` / `assets/…` tokens in a text file
|
|
must exist in the package), and `scripts_parse` (every `scripts/*.py` must pass
|
|
`ast.parse`).
|
|
|
|
**The evaluator never executes package content.** The candidate is an
|
|
agent-proposed revision that has not passed the host security `check` yet, so
|
|
the only safe operations are structural (presence, text, parse-only). A
|
|
malformed case file raises and stops the evaluation rather than silently
|
|
narrowing it.
|
|
|
|
### Verdicts
|
|
|
|
| Verdict | Meaning | Service behavior |
|
|
| --- | --- | --- |
|
|
| `justified` | every case passes on the candidate and at least one fails on the baseline | eligible for `apply` |
|
|
| `unjustified` | every case passes on both sides — nothing recorded was broken | proposal discarded, `apply` refuses |
|
|
| `failed` | some case still fails on the candidate (or none recorded) | proposal discarded, `apply` refuses |
|
|
|
|
## Install and configure
|
|
|
|
From the DeerFlow checkout root:
|
|
|
|
```sh
|
|
make extension-install SOURCE=examples/deerflow-extension-skill-evolution
|
|
```
|
|
|
|
The mutation capability requires an explicit operator grant in `config.yaml`
|
|
(restart required). Prerequisites are the same as any `skill_mutations`
|
|
consumer: durable database backend, `run_events.backend: db`, and a single
|
|
POSIX host with local disk:
|
|
|
|
```yaml
|
|
plugins:
|
|
- name: skill-evolution
|
|
package: deerflow-extension-skill-evolution
|
|
use: deerflow_extension_skill_evolution:install
|
|
enabled: true
|
|
host_access:
|
|
evidence:
|
|
owners: ["your-user-id"]
|
|
skill_mutations:
|
|
owners: ["your-user-id"] # subset of evidence.owners
|
|
operations: [stage, check, commit, revert]
|
|
trigger_agents: [lead_agent]
|
|
target_skills: [example-skill] # canonical names this plugin may revise
|
|
topology: single_host_local
|
|
config:
|
|
cases_dir: null # optional: default bundled cases
|
|
```
|
|
|
|
Without the grant the service fails to start with a `NOT_GRANTED` diagnostic;
|
|
nothing else in the deployment changes.
|
|
|
|
## HTTP surface
|
|
|
|
All routes are under `/api/extension-skill-evolution`:
|
|
|
|
| Route | Purpose |
|
|
| --- | --- |
|
|
| `GET /health` | readiness, evaluator identity, recorded case count |
|
|
| `POST /proposals` | resolve the sealed run evidence, stage a candidate, evaluate, return the verdict + report |
|
|
| `GET /proposals` · `GET /proposals/{id}` | list / inspect proposals and their reports |
|
|
| `POST /proposals/{id}/apply` | host `check`, then commit with the `AssessmentRef` attached |
|
|
| `POST /operations/{id}/revert` | roll back a committed revision |
|
|
|
|
`POST /proposals` takes `{skill_name, candidate_content, thread_id, run_id}`.
|
|
The run must be completed and sealed — it is the evidence anchor the host
|
|
requires for staging (`source_refs`), so every proposal records *which run
|
|
motivated the change*.
|
|
|
|
**Every route binds the caller.** The capability handles injected into the
|
|
service are scoped to the extension by the operator's grant, not to the
|
|
requesting user, so the routes resolve the host principal
|
|
(`deerflow_extension_api.auth.resolve_principal`) and refuse to act on a skill
|
|
owned by anyone else: `propose`, `apply` and `revert` answer 401 without a
|
|
principal and 403 (`FORBIDDEN`) across owners, and `GET /proposals` lists only
|
|
the caller's own proposals (another owner's proposal reads as 404, without
|
|
disclosing that it exists). When the host exposes a request-scoped run-evidence
|
|
reader, the evidence anchor must also be visible to the caller; hosts without
|
|
one fall back to the owner check.
|
|
|
|
## Offline walkthrough (tests)
|
|
|
|
The package ships the demonstration as tests; no Gateway, model, or network:
|
|
|
|
```sh
|
|
cd examples/deerflow-extension-skill-evolution
|
|
uv run --no-project --python 3.12 --with pytest --with pytest-asyncio \
|
|
--with ../../backend/packages/extension-api \
|
|
--with-editable . python -m pytest tests/ -q
|
|
```
|
|
|
|
`tests/test_lifecycle.py` drives the full loop against a fake host that
|
|
implements only the public contract: propose a fix for a broken converter
|
|
argument → verdict `justified` → apply → the committed operation carries the
|
|
report id → revert → the skill is byte-identical to the known-good baseline.
|
|
It also pins the refusal paths: `unjustified` and `failed` proposals are
|
|
discarded and cannot commit, a denied host `check` blocks the commit, and
|
|
double-commit is rejected.
|
|
|
|
## Scope and boundaries
|
|
|
|
- **This is an example, not a production engine.** The evaluator is a
|
|
deterministic static checker; it has no live-model runs, metrics, or signal
|
|
collection.
|
|
- [#6954](https://github.com/bytedance/deer-flow/discussions/6954) proposes a
|
|
distributable evolution plugin package with its own snapshot/versioning
|
|
design; this example stops at the merged #5645 contract and should be
|
|
revisited against that RFC if it lands.
|
|
- [#7066](https://github.com/bytedance/deer-flow/issues/7066) (eval-gated
|
|
evolution) owns the broader signal → attribution → candidate → verification
|
|
loop; the minimal evaluator here is deliberately smaller than that scope.
|
|
- The `skill_manage` direct-write path is unchanged. This plugin is additive
|
|
and opt-in.
|
|
|
|
## A concrete evidence standard (input for the roadmap discussion)
|
|
|
|
The roadmap asks what evidence should be required before enabling automatic
|
|
publication. Rather than propose a standard in the abstract, this example makes
|
|
one concrete and reviewable:
|
|
|
|
1. **Every proposal names its motivating run.** Staging requires sealed
|
|
completed-run evidence (`source_refs`); the host rejects stale or unsealed
|
|
anchors.
|
|
2. **Evidence is recorded, not narrated.** A case is a versioned JSON file with
|
|
deterministic checks; the verdict is computed from the frozen baseline and
|
|
candidate bundles, not from the proposal text.
|
|
3. **The verdict travels with the commit.** `AssessmentRef` pins
|
|
`evaluator`, `evaluator_version`, and a content-addressed `report_id`
|
|
(SHA-256 over the canonical report JSON) to the exact `candidate_hash` and
|
|
`base_revision`. The report bytes themselves are held in this service's
|
|
in-memory cache, not persisted; the id lets anyone holding the report — or
|
|
recomputing it from the same cases and frozen bundles — verify that it is
|
|
the one this commit cites.
|
|
4. **Fail-closed by default.** `unjustified` and `failed` proposals are
|
|
discarded before they can commit; a host `check` denial blocks the commit;
|
|
an assessment mismatch is rejected by the host itself.
|
|
5. **Rollback is as auditable as the change.** `revert` appends a new
|
|
operation referencing the one it undoes.
|
|
|
|
Automatic publication would additionally need: case authorship provenance (who
|
|
recorded the case, and was it derived from the same run that motivated the
|
|
change), a minimum case count, and a policy for `unjustified` proposals that
|
|
keep recurring. Those are deliberately out of scope here.
|