mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-14 16:08:41 +00:00
6d5d7bb1d5
* feat(subagents): add opt-in parent context snapshots * test(subagents): package synthetic snapshot evaluation * fix(subagents): preserve output text and defer snapshot capture * docs(subagents): keep snapshot guidance within chain budget * fix(subagents): omit unpaired tool calls from snapshots * fix(subagents): safely omit unserializable snapshot media
117 lines
6.2 KiB
Markdown
117 lines
6.2 KiB
Markdown
# Synthetic context snapshot evaluation
|
|
|
|
This manual evaluation compares a complete, isolated handoff with an opt-in
|
|
parent-context snapshot. It contains eight synthetic JSON/Python tasks and 51
|
|
predeclared checks. No external dataset is downloaded. Live calls are explicit
|
|
and are not part of pytest or default CI.
|
|
|
|
## Run
|
|
|
|
After installing the backend dependencies, run from `backend/`. Supply an
|
|
OpenAI-compatible Chat Completions endpoint and a tool-capable model that accepts
|
|
the sampling options in `config.json`:
|
|
|
|
```bash
|
|
export CONTEXT_SNAPSHOT_BASE_URL="https://provider.example/v1"
|
|
export CONTEXT_SNAPSHOT_MODEL="your-model-id"
|
|
# Set CONTEXT_SNAPSHOT_API_KEY if the endpoint requires authentication.
|
|
uv run python -m scripts.benchmark.context_snapshot run-live \
|
|
--output-dir /tmp/context-snapshot-run
|
|
```
|
|
|
|
The output directory must be new. It receives a `.gitignore`, artifacts and
|
|
their revisions, local dispatch/check details, usage metadata, `rows.jsonl`,
|
|
`summary.json`, and `run.json`. The latter records the selected model, config,
|
|
seeds, job order, source hashes, package versions, Git revision and dirty state.
|
|
Endpoint values and credentials are read only from the named environment
|
|
variables; provider headers and complete requests/responses are not recorded.
|
|
|
|
Use repeated `--case` options for a smoke run (for example `rate_limits` and
|
|
`invoice_total`), `--repetitions 3` for additional pairs, or `--config path.json`
|
|
for an explicitly versioned parameter variant. The default config uses
|
|
temperature 0.2, reasoning disabled, 4,096 output tokens per request, 50 graph
|
|
steps, a 180-second worker limit and two concurrent jobs. Provider retries are
|
|
disabled. Both arms use seed `1703 + repetition`; ordering uses seed 20260913.
|
|
The model ID is an environment input and is recorded in each run, so a rerun
|
|
must set it explicitly (the historical run used `qwen3.8-flash-next`).
|
|
|
|
Offline commands need no endpoint or credentials:
|
|
|
|
```bash
|
|
uv run python -m scripts.benchmark.context_snapshot summarize \
|
|
scripts/benchmark/context_snapshot/results/2026-09-12.rows.jsonl
|
|
uv run pytest tests/test_bench_context_snapshot.py -q
|
|
```
|
|
|
|
## Scope and scoring
|
|
|
|
Both arms make a real parent-model call using the production `task` schema.
|
|
The isolated arm asks the parent to write a complete handoff; the snapshot arm
|
|
asks for the short current brief and supplies `ParentContextSnapshot` separately.
|
|
Valid prompt rephrasing is accepted and recorded. The worker runs through the
|
|
production `SubagentExecutor._aexecute()` and agent factory, with model creation
|
|
instrumented for per-job usage accounting. The three exercise tools write an
|
|
artifact, read bundled references and run public checks. Hidden checks evaluate
|
|
the final saved artifact separately.
|
|
|
|
This does not run Gateway dispatch/polling, UI, real repository work, network
|
|
research, or automatic mode selection. Worker authorization, memory and
|
|
summarization are disabled. Python exercises use a restricted no-import profile
|
|
with standard computational builtins and a three-second subprocess timeout;
|
|
that grader is not an operating-system sandbox. Run model-generated code in a
|
|
disposable development environment.
|
|
|
|
`artifact_correct` means all predeclared task checks pass. `clean_success`
|
|
additionally requires a fresh public check of the final revision, completed
|
|
executor status, no stop reason and no exception. A correct artifact at the
|
|
turn cap counts only in the first metric. Costs include parent and worker
|
|
requests, including failed/capped runs. Missing provider usage is marked
|
|
incomplete; the mean token count becomes null instead of treating unknown
|
|
usage as zero. Timing uses `perf_counter`, includes parent plus worker execution,
|
|
and excludes queue wait and the final offline hidden-check pass.
|
|
|
|
## Historical result: 2026-09-12
|
|
|
|
The committed rows are a metadata-only export from the original local harness,
|
|
not an output claimed to have been generated by this packaged runner. The
|
|
package preserves the synthetic cases, prompts and inference budgets while
|
|
making configuration, artifact paths and metadata handling portable. Exact
|
|
artifact acknowledgements and grader representations differ; a fresh model
|
|
run is not expected to reproduce outputs or timings byte for byte.
|
|
|
|
| Metric | Complete isolated handoff | Snapshot |
|
|
| --- | ---: | ---: |
|
|
| Paired synthetic tasks | 8 | 8 |
|
|
| Correct final artifacts | 8/8 | 8/8 |
|
|
| Normal completions | 7/8 | 7/8 |
|
|
| Mean parent + worker tokens | 13,196.1 | 11,151.8 |
|
|
| Mean parent + worker seconds | 33.73 | 20.14 |
|
|
| Provider requests | 48 | 42 |
|
|
|
|
Both arms also passed all first-artifact checks. The isolated invoice worker
|
|
and snapshot pagination worker reached the turn cap despite correct artifacts;
|
|
their full costs remain included. Mean tokens were 15.5% lower and mean elapsed
|
|
time 40.3% lower for snapshot in this sample. For the self-contained task with
|
|
unrelated history, tokens increased from 8,359 to 12,360 (+47.9%). This is one
|
|
model and eight synthetic pairs, not evidence of general quality superiority,
|
|
production readiness or guaranteed savings.
|
|
|
|
Earlier 72 exploratory assignments are excluded from this comparison because
|
|
the lead lacked an explicit description of the worker's restricted tool
|
|
capabilities and the grader omitted some valid Python builtins. Nine of those
|
|
assignments were resumed after removing a harness-only literal-copy gate,
|
|
reusing their original parent calls. The controlled 16-run phase disclosed the
|
|
worker capabilities to both arms and accepted schema-valid prompt rephrasing.
|
|
|
|
A subsequent grader audit found that missing `divmod` had caused a false public
|
|
failure in the controlled invoice case. Both invoice workers were rerun after
|
|
fixing that omission, reusing each arm's original parent dispatch, token count
|
|
and elapsed time. The canonical table therefore combines 14 original runs and
|
|
the corrected pair; each parent call is counted once. Failed original workers
|
|
remain local experimental overhead and are not silently counted as canonical
|
|
successes. Task requirements and the 51 hidden checks were unchanged.
|
|
|
|
See `results/2026-09-12.provenance.json` for protocol hashes and correction
|
|
metadata. Raw local logs, full provider payloads, endpoint configuration and
|
|
credentials are deliberately absent from the committed evidence.
|