deer-flow/backend/scripts/benchmark/context_snapshot

Synthetic context snapshot evaluation

This manual evaluation compares a complete, isolated handoff with an opt-in parent-context snapshot. It contains eight synthetic JSON/Python tasks and 51 predeclared checks. No external dataset is downloaded. Live calls are explicit and are not part of pytest or default CI.

Run

After installing the backend dependencies, run from backend/. Supply an OpenAI-compatible Chat Completions endpoint and a tool-capable model that accepts the sampling options in config.json:

export CONTEXT_SNAPSHOT_BASE_URL="https://provider.example/v1"
export CONTEXT_SNAPSHOT_MODEL="your-model-id"
# Set CONTEXT_SNAPSHOT_API_KEY if the endpoint requires authentication.
uv run python -m scripts.benchmark.context_snapshot run-live \
  --output-dir /tmp/context-snapshot-run

The output directory must be new. It receives a .gitignore, artifacts and their revisions, local dispatch/check details, usage metadata, rows.jsonl, summary.json, and run.json. The latter records the selected model, config, seeds, job order, source hashes, package versions, Git revision and dirty state. Endpoint values and credentials are read only from the named environment variables; provider headers and complete requests/responses are not recorded.

Use repeated --case options for a smoke run (for example rate_limits and invoice_total), --repetitions 3 for additional pairs, or --config path.json for an explicitly versioned parameter variant. The default config uses temperature 0.2, reasoning disabled, 4,096 output tokens per request, 50 graph steps, a 180-second worker limit and two concurrent jobs. Provider retries are disabled. Both arms use seed 1703 + repetition; ordering uses seed 20260913. The model ID is an environment input and is recorded in each run, so a rerun must set it explicitly (the historical run used qwen3.8-flash-next).

Offline commands need no endpoint or credentials:

uv run python -m scripts.benchmark.context_snapshot summarize \
  scripts/benchmark/context_snapshot/results/2026-09-12.rows.jsonl
uv run pytest tests/test_bench_context_snapshot.py -q

Scope and scoring

Both arms make a real parent-model call using the production task schema. The isolated arm asks the parent to write a complete handoff; the snapshot arm asks for the short current brief and supplies ParentContextSnapshot separately. Valid prompt rephrasing is accepted and recorded. The worker runs through the production SubagentExecutor._aexecute() and agent factory, with model creation instrumented for per-job usage accounting. The three exercise tools write an artifact, read bundled references and run public checks. Hidden checks evaluate the final saved artifact separately.

This does not run Gateway dispatch/polling, UI, real repository work, network research, or automatic mode selection. Worker authorization, memory and summarization are disabled. Python exercises use a restricted no-import profile with standard computational builtins and a three-second subprocess timeout; that grader is not an operating-system sandbox. Run model-generated code in a disposable development environment.

artifact_correct means all predeclared task checks pass. clean_success additionally requires a fresh public check of the final revision, completed executor status, no stop reason and no exception. A correct artifact at the turn cap counts only in the first metric. Costs include parent and worker requests, including failed/capped runs. Missing provider usage is marked incomplete; the mean token count becomes null instead of treating unknown usage as zero. Timing uses perf_counter, includes parent plus worker execution, and excludes queue wait and the final offline hidden-check pass.

Historical result: 2026-09-12

The committed rows are a metadata-only export from the original local harness, not an output claimed to have been generated by this packaged runner. The package preserves the synthetic cases, prompts and inference budgets while making configuration, artifact paths and metadata handling portable. Exact artifact acknowledgements and grader representations differ; a fresh model run is not expected to reproduce outputs or timings byte for byte.

Metric	Complete isolated handoff	Snapshot
Paired synthetic tasks	8	8
Correct final artifacts	8/8	8/8
Normal completions	7/8	7/8
Mean parent + worker tokens	13,196.1	11,151.8
Mean parent + worker seconds	33.73	20.14
Provider requests	48	42

Both arms also passed all first-artifact checks. The isolated invoice worker and snapshot pagination worker reached the turn cap despite correct artifacts; their full costs remain included. Mean tokens were 15.5% lower and mean elapsed time 40.3% lower for snapshot in this sample. For the self-contained task with unrelated history, tokens increased from 8,359 to 12,360 (+47.9%). This is one model and eight synthetic pairs, not evidence of general quality superiority, production readiness or guaranteed savings.

Earlier 72 exploratory assignments are excluded from this comparison because the lead lacked an explicit description of the worker's restricted tool capabilities and the grader omitted some valid Python builtins. Nine of those assignments were resumed after removing a harness-only literal-copy gate, reusing their original parent calls. The controlled 16-run phase disclosed the worker capabilities to both arms and accepted schema-valid prompt rephrasing.

A subsequent grader audit found that missing divmod had caused a false public failure in the controlled invoice case. Both invoice workers were rerun after fixing that omission, reusing each arm's original parent dispatch, token count and elapsed time. The canonical table therefore combines 14 original runs and the corrected pair; each parent call is counted once. Failed original workers remain local experimental overhead and are not silently counted as canonical successes. Task requirements and the 51 hidden checks were unchanged.

See results/2026-09-12.provenance.json for protocol hashes and correction metadata. Raw local logs, full provider payloads, endpoint configuration and credentials are deliberately absent from the committed evidence.