ChatDev/execorchestra/docs/BENCHMARKS.md
2026-11-05 16:22:12 +08:00

142 lines
6.0 KiB
Markdown

# Benchmark invocation
Run these commands after completing the preparation sequence in README.md.
The distributed candidate list is the 57-resource research profile. For comparisons with published results, match the experimental pool, model configuration, and evaluator. Obtain benchmark datasets separately from their official distributions.
## Terminal-Bench 2.1
### Dataset and container preparation
The reference input is an 89-task export from
[Harbor Hub](https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1/latest),
obtained with Harbor 0.23.0. Install that downloader in a separate Python 3.12+
environment; the framework itself continues to use Python 3.11:
```bash
python3.12 -m venv .cache/harbor-env
.cache/harbor-env/bin/python -m pip install harbor==0.23.0
.cache/harbor-env/bin/harbor auth login
.cache/harbor-env/bin/harbor download terminal-bench/terminal-bench-2-1 --export -o benchmarks
export TB_TASK_ROOT="$PWD/benchmarks/terminal-bench-2-1"
```
`TB_TASK_ROOT` must directly contain the task directories, each with `task.toml`,
`instruction.md`, `environment/`, and `tests/test.sh`. The adapter enforces the
reference tree SHA-256:
```text
c31db162da898b59e8f5d827715a65378ca999e4df0dc052fe2a18f4241e0c0b
```
`configs/tb21_dataset_files.json` lists the reference file hashes. The recorded
export used `@latest`; use a matching snapshot if the current download fails
the adapter's tree-hash check.
Inspect the inventory and pre-pull the exact images declared by the tasks:
```bash
python -m sgar_mvp.benchmarks.terminalbench21.runner inventory --task-root "$TB_TASK_ROOT"
python - <<'PY'
import os, subprocess, tomllib
from pathlib import Path
root = Path(os.environ["TB_TASK_ROOT"])
images = {tomllib.loads(p.read_text())["environment"]["docker_image"]
for p in root.glob("*/task.toml")}
for image in sorted(images):
subprocess.run(["docker", "pull", image], check=True)
PY
```
Docker registry access is needed for this step. Pre-pulling avoids the optional
runtime downloader, which requires `skopeo` and `SGAR_MODEL_PROXY` for missing
registry-qualified images. Configure `SGAR_DOCKER_PROXY` as described in
[SERVICES.md](SERVICES.md) before running tasks with internet access. Images,
CPU/RAM limits, GPU requirements, and verifier timeouts come from `task.toml`.
### Run
For one task, set `TASK_ID` to a directory name in `TB_TASK_ROOT`:
```bash
python -m sgar_mvp.benchmarks.terminalbench21.runner \
--task-root "$TB_TASK_ROOT" --task-id "$TASK_ID" \
--config "$PWD/sgar_mvp/config.json" --output-root "$PWD/runs/tb21" --network
```
Run the complete 89-task benchmark with a new output directory:
```bash
python -m sgar_mvp.benchmarks.terminalbench21.runner \
--task-root "$TB_TASK_ROOT" --task-id all \
--config "$PWD/sgar_mvp/config.json" --output-root "$PWD/runs/tb21-full" --network
```
The adapter uses task-defined images and deadlines, then runs the executable
verifier. The commands above retain those defaults.
## GAIA
Obtain GAIA's 2023 validation split through its
[official distribution](https://huggingface.co/datasets/gaia-benchmark/GAIA).
Accept its access terms and use your own Hugging Face token.
The included GAIA harness separates public task inputs from evaluator answers.
The reference dataset revision is `682dd723ee1e1697e00360edccf2366dc8418dd9`.
Download only the validation material into private local storage:
```bash
export GAIA_REVISION=682dd723ee1e1697e00360edccf2366dc8418dd9
hf auth login
python - <<'PY'
import os
from huggingface_hub import snapshot_download
snapshot_download("gaia-benchmark/GAIA", repo_type="dataset",
revision=os.environ["GAIA_REVISION"], local_dir="data/gaia-source",
allow_patterns=["2023/validation/*"])
PY
export GAIA_SOURCE="$PWD/data/gaia-source/2023/validation"
```
The directory must contain `metadata.jsonl` or
`metadata.parquet` and the referenced attachments; Parquet input additionally
requires `pyarrow`.
```bash
python -m evaluation.gaia.freeze_dataset \
--source "$GAIA_SOURCE" --output data/gaia-frozen --revision "$GAIA_REVISION"
python -m scripts.freeze_gaia_manifest --dataset-root data/gaia-frozen
python -m evaluation.gaia.run_validation \
--dataset-root data/gaia-frozen --sgar-root "$PWD" \
--experiment-manifest sgar_mvp/runtime_state/gaia_manifest.json \
--config "$PWD/sgar_mvp/config.json" --output-root runs/gaia \
--task-timeout-sec 3600 --network-policy declared
```
Freeze the manifest after preparation and source commit. It binds code, models,
resources, index, scorer, and dataset. The command runs all 165 validation tasks
with a 3,600-second agent-phase cap. For one task, add `--task-id ID --max-tasks 1`.
The included scorer uses GAIA/AOrchestra scoring rules.
## Outputs and accounting
The launchers process tasks sequentially within each benchmark. For parallel
execution, assign disjoint tasks to separate processes and output directories,
using the same prepared index. Size concurrency to provider and container limits.
| Output | Contents |
| --- | --- |
| GAIA `validation_results.json` | Per-task scores and aggregate accuracy, cost, calls, and timing. |
| GAIA task `benchmark_result.json`, `trial_record.json` | Official score, delivery status, timeout/failure class, and phase durations. |
| TB `batch_result.json` | Per-task native results and success/failure counts. |
| TB task `result.json` | Verifier reward, verified success, delivery status, timing, and native-run location. |
| Native `cost_summary.json`, `execution_summary.json` | Model usage/cost completeness and observed resource execution. |
SR uses the complete task count (165/89). Cost/Trial divides total trial cost by
that count; CPVS divides it by successful tasks and is undefined with no successes.
Include failed and timed-out trial costs. Costs and calls cover control,
execution, and recovery, with reported cache usage priced by the supplied policy.
Check usage completeness; keep agent-phase and end-to-end time separate.
Optional semantic review is in `sgar_mvp/benchmarks/answer_validation.py`.
It records assisted results and review cost separately from official scoring.