ChatDev/execorchestra/docs/BENCHMARKS.md
2026-11-05 16:22:12 +08:00

6.0 KiB

Benchmark invocation

Run these commands after completing the preparation sequence in README.md. The distributed candidate list is the 57-resource research profile. For comparisons with published results, match the experimental pool, model configuration, and evaluator. Obtain benchmark datasets separately from their official distributions.

Terminal-Bench 2.1

Dataset and container preparation

The reference input is an 89-task export from Harbor Hub, obtained with Harbor 0.23.0. Install that downloader in a separate Python 3.12+ environment; the framework itself continues to use Python 3.11:

python3.12 -m venv .cache/harbor-env
.cache/harbor-env/bin/python -m pip install harbor==0.23.0
.cache/harbor-env/bin/harbor auth login
.cache/harbor-env/bin/harbor download terminal-bench/terminal-bench-2-1 --export -o benchmarks
export TB_TASK_ROOT="$PWD/benchmarks/terminal-bench-2-1"

TB_TASK_ROOT must directly contain the task directories, each with task.toml, instruction.md, environment/, and tests/test.sh. The adapter enforces the reference tree SHA-256:

c31db162da898b59e8f5d827715a65378ca999e4df0dc052fe2a18f4241e0c0b

configs/tb21_dataset_files.json lists the reference file hashes. The recorded export used @latest; use a matching snapshot if the current download fails the adapter's tree-hash check.

Inspect the inventory and pre-pull the exact images declared by the tasks:

python -m sgar_mvp.benchmarks.terminalbench21.runner inventory --task-root "$TB_TASK_ROOT"
python - <<'PY'
import os, subprocess, tomllib
from pathlib import Path
root = Path(os.environ["TB_TASK_ROOT"])
images = {tomllib.loads(p.read_text())["environment"]["docker_image"]
          for p in root.glob("*/task.toml")}
for image in sorted(images):
    subprocess.run(["docker", "pull", image], check=True)
PY

Docker registry access is needed for this step. Pre-pulling avoids the optional runtime downloader, which requires skopeo and SGAR_MODEL_PROXY for missing registry-qualified images. Configure SGAR_DOCKER_PROXY as described in SERVICES.md before running tasks with internet access. Images, CPU/RAM limits, GPU requirements, and verifier timeouts come from task.toml.

Run

For one task, set TASK_ID to a directory name in TB_TASK_ROOT:

python -m sgar_mvp.benchmarks.terminalbench21.runner \
  --task-root "$TB_TASK_ROOT" --task-id "$TASK_ID" \
  --config "$PWD/sgar_mvp/config.json" --output-root "$PWD/runs/tb21" --network

Run the complete 89-task benchmark with a new output directory:

python -m sgar_mvp.benchmarks.terminalbench21.runner \
  --task-root "$TB_TASK_ROOT" --task-id all \
  --config "$PWD/sgar_mvp/config.json" --output-root "$PWD/runs/tb21-full" --network

The adapter uses task-defined images and deadlines, then runs the executable verifier. The commands above retain those defaults.

GAIA

Obtain GAIA's 2023 validation split through its official distribution. Accept its access terms and use your own Hugging Face token.

The included GAIA harness separates public task inputs from evaluator answers. The reference dataset revision is 682dd723ee1e1697e00360edccf2366dc8418dd9. Download only the validation material into private local storage:

export GAIA_REVISION=682dd723ee1e1697e00360edccf2366dc8418dd9
hf auth login
python - <<'PY'
import os
from huggingface_hub import snapshot_download
snapshot_download("gaia-benchmark/GAIA", repo_type="dataset",
    revision=os.environ["GAIA_REVISION"], local_dir="data/gaia-source",
    allow_patterns=["2023/validation/*"])
PY
export GAIA_SOURCE="$PWD/data/gaia-source/2023/validation"

The directory must contain metadata.jsonl or metadata.parquet and the referenced attachments; Parquet input additionally requires pyarrow.

python -m evaluation.gaia.freeze_dataset \
  --source "$GAIA_SOURCE" --output data/gaia-frozen --revision "$GAIA_REVISION"
python -m scripts.freeze_gaia_manifest --dataset-root data/gaia-frozen
python -m evaluation.gaia.run_validation \
  --dataset-root data/gaia-frozen --sgar-root "$PWD" \
  --experiment-manifest sgar_mvp/runtime_state/gaia_manifest.json \
  --config "$PWD/sgar_mvp/config.json" --output-root runs/gaia \
  --task-timeout-sec 3600 --network-policy declared

Freeze the manifest after preparation and source commit. It binds code, models, resources, index, scorer, and dataset. The command runs all 165 validation tasks with a 3,600-second agent-phase cap. For one task, add --task-id ID --max-tasks 1. The included scorer uses GAIA/AOrchestra scoring rules.

Outputs and accounting

The launchers process tasks sequentially within each benchmark. For parallel execution, assign disjoint tasks to separate processes and output directories, using the same prepared index. Size concurrency to provider and container limits.

Output Contents
GAIA validation_results.json Per-task scores and aggregate accuracy, cost, calls, and timing.
GAIA task benchmark_result.json, trial_record.json Official score, delivery status, timeout/failure class, and phase durations.
TB batch_result.json Per-task native results and success/failure counts.
TB task result.json Verifier reward, verified success, delivery status, timing, and native-run location.
Native cost_summary.json, execution_summary.json Model usage/cost completeness and observed resource execution.

SR uses the complete task count (165/89). Cost/Trial divides total trial cost by that count; CPVS divides it by successful tasks and is undefined with no successes. Include failed and timed-out trial costs. Costs and calls cover control, execution, and recovery, with reported cache usage priced by the supplied policy. Check usage completeness; keep agent-phase and end-to-end time separate.

Optional semantic review is in sgar_mvp/benchmarks/answer_validation.py. It records assisted results and review cost separately from official scoring.