# Backend benchmarks (`backend/scripts/benchmark/`)

Standalone, reproducible benchmarks. Each one imports the production functions
and reuses their runtime. Options and commands are in the README of each
benchmark directory. Run them from `backend/`.

```bash
cd backend && python -m pytest tests -q -k "bench"
```

The offline tests need no network, no provider credentials and no downloaded
dataset. Fixtures shaped like a public dataset are synthetic and generated by
the tests.

- **Pin a dataset by immutable revision and SHA-256**, and take it from a path
  the caller supplies. A benchmark does not download data. Label synthetic
  cases.
- **Read provider credentials and endpoints from named environment variables.**
  Version model IDs, inference parameters, prompts, retries, clocks and seeds in
  the evaluation config.
- **Use fixed clocks and a deterministic offline order**, and record the config,
  manifest, prompt, dataset and git revisions in the results.
- **Public raw results hold case IDs, decisions, grades and non-secret
  metadata.** Dataset text, reference answers, memory and full provider payloads
  stay in ignored local run directories. Credentials and response headers are
  never committed.
- **Sandbox benchmarks use the production lifecycle** and fail on a cleanup
  error.
- **The memory comparison reports original-thread and fresh-thread results
  separately.**
- **The DeerMem eviction benchmark compares the `confidence` policy with
  `hybrid-v1` only.** Do not add another strategy to it.
- **The concurrency benchmark uses separate OS processes** and a disposable
  schema outside `public` in a throwaway PostgreSQL database. It fails on a
  crash, a short operation count or any error.