docs(backend): shorten benchmark guidance to fit budget (#8106)

This commit is contained in:
Willem Jiang 2026-11-10 18:39:52 +08:00 committed by GitHub
parent 42da64a25c
commit 72fb20d56d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -37,33 +37,27 @@ Image result URL policy: [image search guide](packages/harness/deerflow/communit
### Backend Benchmarks
`scripts/benchmark/context_snapshot/`: explicit `run-live` needs provider env
vars; `summarize` and pytest are offline. See its README for the protocol.
Keep `scripts/benchmark/` standalone and reproducible. Import production
functions and reuse their runtime.
Benchmarks in `scripts/benchmark/` must be standalone and reproducible. Import
production functions; never duplicate them or introduce an alternative runtime.
`scripts/benchmark/context_memory_comparison/` runs an offline Gateway with
SQLite. Keep original- and fresh-thread results separate.
Sandbox benchmarks reuse production lifecycle. Reject negative/NaN timing values
before provider creation, export no secrets, and fail on final cleanup errors.
See the sandbox benchmark README for supported options and tests.
- Pin every external dataset by immutable revision and SHA-256. Callers provide
the local dataset path; evaluation commands must not silently download data.
- Never commit upstream dataset text, credentials, complete provider requests,
or response headers. Committed manifests may contain stable IDs and source
locators. Synthetic cases must identify themselves as synthetic.
- Read provider credentials and endpoints from named environment variables.
Version model IDs, inference parameters, prompts, retry rules, clocks, and
random seeds in the evaluation config.
- Public raw results may contain case IDs, policy decisions, model hypotheses,
grades, and non-secret response metadata. Keep dataset questions, reference
answers, memory content, and full provider payloads in ignored local run
directories.
- Use fixed clocks and deterministic ordering for offline selection. Results
must record the config, manifest, prompt, dataset, and git revisions used.
- `context_snapshot` needs provider env vars for `run-live`. See its README.
`summarize` and pytest run offline.
- `context_memory_comparison` runs Gateway offline with SQLite. Report
original- and fresh-thread results separately.
- Sandbox benchmarks use the production lifecycle. Reject negative or NaN
timing values before provider creation, export no secrets, and fail on
cleanup errors. See the sandbox benchmark README for options and tests.
- Pin datasets by immutable revision and SHA-256. Require a caller-provided
local path without automatic downloads. Label synthetic cases. Manifests
may include stable IDs and source locators.
- Read provider credentials and endpoints from named env vars. Version model
IDs, inference parameters, prompts, retries, clocks, and seeds in eval config.
- Public raw results may include case IDs, policy decisions, model hypotheses,
grades, and non-secret response metadata. Keep dataset text, reference
answers, memory, and full provider payloads in ignored local run directories.
Exclude credentials and response headers from commits.
- Use fixed clocks and deterministic offline ordering. Record config, manifest,
prompt, dataset, and git revisions in results.
`scripts/benchmark/deermem_eviction/` evaluates the production
`select_facts_for_capacity()` implementation used by DeerMem. It compares only
@ -84,15 +78,12 @@ The offline test suite must not require network access, provider credentials,
or the LongMemEval dataset. Small LongMemEval-shaped fixtures must be synthetic
and generated by tests.
`scripts/benchmark/concurrency/` measures multi-process contention on the
`users` table (N separate OS processes, not asyncio tasks) for SQLite vs
Postgres -- the scenario `CONFIGURATION.md` requires Postgres for. `worker.py`
connects directly via SQLAlchemy (skipping the ~8.5s Alembic bootstrap the
orchestrator already ran once) and mirrors the app's per-connection SQLite
PRAGMAs; `run_concurrency_bench.py` seeds a disposable per-run Postgres schema,
synchronises workers on a READY/GO barrier before timing, and exits non-zero on
any crash, short op count, or `errors > 0`. Postgres runs need a throwaway
database via `--pg-url`; nothing here touches `public`. Run from `backend/`:
`scripts/benchmark/concurrency/` compares SQLite and Postgres contention on
`users` with separate OS processes. `worker.py` connects via SQLAlchemy with
the app's SQLite PRAGMAs after the orchestrator runs Alembic once.
`run_concurrency_bench.py` uses a disposable schema outside `public` in the
throwaway database supplied via `--pg-url`. It starts timing after a READY/GO
barrier and fails on a crash, short op count, or any errors. Run from `backend/`.
```bash
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py \