diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 46f6aa0296..260ea508ef 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -37,33 +37,27 @@ Image result URL policy: [image search guide](packages/harness/deerflow/communit ### Backend Benchmarks -`scripts/benchmark/context_snapshot/`: explicit `run-live` needs provider env -vars; `summarize` and pytest are offline. See its README for the protocol. +Keep `scripts/benchmark/` standalone and reproducible. Import production +functions and reuse their runtime. -Benchmarks in `scripts/benchmark/` must be standalone and reproducible. Import -production functions; never duplicate them or introduce an alternative runtime. - -`scripts/benchmark/context_memory_comparison/` runs an offline Gateway with -SQLite. Keep original- and fresh-thread results separate. - -Sandbox benchmarks reuse production lifecycle. Reject negative/NaN timing values -before provider creation, export no secrets, and fail on final cleanup errors. -See the sandbox benchmark README for supported options and tests. - -- Pin every external dataset by immutable revision and SHA-256. Callers provide - the local dataset path; evaluation commands must not silently download data. -- Never commit upstream dataset text, credentials, complete provider requests, - or response headers. Committed manifests may contain stable IDs and source - locators. Synthetic cases must identify themselves as synthetic. -- Read provider credentials and endpoints from named environment variables. - Version model IDs, inference parameters, prompts, retry rules, clocks, and - random seeds in the evaluation config. -- Public raw results may contain case IDs, policy decisions, model hypotheses, - grades, and non-secret response metadata. Keep dataset questions, reference - answers, memory content, and full provider payloads in ignored local run - directories. -- Use fixed clocks and deterministic ordering for offline selection. Results - must record the config, manifest, prompt, dataset, and git revisions used. +- `context_snapshot` needs provider env vars for `run-live`. See its README. + `summarize` and pytest run offline. +- `context_memory_comparison` runs Gateway offline with SQLite. Report + original- and fresh-thread results separately. +- Sandbox benchmarks use the production lifecycle. Reject negative or NaN + timing values before provider creation, export no secrets, and fail on + cleanup errors. See the sandbox benchmark README for options and tests. +- Pin datasets by immutable revision and SHA-256. Require a caller-provided + local path without automatic downloads. Label synthetic cases. Manifests + may include stable IDs and source locators. +- Read provider credentials and endpoints from named env vars. Version model + IDs, inference parameters, prompts, retries, clocks, and seeds in eval config. +- Public raw results may include case IDs, policy decisions, model hypotheses, + grades, and non-secret response metadata. Keep dataset text, reference + answers, memory, and full provider payloads in ignored local run directories. + Exclude credentials and response headers from commits. +- Use fixed clocks and deterministic offline ordering. Record config, manifest, + prompt, dataset, and git revisions in results. `scripts/benchmark/deermem_eviction/` evaluates the production `select_facts_for_capacity()` implementation used by DeerMem. It compares only @@ -84,15 +78,12 @@ The offline test suite must not require network access, provider credentials, or the LongMemEval dataset. Small LongMemEval-shaped fixtures must be synthetic and generated by tests. -`scripts/benchmark/concurrency/` measures multi-process contention on the -`users` table (N separate OS processes, not asyncio tasks) for SQLite vs -Postgres -- the scenario `CONFIGURATION.md` requires Postgres for. `worker.py` -connects directly via SQLAlchemy (skipping the ~8.5s Alembic bootstrap the -orchestrator already ran once) and mirrors the app's per-connection SQLite -PRAGMAs; `run_concurrency_bench.py` seeds a disposable per-run Postgres schema, -synchronises workers on a READY/GO barrier before timing, and exits non-zero on -any crash, short op count, or `errors > 0`. Postgres runs need a throwaway -database via `--pg-url`; nothing here touches `public`. Run from `backend/`: +`scripts/benchmark/concurrency/` compares SQLite and Postgres contention on +`users` with separate OS processes. `worker.py` connects via SQLAlchemy with +the app's SQLite PRAGMAs after the orchestrator runs Alembic once. +`run_concurrency_bench.py` uses a disposable schema outside `public` in the +throwaway database supplied via `--pg-url`. It starts timing after a READY/GO +barrier and fails on a crash, short op count, or any errors. Run from `backend/`. ```bash uv run python scripts/benchmark/concurrency/run_concurrency_bench.py \