mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-12 14:00:35 +00:00
docs(backend): shorten benchmark guidance to fit budget (#8106)
This commit is contained in:
parent
42da64a25c
commit
72fb20d56d
@ -37,33 +37,27 @@ Image result URL policy: [image search guide](packages/harness/deerflow/communit
|
||||
|
||||
### Backend Benchmarks
|
||||
|
||||
`scripts/benchmark/context_snapshot/`: explicit `run-live` needs provider env
|
||||
vars; `summarize` and pytest are offline. See its README for the protocol.
|
||||
Keep `scripts/benchmark/` standalone and reproducible. Import production
|
||||
functions and reuse their runtime.
|
||||
|
||||
Benchmarks in `scripts/benchmark/` must be standalone and reproducible. Import
|
||||
production functions; never duplicate them or introduce an alternative runtime.
|
||||
|
||||
`scripts/benchmark/context_memory_comparison/` runs an offline Gateway with
|
||||
SQLite. Keep original- and fresh-thread results separate.
|
||||
|
||||
Sandbox benchmarks reuse production lifecycle. Reject negative/NaN timing values
|
||||
before provider creation, export no secrets, and fail on final cleanup errors.
|
||||
See the sandbox benchmark README for supported options and tests.
|
||||
|
||||
- Pin every external dataset by immutable revision and SHA-256. Callers provide
|
||||
the local dataset path; evaluation commands must not silently download data.
|
||||
- Never commit upstream dataset text, credentials, complete provider requests,
|
||||
or response headers. Committed manifests may contain stable IDs and source
|
||||
locators. Synthetic cases must identify themselves as synthetic.
|
||||
- Read provider credentials and endpoints from named environment variables.
|
||||
Version model IDs, inference parameters, prompts, retry rules, clocks, and
|
||||
random seeds in the evaluation config.
|
||||
- Public raw results may contain case IDs, policy decisions, model hypotheses,
|
||||
grades, and non-secret response metadata. Keep dataset questions, reference
|
||||
answers, memory content, and full provider payloads in ignored local run
|
||||
directories.
|
||||
- Use fixed clocks and deterministic ordering for offline selection. Results
|
||||
must record the config, manifest, prompt, dataset, and git revisions used.
|
||||
- `context_snapshot` needs provider env vars for `run-live`. See its README.
|
||||
`summarize` and pytest run offline.
|
||||
- `context_memory_comparison` runs Gateway offline with SQLite. Report
|
||||
original- and fresh-thread results separately.
|
||||
- Sandbox benchmarks use the production lifecycle. Reject negative or NaN
|
||||
timing values before provider creation, export no secrets, and fail on
|
||||
cleanup errors. See the sandbox benchmark README for options and tests.
|
||||
- Pin datasets by immutable revision and SHA-256. Require a caller-provided
|
||||
local path without automatic downloads. Label synthetic cases. Manifests
|
||||
may include stable IDs and source locators.
|
||||
- Read provider credentials and endpoints from named env vars. Version model
|
||||
IDs, inference parameters, prompts, retries, clocks, and seeds in eval config.
|
||||
- Public raw results may include case IDs, policy decisions, model hypotheses,
|
||||
grades, and non-secret response metadata. Keep dataset text, reference
|
||||
answers, memory, and full provider payloads in ignored local run directories.
|
||||
Exclude credentials and response headers from commits.
|
||||
- Use fixed clocks and deterministic offline ordering. Record config, manifest,
|
||||
prompt, dataset, and git revisions in results.
|
||||
|
||||
`scripts/benchmark/deermem_eviction/` evaluates the production
|
||||
`select_facts_for_capacity()` implementation used by DeerMem. It compares only
|
||||
@ -84,15 +78,12 @@ The offline test suite must not require network access, provider credentials,
|
||||
or the LongMemEval dataset. Small LongMemEval-shaped fixtures must be synthetic
|
||||
and generated by tests.
|
||||
|
||||
`scripts/benchmark/concurrency/` measures multi-process contention on the
|
||||
`users` table (N separate OS processes, not asyncio tasks) for SQLite vs
|
||||
Postgres -- the scenario `CONFIGURATION.md` requires Postgres for. `worker.py`
|
||||
connects directly via SQLAlchemy (skipping the ~8.5s Alembic bootstrap the
|
||||
orchestrator already ran once) and mirrors the app's per-connection SQLite
|
||||
PRAGMAs; `run_concurrency_bench.py` seeds a disposable per-run Postgres schema,
|
||||
synchronises workers on a READY/GO barrier before timing, and exits non-zero on
|
||||
any crash, short op count, or `errors > 0`. Postgres runs need a throwaway
|
||||
database via `--pg-url`; nothing here touches `public`. Run from `backend/`:
|
||||
`scripts/benchmark/concurrency/` compares SQLite and Postgres contention on
|
||||
`users` with separate OS processes. `worker.py` connects via SQLAlchemy with
|
||||
the app's SQLite PRAGMAs after the orchestrator runs Alembic once.
|
||||
`run_concurrency_bench.py` uses a disposable schema outside `public` in the
|
||||
throwaway database supplied via `--pg-url`. It starts timing after a READY/GO
|
||||
barrier and fails on a crash, short op count, or any errors. Run from `backend/`.
|
||||
|
||||
```bash
|
||||
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py \
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user