mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-12 22:20:33 +00:00
2a97ec3e58
* docs: turn README into an index and move reference detail to docs/guide The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md. * docs: send user docs to topic guides and budget the root README The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003). * chore: remove unreferenced screenshots and finished plan notes pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them. * docs: route always-loaded AGENTS.md files to scoped guides * docs(agents): cut scoped guides to scope, verify, traps, decisions Each AGENTS.md keeps four kinds of content: the scope of the directory, the verification commands, traps a local test run does not show, and decisions the code cannot state. Values, field lists, function inventories, test names, issue numbers and history are deleted, because the code, the tests and git state them. Search and fetch providers share one community guide. Four guides with no remaining content are removed. The size check now allows 6 KiB per file (hard limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB). * style: format frontend guide table and guidance test imports --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
36 lines
1.7 KiB
Markdown
36 lines
1.7 KiB
Markdown
# Backend benchmarks (`backend/scripts/benchmark/`)
|
|
|
|
Standalone, reproducible benchmarks. Each one imports the production functions
|
|
and reuses their runtime. Options and commands are in the README of each
|
|
benchmark directory. Run them from `backend/`.
|
|
|
|
```bash
|
|
cd backend && python -m pytest tests -q -k "bench"
|
|
```
|
|
|
|
The offline tests need no network, no provider credentials and no downloaded
|
|
dataset. Fixtures shaped like a public dataset are synthetic and generated by
|
|
the tests.
|
|
|
|
- **Pin a dataset by immutable revision and SHA-256**, and take it from a path
|
|
the caller supplies. A benchmark does not download data. Label synthetic
|
|
cases.
|
|
- **Read provider credentials and endpoints from named environment variables.**
|
|
Version model IDs, inference parameters, prompts, retries, clocks and seeds in
|
|
the evaluation config.
|
|
- **Use fixed clocks and a deterministic offline order**, and record the config,
|
|
manifest, prompt, dataset and git revisions in the results.
|
|
- **Public raw results hold case IDs, decisions, grades and non-secret
|
|
metadata.** Dataset text, reference answers, memory and full provider payloads
|
|
stay in ignored local run directories. Credentials and response headers are
|
|
never committed.
|
|
- **Sandbox benchmarks use the production lifecycle** and fail on a cleanup
|
|
error.
|
|
- **The memory comparison reports original-thread and fresh-thread results
|
|
separately.**
|
|
- **The DeerMem eviction benchmark compares the `confidence` policy with
|
|
`hybrid-v1` only.** Do not add another strategy to it.
|
|
- **The concurrency benchmark uses separate OS processes** and a disposable
|
|
schema outside `public` in a throwaway PostgreSQL database. It fails on a
|
|
crash, a short operation count or any error.
|