github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-11-12 14:00:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/backend/scripts/benchmark/AGENTS.md
Xinmin Zeng 2a97ec3e58
docs: turn README into an index and move reference detail to docs/guide (#8113) 
* docs: turn README into an index and move reference detail to docs/guide

The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.

* docs: send user docs to topic guides and budget the root README

The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).

* chore: remove unreferenced screenshots and finished plan notes

pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.

* docs: route always-loaded AGENTS.md files to scoped guides

* docs(agents): cut scoped guides to scope, verify, traps, decisions

Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.

Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).

* style: format frontend guide table and guidance test imports

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 13:26:17 +08:00

1.7 KiB
Raw Blame History

Backend benchmarks (backend/scripts/benchmark/)

Standalone, reproducible benchmarks. Each one imports the production functions and reuses their runtime. Options and commands are in the README of each benchmark directory. Run them from backend/.

cd backend && python -m pytest tests -q -k "bench"

The offline tests need no network, no provider credentials and no downloaded dataset. Fixtures shaped like a public dataset are synthetic and generated by the tests.

  • Pin a dataset by immutable revision and SHA-256, and take it from a path the caller supplies. A benchmark does not download data. Label synthetic cases.
  • Read provider credentials and endpoints from named environment variables. Version model IDs, inference parameters, prompts, retries, clocks and seeds in the evaluation config.
  • Use fixed clocks and a deterministic offline order, and record the config, manifest, prompt, dataset and git revisions in the results.
  • Public raw results hold case IDs, decisions, grades and non-secret metadata. Dataset text, reference answers, memory and full provider payloads stay in ignored local run directories. Credentials and response headers are never committed.
  • Sandbox benchmarks use the production lifecycle and fail on a cleanup error.
  • The memory comparison reports original-thread and fresh-thread results separately.
  • The DeerMem eviction benchmark compares the confidence policy with hybrid-v1 only. Do not add another strategy to it.
  • The concurrency benchmark uses separate OS processes and a disposable schema outside public in a throwaway PostgreSQL database. It fails on a crash, a short operation count or any error.