mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-21 12:21:50 +00:00
docs(tests): align contributor test rules with module-suite hygiene and sync 6-shard config (#8361)
- Update AGENTS.md, backend/AGENTS.md, backend/tests/AGENTS.md, and PR template to fold regressions into owning module suites, prune superseded tests, and forbid inspect.getsource assertions. - Align CONTRIBUTING.md, backend/tests/AGENTS.md, backend/Makefile, and test_make_test_parallel.py with the 6-shard CI configuration.
This commit is contained in:
parent
a50663c2f1
commit
8d60bef1b4
14
.github/pull_request_template.md
vendored
14
.github/pull_request_template.md
vendored
@ -46,13 +46,23 @@ Fixes #
|
||||
|
||||
<!-- Skip (delete) this section if this PR is not a bug fix.
|
||||
|
||||
Bugs should be encoded as a failing test that goes red before the fix.
|
||||
Fold regression cases into the owning module's existing suite instead of
|
||||
creating a new test file per fix.
|
||||
Confirm:
|
||||
- Test path that reproduces the bug:
|
||||
- Existing suite + test case that reproduces the bug:
|
||||
- Did it go red on `main` and green on this branch? (yes / no)
|
||||
- If a red test wasn't cheap to write, explain why and what you did instead. -->
|
||||
|
||||
|
||||
## Test footprint
|
||||
|
||||
<!-- Keep CI shards within budget: add cases to the module's behavioral suite
|
||||
and delete or update superseded tests in the same PR. -->
|
||||
|
||||
- **Net test line delta** (`+<added> / -<removed>` across test files):
|
||||
- **Merged or pruned old tests?** (`yes` — list consolidated/deleted test paths, or `no` — explain why net growth is needed):
|
||||
|
||||
|
||||
## Validation
|
||||
|
||||
<!-- What you actually ran. Run at least the checks for the area you changed:
|
||||
|
||||
@ -86,7 +86,9 @@ make check-agent-guidance # size budget of these guides
|
||||
because loading an extension imports code with Gateway privileges. Install
|
||||
from trusted sources only, and restart the Gateway after every extension
|
||||
change.
|
||||
- **Ship tests with every feature and bug fix.**
|
||||
- **Cover behavior changes in the owning module's suite.** Fold regression cases
|
||||
into that module's existing suite instead of opening a one-fix test file, and
|
||||
update or delete obsolete tests in the same change.
|
||||
- **Document behavior next to its topic, in the same change**: user-facing
|
||||
behavior in `docs/guide/` or `backend/docs/`. `README.md` is an index; change
|
||||
it only to add or re-point a link.
|
||||
|
||||
@ -350,7 +350,7 @@ before review.
|
||||
python -m pip install -r tests/skills/requirements.txt
|
||||
python -m pytest tests/skills -q
|
||||
|
||||
# Default backend tests (four parallel shards; excludes live, blocking-I/O and bundled-extension tests)
|
||||
# Default backend tests (six parallel shards matching CI; excludes live, blocking-I/O and bundled-extension tests)
|
||||
cd backend
|
||||
make test
|
||||
|
||||
@ -406,7 +406,7 @@ setup also verifies SSL and SQLite imports before launching fixture gateways.
|
||||
|
||||
### Test and diagnostics notes
|
||||
|
||||
Backend `make test` runs four duration-balanced shards concurrently and reports
|
||||
Backend `make test` runs six duration-balanced shards concurrently (matching CI) and reports
|
||||
failure if any shard fails. Use `make test TEST_JOBS=1` to run the shards sequentially.
|
||||
It excludes live external-API, blocking-I/O and bundled-extension coverage.
|
||||
Run `cd backend && make test-blocking-io` for strict blocking-I/O checks.
|
||||
|
||||
@ -35,9 +35,11 @@ make format
|
||||
make test-live # calls real APIs; needs DEER_FLOW_RUN_LIVE_TESTS=1
|
||||
```
|
||||
|
||||
Run `make test` and `make test-blocking-io` before and after a change. Every
|
||||
feature and bug fix ships with a test in `tests/`. Live tests stay opt-in, so the
|
||||
default suite needs no network and no API key.
|
||||
Run `make test` and `make test-blocking-io` before and after a change. Cover
|
||||
behavior changes in the owning module's existing suite under `tests/` rather
|
||||
than opening a new file per fix, and update or delete superseded tests in the
|
||||
same change. Live tests stay opt-in, so the default suite needs no network and
|
||||
no API key.
|
||||
|
||||
## Rules
|
||||
|
||||
|
||||
@ -19,15 +19,15 @@ dev:
|
||||
gateway:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run --locked uvicorn app.gateway.app:app --host 127.0.0.1 --port 8001
|
||||
|
||||
# Run all four duration-balanced shards locally; wait for every shard on failure.
|
||||
TEST_JOBS ?= 4
|
||||
TEST_SHARD_TARGETS := test-shard-1 test-shard-2 test-shard-3 test-shard-4
|
||||
# Run all six duration-balanced shards locally (matching CI); wait for every shard on failure.
|
||||
TEST_JOBS ?= 6
|
||||
TEST_SHARD_TARGETS := test-shard-1 test-shard-2 test-shard-3 test-shard-4 test-shard-5 test-shard-6
|
||||
.PHONY: test test-shard test-bundled-extensions $(TEST_SHARD_TARGETS)
|
||||
test:
|
||||
$(MAKE) --no-print-directory -k -j$(TEST_JOBS) $(TEST_SHARD_TARGETS)
|
||||
|
||||
$(TEST_SHARD_TARGETS):
|
||||
$(MAKE) --no-print-directory test-shard SPLITS=4 GROUP=$(patsubst test-shard-%,%,$@)
|
||||
$(MAKE) --no-print-directory test-shard SPLITS=6 GROUP=$(patsubst test-shard-%,%,$@)
|
||||
|
||||
test-live:
|
||||
DEER_FLOW_RUN_LIVE_TESTS=1 PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m live tests/ -v -s
|
||||
@ -41,8 +41,8 @@ test-blocking-io:
|
||||
# read from .test_durations. Shards only READ that file (no --store-durations),
|
||||
# so concurrent CI jobs cannot race writes on it. Refresh it with
|
||||
# `make test-shard-durations` after a meaningful change to the test set.
|
||||
# `make test-shard SPLITS=4 GROUP=2` runs shard 2 of 4; a bare `make test-shard`
|
||||
# runs the whole offline suite as a single split; `make test` runs all four shards.
|
||||
# `make test-shard SPLITS=6 GROUP=2` runs shard 2 of 6; a bare `make test-shard`
|
||||
# runs the whole offline suite as a single split; `make test` runs all six shards.
|
||||
SPLITS ?= 1
|
||||
GROUP ?= 1
|
||||
DURATIONS_FILE ?= .test_durations
|
||||
|
||||
@ -8,7 +8,7 @@ data. What a single test file covers is stated in that file, not here.
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
make test # four offline shards; TEST_JOBS=1 runs serially
|
||||
make test # six offline shards (matches CI); TEST_JOBS=1 runs serially
|
||||
make test-blocking-io # strict gate over tests/blocking_io/
|
||||
python -m pytest tests/test_<feature>.py::test_<name> -q
|
||||
```
|
||||
@ -29,7 +29,11 @@ with a reason when the binary is missing.
|
||||
the fix. Do not encode a known bug as expected behaviour, and do not hide it
|
||||
behind a permanent `xfail`.
|
||||
- **Assert on structure.** Check decoded payloads, stored rows and metadata.
|
||||
User-facing wording is asserted only where the wording is the contract.
|
||||
User-facing wording is asserted only where the wording is the contract. Do not
|
||||
assert on `inspect.getsource` or prose; enforce import boundaries statically.
|
||||
- **Grow module suites, not one-fix files.** Fold regression cases into the
|
||||
owning module's behavioral suite and delete or update obsolete tests when
|
||||
behavior changes.
|
||||
- **Scripts run as copies in a temporary checkout.** Never swap sources in the
|
||||
working tree during a test, and never let a test read the developer's own
|
||||
`config.yaml`, `.env` or runtime home.
|
||||
|
||||
@ -71,25 +71,25 @@ def run_make(project: Path, *args: str, fail_group: str = "", parallel: bool = T
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fail_group", ["", "1"])
|
||||
def test_make_test_runs_all_four_shards_and_reports_any_failure(test_project: Path, fail_group: str, monkeypatch) -> None:
|
||||
def test_make_test_runs_all_six_shards_and_reports_any_failure(test_project: Path, fail_group: str, monkeypatch) -> None:
|
||||
monkeypatch.setenv("TEST_JOBS", "2")
|
||||
result = run_make(test_project, "test", fail_group=fail_group)
|
||||
|
||||
assert (result.returncode == 0) == (not fail_group), result.stdout + result.stderr
|
||||
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4"}
|
||||
for group in range(1, 5):
|
||||
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4", "5", "6"}
|
||||
for group in range(1, 7):
|
||||
args = json.loads((test_project / f"{group}.started").read_text(encoding="utf-8"))
|
||||
assert args[:3] == ["run", "pytest", "-m"]
|
||||
assert args[args.index("-m") + 1] == "not live and not bundled_extension"
|
||||
assert "--ignore=tests/blocking_io" in args
|
||||
assert args[args.index("--splits") + 1] == "4"
|
||||
assert args[args.index("--splits") + 1] == "6"
|
||||
assert args[args.index("--group") + 1] == str(group)
|
||||
assert args[args.index("--splitting-algorithm") + 1] == "least_duration"
|
||||
assert "--store-durations" not in args
|
||||
|
||||
|
||||
def test_ci_can_still_run_one_shard(test_project: Path) -> None:
|
||||
result = run_make(test_project, "test-shard", "SPLITS=4", "GROUP=2", parallel=False)
|
||||
result = run_make(test_project, "test-shard", "SPLITS=6", "GROUP=2", parallel=False)
|
||||
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert {path.stem for path in test_project.glob("*.finished")} == {"2"}
|
||||
@ -99,7 +99,7 @@ def test_make_test_can_limit_concurrency(test_project: Path) -> None:
|
||||
result = run_make(test_project, "test", "TEST_JOBS=1", parallel=False)
|
||||
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4"}
|
||||
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4", "5", "6"}
|
||||
|
||||
|
||||
def test_make_test_reports_missing_durations(test_project: Path) -> None:
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user