docs(tests): align contributor test rules with module-suite hygiene and sync 6-shard config (#8361)

- Update AGENTS.md, backend/AGENTS.md, backend/tests/AGENTS.md, and PR template to fold regressions into owning module suites, prune superseded tests, and forbid inspect.getsource assertions.
- Align CONTRIBUTING.md, backend/tests/AGENTS.md, backend/Makefile, and test_make_test_parallel.py with the 6-shard CI configuration.
This commit is contained in:
Xinmin Zeng 2026-11-18 03:32:38 -08:00 committed by GitHub
parent a50663c2f1
commit 8d60bef1b4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 40 additions and 22 deletions

View File

@ -46,13 +46,23 @@ Fixes #
<!-- Skip (delete) this section if this PR is not a bug fix.
Bugs should be encoded as a failing test that goes red before the fix.
Fold regression cases into the owning module's existing suite instead of
creating a new test file per fix.
Confirm:
- Test path that reproduces the bug:
- Existing suite + test case that reproduces the bug:
- Did it go red on `main` and green on this branch? (yes / no)
- If a red test wasn't cheap to write, explain why and what you did instead. -->
## Test footprint
<!-- Keep CI shards within budget: add cases to the module's behavioral suite
and delete or update superseded tests in the same PR. -->
- **Net test line delta** (`+<added> / -<removed>` across test files):
- **Merged or pruned old tests?** (`yes` — list consolidated/deleted test paths, or `no` — explain why net growth is needed):
## Validation
<!-- What you actually ran. Run at least the checks for the area you changed:

View File

@ -86,7 +86,9 @@ make check-agent-guidance # size budget of these guides
because loading an extension imports code with Gateway privileges. Install
from trusted sources only, and restart the Gateway after every extension
change.
- **Ship tests with every feature and bug fix.**
- **Cover behavior changes in the owning module's suite.** Fold regression cases
into that module's existing suite instead of opening a one-fix test file, and
update or delete obsolete tests in the same change.
- **Document behavior next to its topic, in the same change**: user-facing
behavior in `docs/guide/` or `backend/docs/`. `README.md` is an index; change
it only to add or re-point a link.

View File

@ -350,7 +350,7 @@ before review.
python -m pip install -r tests/skills/requirements.txt
python -m pytest tests/skills -q
# Default backend tests (four parallel shards; excludes live, blocking-I/O and bundled-extension tests)
# Default backend tests (six parallel shards matching CI; excludes live, blocking-I/O and bundled-extension tests)
cd backend
make test
@ -406,7 +406,7 @@ setup also verifies SSL and SQLite imports before launching fixture gateways.
### Test and diagnostics notes
Backend `make test` runs four duration-balanced shards concurrently and reports
Backend `make test` runs six duration-balanced shards concurrently (matching CI) and reports
failure if any shard fails. Use `make test TEST_JOBS=1` to run the shards sequentially.
It excludes live external-API, blocking-I/O and bundled-extension coverage.
Run `cd backend && make test-blocking-io` for strict blocking-I/O checks.

View File

@ -35,9 +35,11 @@ make format
make test-live # calls real APIs; needs DEER_FLOW_RUN_LIVE_TESTS=1
```
Run `make test` and `make test-blocking-io` before and after a change. Every
feature and bug fix ships with a test in `tests/`. Live tests stay opt-in, so the
default suite needs no network and no API key.
Run `make test` and `make test-blocking-io` before and after a change. Cover
behavior changes in the owning module's existing suite under `tests/` rather
than opening a new file per fix, and update or delete superseded tests in the
same change. Live tests stay opt-in, so the default suite needs no network and
no API key.
## Rules

View File

@ -19,15 +19,15 @@ dev:
gateway:
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run --locked uvicorn app.gateway.app:app --host 127.0.0.1 --port 8001
# Run all four duration-balanced shards locally; wait for every shard on failure.
TEST_JOBS ?= 4
TEST_SHARD_TARGETS := test-shard-1 test-shard-2 test-shard-3 test-shard-4
# Run all six duration-balanced shards locally (matching CI); wait for every shard on failure.
TEST_JOBS ?= 6
TEST_SHARD_TARGETS := test-shard-1 test-shard-2 test-shard-3 test-shard-4 test-shard-5 test-shard-6
.PHONY: test test-shard test-bundled-extensions $(TEST_SHARD_TARGETS)
test:
$(MAKE) --no-print-directory -k -j$(TEST_JOBS) $(TEST_SHARD_TARGETS)
$(TEST_SHARD_TARGETS):
$(MAKE) --no-print-directory test-shard SPLITS=4 GROUP=$(patsubst test-shard-%,%,$@)
$(MAKE) --no-print-directory test-shard SPLITS=6 GROUP=$(patsubst test-shard-%,%,$@)
test-live:
DEER_FLOW_RUN_LIVE_TESTS=1 PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m live tests/ -v -s
@ -41,8 +41,8 @@ test-blocking-io:
# read from .test_durations. Shards only READ that file (no --store-durations),
# so concurrent CI jobs cannot race writes on it. Refresh it with
# `make test-shard-durations` after a meaningful change to the test set.
# `make test-shard SPLITS=4 GROUP=2` runs shard 2 of 4; a bare `make test-shard`
# runs the whole offline suite as a single split; `make test` runs all four shards.
# `make test-shard SPLITS=6 GROUP=2` runs shard 2 of 6; a bare `make test-shard`
# runs the whole offline suite as a single split; `make test` runs all six shards.
SPLITS ?= 1
GROUP ?= 1
DURATIONS_FILE ?= .test_durations

View File

@ -8,7 +8,7 @@ data. What a single test file covers is stated in that file, not here.
```bash
cd backend
make test # four offline shards; TEST_JOBS=1 runs serially
make test # six offline shards (matches CI); TEST_JOBS=1 runs serially
make test-blocking-io # strict gate over tests/blocking_io/
python -m pytest tests/test_<feature>.py::test_<name> -q
```
@ -29,7 +29,11 @@ with a reason when the binary is missing.
the fix. Do not encode a known bug as expected behaviour, and do not hide it
behind a permanent `xfail`.
- **Assert on structure.** Check decoded payloads, stored rows and metadata.
User-facing wording is asserted only where the wording is the contract.
User-facing wording is asserted only where the wording is the contract. Do not
assert on `inspect.getsource` or prose; enforce import boundaries statically.
- **Grow module suites, not one-fix files.** Fold regression cases into the
owning module's behavioral suite and delete or update obsolete tests when
behavior changes.
- **Scripts run as copies in a temporary checkout.** Never swap sources in the
working tree during a test, and never let a test read the developer's own
`config.yaml`, `.env` or runtime home.

View File

@ -71,25 +71,25 @@ def run_make(project: Path, *args: str, fail_group: str = "", parallel: bool = T
@pytest.mark.parametrize("fail_group", ["", "1"])
def test_make_test_runs_all_four_shards_and_reports_any_failure(test_project: Path, fail_group: str, monkeypatch) -> None:
def test_make_test_runs_all_six_shards_and_reports_any_failure(test_project: Path, fail_group: str, monkeypatch) -> None:
monkeypatch.setenv("TEST_JOBS", "2")
result = run_make(test_project, "test", fail_group=fail_group)
assert (result.returncode == 0) == (not fail_group), result.stdout + result.stderr
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4"}
for group in range(1, 5):
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4", "5", "6"}
for group in range(1, 7):
args = json.loads((test_project / f"{group}.started").read_text(encoding="utf-8"))
assert args[:3] == ["run", "pytest", "-m"]
assert args[args.index("-m") + 1] == "not live and not bundled_extension"
assert "--ignore=tests/blocking_io" in args
assert args[args.index("--splits") + 1] == "4"
assert args[args.index("--splits") + 1] == "6"
assert args[args.index("--group") + 1] == str(group)
assert args[args.index("--splitting-algorithm") + 1] == "least_duration"
assert "--store-durations" not in args
def test_ci_can_still_run_one_shard(test_project: Path) -> None:
result = run_make(test_project, "test-shard", "SPLITS=4", "GROUP=2", parallel=False)
result = run_make(test_project, "test-shard", "SPLITS=6", "GROUP=2", parallel=False)
assert result.returncode == 0, result.stdout + result.stderr
assert {path.stem for path in test_project.glob("*.finished")} == {"2"}
@ -99,7 +99,7 @@ def test_make_test_can_limit_concurrency(test_project: Path) -> None:
result = run_make(test_project, "test", "TEST_JOBS=1", parallel=False)
assert result.returncode == 0, result.stdout + result.stderr
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4"}
assert {path.stem for path in test_project.glob("*.finished")} == {"1", "2", "3", "4", "5", "6"}
def test_make_test_reports_missing_durations(test_project: Path) -> None: