feat(ci): split backend unit tests into parallel shards (#5137)

* feat(ci): split backend unit tests into parallel CI shards

Split the single offline backend `make test` job into four GitHub Actions
matrix shards (SPLITS=4, GROUP=1..4) via pytest-split, so the ~12k-test suite
runs in parallel instead of in one 15-minute job. Each shard runs on its own
runner with its own Postgres/Redis services; fail-fast: false lets a failing
shard report its owned tests without cancelling its peers.

`make test` stays the canonical full-suite entry point; CI now calls the new
`make test-shard SPLITS=4 GROUP=N`. tests/blocking_io remains owned solely by
the dedicated blocking-I/O workflow (excluded via --ignore), extending #5105.

Fixes #5088

* test(ci): make backend test shards duration-aware and pin the contract

Make `make test-shard` an explicit least_duration split that READS
backend/.test_durations (read-only for shards, so concurrent CI jobs never
race writes on it), and add `make test-shard-durations` to regenerate that
file from the full offline suite. Update the CI unit-test workflow contract to
call `make test-shard SPLITS=4 GROUP=<n>` and assert the shard command carries
--splits 4, --group 2, -m "not live", --ignore=tests/blocking_io and
--splitting-algorithm least_duration. Verified on the real 13,140-test normal
suite that the four shards are pairwise disjoint and their union equals the
unsplit suite.

Refs #5088

* test(ci): fail fast when the duration baseline is missing

`make test-shard` now requires backend/.test_durations and exits with a clear
error instead of letting pytest-split silently degrade to an even (count-based)
split. Harden the CI contract test to pin `--durations-path=.test_durations` and
to assert the repo ships the committed duration baseline.

Refs #5088

* docs: trim backend/AGENTS.md within guidance budget

* test(ci): add backend test duration baseline

Add the duration baseline generated by a full offline backend run on a
GitHub-hosted ubuntu-latest runner (the same runner type the shards use), so
`make test-shard` balances the four matrix shards by real wall-clock cost.

Refs #5088

* test(ci): make the duration writer honor DURATIONS_FILE

`test-shard-durations` now writes `--durations-path=$(DURATIONS_FILE)` instead of a
hard-coded .test_durations, so the reader and writer stay consistent when the
path is overridden.

Refs #5088

* test(ci): address sharding review feedback

* test: isolate subagent execution capacity state

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

.github/workflows/backend-unit-tests.yml

@@ -49,6 +49,15 @@ jobs:
     if: github.event.pull_request.draft == false
     runs-on: ubuntu-latest
     timeout-minutes: 15
+    strategy:
+      # Duration-aware split: `make test-shard` balances the shards by real
+      # wall-clock cost from backend/.test_durations (read-only for shards, so
+      # concurrent jobs never race writes on it). Every test runs exactly once.
+      # Fail-fast is off so a failing shard reports its tests without cancelling
+      # its peers.
+      fail-fast: false
+      matrix:
+        shard: [1, 2, 3, 4]
 
     services:
       postgres:
@@ -96,7 +105,7 @@ jobs:
         working-directory: backend
         run: uv sync --group dev --extra postgres
 
-      - name: Run unit tests of backend
+      - name: Run unit tests of backend (shard ${{ matrix.shard }} of 4)
         working-directory: backend
         env:
           # Expose the job's Postgres service to the cross-pod dedupe integration
@@ -104,4 +113,4 @@ jobs:
           # mapping those tests silently skip and the headline capability is never
           # exercised in CI.
           DEDUPE_TEST_POSTGRES_URL: ${{ env.TEST_POSTGRES_URI }}
-        run: make test
+        run: make test-shard SPLITS=4 GROUP=${{ matrix.shard }}

backend/AGENTS.md

@@ -149,13 +149,15 @@ make stop       # Stop all services
 **Backend directory** (for backend development only):
 ```bash
 make install            # Install backend dependencies
-make dev                # Run Gateway API with runtime-safe reload (port 8001)
-make gateway            # Run Gateway API only (port 8001)
-make test               # Run offline backend tests (excludes live and blocking-I/O tests)
-make test-live          # Explicitly run live DeerFlowClient tests with real APIs
-make test-blocking-io   # Run strict Blockbuster runtime gate on tests/blocking_io/
-make lint               # Lint with ruff
-make format             # Format code with ruff
+make dev                # Gateway API, reload (port 8001)
+make gateway            # Gateway API only (port 8001)
+make test               # offline tests (no live/blocking-io)
+make test-live          # live tests (real APIs)
+make test-blocking-io   # strict Blockbuster gate on tests/blocking_io/
+make test-shard SPLITS=4 GROUP=2  # one duration-aware shard
+make test-shard-durations  # refresh baseline
+make lint               # ruff lint
+make format             # ruff format
 make migrate-rev MSG="..."  # Autogenerate a new alembic revision (see Schema Migrations section)
 ```
 

backend/Makefile

@@ -28,6 +28,28 @@ test-live:
 test-blocking-io:
 	PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/blocking_io -q --tb=short
 
+# Run the backend unit-test suite as one shard of a parallel CI split.
+#   SPLITS = total number of shards; GROUP = 1-based shard index.
+# Duration-aware: `least_duration` balances the shards by real wall-clock cost
+# read from .test_durations. Shards only READ that file (no --store-durations),
+# so concurrent CI jobs cannot race writes on it. Refresh it with
+# `make test-shard-durations` after a meaningful change to the test set.
+# `make test-shard SPLITS=4 GROUP=2` runs shard 2 of 4; a bare `make test-shard`
+# runs the whole (non-live) suite as a single split, matching `make test`.
+SPLITS ?= 1
+GROUP ?= 1
+DURATIONS_FILE ?= .test_durations
+test-shard:
+	@test -f "$(DURATIONS_FILE)" || \
+		(echo "error: $(DURATIONS_FILE) is missing; run 'make test-shard-durations' to regenerate the duration baseline" >&2; exit 1)
+	PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live" --ignore=tests/blocking_io --splits $(SPLITS) --group $(GROUP) --splitting-algorithm least_duration --durations-path=$(DURATIONS_FILE) tests/ -q
+
+# Regenerate .test_durations from the full offline (non-live, non-blocking-I/O)
+# suite so duration-aware sharding reflects real wall-clock cost. Run locally or
+# in a periodic job, then commit the refreshed file.
+test-shard-durations:
+	PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest -m "not live" --ignore=tests/blocking_io --store-durations --clean-durations --durations-path=$(DURATIONS_FILE) tests/ -q
+
 lint:
 	uv run ruff check .
 	uv run ruff format --check .

backend/pyproject.toml

@@ -62,6 +62,7 @@ dev = [
     "prompt-toolkit>=3.0.0",
     "pytest>=9.0.3",
     "pytest-asyncio>=1.3.0",
+    "pytest-split>=0.11.0",
     "ruff>=0.14.11",
     # Monocle tracer (also the deerflow-harness[monocle] extra); kept in the dev
     # group so the tracing tests can import it without forcing it onto installs.

backend/tests/test_client_live_policy.py

@@ -85,9 +85,9 @@ def _assert_collection_skipped(result: subprocess.CompletedProcess[str], reason:
     assert all(node_id not in output for node_id in REPRESENTATIVE_LIVE_NODE_IDS)
 
 
-def _dry_run_make_target(target: str) -> str:
+def _dry_run_make_target(target: str, *extra: str) -> str:
     result = subprocess.run(
-        ["make", "-n", target],
+        ["make", "-n", target, *extra],
         cwd=BACKEND_ROOT,
         capture_output=True,
         text=True,
@@ -158,10 +158,50 @@ def test_documentation_matches_live_test_commands() -> None:
 def test_default_ci_workflow_does_not_opt_in_to_live_tests() -> None:
     workflow = (REPO_ROOT / ".github" / "workflows" / "backend-unit-tests.yml").read_text(encoding="utf-8")
 
-    assert "make test" in workflow
+    # Assert the exact sharded caller, not the substring `make test` (which
+    # `make test-shard` also contains). Otherwise a regression that re-points the
+    # workflow at live tests could slip through as a vacuous substring check.
+    assert "make test-shard" in workflow
     assert LIVE_OPT_IN not in workflow
 
 
+def test_ci_unit_test_workflow_runs_duration_aware_shards() -> None:
+    workflow = (REPO_ROOT / ".github" / "workflows" / "backend-unit-tests.yml").read_text(encoding="utf-8")
+
+    assert "shard: [1, 2, 3, 4]" in workflow
+    assert "make test-shard SPLITS=4" in workflow
+    assert "GROUP=${{ matrix.shard }}" in workflow
+
+    # The sharded command must select the offline suite and balance the shards by
+    # real wall-clock cost (duration-aware), not fall back to an even split.
+    command = _dry_run_make_target("test-shard", "SPLITS=4", "GROUP=2")
+
+    assert "--splits 4" in command
+    assert "--group 2" in command
+    assert 'pytest -m "not live"' in command
+    assert "--ignore=tests/blocking_io" in command
+    assert "--splitting-algorithm least_duration" in command
+    assert "--durations-path=.test_durations" in command
+
+    # The repo must ship a duration baseline. Without it pytest-split silently
+    # degrades to an even (count-based) split, so this is a fail-closed contract.
+    assert (BACKEND_ROOT / ".test_durations").exists()
+
+    # The full-suite entry point stays offline-only.
+    full_command = _dry_run_make_target("test")
+    assert 'pytest -m "not live"' in full_command
+    assert "--ignore=tests/blocking_io" in full_command
+    assert LIVE_OPT_IN not in full_command
+
+
+def test_duration_baseline_target_replaces_stale_entries() -> None:
+    command = _dry_run_make_target("test-shard-durations")
+
+    assert "--store-durations" in command
+    assert "--clean-durations" in command
+    assert "--durations-path=.test_durations" in command
+
+
 def test_blocking_io_ci_workflow_owns_dedicated_suite() -> None:
     command = _dry_run_make_target("test-blocking-io")
     workflow = (REPO_ROOT / ".github" / "workflows" / "backend-blocking-io-tests.yml").read_text(encoding="utf-8")

backend/tests/test_skills_custom_router.py

@@ -992,6 +992,8 @@ def test_public_skill_toggle_clears_all_users_cache(monkeypatch, tmp_path):
 
 
 def test_public_skill_toggle_creates_missing_extensions_config(monkeypatch, tmp_path):
+    from deerflow.config.extensions_config import ExtensionsConfig
+
     backend_dir = tmp_path / "backend"
     backend_dir.mkdir()
     monkeypatch.chdir(backend_dir)
@@ -1006,6 +1008,7 @@ def test_public_skill_toggle_creates_missing_extensions_config(monkeypatch, tmp_
 
     storage = SimpleNamespace(load_skills=_load_skills)
     monkeypatch.setattr(skills_router, "_get_user_skill_storage", lambda _config: storage)
+    monkeypatch.setattr(skills_router, "get_extensions_config", lambda: ExtensionsConfig())
 
     def _resolve_config_path(explicit_path=None):
         if explicit_path is None:

backend/tests/test_subagent_execution_capacity.py

@@ -12,6 +12,13 @@ from deerflow.subagents.capacity import (
 )
 
 
+@pytest.fixture(autouse=True)
+def restore_default_execution_capacity():
+    """Keep capacity scenarios from changing admission for later tests."""
+    yield
+    configure_subagent_execution_capacity(SubagentRuntimeConfig())
+
+
 @pytest.mark.asyncio
 async def test_capacity_queues_without_starting_more_than_configured_slots() -> None:
     configure_subagent_execution_capacity(SubagentRuntimeConfig(max_running=1, max_queued=2, queue_timeout_seconds=5))

backend/tests/test_subagent_executor.py

@@ -2196,8 +2196,19 @@ class TestThreadSafety:
         """Test multiple executors running in parallel via thread pool."""
         from concurrent.futures import ThreadPoolExecutor, as_completed
 
+        from deerflow.config.subagent_runtime_config import SubagentRuntimeConfig
+        from deerflow.subagents.capacity import SubagentExecutionCapacity
+
         SubagentExecutor = classes["SubagentExecutor"]
         SubagentStatus = classes["SubagentStatus"]
+        capacity = SubagentExecutionCapacity(
+            SubagentRuntimeConfig(
+                max_running=3,
+                max_queued=64,
+                admission_policy="queue",
+                queue_timeout_seconds=300,
+            )
+        )
 
         results = []
 
@@ -2221,6 +2232,7 @@ class TestThreadSafety:
                 config=base_config,
                 tools=[],
                 thread_id=f"thread-{task_id}",
+                execution_capacity=capacity,
             )
 
             with patch.object(executor, "_create_agent", return_value=mock_agent):

backend/uv.lock

@@ -859,6 +859,7 @@ dev = [
     { name = "prompt-toolkit" },
     { name = "pytest" },
     { name = "pytest-asyncio" },
+    { name = "pytest-split" },
     { name = "redis" },
     { name = "ruff" },
     { name = "textual" },
@@ -904,6 +905,7 @@ dev = [
     { name = "prompt-toolkit", specifier = ">=3.0.0" },
     { name = "pytest", specifier = ">=9.0.3" },
     { name = "pytest-asyncio", specifier = ">=1.3.0" },
+    { name = "pytest-split", specifier = ">=0.11.0" },
     { name = "redis", specifier = ">=5.0.0" },
     { name = "ruff", specifier = ">=0.14.11" },
     { name = "textual", specifier = ">=0.80" },
@@ -3968,6 +3970,18 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/e5/35/f8b19922b6a25bc0880171a2f1a003eaeb93657475193ab516fd87cac9da/pytest_asyncio-1.3.0-py3-none-any.whl", hash = "sha256:611e26147c7f77640e6d0a92a38ed17c3e9848063698d5c93d5aa7aa11cebff5", size = 15075, upload-time = "2025-11-10T16:07:45.537Z" },
 ]
 
+[[package]]
+name = "pytest-split"
+version = "0.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pytest" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2f/16/8af4c5f2ceb3640bb1f78dfdf5c184556b10dfe9369feaaad7ff1c13f329/pytest_split-0.11.0.tar.gz", hash = "sha256:8ebdb29cc72cc962e8eb1ec07db1eeb98ab25e215ed8e3216f6b9fc7ce0ec2b5", size = 13421, upload-time = "2026-02-03T09:14:31.469Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ae/a1/d4423657caaa8be9b31e491592b49cebdcfd434d3e74512ce71f6ec39905/pytest_split-0.11.0-py3-none-any.whl", hash = "sha256:899d7c0f5730da91e2daf283860eb73b503259cb416851a65599368849c7f382", size = 11911, upload-time = "2026-02-03T09:14:33.708Z" },
+]
+
 [[package]]
 name = "python-dateutil"
 version = "2.9.0.post0"