Mohammed Ansari 867235389d
Add trace-based behavioral tests with Monocle Test Tools (#4025)
* Add Monocle behavioral test suite

Trace-based tests under tests/monocle/ asserting against the agent's Monocle execution traces: 4 offline tests load a recorded trace by file (with_trace_source), one per curated question, plus 1 live end-to-end test. Fluent structural asserts (agent, tools, input/output, token/duration budget); additive only, no app-code changes.

* Address review: restructure suite, move under backend/tests/monocle/

Responds to the review on this PR.

Behavioural coverage is now the live tests, not frozen fixtures. Keep one
offline test as a worked example of the fluent assertion API (loads a recorded
trace by file, no keys), and add two live tests that drive the agent end-to-end
through DeerFlowClient and assert on the trace the real run emits (web-research
and sandbox paths). The live tests skip without OPENAI_API_KEY or the app.

Other review fixes:
- Move the suite under backend/tests/monocle/ so backend pytest collects it.
- Split helpers into _helpers.py; conftest is fixtures-only (run_agent), with
  Monocle setup owned by the validator and .env load scoped to the live path.
- Resolve the model from config.yaml instead of hardcoding gpt-4o; skip the
  live tests when config.yaml is absent.
- Keep one trace with a stable name (web_research_ev_battery.json); drop the
  other three (removes ~2,200 lines of fixture blobs).
- Drop the flaky wall-clock duration bound on live runs.
- Keep monocle_test_tools in a standalone requirements.txt rather than the
  backend dev group: it hard-depends on the ML eval stack (torch, transformers,
  sentence-transformers, ~48 packages, +950 lines in uv.lock), so isolating it
  keeps the app's locked deps clean. importorskip skips the suite when absent.

* docs(monocle tests): explain the golden-trace workflow

Add a "How this is meant to be used" section: capture a run you are happy with
as a golden, labelled trace, turn it into assertions (the offline example), then
point the same assertions at the live agent so every later run has to reproduce
that behaviour.

* Address review: fix docstring pytest paths, state the suite is not run in CI

The docstring commands now use the backend/tests/monocle/ form (matching
the README, which also gains the backend-dir uv variant), and the README
states explicitly that the suite is skipped in CI and run on demand.

* Address review: document the committed trace, pin assertions context

- README: new section on the committed trace. It is a full, unmodified
  real-run recording (system prompt of the recording date + fetched web
  content, no credentials), committed whole so the offline example parses a
  genuine trace. The offline assertions are pinned to this trace and the
  monocle_apptrace 0.8.8 span shapes; re-record when prompt, tools, or model
  change.
- README: note that the monocle_trace_asserter fixture comes from
  monocle_test_tools' auto-registered pytest plugin (pytest11 entry point).
- test_deerflow.py: comment why web_fetch asserts min_count=2 rather than the
  recorded exact count of 5 (fetch counts vary run to run; keep it a floor).
- requirements.txt: loose pin python-dotenv>=1.0.

* Address review: make live tests explicit opt-in, drop OpenAI-only gate

Two execution-gating fixes from review:

- Live tests are now opt-in via MONOCLE_LIVE_TESTS=1 (default off). Previously
  the documented offline command collected the live tests too, and on a
  configured checkout (.env + config.yaml present) they would run for real,
  spending model tokens and hitting the network. Now the plain
  `pytest backend/tests/monocle/` run cannot go live regardless of what
  credentials are present; test_live_gate_defaults_off pins the gate.
- The run_agent fixture no longer requires OPENAI_API_KEY. config.yaml resolves
  the model, which may be any provider (Anthropic, Gemini, Volcengine, ...), so
  a hard-coded OpenAI gate skipped valid configurations and passed invalid
  ones. Credentials are validated by the configured model itself.

README and docstrings updated to match: offline command is offline by
construction, live is MONOCLE_LIVE_TESTS=1, credentials described as the
configured model's rather than OpenAI's.

Verified both modes: default run is 2 passed 2 skipped with no network; opted
in, all 4 pass with real end-to-end runs.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-19 18:26:26 +08:00

43 lines
1.5 KiB
Python

"""Fixtures for the DeerFlow Monocle behavioural tests.
Only fixtures live here. Paths and ``run_deerflow`` are in ``_helpers.py`` so
nothing imports ``conftest`` as a module. The ``sys.path`` insert (mirroring the
backend root ``conftest.py``) makes ``_helpers`` importable under any pytest
import mode. The ``.env`` load is scoped to the live fixture, so collecting or
running the offline test never reads secrets.
"""
from __future__ import annotations
import sys
from collections.abc import Callable
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent))
@pytest.fixture
def run_agent() -> Callable[[str], str]:
"""Live agent runner. Explicit opt-in, so a default run can never go live.
Skips unless ``MONOCLE_LIVE_TESTS=1`` is set, when the DeerFlow app is not
importable (e.g. a test-tools-only venv), or when ``config.yaml`` is absent.
Provider credentials are validated by the configured model itself —
``config.yaml`` may select any provider, not just OpenAI, so there is no
hard-coded key check here.
"""
from _helpers import CONFIG_PATH, REPO_ROOT, live_tests_enabled, run_deerflow
if not live_tests_enabled():
pytest.skip("live tests are opt-in: set MONOCLE_LIVE_TESTS=1")
pytest.importorskip("deerflow", reason="DeerFlow app not importable in this venv")
from dotenv import load_dotenv
load_dotenv(REPO_ROOT / ".env")
if not CONFIG_PATH.exists():
pytest.skip(f"config.yaml not found at {CONFIG_PATH}")
return run_deerflow