mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-26 16:07:53 +00:00
* Add Monocle behavioral test suite Trace-based tests under tests/monocle/ asserting against the agent's Monocle execution traces: 4 offline tests load a recorded trace by file (with_trace_source), one per curated question, plus 1 live end-to-end test. Fluent structural asserts (agent, tools, input/output, token/duration budget); additive only, no app-code changes. * Address review: restructure suite, move under backend/tests/monocle/ Responds to the review on this PR. Behavioural coverage is now the live tests, not frozen fixtures. Keep one offline test as a worked example of the fluent assertion API (loads a recorded trace by file, no keys), and add two live tests that drive the agent end-to-end through DeerFlowClient and assert on the trace the real run emits (web-research and sandbox paths). The live tests skip without OPENAI_API_KEY or the app. Other review fixes: - Move the suite under backend/tests/monocle/ so backend pytest collects it. - Split helpers into _helpers.py; conftest is fixtures-only (run_agent), with Monocle setup owned by the validator and .env load scoped to the live path. - Resolve the model from config.yaml instead of hardcoding gpt-4o; skip the live tests when config.yaml is absent. - Keep one trace with a stable name (web_research_ev_battery.json); drop the other three (removes ~2,200 lines of fixture blobs). - Drop the flaky wall-clock duration bound on live runs. - Keep monocle_test_tools in a standalone requirements.txt rather than the backend dev group: it hard-depends on the ML eval stack (torch, transformers, sentence-transformers, ~48 packages, +950 lines in uv.lock), so isolating it keeps the app's locked deps clean. importorskip skips the suite when absent. * docs(monocle tests): explain the golden-trace workflow Add a "How this is meant to be used" section: capture a run you are happy with as a golden, labelled trace, turn it into assertions (the offline example), then point the same assertions at the live agent so every later run has to reproduce that behaviour. * Address review: fix docstring pytest paths, state the suite is not run in CI The docstring commands now use the backend/tests/monocle/ form (matching the README, which also gains the backend-dir uv variant), and the README states explicitly that the suite is skipped in CI and run on demand. * Address review: document the committed trace, pin assertions context - README: new section on the committed trace. It is a full, unmodified real-run recording (system prompt of the recording date + fetched web content, no credentials), committed whole so the offline example parses a genuine trace. The offline assertions are pinned to this trace and the monocle_apptrace 0.8.8 span shapes; re-record when prompt, tools, or model change. - README: note that the monocle_trace_asserter fixture comes from monocle_test_tools' auto-registered pytest plugin (pytest11 entry point). - test_deerflow.py: comment why web_fetch asserts min_count=2 rather than the recorded exact count of 5 (fetch counts vary run to run; keep it a floor). - requirements.txt: loose pin python-dotenv>=1.0. * Address review: make live tests explicit opt-in, drop OpenAI-only gate Two execution-gating fixes from review: - Live tests are now opt-in via MONOCLE_LIVE_TESTS=1 (default off). Previously the documented offline command collected the live tests too, and on a configured checkout (.env + config.yaml present) they would run for real, spending model tokens and hitting the network. Now the plain `pytest backend/tests/monocle/` run cannot go live regardless of what credentials are present; test_live_gate_defaults_off pins the gate. - The run_agent fixture no longer requires OPENAI_API_KEY. config.yaml resolves the model, which may be any provider (Anthropic, Gemini, Volcengine, ...), so a hard-coded OpenAI gate skipped valid configurations and passed invalid ones. Credentials are validated by the configured model itself. README and docstrings updated to match: offline command is offline by construction, live is MONOCLE_LIVE_TESTS=1, credentials described as the configured model's rather than OpenAI's. Verified both modes: default run is 2 passed 2 skipped with no network; opted in, all 4 pass with real end-to-end runs. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
43 lines
1.5 KiB
Python
43 lines
1.5 KiB
Python
"""Fixtures for the DeerFlow Monocle behavioural tests.
|
|
|
|
Only fixtures live here. Paths and ``run_deerflow`` are in ``_helpers.py`` so
|
|
nothing imports ``conftest`` as a module. The ``sys.path`` insert (mirroring the
|
|
backend root ``conftest.py``) makes ``_helpers`` importable under any pytest
|
|
import mode. The ``.env`` load is scoped to the live fixture, so collecting or
|
|
running the offline test never reads secrets.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
from collections.abc import Callable
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
|
|
|
|
|
@pytest.fixture
|
|
def run_agent() -> Callable[[str], str]:
|
|
"""Live agent runner. Explicit opt-in, so a default run can never go live.
|
|
|
|
Skips unless ``MONOCLE_LIVE_TESTS=1`` is set, when the DeerFlow app is not
|
|
importable (e.g. a test-tools-only venv), or when ``config.yaml`` is absent.
|
|
Provider credentials are validated by the configured model itself —
|
|
``config.yaml`` may select any provider, not just OpenAI, so there is no
|
|
hard-coded key check here.
|
|
"""
|
|
from _helpers import CONFIG_PATH, REPO_ROOT, live_tests_enabled, run_deerflow
|
|
|
|
if not live_tests_enabled():
|
|
pytest.skip("live tests are opt-in: set MONOCLE_LIVE_TESTS=1")
|
|
pytest.importorskip("deerflow", reason="DeerFlow app not importable in this venv")
|
|
|
|
from dotenv import load_dotenv
|
|
|
|
load_dotenv(REPO_ROOT / ".env")
|
|
if not CONFIG_PATH.exists():
|
|
pytest.skip(f"config.yaml not found at {CONFIG_PATH}")
|
|
return run_deerflow
|