mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-26 16:07:53 +00:00
* Add Monocle behavioral test suite Trace-based tests under tests/monocle/ asserting against the agent's Monocle execution traces: 4 offline tests load a recorded trace by file (with_trace_source), one per curated question, plus 1 live end-to-end test. Fluent structural asserts (agent, tools, input/output, token/duration budget); additive only, no app-code changes. * Address review: restructure suite, move under backend/tests/monocle/ Responds to the review on this PR. Behavioural coverage is now the live tests, not frozen fixtures. Keep one offline test as a worked example of the fluent assertion API (loads a recorded trace by file, no keys), and add two live tests that drive the agent end-to-end through DeerFlowClient and assert on the trace the real run emits (web-research and sandbox paths). The live tests skip without OPENAI_API_KEY or the app. Other review fixes: - Move the suite under backend/tests/monocle/ so backend pytest collects it. - Split helpers into _helpers.py; conftest is fixtures-only (run_agent), with Monocle setup owned by the validator and .env load scoped to the live path. - Resolve the model from config.yaml instead of hardcoding gpt-4o; skip the live tests when config.yaml is absent. - Keep one trace with a stable name (web_research_ev_battery.json); drop the other three (removes ~2,200 lines of fixture blobs). - Drop the flaky wall-clock duration bound on live runs. - Keep monocle_test_tools in a standalone requirements.txt rather than the backend dev group: it hard-depends on the ML eval stack (torch, transformers, sentence-transformers, ~48 packages, +950 lines in uv.lock), so isolating it keeps the app's locked deps clean. importorskip skips the suite when absent. * docs(monocle tests): explain the golden-trace workflow Add a "How this is meant to be used" section: capture a run you are happy with as a golden, labelled trace, turn it into assertions (the offline example), then point the same assertions at the live agent so every later run has to reproduce that behaviour. * Address review: fix docstring pytest paths, state the suite is not run in CI The docstring commands now use the backend/tests/monocle/ form (matching the README, which also gains the backend-dir uv variant), and the README states explicitly that the suite is skipped in CI and run on demand. * Address review: document the committed trace, pin assertions context - README: new section on the committed trace. It is a full, unmodified real-run recording (system prompt of the recording date + fetched web content, no credentials), committed whole so the offline example parses a genuine trace. The offline assertions are pinned to this trace and the monocle_apptrace 0.8.8 span shapes; re-record when prompt, tools, or model change. - README: note that the monocle_trace_asserter fixture comes from monocle_test_tools' auto-registered pytest plugin (pytest11 entry point). - test_deerflow.py: comment why web_fetch asserts min_count=2 rather than the recorded exact count of 5 (fetch counts vary run to run; keep it a floor). - requirements.txt: loose pin python-dotenv>=1.0. * Address review: make live tests explicit opt-in, drop OpenAI-only gate Two execution-gating fixes from review: - Live tests are now opt-in via MONOCLE_LIVE_TESTS=1 (default off). Previously the documented offline command collected the live tests too, and on a configured checkout (.env + config.yaml present) they would run for real, spending model tokens and hitting the network. Now the plain `pytest backend/tests/monocle/` run cannot go live regardless of what credentials are present; test_live_gate_defaults_off pins the gate. - The run_agent fixture no longer requires OPENAI_API_KEY. config.yaml resolves the model, which may be any provider (Anthropic, Gemini, Volcengine, ...), so a hard-coded OpenAI gate skipped valid configurations and passed invalid ones. Credentials are validated by the configured model itself. README and docstrings updated to match: offline command is offline by construction, live is MONOCLE_LIVE_TESTS=1, credentials described as the configured model's rather than OpenAI's. Verified both modes: default run is 2 passed 2 skipped with no network; opted in, all 4 pass with real end-to-end runs. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
43 lines
1.5 KiB
Python
43 lines
1.5 KiB
Python
"""Helpers for the DeerFlow Monocle behavioural tests.
|
|
|
|
Kept out of ``conftest.py`` so nothing imports ``conftest`` as a module.
|
|
Monocle instrumentation is owned by the Test Tools validator (installed by the
|
|
``monocle_trace_asserter`` fixture), so ``run_deerflow`` only drives the agent;
|
|
the already-installed instrumentation captures the run's spans.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import uuid
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).resolve().parent
|
|
TRACES = HERE / "traces"
|
|
REPO_ROOT = HERE.parents[2] # backend/tests/monocle -> backend/tests -> backend -> repo root
|
|
CONFIG_PATH = REPO_ROOT / "config.yaml"
|
|
|
|
_TRUTHY = {"1", "true", "yes", "on"}
|
|
|
|
|
|
def live_tests_enabled() -> bool:
|
|
"""Whether the live tests are explicitly opted into via ``MONOCLE_LIVE_TESTS``.
|
|
|
|
Off by default so the plain ``pytest backend/tests/monocle/`` run can never
|
|
spend model tokens, hit the network, or write to a sandbox — even on a fully
|
|
configured checkout where credentials and ``config.yaml`` are present.
|
|
"""
|
|
return os.getenv("MONOCLE_LIVE_TESTS", "").strip().lower() in _TRUTHY
|
|
|
|
|
|
def run_deerflow(message: str) -> str:
|
|
"""Run the DeerFlow agent once and return its response text.
|
|
|
|
The model is resolved from ``config.yaml`` (no hardcoded override) so the
|
|
live test exercises DeerFlow's own model-resolution path.
|
|
"""
|
|
from deerflow.client import DeerFlowClient
|
|
|
|
client = DeerFlowClient(config_path=str(CONFIG_PATH))
|
|
return client.chat(message, thread_id=f"monocle-test-{uuid.uuid4().hex[:8]}")
|