mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-08-01 19:06:01 +00:00
* feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel
Add a restart-required database.checkpoint_channel_mode ("full" default,
"delta") that stores the messages channel via LangGraph 1.2 DeltaChannel,
cutting checkpoint storage from O(n^2) to O(n) for append-only history.
Existing full checkpoints seed delta state transparently; no data migration.
- config: mode schema + freeze-on-first-use with
CheckpointModeReconfigurationError; mode marker persisted in checkpoint
metadata; unsafe delta->full downgrade rejected fail-closed with
CheckpointModeMismatchError (run-level error, failed state read)
- state: delta message state schema; CheckpointStateAccessor centralizes
materialized reads for all consumers (threads API, branches,
regeneration, compaction, state updates, memory, goal workers)
- runtime: raw writers (run durations, interrupted title, thread goal)
parent their checkpoints to the checkpoint they derive from, preserving
delta ancestry; rollback forks the pre-run lineage through a state
mutation graph with Overwrite restores; InMemorySaver delta-history
override delegates to the base walk (fixes dropped first write after
migration, also present upstream)
- tests: conformance suite over {memory, sqlite, postgres} covering
migration replay, stable message IDs, storage shape and writer
preservation; conftest fixture isolates the frozen mode between tests;
stale config fakes refreshed
- ci: backend unit tests gain a postgres service
* fix(checkpoint): close materialization gaps in goal flow, guard public factory
- Route goal-continuation message reads through CheckpointStateAccessor:
raw channel_values reads see the delta sentinel in delta mode, which
disabled goal continuation (stand_down=no_durable_end_of_turn) after
durable assistant turns. Raw tuples remain for tuple-only metadata
(checkpoint id, pending_writes).
- Reject checkpoint_channel_mode='delta' + checkpointer in
create_deerflow_agent at construction: factory-built persisted graphs
bypass mode-marker injection and the fail-closed gate, reproducing
silent mixed-mode state loss. Delta without persistence stays allowed.
- Import the postgres saver lazily (pytest.importorskip in the fixture)
so the documented default install collects the suite; add a CI job
running pytest --collect-only on uv sync --group dev without extras.
- Fix test_checkpointer fallback test to patch get_app_config at its
use site (provider module), making it deterministic when a local
config.yaml selects a persistent backend.
* fix(gateway): preserve extension-owned channels in state mutations, bump config version
- build_state_mutation_graph / build_checkpoint_state_mutation_accessor
accept an explicit state_schema; branch and POST /state now compile the
mutation graph from the thread's effective schema
(graph_state_schema on the assistant graph). The base-ThreadState
fallback silently discarded channels contributed by custom
AgentMiddleware.state_schema on branch (data loss) and returned a
false-success 200 on POST /state.
- POST /state validates values keys against the mutation graph's
channels and rejects unknown fields with 422 instead of ignoring
them; reducer detection covers extension channels
(BinaryOperatorAggregate or DeltaChannel) so Overwrite replace
semantics work for middleware reducers in both modes.
- Endpoint regression: custom AgentMiddleware.state_schema value
survives branch, updates through POST /state, and an unknown field
receives 422.
- config_version 26 -> 27 for the new database.checkpoint_channel_mode
(example, Helm chart values + README, support-bundle fixture), so
existing installs get the outdated-config warning and
make config-upgrade merges the field; covered by a test driving the
real example file and the real config-upgrade script.
* fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite
GET /threads/{id}/state now resolves the thread's assistant_id through a
single reusable boundary (thread metadata -> assistant_id -> effective
graph), so channels contributed by a custom AgentMiddleware.state_schema
are materialized instead of dropped by the default lead schema. POST
/state uses the same boundary instead of resolving the schema ad hoc.
Branch writes wrap every copied reducer channel in Overwrite (derived
from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel),
not just messages, so already-aggregated values are never re-merged.
Regression tests use a real AgentMiddleware.state_schema with a
non-identity reducer in both full and delta modes: GET /state returns the
extension value, POST /state replaces it, branch preserves it
byte-for-byte; the unknown-field 422 is a separate assertion.
* refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests
Address review round 4 on PR #4292:
- Push ahistory/history limit through Pregel into checkpointer.alist
(SQL LIMIT) instead of materializing all rows and breaking in Python
- Fold the read-side mode-compat gate onto the returned snapshot's
metadata; only writes keep the pre-write tuple fetch (fail-closed)
- Cache factory-built accessor graphs per (assistant_id, mode) with
factory-identity revalidation; state reads no longer build a lead
agent per request
- get_thread: one snapshot fetch + one raw pending_writes fetch on the
resolved checkpoint (post-checkpoint __error__ writes never surface
in snapshot.tasks; verified empirically)
- DeerFlowClient.get_thread: single checkpointer.list walk collects
pending_writes per checkpoint instead of N get_tuple calls
- InMemorySaver delta-history patch: stand-down when the upstream
override disappears, try/except guard, validated-version warning,
guard tests
- make_lead_agent mode precedence: first freeze is owned by app_config
(client-supplied configurable key ignored); once frozen, injected
key/app_config must match or fail closed
- Rollback: lock in non-message channel restoration via fork
inheritance with a dedicated reducer-channel test
- Add tests/test_threads_checkpoint_mode.py and
tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and
the PR validation section: lifecycle parity (memory + sqlite),
per-step blob-count storage guard, gateway endpoint parity
Counted-saver tests pin checkpoint round-trips for aget/ahistory so
these regressions cannot silently return.
* fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience
- threads router: map CheckpointModeMismatchError to 409 (with cause and
thread id) and CheckpointModeReconfigurationError to 503 across all state
endpoints instead of swallowing both into a generic 500
- gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and
assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary
through the real route stack
- rollback: compile the restore mutation graph with the thread's effective
state schema per the build_state_mutation_graph contract
- inheritance contract locks: rollback and manual compaction preserve
middleware-contributed channels via checkpoint fork cloning
- services: revalidate the accessor-graph cache against app_config identity
so config.yaml hot-reloads never serve a stale compiled graph
- services: degrade full-mode state reads to raw checkpointer reads when the
agent factory is unavailable (delta gate still applies; delta mode has no
fallback)
- deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently
downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility)
and bump the langchain lower bound to what the lockfile actually resolves
* fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread
- _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while
pregel's get_state_history treats config.checkpoint_id as the inclusive
start; fetch the anchor explicitly so both read paths paginate identically
- extend the degraded-path gateway test: GET /thread returns raw values, and
POST /history with before starts at the anchor checkpoint
* fix(gateway): preserve degraded checkpoint timestamps
* fix(gateway): harden degraded checkpoint access
* fix(gateway): resolve assistants for checkpoint reads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
164 lines
5.6 KiB
Python
164 lines
5.6 KiB
Python
"""Test configuration for the backend test suite.
|
|
|
|
Sets up sys.path and pre-mocks modules that would cause circular import
|
|
issues when unit-testing lightweight config/registry code in isolation.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
import sys
|
|
from pathlib import Path
|
|
from types import SimpleNamespace
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
# Make 'app' and 'deerflow' importable from any working directory
|
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
|
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "scripts"))
|
|
|
|
# Break the circular import chain that exists in production code:
|
|
# deerflow.subagents.__init__
|
|
# -> .executor (SubagentExecutor, SubagentResult)
|
|
# -> deerflow.agents.thread_state
|
|
# -> deerflow.agents.__init__
|
|
# -> lead_agent.agent
|
|
# -> subagent_limit_middleware
|
|
# -> deerflow.subagents.executor <-- circular!
|
|
#
|
|
# By injecting a mock for deerflow.subagents.executor *before* any test module
|
|
# triggers the import, __init__.py's "from .executor import ..." succeeds
|
|
# immediately without running the real executor module.
|
|
_executor_mock = MagicMock()
|
|
_executor_mock.SubagentExecutor = MagicMock
|
|
_executor_mock.SubagentResult = MagicMock
|
|
_executor_mock.SubagentStatus = MagicMock
|
|
_executor_mock.MAX_CONCURRENT_SUBAGENTS = 3
|
|
_executor_mock.get_background_task_result = MagicMock()
|
|
|
|
sys.modules["deerflow.subagents.executor"] = _executor_mock
|
|
|
|
|
|
@pytest.fixture()
|
|
def provisioner_module():
|
|
"""Load docker/provisioner/app.py as an importable test module.
|
|
|
|
Shared by test_provisioner_kubeconfig and test_provisioner_pvc_volumes so
|
|
that any change to the provisioner entry-point path or module name only
|
|
needs to be updated in one place.
|
|
"""
|
|
repo_root = Path(__file__).resolve().parents[2]
|
|
module_path = repo_root / "docker" / "provisioner" / "app.py"
|
|
spec = importlib.util.spec_from_file_location("provisioner_app_test", module_path)
|
|
assert spec is not None
|
|
assert spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
previous_module = sys.modules.get(spec.name)
|
|
sys.modules[spec.name] = module
|
|
try:
|
|
spec.loader.exec_module(module)
|
|
yield module
|
|
finally:
|
|
if previous_module is None:
|
|
sys.modules.pop(spec.name, None)
|
|
else:
|
|
sys.modules[spec.name] = previous_module
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Auto-set user context for every test unless marked no_auto_user
|
|
# ---------------------------------------------------------------------------
|
|
#
|
|
# Repository methods read ``user_id`` from a contextvar by default
|
|
# (see ``deerflow.runtime.user_context``). Without this fixture, every
|
|
# pre-existing persistence test would raise RuntimeError because the
|
|
# contextvar is unset. The fixture sets a default test user on every
|
|
# test; tests that explicitly want to verify behaviour *without* a user
|
|
# context should mark themselves ``@pytest.mark.no_auto_user``.
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_skill_storage_singleton():
|
|
"""Reset the SkillStorage singleton between tests to prevent cross-test contamination."""
|
|
try:
|
|
from deerflow.skills.storage import reset_skill_storage
|
|
except ImportError:
|
|
yield
|
|
return
|
|
reset_skill_storage()
|
|
try:
|
|
yield
|
|
finally:
|
|
reset_skill_storage()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_frozen_checkpoint_channel_mode(monkeypatch):
|
|
"""Reset the process-global frozen checkpoint channel mode between tests.
|
|
|
|
Production treats ``checkpoint_channel_mode`` as restart-required: the
|
|
first client/app freezes it for the process. The test suite builds many
|
|
clients and apps with different modes in one process, so the freeze must
|
|
not leak across tests. Mirrors the per-test ``monkeypatch.setattr``
|
|
resets already used in test_client.py / test_lead_agent_model_resolution.py.
|
|
"""
|
|
from deerflow.runtime import checkpoint_mode
|
|
|
|
monkeypatch.setattr(checkpoint_mode, "_frozen_checkpoint_channel_mode", None)
|
|
yield
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _restore_title_config_singleton():
|
|
"""Reset ``_title_config`` to its pristine default after every test.
|
|
|
|
``AppConfig.from_file()`` writes the on-disk ``title`` block into the
|
|
module-level singleton (``config/app_config.py`` calls
|
|
``load_title_config_from_dict``). Any test that loads the real
|
|
``config.yaml`` therefore leaves the singleton in a state that
|
|
``test_title_middleware_core_logic.py`` does not expect; that suite
|
|
relies on the pristine ``TitleConfig()`` default (``enabled=True``).
|
|
We restore the default after every test so test files stay
|
|
independent regardless of order.
|
|
"""
|
|
try:
|
|
from deerflow.config.title_config import reset_title_config
|
|
except ImportError:
|
|
yield
|
|
return
|
|
|
|
try:
|
|
yield
|
|
finally:
|
|
reset_title_config()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _auto_user_context(request):
|
|
"""Inject a default ``test-user-autouse`` into the contextvar.
|
|
|
|
Opt-out via ``@pytest.mark.no_auto_user``. Uses lazy import so that
|
|
tests which don't touch the persistence layer never pay the cost
|
|
of importing runtime.user_context.
|
|
"""
|
|
if request.node.get_closest_marker("no_auto_user"):
|
|
yield
|
|
return
|
|
|
|
try:
|
|
from deerflow.runtime.user_context import (
|
|
reset_current_user,
|
|
set_current_user,
|
|
)
|
|
except ImportError:
|
|
yield
|
|
return
|
|
|
|
user = SimpleNamespace(id="test-user-autouse", email="test@local")
|
|
token = set_current_user(user)
|
|
try:
|
|
yield
|
|
finally:
|
|
reset_current_user(token)
|