Vanzeren 42baed8c8c
feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel (#4292)
* feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel

Add a restart-required database.checkpoint_channel_mode ("full" default,
"delta") that stores the messages channel via LangGraph 1.2 DeltaChannel,
cutting checkpoint storage from O(n^2) to O(n) for append-only history.
Existing full checkpoints seed delta state transparently; no data migration.

- config: mode schema + freeze-on-first-use with
  CheckpointModeReconfigurationError; mode marker persisted in checkpoint
  metadata; unsafe delta->full downgrade rejected fail-closed with
  CheckpointModeMismatchError (run-level error, failed state read)
- state: delta message state schema; CheckpointStateAccessor centralizes
  materialized reads for all consumers (threads API, branches,
  regeneration, compaction, state updates, memory, goal workers)
- runtime: raw writers (run durations, interrupted title, thread goal)
  parent their checkpoints to the checkpoint they derive from, preserving
  delta ancestry; rollback forks the pre-run lineage through a state
  mutation graph with Overwrite restores; InMemorySaver delta-history
  override delegates to the base walk (fixes dropped first write after
  migration, also present upstream)
- tests: conformance suite over {memory, sqlite, postgres} covering
  migration replay, stable message IDs, storage shape and writer
  preservation; conftest fixture isolates the frozen mode between tests;
  stale config fakes refreshed
- ci: backend unit tests gain a postgres service

* fix(checkpoint): close materialization gaps in goal flow, guard public factory

- Route goal-continuation message reads through CheckpointStateAccessor:
  raw channel_values reads see the delta sentinel in delta mode, which
  disabled goal continuation (stand_down=no_durable_end_of_turn) after
  durable assistant turns. Raw tuples remain for tuple-only metadata
  (checkpoint id, pending_writes).
- Reject checkpoint_channel_mode='delta' + checkpointer in
  create_deerflow_agent at construction: factory-built persisted graphs
  bypass mode-marker injection and the fail-closed gate, reproducing
  silent mixed-mode state loss. Delta without persistence stays allowed.
- Import the postgres saver lazily (pytest.importorskip in the fixture)
  so the documented default install collects the suite; add a CI job
  running pytest --collect-only on uv sync --group dev without extras.
- Fix test_checkpointer fallback test to patch get_app_config at its
  use site (provider module), making it deterministic when a local
  config.yaml selects a persistent backend.

* fix(gateway): preserve extension-owned channels in state mutations, bump config version

- build_state_mutation_graph / build_checkpoint_state_mutation_accessor
  accept an explicit state_schema; branch and POST /state now compile the
  mutation graph from the thread's effective schema
  (graph_state_schema on the assistant graph). The base-ThreadState
  fallback silently discarded channels contributed by custom
  AgentMiddleware.state_schema on branch (data loss) and returned a
  false-success 200 on POST /state.
- POST /state validates values keys against the mutation graph's
  channels and rejects unknown fields with 422 instead of ignoring
  them; reducer detection covers extension channels
  (BinaryOperatorAggregate or DeltaChannel) so Overwrite replace
  semantics work for middleware reducers in both modes.
- Endpoint regression: custom AgentMiddleware.state_schema value
  survives branch, updates through POST /state, and an unknown field
  receives 422.
- config_version 26 -> 27 for the new database.checkpoint_channel_mode
  (example, Helm chart values + README, support-bundle fixture), so
  existing installs get the outdated-config warning and
  make config-upgrade merges the field; covered by a test driving the
  real example file and the real config-upgrade script.

* fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite

GET /threads/{id}/state now resolves the thread's assistant_id through a
single reusable boundary (thread metadata -> assistant_id -> effective
graph), so channels contributed by a custom AgentMiddleware.state_schema
are materialized instead of dropped by the default lead schema. POST
/state uses the same boundary instead of resolving the schema ad hoc.

Branch writes wrap every copied reducer channel in Overwrite (derived
from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel),
not just messages, so already-aggregated values are never re-merged.

Regression tests use a real AgentMiddleware.state_schema with a
non-identity reducer in both full and delta modes: GET /state returns the
extension value, POST /state replaces it, branch preserves it
byte-for-byte; the unknown-field 422 is a separate assertion.

* refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests

Address review round 4 on PR #4292:

- Push ahistory/history limit through Pregel into checkpointer.alist
  (SQL LIMIT) instead of materializing all rows and breaking in Python
- Fold the read-side mode-compat gate onto the returned snapshot's
  metadata; only writes keep the pre-write tuple fetch (fail-closed)
- Cache factory-built accessor graphs per (assistant_id, mode) with
  factory-identity revalidation; state reads no longer build a lead
  agent per request
- get_thread: one snapshot fetch + one raw pending_writes fetch on the
  resolved checkpoint (post-checkpoint __error__ writes never surface
  in snapshot.tasks; verified empirically)
- DeerFlowClient.get_thread: single checkpointer.list walk collects
  pending_writes per checkpoint instead of N get_tuple calls
- InMemorySaver delta-history patch: stand-down when the upstream
  override disappears, try/except guard, validated-version warning,
  guard tests
- make_lead_agent mode precedence: first freeze is owned by app_config
  (client-supplied configurable key ignored); once frozen, injected
  key/app_config must match or fail closed
- Rollback: lock in non-message channel restoration via fork
  inheritance with a dedicated reducer-channel test
- Add tests/test_threads_checkpoint_mode.py and
  tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and
  the PR validation section: lifecycle parity (memory + sqlite),
  per-step blob-count storage guard, gateway endpoint parity

Counted-saver tests pin checkpoint round-trips for aget/ahistory so
these regressions cannot silently return.

* fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience

- threads router: map CheckpointModeMismatchError to 409 (with cause and
  thread id) and CheckpointModeReconfigurationError to 503 across all state
  endpoints instead of swallowing both into a generic 500
- gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and
  assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary
  through the real route stack
- rollback: compile the restore mutation graph with the thread's effective
  state schema per the build_state_mutation_graph contract
- inheritance contract locks: rollback and manual compaction preserve
  middleware-contributed channels via checkpoint fork cloning
- services: revalidate the accessor-graph cache against app_config identity
  so config.yaml hot-reloads never serve a stale compiled graph
- services: degrade full-mode state reads to raw checkpointer reads when the
  agent factory is unavailable (delta gate still applies; delta mode has no
  fallback)
- deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently
  downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility)
  and bump the langchain lower bound to what the lockfile actually resolves

* fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread

- _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while
  pregel's get_state_history treats config.checkpoint_id as the inclusive
  start; fetch the anchor explicitly so both read paths paginate identically
- extend the degraded-path gateway test: GET /thread returns raw values, and
  POST /history with before starts at the anchor checkpoint

* fix(gateway): preserve degraded checkpoint timestamps

* fix(gateway): harden degraded checkpoint access

* fix(gateway): resolve assistants for checkpoint reads

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-22 08:33:29 +08:00

188 lines
7.3 KiB
Python

"""Materialized checkpoint-state access and state-only mutation graphs.
:class:`CheckpointStateAccessor` is the single choke point for thread
checkpoint-state reads and writes. It binds a compiled graph (which carries
the mode-matched channel schema), a checkpointer, and the frozen channel mode:
every operation injects the mode marker into the config and passes the
compatibility gate before touching state. Delta checkpoints store no full
``channel_values`` — raw saver reads see sentinels — so consumers must go
through this accessor instead of calling the checkpointer directly.
:func:`build_state_mutation_graph` compiles a state-only graph (one no-op
node, entry = finish) for wholesale state replacement such as rollback
restore and context compaction: it shares the agent graph's checkpoint
machinery but schedules no pending nodes, so the written head stays idle.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from deerflow.agents.thread_state import get_thread_state_schema
from deerflow.config.database_config import CheckpointChannelMode
from deerflow.runtime.checkpoint_mode import (
aensure_checkpoint_mode_compatible,
ensure_checkpoint_mode_compatible,
inject_checkpoint_mode,
raise_if_snapshot_incompatible,
)
def _finish_state_mutation(_state: dict[str, Any]) -> dict[str, Any]:
return {}
def build_state_mutation_graph(as_node: str, mode: CheckpointChannelMode, state_schema: Any | None = None) -> Any:
"""Compile a state-only graph whose single writer node finishes immediately.
``update_state(..., as_node=...)`` requires the node to be registered in
the graph; a dedicated single-node graph applies reducer writes and
finishes, so the mutation checkpoint schedules no agent nodes and has no
pending ``next`` nodes.
``state_schema`` must be the thread's *effective* schema (the class the
assistant graph was compiled with) whenever the write carries materialized
state: the base ThreadState fallback does not know channels contributed by
custom middleware, and writes to unknown channels are silently discarded.
"""
if not as_node:
raise ValueError("as_node is required for checkpoint state mutation")
from langgraph.graph import StateGraph
builder = StateGraph(state_schema if state_schema is not None else get_thread_state_schema(mode))
builder.add_node(as_node, _finish_state_mutation)
builder.set_entry_point(as_node)
builder.set_finish_point(as_node)
return builder.compile()
def graph_state_schema(graph: Any) -> Any | None:
"""Return the state schema class a compiled graph was built with.
The schema is the first entry of ``StateGraph.schemas`` (state schema is
registered before input/output schemas). Returns ``None`` for stub
accessors in tests that do not wrap a real compiled graph.
"""
schemas = getattr(getattr(graph, "builder", None), "schemas", None)
if not schemas:
return None
return next(iter(schemas))
def graph_writable_channels(graph: Any) -> frozenset[str] | None:
"""Return the user-visible state channel names of a compiled graph.
Excludes Pregel-internal channels (``__*``) and branch fan-in channels
(``branch:*``). Returns ``None`` when the graph does not expose channels
(stub accessors), so callers can fall back to the base ThreadState set.
"""
channels = getattr(graph, "channels", None)
if not channels:
return None
return frozenset(name for name in channels if not name.startswith("__") and not name.startswith("branch:"))
def graph_reducer_channels(graph: Any) -> frozenset[str] | None:
"""Return channel names whose writes merge through a reducer.
Covers classic reducers (``BinaryOperatorAggregate``) and delta channels:
both require ``Overwrite`` wrapping for replace-style writes, in any mode.
Returns ``None`` when the graph does not expose channels (stub
accessors), so callers can fall back to the base ThreadState set.
"""
from langgraph.channels import BinaryOperatorAggregate, DeltaChannel
channels = getattr(graph, "channels", None)
if channels is None:
return None
return frozenset(name for name, channel in channels.items() if isinstance(channel, (BinaryOperatorAggregate, DeltaChannel)))
@dataclass
class CheckpointStateAccessor:
graph: Any
checkpointer: Any
mode: CheckpointChannelMode
@classmethod
def bind(
cls,
graph: Any,
checkpointer: Any,
*,
store: Any | None = None,
mode: CheckpointChannelMode = "full",
) -> CheckpointStateAccessor:
graph.checkpointer = checkpointer
if store is not None:
graph.store = store
return cls(graph=graph, checkpointer=checkpointer, mode=mode)
def _prepare_config(self, config: dict[str, Any]) -> dict[str, Any]:
prepared = {
**config,
"configurable": dict(config.get("configurable", {})),
"metadata": dict(config.get("metadata", {})),
}
inject_checkpoint_mode(prepared, self.mode)
return prepared
def get(self, config: dict[str, Any]) -> Any:
prepared = self._prepare_config(config)
snapshot = self.graph.get_state(prepared)
raise_if_snapshot_incompatible(snapshot, self.mode)
return snapshot
async def aget(self, config: dict[str, Any]) -> Any:
prepared = self._prepare_config(config)
snapshot = await self.graph.aget_state(prepared)
raise_if_snapshot_incompatible(snapshot, self.mode)
return snapshot
def history(self, config: dict[str, Any], *, limit: int | None = None) -> list[Any]:
prepared = self._prepare_config(config)
if limit is not None and limit <= 0:
return []
result = []
for snapshot in self.graph.get_state_history(prepared, limit=limit):
raise_if_snapshot_incompatible(snapshot, self.mode)
result.append(snapshot)
if limit is not None and len(result) >= limit:
break
return result
async def ahistory(self, config: dict[str, Any], *, limit: int | None = None) -> list[Any]:
prepared = self._prepare_config(config)
if limit is not None and limit <= 0:
return []
result = []
async for snapshot in self.graph.aget_state_history(prepared, limit=limit):
raise_if_snapshot_incompatible(snapshot, self.mode)
result.append(snapshot)
if limit is not None and len(result) >= limit:
break
return result
def update(
self,
config: dict[str, Any],
values: dict[str, Any],
*,
as_node: str | None = None,
) -> dict[str, Any]:
prepared = self._prepare_config(config)
ensure_checkpoint_mode_compatible(self.checkpointer, prepared, self.mode)
return self.graph.update_state(prepared, values, as_node=as_node)
async def aupdate(
self,
config: dict[str, Any],
values: dict[str, Any],
*,
as_node: str | None = None,
) -> dict[str, Any]:
prepared = self._prepare_config(config)
await aensure_checkpoint_mode_compatible(self.checkpointer, prepared, self.mode)
return await self.graph.aupdate_state(prepared, values, as_node=as_node)