mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
* fix(gateway): classify duration-only checkpoints without the Postgres-dropped marker On Postgres the savers funnel metadata through langgraph's get_serializable_checkpoint_metadata, which pops writes before the row lands, so persist_run_history_metadata's runtime_run_duration marker never comes back and every replay/history read path treats metadata-only duration leaves as addressable conversation state. is_duration_only_checkpoint now classifies in two tiers: - writes present (memory/SQLite) is authoritative — the per-writer key is exact, so a title leaf carrying runtime_interrupt_title with inherited run_durations is not a duration checkpoint; - writes absent (Postgres) falls back to the surviving stamps (source == run_durations / run_message_ids keys). Stamps alone are not exact: _ensure_interrupted_title and _rollback_to_pre_run_checkpoint copy the head's metadata and inherit them. Read paths tolerate that because every stamp-bearing leaf either copies its parent's messages verbatim or restores an ancestor's exact message state, so a skipped leaf only advances a message-presence scan onto a state-equivalent checkpoint. Destructive callers opt out via stamp_fallback=False: enforce_thread_retention keeps widening the class only through the parent-shape check in _mark_duration_leaves_without_the_marker. * fix(gateway): confirm the duration stamp fallback by parent shape Review on #5988 surfaced a real false positive: write_thread_goal copies the head's metadata and thereby inherits the run_durations/run_message_ids stamps while changing the goal channel, and on Postgres its own writes.goal marker is popped too — so the stamp fallback classified the goal leaf as duration-only and the branch scan walked past it, copying channel values from before the goal write. The stamp fallback now also requires the duration writer's shape: it clones the parent's channel_versions verbatim, while every state-changing leaf (title, goal) bumps at least one version. Callers resolve the parent from their own context — the lineage walk fetches the grandparent only for stamp candidates, the history-window scans (branch, latest-turn, regenerate filter, chronological fallback) index the window they already hold — and stamps without a resolvable parent are refused: scanning one state-equivalent copy is free, skipping a goal leaf is not. Also corrects the mechanism claims from the first pass: langgraph's update_state paths (rollback, compaction, manual updates) persist metadata built from scratch and cannot inherit the stamps at all; the inheriting writers are title and goal. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(gateway): share the stamp gate and reuse the walked grandparent Review round 2 on #5988 — all three items, no functional changes: - extract _has_postgres_stamps as the single gate shared by the classifier fallback tier and the walk probe, so widening one widens the other; - carry the grandparent fetched for a stamp candidate into the next iteration (it is exactly the next parent), turning a chain of N Postgres duration leaves from ~2N+1 ancestor reads into N+1; - pin the unresolvable-grandparent promise at walk level: a stamp candidate whose own parent link was pruned stays addressable and remains the replay base. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(gateway): resolve persisted channel versions for the duration stamp fallback Materialized accessor snapshots expose no checkpoint payload, so the verbatim-copy shape check always failed open and Postgres duration leaves stayed addressable. Carry the version maps alongside the snapshots (raw snapshots expose them directly, graph-backed ones re-read the raw tuple through the accessor) and use them at every replay-base scan site. * perf(gateway): resolve stamp-fallback versions lazily and once per tuple The Postgres stamp fallback needs the raw channel versions for its shape check, and both window scans resolved them eagerly for every entry: a branch request paid up to 2N serial roundtrips for a window of N, and the regenerate scan resolved the same tuples a third time through resolve_history_versions. Resolve them only for stamp candidates - the marker tier and the stamp gate decide without versions - and memoize by config identity, so a marker-decided window pays nothing and each distinct tuple pays at most one raw read. The lineage walk had the same shape: a stamp candidate the shape check refuses re-read its grandparent as the next iteration's parent, because the prefetched slot was only filled when the candidate was skipped. Ancestor reads are now memoized by config identity, so no ancestor is fetched twice. Dropped the dead checkpointer parameter of find_checkpoint_before_message_chronologically: no caller passed it, and wiring one in from an async handler would put a blocking get_tuple on the event loop. --------- Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
619 lines
27 KiB
Python
619 lines
27 KiB
Python
"""Replay-base resolution rules shared by regenerate, edit replay and branching.
|
|
|
|
The load-bearing rule here is that a replay base must be a checkpoint the
|
|
thread was actually at rest in. A mid-run checkpoint still owns the interrupted
|
|
node's pending writes, so resuming from it replays them.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
|
|
|
|
from app.gateway.checkpoint_lineage import (
|
|
CheckpointParentMissingError,
|
|
find_checkpoint_before_message,
|
|
find_checkpoint_before_message_chronologically,
|
|
is_duration_only_checkpoint,
|
|
resolve_channel_versions,
|
|
resolve_stamp_candidate_versions,
|
|
)
|
|
|
|
THREAD_ID = "thread-1"
|
|
|
|
# The metadata a Postgres row carries for a duration-only checkpoint: the
|
|
# writer's ``writes.runtime_run_duration`` stamp was dropped by langgraph's
|
|
# Postgres metadata serialisation, so only the index keys and ``source``
|
|
# survive the round trip.
|
|
_POSTGRES_DURATION_METADATA = {
|
|
"source": "update",
|
|
"step": 3,
|
|
"run_durations": {"run-1": 12},
|
|
"run_message_ids": {"ai-1": "run-1"},
|
|
}
|
|
|
|
# A Postgres-shaped leaf inherits these stamps whenever a metadata-copying
|
|
# writer (title, goal) stacks on a stamped head, but bumps a channel version.
|
|
_POSTGRES_INHERITED_STAMPS = {"source": "update", "step": 4, "run_durations": {"run-1": 12}}
|
|
|
|
|
|
def _snapshot(checkpoint_id: str, messages: list[object], *, next_tasks: tuple[str, ...] = (), parent_id: str | None = None, metadata: dict | None = None, channel_versions: dict | None = None):
|
|
parent_config = None
|
|
if parent_id is not None:
|
|
parent_config = {"configurable": {"thread_id": THREAD_ID, "checkpoint_ns": "", "checkpoint_id": parent_id}}
|
|
return SimpleNamespace(
|
|
values={"messages": messages},
|
|
config={
|
|
"configurable": {
|
|
"thread_id": THREAD_ID,
|
|
"checkpoint_ns": "",
|
|
"checkpoint_id": checkpoint_id,
|
|
"checkpoint_map": None,
|
|
}
|
|
},
|
|
checkpoint={"channel_versions": dict(channel_versions or {})},
|
|
metadata=metadata or {},
|
|
parent_config=parent_config,
|
|
next=next_tasks,
|
|
)
|
|
|
|
|
|
class _Accessor:
|
|
"""Minimal accessor over a fixed snapshot list, keyed by checkpoint id."""
|
|
|
|
def __init__(self, snapshots: list[object]) -> None:
|
|
self.snapshots = snapshots
|
|
|
|
async def aget(self, config):
|
|
checkpoint_id = config.get("configurable", {}).get("checkpoint_id")
|
|
return next(
|
|
(item for item in self.snapshots if item.config["configurable"]["checkpoint_id"] == checkpoint_id),
|
|
SimpleNamespace(values={}, config={}, metadata=None, parent_config=None, next=()),
|
|
)
|
|
|
|
|
|
def _first_turn_history() -> list[object]:
|
|
"""Newest-first history of a thread whose only turn is still its first.
|
|
|
|
``DynamicContextMiddleware`` swaps the first user message's id mid-run:
|
|
the injected reminder takes ``{id}`` and the real user message becomes
|
|
``{id}__user``. So the pre-injection checkpoints hold the very same prompt
|
|
under an id the replay-base lookup does not recognise (#4531).
|
|
"""
|
|
system = SystemMessage(id="h1", content="<system-reminder>date</system-reminder>")
|
|
swapped_human = HumanMessage(id="h1__user", content="question")
|
|
raw_human = HumanMessage(id="h1", content="question")
|
|
ai = AIMessage(id="ai-1", content="answer")
|
|
return [
|
|
_snapshot("ckpt-head", [system, swapped_human, ai], parent_id="ckpt-after-inject"),
|
|
_snapshot("ckpt-after-inject", [system, swapped_human], next_tasks=("LoopDetectionMiddleware.before_agent",), parent_id="ckpt-mid"),
|
|
_snapshot("ckpt-mid", [raw_human], next_tasks=("DynamicContextMiddleware.before_agent",), parent_id="ckpt-input"),
|
|
_snapshot("ckpt-input", [], next_tasks=("__start__",), parent_id="ckpt-empty"),
|
|
_snapshot("ckpt-empty", []),
|
|
]
|
|
|
|
|
|
def test_chronological_scan_skips_checkpoints_with_pending_tasks():
|
|
history = _first_turn_history()
|
|
|
|
base, found = find_checkpoint_before_message_chronologically(history, "h1__user")
|
|
|
|
assert found is True
|
|
assert base is not None
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-empty"
|
|
|
|
|
|
def test_lineage_walk_skips_checkpoints_with_pending_tasks():
|
|
history = _first_turn_history()
|
|
accessor = _Accessor(history)
|
|
|
|
base = asyncio.run(find_checkpoint_before_message(accessor, history[0], "h1__user", max_depth=10))
|
|
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-empty"
|
|
|
|
|
|
def test_chronological_scan_prefers_the_previous_turn_boundary():
|
|
"""A later turn resolves to the previous run's settled tail, not its input checkpoint."""
|
|
human1 = HumanMessage(id="h1__user", content="first")
|
|
ai1 = AIMessage(id="ai-1", content="first answer")
|
|
human2 = HumanMessage(id="h2", content="second")
|
|
history = [
|
|
_snapshot("ckpt-turn2-head", [human1, ai1, human2, AIMessage(id="ai-2", content="second answer")]),
|
|
_snapshot("ckpt-turn2-mid", [human1, ai1, human2], next_tasks=("model",)),
|
|
_snapshot("ckpt-turn2-input", [human1, ai1], next_tasks=("__start__",)),
|
|
_snapshot("ckpt-turn1-tail", [human1, ai1]),
|
|
]
|
|
|
|
base, found = find_checkpoint_before_message_chronologically(history, "h2")
|
|
|
|
assert found is True
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-turn1-tail"
|
|
|
|
|
|
def test_lineage_walk_reports_missing_parent_when_no_settled_ancestor_exists():
|
|
"""Fail closed rather than fork a mid-run checkpoint."""
|
|
human = HumanMessage(id="h1__user", content="question")
|
|
history = [
|
|
_snapshot("ckpt-head", [human, AIMessage(id="ai-1", content="answer")], parent_id="ckpt-mid"),
|
|
_snapshot("ckpt-mid", [HumanMessage(id="h1", content="question")], next_tasks=("DynamicContextMiddleware.before_agent",)),
|
|
]
|
|
accessor = _Accessor(history)
|
|
|
|
with pytest.raises(CheckpointParentMissingError):
|
|
asyncio.run(find_checkpoint_before_message(accessor, history[0], "h1__user", max_depth=10))
|
|
|
|
|
|
def test_unknown_pending_tasks_do_not_block_selection():
|
|
"""Raw full-mode reads cannot derive tasks; absence of evidence stays permissive."""
|
|
human = HumanMessage(id="h1", content="question")
|
|
history = [
|
|
SimpleNamespace(
|
|
values={"messages": [human]},
|
|
config={"configurable": {"thread_id": THREAD_ID, "checkpoint_ns": "", "checkpoint_id": "ckpt-head"}},
|
|
metadata={},
|
|
),
|
|
SimpleNamespace(
|
|
values={"messages": []},
|
|
config={"configurable": {"thread_id": THREAD_ID, "checkpoint_ns": "", "checkpoint_id": "ckpt-base"}},
|
|
metadata={},
|
|
),
|
|
]
|
|
|
|
base, found = find_checkpoint_before_message_chronologically(history, "h1")
|
|
|
|
assert found is True
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-base"
|
|
|
|
|
|
def test_memory_backed_duration_checkpoint_is_recognised():
|
|
"""The marker memory and SQLite savers round-trip unchanged."""
|
|
snapshot = _snapshot(
|
|
"ckpt-duration",
|
|
[],
|
|
metadata={"source": "update", "step": 3, "writes": {"runtime_run_duration": {"run_ids": ["run-1"], "message_ids": []}}},
|
|
)
|
|
|
|
assert is_duration_only_checkpoint(snapshot) is True
|
|
|
|
|
|
def test_postgres_duration_checkpoint_is_recognised_without_the_writes_marker():
|
|
"""Postgres serialisation pops ``writes``; stamps plus the verbatim shape identify the leaf."""
|
|
parent = _snapshot("ckpt-real", [], channel_versions={"messages": 5})
|
|
snapshot = _snapshot(
|
|
"ckpt-duration",
|
|
[],
|
|
parent_id="ckpt-real",
|
|
metadata=dict(_POSTGRES_DURATION_METADATA),
|
|
channel_versions={"messages": 5},
|
|
)
|
|
|
|
assert is_duration_only_checkpoint(snapshot, parent=parent) is True
|
|
|
|
|
|
def test_stamp_fallback_refuses_stamps_without_a_parent():
|
|
"""Stamps alone are evidence, not proof — with no parent there is no verdict.
|
|
|
|
A title or goal leaf at a window boundary carries the same stamps; the
|
|
conservative answer keeps it addressable, which only ever costs one
|
|
scanned state-equivalent copy.
|
|
"""
|
|
snapshot = _snapshot("ckpt-duration", [], metadata=dict(_POSTGRES_DURATION_METADATA), channel_versions={"messages": 5})
|
|
|
|
assert is_duration_only_checkpoint(snapshot) is False
|
|
assert is_duration_only_checkpoint(snapshot, parent=None) is False
|
|
|
|
|
|
def test_goal_leaf_inheriting_stamps_is_not_duration_only():
|
|
"""``write_thread_goal`` copies the head's metadata and changes the goal channel.
|
|
|
|
On Postgres its own ``writes.goal`` marker is popped too, so the inherited
|
|
stamps are the only trace of the duration writer — but the bumped ``goal``
|
|
version breaks the verbatim shape. Classifying it as duration-only made
|
|
the branch scan walk past the leaf and copy channel values from before
|
|
the goal write.
|
|
"""
|
|
parent = _snapshot("ckpt-stamped-head", [], channel_versions={"messages": 5, "goal": 2})
|
|
goal_leaf = _snapshot(
|
|
"ckpt-goal",
|
|
[],
|
|
parent_id="ckpt-stamped-head",
|
|
metadata=dict(_POSTGRES_INHERITED_STAMPS),
|
|
channel_versions={"messages": 5, "goal": 3},
|
|
)
|
|
|
|
assert is_duration_only_checkpoint(goal_leaf, parent=parent) is False
|
|
|
|
|
|
def test_postgres_attribution_only_duration_checkpoint_is_recognised():
|
|
"""A leaf stamped for message attribution alone has an empty ``run_durations`` map."""
|
|
parent = _snapshot("ckpt-real", [], channel_versions={"messages": 5})
|
|
snapshot = _snapshot(
|
|
"ckpt-duration",
|
|
[],
|
|
parent_id="ckpt-real",
|
|
metadata={"source": "update", "step": 3, "run_durations": {}, "run_message_ids": {"ai-1": "run-1"}},
|
|
channel_versions={"messages": 5},
|
|
)
|
|
|
|
assert is_duration_only_checkpoint(snapshot, parent=parent) is True
|
|
|
|
|
|
def test_client_update_state_is_not_a_duration_checkpoint():
|
|
"""A leaf from langgraph's ``update_state`` carries none of the writer's stamps.
|
|
|
|
update_state persists metadata built from scratch on the pinned langgraph
|
|
(``{"source": "update", "step", "parents", ...}`` — no ``writes`` key at
|
|
all), so this shape reaches the stamp fallback tier and stays addressable
|
|
because the index keys are absent.
|
|
"""
|
|
snapshot = _snapshot("ckpt-update", [], metadata={"source": "update", "step": 3})
|
|
|
|
assert is_duration_only_checkpoint(snapshot) is False
|
|
|
|
|
|
def test_graph_step_checkpoints_are_not_duration_checkpoints():
|
|
for source in ("input", "loop", "step"):
|
|
snapshot = _snapshot("ckpt-step", [], metadata={"source": source, "step": 1, "writes": {"messages": "..."}})
|
|
assert is_duration_only_checkpoint(snapshot) is False
|
|
assert is_duration_only_checkpoint(_snapshot("ckpt-plain", [])) is False
|
|
assert is_duration_only_checkpoint(SimpleNamespace(values={}, config={}, metadata=None)) is False
|
|
|
|
|
|
def test_chronological_scan_skips_postgres_shaped_duration_checkpoints():
|
|
"""An interleaved import must not hand the replay base to a metadata-only leaf.
|
|
|
|
The duration leaf belongs to an older incarnation whose timestamps interleave
|
|
with the live branch; without the Postgres-shape classification it would be
|
|
selected as the replay base for a message that only the newer real state and
|
|
the head contain.
|
|
"""
|
|
human = HumanMessage(id="h1", content="question")
|
|
answer = AIMessage(id="ai-1", content="answer")
|
|
history = [
|
|
_snapshot("ckpt-x-head", [human, answer]),
|
|
_snapshot("ckpt-a-early", [], channel_versions={"messages": 3}),
|
|
_snapshot("ckpt-a-duration", [human], parent_id="ckpt-a-early", metadata=dict(_POSTGRES_DURATION_METADATA), channel_versions={"messages": 3}),
|
|
_snapshot("ckpt-a-input", [], channel_versions={"messages": 2}),
|
|
]
|
|
|
|
base, found = find_checkpoint_before_message_chronologically(history, "h1")
|
|
|
|
assert found is True
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-a-early"
|
|
|
|
|
|
def test_chronological_scan_keeps_goal_leaf_addressable():
|
|
"""The imported-history fallback must not walk past a goal write either.
|
|
|
|
The goal leaf inherits the stamps but bumped the goal version, so the
|
|
verbatim shape fails and it stays a replay-base candidate.
|
|
"""
|
|
human = HumanMessage(id="h1", content="question")
|
|
ai = AIMessage(id="ai-1", content="answer")
|
|
human2 = HumanMessage(id="h2", content="follow-up")
|
|
ai2 = AIMessage(id="ai-2", content="answer-2")
|
|
history = [
|
|
_snapshot("ckpt-turn2-head", [human, ai, human2, ai2]),
|
|
_snapshot("ckpt-goal", [human, ai], parent_id="ckpt-turn1-tail", metadata=dict(_POSTGRES_INHERITED_STAMPS), channel_versions={"messages": 5, "goal": 3}),
|
|
_snapshot("ckpt-turn1-tail", [human, ai], channel_versions={"messages": 5}),
|
|
]
|
|
|
|
base, found = find_checkpoint_before_message_chronologically(history, "h2")
|
|
|
|
assert found is True
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-goal"
|
|
|
|
|
|
def test_lineage_walk_crosses_chained_postgres_duration_checkpoints():
|
|
"""The lineage walk skips duration-only parents even when ``writes`` never comes back."""
|
|
swapped_human = HumanMessage(id="h1__user", content="question")
|
|
raw_human = HumanMessage(id="h1", content="question")
|
|
ai = AIMessage(id="ai-1", content="answer")
|
|
history = [
|
|
_snapshot("ckpt-head", [swapped_human, ai], parent_id="ckpt-leaf-2", channel_versions={"messages": 7}),
|
|
_snapshot("ckpt-leaf-2", [swapped_human, ai], parent_id="ckpt-leaf-1", metadata=dict(_POSTGRES_DURATION_METADATA), channel_versions={"messages": 5}),
|
|
_snapshot("ckpt-leaf-1", [swapped_human, ai], parent_id="ckpt-turn1-tail", metadata=dict(_POSTGRES_DURATION_METADATA), channel_versions={"messages": 5}),
|
|
_snapshot("ckpt-turn1-tail", [raw_human], channel_versions={"messages": 5}),
|
|
]
|
|
accessor = _Accessor(history)
|
|
|
|
base = asyncio.run(find_checkpoint_before_message(accessor, history[0], "h1__user", max_depth=10))
|
|
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-turn1-tail"
|
|
|
|
|
|
def test_lineage_walk_keeps_goal_leaf_addressable():
|
|
"""A goal write stacked on a stamped head must remain a replay-base candidate.
|
|
|
|
The goal leaf inherits the duration stamps on Postgres but bumped the goal
|
|
version; treating it as duration-only would walk past it and hand back a
|
|
base whose channel values predate the goal write.
|
|
"""
|
|
human = HumanMessage(id="h1", content="question")
|
|
ai = AIMessage(id="ai-1", content="answer")
|
|
human2 = HumanMessage(id="h2", content="follow-up")
|
|
history = [
|
|
_snapshot("ckpt-turn2-head", [human, ai, human2], parent_id="ckpt-goal", channel_versions={"messages": 6}),
|
|
_snapshot("ckpt-goal", [human, ai], parent_id="ckpt-turn1-tail", metadata=dict(_POSTGRES_INHERITED_STAMPS), channel_versions={"messages": 5, "goal": 3}),
|
|
_snapshot("ckpt-turn1-tail", [human, ai], channel_versions={"messages": 5}),
|
|
]
|
|
accessor = _Accessor(history)
|
|
|
|
base = asyncio.run(find_checkpoint_before_message(accessor, history[0], "h2", max_depth=10))
|
|
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-goal"
|
|
|
|
|
|
def test_lineage_walk_keeps_stamp_candidate_addressable_when_grandparent_is_gone():
|
|
"""A stamp candidate whose own parent link was pruned stays addressable.
|
|
|
|
Retention may already have removed the grandparent; the walk must then
|
|
treat the candidate as addressable rather than skipping it on inherited
|
|
stamps — the returned replay base keeps the candidate's channel values.
|
|
"""
|
|
human = HumanMessage(id="h1", content="question")
|
|
ai = AIMessage(id="ai-1", content="answer")
|
|
human2 = HumanMessage(id="h2", content="follow-up")
|
|
history = [
|
|
_snapshot("ckpt-turn2-head", [human, ai, human2], parent_id="ckpt-orphan-duration"),
|
|
_snapshot("ckpt-orphan-duration", [human, ai], parent_id="ckpt-pruned", metadata=dict(_POSTGRES_INHERITED_STAMPS), channel_versions={"messages": 5}),
|
|
]
|
|
accessor = _Accessor(history)
|
|
|
|
base = asyncio.run(find_checkpoint_before_message(accessor, history[0], "h2", max_depth=10))
|
|
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-orphan-duration"
|
|
|
|
|
|
def test_title_leaf_inheriting_duration_stamps_is_not_duration_only():
|
|
"""``_ensure_interrupted_title`` copies the head's metadata, stamps included.
|
|
|
|
On memory/SQLite the leaf keeps its own ``writes.runtime_interrupt_title``
|
|
marker, which must win over the inherited ``run_durations`` — otherwise a
|
|
title write stacked on a stamped head would be classified as a duration
|
|
checkpoint even where the marker is authoritative.
|
|
"""
|
|
snapshot = _snapshot(
|
|
"ckpt-title",
|
|
[],
|
|
metadata={
|
|
"source": "update",
|
|
"step": 4,
|
|
"writes": {"runtime_interrupt_title": {"title": "hi"}},
|
|
"run_durations": {"run-1": 12},
|
|
"run_message_ids": {"ai-1": "run-1"},
|
|
},
|
|
)
|
|
|
|
assert is_duration_only_checkpoint(snapshot) is False
|
|
|
|
|
|
def test_stamp_fallback_disabled_requires_the_writes_marker():
|
|
"""Destructive callers must see a Postgres-shaped leaf as *not* duration-only.
|
|
|
|
Without the ``writes`` marker the stamps are only evidence, not proof —
|
|
``checkpoint_retention`` refuses them here and widens the class solely via
|
|
the parent-shape check in ``_mark_duration_leaves_without_the_marker``.
|
|
"""
|
|
parent = _snapshot("ckpt-real", [], channel_versions={"messages": 5})
|
|
postgres_shaped = _snapshot("ckpt-duration", [], metadata=dict(_POSTGRES_DURATION_METADATA), channel_versions={"messages": 5})
|
|
marker_backed = _snapshot(
|
|
"ckpt-duration",
|
|
[],
|
|
metadata={"source": "update", "step": 3, "writes": {"runtime_run_duration": {"run_ids": [], "message_ids": []}}},
|
|
)
|
|
|
|
assert is_duration_only_checkpoint(postgres_shaped, parent=parent, stamp_fallback=False) is False
|
|
assert is_duration_only_checkpoint(marker_backed, parent=parent, stamp_fallback=False) is True
|
|
|
|
|
|
async def _post_strip_writes(saver, tup) -> None:
|
|
"""Rewrite *tup*'s persisted metadata without ``writes``, in place."""
|
|
|
|
checkpoint = dict(getattr(tup, "checkpoint", {}) or {})
|
|
metadata = dict(getattr(tup, "metadata", {}) or {})
|
|
metadata.pop("writes", None)
|
|
parent_config = getattr(tup, "parent_config", None)
|
|
write_config = parent_config if isinstance(parent_config, dict) else {"configurable": {"thread_id": tup.config["configurable"]["thread_id"], "checkpoint_ns": "", "checkpoint_id": None}}
|
|
await saver.aput(write_config, checkpoint, metadata, {})
|
|
|
|
|
|
async def _seed_lineage(mode: str):
|
|
from langgraph.checkpoint.memory import InMemorySaver
|
|
from langgraph.types import Overwrite
|
|
|
|
from deerflow.runtime.checkpoint_state import CheckpointStateAccessor, build_state_mutation_graph
|
|
|
|
saver = InMemorySaver()
|
|
accessor = CheckpointStateAccessor.bind(build_state_mutation_graph("seed", mode), saver, mode=mode)
|
|
config = {"configurable": {"thread_id": THREAD_ID, "checkpoint_ns": ""}}
|
|
await accessor.aupdate(config, {"messages": Overwrite([HumanMessage(content="q", id="h1"), AIMessage(content="a", id="ai-1")])}, as_node="seed")
|
|
return saver, accessor, config
|
|
|
|
|
|
async def _stamp_duration(saver, accessor, config):
|
|
from deerflow.runtime.runs.worker import persist_run_history_metadata
|
|
|
|
await persist_run_history_metadata(checkpointer=saver, thread_id=THREAD_ID, durations={"run-1": 12}, message_run_ids={"ai-1": "run-1"})
|
|
|
|
|
|
def test_classifier_uses_accessor_snapshots_in_full_mode() -> None:
|
|
"""Real ``StateSnapshot`` objects must classify Postgres duration leaves (full)."""
|
|
|
|
async def scenario() -> None:
|
|
saver, accessor, config = await _seed_lineage("full")
|
|
await _stamp_duration(saver, accessor, config)
|
|
head = await saver.aget_tuple(config)
|
|
await _post_strip_writes(saver, head)
|
|
|
|
history = await accessor.ahistory(config)
|
|
leaf = history[0]
|
|
parent = history[1] if len(history) > 1 else None
|
|
assert is_duration_only_checkpoint(leaf, parent=parent, versions=await resolve_channel_versions(accessor, leaf), parent_versions=await resolve_channel_versions(accessor, parent) if parent is not None else None) is True
|
|
|
|
asyncio.run(scenario())
|
|
|
|
|
|
def test_classifier_uses_accessor_snapshots_in_delta_mode() -> None:
|
|
"""Same integration gap exists on the delta read path; both must be covered."""
|
|
|
|
async def scenario() -> None:
|
|
saver, accessor, config = await _seed_lineage("delta")
|
|
await _stamp_duration(saver, accessor, config)
|
|
head = await saver.aget_tuple(config)
|
|
await _post_strip_writes(saver, head)
|
|
|
|
history = await accessor.ahistory(config)
|
|
leaf = history[0]
|
|
parent = history[1] if len(history) > 1 else None
|
|
assert is_duration_only_checkpoint(leaf, parent=parent, versions=await resolve_channel_versions(accessor, leaf), parent_versions=await resolve_channel_versions(accessor, parent) if parent is not None else None) is True
|
|
|
|
asyncio.run(scenario())
|
|
|
|
|
|
def test_goal_leaf_stays_addressable_with_accessor_snapshots() -> None:
|
|
"""A goal write stacked on a stamped head bumps the goal version and must stay."""
|
|
|
|
async def scenario() -> None:
|
|
saver, accessor, config = await _seed_lineage("full")
|
|
await _stamp_duration(saver, accessor, config)
|
|
|
|
from deerflow.runtime.goal import write_thread_goal
|
|
|
|
await write_thread_goal(saver, THREAD_ID, {"objective": "ship", "continuation_count": 0, "no_progress_count": 0, "updated_at": "2026-10-08T00:00:00Z"})
|
|
goal_tup = await saver.aget_tuple(config)
|
|
await _post_strip_writes(saver, goal_tup)
|
|
|
|
history = await accessor.ahistory(config)
|
|
leaf = history[0]
|
|
parent = history[1] if len(history) > 1 else None
|
|
assert is_duration_only_checkpoint(leaf, parent=parent, versions=await resolve_channel_versions(accessor, leaf), parent_versions=await resolve_channel_versions(accessor, parent) if parent is not None else None) is False
|
|
|
|
asyncio.run(scenario())
|
|
|
|
|
|
def test_degraded_raw_read_accessor_classifies_via_carried_versions() -> None:
|
|
"""The degraded full-mode accessor ships channel versions on its snapshots."""
|
|
|
|
async def scenario() -> None:
|
|
saver, accessor, config = await _seed_lineage("full")
|
|
await _stamp_duration(saver, accessor, config)
|
|
head = await saver.aget_tuple(config)
|
|
await _post_strip_writes(saver, head)
|
|
|
|
from app.gateway.services import _RawCheckpointReadAccessor
|
|
|
|
raw = _RawCheckpointReadAccessor(saver, "full")
|
|
leaf_snap = await raw.aget(config)
|
|
parent_config = getattr(leaf_snap, "parent_config", None)
|
|
parent_snap = await raw.aget(parent_config) if isinstance(parent_config, dict) else None
|
|
assert leaf_snap.channel_versions is not None
|
|
assert is_duration_only_checkpoint(leaf_snap, parent=parent_snap, versions=_snapshot_versions_of(leaf_snap), parent_versions=_snapshot_versions_of(parent_snap)) is True
|
|
|
|
asyncio.run(scenario())
|
|
|
|
|
|
def _snapshot_versions_of(obj):
|
|
checkpoint = getattr(obj, "checkpoint", None)
|
|
if isinstance(checkpoint, dict):
|
|
versions = checkpoint.get("channel_versions")
|
|
if isinstance(versions, dict):
|
|
return versions
|
|
return getattr(obj, "channel_versions", None)
|
|
|
|
|
|
class _CountingAccessor(_Accessor):
|
|
"""``_Accessor`` that records every read, pinning the walk's read budget."""
|
|
|
|
def __init__(self, snapshots: list[object]) -> None:
|
|
super().__init__(snapshots)
|
|
self.reads: list[str] = []
|
|
|
|
async def aget(self, config):
|
|
self.reads.append(config.get("configurable", {}).get("checkpoint_id"))
|
|
return await super().aget(config)
|
|
|
|
|
|
class _RawVersionAccessor:
|
|
"""Accessor whose snapshots hide the payload, like langgraph's materialized ones.
|
|
|
|
``aget_tuple`` is the only way to reach ``channel_versions`` from such a
|
|
snapshot, so every call models one serial Postgres roundtrip.
|
|
"""
|
|
|
|
def __init__(self, tuples: dict[str, object]) -> None:
|
|
self.tuples = tuples
|
|
self.raw_reads: list[str] = []
|
|
|
|
async def aget_tuple(self, config):
|
|
checkpoint_id = config.get("configurable", {}).get("checkpoint_id")
|
|
self.raw_reads.append(checkpoint_id)
|
|
return self.tuples.get(checkpoint_id)
|
|
|
|
|
|
def _hidden_snapshot(checkpoint_id: str, metadata: dict) -> SimpleNamespace:
|
|
"""A snapshot exposing neither ``checkpoint`` nor ``channel_versions``."""
|
|
|
|
return SimpleNamespace(
|
|
config={"configurable": {"thread_id": THREAD_ID, "checkpoint_ns": "", "checkpoint_id": checkpoint_id}},
|
|
metadata=dict(metadata),
|
|
)
|
|
|
|
|
|
def test_lineage_walk_reads_a_refused_stamp_candidate_grandparent_once():
|
|
"""A refused stamp candidate must not re-read its own parent.
|
|
|
|
The goal leaf inherits the duration stamps but bumped a channel version, so
|
|
the walk reads its parent for the shape check and then continues from the
|
|
candidate — whose parent link points at that very tuple. Both reads must hit
|
|
one entry in the walk's read cache.
|
|
"""
|
|
human = HumanMessage(id="h1", content="question")
|
|
ai = AIMessage(id="ai-1", content="answer")
|
|
human2 = HumanMessage(id="h2", content="follow-up")
|
|
history = [
|
|
_snapshot("ckpt-head", [human, ai, human2], parent_id="ckpt-goal", channel_versions={"messages": 6}),
|
|
_snapshot("ckpt-goal", [human, ai, human2], parent_id="ckpt-turn1-tail", metadata=dict(_POSTGRES_INHERITED_STAMPS), channel_versions={"messages": 5, "goal": 3}),
|
|
_snapshot("ckpt-turn1-tail", [human, ai], channel_versions={"messages": 5}),
|
|
]
|
|
accessor = _CountingAccessor(history)
|
|
|
|
base = asyncio.run(find_checkpoint_before_message(accessor, history[0], "h2", max_depth=10))
|
|
|
|
assert base.config["configurable"]["checkpoint_id"] == "ckpt-turn1-tail"
|
|
assert accessor.reads.count("ckpt-turn1-tail") == 1
|
|
|
|
|
|
def test_stamp_candidate_versions_are_lazy_and_memoized():
|
|
"""Only stamp candidates pay for version maps, and each tuple pays once.
|
|
|
|
Behind a materialized snapshot every lookup is a serial raw read, so a
|
|
window the marker tier or the stamp gate already decided must issue none,
|
|
and resolving a candidate must reuse the pass that resolved its parent.
|
|
"""
|
|
candidate = _hidden_snapshot("ckpt-candidate", _POSTGRES_DURATION_METADATA)
|
|
parent = _hidden_snapshot("ckpt-parent", _POSTGRES_DURATION_METADATA)
|
|
marker_decided = _hidden_snapshot("ckpt-marker", {"source": "update", "step": 3, "writes": {"runtime_run_duration": {"run_id": "run-1"}}})
|
|
accessor = _RawVersionAccessor(
|
|
{
|
|
"ckpt-candidate": SimpleNamespace(checkpoint={"channel_versions": {"messages": 5}}),
|
|
"ckpt-parent": SimpleNamespace(checkpoint={"channel_versions": {"messages": 5}}),
|
|
}
|
|
)
|
|
cache: dict[tuple[str, str, str], object] = {}
|
|
|
|
assert asyncio.run(resolve_stamp_candidate_versions(accessor, marker_decided, parent, cache)) == (None, None)
|
|
assert accessor.raw_reads == []
|
|
|
|
assert asyncio.run(resolve_stamp_candidate_versions(accessor, candidate, parent, cache)) == ({"messages": 5}, {"messages": 5})
|
|
assert accessor.raw_reads == ["ckpt-candidate", "ckpt-parent"]
|
|
|
|
assert asyncio.run(resolve_stamp_candidate_versions(accessor, parent, candidate, cache)) == ({"messages": 5}, {"messages": 5})
|
|
assert accessor.raw_reads == ["ckpt-candidate", "ckpt-parent"]
|
|
|
|
assert asyncio.run(resolve_stamp_candidate_versions(accessor, candidate, None, cache)) == (None, None)
|
|
assert accessor.raw_reads == ["ckpt-candidate", "ckpt-parent"]
|