mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-12 22:20:33 +00:00
chore(checkpoint): require fixed DeltaChannel replay backends (#8139)
* chore(checkpoint): require fixed DeltaChannel replay backends * fix(checkpoint): correct Postgres floor and refresh docs * test(checkpoint): update Postgres dependency floor assertion --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Zeren Wang
GitHub
parent
51cc489c8f
commit
95514245e2
@ -58,6 +58,14 @@ This section accumulates work toward the **2.3.0** milestone
|
||||
connect an ordinary StepFun API account with an API key and model ID.
|
||||
([#7988](https://github.com/bytedance/deer-flow/pull/7988))
|
||||
|
||||
### Changed
|
||||
|
||||
- **checkpoint:** DeltaChannel deployments now require `langgraph-checkpoint`
|
||||
4.3.0, SQLite saver 3.1.2, and Postgres saver 3.1.3, excluding versions that
|
||||
could reorder parallel writes during replay or lose paged ancestor history.
|
||||
The ordering and Postgres pagination contracts now fail on regression instead
|
||||
of skipping. ([#8139])
|
||||
|
||||
### Fixed
|
||||
|
||||
- **channels:** WeCom no longer drops images between 2 MB and 10 MB. The
|
||||
@ -16346,5 +16354,6 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
||||
[#8118]: https://github.com/bytedance/deer-flow/pull/8118
|
||||
[#8126]: https://github.com/bytedance/deer-flow/pull/8126
|
||||
[#8131]: https://github.com/bytedance/deer-flow/pull/8131
|
||||
[#8139]: https://github.com/bytedance/deer-flow/pull/8139
|
||||
[#8151]: https://github.com/bytedance/deer-flow/pull/8151
|
||||
[#8174]: https://github.com/bytedance/deer-flow/pull/8174
|
||||
|
||||
@ -24,6 +24,13 @@
|
||||
- **模型：** 管理员可在 **设置 → 模型** 中选择 StepFun（阶跃星辰），
|
||||
填写 API Key 和模型 ID 即可接入普通 API。([#7988](https://github.com/bytedance/deer-flow/pull/7988))
|
||||
|
||||
### 变更
|
||||
|
||||
- **Checkpoint：** DeltaChannel 部署现在要求 `langgraph-checkpoint` 4.3.0、
|
||||
SQLite saver 3.1.2 和 Postgres saver 3.1.3，排除会在重放时重排并行写入或
|
||||
丢失分页祖先历史的版本。顺序和 Postgres 分页契约在回归时现在会直接失败，
|
||||
不再跳过。([#8139])
|
||||
|
||||
### 修复
|
||||
|
||||
- **通道：** 企业微信不再丢弃 2–10 MB 的图片。此前适配器把图片上限定为 2 MB（同一协议中语音的限制），
|
||||
@ -12875,4 +12882,5 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
||||
[#8064]: https://github.com/bytedance/deer-flow/pull/8064
|
||||
[#8118]: https://github.com/bytedance/deer-flow/pull/8118
|
||||
[#8126]: https://github.com/bytedance/deer-flow/pull/8126
|
||||
[#8139]: https://github.com/bytedance/deer-flow/pull/8139
|
||||
[#8174]: https://github.com/bytedance/deer-flow/pull/8174
|
||||
|
||||
@ -343,7 +343,7 @@ Checkpointer storage runs in one of two channel modes, selected by `checkpoint_c
|
||||
**Where things live**:
|
||||
- `runtime/checkpoint_mode.py` — mode + snapshot-frequency freeze, marker injection, delta detection, compatibility gate, both error types
|
||||
- `runtime/checkpoint_state.py` — `CheckpointStateAccessor`, `build_state_mutation_graph`, `RollbackPoint`
|
||||
- `checkpoint_patches.py` (package root) — the one remaining patch: `BinaryOperatorAggregate` unwrapping an `Overwrite` first write into an empty channel (#4380; probe-guarded). The `InMemorySaver` delta-history patch is gone: `langgraph-checkpoint` 4.2.0 fixed that write loss upstream (#8526) while keeping its override, so the dependency floor plus the full → delta migration contract test are the gate — never re-add a version-guarded saver patch.
|
||||
- `checkpoint_patches.py` (package root) — the one remaining patch: `BinaryOperatorAggregate` unwrapping an `Overwrite` first write into an empty channel (#4380; probe-guarded). The `InMemorySaver` delta-history patch is gone: `langgraph-checkpoint` 4.2.0 fixed that write loss upstream (#8526) while keeping its override. The 4.3.0 floor also keeps same-superstep replay in live task-path order; SQLite 3.1.2 and Postgres 3.1.3 carry the matching saver behavior, including paged Postgres ancestor walks. Dependency floors plus the migration, ordering and pagination contract tests are the gates — never re-add version-guarded saver patches or defect-triggered skips.
|
||||
- `agents/thread_state.py` — `ThreadState`/`DeltaThreadState`, `delta_messages_field` / `DELTA_MESSAGES_FIELD` (`DeltaChannel` at the configured `snapshot_frequency`, default 10), schema adaptation helpers
|
||||
- `runtime/context_compaction.py` — compaction via accessor + mutation graph (reference consumer). Runs stamp their effective agent into server-owned checkpoint metadata; manual compaction uses that binding—not request `agent_name`—for memory policy and bucket. Missing/invalid legacy bindings and unreadable agent configs fail closed by skipping the optional flush while compaction may continue with the default model; a missing pre-binding checkpoint emits a warning so the skipped write is observable.
|
||||
- `runtime/checkpoint_cache/` + `runtime/checkpointer/cached_saver.py` — delta-mode checkpoint history cache; checkpoint state reads MUST go through `CheckpointStateAccessor`, and the checkpointer may be a `CachedHistorySaver` wrapper — never rely on concrete saver types
|
||||
|
||||
@ -47,13 +47,11 @@ dependencies = [
|
||||
"ddgs>=9.10.0",
|
||||
"duckdb>=1.4.4",
|
||||
"langchain-google-genai>=4.2.1",
|
||||
# Floor matters: the InMemorySaver delta-history patch was removed once
|
||||
# 4.2.0 fixed the dropped first post-migration write upstream (#8526) and
|
||||
# kept its own override, so a 4.1.x resolution would silently reintroduce
|
||||
# the bug. langgraph's own constraint (langgraph-checkpoint>=4.1.0) does
|
||||
# not exclude that version.
|
||||
"langgraph-checkpoint>=4.2.0,<5.0",
|
||||
"langgraph-checkpoint-sqlite>=3.1.1,<3.2",
|
||||
# Floor matters: 4.2.0 fixed the dropped first post-migration write, and
|
||||
# 4.3.0 makes same-superstep delta replay preserve live task-path order.
|
||||
# Older resolutions can silently reintroduce both replay defects.
|
||||
"langgraph-checkpoint>=4.3.0,<5.0",
|
||||
"langgraph-checkpoint-sqlite>=3.1.2,<3.2",
|
||||
"langgraph-sdk>=0.1.51",
|
||||
"sqlalchemy[asyncio]>=2.0,<3.0",
|
||||
"aiosqlite>=0.19",
|
||||
@ -80,9 +78,9 @@ groundroute = []
|
||||
ollama = ["langchain-ollama>=0.3.0"]
|
||||
postgres = [
|
||||
"asyncpg>=0.29",
|
||||
# 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.1 walks past
|
||||
# them to the thread root instead.
|
||||
"langgraph-checkpoint-postgres>=3.1.2,<3.2",
|
||||
# 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.3 fixes the
|
||||
# paged ancestor-walk cursor (upstream #8556).
|
||||
"langgraph-checkpoint-postgres>=3.1.3,<3.2",
|
||||
"psycopg[binary]>=3.3.3",
|
||||
"psycopg-pool>=3.3.0",
|
||||
]
|
||||
|
||||
@ -208,7 +208,7 @@ class TestHarnessPackaging:
|
||||
assert "postgres" in optional_dependencies
|
||||
assert optional_dependencies["postgres"] == [
|
||||
"asyncpg>=0.29",
|
||||
"langgraph-checkpoint-postgres>=3.1.2,<3.2",
|
||||
"langgraph-checkpoint-postgres>=3.1.3,<3.2",
|
||||
"psycopg[binary]>=3.3.3",
|
||||
"psycopg-pool>=3.3.0",
|
||||
]
|
||||
|
||||
@ -16,14 +16,11 @@ Every contract runs against InMemorySaver, AsyncSqliteSaver, and - when
|
||||
``TEST_POSTGRES_URI`` is set - AsyncPostgresSaver, because each backend
|
||||
implements blob/version handling slightly differently.
|
||||
|
||||
Two further tests pin currently-unfixed upstream defects (langgraph #8382
|
||||
parallel-superstep replay order, #8448 Postgres paginated delta walk). They
|
||||
assert the correct contract and trip - skip, naming the issue - while the
|
||||
defect is present, becoming live gates once a dependency bump lands a fix. The
|
||||
#8448 trip is scoped to the Postgres parameter, the only backend that pages its
|
||||
stage-1 scan: on memory/sqlite it stays a live differential assertion against
|
||||
``InMemorySaver``, so a regression there fails instead of blaming the
|
||||
Postgres-only upstream issue.
|
||||
Two further tests pin parallel-superstep replay order and Postgres paginated
|
||||
delta walks. The dependency floors carry the upstream fixes, so these are live
|
||||
regression gates rather than defect-triggered skips. The pagination contract
|
||||
uses Postgres for its paged stage-1 scan and retains a live differential
|
||||
assertion against ``InMemorySaver`` on the other backends.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -331,16 +328,12 @@ async def test_non_delta_writers_preserve_delta_messages_and_markers(saver_env:
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known-unfixed upstream DeltaChannel defects.
|
||||
# DeltaChannel replay ordering and pagination regression gates.
|
||||
#
|
||||
# langgraph #8382 (parallel-superstep delta replay order) and #8448 (Postgres
|
||||
# paginated delta-walk cursor) are both still open against the pinned
|
||||
# langgraph-checkpoint 4.2.0 / langgraph-checkpoint-postgres 3.1.2, with fixes
|
||||
# (PR #8544 / PR #8556) unmerged and no released version carrying them. Each
|
||||
# test below asserts the CORRECT contract; while the defect is present it trips
|
||||
# (skips, naming the upstream issue) instead of failing, so a later dependency
|
||||
# bump that lands the fix flips it into a live gate. The contract is never
|
||||
# asserted as buggy behaviour.
|
||||
# langgraph-checkpoint 4.3.0 and the matching saver releases preserve live
|
||||
# task-path write order and keep the Postgres paged ancestor-walk cursor valid.
|
||||
# The dependency floors exclude older implementations, so any regression must
|
||||
# fail these tests rather than being converted into a skip.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_PARALLEL_LABELS = tuple("abcdefgh")
|
||||
@ -399,10 +392,7 @@ async def _long_chain_history(saver: Any, thread_id: str, steps: int) -> list[tu
|
||||
@pytest.mark.anyio
|
||||
async def test_long_chain_history_survives_pagination(saver_env: _SaverEnv, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Every checkpoint of a long delta chain must materialize its own ancestor
|
||||
prefix. Postgres resolves the history through a stage-1 scan paged
|
||||
newest-first; when the target checkpoint is not on the first page the
|
||||
unpatched cursor is derived from a not-yet-loaded parent and parks at
|
||||
``None`` for good, so old checkpoints hydrate empty (upstream #8448)."""
|
||||
prefix, including when Postgres has to page beyond the first stage-1 scan."""
|
||||
if saver_env.kind == "postgres":
|
||||
pg_aio = pytest.importorskip("langgraph.checkpoint.postgres.aio", reason="postgres extra not installed")
|
||||
monkeypatch.setattr(pg_aio, "_DELTA_PAGE_SIZE", _PAGINATION_PAGE_SIZE)
|
||||
@ -415,8 +405,6 @@ async def test_long_chain_history_survives_pagination(saver_env: _SaverEnv, monk
|
||||
assert oracle[0] == (_expected_long_chain(_PAGINATION_STEPS), ())
|
||||
assert all(digest for digest, next_key in oracle if next_key != ("__start__",))
|
||||
|
||||
if saver_env.kind == "postgres" and got != oracle:
|
||||
pytest.skip("upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty")
|
||||
assert got == oracle
|
||||
|
||||
|
||||
@ -438,11 +426,8 @@ def _build_fanout_graph(saver: Any) -> Any:
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_parallel_superstep_replay_matches_live_write_order(saver_env: _SaverEnv) -> None:
|
||||
"""When several parallel tasks write the same delta channel in one
|
||||
superstep, the live ``apply_writes`` order is path-sorted, but the delta
|
||||
history replay sorts by the hashed ``(task_id, idx)`` - an unrelated order
|
||||
(upstream #8382). ``get_state`` on the same thread must report the order the
|
||||
run actually produced."""
|
||||
"""Parallel delta writes must replay in the path order used by the live
|
||||
``apply_writes`` pass, not in an unrelated hashed ``task_id`` order."""
|
||||
graph = _build_fanout_graph(saver_env.saver)
|
||||
config = _config(_thread_id())
|
||||
live = await graph.ainvoke({"messages": []}, config)
|
||||
@ -455,6 +440,4 @@ async def test_parallel_superstep_replay_matches_live_write_order(saver_env: _Sa
|
||||
replayed = await graph.aget_state(config)
|
||||
replayed_order = [m.content for m in replayed.values["messages"]]
|
||||
|
||||
if replayed_order != live_order:
|
||||
pytest.skip("upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes")
|
||||
assert replayed_order == live_order
|
||||
|
||||
6
backend/uv.lock
generated
6
backend/uv.lock
generated
@ -1093,9 +1093,9 @@ requires-dist = [
|
||||
{ name = "langfuse", specifier = ">=3.4.1" },
|
||||
{ name = "langgraph", specifier = ">=1.2.9,<1.3" },
|
||||
{ name = "langgraph-api", specifier = ">=0.8.1" },
|
||||
{ name = "langgraph-checkpoint", specifier = ">=4.2.0,<5.0" },
|
||||
{ name = "langgraph-checkpoint-postgres", marker = "extra == 'postgres'", specifier = ">=3.1.2,<3.2" },
|
||||
{ name = "langgraph-checkpoint-sqlite", specifier = ">=3.1.1,<3.2" },
|
||||
{ name = "langgraph-checkpoint", specifier = ">=4.3.0,<5.0" },
|
||||
{ name = "langgraph-checkpoint-postgres", marker = "extra == 'postgres'", specifier = ">=3.1.3,<3.2" },
|
||||
{ name = "langgraph-checkpoint-sqlite", specifier = ">=3.1.2,<3.2" },
|
||||
{ name = "langgraph-cli", specifier = ">=0.4.24" },
|
||||
{ name = "langgraph-runtime-inmem", specifier = ">=0.30.0" },
|
||||
{ name = "langgraph-sdk", specifier = ">=0.1.51" },
|
||||
|
||||
@ -128,13 +128,13 @@ The dependency floor is what keeps the removed saver bug out, so the floors are
|
||||
|
||||
| Distribution | Floor | Why |
|
||||
| --- | --- | --- |
|
||||
| `langgraph-checkpoint` | `>=4.2.0,<5.0` | 4.2.0 fixes the dropped first post-migration write (upstream #8526). A 4.1.x resolution would silently reintroduce it, and `langgraph`'s own `langgraph-checkpoint>=4.1.0` constraint does not exclude that version |
|
||||
| `langgraph-checkpoint-postgres` | `>=3.1.2,<3.2` | 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.1 walks past them to the thread root |
|
||||
| `langgraph-checkpoint-sqlite` | `>=3.1.1,<3.2` | Pinned alongside the other savers |
|
||||
| `langgraph-checkpoint` | `>=4.3.0,<5.0` | 4.2.0 fixes the dropped first post-migration write (upstream #8526); 4.3.0 preserves live task-path order when replaying parallel same-superstep writes (upstream #8544) |
|
||||
| `langgraph-checkpoint-postgres` | `>=3.1.3,<3.2` | 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.3 fixes the paged ancestor-walk cursor (upstream #8556) |
|
||||
| `langgraph-checkpoint-sqlite` | `>=3.1.2,<3.2` | Matches the checkpoint replay-order fix |
|
||||
|
||||
The regression gate is `backend/tests/test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`: it fails when the 4.2.0 fix is absent, and a full-to-delta migration must keep replaying on the same thread. The pinned environment resolves `langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, and `langgraph-checkpoint-postgres 3.1.2`.
|
||||
The regression gates in `backend/tests/test_delta_channel_checkpointers.py` cover full-to-delta migration, parallel-write replay order, and paged ancestor walks. The locked environment resolves `langgraph 1.2.14`, `langgraph-checkpoint 4.3.0`, and `langgraph-checkpoint-postgres 3.1.3`.
|
||||
|
||||
There is no runtime version check or warning. The only version signal an operator sees is a trigger-skipped test naming the pinned version, for example the `#8448` Postgres skip or the `#8382` replay-order skip.
|
||||
There is no runtime version check or warning. The dependency floors exclude affected versions, and all three regression contracts fail rather than skip when a defect returns.
|
||||
|
||||
## Running the benchmarks
|
||||
|
||||
|
||||
@ -36,7 +36,7 @@ Evidence column values are repo-relative `path:line` references using the aliase
|
||||
| `context_usage` | `backend/app/gateway/context_usage.py` |
|
||||
| `example` | `config.example.yaml` |
|
||||
|
||||
The pinned dependency set referenced by the defect table is `langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, and `langgraph-checkpoint-postgres 3.1.2`.
|
||||
The locked dependency set referenced by the defect table is `langgraph 1.2.14`, `langgraph-checkpoint 4.3.0`, and `langgraph-checkpoint-postgres 3.1.3`.
|
||||
|
||||
## Configuration keys
|
||||
|
||||
@ -288,7 +288,7 @@ Tests a reader can run or cite as the executable contract. All paths are under `
|
||||
- `test_delta_channel_state.py::test_production_message_forms_keep_assigned_ids_across_delta_replay`
|
||||
- `test_delta_channel_state.py::test_delta_normalization_compiles_stable_channel_without_mutating_middleware`
|
||||
|
||||
`test_long_chain_history_survives_pagination` and `test_parallel_superstep_replay_matches_live_write_order` assert the correct contract and trigger-skip while the pinned dependency still has the defect; they are the gates that flip live on a dependency bump.
|
||||
`test_long_chain_history_survives_pagination` and `test_parallel_superstep_replay_matches_live_write_order` unconditionally assert the correct replay contracts and fail if either upstream defect returns.
|
||||
|
||||
### History cache
|
||||
|
||||
@ -351,14 +351,14 @@ Tests a reader can run or cite as the executable contract. All paths are under `
|
||||
|
||||
## Pinned upstream defects
|
||||
|
||||
State at the pinned set (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`). Each local test asserts the correct contract and trigger-skips while the defect is present, so a later dependency bump flips it into a live gate; the buggy behaviour is never asserted as correct.
|
||||
State at the locked set (`langgraph 1.2.14`, `langgraph-checkpoint 4.3.0`, `langgraph-checkpoint-postgres 3.1.3`). Fixed replay defects have unconditional regression gates; defects that remain open retain their documented local handling.
|
||||
|
||||
| id | Kind / status | Scenario that breaks | Local handling | Pinned version | Evidence |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `#4380` | DeerFlow issue; patched locally | A replace-style write into an empty Union-typed `BinaryOperatorAggregate` channel persists the `Overwrite` wrapper; the next read crashes with `TypeError: 'Overwrite' object is not subscriptable` | Probe-guarded patch in `checkpoint_patches.py`; patch tests; branch/state-update anchors in `test_threads_router.py` | `langgraph 1.2.9` | `checkpoint_patches:91-107`, `backend/tests/test_checkpoint_patches.py:31-95` |
|
||||
| `#8526` (fixes upstream `#8384`), plus postgres `#8535` | upstream langgraph PR; merged 2026-08-07; fixed in `langgraph-checkpoint` 4.2.0 | After switching a thread from `BinaryOperatorAggregate` to `DeltaChannel`, the first post-migration write survives the live `invoke` return value but disappears from every subsequent read; on postgres, with no seed the walk runs to the root and replays everything | Local `InMemorySaver` patch deleted; dependency floor `langgraph-checkpoint>=4.2.0,<5.0`; regression gate `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`; a version-guarded saver patch must not be re-added | `langgraph-checkpoint 4.2.0` | `checkpoint_patches:11-17`, `backend/packages/harness/pyproject.toml:49-56`, `CHANGELOG.md:3040-3048` |
|
||||
| `#8382` | upstream langgraph issue; **open**; fix PR `#8544` unmerged, no released version | Delta replay order diverges from live execution order for parallel-superstep writes: live `invoke` returns path-sorted `[a..h]`, `get_state` replays `[f,b,g,h,a,c,d,e]` (a pure permutation); reproduced on memory, sqlite, and postgres | Trigger-skip naming the issue; `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.2.0` | `test_delta_channel_checkpointers.py:419,440-460` |
|
||||
| `#8448` | upstream langgraph issue; **open**; fix PR `#8556` unmerged | `PostgresSaver.get_delta_channel_history` permanently poisons the walk cursor when the target checkpoint is not in the first pagination page, so old checkpoints hydrate empty; only Postgres pages its stage-1 scan | Assertion stays live on memory/sqlite (differential against an `InMemorySaver` oracle); Postgres trigger-skips; `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` | `langgraph-checkpoint-postgres 3.1.2` | `test_delta_channel_checkpointers.py:459` |
|
||||
| `#8526` (fixes upstream `#8384`), plus postgres `#8535` | upstream langgraph PR; merged 2026-08-07; fixed in `langgraph-checkpoint` 4.2.0 | After switching a thread from `BinaryOperatorAggregate` to `DeltaChannel`, the first post-migration write survives the live `invoke` return value but disappears from every subsequent read; on postgres, with no seed the walk runs to the root and replays everything | Local `InMemorySaver` patch deleted; dependency floor `langgraph-checkpoint>=4.3.0,<5.0`; regression gate `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`; a version-guarded saver patch must not be re-added | `langgraph-checkpoint 4.3.0` | `checkpoint_patches:11-17`, `backend/packages/harness/pyproject.toml:50-54`, `CHANGELOG.md:3040-3048` |
|
||||
| `#8382` | upstream langgraph issue; fixed by PR `#8544` in `langgraph-checkpoint` 4.3.0 | Delta replay order diverges from live execution order for parallel-superstep writes: live `invoke` returns path-sorted `[a..h]`, while affected versions replay a pure permutation such as `[f,b,g,h,a,c,d,e]`; reproduced on memory, sqlite, and postgres | Dependency floor `langgraph-checkpoint>=4.3.0,<5.0`; unconditional regression gate `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.3.0` | `test_delta_channel_checkpointers.py:427-443` |
|
||||
| `#8448` | upstream langgraph issue; fixed by PR `#8556` in `langgraph-checkpoint-postgres` 3.1.3 | `PostgresSaver.get_delta_channel_history` permanently poisons the walk cursor when the target checkpoint is not in the first pagination page, so old checkpoints hydrate empty; only Postgres pages its stage-1 scan | Dependency floor `langgraph-checkpoint-postgres>=3.1.3,<3.2`; unconditional regression gate `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` against an `InMemorySaver` oracle | `langgraph-checkpoint-postgres 3.1.3` | `test_delta_channel_checkpointers.py:392-408` |
|
||||
| `#8548` | upstream langgraph PR; **open**; fixes upstream `#8443`, with related issue `#8551` still waiting on it | Does not replay an abandoned branch into a `DeltaChannel` fork | While unmerged the local linearization cannot be removed: `_linearize_delta_checkpoint_resume` is retained | `langgraph 1.2.9` | `worker:2267-2279`, `test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head` |
|
||||
| `#4458` | DeerFlow issue; the user-visible symptom of the fork bug | Regenerating in a branched thread resurrected the old assistant message beside the new one after a reload; reproduced on postgres, sqlite, and the in-memory saver, and it also broke branch history when seeded rows shared one run id | Linearization plus per-turn seeded run ids; `test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer` | `langgraph 1.2.9` | `worker:2267-2289`, `test_run_worker_delta_resume.py:204` |
|
||||
| `#4189` item 3 | DeerFlow design discussion | Retention and deletion of checkpoint rows; what may be deleted | `backend/docs/checkpoint-retention-contract.md`, `backend/tests/test_checkpoint_retention_contract.py`, `backend/app/gateway/checkpoint_retention.py`; bounded cleanup runs at durable admission before checkpoint reads, protects explicit selections, and removes only the root trailing duration leaf | — | `checkpoint_retention.py`, `backend/docs/checkpoint-retention-contract.md:31-77` |
|
||||
|
||||
@ -149,29 +149,17 @@ The upstream fix is PR [#8548](https://github.com/langchain-ai/langgraph/pull/85
|
||||
|
||||
**Symptom**: on a parallel fan-out that writes one delta channel in a single super-step, the live run returns writes in path order but `get_state` / resume replays them in a different order.
|
||||
|
||||
**Cause**: pinned upstream defect ([langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)), open, fix PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) unmerged, no released version. Delta replay orders writes by hashed `(task_id, idx)` instead of the path order `apply_writes` used live, so same-superstep writes are permuted. Present in `langgraph-checkpoint 4.2.0`; reproduced on memory, SQLite, and Postgres.
|
||||
**Cause**: upstream defect [langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382), fixed by PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) in `langgraph-checkpoint` 4.3.0. Affected versions order delta replay writes by hashed `(task_id, idx)` instead of the path order `apply_writes` uses live, permuting same-superstep writes on memory, SQLite, and Postgres.
|
||||
|
||||
**Fix**: none locally. DeerFlow's test asserts the correct contract and trigger-skips while the defect is present, naming the issue:
|
||||
|
||||
```
|
||||
upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes
|
||||
```
|
||||
|
||||
If ordering within one super-step matters to your consumer, avoid fanning parallel writes into a single delta channel.
|
||||
**Fix**: use `langgraph-checkpoint>=4.3.0,<5.0`. DeerFlow's dependency floor excludes affected versions, and `test_parallel_superstep_replay_matches_live_write_order` unconditionally asserts that replay matches live write order.
|
||||
|
||||
### Postgres hydrates old checkpoints empty
|
||||
|
||||
**Symptom**: on Postgres, an older delta checkpoint materializes empty while a recent one is fine.
|
||||
|
||||
**Cause**: pinned upstream defect ([langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)), open, fix PR #8556 unmerged. `PostgresSaver.get_delta_channel_history` permanently poisons its walk cursor when the target checkpoint is not in the first pagination page: the unpatched cursor is derived from a not-yet-loaded parent and parks at `None` for good, so old checkpoints hydrate empty. Only Postgres pages its stage-1 scan. Present in `langgraph-checkpoint-postgres 3.1.2`.
|
||||
**Cause**: upstream defect [langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448), fixed by PR #8556 in `langgraph-checkpoint-postgres` 3.1.3. In affected versions, `PostgresSaver.get_delta_channel_history` derives its walk cursor from a not-yet-loaded parent when the target checkpoint is beyond the first pagination page; the cursor then remains `None`, so old checkpoints hydrate empty.
|
||||
|
||||
**Fix**: none locally; the Postgres case trigger-skips, naming the issue:
|
||||
|
||||
```
|
||||
upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty
|
||||
```
|
||||
|
||||
Cadence bounds replay depth, so a smaller `snapshot_frequency` keeps the walk shorter; that is the only lever this repository exposes, not a fix for the cursor bug.
|
||||
**Fix**: use `langgraph-checkpoint-postgres>=3.1.3,<3.2`. DeerFlow's dependency floor excludes affected versions, and `test_long_chain_history_survives_pagination` unconditionally checks every materialized checkpoint against an `InMemorySaver` oracle.
|
||||
|
||||
## Cache
|
||||
|
||||
|
||||
@ -125,13 +125,13 @@ run 获得持久化的同线程 admission 后，Gateway 会在首次 checkpoint
|
||||
|
||||
| 发行包 | 下限 | 原因 |
|
||||
| --- | --- | --- |
|
||||
| `langgraph-checkpoint` | `>=4.2.0,<5.0` | 4.2.0 修掉了迁移后首个写入丢失（上游 #8526）。解析到 4.1.x 会静默重新引入该缺陷，而 `langgraph` 自己的 `langgraph-checkpoint>=4.1.0` 约束并不排除该版本 |
|
||||
| `langgraph-checkpoint-postgres` | `>=3.1.2,<3.2` | 3.1.2 能找到纯值形式的增量种子（上游 #8535）；3.1.1 会越过它一路走到线程根 |
|
||||
| `langgraph-checkpoint-sqlite` | `>=3.1.1,<3.2` | 与其他 saver 一同锁定 |
|
||||
| `langgraph-checkpoint` | `>=4.3.0,<5.0` | 4.2.0 修掉了迁移后首个写入丢失（上游 #8526）；4.3.0 让并行 super-step 写入在重放时保持实时任务路径顺序（上游 #8544） |
|
||||
| `langgraph-checkpoint-postgres` | `>=3.1.3,<3.2` | 3.1.2 能找到纯值形式的增量种子（上游 #8535）；3.1.3 修复分页祖先回走的游标（上游 #8556） |
|
||||
| `langgraph-checkpoint-sqlite` | `>=3.1.2,<3.2` | 与 checkpoint 的回放顺序修复配套 |
|
||||
|
||||
回归门禁是 `backend/tests/test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`：当 4.2.0 的修复缺失时它会失败，保证 full→delta 迁移仍能在同一线程上回放。锁定环境解析到 `langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`。
|
||||
`backend/tests/test_delta_channel_checkpointers.py` 中的回归门禁覆盖 full→delta 迁移、并行写入回放顺序和分页祖先回走。锁定环境解析到 `langgraph 1.2.14`、`langgraph-checkpoint 4.3.0`、`langgraph-checkpoint-postgres 3.1.3`。
|
||||
|
||||
运行时没有版本检查，也没有告警。运维能看到的唯一版本信号，是一条带有锁定版本号的触发式跳过测试，例如 `#8448` 的 Postgres 跳过或 `#8382` 的回放顺序跳过。
|
||||
运行时没有版本检查，也没有告警。依赖下限会排除受影响的版本；这三个回归契约在缺陷复现时都会直接失败，不再跳过。
|
||||
|
||||
## 如何跑基准
|
||||
|
||||
|
||||
@ -36,7 +36,7 @@ import { Callout } from "nextra/components";
|
||||
| `context_usage` | `backend/app/gateway/context_usage.py` |
|
||||
| `example` | `config.example.yaml` |
|
||||
|
||||
缺陷表引用的固定依赖集为 `langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`。
|
||||
缺陷表引用的锁定依赖集为 `langgraph 1.2.14`、`langgraph-checkpoint 4.3.0`、`langgraph-checkpoint-postgres 3.1.3`。
|
||||
|
||||
## 配置键
|
||||
|
||||
@ -288,7 +288,7 @@ wait 响应仍为 HTTP `200`。Python SDK 默认使用 `raise_error=True`，识
|
||||
- `test_delta_channel_state.py::test_production_message_forms_keep_assigned_ids_across_delta_replay`
|
||||
- `test_delta_channel_state.py::test_delta_normalization_compiles_stable_channel_without_mutating_middleware`
|
||||
|
||||
`test_long_chain_history_survives_pagination` 与 `test_parallel_superstep_replay_matches_live_write_order` 断言的是正确契约，在固定依赖仍带缺陷时触发 skip；依赖升级后它们会成为实际门禁。
|
||||
`test_long_chain_history_survives_pagination` 与 `test_parallel_superstep_replay_matches_live_write_order` 无条件断言正确的回放契约；任一上游缺陷复现时都会直接失败。
|
||||
|
||||
### 历史缓存
|
||||
|
||||
@ -351,14 +351,14 @@ wait 响应仍为 HTTP `200`。Python SDK 默认使用 `raise_error=True`，识
|
||||
|
||||
## 上游缺陷固定表
|
||||
|
||||
状态对应固定依赖集（`langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`）。每条本地测试断言的都是正确契约，缺陷存在期间触发 skip，因此后续依赖升级会把它变成实际门禁；有缺陷的行为从不被当作正确行为断言。
|
||||
状态对应锁定依赖集（`langgraph 1.2.14`、`langgraph-checkpoint 4.3.0`、`langgraph-checkpoint-postgres 3.1.3`）。已修复的回放缺陷都有无条件回归门禁；仍未修复的缺陷保留文档所述的本地处理。
|
||||
|
||||
| 编号 | 种类 / 状态 | 触发场景 | 本地应对 | 固定版本 | 证据 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `#4380` | DeerFlow issue；已在本地打补丁 | 向空的 Union 类型 `BinaryOperatorAggregate` 通道做替换式写入会把 `Overwrite` 包装器持久化；下一次读取以 `TypeError: 'Overwrite' object is not subscriptable` 崩溃 | `checkpoint_patches.py` 中的探针守卫补丁；补丁测试；`test_threads_router.py` 中的分支/状态更新锚点 | `langgraph 1.2.9` | `checkpoint_patches:91-107`、`backend/tests/test_checkpoint_patches.py:31-95` |
|
||||
| `#8526`（修复上游 `#8384`），以及 postgres `#8535` | 上游 langgraph PR；2026-08-07 合并；`langgraph-checkpoint` 4.2.0 修复 | 线程从 `BinaryOperatorAggregate` 切换到 `DeltaChannel` 后，迁移后的第一次写入能从实时 `invoke` 返回值中看到，却从之后每次读取中消失；postgres 侧在没有 seed 时走到根并重放全部内容 | 删除本地 `InMemorySaver` 补丁；依赖下限 `langgraph-checkpoint>=4.2.0,<5.0`；回归门禁 `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`；禁止再引入版本守卫式 saver 补丁 | `langgraph-checkpoint 4.2.0` | `checkpoint_patches:11-17`、`backend/packages/harness/pyproject.toml:49-56`、`CHANGELOG.md:3040-3048` |
|
||||
| `#8382` | 上游 langgraph issue；**open**；修复 PR `#8544` 未合并，无发布版本 | 并行 super-step 写入的 delta 重放顺序与实时执行顺序不一致：实时 `invoke` 返回按路径排序的 `[a..h]`，`get_state` 重放为 `[f,b,g,h,a,c,d,e]`（纯置换）；在 memory、sqlite、postgres 上均可复现 | 触发 skip 并点名该 issue；`test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.2.0` | `test_delta_channel_checkpointers.py:419,440-460` |
|
||||
| `#8448` | 上游 langgraph issue；**open**；修复 PR `#8556` 未合并 | 目标检查点不在第一页分页结果中时，`PostgresSaver.get_delta_channel_history` 会永久污染游标，导致老检查点物化为空；只有 Postgres 会对 stage-1 扫描分页 | memory/sqlite 上断言保持生效（与 `InMemorySaver` 预言机做差分）；Postgres 触发 skip；`test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` | `langgraph-checkpoint-postgres 3.1.2` | `test_delta_channel_checkpointers.py:459` |
|
||||
| `#8526`（修复上游 `#8384`），以及 postgres `#8535` | 上游 langgraph PR；2026-08-07 合并；`langgraph-checkpoint` 4.2.0 修复 | 线程从 `BinaryOperatorAggregate` 切换到 `DeltaChannel` 后，迁移后的第一次写入能从实时 `invoke` 返回值中看到，却从之后每次读取中消失；postgres 侧在没有 seed 时走到根并重放全部内容 | 删除本地 `InMemorySaver` 补丁；依赖下限 `langgraph-checkpoint>=4.3.0,<5.0`；回归门禁 `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`；禁止再引入版本守卫式 saver 补丁 | `langgraph-checkpoint 4.3.0` | `checkpoint_patches:11-17`、`backend/packages/harness/pyproject.toml:50-54`、`CHANGELOG.md:3040-3048` |
|
||||
| `#8382` | 上游 langgraph issue；已由 PR `#8544` 在 `langgraph-checkpoint` 4.3.0 中修复 | 并行 super-step 写入的 delta 重放顺序与实时执行顺序不一致：实时 `invoke` 返回按路径排序的 `[a..h]`，受影响版本则会重放为 `[f,b,g,h,a,c,d,e]` 一类的纯置换；在 memory、sqlite、postgres 上均可复现 | 依赖下限 `langgraph-checkpoint>=4.3.0,<5.0`；无条件回归门禁 `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.3.0` | `test_delta_channel_checkpointers.py:427-443` |
|
||||
| `#8448` | 上游 langgraph issue；已由 PR `#8556` 在 `langgraph-checkpoint-postgres` 3.1.3 中修复 | 目标检查点不在第一页分页结果中时，`PostgresSaver.get_delta_channel_history` 会永久污染游标，导致老检查点物化为空；只有 Postgres 会对 stage-1 扫描分页 | 依赖下限 `langgraph-checkpoint-postgres>=3.1.3,<3.2`；无条件回归门禁 `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination`，与 `InMemorySaver` 预言机对照 | `langgraph-checkpoint-postgres 3.1.3` | `test_delta_channel_checkpointers.py:392-408` |
|
||||
| `#8548` | 上游 langgraph PR；**open**；修复上游 `#8443`，相关 issue `#8551` 仍在等它 | 不把被放弃的分支重放进 `DeltaChannel` 分叉 | 未合并期间本地线性化不能移除：保留 `_linearize_delta_checkpoint_resume` | `langgraph 1.2.9` | `worker:2267-2279`、`test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head` |
|
||||
| `#4458` | DeerFlow issue；分叉缺陷对用户可见的症状 | 在分叉线程中重新生成后，重载时旧助手消息会与新回答并排复活；在 postgres、sqlite 与内存 saver 上均可复现，且当种子行共用一个 run id 时还会破坏分支历史 | 线性化加按轮次播种的 run id；`test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer` | `langgraph 1.2.9` | `worker:2267-2289`、`test_run_worker_delta_resume.py:204` |
|
||||
| `#4189` 第 3 项 | DeerFlow 设计讨论 | 检查点行的保留与删除；哪些内容可以被删除 | `backend/docs/checkpoint-retention-contract.md`、`backend/tests/test_checkpoint_retention_contract.py`、`backend/app/gateway/checkpoint_retention.py`；目前没有接入生产触发点 | — | `checkpoint_retention.py:1-40`、`backend/docs/checkpoint-retention-contract.md:31-77` |
|
||||
|
||||
@ -147,29 +147,17 @@ Run {run_id} could not materialize resume checkpoint {checkpoint_id}
|
||||
|
||||
**症状**：在并行扇出向同一 delta 通道写入的同一个 super-step 中，实时 run 按路径顺序返回写入，但 `get_state` / 恢复以不同顺序回放。
|
||||
|
||||
**原因**：已固定的上游缺陷（[langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)），open，修复 PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) 未合并，无已发布版本。delta 回放按哈希后的 `(task_id, idx)` 排序写入，而不是实时 `apply_writes` 使用的路径顺序，于是同一 super-step 的写入被重排。存在于 `langgraph-checkpoint 4.2.0`；已在 memory、SQLite 与 Postgres 上复现。
|
||||
**原因**：上游缺陷 [langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)，已由 PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) 在 `langgraph-checkpoint` 4.3.0 中修复。受影响的版本按哈希后的 `(task_id, idx)` 排序 delta 回放写入，而不是实时 `apply_writes` 使用的路径顺序，导致 memory、SQLite 与 Postgres 上同一 super-step 的写入被重排。
|
||||
|
||||
**处理**：本地无解。DeerFlow 的测试断言正确契约，并在缺陷存在时触发 skip，同时点名该 issue：
|
||||
|
||||
```
|
||||
upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes
|
||||
```
|
||||
|
||||
如果你的消费方在意同一 super-step 内的顺序，请避免把并行写入扇出到同一个 delta 通道。
|
||||
**处理**：使用 `langgraph-checkpoint>=4.3.0,<5.0`。DeerFlow 的依赖下限会排除受影响版本，`test_parallel_superstep_replay_matches_live_write_order` 无条件断言回放顺序与实时写入顺序一致。
|
||||
|
||||
### Postgres 物化旧检查点为空
|
||||
|
||||
**症状**：在 Postgres 上，较旧的 delta 检查点物化为空，较新的却正常。
|
||||
|
||||
**原因**：已固定的上游缺陷（[langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)），open，修复 PR #8556 未合并。当目标检查点不在第一页分页结果里时，`PostgresSaver.get_delta_channel_history` 会永久污染其回走游标：未修补的游标取自一个尚未加载的父级，永久停在 `None`，于是旧检查点物化为空。只有 Postgres 对其 stage-1 扫描分页。存在于 `langgraph-checkpoint-postgres 3.1.2`。
|
||||
**原因**：上游缺陷 [langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)，已由 PR #8556 在 `langgraph-checkpoint-postgres` 3.1.3 中修复。受影响的版本在目标检查点超出第一页分页结果时，从尚未加载的父级推导回走游标；游标随后一直停在 `None`，导致旧检查点物化为空。
|
||||
|
||||
**处理**：本地无解；Postgres 用例触发 skip，并点名该 issue：
|
||||
|
||||
```
|
||||
upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty
|
||||
```
|
||||
|
||||
节奏约束回放深度，因此更小的 `snapshot_frequency` 能让回走更短；这是本仓库暴露的唯一杠杆，而不是对该游标缺陷的修复。
|
||||
**处理**：使用 `langgraph-checkpoint-postgres>=3.1.3,<3.2`。DeerFlow 的依赖下限会排除受影响版本，`test_long_chain_history_survives_pagination` 无条件将每个物化检查点与 `InMemorySaver` 预言机对照。
|
||||
|
||||
## 缓存
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user