From 95514245e2802f62e10fd153bbf5eff879d91818 Mon Sep 17 00:00:00 2001 From: Zeren Wang <53075619+Vanzeren@users.noreply.github.com> Date: Thu, 12 Nov 2026 10:49:41 +0100 Subject: [PATCH] chore(checkpoint): require fixed DeltaChannel replay backends (#8139) * chore(checkpoint): require fixed DeltaChannel replay backends * fix(checkpoint): correct Postgres floor and refresh docs * test(checkpoint): update Postgres dependency floor assertion --------- Co-authored-by: Willem Jiang --- CHANGELOG.md | 9 ++++ CHANGELOG_zh.md | 8 ++++ backend/docs/runtime-guidance-details.md | 2 +- backend/packages/harness/pyproject.toml | 18 ++++---- backend/tests/test_checkpointer.py | 2 +- .../tests/test_delta_channel_checkpointers.py | 43 ++++++------------- backend/uv.lock | 6 +-- .../en/harness/checkpoints/operations.mdx | 10 ++--- .../en/harness/checkpoints/reference.mdx | 12 +++--- .../harness/checkpoints/troubleshooting.mdx | 20 ++------- .../zh/harness/checkpoints/operations.mdx | 10 ++--- .../zh/harness/checkpoints/reference.mdx | 12 +++--- .../harness/checkpoints/troubleshooting.mdx | 20 ++------- 13 files changed, 73 insertions(+), 99 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b78a05d27..9b4cea9339 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -58,6 +58,14 @@ This section accumulates work toward the **2.3.0** milestone connect an ordinary StepFun API account with an API key and model ID. ([#7988](https://github.com/bytedance/deer-flow/pull/7988)) +### Changed + +- **checkpoint:** DeltaChannel deployments now require `langgraph-checkpoint` + 4.3.0, SQLite saver 3.1.2, and Postgres saver 3.1.3, excluding versions that + could reorder parallel writes during replay or lose paged ancestor history. + The ordering and Postgres pagination contracts now fail on regression instead + of skipping. ([#8139]) + ### Fixed - **channels:** WeCom no longer drops images between 2 MB and 10 MB. The @@ -16346,5 +16354,6 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#8118]: https://github.com/bytedance/deer-flow/pull/8118 [#8126]: https://github.com/bytedance/deer-flow/pull/8126 [#8131]: https://github.com/bytedance/deer-flow/pull/8131 +[#8139]: https://github.com/bytedance/deer-flow/pull/8139 [#8151]: https://github.com/bytedance/deer-flow/pull/8151 [#8174]: https://github.com/bytedance/deer-flow/pull/8174 diff --git a/CHANGELOG_zh.md b/CHANGELOG_zh.md index a82b8079e2..6324927e91 100644 --- a/CHANGELOG_zh.md +++ b/CHANGELOG_zh.md @@ -24,6 +24,13 @@ - **模型:** 管理员可在 **设置 → 模型** 中选择 StepFun(阶跃星辰), 填写 API Key 和模型 ID 即可接入普通 API。([#7988](https://github.com/bytedance/deer-flow/pull/7988)) +### 变更 + +- **Checkpoint:** DeltaChannel 部署现在要求 `langgraph-checkpoint` 4.3.0、 + SQLite saver 3.1.2 和 Postgres saver 3.1.3,排除会在重放时重排并行写入或 + 丢失分页祖先历史的版本。顺序和 Postgres 分页契约在回归时现在会直接失败, + 不再跳过。([#8139]) + ### 修复 - **通道:** 企业微信不再丢弃 2–10 MB 的图片。此前适配器把图片上限定为 2 MB(同一协议中语音的限制), @@ -12875,4 +12882,5 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写,核心包含子 [#8064]: https://github.com/bytedance/deer-flow/pull/8064 [#8118]: https://github.com/bytedance/deer-flow/pull/8118 [#8126]: https://github.com/bytedance/deer-flow/pull/8126 +[#8139]: https://github.com/bytedance/deer-flow/pull/8139 [#8174]: https://github.com/bytedance/deer-flow/pull/8174 diff --git a/backend/docs/runtime-guidance-details.md b/backend/docs/runtime-guidance-details.md index 5b0f401123..8b7d8754e5 100644 --- a/backend/docs/runtime-guidance-details.md +++ b/backend/docs/runtime-guidance-details.md @@ -343,7 +343,7 @@ Checkpointer storage runs in one of two channel modes, selected by `checkpoint_c **Where things live**: - `runtime/checkpoint_mode.py` — mode + snapshot-frequency freeze, marker injection, delta detection, compatibility gate, both error types - `runtime/checkpoint_state.py` — `CheckpointStateAccessor`, `build_state_mutation_graph`, `RollbackPoint` -- `checkpoint_patches.py` (package root) — the one remaining patch: `BinaryOperatorAggregate` unwrapping an `Overwrite` first write into an empty channel (#4380; probe-guarded). The `InMemorySaver` delta-history patch is gone: `langgraph-checkpoint` 4.2.0 fixed that write loss upstream (#8526) while keeping its override, so the dependency floor plus the full → delta migration contract test are the gate — never re-add a version-guarded saver patch. +- `checkpoint_patches.py` (package root) — the one remaining patch: `BinaryOperatorAggregate` unwrapping an `Overwrite` first write into an empty channel (#4380; probe-guarded). The `InMemorySaver` delta-history patch is gone: `langgraph-checkpoint` 4.2.0 fixed that write loss upstream (#8526) while keeping its override. The 4.3.0 floor also keeps same-superstep replay in live task-path order; SQLite 3.1.2 and Postgres 3.1.3 carry the matching saver behavior, including paged Postgres ancestor walks. Dependency floors plus the migration, ordering and pagination contract tests are the gates — never re-add version-guarded saver patches or defect-triggered skips. - `agents/thread_state.py` — `ThreadState`/`DeltaThreadState`, `delta_messages_field` / `DELTA_MESSAGES_FIELD` (`DeltaChannel` at the configured `snapshot_frequency`, default 10), schema adaptation helpers - `runtime/context_compaction.py` — compaction via accessor + mutation graph (reference consumer). Runs stamp their effective agent into server-owned checkpoint metadata; manual compaction uses that binding—not request `agent_name`—for memory policy and bucket. Missing/invalid legacy bindings and unreadable agent configs fail closed by skipping the optional flush while compaction may continue with the default model; a missing pre-binding checkpoint emits a warning so the skipped write is observable. - `runtime/checkpoint_cache/` + `runtime/checkpointer/cached_saver.py` — delta-mode checkpoint history cache; checkpoint state reads MUST go through `CheckpointStateAccessor`, and the checkpointer may be a `CachedHistorySaver` wrapper — never rely on concrete saver types diff --git a/backend/packages/harness/pyproject.toml b/backend/packages/harness/pyproject.toml index 39478adcca..03c8e054c3 100644 --- a/backend/packages/harness/pyproject.toml +++ b/backend/packages/harness/pyproject.toml @@ -47,13 +47,11 @@ dependencies = [ "ddgs>=9.10.0", "duckdb>=1.4.4", "langchain-google-genai>=4.2.1", - # Floor matters: the InMemorySaver delta-history patch was removed once - # 4.2.0 fixed the dropped first post-migration write upstream (#8526) and - # kept its own override, so a 4.1.x resolution would silently reintroduce - # the bug. langgraph's own constraint (langgraph-checkpoint>=4.1.0) does - # not exclude that version. - "langgraph-checkpoint>=4.2.0,<5.0", - "langgraph-checkpoint-sqlite>=3.1.1,<3.2", + # Floor matters: 4.2.0 fixed the dropped first post-migration write, and + # 4.3.0 makes same-superstep delta replay preserve live task-path order. + # Older resolutions can silently reintroduce both replay defects. + "langgraph-checkpoint>=4.3.0,<5.0", + "langgraph-checkpoint-sqlite>=3.1.2,<3.2", "langgraph-sdk>=0.1.51", "sqlalchemy[asyncio]>=2.0,<3.0", "aiosqlite>=0.19", @@ -80,9 +78,9 @@ groundroute = [] ollama = ["langchain-ollama>=0.3.0"] postgres = [ "asyncpg>=0.29", - # 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.1 walks past - # them to the thread root instead. - "langgraph-checkpoint-postgres>=3.1.2,<3.2", + # 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.3 fixes the + # paged ancestor-walk cursor (upstream #8556). + "langgraph-checkpoint-postgres>=3.1.3,<3.2", "psycopg[binary]>=3.3.3", "psycopg-pool>=3.3.0", ] diff --git a/backend/tests/test_checkpointer.py b/backend/tests/test_checkpointer.py index 2454cbc5af..abc87b42ec 100644 --- a/backend/tests/test_checkpointer.py +++ b/backend/tests/test_checkpointer.py @@ -208,7 +208,7 @@ class TestHarnessPackaging: assert "postgres" in optional_dependencies assert optional_dependencies["postgres"] == [ "asyncpg>=0.29", - "langgraph-checkpoint-postgres>=3.1.2,<3.2", + "langgraph-checkpoint-postgres>=3.1.3,<3.2", "psycopg[binary]>=3.3.3", "psycopg-pool>=3.3.0", ] diff --git a/backend/tests/test_delta_channel_checkpointers.py b/backend/tests/test_delta_channel_checkpointers.py index c7c0106f2f..f29182d3d1 100644 --- a/backend/tests/test_delta_channel_checkpointers.py +++ b/backend/tests/test_delta_channel_checkpointers.py @@ -16,14 +16,11 @@ Every contract runs against InMemorySaver, AsyncSqliteSaver, and - when ``TEST_POSTGRES_URI`` is set - AsyncPostgresSaver, because each backend implements blob/version handling slightly differently. -Two further tests pin currently-unfixed upstream defects (langgraph #8382 -parallel-superstep replay order, #8448 Postgres paginated delta walk). They -assert the correct contract and trip - skip, naming the issue - while the -defect is present, becoming live gates once a dependency bump lands a fix. The -#8448 trip is scoped to the Postgres parameter, the only backend that pages its -stage-1 scan: on memory/sqlite it stays a live differential assertion against -``InMemorySaver``, so a regression there fails instead of blaming the -Postgres-only upstream issue. +Two further tests pin parallel-superstep replay order and Postgres paginated +delta walks. The dependency floors carry the upstream fixes, so these are live +regression gates rather than defect-triggered skips. The pagination contract +uses Postgres for its paged stage-1 scan and retains a live differential +assertion against ``InMemorySaver`` on the other backends. """ from __future__ import annotations @@ -331,16 +328,12 @@ async def test_non_delta_writers_preserve_delta_messages_and_markers(saver_env: # --------------------------------------------------------------------------- -# Known-unfixed upstream DeltaChannel defects. +# DeltaChannel replay ordering and pagination regression gates. # -# langgraph #8382 (parallel-superstep delta replay order) and #8448 (Postgres -# paginated delta-walk cursor) are both still open against the pinned -# langgraph-checkpoint 4.2.0 / langgraph-checkpoint-postgres 3.1.2, with fixes -# (PR #8544 / PR #8556) unmerged and no released version carrying them. Each -# test below asserts the CORRECT contract; while the defect is present it trips -# (skips, naming the upstream issue) instead of failing, so a later dependency -# bump that lands the fix flips it into a live gate. The contract is never -# asserted as buggy behaviour. +# langgraph-checkpoint 4.3.0 and the matching saver releases preserve live +# task-path write order and keep the Postgres paged ancestor-walk cursor valid. +# The dependency floors exclude older implementations, so any regression must +# fail these tests rather than being converted into a skip. # --------------------------------------------------------------------------- _PARALLEL_LABELS = tuple("abcdefgh") @@ -399,10 +392,7 @@ async def _long_chain_history(saver: Any, thread_id: str, steps: int) -> list[tu @pytest.mark.anyio async def test_long_chain_history_survives_pagination(saver_env: _SaverEnv, monkeypatch: pytest.MonkeyPatch) -> None: """Every checkpoint of a long delta chain must materialize its own ancestor - prefix. Postgres resolves the history through a stage-1 scan paged - newest-first; when the target checkpoint is not on the first page the - unpatched cursor is derived from a not-yet-loaded parent and parks at - ``None`` for good, so old checkpoints hydrate empty (upstream #8448).""" + prefix, including when Postgres has to page beyond the first stage-1 scan.""" if saver_env.kind == "postgres": pg_aio = pytest.importorskip("langgraph.checkpoint.postgres.aio", reason="postgres extra not installed") monkeypatch.setattr(pg_aio, "_DELTA_PAGE_SIZE", _PAGINATION_PAGE_SIZE) @@ -415,8 +405,6 @@ async def test_long_chain_history_survives_pagination(saver_env: _SaverEnv, monk assert oracle[0] == (_expected_long_chain(_PAGINATION_STEPS), ()) assert all(digest for digest, next_key in oracle if next_key != ("__start__",)) - if saver_env.kind == "postgres" and got != oracle: - pytest.skip("upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty") assert got == oracle @@ -438,11 +426,8 @@ def _build_fanout_graph(saver: Any) -> Any: @pytest.mark.anyio async def test_parallel_superstep_replay_matches_live_write_order(saver_env: _SaverEnv) -> None: - """When several parallel tasks write the same delta channel in one - superstep, the live ``apply_writes`` order is path-sorted, but the delta - history replay sorts by the hashed ``(task_id, idx)`` - an unrelated order - (upstream #8382). ``get_state`` on the same thread must report the order the - run actually produced.""" + """Parallel delta writes must replay in the path order used by the live + ``apply_writes`` pass, not in an unrelated hashed ``task_id`` order.""" graph = _build_fanout_graph(saver_env.saver) config = _config(_thread_id()) live = await graph.ainvoke({"messages": []}, config) @@ -455,6 +440,4 @@ async def test_parallel_superstep_replay_matches_live_write_order(saver_env: _Sa replayed = await graph.aget_state(config) replayed_order = [m.content for m in replayed.values["messages"]] - if replayed_order != live_order: - pytest.skip("upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes") assert replayed_order == live_order diff --git a/backend/uv.lock b/backend/uv.lock index 1aa38b02d5..c448d25764 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -1093,9 +1093,9 @@ requires-dist = [ { name = "langfuse", specifier = ">=3.4.1" }, { name = "langgraph", specifier = ">=1.2.9,<1.3" }, { name = "langgraph-api", specifier = ">=0.8.1" }, - { name = "langgraph-checkpoint", specifier = ">=4.2.0,<5.0" }, - { name = "langgraph-checkpoint-postgres", marker = "extra == 'postgres'", specifier = ">=3.1.2,<3.2" }, - { name = "langgraph-checkpoint-sqlite", specifier = ">=3.1.1,<3.2" }, + { name = "langgraph-checkpoint", specifier = ">=4.3.0,<5.0" }, + { name = "langgraph-checkpoint-postgres", marker = "extra == 'postgres'", specifier = ">=3.1.3,<3.2" }, + { name = "langgraph-checkpoint-sqlite", specifier = ">=3.1.2,<3.2" }, { name = "langgraph-cli", specifier = ">=0.4.24" }, { name = "langgraph-runtime-inmem", specifier = ">=0.30.0" }, { name = "langgraph-sdk", specifier = ">=0.1.51" }, diff --git a/frontend/src/content/en/harness/checkpoints/operations.mdx b/frontend/src/content/en/harness/checkpoints/operations.mdx index 5351931775..ba69cd1f54 100644 --- a/frontend/src/content/en/harness/checkpoints/operations.mdx +++ b/frontend/src/content/en/harness/checkpoints/operations.mdx @@ -128,13 +128,13 @@ The dependency floor is what keeps the removed saver bug out, so the floors are | Distribution | Floor | Why | | --- | --- | --- | -| `langgraph-checkpoint` | `>=4.2.0,<5.0` | 4.2.0 fixes the dropped first post-migration write (upstream #8526). A 4.1.x resolution would silently reintroduce it, and `langgraph`'s own `langgraph-checkpoint>=4.1.0` constraint does not exclude that version | -| `langgraph-checkpoint-postgres` | `>=3.1.2,<3.2` | 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.1 walks past them to the thread root | -| `langgraph-checkpoint-sqlite` | `>=3.1.1,<3.2` | Pinned alongside the other savers | +| `langgraph-checkpoint` | `>=4.3.0,<5.0` | 4.2.0 fixes the dropped first post-migration write (upstream #8526); 4.3.0 preserves live task-path order when replaying parallel same-superstep writes (upstream #8544) | +| `langgraph-checkpoint-postgres` | `>=3.1.3,<3.2` | 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.3 fixes the paged ancestor-walk cursor (upstream #8556) | +| `langgraph-checkpoint-sqlite` | `>=3.1.2,<3.2` | Matches the checkpoint replay-order fix | -The regression gate is `backend/tests/test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`: it fails when the 4.2.0 fix is absent, and a full-to-delta migration must keep replaying on the same thread. The pinned environment resolves `langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, and `langgraph-checkpoint-postgres 3.1.2`. +The regression gates in `backend/tests/test_delta_channel_checkpointers.py` cover full-to-delta migration, parallel-write replay order, and paged ancestor walks. The locked environment resolves `langgraph 1.2.14`, `langgraph-checkpoint 4.3.0`, and `langgraph-checkpoint-postgres 3.1.3`. -There is no runtime version check or warning. The only version signal an operator sees is a trigger-skipped test naming the pinned version, for example the `#8448` Postgres skip or the `#8382` replay-order skip. +There is no runtime version check or warning. The dependency floors exclude affected versions, and all three regression contracts fail rather than skip when a defect returns. ## Running the benchmarks diff --git a/frontend/src/content/en/harness/checkpoints/reference.mdx b/frontend/src/content/en/harness/checkpoints/reference.mdx index f12ff25128..0bcb2ac77b 100644 --- a/frontend/src/content/en/harness/checkpoints/reference.mdx +++ b/frontend/src/content/en/harness/checkpoints/reference.mdx @@ -36,7 +36,7 @@ Evidence column values are repo-relative `path:line` references using the aliase | `context_usage` | `backend/app/gateway/context_usage.py` | | `example` | `config.example.yaml` | -The pinned dependency set referenced by the defect table is `langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, and `langgraph-checkpoint-postgres 3.1.2`. +The locked dependency set referenced by the defect table is `langgraph 1.2.14`, `langgraph-checkpoint 4.3.0`, and `langgraph-checkpoint-postgres 3.1.3`. ## Configuration keys @@ -288,7 +288,7 @@ Tests a reader can run or cite as the executable contract. All paths are under ` - `test_delta_channel_state.py::test_production_message_forms_keep_assigned_ids_across_delta_replay` - `test_delta_channel_state.py::test_delta_normalization_compiles_stable_channel_without_mutating_middleware` -`test_long_chain_history_survives_pagination` and `test_parallel_superstep_replay_matches_live_write_order` assert the correct contract and trigger-skip while the pinned dependency still has the defect; they are the gates that flip live on a dependency bump. +`test_long_chain_history_survives_pagination` and `test_parallel_superstep_replay_matches_live_write_order` unconditionally assert the correct replay contracts and fail if either upstream defect returns. ### History cache @@ -351,14 +351,14 @@ Tests a reader can run or cite as the executable contract. All paths are under ` ## Pinned upstream defects -State at the pinned set (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`). Each local test asserts the correct contract and trigger-skips while the defect is present, so a later dependency bump flips it into a live gate; the buggy behaviour is never asserted as correct. +State at the locked set (`langgraph 1.2.14`, `langgraph-checkpoint 4.3.0`, `langgraph-checkpoint-postgres 3.1.3`). Fixed replay defects have unconditional regression gates; defects that remain open retain their documented local handling. | id | Kind / status | Scenario that breaks | Local handling | Pinned version | Evidence | | --- | --- | --- | --- | --- | --- | | `#4380` | DeerFlow issue; patched locally | A replace-style write into an empty Union-typed `BinaryOperatorAggregate` channel persists the `Overwrite` wrapper; the next read crashes with `TypeError: 'Overwrite' object is not subscriptable` | Probe-guarded patch in `checkpoint_patches.py`; patch tests; branch/state-update anchors in `test_threads_router.py` | `langgraph 1.2.9` | `checkpoint_patches:91-107`, `backend/tests/test_checkpoint_patches.py:31-95` | -| `#8526` (fixes upstream `#8384`), plus postgres `#8535` | upstream langgraph PR; merged 2026-08-07; fixed in `langgraph-checkpoint` 4.2.0 | After switching a thread from `BinaryOperatorAggregate` to `DeltaChannel`, the first post-migration write survives the live `invoke` return value but disappears from every subsequent read; on postgres, with no seed the walk runs to the root and replays everything | Local `InMemorySaver` patch deleted; dependency floor `langgraph-checkpoint>=4.2.0,<5.0`; regression gate `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`; a version-guarded saver patch must not be re-added | `langgraph-checkpoint 4.2.0` | `checkpoint_patches:11-17`, `backend/packages/harness/pyproject.toml:49-56`, `CHANGELOG.md:3040-3048` | -| `#8382` | upstream langgraph issue; **open**; fix PR `#8544` unmerged, no released version | Delta replay order diverges from live execution order for parallel-superstep writes: live `invoke` returns path-sorted `[a..h]`, `get_state` replays `[f,b,g,h,a,c,d,e]` (a pure permutation); reproduced on memory, sqlite, and postgres | Trigger-skip naming the issue; `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.2.0` | `test_delta_channel_checkpointers.py:419,440-460` | -| `#8448` | upstream langgraph issue; **open**; fix PR `#8556` unmerged | `PostgresSaver.get_delta_channel_history` permanently poisons the walk cursor when the target checkpoint is not in the first pagination page, so old checkpoints hydrate empty; only Postgres pages its stage-1 scan | Assertion stays live on memory/sqlite (differential against an `InMemorySaver` oracle); Postgres trigger-skips; `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` | `langgraph-checkpoint-postgres 3.1.2` | `test_delta_channel_checkpointers.py:459` | +| `#8526` (fixes upstream `#8384`), plus postgres `#8535` | upstream langgraph PR; merged 2026-08-07; fixed in `langgraph-checkpoint` 4.2.0 | After switching a thread from `BinaryOperatorAggregate` to `DeltaChannel`, the first post-migration write survives the live `invoke` return value but disappears from every subsequent read; on postgres, with no seed the walk runs to the root and replays everything | Local `InMemorySaver` patch deleted; dependency floor `langgraph-checkpoint>=4.3.0,<5.0`; regression gate `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`; a version-guarded saver patch must not be re-added | `langgraph-checkpoint 4.3.0` | `checkpoint_patches:11-17`, `backend/packages/harness/pyproject.toml:50-54`, `CHANGELOG.md:3040-3048` | +| `#8382` | upstream langgraph issue; fixed by PR `#8544` in `langgraph-checkpoint` 4.3.0 | Delta replay order diverges from live execution order for parallel-superstep writes: live `invoke` returns path-sorted `[a..h]`, while affected versions replay a pure permutation such as `[f,b,g,h,a,c,d,e]`; reproduced on memory, sqlite, and postgres | Dependency floor `langgraph-checkpoint>=4.3.0,<5.0`; unconditional regression gate `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.3.0` | `test_delta_channel_checkpointers.py:427-443` | +| `#8448` | upstream langgraph issue; fixed by PR `#8556` in `langgraph-checkpoint-postgres` 3.1.3 | `PostgresSaver.get_delta_channel_history` permanently poisons the walk cursor when the target checkpoint is not in the first pagination page, so old checkpoints hydrate empty; only Postgres pages its stage-1 scan | Dependency floor `langgraph-checkpoint-postgres>=3.1.3,<3.2`; unconditional regression gate `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` against an `InMemorySaver` oracle | `langgraph-checkpoint-postgres 3.1.3` | `test_delta_channel_checkpointers.py:392-408` | | `#8548` | upstream langgraph PR; **open**; fixes upstream `#8443`, with related issue `#8551` still waiting on it | Does not replay an abandoned branch into a `DeltaChannel` fork | While unmerged the local linearization cannot be removed: `_linearize_delta_checkpoint_resume` is retained | `langgraph 1.2.9` | `worker:2267-2279`, `test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head` | | `#4458` | DeerFlow issue; the user-visible symptom of the fork bug | Regenerating in a branched thread resurrected the old assistant message beside the new one after a reload; reproduced on postgres, sqlite, and the in-memory saver, and it also broke branch history when seeded rows shared one run id | Linearization plus per-turn seeded run ids; `test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer` | `langgraph 1.2.9` | `worker:2267-2289`, `test_run_worker_delta_resume.py:204` | | `#4189` item 3 | DeerFlow design discussion | Retention and deletion of checkpoint rows; what may be deleted | `backend/docs/checkpoint-retention-contract.md`, `backend/tests/test_checkpoint_retention_contract.py`, `backend/app/gateway/checkpoint_retention.py`; bounded cleanup runs at durable admission before checkpoint reads, protects explicit selections, and removes only the root trailing duration leaf | — | `checkpoint_retention.py`, `backend/docs/checkpoint-retention-contract.md:31-77` | diff --git a/frontend/src/content/en/harness/checkpoints/troubleshooting.mdx b/frontend/src/content/en/harness/checkpoints/troubleshooting.mdx index 148b3cd11c..6766357a8a 100644 --- a/frontend/src/content/en/harness/checkpoints/troubleshooting.mdx +++ b/frontend/src/content/en/harness/checkpoints/troubleshooting.mdx @@ -149,29 +149,17 @@ The upstream fix is PR [#8548](https://github.com/langchain-ai/langgraph/pull/85 **Symptom**: on a parallel fan-out that writes one delta channel in a single super-step, the live run returns writes in path order but `get_state` / resume replays them in a different order. -**Cause**: pinned upstream defect ([langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)), open, fix PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) unmerged, no released version. Delta replay orders writes by hashed `(task_id, idx)` instead of the path order `apply_writes` used live, so same-superstep writes are permuted. Present in `langgraph-checkpoint 4.2.0`; reproduced on memory, SQLite, and Postgres. +**Cause**: upstream defect [langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382), fixed by PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) in `langgraph-checkpoint` 4.3.0. Affected versions order delta replay writes by hashed `(task_id, idx)` instead of the path order `apply_writes` uses live, permuting same-superstep writes on memory, SQLite, and Postgres. -**Fix**: none locally. DeerFlow's test asserts the correct contract and trigger-skips while the defect is present, naming the issue: - -``` -upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes -``` - -If ordering within one super-step matters to your consumer, avoid fanning parallel writes into a single delta channel. +**Fix**: use `langgraph-checkpoint>=4.3.0,<5.0`. DeerFlow's dependency floor excludes affected versions, and `test_parallel_superstep_replay_matches_live_write_order` unconditionally asserts that replay matches live write order. ### Postgres hydrates old checkpoints empty **Symptom**: on Postgres, an older delta checkpoint materializes empty while a recent one is fine. -**Cause**: pinned upstream defect ([langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)), open, fix PR #8556 unmerged. `PostgresSaver.get_delta_channel_history` permanently poisons its walk cursor when the target checkpoint is not in the first pagination page: the unpatched cursor is derived from a not-yet-loaded parent and parks at `None` for good, so old checkpoints hydrate empty. Only Postgres pages its stage-1 scan. Present in `langgraph-checkpoint-postgres 3.1.2`. +**Cause**: upstream defect [langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448), fixed by PR #8556 in `langgraph-checkpoint-postgres` 3.1.3. In affected versions, `PostgresSaver.get_delta_channel_history` derives its walk cursor from a not-yet-loaded parent when the target checkpoint is beyond the first pagination page; the cursor then remains `None`, so old checkpoints hydrate empty. -**Fix**: none locally; the Postgres case trigger-skips, naming the issue: - -``` -upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty -``` - -Cadence bounds replay depth, so a smaller `snapshot_frequency` keeps the walk shorter; that is the only lever this repository exposes, not a fix for the cursor bug. +**Fix**: use `langgraph-checkpoint-postgres>=3.1.3,<3.2`. DeerFlow's dependency floor excludes affected versions, and `test_long_chain_history_survives_pagination` unconditionally checks every materialized checkpoint against an `InMemorySaver` oracle. ## Cache diff --git a/frontend/src/content/zh/harness/checkpoints/operations.mdx b/frontend/src/content/zh/harness/checkpoints/operations.mdx index 68034c7375..b6825169c0 100644 --- a/frontend/src/content/zh/harness/checkpoints/operations.mdx +++ b/frontend/src/content/zh/harness/checkpoints/operations.mdx @@ -125,13 +125,13 @@ run 获得持久化的同线程 admission 后,Gateway 会在首次 checkpoint | 发行包 | 下限 | 原因 | | --- | --- | --- | -| `langgraph-checkpoint` | `>=4.2.0,<5.0` | 4.2.0 修掉了迁移后首个写入丢失(上游 #8526)。解析到 4.1.x 会静默重新引入该缺陷,而 `langgraph` 自己的 `langgraph-checkpoint>=4.1.0` 约束并不排除该版本 | -| `langgraph-checkpoint-postgres` | `>=3.1.2,<3.2` | 3.1.2 能找到纯值形式的增量种子(上游 #8535);3.1.1 会越过它一路走到线程根 | -| `langgraph-checkpoint-sqlite` | `>=3.1.1,<3.2` | 与其他 saver 一同锁定 | +| `langgraph-checkpoint` | `>=4.3.0,<5.0` | 4.2.0 修掉了迁移后首个写入丢失(上游 #8526);4.3.0 让并行 super-step 写入在重放时保持实时任务路径顺序(上游 #8544) | +| `langgraph-checkpoint-postgres` | `>=3.1.3,<3.2` | 3.1.2 能找到纯值形式的增量种子(上游 #8535);3.1.3 修复分页祖先回走的游标(上游 #8556) | +| `langgraph-checkpoint-sqlite` | `>=3.1.2,<3.2` | 与 checkpoint 的回放顺序修复配套 | -回归门禁是 `backend/tests/test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`:当 4.2.0 的修复缺失时它会失败,保证 full→delta 迁移仍能在同一线程上回放。锁定环境解析到 `langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`。 +`backend/tests/test_delta_channel_checkpointers.py` 中的回归门禁覆盖 full→delta 迁移、并行写入回放顺序和分页祖先回走。锁定环境解析到 `langgraph 1.2.14`、`langgraph-checkpoint 4.3.0`、`langgraph-checkpoint-postgres 3.1.3`。 -运行时没有版本检查,也没有告警。运维能看到的唯一版本信号,是一条带有锁定版本号的触发式跳过测试,例如 `#8448` 的 Postgres 跳过或 `#8382` 的回放顺序跳过。 +运行时没有版本检查,也没有告警。依赖下限会排除受影响的版本;这三个回归契约在缺陷复现时都会直接失败,不再跳过。 ## 如何跑基准 diff --git a/frontend/src/content/zh/harness/checkpoints/reference.mdx b/frontend/src/content/zh/harness/checkpoints/reference.mdx index dfd1a1d4b7..98e50bac04 100644 --- a/frontend/src/content/zh/harness/checkpoints/reference.mdx +++ b/frontend/src/content/zh/harness/checkpoints/reference.mdx @@ -36,7 +36,7 @@ import { Callout } from "nextra/components"; | `context_usage` | `backend/app/gateway/context_usage.py` | | `example` | `config.example.yaml` | -缺陷表引用的固定依赖集为 `langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`。 +缺陷表引用的锁定依赖集为 `langgraph 1.2.14`、`langgraph-checkpoint 4.3.0`、`langgraph-checkpoint-postgres 3.1.3`。 ## 配置键 @@ -288,7 +288,7 @@ wait 响应仍为 HTTP `200`。Python SDK 默认使用 `raise_error=True`,识 - `test_delta_channel_state.py::test_production_message_forms_keep_assigned_ids_across_delta_replay` - `test_delta_channel_state.py::test_delta_normalization_compiles_stable_channel_without_mutating_middleware` -`test_long_chain_history_survives_pagination` 与 `test_parallel_superstep_replay_matches_live_write_order` 断言的是正确契约,在固定依赖仍带缺陷时触发 skip;依赖升级后它们会成为实际门禁。 +`test_long_chain_history_survives_pagination` 与 `test_parallel_superstep_replay_matches_live_write_order` 无条件断言正确的回放契约;任一上游缺陷复现时都会直接失败。 ### 历史缓存 @@ -351,14 +351,14 @@ wait 响应仍为 HTTP `200`。Python SDK 默认使用 `raise_error=True`,识 ## 上游缺陷固定表 -状态对应固定依赖集(`langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`)。每条本地测试断言的都是正确契约,缺陷存在期间触发 skip,因此后续依赖升级会把它变成实际门禁;有缺陷的行为从不被当作正确行为断言。 +状态对应锁定依赖集(`langgraph 1.2.14`、`langgraph-checkpoint 4.3.0`、`langgraph-checkpoint-postgres 3.1.3`)。已修复的回放缺陷都有无条件回归门禁;仍未修复的缺陷保留文档所述的本地处理。 | 编号 | 种类 / 状态 | 触发场景 | 本地应对 | 固定版本 | 证据 | | --- | --- | --- | --- | --- | --- | | `#4380` | DeerFlow issue;已在本地打补丁 | 向空的 Union 类型 `BinaryOperatorAggregate` 通道做替换式写入会把 `Overwrite` 包装器持久化;下一次读取以 `TypeError: 'Overwrite' object is not subscriptable` 崩溃 | `checkpoint_patches.py` 中的探针守卫补丁;补丁测试;`test_threads_router.py` 中的分支/状态更新锚点 | `langgraph 1.2.9` | `checkpoint_patches:91-107`、`backend/tests/test_checkpoint_patches.py:31-95` | -| `#8526`(修复上游 `#8384`),以及 postgres `#8535` | 上游 langgraph PR;2026-08-07 合并;`langgraph-checkpoint` 4.2.0 修复 | 线程从 `BinaryOperatorAggregate` 切换到 `DeltaChannel` 后,迁移后的第一次写入能从实时 `invoke` 返回值中看到,却从之后每次读取中消失;postgres 侧在没有 seed 时走到根并重放全部内容 | 删除本地 `InMemorySaver` 补丁;依赖下限 `langgraph-checkpoint>=4.2.0,<5.0`;回归门禁 `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`;禁止再引入版本守卫式 saver 补丁 | `langgraph-checkpoint 4.2.0` | `checkpoint_patches:11-17`、`backend/packages/harness/pyproject.toml:49-56`、`CHANGELOG.md:3040-3048` | -| `#8382` | 上游 langgraph issue;**open**;修复 PR `#8544` 未合并,无发布版本 | 并行 super-step 写入的 delta 重放顺序与实时执行顺序不一致:实时 `invoke` 返回按路径排序的 `[a..h]`,`get_state` 重放为 `[f,b,g,h,a,c,d,e]`(纯置换);在 memory、sqlite、postgres 上均可复现 | 触发 skip 并点名该 issue;`test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.2.0` | `test_delta_channel_checkpointers.py:419,440-460` | -| `#8448` | 上游 langgraph issue;**open**;修复 PR `#8556` 未合并 | 目标检查点不在第一页分页结果中时,`PostgresSaver.get_delta_channel_history` 会永久污染游标,导致老检查点物化为空;只有 Postgres 会对 stage-1 扫描分页 | memory/sqlite 上断言保持生效(与 `InMemorySaver` 预言机做差分);Postgres 触发 skip;`test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` | `langgraph-checkpoint-postgres 3.1.2` | `test_delta_channel_checkpointers.py:459` | +| `#8526`(修复上游 `#8384`),以及 postgres `#8535` | 上游 langgraph PR;2026-08-07 合并;`langgraph-checkpoint` 4.2.0 修复 | 线程从 `BinaryOperatorAggregate` 切换到 `DeltaChannel` 后,迁移后的第一次写入能从实时 `invoke` 返回值中看到,却从之后每次读取中消失;postgres 侧在没有 seed 时走到根并重放全部内容 | 删除本地 `InMemorySaver` 补丁;依赖下限 `langgraph-checkpoint>=4.3.0,<5.0`;回归门禁 `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`;禁止再引入版本守卫式 saver 补丁 | `langgraph-checkpoint 4.3.0` | `checkpoint_patches:11-17`、`backend/packages/harness/pyproject.toml:50-54`、`CHANGELOG.md:3040-3048` | +| `#8382` | 上游 langgraph issue;已由 PR `#8544` 在 `langgraph-checkpoint` 4.3.0 中修复 | 并行 super-step 写入的 delta 重放顺序与实时执行顺序不一致:实时 `invoke` 返回按路径排序的 `[a..h]`,受影响版本则会重放为 `[f,b,g,h,a,c,d,e]` 一类的纯置换;在 memory、sqlite、postgres 上均可复现 | 依赖下限 `langgraph-checkpoint>=4.3.0,<5.0`;无条件回归门禁 `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.3.0` | `test_delta_channel_checkpointers.py:427-443` | +| `#8448` | 上游 langgraph issue;已由 PR `#8556` 在 `langgraph-checkpoint-postgres` 3.1.3 中修复 | 目标检查点不在第一页分页结果中时,`PostgresSaver.get_delta_channel_history` 会永久污染游标,导致老检查点物化为空;只有 Postgres 会对 stage-1 扫描分页 | 依赖下限 `langgraph-checkpoint-postgres>=3.1.3,<3.2`;无条件回归门禁 `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination`,与 `InMemorySaver` 预言机对照 | `langgraph-checkpoint-postgres 3.1.3` | `test_delta_channel_checkpointers.py:392-408` | | `#8548` | 上游 langgraph PR;**open**;修复上游 `#8443`,相关 issue `#8551` 仍在等它 | 不把被放弃的分支重放进 `DeltaChannel` 分叉 | 未合并期间本地线性化不能移除:保留 `_linearize_delta_checkpoint_resume` | `langgraph 1.2.9` | `worker:2267-2279`、`test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head` | | `#4458` | DeerFlow issue;分叉缺陷对用户可见的症状 | 在分叉线程中重新生成后,重载时旧助手消息会与新回答并排复活;在 postgres、sqlite 与内存 saver 上均可复现,且当种子行共用一个 run id 时还会破坏分支历史 | 线性化加按轮次播种的 run id;`test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer` | `langgraph 1.2.9` | `worker:2267-2289`、`test_run_worker_delta_resume.py:204` | | `#4189` 第 3 项 | DeerFlow 设计讨论 | 检查点行的保留与删除;哪些内容可以被删除 | `backend/docs/checkpoint-retention-contract.md`、`backend/tests/test_checkpoint_retention_contract.py`、`backend/app/gateway/checkpoint_retention.py`;目前没有接入生产触发点 | — | `checkpoint_retention.py:1-40`、`backend/docs/checkpoint-retention-contract.md:31-77` | diff --git a/frontend/src/content/zh/harness/checkpoints/troubleshooting.mdx b/frontend/src/content/zh/harness/checkpoints/troubleshooting.mdx index 8b1ee915b2..f872439e4f 100644 --- a/frontend/src/content/zh/harness/checkpoints/troubleshooting.mdx +++ b/frontend/src/content/zh/harness/checkpoints/troubleshooting.mdx @@ -147,29 +147,17 @@ Run {run_id} could not materialize resume checkpoint {checkpoint_id} **症状**:在并行扇出向同一 delta 通道写入的同一个 super-step 中,实时 run 按路径顺序返回写入,但 `get_state` / 恢复以不同顺序回放。 -**原因**:已固定的上游缺陷([langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)),open,修复 PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) 未合并,无已发布版本。delta 回放按哈希后的 `(task_id, idx)` 排序写入,而不是实时 `apply_writes` 使用的路径顺序,于是同一 super-step 的写入被重排。存在于 `langgraph-checkpoint 4.2.0`;已在 memory、SQLite 与 Postgres 上复现。 +**原因**:上游缺陷 [langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382),已由 PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) 在 `langgraph-checkpoint` 4.3.0 中修复。受影响的版本按哈希后的 `(task_id, idx)` 排序 delta 回放写入,而不是实时 `apply_writes` 使用的路径顺序,导致 memory、SQLite 与 Postgres 上同一 super-step 的写入被重排。 -**处理**:本地无解。DeerFlow 的测试断言正确契约,并在缺陷存在时触发 skip,同时点名该 issue: - -``` -upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes -``` - -如果你的消费方在意同一 super-step 内的顺序,请避免把并行写入扇出到同一个 delta 通道。 +**处理**:使用 `langgraph-checkpoint>=4.3.0,<5.0`。DeerFlow 的依赖下限会排除受影响版本,`test_parallel_superstep_replay_matches_live_write_order` 无条件断言回放顺序与实时写入顺序一致。 ### Postgres 物化旧检查点为空 **症状**:在 Postgres 上,较旧的 delta 检查点物化为空,较新的却正常。 -**原因**:已固定的上游缺陷([langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)),open,修复 PR #8556 未合并。当目标检查点不在第一页分页结果里时,`PostgresSaver.get_delta_channel_history` 会永久污染其回走游标:未修补的游标取自一个尚未加载的父级,永久停在 `None`,于是旧检查点物化为空。只有 Postgres 对其 stage-1 扫描分页。存在于 `langgraph-checkpoint-postgres 3.1.2`。 +**原因**:上游缺陷 [langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448),已由 PR #8556 在 `langgraph-checkpoint-postgres` 3.1.3 中修复。受影响的版本在目标检查点超出第一页分页结果时,从尚未加载的父级推导回走游标;游标随后一直停在 `None`,导致旧检查点物化为空。 -**处理**:本地无解;Postgres 用例触发 skip,并点名该 issue: - -``` -upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty -``` - -节奏约束回放深度,因此更小的 `snapshot_frequency` 能让回走更短;这是本仓库暴露的唯一杠杆,而不是对该游标缺陷的修复。 +**处理**:使用 `langgraph-checkpoint-postgres>=3.1.3,<3.2`。DeerFlow 的依赖下限会排除受影响版本,`test_long_chain_history_survives_pagination` 无条件将每个物化检查点与 `InMemorySaver` 预言机对照。 ## 缓存