docs(checkpoints): add a ten-chapter checkpoint storage manual in both languages (#5845)

* docs(checkpoints): add a bilingual checkpoint storage manual

Add the missing docs-site coverage for dual-mode checkpoint storage (`database.checkpoint_channel_mode: full | delta`) as a ten-chapter manual under `harness/checkpoints/` in both languages: concepts, quick start, channel modes, snapshot cadence, history cache, resume and rollback, operating checkpoints, observability, troubleshooting by symptom, and a reference of every configuration key, error message, HTTP status mapping, public symbol, load-bearing test, and pinned upstream defect.

The index page is `asIndexPage`, so `/docs/harness/checkpoints` is the section itself and its `_meta.ts` lists only the nine child chapters. Discovery: `harness/_meta.ts` registers the section, the harness configuration module table gains a `database` row, and the thread persistence section of `agents-and-threads` links to the manual.

Content is grounded in the code. Only the three measured values recorded in `backend/docs/runtime-guidance-details.md` are quoted; the upstream-derived 41x storage figure in the untracked delta-channel RFC is deliberately not cited. Pinned upstream defects are documented with their issue and PR numbers and the versions they are pinned at (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`).

* docs(checkpoints): correct the history-cache restart requirement

The history cache is built with the checkpointer at Gateway startup (`langgraph_runtime()` -> `make_checkpointer()` -> `make_checkpoint_cache()`), so editing `database.checkpoint_cache.*` does not change a running process. The manual previously marked the whole section "not restart-required", which conflated "not frozen for correctness" with "hot-reloadable" and could mislead an operator changing capacity, credentials, or retention.

Distinguish the two properties everywhere the section is described: history-cache.mdx marks every field as needing a restart and adds a callout explaining the difference, reference.mdx redefines the Restart column and fixes the five rows and the effect list (adding the `asyncprov` evidence alias), troubleshooting.mdx replaces the "two keys need no restart" claim and gains a dedicated symptom entry, and observability.mdx notes that changed counters need a restart. The Chinese text is corrected in parallel.

Verified: the docs link test and the docs unit suite pass, frontmatter parses for all 20 pages, and every in-page anchor resolves on the rendered English and Chinese history-cache, troubleshooting, reference, and observability pages.

* docs(checkpoints): account for periodic snapshots in the storage-growth claim

Every periodic snapshot stores the entire message list as of that step (`_DeltaSnapshot(ch.get())`, persisted as a checkpoint blob), so with a fixed snapshot frequency K the retained total after N turns is O(N²/K + N), not O(N). The manual claimed unconditional linear growth, which understated capacity planning — the more so because nothing in DeerFlow prunes those snapshots yet.

Distinguish per-step delta writes from total retained storage in index.mdx (prose and table), modes.mdx (step-cost row), and operations.mdx; note in cadence.mdx that K is also the capacity knob that scales the snapshot bytes. The English and Chinese chapters are corrected in parallel. The reference chapter keeps the released #4292 wording as a verbatim quotation and now names the retained-storage bound beside it.

Verified: `_DeltaSnapshot(ch.get())` and its blob serialization in the pinned langgraph (channels/delta.py:196-198, checkpoint/serde/types.py:19-25) confirm the premise; the docs link test and docs unit suite pass; all ten changed pages render with resolving anchors and no stale linear-growth wording.

* docs(checkpoints): correct the run-error surfaces and repair two evidence pointers

Review follow-ups on the checkpoint manual, both languages.

Error-surface corrections:
- Run creation is not always asynchronous: when a thread's run-event feed is
  empty but a checkpoint head exists, ensure_checkpoint_history_seeded reads
  that head inside start_run, where only ConflictError (409) and
  UnsupportedStrategyError (501) are mapped, so a mode mismatch returns a
  plain 500 before admission. Troubleshooting, observability and the
  reference route tables now scope the "does not pre-gate" claim.
- The 503 symptom is retargeted. Only Gateway startup, make_lead_agent and
  DeerFlowClient.__init__ call the freeze helpers, so a config.yaml edit
  cannot make a state/history/thread route answer 503. A mode edit is
  injected over by the run worker and ignored until a restart; a cadence edit
  fails the next run with the restart-required message. The router's 503
  mapping is marked defensive in modes and the reference.
- The regenerate section title no longer groups the state endpoints, which
  map a mismatch to 409.
- operations: the Textual TUI catches the mismatch through stream_actions();
  only the headless --print/--json runners print a traceback.

Evidence-pointer repairs:
- reference: this branch's CHANGELOG insertion shifted the #5734 entry from
  3032-3039 to 3040-3048; the citation is re-pointed.
- reference: the #8548 row cited an untracked worktree plan file; replaced by
  the in-repo test anchor, and "mirror issue #8551" reworded to "related".
- reference: new `agent` alias for the agent-factory freeze sites.
- history-cache: the bare `deps:`/`asyncprov:` aliases are expanded to full
  paths; the alias legend lives only on the reference page.

* docs(checkpoints): say a delta fork is accepted, not refused

"Delta mode cannot fork" overstated the constraint. Nothing in the stack
rejects the fork: LangGraph stores the second child, no saver checks for an
existing sibling, and ensure_checkpoint_mode_compatible returns immediately
when the process mode is delta. What breaks is the read — the walk replays
the abandoned sibling's writes stored under the shared parent.

Reproduced on the pinned stack (langgraph 1.2.9 / langgraph-checkpoint
4.2.0, AsyncSqliteSaver) by running this repo's own #4458 fixture without
the linearization call: the forked resume returns
["h1", "a1", "h2", "a2", "a2-new"] instead of ["h1", "a1", "h2", "a2-new"],
i.e. the superseded answer comes back.

- resume-and-rollback: title/description/heading now say "cannot fork
  correctly"; new "Forks are accepted, not refused" subsection covering the
  unguarded write side.
- troubleshooting: the "old answer reappears" cause states the same split.
- reference: the POST /threads/{id}/state row notes that a client-supplied
  checkpoint_id older than the head is applied as a delta fork with no
  linearization — the write shape upstream tracks as #8551 (open), which the
  run-path linearization does not cover.

* docs(checkpoints): scope the unguarded state-update fork to what the probe showed

The previous commit said a mutation against an older checkpoint stores its
writes under the shared parent and implied the sibling branch suffers for it.
Probing the pinned stack (AsyncSqliteSaver, delta messages channel) shows:

- the write does land under the shared parent and joins the write set the
  parent's other children walk (raw walk for a sibling includes the abandoned
  head's own write);
- but reducer channels are written with Overwrite, which replaces the
  accumulated value, so the forked branch reads clean and the abandoned
  branch is unchanged in that shape — #8551 did not reproduce;

state the masking mechanism and the upstream tracking issue, not a corruption
claim.

* docs(checkpoints): fix a frontmatter description that broke the docs build

`resume-and-rollback.mdx` shipped `description: Delta mode cannot fork a
thread correctly: a forked delta read ...` in the previous commit. The
unquoted ": " turns the value into a nested mapping, so Nextra's YAML
frontmatter parse failed and the e2e lane could not start its web server
(YAMLParseError at line 2, column 14) — the exact failure class this manual
warns docs authors about, introduced by me two commits ago.

- Replace the colon with an em dash in the English description (the Chinese
  one uses a full-width colon and was unaffected).
- Add tests/unit/content/docs-frontmatter.test.ts: every `src/content/**/*.mdx`
  frontmatter block must be YAML-parseable, implemented as the one rule that
  actually breaks it (an unquoted scalar containing ": "), because the unit
  suite never compiles MDX and only a build or a dev render catches it.
  Verified the guard fails on the broken line and passes on the fix.

Verified with a clean `BETTER_AUTH_SECRET=local-dev-secret pnpm build` on this
tree: all pages prerender, resume-and-rollback EN/ZH included.

CHANGELOG.md

@@ -3026,6 +3026,14 @@ This release closes that milestone with **765 merged pull requests**.
   with a per-user route example; and plugin troubleshooting and operations
   notes. Also correct the plugin `mount` return value in
   `docs/full-stack-plugins.md`.
+- **docs:** Add a checkpoint storage manual under `harness/checkpoints/` in both
+  languages, covering the `full` and `delta` channel modes: concepts, quick
+  start, the mode marker and its fail-closed gate, the snapshot cadence, the
+  delta history cache, resume and rollback linearization, operating and
+  retention constraints, observability, troubleshooting by symptom, and a
+  reference of every configuration key, error message, and pinned upstream
+  defect. The section is a new directory rather than a restructured page, so
+  the existing checkpointer documentation is unchanged.
 
 ### Internal
 

CHANGELOG_zh.md

@@ -2246,6 +2246,11 @@
   插件章节，涵盖 `registry.plugin()`、浏览器模块与打包资源、后端动作、模型工具和
   设置；新增请求级运行证据读取器及按用户的路由示例；补充插件的排查与运维说明。同时
   修正 `docs/full-stack-plugins.md` 中插件 `mount` 返回值的描述。
+- **文档：** 新增中英文检查点存储手册（`harness/checkpoints/`），覆盖 `full` 与
+  `delta` 两种通道模式：概念、快速上手、模式标记与失败关闭门禁、快照节奏、delta
+  历史缓存、恢复与回滚的线性化、运维与保留约束、可观测性、按症状排查，以及列出全部
+  配置键、错误信息和已固定上游缺陷的参考章节。该章节为新增目录而非页面重构，原有
+  检查点相关文档保持不变。
 
 ### 内部改进
 

frontend/src/content/en/application/agents-and-threads.mdx

@@ -118,6 +118,11 @@ database:
   <code>database</code> for new deployments.
 </Callout>
 
+How thread state is represented in that backend — full-value checkpoints or
+LangGraph's `DeltaChannel` — is a separate switch,
+`database.checkpoint_channel_mode`. See
+[Checkpoints and Delta Channels](/docs/harness/checkpoints).
+
 ### Thread data storage
 
 Working files produced during a thread (uploads, intermediate files, output artifacts) are stored under:

frontend/src/content/en/harness/_meta.ts

@@ -34,6 +34,9 @@ const meta: MetaRecord = {
   subagents: {
     title: "Subagents",
   },
+  checkpoints: {
+    title: "Checkpoints",
+  },
   mcp: {
     title: "MCP Integration",
   },

frontend/src/content/en/harness/checkpoints/_meta.ts

@@ -0,0 +1,33 @@
+import type { MetaRecord } from "nextra";
+
+const meta: MetaRecord = {
+  "quick-start": {
+    title: "Quick Start",
+  },
+  modes: {
+    title: "Channel Modes",
+  },
+  cadence: {
+    title: "Snapshot Cadence",
+  },
+  "history-cache": {
+    title: "History Cache",
+  },
+  "resume-and-rollback": {
+    title: "Resume and Rollback",
+  },
+  operations: {
+    title: "Operating Checkpoints",
+  },
+  observability: {
+    title: "Observability",
+  },
+  troubleshooting: {
+    title: "Troubleshooting",
+  },
+  reference: {
+    title: "Reference",
+  },
+};
+
+export default meta;

frontend/src/content/en/harness/checkpoints/cadence.mdx

@@ -0,0 +1,112 @@
+---
+title: Snapshot Cadence
+description: How database.checkpoint_delta.snapshot_frequency trades checkpoint size against materialization cost, its interaction with LangGraph's system-wide superstep bound, why it is frozen with the mode and never stamped into checkpoint metadata, and the 1000 to 10 default change with its legacy-key warnings.
+---
+
+import { Callout } from "nextra/components";
+
+# Snapshot Cadence
+
+In `delta` mode a checkpoint stores writes, and the message list is reconstructed on read. A full messages snapshot is written out every so often so reconstruction does not have to walk the ancestor chain forever. `database.checkpoint_delta.snapshot_frequency` sets that interval.
+
+<Callout type="info">
+  This knob only applies in `delta` mode. `checkpoint_delta` is ignored when
+  `checkpoint_channel_mode` is `full`; see [Channel Modes](/docs/harness/checkpoints/modes).
+</Callout>
+
+## The knob
+
+| Key                                       | Type  | Default | Range                        | Restart required |
+| ----------------------------------------- | ----- | ------- | ---------------------------- | ---------------- |
+| `database.checkpoint_delta.snapshot_frequency` | `int` | `10`    | `ge=1` (1 or more, no upper bound) | Yes        |
+
+A full messages snapshot is stored every N per-step writes (higher = smaller checkpoints, slower materialization). The value reaches the compiled channel table: the messages field is built as `DeltaChannel(merge_message_writes, snapshot_frequency=…)`, so the cadence is a property of the compiled graph rather than of an individual checkpoint.
+
+```yaml
+database:
+  checkpoint_channel_mode: delta   # snapshot_frequency only means anything here
+  checkpoint_delta:
+    snapshot_frequency: 10         # every 10 per-step writes, one _DeltaSnapshot blob
+```
+
+## The tradeoff
+
+| Cadence        | Checkpoint size                                              | Materialization                                  | Materialized state |
+| -------------- | ------------------------------------------------------------ | ------------------------------------------------ | ------------------ |
+| Low N (for example 1) | A snapshot blob on every step; the storage win shrinks toward `full` mode's size | Short ancestor walk; fastest reads             | Identical          |
+| Default `10`   | One snapshot blob per 10 per-step writes                     | Moderate walk                                    | Identical          |
+| High N         | Fewest blobs, smallest store                                 | Longest walk; slowest reads                      | Identical          |
+
+The knob buys storage against read cost, never a different answer. A low cadence snapshots the message list far more often than the default while materialized state stays identical, and the tests pin both halves: the configured frequency must reach the compiled channel table, and materialization must produce the same values at any cadence.
+
+It is also a first-order capacity knob, because those snapshots are what dominate the retained total: with K = `snapshot_frequency`, N turns retain O(N²/K + N) rather than O(N²), while the per-step write payloads are independent of K. Sizing a store from the per-step cost alone understates it; see [Checkpoints and Delta Channels](/docs/harness/checkpoints) for the derivation.
+
+## Interaction with LangGraph's superstep bound
+
+Snapshot writing is driven by two counters, and whichever trips first wins:
+
+| Counter                              | Scope        | Default    | Set by                          |
+| ------------------------------------ | ------------ | ---------- | ------------------------------- |
+| Per-channel update count             | This channel | `snapshot_frequency` (`10`) | `database.checkpoint_delta.snapshot_frequency` |
+| Supersteps since the last snapshot   | System-wide  | `5000`     | LangGraph's `DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT` (env `LANGGRAPH_DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT`) |
+
+The second counter bounds replay depth even for a channel that stops receiving writes. Raising `snapshot_frequency` therefore cannot make replay depth unbounded: a thread that keeps running without touching messages still snapshots under the 5000-superstep bound. That bound belongs to LangGraph, not to the `database:` section, and applies to every delta channel in the process.
+
+## Frozen with the mode, never stored in a checkpoint
+
+The snapshot cadence is baked into each compiled graph's channel table, not stored in the checkpoint. That has direct operational consequences:
+
+| Fact | Consequence |
+| --- | --- |
+| Resolution order is explicit argument, then process-frozen value, then the config default `10` | Explicit values exist for tests and ephemeral graphs; the assembly paths pass the process value, so a running deployment compiles at the frozen cadence |
+| Frozen on the first graph build, next to the mode | All processes sharing one checkpoint database must use the same value; there is no per-checkpoint cadence to compare against |
+| A later different value raises `CheckpointModeReconfigurationError("checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process")` | Editing `config.yaml` does not take effect until restart; the failure is loud, not silent |
+| A non-positive value raises `ValueError("snapshot frequency must be positive")` | The freeze helper and config validation (`ge=1`) agree |
+| The value is read from the application config, not from a per-request configurable key | A forged or client-supplied value cannot recompile the channel table |
+
+The cadence is deliberately **not** stamped into checkpoint metadata: the mode marker contract (absence = full) and the full to delta migration semantics are unchanged by the frequency value. Because it is absent from every checkpoint, a process running a different cadence applies it silently to the same threads — which is why the value is an operator responsibility documented on the config field, not something the runtime can detect.
+
+Contrast with the mode marker, which *is* written: delta checkpoints carry `deerflow_checkpoint_channel_mode: "delta"` in their metadata, and absence means `full`. Cadence has no equivalent.
+
+## Storage-shape contract
+
+| Checkpoint kind     | `channel_values.messages`                  | Where the step's payload lives                  |
+| ------------------- | ------------------------------------------ | ----------------------------------------------- |
+| Non-snapshot step   | Absent — no `messages` key at all          | In the checkpoint's per-step writes             |
+| Snapshot step       | A `_DeltaSnapshot` blob holding the list   | In the checkpoint itself; the ancestor walk stops here |
+| `full` mode         | The whole message list in every checkpoint | Nowhere to replay                               |
+
+The contract is pinned two ways: non-snapshot checkpoints must not carry `messages` in `channel_values`, every `snapshot_frequency` updates instead carry a `_DeltaSnapshot` blob, and materialization stays correct across both shapes; and the storage-level guard compares the two modes over the same turns: delta carries at most the periodic snapshot blob, while `full` writes one per message-writing step. A snapshot is therefore not an extra copy — in delta mode it is the only full message payload that exists.
+
+## Default change: 1000 to 10
+
+The cadence used to live at the flat key `database.checkpoint_delta_snapshot_frequency` with a default of `1000`. It moved to the nested key `database.checkpoint_delta.snapshot_frequency` with a default of `10` — the same field, but threads now snapshot **100x more often** by default. Upstream `DeltaChannel` still defaults its parameter to 1000; DeerFlow always passes an explicit value, so no DeerFlow thread uses upstream's default. The previous cadence is still available by setting the value explicitly:
+
+```yaml
+database:
+  checkpoint_delta:
+    snapshot_frequency: 1000   # the pre-rename default cadence
+```
+
+Three migration warnings can appear while the old flat key is still present. All three are logged, not errors, and the nested key always wins:
+
+| Condition | Warning (verbatim) |
+| --- | --- |
+| Both keys set | `Both database.checkpoint_delta_snapshot_frequency (deprecated) and database.checkpoint_delta.snapshot_frequency are set; the nested key wins.` |
+| `checkpoint_delta` is already a config object | `Ignoring deprecated database.checkpoint_delta_snapshot_frequency because database.checkpoint_delta is already set.` |
+| Legacy value carried forward | `database.checkpoint_delta_snapshot_frequency is deprecated; use database.checkpoint_delta.snapshot_frequency instead. Carried the legacy value (%r) forward.` |
+
+`DatabaseConfig` ignores unknown keys, so without this shim a `config.yaml` still written against the old flat key would fall back to `10` instead of the operator's chosen value. Deleting the flat key once the nested one is set removes all three warnings.
+
+## Choosing a value
+
+The default is not a rule of thumb: the production benchmark emits `snapshot_write_spike` (delta checkpoint-write p99 over p50) and `cache_effect_ms` (state cold minus warm p50) as the decision inputs for the production snapshot-frequency and accessor-cache defaults. The channel benchmark reports backend-neutral row and byte counts plus warm and cold read times, with ratios always `delta/full`; timing thresholds are not CI gates. See [Operating Checkpoints](/docs/harness/checkpoints/operations) for how to run them.
+
+Because the value is frozen and must match across processes, change it deliberately: pick a cadence, restart every process sharing the database, and verify with the storage-shape check rather than watching a single request.
+
+## Related
+
+- [Channel Modes](/docs/harness/checkpoints/modes) — what the two storage shapes mean for reads and writes.
+- [History Cache](/docs/harness/checkpoints/history-cache) — the read path this cadence feeds.
+- [Operations](/docs/harness/checkpoints/operations) — retention, benchmarks, and backend choice.
+- [Reference](/docs/harness/checkpoints/reference) — the full config-key and error table.

frontend/src/content/en/harness/checkpoints/history-cache.mdx

@@ -0,0 +1,195 @@
+---
+title: History Cache
+description: The delta-mode history cache — where it sits, its entry shape and immutability argument, every database.checkpoint_cache field with its default and env fallback, purge and redis failure semantics, stats, and the compiled accessor-graph cache cap.
+---
+
+import { Callout } from "nextra/components";
+
+# History Cache
+
+In `delta` mode a read has no message list to return: it reconstructs one by replaying ancestor writes back to the nearest `_DeltaSnapshot` (see [Snapshot Cadence](/docs/harness/checkpoints/cadence)). That walk is the materialization cost. The history cache stores the reconstructed histories so the next read pays a lookup instead of a walk, and it wraps the raw saver on the delta path.
+
+<Callout type="info">
+  The cache is performance-only. Nothing about correctness depends on it: no
+  field here is frozen, none must match across workers, and a full cache, an
+  empty cache, and no cache must all materialize identical state.
+</Callout>
+
+## Where the cache sits
+
+| Path | Wrapper | Backend |
+| --- | --- | --- |
+| Gateway / async | `CachedHistorySaver` wraps the selected saver when the effective mode is `delta`; the cache's lifetime equals the checkpointer context manager's | `memory` or `redis` |
+| Embedded / TUI (sync) | `_wrap_sync_if_delta` wraps with the memory backend only | `memory` only |
+| `full` mode | No wrapper at all — there is nothing to materialize | — |
+
+`CachedHistorySaver` overrides exactly one behaviour, `get_delta_channel_history` / `aget_delta_channel_history`. Everything else — tuple reads, writes, listing, copy, delete, prune — is delegated to the inner saver unchanged. The sync path shares one process-local memory cache and recreates it when `max_entries` or the derived key prefix changes, because entries under a stale prefix would be unreachable and no longer covered by thread purges.
+
+`redis` on the sync path is rejected outright: `database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.`
+
+## Entry shape
+
+An entry is a `DeltaChannelHistory`-shaped dict, exactly what the raw saver would return for that checkpoint and channel:
+
+| Field | Meaning |
+| --- | --- |
+| `writes` | On-path deltas for one channel, oldest to newest, as `PendingWrite` tuples. Always present, possibly empty. Writes stored at the target checkpoint itself are pending for the next super-step and are excluded from its own history. |
+| `seed` | The stored value at the nearest ancestor whose `channel_values[channel]` is populated. Omitted when the walk reaches the root without finding a stored value; a consumer treats absence as "start empty". |
+
+Reads are copy-on-read: `writes` comes back as a fresh list, while `seed` is shared by reference and never mutated in place.
+
+## Immutability: why there is no invalidation
+
+A checkpoint's delta history is a pure function of its sealed ancestor chain — the LangGraph contract excludes the target's own pending writes, parent links are fixed at creation, and an ancestor's writes are sealed once its child exists. Entries keyed by thread, namespace, checkpoint id, and channel are therefore immutable.
+
+| Property | Consequence |
+| --- | --- |
+| The key contains only immutable components | An entry never changes once written, so there is no invalidation path, no version field, and no coordination |
+| Ancestors are sealed when their child exists | A shared backend (redis) is coherent across processes without any handshake |
+| The cache never caches the "latest checkpoint" resolution | The target is resolved once through the inner saver before keys are built; only resolved, immutable checkpoint ids are keyed |
+| Thread deletion and prune are lifecycle events, not invalidation | They purge entries because the source checkpoints are gone, not because an entry went stale |
+
+That is why the whole policy is performance-only and safe to differ per worker.
+
+## Keys and namespace
+
+`make_history_key(key_prefix, thread_id, checkpoint_ns, checkpoint_id, channel)` builds:
+
+```
+{key_prefix}:{thread_id}:{sha256(checkpoint_ns \x00 checkpoint_id \x00 channel)[:24]}
+```
+
+| Fact | Detail |
+| --- | --- |
+| `thread_id` is left readable | Ops can identify a thread's keys without recomputing the digest |
+| The remaining components are hashed with NUL separators | A namespace containing `:` cannot produce an ambiguous key |
+| Thread purge stem | `{key_prefix}:{thread_id}:` — a prefix match, never per-key invalidation |
+| Default prefix | `ckpt-hist:v{CACHE_FORMAT_VERSION}:{hash12(database identity)}`, with `CACHE_FORMAT_VERSION = 1` |
+| `database.checkpoint_cache.key_prefix` | Replaces the whole derived prefix |
+| Database identity | Credential-free: postgres `host:port/database:schema`, the sqlite path, else `memory`. Rotating credentials does not change the namespace, so a credential rotation cannot cold-start the cache and orphan keys until TTL. Two deployments sharing one redis get different prefixes by default. |
+
+## Compose instead of walk
+
+Real runs create several checkpoints per super-step and only some are ever materialized as targets, so a target's direct parent is usually an unwarmed intermediate checkpoint. Composing only one level deep therefore measured zero cache hits on a 500-step sqlite run. The wrapper composes instead:
+
+| Step | Behaviour |
+| --- | --- |
+| Miss on the requested channel | Compose from the nearest warm ancestor, recursing one level per intermediate checkpoint |
+| Depth budget | `_COMPOSE_MAX_DEPTH = 8` |
+| Steady state | Each composed level is cached, so the warm frontier follows the run and lands on a warmed ancestor within about two levels |
+| Snapshot parent | The parent already holds the channel value, so the entry is built from the parent's writes plus its seed with no parent-history lookup at all |
+| Cold chain at depth 0 | Delegates exactly **one** inner fast-path walk (two SQL statements) for that level instead of crawling ancestors tuple by tuple. Ancestors below it stay cold and are resolved later from the nearest warm level. |
+| Root or missing parent | `{"writes": []}` |
+| Channel missing on every path | `{"writes": []}` |
+| Cache disabled | Passes straight through to the raw saver — composing over all-miss entries would be strictly more work than the walk |
+
+The sync path mirrors this recursion and shares the same cache and depth budget. Cold composition never calls the inner saver's history method: it either recurses or walks itself, so the fallback walk is the only place the raw saver is consulted.
+
+## Configuration
+
+The cache is built **with the checkpointer, once per process**. `langgraph_runtime()` constructs the
+checkpointer from the startup `AppConfig` snapshot (`backend/app/gateway/deps.py:463`), and the delta saver is wrapped in
+`CachedHistorySaver` around whatever `make_checkpoint_cache()` returned at that moment
+(`backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py:250-254`). Nothing re-reads this section at runtime, so every field below keeps its
+startup value for the process's lifetime.
+
+| Key | Type | Default | Range | Restart to change | Notes |
+| --- | --- | --- | --- | --- | --- |
+| `database.checkpoint_cache.type` | `memory` \| `redis` | `memory` | — | Yes | `memory` is a process-local LRU; `redis` is a shared cache for multi-worker deployments, async/Gateway path only |
+| `database.checkpoint_cache.max_entries` | `int` | `128` | `ge=0` | Yes | LRU capacity of the memory backend; `0` disables the cache |
+| `database.checkpoint_cache.redis_url` | `str` or null | `null` | Any redis URL | Yes | Falls back to `DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL`, then `REDIS_URL`, then `redis://localhost:6379/0` |
+| `database.checkpoint_cache.ttl_seconds` | `int` | `86400` | `ge=0` | Yes | Redis entry TTL, a leak safety net rather than a correctness mechanism; `0` disables expiry |
+| `database.checkpoint_cache.key_prefix` | `str` | `""` | Any string | Yes | Empty means the derived hash of the database identity |
+
+<Callout type="warning">
+  "Not frozen" is not "hot-reloadable". These settings are **not frozen** for correctness: they
+  cannot corrupt a thread, results are identical with the cache disabled, and processes sharing one
+  checkpoint database may safely run different values. But a running process never picks up an edit —
+  backend, capacity, Redis connection, prefix, and TTL all stay at their startup values — so changing
+  any of them needs a Gateway restart. The embedded and TUI paths behave the same way: their
+  checkpointer singleton is created once and reused for the life of the process. Only
+  [`database.checkpoint_graph_cache.accessor_graph_max`](#compiled-accessor-graph-cache) is re-read
+  at runtime.
+</Callout>
+
+```yaml
+database:
+  checkpoint_channel_mode: delta
+  checkpoint_cache:
+    type: memory            # memory | redis (redis is Gateway/async only)
+    max_entries: 128        # 0 disables the cache
+    redis_url: null         # or DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL / REDIS_URL
+    ttl_seconds: 86400      # redis leak safety net; purge itself is immediate
+    key_prefix: ""          # default: hash of the database identity
+```
+
+Only the redis URL has environment fallbacks; nothing else here is read from the environment, and `$VAR` interpolation inside config values is handled centrally before this section is parsed.
+
+`max_entries: 0` disables the cache uniformly for both types: the factory yields a disabled memory backend (`enabled` is `max_entries > 0`), so the wrapper never needs a None check. A zero-entry cache must behave exactly like the raw inner saver — same digests, `hits == 0` — and that parity is pinned by test. The memory backend rejects negatives with `max_entries must be >= 0`. Eviction is strict LRU: hits and writes both refresh recency and the oldest entries are dropped while over capacity, with a cumulative `evictions` counter. A one-entry cache thrashes on every read but stays correct.
+
+The redis client is imported lazily; without the optional extra, constructing a redis backend raises `redis is required for the redis checkpoint cache backend. Install it with: uv sync --extra redis`. An unrecognised `type` raises `Unknown checkpoint cache type: {config.type!r}`.
+
+## Purge and lifecycle
+
+| Operation | Cache action |
+| --- | --- |
+| `delete_thread` / `adelete_thread` | Purge that thread's entries by key-prefix match, after the source delete |
+| `prune` / `aprune` | Purge the rewritten threads' entries: a pruned chain must not keep pre-prune cached histories or references to a deleted ancestor |
+| `delete_for_runs` / `adelete_for_runs` | **No purge.** Run-scoped deletes cannot be mapped back to threads without an extra query, so entries stay in place: they remain correct (the sealing argument is per-chain) and residual retention is bounded by LRU/TTL. There are no in-tree callers today. |
+| `copy_thread` / `acopy_thread` | Delegated; no purge |
+
+The rationale is data lifecycle: source-of-truth removal must not leave residual history payloads behind in the cache. It is never a correctness requirement.
+
+## Redis failure behaviour
+
+Every redis failure is performance-only. The source of truth has already been updated by the time the cache is touched, so an outage costs hits, never availability:
+
+| Failure | Warning (verbatim) | Effect |
+| --- | --- | --- |
+| `mget` fails | `checkpoint history cache mget failed; treating as all-miss: %s` | Every requested key counts as a miss; histories are recomposed |
+| Write fails | `checkpoint history cache write failed; skipping: %s` | The write is skipped; the next read simply recomputes |
+| Thread purge fails | `checkpoint history cache thread purge failed; residual entries expire via TTL: %s` | The source delete already happened; residual copies expire via `ttl_seconds`; this path never raises |
+
+The TTL is a leak safety net, not a correctness mechanism, because entries are immutable and no read depends on them. `ttl_seconds: 0` opts out of expiry entirely, so orphaned keys then rely on the redis `maxmemory` policy alone. A failed purge is the only way a deleted thread's history lingers, and only until the TTL.
+
+## Stats
+
+`CachedHistorySaver.stats()` merges the backend's counters with the wrapper's own composition counters. It is the only cache surface: no gateway endpoint or health check reports cache state.
+
+| Counter | Source | Meaning |
+| --- | --- | --- |
+| `hits` / `misses` | Backend | Per-key lookups |
+| `evictions` / `entries` | Memory backend only | LRU drops and the current entry count. Redis reports `0` for both — it keeps no local LRU or size, and its own `maxmemory` policy is the bound |
+| `compose_hits` | Wrapper | Levels composed from a stored parent value or from a cached history |
+| `full_walks` | Wrapper | Cold fallbacks that delegated an inner delta walk |
+
+A steady-state process should show `compose_hits` growing while `full_walks` stays low; a persistently high `full_walks` means composition keeps hitting cold chains, and `hits` pinned at `0` on a warm process means lookups never reach entries — usually a disabled cache or a namespace that does not match the entries' prefix, since the prefix is derived from the database identity. See [Troubleshooting](/docs/harness/checkpoints/troubleshooting).
+
+## Compiled accessor-graph cache
+
+Delta reads must go through the thread-state accessor, whose compiled graph is cached per assistant, channel mode, snapshot cadence, and app config. That cache has its own size cap:
+
+| Key | Type | Default | Range | Restart |
+| --- | --- | --- | --- | --- |
+| `database.checkpoint_graph_cache.accessor_graph_max` | `int` | `64` | `ge=1` | Not required — hot-reloadable |
+
+| Fact | Consequence |
+| --- | --- |
+| A larger or smaller cap only changes when the cache evicts, never graph semantics | The value is re-read on every eviction check, so a hot reload takes effect without a restart |
+| At the cap the whole cache is cleared (no LRU here) | The cap bounds distinct assistants and cadences, not recency |
+| A config reload rebuilds the `AppConfig` object, and cached entries re-validate factory and config identity | A hot reload never serves a graph compiled under the old config |
+| Unusable values (non-int, boolean, below 1) fall back to the in-code default `64` | A stub or partial config cannot break accessor caching |
+
+This cap applies in both modes and is unrelated to the history cache, which only wraps delta savers. It exposes no counters; it is bounded by size only.
+
+## Measuring the cache
+
+`cache_effect_ms` (state cold minus warm p50) is the decision metric for the production accessor-cache defaults, and it comes out of the production checkpoint benchmark, which also reports `state_cold_ms`, `state_warm_p50_ms`, `state_warm_p95_ms`, `db_bytes`, and per-limit history read times. The channel benchmark reports `warm_read_ms`, `cold_read_ms`, and `reducer_replay_ms` with `delta/full` ratios. Timing thresholds are not CI gates. See [Operating Checkpoints](/docs/harness/checkpoints/operations).
+
+## Related
+
+- [Snapshot Cadence](/docs/harness/checkpoints/cadence) — the walk this cache memoizes.
+- [Channel Modes](/docs/harness/checkpoints/modes) — the accessor and the mode marker.
+- [Observability](/docs/harness/checkpoints/observability) — where these counters surface.
+- [Troubleshooting](/docs/harness/checkpoints/troubleshooting) — redis errors on the embedded path and zero-hit caches.
+- [Reference](/docs/harness/checkpoints/reference) — the full config-key table.

frontend/src/content/en/harness/checkpoints/index.mdx

@@ -0,0 +1,85 @@
+---
+title: Checkpoints and Delta Channels
+description: How DeerFlow persists thread state, and how database.checkpoint_channel_mode chooses between full-message checkpoints and LangGraph DeltaChannel delta checkpoints. Concepts, storage shapes, and reading routes for the manual.
+asIndexPage: true
+---
+
+import { Callout } from "nextra/components";
+
+# Checkpoints and Delta Channels
+
+<Callout type="info" emoji="🗃️">
+  A checkpoint is one thread's persisted state at one step. `database.checkpoint_channel_mode`
+  decides how the message channel inside it is stored: `full` keeps the complete message list in
+  every checkpoint, `delta` leaves the list out of per-step checkpoints, appends each step's writes,
+  and reconstructs the list when a read needs it. The default is `full`.
+</Callout>
+
+Every DeerFlow run writes thread state to a checkpoint after each step: the message list, the goal, sandbox handles, durable context, and any channels contributed by middleware. The message list dominates that state in a long run, so how it is represented on disk decides how much a store retains after N turns: `full` mode retains a quadratic amount of it, and `delta` mode shrinks that quadratic component by a factor set by the snapshot cadence rather than removing it. This manual covers both representations, the switch that selects one, and what each choice costs an operator.
+
+## How to read this manual
+
+| Reader                 | Start with                                                                                                                                                                                                                                                                                                                        |
+| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Chat users             | This page, [Quick Start](/docs/harness/checkpoints/quick-start), [Channel Modes](/docs/harness/checkpoints/modes)                                                                                                                                                                                                                  |
+| Operators              | [Quick Start](/docs/harness/checkpoints/quick-start), [Snapshot Cadence](/docs/harness/checkpoints/cadence), [History Cache](/docs/harness/checkpoints/history-cache), [Operating Checkpoints](/docs/harness/checkpoints/operations), [Observability](/docs/harness/checkpoints/observability), [Troubleshooting](/docs/harness/checkpoints/troubleshooting) |
+| Integration developers | [Channel Modes](/docs/harness/checkpoints/modes), [Resume and Rollback](/docs/harness/checkpoints/resume-and-rollback), [Reference](/docs/harness/checkpoints/reference)                                                                                                                                                            |
+
+## Concepts
+
+- **Checkpoint**: one thread's channel values at one step, plus the metadata LangGraph attaches to it — checkpoint id, parent link, pending writes, counters. Reads, resumes, and branches all start from a checkpoint.
+- **Channel**: one field of the thread state schema, with its own value type and update rule. A compiled graph carries a channel table that decides how each channel is stored and merged. The mode changes only the `messages` channel.
+- **Reducer**: the function that folds incoming writes into a channel's value. Delta mode's message reducer is `merge_message_writes`, which applies `add_messages` semantics in linear time.
+
+## Why delta exists
+
+In `full` mode every checkpoint carries the complete channel values, so each step re-serializes the whole message list, and all of those checkpoints are retained: total storage grows O(N²) in turns.
+
+In `delta` mode the message list is a LangGraph `DeltaChannel`. A non-snapshot checkpoint stores no message value in `channel_values`, each step's updates are appended as writes, and a read reconstructs the list by replaying the ancestor writes through the reducer. That removes the whole-list serialization from most checkpoints — but it does not remove the quadratic term, because every `snapshot_frequency` writes a `_DeltaSnapshot` holds the **entire** message list as of that step, and those snapshots are retained like any other checkpoint.
+
+Let K be `snapshot_frequency`. The two quantities a capacity plan depends on are different:
+
+| Quantity                   | `full`                                                                     | `delta`                                                                                                                   |
+| -------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| Marginal cost of one step  | Re-serializes the whole list into one more retained checkpoint             | Appends that step's writes; every K writes also stores a full-list snapshot                                                |
+| Total retained after N turns | O(N²)                                                                    | O(N²/K + N)                                                                                                                |
+
+Delta therefore divides the quadratic component by roughly K — `10` by default — instead of deleting it. Fewer snapshots mean a smaller residual *and* slower reads, since materialization replays from the nearest one ([Snapshot Cadence](/docs/harness/checkpoints/cadence)). Nothing in DeerFlow prunes those snapshots yet, so size a store from the retained total rather than from the per-step write cost ([Operating Checkpoints](/docs/harness/checkpoints/operations)).
+
+|                      | `full`                                                                     | `delta`                                                                                                                   |
+| -------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| Checkpoint payload   | Complete `channel_values`, message list included                           | No message blob on non-snapshot steps; a sentinel `_DeltaSnapshot` blob appears every `snapshot_frequency` writes          |
+| Step updates         | Folded into the next checkpoint's payload                                  | Stored as per-step writes                                                                                                  |
+| Read path            | Stored values are used directly                                            | Reconstructed from the nearest stored snapshot plus replayed writes                                                        |
+| Total retained after N turns | O(N²)                                                              | O(N²/K + N), where K = `snapshot_frequency`                                                                                |
+
+The mode is a **process-level, restart-required choice**. Each process resolves it once, before the first graph is compiled, and cannot change it afterwards; every process sharing one checkpoint database must use the same value. [Quick Start](/docs/harness/checkpoints/quick-start) covers the switch, and [Snapshot Cadence](/docs/harness/checkpoints/cadence) covers the second frozen knob, the snapshot cadence.
+
+## Terminology
+
+| Term                       | Meaning                                                                                                                                                |
+| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Checkpoint                 | A thread's persisted channel values at one step, plus LangGraph's checkpoint metadata.                                                                  |
+| Channel                    | One field of the thread state schema, with its own value type and update rule.                                                                          |
+| Reducer                    | The function that folds writes into a channel value.                                                                                                    |
+| Delta channel (`DeltaChannel`) | LangGraph's reducer channel that keeps no full value in the checkpoint and reconstructs it by replaying ancestor writes.                           |
+| Sentinel                   | The marker stored in a delta channel's place in `channel_values`; a stored `_DeltaSnapshot` terminates the ancestor walk for that channel.               |
+| Seed                       | The stored value at the nearest ancestor whose `channel_values` is populated. Reconstruction starts from it.                                            |
+| Writes                     | Per-step channel updates stored in their own table. In delta mode they are load-bearing replay state, not garbage.                                       |
+| Materialize                | Reconstruct a channel's value from the seed plus replayed writes.                                                                                       |
+| Cadence (`snapshot_frequency`) | How many per-step writes pass before a full messages snapshot is stored.                                                                            |
+| Ancestor chain             | The checkpoint parent chain. A delta checkpoint's history is a pure function of its sealed ancestor chain.                                              |
+| Fork                       | A second child of the same checkpoint. Delta mode cannot fork correctly; a resume is linearized onto the head instead.                                   |
+| Linearization              | Re-expressing a resume from an older checkpoint as a replace-style write on the current head.                                                           |
+| Rollback                   | Restoring the pre-run head after a cancelled run.                                                                                                       |
+| Compaction                 | Replacing the message list with a summary, written through an `Overwrite` so the replace survives the append reducer.                                    |
+| Retention / prune          | Deleting checkpoint rows. In delta mode the writes rows that reconstruction depends on must survive.                                                     |
+| Cache                      | The delta-only history cache that memoizes materialized channel histories.                                                                              |
+| Frozen                     | Resolved once per process before the first graph is built; a different value later is an error, and changing it needs a restart.                        |
+| Gateway / embedded         | The async Gateway path versus the sync TUI or in-process `DeerFlowClient` path.                                                                         |
+
+<Cards num={3}>
+  <Cards.Card title="Quick Start" href="/docs/harness/checkpoints/quick-start" />
+  <Cards.Card title="Channel Modes" href="/docs/harness/checkpoints/modes" />
+  <Cards.Card title="Snapshot Cadence" href="/docs/harness/checkpoints/cadence" />
+</Cards>

frontend/src/content/en/harness/checkpoints/modes.mdx

@@ -0,0 +1,90 @@
+---
+title: Channel Modes
+description: What full and delta checkpoints look like on disk, how each read path materializes state, the mode marker contract, the fail-closed gate, the 409/503 error mapping, the cross-process invariant, and the state-schema symbols each mode compiles.
+---
+
+import { Callout } from "nextra/components";
+
+# Channel Modes
+
+`database.checkpoint_channel_mode` has two values. The mode decides how the message channel is written and how a read materializes it. It is frozen per process, so the value a thread was written with and the value a reading process froze on startup must agree.
+
+## Storage shape and read path
+
+| | `full` | `delta` |
+| --- | --- | --- |
+| Where the message list lives | In every checkpoint's `channel_values` | In the per-step writes. A non-snapshot step leaves the channel out of `channel_values`; a `_DeltaSnapshot` blob is stored every `snapshot_frequency` writes |
+| What one step costs | Re-serializing the whole list into one more retained checkpoint | Appending that step's writes; every K writes also stores a full-list `_DeltaSnapshot` (total retained after N turns: O(N²) vs O(N²/K + N)) |
+| Snapshot trigger | n/a — every step is a full payload | The update count reaches `snapshot_frequency`, or the supersteps count reaches LangGraph's `DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT` bound (5000) |
+| Read path | Stored values are used directly | Hydrate channels; delta channels absent from `channel_values` are batched into a single saver call; then `from_checkpoint(seed)` plus `replay_writes` |
+| Raw checkpointer read | Returns the stored value | Returns a sentinel, not the message list |
+
+<Callout type="warning">
+  Delta checkpoints store no full `channel_values` — raw saver reads see sentinels — so consumers must go
+  through `CheckpointStateAccessor` instead of calling the checkpointer directly. The accessor also
+  re-injects the mode marker on every prepared config, on a copy of `configurable` and `metadata`, so a
+  caller's dicts are not mutated.
+</Callout>
+
+The supersteps bound is what keeps replay depth bounded even for a channel that stops receiving writes; the cadence is the knob you tune. See [Snapshot Cadence](/docs/harness/checkpoints/cadence) for the trade-off and [History Cache](/docs/harness/checkpoints/history-cache) for what materialization costs.
+
+## Marker contract
+
+| Constant | Key and location | Written when | Meaning |
+| --- | --- | --- | --- |
+| `CHECKPOINT_MODE_METADATA_KEY` | `deerflow_checkpoint_channel_mode` in `config["metadata"]` | `delta` only, value `"delta"` | The persisted mode marker. Full mode removes the key, so **absence means full** and pre-feature checkpoints need no migration |
+| `counters_since_delta_snapshot` | LangGraph's own checkpoint metadata key, a dict containing `"messages"` | Written by LangGraph | Fallback detection: its presence is treated as delta even when the DeerFlow marker is absent |
+| `INTERNAL_CHECKPOINT_MODE_KEY` | `__deerflow_checkpoint_channel_mode` in `config["configurable"]` | Every request, both modes | Internal routing only, set by `inject_checkpoint_mode`. It is never a stored metadata key and never reaches the wire; a caller-supplied value is discarded by the Gateway's run-config builder, ignored on first freeze, and rejected once the process is frozen |
+
+The snapshot cadence is deliberately **not** stamped into checkpoint metadata, which is why the marker contract is unaffected by the cadence value:
+
+> It is deliberately NOT stamped into checkpoint metadata: the mode marker contract (absence = full) and the full -> delta migration semantics are unchanged by the frequency value.
+
+## The fail-closed gate
+
+The gate runs on both sides of an access, and it is asymmetric: a delta-mode process reads legacy full checkpoints transparently, while a full-mode process refuses a delta thread instead of silently materializing empty state.
+
+| Gate | Runs on | Behaviour |
+| --- | --- | --- |
+| `raise_if_snapshot_incompatible(snapshot, mode)` | Materialized reads (`get_state`, `get_state_history`) | Raises on the materialized `StateSnapshot`; one checkpoint fetch, no extra round-trip |
+| `raise_if_checkpoint_tuple_incompatible(tuple, mode)` | Metadata-only reads (`get_metadata`, `aget_metadata`) | Keeps the gate without materializing anything |
+| `ensure_checkpoint_mode_compatible(checkpointer, config, mode)` / `aensure_checkpoint_mode_compatible(...)` | Before a write | Fails closed before the write is persisted — a write cannot be un-applied. Both return immediately when the process mode is `delta` |
+
+A rejected access raises `CheckpointModeMismatchError` with the message `Thread requires delta mode; materialize and convert its checkpoints before using full mode.` The same string is duplicated in the degraded full-mode read accessor, so the message survives a broken agent factory.
+
+## HTTP error mapping
+
+| Error | Status | `detail` | Meaning |
+| --- | --- | --- | --- |
+| `CheckpointModeMismatchError` | 409 | `Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode.` | The thread's persisted checkpoints conflict with the process's frozen mode |
+| `CheckpointModeReconfigurationError` | 503 | `checkpoint_channel_mode is restart-required and cannot change in a running process` (or the `checkpoint_delta.snapshot_frequency` twin) | The process itself is mid mode-flip, so the request is transiently unserviceable |
+
+The router records the rationale for the split:
+
+> A mismatch means the thread's persisted checkpoints conflict with the process's frozen mode (operator-actionable, 409); a reconfiguration means the process itself is mid mode-flip (transient, 503).
+
+Only the threads router maps these two errors; run-creation routes surface the same mismatch text asynchronously in the run record or the SSE `error` frame — except when the thread's run-event feed is empty and its checkpoint head has to be materialized before admission, where the seeding read turns the mismatch into a plain `500`. The `503` half of the mapping is defensive: reconfiguration errors are raised when a graph is built (Gateway startup, `make_lead_agent`, `DeerFlowClient.__init__`) and no state, history, or thread route builds an agent, so an operator who edited `config.yaml` without restarting sees a failed run rather than a `503`. See [Observability](/docs/harness/checkpoints/observability) and [Troubleshooting](/docs/harness/checkpoints/troubleshooting).
+
+## Cross-process invariant
+
+- The mode is compiled into each graph's channel table, not stored per checkpoint, so it must match in **every** process sharing one checkpoint database.
+- The cadence must match as well, but it is not stamped, so a per-checkpoint mismatch is undetectable: there is no runtime cadence comparison at all. Enforcement is fail-closed at the mode level only, by marker presence or absence. Treat identical `config.yaml` files as a deployment requirement, not a suggestion.
+- The freeze is module-global per OS process. Each process freezes independently from its own startup config, so nothing in the runtime reconciles a divergent process for you.
+
+## State schemas
+
+The mode picks the compiled state schema and adapts middleware-contributed channels to it.
+
+| Symbol | Module | Role |
+| --- | --- | --- |
+| `ThreadState` | `deerflow.agents.thread_state` | The full-mode state schema |
+| `DeltaThreadState` | `deerflow.agents.thread_state` | The delta schema at the default cadence; the name is preserved only when the cadence is the default |
+| `delta_messages_field(snapshot_frequency=…)` | `deerflow.agents.thread_state` | Builds `Annotated[list[AnyMessage], DeltaChannel(merge_message_writes, snapshot_frequency=…)]` |
+| `DELTA_MESSAGES_FIELD` | `deerflow.agents.thread_state` | The module-level delta field at the default cadence |
+| `merge_message_writes(state, writes)` | `deerflow.agents.thread_state` | The linear-time `add_messages`-equivalent delta reducer |
+| `get_thread_state_schema(mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | Returns `ThreadState` for non-delta modes and a cached delta schema otherwise; non-default cadences get a cached, cadence-specific schema |
+| `adapt_state_schema_for_mode(schema, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | Adapts a state schema for the mode, so middleware channels compile against the right channel types |
+| `normalize_middleware_state_schemas(middleware, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | Returns the middleware list with adapted `state_schema` copies |
+| `THREAD_STATE_REDUCER_FIELDS` | `deerflow.agents.thread_state` | The reducer channels, including `messages`; used to decide which channels need `Overwrite` wrapping for a replace-style write |
+
+The full key table, the remaining constants, and the load-bearing test anchors are in [Reference](/docs/harness/checkpoints/reference).

frontend/src/content/en/harness/checkpoints/observability.mdx

@@ -0,0 +1,117 @@
+---
+title: Observability
+description: Nothing reports the checkpoint mode. This chapter covers what a client can and cannot see, the log lines and SSE frames a mismatch produces, what a browser user actually sees, the cache counters an operator can read, and how to determine a running process's or a thread's mode.
+---
+
+import { Callout } from "nextra/components";
+
+# Observability
+
+## Nothing exposes the mode
+
+The checkpoint mode is server-side state. Neither the mode nor the snapshot cadence appears in any health, readiness, features, capabilities, or console field, and none of them appears in a response model, a response header, or an event field.
+
+- The health and readiness probes only test backend reachability. No route reads `checkpoint_channel_mode` except the threads router's error mapping.
+- The only custom SSE headers are `Cache-Control`, `Connection`, `X-Accel-Buffering`, and `Content-Location`.
+- The Gateway consumes the mode server-side only, into `app.state.checkpoint_channel_mode` / `app.state.checkpoint_snapshot_frequency` and the run context. Nothing serializes it outward.
+
+There is no field to poll for "is this deployment in delta mode?". You infer it from a thread's metadata, from a `409` on a gated route, or from inside the process.
+
+## What a client can and cannot see
+
+Two mode keys exist, and only one of them can reach a client.
+
+| Key | Where it is written | On the wire |
+| --- | --- | --- |
+| `INTERNAL_CHECKPOINT_MODE_KEY` = `"__deerflow_checkpoint_channel_mode"` | `config["configurable"]`, always, both modes | Never. Server-side only. |
+| `CHECKPOINT_MODE_METADATA_KEY` = `"deerflow_checkpoint_channel_mode"` | `config["metadata"]`, **only** in delta mode; removed in full mode | Can pass through inside response `metadata` objects |
+
+Because the marker is only written in delta mode, a stored delta checkpoint carries it and a stored full checkpoint does not; absence means full.
+
+Two read endpoints build their response `metadata` from the stored snapshot's metadata minus a strip list, and that list does not contain the mode marker or LangGraph's own `counters_since_delta_snapshot`, so both can reach a client:
+
+| Endpoint | Response `metadata` source |
+| --- | --- |
+| `GET /api/threads/{thread_id}` | `snapshot.metadata` minus `created_at`, `updated_at`, `step`, `source`, `writes`, `parents` — but only when no `threads_meta` row exists; otherwise the store row supplies `metadata` |
+| `POST /api/threads/{thread_id}/history` | each entry's `snapshot.metadata` minus the same list plus `run_durations` and the run-message-ids key, then `step` is re-added |
+
+This passthrough follows from the strip lists in the handlers; no test asserts the marker on the wire, so treat it as an implementation detail rather than a contract.
+
+Stream frames carry no mode. The SSE `metadata` event payload is only `{"run_id", "thread_id"}`, and the SSE `error` event payload is `{"message", "name"}`, where `name` is the exception class name.
+
+## Log and stream surface
+
+A mode problem produces different evidence depending on which path hit it.
+
+| Situation | Surface | Verbatim text |
+| --- | --- | --- |
+| A run hits a mismatch | run worker log (`exc_info=True`) | `Run %s failed: %s` with the mismatch text as `%s` |
+| A run hits a mismatch | run record / `/wait` body / SSE `error` frame | `Thread requires delta mode; materialize and convert its checkpoints before using full mode.` |
+| A run hits a restart-required violation | run worker log (`exc_info=True`), run record, SSE `error` frame | `Run %s failed: %s` with `checkpoint_channel_mode is restart-required and cannot change in a running process` (or the `checkpoint_delta.snapshot_frequency` twin) as `%s` |
+| Context usage hits a mismatch | log | `Failed to load checkpoint for context usage on thread %s` |
+| Regenerate cannot read the latest checkpoint | log, then HTTP 500 `detail` | `Failed to read latest checkpoint for regenerate thread %s` → `Failed to read latest checkpoint` |
+| Regenerate cannot list checkpoint history | log, then HTTP 500 `detail` | `Failed to list checkpoints for regenerate thread %s` → `Failed to inspect checkpoint history` |
+
+The failure payloads differ by route. Run creation does **not** pre-gate on the threads router: `POST /api/threads/{thread_id}/runs` returns a `200` run record that later turns `status: error` with the mismatch text, `POST /runs/stream` returns `200 text/event-stream` and then an `event: error` frame with `{"message": ..., "name": "CheckpointModeMismatchError"}`, and `POST /runs/wait` returns `200` JSON with `{"status": "error", "error": "<mismatch text>"}`. The exception is a thread whose run-event feed is empty but whose checkpoint head exists: `ensure_checkpoint_history_seeded` materializes that head inside `start_run`, where the mode errors are not mapped, so the same mismatch returns a plain `500` before the run is admitted. The stateless `/api/runs/*` routes share that path.
+
+<Callout type="warning">
+  The mode gate itself logs nothing. `checkpoint_mode.py` has no logger,
+  and the threads router converts mode errors into `HTTPException`s without
+  logging, so a `409` leaves no log line of its own. The evidence lives on
+  the caller: the run worker's `Run %s failed: %s`, the run record, or the
+  client-side toast.
+</Callout>
+
+## What a browser user sees
+
+The UI has no mode surface at all: no badge, setting, column, or error copy, and no i18n string for a mismatch. But the chat page does hit the gated routes, so the failure is visible without naming the mode.
+
+- The chat page loads durable history by default (`fetchStateHistory: { limit: 1 }`), which calls `POST /api/threads/{thread_id}/history`. On a delta thread read by a full-mode process, that is the `409` route.
+- The SDK turns the response into an `HTTPError` whose message is `HTTP 409: <raw response body text>`; `409` is in the SDK's no-retry set, so the error is thrown immediately without retries.
+- That error reaches the stream `onError`, which shows a toast of the error message. The user therefore sees the raw server text, approximately:
+
+  ```
+  HTTP 409: {"detail":"Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode."}
+  ```
+
+  Both halves are individually verified (`HTTP 409: ` from the SDK, the `detail` body from the router); the concatenated string itself was not observed at runtime.
+- Sidebar **Export** calls `getState()` → `GET /api/threads/{thread_id}/state` → `409`, but a bare `catch {}` discards the detail and shows only the generic `exportFailed` toast.
+- **Rename** calls `updateState()` → `POST /api/threads/{thread_id}/state` → `409` → `toast.error(error.message ?? renameFailed)`, so on this path the raw HTTPError string can surface.
+- Stream-gap recovery wraps a failing `getState` into `StreamReplayGapError`; the other internal `getState` (the reconnect input snapshot) swallows every non-abort error and returns `undefined`.
+- The only checkpoint-ish toast text in the UI, `Failed to load thread history.`, belongs to the messages-page route, not to a mode-gated one.
+
+## Cache counters
+
+The delta history cache wraps the saver only in delta mode on the Gateway (async) path. `CachedHistorySaver.stats()` merges the backend counters with two wrapper counters. No HTTP route exposes them; read them in-process.
+
+| Counter | Meaning | Reported by |
+| --- | --- | --- |
+| `hits` | per-channel history lookups served from the cache | memory, redis |
+| `misses` | lookups that had to be composed or walked | memory, redis |
+| `evictions` | entries dropped by the LRU bound | memory only |
+| `entries` | current number of cached entries | memory only |
+| `compose_hits` | ancestor levels resolved by composing from a warm ancestor (cached entry or snapshot-bearing parent) | wrapper |
+| `full_walks` | fallbacks to the raw inner saver's own walk (cold chain at the depth budget, disabled cache, or no target found) | wrapper |
+
+The redis backend reports only `hits` and `misses`; its `evictions` and `entries` stay at `0`. `max_entries: 0` disables the cache uniformly, and a disabled cache passes straight through to the raw saver without composing at all. Changing any `database.checkpoint_cache.*` field takes a restart before a running Gateway reports different counters, because the cache is built with the checkpointer at startup; see [History Cache](/docs/harness/checkpoints/history-cache).
+
+## Determining the mode
+
+### A running process
+
+The mode is a module-global frozen the first time a process builds an agent. `frozen_checkpoint_channel_mode()` returns the frozen value or `None`; `frozen_checkpoint_snapshot_frequency()` does the same for the cadence. The Gateway additionally stores the startup snapshot at `app.state.checkpoint_channel_mode` and `app.state.checkpoint_snapshot_frequency`.
+
+Nothing prints either value at startup. In practice:
+
+1. Read `database.checkpoint_channel_mode` (and `database.checkpoint_delta.snapshot_frequency`) from `config.yaml`.
+2. Confirm the process started after the last edit — an edit without a restart has no effect and fails loudly once a new graph is built (see [Troubleshooting](/docs/harness/checkpoints/troubleshooting)).
+3. Remember that every process sharing the checkpoint database must use the same value, and that a client-supplied `configurable.__deerflow_checkpoint_channel_mode` cannot change a process's mode.
+
+### A thread
+
+A thread's mode is recorded in its stored checkpoints:
+
+- A delta checkpoint carries `deerflow_checkpoint_channel_mode: "delta"` in its metadata; a full checkpoint does not.
+- LangGraph's own `counters_since_delta_snapshot` metadata is a second delta signal.
+- Through HTTP, the marker can be read from the `metadata` objects of `GET /api/threads/{thread_id}` (when no `threads_meta` row exists) and of each entry in `POST /api/threads/{thread_id}/history`.
+- Or open the thread in a process you believe is `full`: a delta thread answers with `409`.

frontend/src/content/en/harness/checkpoints/operations.mdx

@@ -0,0 +1,218 @@
+---
+title: Operating Checkpoints
+description: Backend independence, the retention and prune constraint in delta mode, the one remaining import-time patch, dependency floors, how to run both checkpoint benchmarks, and what differs on the TUI and embedded paths.
+---
+
+import { Callout } from "nextra/components";
+
+# Operating Checkpoints
+
+Delta mode changes where thread state is stored, not which backend stores it. This chapter covers the operational consequences: backend choice, retention and prune constraints, the one remaining compatibility patch, the dependency floors that keep upstream bugs out, how to measure both modes, and the differences an operator sees on the TUI and embedded path.
+
+## Backends
+
+The mode is a property of the compiled graph's channel table, not of the saver, so all three backends serve both modes:
+
+| `database.backend` | Saver | Both modes | Notes |
+| --- | --- | --- | --- |
+| `memory` | `InMemorySaver` | Yes | Process-local, lost on restart; mode semantics are identical |
+| `sqlite` | SQLite saver | Yes | Channel values live inside the serialized checkpoint payload; there is no separate blob table |
+| `postgres` | Postgres saver | Yes | `checkpoints` plus a separate `checkpoint_blobs` table; writes live in `checkpoint_writes` |
+
+Two process-level rules bind regardless of backend, and neither is enforced by the backend itself:
+
+- **Mode must match.** Every process sharing one checkpoint database must use the same `database.checkpoint_channel_mode`. The mode is baked into each compiled graph's channel table, not stored per checkpoint.
+- **Cadence must match.** `database.checkpoint_delta.snapshot_frequency` is baked into the channel table the same way. It is deliberately not stamped into checkpoint metadata, so a mismatch is undetectable per checkpoint and remains an operator responsibility. The fail-closed gate enforces the *mode* only; there is no runtime cadence comparison.
+
+Delta mode changes the storage shape on every backend: per-step checkpoints carry no full `messages` blob, so the marginal cost of a step no longer scales with the history. Retained storage still includes a full-list `_DeltaSnapshot` every `snapshot_frequency` writes, so after N turns it grows as O(N²/K + N) with K = `snapshot_frequency` — reduced by roughly K, not linear. Plan capacity from the retained total; see [Checkpoints and Delta Channels](/docs/harness/checkpoints) and [Channel Modes](/docs/harness/checkpoints/modes).
+
+## Retention and pruning
+
+### Writes rows are replay state, not garbage
+
+In `full` mode every checkpoint re-snapshots the accumulated messages, so discarding an intermediate checkpoint is at least conceptually possible. In `delta` mode the per-step payloads live in the **writes** table, and they are load-bearing replay state: deleting them severs the ancestor chain a later materialization depends on.
+
+| Mode | Where per-step payloads live |
+| --- | --- |
+| `full` | Re-snapshotted into the checkpoint payload |
+| `delta` | The writes table; a full value is snapshotted only every `snapshot_frequency` steps |
+
+The retention contract suite pins this shape directly. Its growth baseline asserts that a delta thread ends with `write_rows > 0` ("delta mode must land per-step payloads in writes"), while full mode re-snapshots everything into the checkpoint payload. Byte comparisons between the two are cadence- and backend-dependent, so only the shape is asserted.
+
+### The naive prune failure
+
+<Callout type="warning">
+  A naive `keep_latest` that drops intermediate checkpoints and their writes can
+  sever the chain: the surviving "latest" checkpoint is rarely a snapshot point
+  itself, so its delta channels would silently reconstruct as empty. No error is
+  raised — `get_delta_channel_history` simply returns no `seed`.
+</Callout>
+
+Upstream documents three safe options for a graph that uses `DeltaChannel`:
+
+- Walk back from each kept checkpoint and preserve every ancestor (plus its writes) up to the nearest one whose `channel_values` already contains a `_DeltaSnapshot` for every delta-backed key.
+- Force a fresh snapshot on the kept checkpoint before deleting ancestors, rewriting `channel_values[k] = _DeltaSnapshot(value)` for each delta channel `k`.
+- Skip pruning threads whose graph uses `DeltaChannel` until one of the above is implemented.
+
+The same warning extends to `adelete_for_runs` and `copy_thread`: deleting a run's ancestor writes (or the only `_DeltaSnapshot` blob) breaks reconstruction for any delta channel that depended on those rows.
+
+### The protected set
+
+| # | Never delete without the stated compensation |
+| --- | --- |
+| 1 | **Explicit resume targets** — any `checkpoint_id` a client may still resume to. A policy may expire these, but only with an explicit TTL semantic agreed up front |
+| 2 | **Branch ancestors** — every checkpoint on the parent chain from a branchable head back to, and including, the checkpoint before the oldest branchable message. Deleting any node on that walk breaks branch and regenerate, and it fails loudly rather than silently |
+| 3 | **Pending writes** — rows in the writes table are uncommitted or in-flight state, not garbage |
+| 4 | **Duration-only chain links** — a duration-only checkpoint a later run has forked from is a chain link; deleting it requires grafting the fork onto the grandparent in the same change. A bare leaf is safe; a link is not |
+| 5 | **Latest resumable state per thread** — the newest checkpoint must stay addressable so a thread can always continue |
+
+### What may actually be deleted today
+
+Exactly two shapes are proven safe, and the retention service implements only those:
+
+| Shape | Default | Why it is safe |
+| --- | --- | --- |
+| Trailing duration-only leaves | On | `persist_run_durations` appends metadata-only checkpoints after a run finishes; while no later run has forked from one, it is nobody's ancestor |
+| Leaf sibling branches | Off (opt-in) | A checkpoint forked off an older turn that has no children. Opt-in because a superseded line's checkpoints may still be explicit resume targets a client holds, hence the `protect_checkpoint_ids` escape hatch |
+
+Two mechanical constraints come with deletion:
+
+- **Joint table deletion, orphan-first.** Deleting a checkpoint row requires deciding which writes and blob rows are orphaned; a row is an orphan only if no *surviving* checkpoint references it, computed in a whole-thread pass. A real duration-only leaf is not payload-free — it materializes the parent's values — and on Postgres its blob rows are the same rows backing its parent.
+- **All-or-nothing.** A partial deletion that leaves a dangling parent link converts a cleanup into a thread-level outage: branch and regenerate fail loudly for every later turn.
+
+There is also a history fast-path interaction to sequence around: the trailing duration-only leaf carries the `run_durations` and `run_message_ids` maps that `get_thread_history` reads from the latest checkpoint. Deleting the leaf makes the next history read fall back to store scans and re-persist a fresh leaf, so any wiring must sequence retention away from history reads or spare cache-carrying leaves.
+
+### Policy defaults
+
+| Field | Default | Meaning |
+| --- | --- | --- |
+| `prune_trailing_duration_leaves` | `True` | Delete a duration-only leaf no later run has forked from |
+| `prune_leaf_sibling_branches` | `False` | Delete leaf sibling branches (opt-in) |
+| `protect_checkpoint_ids` | `frozenset()` | Explicit ids that are never deleted |
+| `strict_pending_write_guard` | `True` | Treat any checkpoint that still owns writes rows as protected |
+| `max_delete_per_run` | `None` | Optional cap on deletions per run |
+
+### There is no production trigger yet
+
+The retention service (`backend/app/gateway/checkpoint_retention.py`) ships **without a production trigger**: where retention is invoked from — post-run hook, scheduler, or explicit admin action — is a maintainer decision that lands with the contract itself. Nothing in DeerFlow calls it, and there is no production caller of `prune` or `aprune` either: in the pinned environment only the base implementation defines them, and it raises `NotImplementedError`.
+
+Treat this as unfinished work rather than a safe default. The executable contract proves the branch-ancestor, explicit-resume-target, pending-writes, leaf-duration, and leaf-sibling scenarios against the **full-mode** state schema; only the growth baseline is delta-shaped. Delta-specific prune safety is neither proved by the suite nor implemented.
+
+## The one remaining patch
+
+One compatibility patch remains, and it is applied at import time of the module itself. The module lives at the package root so it can be imported from `deerflow.agents.thread_state` without pulling in the heavier `deerflow.runtime` package, and it is anchored there so every process that builds a DeerFlow graph — Gateway, workers, in-process LangGraph runtime, tests — runs with it in place.
+
+| Aspect | Detail |
+| --- | --- |
+| Subject | `BinaryOperatorAggregate` unwrapping an `Overwrite` first write into an empty (MISSING) channel |
+| Tracking issue | DeerFlow **#4380** |
+| Symptom it prevents | Channels whose type is a Union (`SandboxState \| None`, `GoalState \| None`, and similar) have no constructible default, so they start MISSING. A replace-style write into a fresh thread (thread branching) or a never-written channel (state update) then persists the `Overwrite` wrapper itself into the checkpoint, and the next consumer crashes with `TypeError: 'Overwrite' object is not subscriptable` |
+| Scope | Intercepts only the empty-channel plus leading-`Overwrite` case. Later plain values are ignored, and a second `Overwrite` in the same super-step raises `InvalidUpdateError` with `Can receive only one Overwrite value per super-step.`. Everything else delegates to the upstream implementation |
+| Consistency side effect | `DeltaChannel.update` already unwraps in the same situation, so the patch also removes a behavioural inconsistency between the two reducer channel types |
+
+The guard is a **behavioural probe, not a version pin**. The probe builds a `BinaryOperatorAggregate` over a Union type (so the channel starts MISSING) and checks whether reading it yields the `Overwrite` wrapper; if upstream already unwraps, the probe reports the bug as absent and the patch stands down. Application is idempotent, and on unexpected failure it logs `Failed to apply the BinaryOperatorAggregate Overwrite first-write patch; leaving the upstream implementation untouched.` and leaves upstream intact.
+
+<Callout type="warning">
+  Do not re-add a version-guarded saver patch. The previous guard here read
+  `langgraph`'s version, but `InMemorySaver` ships in the independently released
+  `langgraph-checkpoint` distribution, so that guard could not see the real
+  dependency. The former `InMemorySaver` delta-history patch was removed rather
+  than guarded: upstream fixed the dropped first post-migration write in
+  `langgraph-checkpoint` 4.2.0 (upstream #8526) while keeping its own override,
+  which a behavioural probe cannot detect.
+</Callout>
+
+## Dependency floors and the regression gate
+
+The dependency floor is what keeps the removed saver bug out, so the floors are load-bearing rather than cosmetic:
+
+| Distribution | Floor | Why |
+| --- | --- | --- |
+| `langgraph-checkpoint` | `>=4.2.0,<5.0` | 4.2.0 fixes the dropped first post-migration write (upstream #8526). A 4.1.x resolution would silently reintroduce it, and `langgraph`'s own `langgraph-checkpoint>=4.1.0` constraint does not exclude that version |
+| `langgraph-checkpoint-postgres` | `>=3.1.2,<3.2` | 3.1.2 finds plain-value delta seeds (upstream #8535); 3.1.1 walks past them to the thread root |
+| `langgraph-checkpoint-sqlite` | `>=3.1.1,<3.2` | Pinned alongside the other savers |
+
+The regression gate is `backend/tests/test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`: it fails when the 4.2.0 fix is absent, and a full-to-delta migration must keep replaying on the same thread. The pinned environment resolves `langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, and `langgraph-checkpoint-postgres 3.1.2`.
+
+There is no runtime version check or warning. The only version signal an operator sees is a trigger-skipped test naming the pinned version, for example the `#8448` Postgres skip or the `#8382` replay-order skip.
+
+## Running the benchmarks
+
+Two scripts cover the two layers, and each has a summarizer. All ratios they emit are `delta/full`, and timing thresholds are not CI gates.
+
+| Script | Layer | What it measures |
+| --- | --- | --- |
+| `scripts/benchmark/checkpoint/bench_channels.py` | Channel layer | Paired `full` and `delta` message-only StateGraphs in a fresh child process per case, using a sync `InMemorySaver` or `SqliteSaver`, so reducer, serialization, and saver costs stay separate from Gateway scheduling. Postgres cases are opt-in through `TEST_POSTGRES_URI` |
+| `scripts/benchmark/checkpoint/bench_production.py` | Production layer | Graph-level `ainvoke` turns through the real lead-agent graph (scripted deterministic model, real `AsyncSqliteSaver`), then `GET /threads/{id}/state` and `POST /threads/{id}/history` through the real Gateway route stack, split into cold and warm samples |
+
+```bash
+cd backend
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/bench_channels.py \
+  --backends sqlite --updates 100,500,999,1000,1001 --payload-bytes 128 \
+  --repetitions 7 --output /tmp/checkpoint-bench.jsonl
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/summarize_channels.py \
+  /tmp/checkpoint-bench.jsonl
+```
+
+```bash
+cd backend
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/bench_production.py \
+  --turns 10,100,500,1000,2000 --payload-bytes 128 \
+  --snapshot-frequencies 10,50,100,500,1000 \
+  --repetitions 7 --output /tmp/production-bench.jsonl
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/summarize_production.py \
+  /tmp/production-bench.jsonl
+```
+
+The channel controller alternates mode order between cases and rejects performance data when the paired modes materialize different state; the production controller pairs every delta frequency against the same full row and fails both rows of a pair when materialized or wire digests diverge.
+
+### The measured numbers that exist in the repository
+
+Only three measured values are committed, all from the operational-limits notes on the first benchmark runs:
+
+| Observation | Value |
+| --- | --- |
+| Delta mode at `snapshot_frequency=1000`, 500 turns | Roughly 1100–1200 s wall clock, so the default `--timeout-seconds 900` is insufficient. Pass an explicit timeout for any large matrix |
+| Delta mode at `snapshot_frequency=1000`, 2000 turns | Roughly 45 minutes; treat the 2000-turn corner as practical only at small snapshot frequencies |
+| Full mode, 2000 turns, SQLite | Roughly a 33 GB database file. Point `TMPDIR` at real disk, not tmpfs, or the run dies mid-case |
+
+One related route constraint is often mistaken for a benchmark limit: the history route clamps `limit` to 100, so `--history-limits` values above 100 are measured and reported at the clamped limit.
+
+<Callout type="warning">
+  Two classes of number must not be presented as measurements. **Synthetic test
+  fixtures** — the ratio literals the summarizer unit tests feed in, such as
+  `ratio_write_total_ms: 0.5`, `ratio_logical_checkpoint_bytes: 0.2`,
+  `delta_snapshot_write_spike: 5.0`, and `delta_cache_effect_ms: 45.0` —
+  describe fixture inputs, not observed runs. And a storage-reduction ratio
+  quoted for a 200-turn run in the untracked worktree RFC is upstream-derived,
+  not measured here, and the local follow-up plan explicitly forbids citing it
+  for the current configuration — do not use it in performance claims. No
+  benchmark result files are committed at all.
+</Callout>
+
+## TUI and embedded differences
+
+The `deerflow` console script is `deerflow.tui.cli:main`; `--print` and `--json` are headless variations of it, and the rest is a Textual TUI. Both use the embedded client, which differs from the Gateway in ways that matter for delta mode.
+
+| Aspect | Gateway (async) | TUI / embedded (sync) |
+| --- | --- | --- |
+| Checkpointer | The async provider builds and wraps the saver per process | `get_checkpointer()` returns the process-wide **sync** singleton |
+| History cache backend | `memory` and `redis` | `memory` only |
+| Failed agent build | `full` mode degrades to raw checkpointer reads; `delta` re-raises | No degraded path at all |
+| Cross-mode mismatch | HTTP 409 on state and history routes; a run fails with an SSE `error` frame or an error run record | Raised in-process before the agent is built or the graph streams |
+| Freeze | Frozen at Gateway startup from the startup config snapshot | `DeerFlowClient.__init__` freezes mode and cadence from its own `AppConfig`, with the same restart-required semantics |
+
+Three consequences are worth stating plainly:
+
+- **Redis is rejected on the sync path.** Selecting `database.checkpoint_cache.type: redis` raises `ValueError("database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.")` at checkpointer construction, so a configuration that is valid for the Gateway can break the TUI. The rejection is honest rather than a fallback: the sync path is process-local anyway.
+- **There is no degraded read path.** Delta materialization needs the graph's channel table, and the TUI has no equivalent to the Gateway's raw-checkpoint fallback.
+- **An uncaught mismatch is a traceback on the headless runners.** The embedded client raises `CheckpointModeMismatchError` in-process — no HTTP status, no run record, no SSE frame — and the CLI dispatches straight to its print or JSON runner without a handler, so `--print` and `--json` fail with a Python traceback on stderr. The Textual TUI is the exception: its `stream_actions()` helper catches every exception from `client.stream()` and emits `AssistantError` plus `RunEnded`, so an interactive user sees the failure in the UI instead.
+
+Third-party embedders hit a related guard: `create_deerflow_agent` with `checkpoint_channel_mode="delta"` **and** a checkpointer is rejected with `ValueError`, because graphs built there bypass checkpoint-mode marker injection and the fail-closed compatibility gate, so a mixed-mode store would silently corrupt thread state. Delta without a checkpointer is ephemeral and allowed; persisted delta graphs must go through the guarded application paths (`make_lead_agent` or `DeerFlowClient`).
+
+## See also
+
+- [Channel Modes](/docs/harness/checkpoints/modes) — the mode marker, the fail-closed gate, and the HTTP boundary.
+- [Snapshot Cadence](/docs/harness/checkpoints/cadence) — how `snapshot_frequency` trades checkpoint size against materialization cost.
+- [History Cache](/docs/harness/checkpoints/history-cache) — the performance-only cache whose policies are safe to differ across workers.
+- [Troubleshooting](/docs/harness/checkpoints/troubleshooting) — the symptom index for every failure signature above.

frontend/src/content/en/harness/checkpoints/quick-start.mdx

@@ -0,0 +1,73 @@
+---
+title: Quick Start
+description: How to switch DeerFlow checkpoint storage to delta mode — the two config keys and their defaults, why a restart is required, what a fresh install gets, how to confirm the switch took effect, and the supported migration direction.
+---
+
+import { Callout } from "nextra/components";
+
+# Quick Start
+
+Two keys under `database:` select the checkpoint representation and how often delta mode stores a full message snapshot. Both are restart-required.
+
+```yaml
+database:
+  checkpoint_channel_mode: full     # full | delta
+  checkpoint_delta:
+    snapshot_frequency: 10          # >= 1
+```
+
+| Key | Type | Default | Range | Restart required |
+| --- | --- | --- | --- | --- |
+| `database.checkpoint_channel_mode` | `"full"` \| `"delta"` | `"full"` | `full`, `delta` | Yes |
+| `database.checkpoint_delta.snapshot_frequency` | int | `10` | `>= 1`, no documented upper bound | Yes |
+
+No environment variable selects the mode or the cadence; both come only from `config.yaml`. An unknown mode value fails pydantic validation at config load, and so does a `snapshot_frequency` below 1. `snapshot_frequency` applies only in `delta` mode; `config.example.yaml` ships both keys at their defaults.
+
+## Why a restart is required
+
+Both values are frozen on first use: the process stores them in two module globals, and a later value that differs raises `CheckpointModeReconfigurationError`:
+
+- `checkpoint_channel_mode is restart-required and cannot change in a running process`
+- `checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process`
+
+The cadence is frozen alongside the mode because it is not stored in the checkpoint: it is baked into each compiled graph's channel table when the graph is built. A process started with a different cadence would apply that cadence to the same threads, and since the cadence is deliberately not stamped into checkpoint metadata there is nothing at read time to detect the mismatch against. The same reasoning makes the mode a cross-process invariant: every process sharing one checkpoint database must use the same value.
+
+The freeze happens before graph compilation, on every path that builds a graph: the Gateway lifespan, the lead agent assembly, and `DeerFlowClient.__init__`. There is no hot reload for either value.
+
+## What a fresh install gets
+
+| Setting | Value on a fresh install |
+| --- | --- |
+| `database.backend` | `sqlite`, with `sqlite_dir: .deer-flow/data`, when the `database:` section or the key is absent |
+| `database.checkpoint_channel_mode` | `full` |
+| `database.checkpoint_delta.snapshot_frequency` | `10` |
+| `database.checkpoint_graph_cache.accessor_graph_max` | `64` |
+| `database.checkpoint_cache` | Process-local memory LRU: `max_entries: 128`, `redis_url: null`, `ttl_seconds: 86400`, `key_prefix: ""`. The example file leaves the whole block commented out |
+
+A missing `database:` section resolves to `sqlite` plus `full`, not to the in-memory backend: `memory` is only the default of a directly constructed `DatabaseConfig()`. If your `config.yaml` predates the key, `make config-upgrade` merges it in with the safe default `full`.
+
+## Confirming the switch took effect
+
+1. **In-process.** `frozen_checkpoint_channel_mode()` returns the frozen mode, or `None` before the process has frozen one; `frozen_checkpoint_snapshot_frequency()` is its cadence twin. Both live in `deerflow.runtime.checkpoint_mode`.
+2. **On the thread.** A checkpoint written by a delta-mode process carries the metadata marker `deerflow_checkpoint_channel_mode: "delta"`. Full mode removes that key, so absence means full — existing checkpoints need no migration.
+3. **On a mismatch.** If the process is still in `full` mode, opening a delta thread returns HTTP 409 with `Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode.` That is the fail-closed gate working, not an error to silence; see [Channel Modes](/docs/harness/checkpoints/modes).
+
+There is no health, readiness, or capabilities endpoint that reports the mode, so the three checks above are the ones available.
+
+## Migration direction
+
+Delta-mode processes read legacy full checkpoints transparently, so **full → delta is the supported direction**: existing threads keep working with no data rewrite. The reverse direction is not symmetric. From the runtime module docstring:
+
+> a full-mode process opening a delta thread raises CheckpointModeMismatchError instead of silently materializing empty state. Delta-mode processes read legacy full checkpoints transparently, so full -> delta is the supported migration path.
+
+Switching a store back to `full` therefore requires materializing and converting those threads' checkpoints first, which is what the error text tells the operator to do. Until that is done, keep `delta` in every process that touches the store.
+
+<Callout type="warning">
+  Editing `checkpoint_channel_mode` in `config.yaml` without restarting does not take effect: the run
+  worker injects the frozen mode, so the edit is ignored until a restart. Editing
+  `checkpoint_delta.snapshot_frequency` without restarting is louder — the next run fails with
+  `CheckpointModeReconfigurationError` in its run record and SSE `error` frame. Neither edit makes a
+  request return `503`; see
+  [Troubleshooting](/docs/harness/checkpoints/troubleshooting#a-run-fails-after-a-config-edit). Roll
+  either change out by restarting every process that shares the checkpoint database.
+</Callout>

frontend/src/content/en/harness/checkpoints/reference.mdx

@@ -0,0 +1,388 @@
+---
+title: Reference
+description: Every configuration key, constant, exception, HTTP status, public and internal symbol, load-bearing test anchor, pinned upstream defect, and change-log entry behind dual-mode checkpoint storage.
+---
+
+import { Callout } from "nextra/components";
+
+# Reference
+
+Every name, key, string, and test anchor behind `database.checkpoint_channel_mode`. This page is for looking things up, not for learning the model; start with [Channel Modes](/docs/harness/checkpoints/modes) if you need the concepts.
+
+Evidence column values are repo-relative `path:line` references using the aliases below. A fact that appears here is traceable to one of those lines.
+
+| Alias | Path |
+| --- | --- |
+| `dbcfg` | `backend/packages/harness/deerflow/config/database_config.py` |
+| `appcfg` | `backend/packages/harness/deerflow/config/app_config.py` |
+| `reload` | `backend/packages/harness/deerflow/config/reload_boundary.py` |
+| `cmode` | `backend/packages/harness/deerflow/runtime/checkpoint_mode.py` |
+| `cstate` | `backend/packages/harness/deerflow/runtime/checkpoint_state.py` |
+| `tstate` | `backend/packages/harness/deerflow/agents/thread_state.py` |
+| `factory` | `backend/packages/harness/deerflow/agents/factory.py` |
+| `client` | `backend/packages/harness/deerflow/client.py` |
+| `worker` | `backend/packages/harness/deerflow/runtime/runs/worker.py` |
+| `agent` | `backend/packages/harness/deerflow/agents/lead_agent/agent.py` |
+| `goalsrc` | `backend/packages/harness/deerflow/runtime/goal.py` |
+| `checkpoint_patches` | `backend/packages/harness/deerflow/checkpoint_patches.py` |
+| `syncprov` | `backend/packages/harness/deerflow/runtime/checkpointer/provider.py` |
+| `asyncprov` | `backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py` |
+| `cacheprov` | `backend/packages/harness/deerflow/runtime/checkpoint_cache/provider.py` |
+| `threads` | `backend/app/gateway/routers/threads.py` |
+| `thread_runs` | `backend/app/gateway/routers/thread_runs.py` |
+| `runs` | `backend/app/gateway/routers/runs.py` |
+| `services` | `backend/app/gateway/services.py` |
+| `deps` | `backend/app/gateway/deps.py` |
+| `context_usage` | `backend/app/gateway/context_usage.py` |
+| `example` | `config.example.yaml` |
+
+The pinned dependency set referenced by the defect table is `langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, and `langgraph-checkpoint-postgres 3.1.2`.
+
+## Configuration keys
+
+Every `database.*` key related to checkpoints, with the constraint the loader actually enforces. "Restart" means the running process captured the value at startup — frozen into process state, compiled into a channel table, or bound when a singleton such as the checkpointer and its history cache was constructed — so an edit does not take effect until that process restarts.
+
+| Key | Type | Default | Range | Restart | Evidence |
+| --- | --- | --- | --- | --- | --- |
+| `database.backend` | `Literal["memory", "sqlite", "postgres"]` | `"memory"` (pydantic); `sqlite` when the key is absent from `config.yaml` | `memory` \| `sqlite` \| `postgres` | Yes | `dbcfg:143-146`, `appcfg:67-70`, `appcfg:519-527` |
+| `database.checkpoint_channel_mode` | `CheckpointChannelMode` = `Literal["full", "delta"]` | `"full"` | `full` \| `delta` | Yes — frozen on first use; a later different value raises `CheckpointModeReconfigurationError` | `dbcfg:59`, `dbcfg:147-155`, `cmode:39-45` |
+| `database.checkpoint_delta.snapshot_frequency` | `int` | `10` (`DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY`) | `ge=1` (no upper bound) | Yes — frozen alongside the mode | `dbcfg:61`, `dbcfg:74-84`, `cmode:53-69` |
+| `database.checkpoint_graph_cache.accessor_graph_max` | `int` | `64` | `ge=1` | No — hot-reloadable, re-read on every eviction check | `dbcfg:96-99`, `services:1133-1138` |
+| `database.checkpoint_cache.type` | `Literal["memory", "redis"]` | `"memory"` | `memory` \| `redis`; `redis` is async/Gateway only | Yes — captured when the checkpointer is built | `dbcfg:113-116`, `syncprov:181-182`, `asyncprov:250-254` |
+| `database.checkpoint_cache.max_entries` | `int` | `128` | `ge=0`; `0` disables the cache entirely | Yes — captured when the checkpointer is built | `dbcfg:117-121`, `cacheprov:76-84`, `deps:463` |
+| `database.checkpoint_cache.redis_url` | `str \| None` | `None` | any redis URL; when omitted, `DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL`, then `REDIS_URL`, then `redis://localhost:6379/0` | Yes — captured when the checkpointer is built | `dbcfg:122-125`, `cacheprov:19-23` |
+| `database.checkpoint_cache.ttl_seconds` | `int` | `86400` | `ge=0`; `0` explicitly disables expiry | Yes — captured when the checkpointer is built | `dbcfg:126-135` |
+| `database.checkpoint_cache.key_prefix` | `str` | `""` | any string; empty derives a hash of the database identity | Yes — captured when the checkpointer is built | `dbcfg:136-139`, `cacheprov:60-64` |
+| `database.checkpoint_delta_snapshot_frequency` (deprecated, flat) | `int` | none — carried onto the nested key | `ge=1` after migration | n/a | `dbcfg:218-254` |
+
+### What each key affects
+
+- `database.checkpoint_channel_mode` selects how the `messages` channel is stored: whole-value checkpoints (`full`) or a LangGraph `DeltaChannel` sentinel plus per-step writes (`delta`). It affects every read and write of a thread.
+- `database.checkpoint_delta.snapshot_frequency` is the cadence at which delta mode writes a full messages snapshot every N per-step writes. Higher means smaller checkpoints and slower materialization. It is ignored in `full` mode.
+- `database.checkpoint_graph_cache.accessor_graph_max` caps the compiled thread-state accessor graphs a process caches, keyed per assistant, channel mode, and snapshot cadence. Unlike the mode and cadence it is not restart-required, because a different cap only changes when the cache evicts, never graph semantics.
+- `database.checkpoint_cache.*` configures the delta-mode materialized-history cache only. It is performance-only and never frozen — no correctness invariant requires cross-process agreement, and results are identical with it disabled — but it is **not hot-reloadable**: the cache object is constructed with the checkpointer at startup (`deps:463`, `asyncprov:250-254`), inside a `database` section that `reload` registers as startup-only (`reload:47`), so changing any of these fields needs a restart. The memory backend is bounded by `max_entries`, the redis backend by `ttl_seconds` and the server's own maxmemory policy.
+- `database.backend` selects which saver serves the threads (`InMemorySaver`, SQLite, or Postgres). Mode semantics live in the compiled channel table rather than in the saver, so both modes work on all three backends.
+
+The verbatim `Field(description=...)` text for the two frozen keys is worth reading once, because it states the cross-process rule:
+
+```text
+"Checkpoint representation for accumulating channels. 'full' preserves full-value message checkpoints;
+'delta' uses LangGraph DeltaChannel for messages. Restart is required, and all processes sharing one
+checkpoint database must use the same value."
+```
+
+```text
+"DeltaChannel snapshot cadence: a full messages snapshot is stored every N per-step writes
+(higher = smaller checkpoints, slower materialization). Restart is required, and all processes
+sharing one checkpoint database must use the same value."
+```
+
+### Validation
+
+| Input | Result | Evidence |
+| --- | --- | --- |
+| `checkpoint_channel_mode` set to anything outside `full`/`delta` (for example `auto`) | pydantic `ValidationError` at config load | `backend/tests/test_app_config_reload.py:118-120` |
+| `snapshot_frequency` set to `0` or `-1` | pydantic `ValidationError` (constraint `ge=1`) | `dbcfg:76`, `backend/tests/test_app_config_reload.py:131-134` |
+| Deprecated flat key set to `0` or `-1` | pydantic `ValidationError` after the value is carried onto the nested key | `backend/tests/test_app_config_reload.py:154-157` |
+| Unknown keys under `database:` | ignored (pydantic `extra="ignore"`), which is why the legacy-key shim exists | `dbcfg:224-227` |
+| Both the flat key and `checkpoint_delta.snapshot_frequency` set | the nested key wins, with a warning | `dbcfg:218-240` |
+
+### Deprecated flat key and migration warnings
+
+The old top-level `database.checkpoint_delta_snapshot_frequency` is still honored by the pydantic `@model_validator(mode="before")` named `_migrate_legacy_snapshot_frequency`. It logs one of exactly three strings, all operator-visible:
+
+| Condition | Verbatim string | Evidence |
+| --- | --- | --- |
+| Both keys set, `checkpoint_delta` is a dict | `Both database.checkpoint_delta_snapshot_frequency (deprecated) and database.checkpoint_delta.snapshot_frequency are set; the nested key wins.` | `dbcfg:236-238` |
+| `checkpoint_delta` is an already-built `CheckpointDeltaConfig` object | `Ignoring deprecated database.checkpoint_delta_snapshot_frequency because database.checkpoint_delta is already set.` | `dbcfg:246-248` |
+| Legacy value carried forward | `database.checkpoint_delta_snapshot_frequency is deprecated; use database.checkpoint_delta.snapshot_frequency instead. Carried the legacy value (%r) forward.` | `dbcfg:250-253` |
+
+### Environment variables
+
+No environment variable selects the checkpoint mode or its cadence. Both come only from `config.yaml`. The only mode-feature environment variables are the history-cache redis URL fallbacks (`DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL`, then `REDIS_URL`, then `redis://localhost:6379/0`) at `cacheprov:19-23`. `$VAR` interpolation in config values is resolved centrally by `AppConfig.resolve_env_variables()` before `DatabaseConfig` is constructed, so `DatabaseConfig` itself does no environment processing (`dbcfg:13-22`, `appcfg:442`).
+
+### Config version
+
+The schema change shipped with `config_version 27`; `config.example.yaml` is currently `config_version: 47` and a missing version counts as `0` (`example:23`, `appcfg:531-575`). An outdated `config.yaml` logs:
+
+```text
+Your config.yaml (version %d) is outdated — the latest version is %d. Run `make config-upgrade` to merge new fields into your config.
+```
+
+`make config-upgrade` merges the new key with the safe default `full` (`appcfg:570-575`, `backend/tests/test_config_version.py:151-203`). See [Configuration](/docs/harness/configuration) for the general upgrade flow.
+
+## Constants and keys
+
+| Symbol | Value | Where it lives | Evidence |
+| --- | --- | --- | --- |
+| `CheckpointChannelMode` | `Literal["full", "delta"]` | `deerflow.config.database_config` | `dbcfg:59` |
+| `DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY` | `10` | `deerflow.config.database_config` | `dbcfg:61` |
+| `INTERNAL_CHECKPOINT_MODE_KEY` | `"__deerflow_checkpoint_channel_mode"` (a `RunnableConfig` `configurable` key) | `deerflow.runtime.checkpoint_mode` | `cmode:18` |
+| `CHECKPOINT_MODE_METADATA_KEY` | `"deerflow_checkpoint_channel_mode"` (a checkpoint metadata key; the value written is `"delta"`) | `deerflow.runtime.checkpoint_mode` | `cmode:19`, `cmode:86` |
+| `counters_since_delta_snapshot` | LangGraph's own metadata key; a dict containing `"messages"` is treated as delta even without the DeerFlow marker | upstream `langgraph-checkpoint`; detected in `deerflow.runtime.checkpoint_mode` | `cmode:97-98` |
+
+Marker semantics, in one place: `inject_checkpoint_mode(config, mode)` always sets `config["configurable"]["__deerflow_checkpoint_channel_mode"]`. In delta mode it additionally sets `config["metadata"]["deerflow_checkpoint_channel_mode"] = "delta"`; in full mode it **removes** that metadata key, so absence means full and pre-feature checkpoints need no migration (`cmode:81-88`). The snapshot cadence is deliberately **not** stamped into checkpoint metadata, because the mode marker contract and the full-to-delta migration semantics are unchanged by the frequency value (`cmode:53-60`).
+
+## Exceptions and HTTP status
+
+| Class | Base | Docstring | Evidence |
+| --- | --- | --- | --- |
+| `CheckpointModeMismatchError` | `RuntimeError` | `"Raised before a full-mode graph reads a Delta checkpoint."` | `cmode:22-23` |
+| `CheckpointModeReconfigurationError` | `RuntimeError` | `"Raised when a process attempts to hot-switch its persistence mode."` | `cmode:26-27` |
+
+Verbatim messages raised by the mode machinery and its guards:
+
+| Message (verbatim) | Raised by | Evidence |
+| --- | --- | --- |
+| `checkpoint_channel_mode is restart-required and cannot change in a running process` | `freeze_checkpoint_channel_mode` → `CheckpointModeReconfigurationError` | `cmode:44` |
+| `checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process` | `freeze_checkpoint_snapshot_frequency` → `CheckpointModeReconfigurationError` | `cmode:68` |
+| `snapshot frequency must be positive` | `freeze_checkpoint_snapshot_frequency` → `ValueError`, when a non-positive value bypasses pydantic | `cmode:63-64` |
+| `Thread requires delta mode; materialize and convert its checkpoints before using full mode.` | `CheckpointModeMismatchError`, from `raise_if_snapshot_incompatible` and `raise_if_checkpoint_tuple_incompatible`, and duplicated in the degraded gateway accessor | `cmode:123`, `cmode:129`, `services:1229` |
+| `database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.` | `ValueError` in `_wrap_sync_if_delta` | `syncprov:181-182` |
+| `f"Unknown checkpoint cache type: {config.type!r}"` | `ValueError` in `make_checkpoint_cache` | `cacheprov:102` |
+| `create_deerflow_agent does not support checkpoint_channel_mode='delta' with a checkpointer: persisted graphs built here bypass checkpoint mode marker injection and the fail-closed compatibility gate (see deerflow.runtime.checkpoint_mode), so a mixed-mode store would silently corrupt thread state.  Use the guarded application paths (make_lead_agent or DeerFlowClient) for delta persistence; delta without a checkpointer is ephemeral and allowed.` | `ValueError` in `create_deerflow_agent` | `factory:135-141` |
+
+### HTTP status mapping
+
+Only the threads router imports these errors. It registers the gate tuple `_CHECKPOINT_MODE_ERRORS = (CheckpointModeMismatchError, CheckpointModeReconfigurationError)` (`threads:82`) and maps it as follows. Bodies are FastAPI's default `{"detail": "<string>"}`.
+
+| Condition | Status | `detail` | Evidence |
+| --- | --- | --- | --- |
+| `CheckpointModeMismatchError` | `409` | `f"Thread {thread_id}: {exc}"`, i.e. `Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode.` | `threads:105-106` |
+| `CheckpointModeReconfigurationError` | `503` | `str(exc)` | `threads:107` |
+
+The router states the rationale: a mismatch means the thread's persisted checkpoints conflict with the process's frozen mode (operator-actionable, 409); a reconfiguration means the process itself is mid mode-flip (transient, 503) (`threads:99-102`). The `503` half is defensive — nothing in these routes can raise it. The freeze helpers are called only by Gateway startup (`deps:451-452`), `agent:835,839`, and `client:230-231`, and no state, history, or thread route builds an agent: a `checkpoint_channel_mode` edit is injected over by the run worker (`worker:1018-1019`, preferred at `agent:831-835`) and ignored until a restart, and a `snapshot_frequency` edit makes the next run fail with the message (`worker:1450-1462`) instead of returning a status.
+
+### Routes that return 409
+
+| Route | Handler | Notes | Evidence |
+| --- | --- | --- | --- |
+| `GET /api/threads/{thread_id}` | `get_thread` | checked before the `404 Thread <id> not found` | `threads:1276,1291-1292,1298-1299,1303` |
+| `GET /api/threads/{thread_id}/state` | `get_thread_state` | | `threads:1446,1454-1455,1458-1459` |
+| `POST /api/threads/{thread_id}/state` | `update_thread_state` | a client-supplied `checkpoint_id` older than the head is applied as a delta fork with no linearization — see [Forks are accepted, not refused](/docs/harness/checkpoints/resume-and-rollback#forks-are-accepted-not-refused) | `threads:1495,1559-1560` |
+| `POST /api/threads/{thread_id}/history` | `get_thread_history` | | `threads:1695,1715-1716,1907-1908` |
+| `POST /api/threads/{thread_id}/branches` | `branch_thread` → `_branch_thread_with_reservation` | the detail names the source thread id; no test pins this route | `threads:953,231-232,298-299,1105-1106` |
+| `POST /api/threads/{thread_id}/compact` | `compact_thread` | `409` is also returned when compaction is disabled; no test pins this route | `threads:1399,1413-1414,1429-1430,1431-1432` |
+
+### Routes where the mismatch fails asynchronously
+
+| Route | Immediate response | Failure payload | Evidence |
+| --- | --- | --- | --- |
+| `POST /api/threads/{thread_id}/runs` | `200` run record, no pre-gate — but `500` when the thread's run-event feed is empty, so `ensure_checkpoint_history_seeded` materializes the head inside `start_run` first | the run record later has `status: error` and the mismatch text as `error` | `thread_runs:945-960`, `worker:1018-1030,1044-1047`, `worker:1450-1462`, `services:1503-1520,2055` |
+| `POST /api/threads/{thread_id}/runs/stream` | `200` `text/event-stream` | an `event: error` frame whose `data` is `{"message": <mismatch text>, "name": "CheckpointModeMismatchError"}` | `worker:1466-1472`, `services:2374-2378` |
+| `POST /api/threads/{thread_id}/runs/wait` | `200` JSON, never `409` | `{"status": "error", "error": "Thread requires delta mode; materialize and convert its checkpoints before using full mode."}` | `thread_runs:1018-1076` |
+| `POST /api/runs/stream`, `POST /api/runs/wait` (stateless) | same as above | same `{"status", "error"}` body | `runs:59-91` |
+
+### Routes with no mode mapping
+
+| Route | Actual behaviour | Evidence |
+| --- | --- | --- |
+| `POST /api/threads/{thread_id}/runs/regenerate/prepare` | `500` with `detail: "Failed to read latest checkpoint"` — the accessor build sits outside any `try` | `thread_runs:747,749-752` |
+| same route, lineage/history scan | `500` with `detail: "Failed to inspect checkpoint history"` | `thread_runs:679-681` |
+| `POST /api/threads/{thread_id}/runs/edit-regenerate/prepare` | the same two `500` paths | `thread_runs:838-841`, `thread_runs:633-697` |
+| `POST /api/threads/{thread_id}/runs`, `/runs/stream`, `/runs/wait`, and the stateless `/api/runs/*` routes, when the thread's run-event feed is empty and a checkpoint head exists | `500`: `ensure_checkpoint_history_seeded` reads that head through the gated accessor before admission, and `start_run` maps only `ConflictError` (`409`) and `UnsupportedStrategyError` (`501`) | `services:1503-1520`, `services:2055`, `services:2126-2129` |
+| `GET /api/threads/{thread_id}/token-usage` | `200` with `context_usage: null` and a warning log only | `context_usage.py:73-78`, `thread_runs:1872-1873` |
+| `GET` / `PUT` / `DELETE /api/threads/{thread_id}/goal` | not gated at all: raw `checkpointer.aget_tuple` / `aput` on `channel_values["goal"]` | `threads:1329-1383`, `goalsrc:442-453,468-477,480-542` |
+| `DELETE /api/threads/{thread_id}`, `POST /search`, `PATCH`, `POST /move` | metadata store only, so mode-blind | `threads:706,1173,1216,1255` |
+
+### No reporting surface
+
+No health, readiness, features, capabilities, or console field reports the mode or cadence. No response model, header, or event field carries it either: the only custom SSE headers are `Cache-Control`, `Connection`, `X-Accel-Buffering`, and `Content-Location`. The mode is consumed server-side only, through `app.state.checkpoint_channel_mode`, `app.state.checkpoint_snapshot_frequency`, and `RunContext` (`deps:451-452,767-768`, `services:1093,1283-1312`). A latent exception is that the response `metadata` on `GET /api/threads/{thread_id}` and on `POST /api/threads/{thread_id}/history` strips other internal keys but not the delta marker or `counters_since_delta_snapshot`, so those can pass through to a client (`threads:1308-1315`, `threads:1890-1900`). The SSE `metadata` frame carries only `{"run_id", "thread_id"}` (`worker:1067-1074`). See [Observability](/docs/harness/checkpoints/observability).
+
+## Public symbols
+
+Import paths a consumer may rely on. Everything listed here is a Python import; the route contracts are in the HTTP status mapping section above.
+
+| Symbol | Import path | Role | Evidence |
+| --- | --- | --- | --- |
+| `CheckpointChannelMode` | `deerflow.config.database_config` | mode literal | `dbcfg:59` |
+| `DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY` | `deerflow.config.database_config` | default cadence, `10` | `dbcfg:61` |
+| `CheckpointDeltaConfig` | `deerflow.config.database_config` | delta tuning model (`snapshot_frequency`) | `dbcfg:64` |
+| `CheckpointGraphCacheConfig` | `deerflow.config.database_config` | graph-cache cap model (`accessor_graph_max`) | `dbcfg:86` |
+| `CheckpointCacheConfig` | `deerflow.config.database_config` | history-cache policy model | `dbcfg:103` |
+| `DatabaseConfig` | `deerflow.config.database_config` | the whole `database:` section | `dbcfg:142` |
+| `resolve_checkpoint_graph_cache_max(database_config, field_name, default)` | `deerflow.config.database_config` | tolerant hot-reloadable cap reader | `dbcfg:45-57` |
+| `INTERNAL_CHECKPOINT_MODE_KEY` | `deerflow.runtime.checkpoint_mode` | configurable key | `cmode:18` |
+| `CHECKPOINT_MODE_METADATA_KEY` | `deerflow.runtime.checkpoint_mode` | checkpoint metadata key | `cmode:19` |
+| `CheckpointModeMismatchError` | `deerflow.runtime.checkpoint_mode` | fail-closed cross-mode error | `cmode:22` |
+| `CheckpointModeReconfigurationError` | `deerflow.runtime.checkpoint_mode` | restart-required violation error | `cmode:26` |
+| `frozen_checkpoint_channel_mode()` | `deerflow.runtime.checkpoint_mode` | read the frozen mode or `None` | `cmode:34-36` |
+| `freeze_checkpoint_channel_mode(mode)` | `deerflow.runtime.checkpoint_mode` | freeze on first call, reject changes | `cmode:39-45` |
+| `frozen_checkpoint_snapshot_frequency()` | `deerflow.runtime.checkpoint_mode` | read the frozen cadence or `None` | `cmode:48-50` |
+| `freeze_checkpoint_snapshot_frequency(n)` | `deerflow.runtime.checkpoint_mode` | freeze cadence; `ValueError` if non-positive | `cmode:53-69` |
+| `resolve_checkpoint_snapshot_frequency(n=None)` | `deerflow.runtime.checkpoint_mode` | explicit → frozen → default | `cmode:72-78` |
+| `inject_checkpoint_mode(config, mode)` | `deerflow.runtime.checkpoint_mode` | stamp the configurable key and metadata marker | `cmode:81-88` |
+| `checkpoint_metadata_uses_delta(metadata)` | `deerflow.runtime.checkpoint_mode` | marker/counters detection | `cmode:91-98` |
+| `checkpoint_tuple_uses_delta(tuple)` | `deerflow.runtime.checkpoint_mode` | tuple-level detection | `cmode:101-104` |
+| `state_snapshot_uses_delta(snapshot)` | `deerflow.runtime.checkpoint_mode` | snapshot-level detection | `cmode:107-111` |
+| `raise_if_snapshot_incompatible(snapshot, mode)` | `deerflow.runtime.checkpoint_mode` | read gate | `cmode:114-123` |
+| `raise_if_checkpoint_tuple_incompatible(tuple, mode)` | `deerflow.runtime.checkpoint_mode` | raw-tuple gate | `cmode:126-129` |
+| `ensure_checkpoint_mode_compatible(checkpointer, config, mode)` | `deerflow.runtime.checkpoint_mode` | sync pre-write gate | `cmode:132-140` |
+| `aensure_checkpoint_mode_compatible(checkpointer, config, mode)` | `deerflow.runtime.checkpoint_mode` | async pre-write gate | `cmode:143-146` |
+| `ThreadState` | `deerflow.agents.thread_state` | full-mode state schema | `tstate:280` |
+| `DeltaThreadState` | `deerflow.agents.thread_state` | delta schema at the default cadence | `tstate:396-397,423-426` |
+| `delta_messages_field(snapshot_frequency=DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY)` | `deerflow.agents.thread_state` | `Annotated[list[AnyMessage], DeltaChannel(merge_message_writes, snapshot_frequency=…)]` | `tstate:385-390` |
+| `DELTA_MESSAGES_FIELD` | `deerflow.agents.thread_state` | module-level default-cadence delta field | `tstate:393` |
+| `merge_message_writes(state, writes)` | `deerflow.agents.thread_state` | linear-time `add_messages`-equivalent reducer | `tstate:328-382` |
+| `get_thread_state_schema(mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | `ThreadState` for non-delta, cached delta schema otherwise | `tstate:416-419` |
+| `adapt_state_schema_for_mode(schema, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | middleware/state schema adaptation | `tstate:437-440` |
+| `normalize_middleware_state_schemas(middleware, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | middleware list with adapted `state_schema` copies | `tstate:454-467` |
+| `THREAD_STATE_REDUCER_FIELDS` | `deerflow.agents.thread_state` | frozenset of reducer channels (including `"messages"`), used to decide `Overwrite` wrapping | `tstate:400-413`, `threads:1065-1066` |
+| `create_deerflow_agent(..., checkpoint_channel_mode="full", checkpoint_snapshot_frequency=None, ...)` | `deerflow.agents.factory` | public agent factory; delta plus a checkpointer is rejected | `factory:66-77`, `factory:135-141` |
+| `CheckpointStateAccessor`, `build_state_mutation_graph` | `deerflow.runtime` (package re-export) | accessor choke point and mutation graph | `deerflow/runtime/__init__.py:8,21-22` |
+| `ThreadState`, `DeltaThreadState`, `SandboxState` | `deerflow.agents` (package re-export) | lazily exported state schemas | `agents/__init__.py:14-16,40-47` |
+| `CheckpointStateAccessor.bind(graph, checkpointer, *, store=None, mode="full")` | `deerflow.runtime.checkpoint_state` | bind graph, saver, and mode | `cstate:112-134` |
+| `build_state_mutation_graph(as_node, mode, state_schema=None, *, snapshot_frequency=None)` | `deerflow.runtime.checkpoint_state` | state-only writer graph for `Overwrite`-style writes | `cstate:39-65` |
+
+<Callout type="warning">
+  In delta mode a raw saver read sees sentinels, not messages. Consumers must read
+  state through `CheckpointStateAccessor` instead of calling the checkpointer
+  directly; the accessor docstring says so explicitly.
+</Callout>
+
+## Internal symbols
+
+These are implementation details. Do not import them, and do not build on their names or shapes.
+
+| Symbol | Why it is not API | Evidence |
+| --- | --- | --- |
+| `_frozen_checkpoint_channel_mode`, `_frozen_checkpoint_snapshot_frequency` | module globals, only touched through the freeze/frozen helpers; tests monkeypatch them | `cmode:30-31`, `backend/tests/conftest.py:96-111` |
+| `_delta_thread_state_schema` | `@cache`-decorated internal builder; generates `DeltaThreadState_f{freq}` | `tstate:422-434` |
+| `_adapt_state_schema_for_delta` | `@cache`-decorated internal adapter; generates `Delta{module_underscored}_{Name}_f{freq}` | `tstate:443-451` |
+| `_resolve_snapshot_frequency` | internal lazy-import resolver | `tstate:25-33` |
+| `_STATE_ACCESSOR_GRAPH_CACHE_MAX` | in-code fallback constant `64`, used only when the configured value is unusable | `services:1127,1133-1138` |
+| `_RawCheckpointReadAccessor`, `_RawCheckpointSnapshot` | degraded full-mode read-only fallback | `services:1199-1263` |
+| `_checkpoint_mode_http_error`, `_CHECKPOINT_MODE_ERRORS` | router-internal HTTP mapping | `threads:82,96-107` |
+| `_wrap_sync_if_delta` | sync provider internal | `syncprov:164-192` |
+| `_migrate_legacy_snapshot_frequency` | pydantic `mode="before"` validator, labelled `-- Legacy key migration (not user-configured) --` | `dbcfg:216-254` |
+| `_finish_state_mutation` | no-op writer node | `cstate:32-33` |
+
+## Load-bearing tests
+
+Tests a reader can run or cite as the executable contract. All paths are under `backend/tests/`.
+
+### Mode gate, freezing, and configuration
+
+- `test_checkpoint_mode.py` (freeze, detection, gate) — the module named as the mode gate in `backend/packages/harness/deerflow/runtime/AGENTS.md:133`
+- `test_checkpoint_mode.py::test_yaml_mode_change_is_rejected_when_graph_is_reconstructed`
+- `test_checkpoint_mode.py::test_delta_mode_accepts_plain_full_checkpoint`
+- `test_checkpoint_mode.py::test_gateway_runtime_rejects_mode_different_from_frozen_process`
+- `test_threads_checkpoint_mode.py` (router-level mode behaviour)
+- `test_threads_checkpoint_mode.py::test_delta_snapshot_frequency_controls_snapshot_cadence`
+- `test_gateway_checkpoint_mode.py` (dual-mode end-to-end parity)
+- `test_app_config_reload.py::test_checkpoint_channel_mode_rejects_unknown_value`
+- `test_config_version.py` (config-version migration)
+
+### Materialization, storage shape, and migration
+
+- `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`
+- `test_delta_channel_checkpointers.py::test_materialization_is_deterministic_and_message_ids_are_stable`
+- `test_delta_channel_checkpointers.py::test_delta_storage_shape_and_snapshot_cadence`
+- `test_delta_channel_checkpointers.py::test_non_delta_writers_preserve_delta_messages_and_markers`
+- `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination`
+- `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order`
+- `test_delta_channel_state.py::test_mode_selects_expected_state_schema`
+- `test_delta_channel_state.py::test_compiled_delta_graph_bakes_configured_snapshot_frequency`
+- `test_delta_channel_state.py::test_production_message_forms_keep_assigned_ids_across_delta_replay`
+- `test_delta_channel_state.py::test_delta_normalization_compiles_stable_channel_without_mutating_middleware`
+
+`test_long_chain_history_survives_pagination` and `test_parallel_superstep_replay_matches_live_write_order` assert the correct contract and trigger-skip while the pinned dependency still has the defect; they are the gates that flip live on a dependency bump.
+
+### History cache
+
+- `test_cached_history_saver.py::test_composition_matches_full_walk_and_avoids_it`
+- `test_cached_history_saver.py::test_latest_config_caches_under_resolved_checkpoint_id`
+- `test_cached_history_saver.py::test_recursive_resolve_caches_intermediate_levels`
+- `test_cached_history_saver.py::test_deep_cold_chain_delegates_one_inner_walk_at_depth_limit`
+- `test_cached_history_saver.py::test_cold_resolve_never_delegates_to_inner_history`
+- `test_cached_history_saver.py::test_sync_cold_resolve_never_delegates_to_inner_history`
+- `test_cached_history_saver.py::test_snapshot_parent_composes_without_parent_history_lookup`
+- `test_cached_history_saver.py::test_root_checkpoint_history_is_empty_writes`
+- `test_cached_history_saver.py::test_eviction_falls_back_to_walk_but_stays_correct`
+- `test_cached_history_saver.py::test_sync_path_matches_async`
+- `test_cached_history_saver.py::test_stats_expose_composition_counters`
+- `test_cached_history_saver.py::test_adelete_thread_purges_only_that_threads_cache_entries`
+- `test_cached_history_saver.py::test_sync_delete_thread_purges_cache_entries`
+- `test_cached_history_saver.py::test_prune_purges_rewritten_threads_cache_entries`
+- `test_cached_history_saver.py::test_delete_for_runs_delegates_without_cache_purge`
+- `test_cached_history_saver_integration.py::test_sequential_run_composes_without_inner_walks`
+- `test_cached_history_saver_integration.py::test_cache_disabled_parity`
+- `test_cached_history_saver_integration.py::test_branch_divergence_no_cross_contamination`
+- `test_cached_history_saver_integration.py::test_interrupt_resume_appended_head_writes`
+- `test_cached_history_saver_integration.py::test_eviction_only_costs_performance`
+- `test_cached_history_saver_integration.py::test_rollback_supersede_does_not_pollute`
+- `test_cached_history_saver_integration.py::test_full_to_delta_migration_under_cache_states`
+
+### Resume linearization and rollback
+
+- `test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head`
+- `test_run_worker_delta_resume.py::test_linearization_restores_all_selected_state_and_clears_newer_channels`
+- `test_run_worker_delta_resume.py::test_linearized_resume_preserves_selected_checkpoint_agent_binding`
+- `test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer`
+- `test_run_worker_delta_resume.py::test_run_agent_streams_from_the_linearized_delta_resume`
+- `test_run_worker_delta_resume.py::test_run_agent_serializes_resume_preparation_with_checkpoint_writes`
+- `test_run_worker_delta_resume.py::test_cancelled_delta_resume_rolls_back_to_pre_linearization_head`
+- `test_run_worker_delta_resume.py::test_full_mode_keeps_the_fork`
+- `test_run_worker_delta_resume.py::test_ordinary_run_without_a_checkpoint_selector_is_untouched`
+- `test_run_worker_delta_resume.py::test_selecting_the_head_is_already_linear`
+- `test_run_worker_delta_resume.py::test_unmaterializable_resume_state_fails_closed`
+- `test_run_worker_rollback.py::test_rollback_linearizes_delta_restore_onto_cancelled_head`
+- `test_run_worker_rollback.py::test_rollback_linearizes_delta_restore_sqlite_reopen`
+- `test_run_worker_rollback.py::test_rollback_restores_pre_run_pending_writes_for_delta_checkpoints`
+- `test_run_worker_rollback.py::test_rollback_preserves_middleware_contributed_channels[delta]`
+- `test_run_worker_rollback.py::test_rollback_preserves_agent_binding_for_manual_compaction[delta]`
+- `test_run_worker_rollback.py::test_run_agent_marks_rollback_unusable_when_capture_fails`
+
+### Retention
+
+- `test_checkpoint_retention_contract.py::test_growth_baseline_full_vs_delta`
+- `test_checkpoint_retention_contract.py::test_deleting_branch_ancestor_breaks_lineage_loudly`
+- `test_checkpoint_retention_contract.py::test_deleting_explicit_resume_target_breaks_resume`
+- `test_checkpoint_retention_contract.py::test_pending_writes_are_retained_state_not_garbage`
+- `test_checkpoint_retention_contract.py::test_leaf_duration_checkpoint_deletion_is_safe`
+- `test_checkpoint_retention_contract.py::test_leaf_sibling_branch_deletion_is_safe`
+
+### Patches
+
+- `test_checkpoint_patches.py::test_binop_overwrite_patch_is_active`
+- `test_checkpoint_patches.py::test_binop_overwrite_patch_stands_down_when_upstream_fixed`
+
+## Pinned upstream defects
+
+State at the pinned set (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`). Each local test asserts the correct contract and trigger-skips while the defect is present, so a later dependency bump flips it into a live gate; the buggy behaviour is never asserted as correct.
+
+| id | Kind / status | Scenario that breaks | Local handling | Pinned version | Evidence |
+| --- | --- | --- | --- | --- | --- |
+| `#4380` | DeerFlow issue; patched locally | A replace-style write into an empty Union-typed `BinaryOperatorAggregate` channel persists the `Overwrite` wrapper; the next read crashes with `TypeError: 'Overwrite' object is not subscriptable` | Probe-guarded patch in `checkpoint_patches.py`; patch tests; branch/state-update anchors in `test_threads_router.py` | `langgraph 1.2.9` | `checkpoint_patches:91-107`, `backend/tests/test_checkpoint_patches.py:31-95` |
+| `#8526` (fixes upstream `#8384`), plus postgres `#8535` | upstream langgraph PR; merged 2026-08-07; fixed in `langgraph-checkpoint` 4.2.0 | After switching a thread from `BinaryOperatorAggregate` to `DeltaChannel`, the first post-migration write survives the live `invoke` return value but disappears from every subsequent read; on postgres, with no seed the walk runs to the root and replays everything | Local `InMemorySaver` patch deleted; dependency floor `langgraph-checkpoint>=4.2.0,<5.0`; regression gate `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`; a version-guarded saver patch must not be re-added | `langgraph-checkpoint 4.2.0` | `checkpoint_patches:11-17`, `backend/packages/harness/pyproject.toml:49-56`, `CHANGELOG.md:3040-3048` |
+| `#8382` | upstream langgraph issue; **open**; fix PR `#8544` unmerged, no released version | Delta replay order diverges from live execution order for parallel-superstep writes: live `invoke` returns path-sorted `[a..h]`, `get_state` replays `[f,b,g,h,a,c,d,e]` (a pure permutation); reproduced on memory, sqlite, and postgres | Trigger-skip naming the issue; `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.2.0` | `test_delta_channel_checkpointers.py:419,440-460` |
+| `#8448` | upstream langgraph issue; **open**; fix PR `#8556` unmerged | `PostgresSaver.get_delta_channel_history` permanently poisons the walk cursor when the target checkpoint is not in the first pagination page, so old checkpoints hydrate empty; only Postgres pages its stage-1 scan | Assertion stays live on memory/sqlite (differential against an `InMemorySaver` oracle); Postgres trigger-skips; `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` | `langgraph-checkpoint-postgres 3.1.2` | `test_delta_channel_checkpointers.py:459` |
+| `#8548` | upstream langgraph PR; **open**; fixes upstream `#8443`, with related issue `#8551` still waiting on it | Does not replay an abandoned branch into a `DeltaChannel` fork | While unmerged the local linearization cannot be removed: `_linearize_delta_checkpoint_resume` is retained | `langgraph 1.2.9` | `worker:2267-2279`, `test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head` |
+| `#4458` | DeerFlow issue; the user-visible symptom of the fork bug | Regenerating in a branched thread resurrected the old assistant message beside the new one after a reload; reproduced on postgres, sqlite, and the in-memory saver, and it also broke branch history when seeded rows shared one run id | Linearization plus per-turn seeded run ids; `test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer` | `langgraph 1.2.9` | `worker:2267-2289`, `test_run_worker_delta_resume.py:204` |
+| `#4189` item 3 | DeerFlow design discussion | Retention and deletion of checkpoint rows; what may be deleted | `backend/docs/checkpoint-retention-contract.md`, `backend/tests/test_checkpoint_retention_contract.py`, `backend/app/gateway/checkpoint_retention.py`; no production trigger is wired | — | `checkpoint_retention.py:1-40`, `backend/docs/checkpoint-retention-contract.md:31-77` |
+| `#3859` (review) | DeerFlow review reference | An interrupted-title write must bump `channel_versions["title"]` and declare it in `new_versions`, or database savers drop the blob | `backend/tests/test_run_worker_rollback.py:3501-3504` pins the shape | — | `backend/tests/test_run_worker_rollback.py:3501-3504` |
+
+See [Operating Checkpoints](/docs/harness/checkpoints/operations) for what the remaining patch does and how the dependency floor is enforced, and [Troubleshooting](/docs/harness/checkpoints/troubleshooting) for the symptoms each defect produces for a user.
+
+## Change log (June to September 2026)
+
+Pull requests whose changelog entries concern dual-mode checkpoint storage. English entries under `## [Unreleased]` accumulate toward milestone 2.1.0; the `## [2.0.0]` entries are from the 2026-06-15 release. The section column names the changelog section each entry sits under.
+
+| PR | Changelog section | Change |
+| --- | --- | --- |
+| [#4516](https://github.com/bytedance/deer-flow/pull/4516) | `### ⚠ Breaking changes` | The config key moved from `database.checkpoint_delta_snapshot_frequency` to `database.checkpoint_delta.snapshot_frequency`, and the default changed from `1000` to `10`. The legacy top-level value is still honored with a deprecation warning and an explicit nested key wins. Threads now snapshot 100x more often in delta mode; set `…snapshot_frequency: 1000` explicitly to keep the previous cadence. |
+| [#4292](https://github.com/bytedance/deer-flow/pull/4292) | `### Added` | Dual-mode checkpoint storage with LangGraph `DeltaChannel` cuts thread storage from O(N²) to near-linear for long research and coding runs. |
+| [#4638](https://github.com/bytedance/deer-flow/pull/4638) | `### Added` | Delta-mode checkpoint history cache (memory/redis) with O(1) incremental composition, configured via `database.checkpoint_cache`. |
+| [#4437](https://github.com/bytedance/deer-flow/pull/4437), [#4460](https://github.com/bytedance/deer-flow/pull/4460), [#4468](https://github.com/bytedance/deer-flow/pull/4468) | `### Fixed` | Serialize checkpoint writes with active runs, linearize delta-mode checkpoint resume, and accept the SDK's default `stream_resumable=false` to avoid resume races. |
+| [#4383](https://github.com/bytedance/deer-flow/pull/4383) | `### Fixed` | **checkpoint:** unwrap `Overwrite` first writes into empty channels. |
+| [#5255](https://github.com/bytedance/deer-flow/pull/5255) | `### Fixed` | Checkpoint-retention contract suite: per-step growth baselines for both full and delta schemas, deleting a branch ancestor fails loudly with `CheckpointLineageError`, and pending writes are treated as retained state. |
+| [#4140](https://github.com/bytedance/deer-flow/pull/4140) | `### Security` | **storage:** stop persisting base64 image data in checkpoint state. Adjacent rather than mode-specific. |
+| [#5734](https://github.com/bytedance/deer-flow/pull/5734) | `### Internal` | Raise `langgraph-checkpoint` to `>=4.2.0,<5.0` and `langgraph-checkpoint-postgres` to `>=3.1.2,<3.2`, and drop the `InMemorySaver` delta-history patch: upstream 4.2.0 fixes the dropped first write after full→delta (`langgraph#8526`) and postgres locates plain-value seeds (`#8535`). The migration contract test remains the gate. |
+| [#4395](https://github.com/bytedance/deer-flow/pull/4395) | `### Internal` | **bench:** add an isolated checkpoint channel-mode benchmark comparing `full` and `delta` across latency, storage, and replay metrics. |
+| [#5051](https://github.com/bytedance/deer-flow/pull/5051) | `### Internal` | **bench:** measure Postgres checkpoint, blob, and write storage growth in the checkpoint benchmark alongside memory and SQLite. |
+| [#2582](https://github.com/bytedance/deer-flow/pull/2582) | `## [2.0.0]` → `### Fixed` → `#### Runtime, gateway & persistence` | Rollback restore checkpoint now supersedes newer checkpoints. |
+| [#3559](https://github.com/bytedance/deer-flow/pull/3559) | `## [2.0.0]` → `### Fixed` | **subagent:** isolate subagents from the parent run's checkpointer. |
+
+No entry in this list records a change to the mode default, which has been `full` throughout.
+
+The `#4292` row quotes the released changelog entry verbatim. Its "near-linear" wording describes the per-step representation; retained storage still adds up to O(N²/K + N) with K = `snapshot_frequency`, because every periodic snapshot holds the whole growing list — see [Checkpoints and Delta Channels](/docs/harness/checkpoints).

frontend/src/content/en/harness/checkpoints/resume-and-rollback.mdx

@@ -0,0 +1,140 @@
+---
+title: Resume and Rollback
+description: Delta mode cannot fork a thread correctly — a forked delta read replays the abandoned branch's writes. How DeerFlow linearizes a checkpoint resume onto the current head instead, when the rewrite is a no-op, how it fails closed, and what rollback captures first.
+---
+
+import { Callout } from "nextra/components";
+
+# Resume and Rollback
+
+Resuming from an older checkpoint is a fork. In `full` mode that is a normal, well-defined operation. In `delta` mode it produces a wrong read rather than an error — nothing refuses the fork: LangGraph stores the second child, and the delta-mode pre-write gate returns before any compatibility check — so DeerFlow rewrites the fork as a linear write on the current head before the run starts. This chapter covers that rewrite, its failure mode, the rollback point captured around it, the write path it does not cover, and how manual context compaction writes state in the same mode.
+
+## Why a delta run cannot fork correctly
+
+A delta checkpoint stores no full channel values, so its state is reconstructed by replaying ancestor writes. That walk collects **every** `pending_writes` entry stored on each ancestor lying on the path to the target checkpoint — and a shared parent also carries the writes of the sibling child that was abandoned.
+
+```
+            ┌─ C1   abandoned sibling; its writes live under P
+P (shared) ─┤
+            └─ C2   current head
+```
+
+Resuming from `P` means "start again from P's state". But `C1`'s writes are still stored under `P`, so the walk replays them into the new fork: the run starts from a message list that still contains the answer it was supposed to replace.
+
+| Mode | Checkpoint content | Fork behaviour |
+| --- | --- | --- |
+| `full` | Complete `channel_values`, including the whole `messages` list | Fork materializes correctly; LangGraph's branching semantics apply unchanged |
+| `delta` | A `_DeltaSnapshot` sentinel plus per-step writes rows | A shared parent replays the abandoned sibling's writes into the fork |
+
+DeerFlow reproduced this on Postgres, SQLite, and the in-memory saver. The user-visible symptom was **DeerFlow #4458**: regenerating in a branched thread showed the old assistant message reappearing beside the new one after a reload.
+
+Upstream tracks the same defect. PR **#8548** ("don't replay an abandoned branch into a `DeltaChannel` fork") fixes upstream issue **#8443** and is still open; while it is unmerged, the local linearization below cannot be removed.
+
+### Forks are accepted, not refused
+
+Nothing in the stack rejects a delta fork, so "cannot fork" only ever describes the read, never the write. `aupdate_state` against an older checkpoint — or a run resumed from one — creates a second child of that checkpoint with no error: no saver checks for an existing sibling, and `ensure_checkpoint_mode_compatible` returns immediately when the process mode is `delta`. The cost is paid at materialization instead.
+
+Two consequences follow:
+
+- **A resume is the half DeerFlow covers.** The run paths linearize it, so the Gateway and the embedded client never produce this fork on a resume or a rollback.
+- **A state update is not covered.** `POST /api/threads/{thread_id}/state` forwards a client-supplied `checkpoint_id` straight into the accessor's write, and the mutation accessor used while preparing `POST .../branches` accepts the same selector. Nothing linearizes it, so the write is stored under that older checkpoint — the shared parent — and joins the write set that every other child of that parent walks. Reducer channels such as `messages` are written with `Overwrite`, which replaces the accumulated value and therefore masks the parent's other writes for the new branch; the general case is tracked upstream as issue **#8551** ("`update_state` against an older checkpoint writes into the branch it forked away from"), still open. Until it lands, do not select an older `checkpoint_id` on a delta thread; update the head instead.
+
+## What the linearization does
+
+Rather than reimplement write-to-child ownership — that belongs to the saver's `get_delta_channel_history` contract — DeerFlow expresses a resume as what it actually means:
+
+1. Materialize the selected checkpoint's state through the `CheckpointStateAccessor`.
+2. Write that state with **replace semantics** on the **current head**. The head has no other children, so no sibling writes can be replayed into it.
+3. Let the run continue linearly from the rewritten head.
+
+The abandoned turn stays in checkpoint history as the rewritten head's ancestry; it is not deleted.
+
+### Invariants
+
+| Invariant | Behaviour |
+| --- | --- |
+| Every materialized channel is restored | The replacement is built from the selected checkpoint's materialized values |
+| Head-only channels are reset, not inherited | A channel that exists only on the newer head resets to its schema default (for example `[]` or `{}`), or to `None` when the channel has no constructible default. LangGraph has no public "unset channel" update, so this is the closest equivalent |
+| Reducer channels replace instead of merging | The whole-state write wraps every reducer channel — the delta `messages` channel and classic `BinaryOperatorAggregate` channels alike — in `Overwrite`. Reducer channels require that wrapping for replace-style writes in **any** mode; without it the replacement would be merged into the already-aggregated value |
+| Middleware channels are covered | The write goes through the thread's *effective* schema, so channels contributed by custom middleware are included. The base `ThreadState` fallback does not know them, and writes to unknown channels are silently discarded |
+| Agent binding survives | The selected checkpoint's server-authored agent-binding metadata is copied onto the rewritten head |
+| The selector is consumed | Afterwards `checkpoint_id` and `checkpoint_map` are popped from `configurable`, and the run's initial `RunnableConfig` is rebuilt so streaming starts from the rewritten head |
+
+## When the rewrite is a no-op
+
+The helper returns `None` — leaving an ordinary run completely untouched — when any of these holds:
+
+| Condition | Why |
+| --- | --- |
+| `full` mode | Full checkpoints carry complete channel values, so the fork materializes correctly and LangGraph's branching semantics stay untouched |
+| No checkpointer | Nothing is persisted to rewrite |
+| `configurable` is not a dict | There is no selector to interpret |
+| Missing or non-string `checkpoint_id` | No resume target was selected |
+| Non-empty `checkpoint_ns` | Subgraph namespaces have their own lineage, and the Gateway only selects root checkpoints |
+| The selector already names the head | Selecting the head is already linear — no sibling can exist yet |
+
+## Fail-closed behaviour
+
+The linearization never degrades into "fork anyway". Failures propagate, because silently falling back to the fork would persist the corrupted history this code exists to prevent.
+
+| Failure | Error text |
+| --- | --- |
+| The resume state cannot be materialized | `Run {run_id} could not materialize resume checkpoint {checkpoint_id}` |
+| The state schema cannot be inspected for the replacement | `Run {run_id} could not inspect the state schema for {operation}` |
+
+Both are `RuntimeError`s raised before any write, so the run fails rather than persisting a partial replacement.
+
+<Callout type="warning">
+  Delta mode has no degraded read path in the Gateway either. When the agent
+  factory is unavailable, `full` mode falls back to raw checkpointer reads,
+  but delta mode re-raises: materialization needs the graph's channel table,
+  so a fallback could only return sentinels.
+</Callout>
+
+## Rollback
+
+A cancelled run restores the pre-run state. Delta rollback has the same write-ownership problem as a resume: the captured parent now carries writes from the cancelled sibling, so restoring by forking that parent would replay them. Delta rollback therefore restores **linearly on the current head**; `full` mode keeps forking, where every channel inherits from the parent.
+
+### What is captured
+
+| Captured item | How | Notes |
+| --- | --- | --- |
+| Materialized snapshot | Via the `CheckpointStateAccessor` | Raw checkpoint blobs cannot reconstruct delta messages — delta checkpoints omit the materialized value — so the materialized snapshot is the only source for them |
+| Raw `pending_writes` | Via `aget_tuple` | The raw tuple remains valid for tuple-level metadata and in-flight writes |
+| `state_values` | Deep copy of every non-`messages` channel | Captured **only in delta mode**; in full mode the fork inherits those channels from its parent instead |
+| Metadata | From the materialized snapshot | Restored alongside the values |
+
+If capture fails, the run is marked `snapshot_capture_failed` and rollback is disabled — restoring an empty or partial history would silently truncate the thread. If the thread has no checkpoint at all, rollback takes the `adelete_thread` reset path instead.
+
+### Ordering and the thread lock
+
+The pre-run rollback point is captured **before** the resume rewrite. That ordering is what makes cancel-with-rollback restore the real pre-run head rather than the state the linearization just wrote.
+
+Both steps run inside `_checkpoint_thread_lock(thread_id)`, which serializes checkpoint mutations for one thread without blocking goal commands. The lock is non-reentrant and must not be reacquired inside the helper. A previous successful run may still be persisting duration metadata after its admission slot was released, so sharing the lock turns the capture and the rewrite into one uninterrupted read/write sequence against the head.
+
+## Manual compaction
+
+Context compaction replaces a thread's message list with a summary. In delta mode the `messages` channel is append-only, so the replacement only takes effect when it is wrapped:
+
+```python
+updated_config = await accessor.aupdate(
+    update_config,
+    {
+        "messages": Overwrite(list(result.preserved_messages)),
+        "summary_text": result.summary_text,
+        "task_history": result.task_history,
+    },
+    as_node="manual_compaction",
+)
+```
+
+The write goes through the accessor — it needs a materialized `checkpoint_id` — and `Overwrite` is what makes the replace survive the delta channel's append reducer; an unwrapped list would append to the history instead of replacing it. The same rule covers reducer channels in full mode.
+
+Compaction failures keep their existing surface: a summary generation failure raises `ContextCompactionFailed("summary generation failed")` and becomes HTTP 500 `Failed to compact thread context.`; a disabled compaction switch is a 409 `Context compaction is disabled.`
+
+## See also
+
+- [Channel Modes](/docs/harness/checkpoints/modes) — the fail-closed mode gate that a cross-mode resume hits first.
+- [Snapshot Cadence](/docs/harness/checkpoints/cadence) — where `_DeltaSnapshot` blobs come from and why replay depth is bounded.
+- [History Cache](/docs/harness/checkpoints/history-cache) — the ancestor-chain immutability argument that makes cached delta histories safe.
+- [Troubleshooting](/docs/harness/checkpoints/troubleshooting) — the "old answer reappears" symptom and the other delta-mode failure signatures.

frontend/src/content/en/harness/checkpoints/troubleshooting.mdx

@@ -0,0 +1,221 @@
+---
+title: Troubleshooting
+description: Checkpoint problems indexed by symptom. Mode mismatches and restart-required errors, configuration that seems ignored, storage that did not shrink, regenerate and history replay defects, Postgres and redis failures, integration rejections, and cache counters that never hit, each with cause, fix, and the pinned upstream issue.
+---
+
+import { Callout } from "nextra/components";
+
+# Troubleshooting
+
+This chapter is organized by **what you see**. Each entry gives the cause, the fix, and — where the behavior is an upstream defect — the pinned issue and dependency version, so you can match it to your build.
+
+## Symptom index
+
+| Symptom | Most likely cause | Section |
+| --- | --- | --- |
+| `409` from a state/history/thread read | a full-mode process reading a delta thread | [HTTP 409 on a delta thread](#http-409-on-a-delta-thread) |
+| A run request returns `500` before admission | the thread's run-event feed is empty, so the seeding read runs under the mode gate | [HTTP 409 on a delta thread](#http-409-on-a-delta-thread) |
+| A run fails with a restart-required message | a config edit the frozen process rejects when it builds the graph | [A run fails after a config edit](#a-run-fails-after-a-config-edit) |
+| A config edit seems ignored | mode and cadence are frozen at agent-build time | [A config change appears to have no effect](#a-config-change-appears-to-have-no-effect) |
+| A history-cache edit seems ignored | the cache is built with the checkpointer at startup | [A history-cache change appears to have no effect](#a-history-cache-change-appears-to-have-no-effect) |
+| Regenerate endpoints return `500` | the run-runs router does not map the mode errors | [Regenerate endpoints return 500 instead of 409](#regenerate-endpoints-return-500-instead-of-409) |
+| Delta did not shrink storage | still in full mode, or the writes table was not counted | [Delta did not shrink storage](#delta-did-not-shrink-storage) |
+| Materialization got slower | snapshot cadence raised too far | [Materialization got slower after raising the cadence](#materialization-got-slower-after-raising-the-cadence) |
+| The replaced answer comes back after regenerate | delta fork replay (#4458) | [The old answer reappears after regenerate](#the-old-answer-reappears-after-regenerate) |
+| History order differs from the live run | upstream replay ordering (#8382) | [History order differs from the live run](#history-order-differs-from-the-live-run) |
+| Postgres hydrates old checkpoints empty | upstream paged walk (#8448) | [Postgres hydrates old checkpoints empty](#postgres-hydrates-old-checkpoints-empty) |
+| `redis` cache rejected on the TUI | sync path supports `memory` only | [`redis` cache is rejected on the TUI or embedded path](#redis-cache-is-rejected-on-the-tui-or-embedded-path) |
+| Cache counters never hit | cache disabled, wrong mode, or every read is new | [The cache counters never hit](#the-cache-counters-never-hit) |
+| `create_deerflow_agent` rejects delta | delta plus a checkpointer is refused | [`create_deerflow_agent` rejects delta with a checkpointer](#create_deerflow_agent-rejects-delta-with-a-checkpointer) |
+
+## Mode gates and configuration
+
+### HTTP 409 on a delta thread
+
+**Symptom**: a state, history, or thread read returns `409` with
+
+```
+{"detail": "Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode."}
+```
+
+**Cause**: a process running `full` read a thread whose checkpoints were written in `delta`. The read gate raises `CheckpointModeMismatchError` before materializing, so a full-mode process never silently returns empty state. The routes that return `409` are `GET /api/threads/{thread_id}`, `GET` and `POST /api/threads/{thread_id}/state`, and `POST /api/threads/{thread_id}/history` (all covered by tests); `POST /api/threads/{thread_id}/branches` and `POST .../compact` gate the same way by code path. `compact` also uses `409` for a different reason, with detail `Context compaction is disabled.`
+
+The threads router does not pre-gate run creation, so a mismatch there fails asynchronously: `POST .../runs` returns a `200` run record that later turns `status: error`, `POST .../runs/stream` returns `200 text/event-stream` and then an `event: error` frame, and `POST .../runs/wait` returns `200` JSON with `{"status": "error", "error": "<the mismatch text>"}`. One run path reads a checkpoint before admission, though: when a thread's run-event feed is empty but a checkpoint head exists, `ensure_checkpoint_history_seeded` materializes that head inside `start_run`, where only `ConflictError` and `UnsupportedStrategyError` are mapped, so the mismatch escapes as an unhandled `500` on `POST /api/threads/{thread_id}/runs`, `/runs/stream`, and `/runs/wait`. With the default `run_events.backend: memory` a Gateway restart empties every existing thread's feed, so that branch is reachable after a restart.
+
+In the embedded TUI the client raises in-process before the graph streams: the Textual runner turns it into an `AssistantError` plus `RunEnded` pair, while the headless `--print` and `--json` runners have no handler and print an uncaught traceback instead of a status code.
+
+**Fix**: pick one of two exits.
+
+1. Run the process in `delta`: set `database.checkpoint_channel_mode: delta` and restart every process sharing the database. A delta process reads legacy full checkpoints transparently, so full to delta is the supported direction.
+2. Keep the process in `full`, which requires the thread's checkpoints to be materialized and converted first. The error text names this step, but the repository ships no conversion command, and a raw full-mode read of delta checkpoints would only see sentinels — do not work around it by deleting rows.
+
+See [Channel Modes](/docs/harness/checkpoints/modes) for the marker semantics and [Quick Start](/docs/harness/checkpoints/quick-start) for the switch procedure.
+
+### A run fails after a config edit
+
+**Symptom**: after editing `config.yaml` without restarting, the next run fails with one of
+
+```
+checkpoint_channel_mode is restart-required and cannot change in a running process
+checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process
+```
+
+in its run record, its SSE `error` frame, or the worker log line `Run %s failed: %s`.
+
+**Cause**: `CheckpointModeReconfigurationError`. The process froze its mode and cadence at startup, and something built a graph with a different value — a config hot-reload that diverged from the frozen startup snapshot. Only Gateway startup, `make_lead_agent`, and `DeerFlowClient.__init__` call the freeze helpers, and no state, history, or thread route builds an agent — those routes read the startup-frozen `app.state` values — so this is a run-side failure: none of them answers `503` for a config edit. The router-level `503` mapping exists for that error class but nothing in the threads routes raises it.
+
+Each knob behaves differently:
+
+- `database.checkpoint_channel_mode`: no failure. The run worker injects the frozen mode into the run config and the agent factory prefers that injected value over the request-time config, so the edited value is ignored until a restart — that is [A config change appears to have no effect](#a-config-change-appears-to-have-no-effect).
+- `database.checkpoint_delta.snapshot_frequency`: the next run fails, because the cadence has no injected override and the agent factory freezes it from the reloaded config.
+
+**Fix**: restart the process so the new value is frozen, then re-run. The snapshot cadence is part of the failure because it is compiled into each graph's channel table and is likewise restart-required.
+
+### A config change appears to have no effect
+
+**Symptom**: you edited `database.checkpoint_channel_mode` or `database.checkpoint_delta.snapshot_frequency`, but threads keep behaving as before.
+
+**Cause**: the mode is frozen process-globally the first time an agent is built, from the startup snapshot; the cadence is frozen alongside it. On the first freeze a client-supplied `configurable.__deerflow_checkpoint_channel_mode` is ignored, and once frozen any differing value fails closed. The embedded `DeerFlowClient` freezes mode and cadence the same way. Also note `checkpoint_delta.*` is ignored in full mode.
+
+**Fix**: restart every process that shares the checkpoint database; all of them must use the same value. Confirm the switch by reading a thread's marker (see [Observability](/docs/harness/checkpoints/observability)). Only one related key is re-read at runtime: `database.checkpoint_graph_cache.accessor_graph_max` (on each eviction check). `database.checkpoint_cache.*` is *not frozen* — it cannot corrupt a thread and may differ across workers — but it is also not hot-reloadable; see [the next entry](#a-history-cache-change-appears-to-have-no-effect).
+
+### A history-cache change appears to have no effect
+
+**Symptom**: you changed `database.checkpoint_cache.max_entries`, switched `type` to `redis`, or edited `redis_url`, `ttl_seconds`, or `key_prefix`, and the running Gateway keeps its old behaviour — the old capacity, no redis traffic, the previous keys.
+
+**Cause**: the history cache is not hot-reloadable. `langgraph_runtime()` builds the checkpointer once from the startup `AppConfig` snapshot, and the delta saver is wrapped around whatever `make_checkpoint_cache()` returned at that moment (`backend/app/gateway/deps.py`, `backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py`), so backend, capacity, connection, TTL, and prefix are captured for the process's lifetime. The whole `database` section is registered startup-only for exactly this reason (`backend/packages/harness/deerflow/config/reload_boundary.py`).
+
+This is not the frozen-for-correctness case: nothing about the cache can corrupt a thread, results are identical with it disabled, and workers sharing one checkpoint database may safely run different cache settings. It is simply captured once.
+
+**Fix**: restart the Gateway after editing the section — and on the embedded or TUI path, restart that process, whose checkpointer singleton is also created once and reused. Only `database.checkpoint_graph_cache.accessor_graph_max` applies without a restart, because it is re-read on every eviction check.
+
+### Regenerate endpoints return 500 instead of 409
+
+**Symptom**: regenerate preparation returns `500` with `detail: "Failed to read latest checkpoint"` or `detail: "Failed to inspect checkpoint history"`, even though the same thread yields `409` elsewhere.
+
+**Cause**: `POST /api/threads/{thread_id}/runs/regenerate/prepare` (and `/runs/edit-regenerate/prepare`) build the state accessor outside any `try`, and this module does not map the checkpoint-mode errors, so a mismatch surfaces as an unhandled `500`. The log lines are
+
+```
+Failed to read latest checkpoint for regenerate thread %s
+Failed to list checkpoints for regenerate thread %s
+```
+
+Two other endpoints fail more quietly for the same reason: `GET /api/threads/{thread_id}/token-usage` returns `200` with `context_usage: null` (only `Failed to load checkpoint for context usage on thread %s` is logged), and `/runs/wait` returns `200` with an error body rather than `409`.
+
+**Fix**: read the `500` as a mode mismatch. Check the thread's marker, then restart the process in the thread's mode. There is no local fix that turns these into `409`.
+
+## Storage and materialization
+
+### Delta did not shrink storage
+
+**Symptom**: you switched to delta (or believe you did), but the database is the same size or larger.
+
+**Cause**: usually one of these.
+
+- The process is still in `full` — the default — because the key was never set, or was set without a restart. `checkpoint_delta.*` does nothing in full mode, and the history cache wraps the saver only in delta mode.
+- You compared the wrong table. Delta moves per-step payloads into the writes table, so a fair comparison counts checkpoint payload bytes **plus** write rows. The storage guard asserts delta keeps at most one snapshot blob where full keeps at least four.
+
+A related caution: retention that drops intermediate checkpoints must not drop the ancestor checkpoints or writes between the kept checkpoint and the nearest `_DeltaSnapshot`, or delta channels silently reconstruct empty with no error raised. DeerFlow ships no production trigger for `prune`/`aprune`, so this is a constraint on any wiring, not a live failure.
+
+**Fix**: confirm the mode from a thread's marker, restart if needed, and compare checkpoint payload plus writes rows. See [History Cache](/docs/harness/checkpoints/history-cache) and [Operating Checkpoints](/docs/harness/checkpoints/operations).
+
+### Materialization got slower after raising the cadence
+
+**Symptom**: after raising `snapshot_frequency`, reads and state materialization are slower.
+
+**Cause**: `snapshot_frequency` (default `10`) is the number of per-step writes between full snapshots. Higher means fewer, smaller snapshots but a longer replay walk to materialize any given checkpoint. The materialized state is identical either way; only size and read cost trade off.
+
+**Fix**: lower `snapshot_frequency` to trade checkpoint size back for read speed. It is restart-required, like the mode. Repeated materialization of the same checkpoints is what the history cache absorbs, so check its counters too (below).
+
+## History and replay
+
+### The old answer reappears after regenerate
+
+**Symptom**: regenerating in a branched thread leaves the replaced assistant message beside the new one after a reload, and branch history can break when seeded rows share one run id ([#4458](https://github.com/bytedance/deer-flow/issues/4458)).
+
+**Cause**: delta cannot fork correctly — the fork itself is accepted, the read is what breaks. Resuming from an older checkpoint forks the lineage, and the delta history walk collects **every** `pending_writes` entry on each on-path ancestor — but a shared parent also carries the writes of the sibling that was abandoned, so those writes are replayed into the fork and the run starts from a message list that still contains the answer it was supposed to replace. Reproduced on Postgres, SQLite, and the in-memory saver; full mode is unaffected because its checkpoints carry complete channel values.
+
+**Fix**: DeerFlow linearizes instead of forking. It materializes the requested checkpoint's state and writes it with replace semantics on the current head (which has no other children), resets channels that exist only on the newer head to their schema default or `None`, consumes the selector, then runs linearly. Linearization is a no-op when the mode is full, there is no checkpointer, the selector already names the head, or the namespace is non-root; it **fails closed** rather than falling back to the fork:
+
+```
+Run {run_id} could not materialize resume checkpoint {checkpoint_id}
+```
+
+The upstream fix is PR [#8548](https://github.com/langchain-ai/langgraph/pull/8548) (fixing upstream [#8443](https://github.com/langchain-ai/langgraph/issues/8443); mirror #8551), still open, which is why the local linearization is retained. See [Resume and Rollback](/docs/harness/checkpoints/resume-and-rollback).
+
+### History order differs from the live run
+
+**Symptom**: on a parallel fan-out that writes one delta channel in a single super-step, the live run returns writes in path order but `get_state` / resume replays them in a different order.
+
+**Cause**: pinned upstream defect ([langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)), open, fix PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) unmerged, no released version. Delta replay orders writes by hashed `(task_id, idx)` instead of the path order `apply_writes` used live, so same-superstep writes are permuted. Present in `langgraph-checkpoint 4.2.0`; reproduced on memory, SQLite, and Postgres.
+
+**Fix**: none locally. DeerFlow's test asserts the correct contract and trigger-skips while the defect is present, naming the issue:
+
+```
+upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes
+```
+
+If ordering within one super-step matters to your consumer, avoid fanning parallel writes into a single delta channel.
+
+### Postgres hydrates old checkpoints empty
+
+**Symptom**: on Postgres, an older delta checkpoint materializes empty while a recent one is fine.
+
+**Cause**: pinned upstream defect ([langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)), open, fix PR #8556 unmerged. `PostgresSaver.get_delta_channel_history` permanently poisons its walk cursor when the target checkpoint is not in the first pagination page: the unpatched cursor is derived from a not-yet-loaded parent and parks at `None` for good, so old checkpoints hydrate empty. Only Postgres pages its stage-1 scan. Present in `langgraph-checkpoint-postgres 3.1.2`.
+
+**Fix**: none locally; the Postgres case trigger-skips, naming the issue:
+
+```
+upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty
+```
+
+Cadence bounds replay depth, so a smaller `snapshot_frequency` keeps the walk shorter; that is the only lever this repository exposes, not a fix for the cursor bug.
+
+## Cache
+
+### `redis` cache is rejected on the TUI or embedded path
+
+**Symptom**: the TUI or a headless `--print` / `--json` run fails at checkpointer construction with
+
+```
+database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.
+```
+
+**Cause**: the TUI and headless modes build `DeerFlowClient(checkpointer=get_checkpointer())` — the **sync** singleton checkpointer — and only the `memory` backend is supported on the sync path (it is process-local anyway). A config that is valid for the Gateway can therefore break the TUI.
+
+**Fix**: set `database.checkpoint_cache.type: memory`. Gateway-only deployments can keep `redis`, which is the async path's shared cache.
+
+### The cache counters never hit
+
+**Symptom**: `hits` stays at `0` while `misses` climbs, or `compose_hits` and `full_walks` look wrong.
+
+**Cause**: several legitimate reasons.
+
+- The cache is installed only in delta mode, on the async/Gateway path. In full mode there is no `CachedHistorySaver` at all.
+- `max_entries: 0` disables the cache uniformly, and a disabled cache passes straight through without composing.
+- The wrapper never caches the "latest checkpoint" resolution: it resolves the target to an immutable checkpoint id first, so a client that only ever reads brand-new checkpoints misses by design.
+- Composition depth is bounded (`_COMPOSE_MAX_DEPTH = 8`) and a cold chain falls back to one inner walk; single-level composition was measured to yield `0` cache hits on a 500-step SQLite run before recursive composition was added.
+- A `max_entries: 1` LRU thrashes on every read — correct, but no hits.
+- The redis backend reports only `hits` and `misses`; its `evictions` and `entries` stay `0`.
+
+**Fix**: confirm delta mode and `max_entries > 0` — and restart before drawing conclusions from a settings change, because the cache is read only when it is built at startup. Then read `compose_hits` and `full_walks` to see whether composition (not the top-level cache) is doing the work. Interpret `hits`/`misses` together with those two counters rather than alone. See [Cache counters](/docs/harness/checkpoints/observability).
+
+## Integration
+
+### `create_deerflow_agent` rejects delta with a checkpointer
+
+**Symptom**: calling `create_deerflow_agent(..., checkpoint_channel_mode="delta", checkpointer=...)` raises a `ValueError`:
+
+```
+create_deerflow_agent does not support checkpoint_channel_mode='delta' with a checkpointer: persisted graphs built here bypass checkpoint mode marker injection and the fail-closed compatibility gate (see deerflow.runtime.checkpoint_mode), so a mixed-mode store would silently corrupt thread state.  Use the guarded application paths (make_lead_agent or DeerFlowClient) for delta persistence; delta without a checkpointer is ephemeral and allowed.
+```
+
+**Cause**: a graph built directly this way bypasses marker injection and the fail-closed compatibility gate, so a mixed-mode store could silently corrupt thread state. Delta **without** a checkpointer is ephemeral and allowed.
+
+**Fix**: for delta persistence, go through the guarded paths — `make_lead_agent` or `DeerFlowClient`. If you only need an ephemeral delta graph, drop the checkpointer.
+
+<Callout type="tip">
+  Before chasing a symptom, get two facts: the process's frozen mode (from
+  `config.yaml` plus whether it was restarted) and the thread's marker (from
+  its `metadata`). Almost every entry above resolves once those agree.
+</Callout>

frontend/src/content/en/harness/configuration.mdx

@@ -120,6 +120,7 @@ The following table maps each top-level `config.yaml` section to its documentati
 | `memory`          | Cross-session memory storage                     | [Memory](/docs/harness/memory)                           |
 | `summarization`   | Conversation summarization                       | [Middlewares](/docs/harness/middlewares)                 |
 | `title`           | Automatic thread title generation                | [Middlewares](/docs/harness/middlewares)                 |
+| `database`        | Checkpoint channel mode, snapshot cadence, and history cache | [Checkpoints and Delta Channels](/docs/harness/checkpoints) |
 | `checkpointer`    | Thread state persistence                         | [Agents & Threads](/docs/application/agents-and-threads) |
 | `guardrails`      | Tool call authorization                          | —                                                        |
 | `stream_bridge`   | Streaming configuration                          | —                                                        |

frontend/src/content/zh/application/agents-and-threads.mdx

@@ -100,6 +100,8 @@ database:
 
 旧的 `checkpointer` 配置仍然兼容，用于 LangGraph 状态持久化；如果配置了它，会优先于 `database` 的检查点设置。新的部署应优先使用 `database`。
 
+线程状态在该后端中以何种形式存储——整份快照，还是 LangGraph 的 `DeltaChannel` 增量——由另一个开关 `database.checkpoint_channel_mode` 控制，见[检查点与增量通道](/docs/harness/checkpoints)。
+
 ### 线程数据目录
 
 每个线程在 `.deer-flow/threads/{thread_id}/` 下有一个专用目录：

frontend/src/content/zh/harness/_meta.ts

@@ -34,6 +34,9 @@ const meta: MetaRecord = {
   subagents: {
     title: "子 Agent",
   },
+  checkpoints: {
+    title: "检查点",
+  },
   mcp: {
     title: "MCP 集成",
   },

frontend/src/content/zh/harness/checkpoints/_meta.ts

@@ -0,0 +1,33 @@
+import type { MetaRecord } from "nextra";
+
+const meta: MetaRecord = {
+  "quick-start": {
+    title: "快速上手",
+  },
+  modes: {
+    title: "通道模式",
+  },
+  cadence: {
+    title: "快照节奏",
+  },
+  "history-cache": {
+    title: "历史缓存",
+  },
+  "resume-and-rollback": {
+    title: "恢复与回滚",
+  },
+  operations: {
+    title: "运维检查点",
+  },
+  observability: {
+    title: "可观测性",
+  },
+  troubleshooting: {
+    title: "故障排查",
+  },
+  reference: {
+    title: "参考附录",
+  },
+};
+
+export default meta;

frontend/src/content/zh/harness/checkpoints/cadence.mdx

@@ -0,0 +1,111 @@
+---
+title: 快照节奏
+description: database.checkpoint_delta.snapshot_frequency 如何在检查点体积与物化成本之间取舍、它与 LangGraph 全局 superstep 上界的关系、为什么它与模式一同冻结且从不写入检查点元数据，以及默认值 1000 改为 10 带来的历史变更与三条旧键迁移告警。
+---
+
+import { Callout } from "nextra/components";
+
+# 快照节奏
+
+`delta` 模式下检查点存的是写入，消息列表在读取时重建。为了让重建不必无限回溯祖先链，系统会定期落下一份完整的消息快照。`database.checkpoint_delta.snapshot_frequency` 决定这个间隔。
+
+<Callout type="info">
+  这个旋钮只在 `delta` 模式下有效。`checkpoint_channel_mode` 为 `full` 时，`checkpoint_delta` 整个小节都会被忽略；见[通道模式](/docs/harness/checkpoints/modes)。
+</Callout>
+
+## 配置键
+
+| 配置键                                          | 类型  | 默认值 | 范围                          | 是否重启 |
+| ----------------------------------------------- | ----- | ------ | ----------------------------- | -------- |
+| `database.checkpoint_delta.snapshot_frequency`  | `int` | `10`   | `ge=1`（大于等于 1，无上界）  | 是       |
+
+语义是：每 N 次逐步写入落一份完整的消息快照（值越大检查点越小、物化越慢）。该值会进入编译后的通道表：messages 字段被构造成 `DeltaChannel(merge_message_writes, snapshot_frequency=…)`，因此节奏是"编译出的图"的属性，而不是某个检查点自身的属性。
+
+```yaml
+database:
+  checkpoint_channel_mode: delta   # 只有在这个模式下 snapshot_frequency 才有意义
+  checkpoint_delta:
+    snapshot_frequency: 10         # 每 10 次逐步写入落一份 _DeltaSnapshot
+```
+
+## 权衡
+
+| 节奏             | 检查点体积                                          | 物化                                      | 物化结果 |
+| ---------------- | --------------------------------------------------- | ----------------------------------------- | -------- |
+| 小 N（例如 1）   | 每一步都带快照；存储优势退化到接近 `full` 模式        | 祖先链最短，读取最快                      | 完全相同 |
+| 默认 `10`        | 每 10 次逐步写入一份快照                             | 中等长度的回溯                            | 完全相同 |
+| 大 N             | 快照最少，存储最小                                   | 回溯最长，读取最慢                        | 完全相同 |
+
+这个旋钮买的是「存储」与「读取成本」之间的取舍，从不改变答案。低节奏会比默认值频繁得多地为消息列表落快照，而物化结果保持一致；两条测试分别钉住这两半：配置值必须真正到达编译后的通道表，且任何节奏下物化都必须产出相同的值。
+
+它同时是一阶的容量旋钮，因为占住保留总量的正是这些快照：设 K 为 `snapshot_frequency`，N 轮保留的是 O(N²/K + N) 而不是 O(N²)，而逐步写入的负载与 K 无关。只按单步成本估算容量会低估占用；推导见[检查点与增量通道](/docs/harness/checkpoints)。
+
+## 与 LangGraph superstep 上界的关系
+
+是否落快照由两个计数器共同决定，谁先到达就由谁触发：
+
+| 计数器                       | 作用域       | 默认值                    | 由谁设置                     |
+| ---------------------------- | ------------ | ------------------------- | ---------------------------- |
+| 本通道的更新次数             | 当前通道     | `snapshot_frequency`（`10`） | `database.checkpoint_delta.snapshot_frequency` |
+| 距上次快照的 superstep 数    | 进程全局     | `5000`                    | LangGraph 的 `DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT`（环境变量 `LANGGRAPH_DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT`） |
+
+第二个计数器为「不再收到写入」的通道也兜住回溯深度。因此把 `snapshot_frequency` 调得再大，回溯深度也不会无界：一个持续运行却不再触碰 messages 的线程，最迟会在 5000 个 superstep 的界内落快照。这个界属于 LangGraph，不属于 `database:` 小节，且对整个进程内的所有 delta 通道生效。
+
+## 与模式一同冻结，且不写入检查点
+
+快照节奏被编译进每个图的通道表，而不是存进检查点。由此带来几条直接的运维后果：
+
+| 事实 | 后果 |
+| --- | --- |
+| 解析顺序是：显式参数 → 进程内已冻结值 → 配置默认值 `10` | 显式值主要供测试与临时图使用；装配路径始终传入进程值，因此运行中的部署总是按冻结节奏编译 |
+| 与模式一样，在第一次建图时冻结 | 共享同一个检查点数据库的所有进程必须使用同一个值；检查点里没有可供比对的节奏信息 |
+| 运行中改为不同值会抛 `CheckpointModeReconfigurationError("checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process")` | 改 `config.yaml` 不重启不生效，且失败是响亮的，不会静默降级 |
+| 非正值会抛 `ValueError("snapshot frequency must be positive")` | 冻结入口与配置校验（`ge=1`）口径一致 |
+| 该值只从应用配置读取，不接受逐请求的 configurable 键 | 伪造或客户端传入的值无法重编译通道表 |
+
+节奏被**刻意**不写入检查点元数据：模式标记契约（缺失即 `full`）以及 full → delta 的迁移语义都不随频率值改变。正因为它不在任何检查点里，运行着不同节奏的进程会静默地把自己的节奏套用到同一批线程上——所以这是一个写进配置字段说明的运维责任，运行时无法检测。
+
+对比一下**会**写入的模式标记：delta 检查点的元数据里带有 `deerflow_checkpoint_channel_mode: "delta"`，缺失即表示 `full`。节奏没有对应的标记。
+
+## 存储形态契约
+
+| 检查点类型     | `channel_values.messages`            | 该步的负载存在哪里                        |
+| -------------- | ------------------------------------ | ----------------------------------------- |
+| 非快照步       | 不存在——`channel_values` 里没有 `messages` 这个键 | 在该检查点的逐步写入中                    |
+| 快照步         | 一份 `_DeltaSnapshot`，内含完整消息列表 | 就在检查点本身；祖先回溯到此终止          |
+| `full` 模式    | 每个检查点都带完整消息列表           | 无需重放                                  |
+
+这条契约有两处钉死：第一，非快照检查点不得在 `channel_values` 中携带 `messages`，而每 `snapshot_frequency` 次更新必须改为携带一份 `_DeltaSnapshot`，且两种形态下物化都必须正确；第二，存储层面的守卫在相同轮数下比较两种模式：delta 至多只有周期性快照 blob，而 `full` 模式每个写了消息的检查点都会有一个。所以在 delta 模式下，快照不是"多存的一份拷贝"——它是唯一存在的完整消息负载。
+
+## 默认值变更：1000 改为 10
+
+节奏原本位于扁平键 `database.checkpoint_delta_snapshot_frequency`，默认值 `1000`。现在它迁到嵌套键 `database.checkpoint_delta.snapshot_frequency`，默认值 `10`——同一个字段，但默认情况下线程落快照的频率提高了 **100 倍**。上游 `DeltaChannel` 的参数默认值仍是 1000；DeerFlow 始终显式传值，因此由 DeerFlow 构建的线程不会使用上游默认值。想恢复旧节奏，显式写出该值即可：
+
+```yaml
+database:
+  checkpoint_delta:
+    snapshot_frequency: 1000   # 重命名前的默认节奏
+```
+
+在旧扁平键仍存在期间，可能出现三条迁移告警。三条都只记日志、不报错，且嵌套键始终优先：
+
+| 触发条件 | 告警原文 |
+| --- | --- |
+| 两个键同时存在 | `Both database.checkpoint_delta_snapshot_frequency (deprecated) and database.checkpoint_delta.snapshot_frequency are set; the nested key wins.` |
+| `checkpoint_delta` 已经是配置对象 | `Ignoring deprecated database.checkpoint_delta_snapshot_frequency because database.checkpoint_delta is already set.` |
+| 旧值被继承 | `database.checkpoint_delta_snapshot_frequency is deprecated; use database.checkpoint_delta.snapshot_frequency instead. Carried the legacy value (%r) forward.` |
+
+`DatabaseConfig` 会忽略未知键，所以没有这层兼容代码时，仍写扁平旧键的 `config.yaml` 会静默回落到 `10`，而不是运维选定的值。嵌套键写好后删掉扁平键，这三条告警就都消失了。
+
+## 如何选值
+
+默认值不是拍脑袋定的：生产基准会输出 `snapshot_write_spike`（delta 检查点写入的 p99 与 p50 之比）与 `cache_effect_ms`（状态冷读减去温读的 p50），二者正是生产环境「快照频率」与「accessor 缓存」默认值的决策输入。通道基准报告与后端无关的行数/字节数以及温读、冷读耗时，比值一律是 `delta/full`；耗时阈值不构成 CI 门禁。运行方式见[运维检查点](/docs/harness/checkpoints/operations)。
+
+由于该值会被冻结、且跨进程必须一致，变更要刻意为之：先选定节奏，重启所有共享该数据库的进程，再用存储形态检查确认，而不是盯着单个请求。
+
+## 相关章节
+
+- [通道模式](/docs/harness/checkpoints/modes)——两种存储形态对读写意味着什么。
+- [历史缓存](/docs/harness/checkpoints/history-cache)——本节奏所服务的读取路径。
+- [运维检查点](/docs/harness/checkpoints/operations)——保留、基准测试与后端选择。
+- [参考附录](/docs/harness/checkpoints/reference)——完整的配置键与错误对照表。

frontend/src/content/zh/harness/checkpoints/history-cache.mdx

@@ -0,0 +1,183 @@
+---
+title: 历史缓存
+description: 增量模式的历史缓存：它位于哪一层、条目形态与不可变性论证、database.checkpoint_cache 每个字段的默认值与环境变量回退、清理与 redis 故障语义、可见 stats，以及编译后 accessor 图的容量上限。
+---
+
+import { Callout } from "nextra/components";
+
+# 历史缓存
+
+`delta` 模式下读取没有现成的消息列表可返回：它必须回溯祖先写入直到最近的 `_DeltaSnapshot`，再据此重建（见[快照节奏](/docs/harness/checkpoints/cadence)）。这段回溯就是物化成本。历史缓存把重建出的历史存下来，让下一次读取只需一次查表而不是一次回溯；它包在 delta 路径的原始 saver 外面。
+
+<Callout type="info">
+  缓存纯粹是性能优化。正确性不依赖它：这里没有任何字段会被冻结，不要求跨 worker 一致，
+  而且缓存满、缓存空、完全没有缓存三种情况下物化结果必须完全相同。
+</Callout>
+
+## 缓存位于哪一层
+
+| 路径 | 包装方式 | 后端 |
+| --- | --- | --- |
+| Gateway / 异步 | 生效模式为 `delta` 时，`CachedHistorySaver` 包住选定的 saver；缓存生命周期与 checkpointer 上下文管理器一致 | `memory` 或 `redis` |
+| 内嵌 / TUI（同步） | `_wrap_sync_if_delta` 只允许内存后端 | 仅 `memory` |
+| `full` 模式 | 完全不包装——没有东西需要物化 | — |
+
+`CachedHistorySaver` 只覆盖一个行为：`get_delta_channel_history` / `aget_delta_channel_history`。其余全部——元组读取、写入、列举、复制、删除、裁剪——原样委派给内部 saver。同步路径共享一个进程内内存缓存，并在 `max_entries` 或派生出的键前缀变化时重建，因为旧前缀下的条目既不可达、也不再被线程清理覆盖。
+
+同步路径直接拒绝 `redis`：`database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.`
+
+## 条目形态
+
+一个条目就是 `DeltaChannelHistory` 形状的字典，与原始 saver 对同一检查点、同一通道返回的内容完全一致：
+
+| 字段 | 含义 |
+| --- | --- |
+| `writes` | 该通道在路径上的增量，从旧到新，元素是 `PendingWrite` 元组。始终存在，可能为空。存在目标检查点自身的写入是留给下一个 super-step 的 pending 写入，不计入它自己的历史。 |
+| `seed` | 最近一个 `channel_values[channel]` 有值的祖先上存的值。若回溯到根都没找到存值则省略；使用方把"缺失"视为"从空开始"。 |
+
+读取是读时复制：`writes` 返回一份新列表，`seed` 按引用共享且绝不原地修改。
+
+## 不可变性：为什么不需要失效
+
+一个检查点的增量历史是它那条已封存祖先链的纯函数——LangGraph 契约把目标自身的 pending 写入排除在外，父链接在创建时就固定，而祖先的写入在它的子检查点存在后就已封存。因此以「线程、命名空间、检查点 id、通道」为键的条目是不可变的。
+
+| 性质 | 后果 |
+| --- | --- |
+| 键只由不可变成分组成 | 条目一旦写入就不再变化，于是没有失效路径、没有版本字段、也不需要协调 |
+| 子检查点存在即封存祖先 | 共享后端（redis）无需任何握手，跨进程天然一致 |
+| 缓存从不缓存"最新检查点"的解析结果 | 目标先经内部 saver 解析一次再构造键；只有已解析的、不可变的检查点 id 才会成为键 |
+| 线程删除与裁剪是生命周期事件，不是失效 | 它们清理条目是因为源检查点已不存在，而不是因为条目过期 |
+
+正因如此，整套策略是纯性能的，允许每个 worker 配置不同。
+
+## 键与命名空间
+
+`make_history_key(key_prefix, thread_id, checkpoint_ns, checkpoint_id, channel)` 构造：
+
+```
+{key_prefix}:{thread_id}:{sha256(checkpoint_ns \x00 checkpoint_id \x00 channel)[:24]}
+```
+
+| 事实 | 细节 |
+| --- | --- |
+| `thread_id` 保持可读 | 运维不必重算摘要就能认出某个线程的键 |
+| 其余成分以 NUL 分隔后哈希 | 含 `:` 的命名空间不可能产生有歧义的键 |
+| 线程清理前缀 | `{key_prefix}:{thread_id}:`，按前缀匹配，绝不做逐键失效 |
+| 默认前缀 | `ckpt-hist:v{CACHE_FORMAT_VERSION}:{hash12(database identity)}`，其中 `CACHE_FORMAT_VERSION = 1` |
+| `database.checkpoint_cache.key_prefix` | 整体替换派生前缀 |
+| 数据库身份 | 不含凭据：postgres 为 `host:port/database:schema`，sqlite 为文件路径，其余为 `memory`。轮换凭据不会改变命名空间，否则一次凭据轮换就会让缓存变冷、并让旧键一直孤儿到 TTL。两个部署共用同一个 redis 时，默认前缀不同。 |
+
+## 用合成代替回溯
+
+真实运行中每个 super-step 会创建多个检查点，只有其中一部分会被真正物化成读取目标，因此目标的直接父级通常是尚未预热的中间检查点。只合成一层时，实测在 500 步的 sqlite 运行上是零命中。于是包装器改为逐层合成：
+
+| 步骤 | 行为 |
+| --- | --- |
+| 请求的通道未命中 | 从最近的"已预热或已缓存"祖先开始合成，每个中间检查点递归一层 |
+| 深度预算 | `_COMPOSE_MAX_DEPTH = 8` |
+| 稳态 | 每一层合成结果都会被缓存，于是预热边界跟着运行前移，通常约两层内就能命中已预热的祖先 |
+| 父级是快照 | 父级本身持有通道值，直接用父级的写入加它的 seed 构造条目，完全不查父级历史 |
+| 深度 0 的冷链 | 该层只委派**一次**内部快速回溯（两条 SQL），而不是逐个元组爬祖先。它下面的祖先保持冷态，之后会从最近的预热层继续解析 |
+| 根或父级缺失 | `{"writes": []}` |
+| 所有路径上都没有该通道 | `{"writes": []}` |
+| 缓存已禁用 | 直接透传给原始 saver——在全未命中的条目上做合成严格比直接回溯更费 |
+
+同步路径镜像同一套递归逻辑，并共享同一个缓存与深度预算。冷合成从不调用内部 saver 的历史方法：它要么递归，要么自己回溯，因此回退回溯是唯一会触碰原始 saver 的地方。
+
+## 配置
+
+缓存是**随 checkpointer 一起、每个进程构造一次**的。`langgraph_runtime()` 用启动时的 `AppConfig` 快照构造 checkpointer（`backend/app/gateway/deps.py:463`），当时 `make_checkpoint_cache()` 返回什么，delta saver 就被 `CachedHistorySaver` 包住什么（`backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py:250-254`）。运行期不会再读本节，因此下表每个字段在整个进程生命周期内都保持启动时的取值。
+
+| 配置键 | 类型 | 默认值 | 范围 | 是否需重启才生效 | 说明 |
+| --- | --- | --- | --- | --- | --- |
+| `database.checkpoint_cache.type` | `memory` \| `redis` | `memory` | — | 是 | `memory` 是进程内 LRU；`redis` 供多 worker 共享，仅异步/Gateway 路径 |
+| `database.checkpoint_cache.max_entries` | `int` | `128` | `ge=0` | 是 | 内存后端 LRU 容量；`0` 表示禁用缓存 |
+| `database.checkpoint_cache.redis_url` | `str` 或 null | `null` | 任意 redis URL | 是 | 依次回退到 `DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL`、`REDIS_URL`、`redis://localhost:6379/0` |
+| `database.checkpoint_cache.ttl_seconds` | `int` | `86400` | `ge=0` | 是 | redis 条目 TTL，是防泄漏网而非正确性机制；`0` 表示显式关闭过期 |
+| `database.checkpoint_cache.key_prefix` | `str` | `""` | 任意字符串 | 是 | 为空表示使用数据库身份派生的哈希 |
+
+<Callout type="warning">
+  「不冻结」不等于「热重载」。这些设置**在正确性意义上不冻结**：它们不会损坏线程，禁用缓存时结果完全相同，共享同一检查点数据库的多个进程也可以安全地使用不同取值。但运行中的进程不会读到改动——后端、容量、redis 连接、前缀与 TTL 都停留在启动时的取值——因此改动其中任何一项都需要重启 Gateway。内嵌与 TUI 路径同理：它们的 checkpointer 单例只在进程内创建一次并一直复用。只有 [`database.checkpoint_graph_cache.accessor_graph_max`](#编译后的-accessor-图缓存) 会在运行期重新读取。
+</Callout>
+
+```yaml
+database:
+  checkpoint_channel_mode: delta
+  checkpoint_cache:
+    type: memory            # memory | redis（redis 仅 Gateway/异步可用）
+    max_entries: 128        # 0 表示禁用缓存
+    redis_url: null         # 也可用 DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL / REDIS_URL
+    ttl_seconds: 86400      # redis 防泄漏网；清理本身是即时的
+    key_prefix: ""          # 默认：数据库身份的哈希
+```
+
+只有 redis URL 有环境变量回退；本节其他字段都不从环境读取，而配置值里的 `$VAR` 插值由中心化逻辑在本节被解析之前处理。
+
+`max_entries: 0` 对两种类型统一地禁用缓存：工厂返回一个已禁用的内存后端（`enabled` 即 `max_entries > 0`），因此包装器不需要判 None。零容量缓存的行为必须与原始内部 saver 完全一致——摘要相同、`hits == 0`——这条等价性有测试钉住。内存后端拒绝负值，抛 `max_entries must be >= 0`。淘汰是严格 LRU：命中和写入都会刷新最近使用顺序，超容量时丢最旧的条目，并累计 `evictions` 计数。容量为 1 的缓存在每次读时都会抖动，但结果依然正确。
+
+redis 客户端是惰性导入的；缺少可选依赖时构造 redis 后端会抛 `redis is required for the redis checkpoint cache backend. Install it with: uv sync --extra redis`。无法识别的 `type` 会抛 `Unknown checkpoint cache type: {config.type!r}`。
+
+## 清理与生命周期
+
+| 操作 | 缓存动作 |
+| --- | --- |
+| `delete_thread` / `adelete_thread` | 源数据删除后，按键前缀匹配清理该线程的条目 |
+| `prune` / `aprune` | 清理被重写线程的条目：被裁剪过的链不得保留裁剪前的缓存历史，也不得残留对已删除祖先的引用 |
+| `delete_for_runs` / `adelete_for_runs` | **不清理。** 以 run 为单位的删除无法在不额外查询的前提下映射回线程，因此条目原样保留：它们仍然正确（封存论证是逐链成立的），残余保留量由 LRU/TTL 兜底。当前仓库内没有调用方。 |
+| `copy_thread` / `acopy_thread` | 直接委派；不清理 |
+
+理由是数据生命周期：源的删除不应在缓存里留下残余的历史负载。这从来不是正确性要求。
+
+## redis 故障行为
+
+所有 redis 故障都只影响性能。触达缓存时源数据早已更新完毕，所以故障只是损失命中，从不影响可用性：
+
+| 故障 | 告警原文 | 效果 |
+| --- | --- | --- |
+| `mget` 失败 | `checkpoint history cache mget failed; treating as all-miss: %s` | 所有请求的键都按未命中处理，历史被重新合成 |
+| 写入失败 | `checkpoint history cache write failed; skipping: %s` | 跳过该次写入，下次读取直接重算 |
+| 线程清理失败 | `checkpoint history cache thread purge failed; residual entries expire via TTL: %s` | 源删除已经完成；残余副本会随 `ttl_seconds` 过期；这条路径从不抛异常 |
+
+TTL 是防泄漏网而非正确性机制，因为条目不可变、没有任何读取依赖它。`ttl_seconds: 0` 表示彻底关闭过期，此时孤儿键只依赖 redis 的 `maxmemory` 策略。清理失败是已删除线程的历史残留下来的唯一途径，且只残留到 TTL 到期。
+
+## stats
+
+`CachedHistorySaver.stats()` 把后端的计数与包装器自身的合成计数合并在一起。这是唯一暴露的缓存观测面：没有任何 Gateway 接口或健康检查会报告缓存状态。
+
+| 计数 | 来源 | 含义 |
+| --- | --- | --- |
+| `hits` / `misses` | 后端 | 逐键查询的命中与未命中 |
+| `evictions` / `entries` | 仅内存后端 | LRU 淘汰次数与当前条目数。redis 后端这两项恒为 `0`——它不维护本地 LRU 或容量，容量由它自己的 `maxmemory` 策略兜住 |
+| `compose_hits` | 包装器 | 从已存父值或已缓存历史合成出来的层级数 |
+| `full_walks` | 包装器 | 委派内部增量回溯的冷回退次数 |
+
+稳态进程应表现为 `compose_hits` 持续增长而 `full_walks` 保持低位；`full_walks` 长期偏高说明合成总落在冷链上；而预热进程里 `hits` 一直为 `0` 说明查询始终没碰到条目——通常是缓存被禁用，或命名空间与条目前缀不一致（前缀由数据库身份派生）。见[故障排查](/docs/harness/checkpoints/troubleshooting)。
+
+## 编译后的 accessor 图缓存
+
+delta 读取必须走线程状态 accessor，它编译出的图按「assistant、通道模式、快照节奏、应用配置」缓存。该缓存有自己的容量上限：
+
+| 配置键 | 类型 | 默认值 | 范围 | 是否重启 |
+| --- | --- | --- | --- | --- |
+| `database.checkpoint_graph_cache.accessor_graph_max` | `int` | `64` | `ge=1` | 否——热重载生效 |
+
+| 事实 | 后果 |
+| --- | --- |
+| 调大或调小只改变缓存何时淘汰，从不改变图的语义 | 该值在每次淘汰检查时重新读取，因此热重载无需重启即生效 |
+| 到达上限时整个缓存被清空（这里没有 LRU） | 该上限约束的是不同 assistant 与节奏的数量，而非最近使用顺序 |
+| 配置热重载会重建 `AppConfig` 对象，而缓存条目会重新校验工厂与配置身份 | 热重载绝不会返回按旧配置编译的图 |
+| 不可用的值（非整数、布尔、小于 1）回退到代码内默认值 `64` | 桩配置或残缺配置不会破坏 accessor 缓存 |
+
+这个上限对两种模式都生效，与历史缓存无关——后者只包装 delta saver。它不暴露任何计数，只按容量设界。
+
+## 如何度量缓存效果
+
+`cache_effect_ms`（状态冷读减去温读的 p50）是生产环境 accessor 缓存默认值的决策指标，来自生产检查点基准，该基准还报告 `state_cold_ms`、`state_warm_p50_ms`、`state_warm_p95_ms`、`db_bytes` 以及各 `limit` 下的历史读取耗时。通道基准报告 `warm_read_ms`、`cold_read_ms`、`reducer_replay_ms`，比值一律为 `delta/full`。耗时阈值不构成 CI 门禁。见[运维检查点](/docs/harness/checkpoints/operations)。
+
+## 相关章节
+
+- [快照节奏](/docs/harness/checkpoints/cadence)——本缓存所记忆的那次回溯。
+- [通道模式](/docs/harness/checkpoints/modes)——accessor 与模式标记。
+- [可观测性](/docs/harness/checkpoints/observability)——这些计数在哪里出现。
+- [故障排查](/docs/harness/checkpoints/troubleshooting)——内嵌路径的 redis 报错与零命中缓存。
+- [参考附录](/docs/harness/checkpoints/reference)——完整配置键表。

frontend/src/content/zh/harness/checkpoints/index.mdx

@@ -0,0 +1,84 @@
+---
+title: 检查点与增量通道
+description: DeerFlow 如何持久化线程状态，以及 database.checkpoint_channel_mode 如何在整份消息检查点与 LangGraph DeltaChannel 增量检查点之间取舍。含概念、存储形态与本手册阅读路线。
+asIndexPage: true
+---
+
+import { Callout } from "nextra/components";
+
+# 检查点与增量通道
+
+<Callout type="info" emoji="🗃️">
+  检查点是一个线程在某一步的持久化状态。`database.checkpoint_channel_mode` 决定其中消息通道的存储方式：
+  `full` 把完整消息列表写进每一份检查点；`delta` 不把列表放进逐步检查点，而是追加每步的写入，
+  在读需要时重建列表。默认值是 `full`。
+</Callout>
+
+DeerFlow 的每次运行都会在每一步之后把线程状态写成检查点：消息列表、目标、沙箱句柄、持久上下文，以及中间件贡献的各类通道。在长任务里消息列表占据绝大部分体积，因此它在磁盘上的表示方式，决定了 N 轮之后这个存储到底保留了多少：`full` 模式保留的量是平方级，`delta` 模式则按快照节奏把这一平方项缩小，而不是把它消掉。本手册讲这两种表示、切换它们的开关，以及每种选择的运维代价。
+
+## 如何阅读本手册
+
+| 读者       | 从这里开始                                                                                                                                                                                                                                                                                                                        |
+| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 聊天用户   | 本页、[快速上手](/docs/harness/checkpoints/quick-start)、[通道模式](/docs/harness/checkpoints/modes)                                                                                                                                                                                                                               |
+| 运维人员   | [快速上手](/docs/harness/checkpoints/quick-start)、[快照节奏](/docs/harness/checkpoints/cadence)、[历史缓存](/docs/harness/checkpoints/history-cache)、[运维检查点](/docs/harness/checkpoints/operations)、[可观测性](/docs/harness/checkpoints/observability)、[故障排查](/docs/harness/checkpoints/troubleshooting)                     |
+| 集成开发者 | [通道模式](/docs/harness/checkpoints/modes)、[恢复与回滚](/docs/harness/checkpoints/resume-and-rollback)、[参考附录](/docs/harness/checkpoints/reference)                                                                                                                                                                          |
+
+## 概念
+
+- **检查点（checkpoint）**：一个线程在某一步的通道取值，加上 LangGraph 附加的元数据——检查点 id、父链接、待处理写入、计数器。读取、恢复、分叉都从检查点出发。
+- **通道（channel）**：线程状态 schema 中的一个字段，有自己的值类型和更新规则。编译后的图带有一张 channel table，决定每个通道如何存储与归并。模式只改变 `messages` 这一个通道。
+- **归并器（reducer）**：把传入的写入折叠进通道取值的函数。增量模式的消息归并器是 `merge_message_writes`，以线性时间实现 `add_messages` 的语义。
+
+## 为什么需要增量
+
+`full` 模式下每份检查点都携带完整的通道取值，因此每一步都要重新序列化整个消息列表，而这些检查点都会被保留下来：总存储随轮次按 O(N²) 增长。
+
+`delta` 模式下消息列表是 LangGraph 的 `DeltaChannel`。非快照检查点的 `channel_values` 里不含消息取值，每步的更新以写入的形式追加，读取时通过归并器重放祖先写入来重建列表。这让大多数检查点不再序列化整份列表——但它**没有消除平方项**：每 `snapshot_frequency` 次写入都会落一份 `_DeltaSnapshot`，里面存的是**当时整份**消息列表，而这些快照和别的检查点一样会被保留。
+
+设 K 为 `snapshot_frequency`。容量规划依赖的两个量并不相同：
+
+| 量               | `full`                                           | `delta`                                                                                          |
+| ---------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
+| 单步边际成本     | 把整份列表重新序列化进又一份会被保留的检查点     | 追加该步的写入；另外每 K 次写入还会落一份整份快照                                                  |
+| N 轮后保留的总量 | O(N²)                                            | O(N²/K + N)                                                                                      |
+
+也就是说，delta 是把平方项按 K 大致缩小（默认 `10`），而不是删掉它。快照越少，残余越小，读也越慢——物化要从最近一份快照开始重放（见[快照节奏](/docs/harness/checkpoints/cadence)）。DeerFlow 目前没有任何机制清理这些快照，因此容量应按**保留总量**估算，而不是按单步写入成本（见[运维检查点](/docs/harness/checkpoints/operations)）。
+
+|                  | `full`                                           | `delta`                                                                                          |
+| ---------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
+| 检查点载荷       | 完整 `channel_values`，含整份消息列表            | 非快照步骤不含消息大对象；每 `snapshot_frequency` 次写入出现一个哨兵 `_DeltaSnapshot` 大对象      |
+| 单步更新         | 折叠进下一份检查点的载荷                         | 以逐步写入的形式存储                                                                             |
+| 读取路径         | 直接使用已存储的取值                             | 由最近一份已存快照加重放的写入重建                                                               |
+| N 轮后保留的总量 | O(N²)                                            | O(N²/K + N)，K 为 `snapshot_frequency`                                                           |
+
+模式是**进程级、需要重启的选择**。每个进程在第一次编译图之前解析一次，之后不能改变；共享同一个检查点数据库的所有进程必须使用同一个值。[快速上手](/docs/harness/checkpoints/quick-start)讲切换，[快照节奏](/docs/harness/checkpoints/cadence)讲第二个被冻结的开关——快照节奏。
+
+## 术语
+
+| 术语                          | 含义                                                                                                                                          |
+| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| 检查点（checkpoint）          | 一个线程在某一步持久化的通道取值，加上 LangGraph 的检查点元数据。                                                                             |
+| 通道（channel）               | 线程状态 schema 中的一个字段，有自己的值类型和更新规则。                                                                                       |
+| 归并器（reducer）             | 把写入折叠进通道取值的函数。                                                                                                                   |
+| 增量通道（`DeltaChannel`）    | LangGraph 的归并器通道，检查点中不保存完整取值，而是通过重放祖先写入来重建。                                                                  |
+| 哨兵（sentinel）              | 增量通道在 `channel_values` 中占位存放的标记；已存的 `_DeltaSnapshot` 会终止该通道的祖先遍历。                                                |
+| 种子（seed）                  | 最近的、`channel_values` 中有取值的祖先上存储的值，重建从它出发。                                                                             |
+| 写入（writes）                | 存在独立表中的逐步通道更新。增量模式下它们是承重的重放状态，不是垃圾数据。                                                                     |
+| 物化（materialize）           | 由种子加重放的写入重建通道取值。                                                                                                               |
+| 节奏（`snapshot_frequency`）  | 经过多少次逐步写入之后落一份完整消息快照。                                                                                                     |
+| 祖先链（ancestor chain）      | 检查点的父链。增量检查点的历史是其已封闭祖先链的纯函数。                                                                                       |
+| 分叉（fork）                  | 同一检查点的第二个子节点。增量模式无法正确分叉，恢复会改为在当前 head 上线性化。                                                              |
+| 线性化（linearization）       | 把"从较旧检查点恢复"改写为在当前 head 上的一次替换式写入。                                                                                     |
+| 回滚（rollback）              | 一次被取消的运行之后，把 head 恢复到运行前的状态。                                                                                             |
+| 上下文压缩（compaction）      | 用摘要替换消息列表，通过 `Overwrite` 写入，使替换能穿过追加式归并器。                                                                          |
+| 保留 / 裁剪（retention / prune） | 删除检查点行。增量模式下，重建所依赖的 writes 行必须保留。                                                                                  |
+| 缓存（cache）                 | 增量模式专用的历史缓存，缓存已物化的通道历史。                                                                                                 |
+| 冻结（frozen）                | 在第一次构建图之前每个进程解析一次；之后出现不同取值就是错误，改动需要重启。                                                                   |
+| 网关 / 内嵌（Gateway / embedded） | 异步的 Gateway 路径，对比同步的 TUI 或进程内 `DeerFlowClient` 路径。                                                                       |
+
+<Cards num={3}>
+  <Cards.Card title="快速上手" href="/docs/harness/checkpoints/quick-start" />
+  <Cards.Card title="通道模式" href="/docs/harness/checkpoints/modes" />
+  <Cards.Card title="快照节奏" href="/docs/harness/checkpoints/cadence" />
+</Cards>

frontend/src/content/zh/harness/checkpoints/modes.mdx

@@ -0,0 +1,89 @@
+---
+title: 通道模式
+description: 两种模式下检查点在磁盘上的形态、各自的读取路径、模式标记契约、失败关闭的闸门、409/503 错误映射、跨进程不变量，以及各模式编译出的状态 schema 符号。
+---
+
+import { Callout } from "nextra/components";
+
+# 通道模式
+
+`database.checkpoint_channel_mode` 只有两个取值。模式既决定消息通道怎么写，也决定读取时走哪条物化路径。它按进程冻结，因此线程写入时用的值，与读取进程启动时冻结的值必须一致。
+
+## 存储形态与读取路径
+
+| | `full` | `delta` |
+| --- | --- | --- |
+| 消息列表存放位置 | 每份检查点的 `channel_values` 里 | 逐步写入里。非快照步骤不把该通道放进 `channel_values`；每 `snapshot_frequency` 次写入落一份 `_DeltaSnapshot` 大对象 |
+| 单步代价 | 把整份列表重新序列化进又一份会被保留的检查点 | 追加该步写入；另外每 K 次写入落一份整份 `_DeltaSnapshot`（N 轮后保留总量：O(N²) 对 O(N²/K + N)） |
+| 快照触发 | 不适用——每一步都是完整载荷 | 更新计数达到 `snapshot_frequency`，或 superstep 计数达到 LangGraph 的 `DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT` 上限（5000） |
+| 读取路径 | 直接使用已存储的取值 | 先补齐通道；`channel_values` 中缺失的增量通道合并为一次 saver 调用；再做 `from_checkpoint(seed)` 加 `replay_writes` |
+| 直接读 checkpointer | 返回已存储的取值 | 返回哨兵，而不是消息列表 |
+
+<Callout type="warning">
+  增量检查点不存完整的 `channel_values`——直接读 saver 只能看到哨兵——所以消费者必须走
+  `CheckpointStateAccessor`，不要直接调用 checkpointer。该访问器还会在每份准备好的 config 上重新注入模式标记，
+  写入的是 `configurable` 与 `metadata` 的副本，不会改动调用方的字典。
+</Callout>
+
+superstep 上限的作用，是让停止接收写入的通道重放深度也有界；节奏才是你要调的旋钮。权衡见[快照节奏](/docs/harness/checkpoints/cadence)，物化代价见[历史缓存](/docs/harness/checkpoints/history-cache)。
+
+## 标记契约
+
+| 常量 | 键与位置 | 写入时机 | 含义 |
+| --- | --- | --- | --- |
+| `CHECKPOINT_MODE_METADATA_KEY` | `config["metadata"]` 中的 `deerflow_checkpoint_channel_mode` | 仅 `delta`，取值 `"delta"` | 持久化的模式标记。`full` 模式会移除该键，因此**缺失即 `full`**，功能上线前的检查点无需迁移 |
+| `counters_since_delta_snapshot` | LangGraph 自己的检查点元数据键，一个含 `"messages"` 的字典 | 由 LangGraph 写入 | 兜底识别：即使没有 DeerFlow 标记，只要它存在也按增量处理 |
+| `INTERNAL_CHECKPOINT_MODE_KEY` | `config["configurable"]` 中的 `__deerflow_checkpoint_channel_mode` | 每个请求，两种模式都写 | 仅供内部分流，由 `inject_checkpoint_mode` 写入。它永远不是持久化的元数据键，也不会出现在线上报文里；调用方传入的取值会被 Gateway 的 run-config 构建器丢弃，首次冻结时被忽略，冻结之后再传不同取值则被拒绝 |
+
+快照节奏刻意**不**写入检查点元数据，这也是标记契约不受节奏取值影响的原因：
+
+> It is deliberately NOT stamped into checkpoint metadata: the mode marker contract (absence = full) and the full -> delta migration semantics are unchanged by the frequency value.
+
+## 失败关闭的闸门
+
+闸门在一次访问的两侧都会运行，而且是不对称的：增量模式进程可以透明读取旧的整份检查点，而 `full` 模式进程会拒绝增量线程，而不是悄悄物化出空状态。
+
+| 闸门 | 作用位置 | 行为 |
+| --- | --- | --- |
+| `raise_if_snapshot_incompatible(snapshot, mode)` | 物化读取（`get_state`、`get_state_history`） | 在物化后的 `StateSnapshot` 上抛出；只多一次检查点取数，不额外往返 |
+| `raise_if_checkpoint_tuple_incompatible(tuple, mode)` | 仅元数据读取（`get_metadata`、`aget_metadata`） | 在不做任何物化的前提下保留闸门 |
+| `ensure_checkpoint_mode_compatible(checkpointer, config, mode)` / `aensure_checkpoint_mode_compatible(...)` | 写入之前 | 在写入落库之前失败关闭——写入无法撤销。进程模式为 `delta` 时两者都立即返回 |
+
+被拒绝的访问抛出 `CheckpointModeMismatchError`，消息为 `Thread requires delta mode; materialize and convert its checkpoints before using full mode.`。同一串文本在降级的整份模式只读访问器里也复制了一份，因此即使 agent factory 坏掉，这条消息依然会出现。
+
+## HTTP 错误映射
+
+| 错误 | 状态码 | `detail` | 含义 |
+| --- | --- | --- | --- |
+| `CheckpointModeMismatchError` | 409 | `Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode.` | 线程已持久化的检查点与进程冻结的模式冲突 |
+| `CheckpointModeReconfigurationError` | 503 | `checkpoint_channel_mode is restart-required and cannot change in a running process`（或 `checkpoint_delta.snapshot_frequency` 的对应文本） | 进程自身正处于模式切换中，请求暂时无法服务 |
+
+路由里记录了这一区分的理由：
+
+> A mismatch means the thread's persisted checkpoints conflict with the process's frozen mode (operator-actionable, 409); a reconfiguration means the process itself is mid mode-flip (transient, 503).
+
+只有线程路由映射这两个错误；创建运行的路径会把同一段不一致文本异步地写进 run 记录或 SSE `error` 帧 —— 例外是线程的 run event 流为空、检查点头部必须在准入前被物化时，播种读取会把这个不一致变成普通的 `500`。映射中的 `503` 一半属于防御性代码：重新配置错误在构建图时抛出（Gateway 启动、`make_lead_agent`、`DeerFlowClient.__init__`），而没有任何状态、历史或线程路由会构建 agent，所以改完 `config.yaml` 却没重启的运维人员看到的是失败的 run，而不是 `503`。见[可观测性](/docs/harness/checkpoints/observability)与[故障排查](/docs/harness/checkpoints/troubleshooting)。
+
+## 跨进程不变量
+
+- 模式被编译进每张图的 channel table，而不是按检查点存储，因此共享同一个检查点数据库的**每个**进程都必须一致。
+- 节奏同样必须一致，但它不被打标记，所以逐检查点的不一致无法被发现：运行时根本不做节奏比较。强制只落实在模式层面，靠标记存在与否做失败关闭。请把"各进程 `config.yaml` 完全一致"当作部署要求，而不是建议。
+- 冻结状态是每个操作系统进程内的模块级全局变量。每个进程按自己的启动配置独立冻结，运行时不会替你调和任何一个跑偏的进程。
+
+## 状态 schema
+
+模式决定编译出哪张状态 schema，并把中间件贡献的通道适配到它上面。
+
+| 符号 | 模块 | 作用 |
+| --- | --- | --- |
+| `ThreadState` | `deerflow.agents.thread_state` | 整份模式的 state schema |
+| `DeltaThreadState` | `deerflow.agents.thread_state` | 默认节奏下的增量 schema；只有节奏等于默认值时该名字才保持 |
+| `delta_messages_field(snapshot_frequency=…)` | `deerflow.agents.thread_state` | 构造 `Annotated[list[AnyMessage], DeltaChannel(merge_message_writes, snapshot_frequency=…)]` |
+| `DELTA_MESSAGES_FIELD` | `deerflow.agents.thread_state` | 默认节奏下的模块级增量字段 |
+| `merge_message_writes(state, writes)` | `deerflow.agents.thread_state` | 线性时间、等价于 `add_messages` 的增量归并器 |
+| `get_thread_state_schema(mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | 非增量模式返回 `ThreadState`，否则返回带缓存的增量 schema；非默认节奏会得到按节奏缓存的专用 schema |
+| `adapt_state_schema_for_mode(schema, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | 按模式适配状态 schema，让中间件通道编译到正确的通道类型 |
+| `normalize_middleware_state_schemas(middleware, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | 返回带上已适配 `state_schema` 副本的中间件列表 |
+| `THREAD_STATE_REDUCER_FIELDS` | `deerflow.agents.thread_state` | 归并器通道集合，含 `messages`；用于判断哪些通道的替换式写入需要 `Overwrite` 包装 |
+
+完整的配置键表、其余常量与承重测试锚点见[参考附录](/docs/harness/checkpoints/reference)。

frontend/src/content/zh/harness/checkpoints/observability.mdx

@@ -0,0 +1,116 @@
+---
+title: 可观测性
+description: 没有任何接口暴露检查点模式。本章说明客户端能看到与不能看到什么、模式不匹配时留下的日志行与 SSE 帧、浏览器用户实际看到什么、运维可读的缓存计数器，以及如何判断某个进程或某个线程的模式。
+---
+
+import { Callout } from "nextra/components";
+
+# 可观测性
+
+## 没有任何接口暴露模式
+
+检查点模式是服务端状态。模式与快照节奏都不会出现在任何 health、readiness、features、capabilities 或 console 字段里，也不会出现在任何响应模型、响应头或事件字段里。
+
+- 健康与就绪探针只检测后端可达性；除了 threads 路由的错误映射之外，没有任何路由读取 `checkpoint_channel_mode`。
+- 仅有的自定义 SSE 响应头是 `Cache-Control`、`Connection`、`X-Accel-Buffering` 和 `Content-Location`。
+- Gateway 只在服务端消费该模式，写入 `app.state.checkpoint_channel_mode` / `app.state.checkpoint_snapshot_frequency` 与 run 上下文，不向外序列化。
+
+因此不存在可以轮询的「本部署是否为 delta 模式」字段。你只能从线程的 metadata、受门禁路由上的一次 `409`，或者在进程内部自行判断。
+
+## 客户端能看到与不能看到什么
+
+存在两个模式键，只有一个可能到达客户端。
+
+| 键 | 写入位置 | 是否上线 |
+| --- | --- | --- |
+| `INTERNAL_CHECKPOINT_MODE_KEY` = `"__deerflow_checkpoint_channel_mode"` | `config["configurable"]`，两种模式都写 | 从不。仅服务端内部。 |
+| `CHECKPOINT_MODE_METADATA_KEY` = `"deerflow_checkpoint_channel_mode"` | `config["metadata"]`，**只有** delta 模式写；full 模式移除 | 可以在响应的 `metadata` 对象中透传 |
+
+由于标记只在 delta 模式写入，已落库的 delta 检查点带有它，full 检查点没有；缺省即 full。
+
+有两个读接口用「存储快照的 metadata 减去一个剥离清单」构造响应的 `metadata`，而该清单不包含模式标记与 LangGraph 自己的 `counters_since_delta_snapshot`，因此两者都可能到达客户端：
+
+| 接口 | 响应 `metadata` 来源 |
+| --- | --- |
+| `GET /api/threads/{thread_id}` | `snapshot.metadata` 减去 `created_at`、`updated_at`、`step`、`source`、`writes`、`parents` —— 但仅当不存在 `threads_meta` 行时；否则 `metadata` 来自存储行 |
+| `POST /api/threads/{thread_id}/history` | 每条条目的 `snapshot.metadata` 减去同一清单再加上 `run_durations` 与 run-message-ids 键，然后重新加回 `step` |
+
+这种透传来自处理函数里的剥离清单；没有任何测试断言标记会出现在线上响应中，所以请把它当作实现细节而不是契约。
+
+流帧不携带模式。SSE 的 `metadata` 事件载荷只有 `{"run_id", "thread_id"}`，SSE 的 `error` 事件载荷是 `{"message", "name"}`，其中 `name` 是异常类名。
+
+## 日志与流面
+
+模式问题留下的证据取决于命中它的路径。
+
+| 场景 | 面 | 逐字文本 |
+| --- | --- | --- |
+| run 命中不匹配 | run worker 日志（`exc_info=True`） | `Run %s failed: %s`，`%s` 为不匹配文本 |
+| run 命中不匹配 | run 记录 / `/wait` 响应体 / SSE `error` 帧 | `Thread requires delta mode; materialize and convert its checkpoints before using full mode.` |
+| run 命中重启要求违例 | run worker 日志（`exc_info=True`）、run 记录、SSE `error` 帧 | `Run %s failed: %s`，`%s` 为 `checkpoint_channel_mode is restart-required and cannot change in a running process`（或 `checkpoint_delta.snapshot_frequency` 的对应文本） |
+| 上下文用量命中不匹配 | 日志 | `Failed to load checkpoint for context usage on thread %s` |
+| 重新生成无法读取最新检查点 | 日志，随后 HTTP 500 `detail` | `Failed to read latest checkpoint for regenerate thread %s` → `Failed to read latest checkpoint` |
+| 重新生成无法列出检查点历史 | 日志，随后 HTTP 500 `detail` | `Failed to list checkpoints for regenerate thread %s` → `Failed to inspect checkpoint history` |
+
+失败载荷随路由不同。创建 run 在 threads 路由上**不会**提前做门禁：`POST /api/threads/{thread_id}/runs` 返回 `200` run 记录，随后变为 `status: error` 并带上不匹配文本；`POST /runs/stream` 返回 `200 text/event-stream`，随后发出 `event: error` 帧，载荷为 `{"message": ..., "name": "CheckpointModeMismatchError"}`；`POST /runs/wait` 返回 `200` JSON，内容为 `{"status": "error", "error": "<不匹配文本>"}`。例外是 run event 流为空但存在检查点头部的线程：`ensure_checkpoint_history_seeded` 会在 `start_run` 内物化该头部，而那里没有映射模式错误，于是同一个不匹配会在 run 被准入之前返回普通的 `500`。无状态的 `/api/runs/*` 路由共享这条路径。
+
+<Callout type="warning">
+  模式门自身不记任何日志。`checkpoint_mode.py` 没有 logger，threads
+  路由把模式错误转成 `HTTPException` 时也不记日志，所以一次 `409` 不会
+  单独留下日志行。证据在调用方：run worker 的 `Run %s failed: %s`、run
+  记录，或客户端侧的 toast。
+</Callout>
+
+## 浏览器用户看到什么
+
+前端完全没有模式界面：没有徽章、设置、列或错误文案，也没有针对模式不匹配的 i18n 字符串。但聊天页确实会命中受门禁的路由，所以用户能在不知道「模式」这个词的情况下看到失败。
+
+- 聊天页默认加载持久历史（`fetchStateHistory: { limit: 1 }`），调用 `POST /api/threads/{thread_id}/history`。当 full 模式进程读取 delta 线程时，这正是 `409` 路由。
+- SDK 把响应转成 `HTTPError`，其 message 为 `HTTP 409: <原始响应体文本>`；`409` 在 SDK 的不可重试集合里，因此立即抛出、不重试。
+- 该错误到达流的 `onError`，显示为错误 message 的 toast。用户实际看到的是原始服务端文本，大致为：
+
+  ```
+  HTTP 409: {"detail":"Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode."}
+  ```
+
+  两侧都各自经过核实（`HTTP 409: ` 前缀来自 SDK，`detail` 正文来自路由）；拼接后的整串未在运行期实测。
+- 侧边栏**导出**调用 `getState()` → `GET /api/threads/{thread_id}/state` → `409`，但一个裸 `catch {}` 丢弃了详情，只显示通用的 `exportFailed` toast。
+- **重命名**调用 `updateState()` → `POST /api/threads/{thread_id}/state` → `409` → `toast.error(error.message ?? renameFailed)`，因此这条路径上原始的 HTTPError 字符串可能显现。
+- 流缺口恢复会把失败的 `getState` 包装成 `StreamReplayGapError`；另一处内部 `getState`（重连输入快照）吞掉所有非 abort 错误并返回 `undefined`。
+- 界面里唯一带检查点意味的 toast 文本 `Failed to load thread history.` 属于消息分页路由，不属于任何受模式门禁的路由。
+
+## 缓存计数器
+
+delta 历史缓存只在 Gateway（异步）路径的 delta 模式下包裹 saver。`CachedHistorySaver.stats()` 把后端计数器与两个 wrapper 计数器合并。没有任何 HTTP 路由暴露它们，需在进程内读取。
+
+| 计数器 | 含义 | 由谁报告 |
+| --- | --- | --- |
+| `hits` | 由缓存直接命中的按通道历史查询 | memory、redis |
+| `misses` | 需要组合或回走的查询 | memory、redis |
+| `evictions` | 因 LRU 上限被丢弃的条目 | 仅 memory |
+| `entries` | 当前缓存条目数 | 仅 memory |
+| `compose_hits` | 由「热祖先」（已缓存条目或带快照的父级）组合出的祖先层级数 | wrapper |
+| `full_walks` | 回退到原始内层 saver 自行回走的次数（冷链到达深度预算、缓存被禁用、未找到目标） | wrapper |
+
+redis 后端只报告 `hits` 与 `misses`；它的 `evictions` 与 `entries` 恒为 `0`。`max_entries: 0` 统一禁用缓存，被禁用的缓存会直接透传给原始 saver，完全不做组合。改动 `database.checkpoint_cache.*` 的任何字段后，运行中的 Gateway 需要重启才会报告不同的计数，因为缓存是随 checkpointer 在启动时构造的；见[历史缓存](/docs/harness/checkpoints/history-cache)。
+
+## 判断模式
+
+### 运行中的进程
+
+模式是一个模块级全局量，在进程首次构建 agent 时冻结。`frozen_checkpoint_channel_mode()` 返回冻结值或 `None`，`frozen_checkpoint_snapshot_frequency()` 对节奏同理。Gateway 另外把启动快照存到 `app.state.checkpoint_channel_mode` 与 `app.state.checkpoint_snapshot_frequency`。
+
+启动时不会打印这两个值。实践上：
+
+1. 从 `config.yaml` 读取 `database.checkpoint_channel_mode`（以及 `database.checkpoint_delta.snapshot_frequency`）。
+2. 确认进程是在最后一次改动之后启动的 —— 改配置而不重启不生效，并会在构建新图时明确报错（见[故障排查](/docs/harness/checkpoints/troubleshooting)）。
+3. 记住共享同一检查点数据库的每个进程都必须使用相同的值，且客户端传入的 `configurable.__deerflow_checkpoint_channel_mode` 无法改变进程的模式。
+
+### 某个线程
+
+线程的模式记录在它已落库的检查点里：
+
+- delta 检查点的 metadata 带有 `deerflow_checkpoint_channel_mode: "delta"`；full 检查点没有。
+- LangGraph 自己的 `counters_since_delta_snapshot` metadata 是第二个 delta 信号。
+- 通过 HTTP，可从 `GET /api/threads/{thread_id}`（不存在 `threads_meta` 行时）的 `metadata` 对象，以及 `POST /api/threads/{thread_id}/history` 每条条目的 `metadata` 中读到该标记。
+- 或者在一个你认为是 `full` 的进程里打开该线程：delta 线程会以 `409` 回应。

frontend/src/content/zh/harness/checkpoints/operations.mdx

@@ -0,0 +1,212 @@
+---
+title: 运维检查点
+description: 后端无关性、增量模式下的保留与裁剪约束、仅存的一个导入期补丁、依赖下限、两个检查点基准的跑法，以及 TUI 与内嵌路径的差异。
+---
+
+import { Callout } from "nextra/components";
+
+# 运维检查点
+
+增量模式改变的是线程状态存放的位置，而不是由哪个后端存放。本章讲它的运维后果：后端选择、保留与裁剪约束、仅存的一个兼容补丁、挡住上游缺陷的依赖下限、如何测量两种模式，以及运维在 TUI 与内嵌路径上看到的差异。
+
+## 后端
+
+模式是编译后图的通道表属性，不是 saver 的属性，因此三个后端都能跑两种模式：
+
+| `database.backend` | Saver | 支持两种模式 | 说明 |
+| --- | --- | --- | --- |
+| `memory` | `InMemorySaver` | 是 | 进程内、重启即丢；模式语义完全相同 |
+| `sqlite` | SQLite saver | 是 | 通道值内嵌在序列化后的检查点负载里，没有独立的 blob 表 |
+| `postgres` | Postgres saver | 是 | `checkpoints` 加独立的 `checkpoint_blobs` 表；写入放在 `checkpoint_writes` |
+
+有两条进程级规则与后端无关，且都不由后端来约束：
+
+- **模式必须一致。** 共享同一个检查点数据库的所有进程必须使用相同的 `database.checkpoint_channel_mode`。模式被编译进每个图的通道表，不随检查点存储。
+- **节奏必须一致。** `database.checkpoint_delta.snapshot_frequency` 同样被编译进通道表。它被刻意不写入检查点元数据，因此逐检查点无法检测不一致，这属于运维责任。失败关闭的门只约束*模式*，运行时不做节奏比对。
+
+增量模式在所有后端上都改变存储形态：逐步检查点不再携带整份 `messages` 负载，因此单步的边际成本不再随历史增长。但保留的存储里仍然有每 `snapshot_frequency` 次写入落下的整份 `_DeltaSnapshot`，所以 N 轮之后总量按 O(N²/K + N) 增长（K 为 `snapshot_frequency`）——是按 K 大致缩小，而不是线性。请按保留总量做容量规划，见[检查点与增量通道](/docs/harness/checkpoints)与[通道模式](/docs/harness/checkpoints/modes)。
+
+## 保留与裁剪
+
+### writes 行是重放状态，不是垃圾
+
+`full` 模式下每个检查点都会重新快照累计消息，丢弃中间检查点至少在概念上可行。`delta` 模式下逐步负载放在 **writes** 表里，它们是承重的重放状态：删掉就会切断后续物化所依赖的祖先链。
+
+| 模式 | 逐步负载存放位置 |
+| --- | --- |
+| `full` | 重新快照进检查点负载 |
+| `delta` | writes 表；每 `snapshot_frequency` 步才把完整值快照一次 |
+
+保留契约测试套件直接钉住了这个形态：其增长基线断言增量线程最终 `write_rows > 0`（"delta mode must land per-step payloads in writes"），而 full 模式把所有内容重新快照进检查点负载。两者的字节对比依赖节奏与后端，因此只断言形态。
+
+### 朴素裁剪的失败方式
+
+<Callout type="warning">
+  朴素的 `keep_latest` 若删掉中间检查点及其 writes，就会切断这条链：幸存的
+  「最新」检查点本身很少是快照点，于是它的增量通道会被静默重建为**空**。
+  不会报错——`get_delta_channel_history` 只是返回不了 `seed`。
+</Callout>
+
+上游为使用 `DeltaChannel` 的图给出三种安全做法：
+
+- 从每个保留的检查点往回走，把沿途所有祖先（连同其 writes）一直保留到最近一个 `channel_values` 里已包含每个增量通道 `_DeltaSnapshot` 的祖先。
+- 删祖先之前，先在被保留的检查点上强制落一份新快照，对每个增量通道 `k` 重写 `channel_values[k] = _DeltaSnapshot(value)`。
+- 在上述任一做法实现之前，跳过对使用 `DeltaChannel` 的线程做裁剪。
+
+同样的警告也适用于 `adelete_for_runs` 与 `copy_thread`：删除某个 run 的祖先 writes（或唯一的 `_DeltaSnapshot` blob），会破坏任何依赖这些行的增量通道的重建。
+
+### 保护集
+
+| # | 不得删除，除非满足所述补偿条件 |
+| --- | --- |
+| 1 | **显式恢复目标**——客户端可能仍会恢复到的任意 `checkpoint_id`。策略可以让其过期，但必须先就有明确的 TTL 语义达成一致 |
+| 2 | **分支祖先**——从可分叉 head 回溯到最旧可分支消息之前的那个检查点（含）为止的整条父链。删掉这条链上任何节点都会破坏分支与重新生成，而且是响亮失败而非静默失败 |
+| 3 | **待处理写入**——writes 表里的行是未提交或在途状态，不是垃圾 |
+| 4 | **仅耗时链环**——后续 run 已从中分叉的仅耗时检查点是链环；删除它必须在同一次变更中把分叉嫁接到祖父节点上。孤立叶子是安全的，链环不是 |
+| 5 | **每线程最新的可恢复状态**——最新检查点必须始终可寻址，线程才能一直续跑 |
+
+### 目前真正允许删除的形态
+
+只有两种形态被证明安全，保留服务也只实现这两种：
+
+| 形态 | 默认 | 为什么安全 |
+| --- | --- | --- |
+| 尾部仅耗时叶子 | 开启 | `persist_run_durations` 在 run 结束后追加仅元数据检查点；在后续 run 尚未从中分叉时，它不属于任何人的祖先 |
+| 叶子兄弟分支 | 关闭（需显式开启） | 从较旧轮次分叉且没有子节点的检查点。之所以默认关闭，是因为被取代的那条线上的检查点可能仍是客户端持有的显式恢复目标，因此提供了 `protect_checkpoint_ids` 逃生口 |
+
+删除还带两条机械约束：
+
+- **联合表删除、先算孤儿。** 删除检查点行时必须判定哪些 writes 与 blob 行成了孤儿；只有当**幸存**检查点都不再引用某行时它才算孤儿，且要在整线程范围内计算。真实的仅耗时叶子并非没有负载——它会物化父节点的值——在 Postgres 上它的 blob 行与支撑父节点的正是同一批行。
+- **要么全做，要么不做。** 留下悬空父链接的部分删除会把一次清理变成线程级故障：此后每一轮的分支与重新生成都会响亮失败。
+
+还有一条历史快路径需要排期：尾部的仅耗时叶子同时承载 `run_durations` 与 `run_message_ids` 映射，`get_thread_history` 会从最新检查点读取它们。删掉该叶子会让下一次历史读取退化为扫描事件存储并重新落一份新叶子，因此接线时必须把保留动作与历史读错开，或采用会保留「带缓存的叶子」的策略。
+
+### 策略默认值
+
+| 字段 | 默认值 | 含义 |
+| --- | --- | --- |
+| `prune_trailing_duration_leaves` | `True` | 删除没有后续 run 分叉过的仅耗时叶子 |
+| `prune_leaf_sibling_branches` | `False` | 删除叶子兄弟分支（需显式开启） |
+| `protect_checkpoint_ids` | `frozenset()` | 永不删除的显式 id 集合 |
+| `strict_pending_write_guard` | `True` | 仍持有 writes 行的检查点一律视为受保护 |
+| `max_delete_per_run` | `None` | 单次 run 删除数量的可选上限 |
+
+### 目前没有生产触发点
+
+保留服务（`backend/app/gateway/checkpoint_retention.py`）**没有生产触发点**：保留动作从哪里调用——run 结束后钩子、调度器还是显式管理操作——是维护者随契约一起决定的。DeerFlow 中没有任何地方调用它；`prune` 与 `aprune` 也没有生产调用方：在锁定环境中只有基类实现定义了它们，而基类实现直接抛 `NotImplementedError`。
+
+请把它当作未完成的工作，而不是安全默认值。可执行契约只在 **full 模式**的状态 schema 上证明了分支祖先、显式恢复目标、待处理写入、叶子耗时、叶子兄弟这几类场景；只有增长基线是增量形态的。增量专属的裁剪安全性既未被测试套件证明，也未被实现。
+
+## 仅存的一个补丁
+
+只剩一个兼容补丁，它在模块自身导入时应用。该模块位于包根目录，以便从 `deerflow.agents.thread_state` 导入时不必拉进更重的 `deerflow.runtime` 包；补丁锚定在那里，因此所有会构建 DeerFlow 图的进程——网关、worker、进程内 LangGraph 运行时、测试——都带着该修复运行。
+
+| 方面 | 细节 |
+| --- | --- |
+| 对象 | `BinaryOperatorAggregate` 把写入空（MISSING）通道的首个 `Overwrite` 解开 |
+| 跟踪 issue | DeerFlow **#4380** |
+| 它避免的症状 | 类型为 Union 的通道（`SandboxState \| None`、`GoalState \| None` 等）没有可构造默认值，因此以 MISSING 起始。此时对全新线程做替换式写入（线程分叉）或对从未写过的通道做状态更新，会把 `Overwrite` 包装本身持久化进检查点，下一个消费者就读到崩溃：`TypeError: 'Overwrite' object is not subscriptable` |
+| 作用范围 | 只拦截「空通道 + 首个 `Overwrite`」这一种情况。后续的普通值被忽略；同一个 super-step 里的第二个 `Overwrite` 抛出 `InvalidUpdateError`，消息为 `Can receive only one Overwrite value per super-step.`。其余情况全部委托给上游实现 |
+| 一致性副作用 | `DeltaChannel.update` 在同样情况下本来就会解开，因此该补丁同时消除了两类归并器通道之间的行为差异 |
+
+它的守卫是**行为探针，不是版本钉**。探针构造一个 Union 类型的 `BinaryOperatorAggregate`（通道因此以 MISSING 起始），再读取它判断是否得到 `Overwrite` 包装；若上游已经解开，探针就会报告缺陷不存在，补丁自动退场。应用是幂等的；意外失败时记录 `Failed to apply the BinaryOperatorAggregate Overwrite first-write patch; leaving the upstream implementation untouched.` 并保持上游实现不变。
+
+<Callout type="warning">
+  不要重新引入版本守卫式的 saver 补丁。此前这里的守卫读取的是 `langgraph` 的
+  版本，但 `InMemorySaver` 属于独立发布的 `langgraph-checkpoint` 包，那样
+  的守卫看不到真正的依赖。前一个 `InMemorySaver` 增量历史补丁是被**删除**
+  而非加守卫的：上游在 `langgraph-checkpoint` 4.2.0 修掉了迁移后首个写入
+  丢失的问题（上游 #8526），同时保留了自己的 override，行为探针无法察觉。
+</Callout>
+
+## 依赖下限与回归门禁
+
+依赖下限正是挡住已删除的 saver 缺陷的东西，因此这些下限是承重的，不是装饰：
+
+| 发行包 | 下限 | 原因 |
+| --- | --- | --- |
+| `langgraph-checkpoint` | `>=4.2.0,<5.0` | 4.2.0 修掉了迁移后首个写入丢失（上游 #8526）。解析到 4.1.x 会静默重新引入该缺陷，而 `langgraph` 自己的 `langgraph-checkpoint>=4.1.0` 约束并不排除该版本 |
+| `langgraph-checkpoint-postgres` | `>=3.1.2,<3.2` | 3.1.2 能找到纯值形式的增量种子（上游 #8535）；3.1.1 会越过它一路走到线程根 |
+| `langgraph-checkpoint-sqlite` | `>=3.1.1,<3.2` | 与其他 saver 一同锁定 |
+
+回归门禁是 `backend/tests/test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`：当 4.2.0 的修复缺失时它会失败，保证 full→delta 迁移仍能在同一线程上回放。锁定环境解析到 `langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`。
+
+运行时没有版本检查，也没有告警。运维能看到的唯一版本信号，是一条带有锁定版本号的触发式跳过测试，例如 `#8448` 的 Postgres 跳过或 `#8382` 的回放顺序跳过。
+
+## 如何跑基准
+
+两个脚本覆盖两层，各有一个汇总脚本。它们输出的比例一律是 `delta/full`，且时间阈值不是 CI 门禁。
+
+| 脚本 | 层次 | 测什么 |
+| --- | --- | --- |
+| `scripts/benchmark/checkpoint/bench_channels.py` | 通道层 | 每个用例在全新的子进程里跑成对的 `full` / `delta` 纯消息 StateGraph，使用同步 `InMemorySaver` 或 `SqliteSaver`，从而把归并器、序列化与 saver 的成本同网关调度分开。Postgres 用例需通过 `TEST_POSTGRES_URI` 显式开启 |
+| `scripts/benchmark/checkpoint/bench_production.py` | 生产层 | 在真实 Lead Agent 图上跑图层级 `ainvoke`（脚本化确定性模型、真实 `AsyncSqliteSaver`），再经真实网关路由栈测 `GET /threads/{id}/state` 与 `POST /threads/{id}/history`，区分冷/热样本 |
+
+```bash
+cd backend
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/bench_channels.py \
+  --backends sqlite --updates 100,500,999,1000,1001 --payload-bytes 128 \
+  --repetitions 7 --output /tmp/checkpoint-bench.jsonl
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/summarize_channels.py \
+  /tmp/checkpoint-bench.jsonl
+```
+
+```bash
+cd backend
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/bench_production.py \
+  --turns 10,100,500,1000,2000 --payload-bytes 128 \
+  --snapshot-frequencies 10,50,100,500,1000 \
+  --repetitions 7 --output /tmp/production-bench.jsonl
+PYTHONPATH=. uv run python scripts/benchmark/checkpoint/summarize_production.py \
+  /tmp/production-bench.jsonl
+```
+
+通道层控制器会在用例之间交替模式顺序，并在成对模式物化出不同状态时拒绝性能数据；生产层控制器把每个增量频率与同一行 full 配对，一旦物化摘要或线上摘要不一致就把这一对两行都判失败。
+
+### 仓库中仅有的实测数字
+
+已提交的实测值只有三个，都来自首次基准运行的运维限制记录：
+
+| 观测 | 数值 |
+| --- | --- |
+| 增量模式、`snapshot_frequency=1000`、500 轮 | 约 1100–1200 秒墙钟，因此默认的 `--timeout-seconds 900` 不够。任何大规模矩阵都要显式传超时 |
+| 增量模式、`snapshot_frequency=1000`、2000 轮 | 约 45 分钟；2000 轮这个角落只在较小快照频率下才实际可行 |
+| full 模式、2000 轮、SQLite | 约 33 GB 的数据库文件。把 `TMPDIR` 指向真实磁盘而非 tmpfs，否则用例中途就会挂掉 |
+
+另有一条常被误当成基准限制的路由约束：历史路由把 `limit` 截到 100，因此大于 100 的 `--history-limits` 实际是按截断后的上限测量并上报的。
+
+<Callout type="warning">
+  有两类数字不得当作实测呈现。**合成测试夹具**——汇总器单测里喂进去的比例字面量，
+  例如 `ratio_write_total_ms: 0.5`、`ratio_logical_checkpoint_bytes: 0.2`、
+  `delta_snapshot_write_spike: 5.0`、`delta_cache_effect_ms: 45.0`——描述的是夹具
+  输入，不是真实运行。另外，未跟踪 worktree RFC 中针对 200 轮的某个存储缩减倍数
+  来自上游、并非本地实测，且本地后续计划明确禁止在当前配置下引用它——不要把它
+  写进性能结论。仓库里完全没有提交过任何基准结果文件。
+</Callout>
+
+## TUI 与内嵌路径的差异
+
+`deerflow` 控制台脚本是 `deerflow.tui.cli:main`；`--print` 与 `--json` 是它的无界面变体，其余情况是 Textual TUI。两者都使用内嵌客户端，而内嵌客户端与网关的差异在增量模式下有实际影响。
+
+| 方面 | 网关（异步） | TUI / 内嵌（同步） |
+| --- | --- | --- |
+| Checkpointer | 异步 provider 按进程构建并包装 saver | `get_checkpointer()` 返回进程级**同步**单例 |
+| 历史缓存后端 | `memory` 与 `redis` | 仅 `memory` |
+| Agent 构建失败 | `full` 模式降级为原始 checkpointer 读取；`delta` 原样抛出 | 完全没有降级路径 |
+| 跨模式不一致 | 状态与历史路由返回 HTTP 409；run 以 SSE `error` 帧或 error 运行记录失败 | 在构建 Agent 或开始流的图之前就在进程内抛出 |
+| 冻结 | 在网关启动时按启动配置快照冻结 | `DeerFlowClient.__init__` 按自己的 `AppConfig` 冻结模式与节奏，重启要求语义相同 |
+
+有三点后果值得直说：
+
+- **同步路径拒绝 redis。** 选择 `database.checkpoint_cache.type: redis` 会在构建 checkpointer 时抛出 `ValueError("database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.")`，因此对网关合法的配置可能让 TUI 起不来。这是诚实的拒绝而非降级：同步路径本来就是进程内的。
+- **没有降级读取路径。** 增量物化需要图的通道表，TUI 没有网关那种退回原始 checkpointer 读取的等价物。
+- **未捕获的模式不一致在无头入口上会变成 traceback。** 内嵌客户端在进程内抛出 `CheckpointModeMismatchError`——没有 HTTP 状态码、没有运行记录、没有 SSE 帧——而 CLI 直接分派到它的 print 或 JSON 入口且没有处理器，于是 `--print`、`--json` 在 stderr 上打印一段 Python traceback。Textual TUI 是例外：它的 `stream_actions()` 会捕获来自 `client.stream()` 的每个异常并发出 `AssistantError` 与 `RunEnded`，所以交互用户是在界面里看到这次失败。
+
+第三方嵌入者还会撞上相关守卫：`create_deerflow_agent` 在 `checkpoint_channel_mode="delta"` **且**传入 checkpointer 时抛出 `ValueError`，因为在那里构建的图绕过了检查点模式标记注入与失败关闭的兼容门，混合模式的存储会静默损坏线程状态。不带 checkpointer 的 delta 是临时的，允许使用；需要持久化的增量图必须走受守卫的应用路径（`make_lead_agent` 或 `DeerFlowClient`）。
+
+## 相关章节
+
+- [通道模式](/docs/harness/checkpoints/modes) —— 模式标记、失败关闭的门以及 HTTP 边界。
+- [快照节奏](/docs/harness/checkpoints/cadence) —— `snapshot_frequency` 如何在检查点体积与物化成本之间取舍。
+- [历史缓存](/docs/harness/checkpoints/history-cache) —— 纯性能缓存，其策略允许在各 worker 之间不同。
+- [故障排查](/docs/harness/checkpoints/troubleshooting) —— 上述每种失败特征对应的症状索引。

frontend/src/content/zh/harness/checkpoints/quick-start.mdx

@@ -0,0 +1,72 @@
+---
+title: 快速上手
+description: 把 DeerFlow 的检查点存储切换到增量模式：两个配置键及其默认值、为什么需要重启、新装默认拿到什么、如何确认切换已生效，以及受支持的迁移方向。
+---
+
+import { Callout } from "nextra/components";
+
+# 快速上手
+
+`database:` 下的两个键决定检查点的表示方式，以及增量模式下多久落一份完整消息快照。两者都需要重启。
+
+```yaml
+database:
+  checkpoint_channel_mode: full     # full | delta
+  checkpoint_delta:
+    snapshot_frequency: 10          # >= 1
+```
+
+| 键 | 类型 | 默认值 | 范围 | 是否重启 |
+| --- | --- | --- | --- | --- |
+| `database.checkpoint_channel_mode` | `"full"` \| `"delta"` | `"full"` | `full`、`delta` | 是 |
+| `database.checkpoint_delta.snapshot_frequency` | int | `10` | `>= 1`，无文档化的上限 | 是 |
+
+没有任何环境变量能选择模式或节奏，两者只来自 `config.yaml`。未知的模式取值会在加载配置时被 pydantic 校验拒绝，`snapshot_frequency` 小于 1 同样如此。`snapshot_frequency` 只在 `delta` 模式下起作用；`config.example.yaml` 里两个键都以默认值出现。
+
+## 为什么需要重启
+
+两个值都在首次使用时冻结：进程把它们存进两个模块级全局变量，之后再出现不同的取值就会抛出 `CheckpointModeReconfigurationError`：
+
+- `checkpoint_channel_mode is restart-required and cannot change in a running process`
+- `checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process`
+
+节奏和模式一起被冻结，因为它并不存在检查点里：它在图被编译时就已经编进每张图的 channel table。用不同节奏启动的进程会把那个节奏施加到同一批线程上，而节奏又刻意不写入检查点元数据，读取时没有任何东西可以用来发现这种不一致。同样的推理让模式成为一个跨进程不变量：共享同一个检查点数据库的所有进程都必须使用同一个值。
+
+冻结发生在编译图之前，覆盖每一条会构建图的路径：Gateway 的 lifespan、Lead Agent 装配，以及 `DeerFlowClient.__init__`。两个值都没有热重载。
+
+## 新装默认拿到什么
+
+| 配置项 | 全新安装的取值 |
+| --- | --- |
+| `database.backend` | 缺少 `database:` 段或该键时为 `sqlite`，`sqlite_dir: .deer-flow/data` |
+| `database.checkpoint_channel_mode` | `full` |
+| `database.checkpoint_delta.snapshot_frequency` | `10` |
+| `database.checkpoint_graph_cache.accessor_graph_max` | `64` |
+| `database.checkpoint_cache` | 进程内 memory LRU：`max_entries: 128`、`redis_url: null`、`ttl_seconds: 86400`、`key_prefix: ""`。示例文件把整段注释掉了 |
+
+缺少 `database:` 段时解析结果是 `sqlite` 加 `full`，不会是内存后端：`memory` 只是直接构造 `DatabaseConfig()` 时的默认值。如果你的 `config.yaml` 早于这个键，`make config-upgrade` 会以安全默认值 `full` 合并进来。
+
+## 确认切换已生效
+
+1. **进程内。** `frozen_checkpoint_channel_mode()` 返回已冻结的模式；进程尚未冻结时返回 `None`。`frozen_checkpoint_snapshot_frequency()` 是它的节奏版本。两者都在 `deerflow.runtime.checkpoint_mode` 里。
+2. **线程上。** 增量模式进程写出的检查点带有元数据标记 `deerflow_checkpoint_channel_mode: "delta"`。`full` 模式会移除该键，因此缺失即 `full`，已有检查点不需要迁移。
+3. **不一致时。** 如果进程仍处于 `full` 模式，打开一个增量线程会返回 HTTP 409，`detail` 为 `Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode.`。这是失败关闭的闸门在正常工作，不是需要消除的错误；见[通道模式](/docs/harness/checkpoints/modes)。
+
+没有任何健康检查、就绪或能力接口会报告模式，所以能用的就是上面这三条。
+
+## 迁移方向
+
+增量模式进程可以透明地读取旧的整份检查点，因此 **full → delta 是受支持的方向**：已有线程无需重写数据即可继续工作。反方向并不对称。运行时模块文档字符串写道：
+
+> a full-mode process opening a delta thread raises CheckpointModeMismatchError instead of silently materializing empty state. Delta-mode processes read legacy full checkpoints transparently, so full -> delta is the supported migration path.
+
+因此把存储切回 `full` 之前，必须先物化并转换这些线程的检查点，错误文本本身就是在告诉运维人员这件事。在完成之前，请让所有接触该存储的进程保持 `delta`。
+
+<Callout type="warning">
+  只改 `config.yaml` 里的 `checkpoint_channel_mode` 而不重启不会生效：run worker 会注入冻结的模式，
+  因此这次改动在重启前一直被忽略。改 `checkpoint_delta.snapshot_frequency` 而不重启则更响——
+  下一次 run 会失败，run 记录与 SSE `error` 帧里带着 `CheckpointModeReconfigurationError`。
+  这两种改动都不会让任何请求返回 `503`；见
+  [故障排查](/docs/harness/checkpoints/troubleshooting#改完配置后-run-失败)。请通过重启所有共享
+  该检查点数据库的进程来发布改动。
+</Callout>

frontend/src/content/zh/harness/checkpoints/reference.mdx

@@ -0,0 +1,388 @@
+---
+title: 参考附录
+description: 双模检查点存储涉及的全部配置键、常量、异常、HTTP 状态、对外与内部符号、承重测试锚点、上游缺陷固定表，以及本特性的变更记录。
+---
+
+import { Callout } from "nextra/components";
+
+# 参考附录
+
+`database.checkpoint_channel_mode` 涉及的每一个名称、键、字符串与测试锚点都汇总在本页。本页的用途是查阅而不是理解模型；需要概念请先读[通道模式](/docs/harness/checkpoints/modes)。
+
+「证据」列的取值是仓库相对路径的 `path:line`，使用下表的别名。本页出现的每条事实都能追溯到这些行。
+
+| 别名 | 路径 |
+| --- | --- |
+| `dbcfg` | `backend/packages/harness/deerflow/config/database_config.py` |
+| `appcfg` | `backend/packages/harness/deerflow/config/app_config.py` |
+| `reload` | `backend/packages/harness/deerflow/config/reload_boundary.py` |
+| `cmode` | `backend/packages/harness/deerflow/runtime/checkpoint_mode.py` |
+| `cstate` | `backend/packages/harness/deerflow/runtime/checkpoint_state.py` |
+| `tstate` | `backend/packages/harness/deerflow/agents/thread_state.py` |
+| `factory` | `backend/packages/harness/deerflow/agents/factory.py` |
+| `client` | `backend/packages/harness/deerflow/client.py` |
+| `worker` | `backend/packages/harness/deerflow/runtime/runs/worker.py` |
+| `agent` | `backend/packages/harness/deerflow/agents/lead_agent/agent.py` |
+| `goalsrc` | `backend/packages/harness/deerflow/runtime/goal.py` |
+| `checkpoint_patches` | `backend/packages/harness/deerflow/checkpoint_patches.py` |
+| `syncprov` | `backend/packages/harness/deerflow/runtime/checkpointer/provider.py` |
+| `asyncprov` | `backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py` |
+| `cacheprov` | `backend/packages/harness/deerflow/runtime/checkpoint_cache/provider.py` |
+| `threads` | `backend/app/gateway/routers/threads.py` |
+| `thread_runs` | `backend/app/gateway/routers/thread_runs.py` |
+| `runs` | `backend/app/gateway/routers/runs.py` |
+| `services` | `backend/app/gateway/services.py` |
+| `deps` | `backend/app/gateway/deps.py` |
+| `context_usage` | `backend/app/gateway/context_usage.py` |
+| `example` | `config.example.yaml` |
+
+缺陷表引用的固定依赖集为 `langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`。
+
+## 配置键
+
+下表列出与检查点相关的全部 `database.*` 键，以及加载器实际强制执行的约束。「重启」表示运行中的进程在启动时就捕获了该值——冻结进进程状态、编译进通道表，或在构造 checkpointer 及其历史缓存这类单例时绑定——因此改动要到该进程重启后才生效。
+
+| 键 | 类型 | 默认值 | 范围 | 重启 | 证据 |
+| --- | --- | --- | --- | --- | --- |
+| `database.backend` | `Literal["memory", "sqlite", "postgres"]` | pydantic 默认 `"memory"`；`config.yaml` 缺该键时为 `sqlite` | `memory` \| `sqlite` \| `postgres` | 是 | `dbcfg:143-146`、`appcfg:67-70`、`appcfg:519-527` |
+| `database.checkpoint_channel_mode` | `CheckpointChannelMode` = `Literal["full", "delta"]` | `"full"` | `full` \| `delta` | 是——首次使用时冻结；之后更改为不同值会抛 `CheckpointModeReconfigurationError` | `dbcfg:59`、`dbcfg:147-155`、`cmode:39-45` |
+| `database.checkpoint_delta.snapshot_frequency` | `int` | `10`（`DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY`） | `ge=1`（无上界） | 是——与模式一同冻结 | `dbcfg:61`、`dbcfg:74-84`、`cmode:53-69` |
+| `database.checkpoint_graph_cache.accessor_graph_max` | `int` | `64` | `ge=1` | 否——热重载，每次淘汰检查时重读 | `dbcfg:96-99`、`services:1133-1138` |
+| `database.checkpoint_cache.type` | `Literal["memory", "redis"]` | `"memory"` | `memory` \| `redis`；`redis` 仅限异步/Gateway | 是——构造 checkpointer 时捕获 | `dbcfg:113-116`、`syncprov:181-182`、`asyncprov:250-254` |
+| `database.checkpoint_cache.max_entries` | `int` | `128` | `ge=0`；`0` 完全禁用缓存 | 是——构造 checkpointer 时捕获 | `dbcfg:117-121`、`cacheprov:76-84`、`deps:463` |
+| `database.checkpoint_cache.redis_url` | `str \| None` | `None` | 任意 redis URL；省略时依次取 `DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL`、`REDIS_URL`、`redis://localhost:6379/0` | 是——构造 checkpointer 时捕获 | `dbcfg:122-125`、`cacheprov:19-23` |
+| `database.checkpoint_cache.ttl_seconds` | `int` | `86400` | `ge=0`；`0` 显式关闭过期 | 是——构造 checkpointer 时捕获 | `dbcfg:126-135` |
+| `database.checkpoint_cache.key_prefix` | `str` | `""` | 任意字符串；空值则由数据库标识派生哈希 | 是——构造 checkpointer 时捕获 | `dbcfg:136-139`、`cacheprov:60-64` |
+| `database.checkpoint_delta_snapshot_frequency`（已废弃的扁平键） | `int` | 无——会迁移到嵌套键 | 迁移后为 `ge=1` | 不适用 | `dbcfg:218-254` |
+
+### 各键影响什么
+
+- `database.checkpoint_channel_mode` 决定 `messages` 通道的存储形态：整份取值（`full`）还是 LangGraph `DeltaChannel` 哨兵加逐步写入（`delta`）。它影响一个线程的每一次读写。
+- `database.checkpoint_delta.snapshot_frequency` 是 delta 模式下每 N 次逐步写入落一份完整 messages 快照的节奏。值越大检查点越小、物化越慢。`full` 模式下被忽略。
+- `database.checkpoint_graph_cache.accessor_graph_max` 限制进程缓存的已编译线程状态访问器图数量，键按 assistant、通道模式与快照节奏组合。与模式和节奏不同，它不需要重启：容量不同只会改变缓存何时淘汰，不会改变图的语义。
+- `database.checkpoint_cache.*` 只配置 delta 模式下的物化历史缓存。它纯属性能优化、永不冻结——没有任何正确性不变量要求跨进程一致，禁用缓存时结果也完全相同——但它**不是热重载**：缓存对象在启动时随 checkpointer 一起构造（`deps:463`、`asyncprov:250-254`），而 `reload` 把整个 `database` 段登记为 startup-only（`reload:47`），所以改动其中任何一项都需要重启。memory 后端受 `max_entries` 约束，redis 后端受 `ttl_seconds` 与服务器自身 maxmemory 策略约束。
+- `database.backend` 决定由哪种 saver 承载线程（`InMemorySaver`、SQLite 或 Postgres）。模式语义位于已编译的通道表而不是 saver，所以三种后端都支持两种模式。
+
+两个冻结键的 `Field(description=...)` 原文值得读一遍，因为它写明了跨进程规则：
+
+```text
+"Checkpoint representation for accumulating channels. 'full' preserves full-value message checkpoints;
+'delta' uses LangGraph DeltaChannel for messages. Restart is required, and all processes sharing one
+checkpoint database must use the same value."
+```
+
+```text
+"DeltaChannel snapshot cadence: a full messages snapshot is stored every N per-step writes
+(higher = smaller checkpoints, slower materialization). Restart is required, and all processes
+sharing one checkpoint database must use the same value."
+```
+
+### 校验行为
+
+| 输入 | 结果 | 证据 |
+| --- | --- | --- |
+| `checkpoint_channel_mode` 取 `full`/`delta` 之外的值（例如 `auto`） | 加载配置时抛 pydantic `ValidationError` | `backend/tests/test_app_config_reload.py:118-120` |
+| `snapshot_frequency` 取 `0` 或 `-1` | 抛 pydantic `ValidationError`（约束 `ge=1`） | `dbcfg:76`、`backend/tests/test_app_config_reload.py:131-134` |
+| 已废弃的扁平键取 `0` 或 `-1` | 取值迁移到嵌套键后同样抛 `ValidationError` | `backend/tests/test_app_config_reload.py:154-157` |
+| `database:` 下的未知键 | 被忽略（pydantic `extra="ignore"`），这正是旧键兼容垫片存在的原因 | `dbcfg:224-227` |
+| 扁平键与 `checkpoint_delta.snapshot_frequency` 同时设置 | 嵌套键胜出，并记一条警告 | `dbcfg:218-240` |
+
+### 已废弃的扁平键与迁移告警
+
+旧的顶层键 `database.checkpoint_delta_snapshot_frequency` 仍被 pydantic 的 `@model_validator(mode="before")`（名为 `_migrate_legacy_snapshot_frequency`）接受。它只会记录以下三条字符串之一，全部对运维可见：
+
+| 条件 | 原文 | 证据 |
+| --- | --- | --- |
+| 两个键都设置，且 `checkpoint_delta` 是 dict | `Both database.checkpoint_delta_snapshot_frequency (deprecated) and database.checkpoint_delta.snapshot_frequency are set; the nested key wins.` | `dbcfg:236-238` |
+| `checkpoint_delta` 是已构造好的 `CheckpointDeltaConfig` 对象 | `Ignoring deprecated database.checkpoint_delta_snapshot_frequency because database.checkpoint_delta is already set.` | `dbcfg:246-248` |
+| 旧值被前移 | `database.checkpoint_delta_snapshot_frequency is deprecated; use database.checkpoint_delta.snapshot_frequency instead. Carried the legacy value (%r) forward.` | `dbcfg:250-253` |
+
+### 环境变量
+
+没有任何环境变量能选择检查点模式或快照节奏，两者只来自 `config.yaml`。本特性唯一相关的环境变量是历史缓存 redis URL 的回退链（`DEER_FLOW_CHECKPOINT_CACHE_REDIS_URL`、`REDIS_URL`、`redis://localhost:6379/0`），见 `cacheprov:19-23`。配置值里的 `$VAR` 插值由 `AppConfig.resolve_env_variables()` 在构造 `DatabaseConfig` 之前统一处理，因此 `DatabaseConfig` 自身不做任何环境处理（`dbcfg:13-22`、`appcfg:442`）。
+
+### 配置版本
+
+该 schema 变更随 `config_version 27` 发布；`config.example.yaml` 当前是 `config_version: 47`，缺少版本号按 `0` 处理（`example:23`、`appcfg:531-575`）。过旧的 `config.yaml` 会记录：
+
+```text
+Your config.yaml (version %d) is outdated — the latest version is %d. Run `make config-upgrade` to merge new fields into your config.
+```
+
+`make config-upgrade` 会以安全默认值 `full` 合并新键（`appcfg:570-575`、`backend/tests/test_config_version.py:151-203`）。通用升级流程见[配置](/docs/harness/configuration)。
+
+## 常量与键
+
+| 符号 | 取值 | 所在模块 | 证据 |
+| --- | --- | --- | --- |
+| `CheckpointChannelMode` | `Literal["full", "delta"]` | `deerflow.config.database_config` | `dbcfg:59` |
+| `DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY` | `10` | `deerflow.config.database_config` | `dbcfg:61` |
+| `INTERNAL_CHECKPOINT_MODE_KEY` | `"__deerflow_checkpoint_channel_mode"`（`RunnableConfig` 的 `configurable` 键） | `deerflow.runtime.checkpoint_mode` | `cmode:18` |
+| `CHECKPOINT_MODE_METADATA_KEY` | `"deerflow_checkpoint_channel_mode"`（检查点 metadata 键；写入的值为 `"delta"`） | `deerflow.runtime.checkpoint_mode` | `cmode:19`、`cmode:86` |
+| `counters_since_delta_snapshot` | LangGraph 自有的 metadata 键；只要其值是含 `"messages"` 的 dict，即使没有 DeerFlow 标记也按 delta 处理 | 上游 `langgraph-checkpoint`；检测位于 `deerflow.runtime.checkpoint_mode` | `cmode:97-98` |
+
+标记语义集中在一处：`inject_checkpoint_mode(config, mode)` 总是写入 `config["configurable"]["__deerflow_checkpoint_channel_mode"]`；delta 模式下还会写入 `config["metadata"]["deerflow_checkpoint_channel_mode"] = "delta"`，而 full 模式下会**移除**该 metadata 键，所以缺失即 full，特性上线前的检查点无需迁移（`cmode:81-88`）。快照节奏被刻意**不**写进检查点 metadata，因为模式标记契约与 full→delta 迁移语义都不随频率取值改变（`cmode:53-60`）。
+
+## 异常与 HTTP 状态
+
+| 类 | 基类 | docstring | 证据 |
+| --- | --- | --- | --- |
+| `CheckpointModeMismatchError` | `RuntimeError` | `"Raised before a full-mode graph reads a Delta checkpoint."` | `cmode:22-23` |
+| `CheckpointModeReconfigurationError` | `RuntimeError` | `"Raised when a process attempts to hot-switch its persistence mode."` | `cmode:26-27` |
+
+模式机制及其守卫抛出的原文消息：
+
+| 原文消息 | 抛出位置 | 证据 |
+| --- | --- | --- |
+| `checkpoint_channel_mode is restart-required and cannot change in a running process` | `freeze_checkpoint_channel_mode` → `CheckpointModeReconfigurationError` | `cmode:44` |
+| `checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process` | `freeze_checkpoint_snapshot_frequency` → `CheckpointModeReconfigurationError` | `cmode:68` |
+| `snapshot frequency must be positive` | `freeze_checkpoint_snapshot_frequency` → `ValueError`，仅当绕过 pydantic 传入非正值 | `cmode:63-64` |
+| `Thread requires delta mode; materialize and convert its checkpoints before using full mode.` | `CheckpointModeMismatchError`，由 `raise_if_snapshot_incompatible` 与 `raise_if_checkpoint_tuple_incompatible` 抛出，并在降级网关访问器中重复 | `cmode:123`、`cmode:129`、`services:1229` |
+| `database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.` | `_wrap_sync_if_delta` 中的 `ValueError` | `syncprov:181-182` |
+| `f"Unknown checkpoint cache type: {config.type!r}"` | `make_checkpoint_cache` 中的 `ValueError` | `cacheprov:102` |
+| `create_deerflow_agent does not support checkpoint_channel_mode='delta' with a checkpointer: persisted graphs built here bypass checkpoint mode marker injection and the fail-closed compatibility gate (see deerflow.runtime.checkpoint_mode), so a mixed-mode store would silently corrupt thread state.  Use the guarded application paths (make_lead_agent or DeerFlowClient) for delta persistence; delta without a checkpointer is ephemeral and allowed.` | `create_deerflow_agent` 中的 `ValueError` | `factory:135-141` |
+
+### HTTP 状态映射
+
+只有 threads 路由导入这些异常。它注册门禁元组 `_CHECKPOINT_MODE_ERRORS = (CheckpointModeMismatchError, CheckpointModeReconfigurationError)`（`threads:82`），映射如下。响应体是 FastAPI 默认的 `{"detail": "<string>"}`。
+
+| 条件 | 状态码 | `detail` | 证据 |
+| --- | --- | --- | --- |
+| `CheckpointModeMismatchError` | `409` | `f"Thread {thread_id}: {exc}"`，即 `Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode.` | `threads:105-106` |
+| `CheckpointModeReconfigurationError` | `503` | `str(exc)` | `threads:107` |
+
+路由给出的理由：不匹配说明线程持久化的检查点与进程冻结的模式冲突（运维可处理，409）；重新配置说明进程自身正处于模式切换中（暂时性，503）（`threads:99-102`）。`503` 这一半属于防御性代码 —— 这些路由中没有任何路径会抛出它。冻结辅助函数只被 Gateway 启动（`deps:451-452`）、`agent:835,839` 与 `client:230-231` 调用，而没有任何状态、历史或线程路由会构建 agent：`checkpoint_channel_mode` 的改动会被 run worker 注入的值覆盖（`worker:1018-1019`，优先采用于 `agent:831-835`），在重启前被忽略；`snapshot_frequency` 的改动则会让下一次 run 失败（`worker:1450-1462`），而不是返回某个状态码。
+
+### 返回 409 的路由
+
+| 路由 | 处理器 | 说明 | 证据 |
+| --- | --- | --- | --- |
+| `GET /api/threads/{thread_id}` | `get_thread` | 在 `404 Thread <id> not found` 之前检查 | `threads:1276,1291-1292,1298-1299,1303` |
+| `GET /api/threads/{thread_id}/state` | `get_thread_state` | | `threads:1446,1454-1455,1458-1459` |
+| `POST /api/threads/{thread_id}/state` | `update_thread_state` | 调用方传入的 `checkpoint_id` 若比 head 更旧，会作为一次增量分叉被直接应用，没有线性化 —— 见[分叉会被接受，而不是被拒绝](/docs/harness/checkpoints/resume-and-rollback#分叉会被接受而不是被拒绝) | `threads:1495,1559-1560` |
+| `POST /api/threads/{thread_id}/history` | `get_thread_history` | | `threads:1695,1715-1716,1907-1908` |
+| `POST /api/threads/{thread_id}/branches` | `branch_thread` → `_branch_thread_with_reservation` | detail 中的线程 id 是源线程；无测试覆盖该路由 | `threads:953,231-232,298-299,1105-1106` |
+| `POST /api/threads/{thread_id}/compact` | `compact_thread` | 压缩被禁用时同样返回 `409`；无测试覆盖该路由 | `threads:1399,1413-1414,1429-1430,1431-1432` |
+
+### 不匹配异步失败的路由
+
+| 路由 | 即时响应 | 失败载荷 | 证据 |
+| --- | --- | --- | --- |
+| `POST /api/threads/{thread_id}/runs` | `200` run 记录，无前置门禁 —— 但当线程的 run event 流为空时，`ensure_checkpoint_history_seeded` 会先在 `start_run` 内物化头部，此时返回 `500` | run 记录稍后变为 `status: error`，`error` 为不匹配原文 | `thread_runs:945-960`、`worker:1018-1030,1044-1047`、`worker:1450-1462`、`services:1503-1520,2055` |
+| `POST /api/threads/{thread_id}/runs/stream` | `200` `text/event-stream` | 一帧 `event: error`，其 `data` 为 `{"message": <不匹配原文>, "name": "CheckpointModeMismatchError"}` | `worker:1466-1472`、`services:2374-2378` |
+| `POST /api/threads/{thread_id}/runs/wait` | `200` JSON，从不返回 `409` | `{"status": "error", "error": "Thread requires delta mode; materialize and convert its checkpoints before using full mode."}` | `thread_runs:1018-1076` |
+| `POST /api/runs/stream`、`POST /api/runs/wait`（无状态） | 同上 | 同样的 `{"status", "error"}` 载荷 | `runs:59-91` |
+
+### 未做模式映射的路由
+
+| 路由 | 实际行为 | 证据 |
+| --- | --- | --- |
+| `POST /api/threads/{thread_id}/runs/regenerate/prepare` | `500`，`detail: "Failed to read latest checkpoint"`——访问器构建位于任何 `try` 之外 | `thread_runs:747,749-752` |
+| 同一路由的谱系/历史扫描 | `500`，`detail: "Failed to inspect checkpoint history"` | `thread_runs:679-681` |
+| `POST /api/threads/{thread_id}/runs/edit-regenerate/prepare` | 同样两条 `500` 路径 | `thread_runs:838-841`、`thread_runs:633-697` |
+| `POST /api/threads/{thread_id}/runs`、`/runs/stream`、`/runs/wait` 及无状态 `/api/runs/*`，当线程的 run event 流为空且存在检查点头部时 | `500`：`ensure_checkpoint_history_seeded` 在准入前通过受门禁的访问器读取该头部，而 `start_run` 只映射 `ConflictError`（`409`）与 `UnsupportedStrategyError`（`501`） | `services:1503-1520`、`services:2055`、`services:2126-2129` |
+| `GET /api/threads/{thread_id}/token-usage` | `200`，`context_usage: null`，仅记一条警告日志 | `context_usage.py:73-78`、`thread_runs:1872-1873` |
+| `GET` / `PUT` / `DELETE /api/threads/{thread_id}/goal` | 完全没有门禁：直接对 `channel_values["goal"]` 调用 `checkpointer.aget_tuple` / `aput` | `threads:1329-1383`、`goalsrc:442-453,468-477,480-542` |
+| `DELETE /api/threads/{thread_id}`、`POST /search`、`PATCH`、`POST /move` | 只操作元数据库，对模式无感 | `threads:706,1173,1216,1255` |
+
+### 没有任何上报面
+
+没有任何健康、就绪、特性、能力或控制台字段会上报模式或节奏。响应模型、响应头、事件字段也都不携带它：仅有的自定义 SSE 头是 `Cache-Control`、`Connection`、`X-Accel-Buffering`、`Content-Location`。模式只在服务端被消费，经由 `app.state.checkpoint_channel_mode`、`app.state.checkpoint_snapshot_frequency` 与 `RunContext`（`deps:451-452,767-768`、`services:1093,1283-1312`）。一处潜在例外：`GET /api/threads/{thread_id}` 与 `POST /api/threads/{thread_id}/history` 的响应 `metadata` 会剥离其他内部键，但不剥离 delta 标记与 `counters_since_delta_snapshot`，因此它们可能透传到客户端（`threads:1308-1315`、`threads:1890-1900`）。SSE 的 `metadata` 帧只带 `{"run_id", "thread_id"}`（`worker:1067-1074`）。见[可观测性](/docs/harness/checkpoints/observability)。
+
+## 对外符号
+
+消费者可以依赖的导入路径。这里列出的都是 Python 导入；路由契约见上文「异常与 HTTP 状态」一节。
+
+| 符号 | 导入路径 | 作用 | 证据 |
+| --- | --- | --- | --- |
+| `CheckpointChannelMode` | `deerflow.config.database_config` | 模式字面量 | `dbcfg:59` |
+| `DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY` | `deerflow.config.database_config` | 默认节奏 `10` | `dbcfg:61` |
+| `CheckpointDeltaConfig` | `deerflow.config.database_config` | delta 调优模型（`snapshot_frequency`） | `dbcfg:64` |
+| `CheckpointGraphCacheConfig` | `deerflow.config.database_config` | 图缓存上限模型（`accessor_graph_max`） | `dbcfg:86` |
+| `CheckpointCacheConfig` | `deerflow.config.database_config` | 历史缓存策略模型 | `dbcfg:103` |
+| `DatabaseConfig` | `deerflow.config.database_config` | 整个 `database:` 段 | `dbcfg:142` |
+| `resolve_checkpoint_graph_cache_max(database_config, field_name, default)` | `deerflow.config.database_config` | 容错的热重载上限读取器 | `dbcfg:45-57` |
+| `INTERNAL_CHECKPOINT_MODE_KEY` | `deerflow.runtime.checkpoint_mode` | `configurable` 键 | `cmode:18` |
+| `CHECKPOINT_MODE_METADATA_KEY` | `deerflow.runtime.checkpoint_mode` | 检查点 metadata 键 | `cmode:19` |
+| `CheckpointModeMismatchError` | `deerflow.runtime.checkpoint_mode` | 失败关闭的跨模式异常 | `cmode:22` |
+| `CheckpointModeReconfigurationError` | `deerflow.runtime.checkpoint_mode` | 需要重启却被改动的异常 | `cmode:26` |
+| `frozen_checkpoint_channel_mode()` | `deerflow.runtime.checkpoint_mode` | 读取已冻结的模式或 `None` | `cmode:34-36` |
+| `freeze_checkpoint_channel_mode(mode)` | `deerflow.runtime.checkpoint_mode` | 首次调用时冻结，之后拒绝变更 | `cmode:39-45` |
+| `frozen_checkpoint_snapshot_frequency()` | `deerflow.runtime.checkpoint_mode` | 读取已冻结的节奏或 `None` | `cmode:48-50` |
+| `freeze_checkpoint_snapshot_frequency(n)` | `deerflow.runtime.checkpoint_mode` | 冻结节奏；非正值抛 `ValueError` | `cmode:53-69` |
+| `resolve_checkpoint_snapshot_frequency(n=None)` | `deerflow.runtime.checkpoint_mode` | 显式参数 → 进程冻结值 → 默认值 | `cmode:72-78` |
+| `inject_checkpoint_mode(config, mode)` | `deerflow.runtime.checkpoint_mode` | 写入 `configurable` 键与 metadata 标记 | `cmode:81-88` |
+| `checkpoint_metadata_uses_delta(metadata)` | `deerflow.runtime.checkpoint_mode` | 标记/计数器识别 | `cmode:91-98` |
+| `checkpoint_tuple_uses_delta(tuple)` | `deerflow.runtime.checkpoint_mode` | 元组级识别 | `cmode:101-104` |
+| `state_snapshot_uses_delta(snapshot)` | `deerflow.runtime.checkpoint_mode` | 快照级识别 | `cmode:107-111` |
+| `raise_if_snapshot_incompatible(snapshot, mode)` | `deerflow.runtime.checkpoint_mode` | 读取门禁 | `cmode:114-123` |
+| `raise_if_checkpoint_tuple_incompatible(tuple, mode)` | `deerflow.runtime.checkpoint_mode` | 裸元组门禁 | `cmode:126-129` |
+| `ensure_checkpoint_mode_compatible(checkpointer, config, mode)` | `deerflow.runtime.checkpoint_mode` | 同步写前门禁 | `cmode:132-140` |
+| `aensure_checkpoint_mode_compatible(checkpointer, config, mode)` | `deerflow.runtime.checkpoint_mode` | 异步写前门禁 | `cmode:143-146` |
+| `ThreadState` | `deerflow.agents.thread_state` | full 模式状态 schema | `tstate:280` |
+| `DeltaThreadState` | `deerflow.agents.thread_state` | 默认节奏下的 delta schema | `tstate:396-397,423-426` |
+| `delta_messages_field(snapshot_frequency=DEFAULT_CHECKPOINT_SNAPSHOT_FREQUENCY)` | `deerflow.agents.thread_state` | `Annotated[list[AnyMessage], DeltaChannel(merge_message_writes, snapshot_frequency=…)]` | `tstate:385-390` |
+| `DELTA_MESSAGES_FIELD` | `deerflow.agents.thread_state` | 模块级默认节奏 delta 字段 | `tstate:393` |
+| `merge_message_writes(state, writes)` | `deerflow.agents.thread_state` | 线性时间、等价 `add_messages` 的 delta 归并器 | `tstate:328-382` |
+| `get_thread_state_schema(mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | 非 delta 返回 `ThreadState`，delta 返回缓存 schema | `tstate:416-419` |
+| `adapt_state_schema_for_mode(schema, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | 中间件/状态 schema 适配 | `tstate:437-440` |
+| `normalize_middleware_state_schemas(middleware, mode, snapshot_frequency=None)` | `deerflow.agents.thread_state` | 返回带适配 `state_schema` 副本的中间件列表 | `tstate:454-467` |
+| `THREAD_STATE_REDUCER_FIELDS` | `deerflow.agents.thread_state` | 归并器通道的 frozenset（含 `"messages"`），用于决定是否包 `Overwrite` | `tstate:400-413`、`threads:1065-1066` |
+| `create_deerflow_agent(..., checkpoint_channel_mode="full", checkpoint_snapshot_frequency=None, ...)` | `deerflow.agents.factory` | 对外 agent 工厂；delta 加 checkpointer 会被拒绝 | `factory:66-77`、`factory:135-141` |
+| `CheckpointStateAccessor`、`build_state_mutation_graph` | `deerflow.runtime`（包级再导出） | 访问器收口点与变更图 | `deerflow/runtime/__init__.py:8,21-22` |
+| `ThreadState`、`DeltaThreadState`、`SandboxState` | `deerflow.agents`（包级再导出） | 惰性导出的状态 schema | `agents/__init__.py:14-16,40-47` |
+| `CheckpointStateAccessor.bind(graph, checkpointer, *, store=None, mode="full")` | `deerflow.runtime.checkpoint_state` | 绑定 graph、saver 与模式 | `cstate:112-134` |
+| `build_state_mutation_graph(as_node, mode, state_schema=None, *, snapshot_frequency=None)` | `deerflow.runtime.checkpoint_state` | 面向 `Overwrite` 式写入的纯状态写入图 | `cstate:39-65` |
+
+<Callout type="warning">
+  delta 模式下裸读 saver 看到的是哨兵而不是消息。消费者必须通过
+  `CheckpointStateAccessor` 读取状态，而不是直接调用 checkpointer；
+  访问器的 docstring 明确写了这一点。
+</Callout>
+
+## 内部符号
+
+以下都是实现细节。不要导入它们，也不要依赖其名称或形状。
+
+| 符号 | 为何不是 API | 证据 |
+| --- | --- | --- |
+| `_frozen_checkpoint_channel_mode`、`_frozen_checkpoint_snapshot_frequency` | 模块级全局变量，只通过 freeze/frozen 辅助函数触碰；测试用 monkeypatch 重置 | `cmode:30-31`、`backend/tests/conftest.py:96-111` |
+| `_delta_thread_state_schema` | `@cache` 装饰的内部构建器；生成 `DeltaThreadState_f{freq}` | `tstate:422-434` |
+| `_adapt_state_schema_for_delta` | `@cache` 装饰的内部适配器；生成 `Delta{module_underscored}_{Name}_f{freq}` | `tstate:443-451` |
+| `_resolve_snapshot_frequency` | 内部惰性导入解析器 | `tstate:25-33` |
+| `_STATE_ACCESSOR_GRAPH_CACHE_MAX` | 代码内回退常量 `64`，仅在配置取值不可用时使用 | `services:1127,1133-1138` |
+| `_RawCheckpointReadAccessor`、`_RawCheckpointSnapshot` | full 模式的降级只读回退 | `services:1199-1263` |
+| `_checkpoint_mode_http_error`、`_CHECKPOINT_MODE_ERRORS` | 路由内部的 HTTP 映射 | `threads:82,96-107` |
+| `_wrap_sync_if_delta` | 同步 provider 内部实现 | `syncprov:164-192` |
+| `_migrate_legacy_snapshot_frequency` | pydantic `mode="before"` 校验器，标注为 `-- Legacy key migration (not user-configured) --` | `dbcfg:216-254` |
+| `_finish_state_mutation` | 空操作写入节点 | `cstate:32-33` |
+
+## 承重测试
+
+读者可以运行或引用的测试，即这份可执行契约。路径均位于 `backend/tests/` 下。
+
+### 模式门禁、冻结与配置
+
+- `test_checkpoint_mode.py`（冻结、识别、门禁）——`backend/packages/harness/deerflow/runtime/AGENTS.md:133` 中列为模式门禁的模块
+- `test_checkpoint_mode.py::test_yaml_mode_change_is_rejected_when_graph_is_reconstructed`
+- `test_checkpoint_mode.py::test_delta_mode_accepts_plain_full_checkpoint`
+- `test_checkpoint_mode.py::test_gateway_runtime_rejects_mode_different_from_frozen_process`
+- `test_threads_checkpoint_mode.py`（路由层模式行为）
+- `test_threads_checkpoint_mode.py::test_delta_snapshot_frequency_controls_snapshot_cadence`
+- `test_gateway_checkpoint_mode.py`（双模式端到端一致性）
+- `test_app_config_reload.py::test_checkpoint_channel_mode_rejects_unknown_value`
+- `test_config_version.py`（配置版本迁移）
+
+### 物化、存储形态与迁移
+
+- `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`
+- `test_delta_channel_checkpointers.py::test_materialization_is_deterministic_and_message_ids_are_stable`
+- `test_delta_channel_checkpointers.py::test_delta_storage_shape_and_snapshot_cadence`
+- `test_delta_channel_checkpointers.py::test_non_delta_writers_preserve_delta_messages_and_markers`
+- `test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination`
+- `test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order`
+- `test_delta_channel_state.py::test_mode_selects_expected_state_schema`
+- `test_delta_channel_state.py::test_compiled_delta_graph_bakes_configured_snapshot_frequency`
+- `test_delta_channel_state.py::test_production_message_forms_keep_assigned_ids_across_delta_replay`
+- `test_delta_channel_state.py::test_delta_normalization_compiles_stable_channel_without_mutating_middleware`
+
+`test_long_chain_history_survives_pagination` 与 `test_parallel_superstep_replay_matches_live_write_order` 断言的是正确契约，在固定依赖仍带缺陷时触发 skip；依赖升级后它们会成为实际门禁。
+
+### 历史缓存
+
+- `test_cached_history_saver.py::test_composition_matches_full_walk_and_avoids_it`
+- `test_cached_history_saver.py::test_latest_config_caches_under_resolved_checkpoint_id`
+- `test_cached_history_saver.py::test_recursive_resolve_caches_intermediate_levels`
+- `test_cached_history_saver.py::test_deep_cold_chain_delegates_one_inner_walk_at_depth_limit`
+- `test_cached_history_saver.py::test_cold_resolve_never_delegates_to_inner_history`
+- `test_cached_history_saver.py::test_sync_cold_resolve_never_delegates_to_inner_history`
+- `test_cached_history_saver.py::test_snapshot_parent_composes_without_parent_history_lookup`
+- `test_cached_history_saver.py::test_root_checkpoint_history_is_empty_writes`
+- `test_cached_history_saver.py::test_eviction_falls_back_to_walk_but_stays_correct`
+- `test_cached_history_saver.py::test_sync_path_matches_async`
+- `test_cached_history_saver.py::test_stats_expose_composition_counters`
+- `test_cached_history_saver.py::test_adelete_thread_purges_only_that_threads_cache_entries`
+- `test_cached_history_saver.py::test_sync_delete_thread_purges_cache_entries`
+- `test_cached_history_saver.py::test_prune_purges_rewritten_threads_cache_entries`
+- `test_cached_history_saver.py::test_delete_for_runs_delegates_without_cache_purge`
+- `test_cached_history_saver_integration.py::test_sequential_run_composes_without_inner_walks`
+- `test_cached_history_saver_integration.py::test_cache_disabled_parity`
+- `test_cached_history_saver_integration.py::test_branch_divergence_no_cross_contamination`
+- `test_cached_history_saver_integration.py::test_interrupt_resume_appended_head_writes`
+- `test_cached_history_saver_integration.py::test_eviction_only_costs_performance`
+- `test_cached_history_saver_integration.py::test_rollback_supersede_does_not_pollute`
+- `test_cached_history_saver_integration.py::test_full_to_delta_migration_under_cache_states`
+
+### 恢复线性化与回滚
+
+- `test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head`
+- `test_run_worker_delta_resume.py::test_linearization_restores_all_selected_state_and_clears_newer_channels`
+- `test_run_worker_delta_resume.py::test_linearized_resume_preserves_selected_checkpoint_agent_binding`
+- `test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer`
+- `test_run_worker_delta_resume.py::test_run_agent_streams_from_the_linearized_delta_resume`
+- `test_run_worker_delta_resume.py::test_run_agent_serializes_resume_preparation_with_checkpoint_writes`
+- `test_run_worker_delta_resume.py::test_cancelled_delta_resume_rolls_back_to_pre_linearization_head`
+- `test_run_worker_delta_resume.py::test_full_mode_keeps_the_fork`
+- `test_run_worker_delta_resume.py::test_ordinary_run_without_a_checkpoint_selector_is_untouched`
+- `test_run_worker_delta_resume.py::test_selecting_the_head_is_already_linear`
+- `test_run_worker_delta_resume.py::test_unmaterializable_resume_state_fails_closed`
+- `test_run_worker_rollback.py::test_rollback_linearizes_delta_restore_onto_cancelled_head`
+- `test_run_worker_rollback.py::test_rollback_linearizes_delta_restore_sqlite_reopen`
+- `test_run_worker_rollback.py::test_rollback_restores_pre_run_pending_writes_for_delta_checkpoints`
+- `test_run_worker_rollback.py::test_rollback_preserves_middleware_contributed_channels[delta]`
+- `test_run_worker_rollback.py::test_rollback_preserves_agent_binding_for_manual_compaction[delta]`
+- `test_run_worker_rollback.py::test_run_agent_marks_rollback_unusable_when_capture_fails`
+
+### 保留
+
+- `test_checkpoint_retention_contract.py::test_growth_baseline_full_vs_delta`
+- `test_checkpoint_retention_contract.py::test_deleting_branch_ancestor_breaks_lineage_loudly`
+- `test_checkpoint_retention_contract.py::test_deleting_explicit_resume_target_breaks_resume`
+- `test_checkpoint_retention_contract.py::test_pending_writes_are_retained_state_not_garbage`
+- `test_checkpoint_retention_contract.py::test_leaf_duration_checkpoint_deletion_is_safe`
+- `test_checkpoint_retention_contract.py::test_leaf_sibling_branch_deletion_is_safe`
+
+### 补丁
+
+- `test_checkpoint_patches.py::test_binop_overwrite_patch_is_active`
+- `test_checkpoint_patches.py::test_binop_overwrite_patch_stands_down_when_upstream_fixed`
+
+## 上游缺陷固定表
+
+状态对应固定依赖集（`langgraph 1.2.9`、`langgraph-checkpoint 4.2.0`、`langgraph-checkpoint-postgres 3.1.2`）。每条本地测试断言的都是正确契约，缺陷存在期间触发 skip，因此后续依赖升级会把它变成实际门禁；有缺陷的行为从不被当作正确行为断言。
+
+| 编号 | 种类 / 状态 | 触发场景 | 本地应对 | 固定版本 | 证据 |
+| --- | --- | --- | --- | --- | --- |
+| `#4380` | DeerFlow issue；已在本地打补丁 | 向空的 Union 类型 `BinaryOperatorAggregate` 通道做替换式写入会把 `Overwrite` 包装器持久化；下一次读取以 `TypeError: 'Overwrite' object is not subscriptable` 崩溃 | `checkpoint_patches.py` 中的探针守卫补丁；补丁测试；`test_threads_router.py` 中的分支/状态更新锚点 | `langgraph 1.2.9` | `checkpoint_patches:91-107`、`backend/tests/test_checkpoint_patches.py:31-95` |
+| `#8526`（修复上游 `#8384`），以及 postgres `#8535` | 上游 langgraph PR；2026-08-07 合并；`langgraph-checkpoint` 4.2.0 修复 | 线程从 `BinaryOperatorAggregate` 切换到 `DeltaChannel` 后，迁移后的第一次写入能从实时 `invoke` 返回值中看到，却从之后每次读取中消失；postgres 侧在没有 seed 时走到根并重放全部内容 | 删除本地 `InMemorySaver` 补丁；依赖下限 `langgraph-checkpoint>=4.2.0,<5.0`；回归门禁 `test_delta_channel_checkpointers.py::test_full_to_delta_migration_replays_on_same_thread`；禁止再引入版本守卫式 saver 补丁 | `langgraph-checkpoint 4.2.0` | `checkpoint_patches:11-17`、`backend/packages/harness/pyproject.toml:49-56`、`CHANGELOG.md:3040-3048` |
+| `#8382` | 上游 langgraph issue；**open**；修复 PR `#8544` 未合并，无发布版本 | 并行 super-step 写入的 delta 重放顺序与实时执行顺序不一致：实时 `invoke` 返回按路径排序的 `[a..h]`，`get_state` 重放为 `[f,b,g,h,a,c,d,e]`（纯置换）；在 memory、sqlite、postgres 上均可复现 | 触发 skip 并点名该 issue；`test_delta_channel_checkpointers.py::test_parallel_superstep_replay_matches_live_write_order` | `langgraph-checkpoint 4.2.0` | `test_delta_channel_checkpointers.py:419,440-460` |
+| `#8448` | 上游 langgraph issue；**open**；修复 PR `#8556` 未合并 | 目标检查点不在第一页分页结果中时，`PostgresSaver.get_delta_channel_history` 会永久污染游标，导致老检查点物化为空；只有 Postgres 会对 stage-1 扫描分页 | memory/sqlite 上断言保持生效（与 `InMemorySaver` 预言机做差分）；Postgres 触发 skip；`test_delta_channel_checkpointers.py::test_long_chain_history_survives_pagination` | `langgraph-checkpoint-postgres 3.1.2` | `test_delta_channel_checkpointers.py:459` |
+| `#8548` | 上游 langgraph PR；**open**；修复上游 `#8443`，相关 issue `#8551` 仍在等它 | 不把被放弃的分支重放进 `DeltaChannel` 分叉 | 未合并期间本地线性化不能移除：保留 `_linearize_delta_checkpoint_resume` | `langgraph 1.2.9` | `worker:2267-2279`、`test_run_worker_delta_resume.py::test_linearizes_a_delta_resume_onto_the_head` |
+| `#4458` | DeerFlow issue；分叉缺陷对用户可见的症状 | 在分叉线程中重新生成后，重载时旧助手消息会与新回答并排复活；在 postgres、sqlite 与内存 saver 上均可复现，且当种子行共用一个 run id 时还会破坏分支历史 | 线性化加按轮次播种的 run id；`test_run_worker_delta_resume.py::test_regenerating_in_a_branched_thread_does_not_resurrect_the_old_answer` | `langgraph 1.2.9` | `worker:2267-2289`、`test_run_worker_delta_resume.py:204` |
+| `#4189` 第 3 项 | DeerFlow 设计讨论 | 检查点行的保留与删除；哪些内容可以被删除 | `backend/docs/checkpoint-retention-contract.md`、`backend/tests/test_checkpoint_retention_contract.py`、`backend/app/gateway/checkpoint_retention.py`；目前没有接入生产触发点 | — | `checkpoint_retention.py:1-40`、`backend/docs/checkpoint-retention-contract.md:31-77` |
+| `#3859`（评审） | DeerFlow 评审参考 | 被中断的标题写入必须递增 `channel_versions["title"]` 并在 `new_versions` 中声明，否则数据库 saver 会丢弃该 blob | `backend/tests/test_run_worker_rollback.py:3501-3504` 固定了这一形状 | — | `backend/tests/test_run_worker_rollback.py:3501-3504` |
+
+补丁的内容与依赖下限的执行方式见[运维检查点](/docs/harness/checkpoints/operations)，每个缺陷对用户表现出的症状见[故障排查](/docs/harness/checkpoints/troubleshooting)。
+
+## 变更记录（2026 年 6 月至 9 月）
+
+变更记录条目涉及双模检查点存储的 PR。`## [未发布]` 下的英文条目累积发往里程碑 2.1.0；`## [2.0.0]` 下的条目来自 2026-06-15 的发布。「变更记录小节」列给出该条目所在的小节名。
+
+| PR | 变更记录小节 | 变更内容 |
+| --- | --- | --- |
+| [#4516](https://github.com/bytedance/deer-flow/pull/4516) | `### ⚠ 不兼容变更（Breaking Changes）` | 配置键从 `database.checkpoint_delta_snapshot_frequency` 迁移到 `database.checkpoint_delta.snapshot_frequency`，默认值从 `1000` 改为 `10`。旧顶层键仍被接受并给出弃用警告，显式设置的嵌套键胜出。delta 模式下快照频率提高 100 倍；要保留原节奏请显式设置 `…snapshot_frequency: 1000`。 |
+| [#4292](https://github.com/bytedance/deer-flow/pull/4292) | `### 新增` | 基于 LangGraph `DeltaChannel` 的双模检查点存储，把长研究/编码任务的线程存储从 O(N²) 降到近线性。 |
+| [#4638](https://github.com/bytedance/deer-flow/pull/4638) | `### 新增` | delta 模式的检查点历史缓存（memory/redis），用 O(1) 增量组合，通过 `database.checkpoint_cache` 配置。 |
+| [#4437](https://github.com/bytedance/deer-flow/pull/4437)、[#4460](https://github.com/bytedance/deer-flow/pull/4460)、[#4468](https://github.com/bytedance/deer-flow/pull/4468) | `### 修复` | 让检查点写入与活跃 run 串行化、线性化 delta 模式的检查点恢复，并接受 SDK 默认的 `stream_resumable=false` 以避免恢复竞态。 |
+| [#4383](https://github.com/bytedance/deer-flow/pull/4383) | `### 修复` | **checkpoint：** 把写入空通道的首个 `Overwrite` 解包。 |
+| [#5255](https://github.com/bytedance/deer-flow/pull/5255) | `### 修复` | 检查点保留契约测试套件：为 full 与 delta 两种 schema 建立按步的增长基线，删除分支祖先会以 `CheckpointLineageError` 明确失败，pending writes 被视为保留状态。 |
+| [#4140](https://github.com/bytedance/deer-flow/pull/4140) | `### 安全` | **storage：** 不再把 base64 图片数据持久化进检查点状态。属于相邻改动，不针对模式。 |
+| [#5734](https://github.com/bytedance/deer-flow/pull/5734) | `### 内部改进` | 将 `langgraph-checkpoint` 提升到 `>=4.2.0,<5.0`、`langgraph-checkpoint-postgres` 提升到 `>=3.1.2,<3.2`，并删除 `InMemorySaver` 的 delta 历史补丁：上游 4.2.0 修复了 full→delta 后首次写入丢失（`langgraph#8526`），postgres 也能定位普通值种子（`#8535`）。迁移契约测试仍是门禁。 |
+| [#4395](https://github.com/bytedance/deer-flow/pull/4395) | `### 内部改进` | **bench：** 新增隔离的检查点通道模式基准，在延迟、存储与重放指标上比较 `full` 与 `delta`。 |
+| [#5051](https://github.com/bytedance/deer-flow/pull/5051) | `### 内部改进` | **bench：** 在检查点基准中与 memory、SQLite 一并测量 Postgres 的检查点/块/写入存储增长。 |
+| [#2582](https://github.com/bytedance/deer-flow/pull/2582) | `## [2.0.0]` → `### 修复` → `#### 运行时、网关与持久化` | 回滚恢复检查点现在会取代更新的检查点。 |
+| [#3559](https://github.com/bytedance/deer-flow/pull/3559) | `## [2.0.0]` → `### 修复` | **subagent：** 让子 Agent 与父 run 的 checkpointer 隔离。 |
+
+本列表没有任何条目改动过模式默认值，它一直是 `full`。
+
+`#4292` 一行是对已发布 changelog 条目的逐字引用。其中「近线性」的说法描述的是单步表示；保留的存储总量仍然叠加到 O(N²/K + N)（K 为 `snapshot_frequency`），因为每一份周期性快照都保存着当时整份增长中的列表——见[检查点与增量通道](/docs/harness/checkpoints)。

frontend/src/content/zh/harness/checkpoints/resume-and-rollback.mdx

@@ -0,0 +1,139 @@
+---
+title: 恢复与回滚
+description: 增量模式无法正确地分叉线程：分叉后的增量读取会回放被放弃分支的写入。本章说明 DeerFlow 如何改为把检查点恢复线性化到当前 head、何时该重写是空操作、如何失败关闭，以及回滚在重写之前捕获什么。
+---
+
+import { Callout } from "nextra/components";
+
+# 恢复与回滚
+
+从较旧的检查点恢复等于一次分叉。在 `full` 模式下这是定义清晰的常规操作；在 `delta` 模式下它带来的是错误的读取结果，而不是报错——没有任何环节会拒绝这次分叉：LangGraph 会把第二个子节点存下来，而 delta 模式的前置写门在任何兼容性检查之前就直接返回——所以 DeerFlow 会在运行开始前把这次分叉改写成当前 head 上的一次线性写入。本章讲这次重写、它的失败形态、围绕它捕获的回滚点、它没有覆盖的写入路径，以及手动上下文压缩在同一模式下的写入方式。
+
+## 增量模式为什么无法正确分叉
+
+增量检查点不保存完整的通道值，它的状态靠回放祖先写入重建。这条回放路径会收集目标检查点沿途每个祖先上的**全部** `pending_writes` 条目——而共享父节点同时还留着被放弃的那条兄弟分支的写入。
+
+```
+            ┌─ C1   被放弃的兄弟分支；它的写入存在 P 下
+P（共享）  ─┤
+            └─ C2   当前 head
+```
+
+从 `P` 恢复的本意是「从 P 的状态重新开始」。但 `C1` 的写入仍然存在 `P` 下，于是回放会把它们灌进新的分叉：这次运行起手拿到的消息列表里，还带着它本该替换掉的那个回答。
+
+| 模式 | 检查点内容 | 分叉行为 |
+| --- | --- | --- |
+| `full` | 完整的 `channel_values`，包含整份 `messages` 列表 | 分叉可正确物化，LangGraph 的分支语义原样保留 |
+| `delta` | 一个 `_DeltaSnapshot` 哨兵加上逐步 writes 行 | 共享父节点会把被放弃兄弟分支的写入回放进分叉 |
+
+DeerFlow 在 Postgres、SQLite 和内存 saver 上都复现过该问题。用户可见的症状是 **DeerFlow #4458**：在已分支的线程里重新生成，刷新后会看到旧助手消息和新回答并列出现。
+
+上游跟踪同一个缺陷。PR **#8548**（"don't replay an abandoned branch into a `DeltaChannel` fork"）修复上游 issue **#8443**，目前仍未合并；它合并之前，下面的本地线性化不能移除。
+
+### 分叉会被接受，而不是被拒绝
+
+这套栈里没有任何环节拒绝增量分叉，所以「不能分叉」永远只描述读取侧，而不是写入侧。针对较旧检查点调用 `aupdate_state`（或从该检查点恢复一次运行）会为它建出第二个子节点，且不会报错：没有任何 saver 会检查是否已存在兄弟分支，而进程模式为 `delta` 时 `ensure_checkpoint_mode_compatible` 会直接返回。代价在物化时才付出。
+
+由此有两点：
+
+- **恢复是 DeerFlow 覆盖的那一半。** 运行路径会做线性化，因此 Gateway 与内嵌客户端在恢复或回滚时不会产生这种分叉。
+- **状态更新没有被覆盖。** `POST /api/threads/{thread_id}/state` 会把调用方传入的 `checkpoint_id` 直接送进访问器的写入，准备 `POST .../branches` 时用的变更访问器也接受同一个选择器。没有任何环节对它做线性化，因此写入会落在那个较旧的检查点（即共享父节点）下，进入该父节点每个其他孩子都会回走的写入集合。`messages` 这类 reducer 通道以 `Overwrite` 写入，会替换已累积的值，从而对新分支遮蔽该父节点的其他写入；一般情形由上游 issue **#8551**（"`update_state` against an older checkpoint writes into the branch it forked away from"）跟踪，目前仍未修复。在它落地之前，不要为增量线程选择较旧的 `checkpoint_id`；改为更新 head。
+
+## 线性化做了什么
+
+DeerFlow 没有重新实现「写入归属哪个子节点」——那属于 saver 的 `get_delta_channel_history` 契约——而是把恢复表达成它真正的含义：
+
+1. 通过 `CheckpointStateAccessor` 物化所选检查点的状态。
+2. 以**替换语义**把该状态写到**当前 head** 上。head 没有其他子节点，因此不会有兄弟写入被回放进它。
+3. 让运行从重写后的 head 开始线性续跑。
+
+被放弃的那一轮仍留在检查点历史里，成为重写后 head 的祖先，不会被删除。
+
+### 不变量
+
+| 不变量 | 行为 |
+| --- | --- |
+| 所有已物化的通道都被还原 | 替换值由所选检查点的物化值构造 |
+| 仅存在于 head 的通道被重置而非继承 | 只存在于较新 head 上的通道重置为其 schema 默认值（例如 `[]` 或 `{}`）；没有可构造默认值的通道重置为 `None`。LangGraph 没有公开的「取消通道」更新，这是最接近的等价做法 |
+| 归并器通道是替换而不是再归并 | 整份状态写入会把每个归并器通道——增量 `messages` 通道与经典 `BinaryOperatorAggregate` 通道都一样——包进 `Overwrite`。在任何模式下，归并器通道的替换式写入都需要这层包装；否则替换值会被再归并进已经聚合过的旧值 |
+| 中间件通道被覆盖 | 写入走线程的*有效* schema，因此自定义中间件贡献的通道也会被包含。基础 `ThreadState` 兜底 schema 不认识这些通道，而对未知通道的写入会被静默丢弃 |
+| Agent 绑定保留 | 所选检查点上由服务端写入的 Agent 绑定元数据会复制到重写后的 head 上 |
+| 选择器被消费 | 重写后 `configurable` 里的 `checkpoint_id` 与 `checkpoint_map` 会被弹出，运行初始的 `RunnableConfig` 也会重建，使流式输出从重写后的 head 开始 |
+
+## 何时是空操作
+
+以下任一条件成立时，该辅助函数返回 `None`，普通运行完全不受影响：
+
+| 条件 | 原因 |
+| --- | --- |
+| `full` 模式 | 完整检查点自带全部通道值，分叉可正确物化，LangGraph 的分支语义保持原样 |
+| 没有 checkpointer | 没有需要重写的持久化内容 |
+| `configurable` 不是 dict | 没有可解析的选择器 |
+| `checkpoint_id` 缺失或不是字符串 | 没有选中任何恢复目标 |
+| `checkpoint_ns` 非空 | 子图命名空间有自己的谱系，而网关只选择根检查点 |
+| 选择器指向的就是 head | 选择 head 本身就是线性的——此时还不可能存在兄弟分支 |
+
+## 失败关闭
+
+线性化不会退化成「那就照样分叉」。失败会向上抛出，因为悄悄回退到分叉会持久化它本该阻止的损坏历史。
+
+| 失败 | 错误原文 |
+| --- | --- |
+| 恢复状态无法物化 | `Run {run_id} could not materialize resume checkpoint {checkpoint_id}` |
+| 无法为替换值检视状态 schema | `Run {run_id} could not inspect the state schema for {operation}` |
+
+两者都是在任何写入之前抛出的 `RuntimeError`，所以运行会直接失败，而不是持久化一份不完整的替换。
+
+<Callout type="warning">
+  网关对增量模式也没有降级读取路径。Agent 工厂不可用时，`full` 模式会退化为
+  原始 checkpointer 读取，但 `delta` 模式会原样抛出：物化需要图的通道表，
+  降级读取只能拿到哨兵。
+</Callout>
+
+## 回滚
+
+被取消的运行需要还原运行前状态。增量回滚和恢复有同样的写入归属问题：捕获的父节点现在带着被取消兄弟分支的写入，靠分叉该父节点来还原会把它们回放进来。因此增量回滚改为在**当前 head 上线性还原**；`full` 模式继续使用分叉，所有通道都从父节点继承。
+
+### 捕获了什么
+
+| 捕获项 | 方式 | 说明 |
+| --- | --- | --- |
+| 已物化快照 | 通过 `CheckpointStateAccessor` | 原始检查点 blob 无法重建增量消息——增量检查点不存物化值——所以物化快照是唯一的来源 |
+| 原始 `pending_writes` | 通过 `aget_tuple` | 原始元组对元组级元数据和在途写入仍然有效 |
+| `state_values` | 每个非 `messages` 通道的深拷贝 | **仅在增量模式下**捕获；full 模式下这些通道由分叉从父节点继承 |
+| 元数据 | 取自物化快照 | 与值一起还原 |
+
+捕获失败时，本次运行被标记为 `snapshot_capture_failed` 并禁用回滚——还原空或部分历史会静默截断线程。线程完全没有检查点时，回滚改走 `adelete_thread` 重置路径。
+
+### 顺序与线程锁
+
+运行前的回滚点在恢复重写**之前**捕获。正是这个顺序使「取消并回滚」能还原真正的运行前 head，而不是线性化刚写下的状态。
+
+两步都在 `_checkpoint_thread_lock(thread_id)` 内执行，该锁串行化单个线程的检查点变更，同时不阻塞 goal 命令。锁不可重入，辅助函数内部不得再次获取。上一个成功的运行可能在释放准入槽位后仍在持久化耗时元数据，因此共用这把锁能把捕获与重写合成对 head 的一次不中断读改写序列。
+
+## 手动压缩
+
+上下文压缩用一个摘要替换线程的消息列表。增量模式下 `messages` 通道是追加式的，因此只有包一层替换才会生效：
+
+```python
+updated_config = await accessor.aupdate(
+    update_config,
+    {
+        "messages": Overwrite(list(result.preserved_messages)),
+        "summary_text": result.summary_text,
+        "task_history": result.task_history,
+    },
+    as_node="manual_compaction",
+)
+```
+
+写入通过 accessor 完成——它需要一个已物化的 `checkpoint_id`——而 `Overwrite` 正是让替换能穿过增量通道追加式归并器的关键；不加包装的列表会被追加到历史之后，而不是替换它。同一条规则也适用于 full 模式下的归并器通道。
+
+压缩失败保持原有对外表现：摘要生成失败抛出 `ContextCompactionFailed("summary generation failed")`，对应 HTTP 500 `Failed to compact thread context.`；压缩开关关闭则是 409 `Context compaction is disabled.`
+
+## 相关章节
+
+- [通道模式](/docs/harness/checkpoints/modes) —— 跨模式恢复首先会撞上的失败关闭模式门。
+- [快照节奏](/docs/harness/checkpoints/cadence) —— `_DeltaSnapshot` blob 从何而来，以及回放深度为何有界。
+- [历史缓存](/docs/harness/checkpoints/history-cache) —— 使缓存的增量历史安全可用的祖先链不变性论证。
+- [故障排查](/docs/harness/checkpoints/troubleshooting) —— 「旧回答复活」症状以及其他增量模式失败特征。

frontend/src/content/zh/harness/checkpoints/troubleshooting.mdx

@@ -0,0 +1,219 @@
+---
+title: 故障排查
+description: 按现象索引的检查点常见问题。模式不匹配与需要重启的错误、看似被忽略的配置、没有缩小的存储、重新生成与历史回放缺陷、Postgres 与 redis 故障、集成被拒、始终不命中的缓存计数器，每条给出原因、处理办法与上游 issue。
+---
+
+import { Callout } from "nextra/components";
+
+# 故障排查
+
+本章按**你看到的现象**组织。每条给出原因、处理办法，以及（当某种行为属于上游缺陷时）对应的上游 issue 与依赖版本，方便对照你的构建。
+
+## 症状索引
+
+| 症状 | 最可能的原因 | 章节 |
+| --- | --- | --- |
+| 状态/历史/线程读接口返回 `409` | full 模式进程读取了 delta 线程 | [delta 线程上的 HTTP 409](#delta-线程上的-http-409) |
+| run 请求在准入前返回 `500` | 线程的 run event 流为空，播种读取落在模式门下 | [delta 线程上的 HTTP 409](#delta-线程上的-http-409) |
+| run 失败并带重启要求信息 | 配置改动在被冻结的进程构建图时被拒绝 | [改完配置后 run 失败](#改完配置后-run-失败) |
+| 配置改动似乎被忽略 | 模式与节奏在构建 agent 时已冻结 | [配置改动似乎没有生效](#配置改动似乎没有生效) |
+| 历史缓存改动似乎被忽略 | 缓存在启动时随 checkpointer 一起构造 | [历史缓存的改动没有生效](#历史缓存的改动没有生效) |
+| 重新生成接口返回 `500` | run-runs 路由没有映射模式错误 | [重新生成接口返回 500 而不是 409](#重新生成接口返回-500-而不是-409) |
+| delta 没有缩小存储 | 仍在 full 模式，或没统计 writes 表 | [delta 没有缩小存储](#delta-没有缩小存储) |
+| 物化变慢 | 快照节奏调得过高 | [提高节奏后物化变慢](#提高节奏后物化变慢) |
+| 重新生成后被替换的回答复活 | delta 分叉回放（#4458） | [重新生成后旧回答复活](#重新生成后旧回答复活) |
+| 历史顺序与实时不一致 | 上游回放顺序缺陷（#8382） | [历史顺序与实时运行不一致](#历史顺序与实时运行不一致) |
+| Postgres 物化旧检查点为空 | 上游分页回走缺陷（#8448） | [Postgres 物化旧检查点为空](#postgres-物化旧检查点为空) |
+| TUI 上 `redis` 缓存被拒 | 同步路径只支持 `memory` | [TUI 或内嵌路径拒绝 redis 缓存](#tui-或内嵌路径拒绝-redis-缓存) |
+| 缓存计数器始终不命中 | 缓存被禁用、模式不对，或每次读的都是新检查点 | [缓存计数器始终不命中](#缓存计数器始终不命中) |
+| `create_deerflow_agent` 拒绝 delta | delta 加检查点器会被拒绝 | [`create_deerflow_agent` 拒绝 delta 加检查点器](#create_deerflow_agent-拒绝-delta-加检查点器) |
+
+## 模式门与配置
+
+### delta 线程上的 HTTP 409
+
+**症状**：状态、历史或线程读接口返回 `409`，内容为
+
+```
+{"detail": "Thread <thread_id>: Thread requires delta mode; materialize and convert its checkpoints before using full mode."}
+```
+
+**原因**：一个运行在 `full` 的进程读取了以 `delta` 写入检查点的线程。读门在物化之前抛出 `CheckpointModeMismatchError`，因此 full 模式进程绝不会静默返回空状态。返回 `409` 的路由是 `GET /api/threads/{thread_id}`、`GET` 与 `POST /api/threads/{thread_id}/state`、`POST /api/threads/{thread_id}/history`（均有测试覆盖）；`POST /api/threads/{thread_id}/branches` 与 `POST .../compact` 在代码路径上以同样方式门禁。`compact` 也会因另一原因返回 `409`，detail 为 `Context compaction is disabled.`
+
+threads 路由不会对创建 run 提前门禁，因此那里的不匹配改为异步失败：`POST .../runs` 返回 `200` run 记录，随后变为 `status: error`；`POST .../runs/stream` 返回 `200 text/event-stream` 后发出 `event: error` 帧；`POST .../runs/wait` 返回 `200` JSON，内容为 `{"status": "error", "error": "<不匹配文本>"}`。不过有一条 run 路径会在准入前读取检查点：当线程的 run event 流为空但存在检查点头部时，`ensure_checkpoint_history_seeded` 会在 `start_run` 内物化该头部，而那里只映射 `ConflictError` 与 `UnsupportedStrategyError`，因此不匹配会以未处理的 `500` 逃出，作用于 `POST /api/threads/{thread_id}/runs`、`/runs/stream` 与 `/runs/wait`。默认的 `run_events.backend: memory` 会让 Gateway 重启后每个既有线程的流都变空，所以这条分支在重启后可达。
+
+在内嵌 TUI 里，client 在图开始流式输出前就在进程内抛出：Textual runner 会把它变成 `AssistantError` 与 `RunEnded` 的组合，而无头的 `--print`、`--json` runner 没有对应处理器，会打印未捕获的 traceback，而不是状态码。
+
+**处理**：二选一。
+
+1. 让进程运行在 `delta`：设置 `database.checkpoint_channel_mode: delta`，并重启共享该数据库的每个进程。delta 进程可以透明读取遗留的 full 检查点，所以 full → delta 是受支持的方向。
+2. 让进程保持 `full`，这需要先物化并转换该线程的检查点。错误文本点明了这一步，但仓库不提供任何转换命令，而 full 模式直接读 delta 检查点只会看到哨兵 —— 不要靠删行来绕过。
+
+标记语义见[通道模式](/docs/harness/checkpoints/modes)，切换步骤见[快速上手](/docs/harness/checkpoints/quick-start)。
+
+### 改完配置后 run 失败
+
+**症状**：在未重启的情况下改完 `config.yaml` 后，下一次 run 失败，其 run 记录、SSE `error` 帧或 worker 日志行 `Run %s failed: %s` 中出现下二者之一
+
+```
+checkpoint_channel_mode is restart-required and cannot change in a running process
+checkpoint_delta.snapshot_frequency is restart-required and cannot change in a running process
+```
+
+**原因**：`CheckpointModeReconfigurationError`。进程在启动时冻结了模式与节奏，而某处用不同的值构建了图 —— 即一次与启动冻结值分歧的配置热重载。只有 Gateway 启动、`make_lead_agent` 与 `DeerFlowClient.__init__` 会调用冻结辅助函数，而没有任何状态、历史或线程路由会构建 agent（这些路由读取启动时冻结的 `app.state` 值），所以这是 run 侧的失败：它们都不会因配置改动返回 `503`。路由层的 `503` 映射为该错误类而存在，但 threads 路由中没有任何路径会抛出它。
+
+两个开关的行为不同：
+
+- `database.checkpoint_channel_mode`：完全不会失败。run worker 会把冻结的模式注入 run 配置，而 agent 工厂优先采用该注入值而非请求期配置，因此改动在重启前被忽略 —— 即[配置改动似乎没有生效](#配置改动似乎没有生效)。
+- `database.checkpoint_delta.snapshot_frequency`：下一次 run 会失败，因为节奏没有注入覆盖，agent 工厂会按重载后的配置冻结它。
+
+**处理**：重启进程以冻结新值，然后重跑。快照节奏也参与其中，因为它被编译进每张图的通道表，同样需要重启。
+
+### 配置改动似乎没有生效
+
+**症状**：你改了 `database.checkpoint_channel_mode` 或 `database.checkpoint_delta.snapshot_frequency`，但线程行为照旧。
+
+**原因**：模式在进程首次构建 agent 时按启动快照全局冻结；节奏随之一起冻结。首次冻结时，客户端传入的 `configurable.__deerflow_checkpoint_channel_mode` 会被忽略；一旦冻结，任何不同的值都会 fail-closed。内嵌 `DeerFlowClient` 也以同样方式冻结模式与节奏。另注意 `checkpoint_delta.*` 在 full 模式下被忽略。
+
+**处理**：重启共享该检查点数据库的每个进程；它们必须使用相同的值。通过读取线程标记确认切换（见[可观测性](/docs/harness/checkpoints/observability)）。只有一个相关键会在运行期重新读取：`database.checkpoint_graph_cache.accessor_graph_max`（每次淘汰检查时）。`database.checkpoint_cache.*` **不冻结**——它不会损坏线程，也允许多 worker 取值不同——但它同样不是热重载，见[下一条](#历史缓存的改动没有生效)。
+
+### 历史缓存的改动没有生效
+
+**症状**：你改了 `database.checkpoint_cache.max_entries`、把 `type` 换成 `redis`，或改了 `redis_url`、`ttl_seconds`、`key_prefix`，但运行中的 Gateway 行为照旧——容量没变、没有 redis 流量、仍是原来的键前缀。
+
+**原因**：历史缓存不是热重载。`langgraph_runtime()` 用启动时的 `AppConfig` 快照只构造一次 checkpointer，delta saver 就被包在当时 `make_checkpoint_cache()` 返回的对象上（`backend/app/gateway/deps.py`、`backend/packages/harness/deerflow/runtime/checkpointer/async_provider.py`），因此后端、容量、连接、TTL 与前缀都被捕获到进程生命周期结束。整个 `database` 段被登记为 startup-only 正是这个原因（`backend/packages/harness/deerflow/config/reload_boundary.py`）。
+
+这不属于「为正确性而冻结」：缓存不会损坏线程，禁用后结果完全相同，共享同一检查点数据库的多个 worker 也可以安全地使用不同设置。区别只在于它只在启动时被读取一次。
+
+**处理**：改完该段后重启 Gateway；内嵌或 TUI 路径则重启那个进程，因为它的 checkpointer 单例同样只创建一次并一直复用。只有 `database.checkpoint_graph_cache.accessor_graph_max` 无需重启，因为它在每次淘汰检查时都会重读。
+
+### 重新生成接口返回 500 而不是 409
+
+**症状**：重新生成准备返回 `500`，detail 为 `"Failed to read latest checkpoint"` 或 `"Failed to inspect checkpoint history"`，而同一线程在别处却返回 `409`。
+
+**原因**：`POST /api/threads/{thread_id}/runs/regenerate/prepare`（以及 `/runs/edit-regenerate/prepare`）在任何 `try` 之外构建状态访问器，而该模块不映射检查点模式错误，于是不匹配表现为未处理的 `500`。日志行为
+
+```
+Failed to read latest checkpoint for regenerate thread %s
+Failed to list checkpoints for regenerate thread %s
+```
+
+另有两个接口因同样原因更安静地失败：`GET /api/threads/{thread_id}/token-usage` 返回 `200` 且 `context_usage: null`（只记录 `Failed to load checkpoint for context usage on thread %s`），`/runs/wait` 返回 `200` 错误体而不是 `409`。
+
+**处理**：把该 `500` 读作模式不匹配。检查线程标记，然后在与线程匹配的模式下重启进程。本地没有把这些变成 `409` 的修法。
+
+## 存储与物化
+
+### delta 没有缩小存储
+
+**症状**：你切到了 delta（或自认为切了），但数据库体积不变甚至更大。
+
+**原因**：通常是下列之一。
+
+- 进程仍在 `full`（默认值）—— 键从未设置，或设置了但未重启。`checkpoint_delta.*` 在 full 模式下不起作用，历史缓存也只在 delta 模式下包裹 saver。
+- 你比较错了表。delta 把每步载荷移入 writes 表，因此公平的比较要同时统计检查点载荷字节**与** writes 行。存储守卫测试断言 delta 至多保留一个快照 blob，而 full 至少保留四个。
+
+一个相关提醒：任何保留策略在删除中间检查点时，都不得删掉保留下来的检查点与最近 `_DeltaSnapshot` 之间的祖先检查点或 writes，否则 delta 通道会静默重建为空且不报错。DeerFlow 不提供 `prune`/`aprune` 的生产触发点，因此这只是对接线方式的约束，不是现网故障。
+
+**处理**：从线程标记确认模式，必要时重启，并比较检查点载荷加 writes 行。见[历史缓存](/docs/harness/checkpoints/history-cache)与[运维检查点](/docs/harness/checkpoints/operations)。
+
+### 提高节奏后物化变慢
+
+**症状**：调高 `snapshot_frequency` 后，读取与状态物化变慢。
+
+**原因**：`snapshot_frequency`（默认 `10`）是两次完整快照之间的逐步写入次数。越大表示快照更少更小，但物化任意检查点需要回走的链更长。物化结果两者相同，只有体积与读取成本在权衡。
+
+**处理**：调低 `snapshot_frequency`，用检查点体积换回读取速度。它与模式一样需要重启。对同一检查点的重复物化正是历史缓存要吸收的，所以也检查它的计数器（见下）。
+
+## 历史与回放
+
+### 重新生成后旧回答复活
+
+**症状**：在分叉线程里重新生成后，刷新页面时被替换掉的助手消息和新回答并列存在，且当种子行共用一个 run id 时分支历史可能损坏（[#4458](https://github.com/bytedance/deer-flow/issues/4458)）。
+
+**原因**：delta 无法正确分叉 —— 分叉本身会被接受，坏的是读取。从较旧的检查点恢复会分叉谱系，而 delta 历史回走会收集每条在路径上的祖先的**全部** `pending_writes` —— 但共享父级还带着被放弃的兄弟孩子的写入，于是这些写入被回放进分叉，run 从一份仍包含「本应被替换掉的回答」的消息列表开始。已在 Postgres、SQLite 与内存 saver 上复现；full 模式不受影响，因为它的检查点携带完整通道值。
+
+**处理**：DeerFlow 改为线性化而不分叉。它物化所选检查点的状态，以替换语义写到当前 head（head 没有其他孩子），把只存在于较新 head 的通道重置为 schema 默认值或 `None`，消费掉选择器，然后线性续跑。当模式为 full、没有 checkpointer、选择器已经指向 head，或命名空间非根时，线性化是空操作；它**fail-closed**，绝不回退到分叉：
+
+```
+Run {run_id} could not materialize resume checkpoint {checkpoint_id}
+```
+
+上游修复是 PR [#8548](https://github.com/langchain-ai/langgraph/pull/8548)（修复上游 [#8443](https://github.com/langchain-ai/langgraph/issues/8443)；镜像 #8551），仍未合并，这正是本地线性化被保留的原因。见[恢复与回滚](/docs/harness/checkpoints/resume-and-rollback)。
+
+### 历史顺序与实时运行不一致
+
+**症状**：在并行扇出向同一 delta 通道写入的同一个 super-step 中，实时 run 按路径顺序返回写入，但 `get_state` / 恢复以不同顺序回放。
+
+**原因**：已固定的上游缺陷（[langgraph #8382](https://github.com/langchain-ai/langgraph/issues/8382)），open，修复 PR [#8544](https://github.com/langchain-ai/langgraph/pull/8544) 未合并，无已发布版本。delta 回放按哈希后的 `(task_id, idx)` 排序写入，而不是实时 `apply_writes` 使用的路径顺序，于是同一 super-step 的写入被重排。存在于 `langgraph-checkpoint 4.2.0`；已在 memory、SQLite 与 Postgres 上复现。
+
+**处理**：本地无解。DeerFlow 的测试断言正确契约，并在缺陷存在时触发 skip，同时点名该 issue：
+
+```
+upstream langgraph#8382 unfixed in langgraph-checkpoint 4.2.0: delta replay orders writes by hashed (task_id, idx) instead of the path order apply_writes used live, so get_state/resume reorders same-superstep writes
+```
+
+如果你的消费方在意同一 super-step 内的顺序，请避免把并行写入扇出到同一个 delta 通道。
+
+### Postgres 物化旧检查点为空
+
+**症状**：在 Postgres 上，较旧的 delta 检查点物化为空，较新的却正常。
+
+**原因**：已固定的上游缺陷（[langgraph #8448](https://github.com/langchain-ai/langgraph/issues/8448)），open，修复 PR #8556 未合并。当目标检查点不在第一页分页结果里时，`PostgresSaver.get_delta_channel_history` 会永久污染其回走游标：未修补的游标取自一个尚未加载的父级，永久停在 `None`，于是旧检查点物化为空。只有 Postgres 对其 stage-1 扫描分页。存在于 `langgraph-checkpoint-postgres 3.1.2`。
+
+**处理**：本地无解；Postgres 用例触发 skip，并点名该 issue：
+
+```
+upstream langgraph#8448 unfixed in langgraph-checkpoint-postgres 3.1.2: the paged delta walk poisons the channel cursor for a target past the first stage-1 page, hydrating old checkpoints empty
+```
+
+节奏约束回放深度，因此更小的 `snapshot_frequency` 能让回走更短；这是本仓库暴露的唯一杠杆，而不是对该游标缺陷的修复。
+
+## 缓存
+
+### TUI 或内嵌路径拒绝 redis 缓存
+
+**症状**：TUI 或无头的 `--print` / `--json` 运行在构造检查点器时失败：
+
+```
+database.checkpoint_cache.type 'redis' is not supported on the sync checkpointer path (TUI/embedded); use 'memory'.
+```
+
+**原因**：TUI 与无头模式构建的是 `DeerFlowClient(checkpointer=get_checkpointer())` —— **同步**单例检查点器 —— 而同步路径只支持 `memory` 后端（它本来就是进程本地的）。因此对 Gateway 有效的配置可能让 TUI 起不来。
+
+**处理**：设置 `database.checkpoint_cache.type: memory`。仅跑 Gateway 的部署可以保留 `redis`，它是异步路径的共享缓存。
+
+### 缓存计数器始终不命中
+
+**症状**：`hits` 一直是 `0` 而 `misses` 持续增长，或者 `compose_hits` 与 `full_walks` 看起来不对劲。
+
+**原因**：有若干合理原因。
+
+- 缓存只在 delta 模式、异步/Gateway 路径上安装。full 模式下根本没有 `CachedHistorySaver`。
+- `max_entries: 0` 统一禁用缓存，被禁用的缓存直接透传，不做组合。
+- wrapper 从不缓存「最新检查点」的解析结果：它先把目标解析成不可变的检查点 id，因此只读最新检查点的客户端按设计就是 miss。
+- 组合深度有上限（`_COMPOSE_MAX_DEPTH = 8`），冷链会回退到一次内层回走；在加入递归组合之前，实测单层组合在 500 步 SQLite run 上得到 `0` 次缓存命中。
+- `max_entries: 1` 的 LRU 每次读都会抖动 —— 结果正确，但没有命中。
+- redis 后端只报告 `hits` 与 `misses`；它的 `evictions` 与 `entries` 恒为 `0`。
+
+**处理**：确认处于 delta 模式且 `max_entries > 0`——并且在根据设置改动下结论前先重启，因为缓存只在启动构造时被读取一次。然后读 `compose_hits` 与 `full_walks`，判断是否由组合（而非顶层缓存）在承担工作。解释 `hits`/`misses` 时要与这两个计数器一起看，而不是单独看。见[缓存计数器](/docs/harness/checkpoints/observability)。
+
+## 集成
+
+### `create_deerflow_agent` 拒绝 delta 加检查点器
+
+**症状**：调用 `create_deerflow_agent(..., checkpoint_channel_mode="delta", checkpointer=...)` 抛出 `ValueError`：
+
+```
+create_deerflow_agent does not support checkpoint_channel_mode='delta' with a checkpointer: persisted graphs built here bypass checkpoint mode marker injection and the fail-closed compatibility gate (see deerflow.runtime.checkpoint_mode), so a mixed-mode store would silently corrupt thread state.  Use the guarded application paths (make_lead_agent or DeerFlowClient) for delta persistence; delta without a checkpointer is ephemeral and allowed.
+```
+
+**原因**：这样直接构建的图绕过了标记注入与 fail-closed 兼容门，混合模式的存储可能静默破坏线程状态。**不带**检查点器的 delta 是临时的，允许使用。
+
+**处理**：需要 delta 持久化时走受保护路径 —— `make_lead_agent` 或 `DeerFlowClient`。若只需一个临时的 delta 图，去掉 checkpointer 即可。
+
+<Callout type="tip">
+  排查任何症状前先拿到两个事实：进程的冻结模式（来自 `config.yaml`
+  以及它是否重启过）与线程的标记（来自其 `metadata`）。两者一致后，
+  上面几乎所有条目都会消解。
+</Callout>

frontend/src/content/zh/harness/configuration.mdx

@@ -119,6 +119,7 @@ make config-upgrade
 | `memory`          | 跨会话记忆存储                               | [记忆系统](/docs/harness/memory)                     |
 | `summarization`   | 对话摘要压缩                                 | [中间件](/docs/harness/middlewares)                  |
 | `title`           | 自动生成线程标题                             | [中间件](/docs/harness/middlewares)                  |
+| `database`        | 检查点通道模式、快照节奏与历史缓存           | [检查点与增量通道](/docs/harness/checkpoints)        |
 | `checkpointer`    | 线程状态持久化                               | [Agent 与线程](/docs/application/agents-and-threads) |
 | `guardrails`      | 工具调用授权                                 | —                                                    |
 | `uploads`         | 文件上传设置（PDF 转换器）                   | —                                                    |

frontend/tests/unit/content/docs-frontmatter.test.ts

@@ -0,0 +1,52 @@
+import { readFileSync, readdirSync } from "node:fs";
+import { join } from "node:path";
+
+import { describe, expect, it } from "@rstest/core";
+
+const CONTENT_ROOT = join(process.cwd(), "src/content");
+// An unquoted frontmatter scalar containing ": " parses as a nested mapping and
+// makes Nextra's YAML frontmatter fail, which 500s every page in the section.
+// Only `next build` (or a dev-server render) compiles MDX, so the unit suite has
+// to keep this guard on its own.
+const UNQUOTED_MAPPING_VALUE = /^([A-Za-z_][\w-]*):[ \t]*(?![|>])/;
+const SAFE_VALUE = /^(".*"|'.*'|\[.*\]|\{.*\}|[|>]|$)/;
+
+describe("docs frontmatter", () => {
+  const files = readdirSync(CONTENT_ROOT, { recursive: true })
+    .map((entry) => join(CONTENT_ROOT, String(entry)))
+    .filter((path) => path.endsWith(".mdx"));
+
+  it("finds the content tree it is meant to guard", () => {
+    expect(files.length).toBeGreaterThan(0);
+  });
+
+  it("keeps every frontmatter block YAML-parseable", () => {
+    const offenders = files.flatMap((path) => {
+      const source = readFileSync(path, "utf8");
+      if (!source.startsWith("---\n")) {
+        return [];
+      }
+      const displayPath = path
+        .slice(CONTENT_ROOT.length + 1)
+        .split("\\")
+        .join("/");
+      return source
+        .slice(4, source.indexOf("\n---", 4))
+        .split("\n")
+        .flatMap((line, index) => {
+          const key = UNQUOTED_MAPPING_VALUE.exec(line);
+          if (key === null) {
+            return [];
+          }
+          const value = line.slice(key[0].length);
+          return !SAFE_VALUE.test(value) && value.includes(": ")
+            ? [
+                `${displayPath}:${index + 2}: unquoted "${key[1]}" value contains ": "`,
+              ]
+            : [];
+        });
+    });
+
+    expect(offenders).toEqual([]);
+  });
+});