deer-flow/backend/docs/checkpoint-retention-contract.md

Checkpoint Retention Contract (DRAFT)

Status: draft — the deletion contract for #4189 item 3. No retention or deletion implementation should land before this contract (or a successor revision of it) is accepted, and every deletion proposal must be validated against backend/tests/test_checkpoint_retention_contract.py.

Why a contract is needed

LangGraph checkpoints form a per-thread parent chain. Gateway features depend on that chain being intact:

• Branch / regenerate resolves the replay base by walking parent_config links from a checkpoint that contains the target message (app/gateway/checkpoint_lineage.py::find_checkpoint_before_message).
• Explicit resume replays from a checkpoint_id a client still holds.

Deleting checkpoint rows by recency or table size can therefore break those features silently — a missing ancestor surfaces as CheckpointLineageIntegrityError at branch time, or as a lost resume target, never as an obvious storage bug. The contract below separates deletable rows from protected rows and pins the verification method.

Data model

Backend	State rows	Writes rows
SQLite	checkpoints	writes
Postgres	checkpoints, checkpoint_blobs	checkpoint_writes
Memory	saver.storage, saver.blobs	saver.writes

(Note: SQLite has no separate blob table; channel values live inside the serialized checkpoint payload. Postgres splits blobs out.)

Measurement shape: per-thread rows + bytes per table, normalized by bench_channels._normalized_storage_stats.

Protected set (MUST NOT delete without the stated compensation)

1. Explicit resume targets — any checkpoint_id a client may still resume to. Deleting it removes the replay surface (test_deleting_explicit_resume_target_breaks_resume). A retention policy may expire these, but only with an explicit TTL semantic agreed here.
2. Branch ancestors — every checkpoint on the parent chain from a branchable head back to (and including) the checkpoint before the oldest branchable message. Deleting any node on that walk breaks branch/regenerate with CheckpointLineageIntegrityError (test_deleting_branch_ancestor_breaks_lineage_loudly).
3. Pending writes — rows in the writes table are uncommitted/in-flight state, not garbage (test_pending_writes_are_retained_state_not_garbage).
4. Duration-only chain links — persist_run_durations appends metadata-only checkpoints. A duration-only checkpoint that a later run has forked from is a chain link: the walk steps through it, so deleting it requires grafting the fork onto the grandparent (rewriting the fork's parent_config) in the same change. A bare leaf (below) is safe; a link is not. The link shape can only be produced by the real runtime, so the graft path is specified here and intentionally not covered by a storage-level test.
5. Latest resumable state per thread — the newest checkpoint must remain addressable so a thread can always continue.

Provably safe forms (validated by tests)

1. Leaf sibling branches — a checkpoint forked off an older turn that has no children (test_leaf_sibling_branch_deletion_is_safe). Pruning it does not affect the main line's walk, explicit resume, or head.
2. Trailing duration-only leaves — a duration-only checkpoint no later run has forked from (test_leaf_duration_checkpoint_deletion_is_safe).

New deletion proposals must add their shape as a test here: construct the chain, delete, then verify (a) latest resume, (b) explicit checkpoint_id resume, (c) branch from an older visible turn, and (d) orphan row counts.

Deletion mechanics

• Deletion must cover the backend's tables jointly and account for orphans. General retention computes blob reachability from the surviving checkpoints in a whole-thread pass: after deleting a checkpoint row, a checkpoint_blobs / checkpoint_writes row is an orphan only if no surviving checkpoint references it. The shared-version case is not hypothetical — the real duration-only checkpoint is a copy of the head checkpoint dict (persist_run_history_metadata replaces only id/ts), so it inherits the parent's channel_versions verbatim, and on Postgres the blob rows reachable from the deleted duration row are the same rows backing its parent. An implementation that deletes blobs keyed by the removed checkpoint's own channel_versions would corrupt the thread's newest surviving state — exactly the failure class this contract exists to prevent. (For the same reason a real duration-only leaf is not payload-free: it materializes the parent's values under {"writes": {"runtime_run_duration": {...}}, "source": "update", "step": ...} metadata, which is what makes reclaiming it worthwhile.)
• Bounded admission cleanup instead proves that the root duration leaf and its addressable parent have identical channel_versions maps. It deletes only that leaf's checkpoint/writes rows and leaves all blobs intact: the parent still references every version the leaf did. No whole-thread blob scan or GC is needed for this shape.
• Failure semantics: if a proposed deletion cannot be proven safe against the protected set, it must not ship. Partial deletion that leaves a dangling parent_config converts a cleanup into a thread-level outage (branch and regenerate fail loudly for every later turn).
• Measurement first: proposals must include before/after numbers from ../scripts/benchmark/checkpoint/bench_channels.py (per-thread rows/bytes, SQLite and Postgres) plus the contract test suite passing.

History fast-path interaction (wiring requirement)

The trailing duration-only leaf is also the carrier of the run-history metadata cache: persist_run_history_metadata accumulates run_durations and run_message_ids in the leaf's metadata, and app/gateway/routers/threads.py::get_thread_history reads that map from the latest checkpoint (_checkpoint_run_durations / _checkpoint_run_message_ids, gated on is_latest_checkpoint) to answer every known turn's duration and message-to-run attribution without scanning the event store. The parent checkpoint the leaf clones does not carry that map.

Deleting the leaf (scenario E1) therefore removes the fast-path cache: the next history read sees no durations, falls back to event-store + run-manager scans, and _persist_run_history_metadata_background re-writes a fresh duration-only leaf — which the next retention pass deletes again. Net effect without sequencing: the reclaimed row comes straight back, plus recurring store scans and an extra write per read.

Gateway runs retention only when the next run has won durable same-thread admission, before that worker's first checkpoint read, and under the local checkpoint lock. The cache leaf remains available while the thread is idle; cleanup does not run from the completion callback. The admitted run's checkpoint_id and every checkpoint_map selector are explicitly protected.

The production enforce_completed_run_retention entrypoint delegates to enforce_trailing_duration_retention: at most two checkpoint lookups (the root head and its parent), one checkpoint-scoped pending-write existence probe, and one leaf deletion. It performs no alist, whole-thread writes audit, statistics or blob GC; work does not grow with retained history length. The exhaustive enforce_thread_retention engine remains separate for maintenance and contract measurements. A missing parent, changed channel versions, or writes on the candidate makes admission cleanup a no-op.

Wrapper histories are invalidated through the public cache lifecycle API before source deletion; an invalidation failure leaves the checkpoint intact. Cleanup failures are non-fatal to admission. Unpinned delta state reads retry once if their resolved target disappeared during hydration; explicit historical reads never silently move to a different checkpoint.

Item 4 note (large tool results)

ToolOutputBudgetMiddleware externalizes oversized tool outputs before they reach state (preview + file reference under .tool-results/), so the "50 KB result re-snapshotted every step" scenario from the original report depends on which tools/paths bypass it. The probe (../scripts/benchmark/checkpoint/bench_tool_result_probe.py) measures the on-disk checkpoint delta for the wrapped vs unwrapped paths on the lead graph; subagent chains instantiate the same middleware by default. Any PR claiming a residual gap must name the concrete bypassing path and show its probe numbers.