deer-flow/backend/docs/checkpoint-retention-contract.md

Checkpoint Retention Contract (DRAFT)

Status: draft — the deletion contract for #4189 item 3. No retention or deletion implementation should land before this contract (or a successor revision of it) is accepted, and every deletion proposal must be validated against backend/tests/test_checkpoint_retention_contract.py.

Why a contract is needed

LangGraph checkpoints form a per-thread parent chain. Gateway features depend on that chain being intact:

• Branch / regenerate resolves the replay base by walking parent_config links from a checkpoint that contains the target message (app/gateway/checkpoint_lineage.py::find_checkpoint_before_message).
• Explicit resume replays from a checkpoint_id a client still holds.

Deleting checkpoint rows by recency or table size can therefore break those features silently — a missing ancestor surfaces as CheckpointLineageIntegrityError at branch time, or as a lost resume target, never as an obvious storage bug. The contract below separates deletable rows from protected rows and pins the verification method.

Data model

Backend	State rows	Writes rows
SQLite	checkpoints	writes
Postgres	checkpoints, checkpoint_blobs	checkpoint_writes
Memory	saver.storage, saver.blobs	saver.writes

(Note: SQLite has no separate blob table; channel values live inside the serialized checkpoint payload. Postgres splits blobs out.)

Measurement shape: per-thread rows + bytes per table, normalized by bench_channels._normalized_storage_stats.

Protected set (MUST NOT delete without the stated compensation)

1. Explicit resume targets — any checkpoint_id a client may still resume to. Deleting it removes the replay surface (test_deleting_explicit_resume_target_breaks_resume). A retention policy may expire these, but only with an explicit TTL semantic agreed here.
2. Branch ancestors — every checkpoint on the parent chain from a branchable head back to (and including) the checkpoint before the oldest branchable message. Deleting any node on that walk breaks branch/regenerate with CheckpointLineageIntegrityError (test_deleting_branch_ancestor_breaks_lineage_loudly).
3. Pending writes — rows in the writes table are uncommitted/in-flight state, not garbage (test_pending_writes_are_retained_state_not_garbage).
4. Duration-only chain links — persist_run_durations appends metadata-only checkpoints. A duration-only checkpoint that a later run has forked from is a chain link: the walk steps through it, so deleting it requires grafting the fork onto the grandparent (rewriting the fork's parent_config) in the same change. A bare leaf (below) is safe; a link is not. The link shape can only be produced by the real runtime, so the graft path is specified here and intentionally not covered by a storage-level test.
5. Latest resumable state per thread — the newest checkpoint must remain addressable so a thread can always continue.

Provably safe forms (validated by tests)

1. Leaf sibling branches — a checkpoint forked off an older turn that has no children (test_leaf_sibling_branch_deletion_is_safe). Pruning it does not affect the main line's walk, explicit resume, or head.
2. Trailing duration-only leaves — a duration-only checkpoint no later run has forked from (test_leaf_duration_checkpoint_deletion_is_safe).

New deletion proposals must add their shape as a test here: construct the chain, delete, then verify (a) latest resume, (b) explicit checkpoint_id resume, (c) branch from an older visible turn, and (d) orphan row counts.

Deletion mechanics

• Deletion must cover the backend's tables jointly and account for orphans, and blob reachability must be computed from the surviving checkpoints in a whole-thread pass: after deleting a checkpoint row, a checkpoint_blobs / checkpoint_writes row is an orphan only if no surviving checkpoint references it. The shared-version case is not hypothetical — the real duration-only checkpoint is a copy of the head checkpoint dict (persist_run_history_metadata replaces only id/ts), so it inherits the parent's channel_versions verbatim, and on Postgres the blob rows reachable from the deleted duration row are the same rows backing its parent. An implementation that deletes blobs keyed by the removed checkpoint's own channel_versions would corrupt the thread's newest surviving state — exactly the failure class this contract exists to prevent. (For the same reason a real duration-only leaf is not payload-free: it materializes the parent's values under {"writes": {"runtime_run_duration": {...}}, "source": "update", "step": ...} metadata, which is what makes reclaiming it worthwhile.)
• Failure semantics: if a proposed deletion cannot be proven safe against the protected set, it must not ship. Partial deletion that leaves a dangling parent_config converts a cleanup into a thread-level outage (branch and regenerate fail loudly for every later turn).
• Measurement first: proposals must include before/after numbers from ../scripts/benchmark/checkpoint/bench_channels.py (per-thread rows/bytes, SQLite and Postgres) plus the contract test suite passing.

History fast-path interaction (wiring requirement)

The trailing duration-only leaf is also the carrier of the run-history metadata cache: persist_run_history_metadata accumulates run_durations and run_message_ids in the leaf's metadata, and app/gateway/routers/threads.py::get_thread_history reads that map from the latest checkpoint (_checkpoint_run_durations / _checkpoint_run_message_ids, gated on is_latest_checkpoint) to answer every known turn's duration and message-to-run attribution without scanning the event store. The parent checkpoint the leaf clones does not carry that map.

Deleting the leaf (scenario E1) therefore removes the fast-path cache: the next history read sees no durations, falls back to event-store + run-manager scans, and _persist_run_history_metadata_background re-writes a fresh duration-only leaf — which the next retention pass deletes again. Net effect without sequencing: the reclaimed row comes straight back, plus recurring store scans and an extra write per read.

The wiring PR that introduces the production trigger must therefore either:

1. Sequence retention away from history reads — e.g. run retention on a schedule whose next pass re-reclaims the re-created leaf, or run it when the thread is not being read; or
2. Adopt a policy that spares cache-carrying leaves — e.g. a RetentionPolicy flag that keeps any trailing duration-only leaf whose metadata still carries run_durations / run_message_ids (same spirit as the strict pending-writes guard), at the cost of not reclaiming that leaf's rows.

Without either, E1 pruning and history reads churn against each other. This decision belongs to the wiring PR, not to the storage-level service: the service cannot tell a cache-carrying leaf from a payload-free one on the alist path without re-implementing the writer's merge semantics.

Item 4 note (large tool results)

ToolOutputBudgetMiddleware externalizes oversized tool outputs before they reach state (preview + file reference under .tool-results/), so the "50 KB result re-snapshotted every step" scenario from the original report depends on which tools/paths bypass it. The probe (../scripts/benchmark/checkpoint/bench_tool_result_probe.py) measures the on-disk checkpoint delta for the wrapped vs unwrapped paths on the lead graph; subagent chains instantiate the same middleware by default. Any PR claiming a residual gap must name the concrete bypassing path and show its probe numbers.