3 Commits

Author	SHA1	Message	Date
Zeren Wang	7cf0abe584	test(checkpoint): delta-history cache migration coverage + pin two open DeltaChannel defects (#5797) ... * test(checkpoint): cover delta-history cache migration and pin two open DeltaChannel defects - Extend the full -> delta migration contract to {cache disabled, cold, warm} with a differential raw-saver oracle; the warm state also asserts a cache-served re-read with no extra inner history walks. - Pin two still-unfixed upstream DeltaChannel defects as tripwire regressions: #8382 (parallel-superstep delta replay order diverges from live order) and #8448 (Postgres paged delta walk hydrates old checkpoints empty). Each asserts the correct contract and skips, naming the issue and pinned version, while the defect is present - so a later dependency bump that lands the fix turns it into a live gate. The contract is never asserted as buggy behaviour (see backend/tests/AGENTS.md). * test(checkpoint): scope the #8448 skip to the postgres parameter The paged stage-1 walk that langgraph#8448 poisons only exists in langgraph.checkpoint.postgres; memory and sqlite never paginate. An unscoped skip let a future regression in long-chain materialization on those backends hide behind the Postgres-only upstream issue instead of failing the InMemorySaver differential oracle. Keep the assertion live there and trip only where the defect can occur.	2026-09-24 15:16:08 +08:00
Zeren Wang	31bfd40f38	chore(deps): bump langgraph-checkpoint to 4.2.0 / postgres 3.1.2 and delete the saver patch (#5734) ... * chore(checkpoint): bump to checkpoint 4.2.0 and drop the stale saver patch langgraph-checkpoint 4.2.0 fixes the write dropped after a full -> delta migration (upstream langchain-ai/langgraph#8526) and keeps its own InMemorySaver override, so the compatibility patch in checkpoint_patches.py could neither detect the fix nor stand down: it read langgraph's version while InMemorySaver ships in langgraph-checkpoint, and it would have kept shadowing the upstream implementation on 4.2.0. The postgres release 3.1.2 locates plain-value delta seeds (upstream #8535). - floor langgraph-checkpoint>=4.2.0,<5.0 in the harness and raise the postgres extra to >=3.1.2,<3.2; langgraph's own >=4.1.0 would otherwise let a resolver reintroduce the bug now that the patch is gone - delete the patch, its version guard, and the three implementation-assertion tests; test_full_to_delta_migration_replays_on_same_thread stays as the gate and fails on 4.1.1 without the patch (verified by a one-off unpatch run) - langgraph stays 1.2.9 and langgraph-checkpoint-sqlite stays 3.1.1 Verification: make lint; delta checkpointer/cache/worker-resume suites green with the migration gate passing on memory and sqlite (postgres params skip without TEST_POSTGRES_URI). The full make test run before the packaging-pin fix below was 13 failed / 18523 passed; 12 of those failures reproduce on main with the same environment (git stash baseline, identical failure sets), 7 of them because the local config.yaml runs checkpoint_channel_mode: delta. The remaining one, the postgres-extra string pin in tests/test_checkpointer.py, is updated here. Upstream status for the surrounding gaps: cache parity across off/cold/warm states, postgres pagination (langchain-ai/langgraph#8448) and parallel-superstep replay order (#8382) are still open upstream, as is the abandoned-branch fork fix (#8548) that the delta resume linearization in runtime/runs/worker.py works around, so that path stays. * docs(changelog): keep checkpoint upgrade in Unreleased --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-23 09:18:26 +08:00
Vanzeren	42baed8c8c	feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel (#4292) ... * feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel Add a restart-required database.checkpoint_channel_mode ("full" default, "delta") that stores the messages channel via LangGraph 1.2 DeltaChannel, cutting checkpoint storage from O(n^2) to O(n) for append-only history. Existing full checkpoints seed delta state transparently; no data migration. - config: mode schema + freeze-on-first-use with CheckpointModeReconfigurationError; mode marker persisted in checkpoint metadata; unsafe delta->full downgrade rejected fail-closed with CheckpointModeMismatchError (run-level error, failed state read) - state: delta message state schema; CheckpointStateAccessor centralizes materialized reads for all consumers (threads API, branches, regeneration, compaction, state updates, memory, goal workers) - runtime: raw writers (run durations, interrupted title, thread goal) parent their checkpoints to the checkpoint they derive from, preserving delta ancestry; rollback forks the pre-run lineage through a state mutation graph with Overwrite restores; InMemorySaver delta-history override delegates to the base walk (fixes dropped first write after migration, also present upstream) - tests: conformance suite over {memory, sqlite, postgres} covering migration replay, stable message IDs, storage shape and writer preservation; conftest fixture isolates the frozen mode between tests; stale config fakes refreshed - ci: backend unit tests gain a postgres service * fix(checkpoint): close materialization gaps in goal flow, guard public factory - Route goal-continuation message reads through CheckpointStateAccessor: raw channel_values reads see the delta sentinel in delta mode, which disabled goal continuation (stand_down=no_durable_end_of_turn) after durable assistant turns. Raw tuples remain for tuple-only metadata (checkpoint id, pending_writes). - Reject checkpoint_channel_mode='delta' + checkpointer in create_deerflow_agent at construction: factory-built persisted graphs bypass mode-marker injection and the fail-closed gate, reproducing silent mixed-mode state loss. Delta without persistence stays allowed. - Import the postgres saver lazily (pytest.importorskip in the fixture) so the documented default install collects the suite; add a CI job running pytest --collect-only on uv sync --group dev without extras. - Fix test_checkpointer fallback test to patch get_app_config at its use site (provider module), making it deterministic when a local config.yaml selects a persistent backend. * fix(gateway): preserve extension-owned channels in state mutations, bump config version - build_state_mutation_graph / build_checkpoint_state_mutation_accessor accept an explicit state_schema; branch and POST /state now compile the mutation graph from the thread's effective schema (graph_state_schema on the assistant graph). The base-ThreadState fallback silently discarded channels contributed by custom AgentMiddleware.state_schema on branch (data loss) and returned a false-success 200 on POST /state. - POST /state validates values keys against the mutation graph's channels and rejects unknown fields with 422 instead of ignoring them; reducer detection covers extension channels (BinaryOperatorAggregate or DeltaChannel) so Overwrite replace semantics work for middleware reducers in both modes. - Endpoint regression: custom AgentMiddleware.state_schema value survives branch, updates through POST /state, and an unknown field receives 422. - config_version 26 -> 27 for the new database.checkpoint_channel_mode (example, Helm chart values + README, support-bundle fixture), so existing installs get the outdated-config warning and make config-upgrade merges the field; covered by a test driving the real example file and the real config-upgrade script. * fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite GET /threads/{id}/state now resolves the thread's assistant_id through a single reusable boundary (thread metadata -> assistant_id -> effective graph), so channels contributed by a custom AgentMiddleware.state_schema are materialized instead of dropped by the default lead schema. POST /state uses the same boundary instead of resolving the schema ad hoc. Branch writes wrap every copied reducer channel in Overwrite (derived from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel), not just messages, so already-aggregated values are never re-merged. Regression tests use a real AgentMiddleware.state_schema with a non-identity reducer in both full and delta modes: GET /state returns the extension value, POST /state replaces it, branch preserves it byte-for-byte; the unknown-field 422 is a separate assertion. * refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests Address review round 4 on PR #4292: - Push ahistory/history limit through Pregel into checkpointer.alist (SQL LIMIT) instead of materializing all rows and breaking in Python - Fold the read-side mode-compat gate onto the returned snapshot's metadata; only writes keep the pre-write tuple fetch (fail-closed) - Cache factory-built accessor graphs per (assistant_id, mode) with factory-identity revalidation; state reads no longer build a lead agent per request - get_thread: one snapshot fetch + one raw pending_writes fetch on the resolved checkpoint (post-checkpoint __error__ writes never surface in snapshot.tasks; verified empirically) - DeerFlowClient.get_thread: single checkpointer.list walk collects pending_writes per checkpoint instead of N get_tuple calls - InMemorySaver delta-history patch: stand-down when the upstream override disappears, try/except guard, validated-version warning, guard tests - make_lead_agent mode precedence: first freeze is owned by app_config (client-supplied configurable key ignored); once frozen, injected key/app_config must match or fail closed - Rollback: lock in non-message channel restoration via fork inheritance with a dedicated reducer-channel test - Add tests/test_threads_checkpoint_mode.py and tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and the PR validation section: lifecycle parity (memory + sqlite), per-step blob-count storage guard, gateway endpoint parity Counted-saver tests pin checkpoint round-trips for aget/ahistory so these regressions cannot silently return. * fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience - threads router: map CheckpointModeMismatchError to 409 (with cause and thread id) and CheckpointModeReconfigurationError to 503 across all state endpoints instead of swallowing both into a generic 500 - gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary through the real route stack - rollback: compile the restore mutation graph with the thread's effective state schema per the build_state_mutation_graph contract - inheritance contract locks: rollback and manual compaction preserve middleware-contributed channels via checkpoint fork cloning - services: revalidate the accessor-graph cache against app_config identity so config.yaml hot-reloads never serve a stale compiled graph - services: degrade full-mode state reads to raw checkpointer reads when the agent factory is unavailable (delta gate still applies; delta mode has no fallback) - deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility) and bump the langchain lower bound to what the lockfile actually resolves * fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread - _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while pregel's get_state_history treats config.checkpoint_id as the inclusive start; fetch the anchor explicitly so both read paths paginate identically - extend the degraded-path gateway test: GET /thread returns raw values, and POST /history with before starts at the anchor checkpoint * fix(gateway): preserve degraded checkpoint timestamps * fix(gateway): harden degraded checkpoint access * fix(gateway): resolve assistants for checkpoint reads --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-22 08:33:29 +08:00