* fix(backend): stop forcing WindowsSelectorEventLoopPolicy on Windows
backend/sitecustomize.py switched every Python process with backend/ on
sys.path to WindowsSelectorEventLoopPolicy (added by #3236 for 'make dev'
on Windows). On Windows the selector loop cannot spawn asyncio
subprocesses - asyncio.create_subprocess_exec raises a bare
NotImplementedError - so ACP agents, stdio MCP servers, and local sandbox
command execution are all broken on native Windows, and the swallowed
exception surfaces as 'Error invoking ACP agent x: ' with no reason.
The forcing has no beneficiary left on Windows: the async postgres driver
is asyncpg (not psycopg, as #3236 states), and asyncpg 0.31.0 ships no
Windows wheels, so the postgres backend is unavailable on native Windows
regardless of the loop; the default SQLite backend (aiosqlite) runs on
either loop. The make dev startup fix in #3236 came from the serve.sh /
wait-for-port.sh changes, which are untouched.
With the Python-default ProactorEventLoop restored, subprocess features
work again on native Windows.
* fix(benchmark): scope a selector loop to the postgres checkpoint benchmark on Windows
Static review of the sitecustomize removal flagged the one caller that
relied on the old startup policy: bench_channels.py postgres workers run
bare asyncio.run() and enter AsyncPostgresSaver.from_conn_string(), which
the lockfile-pinned psycopg 3.3.3 rejects on Windows under
ProactorEventLoop before connecting.
Instead of restoring the global selector policy, the postgres branch of
the benchmark now scopes WindowsSelectorEventLoopPolicy to itself right
before asyncio.run(); memory/sqlite cases and every other backend process
keep the subprocess-capable Proactor default.
* test(checkpoint): retention deletion contract + growth baseline
Six contract scenarios x memory/sqlite/postgres pin what retention deletions
must never break (branch ancestors, explicit resume targets, pending writes,
duration-only chain links), prove the two safe shapes (leaf sibling branches,
trailing duration leaves), record the full-vs-delta growth baseline in the
normalized bench shape, and add an item 4 probe showing the default
ToolOutputBudgetMiddleware already externalizes oversized tool results.
Refs #4189
* test(checkpoint): make the retention contract load-bearing per review
Review findings from willem-bd and Ricky-7-Yan:
- scenario D pins its own row: before/after stats delta plus a serde
round-trip of the stored write, instead of an always-true > 0 check
- _delete_checkpoint now performs the joint delete the doc mandates
(checkpoint row + writes rows + blobs unreachable from surviving
checkpoints), so E1/E2 exercise the shape they prescribe
- E1 builds the real runtime duration shape via persist_run_durations
(parent dict clone, fresh id/ts, real metadata), which surfaces the
shared-version case: the leaf's blobs are the surviving parent's rows
- contract doc: blob reachability must be computed from surviving
checkpoints in a whole-thread pass; shared-version/duration-only
hazard called out explicitly; memory data model includes saver.blobs
- _stats counts memory blob rows and returns the full normalized shape
(logical byte totals included)
- probe: drops the unused middleware/outputs_dir graph parameters and
discloses the manual-harness scope limit in the module docstring
- E1/E2 assert default head resolution (protected set item 5); unused
graph_for helper and DURATION_ONLY_METADATA stand-in removed
Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
* fix(checkpoint): scope probe cleanup to owned dirs, key report by backend
Second-round review findings on #5255:
- [P1] bench_tool_result_probe.py removed the whole user-supplied
--outputs-dir (and the shared .probe-tmp) in its finally block, so
pre-existing files were deleted on success and failure alike. The run
now writes into (and removes) a fresh owned probe-run-* child beneath
the requested directory, and SQLite databases live in a unique
mkdtemp'd temp directory that is removed with the run. Regression
tests pin that unrelated pre-existing files survive both a successful
and a simulated failing run.
- [P2] the optional retention report keyed every backend's measurements
under one shared name, so a multi-backend invocation kept only the
last backend's numbers. _report() now takes the parameterized backend
explicitly (saver_env.kind); regression pins that memory and sqlite
entries coexist in one report file.
Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
---------
Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
* feat(checkpoint-cache): delta-mode checkpoint history cache with recursive compose
Read-only, invalidation-free cache for LangGraph delta-channel history
({writes, seed}) at the get_delta_channel_history choke point:
- database.checkpoint_cache config (memory|redis; max_entries 0=disabled;
redis bounded by TTL, Gateway/async only)
- memory LRU backend (copy-on-read, zero-serde hit path) and redis backend
(lazy import, degrades to all-miss on outage)
- CachedHistorySaver: recursive composition from the nearest warm ancestor
(depth budget 8), caching each level; depth-0 cold chains delegate one
inner fast-path walk. Entries keyed by immutable
(db, thread, ns, checkpoint_id, channel) — no invalidation, coherent
across workers
- provider wiring: wraps in delta mode only (async + sync), full mode
untouched; sync path is memory-only
- bench opt-in: DEERFLOW_CHECKPOINT_BENCH_HISTORY_CACHE=1
sqlite bench (500 updates, payload 2KB): write phase 2.28x at f=250,
1.32x at f=10; one delegated walk per thread cold start.
* chore(config): bump config_version to 32 for database.checkpoint_cache
The checkpoint history cache feature added the database.checkpoint_cache
section to config.example.yaml; bump the schema version so existing
deployments get the outdated-config warning and can run make config-upgrade.
* chore(helm): bump config_version to 32 in chart values and README
* fix(checkpoint-cache): purge thread history entries on delete paths
Addresses review on #4638: delete_thread/prune removed source-of-truth
checkpoints but left the thread's materialized history payloads in the
cache (memory: until LRU eviction; redis: until TTL, default 1 day) — a
data-lifecycle gap for tenant offboarding / GDPR-style erasure.
- Cache contract gains thread-scoped adelete_thread/delete_thread
(lifecycle purge, not invalidation; entries remain immutable)
- Memory backend: stem scan over the LRU map; redis: SCAN MATCH + UNLINK,
outage degrades to TTL-bounded retention without raising
- CachedHistorySaver purges on delete_thread/adelete_thread and
prune/aprune (prune rewrites chains, so pre-prune histories must go);
delete_for_runs stays delegation-only (run->thread mapping unavailable,
no in-tree callers), documented in code
- ttl_seconds description documents the residual-retention window
- Tests: thread-scoped purge on both backends, saver-level delete/prune
purge, prefix-safety (t1 vs t10), redis outage degradation, and the
pinned no-purge behavior of delete_for_runs
* fix(checkpoint-cache): stable db identity, prefix-aware sync singleton, explicit zero TTL
Addresses Copilot review on #4638:
- checkpoint_cache_db_hash now hashes the credential-free postgres
identity (host:port/database + schema): credential rotation no longer
changes the cache namespace (cold cache + orphaned keys until TTL).
Unparseable URLs fall back to the raw string.
- The sync-path memory cache singleton is also keyed by its key_prefix:
a namespace change (db identity change or operator override) recreates
the cache instead of leaving stale-prefix entries unreachable and
unpurgeable.
- ttl_seconds=0 is now an explicit, documented opt-out of redis expiry
(SET without EX; redis maxmemory policy only) instead of a silent
'ttl_seconds or None' coercion.
Tests: credential-rotation hash stability, unparseable-URL fallback,
prefix-change singleton recreation, same-prefix singleton reuse, and
zero-TTL wire behavior (ex=None).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(checkpoint): make delta snapshot_frequency configurable
* fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning
Addresses review on #4516: the rename from the flat
database.checkpoint_delta_snapshot_frequency key to nested
database.checkpoint_delta.snapshot_frequency silently dropped the old
value (pydantic extra="ignore"). Add a before-validator that maps the
legacy key onto the nested one with a deprecation warning (nested key
wins when both are set), plus a CHANGELOG breaking-change note covering
the rename and the 1000 -> 10 default change.
* fix(checkpoint): validate frozen snapshot frequency
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency
- Group benchmark scripts into per-family folders (checkpoint/, sandbox/)
- Extract shared benchmark infrastructure into checkpoint_bench_common.py
- Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen);
freeze it in make_lead_agent and DeerFlowClient; key the state-schema
adaptation cache by resolved frequency
- New bench_production.py: per-case child processes run N ainvoke turns through
the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver),
then measure GET /state + POST /history through the real Gateway route stack
in one event loop (httpx ASGITransport), cold/warm accessor-cache split,
cross-mode digest gates
- New summarize_production.py: delta/full ratios plus decision metrics
(snapshot_write_spike, cache_effect_ms, checkpoint_write_share,
auto-discovered history per-limit ratios)
* fix(checkpoint): address production benchmark review
