22 backend tests assert facilities that only exist on a POSIX host and
hard-fail on Windows (CI is ubuntu-latest, so upstream never sees them):
- test_extension_dependency_sync.py: 11 tests spawn bare 'make -n';
hosts without GNU make get WinError 2 (make is not part of the
documented Windows setup).
- test_lark_broker.py: 10 tests execute the '#!' fake lark-cli / shim /
launcher scripts; CreateProcess cannot run shebang scripts
(WinError 193).
- test_bench_sandbox_provider.py: the boxlite shim workaround test
asserts POSIX exec-bit semantics after os.chmod; Windows has no
execute bit and boxlite sandboxes are Linux-only.
Each now skips with an explicit reason when its facility is absent;
Linux/CI coverage is unchanged. On a Windows host the three files go
from 22 failed to 0 failed / 22 skipped.
Fixes#5930
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency
- Group benchmark scripts into per-family folders (checkpoint/, sandbox/)
- Extract shared benchmark infrastructure into checkpoint_bench_common.py
- Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen);
freeze it in make_lead_agent and DeerFlowClient; key the state-schema
adaptation cache by resolved frequency
- New bench_production.py: per-case child processes run N ainvoke turns through
the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver),
then measure GET /state + POST /history through the real Gateway route stack
in one event loop (httpx ASGITransport), cold/warm accessor-cache split,
cross-mode digest gates
- New summarize_production.py: delta/full ratios plus decision metrics
(snapshot_write_spike, cache_effect_ms, checkpoint_write_share,
auto-discovered history per-limit ratios)
* fix(checkpoint): address production benchmark review
* bench: add provider-agnostic sandbox benchmark with BoxLite warm pool results
- scripts/bench_sandbox_provider.py: CLI for measuring acquire/run/release
across providers, scenarios, workloads, and concurrency levels
- scripts/summarize_bench.py: JSONL aggregation with p50/p95/p99 tables
- bench_results.jsonl: 110 turns across 7 scenarios on real BoxLite 0.9.7
Key findings:
cold acquire: ~860ms
warm reclaim: ~14ms (60x speedup)
release: ~0ms
warm_hit_rate: 95% (warm_same_thread)
* perf(boxlite): skip health check for recently-released warm pool boxes
Boxes released within health_check_skip_seconds (default 5.0s)
are promoted directly without the ~14ms echo-ok round-trip.
A VM alive seconds ago is overwhelmingly likely to still be alive.
Add sandbox.health_check_skip_seconds config option.
Set to 0 to always health-check (old behaviour).
Benchmark (warm_same_thread, noop, 20 iters):
acquire p50: 14.9ms → 0.0ms
total p50: 29.9ms → 14.0ms
* chore: move benchmark scripts into backend/scripts/benchmark/
* fix: address BoxLite benchmark review findings
* fix(boxlite): only skip warm reclaim checks for released boxes
* fix(benchmark): keep BoxLite shim workaround off the event loop
* fix(boxlite): invalidate dead boxes from command path
* test(boxlite): cover skip window and invalidation edge cases
* fix(boxlite): treat sandbox-has-been-closed as terminal in _exec
* fix(boxlite): harden warm-pool reclaim and benchmark accounting
* fix(boxlite): validate warm-pool reclaims by default
* fix(config): expose boxlite health-check skip setting
* fix(boxlite): tighten failure classification and benchmark workaround
* Update config_version to 21 in values.yaml
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
