4 Commits

Author SHA1 Message Date
yinx65588-boop
4ec0fe1bb4
fix(tenki): retain cleanup ownership across provider retirement (#7311)
* fix(tenki): retain sandbox ownership through provider retirement

* fix(tenki): fence terminal callbacks by sandbox generation

* docs(sandbox): keep Tenki lifecycle guidance within the chain budget

* fix(tenki): report pending exit cleanup without a traceback

---------

Co-authored-by: yinx65588-boop <yinx65588-boop@users.noreply.github.com>
Co-authored-by: yinx65588-boop <272180046+yinx65588-boop@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-24 08:49:35 +08:00
FanouZeng-TT
eb7c342fb1
fix(sandbox): measure warm-pool idle age on a monotonic clock (#7035)
* fix(sandbox): measure warm-pool idle age on a monotonic clock

Warm-pool expiry measures a duration, but every provider stamped and reaped
entries using the wall clock. A forward correction could therefore destroy
fresh sandboxes, while a backward correction could retain expired ones until
the clock caught up.

Move every warm-pool writer and reader to time.monotonic(), including the
Tenki and OpenSandbox reset paths missed by the earlier closed attempt. Keep
absolute SandboxInfo.created_at and reconciliation grace timestamps on the
wall clock. Convert AIO's sibling active-idle timestamps as one clock domain,
and update finite clock mocks for the additional duration read.

Fixes #6970

* test(sandbox): keep active-idle race fixtures in the monotonic domain

The production AIO activity clock moved to monotonic time, but the shared
orphan-reconciliation helper still stamped active fixtures with wall time.
CI then computed a negative idle age, so two teardown race tests never reached
the ownership gate. Use the same monotonic domain as production.

* test(sandbox): read warm-pool idle fixtures on the monotonic clock

The idle-age conversion left several fixtures seeding the fields the reaper
now reads with values from the clock it no longer reads:

- `test_idle_reaping_forces_a_new_remote` froze `warm_pool_lifecycle.time.time`,
  so the benchmark's reaper compared against real elapsed monotonic seconds —
  zero over the test's lifetime. Nothing expired, the entry survived, and the
  second turn came back as a warm hit.
- Three orphan-reconciliation fixtures seeded `_last_activity` with `0.0` and
  two aio fixtures with `time.time()`, both of which are far below a running
  monotonic clock: the computed idle age went negative, so the reaper never
  reached the ownership claim these tests are built to exercise.

Seed them from `time.monotonic()` instead, and keep `created_at` (a wall-clock
field) on `time.time()`.
2026-10-20 19:36:04 +08:00
NanPan
68dcaa6025
fix(sandbox): keep shutdown retryable while idle reaper is live (#6231)
* fix(sandbox): fail shutdown while idle reaper is live

* test(sandbox): pin live idle reaper timeout

* fix(sandbox): keep shutdown retryable after reaper timeout

* fix(sandbox): keep shutdown retryable after reaper timeout

* fix(sandbox): keep shutdown retryable after reaper timeout

* fix(sandbox): keep shutdown retryable after reaper timeout

* docs(sandbox): pin idle-reaper shutdown ownership

* fix(sandbox): keep AIO warm entries retryable

* test(sandbox): fix AIO retry regression import

* test(sandbox): cover retryable provider shutdown guards

* test(sandbox): assert warm ownership survives failed shutdown
2026-10-04 07:01:13 +08:00
Vanzeren
67c8ade375
feat(sandbox): warm pool for boxlite (#3951)
* feat: add boxlite SDK as harness dependency

* test: add fake SimpleBox fixtures for BoxLite warm pool tests

* feat: add deterministic sandbox_id and warm pool fields to BoxliteProvider

* feat: pass deterministic sandbox_id to SimpleBox name

* feat: warm pool lifecycle — park on release, reclaim on acquire

- release(): parks VMs in _warm_pool with timestamp instead of closing
- _reclaim_warm_pool(): health checks warm boxes via echo ok
- acquire(): tries warm pool reclaim before creating new boxes
- Deterministic sandbox_id ensures thread isolation

* feat(boxlite): idle reaper, replica enforcement, warm-pool shutdown/reset

- Task 6: idle reaper daemon thread destroys expired warm-pool boxes
- Task 7: replica enforcement evicts oldest warm-pool box when at capacity
- Task 8: shutdown() stops idle checker first, destroys all boxes (active+warm);
  reset() clears warm pool

Tests: 17 passed (4 new: idle reaper, replica enforcement, shutdown, reset)

* fix(boxlite): harden warm pool lifecycle races

* docs(boxlite): document warm pool configuration

* fix(sandbox): log warning when evicting oldest warm box is failed

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(boxlite): stop provider lifecycle on reset

* fix(boxlite): make runtime an optional dependency

* test(boxlite): remove unused warm pool lookup

* docs(boxlite): clarify optional runtime support

* refactor: extract shared WarmPoolLifecycleMixin for sandbox warm-pool lifecycle

Introduce deerflow.community.warm_pool_lifecycle.WarmPoolLifecycleMixin
owning idle-checker loop, warm-pool expiry, oldest-warm eviction,
replica counting, and soft-cap logging. Move AioSandboxProvider and
BoxliteProvider onto the mixin; keep AIO active-idle cleanup local and
delegate only warm-pool expiry to the shared helper.

BoxliteProvider also gains:
- Prefixed box names (deer-flow-boxlite-*) for startup orphan reconciliation
- _reconcile_orphans() adopting surviving boxes from a prior process
- Pinned timeout forwarding: command timeout now bounds both BoxLite
  SDK exec(timeout=...) and the loop bridge .result(timeout)
- reset() reworked as lightweight registry clear (boxes -> warm pool,
  no close, no idle-reaper stop, no loop close) so reset_sandbox_provider()
  config switches are safe; shutdown() remains the teardown path

Backward-compatible: AIO DEFAULT_IDLE_TIMEOUT/DEFAULT_REPLICAS/
IDLE_CHECK_INTERVAL stay importable; Boxlite IDLE_CHECK_INTERVAL
stays monkeypatchable.

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-07-06 07:21:12 +08:00