c668716737
* fix(sandbox): stop E2B reconciliation from reviving warm-pool sandboxes Periodic reconciliation probed every discovered remote sandbox with Sandbox.connect() before the locality check, and the check itself only consulted _sandboxes, not _warm_pool. A sandbox parked by release() was therefore adopted back to active on the first pass, and because the SDK normalizes connect(timeout=None) to its 300s default and the control plane extends a running sandbox's expiry when now+timeout is later, each 60s pass kept pushing the expiry forward — idle warm sandboxes never hit their configured idle_timeout. Treat _sandboxes and _warm_pool ids as locally tracked up front: skip probing them (no timeout-mutating connect), keep them canonical, and route only genuinely remote candidates through the duplicate-reap path. Extend the post-probe adoption recheck to _warm_pool so a release that lands mid-probe cannot be promoted back to active either. Fixes #5550 * fix(sandbox): keep active E2B VMs alive and sweep expired warm entries Address review on #5562: - Reconciliation now refreshes the remote TTL of locally active sandboxes through their cached client (never connect()), restoring the keepalive for turns that outlive idle_timeout without reviving warm-pool VMs. - Warm-pool entries parked longer than idle_timeout are dropped during reconciliation — their VMs are expected to be reaped by the control plane — releasing the ownership lease and the capacity slot they would otherwise pin until reclaim, eviction, or shutdown. - Remove the now-dead thread-local canonical sort; locally tracked ids are skipped unconditionally, so the ordering hint had no effect. * fix(sandbox): preserve active E2B keepalive and shared capacity * fix(sandbox): serialize E2B reconciliation lifecycle transitions * fix(sandbox): fence E2B ownership and timeout lifecycle writes * fix(sandbox): isolate ownership heartbeats from E2B timeout IO --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
1.7 KiB
E2B lifecycle
Reconciliation floors active-client TTL renewals at the SDK's default timeout and twice the sum of the configured reconciliation interval and pass budget (subject to the E2B timeout cap). Release still uses the configured idle timeout. Sweeping old local warm entries must not release deployment capacity: a peer may have renewed the VM. Leave shared removal to revision-checked remote inventory and its missing-entry grace period; partial/failed inventory cannot prove absence.
Per-VM _sandbox_lifecycle locks have independent timeout and ownership
domains. Timeout locks serialize active TTL writes with removal from the active
map. Ownership locks serialize publication/claim/renewal/release with warm-entry
cleanup. Never hold both domains: slow E2B timeout requests must not block lease
heartbeats for this VM or subsequent VMs in the renewal pass.
Snapshot readers recheck sandbox identity, parked-entry identity and acquisition
intent under that lock before acting. Lock order is thread key, lifecycle, then _lock;
never wait for a lifecycle lock while holding the metadata lock or perform
remote I/O under _lock. Holders and waiters retain one refcounted RLock per domain/ID,
reclaimed on the last exit. Cleanup remains available during shutdown without
an executor or a permanent per-sandbox lock table.
Release must own the final idle-TTL write, and cleanup must finish before a new
lease can be published. Do not renew ownership removed by a concurrent cleanup.
Release only needs the VM lock while leaving active state; do not hold it
during output sync, which must not prevent ownership heartbeats.
Track that release in _remote_ops_in_progress until it completes so
reconciliation cannot probe or re-adopt a VM between active and warm states.