3 Commits

Author SHA1 Message Date
Nan Gao
a684ae2fe3
fix(sandbox): leave SIGTERM/SIGINT teardown to the Gateway lifespan (#7560)
* fix(sandbox): leave SIGTERM/SIGINT teardown to the Gateway lifespan

AIO and E2B providers built on the event-loop thread installed SIGTERM/SIGINT
handlers that force-destroyed every sandbox synchronously, before uvicorn
started the run drain. The Gateway lifespan now holds sandbox shutdown
ownership from its first step, so providers skip those handlers and the
existing hook closes the provider after run drain and MCP teardown. SIGHUP
keeps the provider fallback. AIO acquire now refuses once shutdown began.

* docs(sandbox): document Gateway-owned sandbox shutdown and AIO admission fence

* test(gateway): inspect the lifespan body now that lifespan wraps it

* docs(changelog): reference #7560 for the sandbox SIGTERM drain-order fix

* ci: retrigger sandbox image smoke after a registry pull timeout
2026-10-26 20:38:32 +08:00
Totoro
eae908c00d
fix(sandbox): make failed AIO teardown safely retryable (#7001)
* fix(sandbox): make failed AIO teardown safely retryable

* docs(sandbox): add sanitized teardown retry verification

* fix(sandbox): clear stale teardown fence on warm reclaim

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-18 09:57:12 +08:00
Czile
30841c3b9f
fix(sandbox): synchronize sandbox provider singleton lifecycle (+ concurrency regression tests) (#3730)
* fix(sandbox): synchronize sandbox provider singleton lifecycle

get_sandbox_provider() used an unsynchronized check-then-create, so two OS
threads (e.g. the main event loop and the Feishu channel thread, which runs
its own loop) could double-initialize the provider. With AioSandboxProvider
the overwritten instance leaks its idle-checker thread, since the only code
that joins it (shutdown()) is reachable only through the reference that was
overwritten.

reset_sandbox_provider(), shutdown_sandbox_provider() and set_sandbox_provider()
also touched the global without a lock, so a reset/shutdown racing an in-flight
create could clear it mid-construction or tear down an instance another thread
was about to return.

Guard all four lifecycle sites with a single module-level threading.Lock and
use double-checked locking in the getter, mirroring get_memory_storage().

* test(sandbox): add concurrent regression tests for provider singleton

- 8 threads racing on cold start, synchronized with a threading.Barrier so
  the check-then-create race fires deterministically; asserts exactly one
  provider instance is created.
- reset racing concurrent gets: asserts every returned value is a fully
  constructed provider (never None / half-built).

* fix(sandbox): lock get read path, run provider callbacks outside the lock

Addresses the three review findings on #3730:

1. get_sandbox_provider()'s read+return ran outside _provider_lock, so a
   concurrent reset/shutdown/set could null or tear the global between the
   check and the return, handing callers None / a torn instance. The hot read
   and the install reconciliation now both happen under the lock.

2. The non-reentrant _provider_lock was held across plugin-supplied callbacks
   (resolve_class import + provider __init__ in get; provider.reset()/shutdown()
   in reset/shutdown). A custom provider that re-entered these lifecycle
   functions would self-deadlock, and a slow teardown blocked every concurrent
   get(). resolve_class + construction now run outside the lock; reset/shutdown
   detach the reference under the lock and invoke the callback outside it.

   Tradeoff: racing cold-start callers may each construct a candidate. Exactly
   one is installed and returned to everyone; the losers (e.g. an AioSandbox
   instance that already started an idle-checker thread) are shut down so they
   do not leak the orphan thread #3721 is about. set_sandbox_provider() documents
   that it replaces but does not shut down the prior instance.

3. test_reset_racing_get reset the singleton to None *before* the barrier, so
   the racing reset was a no-op and never exercised reset-of-a-live-provider.
   It now populates the singleton up front so the reset tears down a live
   instance while getters read it.

Tests: rename the cold-start test to assert "one installed singleton, observed
by all" (construction is no longer single under the new design); add
shutdown-vs-get, set-vs-get, and a losing-racer-shuts-down-its-orphan case.
All five pass; the existing sandbox/middleware/mounts/uploads suites are green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 08:06:17 +08:00