mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-25 14:06:18 +00:00
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Dan Caldr
|
f2857ad3ff
|
fix(sandbox): default to loopback bind on Docker Desktop for DooD sandboxes (#5446)
* fix(sandbox): default to loopback bind on Docker Desktop for DooD sandboxes (#5445) * fix(sandbox): memoize desktop detection and clarify bind host docstring (#5445) * fix(sandbox): latch desktop detection on success only to permit retry on transient failure (#5445) * fix(sandbox): restrict desktop loopback bind to local DooD hostnames (#5445) * fix(sandbox): add Desktop legacy aliases and parametrize DooD host tests (#5445) |
||
|
Onefly
|
d8d110c637
|
fix(sandbox): prevent AIO subagent session eviction (#5178)
* fix(sandbox): prevent AIO subagent session eviction * fix(sandbox): address PR 5178 review issues * fix(sandbox): handle transient session and metadata failures * fix(sandbox): fence capacity upgrades and validate reused limits * docs(sandbox): restore list indentation and trim guidance * fix(ci): stabilize Buzz persistence test and trim sandbox guidance --------- Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Aari
|
0f7d8709d3
|
feat(sandbox): add controlled egress with approvals (#5152)
* feat(sandbox): add controlled egress approvals * Apply batched suggestions from code review * fix(sandbox): harden restricted network policy * fix(sandbox): harden denied egress handling * fix(sandbox): isolate network proxy sidecar * chore: retry sandbox image smoke * fix(sandbox): close remaining network policy gaps * fix(sandbox): harden relay token rejection * fix(sandbox): fence incompatible policy replacement * fix(sandbox): replace containers across network modes * fix(sandbox): close remaining lifecycle gaps --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jun
|
83cb6767b3
|
fix(sandbox): add FOWNER for AIO 1.11 startup (#5163)
* fix(sandbox): add FOWNER for AIO 1.11 startup * test(sandbox): cover FOWNER startup capability * docs(sandbox): document FOWNER capability * test(sandbox): pin FOWNER regression smoke * ci(sandbox): allow pinning FOWNER smoke image * style(sandbox): format FOWNER smoke test --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Aari
|
317577e285
|
fix: enforce custom agent skill allowlists in sandboxes (#5077)
* fix: enforce agent skill allowlists in sandboxes * fix: guard E2B skill projection resets * fix: preserve agent skill isolation across delegation * fix: close sandbox skill isolation bypasses * fix(sandbox): close skill isolation review gaps * fix(sandbox): harden skill isolation lifecycle |
||
|
luo jiyin
|
42796d7086
|
test: isolate Docker bridge gateway fallback from host DNS (#5107)
* test: isolate Docker bridge gateway fallback from host DNS Force the fallback-path test to bypass host DNS resolution. Production bind-host behavior is unchanged. Refs #5106 * test: document Docker fallback isolation Explain why the fallback test must replace host DNS resolution. Refs #5106 |
||
|
陈志谦
|
9e2c1be697
|
fix(sandbox): harden local Docker sandbox containers and port binding (#4986)
* fix(sandbox): harden local Docker sandbox containers and port binding Root causes (security audit SBX-1/SBX-2) in the local container backend: - _resolve_docker_bind_host published sandbox ports on 0.0.0.0 whenever DEER_FLOW_SANDBOX_HOST was non-loopback (docker-compose defaults to host.docker.internal), exposing the unauthenticated /v1/shell/* exec API on every host interface. - _start_container ran every sandbox with seccomp=unconfined and no capability, privilege-escalation, or resource limits, so untrusted model-authored code could exhaust the host, escalate privileges, and reach internal networks / cloud metadata endpoints directly. Hardening changes and defaults: - Port binding: non-loopback sandbox hosts now bind the Docker default bridge gateway instead of 0.0.0.0, discovered dynamically via `docker network inspect bridge` with a static 172.17.0.1 fallback. host.docker.internal resolves to that gateway through host-gateway, so DooD gateways and the Docker host still reach the sandbox while external interfaces no longer see the port. DEER_FLOW_SANDBOX_BIND_HOST=0.0.0.0 restores the legacy broad bind. - seccomp=unconfined is no longer unconditional: sandboxes run with Docker's default seccomp profile; opt back in with DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=1, only when the sandbox image is verified to require syscalls the default profile blocks. - Add --cap-drop=ALL and --security-opt no-new-privileges (Docker only; the Apple Container CLI does not support these flags). - Bounded resources with env overrides: --memory 2g (DEER_FLOW_SANDBOX_MEMORY), --cpus 2 (DEER_FLOW_SANDBOX_CPUS), --pids-limit 512 (DEER_FLOW_SANDBOX_PIDS_LIMIT); each also accepts "0"/"none" to disable the limit. - No --user is forced by default (the default AIO sandbox image's user is upstream-controlled and unverified), but DEER_FLOW_SANDBOX_CONTAINER_USER passes one through for deployments that know their image. - DEER_FLOW_SANDBOX_NETWORK passes --network so sandboxes can be attached to a dedicated egress-controlled network; default networking is unchanged. backend/docs/CONFIGURATION.md documents the new bind behavior and every override; tests cover each default and escape hatch. * fix(sandbox): follow host-gateway mapping for binds; keep image-required seccomp default Review follow-ups on the hardening change: - Bind: resolve the sandbox host itself and bind that address, instead of assuming the default bridge IPv4. host.docker.internal follows the daemon host-gateway-ip mapping (customizable, possibly IPv6), so the resolved address is exactly where the gateway connects — the published port and advertised URL always match. IPv6 is bracketed for docker -p, zone ids stripped, wildcard resolutions ignored; unresolved hosts fall back to the bridge gateway with a warning pointing at DEER_FLOW_SANDBOX_BIND_HOST. - seccomp: the shipped AIO image needs seccomp=unconfined for its Chromium browser (upstream quick-start always passes it; the upstream FAQ documents the browser failing under Docker default profile), so that option returns as the default. Tightening stays possible via DEER_FLOW_SANDBOX_SECCOMP_PROFILE=<path to a restricted, Chromium-compatible profile> or DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=0 for images verified to work with Docker's default profile. - cap-drop/no-new-privileges and the resource limits are unchanged. - Tests updated for both behaviors; 37 pass. * fix(sandbox): bracket bare IPv6 bind overrides; state seccomp default accurately DEER_FLOW_SANDBOX_BIND_HOST was returned verbatim, so a bare IPv6 literal like fd00::1 produced an invalid publish spec (fd00::1:port:8080); Docker requires the bracketed form. Normalize raw and already-bracketed IPv6 literals (IPv4/hostnames untouched), with resolver-level and argv-level tests covering the explicit IPv6 override. The CONFIGURATION.md overview claimed Docker's default seccomp profile stays active, contradicting the seccomp=unconfined default the table (and the code) actually ship for the Chromium-based image; spell out the relaxed default and where to change it. * style(sandbox): apply ruff format to local_backend * fix(sandbox): reject host networking, force builtin seccomp opt-out, resolve hostname binds Review follow-up on #4986 (willem-bd): - P1: DEER_FLOW_SANDBOX_NETWORK=host (and container:<name>) now raise a RuntimeError at start instead of silently voiding the hardened port bind — Docker discards -p/--publish in host mode and shares the network namespace for container:<name>, which would re-expose the unauthenticated exec API on the host's interfaces. Two regression tests cover both rejections. - P2: the seccomp opt-out now passes seccomp=builtin explicitly instead of omitting the option, so a daemon configured with an unconfined or custom default cannot weaken the documented opt-out; the test asserts the flag. - P2: hostname values in DEER_FLOW_SANDBOX_BIND_HOST resolve to an address before use (Docker publish specs require an IP literal as the host part, so host.docker.internal previously produced an invalid spec that prevented every sandbox from starting); unresolvable names raise a clear configuration error. Tests cover resolution and rejection; CONFIGURATION.md updated for all three behaviors. 43/43 pass in tests/test_aio_sandbox_local_backend.py; ruff check + format clean. * fix(sandbox): reject DEER_FLOW_SANDBOX_NETWORK=none (loopback-only, breaks published API port) * fix(sandbox): validate the effective Docker network target; normalize IPv6 sandbox hosts once name=host / name=none dodge raw-string checks but attach like the bare words; strip name= prefixes and validate the effective target (network IDs keep passing). Bracketed IPv6 sandbox hosts now resolve for the bind and bare IPv6 hosts produce bracketed URL authorities — both input forms give identical bind and URL addresses. * fix(sandbox): parse the full Docker network long syntax before validating Docker accepts comma-separated key=value fields in any order (name=, gw-priority=, alias=, ...); a name=host field hides the host network behind surrounding fields. Parse the CSV and validate the parsed name= target (last occurrence wins, fields lowercased, mirroring opts/network.go); no-name values fall through like Docker's own rejection. * fix(sandbox): keep CHOWN/SETUID/SETGID through cap-drop=ALL for the default image The shipped image's entrypoint starts as root, creates the gem user, chowns /opt/jupyter and drops to that user via su; without those three capabilities the set -e script dies before the readiness endpoint exists. no-new-privileges stays (it blocks gaining privileges via exec, not using the added caps). Adds a docker-gated real-image startup smoke test. * fix(sandbox): let pre-initialized non-root images drop the startup capabilities The CHOWN/SETUID/SETGID re-add only exists for the shipped image's root entrypoint handoff. A custom image that never runs as root gets an explicit opt-out (DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0) so those capabilities are not left available to sandboxed code (chown on bind mounts, UID/GID impersonation). * test(sandbox): gate the real-image smoke test behind the live marker The default offline suite (make test = -m 'not live') must not depend on a third-party registry: mark the smoke test live, probe the daemon inside the test body (never at collection time), and allow pinning the image reference via DEER_FLOW_SANDBOX_SMOKE_IMAGE for a dedicated integration job. * test/docs: isolate DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS in tests; add table row; split custom-image guidance _clear_hardening_env now clears the new knob so a developer shell or .env preset cannot flip the default-path tests. CONFIGURATION.md gains the table row, and the custom-image guidance becomes its own paragraph with the no-new-privileges scope stated correctly (it does not mitigate the retained CAP_SETUID/SETGID risk). * test(sandbox): make the live smoke test diagnosable 300s readiness budget (cold pull + cold start must not be conflated with broken capabilities) and dump the container's last 40 log lines on failure so the next live run tells us whether the capability set is incomplete (chown/useradd/su errors) or the services are merely slow. * test(ci): align the smoke test with the 60s provider deadline; add a dedicated live smoke workflow Single-source the readiness deadline as SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT (used by both provider paths and the smoke test) so the validation cannot drift from the production contract again. New sandbox-image-smoke.yml runs the live test on a dedicated job, with the image reference pinnable via the SANDBOX_SMOKE_IMAGE repository variable (digest resolved and recorded in the job summary when falling back to :latest). * test(sandbox): pull the failing program's own logs on smoke failure supervisord only surfaces exit codes in docker logs; nginx's stderr lands in files inside the container. Dump supervisor program logs, nginx -t, and the nginx error log on failure so the next run names the exact broken line. * ci(sandbox): export an immutable repo@digest reference for the smoke run docker pull once on the runner platform, resolve RepoDigests[0], and pass that immutable reference to the test via GITHUB_ENV — the recorded and executed images can no longer diverge when the tag moves, and platform selection is left to the daemon instead of jq over the manifest index. * fix(sandbox): add DAC_OVERRIDE — the root nginx master writes gem-owned logs The image's root nginx master opens /var/log/nginx/{access,error}.log, which belong to the gem user, for the container's lifetime; without CAP_DAC_OVERRIDE it dies with 'open() failed (13: Permission denied)' on every start (FATAL under supervisord) and readiness never arrives. Four capabilities now: CHOWN/SETUID/SETGID for the entrypoint handoff plus this runtime log-write need. |
||
|
Aari
|
5eb59cb130
|
fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes (#4221)
* fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes Docker sandboxes are shared across gateway workers, but each worker kept its own in-memory warm pool. Startup reconciliation adopted every running container, so a peer idle reaper could destroy sandboxes another worker still owned and tool calls hit 502 / Connection refused. Add file-based ownership leases under sandbox-leases/, only adopt true orphans, refuse idle/replica/shutdown destroy while a foreign lease is live, and renew the lease on create/get/release/reclaim. Fixes #4206 * fix(sandbox): close lease fail-open, hot-path IO, and check→destroy race Address review of the multi-worker orphan lease (#4206): - read_lease returns None only for a genuinely-absent lease and raises (CorruptLeaseError/OSError) when a lease is unreadable or corrupt, so the ownership check fails closed instead of mistaking an unprovable peer lease for a free container. clear_lease still removes a stuck/corrupt file. - get() no longer renews the lease (blocking mkdir/fsync/os.replace on the event loop path used by ensure_sandbox_initialized_async); active leases are renewed off the event loop from the idle checker (_renew_active_leases). - The ownership check and container stop run under a per-sandbox flock guard (lease_ownership_guard); every lease write takes the same guard so a peer's touch cannot interleave with a destroy. Same-host multi-worker scope, not a multi-pod distributed lock. Also fixes the ruff format lint on the branch. Adds regression tests: corrupt and unreadable lease fail closed, a tests/blocking_io anchor keeping get() non-blocking on the event loop, and a peer-touch/destroy interleave test. * fix(sandbox): share container ownership across gateway instances Rework of the #4206 fix per review: ownership state is shared through a third-party service instead of being maintained per gateway instance, following the stream_bridge precedent (sandbox.ownership.type: memory | redis). The file lease and its same-host flock guard are deleted, not ported — they only covered workers on one host, while the deployment that hits #4206 is a load-balanced multi-instance gateway. A lease answers "who reaps this container", not "who may use it". Containers are deterministic per (user, thread), so consecutive turns legitimately land on different instances: take() transfers ownership on acquire, while claim() gates every adopt/reap path. Leases carry a state — own: or del: — so a takeover is refused against a teardown in progress. Without it an unconditional take() would overwrite a destroyer's claim and the peer's container stop would land on a sandbox the new owner had already handed to an agent. renew() distinguishes a lapsed lease from one a peer took; only the latter drops the sandbox. Collapsing them meant a Redis restart evicted every in-flight sandbox on every instance at once. Renewal runs on its own thread with a TTL derived from its interval, never from idle_timeout: renewal used to ride the idle checker, which does not start at idle_timeout: 0, so leases silently lapsed on a supported config. Ownership establishment is fail-closed: a sandbox whose ownership cannot be published is never handed out, and a just-created container is destroyed rather than leaked as an adoptable orphan. Every destroy path claims before untracking. The memory store is single-instance only and says so; the resolver reads app_config.stream_bridge and the env var in the bridge's own order, so deployments already using Redis get a redis ownership store without extra config. * fix(sandbox): wait out a recovery grace before adopting a keyless container An absent ownership lease meant two opposite things on two paths. Renewal reads it as LAPSED and re-establishes it: nobody took the lease, so the container is still ours. Reconciliation read the same absent key as "orphan" and adopted on sight. After the store loses its keys (a Redis restart without persistence, or eviction under maxmemory) every owner is alive and merely pre-renewal-tick. Whichever instance reconciled first therefore adopted every live container; each real owner's next renewal reported LOST and dropped a sandbox it was serving mid-turn, leaving it for the adopter to idle-destroy — #4206 through the back door, in the very case the LAPSED handling was added to make safe. Not limited to startup: an already-running instance hits the same window from the idle checker's periodic reconcile. _adoptable_after_grace requires an untracked container to be seen unowned across a full lease TTL before it can be adopted. That rebuilds the delay the state loss erased: a live owner republishes within one renewal interval, shorter than the TTL by construction, while a crashed owner never does, so its containers are still adopted one grace later rather than leaking. A republished lease resets the grace; a pausing-only timer would still expire over a live owner's lease. The peek is read-only — the atomic claim still gates adoption. The grace is skipped when the store cannot coordinate across processes: no peer can hold a lease such a store would show us, so single-instance deployments keep instant orphan cleanup, and a grace could not help a multi-worker gateway on memory anyway. * fix(sandbox): hold the teardown lease for as long as the container stop runs claim(..., for_destroy=True) wrote the del: marker with the ordinary lease TTL and nothing refreshed it. renew() extends only own: and deliberately reports a teardown as LOST, and the destroy paths drop the sandbox from the maps the renewal loop iterates — so a container stop that outlived the TTL let the marker lapse, a peer's take() succeeded against the still-running container, and the stop then landed on the turn that had just been handed it. That is the exact window the del: state exists to close, reopened by its own expiry. The two lease states alone never made the per-sandbox flock redundant, as I claimed when deleting it: a held lock cannot expire, a lease can. The exclusion has to be held deliberately rather than assumed to outlast the work it guards. _held_teardown_lease wraps both _backend.destroy() call sites and re-claims the marker every renewal_interval_seconds until the stop returns. No store change is needed: claim(for_destroy=True) already refreshes an existing del: marker on both backends. Reachable without an abnormal backend. The schema bounds only renewal_interval_seconds (> 0) and ttl_multiplier (>= 2), so a legal config puts the TTL below a normal container stop; and LocalContainerBackend._stop_container passes no timeout to subprocess.run, so a wedged daemon blocks unbounded even at the default 120s TTL. The TTL stays finite on purpose: the heartbeat dies with the process, so a destroyer that crashes mid-stop still releases the container one TTL later instead of marking it undestroyable forever. * fix(sandbox): hold the teardown lease on every del: stop, and pin the claims that had no test 90936b49 said `_held_teardown_lease` wrapped "both" `_backend.destroy()` call sites. There are three. `_drop_unhealthy_sandbox` marks `del:` and then blocks on the same unbounded stop, and it untracks *before* claiming, so `_renew_owned_leases` cannot see the id either — nothing refreshed the marker. Reproduced against a real redis: the peer's `take()` succeeds 1.0s into a 2.5s stop. Same window, third path. That miss came from the habit the rest of this commit addresses: a property asserted in prose, with no test that could falsify it. Auditing every load-bearing claim in this feature — AGENTS.md, the store docstrings, the provider's design comments — against the test that would go red turned up several more, each verified by mutating the code and watching the suite stay green. Tests that could not fail: - `test_reconcile_fails_closed_when_ownership_unknown` reached the grace gate, not the claim. A bare MagicMock answers `owner()` with a truthy mock, so the container read as peer-owned and deferred; `claim()` was never called. It stayed green with `_claim_ownership` failing open. Adding the grace ahead of the claim is what hollowed it out — inserting a gate can silently disarm the tests for the gate behind it. - `test_adoption_grace_restarts_when_a_live_owner_republishes` never distinguished reset from pause. Those diverge only on a *second* lapse, which it never drove, so it passed with the reset deleted. Claims with no test at all, each now pinned (mutation → red, per test): - `destroy()`, `_evict_oldest_warm`, `_reclaim_warm_pool_sandbox`, `_register_created_sandbox` and `shutdown()`'s warm loop were each the one untested sibling of an "every path does X" enumeration. `shutdown()` was never driven with a non-empty warm pool, so a loop bypassing the ownership claim — stopping a live peer's container on our exit — went unnoticed. - Renewal's unknown-is-not-lost rule, the single deliberate exception to fail-closed. Inverting it drops every active and warm sandbox on every instance the moment the store blinks. - Both hops of the stream-bridge redis inference. Deleting either left the suite green while every config.yaml-native multi-instance deployment silently fell back to memory — #4206 reopened on exactly the deployments the inference exists for. Claims narrowed instead, because they promised more than the code delivers: - "run against both backends ... cannot drift" — CI provisions no redis, so the merge gate runs the memory tier only and the Lua never executes there. - "Every destroy path claims before untracking" — `_drop_unhealthy_sandbox` untracks first, deliberately, under its `expected_info` TOCTOU guard. - "Atomic: concurrent claims from different instances cannot both succeed" — true via Lua on redis, vacuous on the single-instance memory store, and pinned by neither, since the contract suite drives sequential calls. A concurrency test against the memory store would make the claim look covered while the mechanism that carries it still never runs in CI. * fix(sandbox): release the teardown marker when a destroy() stop fails The three `del:`-marked stop paths disagreed on failure. `_destroy_warm_entry` releases on both outcomes and says why: the stop failed, so the container is probably still up, and a marker left behind refuses its own thread's `take()` until the TTL lapses. `_drop_unhealthy_sandbox` does the same. `destroy()` had no such guard — a raising backend propagated straight past `_release_ownership`, and the thread could not re-acquire for a full TTL. Fails safe rather than fatal: a stuck marker stops peers from touching the container, it is not the cross-instance kill. But the paths must agree, and this one is the odd one out. Release, then re-raise. Swallowing would be the easier symmetry with `_destroy_warm_entry`'s `return False`, but `destroy()` has no failure return and `shutdown()` logs per sandbox off the exception, so swallowing would silently narrow what callers can see. Found by comparing the three paths after @fancyboi999 asked for release to be handled "consistently with the other destroy paths" on the unhealthy path — which 0d2377b2 already does. This is the sibling that wasn't. * fix(deploy): bump chart config_version to 27 for sandbox.ownership config.example.yaml went to 27 with the new sandbox.ownership section, but the chart embeds its own copy and stayed at 26, so validate-chart failed. A bare bump: the chart already sets stream_bridge.type=redis, which is what resolve_ownership_config infers a redis ownership store from, so no field change is needed. * fix(sandbox): release the teardown lease from its heartbeat, not the caller `_held_teardown_lease` joined its heartbeat only briefly and the caller cleared the `del:` marker right after the stop. A refresh `claim` still in flight (`RedisOwnershipStore` had no socket timeout, so a round trip could block) could land *after* that release and rewrite `del:` on a container whose stop had already completed — refusing a fresh `take()` (or rolling back a fresh create) until the TTL. Move the release into the heartbeat's own `finally`, after its loop stops, so no refresh can run after it. The three destroy paths no longer release after the `with` (`destroy()`'s no-container branch still does, since no lease was held there). Bound every store round trip with a socket timeout so the in-flight refresh — and thus the deferred release — stays finite, and broaden the heartbeat's `except` so an unexpected error cannot strand the marker during a long stop. Also fold in the review follow-ups: stop re-resolving an already-resolved ownership config in the factory, document the Redis-outage-vs-TTL boundary in config.example.yaml, and add a tests/blocking_io anchor pinning that `release()`'s store round trip stays off the event loop. * fix(sandbox): refuse a non-destroy claim that would unwind our own teardown `claim(for_destroy=False)` against our own `del:` lease fell through and overwrote it with `own:`, cancelling a teardown that was already in flight. The container stop cannot be recalled, so downgrading the marker would let a `take()` hand out a container that is about to die -- #4206, self-inflicted. No caller does this today: the two non-destroy callers run against an absent key (the LAPSED re-claim) or an unowned one (post-grace reconcile). The contract has to forbid it rather than rely on that staying true. Fixed in both backends. The redis rule lives in Lua and the memory rule in Python, so fixing one only would let them drift silently -- and the shared contract suite is what is supposed to catch that drift, so it now covers this. Also adds a contention test for `claim`. The suite drove sequential calls only, so it pinned the exclusion predicate but not the atomicity that predicate depends on; eight instances now race for one container and exactly one must win. * fix(sandbox): bound the container stop so it cannot outlive its teardown lease `_stop_container` passed no `timeout` to `subprocess.run`, so a wedged container runtime blocks it forever. The `del:` marker is what keeps a peer from re-acquiring the container while the stop runs, but a marker is a lease and a lease can lapse: a store outage longer than the TTL frees it, a peer's `take()` succeeds against the still-running container, and the stop then lands on the turn that was just handed it -- the exact #4206 failure. The teardown heartbeat already covers the case where the store stays reachable. This bounds the worst case independently of the ownership layer, which is the point: it holds even when the ownership layer is the thing that failed. A timeout is not swallowed like a `CalledProcessError`. That error means the runtime answered "I could not stop it"; a timeout means we do not know, and the container is probably still running -- returning normally would let `_destroy_warm_entry` report a clean stop and drop the warm entry, leaking a running container nothing tracks. * fix(sandbox): exclude this instance's own reapers from its acquire path An ownership lease excludes peers and nothing else. `claim()` and `take()` both succeed against our own `own:` lease by design -- that is what lets a destroy path claim what it already owns -- so `del:` says nothing to this process's other threads. Meanwhile every reaper decides outside `_lock`, because a store round trip must not be held under the lock that guards every acquire. So each reaper acts on a decision its own acquire path may already have invalidated, and the store cannot see the difference. Six paths end in an irreversible act (a container stop, or closing a host-side client) on a decision made outside the lock. All six reproduce: _evict_oldest_warm re-checks warm membership, then releases the lock _reap_expired_warm no re-check at all _cleanup_idle_sandboxes re-verifies idle, then releases the lock _renew_owned_leases acts on a stale renew() -> LOST release() same staleness on its own refresh _drop_unhealthy_sandbox untracks before claiming, opening discovery Both warm reapers are a regression from the deferred pop this branch introduced: `WarmPoolLifecycleMixin` popped under the lock, so a reclaim's membership check failed and the race could not occur. Deferring the pop is still right (popping first loses the container on a refused claim), so the exclusion has to be made explicit instead. The idle path is pre-existing in shape, but this branch widened it from a few instructions to a network round trip by claiming ownership before untracking. Two guards, because the two directions want opposite answers: Reaping -- nothing may promote it. The reaper reserves the id, and every promote path refuses a reserved id exactly as it refuses a peer's `del:` (drop and cold-start). The "is this still reapable?" test travels with the reservation as a predicate and runs in the same critical section, because checking first and reserving second is the window, not a narrower version of it. Forgetting -- the peer legitimately wins, so the promote is what to detect. `_publish_ownership` bumps a per-id acquire epoch; the callers that decide from a store round trip snapshot it first, and the pop is skipped if it moved. Object identity cannot substitute: the reuse path re-publishes ownership while handing out the same tracked `AioSandbox`, so an identity check sees nothing and the pop closes a client mid-turn. `still_reapable` is required rather than defaulting to unconditional -- the safe default is the one that makes a new call site think about it. That diverges from the mixin hook, which is safe because this provider overrides both mixin callers, and loud rather than silent if those are ever dropped. Also closes a client leak on the discover path: "nothing to roll back" was true of the container but not of the HTTP client constructed before the publish, which the sibling create path already closes. The shared-store test view rebound `owner_id` outside the store's lock, so a concurrent claim could execute under the wrong id and read its own lease as a peer's. Serialized, so the heartbeat-hold tests stop flaking. * fix(sandbox): mark acquire intent before the ownership round trip A guard must become visible no later than the transition it guards. The acquire epoch cannot manage that for `take()`: the takeover is durable before `take()` returns -- redis has committed the SET while the reply is still in flight -- and the epoch can only be written afterwards. In that interval the store already says the container is ours while the epoch still reads as it did when a renewal decided `LOST`, so the stale forget walks through, drops the maps and closes the client the acquire is about to hand back. Acquire then returns an id the provider no longer tracks and `get()` answers `None` for the rest of the turn. `_publish_ownership` now publishes an intent mark under `_lock` before the round trip; the epoch keeps covering the other half, "an acquire completed since you decided". `_forget_lost_sandbox` honours the intent mark unconditionally rather than only when an epoch is supplied -- today's epoch-less callers cannot reach the window, but "no epoch" reading as "no guard" is how the next caller of a dangerous primitive gets written. The same invariant had four more instances, all reproduced: reuse returns a decision the forget already invalidated -- before the mark is set a `LOST` is both current and correct, so the forget legitimately runs and the entry reuse decided to hand out is gone. Re-check after publishing and fall through to discovery instead. reclaim installs an entry a reaper reserved after its check -- the warm entry is still visible during the stop, and the reaper's claim succeeds because reclaim's own take() just made the lease ours. Re-check likewise. the reservation was released before the entry was removed -- the pop belonged to the caller, leaving a gap where the container is stopped, the entry is still in `_warm_pool`, and nothing marks it. `_destroy_warm_entry` removes it itself, inside the reservation; the pop stays deferred relative to the stop, just not to the reservation. reconcile adopts a container this instance is tearing down -- adoption is a promote and needs the same reservation check as the others. Neither existing guard excludes it: the claim succeeds because the lease is ours, and on `memory` the recovery grace is skipped outright. The pre-round-trip checks in reuse and reclaim are kept as early-outs, since they skip a health check and a store round trip on a doomed entry, and are pinned to that job rather than to a correctness role they no longer hold. The teardown reservation predicate runs under `_lock`, so it must not touch the lock. Documented rather than engineered around: making the lock reentrant to tolerate it would trade a loud hang for a quiet class of re-entrancy bugs across the rest of the provider. * fix(sandbox): honor local teardown after ownership publish * fix(sandbox): clear a stale warm entry when an id becomes active Active and warm are exclusive states, and the two register paths were the only place that could hold both: they inserted into `_sandboxes` without popping `_warm_pool`, so one container ended up with two reapers. `_reap_expired_warm` judges an entry by its warm timestamp and never consults `_last_activity`, so it stops a container an agent is actively using while `_sandboxes` still hands out its client. Reachable because `_reconcile_orphans` adopts an untracked-but-running container into the warm pool inside the register's publish -> track window, and on the `memory` store it adopts on sight: `_adoptable_after_grace` short-circuits when `supports_cross_process` is False, so an id carrying this process's own lease reads as adoptable. That window is new to this branch -- on main the track was a single locked insert with nothing before it. Both register paths now pop the warm entry inside the same locked section that installs the active one. * fix(sandbox): harden ownership renewal teardown --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
DanielWalnut
|
f401e7baa6
|
[codex] Fix stale AIO sandbox cache reuse (#3494)
* Fix stale AIO sandbox cache reuse * Address AIO sandbox review feedback * Distinguish sandbox health check failures * Keep local discovery recoverable when the runtime check fails LocalContainerBackend.discover() shares _is_container_running, which now raises on transient daemon errors instead of returning False. Discovery has no exception handling in _discover_or_create_with_lock(_async), so a brief Docker hiccup turned a recoverable "could not verify, create instead" into a hard acquire failure. Catch the check failure inside discover() and return None so an unverifiable container is simply not adopted, restoring the pre-change fall-through while keeping raise-on-unknown semantics protecting the destroy path. Reported by fancy-agent on PR #3494. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Narrow the not-found match in container inspect error handling A bare "not found" substring also matches transient failures like "command not found" or "context not found", which would misclassify a check error as "container definitely gone" and bypass the raise-on-unknown contract. Keep Docker's specific "No such object"/"No such container" phrases, and only trust a generic "not found" (Apple Container) when the message names the inspected container or refers to a container/object. Reported by WillemJiang on PR #3494. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Hinotobi
|
74081a85a6
|
[security] fix(sandbox): bind local Docker ports to loopback (#2633)
* fix(sandbox): bind local Docker ports to loopback * fix(sandbox): preserve IPv6 loopback Docker binds * fix(sandbox): log Docker bind host selection |
||
|
DanielWalnut
|
f7dfb88a30
|
fix(aio-sandbox): redact env values in container logs (#2562)
* fix(aio-sandbox): redact env values in container logs Fixes #2534 * fix(aio-sandbox): address env log review comments |
||
|
LYU Yichen
|
3e461d9d08
|
fix: use safe docker bind mount syntax for sandbox mounts (#1655)
Docker's -v host:container syntax is ambiguous for Windows drive-letter paths (e.g. D:/...) because ':' is both the drive separator and the volume separator, causing mount failures on Windows hosts. Introduce _format_container_mount() which uses '--mount type=bind,...' for Docker (unambiguous on all platforms) and keeps '-v' for Apple Container runtime which does not support the --mount flag yet. Adds unit tests covering Windows paths, read-only mounts, and Apple Container pass-through. Made-with: Cursor |