deer-flow/backend/packages/harness/deerflow/sandbox/AGENTS.md

Sandbox System (packages/harness/deerflow/sandbox/)

Interface: Sandbox: execute_command(command, env=None), additive execute_command_in_scope(..., scope_id=...) / release_command_scope(scope_id), read_file, write_file, list_dir, glob, grep. Scoped hooks default to pass-through without server-side sessions, preserving third-party subclasses. grep accepts a text file or directory tree. Per-call env injects secrets: LocalSandbox merges into the subprocess environment; AioSandbox uses fresh bash.exec(env=...) sessions. list_dir: missing path → FileNotFoundError; command/client failure → OSError, never [] (ls_tool: (empty)). Remotes use sandbox/remote_list_dir.py: capture find's status, not | head's (sh -lc lacks pipefail); missing binary → OSError; truncation SIGPIPE → success. Provider Pattern: SandboxProvider exposes acquire, acquire_async, get, release. Async agent/tool paths use async hooks to keep Docker creation, discovery, cross-process locking, readiness polling, and release off-loop. Set supports_agent_skill_isolation=True only when the whole tool surface enforces explicit lead Agent policy: bind mounts use prepared thread roots; upload providers implement sync_agent_skills. Host-backed providers report false if an enabled shell bypasses path mappings. Under explicit policy, middleware rejects unsupported providers before acquire. Shared components (RFC #4741): remote IDs use derive_sandbox_scope_token (sandbox/identity.py); preserve its keyword-only SHA-256/16-hex contract to avoid orphaning containers. AcquireSerializer (sandbox/acquire_serialization.py) serializes selected acquire/release transitions with a bounded, refcounted per-key threading.Lock table and dedicated bounded executor (no event-loop/default-executor blocking). Workers own cancellation cleanup without waiting for cancelled tasks to resume; provider shutdown()/reset() calls idempotent close(). Keys: AIO (user_id, thread_id), E2B (user_id, thread_id, skills_root), BoxLite/Tenki/OpenSandbox derived id. Random-UUID thread_id=None acquires bypass serialization. Execution leases (sandbox/lease.py, #5128): cross-instance ownership decides which Gateway may reap a container; process-local SandboxLeaseManager tracks concurrent lead, subagent, Gateway-request, and channel-upload users of one client. Runs get ephemeral owners, persisted sandboxes are retained idempotently, and the last holder performs any pending SandboxProvider.release. Outer lifecycle fences repeat idempotent release after their complete graph/tool/request batch drains; per-tool terminal Command wrappers never release because sibling handlers may still run. Fork-restored children and upload syncs use non-releasing holders: they fence the client and own scope cleanup without themselves requesting a park; an earlier normal-owner request waits for them, and a missing fork client is replaced by a normal owner. Persisted lookup plus retention is serialized per (user_id, thread_id); stale bindings fall through to acquire, and a post-acquire lookup miss rolls back before raising. Provider I/O does not hold the metadata lock. Repeated cancellation cannot interrupt acquire/rollback/release reconciliation or let a to_thread sandbox operation outlive its enclosing execution/request holder; failures are logged without replacing the original cancellation. Lease/scope context IDs are server-owned: Gateway and worker scrub caller values; only the internal subagent path assigns a task ID. Managers are registered by provider object identity, not hash/equality, so unhashable custom providers remain valid. Subagent owners also serve as sandbox_command_scope_id: AIO gives each scope one ordered persistent shell session, replaces it after ErrorObservation, and cleans it on lease release. Registry identity is revalidated after every scope-lock wait, preventing queued commands from resurrecting released sessions. Env-bearing commands use fresh bash.exec sessions so secrets do not persist. Authorization gate (sandbox:execute, RFC #4063 Phase 3): every sandbox-backed tool call passes through the gate in deerflow/authz/sandbox_authz.py - a binary authorize(principal, "sandbox", "execute", target="*") check before either reusing a persisted sandbox id or calling provider.acquire. Rechecking reuse is required because authorization config and user roles can change while the sandbox remains cached. Sync tool invocations call authorize_sandbox_execution; async tool invocations await authorize_sandbox_execution_async exactly once. A task-local ContextVar scopes that single decision across the complete composed tool invocation, including ReadBeforeWriteMiddleware's pre-write inspection, tool body, and post-read mark; the value is copied into asyncio.to_thread workers. Authorization denial is converted to the normal error ToolMessage at the composed middleware boundary and is explicitly excluded from the gate's generic fail-open handlers. Async config loading and provider class discovery/import are offloaded before aauthorize() so reused sandbox calls do not hash config files or import custom modules on the event loop; provider construction remains on the running event loop because async providers may initialize loop-affine clients. The gate lives at the single tool initialization entry point (ensure_sandbox_initialized / ensure_sandbox_initialized_async in tools.py), while SandboxMiddleware.before_agent / abefore_agent apply the matching sync/async check to eager acquisition. Deny raises SandboxAuthorizationError (sandbox/exceptions.py), which propagates out of ordinary tool execution as a friendly error ToolMessage ("sandbox execution is not permitted for your role") - the eager path catches it and skips acquisition instead, deferring the deny to the first sandbox-touching tool call so both paths share the same semantics. Provider errors (authorization calls and provider resolution) follow authorization.fail_closed / fail_open; no readable config.yaml or authorization.enabled: false makes the gate a no-op (safe_app_config tolerates missing config). Gateway upload/artifact sync calls try_acquire_sandbox_for_request (app/gateway/authz.py), which gates, returns a request lease, and skips sync on deny while preserving the primary operation. Callers release after their last sandbox operation; artifacts request normal parking, uploads do not. Tests: tests/test_sandbox_authorization.py and tests/blocking_io/test_sandbox_authorization.py. Environment policy (sandbox/env_policy.py): execute_command no longer inherits the full os.environ. build_sandbox_env() scrubs secret-looking names (*KEY*/*SECRET*/*TOKEN*/*PASS*/*CREDENTIAL*) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. OPENAI_API_KEY) never leak into skill subprocesses. Benign vars (PATH, HOME, LANG, VIRTUAL_ENV, ...) are preserved. Implementations:

• LocalSandboxProvider - Local filesystem execution. acquire(thread_id) returns a per-user/thread LocalSandbox (id local:{user_id}:{thread_id}) whose path_mappings resolve /mnt/user-data/{workspace,uploads,outputs} and /mnt/acp-workspace to that thread's host directories, so the public Sandbox API honours the /mnt/user-data contract uniformly with AIO. acquire() / acquire(None) keeps the legacy generic singleton (id local) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a threading.Lock. Shared runs use category mappings; a policy-scoped run replaces them with one /mnt/skills root mapping to the coherent thread view, so structured file tools resolve through one managed boundary. This is not a host filesystem security boundary: an enabled host bash subprocess can use canonical paths without PathMapping, so supports_agent_skill_isolation is dynamic and explicit Agent policies fail closed while host bash is enabled. Host-to-virtual output masking scans dynamic per-user/per-thread roots directly instead of compiling path-specific regexes, so evicted thread IDs do not remain in Python's global regex caches; a separate 256-entry root cache prevents repeated realpath() walks for every glob/grep match while bounding dynamic-path retention, and only the small process-stable skill/integration source set uses a bounded compiled cache. On Windows, Git Bash/MSYS argument-conversion exclusions are limited to safe non-root virtual path prefixes; do not restore a blanket conversion disable, because host-native CLI launchers need normal MSYS path conversion for their own installation paths.
• AioSandboxProvider (packages/harness/deerflow/community/) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. get() remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (_renew_owned_leases). Text appends use the AIO file API's native append mode rather than a client-side read-modify-write, so a failed pre-read cannot turn an append into an overwrite. reset() closes the per-instance acquire serializer so replacing the singleton cannot retain its executor workers; full remote sandbox teardown remains shutdown(). uses_thread_data_mounts defaults to backend detection (LocalContainerBackend=True, remote/provisioner backends=False), while the optional sandbox.thread_data_mounts boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it true skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. An explicit Agent policy uses four thread projection category mounts and a distinct deterministic sandbox identity, preventing reuse of an older container created with shared mounts. skills.container_path is a provider-startup snapshot shared by mount construction, sandbox identity, the remote Gateway request, and provisioner validation; custom roots are identity-scoped so a container or Pod created for one destination cannot be reused after the root changes. The Gateway and provisioner independently require one canonical absolute root that does not overlap reserved platform mounts, and both derive the four category allowlist entries from that root. The provisioner accepts all four category overrides; when all are present it suppresses the default hostPath or skills-PVC mount. With USERDATA_PVC_NAME, the thread projection categories use subpaths on that shared data PVC. Readiness probes and agent_sandbox clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set trust_env=False; external FQDNs and public IPs retain environment proxy support.
• E2BSandboxProvider (packages/harness/deerflow/community/e2b_sandbox/) provides E2B remote isolation. New unrestricted sandboxes receive a one-shot upload from the enabled-only public, custom, legacy, and managed integration projections. For a thread with an explicit Agent policy, creation skips that shared upload; after acquire, sync_agent_skills clears only DeerFlow's four managed category directories and signature before strictly uploading the signed thread projection. It rejects non-canonical paths, protected mounts/homes, symlinked roots, and standard operating-system trees before destructive work. E2B rebuilds the managed categories on every policy sync; it deletes legacy sandbox-visible signature markers instead of trusting them as proof that the remote tree is intact. Reset plus all four uploads hold the same per-user/thread and skills-root serializer used by acquire and release, so overlapping policy syncs cannot interleave a wipe with another run's upload. Delegated subagents are non-owners of the lead's thread projection: they reuse that filesystem view and never rebuild it from their discovery/activation policy. The provider snapshots skills.container_path at startup and carries it through mount construction, the warm-pool seed, remote metadata, discovery, reconciliation, and synchronization. A VM from a different root is never adopted and is reaped after the duplicate grace period once no live peer owns it. Acquire and release share an AcquireSerializer hold keyed by (user_id, thread_id, skills_root). The serializer does not cover remote IO. burst_limit adds capacity only for the burst policy. The wait policy fails the turn after acquire_timeout. The runtime does not retry the turn automatically. E2B acquisition uses a bounded executor. Waiting calls do not consume the default asyncio executor. The reject policy can evict one warm VM before it returns an error. With memory ownership, replicas limits one Gateway process. Redis ownership shares one <ownership.key_prefix>:e2b-capacity Hash, making the limit (plus a bounded burst) deployment-wide. Lua atomically manages VM and in-flight-create entries; missing or unavailable state fails closed. E2B reservation metadata repairs interrupted creates. Inventory replacement is revision-CAS guarded, and incomplete inventories never remove entries; complete omissions get a grace period. Uncertain cleanup keeps a tombstone slot. Shutdown tracks owned remote operation IDs. Discovery can find a VM from another Gateway. Shutdown closes an unowned discovery client without destroying its VM. Release ends its transition count when the VM enters the warm pool. Local client cleanup does not consume a second slot. A create that returns after shutdown retries one failed kill through a new client. An unconfirmed remote ID stays tracked. reset() uses full shutdown semantics. It destroys tracked active and warm VMs. It wakes capacity waiters. Callers cannot reuse the old provider instance. A background startup pass and periodic reconciliation list provider-tagged remote sandboxes within page/item/time budgets, probe every candidate until a healthy canonical sandbox is found, adopt only through the shared ownership store, and reap duplicates/orphans only after their configured grace/TTL and an atomic del: claim. Lease renewal is independent of reconciliation. Failed canonical adoption clears both its capacity reservation and acquire-intent marker even if a peer takes ownership between the initial claim and bootstrap cleanup. Shutdown kills only IDs whose leases are owned by this provider instance; peer-owned clients are merely closed.
  • Cross-instance ownership store (aio_sandbox/ownership/, #4206): gateway instances sharing a container backend coordinate container ownership through a pluggable lease store, selected by sandbox.ownership.type (memory | redis) and resolved like stream_bridge (factory.py, lazy per-branch import, redis optional extra, DEER_FLOW_SANDBOX_OWNERSHIP_REDIS_URL env escape hatch; a set DEER_FLOW_STREAM_BRIDGE_REDIS_URL implies a multi-instance deployment and infers redis). memory is single-instance only and declares supports_cross_process = False.
    • A lease answers "who reaps this container", not "who may use it". That splits the interface in two: take() transfers ownership on the acquire path (a container is deterministic per user/thread, so consecutive turns legitimately land on different instances — a conditional claim there would strand the thread until the previous lease expired), while claim() succeeds only if the container is unowned or already ours and gates every adopt/reap path. release() never clears a peer's lease.
    • A lease carries a state, and that is what makes the destroy window safe. own: = responsible for this container; del: = tearing it down (claim(..., for_destroy=True)). take() is refused against a del: lease, so a container cannot be re-acquired between a destroy path's claim and its container stop. Without the two states an unconditional take() would silently overwrite the destroyer's claim and the peer's stop would land on a container the new owner had already handed to an agent — i.e. #4206 again. That pairing is what replaced the previous same-host flock guard, which is gone; Redis makes the scope genuinely multi-instance instead of same-host. A destroyer that dies mid-stop leaves a del: marker that lapses with the TTL. On the acquire path a refused take raises SandboxBeingDestroyedError: the reuse/reclaim paths drop the container and cold-start, and the discover path propagates (falling through to create would collide with the not-yet-removed container name).
    • The del: state has to be held for the stop, not just written before it. The two states alone do not make flock redundant — a held lock cannot expire, whereas a lease can, and claim(..., for_destroy=True) writes the marker with the ordinary lease TTL. Nothing else refreshes it: renew() extends only own: and deliberately reports a teardown as LOST, and the destroy paths drop the sandbox from the maps _renew_owned_leases iterates. So a container stop that outlived the TTL let the marker lapse, a peer's take() succeeded against the still-running container, and the stop then landed on the turn that had just been handed it — the exact window del: exists to close, reopened by its own expiry. _held_teardown_lease wraps every del:-marked stop — _destroy_warm_entry, destroy(), and _drop_unhealthy_sandbox — and re-claims the marker every renewal_interval_seconds until the stop returns. _drop_unhealthy_sandbox needs it most: it untracks before claiming (under its expected_info TOCTOU guard), so _renew_owned_leases cannot see the id either. The final release is the heartbeat's own last act, not the caller's — a refresh claim still in flight when the context exits (the store's socket timeout bounds it, but it can be mid-round-trip) would otherwise land after a caller-side release and rewrite del: on a container whose stop already completed, stranding a fresh take() (or rolling back a fresh create) until the TTL. Releasing from inside the heartbeat, after its loop stops, sequences the release strictly after the last refresh, so no claim can follow it; the context join is bounded and, on a genuine wedge, defers the release to that thread rather than clearing the marker itself. This covers a failed stop too (the container is probably still up, and a marker left behind refuses its own thread's take() until the TTL lapses); destroy() still lets the error propagate out of the with — shutdown() logs per sandbox off it. RedisOwnershipStore sets a socket_timeout so no store round trip — and so no heartbeat refresh — can block unbounded, keeping that deferred release finite. This needs no abnormal backend: the schema bounds only renewal_interval_seconds (> 0) and ttl_multiplier (>= 2), so a legal config puts the TTL below a normal container stop. LocalContainerBackend._stop_container now passes a timeout to subprocess.run (_STOP_TIMEOUT_SECONDS) so a wedged daemon cannot block unbounded — that bounds the residual window independently of the ownership layer, for the case where the del: marker lapses mid-stop (a store outage longer than the TTL) and the stop then lands on a container a peer has been handed. A timed-out stop propagates rather than being swallowed like a CalledProcessError: the container is probably still running, so reporting a clean stop would drop the warm entry and leak it. The TTL stays finite on purpose — the heartbeat dies with the process, so a destroyer that crashes mid-stop still releases the container one TTL later instead of marking it undestroyable forever. Raising a separate teardown TTL instead would only be sufficient if it were bounded above every backend's real stop deadline.
    • Fail-closed both directions. Establishment: a sandbox whose ownership cannot be published is never handed out (a just-created container is destroyed rather than leaked as an adoptable orphan) — acquiring raises OwnershipBackendError, matching the stream bridge's fail-hard v1 policy. Reaping: a store that cannot answer is treated as peer-owned, so an outage never turns live peer containers into orphans. Renewal is the deliberate exception: an unanswerable store there means unknown, not lost, so _refresh_ownership keeps the sandbox and retries — failing closed on that path would evict every live sandbox on every instance the moment the store blinked. The TTL still bounds how long a genuinely dead owner holds a lease. Both paths that stop a container they still track — destroy() and _destroy_warm_entry — claim before untracking, so a refused claim cannot leave a container running and untracked. (_drop_unhealthy_sandbox untracks first, under its expected_info TOCTOU guard, then claims before the stop; a refused claim there leaves the container to the next reconcile, which re-adopts it after the grace.)
    • A lease excludes peers, never ourselves — same-process exclusion is the provider's job. claim() and take() both succeed against this instance's own own: lease by design (that is what lets a destroy path claim what it already owns), so del: says nothing to this process's other threads. Every reaper — idle checker, renewal, warm eviction, unhealthy drop — decides outside self._lock, because a store round trip must not be held under the lock that guards every acquire; so each one acts on a decision its own acquire path may already have invalidated. Two guards cover the two directions, and both live in AioSandboxProvider, not the store:
      • Reaping (_reserve_local_teardown / _local_teardown): the reaper marks the id, and every promote path — _reuse_in_process_sandbox, _reclaim_warm_pool_sandbox, _register_discovered_sandbox — refuses a marked id exactly as it refuses a peer's del: (drop and cold-start). The "is this still reapable?" check runs in the same critical section as the mark, passed down as a still_reapable predicate rather than run by the caller beforehand: checking first and marking second is the window, not a narrower version of it. This matters most where the entry deliberately stays visible during the stop — both warm reapers defer their pop so a refused claim cannot lose the container — and where the maps are cleared first (_drop_unhealthy_sandbox), which leaves backend discovery as the open path. On main the mixin's _evict_oldest_warm / _reap_expired_warm popped under the lock, so the deferred pop is what made this reachable.
      • Forgetting (_acquire_epoch): when renew() reports LOST the peer legitimately wins, so here the promote is the thing to detect. _publish_ownership bumps a per-id acquire epoch; _renew_owned_leases and release() snapshot it before the round trip and hand it to _forget_lost_sandbox, which skips the pop if it moved. Object identity is not enough: the reuse path re-publishes ownership while handing out the same tracked AioSandbox, so an identity check sees nothing and the pop closes a client mid-turn.
      • A guard must become visible no later than the transition it guards. The epoch cannot satisfy that for take(): the takeover is durable before take() returns (redis has committed the SET while the reply is in flight), and the epoch can only be written afterwards, so a renewal holding an older LOST walks through the gap, drops the maps, and closes the client the acquire is about to hand back — acquire then returns an id whose get() is None. _publish_ownership therefore publishes an intent mark (_acquire_inflight) under _lock before the round trip; the epoch covers the other half, "an acquire completed since you decided". _forget_lost_sandbox honours the intent mark unconditionally, not only when an epoch is supplied — "no epoch" must not read as "no guard".
      • A reservation must cover the removal, not just the stop. _destroy_warm_entry pops the warm entry itself, inside the reservation. Releasing the reservation when the stop returns and letting the caller pop afterwards leaves a gap where the container is stopped, the entry is still in _warm_pool, and nothing marks it — a reclaim there hands out a dead container. The pop stays deferred relative to the stop (a refused or failed stop keeps the entry), just no longer relative to the reservation.
      • A check taken before a round trip must be retaken after it. _reuse_in_process_sandbox re-verifies both its map entry and the local teardown reservation, _reclaim_warm_pool_sandbox re-checks the reservation, and _register_discovered_sandbox re-checks before installing its client, all after publishing ownership. Before the intent mark is set a renewal's LOST is both current and correct, so the forget can legitimately remove the entry the acquire decided to hand out; independently, a local reaper can reserve an id while reuse is outside _lock for its health/store calls and deliberately leaves the map entry present until its destroy claim succeeds. Falling through re-discovers or cold-starts instead of returning/installing a client for either stale decision. The pre-round-trip checks remain as early-outs that skip backend and store work on an already-doomed entry.
      • Adoption is a promote too. _reconcile_orphans honours the reservation: a container being torn down is untracked and still running, which is exactly the shape that loop adopts, and neither the claim (ours) nor the recovery grace (skipped entirely on memory, where supports_cross_process is False) excludes it.
      • Active and warm are exclusive, and only a promote can violate it. Both register paths pop _warm_pool inside the same locked section that inserts into _sandboxes: a warm entry for an id is stale the moment that id becomes active, and leaving it gives the container two reapers — _reap_expired_warm judges it by the warm timestamp and never consults _last_activity, so it stops a container an agent is using while _sandboxes still hands out its client. Reachable because reconciliation adopts into the warm pool inside the register's publish → track window, and on memory it adopts on sight (_adoptable_after_grace short-circuits when supports_cross_process is False, so an id carrying this process's own lease reads as adoptable). On main the track was a single locked insert with nothing before it, so the window did not exist. A non-destroy claim() is the one case the store does police against its own owner: it refuses to overwrite our own del:, because the stop it marks is already in flight and downgrading the marker would let a take() hand out a container about to die. Enforced in both backends (Lua and Python) so they cannot drift.
    • Renewal is independent of idle_timeout (_start_lease_renewal, own daemon thread; TTL = renewal_interval_seconds × ttl_multiplier). Renewal used to ride on the idle checker, which __init__ only starts when idle_timeout > 0 — so idle_timeout: 0 ("keep warm VMs until shutdown", a documented config) let every lease lapse. Liveness and reaping must not share a switch. Renewal covers warm entries as well as active ones; losing a lease drops the sandbox from this instance's maps without touching the container (_forget_lost_sandbox) — destroying it there would be the very cross-instance kill this store prevents.
      • A warm teardown is the local exception to that forget rule: _destroy_warm_entry deliberately keeps the entry in _warm_pool until the backend stop succeeds, while its own del: marker makes ordinary renew() report LOST. _forget_lost_sandbox therefore honours _local_teardown; otherwise the renewal thread can pop the retained entry mid-stop and a failed stop leaves a running container untracked.
    • renew() distinguishes lapsed from lost (RenewOutcome), and the two must not be collapsed. LAPSED means the lease is simply absent — nobody took it — so _refresh_ownership re-establishes it; LOST means a peer holds it and it is never re-taken. Treating an absent lease as lost meant a Redis restart without persistence (every key gone) evicted every in-flight sandbox on every instance at once.
      • Renewal's fail-open rule covers both store round trips. If renew() returns LAPSED but the follow-up claim() cannot answer, ownership is still unknown rather than lost, so the provider keeps the sandbox and retries. The ordinary _claim_ownership helper remains fail-closed for adopt/reap callers and is intentionally not used for this re-claim.
    • Teardown join budget covers refresh plus release. Redis bounds each ownership operation at five seconds, and context exit can catch the heartbeat in one final refresh before its finally performs the final release. _TEARDOWN_JOIN_TIMEOUT_SECONDS is therefore 12 seconds — greater than both sequential operation bounds — so a normal pair of socket timeouts does not emit the deferred-release warning; a still-running heartbeat continues to own the release safely.
    • An absent lease means the same thing on both paths, and reconciliation must say so too. The LAPSED rule above only covers an owner renewing its own lease; on its own it does not make state loss safe, because reconciliation reads the same absent key as "orphan, adopt". After a Redis flush (restart without persistence, or eviction under maxmemory) every owner is alive and merely pre-renewal-tick, so whichever instance reconciles first would adopt every live container, each real owner's next renewal would report LOST, and it would drop a sandbox mid-turn for the adopter to idle-destroy — #4206 through the back door. _adoptable_after_grace closes it: an untracked container must be seen unowned (owner(), a read-only peek — the atomic claim() is still what actually gates adoption) across a full lease TTL before it can be adopted, tracked per container in _unowned_since. That rebuilds the delay the flush erased — a live owner republishes within one renewal interval, shorter than the TTL by construction (ttl_multiplier >= 2) — while a genuinely crashed owner never republishes, so its containers are still adopted one grace later rather than leaking. A republished lease resets the grace; a pausing-only timer would still expire over a live owner's lease. The grace is skipped when supports_cross_process is False: no peer can hold a lease such a store would show us, so single-instance deployments keep instant orphan cleanup, and a grace could not help a multi-worker gateway on memory anyway (peers are invisible to each other's leases with or without it).
    • The memory store is single-instance only and says so via supports_cross_process = False; the provider logs a warning at startup when the configured store cannot see peers. A multi-worker gateway on memory has no cross-process coordination at all — same contract as stream_bridge's memory backend. This is why the redis inference matters: it reads app_config.stream_bridge and the env var, in the same order the bridge's own resolver does, so any deployment already pointing the bridge at Redis (i.e. every multi-instance one) gets a redis ownership store without extra config.
    • get() stays a pure in-memory lookup and must never call the store (that is blocking filesystem/network IO on the event loop); anchored by tests/blocking_io/test_aio_sandbox_get.py, which injects a deliberately-blocking probe store so the anchor keeps its teeth regardless of the configured backend. Tests: tests/test_sandbox_ownership_store.py (store contract, defined once for both backends — the redis tier is @pytest.mark.integration, uses DEER_FLOW_TEST_REDIS_URL when set, and otherwise self-skips without a reachable Redis. Backend CI provisions Redis, so the merge gate executes the real Lua tier; there is no fake-redis tier because a fake would not execute the Lua exclusions) and tests/test_sandbox_orphan_reconciliation.py (provider behaviour, two providers sharing one store).
• BoxliteProvider (packages/harness/deerflow/community/boxlite/) - BoxLite micro-VM isolation. The boxlite runtime is optional (deerflow-harness[boxlite]) and lazy-imported only when this provider is selected. The provider owns one private asyncio event loop on a daemon thread because BoxLite handles are loop-affine; sync Sandbox calls marshal onto that loop with run_coroutine_threadsafe. Boxes are named deterministically from user_id:thread_id, released into an in-process warm pool after each agent turn, and reclaimed only by the same user/thread. Warm-pool health checks use a short explicit timeout and forward that timeout through both BoxLite exec(timeout=...) and the private-loop .result(timeout) bridge so a hung VM cannot pin the per-thread acquire lock indefinitely. sandbox.replicas caps active + warm VMs per gateway process; if capacity is exhausted, only warm-pool VMs are evicted. sandbox.idle_timeout stops idle warm VMs after the configured seconds. reset() is intentionally a lightweight registry clear for reset_sandbox_provider() and does not close boxes, stop the idle reaper, or close the private loop; full teardown remains shutdown().
• TenkiSandboxProvider (packages/harness/deerflow/community/tenki/) - Tenki cloud microVM isolation. The tenki SDK is optional (deerflow-harness[tenki], importable as tenki_sandbox) and lazy-imported (_import_client) only when this provider is selected. Unlike Boxlite, the SDK is synchronous, so the adapter calls it directly with no event-loop bridge. File transport uses Tenki's native sandbox.fs API (read_text/read_stream/write_stream/mkdir/stat) — binary-safe and streaming, no base64/shell hop; only directory/content search (list_dir/glob/grep) shells out to busybox-portable find/grep, parsed with the shared deerflow.sandbox.search helpers like community/e2b_sandbox. Sandboxes run as the unprivileged tenki user, so DeerFlow's /mnt/user-data prefix is remapped under a writable HOME (_resolve_path) and best-effort sudo-symlinked at bootstrap. Boxes are named deterministically from sha256(user_id:thread_id)[:16] (64-bit, matching E2B; the warm pool is keyed by this id alone with no full-seed fallback), released into an in-process warm pool, and reclaimed only by the same user/thread after a liveness check. A terminal session error (named SDK errors plus builtin ConnectionError/BrokenPipeError/EOFError) routes through _invalidate_sandbox to evict the dead microVM. Cross-process orphan reconciliation is a follow-up (single-process warm pool today).

Shared warm-pool lifecycle: community sandbox providers that keep released sandboxes alive for fast reuse share deerflow.community.warm_pool_lifecycle.WarmPoolLifecycleMixin. The mixin owns the common DEFAULT_IDLE_TIMEOUT=600, IDLE_CHECK_INTERVAL=60, DEFAULT_REPLICAS=3, idle-checker loop, warm-pool expiry, oldest-warm eviction, replica counting, and soft-cap logging. Providers remain responsible for their own active registries, creation/discovery, health checks, and destroy hook (_destroy_warm_entry): AIO destroys SandboxInfo through its backend; Boxlite closes loop-affine BoxliteBox handles; Tenki closes the microVM session (TenkiSandbox.close, which terminates the remote sandbox). AIO keeps active-idle cleanup outside the mixin and delegates only warm-pool expiry to the shared helper.

Virtual Path System:

• Agent sees: /mnt/user-data/{workspace,uploads,outputs}, /mnt/skills
• Physical: backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...; raw skills stay under deer-flow/skills/ and managed integration storage. Unrestricted sandboxes read backend/.deer-flow/skills_view/public/ and backend/.deer-flow/users/{user_id}/skills_view/{custom,legacy,integrations}/; explicit lead Agent policies read backend/.deer-flow/users/{user_id}/threads/{thread_id}/skills_view/{public,custom,legacy,integrations}/.
• Translation: LocalSandboxProvider builds per-thread PathMappings at acquire time. Sandbox-backed readers resolve only /mnt/user-data/... in the tool layer; skills, ACP workspaces, and configured custom mounts stay virtual so the provider mount table remains the single source of acquire-time identity and visibility. Full reads, ranged reads, and read-before-write hashing share this path. tools.py keeps replace_virtual_path() / replace_virtual_paths_in_command() as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept /mnt/user-data/... natively.
• Detection: is_local_sandbox() accepts both sandbox_id == "local" (legacy / no-thread) and sandbox_id.startswith("local:") (per-thread)

Sandbox Tools (in packages/harness/deerflow/sandbox/tools.py):

• Every sandbox tool keeps a model-visible description field for a human-readable progress label, but the field is optional and defaults to an empty string. Tool execution must depend only on its operational arguments; the frontend supplies localized fallback labels when a provider omits description.
• bash - Execute commands with path translation and error handling. For LocalSandbox (host bash), output on POSIX and Windows is captured through bounded pipe-drain threads and stdin is /dev/null; Windows capture decodes with the platform text encoding and applies universal-newline translation, matching the former subprocess.run(..., text=True) behavior for locale-code-page output, Python UTF-8 Mode, CRLF, and bare CR. That translation is Windows-only so the pre-existing POSIX output contract remains byte-decoded without newline rewriting. On POSIX, a backgrounded long-lived process (server &) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (sandbox.bash_command_timeout, default 600s); on timeout the whole POSIX process group or Windows process tree is killed and the agent gets a notice telling it to background long-lived processes. The shared bash tool description scopes host environment detection to LocalSandbox: start with uname -s, follow with sw_vers on Darwin, and read Linux host system files only when the active policy permits them. Local path and file:// rejections provide the same conditional recovery guidance: command-only probes for environment questions, allowed virtual paths otherwise, and no repetition of the rejected path. The description also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See LocalSandbox.execute_command, its platform runners, and bash_tool's docstring.
• ls - Directory listing (tree format, max 2 levels)
• glob - Find files or directories below a root directory with bounded results
• grep - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results
• read_file - Read file contents with optional line range
• write_file - Write/append to files, creates directories; overwrites by default and exposes the append argument in the model-facing schema for end-of-file writes; subject to the read-before-write gate when read_before_write.enabled (see Middleware Chain)
• str_replace - Substring replacement (single or all occurrences); same-path serialization is scoped to (sandbox.id, path) so isolated sandboxes do not contend on identical virtual paths inside one process; subject to the read-before-write gate when read_before_write.enabled (see Middleware Chain)