github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-19 02:56:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(sandbox): prevent AIO subagent session eviction (#5178)

Browse Source 
* fix(sandbox): prevent AIO subagent session eviction

* fix(sandbox): address PR 5178 review issues

* fix(sandbox): handle transient session and metadata failures

* fix(sandbox): fence capacity upgrades and validate reused limits

* docs(sandbox): restore list indentation and trim guidance

* fix(ci): stabilize Buzz persistence test and trim sandbox guidance

---------

Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Onefly 2026-09-17 07:45:19 +08:00 committed by GitHub
parent cda56aa282
commit d8d110c637
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
16 changed files with 1064 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
backend/docs/CONFIGURATION.md
View File

@ -927,6 +927,38 @@ inside the client. This prevents an inherited proxy from returning a misleading
502 for a healthy local sandbox. Externally hosted sandbox FQDNs and public IPs
continue to use the normal environment proxy configuration.
### AIO shell-session capacity
Each concurrently running native subagent uses one persistent AIO shell session.
The semver AIO images from `1.9.3` through `1.11.0` default
`MAX_SHELL_SESSIONS` to 10 and evict the oldest idle session when an eleventh is
created. If `subagent_runtime.max_running` is greater than nine, DeerFlow sets
the container limit to `max_running + 1`; the extra slot leaves room for the lead
agent's shell. This applies to both locally created containers and provisioner
Pods. Lower concurrency keeps the image's own default unchanged.
The separate `bash.exec` API uses its own `AIO_BASH_MAX_SESSIONS` pool rather
than `MAX_SHELL_SESSIONS`. DeerFlow nevertheless creates and closes an explicit
transient bash session around every env-bearing command, so request-scoped
secrets and completed command sessions are not retained.
You may set `sandbox.environment.MAX_SHELL_SESSIONS` explicitly. It must be a
positive integer at least as large as `subagent_runtime.max_running + 1`, or the
provider fails at startup with the conflicting values. The setting is applied
when a sandbox is created. Persisted local containers and provisioner Pods report
their effective value; DeerFlow replaces one whose capacity is below the current
requirement instead of reusing it. Reuse checks also apply when no explicit
override is needed for new containers: a previously configured lower limit must
still fit the current concurrency. The Gateway waits for existing ownership and
the orphan recovery grace before replacing an incompatible sandbox. A create
request that encounters a lower-capacity Pod returns HTTP 409 without deleting
it; a later acquisition can discover and replace it through that same ownership
check.
If a Service survives deletion of its old Pod, a later create repairs the
missing Pod. A failed capacity read other than a Pod-not-found response remains
an error and does not authorize replacement.
### Building a Custom AIO Sandbox Image
`AioSandboxProvider` talks to the sandbox container through the `agent-sandbox` SDK. The Dockerfile for the default `enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest` image is not part of this repository; DeerFlow treats that image as an upstream AIO sandbox runtime.

163
backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox.py
View File

@ -197,6 +197,25 @@ class AioSandbox(Sandbox):
exit_code = getattr(data, "exit_code", None) if data else None
return output, exit_code
@staticmethod
def _is_missing_shell_session_error(error: ApiError) -> bool:
body = error.body
if error.status_code != 404 or not isinstance(body, dict):
return False
message = body.get("message")
return isinstance(message, str) and "session not found" in message.casefold()
@staticmethod
def _cleanup_bash_session_best_effort(client, session_id: str) -> None:
try:
client.bash.close_session(session_id)
except Exception as cleanup_error:
logger.warning(
"Failed to release transient bash session %s: %s",
session_id,
cleanup_error,
)
def _create_shell_session(self, client) -> str:
session_id = str(uuid.uuid4())
client.shell.create_session(id=session_id)
@ -289,12 +308,24 @@ class AioSandbox(Sandbox):
raise RuntimeError("sandbox client is closed")
if scoped.session_id is None:
scoped.session_id = self._create_shell_session(client)
output, exit_code = self._exec_shell(
client,
command,
session_id=scoped.session_id,
)
if output and _ERROR_OBSERVATION_SIGNATURE in output:
try:
output, exit_code = self._exec_shell(
client,
command,
session_id=scoped.session_id,
)
except ApiError as error:
if not self._is_missing_shell_session_error(error):
raise
logger.warning("Execution-scoped sandbox shell session is missing; recreating it once")
scoped.session_id = None
output, exit_code, scoped.session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=None,
context="execution scope after missing session",
)
if scoped.session_id is not None and output and _ERROR_OBSERVATION_SIGNATURE in output:
logger.warning("ErrorObservation detected in sandbox output for execution scope; rotating session")
output, exit_code, scoped.session_id = self._rotate_and_retry_shell(
client,
@ -372,11 +403,11 @@ class AioSandbox(Sandbox):
command: The command to execute.
env: Optional per-call environment variables (request-scoped secrets,
issue #3861). When provided, the command runs via the ``bash.exec``
API (which supports per-command env) on a fresh auto-created session
so the secrets are scoped to this single command and never persist;
secret values travel in the structured ``env`` field, never in the
command string. When ``None`` the legacy persistent-shell path runs
unchanged.
API (which supports per-command env) on a fresh explicitly released
session, so the secrets are scoped to this single command and never
persist; secret values travel in the structured ``env`` field, never
in the command string. When ``None`` the legacy persistent-shell path
runs unchanged.
timeout: Optional per-call timeout. The current sandbox SDK does not
expose a command-level timeout distinct from its client/request
timeout, so DeerFlow keeps using the backend's default here.
@ -403,13 +434,28 @@ class AioSandbox(Sandbox):
# target it again. A failed replacement is cleaned up and
# the next call starts another explicit session.
self._recovery_session_id = self._create_shell_session(client)
output, exit_code = self._exec_shell(
client,
command,
session_id=self._recovery_session_id,
)
recovered_missing_session = False
try:
output, exit_code = self._exec_shell(
client,
command,
session_id=self._recovery_session_id,
)
except ApiError as error:
if not self._is_missing_shell_session_error(error):
raise
logger.warning("Default sandbox shell session is missing; recreating it once")
self._default_shell_corrupted = True
self._recovery_session_id = None
recovered_missing_session = True
output, exit_code, self._recovery_session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=None,
context="default shell after missing session",
)
if output and _ERROR_OBSERVATION_SIGNATURE in output:
if not recovered_missing_session and output and _ERROR_OBSERVATION_SIGNATURE in output:
self._default_shell_corrupted = True
logger.warning("ErrorObservation detected in sandbox output, retrying on a fresh session")
output, exit_code, self._recovery_session_id = self._rotate_and_retry_shell(
@ -429,10 +475,11 @@ class AioSandbox(Sandbox):
The persistent-shell ``shell.exec_command`` API has no env parameter, so
injected commands use the ``bash.exec`` API which accepts per-command env.
Each call lets the sandbox auto-create a fresh session (no ``session_id``),
so injected request-scoped secrets are scoped to this command and never
persist across calls. Secret values travel in the structured ``env`` field,
never in the command string.
Each call creates an explicit transient session and closes it after the
command, so injected request-scoped secrets are scoped to this command,
never persist across calls, and do not consume the server's session
capacity after completion. Secret values travel in the structured
``env`` field, never in the command string.
Trade-off of the fresh-session choice: consecutive env-bearing bash calls
within the same skill do not share session state (cwd, sourced venv,
@ -463,36 +510,52 @@ class AioSandbox(Sandbox):
return output
def _run_bash_exec(self, command: str, env: dict[str, str]) -> str:
"""Single bash.exec invocation with injected env (one fresh session)."""
"""Single bash.exec invocation in an explicitly released fresh session."""
with self._lock:
try:
result = self._client.bash.exec(
command=command,
env=env,
hard_timeout=self._DEFAULT_HARD_TIMEOUT,
)
data = result.data if result else None
stdout = (data.stdout or "") if data else ""
stderr = (data.stderr or "") if data else ""
exit_code = getattr(data, "exit_code", None) if data else None
output = stdout
if stderr:
output += f"\nStd Error:\n{stderr}" if output else stderr
if exit_code not in (0, None):
# Mirror LocalSandbox: keep the actual shell status in the
# output text (acceptance-checklist evidence).
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
return output if output else "(no output)"
except ApiError as e:
if e.status_code == 404:
self._bash_exec_unsupported = True
logger.error("Sandbox %s does not support bash.exec (/v1/bash/exec returned 404); env-bearing commands are unavailable until the sandbox image is upgraded to all-in-one-sandbox >= 1.9.3", self.id)
return _BASH_EXEC_UNSUPPORTED_ERROR
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
return f"Error: {e}"
except Exception as e:
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
return f"Error: {e}"
for attempt in range(2):
session_id = str(uuid.uuid4())
session_created = False
try:
self._client.bash.create_session(session_id=session_id)
session_created = True
result = self._client.bash.exec(
command=command,
session_id=session_id,
env=env,
hard_timeout=self._DEFAULT_HARD_TIMEOUT,
)
data = result.data if result else None
stdout = (data.stdout or "") if data else ""
stderr = (data.stderr or "") if data else ""
exit_code = getattr(data, "exit_code", None) if data else None
output = stdout
if stderr:
output += f"\nStd Error:\n{stderr}" if output else stderr
if exit_code not in (0, None):
# Mirror LocalSandbox: keep the actual shell status in the
# output text (acceptance-checklist evidence).
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
return output if output else "(no output)"
except ApiError as e:
if self._is_missing_shell_session_error(e):
if attempt == 0:
logger.warning("Transient bash.exec session disappeared; retrying once")
continue
logger.error("Failed to execute command with injected env: bash.exec session disappeared after retry")
return "Error: bash.exec session disappeared after retry"
if e.status_code == 404:
self._bash_exec_unsupported = True
logger.error("Sandbox %s does not support bash.exec (/v1/bash/exec returned 404); env-bearing commands are unavailable until the sandbox image is upgraded to all-in-one-sandbox >= 1.9.3", self.id)
return _BASH_EXEC_UNSUPPORTED_ERROR
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
return f"Error: {e}"
except Exception as e:
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
return f"Error: {e}"
finally:
if session_created:
self._cleanup_bash_session_best_effort(self._client, session_id)
return "Error: bash.exec session disappeared after retry"
def read_file(
self,

33
backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py
View File

@ -70,6 +70,11 @@ DEFAULT_IMAGE = "enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in
DEFAULT_PORT = 8080
DEFAULT_CONTAINER_PREFIX = "deer-flow-sandbox"
IDLE_CHECK_INTERVAL = _SHARED_IDLE_CHECK_INTERVAL
# The supported semver AIO images currently default to ten shell sessions.
# Leave lower-concurrency deployments on the image default; only override it
# when DeerFlow's configured execution capacity cannot fit.
_AIO_DEFAULT_MAX_SHELL_SESSIONS = 10
_SHELL_SESSION_HEADROOM = 1
class SandboxBeingDestroyedError(RuntimeError):
@ -260,7 +265,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
raise RuntimeError("sandbox.network restricted modes are currently supported only by the local Docker AIO backend")
logger.info(f"Using remote sandbox backend with provisioner at {provisioner_url}")
api_key = self._config.get("provisioner_api_key", "")
return RemoteSandboxBackend(provisioner_url=provisioner_url, api_key=api_key)
return RemoteSandboxBackend(
provisioner_url=provisioner_url,
api_key=api_key,
max_shell_sessions=self._config.get("max_shell_sessions"),
required_shell_sessions=self._config.get("required_shell_sessions", 0),
)
logger.info("Using local container sandbox backend")
return LocalContainerBackend(
@ -270,6 +280,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
config_mounts=self._config["mounts"],
environment=self._config["environment"],
network_config=self._config["network"],
required_shell_sessions=self._config.get("required_shell_sessions", 0),
)
# ── Configuration ────────────────────────────────────────────────────
@ -289,6 +300,22 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
if not isinstance(configured_skills_path, str):
configured_skills_path = DEFAULT_SKILLS_CONTAINER_PATH
environment = self._resolve_env_vars(sandbox_config.environment or {})
max_running_subagents = int(getattr(getattr(config, "subagent_runtime", None), "max_running", 3))
required_shell_sessions = max_running_subagents + _SHELL_SESSION_HEADROOM
configured_shell_sessions = environment.get("MAX_SHELL_SESSIONS")
if configured_shell_sessions is None:
max_shell_sessions = required_shell_sessions if required_shell_sessions > _AIO_DEFAULT_MAX_SHELL_SESSIONS else None
if max_shell_sessions is not None:
environment["MAX_SHELL_SESSIONS"] = str(max_shell_sessions)
else:
try:
max_shell_sessions = int(configured_shell_sessions)
except (TypeError, ValueError) as exc:
raise ValueError("sandbox.environment.MAX_SHELL_SESSIONS must be a positive integer") from exc
if max_shell_sessions < required_shell_sessions:
raise ValueError(f"sandbox.environment.MAX_SHELL_SESSIONS must be at least subagent_runtime.max_running + {_SHELL_SESSION_HEADROOM} ({required_shell_sessions} for the current configuration)")
return {
"image": sandbox_config.image or DEFAULT_IMAGE,
"port": sandbox_config.port or DEFAULT_PORT,
@ -297,7 +324,9 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
"replicas": replicas if replicas is not None else DEFAULT_REPLICAS,
"mounts": sandbox_config.mounts or [],
"thread_data_mounts": getattr(sandbox_config, "thread_data_mounts", None),
"environment": self._resolve_env_vars(sandbox_config.environment or {}),
"environment": environment,
"max_shell_sessions": max_shell_sessions,
"required_shell_sessions": required_shell_sessions,
"network": sandbox_config.network.model_dump(),
"ownership": getattr(sandbox_config, "ownership", None),
# A redis stream bridge means the deployment is multi-instance, which

40
backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py
View File

@ -31,6 +31,8 @@ from .sandbox_info import SandboxInfo
logger = logging.getLogger(__name__)
_AIO_DEFAULT_MAX_SHELL_SESSIONS = 10
class _ExistingRestrictedSandbox(RuntimeError):
def __init__(self, info: SandboxInfo):
@ -46,6 +48,7 @@ class _ContainerInspection:
image: str
networks: frozenset[str]
relay_token: str | None = None
max_shell_sessions: int | None = None
@dataclass(frozen=True)
@ -513,6 +516,7 @@ class LocalContainerBackend(SandboxBackend):
config_mounts: list,
environment: dict[str, str],
network_config: dict[str, object] | None = None,
required_shell_sessions: int = 0,
):
"""Initialize the local container backend.
@ -522,12 +526,14 @@ class LocalContainerBackend(SandboxBackend):
container_prefix: Prefix for container names (e.g., "deer-flow-sandbox").
config_mounts: Volume mount configurations from config (list of VolumeMountConfig).
environment: Environment variables to inject into containers.
required_shell_sessions: Minimum usable capacity, independent of image environment overrides.
"""
self._image = image
self._base_port = base_port
self._container_prefix = container_prefix
self._config_mounts = config_mounts
self._environment = environment
self._required_shell_sessions = required_shell_sessions
self._network_config = network_config or {"mode": "open"}
self._network_mode = str(self._network_config.get("mode", "open"))
self._allow_synthetic_dns = False
@ -572,6 +578,18 @@ class LocalContainerBackend(SandboxBackend):
"deerflow.network_mode": self._network_mode,
}
def _has_compatible_shell_capacity(self, inspection: _ContainerInspection) -> bool:
"""Check both the runtime minimum and any explicit environment override."""
configured = self._environment.get("MAX_SHELL_SESSIONS")
required = self._required_shell_sessions
try:
if configured is not None:
required = max(required, int(configured))
except (TypeError, ValueError):
return False
actual = inspection.max_shell_sessions if inspection.max_shell_sessions is not None else _AIO_DEFAULT_MAX_SHELL_SESSIONS
return actual >= required
def _network_policy_digest(self) -> str:
allow_domains = self._network_config.get("allow_domains", [])
canonical_domains = sorted({value for value in allow_domains if isinstance(value, str)}) if isinstance(allow_domains, list) else []
@ -1223,6 +1241,14 @@ class LocalContainerBackend(SandboxBackend):
created_at=created_at,
requires_replacement=True,
)
if not self._has_compatible_shell_capacity(sandbox_inspection):
return SandboxInfo(
sandbox_id=sandbox_id,
sandbox_url="",
container_name=container_name,
created_at=created_at,
requires_replacement=True,
)
if self._network_mode != "open":
proxy_name, _ = self._resource_names(sandbox_id)
@ -1375,7 +1401,7 @@ class LocalContainerBackend(SandboxBackend):
continue
created_at, host_port = data.created_at, data.host_port
request_headers: dict[str, str] = {}
requires_replacement = persisted_mode != self._network_mode
requires_replacement = persisted_mode != self._network_mode or not self._has_compatible_shell_capacity(data)
if not requires_replacement and self._network_mode != "open":
proxy_name, _ = self._resource_names(sandbox_id)
proxy_data = inspections.get(proxy_name)
@ -1555,6 +1581,17 @@ class LocalContainerBackend(SandboxBackend):
host_port = _extract_host_port(entry, 8080)
config = entry.get("Config") or {}
network_settings = entry.get("NetworkSettings") or {}
max_shell_sessions: int | None = None
configured_shell_sessions = _extract_container_environment(config, "MAX_SHELL_SESSIONS")
if configured_shell_sessions is not None:
try:
parsed_shell_sessions = int(configured_shell_sessions)
if parsed_shell_sessions > 0:
max_shell_sessions = parsed_shell_sessions
else:
max_shell_sessions = 0
except ValueError:
max_shell_sessions = 0
out[name] = _ContainerInspection(
created_at=created_at,
host_port=host_port,
@ -1562,6 +1599,7 @@ class LocalContainerBackend(SandboxBackend):
image=str(config.get("Image") or ""),
networks=frozenset(str(value) for value in (network_settings.get("Networks") or {})),
relay_token=_extract_container_environment(config, RELAY_TOKEN_ENV),
max_shell_sessions=max_shell_sessions,
)
return out

39
backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py
View File

@ -54,6 +54,7 @@ _RESERVED_SANDBOX_MOUNT_PATHS = (
_LARK_CLI_RUNTIME_CONTAINER_PATH = "/mnt/integrations/lark-cli/runtime"
_LARK_CLI_CONFIG_CONTAINER_PATH = "/mnt/integrations/lark-cli/config"
_LARK_CLI_DATA_CONTAINER_PATH = "/mnt/integrations/lark-cli/data"
_AIO_DEFAULT_MAX_SHELL_SESSIONS = 10
def _normalize_skills_container_path(container_path: str) -> str:
@ -141,7 +142,14 @@ class RemoteSandboxBackend(SandboxBackend):
provisioner_api_key: $PROVISIONER_API_KEY
"""
def __init__(self, provisioner_url: str, api_key: str = ""):
def __init__(
self,
provisioner_url: str,
api_key: str = "",
max_shell_sessions: int | None = None,
*,
required_shell_sessions: int = 0,
):
"""Initialize with the provisioner service URL and optional API key.
Args:
@ -149,9 +157,14 @@ class RemoteSandboxBackend(SandboxBackend):
(e.g., ``http://provisioner:8002``).
api_key: Value sent as ``X-API-Key`` header on every request.
Leave empty to send no authentication header.
max_shell_sessions: Optional AIO shell-session capacity forwarded
to each provisioned sandbox Pod.
required_shell_sessions: Minimum usable capacity, even when new Pods use the image default.
"""
self._provisioner_url = provisioner_url.rstrip("/")
self._api_key = api_key
self._max_shell_sessions = max_shell_sessions
self._required_shell_sessions = max(required_shell_sessions, max_shell_sessions or 0)
@property
def provisioner_url(self) -> str:
@ -160,6 +173,15 @@ class RemoteSandboxBackend(SandboxBackend):
def _auth_headers(self) -> dict[str, str]:
return {"X-API-Key": self._api_key} if self._api_key else {}
def _requires_shell_capacity_replacement(self, payload: dict[str, object]) -> bool:
if self._required_shell_sessions == 0:
return False
reported = payload.get("max_shell_sessions", _AIO_DEFAULT_MAX_SHELL_SESSIONS)
try:
return int(reported) < self._required_shell_sessions
except (TypeError, ValueError):
return True
# ── SandboxBackend interface ──────────────────────────────────────────
def create(
@ -242,7 +264,13 @@ class RemoteSandboxBackend(SandboxBackend):
sandbox_id = sandbox.get("sandbox_id")
sandbox_url = sandbox.get("sandbox_url")
if isinstance(sandbox_id, str) and sandbox_id and isinstance(sandbox_url, str) and sandbox_url:
infos.append(SandboxInfo(sandbox_id=sandbox_id, sandbox_url=sandbox_url))
infos.append(
SandboxInfo(
sandbox_id=sandbox_id,
sandbox_url=sandbox_url,
requires_replacement=self._requires_shell_capacity_replacement(sandbox),
)
)
logger.info("Provisioner list_running: %d sandbox(es) found", len(infos))
return infos
@ -274,6 +302,8 @@ class RemoteSandboxBackend(SandboxBackend):
"provision_lark_cli_runtime": provision_lark_cli_runtime,
"provision_lark_cli_broker": provision_lark_cli_broker,
}
if self._max_shell_sessions is not None:
payload["max_shell_sessions"] = self._max_shell_sessions
provisioner_extra_mounts = _provisioner_extra_mounts_payload(
extra_mounts,
skills_container_path=normalized_skills_container_path,
@ -291,6 +321,10 @@ class RemoteSandboxBackend(SandboxBackend):
)
resp.raise_for_status()
data = resp.json()
if self._max_shell_sessions is not None and "max_shell_sessions" not in data:
raise RuntimeError("Provisioner did not report max_shell_sessions; Gateway/provisioner version skew prevents shell-capacity validation")
if self._requires_shell_capacity_replacement(data):
raise RuntimeError(f"Provisioner returned sandbox {sandbox_id} with insufficient shell-session capacity")
logger.info(f"Provisioner created sandbox {sandbox_id}: sandbox_url={data['sandbox_url']}")
return SandboxInfo(
sandbox_id=sandbox_id,
@ -349,6 +383,7 @@ class RemoteSandboxBackend(SandboxBackend):
return SandboxInfo(
sandbox_id=sandbox_id,
sandbox_url=data["sandbox_url"],
requires_replacement=self._requires_shell_capacity_replacement(data),
)
except requests.RequestException as exc:
logger.debug(f"Provisioner discover failed for {sandbox_id}: {exc}")

4
backend/packages/harness/deerflow/sandbox/AGENTS.md
View File

@ -3,13 +3,13 @@
**Interface**: `Sandbox`: `execute_command(command, env=None)`, additive `execute_command_in_scope(..., scope_id=...)` / `release_command_scope(scope_id)`, `read_file`, `write_file`, `list_dir`, `glob`, `grep`. Scoped hooks default to pass-through without server-side sessions, preserving third-party subclasses. `grep` accepts a text file or directory tree. Per-call `env` injects secrets: `LocalSandbox` merges into the subprocess environment; `AioSandbox` uses fresh `bash.exec(env=...)` sessions. `list_dir`: missing path → `FileNotFoundError`; command/client failure → `OSError`, never `[]` (`ls_tool`: `(empty)`). Remote `glob`/`grep` share it via `sandbox/remote_search.py`: missing root → `FileNotFoundError`, failed search → `OSError`; only a genuine no-match returns `[]`. The parser takes the command's `limit` and reports `truncated` when output passed it, which providers return after Python-side filtering; tools call an empty truncated result incomplete. Remote `grep(glob=...)` scopes like `glob()` (root-relative `path_matches`), never by basename alone. Remotes use `sandbox/remote_list_dir.py`: capture `find`'s status, not `| head`'s (`sh -lc` lacks `pipefail`); missing binary → `OSError`; truncation SIGPIPE → success.
**Provider Pattern**: `SandboxProvider` exposes `acquire`, `acquire_async`, `get`, `release`. Async agent/tool paths use async hooks to keep Docker creation, discovery, cross-process locking, readiness polling, and release off-loop. Set `supports_agent_skill_isolation=True` only when the whole tool surface enforces explicit lead Agent policy: bind mounts use prepared thread roots; upload providers implement `sync_agent_skills`. Host-backed providers report false if an enabled shell bypasses path mappings. Under explicit policy, middleware rejects unsupported providers before acquire.
**Shared components** (RFC #4741): remote IDs use `derive_sandbox_scope_token` (`sandbox/identity.py`); preserve its keyword-only SHA-256/16-hex contract to avoid orphaning containers. `AcquireSerializer` (`sandbox/acquire_serialization.py`) serializes selected acquire/release transitions with a bounded, refcounted per-key `threading.Lock` table and dedicated bounded executor (no event-loop/default-executor blocking). Workers own cancellation cleanup without waiting for cancelled tasks to resume; provider `shutdown()`/`reset()` calls idempotent `close()`. Keys: AIO `(user_id, thread_id)`, E2B `(user_id, thread_id, skills_root)`, BoxLite/Tenki/OpenSandbox derived id. Random-UUID `thread_id=None` acquires bypass serialization.
**Execution leases** (`sandbox/lease.py`, #5128): cross-instance ownership decides which Gateway may reap a container; process-local `SandboxLeaseManager` tracks concurrent lead, subagent, Gateway-request, and channel-upload users of one client. Runs get ephemeral owners, persisted sandboxes are retained idempotently, and the last holder performs any pending `SandboxProvider.release`. Outer lifecycle fences repeat idempotent release after their complete graph/tool/request batch drains; per-tool terminal `Command` wrappers never release because sibling handlers may still run. Fork-restored children and upload syncs use non-releasing holders: they fence the client and own scope cleanup without themselves requesting a park; an earlier normal-owner request waits for them, and a missing fork client is replaced by a normal owner. Persisted lookup plus retention is serialized per `(user_id, thread_id)`; stale bindings fall through to acquire, and a post-acquire lookup miss rolls back before raising. Provider I/O does not hold the metadata lock. Repeated cancellation cannot interrupt acquire/rollback/release reconciliation or let a `to_thread` sandbox operation outlive its enclosing execution/request holder; failures are logged without replacing the original cancellation. Lease/scope context IDs are server-owned: Gateway and worker scrub caller values; only the internal subagent path assigns a task ID. Managers are registered by provider object identity, not hash/equality, so unhashable custom providers remain valid. Subagent owners also serve as `sandbox_command_scope_id`: AIO gives each scope one ordered persistent shell session, replaces it after `ErrorObservation`, and cleans it on lease release. Registry identity is revalidated after every scope-lock wait, preventing queued commands from resurrecting released sessions. Env-bearing commands use fresh `bash.exec` sessions so secrets do not persist.
**Execution leases** (`sandbox/lease.py`, #5128): cross-instance ownership decides which Gateway may reap a container; process-local `SandboxLeaseManager` tracks concurrent lead, subagent, Gateway-request, and channel-upload users of one client. Runs get ephemeral owners, persisted sandboxes are retained idempotently, and the last holder performs any pending `SandboxProvider.release`. Outer lifecycle fences repeat idempotent release after their complete graph/tool/request batch drains; per-tool terminal `Command` wrappers never release because sibling handlers may still run. Fork-restored children and upload syncs use non-releasing holders: they fence the client and own scope cleanup without themselves requesting a park; an earlier normal-owner request waits for them, and a missing fork client is replaced by a normal owner. Persisted lookup plus retention is serialized per `(user_id, thread_id)`; stale bindings fall through to acquire, and a post-acquire lookup miss rolls back before raising. Provider I/O does not hold the metadata lock. Repeated cancellation cannot interrupt acquire/rollback/release reconciliation or let a `to_thread` sandbox operation outlive its enclosing execution/request holder; failures are logged without replacing the original cancellation. Lease/scope context IDs are server-owned: Gateway and worker scrub caller values; only the internal subagent path assigns a task ID. Managers are registered by provider object identity, not hash/equality, so unhashable custom providers remain valid. Subagent owners also serve as `sandbox_command_scope_id`: AIO gives each scope one ordered persistent shell session, replaces it after `ErrorObservation` or a missing-session 404, and cleans it on lease release. Registry identity is revalidated after every scope-lock wait, preventing queued commands from resurrecting released sessions. Env-bearing commands use fresh `bash.exec` sessions so secrets do not persist.
**Authorization gate** (`sandbox:execute`, RFC #4063 Phase 3): every sandbox-backed tool call passes through the gate in `deerflow/authz/sandbox_authz.py` - a binary `authorize(principal, "sandbox", "execute", target="*")` check before either reusing a persisted sandbox id or calling `provider.acquire`. Rechecking reuse is required because authorization config and user roles can change while the sandbox remains cached. Sync tool invocations call `authorize_sandbox_execution`; async tool invocations await `authorize_sandbox_execution_async` exactly once. A task-local `ContextVar` scopes that single decision across the complete composed tool invocation, including `ReadBeforeWriteMiddleware`'s pre-write inspection, tool body, and post-read mark; the value is copied into `asyncio.to_thread` workers. Authorization denial is converted to the normal error `ToolMessage` at the composed middleware boundary and is explicitly excluded from the gate's generic fail-open handlers. Async config loading and provider class discovery/import are offloaded before `aauthorize()` so reused sandbox calls do not hash config files or import custom modules on the event loop; provider construction remains on the running event loop because async providers may initialize loop-affine clients. The gate lives at the single tool initialization entry point (`ensure_sandbox_initialized` / `ensure_sandbox_initialized_async` in `tools.py`), while `SandboxMiddleware.before_agent` / `abefore_agent` apply the matching sync/async check to eager acquisition. Deny raises `SandboxAuthorizationError` (`sandbox/exceptions.py`), which propagates out of ordinary tool execution as a friendly error `ToolMessage` ("sandbox execution is not permitted for your role") - the eager path catches it and skips acquisition instead, deferring the deny to the first sandbox-touching tool call so both paths share the same semantics. Provider errors (authorization calls and provider resolution) follow `authorization.fail_closed` / `fail_open`; no readable `config.yaml` or `authorization.enabled: false` makes the gate a no-op (`safe_app_config` tolerates missing config). Gateway upload/artifact sync calls `try_acquire_sandbox_for_request` (`app/gateway/authz.py`), which gates, returns a request lease, and skips sync on deny while preserving the primary operation. Callers release after their last sandbox operation; artifacts request normal parking, uploads do not. Tests: `tests/test_sandbox_authorization.py` and `tests/blocking_io/test_sandbox_authorization.py`.
**Environment policy** (`sandbox/env_policy.py`): `execute_command` no longer inherits the full `os.environ`. `build_sandbox_env()` scrubs secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. `OPENAI_API_KEY`) never leak into skill subprocesses. Benign vars (`PATH`, `HOME`, `LANG`, `VIRTUAL_ENV`, ...) are preserved.
**Implementations**:
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-user/thread `LocalSandbox` (id `local:{user_id}:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Shared runs use category mappings; a policy-scoped run replaces them with one `/mnt/skills` root mapping to the coherent thread view, so structured file tools resolve through one managed boundary. This is not a host filesystem security boundary: an enabled host `bash` subprocess can use canonical paths without `PathMapping`, so `supports_agent_skill_isolation` is dynamic and explicit Agent policies fail closed while host bash is enabled. Host-to-virtual output masking scans dynamic per-user/per-thread roots directly instead of compiling path-specific regexes, so evicted thread IDs do not remain in Python's global regex caches; a separate 256-entry root cache prevents repeated `realpath()` walks for every glob/grep match while bounding dynamic-path retention, and only the small process-stable skill/integration source set uses a bounded compiled cache. The shared `path_patterns.py` tail stops at `:`, so `$PATH`-style lists mask every entry; a mount symlink resolving outside all mounts keeps its mount path. On Windows, Git Bash/MSYS argument-conversion exclusions are limited to safe non-root virtual path prefixes; do not restore a blanket conversion disable, because host-native CLI launchers need normal MSYS path conversion for their own installation paths.
PowerShell fallback pairs guarded console UTF-8 setters with explicit UTF-8 pipe decoding; cmd/MSYS retain locale decoding. Encoding regressions include real-pipe probes and Windows-only PowerShell roundtrips with and without a console.
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). Text appends use the AIO file API's native append mode rather than a client-side read-modify-write, so a failed pre-read cannot turn an append into an overwrite. `reset()` closes the per-instance acquire serializer so replacing the singleton cannot retain its executor workers; full remote sandbox teardown remains `shutdown()`. `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner backends=False), while the optional `sandbox.thread_data_mounts` boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it `true` skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. An explicit Agent policy uses four thread projection category mounts and a distinct deterministic sandbox identity, preventing reuse of an older container created with shared mounts. `skills.container_path` is a provider-startup snapshot shared by mount construction, sandbox identity, the remote Gateway request, and provisioner validation; custom roots are identity-scoped so a container or Pod created for one destination cannot be reused after the root changes. The Gateway and provisioner independently require one canonical absolute root that does not overlap reserved platform mounts, and both derive the four category allowlist entries from that root. The provisioner accepts all four category overrides; when all are present it suppresses the default hostPath or skills-PVC mount. With `USERDATA_PVC_NAME`, the thread projection categories use subpaths on that shared data PVC. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support.
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker isolation. Reuse enforces minimum persisted shell capacity even without image-env overrides. Upgrades require ownership-fenced teardown; provisioner create returns 409 for undersized Pods without deleting them. Acquire/reuse checks active-cache and warm-pool entries with the backend, dropping definitively dead containers from all in-process maps before rediscovery/creation. Health-check failures mean unknown, not dead; local discovery skips unverifiable containers and tries creation. Acquire/reclaim publishes ownership; `_renew_owned_leases` refreshes it on a dedicated thread. Text appends use native AIO append, never read-modify-write that could overwrite after a failed read. `reset()` closes the acquire serializer and its workers; `shutdown()` performs full remote teardown. `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner=False); `sandbox.thread_data_mounts` overrides this for deployments sharing thread user-data directories. `true` skips upload acquire/sync; a false positive makes uploads unavailable. Explicit Agent policies use four thread projection category mounts and a distinct deterministic identity, excluding older shared-mount containers. The startup snapshot of `skills.container_path` drives mounts, identity, remote requests, and provisioner validation; root changes prevent container/Pod reuse. Gateway and provisioner independently require a canonical absolute root outside reserved platform mounts and derive four category allowlist entries from it. All four category overrides suppress the provisioner's default hostPath/skills-PVC mount; with `USERDATA_PVC_NAME`, categories use subpaths on that shared PVC. Readiness probes and `agent_sandbox` clients set `trust_env=False` for direct control-plane destinations (loopback/private IPs, single-label cluster hosts, Docker/Podman internal hostnames); external FQDNs and public IPs retain environment proxies.
- `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation.
New unrestricted sandboxes receive a one-shot upload from the enabled-only
public, custom, legacy, and managed integration projections. For a thread

28
backend/tests/blocking_io/test_buzz_channel_seen_events.py
View File

@ -151,7 +151,8 @@ async def test_abandoned_relay_records_are_drained_by_retried_stop_or_restart(tm
await abandoned_relay
assert await seen_events.aseen(_CHANNEL_ID, "late-event")
await asyncio.sleep(0.05)
assert seen_events._flush_handle is None
assert seen_events._flush_task is None
stopped_view = BuzzSeenEventStore(path)
assert not await stopped_view.aseen(_CHANNEL_ID, "late-event")
@ -160,15 +161,28 @@ async def test_abandoned_relay_records_are_drained_by_retried_stop_or_restart(tm
assert await retried_stop_view.aseen(_CHANNEL_ID, "late-event")
await seen_events.arecord(_CHANNEL_ID, "restart-event")
await asyncio.sleep(0.05)
assert seen_events._flush_handle is None
assert seen_events._flush_task is None
still_stopped_view = BuzzSeenEventStore(path)
assert not await still_stopped_view.aseen(_CHANNEL_ID, "restart-event")
flush_completed = asyncio.Event()
original_flush_once = seen_events._flush_once
async def tracked_flush_once() -> bool:
saved = await original_flush_once()
flush_completed.set()
return saved
monkeypatch.setattr(seen_events, "_flush_once", tracked_flush_once)
monkeypatch.setattr(buzz_nostr, "parse_private_key", lambda _value: buzz_nostr.NostrKeys(secret=b"", pubkey_hex=_BOT_PUBLIC))
channel._spawn_connection = lambda: None
await channel.start()
await asyncio.sleep(0.05)
restarted_view = BuzzSeenEventStore(path)
assert await restarted_view.aseen(_CHANNEL_ID, "restart-event")
await channel.stop()
try:
# Wait for the automatically resumed write, not a fixed disk/executor
# latency. Calling aflush() here would hide a broken resume() path.
await asyncio.wait_for(flush_completed.wait(), timeout=5)
restarted_view = BuzzSeenEventStore(path)
assert await restarted_view.aseen(_CHANNEL_ID, "restart-event")
finally:
await channel.stop()

186
backend/tests/test_aio_sandbox.py
View File

@ -294,6 +294,37 @@ class TestErrorObservationRetry:
assert result == "all good"
assert call_count == 1
def test_missing_recovery_session_is_recreated_once_and_reused(self, sandbox):
"""An evicted lead recovery session must not poison every later call."""
from agent_sandbox.core.api_error import ApiError
created_ids: list[str] = []
exec_ids: list[str | None] = []
sandbox._default_shell_corrupted = True
sandbox._recovery_session_id = "evicted-lead-session"
def create_session(id, **kwargs):
created_ids.append(id)
return SimpleNamespace(data=SimpleNamespace(session_id=id))
def exec_command(command, **kwargs):
session_id = kwargs.get("id")
exec_ids.append(session_id)
if session_id == "evicted-lead-session":
raise ApiError(
status_code=404,
body={"message": f"Shell session not found: {session_id}"},
)
return SimpleNamespace(data=SimpleNamespace(output="healthy", exit_code=0))
sandbox._client.shell.create_session = create_session
sandbox._client.shell.exec_command = exec_command
assert sandbox.execute_command("first") == "healthy"
assert sandbox.execute_command("second") == "healthy"
assert len(created_ids) == 1
assert exec_ids == ["evicted-lead-session", created_ids[0], created_ids[0]]
class TestScopedShellSessions:
"""Concurrent subagents use independent persistent shell sessions (#5128)."""
@ -410,6 +441,98 @@ class TestScopedShellSessions:
sandbox.release_command_scope("subagent-a")
assert cleaned_ids == created_ids
def test_missing_scoped_session_is_recreated_and_reused(self, sandbox):
"""A server-side session loss must not pin the scope to a stale id."""
from agent_sandbox.core.api_error import ApiError
created_ids: list[str] = []
exec_ids: list[str] = []
def create_session(id, **kwargs):
created_ids.append(id)
return SimpleNamespace(data=SimpleNamespace(session_id=id))
def exec_command(command, **kwargs):
exec_ids.append(kwargs["id"])
if len(exec_ids) == 1:
raise ApiError(
headers={"server": "nginx/1.18.0 (Ubuntu)"},
status_code=404,
body={"success": False, "message": "Session not found", "data": None, "hint": None},
)
return SimpleNamespace(data=SimpleNamespace(output="ok", exit_code=0))
sandbox._client.shell.create_session = create_session
sandbox._client.shell.exec_command = exec_command
assert sandbox.execute_command_in_scope("first", scope_id="subagent-a") == "ok"
assert len(created_ids) == 2
assert exec_ids == created_ids
assert sandbox.execute_command_in_scope("second", scope_id="subagent-a") == "ok"
assert exec_ids[-1] == created_ids[1]
def test_missing_scoped_session_recovery_is_bounded(self, sandbox):
"""A missing replacement session is reported after one recovery attempt."""
from agent_sandbox.core.api_error import ApiError
created_ids: list[str] = []
exec_ids: list[str] = []
cleaned_ids: list[str] = []
def create_session(id, **kwargs):
created_ids.append(id)
return SimpleNamespace(data=SimpleNamespace(session_id=id))
def exec_command(command, **kwargs):
exec_ids.append(kwargs["id"])
raise ApiError(
headers={"server": "nginx/1.18.0 (Ubuntu)"},
status_code=404,
body={"success": False, "message": "Session not found", "data": None},
)
sandbox._client.shell.create_session = create_session
sandbox._client.shell.exec_command = exec_command
sandbox._client.shell.cleanup_session = lambda session_id, **kwargs: cleaned_ids.append(session_id)
result = sandbox.execute_command_in_scope("first", scope_id="subagent-a")
assert result.startswith("Error:")
assert len(created_ids) == 2
assert exec_ids == created_ids
assert cleaned_ids == [created_ids[1]]
assert sandbox._scoped_shell_sessions["subagent-a"].session_id is None
def test_other_scoped_404_does_not_rotate_session(self, sandbox):
"""Only the structured missing-session response is recoverable."""
from agent_sandbox.core.api_error import ApiError
created_ids: list[str] = []
exec_ids: list[str] = []
def create_session(id, **kwargs):
created_ids.append(id)
return SimpleNamespace(data=SimpleNamespace(session_id=id))
def exec_command(command, **kwargs):
exec_ids.append(kwargs["id"])
raise ApiError(
headers={"server": "nginx/1.18.0 (Ubuntu)"},
status_code=404,
body={"success": False, "message": "Not Found", "data": None},
)
sandbox._client.shell.create_session = create_session
sandbox._client.shell.exec_command = exec_command
result = sandbox.execute_command_in_scope("first", scope_id="subagent-a")
assert result.startswith("Error:")
assert len(created_ids) == 1
assert exec_ids == created_ids
assert sandbox._scoped_shell_sessions["subagent-a"].session_id == created_ids[0]
def test_queued_command_cannot_restart_session_after_scope_release(self, sandbox):
created_ids: list[str] = []
executed_commands: list[str] = []
@ -498,9 +621,47 @@ class TestScopedShellSessions:
== "ok"
)
sandbox._client.bash.exec.assert_called_once()
session_id = sandbox._client.bash.create_session.call_args.kwargs["session_id"]
assert sandbox._client.bash.exec.call_args.kwargs["session_id"] == session_id
sandbox._client.bash.close_session.assert_called_once_with(session_id)
sandbox._client.shell.create_session.assert_not_called()
assert sandbox._scoped_shell_sessions == {}
def test_env_session_cleanup_failure_does_not_mask_command_output(self, sandbox):
sandbox._client.bash.exec = MagicMock(return_value=SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr="", exit_code=0)))
sandbox._client.bash.close_session = MagicMock(side_effect=RuntimeError("cleanup failed"))
assert sandbox.execute_command("echo $TOKEN", env={"TOKEN": "secret"}) == "ok"
def test_missing_session_with_failed_replacement_does_not_execute_third_time(self, sandbox):
from agent_sandbox.core.api_error import ApiError
executions = 0
def exec_command(command, **kwargs):
nonlocal executions
executions += 1
if executions == 1:
raise ApiError(
status_code=404,
body={"message": "session not found while executing command"},
)
return SimpleNamespace(
data=SimpleNamespace(
output="'ErrorObservation' object has no attribute 'exit_code'",
exit_code=None,
)
)
sandbox._client.shell.exec_command = exec_command
sandbox._client.shell.create_session = lambda id, **kwargs: SimpleNamespace(data=SimpleNamespace(session_id=id))
result = sandbox.execute_command_in_scope("unsafe-to-repeat", scope_id="subagent-a")
assert "ErrorObservation" in result
assert executions == 2
assert sandbox._scoped_shell_sessions["subagent-a"].session_id is None
def test_closed_sandbox_rejects_new_scope_without_leaking_session(self, sandbox):
client = sandbox._client
@ -594,6 +755,31 @@ class TestBashExecUnsupportedFailFast:
assert second == "ok"
assert sandbox._client.bash.exec.call_count == 2
def test_bash_exec_missing_session_retries_without_latching_unsupported(self, sandbox):
"""A transient loss after explicit creation must retry once and keep the
capability enabled for subsequent env-bearing commands."""
from agent_sandbox.core.api_error import ApiError
missing = ApiError(status_code=404, body={"message": "Session not found: transient"})
sandbox._client.bash.exec = MagicMock(side_effect=[missing, SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None))])
assert sandbox.execute_command("cmd", env={"TOK": "v"}) == "ok"
assert sandbox._bash_exec_unsupported is False
assert sandbox._client.bash.exec.call_count == 2
assert sandbox._client.bash.create_session.call_count == 2
def test_bash_exec_repeated_missing_session_does_not_latch_capability_gap(self, sandbox):
from agent_sandbox.core.api_error import ApiError
missing = ApiError(status_code=404, body={"message": "Session not found"})
sandbox._client.bash.exec = MagicMock(side_effect=missing)
out = sandbox.execute_command("cmd", env={"TOK": "v"})
assert out == "Error: bash.exec session disappeared after retry"
assert sandbox._bash_exec_unsupported is False
assert sandbox._client.bash.exec.call_count == 2
class TestListDirSerialization:
"""Verify that list_dir also acquires the lock."""

67
backend/tests/test_aio_sandbox_local_backend.py
View File

@ -1176,6 +1176,73 @@ def test_discover_returns_none_when_runtime_check_times_out(monkeypatch):
assert backend.discover("sandbox-timeout") is None
def test_discover_replaces_container_with_insufficient_shell_capacity(monkeypatch):
backend = _backend_for_inspect_tests()
backend._environment["MAX_SHELL_SESSIONS"] = "13"
container_name = "sandbox-existing"
monkeypatch.setattr(backend, "_is_container_running", lambda _name: True)
monkeypatch.setattr(
backend,
"_batch_inspect",
lambda *_args, **_kwargs: {
container_name: _ContainerInspection(
created_at=1.0,
host_port=18080,
labels={
"deerflow.role": "sandbox",
"deerflow.sandbox_id": "existing",
"deerflow.network_mode": "open",
},
image="sandbox:latest",
networks=frozenset({"bridge"}),
max_shell_sessions=10,
)
},
)
info = backend.discover("existing")
assert info is not None
assert info.requires_replacement is True
assert info.sandbox_url == ""
def test_list_running_marks_insufficient_shell_capacity_for_fenced_replacement(monkeypatch):
backend = _backend_for_inspect_tests()
backend._environment["MAX_SHELL_SESSIONS"] = "13"
container_name = "sandbox-existing"
def fake_run(cmd, **kwargs):
assert cmd[:2] == ["docker", "ps"]
return SimpleNamespace(stdout=f"{container_name}\n", stderr="", returncode=0)
monkeypatch.setattr("subprocess.run", fake_run)
monkeypatch.setattr(
backend,
"_batch_inspect",
lambda *_args, **_kwargs: {
container_name: _ContainerInspection(
created_at=1.0,
host_port=18080,
labels={
"deerflow.role": "sandbox",
"deerflow.sandbox_id": "existing",
"deerflow.network_mode": "open",
},
image="sandbox:latest",
networks=frozenset({"bridge"}),
max_shell_sessions=10,
)
},
)
infos = backend.list_running()
assert len(infos) == 1
assert infos[0].requires_replacement is True
assert infos[0].sandbox_url == ""
def test_restricted_discovery_uses_proxy_relay_port(monkeypatch):
backend = _backend_for_inspect_tests()
backend._network_mode = "allowlist"

131
backend/tests/test_aio_sandbox_provider.py
View File

@ -44,8 +44,12 @@ def test_load_config_preserves_thread_data_mounts_override(sandbox_overrides, ex
monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config)
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
assert provider._load_config()["thread_data_mounts"] is expected
assert provider._load_config()["skills_container_path"] == "/mnt/skills"
loaded = provider._load_config()
assert loaded["thread_data_mounts"] is expected
assert loaded["skills_container_path"] == "/mnt/skills"
assert loaded["max_shell_sessions"] is None
assert "MAX_SHELL_SESSIONS" not in loaded["environment"]
def test_load_config_snapshots_custom_skills_container_path(monkeypatch):
@ -64,6 +68,102 @@ def test_load_config_snapshots_custom_skills_container_path(monkeypatch):
assert provider._load_config()["skills_container_path"] == "/custom-skills"
def test_load_config_sizes_aio_shell_capacity_for_subagent_runtime(monkeypatch):
"""Twelve subagents must not exceed AIO 1.11's ten-session default."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
sandbox_config = SandboxConfig(
use="deerflow.community.aio_sandbox:AioSandboxProvider",
)
app_config = SimpleNamespace(
sandbox=sandbox_config,
stream_bridge=None,
subagent_runtime=SimpleNamespace(max_running=12),
)
monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config)
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
loaded = provider._load_config()
assert loaded["max_shell_sessions"] == 13
assert loaded["environment"]["MAX_SHELL_SESSIONS"] == str(loaded["max_shell_sessions"])
@pytest.mark.parametrize("remote", [False, True], ids=["docker", "provisioner"])
@pytest.mark.parametrize("persisted_capacity", [4, 9, 13])
def test_backend_checks_runtime_minimum_without_overriding_image_default(monkeypatch, remote, persisted_capacity):
"""Removing a four-session override must not reuse it for eight subagents."""
from deerflow.community.aio_sandbox import aio_sandbox_provider as aio_mod
from deerflow.community.aio_sandbox import local_backend as local_mod
from deerflow.community.aio_sandbox import remote_backend as remote_mod
app_config = SimpleNamespace(
sandbox=SandboxConfig(
use="deerflow.community.aio_sandbox:AioSandboxProvider",
container_prefix="sandbox",
provisioner_url="http://provisioner:8002" if remote else None,
environment={"MAX_SHELL_SESSIONS": "4"},
),
stream_bridge=None,
subagent_runtime=SimpleNamespace(max_running=3),
)
monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config)
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
assert provider._load_config()["max_shell_sessions"] == 4
app_config.subagent_runtime.max_running = 8
app_config.sandbox.environment = {}
provider._config = provider._load_config()
assert provider._config["max_shell_sessions"] is None
assert "MAX_SHELL_SESSIONS" not in provider._config["environment"]
monkeypatch.setattr(local_mod.LocalContainerBackend, "_detect_runtime", lambda _self: "docker")
backend = provider._create_backend()
if remote:
payload = {"sandbox_id": "example", "sandbox_url": "http://sandbox:8080", "max_shell_sessions": persisted_capacity}
def get(url, **_kwargs):
data = {"sandboxes": [payload]} if url.endswith("/api/sandboxes") else payload
return SimpleNamespace(status_code=200, raise_for_status=lambda: None, json=lambda: data)
monkeypatch.setattr(remote_mod.requests, "get", get)
else:
monkeypatch.setattr(backend, "_is_container_running", lambda _name: True)
monkeypatch.setattr(local_mod, "wait_for_sandbox_ready", lambda *_a, **_kw: True)
monkeypatch.setattr(local_mod.subprocess, "run", lambda *_a, **_kw: SimpleNamespace(returncode=0, stdout="sandbox-example\n"))
inspection = local_mod._ContainerInspection(
created_at=1.0,
host_port=18080,
image="sandbox:latest",
networks=frozenset({"bridge"}),
labels={"deerflow.role": "sandbox", "deerflow.sandbox_id": "example", "deerflow.network_mode": "open"},
max_shell_sessions=persisted_capacity,
)
monkeypatch.setattr(backend, "_batch_inspect", lambda *_a, **_kw: {"sandbox-example": inspection})
discovered = backend.discover("example")
assert discovered is not None
assert discovered.requires_replacement is (persisted_capacity < 9)
listed = backend.list_running()
assert len(listed) == 1
assert listed[0].requires_replacement is (persisted_capacity < 9)
def test_load_config_rejects_shell_capacity_below_subagent_runtime(monkeypatch):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
sandbox_config = SandboxConfig(
use="deerflow.community.aio_sandbox:AioSandboxProvider",
environment={"MAX_SHELL_SESSIONS": "12"},
)
app_config = SimpleNamespace(
sandbox=sandbox_config,
stream_bridge=None,
subagent_runtime=SimpleNamespace(max_running=12),
)
monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config)
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
with pytest.raises(ValueError, match=r"at least subagent_runtime\.max_running \+ 1"):
provider._load_config()
@pytest.mark.parametrize(
("backend_is_local", "override", "expected"),
[
@ -979,6 +1079,33 @@ def test_remote_backend_create_prefers_explicit_user_id(monkeypatch):
assert posted["json"]["include_legacy_skills"] is False
def test_remote_backend_forwards_shell_capacity_to_provisioner(monkeypatch):
remote_mod = importlib.import_module("deerflow.community.aio_sandbox.remote_backend")
backend = remote_mod.RemoteSandboxBackend(
"http://provisioner:8002",
max_shell_sessions=13,
)
posted: dict = {}
class _Response:
def raise_for_status(self):
return None
def json(self):
return {"sandbox_url": "http://sandbox.local", "max_shell_sessions": 13}
def _post(url, json, timeout, headers=None): # noqa: A002 - mirrors requests.post kwarg
posted.update({"url": url, "json": json, "timeout": timeout})
return _Response()
monkeypatch.setattr(remote_mod.requests, "post", _post)
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda _user_id: False)
backend.create("thread-42", "sandbox-42", user_id="user-7")
assert posted["json"]["max_shell_sessions"] == 13
def test_create_sandbox_requests_runtime_when_lark_installed(tmp_path, monkeypatch):
"""The provider must request lark-cli runtime provisioning when Lark is installed."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")

8
backend/tests/test_github_token_plumbing.py
View File

@ -110,11 +110,18 @@ def test_aio_sandbox_env_routes_through_bash_exec() -> None:
captured: dict = {}
class _FakeBash:
def create_session(self, *, session_id):
captured["created_session"] = session_id
def exec(self, *, command, env=None, **kwargs):
captured["command"] = command
captured["env"] = env
captured["exec_session"] = kwargs["session_id"]
return SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None))
def close_session(self, session_id):
captured["closed_session"] = session_id
sbx = AioSandbox.__new__(AioSandbox)
sbx._lock = __import__("threading").Lock()
sbx._client = SimpleNamespace(bash=_FakeBash())
@ -127,6 +134,7 @@ def test_aio_sandbox_env_routes_through_bash_exec() -> None:
assert out == "ok"
assert captured["command"] == "gh pr create"
assert captured["env"] == {"GH_TOKEN": "tok-123"}
assert captured["created_session"] == captured["exec_session"] == captured["closed_session"]
def test_aio_sandbox_no_env_leaves_command_unchanged() -> None:

215
backend/tests/test_provisioner_request_threading.py
View File

@ -4,13 +4,16 @@ from __future__ import annotations
import asyncio
import inspect
import json
import threading
import time
from contextlib import contextmanager
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
import requests
from blockbuster import BlockBuster
from kubernetes.client.rest import ApiException
@ -48,6 +51,194 @@ def test_provisioner_request_defaults_skills_container_path(provisioner_module)
)
assert request.skills_container_path == "/mnt/skills"
assert request.max_shell_sessions is None
def test_provisioner_threads_shell_capacity_into_sandbox_pod(
monkeypatch: pytest.MonkeyPatch,
provisioner_module,
) -> None:
fake_core_v1 = _RecordingCoreV1(
event_loop_thread_id=-1,
ready_after_service_reads={"sandbox-capacity": 1},
)
monkeypatch.setattr(provisioner_module, "core_v1", fake_core_v1)
response = provisioner_module.create_sandbox(
provisioner_module.CreateSandboxRequest(
sandbox_id="sandbox-capacity",
thread_id="thread-1",
max_shell_sessions=13,
)
)
assert response.status == "Running"
pod = fake_core_v1.created_pod_specs["sandbox-capacity"]
env = {item.name: item.value for item in (pod.spec.containers[0].env or [])}
assert env["MAX_SHELL_SESSIONS"] == "13"
def test_provisioner_rejects_insufficient_capacity_without_replacing_existing_pod(
monkeypatch: pytest.MonkeyPatch,
provisioner_module,
) -> None:
fake_core_v1 = _RecordingCoreV1(event_loop_thread_id=-1)
monkeypatch.setattr(provisioner_module, "core_v1", fake_core_v1)
with pytest.raises(provisioner_module.HTTPException) as error:
provisioner_module.create_sandbox(
provisioner_module.CreateSandboxRequest(
sandbox_id="sandbox-existing",
thread_id="thread-1",
max_shell_sessions=13,
)
)
assert error.value.status_code == 409
assert "capacity" in error.value.detail
assert fake_core_v1.created_pods == []
assert fake_core_v1.pod_shell_capacities["sandbox-existing"] == 10
assert "sandbox-existing" in fake_core_v1.service_sandboxes
def test_capacity_replacement_recovers_when_service_outlives_old_pod(monkeypatch, provisioner_module):
"""A create racing asynchronous deletion can leave a Service without a Pod."""
core = _RecordingCoreV1(event_loop_thread_id=-1)
core.pod_shell_capacities.pop("sandbox-existing")
monkeypatch.setattr(provisioner_module, "core_v1", core)
def existing_service(*_args):
raise ApiException(status=409)
monkeypatch.setattr(core, "create_namespaced_service", existing_service)
result = provisioner_module.create_sandbox(provisioner_module.CreateSandboxRequest(sandbox_id="sandbox-existing", thread_id="thread-a", max_shell_sessions=13))
assert result.max_shell_sessions == 13
assert core.created_pods == ["sandbox-existing"]
assert "sandbox-existing" in core.service_sandboxes
@pytest.mark.parametrize("failure", [ApiException(status=403), ApiException(status=503), RuntimeError("invalid persisted capacity")])
def test_capacity_read_failure_does_not_authorize_creation(monkeypatch, provisioner_module, failure):
core = _RecordingCoreV1(event_loop_thread_id=-1)
monkeypatch.setattr(provisioner_module, "core_v1", core)
monkeypatch.setattr(provisioner_module, "_get_pod_shell_capacity", MagicMock(side_effect=failure))
with pytest.raises(provisioner_module.HTTPException) as error:
provisioner_module.create_sandbox(provisioner_module.CreateSandboxRequest(sandbox_id="sandbox-existing", max_shell_sessions=13))
assert error.value.status_code == 500
assert core.created_pods == []
assert core.pod_shell_capacities["sandbox-existing"] == 10
assert "sandbox-existing" in core.service_sandboxes
def test_list_sandboxes_skips_invalid_capacity_metadata(
monkeypatch: pytest.MonkeyPatch,
provisioner_module,
) -> None:
fake_core_v1 = _RecordingCoreV1(event_loop_thread_id=-1)
monkeypatch.setattr(provisioner_module, "core_v1", fake_core_v1)
def invalid_capacity(_sandbox_id: str) -> int:
raise RuntimeError("invalid capacity")
monkeypatch.setattr(
provisioner_module,
"_get_pod_shell_capacity",
invalid_capacity,
)
response = provisioner_module.list_sandboxes()
assert response == {"sandboxes": [], "count": 0}
@pytest.mark.asyncio
@pytest.mark.parametrize("async_acquire", [False, True], ids=["sync", "async"])
async def test_capacity_upgrade_preserves_peer_pod_after_discovery_error(monkeypatch, tmp_path, provisioner_module, async_acquire):
"""Discovery failure must not bypass ownership; a later safe retry replaces."""
from test_sandbox_orphan_reconciliation import _make_provider_for_reconciliation, _make_shared_ownership_store
from deerflow.community.aio_sandbox import aio_sandbox_provider as provider_mod
from deerflow.community.aio_sandbox import remote_backend as remote_mod
from deerflow.community.aio_sandbox.ownership import compute_lease_ttl
from deerflow.config.paths import Paths
shared = _make_shared_ownership_store()
old = _make_provider_for_reconciliation(worker_id="old-gateway", store=shared)
new = _make_provider_for_reconciliation(worker_id="new-gateway", store=shared)
sid = "sandbox-existing"
old._publish_ownership(sid)
new._backend = remote_mod.RemoteSandboxBackend("http://provisioner:8002", max_shell_sessions=13)
core = _RecordingCoreV1(event_loop_thread_id=threading.get_ident())
monkeypatch.setattr(provisioner_module, "core_v1", core)
deleted_under_lease = []
discovery_failed = False
def response(status, payload):
result = requests.Response()
result.status_code = status
result._content = json.dumps(payload).encode()
return result
def get(_url, **_kwargs):
nonlocal discovery_failed
if not discovery_failed:
discovery_failed = True
return response(503, {"detail": "temporarily unavailable"})
return response(200, provisioner_module.get_sandbox(sid).model_dump())
def post(_url, *, json, **_kwargs):
try:
result = provisioner_module.create_sandbox(provisioner_module.CreateSandboxRequest(**json))
except provisioner_module.HTTPException as exc:
return response(exc.status_code, {"detail": exc.detail})
return response(200, result.model_dump())
def delete(_url, **_kwargs):
deleted_under_lease.append((shared.owner(sid), old._ownership.claim(sid)))
return response(200, provisioner_module.destroy_sandbox(sid))
monkeypatch.setattr(requests, "get", get)
monkeypatch.setattr(requests, "post", post)
monkeypatch.setattr(requests, "delete", delete)
monkeypatch.setattr(remote_mod, "user_should_see_legacy_skills", lambda _uid: False)
monkeypatch.setattr(provider_mod, "get_paths", lambda: Paths(base_dir=tmp_path))
monkeypatch.setattr(provider_mod, "wait_for_sandbox_ready", lambda *_a, **_kw: True)
monkeypatch.setattr(provider_mod, "wait_for_sandbox_ready_async", AsyncMock(return_value=True))
monkeypatch.setattr(provider_mod, "AioSandbox", MagicMock())
monkeypatch.setattr(new, "_get_extra_mounts", lambda *_a, **_kw: [])
monkeypatch.setattr(new, "_lark_integration_active", lambda *_a: False)
monkeypatch.setattr(new, "_lark_broker_active", lambda *_a: False)
async def acquire():
if async_acquire:
return await new._discover_or_create_with_lock_async("thread-a", sid, user_id="user-a")
return await asyncio.to_thread(new._discover_or_create_with_lock, "thread-a", sid, user_id="user-a")
try:
with pytest.raises(RuntimeError, match="409"):
await acquire()
assert core.pod_shell_capacities[sid] == 10
assert core.created_pods == []
assert sid in core.service_sandboxes
assert shared.owner(sid) == "old-gateway"
with pytest.raises(provider_mod.SandboxPolicyReplacementDeferredError):
await acquire()
old._ownership.release(sid)
with pytest.raises(provider_mod.SandboxPolicyReplacementDeferredError):
await acquire()
assert deleted_under_lease == []
new._unowned_since[sid] = time.time() - compute_lease_ttl(new._ownership_config) - 1
assert await acquire() == sid
assert deleted_under_lease == [("new-gateway", False)]
assert core.created_pods == [sid]
assert core.pod_shell_capacities[sid] == 13
assert shared.owner(sid) == "new-gateway"
finally:
old._acquire_serializer.close()
new._acquire_serializer.close()
class _RecordingCoreV1:
@ -61,6 +252,10 @@ class _RecordingCoreV1:
self.event_loop_thread_id = event_loop_thread_id
self.thread_ids: list[int] = []
self.service_sandboxes: set[str] = {"sandbox-existing"}
self.pod_shell_capacities: dict[str, int] = {
"sandbox-existing": 10,
"sandbox-listed": 10,
}
self.ready_after_service_reads = ready_after_service_reads or {}
self.service_read_failures = service_read_failures or {}
self.service_read_counts: dict[str, int] = {}
@ -94,13 +289,29 @@ class _RecordingCoreV1:
def read_namespaced_pod(self, _name: str, _namespace: str):
self._record_k8s_call()
return SimpleNamespace(status=SimpleNamespace(phase="Running"))
sandbox_id = _name[len("sandbox-") :]
capacity = self.pod_shell_capacities.get(sandbox_id)
if capacity is None:
raise ApiException(status=404)
return SimpleNamespace(
status=SimpleNamespace(phase="Running"),
spec=SimpleNamespace(
containers=[
SimpleNamespace(
name="sandbox",
env=[SimpleNamespace(name="MAX_SHELL_SESSIONS", value=str(capacity))],
)
]
),
)
def create_namespaced_pod(self, _namespace: str, pod) -> None:
self._record_k8s_call()
sandbox_id = pod.metadata.labels["sandbox-id"]
self.created_pods.append(sandbox_id)
self.created_pod_specs[sandbox_id] = pod
env = {item.name: item.value for item in (pod.spec.containers[0].env or [])}
self.pod_shell_capacities[sandbox_id] = int(env.get("MAX_SHELL_SESSIONS", "10"))
def create_namespaced_service(self, _namespace: str, service) -> None:
self._record_k8s_call()
@ -110,9 +321,11 @@ class _RecordingCoreV1:
def delete_namespaced_service(self, _name: str, _namespace: str) -> None:
self._record_k8s_call()
self.service_sandboxes.discard(_sandbox_id_from_service_name(_name))
def delete_namespaced_pod(self, _name: str, _namespace: str) -> None:
self._record_k8s_call()
self.pod_shell_capacities.pop(_name[len("sandbox-") :], None)
def list_namespaced_service(self, _namespace: str, *, label_selector: str):
self._record_k8s_call()

84
backend/tests/test_remote_sandbox_backend.py
View File

@ -96,6 +96,33 @@ def test_provisioner_list_returns_sandbox_infos_and_filters_invalid_entries(monk
assert infos[0].sandbox_url == "http://k3s:31001"
def test_provisioner_list_marks_insufficient_shell_capacity_for_replacement(monkeypatch):
backend = RemoteSandboxBackend(
"http://provisioner:8002",
max_shell_sessions=13,
)
def mock_get(url: str, timeout: int, headers=None):
return _StubResponse(
payload={
"sandboxes": [
{
"sandbox_id": "abc123",
"sandbox_url": "http://k3s:31001",
"max_shell_sessions": 10,
}
]
}
)
monkeypatch.setattr(requests, "get", mock_get)
infos = backend._provisioner_list()
assert len(infos) == 1
assert infos[0].requires_replacement is True
def test_provisioner_list_sends_auth_header_when_api_key_set(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002", api_key="secret")
captured: list[dict] = []
@ -562,6 +589,63 @@ def test_provisioner_discover_returns_info_on_success(monkeypatch):
assert info.sandbox_url == "http://k3s:31001"
def test_provisioner_discover_marks_insufficient_shell_capacity_for_replacement(monkeypatch):
backend = RemoteSandboxBackend(
"http://provisioner:8002",
max_shell_sessions=13,
)
def mock_get(url: str, timeout: int, headers=None):
return _StubResponse(
payload={
"sandbox_id": "abc123",
"sandbox_url": "http://k3s:31001",
"max_shell_sessions": 10,
}
)
monkeypatch.setattr(requests, "get", mock_get)
info = backend._provisioner_discover("abc123")
assert info is not None
assert info.requires_replacement is True
def test_provisioner_create_reports_version_skew_when_capacity_is_missing(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002", max_shell_sessions=13)
monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda _user_id: False)
def mock_post(url: str, *, json: dict, headers=None, timeout: int):
return _StubResponse(payload={"sandbox_id": "abc123", "sandbox_url": "http://k3s:31001"})
monkeypatch.setattr(requests, "post", mock_post)
with pytest.raises(RuntimeError, match="version skew"):
backend._provisioner_create(None, "abc123")
@pytest.mark.parametrize("reported_capacity", [4, 9, 13, None])
def test_create_checks_runtime_minimum_without_sending_image_override(monkeypatch, reported_capacity):
backend = RemoteSandboxBackend("http://provisioner:8002", required_shell_sessions=9)
monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda _user_id: False)
def post(_url, *, json, **_kwargs):
assert "max_shell_sessions" not in json
payload = {"sandbox_url": "http://sandbox:8080"}
if reported_capacity is not None:
payload["max_shell_sessions"] = reported_capacity
return _StubResponse(payload=payload)
monkeypatch.setattr(requests, "post", post)
if reported_capacity == 4:
with pytest.raises(RuntimeError, match="insufficient shell-session capacity"):
backend.create("thread-a", "example")
else:
# A legacy provisioner without metadata retains the known image default.
assert backend.create("thread-a", "example").sandbox_url == "http://sandbox:8080"
def test_provisioner_discover_returns_none_on_request_exception(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002")

5
config.example.yaml
View File

@ -1538,7 +1538,12 @@ sandbox:
#
# # Optional: Environment variables to inject into the sandbox container
# # Values starting with $ will be resolved from host environment variables
# # AIO semver images default MAX_SHELL_SESSIONS to 10. DeerFlow leaves that
# # image default unchanged while subagent_runtime.max_running + 1 fits, and
# # otherwise injects the required value for local and provisioner sandboxes.
# # An explicit value is allowed but must be at least max_running + 1.
# # environment:
# # MAX_SHELL_SESSIONS: "13" # Example for subagent_runtime.max_running: 12
# # NODE_ENV: production
# # DEBUG: "false"
# # API_KEY: $MY_API_KEY # Reads from host's MY_API_KEY env var

11
docker/provisioner/README.md
View File

@ -164,6 +164,17 @@ The provisioner is configured via environment variables (set in [docker-compose-
| `NODE_HOST` | `host.docker.internal` | Hostname that backend containers use to reach host NodePorts; ignored when `SANDBOX_SERVICE_TYPE=ClusterIP` |
| `K8S_API_SERVER` | (from kubeconfig) | Override K8s API server URL (e.g., `https://host.docker.internal:26443`) |
For new sandbox requests, the Gateway also sends the effective AIO shell-session
capacity derived from `subagent_runtime.max_running`. The provisioner writes it
to the sandbox Pod as `MAX_SHELL_SESSIONS`; requests from older Gateways omit the
field and retain the image default. Discovery responses report the effective
capacity. The Gateway replaces a lower-capacity Pod through its ownership-fenced
replacement path once the previous owner and recovery grace permit it. A create
request for an existing Pod with insufficient capacity returns HTTP 409; the
provisioner does not delete an existing Pod to upgrade its capacity, including
after a transient Gateway discovery failure. If the old Pod has already gone
but its Service remains, create can safely provision the missing Pod.
### Custom sandbox image
Provisioner-created sandbox Pods use the provisioner's `SANDBOX_IMAGE` environment variable. This is separate from `sandbox.image` in `config.yaml`, which applies to local Docker or Apple Container mode.

96
docker/provisioner/app.py
View File

@ -109,6 +109,7 @@ SAFE_THREAD_ID_PATTERN = r"^[A-Za-z0-9_-]{1,64}$"
SAFE_USER_ID_PATTERN = r"^[A-Za-z0-9_\-]+$"
DEFAULT_USER_ID = "default"
DEFAULT_SKILLS_CONTAINER_PATH = "/mnt/skills"
DEFAULT_MAX_SHELL_SESSIONS = 10
MAX_EXTRA_MOUNTS = 10
ALLOWED_EXTRA_MOUNT_PATHS = {
"/mnt/acp-workspace",
@ -483,12 +484,16 @@ class CreateSandboxRequest(BaseModel):
# mounted into the sidecar only, never the sandbox. Supersedes the runtime
# binary + credential mounts when enabled.
provision_lark_cli_broker: bool = False
# New Gateways size this from their process-wide subagent capacity. None
# keeps requests from older Gateways and custom provisioner callers working.
max_shell_sessions: int | None = Field(default=None, gt=0)
class SandboxResponse(BaseModel):
sandbox_id: str
sandbox_url: str
status: str
max_shell_sessions: int | None = None
# ── K8s resource helpers ─────────────────────────────────────────────────
@ -958,11 +963,27 @@ def _build_pod(
skills_container_path: str = DEFAULT_SKILLS_CONTAINER_PATH,
provision_lark_cli_runtime: bool = False,
provision_lark_cli_broker: bool = False,
max_shell_sessions: int | None = None,
) -> k8s_client.V1Pod:
"""Construct a Pod manifest for a single sandbox."""
init_containers = (
_build_lark_cli_init_containers(provision_lark_cli_runtime, provision_lark_cli_broker) or None
)
sandbox_env: list[k8s_client.V1EnvVar] = []
if max_shell_sessions is not None:
sandbox_env.append(
k8s_client.V1EnvVar(
name="MAX_SHELL_SESSIONS",
value=str(max_shell_sessions),
)
)
if _lark_cli_broker_enabled(provision_lark_cli_broker):
sandbox_env.append(
k8s_client.V1EnvVar(
name="DEERFLOW_LARK_BROKER_URL",
value=LARK_BROKER_URL,
)
)
return k8s_client.V1Pod(
metadata=k8s_client.V1ObjectMeta(
name=_pod_name(sandbox_id),
@ -980,11 +1001,7 @@ def _build_pod(
name="sandbox",
image=SANDBOX_IMAGE,
image_pull_policy="IfNotPresent",
env=(
[k8s_client.V1EnvVar(name="DEERFLOW_LARK_BROKER_URL", value=LARK_BROKER_URL)]
if _lark_cli_broker_enabled(provision_lark_cli_broker)
else None
),
env=sandbox_env or None,
ports=[
k8s_client.V1ContainerPort(
name="http",
@ -1129,6 +1146,24 @@ def _get_pod_phase(sandbox_id: str) -> str:
return "NotFound"
def _get_pod_shell_capacity(sandbox_id: str) -> int:
"""Return the effective AIO shell capacity persisted in the sandbox Pod."""
pod = core_v1.read_namespaced_pod(_pod_name(sandbox_id), K8S_NAMESPACE)
containers = getattr(getattr(pod, "spec", None), "containers", None) or []
sandbox_container = next((container for container in containers if getattr(container, "name", None) == "sandbox"), None)
for env_var in getattr(sandbox_container, "env", None) or []:
if getattr(env_var, "name", None) != "MAX_SHELL_SESSIONS":
continue
try:
value = int(env_var.value)
except (TypeError, ValueError) as exc:
raise RuntimeError("Persisted sandbox has an invalid MAX_SHELL_SESSIONS value") from exc
if value <= 0:
raise RuntimeError("Persisted sandbox has an invalid MAX_SHELL_SESSIONS value")
return value
return DEFAULT_MAX_SHELL_SESSIONS
# ── API endpoints ────────────────────────────────────────────────────────
@ -1169,9 +1204,10 @@ def create_sandbox(req: CreateSandboxRequest):
)
provision_lark_cli_runtime = req.provision_lark_cli_runtime
provision_lark_cli_broker = req.provision_lark_cli_broker
max_shell_sessions = req.max_shell_sessions
logger.info(
"Received request to create sandbox '%s' for thread '%s' user '%s' include_legacy_skills=%s skills_container_path=%s provision_lark_cli_runtime=%s provision_lark_cli_broker=%s",
"Received request to create sandbox '%s' for thread '%s' user '%s' include_legacy_skills=%s skills_container_path=%s provision_lark_cli_runtime=%s provision_lark_cli_broker=%s max_shell_sessions=%s",
sandbox_id,
thread_id,
user_id,
@ -1179,16 +1215,33 @@ def create_sandbox(req: CreateSandboxRequest):
skills_container_path,
_lark_cli_runtime_enabled(provision_lark_cli_runtime),
_lark_cli_broker_enabled(provision_lark_cli_broker),
max_shell_sessions,
)
# ── Fast path: sandbox already exists ────────────────────────────
existing_url = _sandbox_access_url(sandbox_id, tolerate_read_errors=True)
if existing_url:
return SandboxResponse(
sandbox_id=sandbox_id,
sandbox_url=existing_url,
status=_get_pod_phase(sandbox_id),
)
try:
existing_shell_capacity = _get_pod_shell_capacity(sandbox_id)
except (ApiException, RuntimeError) as exc:
# A Service can outlive its old Pod during asynchronous replacement.
# Only a confirmed missing Pod may fall through to creation.
if not isinstance(exc, ApiException) or exc.status != 404:
raise HTTPException(status_code=500, detail=f"Could not verify existing sandbox shell capacity: {exc}") from exc
else:
if max_shell_sessions is not None and existing_shell_capacity < max_shell_sessions:
# Only the Gateway can fence replacement against active owners.
# A transient discovery failure can route a live Pod through create.
raise HTTPException(
status_code=409,
detail="Existing sandbox shell capacity is below the requested value; replacement must be coordinated by the Gateway",
)
return SandboxResponse(
sandbox_id=sandbox_id,
sandbox_url=existing_url,
status=_get_pod_phase(sandbox_id),
max_shell_sessions=existing_shell_capacity,
)
# ── Create Pod ───────────────────────────────────────────────────
try:
@ -1203,6 +1256,7 @@ def create_sandbox(req: CreateSandboxRequest):
skills_container_path=skills_container_path,
provision_lark_cli_runtime=provision_lark_cli_runtime,
provision_lark_cli_broker=provision_lark_cli_broker,
max_shell_sessions=max_shell_sessions,
),
)
logger.info(f"Created Pod {_pod_name(sandbox_id)}")
@ -1234,10 +1288,20 @@ def create_sandbox(req: CreateSandboxRequest):
if not sandbox_url:
raise HTTPException(status_code=500, detail="Service access URL was not available in time")
# A concurrent creator can win the 409 race with a lower-capacity Pod.
# Never claim that the requested value was applied without reading it back.
try:
actual_shell_capacity = _get_pod_shell_capacity(sandbox_id)
except (ApiException, RuntimeError) as exc:
raise HTTPException(status_code=500, detail=f"Could not verify created sandbox shell capacity: {exc}") from exc
if max_shell_sessions is not None and actual_shell_capacity < max_shell_sessions:
raise HTTPException(status_code=409, detail="Existing sandbox shell capacity is below the requested value")
return SandboxResponse(
sandbox_id=sandbox_id,
sandbox_url=sandbox_url,
status=_get_pod_phase(sandbox_id),
max_shell_sessions=actual_shell_capacity,
)
@ -1279,6 +1343,7 @@ def get_sandbox(sandbox_id: str):
sandbox_id=sandbox_id,
sandbox_url=sandbox_url,
status=_get_pod_phase(sandbox_id),
max_shell_sessions=_get_pod_shell_capacity(sandbox_id),
)
@ -1301,11 +1366,20 @@ def list_sandboxes():
sandbox_url = _url_from_service(svc, sid)
if not sandbox_url:
continue
try:
shell_capacity = _get_pod_shell_capacity(sid)
except (ApiException, RuntimeError) as exc:
if isinstance(exc, ApiException) and exc.status == 404:
continue
reason = getattr(exc, "reason", str(exc))
logger.warning("Skipping sandbox %s while inspecting shell capacity: %s", sid, reason)
continue
sandboxes.append(
SandboxResponse(
sandbox_id=sid,
sandbox_url=sandbox_url,
status=_get_pod_phase(sid),
max_shell_sessions=shell_capacity,
)
)