fix(sandbox): enforce bash command timeouts in AIO sandbox (#5634)

* fix(sandbox): enforce AIO shell command timeouts

Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results.

* fix(sandbox): contain ambiguous AIO command outcomes

Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative.

* fix(sandbox): wire AIO command timeout from provider config

Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working.

* docs(sandbox): document AIO command timeout semantics

Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget.

* docs(sandbox): scope the wedge-resistance claim to command requests

willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up.

* fix(sandbox): make list_dir honor the shell generation fence

list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.
This commit is contained in:
spud 2026-09-22 11:29:38 +08:00 committed by GitHub
parent e2f19d8335
commit 61a99a1e8a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
10 changed files with 1758 additions and 126 deletions

View File

@ -633,6 +633,13 @@ sandbox:
use: deerflow.community.aio_sandbox:AioSandboxProvider # Docker-based sandbox
```
For AIO images on the supported semver line (`1.9.3` through the recommended
`1.11.0` image), `sandbox.bash_command_timeout` is enforced server-side through
the `hard_timeout` API when the image exposes it. DeerFlow's legacy frozen
`all-in-one-sandbox:latest` image predates that API, so only the host-side
request is bounded there. Timed-out or otherwise ambiguous commands are never
replayed.
**BoxLite micro-VM Sandbox** (runs sandbox code in daemonless OCI micro-VMs):
```yaml
sandbox:

View File

@ -1,6 +1,7 @@
import base64
import errno
import logging
import math
import threading
import uuid
from dataclasses import dataclass, field
@ -65,6 +66,7 @@ class AioSandbox(Sandbox):
base_url: str,
home_dir: str | None = None,
request_headers: dict[str, str] | None = None,
default_command_timeout: float | None = None,
):
"""Initialize the AIO sandbox.
@ -74,8 +76,20 @@ class AioSandbox(Sandbox):
home_dir: Home directory inside the sandbox. If None, will be fetched from the sandbox.
request_headers: Trusted control-plane headers required by a local
relay. These are never injected into sandbox commands.
default_command_timeout: Provider-configured command deadline used
when a command does not provide an explicit timeout.
"""
super().__init__(id)
if default_command_timeout is None:
self._default_command_timeout = self._DEFAULT_HARD_TIMEOUT
else:
try:
resolved_default_timeout = float(default_command_timeout)
except (TypeError, ValueError) as exc:
raise ValueError("default_command_timeout must be positive") from exc
if not math.isfinite(resolved_default_timeout) or resolved_default_timeout <= 0:
raise ValueError("default_command_timeout must be positive")
self._default_command_timeout = resolved_default_timeout
self._base_url = base_url
client_kwargs = {
"base_url": base_url,
@ -179,9 +193,18 @@ class AioSandbox(Sandbox):
logger.warning(f"Error closing AioSandbox client for {self.id}: {e}")
@staticmethod
def _cleanup_session_best_effort(client, session_id: str, *, context: str) -> None:
def _cleanup_session_best_effort(
client,
session_id: str,
*,
context: str,
request_options: dict[str, int] | None = None,
) -> None:
try:
client.shell.cleanup_session(session_id)
client.shell.cleanup_session(
session_id,
**({"request_options": request_options} if request_options is not None else {}),
)
except Exception as cleanup_error:
logger.warning(
"Failed to release shell session %s (%s): %s",
@ -191,11 +214,12 @@ class AioSandbox(Sandbox):
)
@staticmethod
def _format_shell_result(result) -> tuple[str, int | None]:
def _format_shell_result(result) -> tuple[str, int | None, str | None]:
data = result.data if result else None
output = data.output if data else ""
exit_code = getattr(data, "exit_code", None) if data else None
return output, exit_code
status = getattr(data, "status", None) if data else None
return output, exit_code, status
@staticmethod
def _is_missing_shell_session_error(error: ApiError) -> bool:
@ -205,10 +229,12 @@ class AioSandbox(Sandbox):
message = body.get("message")
return isinstance(message, str) and "session not found" in message.casefold()
@staticmethod
def _cleanup_bash_session_best_effort(client, session_id: str) -> None:
def _cleanup_bash_session_best_effort(self, client, session_id: str) -> None:
try:
client.bash.close_session(session_id)
client.bash.close_session(
session_id,
request_options=self._bounded_cleanup_request_options(),
)
except Exception as cleanup_error:
logger.warning(
"Failed to release transient bash session %s: %s",
@ -221,10 +247,32 @@ class AioSandbox(Sandbox):
client.shell.create_session(id=session_id)
return session_id
def _exec_shell(self, client, command: str, *, session_id: str | None) -> tuple[str, int | None]:
def _ensure_default_shell_session_id(self, client) -> str | None:
"""Return the session id that may safely receive default-shell work.
A healthy implicit shell is represented by ``None``. Once that generation
is fenced, all later shell-backed operations must target an explicit
recovery session instead of re-entering the implicit shell.
Caller must hold ``self._lock``.
"""
if self._default_shell_corrupted and self._recovery_session_id is None:
self._recovery_session_id = self._create_shell_session(client)
return self._recovery_session_id
def _exec_shell(
self,
client,
command: str,
*,
session_id: str | None,
timeout: float,
) -> tuple[str, int | None, str | None]:
kwargs = {
"command": command,
"no_change_timeout": self._DEFAULT_NO_CHANGE_TIMEOUT,
"no_change_timeout": self._effective_no_change_timeout(timeout),
"hard_timeout": timeout,
"request_options": self._command_request_options(timeout),
}
if session_id is not None:
kwargs["id"] = session_id
@ -237,35 +285,55 @@ class AioSandbox(Sandbox):
*,
corrupted_session_id: str | None,
context: str,
) -> tuple[str, int | None, str | None]:
timeout: float,
) -> tuple[str, int | None, str | None, str | None]:
cleanup_options = self._bounded_cleanup_request_options()
if corrupted_session_id is not None:
self._cleanup_session_best_effort(
client,
corrupted_session_id,
context=f"corrupted {context}",
request_options=cleanup_options,
)
replacement_id = self._create_shell_session(client)
try:
output, exit_code = self._exec_shell(
output, exit_code, status = self._exec_shell(
client,
command,
session_id=replacement_id,
timeout=timeout,
)
except BaseException:
self._cleanup_session_best_effort(
client,
replacement_id,
context=f"abandoned replacement for {context}",
request_options=cleanup_options,
)
raise
if output and _ERROR_OBSERVATION_SIGNATURE in output:
if self._is_session_invalidating_shell_status(status):
if status == "terminated":
cleanup_context = f"terminated replacement for {context}"
elif status == "no_change_timeout":
cleanup_context = f"ambiguous replacement for {context}"
else:
cleanup_context = f"unexpected replacement status for {context}"
self._cleanup_session_best_effort(
client,
replacement_id,
context=cleanup_context,
request_options=cleanup_options,
)
return output, exit_code, status, None
if status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
self._cleanup_session_best_effort(
client,
replacement_id,
context=f"failed replacement for {context}",
request_options=cleanup_options,
)
return output, exit_code, None
return output, exit_code, replacement_id
return output, exit_code, status, None
return output, exit_code, status, replacement_id
def execute_command_in_scope(
self,
@ -283,7 +351,6 @@ class AioSandbox(Sandbox):
"""
if env or scope_id is None:
return self.execute_command(command, env=env, timeout=timeout)
del timeout
_validate_extra_env(env)
try:
@ -309,31 +376,75 @@ class AioSandbox(Sandbox):
if scoped.session_id is None:
scoped.session_id = self._create_shell_session(client)
try:
output, exit_code = self._exec_shell(
effective_timeout = self._effective_command_timeout(timeout)
output, exit_code, status = self._exec_shell(
client,
command,
session_id=scoped.session_id,
timeout=effective_timeout,
)
except httpx.TimeoutException:
session_id = scoped.session_id
scoped.session_id = None
if session_id is not None:
self._cleanup_session_best_effort(
client,
session_id,
context="execution scope after transport timeout",
request_options=self._bounded_cleanup_request_options(),
)
return self._transport_timeout_error(effective_timeout)
except ApiError as error:
if not self._is_missing_shell_session_error(error):
raise
logger.warning("Execution-scoped sandbox shell session is missing; recreating it once")
scoped.session_id = None
output, exit_code, scoped.session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=None,
context="execution scope after missing session",
)
if scoped.session_id is not None and output and _ERROR_OBSERVATION_SIGNATURE in output:
try:
output, exit_code, status, scoped.session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=None,
context="execution scope after missing session",
timeout=effective_timeout,
)
except httpx.TimeoutException:
return self._transport_timeout_error(effective_timeout)
if self._is_session_invalidating_shell_status(status):
session_id = scoped.session_id
scoped.session_id = None
if session_id is not None:
if status == "terminated":
cleanup_context = "execution scope after terminal session loss"
elif status == "no_change_timeout":
cleanup_context = "execution scope after ambiguous no-change timeout"
else:
cleanup_context = "execution scope after unexpected status"
self._cleanup_session_best_effort(
client,
session_id,
context=cleanup_context,
request_options=self._bounded_cleanup_request_options(),
)
if scoped.session_id is not None and status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
logger.warning("ErrorObservation detected in sandbox output for execution scope; rotating session")
output, exit_code, scoped.session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=scoped.session_id,
context="execution scope",
)
return self._render_shell_output(output, exit_code)
corrupted_session_id = scoped.session_id
scoped.session_id = None
try:
output, exit_code, status, scoped.session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=corrupted_session_id,
context="execution scope",
timeout=effective_timeout,
)
except httpx.TimeoutException:
return self._transport_timeout_error(effective_timeout)
return self._render_shell_output(
output,
exit_code,
status=status,
timeout=effective_timeout,
)
except Exception as e:
logger.error(f"Failed to execute command in sandbox: {e}")
return f"Error: {e}"
@ -355,7 +466,64 @@ class AioSandbox(Sandbox):
scoped.session_id = None
@staticmethod
def _render_shell_output(output: str, exit_code: int | None) -> str:
def _format_timeout_duration(timeout: float) -> str:
return f"{timeout:g}"
@classmethod
def _format_timeout_notice(cls, timeout: float) -> str:
return f"Command timed out after {cls._format_timeout_duration(timeout)} seconds and was terminated."
@classmethod
def _bounded_cleanup_request_options(cls) -> dict[str, int]:
return {
"timeout_in_seconds": cls._CLEANUP_REQUEST_TIMEOUT_SECONDS,
"max_retries": 0,
}
@classmethod
def _transport_timeout_error(cls, timeout: float) -> str:
request_timeout = cls._command_request_options(timeout)["timeout_in_seconds"]
return f"Error: Sandbox command response timed out after {request_timeout} seconds; command outcome is unknown and the command was not retried."
@staticmethod
def _is_unexpected_shell_status(status: str | None) -> bool:
return status not in (None, "completed", "hard_timeout", "no_change_timeout", "terminated")
@classmethod
def _is_session_invalidating_shell_status(cls, status: str | None) -> bool:
return status in ("terminated", "no_change_timeout") or cls._is_unexpected_shell_status(status)
@classmethod
def _unexpected_status_notice(cls, status: str) -> str:
return f"Error: Sandbox command returned an unexpected status '{status}'; command outcome is unknown and the command was not retried."
@classmethod
def _render_shell_output(
cls,
output: str,
exit_code: int | None,
*,
status: str | None,
timeout: float,
) -> str:
if status == "hard_timeout":
notice = cls._format_timeout_notice(timeout)
output = f"{output}\n{notice}" if output else notice
return f"{output}\nExit Code: 124"
if status == "no_change_timeout":
effective_no_change_timeout = cls._effective_no_change_timeout(timeout)
notice = f"Command produced no output change for {effective_no_change_timeout} seconds; it may still be running. Command outcome is unknown and was not retried."
return f"{output}\n{notice}" if output else notice
if status == "terminated":
notice = "Command was terminated because its shell session ended."
return f"{output}\n{notice}" if output else notice
if cls._is_unexpected_shell_status(status):
notice = cls._unexpected_status_notice(status)
return f"{output}\n{notice}" if output else notice
if exit_code not in (0, None):
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
return output if output else "(no output)"
@ -368,13 +536,13 @@ class AioSandbox(Sandbox):
self._home_dir = context.home_dir
return self._home_dir
# Default no_change_timeout for exec_command (seconds). Matches the
# client-level timeout so that long-running commands which produce no
# output are not prematurely terminated by the sandbox's built-in 120 s
# default.
# Default no_change_timeout base idle guard for exec_command (seconds).
# The per-command effective value is max(600, ceil(T + 5)); at the default
# T=600, the value sent to the sandbox is therefore 605, not 600.
_DEFAULT_NO_CHANGE_TIMEOUT = 600
# Wall-clock hard timeout for env-bearing commands routed through bash.exec.
# Fallback command hard timeout for both the legacy shell path and
# env-bearing commands routed through bash.exec when no provider default is set.
# The bash.exec API exposes no idle/no-change timeout (unlike
# shell.exec_command's ``no_change_timeout`` on the legacy path), so
# env-bearing commands are bounded by total elapsed wall-clock time, not
@ -383,6 +551,28 @@ class AioSandbox(Sandbox):
# run; a future SDK that exposes an idle timeout on bash.exec should switch
# this call site to it.
_DEFAULT_HARD_TIMEOUT = 600.0
_REQUEST_TIMEOUT_GRACE_SECONDS = 5.0
_CLEANUP_REQUEST_TIMEOUT_SECONDS = 5
def _effective_command_timeout(self, timeout: float | None) -> float:
return timeout if timeout is not None else (getattr(self, "_default_command_timeout", None) or self._DEFAULT_HARD_TIMEOUT)
@classmethod
def _effective_no_change_timeout(cls, timeout: float) -> int:
return max(
cls._DEFAULT_NO_CHANGE_TIMEOUT,
math.ceil(timeout + cls._REQUEST_TIMEOUT_GRACE_SECONDS),
)
@classmethod
def _command_request_options(cls, timeout: float) -> dict[str, int]:
return {
"timeout_in_seconds": max(
1,
math.ceil(timeout + cls._REQUEST_TIMEOUT_GRACE_SECONDS),
),
"max_retries": 0,
}
def execute_command(
self,
@ -406,41 +596,52 @@ class AioSandbox(Sandbox):
API (which supports per-command env) on a fresh explicitly released
session, so the secrets are scoped to this single command and never
persist; secret values travel in the structured ``env`` field, never
in the command string. When ``None`` the legacy persistent-shell path
runs unchanged.
timeout: Optional per-call timeout. The current sandbox SDK does not
expose a command-level timeout distinct from its client/request
timeout, so DeerFlow keeps using the backend's default here.
in the command string. When ``None``, the legacy persistent-shell
path still uses the provider-wide command timeout via
``hard_timeout``, bounded request options, and returned-status
interpretation.
timeout: Optional per-call command timeout. The legacy shell path
enforces it server-side and gives the request a small additional
response-path grace period.
Returns:
The output of the command.
"""
del timeout
# Validate ``env`` keys before forwarding them to the ``bash.exec`` API.
# The public ``Sandbox.execute_command`` contract accepts arbitrary dict
# keys; enforcing the POSIX env-var name rule keeps the contract
# consistent with the local and e2b sandboxes and catches unsafe keys
# early. ``_validate_extra_env`` is a no-op when ``env`` is None or empty.
_validate_extra_env(env)
effective_timeout = self._effective_command_timeout(timeout)
if env:
return self._execute_with_env(command, env)
return self._execute_with_env(command, env, effective_timeout)
with self._lock:
try:
client = self._client
if getattr(self, "_closed", False) or client is None:
raise RuntimeError("sandbox client is closed")
if self._default_shell_corrupted and self._recovery_session_id is None:
# Once the implicit session emits ErrorObservation, never
# target it again. A failed replacement is cleaned up and
# the next call starts another explicit session.
self._recovery_session_id = self._create_shell_session(client)
session_id = self._ensure_default_shell_session_id(client)
recovered_missing_session = False
try:
output, exit_code = self._exec_shell(
output, exit_code, status = self._exec_shell(
client,
command,
session_id=self._recovery_session_id,
session_id=session_id,
timeout=effective_timeout,
)
except httpx.TimeoutException:
session_id = self._recovery_session_id
self._recovery_session_id = None
self._default_shell_corrupted = True
if session_id is not None:
self._cleanup_session_best_effort(
client,
session_id,
context="default shell after transport timeout",
request_options=self._bounded_cleanup_request_options(),
)
return self._transport_timeout_error(effective_timeout)
except ApiError as error:
if not self._is_missing_shell_session_error(error):
raise
@ -448,29 +649,67 @@ class AioSandbox(Sandbox):
self._default_shell_corrupted = True
self._recovery_session_id = None
recovered_missing_session = True
output, exit_code, self._recovery_session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=None,
context="default shell after missing session",
)
try:
output, exit_code, status, self._recovery_session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=None,
context="default shell after missing session",
timeout=effective_timeout,
)
except httpx.TimeoutException:
return self._transport_timeout_error(effective_timeout)
if not recovered_missing_session and output and _ERROR_OBSERVATION_SIGNATURE in output:
if not recovered_missing_session and status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
self._default_shell_corrupted = True
logger.warning("ErrorObservation detected in sandbox output, retrying on a fresh session")
output, exit_code, self._recovery_session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=self._recovery_session_id,
context="default shell",
)
corrupted_session_id = self._recovery_session_id
self._recovery_session_id = None
try:
output, exit_code, status, self._recovery_session_id = self._rotate_and_retry_shell(
client,
command,
corrupted_session_id=corrupted_session_id,
context="default shell",
timeout=effective_timeout,
)
except httpx.TimeoutException:
return self._transport_timeout_error(effective_timeout)
return self._render_shell_output(output, exit_code)
if self._is_session_invalidating_shell_status(status):
session_id = self._recovery_session_id
self._recovery_session_id = None
self._default_shell_corrupted = True
if session_id is not None:
if status == "terminated":
cleanup_context = "default shell after terminal session loss"
elif status == "no_change_timeout":
cleanup_context = "default shell after ambiguous no-change timeout"
else:
cleanup_context = "default shell after unexpected status"
self._cleanup_session_best_effort(
client,
session_id,
context=cleanup_context,
request_options=self._bounded_cleanup_request_options(),
)
return self._render_shell_output(
output,
exit_code,
status=status,
timeout=effective_timeout,
)
except Exception as e:
logger.error(f"Failed to execute command in sandbox: {e}")
return f"Error: {e}"
def _execute_with_env(self, command: str, env: dict[str, str]) -> str:
def _execute_with_env(
self,
command: str,
env: dict[str, str],
timeout: float,
) -> str:
"""Execute a command with per-call environment variables injected.
The persistent-shell ``shell.exec_command`` API has no env parameter, so
@ -501,15 +740,22 @@ class AioSandbox(Sandbox):
"""
if self._bash_exec_unsupported:
return _BASH_EXEC_UNSUPPORTED_ERROR
output = self._run_bash_exec(command, env)
if output and _ERROR_OBSERVATION_SIGNATURE in output:
output, status = self._run_bash_exec(command, env, timeout)
if status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
logger.warning("ErrorObservation detected in bash.exec output, retrying on a fresh session")
retried = self._run_bash_exec(command, env)
retried, retry_status = self._run_bash_exec(command, env, timeout)
if retry_status not in (None, "completed"):
return retried
if retried and _ERROR_OBSERVATION_SIGNATURE not in retried:
return retried
return output
def _run_bash_exec(self, command: str, env: dict[str, str]) -> str:
def _run_bash_exec(
self,
command: str,
env: dict[str, str],
timeout: float,
) -> tuple[str, str | None]:
"""Single bash.exec invocation in an explicitly released fresh session."""
with self._lock:
for attempt in range(2):
@ -522,40 +768,65 @@ class AioSandbox(Sandbox):
command=command,
session_id=session_id,
env=env,
hard_timeout=self._DEFAULT_HARD_TIMEOUT,
hard_timeout=timeout,
request_options=self._command_request_options(timeout),
)
data = result.data if result else None
stdout = (data.stdout or "") if data else ""
stderr = (data.stderr or "") if data else ""
exit_code = getattr(data, "exit_code", None) if data else None
status = getattr(data, "status", None) if data else None
output = stdout
if stderr:
output += f"\nStd Error:\n{stderr}" if output else stderr
if status == "timed_out":
notice = self._format_timeout_notice(timeout)
output = f"{output}\n{notice}" if output else notice
return f"{output}\nExit Code: 124", status
if status == "killed":
notice = "Command was killed before completion."
output = f"{output}\n{notice}" if output else notice
return output, status
if status == "running":
notice = "Error: Sandbox command returned a non-terminal running status; command outcome is unknown and was not retried."
output = f"{output}\n{notice}" if output else notice
return output, status
if status not in (None, "completed"):
notice = self._unexpected_status_notice(status)
output = f"{output}\n{notice}" if output else notice
return output, status
if exit_code not in (0, None):
# Mirror LocalSandbox: keep the actual shell status in the
# output text (acceptance-checklist evidence).
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
return output if output else "(no output)"
return output if output else "(no output)", status
except httpx.TimeoutException:
return self._transport_timeout_error(timeout), "transport_timeout"
except ApiError as e:
if self._is_missing_shell_session_error(e):
if attempt == 0:
logger.warning("Transient bash.exec session disappeared; retrying once")
continue
logger.error("Failed to execute command with injected env: bash.exec session disappeared after retry")
return "Error: bash.exec session disappeared after retry"
return "Error: bash.exec session disappeared after retry", None
if e.status_code == 404:
self._bash_exec_unsupported = True
logger.error("Sandbox %s does not support bash.exec (/v1/bash/exec returned 404); env-bearing commands are unavailable until the sandbox image is upgraded to all-in-one-sandbox >= 1.9.3", self.id)
return _BASH_EXEC_UNSUPPORTED_ERROR
return _BASH_EXEC_UNSUPPORTED_ERROR, None
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
return f"Error: {e}"
return f"Error: {e}", None
except Exception as e:
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
return f"Error: {e}"
return f"Error: {e}", None
finally:
if session_created:
self._cleanup_bash_session_best_effort(self._client, session_id)
return "Error: bash.exec session disappeared after retry"
return "Error: bash.exec session disappeared after retry", None
def read_file(
self,
@ -639,10 +910,17 @@ class AioSandbox(Sandbox):
resolved = path
with self._lock:
try:
result = self._client.shell.exec_command(
command=remote_list_dir_command(resolved, max_depth),
no_change_timeout=self._DEFAULT_NO_CHANGE_TIMEOUT,
)
client = self._client
session_id = self._ensure_default_shell_session_id(client)
kwargs = {
"command": remote_list_dir_command(resolved, max_depth),
"no_change_timeout": self._DEFAULT_NO_CHANGE_TIMEOUT,
}
if session_id is not None:
kwargs["id"] = session_id
result = client.shell.exec_command(**kwargs)
except Exception as e:
logger.error(f"Failed to list directory in sandbox: {e}")
raise OSError(f"Failed to list directory '{resolved}' in sandbox: {e}") from e

View File

@ -15,12 +15,14 @@ import atexit
import contextlib
import hashlib
import logging
import math
import os
import signal
import threading
import time
import uuid
from collections.abc import Callable
from typing import Any
try:
import fcntl
@ -169,6 +171,16 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
# normally timing-out refresh + release still finishes synchronously.
_TEARDOWN_JOIN_TIMEOUT_SECONDS = 12.0
@staticmethod
def _positive_float(name: str, value: Any, default: float) -> float:
try:
resolved = float(default if value is None else value)
except (TypeError, ValueError) as exc:
raise ValueError(f"sandbox.{name} must be positive") from exc
if not math.isfinite(resolved) or resolved <= 0:
raise ValueError(f"sandbox.{name} must be positive")
return resolved
def __init__(self):
self._lock = threading.Lock()
self._sandboxes: dict[str, AioSandbox] = {} # sandbox_id -> AioSandbox instance
@ -301,6 +313,11 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
configured_skills_path = DEFAULT_SKILLS_CONTAINER_PATH
environment = self._resolve_env_vars(sandbox_config.environment or {})
command_timeout = self._positive_float(
"bash_command_timeout",
getattr(sandbox_config, "bash_command_timeout", None),
AioSandbox._DEFAULT_HARD_TIMEOUT,
)
max_running_subagents = int(getattr(getattr(config, "subagent_runtime", None), "max_running", 3))
required_shell_sessions = max_running_subagents + _SHELL_SESSION_HEADROOM
configured_shell_sessions = environment.get("MAX_SHELL_SESSIONS")
@ -321,6 +338,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
"port": sandbox_config.port or DEFAULT_PORT,
"container_prefix": sandbox_config.container_prefix or DEFAULT_CONTAINER_PREFIX,
"idle_timeout": idle_timeout if idle_timeout is not None else DEFAULT_IDLE_TIMEOUT,
"command_timeout": command_timeout,
"replicas": replicas if replicas is not None else DEFAULT_REPLICAS,
"mounts": sandbox_config.mounts or [],
"thread_data_mounts": getattr(sandbox_config, "thread_data_mounts", None),
@ -1686,7 +1704,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
return None
self._warm_pool_identity.pop(sandbox_id, None)
info, _ = warm_item
sandbox = AioSandbox(id=sandbox_id, base_url=info.sandbox_url, request_headers=info.request_headers)
sandbox = AioSandbox(
id=sandbox_id,
base_url=info.sandbox_url,
request_headers=info.request_headers,
default_command_timeout=self._config.get("command_timeout", AioSandbox._DEFAULT_HARD_TIMEOUT),
)
self._sandboxes[sandbox_id] = sandbox
self._sandbox_infos[sandbox_id] = info
self._active_sandbox_identity[sandbox_id] = key
@ -1731,7 +1754,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
self._assert_active_identity_available_locked(info.sandbox_id, key)
self._assert_warm_identity_available_locked(info.sandbox_id, key)
sandbox = AioSandbox(id=info.sandbox_id, base_url=info.sandbox_url, request_headers=info.request_headers)
sandbox = AioSandbox(
id=info.sandbox_id,
base_url=info.sandbox_url,
request_headers=info.request_headers,
default_command_timeout=self._config.get("command_timeout", AioSandbox._DEFAULT_HARD_TIMEOUT),
)
# Ownership first, so a failure cannot leave a tracked-but-unowned sandbox.
# There is no container to roll back (we did not create it), but the
# host-side HTTP client constructed above is ours and must not leak —
@ -1777,7 +1805,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
def _register_created_sandbox(self, thread_id: str | None, sandbox_id: str, info: SandboxInfo, *, user_id: str | None = None) -> str:
"""Track a newly-created sandbox in the active maps."""
sandbox = AioSandbox(id=sandbox_id, base_url=info.sandbox_url, request_headers=info.request_headers)
sandbox = AioSandbox(
id=sandbox_id,
base_url=info.sandbox_url,
request_headers=info.request_headers,
default_command_timeout=self._config.get("command_timeout", AioSandbox._DEFAULT_HARD_TIMEOUT),
)
key = (
self._thread_key(
thread_id,

View File

@ -258,13 +258,19 @@ class SandboxConfig(BaseModel):
ge=0,
description="Maximum characters to keep from ls tool output. Output exceeding this limit is head-truncated. Set to 0 to disable truncation.",
)
bash_command_timeout: int = Field(
bash_command_timeout: float = Field(
default=600,
gt=0,
allow_inf_nan=False,
description=(
"Maximum wall-clock seconds a bash command may run before it is terminated. LocalSandboxProvider applies it to the host process group; "
"OpenSandboxProvider forwards it to the remote exec service when a call has no explicit timeout. Keeps a blocking foreground command "
"(e.g. an un-backgrounded server) from hanging the turn; background `&` processes return immediately."
"Provider command deadline. AIO images on the supported semver line "
"(1.9.3+, recommended 1.11.0) enforce it server-side through "
"`hard_timeout`; the frozen legacy `all-in-one-sandbox:latest` image "
"only gets the bounded host-side request. `bash_command_timeout` is "
"used by providers that explicitly wire this setting (currently "
"LocalSandbox, AioSandbox, and OpenSandbox). Other providers retain "
"their provider-specific command defaults unless a caller supplies an "
"explicit timeout."
),
)

View File

@ -110,7 +110,7 @@ subshell. Keep parser filtering as a backstop; test ignored roots, metacharacter
symlinks, and visible depth/size bounds.
- Every sandbox tool keeps a model-visible `description` field for a human-readable progress label, but the field is optional and defaults to an empty string. Tool execution must depend only on its operational arguments; the frontend supplies localized fallback labels when a provider omits `description`.
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox` (host bash), output on POSIX and Windows is captured through bounded pipe-drain threads and stdin is `/dev/null`; Windows capture decodes with the platform text encoding and applies universal-newline translation, matching the former `subprocess.run(..., text=True)` behavior for locale-code-page output, Python UTF-8 Mode, CRLF, and bare CR. That translation is Windows-only so the pre-existing POSIX output contract remains byte-decoded without newline rewriting. On POSIX, a backgrounded long-lived process (`server &`) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (`sandbox.bash_command_timeout`, default 600s); on timeout the whole POSIX process group or Windows process tree is killed and the agent gets a notice telling it to background long-lived processes. The shared bash tool description scopes host environment detection to LocalSandbox: start with `uname -s`, follow with `sw_vers` on Darwin, and read Linux host system files only when the active policy permits them. Local path and `file://` rejections provide the same conditional recovery guidance: command-only probes for environment questions, allowed virtual paths otherwise, and no repetition of the rejected path. The description also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See `LocalSandbox.execute_command`, its platform runners, and `bash_tool`'s docstring.
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox`, POSIX/Windows output uses bounded pipe-drain threads with `/dev/null` stdin; Windows decodes locale-code-page/UTF-8/CRLF/bare-CR output with universal-newline translation, while POSIX stays byte-decoded. POSIX background commands return without blocking on inherited pipes; unredirected output is drained without unbounded temp files. Commands that read stdin get immediate EOF. `bash_command_timeout` sets T (600s) for Local/AIO/OpenSandbox; others keep defaults unless explicit. Local: T is a wall-clock process-group deadline. Supported-semver AIO: server-side `hard_timeout`; frozen legacy `all-in-one-sandbox:latest`: command requests get a bounded host `T+5s` wait (`max_retries=0`), so a wedged command request cannot hold the sandbox lock for the SDK's full 600s budget; session-creation and file/list RPCs are not bounded by it. Never replay ambiguous outcomes: transport timeout, terminated, no_change_timeout, unknown statuses. `hard_timeout` => terminated + Exit Code: 124, keep session; `no_change_timeout` => may still be running, fence session generation. The description scopes host-environment probes to LocalSandbox (`uname -s`, then `sw_vers` on Darwin; Linux files only when policy permits), gives conditional recovery for local path/`file://` rejections, and tells the model to background long-lived processes. See `LocalSandbox.execute_command`, its platform runners, and `bash_tool`'s docstring.
- `ls` - Directory listing (tree format, max 2 levels). Remote commands precheck root existence and emit `__DF_FIND_STATUS__:missing`; `find` status 1 is always an incomplete-traversal `OSError`, including when no entries were printed. Do not infer a missing path from status 1 alone.
- `glob` - Find files or directories below a root directory with bounded results
- `grep` - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results

File diff suppressed because it is too large Load Diff

View File

@ -12,6 +12,7 @@ from unittest.mock import MagicMock, patch
import pytest
from _windows_acl_helpers import _windows_acl_owner_sid, _windows_acl_sids
from pydantic import ValidationError
from deerflow.config.paths import Paths, join_host_path
from deerflow.config.sandbox_config import SandboxConfig
@ -68,6 +69,61 @@ def test_load_config_snapshots_custom_skills_container_path(monkeypatch):
assert provider._load_config()["skills_container_path"] == "/custom-skills"
def test_load_config_wires_bash_command_timeout_to_aio_default(monkeypatch):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
sandbox_config = SandboxConfig(
use="deerflow.community.aio_sandbox:AioSandboxProvider",
bash_command_timeout=42.5,
)
app_config = SimpleNamespace(sandbox=sandbox_config, stream_bridge=None)
monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config)
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
assert provider._load_config()["command_timeout"] == 42.5
@pytest.mark.parametrize("invalid_timeout", [float("nan"), float("inf"), float("-inf"), 0, -1])
def test_positive_float_rejects_non_positive_or_non_finite_values(invalid_timeout):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
with pytest.raises(ValueError, match="sandbox.bash_command_timeout must be positive"):
aio_mod.AioSandboxProvider._positive_float("bash_command_timeout", invalid_timeout, 600)
def test_positive_float_accepts_fractional_value():
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
assert aio_mod.AioSandboxProvider._positive_float("bash_command_timeout", 42.5, 600) == 42.5
@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf")])
def test_bash_command_timeout_rejects_non_finite_values(value):
with pytest.raises(ValidationError):
SandboxConfig(bash_command_timeout=value)
def test_register_created_sandbox_forwards_configured_command_timeout(tmp_path):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._config["command_timeout"] = 42
provider._warm_pool = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}
provider._last_activity = {}
provider._publish_ownership = MagicMock()
info = aio_mod.SandboxInfo(sandbox_id="sandbox-timeout", sandbox_url="http://sandbox")
with patch.object(aio_mod, "AioSandbox") as sandbox_cls:
provider._register_created_sandbox("thread-timeout", "sandbox-timeout", info, user_id="user-timeout")
sandbox_cls.assert_called_once_with(
id="sandbox-timeout",
base_url="http://sandbox",
request_headers=info.request_headers,
default_command_timeout=42,
)
def test_load_config_sizes_aio_shell_capacity_for_subagent_runtime(monkeypatch):
"""Twelve subagents must not exceed AIO 1.11's ten-session default."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
@ -231,7 +287,7 @@ def _make_provider(tmp_path):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
with patch.object(aio_mod.AioSandboxProvider, "_start_idle_checker"):
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
provider._config = {"idle_timeout": 600, "replicas": 3}
provider._config = {"command_timeout": 600.0, "idle_timeout": 600, "replicas": 3}
provider._sandboxes = {}
provider._active_sandbox_identity = {}
provider._warm_pool_identity = {}
@ -575,6 +631,7 @@ def test_policy_scoped_create_excludes_local_config_mounts_below_skills_root(
provider = _make_provider(tmp_path)
provider._config = {
"command_timeout": 600.0,
"replicas": 3,
"skills_container_path": "/mnt/skills",
}
@ -631,6 +688,7 @@ def test_remote_create_forwards_configured_skills_container_path(
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._config = {
"command_timeout": 600.0,
"replicas": 3,
"skills_container_path": "/custom-skills",
}
@ -693,6 +751,7 @@ async def test_remote_create_async_forwards_configured_skills_container_path(
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._config = {
"command_timeout": 600.0,
"replicas": 3,
"skills_container_path": "/custom-skills",
}
@ -811,7 +870,7 @@ async def test_acquire_async_uses_async_readiness_polling(monkeypatch):
"""AioSandboxProvider async creation must not use sync readiness polling."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(None)
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._warm_pool = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}
@ -1110,7 +1169,7 @@ def test_create_sandbox_requests_runtime_when_lark_installed(tmp_path, monkeypat
"""The provider must request lark-cli runtime provisioning when Lark is installed."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._warm_pool = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}
@ -1140,7 +1199,7 @@ def test_create_sandbox_requests_broker_when_active(tmp_path, monkeypatch):
"""Broker mode (Pattern B) is requested when the provisioner reports it."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._warm_pool = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}
@ -1170,7 +1229,7 @@ def test_create_sandbox_skips_runtime_when_lark_absent(tmp_path, monkeypatch):
"""No runtime provisioning request when the Lark skill pack is not installed."""
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._warm_pool = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}
@ -1305,7 +1364,7 @@ def test_acquire_drops_dead_cached_sandbox(tmp_path, monkeypatch):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider, sandbox, _ = _make_provider_with_active_sandbox(tmp_path, "sandbox-dead")
provider._thread_sandboxes = {("default", "thread-dead"): "sandbox-dead"}
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._backend.is_alive = MagicMock(return_value=False)
provider._backend.discover = MagicMock(return_value=None)
provider._backend.create = MagicMock(
@ -1389,7 +1448,7 @@ def test_acquire_skips_dead_warm_pool_sandbox(tmp_path, monkeypatch):
0.0,
)
}
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._backend = SimpleNamespace(
is_alive=MagicMock(return_value=False),
destroy=MagicMock(),
@ -1472,7 +1531,7 @@ def test_create_sandbox_evicts_oldest_warm_replica_via_shared_lifecycle(tmp_path
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
provider = _make_provider(tmp_path)
provider._lock = aio_mod.threading.Lock()
provider._config = {"replicas": 2}
provider._config = {"command_timeout": 600.0, "replicas": 2}
provider._sandboxes = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}
@ -1513,7 +1572,7 @@ def _make_tenant_isolation_provider(tmp_path, monkeypatch):
provider._active_sandbox_identity = {}
provider._warm_pool_identity = {}
provider._shutdown_called = False
provider._config = {"replicas": 3, "idle_timeout": 0}
provider._config = {"command_timeout": 600.0, "replicas": 3, "idle_timeout": 0}
create_calls = []
@ -1610,7 +1669,7 @@ def _make_unready_destroy_provider(tmp_path, *, sandbox_id, base_url, monkeypatc
"""
provider = _make_provider(tmp_path)
provider._lock = aio_mod.threading.Lock()
provider._config = {"replicas": 3}
provider._config = {"command_timeout": 600.0, "replicas": 3}
provider._warm_pool = {}
provider._sandbox_infos = {}
provider._thread_sandboxes = {}

View File

@ -119,7 +119,7 @@ def test_aio_sandbox_env_routes_through_bash_exec() -> None:
captured["exec_session"] = kwargs["session_id"]
return SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None))
def close_session(self, session_id):
def close_session(self, session_id, **kwargs):
captured["closed_session"] = session_id
sbx = AioSandbox.__new__(AioSandbox)

View File

@ -13,6 +13,7 @@ from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
import httpx
import pytest
from langchain.agents.middleware.types import ModelRequest
from langchain_core.messages import AIMessage, HumanMessage
@ -101,7 +102,7 @@ class TestAioSandboxEnvInjection:
return AioSandbox(id="test-sandbox", base_url="http://localhost:8080")
def test_env_none_uses_legacy_shell_path(self, sandbox):
"""No injected env → unchanged shell.exec_command path (backward compat)."""
"""No injected env uses the legacy shell path with bounded timeout/status handling."""
sandbox._client.shell.exec_command = MagicMock(return_value=SimpleNamespace(data=SimpleNamespace(output="hello")))
sandbox._client.bash.exec = MagicMock()
out = sandbox.execute_command("echo hello")
@ -122,23 +123,195 @@ class TestAioSandboxEnvInjection:
sandbox._client.shell.exec_command.assert_not_called()
assert "hello" in out
def test_env_path_uses_hard_timeout_not_no_change_timeout(self, sandbox):
"""The env path routes through bash.exec which exposes no idle/no-change
timeout; it must use the dedicated wall-clock ``_DEFAULT_HARD_TIMEOUT``,
not the legacy idle constant (same numeric value today, but distinct
semantics so a future change to one does not silently alter the other)."""
from deerflow.community.aio_sandbox.aio_sandbox import AioSandbox
sandbox._client.bash.exec = MagicMock(return_value=SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None)))
sandbox.execute_command("echo hi", env={"X": "1"})
_, kwargs = sandbox._client.bash.exec.call_args
assert kwargs["hard_timeout"] == AioSandbox._DEFAULT_HARD_TIMEOUT
assert AioSandbox._DEFAULT_HARD_TIMEOUT != AioSandbox._DEFAULT_NO_CHANGE_TIMEOUT or (
# Same numeric value is fine today; the contract is that they are
# named independently so the two call sites evolve independently.
AioSandbox._DEFAULT_HARD_TIMEOUT == AioSandbox._DEFAULT_NO_CHANGE_TIMEOUT
def test_env_path_uses_explicit_command_timeout_and_request_budget(self, sandbox):
sandbox._client.bash.exec = MagicMock(
return_value=SimpleNamespace(
data=SimpleNamespace(
stdout="ok",
stderr=None,
exit_code=0,
status="completed",
)
)
)
sandbox.execute_command(
"echo hi",
env={"X": "1"},
timeout=3,
)
_, kwargs = sandbox._client.bash.exec.call_args
assert kwargs["hard_timeout"] == 3
assert kwargs["request_options"] == {
"timeout_in_seconds": 8,
"max_retries": 0,
}
def test_env_path_uses_default_hard_timeout_when_timeout_is_none(self, sandbox):
sandbox._client.bash.exec = MagicMock(
return_value=SimpleNamespace(
data=SimpleNamespace(
stdout="ok",
stderr=None,
exit_code=0,
status="completed",
)
)
)
sandbox.execute_command("echo hi", env={"X": "1"})
_, kwargs = sandbox._client.bash.exec.call_args
assert kwargs["hard_timeout"] == sandbox._DEFAULT_HARD_TIMEOUT
def test_env_hard_timeout_is_rendered_and_not_retried(self, sandbox):
executions = 0
def bash_exec(**kwargs):
nonlocal executions
executions += 1
return SimpleNamespace(
data=SimpleNamespace(
stdout="partial",
stderr=None,
exit_code=-1,
status="timed_out",
)
)
sandbox._client.bash.exec = bash_exec
out = sandbox.execute_command(
"side-effect; sleep 30",
env={"X": "1"},
timeout=3,
)
assert executions == 1
assert "partial" in out
assert "Command timed out after 3 seconds and was terminated." in out
assert out.endswith("Exit Code: 124")
@pytest.mark.parametrize("status", ["timed_out", "killed"])
def test_env_interrupted_status_is_never_error_observation_retried(
self,
sandbox,
status,
):
executions = 0
def bash_exec(**kwargs):
nonlocal executions
executions += 1
return SimpleNamespace(
data=SimpleNamespace(
stdout="'ErrorObservation' object has no attribute 'exit_code'",
stderr=None,
exit_code=-1,
status=status,
)
)
sandbox._client.bash.exec = bash_exec
sandbox.execute_command(
"unsafe-to-repeat",
env={"X": "1"},
timeout=3,
)
assert executions == 1
def test_env_session_cleanup_is_bounded_and_does_not_mask_output(self, sandbox):
sandbox._client.bash.exec = MagicMock(
return_value=SimpleNamespace(
data=SimpleNamespace(
stdout="ok",
stderr="",
exit_code=0,
status="completed",
)
)
)
sandbox._client.bash.close_session = MagicMock(side_effect=RuntimeError("cleanup failed"))
assert (
sandbox.execute_command(
"echo $TOKEN",
env={"TOKEN": "secret"},
timeout=3,
)
== "ok"
)
_, kwargs = sandbox._client.bash.close_session.call_args
assert kwargs["request_options"] == {
"timeout_in_seconds": sandbox._CLEANUP_REQUEST_TIMEOUT_SECONDS,
"max_retries": 0,
}
@pytest.mark.parametrize(
("status", "notice"),
[
("killed", "Command was killed before completion."),
(
"running",
"Error: Sandbox command returned a non-terminal running status; command outcome is unknown and was not retried.",
),
],
)
def test_env_interrupted_status_is_rendered_without_exit_fallback(
self,
sandbox,
status,
notice,
):
sandbox._client.bash.exec = MagicMock(
return_value=SimpleNamespace(
data=SimpleNamespace(
stdout="partial",
stderr=None,
exit_code=-1,
status=status,
)
)
)
out = sandbox.execute_command(
"unsafe-to-repeat",
env={"X": "1"},
timeout=3,
)
assert notice in out
assert "Exit Code: -1" not in out
def test_env_transport_timeout_is_ambiguous_and_not_retried(self, sandbox):
executions = 0
def bash_exec(**kwargs):
nonlocal executions
executions += 1
raise httpx.ReadTimeout("response stalled")
sandbox._client.bash.exec = bash_exec
out = sandbox.execute_command(
"side-effect; sleep 30",
env={"X": "1"},
timeout=3,
)
assert executions == 1
assert "outcome is unknown" in out
assert "not retried" in out
_, kwargs = sandbox._client.bash.close_session.call_args
assert kwargs["request_options"] == {
"timeout_in_seconds": sandbox._CLEANUP_REQUEST_TIMEOUT_SECONDS,
"max_retries": 0,
}
def test_env_path_retries_on_error_observation_signature(self, sandbox):
"""The env path shares the legacy persistent-shell recovery contract: if
the (unlikely, fresh-session) corruption marker appears, the call is
@ -153,6 +326,89 @@ class TestAioSandboxEnvInjection:
assert "recovered" in out
assert _ERROR_OBSERVATION_SIGNATURE not in out
@pytest.mark.parametrize(
("retry_result", "expected_fragments", "expected_suffix"),
[
pytest.param(
SimpleNamespace(
data=SimpleNamespace(
stdout="authoritative timeout output",
stderr=None,
exit_code=-1,
status="timed_out",
)
),
("Command timed out after 3 seconds and was terminated.",),
"Exit Code: 124",
id="timed-out",
),
pytest.param(
SimpleNamespace(
data=SimpleNamespace(
stdout="authoritative killed output",
stderr=None,
exit_code=-1,
status="killed",
)
),
("Command was killed before completion.",),
None,
id="killed",
),
pytest.param(
SimpleNamespace(
data=SimpleNamespace(
stdout="authoritative running output",
stderr=None,
exit_code=-1,
status="running",
)
),
(
"non-terminal running status",
"outcome is unknown",
"not retried",
),
None,
id="running",
),
pytest.param(
httpx.ReadTimeout("response stalled"),
("outcome is unknown", "not retried"),
None,
id="transport-timeout",
),
],
)
def test_env_error_observation_retry_returns_authoritative_outcome(
self,
sandbox,
retry_result,
expected_fragments,
expected_suffix,
):
from deerflow.community.aio_sandbox.aio_sandbox import _ERROR_OBSERVATION_SIGNATURE
corrupted = SimpleNamespace(
data=SimpleNamespace(
stdout=f"corrupted: {_ERROR_OBSERVATION_SIGNATURE}",
stderr=None,
exit_code=0,
status="completed",
)
)
sandbox._client.bash.exec = MagicMock(side_effect=[corrupted, retry_result])
out = sandbox.execute_command("unsafe-to-repeat", env={"X": "1"}, timeout=3)
assert sandbox._client.bash.exec.call_count == 2
assert "corrupted:" not in out
assert _ERROR_OBSERVATION_SIGNATURE not in out
for expected_fragment in expected_fragments:
assert expected_fragment in out
if expected_suffix is not None:
assert out.endswith(expected_suffix)
class TestEnvPolicy:
"""Platform-secret scrubbing policy for sandbox subprocesses (delta 1)."""
@ -1077,6 +1333,55 @@ class TestBashToolInjectsActiveSecrets:
assert captured["env"] == {"ERP_TOKEN": "tok-456"}
assert captured["timeout"] == 42
def test_remote_bash_does_not_forward_shared_timeout(self):
from deerflow.sandbox import tools as tools_mod
captured = {}
class FakeSandbox:
def execute_command(self, command, env=None, timeout=None):
captured["command"] = command
captured["env"] = env
captured["timeout"] = timeout
return "done"
runtime = SimpleNamespace(
context={},
state={"sandbox": {"sandbox_id": "aio:1"}},
)
fake_cfg = SimpleNamespace(
sandbox=SimpleNamespace(
bash_output_max_chars=321,
bash_command_timeout=42,
)
)
with (
patch.object(
tools_mod,
"ensure_sandbox_initialized",
return_value=FakeSandbox(),
),
patch.object(tools_mod, "is_local_sandbox", return_value=False),
patch.object(
tools_mod,
"ensure_thread_directories_exist",
return_value=None,
),
patch(
"deerflow.config.app_config.get_app_config",
return_value=fake_cfg,
),
):
out = tools_mod.bash_tool.func(
runtime=runtime,
command="echo hi",
description="run remote",
)
assert out == "done"
assert captured["timeout"] is None
_SECRET = "sk-erp-9f3c-DO-NOT-LEAK"

View File

@ -1500,13 +1500,17 @@ sandbox:
read_file_output_max_chars: 50000
ls_output_max_chars: 20000
# Maximum wall-clock seconds a single host bash command may run before it is
# terminated (process group and all). A blocking foreground command — e.g. a
# server started without backgrounding — is killed after this long so the
# agent's turn cannot hang. Start long-lived processes in the background with
# output redirected (e.g. `your-command > /tmp/server.log 2>&1 &`) when you
# need logs; unredirected background output is drained with bounded capture
# and excess output is discarded.
# Provider command deadline (default: 600). AIO images on the supported
# semver line (1.9.3+, recommended 1.11.0) enforce it server-side through
# `hard_timeout`; the frozen legacy `all-in-one-sandbox:latest` image only
# gets the bounded host-side request. `bash_command_timeout` is used by
# providers that explicitly wire this setting (currently LocalSandbox,
# AioSandbox, and OpenSandbox). Other providers retain their provider-specific
# command defaults unless a caller supplies an explicit timeout. For providers
# that enforce this deadline server- or process-side, blocking foreground
# commands are terminated at the configured deadline. Long-lived processes
# should be started in the background with output redirected; unredirected
# background output is drained with bounded capture.
bash_command_timeout: 600
# Option 2: Container-based AIO Sandbox