mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-25 14:06:18 +00:00
fix(sandbox): enforce bash command timeouts in AIO sandbox (#5634)
* fix(sandbox): enforce AIO shell command timeouts Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results. * fix(sandbox): contain ambiguous AIO command outcomes Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative. * fix(sandbox): wire AIO command timeout from provider config Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working. * docs(sandbox): document AIO command timeout semantics Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget. * docs(sandbox): scope the wedge-resistance claim to command requests willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up. * fix(sandbox): make list_dir honor the shell generation fence list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.
This commit is contained in:
parent
e2f19d8335
commit
61a99a1e8a
@ -633,6 +633,13 @@ sandbox:
|
|||||||
use: deerflow.community.aio_sandbox:AioSandboxProvider # Docker-based sandbox
|
use: deerflow.community.aio_sandbox:AioSandboxProvider # Docker-based sandbox
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For AIO images on the supported semver line (`1.9.3` through the recommended
|
||||||
|
`1.11.0` image), `sandbox.bash_command_timeout` is enforced server-side through
|
||||||
|
the `hard_timeout` API when the image exposes it. DeerFlow's legacy frozen
|
||||||
|
`all-in-one-sandbox:latest` image predates that API, so only the host-side
|
||||||
|
request is bounded there. Timed-out or otherwise ambiguous commands are never
|
||||||
|
replayed.
|
||||||
|
|
||||||
**BoxLite micro-VM Sandbox** (runs sandbox code in daemonless OCI micro-VMs):
|
**BoxLite micro-VM Sandbox** (runs sandbox code in daemonless OCI micro-VMs):
|
||||||
```yaml
|
```yaml
|
||||||
sandbox:
|
sandbox:
|
||||||
|
|||||||
@ -1,6 +1,7 @@
|
|||||||
import base64
|
import base64
|
||||||
import errno
|
import errno
|
||||||
import logging
|
import logging
|
||||||
|
import math
|
||||||
import threading
|
import threading
|
||||||
import uuid
|
import uuid
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
@ -65,6 +66,7 @@ class AioSandbox(Sandbox):
|
|||||||
base_url: str,
|
base_url: str,
|
||||||
home_dir: str | None = None,
|
home_dir: str | None = None,
|
||||||
request_headers: dict[str, str] | None = None,
|
request_headers: dict[str, str] | None = None,
|
||||||
|
default_command_timeout: float | None = None,
|
||||||
):
|
):
|
||||||
"""Initialize the AIO sandbox.
|
"""Initialize the AIO sandbox.
|
||||||
|
|
||||||
@ -74,8 +76,20 @@ class AioSandbox(Sandbox):
|
|||||||
home_dir: Home directory inside the sandbox. If None, will be fetched from the sandbox.
|
home_dir: Home directory inside the sandbox. If None, will be fetched from the sandbox.
|
||||||
request_headers: Trusted control-plane headers required by a local
|
request_headers: Trusted control-plane headers required by a local
|
||||||
relay. These are never injected into sandbox commands.
|
relay. These are never injected into sandbox commands.
|
||||||
|
default_command_timeout: Provider-configured command deadline used
|
||||||
|
when a command does not provide an explicit timeout.
|
||||||
"""
|
"""
|
||||||
super().__init__(id)
|
super().__init__(id)
|
||||||
|
if default_command_timeout is None:
|
||||||
|
self._default_command_timeout = self._DEFAULT_HARD_TIMEOUT
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
resolved_default_timeout = float(default_command_timeout)
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise ValueError("default_command_timeout must be positive") from exc
|
||||||
|
if not math.isfinite(resolved_default_timeout) or resolved_default_timeout <= 0:
|
||||||
|
raise ValueError("default_command_timeout must be positive")
|
||||||
|
self._default_command_timeout = resolved_default_timeout
|
||||||
self._base_url = base_url
|
self._base_url = base_url
|
||||||
client_kwargs = {
|
client_kwargs = {
|
||||||
"base_url": base_url,
|
"base_url": base_url,
|
||||||
@ -179,9 +193,18 @@ class AioSandbox(Sandbox):
|
|||||||
logger.warning(f"Error closing AioSandbox client for {self.id}: {e}")
|
logger.warning(f"Error closing AioSandbox client for {self.id}: {e}")
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _cleanup_session_best_effort(client, session_id: str, *, context: str) -> None:
|
def _cleanup_session_best_effort(
|
||||||
|
client,
|
||||||
|
session_id: str,
|
||||||
|
*,
|
||||||
|
context: str,
|
||||||
|
request_options: dict[str, int] | None = None,
|
||||||
|
) -> None:
|
||||||
try:
|
try:
|
||||||
client.shell.cleanup_session(session_id)
|
client.shell.cleanup_session(
|
||||||
|
session_id,
|
||||||
|
**({"request_options": request_options} if request_options is not None else {}),
|
||||||
|
)
|
||||||
except Exception as cleanup_error:
|
except Exception as cleanup_error:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"Failed to release shell session %s (%s): %s",
|
"Failed to release shell session %s (%s): %s",
|
||||||
@ -191,11 +214,12 @@ class AioSandbox(Sandbox):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _format_shell_result(result) -> tuple[str, int | None]:
|
def _format_shell_result(result) -> tuple[str, int | None, str | None]:
|
||||||
data = result.data if result else None
|
data = result.data if result else None
|
||||||
output = data.output if data else ""
|
output = data.output if data else ""
|
||||||
exit_code = getattr(data, "exit_code", None) if data else None
|
exit_code = getattr(data, "exit_code", None) if data else None
|
||||||
return output, exit_code
|
status = getattr(data, "status", None) if data else None
|
||||||
|
return output, exit_code, status
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _is_missing_shell_session_error(error: ApiError) -> bool:
|
def _is_missing_shell_session_error(error: ApiError) -> bool:
|
||||||
@ -205,10 +229,12 @@ class AioSandbox(Sandbox):
|
|||||||
message = body.get("message")
|
message = body.get("message")
|
||||||
return isinstance(message, str) and "session not found" in message.casefold()
|
return isinstance(message, str) and "session not found" in message.casefold()
|
||||||
|
|
||||||
@staticmethod
|
def _cleanup_bash_session_best_effort(self, client, session_id: str) -> None:
|
||||||
def _cleanup_bash_session_best_effort(client, session_id: str) -> None:
|
|
||||||
try:
|
try:
|
||||||
client.bash.close_session(session_id)
|
client.bash.close_session(
|
||||||
|
session_id,
|
||||||
|
request_options=self._bounded_cleanup_request_options(),
|
||||||
|
)
|
||||||
except Exception as cleanup_error:
|
except Exception as cleanup_error:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"Failed to release transient bash session %s: %s",
|
"Failed to release transient bash session %s: %s",
|
||||||
@ -221,10 +247,32 @@ class AioSandbox(Sandbox):
|
|||||||
client.shell.create_session(id=session_id)
|
client.shell.create_session(id=session_id)
|
||||||
return session_id
|
return session_id
|
||||||
|
|
||||||
def _exec_shell(self, client, command: str, *, session_id: str | None) -> tuple[str, int | None]:
|
def _ensure_default_shell_session_id(self, client) -> str | None:
|
||||||
|
"""Return the session id that may safely receive default-shell work.
|
||||||
|
|
||||||
|
A healthy implicit shell is represented by ``None``. Once that generation
|
||||||
|
is fenced, all later shell-backed operations must target an explicit
|
||||||
|
recovery session instead of re-entering the implicit shell.
|
||||||
|
|
||||||
|
Caller must hold ``self._lock``.
|
||||||
|
"""
|
||||||
|
if self._default_shell_corrupted and self._recovery_session_id is None:
|
||||||
|
self._recovery_session_id = self._create_shell_session(client)
|
||||||
|
return self._recovery_session_id
|
||||||
|
|
||||||
|
def _exec_shell(
|
||||||
|
self,
|
||||||
|
client,
|
||||||
|
command: str,
|
||||||
|
*,
|
||||||
|
session_id: str | None,
|
||||||
|
timeout: float,
|
||||||
|
) -> tuple[str, int | None, str | None]:
|
||||||
kwargs = {
|
kwargs = {
|
||||||
"command": command,
|
"command": command,
|
||||||
"no_change_timeout": self._DEFAULT_NO_CHANGE_TIMEOUT,
|
"no_change_timeout": self._effective_no_change_timeout(timeout),
|
||||||
|
"hard_timeout": timeout,
|
||||||
|
"request_options": self._command_request_options(timeout),
|
||||||
}
|
}
|
||||||
if session_id is not None:
|
if session_id is not None:
|
||||||
kwargs["id"] = session_id
|
kwargs["id"] = session_id
|
||||||
@ -237,35 +285,55 @@ class AioSandbox(Sandbox):
|
|||||||
*,
|
*,
|
||||||
corrupted_session_id: str | None,
|
corrupted_session_id: str | None,
|
||||||
context: str,
|
context: str,
|
||||||
) -> tuple[str, int | None, str | None]:
|
timeout: float,
|
||||||
|
) -> tuple[str, int | None, str | None, str | None]:
|
||||||
|
cleanup_options = self._bounded_cleanup_request_options()
|
||||||
if corrupted_session_id is not None:
|
if corrupted_session_id is not None:
|
||||||
self._cleanup_session_best_effort(
|
self._cleanup_session_best_effort(
|
||||||
client,
|
client,
|
||||||
corrupted_session_id,
|
corrupted_session_id,
|
||||||
context=f"corrupted {context}",
|
context=f"corrupted {context}",
|
||||||
|
request_options=cleanup_options,
|
||||||
)
|
)
|
||||||
replacement_id = self._create_shell_session(client)
|
replacement_id = self._create_shell_session(client)
|
||||||
try:
|
try:
|
||||||
output, exit_code = self._exec_shell(
|
output, exit_code, status = self._exec_shell(
|
||||||
client,
|
client,
|
||||||
command,
|
command,
|
||||||
session_id=replacement_id,
|
session_id=replacement_id,
|
||||||
|
timeout=timeout,
|
||||||
)
|
)
|
||||||
except BaseException:
|
except BaseException:
|
||||||
self._cleanup_session_best_effort(
|
self._cleanup_session_best_effort(
|
||||||
client,
|
client,
|
||||||
replacement_id,
|
replacement_id,
|
||||||
context=f"abandoned replacement for {context}",
|
context=f"abandoned replacement for {context}",
|
||||||
|
request_options=cleanup_options,
|
||||||
)
|
)
|
||||||
raise
|
raise
|
||||||
if output and _ERROR_OBSERVATION_SIGNATURE in output:
|
if self._is_session_invalidating_shell_status(status):
|
||||||
|
if status == "terminated":
|
||||||
|
cleanup_context = f"terminated replacement for {context}"
|
||||||
|
elif status == "no_change_timeout":
|
||||||
|
cleanup_context = f"ambiguous replacement for {context}"
|
||||||
|
else:
|
||||||
|
cleanup_context = f"unexpected replacement status for {context}"
|
||||||
|
self._cleanup_session_best_effort(
|
||||||
|
client,
|
||||||
|
replacement_id,
|
||||||
|
context=cleanup_context,
|
||||||
|
request_options=cleanup_options,
|
||||||
|
)
|
||||||
|
return output, exit_code, status, None
|
||||||
|
if status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
|
||||||
self._cleanup_session_best_effort(
|
self._cleanup_session_best_effort(
|
||||||
client,
|
client,
|
||||||
replacement_id,
|
replacement_id,
|
||||||
context=f"failed replacement for {context}",
|
context=f"failed replacement for {context}",
|
||||||
|
request_options=cleanup_options,
|
||||||
)
|
)
|
||||||
return output, exit_code, None
|
return output, exit_code, status, None
|
||||||
return output, exit_code, replacement_id
|
return output, exit_code, status, replacement_id
|
||||||
|
|
||||||
def execute_command_in_scope(
|
def execute_command_in_scope(
|
||||||
self,
|
self,
|
||||||
@ -283,7 +351,6 @@ class AioSandbox(Sandbox):
|
|||||||
"""
|
"""
|
||||||
if env or scope_id is None:
|
if env or scope_id is None:
|
||||||
return self.execute_command(command, env=env, timeout=timeout)
|
return self.execute_command(command, env=env, timeout=timeout)
|
||||||
del timeout
|
|
||||||
_validate_extra_env(env)
|
_validate_extra_env(env)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@ -309,31 +376,75 @@ class AioSandbox(Sandbox):
|
|||||||
if scoped.session_id is None:
|
if scoped.session_id is None:
|
||||||
scoped.session_id = self._create_shell_session(client)
|
scoped.session_id = self._create_shell_session(client)
|
||||||
try:
|
try:
|
||||||
output, exit_code = self._exec_shell(
|
effective_timeout = self._effective_command_timeout(timeout)
|
||||||
|
output, exit_code, status = self._exec_shell(
|
||||||
client,
|
client,
|
||||||
command,
|
command,
|
||||||
session_id=scoped.session_id,
|
session_id=scoped.session_id,
|
||||||
|
timeout=effective_timeout,
|
||||||
)
|
)
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
session_id = scoped.session_id
|
||||||
|
scoped.session_id = None
|
||||||
|
if session_id is not None:
|
||||||
|
self._cleanup_session_best_effort(
|
||||||
|
client,
|
||||||
|
session_id,
|
||||||
|
context="execution scope after transport timeout",
|
||||||
|
request_options=self._bounded_cleanup_request_options(),
|
||||||
|
)
|
||||||
|
return self._transport_timeout_error(effective_timeout)
|
||||||
except ApiError as error:
|
except ApiError as error:
|
||||||
if not self._is_missing_shell_session_error(error):
|
if not self._is_missing_shell_session_error(error):
|
||||||
raise
|
raise
|
||||||
logger.warning("Execution-scoped sandbox shell session is missing; recreating it once")
|
logger.warning("Execution-scoped sandbox shell session is missing; recreating it once")
|
||||||
scoped.session_id = None
|
scoped.session_id = None
|
||||||
output, exit_code, scoped.session_id = self._rotate_and_retry_shell(
|
try:
|
||||||
client,
|
output, exit_code, status, scoped.session_id = self._rotate_and_retry_shell(
|
||||||
command,
|
client,
|
||||||
corrupted_session_id=None,
|
command,
|
||||||
context="execution scope after missing session",
|
corrupted_session_id=None,
|
||||||
)
|
context="execution scope after missing session",
|
||||||
if scoped.session_id is not None and output and _ERROR_OBSERVATION_SIGNATURE in output:
|
timeout=effective_timeout,
|
||||||
|
)
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
return self._transport_timeout_error(effective_timeout)
|
||||||
|
if self._is_session_invalidating_shell_status(status):
|
||||||
|
session_id = scoped.session_id
|
||||||
|
scoped.session_id = None
|
||||||
|
if session_id is not None:
|
||||||
|
if status == "terminated":
|
||||||
|
cleanup_context = "execution scope after terminal session loss"
|
||||||
|
elif status == "no_change_timeout":
|
||||||
|
cleanup_context = "execution scope after ambiguous no-change timeout"
|
||||||
|
else:
|
||||||
|
cleanup_context = "execution scope after unexpected status"
|
||||||
|
self._cleanup_session_best_effort(
|
||||||
|
client,
|
||||||
|
session_id,
|
||||||
|
context=cleanup_context,
|
||||||
|
request_options=self._bounded_cleanup_request_options(),
|
||||||
|
)
|
||||||
|
if scoped.session_id is not None and status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
|
||||||
logger.warning("ErrorObservation detected in sandbox output for execution scope; rotating session")
|
logger.warning("ErrorObservation detected in sandbox output for execution scope; rotating session")
|
||||||
output, exit_code, scoped.session_id = self._rotate_and_retry_shell(
|
corrupted_session_id = scoped.session_id
|
||||||
client,
|
scoped.session_id = None
|
||||||
command,
|
try:
|
||||||
corrupted_session_id=scoped.session_id,
|
output, exit_code, status, scoped.session_id = self._rotate_and_retry_shell(
|
||||||
context="execution scope",
|
client,
|
||||||
)
|
command,
|
||||||
return self._render_shell_output(output, exit_code)
|
corrupted_session_id=corrupted_session_id,
|
||||||
|
context="execution scope",
|
||||||
|
timeout=effective_timeout,
|
||||||
|
)
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
return self._transport_timeout_error(effective_timeout)
|
||||||
|
return self._render_shell_output(
|
||||||
|
output,
|
||||||
|
exit_code,
|
||||||
|
status=status,
|
||||||
|
timeout=effective_timeout,
|
||||||
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to execute command in sandbox: {e}")
|
logger.error(f"Failed to execute command in sandbox: {e}")
|
||||||
return f"Error: {e}"
|
return f"Error: {e}"
|
||||||
@ -355,7 +466,64 @@ class AioSandbox(Sandbox):
|
|||||||
scoped.session_id = None
|
scoped.session_id = None
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _render_shell_output(output: str, exit_code: int | None) -> str:
|
def _format_timeout_duration(timeout: float) -> str:
|
||||||
|
return f"{timeout:g}"
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _format_timeout_notice(cls, timeout: float) -> str:
|
||||||
|
return f"Command timed out after {cls._format_timeout_duration(timeout)} seconds and was terminated."
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _bounded_cleanup_request_options(cls) -> dict[str, int]:
|
||||||
|
return {
|
||||||
|
"timeout_in_seconds": cls._CLEANUP_REQUEST_TIMEOUT_SECONDS,
|
||||||
|
"max_retries": 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _transport_timeout_error(cls, timeout: float) -> str:
|
||||||
|
request_timeout = cls._command_request_options(timeout)["timeout_in_seconds"]
|
||||||
|
return f"Error: Sandbox command response timed out after {request_timeout} seconds; command outcome is unknown and the command was not retried."
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_unexpected_shell_status(status: str | None) -> bool:
|
||||||
|
return status not in (None, "completed", "hard_timeout", "no_change_timeout", "terminated")
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _is_session_invalidating_shell_status(cls, status: str | None) -> bool:
|
||||||
|
return status in ("terminated", "no_change_timeout") or cls._is_unexpected_shell_status(status)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _unexpected_status_notice(cls, status: str) -> str:
|
||||||
|
return f"Error: Sandbox command returned an unexpected status '{status}'; command outcome is unknown and the command was not retried."
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _render_shell_output(
|
||||||
|
cls,
|
||||||
|
output: str,
|
||||||
|
exit_code: int | None,
|
||||||
|
*,
|
||||||
|
status: str | None,
|
||||||
|
timeout: float,
|
||||||
|
) -> str:
|
||||||
|
if status == "hard_timeout":
|
||||||
|
notice = cls._format_timeout_notice(timeout)
|
||||||
|
output = f"{output}\n{notice}" if output else notice
|
||||||
|
return f"{output}\nExit Code: 124"
|
||||||
|
|
||||||
|
if status == "no_change_timeout":
|
||||||
|
effective_no_change_timeout = cls._effective_no_change_timeout(timeout)
|
||||||
|
notice = f"Command produced no output change for {effective_no_change_timeout} seconds; it may still be running. Command outcome is unknown and was not retried."
|
||||||
|
return f"{output}\n{notice}" if output else notice
|
||||||
|
|
||||||
|
if status == "terminated":
|
||||||
|
notice = "Command was terminated because its shell session ended."
|
||||||
|
return f"{output}\n{notice}" if output else notice
|
||||||
|
|
||||||
|
if cls._is_unexpected_shell_status(status):
|
||||||
|
notice = cls._unexpected_status_notice(status)
|
||||||
|
return f"{output}\n{notice}" if output else notice
|
||||||
|
|
||||||
if exit_code not in (0, None):
|
if exit_code not in (0, None):
|
||||||
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
|
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
|
||||||
return output if output else "(no output)"
|
return output if output else "(no output)"
|
||||||
@ -368,13 +536,13 @@ class AioSandbox(Sandbox):
|
|||||||
self._home_dir = context.home_dir
|
self._home_dir = context.home_dir
|
||||||
return self._home_dir
|
return self._home_dir
|
||||||
|
|
||||||
# Default no_change_timeout for exec_command (seconds). Matches the
|
# Default no_change_timeout base idle guard for exec_command (seconds).
|
||||||
# client-level timeout so that long-running commands which produce no
|
# The per-command effective value is max(600, ceil(T + 5)); at the default
|
||||||
# output are not prematurely terminated by the sandbox's built-in 120 s
|
# T=600, the value sent to the sandbox is therefore 605, not 600.
|
||||||
# default.
|
|
||||||
_DEFAULT_NO_CHANGE_TIMEOUT = 600
|
_DEFAULT_NO_CHANGE_TIMEOUT = 600
|
||||||
|
|
||||||
# Wall-clock hard timeout for env-bearing commands routed through bash.exec.
|
# Fallback command hard timeout for both the legacy shell path and
|
||||||
|
# env-bearing commands routed through bash.exec when no provider default is set.
|
||||||
# The bash.exec API exposes no idle/no-change timeout (unlike
|
# The bash.exec API exposes no idle/no-change timeout (unlike
|
||||||
# shell.exec_command's ``no_change_timeout`` on the legacy path), so
|
# shell.exec_command's ``no_change_timeout`` on the legacy path), so
|
||||||
# env-bearing commands are bounded by total elapsed wall-clock time, not
|
# env-bearing commands are bounded by total elapsed wall-clock time, not
|
||||||
@ -383,6 +551,28 @@ class AioSandbox(Sandbox):
|
|||||||
# run; a future SDK that exposes an idle timeout on bash.exec should switch
|
# run; a future SDK that exposes an idle timeout on bash.exec should switch
|
||||||
# this call site to it.
|
# this call site to it.
|
||||||
_DEFAULT_HARD_TIMEOUT = 600.0
|
_DEFAULT_HARD_TIMEOUT = 600.0
|
||||||
|
_REQUEST_TIMEOUT_GRACE_SECONDS = 5.0
|
||||||
|
_CLEANUP_REQUEST_TIMEOUT_SECONDS = 5
|
||||||
|
|
||||||
|
def _effective_command_timeout(self, timeout: float | None) -> float:
|
||||||
|
return timeout if timeout is not None else (getattr(self, "_default_command_timeout", None) or self._DEFAULT_HARD_TIMEOUT)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _effective_no_change_timeout(cls, timeout: float) -> int:
|
||||||
|
return max(
|
||||||
|
cls._DEFAULT_NO_CHANGE_TIMEOUT,
|
||||||
|
math.ceil(timeout + cls._REQUEST_TIMEOUT_GRACE_SECONDS),
|
||||||
|
)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _command_request_options(cls, timeout: float) -> dict[str, int]:
|
||||||
|
return {
|
||||||
|
"timeout_in_seconds": max(
|
||||||
|
1,
|
||||||
|
math.ceil(timeout + cls._REQUEST_TIMEOUT_GRACE_SECONDS),
|
||||||
|
),
|
||||||
|
"max_retries": 0,
|
||||||
|
}
|
||||||
|
|
||||||
def execute_command(
|
def execute_command(
|
||||||
self,
|
self,
|
||||||
@ -406,41 +596,52 @@ class AioSandbox(Sandbox):
|
|||||||
API (which supports per-command env) on a fresh explicitly released
|
API (which supports per-command env) on a fresh explicitly released
|
||||||
session, so the secrets are scoped to this single command and never
|
session, so the secrets are scoped to this single command and never
|
||||||
persist; secret values travel in the structured ``env`` field, never
|
persist; secret values travel in the structured ``env`` field, never
|
||||||
in the command string. When ``None`` the legacy persistent-shell path
|
in the command string. When ``None``, the legacy persistent-shell
|
||||||
runs unchanged.
|
path still uses the provider-wide command timeout via
|
||||||
timeout: Optional per-call timeout. The current sandbox SDK does not
|
``hard_timeout``, bounded request options, and returned-status
|
||||||
expose a command-level timeout distinct from its client/request
|
interpretation.
|
||||||
timeout, so DeerFlow keeps using the backend's default here.
|
timeout: Optional per-call command timeout. The legacy shell path
|
||||||
|
enforces it server-side and gives the request a small additional
|
||||||
|
response-path grace period.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
The output of the command.
|
The output of the command.
|
||||||
"""
|
"""
|
||||||
del timeout
|
|
||||||
# Validate ``env`` keys before forwarding them to the ``bash.exec`` API.
|
# Validate ``env`` keys before forwarding them to the ``bash.exec`` API.
|
||||||
# The public ``Sandbox.execute_command`` contract accepts arbitrary dict
|
# The public ``Sandbox.execute_command`` contract accepts arbitrary dict
|
||||||
# keys; enforcing the POSIX env-var name rule keeps the contract
|
# keys; enforcing the POSIX env-var name rule keeps the contract
|
||||||
# consistent with the local and e2b sandboxes and catches unsafe keys
|
# consistent with the local and e2b sandboxes and catches unsafe keys
|
||||||
# early. ``_validate_extra_env`` is a no-op when ``env`` is None or empty.
|
# early. ``_validate_extra_env`` is a no-op when ``env`` is None or empty.
|
||||||
_validate_extra_env(env)
|
_validate_extra_env(env)
|
||||||
|
effective_timeout = self._effective_command_timeout(timeout)
|
||||||
if env:
|
if env:
|
||||||
return self._execute_with_env(command, env)
|
return self._execute_with_env(command, env, effective_timeout)
|
||||||
with self._lock:
|
with self._lock:
|
||||||
try:
|
try:
|
||||||
client = self._client
|
client = self._client
|
||||||
if getattr(self, "_closed", False) or client is None:
|
if getattr(self, "_closed", False) or client is None:
|
||||||
raise RuntimeError("sandbox client is closed")
|
raise RuntimeError("sandbox client is closed")
|
||||||
if self._default_shell_corrupted and self._recovery_session_id is None:
|
session_id = self._ensure_default_shell_session_id(client)
|
||||||
# Once the implicit session emits ErrorObservation, never
|
|
||||||
# target it again. A failed replacement is cleaned up and
|
|
||||||
# the next call starts another explicit session.
|
|
||||||
self._recovery_session_id = self._create_shell_session(client)
|
|
||||||
recovered_missing_session = False
|
recovered_missing_session = False
|
||||||
try:
|
try:
|
||||||
output, exit_code = self._exec_shell(
|
output, exit_code, status = self._exec_shell(
|
||||||
client,
|
client,
|
||||||
command,
|
command,
|
||||||
session_id=self._recovery_session_id,
|
session_id=session_id,
|
||||||
|
timeout=effective_timeout,
|
||||||
)
|
)
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
session_id = self._recovery_session_id
|
||||||
|
self._recovery_session_id = None
|
||||||
|
self._default_shell_corrupted = True
|
||||||
|
if session_id is not None:
|
||||||
|
self._cleanup_session_best_effort(
|
||||||
|
client,
|
||||||
|
session_id,
|
||||||
|
context="default shell after transport timeout",
|
||||||
|
request_options=self._bounded_cleanup_request_options(),
|
||||||
|
)
|
||||||
|
return self._transport_timeout_error(effective_timeout)
|
||||||
except ApiError as error:
|
except ApiError as error:
|
||||||
if not self._is_missing_shell_session_error(error):
|
if not self._is_missing_shell_session_error(error):
|
||||||
raise
|
raise
|
||||||
@ -448,29 +649,67 @@ class AioSandbox(Sandbox):
|
|||||||
self._default_shell_corrupted = True
|
self._default_shell_corrupted = True
|
||||||
self._recovery_session_id = None
|
self._recovery_session_id = None
|
||||||
recovered_missing_session = True
|
recovered_missing_session = True
|
||||||
output, exit_code, self._recovery_session_id = self._rotate_and_retry_shell(
|
try:
|
||||||
client,
|
output, exit_code, status, self._recovery_session_id = self._rotate_and_retry_shell(
|
||||||
command,
|
client,
|
||||||
corrupted_session_id=None,
|
command,
|
||||||
context="default shell after missing session",
|
corrupted_session_id=None,
|
||||||
)
|
context="default shell after missing session",
|
||||||
|
timeout=effective_timeout,
|
||||||
|
)
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
return self._transport_timeout_error(effective_timeout)
|
||||||
|
|
||||||
if not recovered_missing_session and output and _ERROR_OBSERVATION_SIGNATURE in output:
|
if not recovered_missing_session and status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
|
||||||
self._default_shell_corrupted = True
|
self._default_shell_corrupted = True
|
||||||
logger.warning("ErrorObservation detected in sandbox output, retrying on a fresh session")
|
logger.warning("ErrorObservation detected in sandbox output, retrying on a fresh session")
|
||||||
output, exit_code, self._recovery_session_id = self._rotate_and_retry_shell(
|
corrupted_session_id = self._recovery_session_id
|
||||||
client,
|
self._recovery_session_id = None
|
||||||
command,
|
try:
|
||||||
corrupted_session_id=self._recovery_session_id,
|
output, exit_code, status, self._recovery_session_id = self._rotate_and_retry_shell(
|
||||||
context="default shell",
|
client,
|
||||||
)
|
command,
|
||||||
|
corrupted_session_id=corrupted_session_id,
|
||||||
|
context="default shell",
|
||||||
|
timeout=effective_timeout,
|
||||||
|
)
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
return self._transport_timeout_error(effective_timeout)
|
||||||
|
|
||||||
return self._render_shell_output(output, exit_code)
|
if self._is_session_invalidating_shell_status(status):
|
||||||
|
session_id = self._recovery_session_id
|
||||||
|
self._recovery_session_id = None
|
||||||
|
self._default_shell_corrupted = True
|
||||||
|
if session_id is not None:
|
||||||
|
if status == "terminated":
|
||||||
|
cleanup_context = "default shell after terminal session loss"
|
||||||
|
elif status == "no_change_timeout":
|
||||||
|
cleanup_context = "default shell after ambiguous no-change timeout"
|
||||||
|
else:
|
||||||
|
cleanup_context = "default shell after unexpected status"
|
||||||
|
self._cleanup_session_best_effort(
|
||||||
|
client,
|
||||||
|
session_id,
|
||||||
|
context=cleanup_context,
|
||||||
|
request_options=self._bounded_cleanup_request_options(),
|
||||||
|
)
|
||||||
|
|
||||||
|
return self._render_shell_output(
|
||||||
|
output,
|
||||||
|
exit_code,
|
||||||
|
status=status,
|
||||||
|
timeout=effective_timeout,
|
||||||
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to execute command in sandbox: {e}")
|
logger.error(f"Failed to execute command in sandbox: {e}")
|
||||||
return f"Error: {e}"
|
return f"Error: {e}"
|
||||||
|
|
||||||
def _execute_with_env(self, command: str, env: dict[str, str]) -> str:
|
def _execute_with_env(
|
||||||
|
self,
|
||||||
|
command: str,
|
||||||
|
env: dict[str, str],
|
||||||
|
timeout: float,
|
||||||
|
) -> str:
|
||||||
"""Execute a command with per-call environment variables injected.
|
"""Execute a command with per-call environment variables injected.
|
||||||
|
|
||||||
The persistent-shell ``shell.exec_command`` API has no env parameter, so
|
The persistent-shell ``shell.exec_command`` API has no env parameter, so
|
||||||
@ -501,15 +740,22 @@ class AioSandbox(Sandbox):
|
|||||||
"""
|
"""
|
||||||
if self._bash_exec_unsupported:
|
if self._bash_exec_unsupported:
|
||||||
return _BASH_EXEC_UNSUPPORTED_ERROR
|
return _BASH_EXEC_UNSUPPORTED_ERROR
|
||||||
output = self._run_bash_exec(command, env)
|
output, status = self._run_bash_exec(command, env, timeout)
|
||||||
if output and _ERROR_OBSERVATION_SIGNATURE in output:
|
if status in (None, "completed") and output and _ERROR_OBSERVATION_SIGNATURE in output:
|
||||||
logger.warning("ErrorObservation detected in bash.exec output, retrying on a fresh session")
|
logger.warning("ErrorObservation detected in bash.exec output, retrying on a fresh session")
|
||||||
retried = self._run_bash_exec(command, env)
|
retried, retry_status = self._run_bash_exec(command, env, timeout)
|
||||||
|
if retry_status not in (None, "completed"):
|
||||||
|
return retried
|
||||||
if retried and _ERROR_OBSERVATION_SIGNATURE not in retried:
|
if retried and _ERROR_OBSERVATION_SIGNATURE not in retried:
|
||||||
return retried
|
return retried
|
||||||
return output
|
return output
|
||||||
|
|
||||||
def _run_bash_exec(self, command: str, env: dict[str, str]) -> str:
|
def _run_bash_exec(
|
||||||
|
self,
|
||||||
|
command: str,
|
||||||
|
env: dict[str, str],
|
||||||
|
timeout: float,
|
||||||
|
) -> tuple[str, str | None]:
|
||||||
"""Single bash.exec invocation in an explicitly released fresh session."""
|
"""Single bash.exec invocation in an explicitly released fresh session."""
|
||||||
with self._lock:
|
with self._lock:
|
||||||
for attempt in range(2):
|
for attempt in range(2):
|
||||||
@ -522,40 +768,65 @@ class AioSandbox(Sandbox):
|
|||||||
command=command,
|
command=command,
|
||||||
session_id=session_id,
|
session_id=session_id,
|
||||||
env=env,
|
env=env,
|
||||||
hard_timeout=self._DEFAULT_HARD_TIMEOUT,
|
hard_timeout=timeout,
|
||||||
|
request_options=self._command_request_options(timeout),
|
||||||
)
|
)
|
||||||
data = result.data if result else None
|
data = result.data if result else None
|
||||||
stdout = (data.stdout or "") if data else ""
|
stdout = (data.stdout or "") if data else ""
|
||||||
stderr = (data.stderr or "") if data else ""
|
stderr = (data.stderr or "") if data else ""
|
||||||
exit_code = getattr(data, "exit_code", None) if data else None
|
exit_code = getattr(data, "exit_code", None) if data else None
|
||||||
|
status = getattr(data, "status", None) if data else None
|
||||||
output = stdout
|
output = stdout
|
||||||
if stderr:
|
if stderr:
|
||||||
output += f"\nStd Error:\n{stderr}" if output else stderr
|
output += f"\nStd Error:\n{stderr}" if output else stderr
|
||||||
|
|
||||||
|
if status == "timed_out":
|
||||||
|
notice = self._format_timeout_notice(timeout)
|
||||||
|
output = f"{output}\n{notice}" if output else notice
|
||||||
|
return f"{output}\nExit Code: 124", status
|
||||||
|
|
||||||
|
if status == "killed":
|
||||||
|
notice = "Command was killed before completion."
|
||||||
|
output = f"{output}\n{notice}" if output else notice
|
||||||
|
return output, status
|
||||||
|
|
||||||
|
if status == "running":
|
||||||
|
notice = "Error: Sandbox command returned a non-terminal running status; command outcome is unknown and was not retried."
|
||||||
|
output = f"{output}\n{notice}" if output else notice
|
||||||
|
return output, status
|
||||||
|
|
||||||
|
if status not in (None, "completed"):
|
||||||
|
notice = self._unexpected_status_notice(status)
|
||||||
|
output = f"{output}\n{notice}" if output else notice
|
||||||
|
return output, status
|
||||||
|
|
||||||
if exit_code not in (0, None):
|
if exit_code not in (0, None):
|
||||||
# Mirror LocalSandbox: keep the actual shell status in the
|
# Mirror LocalSandbox: keep the actual shell status in the
|
||||||
# output text (acceptance-checklist evidence).
|
# output text (acceptance-checklist evidence).
|
||||||
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
|
output = f"{output}\nExit Code: {exit_code}" if output else f"Command exited with code {exit_code}"
|
||||||
return output if output else "(no output)"
|
return output if output else "(no output)", status
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
return self._transport_timeout_error(timeout), "transport_timeout"
|
||||||
except ApiError as e:
|
except ApiError as e:
|
||||||
if self._is_missing_shell_session_error(e):
|
if self._is_missing_shell_session_error(e):
|
||||||
if attempt == 0:
|
if attempt == 0:
|
||||||
logger.warning("Transient bash.exec session disappeared; retrying once")
|
logger.warning("Transient bash.exec session disappeared; retrying once")
|
||||||
continue
|
continue
|
||||||
logger.error("Failed to execute command with injected env: bash.exec session disappeared after retry")
|
logger.error("Failed to execute command with injected env: bash.exec session disappeared after retry")
|
||||||
return "Error: bash.exec session disappeared after retry"
|
return "Error: bash.exec session disappeared after retry", None
|
||||||
if e.status_code == 404:
|
if e.status_code == 404:
|
||||||
self._bash_exec_unsupported = True
|
self._bash_exec_unsupported = True
|
||||||
logger.error("Sandbox %s does not support bash.exec (/v1/bash/exec returned 404); env-bearing commands are unavailable until the sandbox image is upgraded to all-in-one-sandbox >= 1.9.3", self.id)
|
logger.error("Sandbox %s does not support bash.exec (/v1/bash/exec returned 404); env-bearing commands are unavailable until the sandbox image is upgraded to all-in-one-sandbox >= 1.9.3", self.id)
|
||||||
return _BASH_EXEC_UNSUPPORTED_ERROR
|
return _BASH_EXEC_UNSUPPORTED_ERROR, None
|
||||||
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
|
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
|
||||||
return f"Error: {e}"
|
return f"Error: {e}", None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
|
logger.error(f"Failed to execute command with injected env in sandbox: {e}")
|
||||||
return f"Error: {e}"
|
return f"Error: {e}", None
|
||||||
finally:
|
finally:
|
||||||
if session_created:
|
if session_created:
|
||||||
self._cleanup_bash_session_best_effort(self._client, session_id)
|
self._cleanup_bash_session_best_effort(self._client, session_id)
|
||||||
return "Error: bash.exec session disappeared after retry"
|
return "Error: bash.exec session disappeared after retry", None
|
||||||
|
|
||||||
def read_file(
|
def read_file(
|
||||||
self,
|
self,
|
||||||
@ -639,10 +910,17 @@ class AioSandbox(Sandbox):
|
|||||||
resolved = path
|
resolved = path
|
||||||
with self._lock:
|
with self._lock:
|
||||||
try:
|
try:
|
||||||
result = self._client.shell.exec_command(
|
client = self._client
|
||||||
command=remote_list_dir_command(resolved, max_depth),
|
session_id = self._ensure_default_shell_session_id(client)
|
||||||
no_change_timeout=self._DEFAULT_NO_CHANGE_TIMEOUT,
|
|
||||||
)
|
kwargs = {
|
||||||
|
"command": remote_list_dir_command(resolved, max_depth),
|
||||||
|
"no_change_timeout": self._DEFAULT_NO_CHANGE_TIMEOUT,
|
||||||
|
}
|
||||||
|
if session_id is not None:
|
||||||
|
kwargs["id"] = session_id
|
||||||
|
|
||||||
|
result = client.shell.exec_command(**kwargs)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to list directory in sandbox: {e}")
|
logger.error(f"Failed to list directory in sandbox: {e}")
|
||||||
raise OSError(f"Failed to list directory '{resolved}' in sandbox: {e}") from e
|
raise OSError(f"Failed to list directory '{resolved}' in sandbox: {e}") from e
|
||||||
|
|||||||
@ -15,12 +15,14 @@ import atexit
|
|||||||
import contextlib
|
import contextlib
|
||||||
import hashlib
|
import hashlib
|
||||||
import logging
|
import logging
|
||||||
|
import math
|
||||||
import os
|
import os
|
||||||
import signal
|
import signal
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import fcntl
|
import fcntl
|
||||||
@ -169,6 +171,16 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
|||||||
# normally timing-out refresh + release still finishes synchronously.
|
# normally timing-out refresh + release still finishes synchronously.
|
||||||
_TEARDOWN_JOIN_TIMEOUT_SECONDS = 12.0
|
_TEARDOWN_JOIN_TIMEOUT_SECONDS = 12.0
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _positive_float(name: str, value: Any, default: float) -> float:
|
||||||
|
try:
|
||||||
|
resolved = float(default if value is None else value)
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise ValueError(f"sandbox.{name} must be positive") from exc
|
||||||
|
if not math.isfinite(resolved) or resolved <= 0:
|
||||||
|
raise ValueError(f"sandbox.{name} must be positive")
|
||||||
|
return resolved
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self._lock = threading.Lock()
|
self._lock = threading.Lock()
|
||||||
self._sandboxes: dict[str, AioSandbox] = {} # sandbox_id -> AioSandbox instance
|
self._sandboxes: dict[str, AioSandbox] = {} # sandbox_id -> AioSandbox instance
|
||||||
@ -301,6 +313,11 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
|||||||
configured_skills_path = DEFAULT_SKILLS_CONTAINER_PATH
|
configured_skills_path = DEFAULT_SKILLS_CONTAINER_PATH
|
||||||
|
|
||||||
environment = self._resolve_env_vars(sandbox_config.environment or {})
|
environment = self._resolve_env_vars(sandbox_config.environment or {})
|
||||||
|
command_timeout = self._positive_float(
|
||||||
|
"bash_command_timeout",
|
||||||
|
getattr(sandbox_config, "bash_command_timeout", None),
|
||||||
|
AioSandbox._DEFAULT_HARD_TIMEOUT,
|
||||||
|
)
|
||||||
max_running_subagents = int(getattr(getattr(config, "subagent_runtime", None), "max_running", 3))
|
max_running_subagents = int(getattr(getattr(config, "subagent_runtime", None), "max_running", 3))
|
||||||
required_shell_sessions = max_running_subagents + _SHELL_SESSION_HEADROOM
|
required_shell_sessions = max_running_subagents + _SHELL_SESSION_HEADROOM
|
||||||
configured_shell_sessions = environment.get("MAX_SHELL_SESSIONS")
|
configured_shell_sessions = environment.get("MAX_SHELL_SESSIONS")
|
||||||
@ -321,6 +338,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
|||||||
"port": sandbox_config.port or DEFAULT_PORT,
|
"port": sandbox_config.port or DEFAULT_PORT,
|
||||||
"container_prefix": sandbox_config.container_prefix or DEFAULT_CONTAINER_PREFIX,
|
"container_prefix": sandbox_config.container_prefix or DEFAULT_CONTAINER_PREFIX,
|
||||||
"idle_timeout": idle_timeout if idle_timeout is not None else DEFAULT_IDLE_TIMEOUT,
|
"idle_timeout": idle_timeout if idle_timeout is not None else DEFAULT_IDLE_TIMEOUT,
|
||||||
|
"command_timeout": command_timeout,
|
||||||
"replicas": replicas if replicas is not None else DEFAULT_REPLICAS,
|
"replicas": replicas if replicas is not None else DEFAULT_REPLICAS,
|
||||||
"mounts": sandbox_config.mounts or [],
|
"mounts": sandbox_config.mounts or [],
|
||||||
"thread_data_mounts": getattr(sandbox_config, "thread_data_mounts", None),
|
"thread_data_mounts": getattr(sandbox_config, "thread_data_mounts", None),
|
||||||
@ -1686,7 +1704,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
|||||||
return None
|
return None
|
||||||
self._warm_pool_identity.pop(sandbox_id, None)
|
self._warm_pool_identity.pop(sandbox_id, None)
|
||||||
info, _ = warm_item
|
info, _ = warm_item
|
||||||
sandbox = AioSandbox(id=sandbox_id, base_url=info.sandbox_url, request_headers=info.request_headers)
|
sandbox = AioSandbox(
|
||||||
|
id=sandbox_id,
|
||||||
|
base_url=info.sandbox_url,
|
||||||
|
request_headers=info.request_headers,
|
||||||
|
default_command_timeout=self._config.get("command_timeout", AioSandbox._DEFAULT_HARD_TIMEOUT),
|
||||||
|
)
|
||||||
self._sandboxes[sandbox_id] = sandbox
|
self._sandboxes[sandbox_id] = sandbox
|
||||||
self._sandbox_infos[sandbox_id] = info
|
self._sandbox_infos[sandbox_id] = info
|
||||||
self._active_sandbox_identity[sandbox_id] = key
|
self._active_sandbox_identity[sandbox_id] = key
|
||||||
@ -1731,7 +1754,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
|||||||
self._assert_active_identity_available_locked(info.sandbox_id, key)
|
self._assert_active_identity_available_locked(info.sandbox_id, key)
|
||||||
self._assert_warm_identity_available_locked(info.sandbox_id, key)
|
self._assert_warm_identity_available_locked(info.sandbox_id, key)
|
||||||
|
|
||||||
sandbox = AioSandbox(id=info.sandbox_id, base_url=info.sandbox_url, request_headers=info.request_headers)
|
sandbox = AioSandbox(
|
||||||
|
id=info.sandbox_id,
|
||||||
|
base_url=info.sandbox_url,
|
||||||
|
request_headers=info.request_headers,
|
||||||
|
default_command_timeout=self._config.get("command_timeout", AioSandbox._DEFAULT_HARD_TIMEOUT),
|
||||||
|
)
|
||||||
# Ownership first, so a failure cannot leave a tracked-but-unowned sandbox.
|
# Ownership first, so a failure cannot leave a tracked-but-unowned sandbox.
|
||||||
# There is no container to roll back (we did not create it), but the
|
# There is no container to roll back (we did not create it), but the
|
||||||
# host-side HTTP client constructed above is ours and must not leak —
|
# host-side HTTP client constructed above is ours and must not leak —
|
||||||
@ -1777,7 +1805,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
|
|||||||
|
|
||||||
def _register_created_sandbox(self, thread_id: str | None, sandbox_id: str, info: SandboxInfo, *, user_id: str | None = None) -> str:
|
def _register_created_sandbox(self, thread_id: str | None, sandbox_id: str, info: SandboxInfo, *, user_id: str | None = None) -> str:
|
||||||
"""Track a newly-created sandbox in the active maps."""
|
"""Track a newly-created sandbox in the active maps."""
|
||||||
sandbox = AioSandbox(id=sandbox_id, base_url=info.sandbox_url, request_headers=info.request_headers)
|
sandbox = AioSandbox(
|
||||||
|
id=sandbox_id,
|
||||||
|
base_url=info.sandbox_url,
|
||||||
|
request_headers=info.request_headers,
|
||||||
|
default_command_timeout=self._config.get("command_timeout", AioSandbox._DEFAULT_HARD_TIMEOUT),
|
||||||
|
)
|
||||||
key = (
|
key = (
|
||||||
self._thread_key(
|
self._thread_key(
|
||||||
thread_id,
|
thread_id,
|
||||||
|
|||||||
@ -258,13 +258,19 @@ class SandboxConfig(BaseModel):
|
|||||||
ge=0,
|
ge=0,
|
||||||
description="Maximum characters to keep from ls tool output. Output exceeding this limit is head-truncated. Set to 0 to disable truncation.",
|
description="Maximum characters to keep from ls tool output. Output exceeding this limit is head-truncated. Set to 0 to disable truncation.",
|
||||||
)
|
)
|
||||||
bash_command_timeout: int = Field(
|
bash_command_timeout: float = Field(
|
||||||
default=600,
|
default=600,
|
||||||
gt=0,
|
gt=0,
|
||||||
|
allow_inf_nan=False,
|
||||||
description=(
|
description=(
|
||||||
"Maximum wall-clock seconds a bash command may run before it is terminated. LocalSandboxProvider applies it to the host process group; "
|
"Provider command deadline. AIO images on the supported semver line "
|
||||||
"OpenSandboxProvider forwards it to the remote exec service when a call has no explicit timeout. Keeps a blocking foreground command "
|
"(1.9.3+, recommended 1.11.0) enforce it server-side through "
|
||||||
"(e.g. an un-backgrounded server) from hanging the turn; background `&` processes return immediately."
|
"`hard_timeout`; the frozen legacy `all-in-one-sandbox:latest` image "
|
||||||
|
"only gets the bounded host-side request. `bash_command_timeout` is "
|
||||||
|
"used by providers that explicitly wire this setting (currently "
|
||||||
|
"LocalSandbox, AioSandbox, and OpenSandbox). Other providers retain "
|
||||||
|
"their provider-specific command defaults unless a caller supplies an "
|
||||||
|
"explicit timeout."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@ -110,7 +110,7 @@ subshell. Keep parser filtering as a backstop; test ignored roots, metacharacter
|
|||||||
symlinks, and visible depth/size bounds.
|
symlinks, and visible depth/size bounds.
|
||||||
|
|
||||||
- Every sandbox tool keeps a model-visible `description` field for a human-readable progress label, but the field is optional and defaults to an empty string. Tool execution must depend only on its operational arguments; the frontend supplies localized fallback labels when a provider omits `description`.
|
- Every sandbox tool keeps a model-visible `description` field for a human-readable progress label, but the field is optional and defaults to an empty string. Tool execution must depend only on its operational arguments; the frontend supplies localized fallback labels when a provider omits `description`.
|
||||||
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox` (host bash), output on POSIX and Windows is captured through bounded pipe-drain threads and stdin is `/dev/null`; Windows capture decodes with the platform text encoding and applies universal-newline translation, matching the former `subprocess.run(..., text=True)` behavior for locale-code-page output, Python UTF-8 Mode, CRLF, and bare CR. That translation is Windows-only so the pre-existing POSIX output contract remains byte-decoded without newline rewriting. On POSIX, a backgrounded long-lived process (`server &`) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (`sandbox.bash_command_timeout`, default 600s); on timeout the whole POSIX process group or Windows process tree is killed and the agent gets a notice telling it to background long-lived processes. The shared bash tool description scopes host environment detection to LocalSandbox: start with `uname -s`, follow with `sw_vers` on Darwin, and read Linux host system files only when the active policy permits them. Local path and `file://` rejections provide the same conditional recovery guidance: command-only probes for environment questions, allowed virtual paths otherwise, and no repetition of the rejected path. The description also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See `LocalSandbox.execute_command`, its platform runners, and `bash_tool`'s docstring.
|
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox`, POSIX/Windows output uses bounded pipe-drain threads with `/dev/null` stdin; Windows decodes locale-code-page/UTF-8/CRLF/bare-CR output with universal-newline translation, while POSIX stays byte-decoded. POSIX background commands return without blocking on inherited pipes; unredirected output is drained without unbounded temp files. Commands that read stdin get immediate EOF. `bash_command_timeout` sets T (600s) for Local/AIO/OpenSandbox; others keep defaults unless explicit. Local: T is a wall-clock process-group deadline. Supported-semver AIO: server-side `hard_timeout`; frozen legacy `all-in-one-sandbox:latest`: command requests get a bounded host `T+5s` wait (`max_retries=0`), so a wedged command request cannot hold the sandbox lock for the SDK's full 600s budget; session-creation and file/list RPCs are not bounded by it. Never replay ambiguous outcomes: transport timeout, terminated, no_change_timeout, unknown statuses. `hard_timeout` => terminated + Exit Code: 124, keep session; `no_change_timeout` => may still be running, fence session generation. The description scopes host-environment probes to LocalSandbox (`uname -s`, then `sw_vers` on Darwin; Linux files only when policy permits), gives conditional recovery for local path/`file://` rejections, and tells the model to background long-lived processes. See `LocalSandbox.execute_command`, its platform runners, and `bash_tool`'s docstring.
|
||||||
- `ls` - Directory listing (tree format, max 2 levels). Remote commands precheck root existence and emit `__DF_FIND_STATUS__:missing`; `find` status 1 is always an incomplete-traversal `OSError`, including when no entries were printed. Do not infer a missing path from status 1 alone.
|
- `ls` - Directory listing (tree format, max 2 levels). Remote commands precheck root existence and emit `__DF_FIND_STATUS__:missing`; `find` status 1 is always an incomplete-traversal `OSError`, including when no entries were printed. Do not infer a missing path from status 1 alone.
|
||||||
- `glob` - Find files or directories below a root directory with bounded results
|
- `glob` - Find files or directories below a root directory with bounded results
|
||||||
- `grep` - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results
|
- `grep` - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@ -12,6 +12,7 @@ from unittest.mock import MagicMock, patch
|
|||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from _windows_acl_helpers import _windows_acl_owner_sid, _windows_acl_sids
|
from _windows_acl_helpers import _windows_acl_owner_sid, _windows_acl_sids
|
||||||
|
from pydantic import ValidationError
|
||||||
|
|
||||||
from deerflow.config.paths import Paths, join_host_path
|
from deerflow.config.paths import Paths, join_host_path
|
||||||
from deerflow.config.sandbox_config import SandboxConfig
|
from deerflow.config.sandbox_config import SandboxConfig
|
||||||
@ -68,6 +69,61 @@ def test_load_config_snapshots_custom_skills_container_path(monkeypatch):
|
|||||||
assert provider._load_config()["skills_container_path"] == "/custom-skills"
|
assert provider._load_config()["skills_container_path"] == "/custom-skills"
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_config_wires_bash_command_timeout_to_aio_default(monkeypatch):
|
||||||
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
|
sandbox_config = SandboxConfig(
|
||||||
|
use="deerflow.community.aio_sandbox:AioSandboxProvider",
|
||||||
|
bash_command_timeout=42.5,
|
||||||
|
)
|
||||||
|
app_config = SimpleNamespace(sandbox=sandbox_config, stream_bridge=None)
|
||||||
|
monkeypatch.setattr(aio_mod, "get_app_config", lambda: app_config)
|
||||||
|
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
|
||||||
|
|
||||||
|
assert provider._load_config()["command_timeout"] == 42.5
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("invalid_timeout", [float("nan"), float("inf"), float("-inf"), 0, -1])
|
||||||
|
def test_positive_float_rejects_non_positive_or_non_finite_values(invalid_timeout):
|
||||||
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="sandbox.bash_command_timeout must be positive"):
|
||||||
|
aio_mod.AioSandboxProvider._positive_float("bash_command_timeout", invalid_timeout, 600)
|
||||||
|
|
||||||
|
|
||||||
|
def test_positive_float_accepts_fractional_value():
|
||||||
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
|
|
||||||
|
assert aio_mod.AioSandboxProvider._positive_float("bash_command_timeout", 42.5, 600) == 42.5
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf")])
|
||||||
|
def test_bash_command_timeout_rejects_non_finite_values(value):
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
SandboxConfig(bash_command_timeout=value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_register_created_sandbox_forwards_configured_command_timeout(tmp_path):
|
||||||
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
|
provider = _make_provider(tmp_path)
|
||||||
|
provider._config["command_timeout"] = 42
|
||||||
|
provider._warm_pool = {}
|
||||||
|
provider._sandbox_infos = {}
|
||||||
|
provider._thread_sandboxes = {}
|
||||||
|
provider._last_activity = {}
|
||||||
|
provider._publish_ownership = MagicMock()
|
||||||
|
info = aio_mod.SandboxInfo(sandbox_id="sandbox-timeout", sandbox_url="http://sandbox")
|
||||||
|
|
||||||
|
with patch.object(aio_mod, "AioSandbox") as sandbox_cls:
|
||||||
|
provider._register_created_sandbox("thread-timeout", "sandbox-timeout", info, user_id="user-timeout")
|
||||||
|
|
||||||
|
sandbox_cls.assert_called_once_with(
|
||||||
|
id="sandbox-timeout",
|
||||||
|
base_url="http://sandbox",
|
||||||
|
request_headers=info.request_headers,
|
||||||
|
default_command_timeout=42,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_load_config_sizes_aio_shell_capacity_for_subagent_runtime(monkeypatch):
|
def test_load_config_sizes_aio_shell_capacity_for_subagent_runtime(monkeypatch):
|
||||||
"""Twelve subagents must not exceed AIO 1.11's ten-session default."""
|
"""Twelve subagents must not exceed AIO 1.11's ten-session default."""
|
||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
@ -231,7 +287,7 @@ def _make_provider(tmp_path):
|
|||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
with patch.object(aio_mod.AioSandboxProvider, "_start_idle_checker"):
|
with patch.object(aio_mod.AioSandboxProvider, "_start_idle_checker"):
|
||||||
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
|
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
|
||||||
provider._config = {"idle_timeout": 600, "replicas": 3}
|
provider._config = {"command_timeout": 600.0, "idle_timeout": 600, "replicas": 3}
|
||||||
provider._sandboxes = {}
|
provider._sandboxes = {}
|
||||||
provider._active_sandbox_identity = {}
|
provider._active_sandbox_identity = {}
|
||||||
provider._warm_pool_identity = {}
|
provider._warm_pool_identity = {}
|
||||||
@ -575,6 +631,7 @@ def test_policy_scoped_create_excludes_local_config_mounts_below_skills_root(
|
|||||||
|
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._config = {
|
provider._config = {
|
||||||
|
"command_timeout": 600.0,
|
||||||
"replicas": 3,
|
"replicas": 3,
|
||||||
"skills_container_path": "/mnt/skills",
|
"skills_container_path": "/mnt/skills",
|
||||||
}
|
}
|
||||||
@ -631,6 +688,7 @@ def test_remote_create_forwards_configured_skills_container_path(
|
|||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._config = {
|
provider._config = {
|
||||||
|
"command_timeout": 600.0,
|
||||||
"replicas": 3,
|
"replicas": 3,
|
||||||
"skills_container_path": "/custom-skills",
|
"skills_container_path": "/custom-skills",
|
||||||
}
|
}
|
||||||
@ -693,6 +751,7 @@ async def test_remote_create_async_forwards_configured_skills_container_path(
|
|||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._config = {
|
provider._config = {
|
||||||
|
"command_timeout": 600.0,
|
||||||
"replicas": 3,
|
"replicas": 3,
|
||||||
"skills_container_path": "/custom-skills",
|
"skills_container_path": "/custom-skills",
|
||||||
}
|
}
|
||||||
@ -811,7 +870,7 @@ async def test_acquire_async_uses_async_readiness_polling(monkeypatch):
|
|||||||
"""AioSandboxProvider async creation must not use sync readiness polling."""
|
"""AioSandboxProvider async creation must not use sync readiness polling."""
|
||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(None)
|
provider = _make_provider(None)
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._warm_pool = {}
|
provider._warm_pool = {}
|
||||||
provider._sandbox_infos = {}
|
provider._sandbox_infos = {}
|
||||||
provider._thread_sandboxes = {}
|
provider._thread_sandboxes = {}
|
||||||
@ -1110,7 +1169,7 @@ def test_create_sandbox_requests_runtime_when_lark_installed(tmp_path, monkeypat
|
|||||||
"""The provider must request lark-cli runtime provisioning when Lark is installed."""
|
"""The provider must request lark-cli runtime provisioning when Lark is installed."""
|
||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._warm_pool = {}
|
provider._warm_pool = {}
|
||||||
provider._sandbox_infos = {}
|
provider._sandbox_infos = {}
|
||||||
provider._thread_sandboxes = {}
|
provider._thread_sandboxes = {}
|
||||||
@ -1140,7 +1199,7 @@ def test_create_sandbox_requests_broker_when_active(tmp_path, monkeypatch):
|
|||||||
"""Broker mode (Pattern B) is requested when the provisioner reports it."""
|
"""Broker mode (Pattern B) is requested when the provisioner reports it."""
|
||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._warm_pool = {}
|
provider._warm_pool = {}
|
||||||
provider._sandbox_infos = {}
|
provider._sandbox_infos = {}
|
||||||
provider._thread_sandboxes = {}
|
provider._thread_sandboxes = {}
|
||||||
@ -1170,7 +1229,7 @@ def test_create_sandbox_skips_runtime_when_lark_absent(tmp_path, monkeypatch):
|
|||||||
"""No runtime provisioning request when the Lark skill pack is not installed."""
|
"""No runtime provisioning request when the Lark skill pack is not installed."""
|
||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._warm_pool = {}
|
provider._warm_pool = {}
|
||||||
provider._sandbox_infos = {}
|
provider._sandbox_infos = {}
|
||||||
provider._thread_sandboxes = {}
|
provider._thread_sandboxes = {}
|
||||||
@ -1305,7 +1364,7 @@ def test_acquire_drops_dead_cached_sandbox(tmp_path, monkeypatch):
|
|||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider, sandbox, _ = _make_provider_with_active_sandbox(tmp_path, "sandbox-dead")
|
provider, sandbox, _ = _make_provider_with_active_sandbox(tmp_path, "sandbox-dead")
|
||||||
provider._thread_sandboxes = {("default", "thread-dead"): "sandbox-dead"}
|
provider._thread_sandboxes = {("default", "thread-dead"): "sandbox-dead"}
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._backend.is_alive = MagicMock(return_value=False)
|
provider._backend.is_alive = MagicMock(return_value=False)
|
||||||
provider._backend.discover = MagicMock(return_value=None)
|
provider._backend.discover = MagicMock(return_value=None)
|
||||||
provider._backend.create = MagicMock(
|
provider._backend.create = MagicMock(
|
||||||
@ -1389,7 +1448,7 @@ def test_acquire_skips_dead_warm_pool_sandbox(tmp_path, monkeypatch):
|
|||||||
0.0,
|
0.0,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._backend = SimpleNamespace(
|
provider._backend = SimpleNamespace(
|
||||||
is_alive=MagicMock(return_value=False),
|
is_alive=MagicMock(return_value=False),
|
||||||
destroy=MagicMock(),
|
destroy=MagicMock(),
|
||||||
@ -1472,7 +1531,7 @@ def test_create_sandbox_evicts_oldest_warm_replica_via_shared_lifecycle(tmp_path
|
|||||||
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._lock = aio_mod.threading.Lock()
|
provider._lock = aio_mod.threading.Lock()
|
||||||
provider._config = {"replicas": 2}
|
provider._config = {"command_timeout": 600.0, "replicas": 2}
|
||||||
provider._sandboxes = {}
|
provider._sandboxes = {}
|
||||||
provider._sandbox_infos = {}
|
provider._sandbox_infos = {}
|
||||||
provider._thread_sandboxes = {}
|
provider._thread_sandboxes = {}
|
||||||
@ -1513,7 +1572,7 @@ def _make_tenant_isolation_provider(tmp_path, monkeypatch):
|
|||||||
provider._active_sandbox_identity = {}
|
provider._active_sandbox_identity = {}
|
||||||
provider._warm_pool_identity = {}
|
provider._warm_pool_identity = {}
|
||||||
provider._shutdown_called = False
|
provider._shutdown_called = False
|
||||||
provider._config = {"replicas": 3, "idle_timeout": 0}
|
provider._config = {"command_timeout": 600.0, "replicas": 3, "idle_timeout": 0}
|
||||||
|
|
||||||
create_calls = []
|
create_calls = []
|
||||||
|
|
||||||
@ -1610,7 +1669,7 @@ def _make_unready_destroy_provider(tmp_path, *, sandbox_id, base_url, monkeypatc
|
|||||||
"""
|
"""
|
||||||
provider = _make_provider(tmp_path)
|
provider = _make_provider(tmp_path)
|
||||||
provider._lock = aio_mod.threading.Lock()
|
provider._lock = aio_mod.threading.Lock()
|
||||||
provider._config = {"replicas": 3}
|
provider._config = {"command_timeout": 600.0, "replicas": 3}
|
||||||
provider._warm_pool = {}
|
provider._warm_pool = {}
|
||||||
provider._sandbox_infos = {}
|
provider._sandbox_infos = {}
|
||||||
provider._thread_sandboxes = {}
|
provider._thread_sandboxes = {}
|
||||||
|
|||||||
@ -119,7 +119,7 @@ def test_aio_sandbox_env_routes_through_bash_exec() -> None:
|
|||||||
captured["exec_session"] = kwargs["session_id"]
|
captured["exec_session"] = kwargs["session_id"]
|
||||||
return SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None))
|
return SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None))
|
||||||
|
|
||||||
def close_session(self, session_id):
|
def close_session(self, session_id, **kwargs):
|
||||||
captured["closed_session"] = session_id
|
captured["closed_session"] = session_id
|
||||||
|
|
||||||
sbx = AioSandbox.__new__(AioSandbox)
|
sbx = AioSandbox.__new__(AioSandbox)
|
||||||
|
|||||||
@ -13,6 +13,7 @@ from pathlib import Path
|
|||||||
from types import SimpleNamespace
|
from types import SimpleNamespace
|
||||||
from unittest.mock import MagicMock, patch
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import httpx
|
||||||
import pytest
|
import pytest
|
||||||
from langchain.agents.middleware.types import ModelRequest
|
from langchain.agents.middleware.types import ModelRequest
|
||||||
from langchain_core.messages import AIMessage, HumanMessage
|
from langchain_core.messages import AIMessage, HumanMessage
|
||||||
@ -101,7 +102,7 @@ class TestAioSandboxEnvInjection:
|
|||||||
return AioSandbox(id="test-sandbox", base_url="http://localhost:8080")
|
return AioSandbox(id="test-sandbox", base_url="http://localhost:8080")
|
||||||
|
|
||||||
def test_env_none_uses_legacy_shell_path(self, sandbox):
|
def test_env_none_uses_legacy_shell_path(self, sandbox):
|
||||||
"""No injected env → unchanged shell.exec_command path (backward compat)."""
|
"""No injected env uses the legacy shell path with bounded timeout/status handling."""
|
||||||
sandbox._client.shell.exec_command = MagicMock(return_value=SimpleNamespace(data=SimpleNamespace(output="hello")))
|
sandbox._client.shell.exec_command = MagicMock(return_value=SimpleNamespace(data=SimpleNamespace(output="hello")))
|
||||||
sandbox._client.bash.exec = MagicMock()
|
sandbox._client.bash.exec = MagicMock()
|
||||||
out = sandbox.execute_command("echo hello")
|
out = sandbox.execute_command("echo hello")
|
||||||
@ -122,23 +123,195 @@ class TestAioSandboxEnvInjection:
|
|||||||
sandbox._client.shell.exec_command.assert_not_called()
|
sandbox._client.shell.exec_command.assert_not_called()
|
||||||
assert "hello" in out
|
assert "hello" in out
|
||||||
|
|
||||||
def test_env_path_uses_hard_timeout_not_no_change_timeout(self, sandbox):
|
def test_env_path_uses_explicit_command_timeout_and_request_budget(self, sandbox):
|
||||||
"""The env path routes through bash.exec which exposes no idle/no-change
|
sandbox._client.bash.exec = MagicMock(
|
||||||
timeout; it must use the dedicated wall-clock ``_DEFAULT_HARD_TIMEOUT``,
|
return_value=SimpleNamespace(
|
||||||
not the legacy idle constant (same numeric value today, but distinct
|
data=SimpleNamespace(
|
||||||
semantics so a future change to one does not silently alter the other)."""
|
stdout="ok",
|
||||||
from deerflow.community.aio_sandbox.aio_sandbox import AioSandbox
|
stderr=None,
|
||||||
|
exit_code=0,
|
||||||
sandbox._client.bash.exec = MagicMock(return_value=SimpleNamespace(data=SimpleNamespace(stdout="ok", stderr=None)))
|
status="completed",
|
||||||
sandbox.execute_command("echo hi", env={"X": "1"})
|
)
|
||||||
_, kwargs = sandbox._client.bash.exec.call_args
|
)
|
||||||
assert kwargs["hard_timeout"] == AioSandbox._DEFAULT_HARD_TIMEOUT
|
|
||||||
assert AioSandbox._DEFAULT_HARD_TIMEOUT != AioSandbox._DEFAULT_NO_CHANGE_TIMEOUT or (
|
|
||||||
# Same numeric value is fine today; the contract is that they are
|
|
||||||
# named independently so the two call sites evolve independently.
|
|
||||||
AioSandbox._DEFAULT_HARD_TIMEOUT == AioSandbox._DEFAULT_NO_CHANGE_TIMEOUT
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
sandbox.execute_command(
|
||||||
|
"echo hi",
|
||||||
|
env={"X": "1"},
|
||||||
|
timeout=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
_, kwargs = sandbox._client.bash.exec.call_args
|
||||||
|
assert kwargs["hard_timeout"] == 3
|
||||||
|
assert kwargs["request_options"] == {
|
||||||
|
"timeout_in_seconds": 8,
|
||||||
|
"max_retries": 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_env_path_uses_default_hard_timeout_when_timeout_is_none(self, sandbox):
|
||||||
|
sandbox._client.bash.exec = MagicMock(
|
||||||
|
return_value=SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="ok",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=0,
|
||||||
|
status="completed",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
sandbox.execute_command("echo hi", env={"X": "1"})
|
||||||
|
|
||||||
|
_, kwargs = sandbox._client.bash.exec.call_args
|
||||||
|
assert kwargs["hard_timeout"] == sandbox._DEFAULT_HARD_TIMEOUT
|
||||||
|
|
||||||
|
def test_env_hard_timeout_is_rendered_and_not_retried(self, sandbox):
|
||||||
|
executions = 0
|
||||||
|
|
||||||
|
def bash_exec(**kwargs):
|
||||||
|
nonlocal executions
|
||||||
|
executions += 1
|
||||||
|
return SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="partial",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=-1,
|
||||||
|
status="timed_out",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
sandbox._client.bash.exec = bash_exec
|
||||||
|
|
||||||
|
out = sandbox.execute_command(
|
||||||
|
"side-effect; sleep 30",
|
||||||
|
env={"X": "1"},
|
||||||
|
timeout=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert executions == 1
|
||||||
|
assert "partial" in out
|
||||||
|
assert "Command timed out after 3 seconds and was terminated." in out
|
||||||
|
assert out.endswith("Exit Code: 124")
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("status", ["timed_out", "killed"])
|
||||||
|
def test_env_interrupted_status_is_never_error_observation_retried(
|
||||||
|
self,
|
||||||
|
sandbox,
|
||||||
|
status,
|
||||||
|
):
|
||||||
|
executions = 0
|
||||||
|
|
||||||
|
def bash_exec(**kwargs):
|
||||||
|
nonlocal executions
|
||||||
|
executions += 1
|
||||||
|
return SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="'ErrorObservation' object has no attribute 'exit_code'",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=-1,
|
||||||
|
status=status,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
sandbox._client.bash.exec = bash_exec
|
||||||
|
|
||||||
|
sandbox.execute_command(
|
||||||
|
"unsafe-to-repeat",
|
||||||
|
env={"X": "1"},
|
||||||
|
timeout=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert executions == 1
|
||||||
|
|
||||||
|
def test_env_session_cleanup_is_bounded_and_does_not_mask_output(self, sandbox):
|
||||||
|
sandbox._client.bash.exec = MagicMock(
|
||||||
|
return_value=SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="ok",
|
||||||
|
stderr="",
|
||||||
|
exit_code=0,
|
||||||
|
status="completed",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
sandbox._client.bash.close_session = MagicMock(side_effect=RuntimeError("cleanup failed"))
|
||||||
|
|
||||||
|
assert (
|
||||||
|
sandbox.execute_command(
|
||||||
|
"echo $TOKEN",
|
||||||
|
env={"TOKEN": "secret"},
|
||||||
|
timeout=3,
|
||||||
|
)
|
||||||
|
== "ok"
|
||||||
|
)
|
||||||
|
|
||||||
|
_, kwargs = sandbox._client.bash.close_session.call_args
|
||||||
|
assert kwargs["request_options"] == {
|
||||||
|
"timeout_in_seconds": sandbox._CLEANUP_REQUEST_TIMEOUT_SECONDS,
|
||||||
|
"max_retries": 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("status", "notice"),
|
||||||
|
[
|
||||||
|
("killed", "Command was killed before completion."),
|
||||||
|
(
|
||||||
|
"running",
|
||||||
|
"Error: Sandbox command returned a non-terminal running status; command outcome is unknown and was not retried.",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_env_interrupted_status_is_rendered_without_exit_fallback(
|
||||||
|
self,
|
||||||
|
sandbox,
|
||||||
|
status,
|
||||||
|
notice,
|
||||||
|
):
|
||||||
|
sandbox._client.bash.exec = MagicMock(
|
||||||
|
return_value=SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="partial",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=-1,
|
||||||
|
status=status,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
out = sandbox.execute_command(
|
||||||
|
"unsafe-to-repeat",
|
||||||
|
env={"X": "1"},
|
||||||
|
timeout=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert notice in out
|
||||||
|
assert "Exit Code: -1" not in out
|
||||||
|
|
||||||
|
def test_env_transport_timeout_is_ambiguous_and_not_retried(self, sandbox):
|
||||||
|
executions = 0
|
||||||
|
|
||||||
|
def bash_exec(**kwargs):
|
||||||
|
nonlocal executions
|
||||||
|
executions += 1
|
||||||
|
raise httpx.ReadTimeout("response stalled")
|
||||||
|
|
||||||
|
sandbox._client.bash.exec = bash_exec
|
||||||
|
|
||||||
|
out = sandbox.execute_command(
|
||||||
|
"side-effect; sleep 30",
|
||||||
|
env={"X": "1"},
|
||||||
|
timeout=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert executions == 1
|
||||||
|
assert "outcome is unknown" in out
|
||||||
|
assert "not retried" in out
|
||||||
|
_, kwargs = sandbox._client.bash.close_session.call_args
|
||||||
|
assert kwargs["request_options"] == {
|
||||||
|
"timeout_in_seconds": sandbox._CLEANUP_REQUEST_TIMEOUT_SECONDS,
|
||||||
|
"max_retries": 0,
|
||||||
|
}
|
||||||
|
|
||||||
def test_env_path_retries_on_error_observation_signature(self, sandbox):
|
def test_env_path_retries_on_error_observation_signature(self, sandbox):
|
||||||
"""The env path shares the legacy persistent-shell recovery contract: if
|
"""The env path shares the legacy persistent-shell recovery contract: if
|
||||||
the (unlikely, fresh-session) corruption marker appears, the call is
|
the (unlikely, fresh-session) corruption marker appears, the call is
|
||||||
@ -153,6 +326,89 @@ class TestAioSandboxEnvInjection:
|
|||||||
assert "recovered" in out
|
assert "recovered" in out
|
||||||
assert _ERROR_OBSERVATION_SIGNATURE not in out
|
assert _ERROR_OBSERVATION_SIGNATURE not in out
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("retry_result", "expected_fragments", "expected_suffix"),
|
||||||
|
[
|
||||||
|
pytest.param(
|
||||||
|
SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="authoritative timeout output",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=-1,
|
||||||
|
status="timed_out",
|
||||||
|
)
|
||||||
|
),
|
||||||
|
("Command timed out after 3 seconds and was terminated.",),
|
||||||
|
"Exit Code: 124",
|
||||||
|
id="timed-out",
|
||||||
|
),
|
||||||
|
pytest.param(
|
||||||
|
SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="authoritative killed output",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=-1,
|
||||||
|
status="killed",
|
||||||
|
)
|
||||||
|
),
|
||||||
|
("Command was killed before completion.",),
|
||||||
|
None,
|
||||||
|
id="killed",
|
||||||
|
),
|
||||||
|
pytest.param(
|
||||||
|
SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout="authoritative running output",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=-1,
|
||||||
|
status="running",
|
||||||
|
)
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"non-terminal running status",
|
||||||
|
"outcome is unknown",
|
||||||
|
"not retried",
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
id="running",
|
||||||
|
),
|
||||||
|
pytest.param(
|
||||||
|
httpx.ReadTimeout("response stalled"),
|
||||||
|
("outcome is unknown", "not retried"),
|
||||||
|
None,
|
||||||
|
id="transport-timeout",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_env_error_observation_retry_returns_authoritative_outcome(
|
||||||
|
self,
|
||||||
|
sandbox,
|
||||||
|
retry_result,
|
||||||
|
expected_fragments,
|
||||||
|
expected_suffix,
|
||||||
|
):
|
||||||
|
from deerflow.community.aio_sandbox.aio_sandbox import _ERROR_OBSERVATION_SIGNATURE
|
||||||
|
|
||||||
|
corrupted = SimpleNamespace(
|
||||||
|
data=SimpleNamespace(
|
||||||
|
stdout=f"corrupted: {_ERROR_OBSERVATION_SIGNATURE}",
|
||||||
|
stderr=None,
|
||||||
|
exit_code=0,
|
||||||
|
status="completed",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
sandbox._client.bash.exec = MagicMock(side_effect=[corrupted, retry_result])
|
||||||
|
|
||||||
|
out = sandbox.execute_command("unsafe-to-repeat", env={"X": "1"}, timeout=3)
|
||||||
|
|
||||||
|
assert sandbox._client.bash.exec.call_count == 2
|
||||||
|
assert "corrupted:" not in out
|
||||||
|
assert _ERROR_OBSERVATION_SIGNATURE not in out
|
||||||
|
for expected_fragment in expected_fragments:
|
||||||
|
assert expected_fragment in out
|
||||||
|
if expected_suffix is not None:
|
||||||
|
assert out.endswith(expected_suffix)
|
||||||
|
|
||||||
|
|
||||||
class TestEnvPolicy:
|
class TestEnvPolicy:
|
||||||
"""Platform-secret scrubbing policy for sandbox subprocesses (delta 1)."""
|
"""Platform-secret scrubbing policy for sandbox subprocesses (delta 1)."""
|
||||||
@ -1077,6 +1333,55 @@ class TestBashToolInjectsActiveSecrets:
|
|||||||
assert captured["env"] == {"ERP_TOKEN": "tok-456"}
|
assert captured["env"] == {"ERP_TOKEN": "tok-456"}
|
||||||
assert captured["timeout"] == 42
|
assert captured["timeout"] == 42
|
||||||
|
|
||||||
|
def test_remote_bash_does_not_forward_shared_timeout(self):
|
||||||
|
from deerflow.sandbox import tools as tools_mod
|
||||||
|
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
class FakeSandbox:
|
||||||
|
def execute_command(self, command, env=None, timeout=None):
|
||||||
|
captured["command"] = command
|
||||||
|
captured["env"] = env
|
||||||
|
captured["timeout"] = timeout
|
||||||
|
return "done"
|
||||||
|
|
||||||
|
runtime = SimpleNamespace(
|
||||||
|
context={},
|
||||||
|
state={"sandbox": {"sandbox_id": "aio:1"}},
|
||||||
|
)
|
||||||
|
fake_cfg = SimpleNamespace(
|
||||||
|
sandbox=SimpleNamespace(
|
||||||
|
bash_output_max_chars=321,
|
||||||
|
bash_command_timeout=42,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch.object(
|
||||||
|
tools_mod,
|
||||||
|
"ensure_sandbox_initialized",
|
||||||
|
return_value=FakeSandbox(),
|
||||||
|
),
|
||||||
|
patch.object(tools_mod, "is_local_sandbox", return_value=False),
|
||||||
|
patch.object(
|
||||||
|
tools_mod,
|
||||||
|
"ensure_thread_directories_exist",
|
||||||
|
return_value=None,
|
||||||
|
),
|
||||||
|
patch(
|
||||||
|
"deerflow.config.app_config.get_app_config",
|
||||||
|
return_value=fake_cfg,
|
||||||
|
),
|
||||||
|
):
|
||||||
|
out = tools_mod.bash_tool.func(
|
||||||
|
runtime=runtime,
|
||||||
|
command="echo hi",
|
||||||
|
description="run remote",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert out == "done"
|
||||||
|
assert captured["timeout"] is None
|
||||||
|
|
||||||
|
|
||||||
_SECRET = "sk-erp-9f3c-DO-NOT-LEAK"
|
_SECRET = "sk-erp-9f3c-DO-NOT-LEAK"
|
||||||
|
|
||||||
|
|||||||
@ -1500,13 +1500,17 @@ sandbox:
|
|||||||
read_file_output_max_chars: 50000
|
read_file_output_max_chars: 50000
|
||||||
ls_output_max_chars: 20000
|
ls_output_max_chars: 20000
|
||||||
|
|
||||||
# Maximum wall-clock seconds a single host bash command may run before it is
|
# Provider command deadline (default: 600). AIO images on the supported
|
||||||
# terminated (process group and all). A blocking foreground command — e.g. a
|
# semver line (1.9.3+, recommended 1.11.0) enforce it server-side through
|
||||||
# server started without backgrounding — is killed after this long so the
|
# `hard_timeout`; the frozen legacy `all-in-one-sandbox:latest` image only
|
||||||
# agent's turn cannot hang. Start long-lived processes in the background with
|
# gets the bounded host-side request. `bash_command_timeout` is used by
|
||||||
# output redirected (e.g. `your-command > /tmp/server.log 2>&1 &`) when you
|
# providers that explicitly wire this setting (currently LocalSandbox,
|
||||||
# need logs; unredirected background output is drained with bounded capture
|
# AioSandbox, and OpenSandbox). Other providers retain their provider-specific
|
||||||
# and excess output is discarded.
|
# command defaults unless a caller supplies an explicit timeout. For providers
|
||||||
|
# that enforce this deadline server- or process-side, blocking foreground
|
||||||
|
# commands are terminated at the configured deadline. Long-lived processes
|
||||||
|
# should be started in the background with output redirected; unredirected
|
||||||
|
# background output is drained with bounded capture.
|
||||||
bash_command_timeout: 600
|
bash_command_timeout: 600
|
||||||
|
|
||||||
# Option 2: Container-based AIO Sandbox
|
# Option 2: Container-based AIO Sandbox
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user