mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-11 06:28:58 +00:00
feat(sandbox): add OpenSandbox provider (#4877)
This commit is contained in:
parent
4e35f0d1d4
commit
917fe595fc
@ -497,6 +497,47 @@ Notes specific to `E2BSandboxProvider`:
|
|||||||
sandbox and surfaced through the standard artifact pipeline) to ship files
|
sandbox and surfaced through the standard artifact pipeline) to ship files
|
||||||
back to the gateway.
|
back to the gateway.
|
||||||
|
|
||||||
|
**OpenSandbox Remote Sandbox** (runs code through an OpenSandbox deployment):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sandbox:
|
||||||
|
use: deerflow.community.opensandbox:OpenSandboxProvider
|
||||||
|
image: python:3.11
|
||||||
|
api_key: $OPEN_SANDBOX_API_KEY # optional when the SDK env var is set
|
||||||
|
domain: localhost:8080 # OPEN_SANDBOX_DOMAIN fallback
|
||||||
|
protocol: http
|
||||||
|
request_timeout: 30 # management request timeout seconds
|
||||||
|
ready_timeout: 30 # create/readiness timeout seconds
|
||||||
|
use_server_proxy: false # proxy execd/file traffic through server
|
||||||
|
sandbox_timeout: 14400 # remote lifetime; 0 = explicit cleanup
|
||||||
|
bash_command_timeout: 600 # default remote command timeout seconds
|
||||||
|
replicas: 3 # active + warm cap per gateway process
|
||||||
|
idle_timeout: 600 # warm seconds before destroy; 0 disables
|
||||||
|
environment:
|
||||||
|
PYTHONUNBUFFERED: "1"
|
||||||
|
```
|
||||||
|
|
||||||
|
Install the optional SDK before selecting this provider:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install "deerflow-harness[opensandbox]"
|
||||||
|
```
|
||||||
|
|
||||||
|
The provider creates a sandbox per effective user/thread scope and parks it in
|
||||||
|
an in-process warm pool after each turn. The same scope can reclaim it after a
|
||||||
|
health check; another user or thread cannot. Create-time readiness and
|
||||||
|
`/mnt/user-data/{workspace,uploads,outputs}` bootstrap failures are cleaned up
|
||||||
|
before `acquire()` returns. Each remote owns an independent SDK transport.
|
||||||
|
Operations renew the configured server-side lifetime, and commands without an
|
||||||
|
explicit timeout use `bash_command_timeout`; a longer explicit timeout extends
|
||||||
|
the renewal horizon to cover the command. Operations on one remote are
|
||||||
|
serialized so a shorter renewal cannot overwrite an in-flight command's
|
||||||
|
horizon. File transfer uses OpenSandbox's native filesystem API; bounded
|
||||||
|
`find`/`grep` commands implement the directory and content-search surface.
|
||||||
|
Downloads are restricted to `/mnt/user-data` and all file paths reject
|
||||||
|
traversal. Multi-process discovery and ownership coordination are not yet
|
||||||
|
implemented, so `replicas` is a per-Gateway-process soft cap.
|
||||||
|
|
||||||
Choose between local execution or Docker-based isolation:
|
Choose between local execution or Docker-based isolation:
|
||||||
|
|
||||||
**Option 1: Local Sandbox** (default, simpler setup):
|
**Option 1: Local Sandbox** (default, simpler setup):
|
||||||
|
|||||||
@ -0,0 +1,75 @@
|
|||||||
|
# OpenSandbox backend
|
||||||
|
|
||||||
|
Runs DeerFlow sandboxes on [OpenSandbox](https://github.com/opensandbox-group/OpenSandbox),
|
||||||
|
using the synchronous Python SDK behind DeerFlow's `Sandbox` and
|
||||||
|
`SandboxProvider` contracts.
|
||||||
|
|
||||||
|
## Installation and configuration
|
||||||
|
|
||||||
|
Install the optional SDK, then select the provider:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install "deerflow-harness[opensandbox]"
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sandbox:
|
||||||
|
use: deerflow.community.opensandbox:OpenSandboxProvider
|
||||||
|
image: python:3.11
|
||||||
|
# api_key: $OPEN_SANDBOX_API_KEY
|
||||||
|
# domain: localhost:8080
|
||||||
|
# protocol: http
|
||||||
|
# request_timeout: 30
|
||||||
|
# ready_timeout: 30
|
||||||
|
# use_server_proxy: false
|
||||||
|
# sandbox_timeout: 14400 # remote lifetime; 0 means explicit cleanup only
|
||||||
|
# bash_command_timeout: 600 # default command timeout
|
||||||
|
# replicas: 3 # active + warm sandboxes per gateway process
|
||||||
|
# idle_timeout: 600 # warm seconds before destroy; 0 disables reaping
|
||||||
|
# environment:
|
||||||
|
# PYTHONUNBUFFERED: "1"
|
||||||
|
```
|
||||||
|
|
||||||
|
`api_key` and `domain` may be omitted when `OPEN_SANDBOX_API_KEY` and
|
||||||
|
`OPEN_SANDBOX_DOMAIN` are set. `use_server_proxy` is useful when DeerFlow can
|
||||||
|
reach the OpenSandbox management service but cannot directly reach sandbox
|
||||||
|
`execd` endpoints.
|
||||||
|
|
||||||
|
Values in `sandbox.environment` that start with `$` are resolved from the
|
||||||
|
Gateway process environment when the provider starts. Missing variables resolve
|
||||||
|
to an empty string, matching the E2B provider.
|
||||||
|
|
||||||
|
## Lifecycle and contract
|
||||||
|
|
||||||
|
The provider derives a stable local ID from `(user_id, thread_id)`. A released
|
||||||
|
sandbox enters an in-process warm pool and only the same scope may reclaim it.
|
||||||
|
Create returns only after the SDK readiness check and DeerFlow's
|
||||||
|
`/mnt/user-data/{workspace,uploads,outputs}` bootstrap succeed. A bootstrap
|
||||||
|
failure explicitly destroys the newly created remote sandbox.
|
||||||
|
Each remote has an independent SDK connection transport. Before an operation,
|
||||||
|
the adapter renews `sandbox_timeout`; commands use `bash_command_timeout` when
|
||||||
|
the caller supplies no timeout and extend the renewal horizon when necessary.
|
||||||
|
Operations are serialized per remote so a shorter renewal cannot overwrite the
|
||||||
|
horizon of an in-flight long command.
|
||||||
|
Setting `sandbox_timeout: 0` selects explicit-cleanup mode and disables renewal.
|
||||||
|
|
||||||
|
The full DeerFlow surface is implemented:
|
||||||
|
|
||||||
|
- `execute_command` forwards per-call environment variables and positive
|
||||||
|
wall-clock timeouts through `RunCommandOpts`, preserving stdout, stderr, and
|
||||||
|
non-zero exit information in DeerFlow's string result.
|
||||||
|
- Text and binary file operations use OpenSandbox's native filesystem API.
|
||||||
|
Append is serialized as a read-modify-write because SDK 0.1.x has no append
|
||||||
|
primitive.
|
||||||
|
- `list_dir`, `glob`, and `grep` use portable `find`/`grep` commands and the
|
||||||
|
shared DeerFlow result parsers.
|
||||||
|
- All paths must be absolute and traversal-free. Artifact downloads are further
|
||||||
|
restricted to `/mnt/user-data`.
|
||||||
|
- Command-path HTTP 404, HTTP 410, unhealthy-session errors, and broken
|
||||||
|
transports evict the dead client so the next acquire cold-starts a
|
||||||
|
replacement. A file-path 404 remains an ordinary missing-file error.
|
||||||
|
|
||||||
|
`reset()` parks active clients for later cleanup; `shutdown()` destroys active
|
||||||
|
and warm remotes. Cross-process discovery and ownership coordination are not
|
||||||
|
implemented yet, so each Gateway process has its own warm pool and capacity
|
||||||
|
accounting.
|
||||||
@ -0,0 +1,6 @@
|
|||||||
|
"""OpenSandbox community provider for DeerFlow."""
|
||||||
|
|
||||||
|
from .provider import OpenSandboxProvider
|
||||||
|
from .sandbox import OpenSandboxSandbox
|
||||||
|
|
||||||
|
__all__ = ["OpenSandboxProvider", "OpenSandboxSandbox"]
|
||||||
@ -0,0 +1,389 @@
|
|||||||
|
"""OpenSandbox-backed community ``SandboxProvider`` for DeerFlow."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import atexit
|
||||||
|
import hashlib
|
||||||
|
import ipaddress
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from datetime import timedelta
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from deerflow.config import get_app_config
|
||||||
|
from deerflow.sandbox.sandbox import Sandbox, _validate_extra_env
|
||||||
|
from deerflow.sandbox.sandbox_provider import SandboxProvider
|
||||||
|
|
||||||
|
from ..warm_pool_lifecycle import WarmPoolLifecycleMixin
|
||||||
|
from .sandbox import OpenSandboxSandbox, format_execution
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from opensandbox.config.connection_sync import ConnectionConfigSync
|
||||||
|
from opensandbox.models.execd import RunCommandOpts
|
||||||
|
from opensandbox.sync import SandboxSync
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DEFAULT_IMAGE = "python:3.11"
|
||||||
|
DEFAULT_READY_TIMEOUT = 30.0
|
||||||
|
DEFAULT_REQUEST_TIMEOUT = 30.0
|
||||||
|
DEFAULT_SANDBOX_TIMEOUT = 4 * 60 * 60
|
||||||
|
DEFAULT_COMMAND_TIMEOUT = 10 * 60
|
||||||
|
_BOOTSTRAP_TIMEOUT = 30.0
|
||||||
|
_BOOTSTRAP_COMMAND = "mkdir -p /mnt/user-data/workspace /mnt/user-data/uploads /mnt/user-data/outputs"
|
||||||
|
|
||||||
|
|
||||||
|
def _uses_insecure_remote_http(domain: Any, protocol: str) -> bool:
|
||||||
|
if not domain:
|
||||||
|
return False
|
||||||
|
value = str(domain)
|
||||||
|
try:
|
||||||
|
parsed = urlsplit(value if "://" in value else f"{protocol}://{value}")
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
if parsed.scheme.lower() != "http" or not parsed.hostname:
|
||||||
|
return False
|
||||||
|
if parsed.hostname.lower() == "localhost":
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
return not ipaddress.ip_address(parsed.hostname).is_loopback
|
||||||
|
except ValueError:
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _import_sdk() -> tuple[type[SandboxSync], type[ConnectionConfigSync], type[RunCommandOpts]]:
|
||||||
|
"""Import the optional OpenSandbox sync SDK only when this provider is used."""
|
||||||
|
try:
|
||||||
|
from opensandbox.config.connection_sync import ConnectionConfigSync
|
||||||
|
from opensandbox.models.execd import RunCommandOpts
|
||||||
|
from opensandbox.sync import SandboxSync
|
||||||
|
except ImportError as exc: # pragma: no cover - depends on optional install state
|
||||||
|
raise ImportError("OpenSandboxProvider requires the optional 'opensandbox' dependency. Install it with: pip install 'deerflow-harness[opensandbox]' or pip install 'opensandbox>=0.1.15,<0.2.0'.") from exc
|
||||||
|
return SandboxSync, ConnectionConfigSync, RunCommandOpts
|
||||||
|
|
||||||
|
|
||||||
|
class OpenSandboxProvider(WarmPoolLifecycleMixin[OpenSandboxSandbox], SandboxProvider):
|
||||||
|
"""Create one OpenSandbox environment per effective user/thread scope."""
|
||||||
|
|
||||||
|
uses_thread_data_mounts = False
|
||||||
|
needs_upload_permission_adjustment = True
|
||||||
|
_idle_checker_thread_name = "opensandbox-idle-reaper"
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
self._sandboxes: dict[str, OpenSandboxSandbox] = {}
|
||||||
|
self._thread_sandboxes: dict[tuple[str, str], str] = {}
|
||||||
|
self._warm_pool: dict[str, tuple[OpenSandboxSandbox, float]] = {}
|
||||||
|
self._acquire_locks: dict[str, threading.Lock] = {}
|
||||||
|
self._idle_checker_stop = threading.Event()
|
||||||
|
self._idle_checker_thread: threading.Thread | None = None
|
||||||
|
self._shutdown_called = False
|
||||||
|
self._sdk: tuple[type[SandboxSync], type[ConnectionConfigSync], type[RunCommandOpts]] | None = None
|
||||||
|
self._config = self._load_config()
|
||||||
|
atexit.register(self.shutdown)
|
||||||
|
self._start_idle_checker()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _positive_float(name: str, value: Any, default: float) -> float:
|
||||||
|
resolved = float(default if value is None else value)
|
||||||
|
if resolved <= 0:
|
||||||
|
raise ValueError(f"sandbox.{name} must be positive")
|
||||||
|
return resolved
|
||||||
|
|
||||||
|
def _load_config(self) -> dict[str, Any]:
|
||||||
|
sandbox_config = get_app_config().sandbox
|
||||||
|
|
||||||
|
def option(name: str, default: Any = None) -> Any:
|
||||||
|
return getattr(sandbox_config, name, default)
|
||||||
|
|
||||||
|
api_key = option("api_key")
|
||||||
|
domain = option("domain")
|
||||||
|
protocol = option("protocol") or "http"
|
||||||
|
effective_domain = domain or os.environ.get("OPEN_SANDBOX_DOMAIN")
|
||||||
|
if not (api_key or os.environ.get("OPEN_SANDBOX_API_KEY")) and not effective_domain:
|
||||||
|
logger.warning("OpenSandboxProvider: no api_key or domain configured (set sandbox.api_key/sandbox.domain in config.yaml or OPEN_SANDBOX_API_KEY/OPEN_SANDBOX_DOMAIN). The SDK will default to unauthenticated localhost:8080.")
|
||||||
|
if _uses_insecure_remote_http(effective_domain, protocol):
|
||||||
|
logger.warning("OpenSandboxProvider: remote OpenSandbox domain uses HTTP; use HTTPS to protect credentials and sandbox traffic.")
|
||||||
|
environment = dict(option("environment") or {})
|
||||||
|
_validate_extra_env(environment)
|
||||||
|
replicas = option("replicas")
|
||||||
|
idle_timeout = option("idle_timeout")
|
||||||
|
raw_sandbox_timeout = option("sandbox_timeout")
|
||||||
|
sandbox_timeout = float(DEFAULT_SANDBOX_TIMEOUT if raw_sandbox_timeout is None else raw_sandbox_timeout)
|
||||||
|
if sandbox_timeout < 0:
|
||||||
|
raise ValueError("sandbox.sandbox_timeout must be non-negative")
|
||||||
|
return {
|
||||||
|
"api_key": api_key,
|
||||||
|
"domain": domain,
|
||||||
|
"protocol": protocol,
|
||||||
|
"request_timeout": self._positive_float("request_timeout", option("request_timeout"), DEFAULT_REQUEST_TIMEOUT),
|
||||||
|
"use_server_proxy": bool(option("use_server_proxy", False)),
|
||||||
|
"image": option("image") or DEFAULT_IMAGE,
|
||||||
|
"ready_timeout": self._positive_float("ready_timeout", option("ready_timeout"), DEFAULT_READY_TIMEOUT),
|
||||||
|
"sandbox_timeout": None if sandbox_timeout == 0 else sandbox_timeout,
|
||||||
|
"command_timeout": self._positive_float("bash_command_timeout", option("bash_command_timeout"), DEFAULT_COMMAND_TIMEOUT),
|
||||||
|
"environment": self._resolve_env_vars(environment),
|
||||||
|
"replicas": replicas if replicas is not None else self.DEFAULT_REPLICAS,
|
||||||
|
"idle_timeout": idle_timeout if idle_timeout is not None else self.DEFAULT_IDLE_TIMEOUT,
|
||||||
|
}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _resolve_env_vars(env_config: dict[str, str]) -> dict[str, str]:
|
||||||
|
resolved: dict[str, str] = {}
|
||||||
|
for key, value in env_config.items():
|
||||||
|
if isinstance(value, str) and value.startswith("$"):
|
||||||
|
resolved[key] = os.environ.get(value[1:], "")
|
||||||
|
else:
|
||||||
|
resolved[key] = "" if value is None else str(value)
|
||||||
|
return resolved
|
||||||
|
|
||||||
|
def _get_sdk(self) -> tuple[type[SandboxSync], type[ConnectionConfigSync], type[RunCommandOpts]]:
|
||||||
|
with self._lock:
|
||||||
|
sdk = self._sdk
|
||||||
|
if sdk is not None:
|
||||||
|
return sdk
|
||||||
|
imported = _import_sdk()
|
||||||
|
with self._lock:
|
||||||
|
if self._sdk is None:
|
||||||
|
self._sdk = imported
|
||||||
|
return imported
|
||||||
|
return self._sdk
|
||||||
|
|
||||||
|
def _new_connection_config(self, connection_config_cls: type[ConnectionConfigSync]) -> ConnectionConfigSync:
|
||||||
|
# SandboxSync.create() derives an SDK-owned transport on a config copy,
|
||||||
|
# and destroy() closes that transport. A fresh base config per remote
|
||||||
|
# ensures no live sandbox can inherit another sandbox's transport.
|
||||||
|
return connection_config_cls(
|
||||||
|
api_key=self._config["api_key"],
|
||||||
|
domain=self._config["domain"],
|
||||||
|
protocol=self._config["protocol"],
|
||||||
|
request_timeout=timedelta(seconds=self._config["request_timeout"]),
|
||||||
|
use_server_proxy=self._config["use_server_proxy"],
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _sandbox_id(thread_id: str, user_id: str) -> str:
|
||||||
|
return hashlib.sha256(f"{user_id}:{thread_id}".encode()).hexdigest()[:16]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _thread_key(thread_id: str, user_id: str | None) -> tuple[str, str]:
|
||||||
|
return (user_id or "", thread_id)
|
||||||
|
|
||||||
|
def _lock_for_sandbox(self, sandbox_id: str) -> threading.Lock:
|
||||||
|
with self._lock:
|
||||||
|
lock = self._acquire_locks.get(sandbox_id)
|
||||||
|
if lock is None:
|
||||||
|
lock = threading.Lock()
|
||||||
|
self._acquire_locks[sandbox_id] = lock
|
||||||
|
return lock
|
||||||
|
|
||||||
|
def _start_idle_checker(self) -> None:
|
||||||
|
if self._config["idle_timeout"] <= 0:
|
||||||
|
return
|
||||||
|
super()._start_idle_checker()
|
||||||
|
|
||||||
|
def _active_count_locked(self) -> int:
|
||||||
|
return len(self._sandboxes)
|
||||||
|
|
||||||
|
def _destroy_warm_entry(self, sandbox_id: str, entry: OpenSandboxSandbox, *, reason: str) -> None:
|
||||||
|
self._destroy_quietly(entry, context=f"warm pool, reason={reason}")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _destroy_quietly(sandbox: OpenSandboxSandbox, *, context: str) -> None:
|
||||||
|
try:
|
||||||
|
sandbox.destroy()
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Error destroying OpenSandbox %s (%s): %s", sandbox.id, context, exc)
|
||||||
|
|
||||||
|
def _invalidate_sandbox(self, sandbox_id: str, reason: str) -> None:
|
||||||
|
with self._lock:
|
||||||
|
active = self._sandboxes.pop(sandbox_id, None)
|
||||||
|
warm_entry = self._warm_pool.pop(sandbox_id, None)
|
||||||
|
for key in [key for key, value in self._thread_sandboxes.items() if value == sandbox_id]:
|
||||||
|
self._thread_sandboxes.pop(key, None)
|
||||||
|
sandbox = active or (warm_entry[0] if warm_entry is not None else None)
|
||||||
|
if sandbox is None:
|
||||||
|
return
|
||||||
|
logger.warning("Invalidating OpenSandbox %s after terminal failure: %s", sandbox_id, reason)
|
||||||
|
self._destroy_quietly(sandbox, context="terminal failure")
|
||||||
|
|
||||||
|
def acquire(self, thread_id: str | None = None, *, user_id: str | None = None) -> str:
|
||||||
|
with self._lock:
|
||||||
|
if self._shutdown_called:
|
||||||
|
raise RuntimeError("OpenSandboxProvider has been shut down")
|
||||||
|
if thread_id is None:
|
||||||
|
sandbox_id = str(uuid.uuid4())[:8]
|
||||||
|
sandbox = self._create_sandbox(sandbox_id, thread_id=None, user_id=user_id)
|
||||||
|
with self._lock:
|
||||||
|
if self._shutdown_called:
|
||||||
|
destroy_after_unlock = True
|
||||||
|
else:
|
||||||
|
self._sandboxes[sandbox_id] = sandbox
|
||||||
|
destroy_after_unlock = False
|
||||||
|
if destroy_after_unlock:
|
||||||
|
self._destroy_quietly(sandbox, context="created during shutdown")
|
||||||
|
raise RuntimeError("OpenSandboxProvider shut down during acquire")
|
||||||
|
return sandbox_id
|
||||||
|
|
||||||
|
key = self._thread_key(thread_id, user_id)
|
||||||
|
sandbox_id = self._sandbox_id(thread_id, user_id or "")
|
||||||
|
with self._lock_for_sandbox(sandbox_id):
|
||||||
|
with self._lock:
|
||||||
|
existing = self._thread_sandboxes.get(key)
|
||||||
|
active = self._sandboxes.get(existing) if existing is not None else None
|
||||||
|
if existing is not None and active is not None:
|
||||||
|
try:
|
||||||
|
active.renew()
|
||||||
|
return existing
|
||||||
|
except Exception:
|
||||||
|
# A terminal renewal failure invokes _invalidate_sandbox,
|
||||||
|
# which removes and closes this exact client. Rebuild in the
|
||||||
|
# same acquire; transient errors leave the registry intact
|
||||||
|
# and must remain visible to the caller.
|
||||||
|
with self._lock:
|
||||||
|
invalidated = self._sandboxes.get(existing) is not active and existing not in self._warm_pool
|
||||||
|
shutting_down = self._shutdown_called
|
||||||
|
if not invalidated or not active.is_closed or shutting_down:
|
||||||
|
raise
|
||||||
|
logger.info("Rebuilding terminal OpenSandbox %s during acquire", existing)
|
||||||
|
reclaimed = self._reclaim_warm_pool(sandbox_id)
|
||||||
|
if reclaimed is not None:
|
||||||
|
with self._lock:
|
||||||
|
if self._shutdown_called:
|
||||||
|
raise RuntimeError("OpenSandboxProvider shut down during acquire")
|
||||||
|
if reclaimed in self._sandboxes:
|
||||||
|
self._thread_sandboxes[key] = reclaimed
|
||||||
|
return reclaimed
|
||||||
|
sandbox = self._create_sandbox(sandbox_id, thread_id=thread_id, user_id=user_id)
|
||||||
|
with self._lock:
|
||||||
|
if self._shutdown_called:
|
||||||
|
destroy_after_unlock = True
|
||||||
|
else:
|
||||||
|
self._sandboxes[sandbox_id] = sandbox
|
||||||
|
self._thread_sandboxes[key] = sandbox_id
|
||||||
|
destroy_after_unlock = False
|
||||||
|
if destroy_after_unlock:
|
||||||
|
self._destroy_quietly(sandbox, context="created during shutdown")
|
||||||
|
raise RuntimeError("OpenSandboxProvider shut down during acquire")
|
||||||
|
return sandbox_id
|
||||||
|
|
||||||
|
def _create_sandbox(self, sandbox_id: str, *, thread_id: str | None, user_id: str | None) -> OpenSandboxSandbox:
|
||||||
|
replicas, total = self._replica_count()
|
||||||
|
if total >= replicas:
|
||||||
|
evicted = self._evict_oldest_warm()
|
||||||
|
self._log_replicas_soft_cap(replicas, sandbox_id, evicted)
|
||||||
|
|
||||||
|
sandbox_cls, connection_config_cls, run_command_opts_cls = self._get_sdk()
|
||||||
|
connection_config = self._new_connection_config(connection_config_cls)
|
||||||
|
metadata = {"deer_flow_provider": "opensandbox"}
|
||||||
|
if thread_id is not None:
|
||||||
|
metadata["deer_flow_thread"] = thread_id
|
||||||
|
if user_id is not None:
|
||||||
|
metadata["deer_flow_user"] = user_id
|
||||||
|
remote = sandbox_cls.create(
|
||||||
|
self._config["image"],
|
||||||
|
timeout=None if self._config["sandbox_timeout"] is None else timedelta(seconds=self._config["sandbox_timeout"]),
|
||||||
|
ready_timeout=timedelta(seconds=self._config["ready_timeout"]),
|
||||||
|
env=self._config["environment"] or None,
|
||||||
|
metadata=metadata,
|
||||||
|
connection_config=connection_config,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
bootstrap = remote.commands.run(
|
||||||
|
_BOOTSTRAP_COMMAND,
|
||||||
|
opts=run_command_opts_cls(
|
||||||
|
timeout=timedelta(seconds=_BOOTSTRAP_TIMEOUT),
|
||||||
|
envs=self._config["environment"] or None,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
exit_code = getattr(bootstrap, "exit_code", None)
|
||||||
|
if exit_code != 0:
|
||||||
|
detail = format_execution(bootstrap).strip() or ("no exit code" if exit_code is None else f"exit code {exit_code}")
|
||||||
|
raise RuntimeError(f"OpenSandbox bootstrap failed: {detail}")
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
remote.destroy()
|
||||||
|
except Exception:
|
||||||
|
logger.warning("Failed to destroy OpenSandbox %s after bootstrap failure", remote.id, exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
return OpenSandboxSandbox(
|
||||||
|
sandbox_id,
|
||||||
|
remote,
|
||||||
|
run_command_opts_cls=run_command_opts_cls,
|
||||||
|
default_env=self._config["environment"],
|
||||||
|
sandbox_timeout=None if self._config["sandbox_timeout"] is None else timedelta(seconds=self._config["sandbox_timeout"]),
|
||||||
|
default_command_timeout=self._config["command_timeout"],
|
||||||
|
on_terminal_failure=self._invalidate_sandbox,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _reclaim_warm_pool(self, sandbox_id: str) -> str | None:
|
||||||
|
with self._lock:
|
||||||
|
warm_entry = self._warm_pool.get(sandbox_id)
|
||||||
|
if warm_entry is None:
|
||||||
|
return None
|
||||||
|
sandbox = warm_entry[0]
|
||||||
|
if not sandbox.ping():
|
||||||
|
with self._lock:
|
||||||
|
removed = self._warm_pool.pop(sandbox_id, None)
|
||||||
|
if removed is not None:
|
||||||
|
self._destroy_warm_entry(sandbox_id, removed[0], reason="health_check_failed")
|
||||||
|
return None
|
||||||
|
with self._lock:
|
||||||
|
removed = self._warm_pool.pop(sandbox_id, None)
|
||||||
|
if removed is None:
|
||||||
|
return None
|
||||||
|
self._sandboxes[sandbox_id] = removed[0]
|
||||||
|
logger.info("Reclaimed warm OpenSandbox %s (remote=%s)", sandbox_id, sandbox.remote_id)
|
||||||
|
return sandbox_id
|
||||||
|
|
||||||
|
def get(self, sandbox_id: str) -> Sandbox | None:
|
||||||
|
with self._lock:
|
||||||
|
return self._sandboxes.get(sandbox_id)
|
||||||
|
|
||||||
|
def release(self, sandbox_id: str) -> None:
|
||||||
|
with self._lock:
|
||||||
|
sandbox = self._sandboxes.pop(sandbox_id, None)
|
||||||
|
for key in [key for key, value in self._thread_sandboxes.items() if value == sandbox_id]:
|
||||||
|
self._thread_sandboxes.pop(key, None)
|
||||||
|
if sandbox is None:
|
||||||
|
return
|
||||||
|
if self._shutdown_called:
|
||||||
|
destroy_after_unlock = True
|
||||||
|
else:
|
||||||
|
self._warm_pool[sandbox_id] = (sandbox, time.time())
|
||||||
|
destroy_after_unlock = False
|
||||||
|
if destroy_after_unlock:
|
||||||
|
self._destroy_quietly(sandbox, context="released during shutdown")
|
||||||
|
|
||||||
|
def reset(self) -> None:
|
||||||
|
"""Park active clients so the detached provider still owns their cleanup."""
|
||||||
|
with self._lock:
|
||||||
|
now = time.time()
|
||||||
|
for sandbox_id, sandbox in self._sandboxes.items():
|
||||||
|
self._warm_pool.setdefault(sandbox_id, (sandbox, now))
|
||||||
|
self._sandboxes.clear()
|
||||||
|
self._thread_sandboxes.clear()
|
||||||
|
self._acquire_locks.clear()
|
||||||
|
|
||||||
|
def shutdown(self) -> None:
|
||||||
|
with self._lock:
|
||||||
|
if self._shutdown_called:
|
||||||
|
return
|
||||||
|
self._shutdown_called = True
|
||||||
|
self._stop_idle_checker()
|
||||||
|
with self._lock:
|
||||||
|
sandboxes = list(self._sandboxes.values()) + [entry for entry, _ in self._warm_pool.values()]
|
||||||
|
self._sandboxes.clear()
|
||||||
|
self._warm_pool.clear()
|
||||||
|
self._thread_sandboxes.clear()
|
||||||
|
self._acquire_locks.clear()
|
||||||
|
for sandbox in sandboxes:
|
||||||
|
self._destroy_quietly(sandbox, context="shutdown")
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["OpenSandboxProvider"]
|
||||||
@ -0,0 +1,417 @@
|
|||||||
|
"""DeerFlow :class:`Sandbox` adapter for an OpenSandbox sync client."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import errno
|
||||||
|
import logging
|
||||||
|
import posixpath
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import threading
|
||||||
|
from datetime import timedelta
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
from deerflow.config.paths import VIRTUAL_PATH_PREFIX
|
||||||
|
from deerflow.sandbox.sandbox import Sandbox, _validate_extra_env
|
||||||
|
from deerflow.sandbox.search import GrepMatch, path_matches, should_ignore_path, truncate_line
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
from opensandbox.sync import SandboxSync
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_TERMINAL_ERROR_NAMES = frozenset({"SandboxUnhealthyException"})
|
||||||
|
_COMMAND_TTL_GRACE = timedelta(seconds=30)
|
||||||
|
_MAX_DOWNLOAD_SIZE = 100 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
def _exception_chain(error: BaseException):
|
||||||
|
"""Yield an exception and its explicit causes without looping forever."""
|
||||||
|
seen: set[int] = set()
|
||||||
|
current: BaseException | None = error
|
||||||
|
while current is not None and id(current) not in seen:
|
||||||
|
seen.add(id(current))
|
||||||
|
yield current
|
||||||
|
current = current.__cause__
|
||||||
|
|
||||||
|
|
||||||
|
def _is_terminal_failure(error: BaseException, *, api_not_found_is_terminal: bool = False) -> bool:
|
||||||
|
"""Return whether an SDK failure means this remote sandbox is unusable."""
|
||||||
|
for item in _exception_chain(error):
|
||||||
|
if isinstance(item, (BrokenPipeError, ConnectionError, EOFError)):
|
||||||
|
return True
|
||||||
|
if type(item).__name__ in _TERMINAL_ERROR_NAMES:
|
||||||
|
return True
|
||||||
|
status_code = getattr(item, "status_code", None)
|
||||||
|
if status_code == 410 or (api_not_found_is_terminal and status_code == 404):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _is_not_found(error: BaseException) -> bool:
|
||||||
|
for item in _exception_chain(error):
|
||||||
|
if isinstance(item, FileNotFoundError) or getattr(item, "status_code", None) == 404:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _join_event_text(chunks) -> str:
|
||||||
|
"""Reconstruct the line-oriented text emitted by OpenSandbox SSE events."""
|
||||||
|
return "\n".join(str(chunk).rstrip("\n") for chunk in chunks)
|
||||||
|
|
||||||
|
|
||||||
|
def _append_output(output: str, value: str) -> str:
|
||||||
|
if not value:
|
||||||
|
return output
|
||||||
|
if not output or output.endswith("\n"):
|
||||||
|
return output + value
|
||||||
|
return f"{output}\n{value}"
|
||||||
|
|
||||||
|
|
||||||
|
def execution_stdout(execution: Any) -> str:
|
||||||
|
return _join_event_text(message.text for message in getattr(getattr(execution, "logs", None), "stdout", []))
|
||||||
|
|
||||||
|
|
||||||
|
def format_execution(execution: Any) -> str:
|
||||||
|
"""Combine stdout, result text, and stderr using DeerFlow's string contract."""
|
||||||
|
output = execution_stdout(execution)
|
||||||
|
result = _join_event_text(item.text for item in getattr(execution, "result", []) if getattr(item, "text", None) is not None)
|
||||||
|
output = _append_output(output, result)
|
||||||
|
stderr = _join_event_text(message.text for message in getattr(getattr(execution, "logs", None), "stderr", []))
|
||||||
|
output = _append_output(output, stderr)
|
||||||
|
error = getattr(execution, "error", None)
|
||||||
|
if error is not None:
|
||||||
|
detail = f"{getattr(error, 'name', type(error).__name__)}: {getattr(error, 'value', error)}"
|
||||||
|
output = _append_output(output, detail)
|
||||||
|
return output
|
||||||
|
|
||||||
|
|
||||||
|
class OpenSandboxSandbox(Sandbox):
|
||||||
|
"""Wrap one live ``opensandbox.sync.SandboxSync`` instance."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
id: str,
|
||||||
|
sandbox: SandboxSync,
|
||||||
|
*,
|
||||||
|
run_command_opts_cls: Callable[..., Any],
|
||||||
|
default_env: dict[str, str] | None = None,
|
||||||
|
sandbox_timeout: timedelta | None = None,
|
||||||
|
default_command_timeout: float = 600,
|
||||||
|
on_terminal_failure: Callable[[str, str], None] | None = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(id)
|
||||||
|
if sandbox_timeout is not None and sandbox_timeout.total_seconds() <= 0:
|
||||||
|
raise ValueError("sandbox_timeout must be positive or None")
|
||||||
|
if default_command_timeout <= 0:
|
||||||
|
raise ValueError("default_command_timeout must be positive")
|
||||||
|
self._sandbox = sandbox
|
||||||
|
self._run_command_opts_cls = run_command_opts_cls
|
||||||
|
self._default_env = dict(default_env or {})
|
||||||
|
self._sandbox_timeout = sandbox_timeout
|
||||||
|
self._default_command_timeout = float(default_command_timeout)
|
||||||
|
self._on_terminal_failure = on_terminal_failure
|
||||||
|
self._state_lock = threading.Lock()
|
||||||
|
# renew() sets an absolute expiration instead of taking a maximum. Keep
|
||||||
|
# each renewal and its operation under one lock so a later short file
|
||||||
|
# operation cannot shorten the horizon of a long-running command.
|
||||||
|
self._operation_lock = threading.Lock()
|
||||||
|
self._append_lock = threading.Lock()
|
||||||
|
self._closed = False
|
||||||
|
|
||||||
|
@property
|
||||||
|
def remote_id(self) -> str:
|
||||||
|
return str(self._sandbox.id)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_closed(self) -> bool:
|
||||||
|
with self._state_lock:
|
||||||
|
return self._closed
|
||||||
|
|
||||||
|
def destroy(self) -> None:
|
||||||
|
"""Terminate the remote sandbox and close its SDK resources."""
|
||||||
|
with self._operation_lock:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._closed:
|
||||||
|
return
|
||||||
|
error: Exception | None = None
|
||||||
|
try:
|
||||||
|
self._sandbox.destroy()
|
||||||
|
except Exception as exc: # SDK errors are normalized below.
|
||||||
|
error = exc
|
||||||
|
finally:
|
||||||
|
# SandboxSync.destroy() closes its transport even when kill fails,
|
||||||
|
# so this client cannot safely be reused after either outcome.
|
||||||
|
with self._state_lock:
|
||||||
|
self._closed = True
|
||||||
|
if error is not None and not _is_terminal_failure(error, api_not_found_is_terminal=True):
|
||||||
|
raise error
|
||||||
|
|
||||||
|
def _note_failure(self, error: Exception, *, api_not_found_is_terminal: bool = False) -> None:
|
||||||
|
if self._on_terminal_failure is None or not _is_terminal_failure(error, api_not_found_is_terminal=api_not_found_is_terminal):
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self._on_terminal_failure(self.id, str(error))
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Terminal OpenSandbox failure callback errored for %s", self.id)
|
||||||
|
|
||||||
|
def renew(self) -> None:
|
||||||
|
"""Refresh this provider-owned remote's server-side lifetime."""
|
||||||
|
if self._sandbox_timeout is None:
|
||||||
|
return
|
||||||
|
with self._operation_lock:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._closed:
|
||||||
|
raise RuntimeError("sandbox has been closed")
|
||||||
|
try:
|
||||||
|
self._sandbox.renew(self._sandbox_timeout)
|
||||||
|
return
|
||||||
|
except Exception as exc:
|
||||||
|
failure = exc
|
||||||
|
self._note_failure(failure, api_not_found_is_terminal=True)
|
||||||
|
raise failure
|
||||||
|
|
||||||
|
def _run(self, command: str, *, env: dict[str, str] | None = None, timeout: float | None = None) -> Any:
|
||||||
|
command_timeout = self._default_command_timeout if timeout is None else float(timeout)
|
||||||
|
if command_timeout <= 0:
|
||||||
|
raise ValueError(f"timeout must be positive, got {timeout}")
|
||||||
|
sdk_timeout = timedelta(seconds=command_timeout)
|
||||||
|
renewal_timeout = self._sandbox_timeout
|
||||||
|
if renewal_timeout is not None:
|
||||||
|
renewal_timeout = max(renewal_timeout, sdk_timeout + _COMMAND_TTL_GRACE)
|
||||||
|
opts = self._run_command_opts_cls(timeout=sdk_timeout, envs=env)
|
||||||
|
with self._operation_lock:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._closed:
|
||||||
|
raise RuntimeError("sandbox has been closed")
|
||||||
|
try:
|
||||||
|
if renewal_timeout is not None:
|
||||||
|
self._sandbox.renew(renewal_timeout)
|
||||||
|
return self._sandbox.commands.run(command, opts=opts)
|
||||||
|
except Exception as exc:
|
||||||
|
failure = exc
|
||||||
|
# A command-path 404 means the execd endpoint/sandbox is gone. File APIs
|
||||||
|
# use 404 for an ordinary missing path, so only command operations opt in.
|
||||||
|
self._note_failure(failure, api_not_found_is_terminal=True)
|
||||||
|
raise failure
|
||||||
|
|
||||||
|
def _file_op(self, operation):
|
||||||
|
with self._operation_lock:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._closed:
|
||||||
|
raise RuntimeError("sandbox has been closed")
|
||||||
|
try:
|
||||||
|
if self._sandbox_timeout is not None:
|
||||||
|
self._sandbox.renew(self._sandbox_timeout)
|
||||||
|
except Exception as exc:
|
||||||
|
failure = exc
|
||||||
|
renewal_failed = True
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
return operation(self._sandbox.files)
|
||||||
|
except Exception as exc:
|
||||||
|
failure = exc
|
||||||
|
renewal_failed = False
|
||||||
|
self._note_failure(failure, api_not_found_is_terminal=renewal_failed)
|
||||||
|
raise failure
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _resolve_path(path: str) -> str:
|
||||||
|
if not isinstance(path, str) or not path:
|
||||||
|
raise ValueError("path must be a non-empty string")
|
||||||
|
normalized = path.replace("\\", "/")
|
||||||
|
if not normalized.startswith("/"):
|
||||||
|
raise ValueError(f"path must be absolute: '{path}'")
|
||||||
|
if any(segment == ".." for segment in normalized.split("/")):
|
||||||
|
raise PermissionError(f"Access denied: path traversal detected in '{path}'")
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _resolve_download_path(cls, path: str) -> str:
|
||||||
|
normalized = cls._resolve_path(path)
|
||||||
|
stripped = normalized.lstrip("/")
|
||||||
|
allowed = VIRTUAL_PATH_PREFIX.lstrip("/")
|
||||||
|
if stripped != allowed and not stripped.startswith(f"{allowed}/"):
|
||||||
|
raise PermissionError(f"Access denied: path must be under '{VIRTUAL_PATH_PREFIX}': '{path}'")
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
def execute_command(self, command: str, env: dict[str, str] | None = None, timeout: float | None = None) -> str:
|
||||||
|
_validate_extra_env(env)
|
||||||
|
merged_env = {**self._default_env, **(env or {})} or None
|
||||||
|
try:
|
||||||
|
execution = self._run(command, env=merged_env, timeout=timeout)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.error("Failed to execute command in OpenSandbox %s: %s", self.id, exc)
|
||||||
|
return f"Error: {exc}"
|
||||||
|
output = format_execution(execution)
|
||||||
|
exit_code = getattr(execution, "exit_code", None)
|
||||||
|
if exit_code is None:
|
||||||
|
detail = output or "no completion or error event"
|
||||||
|
return f"Error: OpenSandbox command completed without an exit code: {detail}"
|
||||||
|
if exit_code != 0 and not output:
|
||||||
|
output = f"Command exited with code {exit_code}"
|
||||||
|
return output if output else "(no output)"
|
||||||
|
|
||||||
|
def read_file(self, path: str, start_line: int | None = None, end_line: int | None = None) -> str:
|
||||||
|
resolved = self._resolve_path(path)
|
||||||
|
try:
|
||||||
|
content = self._file_op(lambda files: files.read_file(resolved))
|
||||||
|
except Exception as exc:
|
||||||
|
logger.error("Failed to read OpenSandbox file %s: %s", resolved, exc)
|
||||||
|
return f"Error: {exc}"
|
||||||
|
if start_line is None and end_line is None:
|
||||||
|
return content or ""
|
||||||
|
lines = (content or "").splitlines()
|
||||||
|
start = start_line or 1
|
||||||
|
end = end_line if end_line is not None else len(lines)
|
||||||
|
return "\n".join(lines[start - 1 : end])
|
||||||
|
|
||||||
|
def write_file(self, path: str, content: str, append: bool = False) -> None:
|
||||||
|
resolved = self._resolve_path(path)
|
||||||
|
if not append:
|
||||||
|
self._file_op(lambda files: files.write_file(resolved, content, mode=644))
|
||||||
|
return
|
||||||
|
with self._append_lock:
|
||||||
|
try:
|
||||||
|
previous = self._file_op(lambda files: files.read_bytes(resolved))
|
||||||
|
except Exception as exc:
|
||||||
|
if not _is_not_found(exc):
|
||||||
|
raise
|
||||||
|
previous = b""
|
||||||
|
data = previous + content.encode("utf-8")
|
||||||
|
self._file_op(lambda files: files.write_file(resolved, data, mode=644))
|
||||||
|
|
||||||
|
def update_file(self, path: str, content: bytes) -> None:
|
||||||
|
resolved = self._resolve_path(path)
|
||||||
|
self._file_op(lambda files: files.write_file(resolved, content, mode=644))
|
||||||
|
|
||||||
|
def download_file(self, path: str) -> bytes:
|
||||||
|
resolved = self._resolve_download_path(path)
|
||||||
|
|
||||||
|
def read_bounded(files) -> bytes:
|
||||||
|
chunks: list[bytes] = []
|
||||||
|
total = 0
|
||||||
|
stream = files.read_bytes_stream(resolved)
|
||||||
|
try:
|
||||||
|
for chunk in stream:
|
||||||
|
total += len(chunk)
|
||||||
|
if total > _MAX_DOWNLOAD_SIZE:
|
||||||
|
raise OSError(errno.EFBIG, f"File exceeds maximum download size of {_MAX_DOWNLOAD_SIZE} bytes", path)
|
||||||
|
chunks.append(chunk)
|
||||||
|
finally:
|
||||||
|
close = getattr(stream, "close", None)
|
||||||
|
if callable(close):
|
||||||
|
close()
|
||||||
|
return b"".join(chunks)
|
||||||
|
|
||||||
|
try:
|
||||||
|
return self._file_op(read_bounded)
|
||||||
|
except OSError:
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
raise OSError(f"cannot read '{path}' from OpenSandbox: {exc}") from exc
|
||||||
|
|
||||||
|
def list_dir(self, path: str, max_depth: int = 2) -> list[str]:
|
||||||
|
depth = int(max_depth)
|
||||||
|
if depth < 0:
|
||||||
|
raise ValueError("max_depth must be non-negative")
|
||||||
|
resolved = self._resolve_path(path)
|
||||||
|
execution = self._run(f"find {shlex.quote(resolved)} -maxdepth {depth} \\( -type f -o -type d \\) 2>/dev/null | head -500")
|
||||||
|
return [line.strip() for line in execution_stdout(execution).splitlines() if line.strip()]
|
||||||
|
|
||||||
|
def glob(self, path: str, pattern: str, *, include_dirs: bool = False, max_results: int = 200) -> tuple[list[str], bool]:
|
||||||
|
if max_results <= 0:
|
||||||
|
raise ValueError("max_results must be positive")
|
||||||
|
resolved = self._resolve_path(path)
|
||||||
|
types = ("f", "d") if include_dirs else ("f",)
|
||||||
|
type_expr = " -o ".join(f"-type {entry_type}" for entry_type in types)
|
||||||
|
hard_limit = max(max_results * 4, max_results + 50)
|
||||||
|
execution = self._run(f"find {shlex.quote(resolved)} \\( {type_expr} \\) -print 2>/dev/null | head -{hard_limit}")
|
||||||
|
|
||||||
|
matches: list[str] = []
|
||||||
|
root = resolved.rstrip("/") or "/"
|
||||||
|
root_prefix = root if root == "/" else f"{root}/"
|
||||||
|
for entry in execution_stdout(execution).splitlines():
|
||||||
|
entry = entry.strip()
|
||||||
|
if not entry or (entry != root and not entry.startswith(root_prefix)) or should_ignore_path(entry):
|
||||||
|
continue
|
||||||
|
relative = entry[len(root) :].lstrip("/")
|
||||||
|
if relative and path_matches(pattern, relative):
|
||||||
|
matches.append(entry)
|
||||||
|
if len(matches) >= max_results:
|
||||||
|
return matches, True
|
||||||
|
return matches, False
|
||||||
|
|
||||||
|
def grep(
|
||||||
|
self,
|
||||||
|
path: str,
|
||||||
|
pattern: str,
|
||||||
|
*,
|
||||||
|
glob: str | None = None,
|
||||||
|
literal: bool = False,
|
||||||
|
case_sensitive: bool = False,
|
||||||
|
max_results: int = 100,
|
||||||
|
) -> tuple[list[GrepMatch], bool]:
|
||||||
|
if max_results <= 0:
|
||||||
|
raise ValueError("max_results must be positive")
|
||||||
|
if not literal:
|
||||||
|
re.compile(pattern, 0 if case_sensitive else re.IGNORECASE)
|
||||||
|
resolved = self._resolve_path(path)
|
||||||
|
flags = ["-r", "-H", "-n", "-I"]
|
||||||
|
if not case_sensitive:
|
||||||
|
flags.append("-i")
|
||||||
|
flags.append("-F" if literal else "-E")
|
||||||
|
portable_flags = list(flags)
|
||||||
|
if glob is not None:
|
||||||
|
include_pattern = glob.split("/")[-1] or glob
|
||||||
|
flags.append(shlex.quote(f"--include={include_pattern}"))
|
||||||
|
per_file_cap = max(max_results, 50)
|
||||||
|
flags.append(f"-m{per_file_cap}")
|
||||||
|
hard_limit = max(max_results * 4, max_results + 50)
|
||||||
|
arguments = f" -e {shlex.quote(pattern)} {shlex.quote(resolved)} 2>/dev/null"
|
||||||
|
primary = "grep " + " ".join(flags) + arguments
|
||||||
|
fallback = "grep " + " ".join(portable_flags) + arguments
|
||||||
|
command = f'{{ {primary}; status=$?; [ "$status" -eq 2 ] && {fallback}; }} | head -{hard_limit}'
|
||||||
|
execution = self._run(command)
|
||||||
|
|
||||||
|
root = resolved.rstrip("/") or "/"
|
||||||
|
root_prefix = root if root == "/" else f"{root}/"
|
||||||
|
matches: list[GrepMatch] = []
|
||||||
|
seen_positions: set[tuple[str, int]] = set()
|
||||||
|
for raw in execution_stdout(execution).splitlines():
|
||||||
|
try:
|
||||||
|
file_path, line_number_text, line = raw.split(":", 2)
|
||||||
|
line_number = int(line_number_text)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if should_ignore_path(file_path):
|
||||||
|
continue
|
||||||
|
if glob is not None:
|
||||||
|
if file_path != root and not file_path.startswith(root_prefix):
|
||||||
|
continue
|
||||||
|
relative = posixpath.basename(file_path) if file_path == root else file_path[len(root) :].lstrip("/")
|
||||||
|
if not path_matches(glob, relative):
|
||||||
|
continue
|
||||||
|
position = (file_path, line_number)
|
||||||
|
if position in seen_positions:
|
||||||
|
continue
|
||||||
|
seen_positions.add(position)
|
||||||
|
matches.append(GrepMatch(path=file_path, line_number=line_number, line=truncate_line(line)))
|
||||||
|
if len(matches) >= max_results:
|
||||||
|
return matches, True
|
||||||
|
return matches, False
|
||||||
|
|
||||||
|
def ping(self, timeout: float = 10) -> bool:
|
||||||
|
if self.is_closed:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
execution = self._run("true", timeout=timeout)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("OpenSandbox %s health check failed: %s", self.id, exc)
|
||||||
|
return False
|
||||||
|
return getattr(execution, "exit_code", None) == 0
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["OpenSandboxSandbox"]
|
||||||
@ -74,8 +74,8 @@ class SandboxConfig(BaseModel):
|
|||||||
allow_host_bash: Enable host-side bash execution for LocalSandboxProvider.
|
allow_host_bash: Enable host-side bash execution for LocalSandboxProvider.
|
||||||
Dangerous and intended only for fully trusted local workflows.
|
Dangerous and intended only for fully trusted local workflows.
|
||||||
|
|
||||||
AioSandboxProvider, BoxliteProvider, and E2BSandboxProvider shared options:
|
AioSandboxProvider, BoxliteProvider, E2BSandboxProvider, and OpenSandboxProvider shared options:
|
||||||
image: Sandbox image to use (Docker/AIO image or BoxLite OCI image)
|
image: Sandbox image to use (Docker/AIO, BoxLite OCI, or OpenSandbox image)
|
||||||
replicas: Positive provider capacity. E2B shares it across Gateway
|
replicas: Positive provider capacity. E2B shares it across Gateway
|
||||||
workers when ownership uses Redis; other modes/providers keep
|
workers when ownership uses Redis; other modes/providers keep
|
||||||
process-local accounting.
|
process-local accounting.
|
||||||
@ -95,6 +95,13 @@ class SandboxConfig(BaseModel):
|
|||||||
AioSandboxProvider and E2BSandboxProvider shared options:
|
AioSandboxProvider and E2BSandboxProvider shared options:
|
||||||
ownership: Cross-instance sandbox ownership store (memory | redis). Multi-instance
|
ownership: Cross-instance sandbox ownership store (memory | redis). Multi-instance
|
||||||
deployments sharing a sandbox backend need redis; see SandboxOwnershipConfig.
|
deployments sharing a sandbox backend need redis; see SandboxOwnershipConfig.
|
||||||
|
|
||||||
|
OpenSandboxProvider specific options:
|
||||||
|
api_key, domain, protocol, request_timeout, use_server_proxy: OpenSandbox
|
||||||
|
management and execd connection settings.
|
||||||
|
ready_timeout: Create/readiness deadline in seconds (default: 30).
|
||||||
|
sandbox_timeout: Remote lifetime in seconds (default: 14400); 0 requires
|
||||||
|
explicit provider cleanup.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
use: str = Field(
|
use: str = Field(
|
||||||
@ -107,7 +114,7 @@ class SandboxConfig(BaseModel):
|
|||||||
)
|
)
|
||||||
image: str | None = Field(
|
image: str | None = Field(
|
||||||
default=None,
|
default=None,
|
||||||
description="Sandbox image to use (Docker/AIO image or BoxLite OCI image)",
|
description="Sandbox image to use (Docker/AIO, BoxLite OCI, or OpenSandbox image)",
|
||||||
)
|
)
|
||||||
port: int | None = Field(
|
port: int | None = Field(
|
||||||
default=None,
|
default=None,
|
||||||
@ -184,8 +191,9 @@ class SandboxConfig(BaseModel):
|
|||||||
default=600,
|
default=600,
|
||||||
gt=0,
|
gt=0,
|
||||||
description=(
|
description=(
|
||||||
"Maximum wall-clock seconds a host bash command may run before it is terminated, process group and all (LocalSandboxProvider). "
|
"Maximum wall-clock seconds a bash command may run before it is terminated. LocalSandboxProvider applies it to the host process group; "
|
||||||
"Keeps a blocking foreground command (e.g. an un-backgrounded server) from hanging the turn; background `&` processes return immediately."
|
"OpenSandboxProvider forwards it to the remote exec service when a call has no explicit timeout. Keeps a blocking foreground command "
|
||||||
|
"(e.g. an un-backgrounded server) from hanging the turn; background `&` processes return immediately."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@ -79,6 +79,9 @@ boxlite = ["boxlite>=0.9.7"]
|
|||||||
# Tenki cloud sandbox provider (deerflow.community.tenki). Optional so a default
|
# Tenki cloud sandbox provider (deerflow.community.tenki). Optional so a default
|
||||||
# install stays free of the Tenki SDK; only pulled in when the provider is used.
|
# install stays free of the Tenki SDK; only pulled in when the provider is used.
|
||||||
tenki = ["tenki-sandbox>=0.4.0"]
|
tenki = ["tenki-sandbox>=0.4.0"]
|
||||||
|
# OpenSandbox remote sandbox provider (deerflow.community.opensandbox). The
|
||||||
|
# sync SDK is loaded only when this provider is selected.
|
||||||
|
opensandbox = ["opensandbox>=0.1.15,<0.2.0"]
|
||||||
# Agent observability (Monocle). Optional so a default install stays free of the
|
# Agent observability (Monocle). Optional so a default install stays free of the
|
||||||
# OpenTelemetry stack; only pulled in when MONOCLE_TRACING is used.
|
# OpenTelemetry stack; only pulled in when MONOCLE_TRACING is used.
|
||||||
monocle = ["monocle_apptrace>=0.8.8"]
|
monocle = ["monocle_apptrace>=0.8.8"]
|
||||||
|
|||||||
@ -14,3 +14,15 @@ def test_boxlite_is_optional_harness_dependency() -> None:
|
|||||||
|
|
||||||
assert not any(dep.startswith("boxlite") for dep in core_dependencies)
|
assert not any(dep.startswith("boxlite") for dep in core_dependencies)
|
||||||
assert any(dep.startswith("boxlite>=0.9.7") for dep in optional_dependencies["boxlite"])
|
assert any(dep.startswith("boxlite>=0.9.7") for dep in optional_dependencies["boxlite"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_opensandbox_is_optional_harness_dependency() -> None:
|
||||||
|
"""OpenSandbox should be installed only when its provider is selected."""
|
||||||
|
pyproject_path = Path(__file__).resolve().parents[1] / "packages" / "harness" / "pyproject.toml"
|
||||||
|
pyproject = tomllib.loads(pyproject_path.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
core_dependencies = pyproject["project"]["dependencies"]
|
||||||
|
optional_dependencies = pyproject["project"]["optional-dependencies"]
|
||||||
|
|
||||||
|
assert not any(dep.startswith("opensandbox") for dep in core_dependencies)
|
||||||
|
assert optional_dependencies["opensandbox"] == ["opensandbox>=0.1.15,<0.2.0"]
|
||||||
|
|||||||
713
backend/tests/test_opensandbox_provider.py
Normal file
713
backend/tests/test_opensandbox_provider.py
Normal file
@ -0,0 +1,713 @@
|
|||||||
|
"""Unit tests for the optional OpenSandbox community provider.
|
||||||
|
|
||||||
|
The real ``opensandbox`` SDK is deliberately not required for this suite. The
|
||||||
|
tests pin DeerFlow's adapter contract with a small synchronous fake: lazy
|
||||||
|
dependency loading, scoped lifecycle reuse, command forwarding, native file
|
||||||
|
transport, search parsing, path guards, and terminal-session eviction.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import errno
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import types
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import timedelta
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from deerflow.community.opensandbox.provider import OpenSandboxProvider, _import_sdk
|
||||||
|
from deerflow.community.opensandbox.sandbox import OpenSandboxSandbox
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _Message:
|
||||||
|
text: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _Result:
|
||||||
|
text: str | None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _Logs:
|
||||||
|
stdout: list[_Message] = field(default_factory=list)
|
||||||
|
stderr: list[_Message] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _Execution:
|
||||||
|
exit_code: int | None = 0
|
||||||
|
logs: _Logs = field(default_factory=_Logs)
|
||||||
|
result: list[_Result] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
def _execution(*, stdout: tuple[str, ...] = (), stderr: tuple[str, ...] = (), result: tuple[str, ...] = (), exit_code: int | None = 0) -> _Execution:
|
||||||
|
return _Execution(
|
||||||
|
exit_code=exit_code,
|
||||||
|
logs=_Logs(
|
||||||
|
stdout=[_Message(text) for text in stdout],
|
||||||
|
stderr=[_Message(text) for text in stderr],
|
||||||
|
),
|
||||||
|
result=[_Result(text) for text in result],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _FakeRunCommandOpts:
|
||||||
|
background: bool = False
|
||||||
|
working_directory: str | None = None
|
||||||
|
timeout: timedelta | None = None
|
||||||
|
uid: int | None = None
|
||||||
|
gid: int | None = None
|
||||||
|
envs: dict[str, str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeFiles:
|
||||||
|
def __init__(self, owner: _FakeRemote) -> None:
|
||||||
|
self._owner = owner
|
||||||
|
self.calls: list[tuple[str, str]] = []
|
||||||
|
|
||||||
|
def _guard(self) -> None:
|
||||||
|
if self._owner.file_error is not None:
|
||||||
|
raise self._owner.file_error
|
||||||
|
|
||||||
|
def read_file(self, path: str, *, encoding: str = "utf-8") -> str:
|
||||||
|
self._guard()
|
||||||
|
self.calls.append(("read_file", path))
|
||||||
|
if path not in self._owner.file_data:
|
||||||
|
raise FileNotFoundError(path)
|
||||||
|
return self._owner.file_data[path].decode(encoding, errors="replace")
|
||||||
|
|
||||||
|
def read_bytes(self, path: str) -> bytes:
|
||||||
|
self._guard()
|
||||||
|
self.calls.append(("read_bytes", path))
|
||||||
|
if path not in self._owner.file_data:
|
||||||
|
raise FileNotFoundError(path)
|
||||||
|
return self._owner.file_data[path]
|
||||||
|
|
||||||
|
def read_bytes_stream(self, path: str):
|
||||||
|
self._guard()
|
||||||
|
self.calls.append(("read_bytes_stream", path))
|
||||||
|
if path not in self._owner.file_data:
|
||||||
|
raise FileNotFoundError(path)
|
||||||
|
data = self._owner.file_data[path]
|
||||||
|
try:
|
||||||
|
yield from (data[index : index + 3] for index in range(0, len(data), 3))
|
||||||
|
finally:
|
||||||
|
self._owner.stream_closed = True
|
||||||
|
|
||||||
|
def write_file(self, path: str, data: str | bytes, *, mode: int = 755) -> None:
|
||||||
|
self._guard()
|
||||||
|
self.calls.append(("write_file", path))
|
||||||
|
self._owner.file_data[path] = data.encode() if isinstance(data, str) else bytes(data)
|
||||||
|
parent = path.rsplit("/", 1)[0]
|
||||||
|
while parent:
|
||||||
|
self._owner.directories.add(parent)
|
||||||
|
parent = parent.rsplit("/", 1)[0]
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeCommands:
|
||||||
|
def __init__(self, owner: _FakeRemote) -> None:
|
||||||
|
self._owner = owner
|
||||||
|
self.calls: list[tuple[str, _FakeRunCommandOpts | None]] = []
|
||||||
|
|
||||||
|
def run(self, command: str, *, opts: _FakeRunCommandOpts | None = None) -> _Execution:
|
||||||
|
self.calls.append((command, opts))
|
||||||
|
if self._owner.command_error is not None:
|
||||||
|
raise self._owner.command_error
|
||||||
|
if command.startswith("mkdir -p /mnt/user-data/"):
|
||||||
|
return _execution(stderr=("bootstrap failed",), exit_code=self._owner.bootstrap_exit_code)
|
||||||
|
if command == "true":
|
||||||
|
return _execution(exit_code=self._owner.health_exit_code)
|
||||||
|
if command == "mixed-output":
|
||||||
|
return _execution(stdout=("out-1", "out-2"), stderr=("err-1",), exit_code=7)
|
||||||
|
if command == "result-output":
|
||||||
|
return _execution(stdout=("stdout",), result=("result",), stderr=("stderr",))
|
||||||
|
if command == "silent-failure":
|
||||||
|
return _execution(exit_code=9)
|
||||||
|
if command == "missing-complete":
|
||||||
|
return _execution(stderr=("stream ended",), exit_code=None)
|
||||||
|
if command.startswith("find "):
|
||||||
|
return self._find(command)
|
||||||
|
if command.startswith(("grep ", "{ grep ")):
|
||||||
|
return self._grep(command)
|
||||||
|
return _execution()
|
||||||
|
|
||||||
|
def _find(self, command: str) -> _Execution:
|
||||||
|
tokens = shlex.split(command)
|
||||||
|
root = tokens[1].rstrip("/") or "/"
|
||||||
|
include_dirs = "d" in tokens
|
||||||
|
paths = list(self._owner.file_data)
|
||||||
|
if include_dirs:
|
||||||
|
paths.extend(self._owner.directories)
|
||||||
|
matches = sorted(path for path in set(paths) if path == root or path.startswith(f"{root}/"))
|
||||||
|
return _execution(stdout=tuple(matches))
|
||||||
|
|
||||||
|
def _grep(self, command: str) -> _Execution:
|
||||||
|
tokens = shlex.split(command)
|
||||||
|
pattern = tokens[tokens.index("-e") + 1]
|
||||||
|
root = tokens[tokens.index("-e") + 2].rstrip("/")
|
||||||
|
flags = 0 if "-i" not in tokens else re.IGNORECASE
|
||||||
|
literal = "-F" in tokens
|
||||||
|
rows: list[str] = []
|
||||||
|
for path, data in sorted(self._owner.file_data.items()):
|
||||||
|
if path != root and not path.startswith(f"{root}/"):
|
||||||
|
continue
|
||||||
|
for line_number, line in enumerate(data.decode(errors="replace").splitlines(), start=1):
|
||||||
|
matched = pattern.lower() in line.lower() if literal and flags else pattern in line if literal else re.search(pattern, line, flags) is not None
|
||||||
|
if matched:
|
||||||
|
rows.append(f"{path}:{line_number}:{line}")
|
||||||
|
if self._owner.grep_duplicate_rows:
|
||||||
|
rows.extend(rows)
|
||||||
|
return _execution(stdout=tuple(rows))
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeRemote:
|
||||||
|
def __init__(self, remote_id: str, *, bootstrap_exit_code: int | None = 0) -> None:
|
||||||
|
self.id = remote_id
|
||||||
|
self.bootstrap_exit_code = bootstrap_exit_code
|
||||||
|
self.health_exit_code: int | None = 0
|
||||||
|
self.command_error: Exception | None = None
|
||||||
|
self.file_error: Exception | None = None
|
||||||
|
self.renew_error: Exception | None = None
|
||||||
|
self.renew_calls: list[timedelta] = []
|
||||||
|
self.destroy_calls = 0
|
||||||
|
self.file_data: dict[str, bytes] = {}
|
||||||
|
self.directories: set[str] = set()
|
||||||
|
self.stream_closed = False
|
||||||
|
self.grep_duplicate_rows = False
|
||||||
|
self.commands = _FakeCommands(self)
|
||||||
|
self.files = _FakeFiles(self)
|
||||||
|
|
||||||
|
def renew(self, timeout: timedelta) -> None:
|
||||||
|
self.renew_calls.append(timeout)
|
||||||
|
if self.renew_error is not None:
|
||||||
|
raise self.renew_error
|
||||||
|
|
||||||
|
def destroy(self) -> None:
|
||||||
|
self.destroy_calls += 1
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeSandboxClass:
|
||||||
|
def __init__(self, remote_factory=None) -> None:
|
||||||
|
self.remote_factory = remote_factory
|
||||||
|
self.create_calls: list[dict[str, Any]] = []
|
||||||
|
self.remotes: list[_FakeRemote] = []
|
||||||
|
|
||||||
|
def create(self, image: str, **kwargs: Any) -> _FakeRemote:
|
||||||
|
self.create_calls.append({"image": image, **kwargs})
|
||||||
|
index = len(self.remotes) + 1
|
||||||
|
remote = self.remote_factory(index) if self.remote_factory is not None else _FakeRemote(f"remote-{index}")
|
||||||
|
self.remotes.append(remote)
|
||||||
|
return remote
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeConnectionConfig:
|
||||||
|
def __init__(self, **kwargs: Any) -> None:
|
||||||
|
self.kwargs = kwargs
|
||||||
|
|
||||||
|
|
||||||
|
class _TerminalApiError(RuntimeError):
|
||||||
|
def __init__(self, message: str, status_code: int = 404) -> None:
|
||||||
|
super().__init__(message)
|
||||||
|
self.status_code = status_code
|
||||||
|
|
||||||
|
|
||||||
|
def _stub_config(attrs: dict[str, Any] | None = None) -> types.SimpleNamespace:
|
||||||
|
values = {"idle_timeout": 0, **(attrs or {})}
|
||||||
|
return types.SimpleNamespace(sandbox=types.SimpleNamespace(**values))
|
||||||
|
|
||||||
|
|
||||||
|
def _install(monkeypatch: pytest.MonkeyPatch, *, sdk: _FakeSandboxClass | None = None, config: dict[str, Any] | None = None) -> tuple[OpenSandboxProvider, _FakeSandboxClass]:
|
||||||
|
fake_sdk = sdk or _FakeSandboxClass()
|
||||||
|
monkeypatch.setattr("deerflow.community.opensandbox.provider.get_app_config", lambda: _stub_config(config))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"deerflow.community.opensandbox.provider._import_sdk",
|
||||||
|
lambda: (fake_sdk, _FakeConnectionConfig, _FakeRunCommandOpts),
|
||||||
|
)
|
||||||
|
return OpenSandboxProvider(), fake_sdk
|
||||||
|
|
||||||
|
|
||||||
|
def _box(
|
||||||
|
remote: _FakeRemote,
|
||||||
|
*,
|
||||||
|
on_terminal_failure=None,
|
||||||
|
default_env=None,
|
||||||
|
sandbox_timeout: timedelta | None = None,
|
||||||
|
default_command_timeout: float = 600,
|
||||||
|
) -> OpenSandboxSandbox:
|
||||||
|
return OpenSandboxSandbox(
|
||||||
|
"sandbox-id",
|
||||||
|
remote,
|
||||||
|
run_command_opts_cls=_FakeRunCommandOpts,
|
||||||
|
default_env=default_env,
|
||||||
|
sandbox_timeout=sandbox_timeout,
|
||||||
|
default_command_timeout=default_command_timeout,
|
||||||
|
on_terminal_failure=on_terminal_failure,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_sdk_has_actionable_error(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
for module_name in (
|
||||||
|
"opensandbox",
|
||||||
|
"opensandbox.sync",
|
||||||
|
"opensandbox.config.connection_sync",
|
||||||
|
"opensandbox.models.execd",
|
||||||
|
):
|
||||||
|
monkeypatch.setitem(sys.modules, module_name, None)
|
||||||
|
with pytest.raises(ImportError, match=r"deerflow-harness\[opensandbox\]"):
|
||||||
|
_import_sdk()
|
||||||
|
|
||||||
|
|
||||||
|
def test_provider_defers_sdk_import_until_acquire(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
monkeypatch.setattr("deerflow.community.opensandbox.provider.get_app_config", lambda: _stub_config())
|
||||||
|
calls = 0
|
||||||
|
|
||||||
|
def fail_if_called():
|
||||||
|
nonlocal calls
|
||||||
|
calls += 1
|
||||||
|
raise AssertionError("SDK imported")
|
||||||
|
|
||||||
|
monkeypatch.setattr("deerflow.community.opensandbox.provider._import_sdk", fail_if_called)
|
||||||
|
provider = OpenSandboxProvider()
|
||||||
|
assert calls == 0
|
||||||
|
with pytest.raises(AssertionError, match="SDK imported"):
|
||||||
|
provider.acquire("thread", user_id="user")
|
||||||
|
assert calls == 1
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_passes_connection_lifetime_scope_and_environment(monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture) -> None:
|
||||||
|
monkeypatch.setenv("OPEN_SANDBOX_TEST_VALUE", "resolved")
|
||||||
|
monkeypatch.delenv("OPEN_SANDBOX_ABSENT_VALUE", raising=False)
|
||||||
|
provider, sdk = _install(
|
||||||
|
monkeypatch,
|
||||||
|
config={
|
||||||
|
"image": "python:3.12",
|
||||||
|
"api_key": "secret",
|
||||||
|
"domain": "sandbox.example",
|
||||||
|
"protocol": "https",
|
||||||
|
"request_timeout": 12,
|
||||||
|
"ready_timeout": 18,
|
||||||
|
"sandbox_timeout": 7200,
|
||||||
|
"use_server_proxy": True,
|
||||||
|
"environment": {
|
||||||
|
"BASE": "1",
|
||||||
|
"FROM_ENV": "$OPEN_SANDBOX_TEST_VALUE",
|
||||||
|
"MISSING_ENV": "$OPEN_SANDBOX_ABSENT_VALUE",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
provider.acquire("thread-1", user_id="user-1")
|
||||||
|
call = sdk.create_calls[0]
|
||||||
|
assert call["image"] == "python:3.12"
|
||||||
|
assert call["timeout"] == timedelta(seconds=7200)
|
||||||
|
assert call["ready_timeout"] == timedelta(seconds=18)
|
||||||
|
assert call["env"] == {"BASE": "1", "FROM_ENV": "resolved", "MISSING_ENV": ""}
|
||||||
|
assert call["metadata"] == {
|
||||||
|
"deer_flow_provider": "opensandbox",
|
||||||
|
"deer_flow_thread": "thread-1",
|
||||||
|
"deer_flow_user": "user-1",
|
||||||
|
}
|
||||||
|
assert call["connection_config"].kwargs == {
|
||||||
|
"api_key": "secret",
|
||||||
|
"domain": "sandbox.example",
|
||||||
|
"protocol": "https",
|
||||||
|
"request_timeout": timedelta(seconds=12),
|
||||||
|
"use_server_proxy": True,
|
||||||
|
}
|
||||||
|
assert "unauthenticated localhost:8080" not in caplog.text
|
||||||
|
assert "remote OpenSandbox domain uses HTTP" not in caplog.text
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_connection_config_warns_about_sdk_default(monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture) -> None:
|
||||||
|
monkeypatch.delenv("OPEN_SANDBOX_API_KEY", raising=False)
|
||||||
|
monkeypatch.delenv("OPEN_SANDBOX_DOMAIN", raising=False)
|
||||||
|
|
||||||
|
provider, _ = _install(monkeypatch)
|
||||||
|
|
||||||
|
assert any(record.levelno == logging.WARNING and "unauthenticated localhost:8080" in record.getMessage() for record in caplog.records)
|
||||||
|
assert "remote OpenSandbox domain uses HTTP" not in caplog.text
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_http_connection_warns_without_logging_api_key(monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture) -> None:
|
||||||
|
caplog.set_level(logging.DEBUG, logger="deerflow.community.opensandbox.provider")
|
||||||
|
provider, _ = _install(
|
||||||
|
monkeypatch,
|
||||||
|
config={"api_key": "not-a-real-secret", "domain": "sandbox.example", "protocol": "http"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert any(record.levelno == logging.WARNING and "remote OpenSandbox domain uses HTTP" in record.getMessage() for record in caplog.records)
|
||||||
|
assert "not-a-real-secret" not in caplog.text
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_null_sandbox_timeout_uses_default(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch, config={"sandbox_timeout": None})
|
||||||
|
|
||||||
|
provider.acquire("thread-1", user_id="user-1")
|
||||||
|
|
||||||
|
assert sdk.create_calls[0]["timeout"] == timedelta(hours=4)
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("exit_code", [17, None])
|
||||||
|
def test_bootstrap_failure_destroys_created_remote(monkeypatch: pytest.MonkeyPatch, exit_code: int | None) -> None:
|
||||||
|
sdk = _FakeSandboxClass(lambda index: _FakeRemote(f"remote-{index}", bootstrap_exit_code=exit_code))
|
||||||
|
provider, _ = _install(monkeypatch, sdk=sdk)
|
||||||
|
with pytest.raises(RuntimeError, match="bootstrap"):
|
||||||
|
provider.acquire("thread-1", user_id="user-1")
|
||||||
|
assert sdk.remotes[0].destroy_calls == 1
|
||||||
|
assert provider._sandboxes == {}
|
||||||
|
assert provider._warm_pool == {}
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_scope_reuse_and_user_thread_isolation(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
first = provider.acquire("thread-1", user_id="user-1")
|
||||||
|
assert provider.acquire("thread-1", user_id="user-1") == first
|
||||||
|
other_user = provider.acquire("thread-1", user_id="user-2")
|
||||||
|
other_thread = provider.acquire("thread-2", user_id="user-1")
|
||||||
|
assert len({first, other_user, other_thread}) == 3
|
||||||
|
assert len(sdk.create_calls) == 3
|
||||||
|
assert sdk.remotes[0].renew_calls == [timedelta(hours=4)]
|
||||||
|
assert len({id(call["connection_config"]) for call in sdk.create_calls}) == 3
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_active_scope_terminal_renewal_failure_rebuilds_in_same_acquire(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||||
|
sdk.remotes[0].renew_error = _TerminalApiError("sandbox expired", status_code=410)
|
||||||
|
|
||||||
|
assert provider.acquire("thread-1", user_id="user-1") == sandbox_id
|
||||||
|
assert len(sdk.create_calls) == 2
|
||||||
|
assert sdk.remotes[0].destroy_calls == 1
|
||||||
|
replacement = provider.get(sandbox_id)
|
||||||
|
assert replacement is not None and replacement.remote_id == "remote-2"
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_active_scope_non_terminal_renewal_failure_is_not_hidden(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||||
|
sdk.remotes[0].renew_error = RuntimeError("temporary management failure")
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="temporary management failure"):
|
||||||
|
provider.acquire("thread-1", user_id="user-1")
|
||||||
|
assert len(sdk.create_calls) == 1
|
||||||
|
assert sdk.remotes[0].destroy_calls == 0
|
||||||
|
assert provider.get(sandbox_id) is not None
|
||||||
|
sdk.remotes[0].renew_error = None
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_release_and_same_scope_warm_reclaim(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||||
|
provider.release(sandbox_id)
|
||||||
|
assert sandbox_id not in provider._sandboxes
|
||||||
|
assert sandbox_id in provider._warm_pool
|
||||||
|
assert provider.acquire("thread-1", user_id="user-1") == sandbox_id
|
||||||
|
assert len(sdk.create_calls) == 1
|
||||||
|
assert sdk.remotes[0].commands.calls[-1][0] == "true"
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("exit_code", [1, None])
|
||||||
|
def test_unhealthy_warm_entry_is_destroyed_and_replaced(monkeypatch: pytest.MonkeyPatch, exit_code: int | None) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||||
|
provider.release(sandbox_id)
|
||||||
|
sdk.remotes[0].health_exit_code = exit_code
|
||||||
|
assert provider.acquire("thread-1", user_id="user-1") == sandbox_id
|
||||||
|
assert sdk.remotes[0].destroy_calls == 1
|
||||||
|
assert len(sdk.create_calls) == 2
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_parks_active_and_shutdown_destroys_active_and_warm(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
active_id = provider.acquire("active", user_id="user")
|
||||||
|
warm_id = provider.acquire("warm", user_id="user")
|
||||||
|
provider.release(warm_id)
|
||||||
|
provider.reset()
|
||||||
|
assert provider._sandboxes == {}
|
||||||
|
assert {active_id, warm_id} == set(provider._warm_pool)
|
||||||
|
provider.shutdown()
|
||||||
|
provider.shutdown()
|
||||||
|
assert [remote.destroy_calls for remote in sdk.remotes] == [1, 1]
|
||||||
|
assert provider._sandboxes == {} and provider._warm_pool == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_shutdown_stops_idle_reaper(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
monkeypatch.setattr(OpenSandboxProvider, "IDLE_CHECK_INTERVAL", 0.01)
|
||||||
|
provider, _ = _install(monkeypatch, config={"idle_timeout": 60})
|
||||||
|
checker = provider._idle_checker_thread
|
||||||
|
provider.shutdown()
|
||||||
|
assert provider._idle_checker_stop.is_set()
|
||||||
|
assert checker is not None and not checker.is_alive()
|
||||||
|
|
||||||
|
|
||||||
|
def test_execute_forwards_env_timeout_and_combines_streams() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
box = _box(remote, default_env={"BASE": "1"})
|
||||||
|
assert box.execute_command("mixed-output", env={"EXTRA": "2"}, timeout=5) == "out-1\nout-2\nerr-1"
|
||||||
|
_, opts = remote.commands.calls[-1]
|
||||||
|
assert opts is not None
|
||||||
|
assert opts.envs == {"BASE": "1", "EXTRA": "2"}
|
||||||
|
assert opts.timeout == timedelta(seconds=5)
|
||||||
|
assert box.execute_command("result-output") == "stdout\nresult\nstderr"
|
||||||
|
assert box.execute_command("silent-failure") == "Command exited with code 9"
|
||||||
|
assert box.execute_command("missing-complete") == "Error: OpenSandbox command completed without an exit code: stream ended"
|
||||||
|
|
||||||
|
|
||||||
|
def test_operations_renew_remote_lifetime_and_bound_default_commands() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
remote.file_data["/mnt/user-data/workspace/note.txt"] = b"note"
|
||||||
|
box = _box(
|
||||||
|
remote,
|
||||||
|
sandbox_timeout=timedelta(seconds=60),
|
||||||
|
default_command_timeout=120,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert box.execute_command("true") == "(no output)"
|
||||||
|
_, opts = remote.commands.calls[-1]
|
||||||
|
assert opts is not None and opts.timeout == timedelta(seconds=120)
|
||||||
|
assert remote.renew_calls[-1] == timedelta(seconds=150)
|
||||||
|
|
||||||
|
assert box.read_file("/mnt/user-data/workspace/note.txt") == "note"
|
||||||
|
assert remote.renew_calls[-1] == timedelta(seconds=60)
|
||||||
|
|
||||||
|
|
||||||
|
def test_short_operation_cannot_shorten_in_flight_command_renewal() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
remote.file_data["/mnt/user-data/workspace/note.txt"] = b"note"
|
||||||
|
box = _box(
|
||||||
|
remote,
|
||||||
|
sandbox_timeout=timedelta(seconds=60),
|
||||||
|
default_command_timeout=120,
|
||||||
|
)
|
||||||
|
command_started = threading.Event()
|
||||||
|
finish_command = threading.Event()
|
||||||
|
file_started = threading.Event()
|
||||||
|
short_renew_attempted = threading.Event()
|
||||||
|
original_run = remote.commands.run
|
||||||
|
original_renew = remote.renew
|
||||||
|
|
||||||
|
def blocking_run(command: str, *, opts: _FakeRunCommandOpts | None = None) -> _Execution:
|
||||||
|
command_started.set()
|
||||||
|
assert finish_command.wait(timeout=2)
|
||||||
|
return original_run(command, opts=opts)
|
||||||
|
|
||||||
|
remote.commands.run = blocking_run # type: ignore[method-assign]
|
||||||
|
|
||||||
|
def observed_renew(timeout: timedelta) -> None:
|
||||||
|
if timeout == timedelta(seconds=60):
|
||||||
|
short_renew_attempted.set()
|
||||||
|
original_renew(timeout)
|
||||||
|
|
||||||
|
remote.renew = observed_renew # type: ignore[method-assign]
|
||||||
|
command_result: list[str] = []
|
||||||
|
file_result: list[str] = []
|
||||||
|
command_thread = threading.Thread(target=lambda: command_result.append(box.execute_command("long-command")))
|
||||||
|
|
||||||
|
def read_file() -> None:
|
||||||
|
file_started.set()
|
||||||
|
file_result.append(box.read_file("/mnt/user-data/workspace/note.txt"))
|
||||||
|
|
||||||
|
file_thread = threading.Thread(target=read_file)
|
||||||
|
command_thread.start()
|
||||||
|
assert command_started.wait(timeout=2)
|
||||||
|
file_thread.start()
|
||||||
|
assert file_started.wait(timeout=2)
|
||||||
|
assert not short_renew_attempted.wait(timeout=0.1)
|
||||||
|
assert file_thread.is_alive()
|
||||||
|
assert remote.renew_calls == [timedelta(seconds=150)]
|
||||||
|
|
||||||
|
finish_command.set()
|
||||||
|
command_thread.join(timeout=2)
|
||||||
|
file_thread.join(timeout=2)
|
||||||
|
assert command_result == ["(no output)"]
|
||||||
|
assert file_result == ["note"]
|
||||||
|
assert remote.renew_calls == [timedelta(seconds=150), timedelta(seconds=60)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_cleanup_mode_skips_renewal() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
box = _box(remote, sandbox_timeout=None)
|
||||||
|
assert box.execute_command("true") == "(no output)"
|
||||||
|
assert remote.renew_calls == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("timeout", [0, -1])
|
||||||
|
def test_execute_rejects_unbounded_or_negative_timeout(timeout: float) -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
box = _box(remote)
|
||||||
|
assert box.execute_command("true", timeout=timeout).startswith("Error: timeout must be positive")
|
||||||
|
assert remote.commands.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_execute_rejects_invalid_environment_key() -> None:
|
||||||
|
box = _box(_FakeRemote("remote"))
|
||||||
|
with pytest.raises(ValueError, match="POSIX"):
|
||||||
|
box.execute_command("true", env={"BAD KEY": "x"})
|
||||||
|
|
||||||
|
|
||||||
|
def test_text_binary_append_and_line_ranges() -> None:
|
||||||
|
box = _box(_FakeRemote("remote"))
|
||||||
|
path = "/mnt/user-data/workspace/note.txt"
|
||||||
|
box.write_file(path, "one\ntwo\nthree")
|
||||||
|
assert box.read_file(path, 2, 3) == "two\nthree"
|
||||||
|
box.write_file(path, "\nfour", append=True)
|
||||||
|
assert box.read_file(path) == "one\ntwo\nthree\nfour"
|
||||||
|
binary_path = "/mnt/user-data/outputs/blob.bin"
|
||||||
|
box.update_file(binary_path, b"\x00\xffpayload")
|
||||||
|
assert box.download_file(binary_path) == b"\x00\xffpayload"
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_rejects_oversize_stream(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
monkeypatch.setattr("deerflow.community.opensandbox.sandbox._MAX_DOWNLOAD_SIZE", 4)
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
path = "/mnt/user-data/outputs/oversize.bin"
|
||||||
|
remote.file_data[path] = b"12345"
|
||||||
|
|
||||||
|
with pytest.raises(OSError) as excinfo:
|
||||||
|
_box(remote).download_file(path)
|
||||||
|
|
||||||
|
assert excinfo.value.errno == errno.EFBIG
|
||||||
|
assert remote.files.calls == [("read_bytes_stream", path)]
|
||||||
|
assert remote.stream_closed
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_glob_and_grep_return_virtual_paths() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
remote.grep_duplicate_rows = True
|
||||||
|
box = _box(remote)
|
||||||
|
box.write_file("/mnt/user-data/workspace/src/a.py", "Needle here\nsecond\n")
|
||||||
|
box.write_file("/mnt/user-data/workspace/vendor/b.py", "needle there\n")
|
||||||
|
assert box.list_dir("/mnt/user-data/workspace") == [
|
||||||
|
"/mnt/user-data/workspace",
|
||||||
|
"/mnt/user-data/workspace/src",
|
||||||
|
"/mnt/user-data/workspace/src/a.py",
|
||||||
|
"/mnt/user-data/workspace/vendor",
|
||||||
|
"/mnt/user-data/workspace/vendor/b.py",
|
||||||
|
]
|
||||||
|
found, truncated = box.glob("/mnt/user-data/workspace", "src/*.py")
|
||||||
|
assert found == ["/mnt/user-data/workspace/src/a.py"]
|
||||||
|
assert truncated is False
|
||||||
|
matches, truncated = box.grep("/mnt/user-data/workspace", "needle", glob="src/*.py", literal=True)
|
||||||
|
assert [(match.path, match.line_number, match.line) for match in matches] == [("/mnt/user-data/workspace/src/a.py", 1, "Needle here")]
|
||||||
|
assert truncated is False
|
||||||
|
grep_tokens = shlex.split(remote.commands.calls[-1][0])
|
||||||
|
assert "--include=*.py" in grep_tokens
|
||||||
|
assert "-m100" in grep_tokens
|
||||||
|
|
||||||
|
box.grep("/mnt/user-data/workspace", "needle", glob="src/*.py; echo injected", literal=True)
|
||||||
|
unsafe_glob_tokens = shlex.split(remote.commands.calls[-1][0])
|
||||||
|
assert "--include=*.py; echo injected" in unsafe_glob_tokens
|
||||||
|
assert unsafe_glob_tokens.count("grep") == 2
|
||||||
|
assert 'status=$?; [ "$status" -eq 2 ] &&' in remote.commands.calls[-1][0]
|
||||||
|
fallback_tokens = unsafe_glob_tokens[unsafe_glob_tokens.index("grep", 2) :]
|
||||||
|
assert not any(token.startswith("--include=") or token.startswith("-m") for token in fallback_tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_rejects_non_positive_limits_and_negative_depth() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
box = _box(remote)
|
||||||
|
with pytest.raises(ValueError, match="max_depth"):
|
||||||
|
box.list_dir("/mnt/user-data/workspace", max_depth=-1)
|
||||||
|
with pytest.raises(ValueError, match="max_results"):
|
||||||
|
box.glob("/mnt/user-data/workspace", "*", max_results=0)
|
||||||
|
with pytest.raises(ValueError, match="max_results"):
|
||||||
|
box.grep("/mnt/user-data/workspace", "text", max_results=-1)
|
||||||
|
assert remote.commands.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"path",
|
||||||
|
["", "relative.txt", "/mnt/user-data/../etc/passwd", "\\mnt\\user-data\\..\\etc\\passwd"],
|
||||||
|
)
|
||||||
|
def test_path_guard_rejects_unsafe_paths(path: str) -> None:
|
||||||
|
box = _box(_FakeRemote("remote"))
|
||||||
|
with pytest.raises((ValueError, PermissionError)):
|
||||||
|
box.read_file(path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_rejects_outside_virtual_prefix_before_sdk_call() -> None:
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
box = _box(remote)
|
||||||
|
with pytest.raises(PermissionError):
|
||||||
|
box.download_file("/etc/passwd")
|
||||||
|
assert remote.files.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_file_api_404_does_not_evict_sandbox() -> None:
|
||||||
|
invalidated: list[tuple[str, str]] = []
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
remote.file_error = _TerminalApiError("file not found", status_code=404)
|
||||||
|
box = _box(remote, on_terminal_failure=lambda sandbox_id, reason: invalidated.append((sandbox_id, reason)))
|
||||||
|
assert box.read_file("/mnt/user-data/workspace/missing.txt").startswith("Error:")
|
||||||
|
assert invalidated == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_renewal_failure_evicts_before_operation() -> None:
|
||||||
|
invalidated: list[tuple[str, str]] = []
|
||||||
|
remote = _FakeRemote("remote")
|
||||||
|
remote.renew_error = _TerminalApiError("sandbox expired")
|
||||||
|
box = _box(
|
||||||
|
remote,
|
||||||
|
sandbox_timeout=timedelta(minutes=5),
|
||||||
|
on_terminal_failure=lambda sandbox_id, reason: invalidated.append((sandbox_id, reason)),
|
||||||
|
)
|
||||||
|
assert box.execute_command("true") == "Error: sandbox expired"
|
||||||
|
assert invalidated == [("sandbox-id", "sandbox expired")]
|
||||||
|
assert remote.commands.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_error_evicts_active_sandbox(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
sandbox_id = provider.acquire("thread-1", user_id="user-1")
|
||||||
|
box = provider.get(sandbox_id)
|
||||||
|
assert box is not None
|
||||||
|
sdk.remotes[0].command_error = _TerminalApiError("sandbox is gone")
|
||||||
|
assert box.execute_command("true") == "Error: sandbox is gone"
|
||||||
|
assert provider.get(sandbox_id) is None
|
||||||
|
assert sdk.remotes[0].destroy_calls == 1
|
||||||
|
provider.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
def test_concurrent_same_scope_acquire_creates_once(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
provider, sdk = _install(monkeypatch)
|
||||||
|
original_create = sdk.create
|
||||||
|
started = threading.Event()
|
||||||
|
|
||||||
|
def slow_create(image: str, **kwargs: Any) -> _FakeRemote:
|
||||||
|
started.set()
|
||||||
|
time.sleep(0.05)
|
||||||
|
return original_create(image, **kwargs)
|
||||||
|
|
||||||
|
sdk.create = slow_create # type: ignore[method-assign]
|
||||||
|
results: list[str] = []
|
||||||
|
|
||||||
|
first = threading.Thread(target=lambda: results.append(provider.acquire("thread", user_id="user")))
|
||||||
|
second = threading.Thread(target=lambda: results.append(provider.acquire("thread", user_id="user")))
|
||||||
|
first.start()
|
||||||
|
assert started.wait(timeout=2)
|
||||||
|
second.start()
|
||||||
|
first.join(timeout=2)
|
||||||
|
second.join(timeout=2)
|
||||||
|
assert len(results) == 2 and results[0] == results[1]
|
||||||
|
assert len(sdk.create_calls) == 1
|
||||||
|
provider.shutdown()
|
||||||
21
backend/uv.lock
generated
21
backend/uv.lock
generated
@ -976,6 +976,9 @@ monocle = [
|
|||||||
ollama = [
|
ollama = [
|
||||||
{ name = "langchain-ollama" },
|
{ name = "langchain-ollama" },
|
||||||
]
|
]
|
||||||
|
opensandbox = [
|
||||||
|
{ name = "opensandbox" },
|
||||||
|
]
|
||||||
postgres = [
|
postgres = [
|
||||||
{ name = "asyncpg" },
|
{ name = "asyncpg" },
|
||||||
{ name = "langgraph-checkpoint-postgres" },
|
{ name = "langgraph-checkpoint-postgres" },
|
||||||
@ -1034,6 +1037,7 @@ requires-dist = [
|
|||||||
{ name = "markdownify", specifier = ">=1.2.2" },
|
{ name = "markdownify", specifier = ">=1.2.2" },
|
||||||
{ name = "markitdown", extras = ["all", "xlsx"], specifier = ">=0.0.1a2" },
|
{ name = "markitdown", extras = ["all", "xlsx"], specifier = ">=0.0.1a2" },
|
||||||
{ name = "monocle-apptrace", marker = "extra == 'monocle'", specifier = ">=0.8.8" },
|
{ name = "monocle-apptrace", marker = "extra == 'monocle'", specifier = ">=0.8.8" },
|
||||||
|
{ name = "opensandbox", marker = "extra == 'opensandbox'", specifier = ">=0.1.15,<0.2.0" },
|
||||||
{ name = "packaging", specifier = ">=24.2" },
|
{ name = "packaging", specifier = ">=24.2" },
|
||||||
{ name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" },
|
{ name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" },
|
||||||
{ name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.3.3" },
|
{ name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.3.3" },
|
||||||
@ -1049,7 +1053,7 @@ requires-dist = [
|
|||||||
{ name = "textual", marker = "extra == 'tui'", specifier = ">=0.80" },
|
{ name = "textual", marker = "extra == 'tui'", specifier = ">=0.80" },
|
||||||
{ name = "tiktoken", specifier = ">=0.8.0" },
|
{ name = "tiktoken", specifier = ">=0.8.0" },
|
||||||
]
|
]
|
||||||
provides-extras = ["tui", "groundroute", "ollama", "postgres", "redis", "pymupdf", "boxlite", "tenki", "monocle", "browser", "memory-zh"]
|
provides-extras = ["tui", "groundroute", "ollama", "postgres", "redis", "pymupdf", "boxlite", "tenki", "opensandbox", "monocle", "browser", "memory-zh"]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "defusedxml"
|
name = "defusedxml"
|
||||||
@ -3000,6 +3004,21 @@ wheels = [
|
|||||||
{ url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" },
|
{ url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "opensandbox"
|
||||||
|
version = "0.1.15"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "attrs" },
|
||||||
|
{ name = "httpx" },
|
||||||
|
{ name = "pydantic" },
|
||||||
|
{ name = "python-dateutil" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/43/21/654a3d69815b09690e926d553f3f4a178640d1206000a1b49f5e22c8eb68/opensandbox-0.1.15.tar.gz", hash = "sha256:017abc9b399b88da51bf077d6fb94ee89b1783e601495e85ba85fed478fed1b0", size = 228729, upload-time = "2026-07-24T09:36:29.988Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/d5/5c/ab87ea696531210790feb8f575471036fccba29dedaff201736f15bbb3a7/opensandbox-0.1.15-py3-none-any.whl", hash = "sha256:992b01490551f4d8e3f99caa25e34cb9d1690f0c5027eeebab912738291957d1", size = 538522, upload-time = "2026-07-24T09:36:28.277Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "opentelemetry-api"
|
name = "opentelemetry-api"
|
||||||
version = "1.41.1"
|
version = "1.41.1"
|
||||||
|
|||||||
@ -1411,6 +1411,26 @@ sandbox:
|
|||||||
# # environment: # injected into every command (and as create-time env)
|
# # environment: # injected into every command (and as create-time env)
|
||||||
# # PYTHONUNBUFFERED: "1"
|
# # PYTHONUNBUFFERED: "1"
|
||||||
|
|
||||||
|
# Option 6: OpenSandbox remote Sandbox
|
||||||
|
# Runs each sandbox through an OpenSandbox server. Released sandboxes stay in an
|
||||||
|
# in-process warm pool and are reclaimed only by the same user/thread. Requires
|
||||||
|
# the optional SDK: pip install "deerflow-harness[opensandbox]".
|
||||||
|
# sandbox:
|
||||||
|
# use: deerflow.community.opensandbox:OpenSandboxProvider
|
||||||
|
# image: python:3.11
|
||||||
|
# # api_key: $OPEN_SANDBOX_API_KEY # optional when the SDK env var is set
|
||||||
|
# # domain: localhost:8080 # optional; OPEN_SANDBOX_DOMAIN fallback
|
||||||
|
# # protocol: http # localhost only; use https for remote domains
|
||||||
|
# # request_timeout: 30 # management API request timeout seconds
|
||||||
|
# # ready_timeout: 30 # create/readiness timeout seconds
|
||||||
|
# # use_server_proxy: false # proxy execd/file traffic through server
|
||||||
|
# # sandbox_timeout: 14400 # remote lifetime seconds; 0 = explicit cleanup
|
||||||
|
# # bash_command_timeout: 600 # default remote command timeout seconds
|
||||||
|
# # replicas: 3 # active + warm cap per gateway process
|
||||||
|
# # idle_timeout: 600 # warm seconds before destroy; 0 disables
|
||||||
|
# # environment: # create-time and per-command defaults
|
||||||
|
# # PYTHONUNBUFFERED: "1"
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Subagents Configuration
|
# Subagents Configuration
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user