Vanzeren dc38d2d003
perf(boxlite): add benchmark-driven warm-pool reclaim tuning (#4001)
* bench: add provider-agnostic sandbox benchmark with BoxLite warm pool results

- scripts/bench_sandbox_provider.py: CLI for measuring acquire/run/release
  across providers, scenarios, workloads, and concurrency levels
- scripts/summarize_bench.py: JSONL aggregation with p50/p95/p99 tables
- bench_results.jsonl: 110 turns across 7 scenarios on real BoxLite 0.9.7

Key findings:
  cold acquire:  ~860ms
  warm reclaim:   ~14ms  (60x speedup)
  release:         ~0ms
  warm_hit_rate:  95%   (warm_same_thread)

* perf(boxlite): skip health check for recently-released warm pool boxes

Boxes released within health_check_skip_seconds (default 5.0s)
are promoted directly without the ~14ms echo-ok round-trip.
A VM alive seconds ago is overwhelmingly likely to still be alive.

Add sandbox.health_check_skip_seconds config option.
Set to 0 to always health-check (old behaviour).

Benchmark (warm_same_thread, noop, 20 iters):
  acquire p50: 14.9ms → 0.0ms
  total p50:   29.9ms → 14.0ms

* chore: move benchmark scripts into backend/scripts/benchmark/

* fix: address BoxLite benchmark review findings

* fix(boxlite): only skip warm reclaim checks for released boxes

* fix(benchmark): keep BoxLite shim workaround off the event loop

* fix(boxlite): invalidate dead boxes from command path

* test(boxlite): cover skip window and invalidation edge cases

* fix(boxlite): treat sandbox-has-been-closed as terminal in _exec

* fix(boxlite): harden warm-pool reclaim and benchmark accounting

* fix(boxlite): validate warm-pool reclaims by default

* fix(config): expose boxlite health-check skip setting

* fix(boxlite): tighten failure classification and benchmark workaround

* Update config_version to 21 in values.yaml

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-09 17:21:25 +08:00

115 lines
5.5 KiB
Python

from pydantic import BaseModel, ConfigDict, Field
class VolumeMountConfig(BaseModel):
"""Configuration for a volume mount."""
host_path: str = Field(
...,
description=(
"Source path for the mount. Resolution depends on the active provider: "
"``LocalSandboxProvider`` checks this path from the gateway process — in "
"``make dev`` that is the host machine, but in Docker deployments "
"(``make up`` / docker-compose) it is the path *inside* the "
"``deer-flow-gateway`` container, so the host directory must also be "
"bind-mounted into the gateway service for the mount to take effect. "
"``AioSandboxProvider`` (DooD) passes this value straight to ``docker -v`` "
"for the sandbox container, where it is resolved by the host Docker daemon "
"from the host machine's perspective."
),
)
container_path: str = Field(..., description="Path inside the container")
read_only: bool = Field(default=False, description="Whether the mount is read-only")
class SandboxConfig(BaseModel):
"""Config section for a sandbox.
Common options:
use: Class path of the sandbox provider (required)
allow_host_bash: Enable host-side bash execution for LocalSandboxProvider.
Dangerous and intended only for fully trusted local workflows.
AioSandboxProvider and BoxliteProvider shared options:
image: Sandbox image to use (Docker/AIO image or BoxLite OCI image)
replicas: Maximum active + warm sandboxes/VMs per gateway process (default: 3). When the limit is reached, warm/least-recently-used sandboxes are evicted to make room; active sandboxes are not forcibly stopped.
idle_timeout: Idle timeout in seconds before released warm sandboxes/VMs are stopped (default: 600 = 10 minutes). Set to 0 to disable.
environment: Environment variables to inject into the sandbox (values starting with $ are resolved from host env)
BoxliteProvider specific options:
health_check_skip_seconds: Optional reclaim-time skip window in seconds for recently released warm VMs. Default behavior is 0.0 = always validate before reuse.
AioSandboxProvider specific options:
port: Base port for sandbox containers (default: 8080)
container_prefix: Prefix for container names (default: deer-flow-sandbox)
mounts: List of volume mounts to share directories with the container
"""
use: str = Field(
...,
description="Class path of the sandbox provider (e.g. deerflow.sandbox.local:LocalSandboxProvider)",
)
allow_host_bash: bool = Field(
default=False,
description="Allow the bash tool to execute directly on the host when using LocalSandboxProvider. Dangerous; intended only for fully trusted local environments.",
)
image: str | None = Field(
default=None,
description="Sandbox image to use (Docker/AIO image or BoxLite OCI image)",
)
port: int | None = Field(
default=None,
description="Base port for sandbox containers",
)
replicas: int | None = Field(
default=None,
description="Maximum active + warm sandboxes/VMs per gateway process (default: 3). Warm/least-recently-used entries are evicted to make room; active sandboxes are not forcibly stopped.",
)
container_prefix: str | None = Field(
default=None,
description="Prefix for container names",
)
idle_timeout: int | None = Field(
default=None,
description="Idle timeout in seconds before released warm sandboxes/VMs are stopped (default: 600 = 10 minutes). Set to 0 to disable.",
)
health_check_skip_seconds: float | None = Field(
default=None,
ge=0,
description="BoxLite-only reclaim skip window in seconds for boxes recently released by this provider instance. Set to 0 to always validate before warm reuse.",
)
mounts: list[VolumeMountConfig] = Field(
default_factory=list,
description="List of volume mounts to share directories between host and container",
)
environment: dict[str, str] = Field(
default_factory=dict,
description="Environment variables to inject into the sandbox container. Values starting with $ will be resolved from host environment variables.",
)
bash_output_max_chars: int = Field(
default=20000,
ge=0,
description="Maximum characters to keep from bash tool output. Output exceeding this limit is middle-truncated (head + tail), preserving the first and last half. Set to 0 to disable truncation.",
)
read_file_output_max_chars: int = Field(
default=50000,
ge=0,
description="Maximum characters to keep from read_file tool output. Output exceeding this limit is head-truncated. Set to 0 to disable truncation.",
)
ls_output_max_chars: int = Field(
default=20000,
ge=0,
description="Maximum characters to keep from ls tool output. Output exceeding this limit is head-truncated. Set to 0 to disable truncation.",
)
bash_command_timeout: int = Field(
default=600,
gt=0,
description=(
"Maximum wall-clock seconds a host bash command may run before it is terminated, process group and all (LocalSandboxProvider). "
"Keeps a blocking foreground command (e.g. an un-backgrounded server) from hanging the turn; background `&` processes return immediately."
),
)
model_config = ConfigDict(extra="allow")