mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-09 21:49:37 +00:00
fix(sandbox): bound Windows command execution (#4946)
* fix(sandbox): bound Windows command execution * fix(sandbox): preserve Windows output encoding * fix(sandbox): honor Python UTF-8 mode * fix(sandbox): normalize captured command newlines * fix(sandbox): scope newline normalization to Windows
This commit is contained in:
parent
d1c06eee96
commit
a4e2a2b934
@ -1,6 +1,6 @@
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command(command, env=None)`, `read_file`, `write_file`, `list_dir`, `glob`, and `grep`. `grep` accepts either one text file or a directory tree. The optional `env` injects per-call environment variables (request-scoped secrets — see Request-Scoped Secrets below); `LocalSandbox` merges it via `subprocess.run(env=...)` and `AioSandbox` routes env-bearing commands through the `bash.exec(env=...)` API on a fresh session.
|
||||
**Interface**: Abstract `Sandbox` with `execute_command(command, env=None)`, `read_file`, `write_file`, `list_dir`, `glob`, and `grep`. `grep` accepts either one text file or a directory tree. The optional `env` injects per-call environment variables (request-scoped secrets — see Request-Scoped Secrets below); `LocalSandbox` merges it into the host subprocess environment and `AioSandbox` routes env-bearing commands through the `bash.exec(env=...)` API on a fresh session.
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop.
|
||||
**Shared components** (RFC #4741): remote providers derive their deterministic sandbox id through `derive_sandbox_scope_token` (`sandbox/identity.py`, keyword-only; the sha256/16-hex derivation is a compatibility contract — changing it orphans existing containers), and serialize provider-selected acquire/release transitions through `AcquireSerializer` (`sandbox/acquire_serialization.py`): per-key `threading.Lock` table with holder/waiter refcount reclamation (no unbounded per-thread lock growth), a bounded dedicated executor so async waits never touch the event loop or the default executor, worker-owned cancellation cleanup that does not depend on a cancelled event loop task resuming, and idempotent `close()` called from provider `shutdown()`/`reset()`. AIO/E2B key by `(user_id, thread_id)`; BoxLite/Tenki/OpenSandbox key by the derived sandbox id. `thread_id=None` acquires (random uuid ids) never enter the serializer.
|
||||
**Authorization gate** (`sandbox:execute`, RFC #4063 Phase 3): every sandbox acquisition passes through `authorize_sandbox_execution` (`deerflow/authz/sandbox_authz.py`) - a binary `authorize(principal, "sandbox", "execute", target="*")` check before `provider.acquire`. The gate lives at the single acquisition entry point (`ensure_sandbox_initialized` / `_acquire_sandbox_async` in `tools.py`, and `SandboxMiddleware.before_agent` / `abefore_agent`), so it cannot be bypassed regardless of which sandbox-dependent tool triggers it; the reuse path (sandbox already in state) skips the re-check. Deny raises `SandboxAuthorizationError` (`sandbox/exceptions.py`), which propagates out of tool execution as a friendly error `ToolMessage` ("sandbox execution is not permitted for your role") - the eager path catches it and skips acquisition instead, deferring the deny to the first sandbox-touching tool call so both paths share the same semantics. Provider errors (both `authorize()` and provider resolution) follow `authorization.fail_closed` / `fail_open`; no readable `config.yaml` or `authorization.enabled: false` makes the gate a no-op (`safe_app_config` tolerates missing config). Gateway auxiliary sync paths (uploads/artifacts routers) call `try_acquire_sandbox_for_request` (`app/gateway/authz.py`), which gates via `authorize_sandbox_for_request` and skips the sync on deny - the upload/artifact edit itself still succeeds. Tests: `tests/test_sandbox_authorization.py`.
|
||||
@ -78,7 +78,7 @@
|
||||
- Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread)
|
||||
|
||||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||||
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox` (host bash), POSIX output is captured through bounded pipe-drain threads and stdin is `/dev/null`, so a backgrounded long-lived process (`server &`) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (`sandbox.bash_command_timeout`, default 600s); on timeout the whole group is killed and the agent gets a notice telling it to background long-lived processes. The bash tool description itself also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See `LocalSandbox.execute_command` / `_run_posix_command` and `bash_tool`'s docstring.
|
||||
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox` (host bash), output on POSIX and Windows is captured through bounded pipe-drain threads and stdin is `/dev/null`; Windows capture decodes with the platform text encoding and applies universal-newline translation, matching the former `subprocess.run(..., text=True)` behavior for locale-code-page output, Python UTF-8 Mode, CRLF, and bare CR. That translation is Windows-only so the pre-existing POSIX output contract remains byte-decoded without newline rewriting. On POSIX, a backgrounded long-lived process (`server &`) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (`sandbox.bash_command_timeout`, default 600s); on timeout the whole POSIX process group or Windows process tree is killed and the agent gets a notice telling it to background long-lived processes. The bash tool description itself also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See `LocalSandbox.execute_command`, its platform runners, and `bash_tool`'s docstring.
|
||||
- `ls` - Directory listing (tree format, max 2 levels)
|
||||
- `glob` - Find files or directories below a root directory with bounded results
|
||||
- `grep` - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
import errno
|
||||
import locale
|
||||
import logging
|
||||
import ntpath
|
||||
import os
|
||||
@ -34,8 +35,16 @@ _PIPE_DRAIN_JOIN_TIMEOUT_SECONDS = 0.2
|
||||
class _BoundedPipeCapture:
|
||||
"""Drain a subprocess pipe while keeping only bounded output in memory."""
|
||||
|
||||
def __init__(self, *, limit_bytes: int = _COMMAND_CAPTURE_LIMIT_BYTES) -> None:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
limit_bytes: int = _COMMAND_CAPTURE_LIMIT_BYTES,
|
||||
encoding: str = "utf-8",
|
||||
normalize_newlines: bool = False,
|
||||
) -> None:
|
||||
self._limit_bytes = limit_bytes
|
||||
self._encoding = encoding
|
||||
self._normalize_newlines = normalize_newlines
|
||||
self._chunks: list[bytes] = []
|
||||
self._kept_bytes = 0
|
||||
self._total_bytes = 0
|
||||
@ -58,7 +67,11 @@ class _BoundedPipeCapture:
|
||||
total_bytes = self._total_bytes
|
||||
kept_bytes = self._kept_bytes
|
||||
|
||||
output = data.decode("utf-8", errors="replace")
|
||||
output = data.decode(self._encoding, errors="replace")
|
||||
if self._normalize_newlines:
|
||||
# Match ``subprocess.run(..., text=True)``: text streams use universal
|
||||
# newlines, translating both CRLF and bare CR to LF.
|
||||
output = output.replace("\r\n", "\n").replace("\r", "\n")
|
||||
if truncated:
|
||||
notice = f"\n... [output truncated after {kept_bytes} of {total_bytes} bytes; remaining output discarded] ..."
|
||||
output += notice
|
||||
@ -154,14 +167,6 @@ class LocalSandbox(Sandbox):
|
||||
"and redirect its output, e.g. `your-command > /mnt/user-data/workspace/server.log 2>&1 &`."
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _coerce_process_output(value: str | bytes | None) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, bytes):
|
||||
return value.decode("utf-8", errors="replace")
|
||||
return value
|
||||
|
||||
@staticmethod
|
||||
def _drain_pipe(fd: int, capture: _BoundedPipeCapture) -> None:
|
||||
try:
|
||||
@ -177,8 +182,14 @@ class LocalSandbox(Sandbox):
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _start_pipe_drain(fd: int, name: str) -> tuple[_BoundedPipeCapture, threading.Thread]:
|
||||
capture = _BoundedPipeCapture()
|
||||
def _start_pipe_drain(
|
||||
fd: int,
|
||||
name: str,
|
||||
*,
|
||||
encoding: str = "utf-8",
|
||||
normalize_newlines: bool = False,
|
||||
) -> tuple[_BoundedPipeCapture, threading.Thread]:
|
||||
capture = _BoundedPipeCapture(encoding=encoding, normalize_newlines=normalize_newlines)
|
||||
thread = threading.Thread(target=LocalSandbox._drain_pipe, args=(fd, capture), name=name, daemon=True)
|
||||
thread.start()
|
||||
return capture, thread
|
||||
@ -522,21 +533,7 @@ class LocalSandbox(Sandbox):
|
||||
"MSYS2_ARG_CONV_EXCL": exclusions,
|
||||
}
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
args,
|
||||
shell=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
env=sandbox_env,
|
||||
)
|
||||
stdout, stderr, returncode = result.stdout, result.stderr, result.returncode
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
timed_out = True
|
||||
stdout = self._coerce_process_output(exc.stdout if exc.stdout is not None else exc.output)
|
||||
stderr = self._coerce_process_output(exc.stderr)
|
||||
returncode = 0
|
||||
stdout, stderr, returncode, timed_out = self._run_windows_command(args, timeout, sandbox_env)
|
||||
else:
|
||||
args = [shell, "-c", resolved_command]
|
||||
stdout, stderr, returncode, timed_out = self._run_posix_command(args, timeout, sandbox_env)
|
||||
@ -554,6 +551,104 @@ class LocalSandbox(Sandbox):
|
||||
# Reverse resolve local paths back to container paths in output
|
||||
return self._reverse_resolve_paths_in_output(final_output)
|
||||
|
||||
@staticmethod
|
||||
def _run_windows_command(
|
||||
args: list[str],
|
||||
timeout: float,
|
||||
env: dict[str, str] | None = None,
|
||||
) -> tuple[str, str, int, bool]:
|
||||
"""Run a Windows command with bounded capture and process-tree timeout."""
|
||||
timed_out = False
|
||||
stdout_read_fd, stdout_write_fd = os.pipe()
|
||||
stderr_read_fd, stderr_write_fd = os.pipe()
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
args,
|
||||
shell=False,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=stdout_write_fd,
|
||||
stderr=stderr_write_fd,
|
||||
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
|
||||
env=env,
|
||||
)
|
||||
except Exception:
|
||||
for fd in (stdout_read_fd, stdout_write_fd, stderr_read_fd, stderr_write_fd):
|
||||
try:
|
||||
os.close(fd)
|
||||
except OSError:
|
||||
# Preserve the original Popen failure; fd cleanup is best-effort.
|
||||
pass
|
||||
raise
|
||||
finally:
|
||||
for fd in (stdout_write_fd, stderr_write_fd):
|
||||
try:
|
||||
os.close(fd)
|
||||
except OSError:
|
||||
# The write fd may already be closed by the exception cleanup above.
|
||||
pass
|
||||
|
||||
encoding = locale.getpreferredencoding(False)
|
||||
stdout_capture, stdout_thread = LocalSandbox._start_pipe_drain(
|
||||
stdout_read_fd,
|
||||
"deerflow-bash-stdout-drain",
|
||||
encoding=encoding,
|
||||
normalize_newlines=True,
|
||||
)
|
||||
stderr_capture, stderr_thread = LocalSandbox._start_pipe_drain(
|
||||
stderr_read_fd,
|
||||
"deerflow-bash-stderr-drain",
|
||||
encoding=encoding,
|
||||
normalize_newlines=True,
|
||||
)
|
||||
|
||||
try:
|
||||
try:
|
||||
process.wait(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
timed_out = True
|
||||
LocalSandbox._terminate_windows_process_tree(process)
|
||||
returncode = process.returncode if process.returncode is not None else 0
|
||||
finally:
|
||||
join_timeout = 10 if timed_out else _PIPE_DRAIN_JOIN_TIMEOUT_SECONDS
|
||||
for thread in (stdout_thread, stderr_thread):
|
||||
thread.join(timeout=join_timeout)
|
||||
if thread.is_alive():
|
||||
logger.debug("Subprocess output drain thread still active after command returned")
|
||||
|
||||
return stdout_capture.read(), stderr_capture.read(), returncode, timed_out
|
||||
|
||||
@staticmethod
|
||||
def _terminate_windows_process_tree(process: subprocess.Popen) -> None:
|
||||
"""Terminate a Windows shell and all descendants, then reap it."""
|
||||
system_root = os.environ.get("SystemRoot", r"C:\Windows")
|
||||
taskkill = ntpath.join(system_root, "System32", "taskkill.exe")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[taskkill, "/PID", str(process.pid), "/T", "/F"],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
timeout=10,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0 and process.poll() is None:
|
||||
try:
|
||||
process.kill()
|
||||
except OSError:
|
||||
logger.debug("Windows process %s exited before fallback kill", process.pid)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
logger.debug("Failed to terminate Windows process tree for pid %s", process.pid, exc_info=True)
|
||||
if process.poll() is None:
|
||||
try:
|
||||
process.kill()
|
||||
except OSError:
|
||||
logger.debug("Windows process %s exited before fallback kill", process.pid)
|
||||
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Process tree for pid %s did not exit after taskkill", process.pid)
|
||||
|
||||
@staticmethod
|
||||
def _run_posix_command(
|
||||
args: list[str],
|
||||
|
||||
@ -17,6 +17,7 @@ concurrent runs on different repos from clobbering each other's token.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, patch
|
||||
@ -59,7 +60,8 @@ def test_local_sandbox_env_overlay_reaches_subprocess(monkeypatch: pytest.Monkey
|
||||
captured["env"] = env
|
||||
return ("", "", 0, False)
|
||||
|
||||
monkeypatch.setattr(LocalSandbox, "_run_posix_command", staticmethod(fake_run_posix))
|
||||
runner = "_run_windows_command" if os.name == "nt" else "_run_posix_command"
|
||||
monkeypatch.setattr(LocalSandbox, runner, staticmethod(fake_run_posix))
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "/bin/bash"))
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": "/usr/bin", "EXISTING": "kept"})
|
||||
|
||||
@ -82,7 +84,8 @@ def test_local_sandbox_no_env_passes_sanitized_environ(monkeypatch: pytest.Monke
|
||||
captured["env"] = env
|
||||
return ("", "", 0, False)
|
||||
|
||||
monkeypatch.setattr(LocalSandbox, "_run_posix_command", staticmethod(fake_run_posix))
|
||||
runner = "_run_windows_command" if os.name == "nt" else "_run_posix_command"
|
||||
monkeypatch.setattr(LocalSandbox, runner, staticmethod(fake_run_posix))
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "/bin/bash"))
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": "/usr/bin", "OPENAI_API_KEY": "sk-leak"})
|
||||
|
||||
@ -240,21 +243,20 @@ def test_local_sandbox_rejects_invalid_env_key(monkeypatch: pytest.MonkeyPatch)
|
||||
"""
|
||||
import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
|
||||
fake_run_called = False
|
||||
fake_popen_called = False
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
nonlocal fake_run_called
|
||||
fake_run_called = True
|
||||
return SimpleNamespace(stdout="", stderr="", returncode=0)
|
||||
def fake_popen(*args, **kwargs):
|
||||
nonlocal fake_popen_called
|
||||
fake_popen_called = True
|
||||
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "Popen", fake_popen)
|
||||
|
||||
with pytest.raises(ValueError, match="extra_env key"):
|
||||
LocalSandbox("local:t").execute_command(
|
||||
"echo hi",
|
||||
env={"X;rm -rf /mnt/user-data;Y": "v"},
|
||||
)
|
||||
assert fake_run_called is False, "subprocess.run must not run when key is invalid"
|
||||
assert fake_popen_called is False, "subprocess.Popen must not run when key is invalid"
|
||||
|
||||
|
||||
def test_aio_sandbox_rejects_invalid_env_key() -> None:
|
||||
|
||||
@ -5,9 +5,8 @@ a backgrounded long-lived process must not keep the bash tool blocked until
|
||||
the timeout, and a genuinely blocking foreground command must be terminated
|
||||
(process group and all) once it exceeds the timeout.
|
||||
|
||||
The POSIX cases exercise real subprocess/process-group semantics, so they are
|
||||
skipped on Windows. Windows keeps the ``subprocess.run`` path, but timeout
|
||||
errors still use the same user-facing notice.
|
||||
The platform-specific cases exercise real subprocess and process-tree/group
|
||||
semantics, while the shared tests pin the user-facing timeout notice.
|
||||
"""
|
||||
|
||||
import os
|
||||
@ -77,19 +76,21 @@ def test_foreground_blocking_command_times_out_with_notice():
|
||||
|
||||
def test_timeout_notice_formats_fractional_and_singular_timeouts(monkeypatch):
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", lambda self: "/bin/sh")
|
||||
monkeypatch.setattr(LocalSandbox, "_run_posix_command", staticmethod(lambda args, timeout, env=None: ("", "", 0, True)))
|
||||
runner = "_run_windows_command" if os.name == "nt" else "_run_posix_command"
|
||||
monkeypatch.setattr(LocalSandbox, runner, staticmethod(lambda args, timeout, env=None: ("", "", 0, True)))
|
||||
|
||||
assert "after 1.5 seconds" in LocalSandbox("t").execute_command("wait", timeout=1.5)
|
||||
assert "after 1 second" in LocalSandbox("t").execute_command("wait", timeout=1)
|
||||
|
||||
|
||||
def test_windows_timeout_expired_returns_notice(monkeypatch):
|
||||
def fake_run(*args, **kwargs):
|
||||
raise local_sandbox.subprocess.TimeoutExpired(cmd=args[0], timeout=kwargs["timeout"], output="partial out", stderr="partial err")
|
||||
|
||||
def test_windows_timeout_returns_notice(monkeypatch):
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", lambda self: "cmd.exe")
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(
|
||||
LocalSandbox,
|
||||
"_run_windows_command",
|
||||
staticmethod(lambda args, timeout, env: ("partial out", "partial err", 0, True)),
|
||||
)
|
||||
|
||||
output = LocalSandbox("t").execute_command("wait", timeout=1.5)
|
||||
|
||||
@ -100,6 +101,34 @@ def test_windows_timeout_expired_returns_notice(monkeypatch):
|
||||
assert "Unexpected error" not in output
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Windows process-tree semantics")
|
||||
def test_windows_foreground_timeout_is_wall_clock_bound(monkeypatch):
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", lambda self: r"C:\Windows\System32\cmd.exe")
|
||||
sandbox = LocalSandbox("t")
|
||||
|
||||
start = time.monotonic()
|
||||
output = sandbox.execute_command("ping -n 5 127.0.0.1", timeout=0.2)
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert elapsed < 2, f"timeout not enforced for process tree, took {elapsed:.1f}s"
|
||||
assert "after 0.2 seconds" in output
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Windows bounded-capture semantics")
|
||||
def test_windows_command_output_is_bounded(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
LocalSandbox,
|
||||
"_get_shell",
|
||||
lambda self: r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe",
|
||||
)
|
||||
emitted_chars = local_sandbox._COMMAND_CAPTURE_LIMIT_BYTES + 1024
|
||||
|
||||
output = LocalSandbox("t").execute_command(f"[Console]::Out.Write('x' * {emitted_chars})", timeout=20)
|
||||
|
||||
assert len(output) < emitted_chars
|
||||
assert "output truncated after 10485760 of 10486784 bytes" in output
|
||||
|
||||
|
||||
@posix_only
|
||||
def test_foreground_timeout_kills_whole_process_group(tmp_path):
|
||||
"""On timeout the entire process group is killed, not just the direct
|
||||
|
||||
@ -1,8 +1,12 @@
|
||||
import builtins
|
||||
from types import SimpleNamespace
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox, PathMapping
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox, PathMapping, _BoundedPipeCapture
|
||||
|
||||
|
||||
def _open(base, file, mode="r", *args, **kwargs):
|
||||
@ -11,6 +15,104 @@ def _open(base, file, mode="r", *args, **kwargs):
|
||||
return base(file, mode, *args, encoding=kwargs.pop("encoding", "gbk"), **kwargs)
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_decodes_non_utf8_output_with_configured_encoding():
|
||||
capture = _BoundedPipeCapture(encoding="cp1252")
|
||||
capture.append("caf\u00e9".encode("cp1252"))
|
||||
|
||||
assert capture.read() == "caf\u00e9"
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_applies_text_mode_newline_normalization_when_enabled():
|
||||
capture = _BoundedPipeCapture(normalize_newlines=True)
|
||||
capture.append(b"crlf\r\nbare-cr\rlf\n")
|
||||
|
||||
assert capture.read() == "crlf\nbare-cr\nlf\n"
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_preserves_posix_newlines_by_default():
|
||||
capture = _BoundedPipeCapture()
|
||||
capture.append(b"crlf\r\nbare-cr\rlf\n")
|
||||
|
||||
assert capture.read() == "crlf\r\nbare-cr\rlf\n"
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX capture semantics")
|
||||
def test_posix_command_capture_preserves_newlines():
|
||||
stdout, stderr, returncode, timed_out = LocalSandbox._run_posix_command(
|
||||
[sys.executable, "-c", "import os; os.write(1, b'crlf\\r\\nbare-cr\\rlf\\n')"],
|
||||
10,
|
||||
)
|
||||
|
||||
assert stdout == "crlf\r\nbare-cr\rlf\n"
|
||||
assert stderr == ""
|
||||
assert returncode == 0
|
||||
assert timed_out is False
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Windows text-mode newline semantics")
|
||||
def test_windows_command_capture_normalizes_newlines():
|
||||
stdout, stderr, returncode, timed_out = LocalSandbox._run_windows_command(
|
||||
[sys.executable, "-c", "import os; os.write(1, b'crlf\\r\\nbare-cr\\rlf\\n')"],
|
||||
10,
|
||||
)
|
||||
|
||||
assert stdout == "crlf\nbare-cr\nlf\n"
|
||||
assert stderr == ""
|
||||
assert returncode == 0
|
||||
assert timed_out is False
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Windows text-mode encoding semantics")
|
||||
@pytest.mark.parametrize(
|
||||
("python_args", "python_utf8"),
|
||||
[([], "0"), ([], "1"), (["-X", "utf8"], "0")],
|
||||
ids=["locale-code-page", "PYTHONUTF8", "-X-utf8"],
|
||||
)
|
||||
def test_windows_capture_matches_subprocess_text_mode_encoding(python_args, python_utf8):
|
||||
probe = r"""
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
|
||||
reference = subprocess.Popen([sys.executable, "-c", ""], stdout=subprocess.PIPE, text=True)
|
||||
encoding = reference.stdout.encoding
|
||||
reference.communicate()
|
||||
|
||||
for expected in ("caf\u00e9", "\u4f60\u597d", "\u65e5\u672c\u8a9e", "\u041f\u0440\u0438\u0432\u0435\u0442"):
|
||||
try:
|
||||
payload = expected.encode(encoding)
|
||||
except UnicodeEncodeError:
|
||||
continue
|
||||
if any(byte >= 0x80 for byte in payload):
|
||||
break
|
||||
else:
|
||||
raise AssertionError(f"no non-ASCII probe text for {encoding}")
|
||||
|
||||
stdout, stderr, returncode, timed_out = LocalSandbox._run_windows_command(
|
||||
[sys.executable, "-c", f"import sys; sys.stdout.buffer.write(bytes.fromhex('{payload.hex()}'))"],
|
||||
10,
|
||||
)
|
||||
assert stdout == expected, (encoding, stdout)
|
||||
assert stderr == ""
|
||||
assert returncode == 0
|
||||
assert timed_out is False
|
||||
"""
|
||||
env = os.environ.copy()
|
||||
env["PYTHONUTF8"] = python_utf8
|
||||
|
||||
result = subprocess.run(
|
||||
[sys.executable, *python_args, "-c", probe],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
env=env,
|
||||
check=False,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
|
||||
|
||||
def test_read_file_uses_utf8_on_windows_locale(tmp_path, monkeypatch):
|
||||
path = tmp_path / "utf8.txt"
|
||||
text = "\u201cutf8\u201d"
|
||||
@ -79,16 +181,16 @@ def test_get_shell_uses_cmd_as_last_windows_fallback(monkeypatch):
|
||||
|
||||
|
||||
def test_execute_command_uses_powershell_command_mode_on_windows(monkeypatch):
|
||||
calls: list[tuple[object, dict]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
calls.append((args[0], kwargs))
|
||||
return SimpleNamespace(stdout="ok", stderr="", returncode=0)
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows", "OPENAI_API_KEY": "should-not-leak"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"))
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
output = LocalSandbox("t").execute_command("Write-Output hello")
|
||||
|
||||
@ -104,29 +206,24 @@ def test_execute_command_uses_powershell_command_mode_on_windows(monkeypatch):
|
||||
"-Command",
|
||||
"Write-Output hello",
|
||||
],
|
||||
{
|
||||
"shell": False,
|
||||
"capture_output": True,
|
||||
"text": True,
|
||||
"timeout": 600,
|
||||
"env": {"PATH": r"C:\Windows"},
|
||||
},
|
||||
600,
|
||||
{"PATH": r"C:\Windows"},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_execute_command_keeps_msys_path_conversion_for_host_commands_on_windows(monkeypatch):
|
||||
calls: list[tuple[object, dict]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
calls.append((args[0], kwargs))
|
||||
return SimpleNamespace(stdout="ok", stderr="", returncode=0)
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Program Files\Git\bin"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Program Files\Git\bin\sh.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_msys_path_conversion_exclusions", lambda self: "/mnt/user-data")
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
output = LocalSandbox("t").execute_command("echo hello")
|
||||
|
||||
@ -134,59 +231,54 @@ def test_execute_command_keeps_msys_path_conversion_for_host_commands_on_windows
|
||||
assert calls == [
|
||||
(
|
||||
[r"C:\Program Files\Git\bin\sh.exe", "-c", "echo hello"],
|
||||
600,
|
||||
{
|
||||
"shell": False,
|
||||
"capture_output": True,
|
||||
"text": True,
|
||||
"timeout": 600,
|
||||
"env": {
|
||||
"PATH": r"C:\Program Files\Git\bin",
|
||||
"MSYS2_ARG_CONV_EXCL": "/mnt/user-data",
|
||||
},
|
||||
"PATH": r"C:\Program Files\Git\bin",
|
||||
"MSYS2_ARG_CONV_EXCL": "/mnt/user-data",
|
||||
},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_execute_command_scopes_msys_path_conversion_exclusions_on_windows(monkeypatch):
|
||||
calls: list[tuple[object, dict]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
calls.append((args[0], kwargs))
|
||||
return SimpleNamespace(stdout="ok", stderr="", returncode=0)
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Program Files\Git\bin"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Program Files\Git\bin\sh.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_msys_path_conversion_exclusions", lambda self: "/mnt/user-data")
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
output = LocalSandbox("t").execute_command("cat /mnt/user-data/workspace/input.txt")
|
||||
|
||||
assert output == "ok"
|
||||
assert calls[0][1]["env"] == {
|
||||
assert calls[0][2] == {
|
||||
"PATH": r"C:\Program Files\Git\bin",
|
||||
"MSYS2_ARG_CONV_EXCL": "/mnt/user-data",
|
||||
}
|
||||
|
||||
|
||||
def test_execute_command_ignores_root_msys_mapping_for_host_commands_on_windows(monkeypatch):
|
||||
calls: list[tuple[object, dict]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
calls.append((args[0], kwargs))
|
||||
return SimpleNamespace(stdout="ok", stderr="", returncode=0)
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Program Files\Git\bin"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Program Files\Git\bin\sh.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_msys_path_conversion_exclusions", lambda self: "")
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
output = LocalSandbox("t").execute_command("echo hello")
|
||||
|
||||
assert output == "ok"
|
||||
assert calls[0][1]["env"] == {"PATH": r"C:\Program Files\Git\bin"}
|
||||
assert calls[0][2] == {"PATH": r"C:\Program Files\Git\bin"}
|
||||
|
||||
|
||||
def test_msys_path_conversion_exclusions_omit_blanket_patterns():
|
||||
@ -204,37 +296,37 @@ def test_msys_path_conversion_exclusions_omit_blanket_patterns():
|
||||
|
||||
|
||||
def test_execute_command_does_not_set_msys_env_for_non_msys_posix_shell_on_windows(monkeypatch):
|
||||
calls: list[tuple[object, dict]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
calls.append((args[0], kwargs))
|
||||
return SimpleNamespace(stdout="ok", stderr="", returncode=0)
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\tools"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\tools\busybox\sh.exe"))
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
output = LocalSandbox("t").execute_command("echo /mnt/skills/demo")
|
||||
|
||||
assert output == "ok"
|
||||
# Non-MSYS posix shell adds no MSYS_* vars; the env is the scrubbed inherited
|
||||
# environment, not None (#3861).
|
||||
assert calls[0][1]["env"] == {"PATH": r"C:\tools"}
|
||||
assert "MSYS_NO_PATHCONV" not in calls[0][1]["env"]
|
||||
assert calls[0][2] == {"PATH": r"C:\tools"}
|
||||
assert "MSYS_NO_PATHCONV" not in calls[0][2]
|
||||
|
||||
|
||||
def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch):
|
||||
calls: list[tuple[object, dict]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
|
||||
def fake_run(*args, **kwargs):
|
||||
calls.append((args[0], kwargs))
|
||||
return SimpleNamespace(stdout="ok", stderr="", returncode=0)
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows", "GITHUB_TOKEN": "should-not-leak"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\cmd.exe"))
|
||||
monkeypatch.setattr(local_sandbox.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
output = LocalSandbox("t").execute_command("echo hello")
|
||||
|
||||
@ -244,12 +336,7 @@ def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch):
|
||||
assert calls == [
|
||||
(
|
||||
[r"C:\Windows\System32\cmd.exe", "/c", "echo hello"],
|
||||
{
|
||||
"shell": False,
|
||||
"capture_output": True,
|
||||
"text": True,
|
||||
"timeout": 600,
|
||||
"env": {"PATH": r"C:\Windows"},
|
||||
},
|
||||
600,
|
||||
{"PATH": r"C:\Windows"},
|
||||
)
|
||||
]
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user