mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 13:40:37 +00:00
fix(sandbox): decode cmd.exe output with the ANSI code page under UTF-8 mode (#6777)
* fix(sandbox): decode cmd.exe output with the ANSI code page under UTF-8 mode The cmd.exe branch left its pipe decoder to the `encoding=None` default of `_run_windows_command`, which follows Python text-mode semantics via `locale.getpreferredencoding(False)`. Under Python UTF-8 mode (`PYTHONUTF8=1`, `-X utf8`, the Windows installer's UTF-8 opt-in, and the default once PEP 686 lands) that call reports `"utf-8"`, while cmd.exe and the children it spawns keep writing the host ANSI code page - so every non-ASCII byte in cmd output collapses to U+FFFD, with no fallback path. Measured on a real zh-CN host under `PYTHONUTF8=1` (`getpreferredencoding` -> `utf-8`, `getencoding` -> `cp936`): - through the `encoding=None` default: `'��� �����Z '` - pinned to the ANSI code page: `'你好 日本語 '` This branch pins the cmd.exe pipe decoder to `locale.getencoding()` (3.11+, repo floor is 3.12), the same selection the MSYS and PowerShell branches already use for their fallback (#6727, #6548). ## Scope note The `encoding=None` default itself is deliberately unchanged: it mirrors Python's own `text=True` semantics and is pinned by `test_windows_capture_matches_subprocess_text_mode_encoding`. Only the cmd.exe branch - whose output is known to be the ANSI code page - overrides it, the same way the MSYS and PowerShell branches already pin their own decoders. Follow-up to willem-bd's review note on #6727 (inline comment 4247281255) flagging this path. ## Testing - new (POSIX-runnable, mocked): the cmd branch passes `encoding=locale.getencoding()` to `_run_windows_command`, verified under a simulated UTF-8-mode host (`getpreferredencoding` -> `utf-8`, `getencoding` -> `cp936`). - updated: `test_execute_command_uses_cmd_command_mode_on_windows` now also pins the new `encoding` kwarg. - new (Windows; skipped when the console output CP does not match the ANSI page - sibling tests can change it process-wide): real `cmd.exe /c echo` roundtrip through `execute_command` returns the text intact (passes on a zh-CN host when run standalone). - full `tests/test_local_sandbox_encoding.py`: 40 passed, 2 skipped under both `PYTHONUTF8=0` and `PYTHONUTF8=1` (Windows 11, Python 3.13.12, repo venv); the text-mode contract test stays green in all three parametrized modes (`locale-code-page`, `PYTHONUTF8`, `-X-utf8`). Red run against unmodified main: both new/updated cmd-branch tests fail (`KeyError: 'encoding'`). - `ruff check` + `ruff format --check` clean on both files. * fix(sandbox): use the cmd console output code page --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
sjr666666
GitHub
parent
3343522fa2
commit
19e0242476
@ -1402,6 +1402,12 @@ names and base URLs readable.
|
||||
|
||||
For `LocalSandboxProvider`, this is a managed tool-path boundary rather than host filesystem isolation. Explicit per-Agent skill policies are accepted only while host bash is disabled (the default), because a host subprocess can address canonical paths without using the provider's virtual-path mappings. Use Docker/AIO, the Kubernetes provisioner, or E2B when the filesystem boundary must remain enforceable alongside shell access.
|
||||
|
||||
On Windows, the local sandbox's `cmd.exe` fallback decodes builtin output using
|
||||
the console output code page, including UTF-8 after `chcp 65001`, independently
|
||||
of Python's UTF-8 mode. If no console page is available, it uses the system OEM
|
||||
page, with ANSI as a last resort. Native programs can choose a different output
|
||||
encoding; configure those programs to match the console page when needed.
|
||||
|
||||
Managed integrations install shared read-only skill packs without mixing them
|
||||
into custom skills. The Lark/Feishu CLI integration is available under
|
||||
`Capability Center → Plugins → Lark / Feishu`; an administrator installs or
|
||||
|
||||
@ -21,7 +21,7 @@ Holders stay bound through scope cleanup; AIO reserves idle teardown atomically.
|
||||
Fork lookups reject invalidated clients; cancelled acquires leave borrowers a pending release.
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-user/thread `LocalSandbox` (id `local:{user_id}:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Shared runs use category mappings; a policy-scoped run replaces them with one `/mnt/skills` root mapping to the coherent thread view, so structured file tools resolve through one managed boundary. This is not a host filesystem security boundary: an enabled host `bash` subprocess can use canonical paths without `PathMapping`, so `supports_agent_skill_isolation` is dynamic and explicit Agent policies fail closed while host bash is enabled; unrestricted runs then get the shared mappings, never a leftover thread view. Host-to-virtual output masking scans dynamic per-user/per-thread roots directly instead of compiling path-specific regexes, so evicted thread IDs do not remain in Python's global regex caches; a separate 256-entry root cache prevents repeated `realpath()` walks for every glob/grep match while bounding dynamic-path retention, and only the small process-stable skill/integration source set uses a bounded compiled cache. The shared `path_patterns.py` tail stops at `:`, so `$PATH`-style lists mask every entry; a mount symlink resolving outside all mounts keeps its mount path. On Windows, Git Bash/MSYS argument-conversion exclusions are limited to safe non-root virtual path prefixes; do not restore a blanket conversion disable, because host-native CLI launchers need normal MSYS path conversion for their own installation paths.
|
||||
PowerShell fallback pairs guarded console UTF-8 setters with explicit UTF-8 pipe decoding; cmd/MSYS retain locale decoding. Encoding regressions include real-pipe probes and Windows-only PowerShell roundtrips with and without a console.
|
||||
PowerShell fallback pairs guarded console UTF-8 setters with explicit UTF-8 pipe decoding; MSYS prefers UTF-8 with an ANSI fallback. Cmd builtin output uses `GetConsoleOutputCP()`, then `GetOEMCP()` when no console page is available, and finally `locale.getencoding()` if the APIs cannot supply a page. Query per command because the console page can change; keep `_run_windows_command(encoding=None)` aligned with Python text-mode semantics. Native tools may use their own encodings. Encoding regressions include real pipes with distinct console/ANSI pages and Windows-only cmd roundtrips in private consoles, plus PowerShell roundtrips with and without a console.
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker isolation. Reuse enforces minimum persisted shell capacity even without image-env overrides. Upgrades require ownership-fenced teardown; provisioner create returns 409 for undersized Pods without deleting them. Acquire/reuse drops definitively dead active/warm entries before rediscovery/creation. Health-check failures mean unknown, so local discovery skips unverifiable containers. Acquire/reclaim publishes ownership; `_renew_owned_leases` refreshes it. Text appends use native AIO append, never read-modify-write. `reset()` closes acquire pools; `shutdown()` performs full teardown. Async acquire separates lock waiters from holder work; started cache/warm workers drain, queued work/skills projection stay cancellable. Create polls flock; owned work/readiness/registration/cleanup drain before unlock. `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner=False); `sandbox.thread_data_mounts` overrides this for deployments sharing thread user-data directories. `true` skips upload acquire/sync; a false positive makes uploads unavailable. Explicit Agent policies use four thread projection category mounts and a distinct deterministic identity, excluding older shared-mount containers. The startup snapshot of `skills.container_path` drives mounts, identity, remote requests, and provisioner validation; root changes prevent container/Pod reuse. Gateway and provisioner independently require a canonical absolute root outside reserved platform mounts and derive four category allowlist entries from it. All four category overrides suppress the provisioner's default hostPath/skills-PVC mount; with `USERDATA_PVC_NAME`, categories use subpaths on that shared PVC. Readiness probes and `agent_sandbox` clients set `trust_env=False` for direct control-plane destinations (loopback/private IPs, single-label cluster hosts, Docker/Podman internal hostnames); external FQDNs and public IPs retain environment proxies.
|
||||
- `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation.
|
||||
New unrestricted sandboxes receive a one-shot upload from the enabled-only
|
||||
|
||||
@ -1,3 +1,4 @@
|
||||
import ctypes
|
||||
import errno
|
||||
import locale
|
||||
import logging
|
||||
@ -567,6 +568,24 @@ class LocalSandbox(Sandbox):
|
||||
|
||||
return pattern.sub(replace_match, content)
|
||||
|
||||
@staticmethod
|
||||
def _get_cmd_output_encoding() -> str:
|
||||
"""Select cmd builtins' console page, or the OEM default without a console."""
|
||||
try:
|
||||
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
|
||||
for name in ("GetConsoleOutputCP", "GetOEMCP"):
|
||||
get_code_page = getattr(kernel32, name)
|
||||
get_code_page.argtypes = []
|
||||
get_code_page.restype = ctypes.c_uint
|
||||
code_page = get_code_page()
|
||||
if code_page:
|
||||
return f"cp{code_page}"
|
||||
except (AttributeError, OSError):
|
||||
pass
|
||||
# Last resort when the Windows APIs are unavailable. Unlike
|
||||
# getpreferredencoding(), this remains independent of Python UTF-8 mode.
|
||||
return locale.getencoding()
|
||||
|
||||
@staticmethod
|
||||
def _get_shell() -> str:
|
||||
"""Detect available shell executable with fallback."""
|
||||
@ -679,6 +698,19 @@ class LocalSandbox(Sandbox):
|
||||
# ``PYTHONUTF8=1``, silently making this fallback a no-op.
|
||||
fallback_encoding=locale.getencoding(),
|
||||
)
|
||||
elif self._is_cmd_shell(shell):
|
||||
# cmd builtins use the console output page, which can differ
|
||||
# from ANSI (e.g. cp437 vs cp1252) or be UTF-8 after chcp 65001.
|
||||
# Native children can choose their own encoding; configure
|
||||
# those tools explicitly when it differs from the console.
|
||||
# Keep the encoding=None default's Python text-mode contract
|
||||
# unchanged and override it only for this shell.
|
||||
stdout, stderr, returncode, timed_out = self._run_windows_command(
|
||||
args,
|
||||
timeout,
|
||||
sandbox_env,
|
||||
encoding=self._get_cmd_output_encoding(),
|
||||
)
|
||||
else:
|
||||
stdout, stderr, returncode, timed_out = self._run_windows_command(args, timeout, sandbox_env)
|
||||
else:
|
||||
|
||||
@ -102,7 +102,7 @@ def test_windows_timeout_returns_notice(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
LocalSandbox,
|
||||
"_run_windows_command",
|
||||
staticmethod(lambda args, timeout, env: ("partial out", "partial err", 0, True)),
|
||||
staticmethod(lambda args, timeout, env, **kwargs: ("partial out", "partial err", 0, True)),
|
||||
)
|
||||
|
||||
output = LocalSandbox("t").execute_command("wait", timeout=1.5)
|
||||
|
||||
@ -1,8 +1,10 @@
|
||||
import builtins
|
||||
import ctypes
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from support.shell import require_posix_sh
|
||||
@ -645,17 +647,23 @@ def test_execute_command_does_not_set_msys_env_for_non_msys_posix_shell_on_windo
|
||||
assert "MSYS_NO_PATHCONV" not in calls[0][2]
|
||||
|
||||
|
||||
def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch):
|
||||
calls: list[tuple[list[str], float, dict[str, str]]] = []
|
||||
def _mock_cmd_code_pages(monkeypatch, console_cp, oem_cp):
|
||||
kernel32 = SimpleNamespace(GetConsoleOutputCP=lambda: console_cp, GetOEMCP=lambda: oem_cp)
|
||||
monkeypatch.setattr(ctypes, "WinDLL", lambda *args, **kwargs: kernel32, raising=False)
|
||||
|
||||
def fake_run(args, timeout, env):
|
||||
calls.append((args, timeout, env))
|
||||
|
||||
def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch):
|
||||
calls: list[tuple[list[str], float, dict[str, str], dict]] = []
|
||||
|
||||
def fake_run(args, timeout, env, **kwargs):
|
||||
calls.append((args, timeout, env, kwargs))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows", "GITHUB_TOKEN": "should-not-leak"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\cmd.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
_mock_cmd_code_pages(monkeypatch, 850, 437)
|
||||
|
||||
output = LocalSandbox("t").execute_command("echo hello")
|
||||
|
||||
@ -667,5 +675,136 @@ def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch):
|
||||
[r"C:\Windows\System32\cmd.exe", "/c", "echo hello"],
|
||||
600,
|
||||
{"PATH": r"C:\Windows"},
|
||||
{"encoding": "cp850"},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_execute_command_cmd_branch_ignores_utf8_mode_for_console_encoding(monkeypatch):
|
||||
"""Select the console page independently of Python UTF-8 mode and ANSI."""
|
||||
monkeypatch.setattr(local_sandbox.locale, "getpreferredencoding", lambda _: "utf-8")
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252")
|
||||
_mock_cmd_code_pages(monkeypatch, 936, 437)
|
||||
|
||||
kwargs: dict = {}
|
||||
|
||||
def fake_run(args, timeout, env, **kw):
|
||||
kwargs.update(kw)
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows"})
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\cmd.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
LocalSandbox("t").execute_command("echo hello")
|
||||
|
||||
assert kwargs["encoding"] == "cp936"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"console_cp,oem_cp,pipe_encoding,probe",
|
||||
[(437, 850, "cp437", "café"), (850, 437, "cp850", "café £"), (65001, 437, "utf-8", "你好 日本語"), (0, 850, "cp850", "café £"), (0, 0, "cp1252", "café")],
|
||||
ids=["oem-437", "oem-850", "utf8-console", "no-console", "ansi-fallback"],
|
||||
)
|
||||
def test_execute_command_cmd_decodes_console_code_page(monkeypatch, console_cp, oem_cp, pipe_encoding, probe):
|
||||
"""Decode real stdout/stderr pipes with differing console and ANSI pages."""
|
||||
_mock_cmd_code_pages(monkeypatch, console_cp, oem_cp)
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252")
|
||||
monkeypatch.setattr(local_sandbox.locale, "getpreferredencoding", lambda _: "utf-8")
|
||||
if os.name != "nt":
|
||||
monkeypatch.setattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0, raising=False)
|
||||
run_windows_command = LocalSandbox._run_windows_command
|
||||
payload = (probe + "\r\n").encode(pipe_encoding)
|
||||
script = f"import os; p=bytes.fromhex('{payload.hex()}'); os.write(1,p); os.write(2,p); raise SystemExit(3)"
|
||||
|
||||
def run_pipe_probe(args, timeout, env, **kwargs):
|
||||
return run_windows_command([sys.executable, "-c", script], timeout, env, **kwargs)
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "cmd.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(run_pipe_probe))
|
||||
|
||||
output = LocalSandbox("t").execute_command("echo probe", timeout=10)
|
||||
|
||||
assert output == f"{probe}\n\nStd Error:\n{probe}\n\nExit Code: 3"
|
||||
|
||||
|
||||
def test_execute_command_cmd_rechecks_console_page_between_commands(monkeypatch):
|
||||
pages = iter([437, 65001])
|
||||
kernel32 = SimpleNamespace(GetConsoleOutputCP=lambda: next(pages), GetOEMCP=lambda: 437)
|
||||
monkeypatch.setattr(ctypes, "WinDLL", lambda *args, **kwargs: kernel32, raising=False)
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252")
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "cmd.exe"))
|
||||
payloads = iter(["café".encode("cp437"), "你好".encode()])
|
||||
|
||||
def fake_run(args, timeout, env, **kwargs):
|
||||
capture = _BoundedPipeCapture(**kwargs)
|
||||
capture.append(next(payloads))
|
||||
return capture.read(), "", 0, False
|
||||
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
sandbox = LocalSandbox("t")
|
||||
|
||||
assert sandbox.execute_command("echo café") == "café"
|
||||
assert sandbox.execute_command("echo 你好") == "你好"
|
||||
|
||||
|
||||
def test_execute_command_cmd_falls_back_when_windows_api_is_unavailable(monkeypatch):
|
||||
def unavailable(*args, **kwargs):
|
||||
raise OSError("kernel32 unavailable")
|
||||
|
||||
monkeypatch.setattr(ctypes, "WinDLL", unavailable, raising=False)
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252")
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "cmd.exe"))
|
||||
kwargs = {}
|
||||
|
||||
def fake_run(args, timeout, env, **kw):
|
||||
kwargs.update(kw)
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
assert LocalSandbox("t").execute_command("echo hello") == "ok"
|
||||
assert kwargs["encoding"] == "cp1252"
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Requires real Windows cmd.exe")
|
||||
@pytest.mark.parametrize("code_page", [437, 850, 936, 65001])
|
||||
def test_execute_command_cmd_branch_roundtrip_with_console_code_page(code_page):
|
||||
"""Use a private console so changing its page cannot affect sibling tests."""
|
||||
probe = r"""
|
||||
import ctypes
|
||||
import os
|
||||
import sys
|
||||
import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
|
||||
code_page = int(sys.argv[1])
|
||||
assert ctypes.windll.kernel32.SetConsoleOutputCP(code_page)
|
||||
# Make ANSI and console encoding differ even on a zh-CN or UTF-8 runner.
|
||||
local_sandbox.locale.getencoding = lambda: "cp1252"
|
||||
shell = os.path.join(os.environ.get("SystemRoot", r"C:\Windows"), "System32", "cmd.exe")
|
||||
LocalSandbox._get_shell = staticmethod(lambda: shell)
|
||||
expected = "caf\u00e9" if code_page in (437, 850) else "\u4f60\u597d \u65e5\u672c\u8a9e"
|
||||
output = LocalSandbox("cmd-encoding-probe").execute_command(f"echo {expected}", timeout=15)
|
||||
assert output == expected + "\n", ascii(output)
|
||||
"""
|
||||
startupinfo = subprocess.STARTUPINFO()
|
||||
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
|
||||
startupinfo.wShowWindow = subprocess.SW_HIDE
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", probe, str(code_page)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
env={**os.environ, "PYTHONUTF8": "1"},
|
||||
creationflags=subprocess.CREATE_NEW_CONSOLE,
|
||||
startupinfo=startupinfo,
|
||||
timeout=30,
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user