diff --git a/README.md b/README.md index 5f6fcec002..6e77f757c0 100644 --- a/README.md +++ b/README.md @@ -1402,6 +1402,12 @@ names and base URLs readable. For `LocalSandboxProvider`, this is a managed tool-path boundary rather than host filesystem isolation. Explicit per-Agent skill policies are accepted only while host bash is disabled (the default), because a host subprocess can address canonical paths without using the provider's virtual-path mappings. Use Docker/AIO, the Kubernetes provisioner, or E2B when the filesystem boundary must remain enforceable alongside shell access. +On Windows, the local sandbox's `cmd.exe` fallback decodes builtin output using +the console output code page, including UTF-8 after `chcp 65001`, independently +of Python's UTF-8 mode. If no console page is available, it uses the system OEM +page, with ANSI as a last resort. Native programs can choose a different output +encoding; configure those programs to match the console page when needed. + Managed integrations install shared read-only skill packs without mixing them into custom skills. The Lark/Feishu CLI integration is available under `Capability Center → Plugins → Lark / Feishu`; an administrator installs or diff --git a/backend/packages/harness/deerflow/sandbox/AGENTS.md b/backend/packages/harness/deerflow/sandbox/AGENTS.md index 35d6494708..71d7d96459 100644 --- a/backend/packages/harness/deerflow/sandbox/AGENTS.md +++ b/backend/packages/harness/deerflow/sandbox/AGENTS.md @@ -21,7 +21,7 @@ Holders stay bound through scope cleanup; AIO reserves idle teardown atomically. Fork lookups reject invalidated clients; cancelled acquires leave borrowers a pending release. **Implementations**: - `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-user/thread `LocalSandbox` (id `local:{user_id}:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Shared runs use category mappings; a policy-scoped run replaces them with one `/mnt/skills` root mapping to the coherent thread view, so structured file tools resolve through one managed boundary. This is not a host filesystem security boundary: an enabled host `bash` subprocess can use canonical paths without `PathMapping`, so `supports_agent_skill_isolation` is dynamic and explicit Agent policies fail closed while host bash is enabled; unrestricted runs then get the shared mappings, never a leftover thread view. Host-to-virtual output masking scans dynamic per-user/per-thread roots directly instead of compiling path-specific regexes, so evicted thread IDs do not remain in Python's global regex caches; a separate 256-entry root cache prevents repeated `realpath()` walks for every glob/grep match while bounding dynamic-path retention, and only the small process-stable skill/integration source set uses a bounded compiled cache. The shared `path_patterns.py` tail stops at `:`, so `$PATH`-style lists mask every entry; a mount symlink resolving outside all mounts keeps its mount path. On Windows, Git Bash/MSYS argument-conversion exclusions are limited to safe non-root virtual path prefixes; do not restore a blanket conversion disable, because host-native CLI launchers need normal MSYS path conversion for their own installation paths. - PowerShell fallback pairs guarded console UTF-8 setters with explicit UTF-8 pipe decoding; cmd/MSYS retain locale decoding. Encoding regressions include real-pipe probes and Windows-only PowerShell roundtrips with and without a console. + PowerShell fallback pairs guarded console UTF-8 setters with explicit UTF-8 pipe decoding; MSYS prefers UTF-8 with an ANSI fallback. Cmd builtin output uses `GetConsoleOutputCP()`, then `GetOEMCP()` when no console page is available, and finally `locale.getencoding()` if the APIs cannot supply a page. Query per command because the console page can change; keep `_run_windows_command(encoding=None)` aligned with Python text-mode semantics. Native tools may use their own encodings. Encoding regressions include real pipes with distinct console/ANSI pages and Windows-only cmd roundtrips in private consoles, plus PowerShell roundtrips with and without a console. - `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker isolation. Reuse enforces minimum persisted shell capacity even without image-env overrides. Upgrades require ownership-fenced teardown; provisioner create returns 409 for undersized Pods without deleting them. Acquire/reuse drops definitively dead active/warm entries before rediscovery/creation. Health-check failures mean unknown, so local discovery skips unverifiable containers. Acquire/reclaim publishes ownership; `_renew_owned_leases` refreshes it. Text appends use native AIO append, never read-modify-write. `reset()` closes acquire pools; `shutdown()` performs full teardown. Async acquire separates lock waiters from holder work; started cache/warm workers drain, queued work/skills projection stay cancellable. Create polls flock; owned work/readiness/registration/cleanup drain before unlock. `uses_thread_data_mounts` defaults to backend detection (`LocalContainerBackend=True`, remote/provisioner=False); `sandbox.thread_data_mounts` overrides this for deployments sharing thread user-data directories. `true` skips upload acquire/sync; a false positive makes uploads unavailable. Explicit Agent policies use four thread projection category mounts and a distinct deterministic identity, excluding older shared-mount containers. The startup snapshot of `skills.container_path` drives mounts, identity, remote requests, and provisioner validation; root changes prevent container/Pod reuse. Gateway and provisioner independently require a canonical absolute root outside reserved platform mounts and derive four category allowlist entries from it. All four category overrides suppress the provisioner's default hostPath/skills-PVC mount; with `USERDATA_PVC_NAME`, categories use subpaths on that shared PVC. Readiness probes and `agent_sandbox` clients set `trust_env=False` for direct control-plane destinations (loopback/private IPs, single-label cluster hosts, Docker/Podman internal hostnames); external FQDNs and public IPs retain environment proxies. - `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation. New unrestricted sandboxes receive a one-shot upload from the enabled-only diff --git a/backend/packages/harness/deerflow/sandbox/local/local_sandbox.py b/backend/packages/harness/deerflow/sandbox/local/local_sandbox.py index 5d25b13d6e..4ffe168c20 100644 --- a/backend/packages/harness/deerflow/sandbox/local/local_sandbox.py +++ b/backend/packages/harness/deerflow/sandbox/local/local_sandbox.py @@ -1,3 +1,4 @@ +import ctypes import errno import locale import logging @@ -567,6 +568,24 @@ class LocalSandbox(Sandbox): return pattern.sub(replace_match, content) + @staticmethod + def _get_cmd_output_encoding() -> str: + """Select cmd builtins' console page, or the OEM default without a console.""" + try: + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + for name in ("GetConsoleOutputCP", "GetOEMCP"): + get_code_page = getattr(kernel32, name) + get_code_page.argtypes = [] + get_code_page.restype = ctypes.c_uint + code_page = get_code_page() + if code_page: + return f"cp{code_page}" + except (AttributeError, OSError): + pass + # Last resort when the Windows APIs are unavailable. Unlike + # getpreferredencoding(), this remains independent of Python UTF-8 mode. + return locale.getencoding() + @staticmethod def _get_shell() -> str: """Detect available shell executable with fallback.""" @@ -679,6 +698,19 @@ class LocalSandbox(Sandbox): # ``PYTHONUTF8=1``, silently making this fallback a no-op. fallback_encoding=locale.getencoding(), ) + elif self._is_cmd_shell(shell): + # cmd builtins use the console output page, which can differ + # from ANSI (e.g. cp437 vs cp1252) or be UTF-8 after chcp 65001. + # Native children can choose their own encoding; configure + # those tools explicitly when it differs from the console. + # Keep the encoding=None default's Python text-mode contract + # unchanged and override it only for this shell. + stdout, stderr, returncode, timed_out = self._run_windows_command( + args, + timeout, + sandbox_env, + encoding=self._get_cmd_output_encoding(), + ) else: stdout, stderr, returncode, timed_out = self._run_windows_command(args, timeout, sandbox_env) else: diff --git a/backend/tests/test_local_sandbox_command_timeout.py b/backend/tests/test_local_sandbox_command_timeout.py index 8727f8d66c..5d49ab8394 100644 --- a/backend/tests/test_local_sandbox_command_timeout.py +++ b/backend/tests/test_local_sandbox_command_timeout.py @@ -102,7 +102,7 @@ def test_windows_timeout_returns_notice(monkeypatch): monkeypatch.setattr( LocalSandbox, "_run_windows_command", - staticmethod(lambda args, timeout, env: ("partial out", "partial err", 0, True)), + staticmethod(lambda args, timeout, env, **kwargs: ("partial out", "partial err", 0, True)), ) output = LocalSandbox("t").execute_command("wait", timeout=1.5) diff --git a/backend/tests/test_local_sandbox_encoding.py b/backend/tests/test_local_sandbox_encoding.py index 079b372043..d5a520b988 100644 --- a/backend/tests/test_local_sandbox_encoding.py +++ b/backend/tests/test_local_sandbox_encoding.py @@ -1,8 +1,10 @@ import builtins +import ctypes import os import shutil import subprocess import sys +from types import SimpleNamespace import pytest from support.shell import require_posix_sh @@ -645,17 +647,23 @@ def test_execute_command_does_not_set_msys_env_for_non_msys_posix_shell_on_windo assert "MSYS_NO_PATHCONV" not in calls[0][2] -def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch): - calls: list[tuple[list[str], float, dict[str, str]]] = [] +def _mock_cmd_code_pages(monkeypatch, console_cp, oem_cp): + kernel32 = SimpleNamespace(GetConsoleOutputCP=lambda: console_cp, GetOEMCP=lambda: oem_cp) + monkeypatch.setattr(ctypes, "WinDLL", lambda *args, **kwargs: kernel32, raising=False) - def fake_run(args, timeout, env): - calls.append((args, timeout, env)) + +def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch): + calls: list[tuple[list[str], float, dict[str, str], dict]] = [] + + def fake_run(args, timeout, env, **kwargs): + calls.append((args, timeout, env, kwargs)) return "ok", "", 0, False monkeypatch.setattr(local_sandbox.os, "name", "nt") monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows", "GITHUB_TOKEN": "should-not-leak"}) monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\cmd.exe")) monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run)) + _mock_cmd_code_pages(monkeypatch, 850, 437) output = LocalSandbox("t").execute_command("echo hello") @@ -667,5 +675,136 @@ def test_execute_command_uses_cmd_command_mode_on_windows(monkeypatch): [r"C:\Windows\System32\cmd.exe", "/c", "echo hello"], 600, {"PATH": r"C:\Windows"}, + {"encoding": "cp850"}, ) ] + + +def test_execute_command_cmd_branch_ignores_utf8_mode_for_console_encoding(monkeypatch): + """Select the console page independently of Python UTF-8 mode and ANSI.""" + monkeypatch.setattr(local_sandbox.locale, "getpreferredencoding", lambda _: "utf-8") + monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252") + _mock_cmd_code_pages(monkeypatch, 936, 437) + + kwargs: dict = {} + + def fake_run(args, timeout, env, **kw): + kwargs.update(kw) + return "ok", "", 0, False + + monkeypatch.setattr(local_sandbox.os, "name", "nt") + monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows"}) + monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\cmd.exe")) + monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run)) + + LocalSandbox("t").execute_command("echo hello") + + assert kwargs["encoding"] == "cp936" + + +@pytest.mark.parametrize( + "console_cp,oem_cp,pipe_encoding,probe", + [(437, 850, "cp437", "café"), (850, 437, "cp850", "café £"), (65001, 437, "utf-8", "你好 日本語"), (0, 850, "cp850", "café £"), (0, 0, "cp1252", "café")], + ids=["oem-437", "oem-850", "utf8-console", "no-console", "ansi-fallback"], +) +def test_execute_command_cmd_decodes_console_code_page(monkeypatch, console_cp, oem_cp, pipe_encoding, probe): + """Decode real stdout/stderr pipes with differing console and ANSI pages.""" + _mock_cmd_code_pages(monkeypatch, console_cp, oem_cp) + monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252") + monkeypatch.setattr(local_sandbox.locale, "getpreferredencoding", lambda _: "utf-8") + if os.name != "nt": + monkeypatch.setattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0, raising=False) + run_windows_command = LocalSandbox._run_windows_command + payload = (probe + "\r\n").encode(pipe_encoding) + script = f"import os; p=bytes.fromhex('{payload.hex()}'); os.write(1,p); os.write(2,p); raise SystemExit(3)" + + def run_pipe_probe(args, timeout, env, **kwargs): + return run_windows_command([sys.executable, "-c", script], timeout, env, **kwargs) + + monkeypatch.setattr(local_sandbox.os, "name", "nt") + monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "cmd.exe")) + monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(run_pipe_probe)) + + output = LocalSandbox("t").execute_command("echo probe", timeout=10) + + assert output == f"{probe}\n\nStd Error:\n{probe}\n\nExit Code: 3" + + +def test_execute_command_cmd_rechecks_console_page_between_commands(monkeypatch): + pages = iter([437, 65001]) + kernel32 = SimpleNamespace(GetConsoleOutputCP=lambda: next(pages), GetOEMCP=lambda: 437) + monkeypatch.setattr(ctypes, "WinDLL", lambda *args, **kwargs: kernel32, raising=False) + monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252") + monkeypatch.setattr(local_sandbox.os, "name", "nt") + monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "cmd.exe")) + payloads = iter(["café".encode("cp437"), "你好".encode()]) + + def fake_run(args, timeout, env, **kwargs): + capture = _BoundedPipeCapture(**kwargs) + capture.append(next(payloads)) + return capture.read(), "", 0, False + + monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run)) + sandbox = LocalSandbox("t") + + assert sandbox.execute_command("echo café") == "café" + assert sandbox.execute_command("echo 你好") == "你好" + + +def test_execute_command_cmd_falls_back_when_windows_api_is_unavailable(monkeypatch): + def unavailable(*args, **kwargs): + raise OSError("kernel32 unavailable") + + monkeypatch.setattr(ctypes, "WinDLL", unavailable, raising=False) + monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp1252") + monkeypatch.setattr(local_sandbox.os, "name", "nt") + monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "cmd.exe")) + kwargs = {} + + def fake_run(args, timeout, env, **kw): + kwargs.update(kw) + return "ok", "", 0, False + + monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run)) + + assert LocalSandbox("t").execute_command("echo hello") == "ok" + assert kwargs["encoding"] == "cp1252" + + +@pytest.mark.skipif(os.name != "nt", reason="Requires real Windows cmd.exe") +@pytest.mark.parametrize("code_page", [437, 850, 936, 65001]) +def test_execute_command_cmd_branch_roundtrip_with_console_code_page(code_page): + """Use a private console so changing its page cannot affect sibling tests.""" + probe = r""" +import ctypes +import os +import sys +import deerflow.sandbox.local.local_sandbox as local_sandbox +from deerflow.sandbox.local.local_sandbox import LocalSandbox + +code_page = int(sys.argv[1]) +assert ctypes.windll.kernel32.SetConsoleOutputCP(code_page) +# Make ANSI and console encoding differ even on a zh-CN or UTF-8 runner. +local_sandbox.locale.getencoding = lambda: "cp1252" +shell = os.path.join(os.environ.get("SystemRoot", r"C:\Windows"), "System32", "cmd.exe") +LocalSandbox._get_shell = staticmethod(lambda: shell) +expected = "caf\u00e9" if code_page in (437, 850) else "\u4f60\u597d \u65e5\u672c\u8a9e" +output = LocalSandbox("cmd-encoding-probe").execute_command(f"echo {expected}", timeout=15) +assert output == expected + "\n", ascii(output) +""" + startupinfo = subprocess.STARTUPINFO() + startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW + startupinfo.wShowWindow = subprocess.SW_HIDE + result = subprocess.run( + [sys.executable, "-c", probe, str(code_page)], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + env={**os.environ, "PYTHONUTF8": "1"}, + creationflags=subprocess.CREATE_NEW_CONSOLE, + startupinfo=startupinfo, + timeout=30, + check=False, + ) + assert result.returncode == 0, result.stdout + result.stderr