mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 05:30:39 +00:00
fix(sandbox): decode PowerShell native-child output with the locale fallback (#6727)
* fix(sandbox): decode PowerShell native-child output with the locale fallback Fixes #6646 ## What The Windows PowerShell branch of `LocalSandbox.execute_command` passed `encoding="utf-8"` to `_run_windows_command` with no `fallback_encoding`. The guarded console-UTF-8 preamble covers PowerShell's *own* output, but native children it runs (`python.exe`, CLI shims) inherit the pipe and keep writing the host code page, so the pipe decoder's `errors="replace"` turned every such byte into U+FFFD. This passes `fallback_encoding=locale.getpreferredencoding(False)` there, mirroring what the MSYS branch above already applies (#6548). ## Why On Windows hosts whose ANSI code page is not UTF-8 (GBK on zh-CN hosts, cp932, cp949, cp1251, ...), non-ASCII tool results printed by native children reach the model garbled (#6646). PowerShell's own output stays correct because the preamble pins it to UTF-8, so the fallback only engages when a buffer is not valid UTF-8 — exactly the case where a native child wrote host-code-page bytes. Pure-ASCII and PowerShell-emitted UTF-8 buffers never touch the fallback, matching the tradeoff the MSYS branch already accepts. ## How checked Reproduced on a real zh-CN Windows host (ANSI code page 936): - Red on `main` (3f7a0769): a native `python.exe` child printing CJK text through `powershell.exe` / `pwsh.exe` returned `'\ufffd\ufffd\ufffd \ufffd\u0571\ufffd\ufffdZ\n'` — the new regression test failed on both shells: ``` FAILED tests/test_local_sandbox_encoding.py::test_windows_powershell_native_child_cjk_roundtrip[powershell.exe] FAILED tests/test_local_sandbox_encoding.py::test_windows_powershell_native_child_cjk_roundtrip[pwsh.exe] ``` - Green with the fix: both pass, and the child's text round-trips exactly (`'\u4f60\u597d \u65e5\u672c\u8a9e\n'`). - `tests/test_local_sandbox_encoding.py`: 34 passed, 1 skipped (Windows 11, CPython 3.13, `uv sync --frozen` environment). The two pre-existing PowerShell unit tests were extended so their `_run_windows_command` fakes accept the new `fallback_encoding` kwarg and assert it is the locale code page. - `ruff check` and `ruff format --check` pass on both changed files. * fix(sandbox): decode fallback must survive Python UTF-8 mode and mid-character captures Follow-up to review feedback on #6727. - Use ``locale.getencoding()`` (3.11+, repo floor is 3.12) instead of ``locale.getpreferredencoding(False)`` for both fallback call sites (MSYS and the native-child guard): ``getpreferredencoding`` reports ``"utf-8"`` whenever the Gateway runs under Python UTF-8 mode (``PYTHONUTF8=1``, ``-X utf8``), silently disabling the fallback and leaving #6646 unfixed on those hosts. ``getencoding()`` deliberately ignores UTF-8 mode and returns the code page the child's CRT used. - After a failed strict UTF-8 decode, retry once without a trailing incomplete multi-byte sequence, so a capture cut mid-character (the 10MB capture-cap boundary, or a timeout kill interrupting the child mid-write) cannot cascade the whole buffer to the locale fallback. Complete non-UTF-8 buffers still fail the retried attempt and reach the fallback untouched. - Tests: the Windows roundtrip probes now run under ``PYTHONUTF8=1`` with ``locale.getencoding`` pinned, so a green run proves the fallback source ignores UTF-8 mode; added POSIX-runnable unit tests for the cap-split and timeout-kill partial tails. Test results (Windows host, ACP=936, repo venv via ``uv sync --frozen``): ``tests/test_local_sandbox_encoding.py`` 36 passed, 1 skipped, both with ``PYTHONUTF8=0`` and ``PYTHONUTF8=1``; ``ruff check`` and ``ruff format --check`` clean on both files. * fix(sandbox): trim only on cut captures, preserving complete locale tails Willem-bd's follow-up review pinned a real P2: on a cp1252 host the MSYS branch receives b'caf\xe9' (a complete e-acute), the trailing E9 is trimmed as an incomplete UTF-8 lead, and the ASCII prefix is accepted as valid UTF-8 - so the locale fallback never runs and the byte is silently dropped. No capture cap or timeout is needed to trigger it. Fix: the strict-retry trim is now authorized only when the capture may actually have been cut mid-character - a byte-cap cut (signalled by the capture's own truncated state) or a timeout kill (new mark_split_tail(), called from _run_windows_command right after the process tree is killed). A complete buffer, even one ending in a byte shaped like a bare UTF-8 lead, now goes to the fallback with every byte intact. Tests: - new: cp1252 complete-tail roundtrip (red on b62ef75: returned 'caf'). - new: the same partial-shaped tail without a cut signal must fall back intact instead of being accepted as a truncated UTF-8 prefix. - new: POSIX-runnable _run_windows_command timeout integration test - the child writes a partial UTF-8 sequence, hangs, is killed at the 1s timeout; the trimmed prefix survives (this goes red if the kill path forgets to authorize the trim, since the whole buffer cascades to the fallback). - updated: the timeout-kill unit test now marks the capture explicitly, mirroring the real call site. Evidence: Windows 11, Python 3.13.12 (repo venv), pytest tests/test_local_sandbox_encoding.py -> 39 passed, 1 skipped under both PYTHONUTF8=0 and PYTHONUTF8=1; ruff check + ruff format --check clean. Red run against b62ef75: the three new/updated tests fail exactly as described above.
This commit is contained in:
sjr666666
GitHub
parent
62588f3675
commit
e81fdec459
@ -51,23 +51,53 @@ class _BoundedPipeCapture:
|
||||
self._chunks: list[bytes] = []
|
||||
self._kept_bytes = 0
|
||||
self._total_bytes = 0
|
||||
# Set when the child may have been killed mid-write (timeout path): the
|
||||
# capture may end in a partial multi-byte sequence that the strict
|
||||
# retry is allowed to trim. A complete buffer — no byte-cap cut, no
|
||||
# kill — never trims, so a valid locale-encoded tail (e.g. cp1252
|
||||
# ``b'caf\xe9'``) reaches the fallback with every byte intact.
|
||||
self._split_tail = False
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _decode(self, data: bytes) -> str:
|
||||
def mark_split_tail(self) -> None:
|
||||
"""Authorize the trim retry because the child may have been killed
|
||||
mid-write, leaving a partial multi-byte sequence at the buffer end."""
|
||||
with self._lock:
|
||||
self._split_tail = True
|
||||
|
||||
def _decode(self, data: bytes, *, allow_trim: bool) -> str:
|
||||
"""Decode captured bytes, preferring the primary encoding when valid.
|
||||
|
||||
A forced ``encoding`` describes the shell's *own* output. Windows-native
|
||||
children (``python.exe``, CLI shims) inherit the pipe and keep writing the
|
||||
host code page, which is not valid in the forced encoding. Rather than
|
||||
replacing every such byte with U+FFFD, retry the whole buffer with
|
||||
``fallback_encoding`` (the locale) when the primary decode fails.
|
||||
``fallback_encoding`` (the locale) when the primary decode fails. Only a
|
||||
capture that may have been cut mid-character (a byte-cap cut or a
|
||||
timeout kill, signalled via ``allow_trim``) retries the strict attempt
|
||||
without the trailing partial sequence; a complete buffer — even one
|
||||
that merely *ends* in a byte shaped like a bare UTF-8 lead — goes to
|
||||
the fallback untouched so no locale-encoded tail byte is dropped.
|
||||
"""
|
||||
if self._fallback_encoding is None:
|
||||
return data.decode(self._encoding, errors="replace")
|
||||
try:
|
||||
return data.decode(self._encoding)
|
||||
except UnicodeDecodeError:
|
||||
return data.decode(self._fallback_encoding, errors="replace")
|
||||
pass
|
||||
if allow_trim and self._encoding.replace("_", "-").lower() == "utf-8":
|
||||
# The capture may have been cut mid-character (capture-cap
|
||||
# boundary or a timeout kill). Retry the strict attempt without
|
||||
# that tail so one split character cannot cascade the whole
|
||||
# buffer to the fallback; a tail that survives the trim still
|
||||
# fails the retried attempt and reaches the fallback untouched.
|
||||
trimmed = _trim_trailing_partial_utf8(data)
|
||||
if len(trimmed) != len(data):
|
||||
try:
|
||||
return trimmed.decode(self._encoding)
|
||||
except UnicodeDecodeError:
|
||||
pass
|
||||
return data.decode(self._fallback_encoding, errors="replace")
|
||||
|
||||
def append(self, chunk: bytes) -> None:
|
||||
with self._lock:
|
||||
@ -83,10 +113,14 @@ class _BoundedPipeCapture:
|
||||
with self._lock:
|
||||
data = b"".join(self._chunks)
|
||||
truncated = self._total_bytes > self._kept_bytes
|
||||
split_tail = self._split_tail
|
||||
total_bytes = self._total_bytes
|
||||
kept_bytes = self._kept_bytes
|
||||
|
||||
output = self._decode(data)
|
||||
# Trimming is only justified on a capture that may have been cut
|
||||
# mid-character — a byte-cap cut or a timeout kill. A complete buffer
|
||||
# goes to the fallback with every byte intact.
|
||||
output = self._decode(data, allow_trim=truncated or split_tail)
|
||||
if self._normalize_newlines:
|
||||
# Match ``subprocess.run(..., text=True)``: text streams use universal
|
||||
# newlines, translating both CRLF and bare CR to LF.
|
||||
@ -97,6 +131,31 @@ class _BoundedPipeCapture:
|
||||
return output
|
||||
|
||||
|
||||
def _trim_trailing_partial_utf8(data: bytes) -> bytes:
|
||||
"""Strip an incomplete multi-byte UTF-8 sequence from the end of ``data``.
|
||||
|
||||
A bounded capture can stop mid-character: the byte cap may cut between a
|
||||
lead byte and its continuations, and a timeout kill can interrupt the
|
||||
child mid-write. Only invoked when the caller has signalled such a cut
|
||||
(``allow_trim``) — a complete buffer is never trimmed, so a locale-encoded
|
||||
tail that merely *looks* like a bare UTF-8 lead (for example the ``E9``
|
||||
byte of a cp1252 ``café``) reaches the locale fallback with every byte
|
||||
intact instead of being dropped as an incomplete sequence.
|
||||
"""
|
||||
for offset in range(min(len(data), 4)):
|
||||
byte = data[len(data) - 1 - offset]
|
||||
if byte < 0x80 or 0xC0 <= byte <= 0xC1 or byte > 0xF4:
|
||||
# ASCII or an invalid byte: whatever precedes it is either complete
|
||||
# or unrecoverable either way; stop without trimming.
|
||||
break
|
||||
if byte >= 0xC2: # lead byte found
|
||||
expected = 2 if byte < 0xE0 else (3 if byte < 0xF0 else 4)
|
||||
if offset + 1 < expected:
|
||||
return data[: len(data) - 1 - offset]
|
||||
break
|
||||
return data
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PathMapping:
|
||||
"""A path mapping from a container path to a local path with optional read-only flag."""
|
||||
@ -592,14 +651,34 @@ class LocalSandbox(Sandbox):
|
||||
timeout,
|
||||
sandbox_env,
|
||||
encoding="utf-8",
|
||||
fallback_encoding=locale.getpreferredencoding(False),
|
||||
# ``locale.getencoding()`` (3.11+, repo floor is 3.12)
|
||||
# deliberately ignores Python UTF-8 mode and returns the code
|
||||
# page the child's CRT actually used — ``getpreferredencoding``
|
||||
# would report ``"utf-8"`` on hosts running under
|
||||
# ``PYTHONUTF8=1``, silently making this fallback a no-op.
|
||||
fallback_encoding=locale.getencoding(),
|
||||
)
|
||||
elif self._is_powershell(shell):
|
||||
# PowerShell is pinned to UTF-8 by the preamble above, so its own
|
||||
# output must be decoded as UTF-8: any other code page (GBK on
|
||||
# zh-CN) mangles non-ASCII output because the pipe decoder
|
||||
# replaces instead of raising.
|
||||
stdout, stderr, returncode, timed_out = self._run_windows_command(args, timeout, sandbox_env, encoding="utf-8")
|
||||
# replaces instead of raising. Native children PowerShell runs
|
||||
# (python.exe, CLI shims) inherit the pipe and keep writing the
|
||||
# host code page, so a buffer that is not valid UTF-8 falls back
|
||||
# to the locale instead of being replaced wholesale — the same
|
||||
# guard the MSYS branch above already applies.
|
||||
stdout, stderr, returncode, timed_out = self._run_windows_command(
|
||||
args,
|
||||
timeout,
|
||||
sandbox_env,
|
||||
encoding="utf-8",
|
||||
# ``locale.getencoding()`` (3.11+, repo floor is 3.12)
|
||||
# deliberately ignores Python UTF-8 mode and returns the code
|
||||
# page the child's CRT actually used — ``getpreferredencoding``
|
||||
# would report ``"utf-8"`` on hosts running under
|
||||
# ``PYTHONUTF8=1``, silently making this fallback a no-op.
|
||||
fallback_encoding=locale.getencoding(),
|
||||
)
|
||||
else:
|
||||
stdout, stderr, returncode, timed_out = self._run_windows_command(args, timeout, sandbox_env)
|
||||
else:
|
||||
@ -685,6 +764,11 @@ class LocalSandbox(Sandbox):
|
||||
except subprocess.TimeoutExpired:
|
||||
timed_out = True
|
||||
LocalSandbox._terminate_windows_process_tree(process)
|
||||
# The kill can interrupt the child mid-write, leaving a partial
|
||||
# multi-byte sequence at the capture end; authorize the trim
|
||||
# retry so the valid prefix survives the strict decode.
|
||||
stdout_capture.mark_split_tail()
|
||||
stderr_capture.mark_split_tail()
|
||||
returncode = process.returncode if process.returncode is not None else 0
|
||||
finally:
|
||||
join_timeout = 10 if timed_out else _PIPE_DRAIN_JOIN_TIMEOUT_SECONDS
|
||||
|
||||
@ -49,6 +49,75 @@ def test_bounded_pipe_capture_without_fallback_replaces_non_utf8_bytes():
|
||||
assert "\u4f60\u597d" not in decoded
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_trims_capture_cap_split_multibyte_tail():
|
||||
"""The byte cap can cut a multi-byte UTF-8 char in half; strict-decoding the
|
||||
partial tail would fail the whole-buffer gate and cascade everything to the
|
||||
locale fallback, so the trailing incomplete sequence is trimmed first."""
|
||||
capture = _BoundedPipeCapture(limit_bytes=10, encoding="utf-8", fallback_encoding="cp936")
|
||||
capture.append(b"abcdefgh")
|
||||
# 2 bytes of the 6-byte "你好" fit under the cap, splitting 你 (e4 bd a0)
|
||||
# after its continuation byte — a partial sequence at the buffer end.
|
||||
capture.append("\u4f60\u597d".encode("utf-8"))
|
||||
|
||||
assert capture.read() == "abcdefgh\n... [output truncated after 10 of 14 bytes; remaining output discarded] ..."
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_preserves_complete_locale_tail_for_fallback():
|
||||
"""A complete buffer that ends in a byte shaped like an incomplete UTF-8
|
||||
lead must reach the locale fallback untouched: on a cp1252 host the child
|
||||
writes ``caf\xe9`` (a complete é), and trimming that byte would silently
|
||||
drop it while accepting the ASCII prefix as valid UTF-8."""
|
||||
capture = _BoundedPipeCapture(encoding="utf-8", fallback_encoding="cp1252")
|
||||
capture.append(b"caf\xe9")
|
||||
|
||||
assert capture.read() == "caf\u00e9"
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_trims_partial_tail_left_by_timeout_kill():
|
||||
"""A child killed mid-write can leave an incomplete multi-byte sequence at
|
||||
the end of the capture; marking the split tail authorizes the trim retry
|
||||
so the valid prefix stays decodable."""
|
||||
capture = _BoundedPipeCapture(encoding="utf-8", fallback_encoding="cp936")
|
||||
capture.append(b"abc")
|
||||
capture.append("\u4f60".encode("utf-8")[:-1])
|
||||
capture.mark_split_tail()
|
||||
|
||||
assert capture.read() == "abc"
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_without_cut_signal_falls_back_on_partial_shaped_tail():
|
||||
"""The same partial-shaped tail without a cut signal (no cap cut, no
|
||||
timeout kill) must NOT be trimmed away: the whole buffer goes to the
|
||||
locale fallback instead of being accepted as a truncated UTF-8 prefix.
|
||||
cp1252 is used as the fallback so every byte has a defined mapping."""
|
||||
capture = _BoundedPipeCapture(encoding="utf-8", fallback_encoding="cp1252")
|
||||
capture.append(b"abc")
|
||||
capture.append("\u4f60".encode("utf-8")[:-1])
|
||||
|
||||
assert capture.read() == "abcä½"
|
||||
|
||||
|
||||
def test_windows_command_timeout_marks_capture_for_trim(monkeypatch):
|
||||
"""A real timeout kill must authorize the trim retry through
|
||||
``_run_windows_command``: the child wrote a partial UTF-8 sequence and was
|
||||
killed mid-command, so the trimmed empty prefix survives instead of the
|
||||
whole buffer cascading to the locale fallback as U+FFFD."""
|
||||
if os.name != "nt":
|
||||
monkeypatch.setattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0, raising=False)
|
||||
partial = "\u4f60".encode("utf-8")[:-1]
|
||||
script = f"import os, time; os.write(1, bytes.fromhex('{partial.hex()}')); time.sleep(30)"
|
||||
|
||||
stdout, stderr, returncode, timed_out = LocalSandbox._run_windows_command(
|
||||
[sys.executable, "-c", script],
|
||||
1,
|
||||
encoding="utf-8",
|
||||
fallback_encoding="cp936",
|
||||
)
|
||||
|
||||
assert timed_out is True
|
||||
assert stdout == ""
|
||||
|
||||
|
||||
def test_bounded_pipe_capture_applies_text_mode_newline_normalization_when_enabled():
|
||||
capture = _BoundedPipeCapture(normalize_newlines=True)
|
||||
capture.append(b"crlf\r\nbare-cr\rlf\n")
|
||||
@ -96,14 +165,17 @@ import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
|
||||
# Keep this regression effective even on an English or UTF-8 Windows runner.
|
||||
local_sandbox.locale.getpreferredencoding = lambda _: "cp936"
|
||||
local_sandbox.locale.getencoding = lambda: "cp936"
|
||||
LocalSandbox._get_shell = staticmethod(lambda: sys.argv[1])
|
||||
expected = "\u4f60\u597d \u65e5\u672c\u8a9e"
|
||||
command = f"Write-Output '{expected}'; [Console]::Error.WriteLine('{expected}'); exit 3"
|
||||
output = LocalSandbox("encoding-probe").execute_command(command, timeout=15)
|
||||
assert output == expected + "\n\nStd Error:\n" + expected + "\n\nExit Code: 3", ascii(output)
|
||||
"""
|
||||
env = {**os.environ, "PYTHONUTF8": "0"}
|
||||
# Run the probe under Python UTF-8 mode on purpose: ``getpreferredencoding``
|
||||
# would report ``"utf-8"`` there and silently disable the locale fallback,
|
||||
# so a green run proves the fallback source is ``locale.getencoding()``.
|
||||
env = {**os.environ, "PYTHONUTF8": "1"}
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", probe, shell],
|
||||
capture_output=True,
|
||||
@ -130,13 +202,16 @@ import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
|
||||
# Keep this regression effective even on an English or UTF-8 Windows runner.
|
||||
local_sandbox.locale.getpreferredencoding = lambda _: "cp936"
|
||||
local_sandbox.locale.getencoding = lambda: "cp936"
|
||||
LocalSandbox._get_shell = staticmethod(lambda: sys.argv[1])
|
||||
expected = "\u4f60\u597d \u65e5\u672c\u8a9e"
|
||||
output = LocalSandbox("encoding-probe").execute_command(f"printf '%s\n' '{expected}'", timeout=15)
|
||||
assert output == expected + "\n", ascii(output)
|
||||
"""
|
||||
env = {**os.environ, "PYTHONUTF8": "0"}
|
||||
# Run the probe under Python UTF-8 mode on purpose: ``getpreferredencoding``
|
||||
# would report ``"utf-8"`` there and silently disable the locale fallback,
|
||||
# so a green run proves the fallback source is ``locale.getencoding()``.
|
||||
env = {**os.environ, "PYTHONUTF8": "1"}
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", probe, shell],
|
||||
capture_output=True,
|
||||
@ -164,7 +239,7 @@ import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
|
||||
# Pin the fallback code page so the regression holds on non-zh-CN runners.
|
||||
local_sandbox.locale.getpreferredencoding = lambda _: "cp936"
|
||||
local_sandbox.locale.getencoding = lambda: "cp936"
|
||||
LocalSandbox._get_shell = staticmethod(lambda: sys.argv[1])
|
||||
expected = "\u4f60\u597d \u65e5\u672c\u8a9e"
|
||||
interpreter = sys.executable.replace("\\", "/")
|
||||
@ -174,7 +249,53 @@ command = f"'{interpreter}' -c \"print('{expected}')\""
|
||||
output = LocalSandbox("encoding-probe").execute_command(command, timeout=30, env={"PYTHONIOENCODING": "cp936"})
|
||||
assert output == expected + "\n", ascii(output)
|
||||
"""
|
||||
env = {**os.environ, "PYTHONUTF8": "0"}
|
||||
# Run the probe under Python UTF-8 mode on purpose: ``getpreferredencoding``
|
||||
# would report ``"utf-8"`` there and silently disable the locale fallback,
|
||||
# so a green run proves the fallback source is ``locale.getencoding()``.
|
||||
env = {**os.environ, "PYTHONUTF8": "1"}
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", probe, shell],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
env=env,
|
||||
timeout=60,
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Requires real Windows PowerShell")
|
||||
@pytest.mark.parametrize("shell_name", ["powershell.exe", "pwsh.exe"])
|
||||
def test_windows_powershell_native_child_cjk_roundtrip(shell_name):
|
||||
"""A Windows-native child PowerShell runs (python.exe) inherits the pipe and
|
||||
keeps writing the host code page; the guarded console-UTF-8 preamble only
|
||||
covers PowerShell's own output, so the forced UTF-8 decode must fall back to
|
||||
the locale for those bytes instead of replacing the text with U+FFFD."""
|
||||
shell = shutil.which(shell_name)
|
||||
if shell is None:
|
||||
pytest.skip(f"{shell_name} is not installed")
|
||||
probe = r"""
|
||||
import sys
|
||||
import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
|
||||
# Pin the fallback code page so the regression holds on non-zh-CN runners.
|
||||
local_sandbox.locale.getencoding = lambda: "cp936"
|
||||
LocalSandbox._get_shell = staticmethod(lambda: sys.argv[1])
|
||||
expected = "\u4f60\u597d \u65e5\u672c\u8a9e"
|
||||
interpreter = sys.executable.replace("\\", "/")
|
||||
command = f"& '{interpreter}' -c \"print('{expected}')\""
|
||||
# PYTHONIOENCODING makes the native child emit CP936 even on a UTF-8 host,
|
||||
# mirroring a zh-CN host's native Python writing to a redirected pipe.
|
||||
output = LocalSandbox("encoding-probe").execute_command(command, timeout=30, env={"PYTHONIOENCODING": "cp936"})
|
||||
assert output == expected + "\n", ascii(output)
|
||||
"""
|
||||
# Run the probe under Python UTF-8 mode on purpose: ``getpreferredencoding``
|
||||
# would report ``"utf-8"`` there and silently disable the locale fallback,
|
||||
# so a green run proves the fallback source is ``locale.getencoding()``.
|
||||
env = {**os.environ, "PYTHONUTF8": "1"}
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", probe, shell],
|
||||
capture_output=True,
|
||||
@ -333,14 +454,15 @@ def test_get_shell_uses_cmd_as_last_windows_fallback(monkeypatch):
|
||||
|
||||
|
||||
def test_execute_command_uses_powershell_command_mode_on_windows(monkeypatch):
|
||||
calls: list[tuple[list[str], float, dict[str, str], str | None]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str], str | None, str | None]] = []
|
||||
|
||||
def fake_run(args, timeout, env, *, encoding=None):
|
||||
calls.append((args, timeout, env, encoding))
|
||||
def fake_run(args, timeout, env, *, encoding=None, fallback_encoding=None):
|
||||
calls.append((args, timeout, env, encoding, fallback_encoding))
|
||||
return "ok", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows", "OPENAI_API_KEY": "should-not-leak"})
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp936")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
@ -361,6 +483,7 @@ def test_execute_command_uses_powershell_command_mode_on_windows(monkeypatch):
|
||||
600,
|
||||
{"PATH": r"C:\Windows"},
|
||||
"utf-8",
|
||||
"cp936",
|
||||
)
|
||||
]
|
||||
|
||||
@ -369,14 +492,15 @@ def test_execute_command_forces_utf8_console_for_powershell_cjk_output(monkeypat
|
||||
"""PowerShell 5.1 defaults console output to the OEM codepage (GBK on
|
||||
zh-CN); without the UTF-8 preamble, CJK output is garbled by the UTF-8
|
||||
pipe reader even though decoding never raises (errors=replace)."""
|
||||
calls: list[tuple[list[str], float, dict[str, str], str | None]] = []
|
||||
calls: list[tuple[list[str], float, dict[str, str], str | None, str | None]] = []
|
||||
|
||||
def fake_run(args, timeout, env, *, encoding=None):
|
||||
calls.append((args, timeout, env, encoding))
|
||||
def fake_run(args, timeout, env, *, encoding=None, fallback_encoding=None):
|
||||
calls.append((args, timeout, env, encoding, fallback_encoding))
|
||||
return "你好", "", 0, False
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Windows"})
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp936")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: "pwsh"))
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
@ -387,6 +511,7 @@ def test_execute_command_forces_utf8_console_for_powershell_cjk_output(monkeypat
|
||||
assert cmd.startswith("try{[Console]::InputEncoding=[System.Text.Encoding]::UTF8}catch{};try{[Console]::OutputEncoding=[System.Text.Encoding]::UTF8}catch{};$OutputEncoding=[System.Text.Encoding]::UTF8;")
|
||||
assert cmd.endswith("Write-Output 你好")
|
||||
assert calls[0][3] == "utf-8"
|
||||
assert calls[0][4] == "cp936"
|
||||
|
||||
|
||||
def test_execute_command_forces_utf8_decoding_with_locale_fallback_for_msys(monkeypatch):
|
||||
@ -402,7 +527,7 @@ def test_execute_command_forces_utf8_decoding_with_locale_fallback_for_msys(monk
|
||||
|
||||
monkeypatch.setattr(local_sandbox.os, "name", "nt")
|
||||
monkeypatch.setattr(local_sandbox.os, "environ", {"PATH": r"C:\Program Files\Git\bin"})
|
||||
monkeypatch.setattr(local_sandbox.locale, "getpreferredencoding", lambda _: "cp936")
|
||||
monkeypatch.setattr(local_sandbox.locale, "getencoding", lambda: "cp936")
|
||||
monkeypatch.setattr(LocalSandbox, "_get_shell", staticmethod(lambda: r"C:\Program Files\Git\bin\sh.exe"))
|
||||
monkeypatch.setattr(LocalSandbox, "_run_windows_command", staticmethod(fake_run))
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user