mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 13:40:37 +00:00
e81fdec459
* fix(sandbox): decode PowerShell native-child output with the locale fallback Fixes #6646 ## What The Windows PowerShell branch of `LocalSandbox.execute_command` passed `encoding="utf-8"` to `_run_windows_command` with no `fallback_encoding`. The guarded console-UTF-8 preamble covers PowerShell's *own* output, but native children it runs (`python.exe`, CLI shims) inherit the pipe and keep writing the host code page, so the pipe decoder's `errors="replace"` turned every such byte into U+FFFD. This passes `fallback_encoding=locale.getpreferredencoding(False)` there, mirroring what the MSYS branch above already applies (#6548). ## Why On Windows hosts whose ANSI code page is not UTF-8 (GBK on zh-CN hosts, cp932, cp949, cp1251, ...), non-ASCII tool results printed by native children reach the model garbled (#6646). PowerShell's own output stays correct because the preamble pins it to UTF-8, so the fallback only engages when a buffer is not valid UTF-8 — exactly the case where a native child wrote host-code-page bytes. Pure-ASCII and PowerShell-emitted UTF-8 buffers never touch the fallback, matching the tradeoff the MSYS branch already accepts. ## How checked Reproduced on a real zh-CN Windows host (ANSI code page 936): - Red on `main` (3f7a0769): a native `python.exe` child printing CJK text through `powershell.exe` / `pwsh.exe` returned `'\ufffd\ufffd\ufffd \ufffd\u0571\ufffd\ufffdZ\n'` — the new regression test failed on both shells: ``` FAILED tests/test_local_sandbox_encoding.py::test_windows_powershell_native_child_cjk_roundtrip[powershell.exe] FAILED tests/test_local_sandbox_encoding.py::test_windows_powershell_native_child_cjk_roundtrip[pwsh.exe] ``` - Green with the fix: both pass, and the child's text round-trips exactly (`'\u4f60\u597d \u65e5\u672c\u8a9e\n'`). - `tests/test_local_sandbox_encoding.py`: 34 passed, 1 skipped (Windows 11, CPython 3.13, `uv sync --frozen` environment). The two pre-existing PowerShell unit tests were extended so their `_run_windows_command` fakes accept the new `fallback_encoding` kwarg and assert it is the locale code page. - `ruff check` and `ruff format --check` pass on both changed files. * fix(sandbox): decode fallback must survive Python UTF-8 mode and mid-character captures Follow-up to review feedback on #6727. - Use ``locale.getencoding()`` (3.11+, repo floor is 3.12) instead of ``locale.getpreferredencoding(False)`` for both fallback call sites (MSYS and the native-child guard): ``getpreferredencoding`` reports ``"utf-8"`` whenever the Gateway runs under Python UTF-8 mode (``PYTHONUTF8=1``, ``-X utf8``), silently disabling the fallback and leaving #6646 unfixed on those hosts. ``getencoding()`` deliberately ignores UTF-8 mode and returns the code page the child's CRT used. - After a failed strict UTF-8 decode, retry once without a trailing incomplete multi-byte sequence, so a capture cut mid-character (the 10MB capture-cap boundary, or a timeout kill interrupting the child mid-write) cannot cascade the whole buffer to the locale fallback. Complete non-UTF-8 buffers still fail the retried attempt and reach the fallback untouched. - Tests: the Windows roundtrip probes now run under ``PYTHONUTF8=1`` with ``locale.getencoding`` pinned, so a green run proves the fallback source ignores UTF-8 mode; added POSIX-runnable unit tests for the cap-split and timeout-kill partial tails. Test results (Windows host, ACP=936, repo venv via ``uv sync --frozen``): ``tests/test_local_sandbox_encoding.py`` 36 passed, 1 skipped, both with ``PYTHONUTF8=0`` and ``PYTHONUTF8=1``; ``ruff check`` and ``ruff format --check`` clean on both files. * fix(sandbox): trim only on cut captures, preserving complete locale tails Willem-bd's follow-up review pinned a real P2: on a cp1252 host the MSYS branch receives b'caf\xe9' (a complete e-acute), the trailing E9 is trimmed as an incomplete UTF-8 lead, and the ASCII prefix is accepted as valid UTF-8 - so the locale fallback never runs and the byte is silently dropped. No capture cap or timeout is needed to trigger it. Fix: the strict-retry trim is now authorized only when the capture may actually have been cut mid-character - a byte-cap cut (signalled by the capture's own truncated state) or a timeout kill (new mark_split_tail(), called from _run_windows_command right after the process tree is killed). A complete buffer, even one ending in a byte shaped like a bare UTF-8 lead, now goes to the fallback with every byte intact. Tests: - new: cp1252 complete-tail roundtrip (red on b62ef75: returned 'caf'). - new: the same partial-shaped tail without a cut signal must fall back intact instead of being accepted as a truncated UTF-8 prefix. - new: POSIX-runnable _run_windows_command timeout integration test - the child writes a partial UTF-8 sequence, hangs, is killed at the 1s timeout; the trimmed prefix survives (this goes red if the kill path forgets to authorize the trim, since the whole buffer cascades to the fallback). - updated: the timeout-kill unit test now marks the capture explicitly, mirroring the real call site. Evidence: Windows 11, Python 3.13.12 (repo venv), pytest tests/test_local_sandbox_encoding.py -> 39 passed, 1 skipped under both PYTHONUTF8=0 and PYTHONUTF8=1; ruff check + ruff format --check clean. Red run against b62ef75: the three new/updated tests fail exactly as described above.