* fix(sandbox): end the host-path tail at a comma like the other list separators
The reverse-masking tail stops at whitespace and at " ; & | < > ( ) :",
but not at ",". A ,-joined list of host paths in command output therefore
lets the first match's tail swallow every later entry, and those paths
reach the model unmasked -- the same failure mode fixed for :-joined
lists ($PATH, $PYTHONPATH) earlier. The forward patterns have the mirror
defect: a ,-joined list of virtual paths resolves only the first entry.
"," also joins paths in real output (pytest/mypy/sed-style lists), and a
"," inside one path only shortens the match, so no path can be corrupted
by terminating the tail there.
* docs(sandbox): extend the tail-terminator rationale to the comma
Review nits: the _PATH_TAIL rationale comment now explains why `,` ends
the tail alongside `:` (a comma-joined argument listing would otherwise
hide later entries inside the first match's tail), and the snapshot
test's prose no longer calls the `:` stop the only tail change -- the
docstring and the literal's comment both name the `,` delta.
---------
Co-authored-by: laoyyu <laoyyu@users.noreply.github.com>
* fix(sandbox): match local grep lines without their terminator
find_grep_matches ran the regex on each line with its trailing "\n", so
patterns that can consume the newline (\s$, [^a-z]$, \n) matched every
newline-ended line while the returned line was shown stripped. Strip the
LF or CRLF terminator before matching, as grep(1) and the remote sandbox
providers do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(sandbox): clarify local grep CRLF handling
* test(sandbox): cover literal grep line terminators
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs: turn README into an index and move reference detail to docs/guide
The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.
* docs: send user docs to topic guides and budget the root README
The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).
* chore: remove unreferenced screenshots and finished plan notes
pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.
* docs: route always-loaded AGENTS.md files to scoped guides
* docs(agents): cut scoped guides to scope, verify, traps, decisions
Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.
Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).
* style: format frontend guide table and guidance test imports
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(agent): ground runtime decisions in structured evidence
Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.
Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.
* test(sandbox): expect read_file initialization error callback
* fix(agent): address evidence review follow-ups
* fix(agent): preserve retry budgets for OpenAI transport errors
* fix(agent): expose tool execution status in model requests
Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): exempt expression division after closers from the host-bash guard
A single-segment absolute-path match directly after ')', ']' or '}' is
division in an inline expression (python3 -c, perl, node -e, $((...)) ),
not a host path; #7023 exempted this shape only inside single-quoted awk
programs. Per-closer rules keep unknown-value path builds rejected:
digits only after ')' and '}' ($1/1024 rationale), an identifier after
')' only when the paren closes a name(...) call, any segment after ']'.
Literal tokens ('ls /2') are unchanged.
Fixes#7444
* fix(sandbox): scope division exemptions to parsed expressions
* docs(sandbox): fit inherited guidance within the hard limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): defer fresh AIO skill-policy acquisition
Prepare the authorized thread skill view before the model runs without
acquiring an AIO container for a fresh lazy run. Keep existing sandbox
bindings and upload-based providers on the eager rebind/sync path.
Add sync/async regressions for first-tool authorization, strict rebinding,
filtered read-only AIO mounts, and policy-scoped warm reuse.
AI assistance: Codex; the account owner confirmed human review of the
complete seven-file patch and responsibility for the change.
* docs(sandbox): keep lazy-isolation guidance within inherited budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): require a path start before translating virtual and host roots
Path translation only checked the right-hand segment boundary (#4035,
#4053), so a root embedded in a longer path or URL was rewritten too.
DeerFlow's own artifact URLs contain /mnt/user-data, so
curl .../artifacts/mnt/user-data/outputs/x.html had the thread's host
directory spliced into the URL, and a mount at /data turned an
unrelated /var/data/app.log into /var/mnt/data/app.log in model-visible
output.
Add a shared left-hand PATH_START_BOUNDARY: a root must not follow a
word character, '.', '~' or '-'. '/' is still allowed so
file:///mnt/user-data keeps mapping. It applies to the bash command
rewrite, LocalSandbox command/content resolution, and both reverse
masking paths.
* refactor(sandbox): derive the path-start lookbehind from one class
Build PATH_START_BOUNDARY from _PATH_START_BLOCKERS so the regex and
scanner paths cannot drift, and document the attached short-option
value (-o/data/x) as a known false negative of the start rule.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): leave SIGTERM/SIGINT teardown to the Gateway lifespan
AIO and E2B providers built on the event-loop thread installed SIGTERM/SIGINT
handlers that force-destroyed every sandbox synchronously, before uvicorn
started the run drain. The Gateway lifespan now holds sandbox shutdown
ownership from its first step, so providers skip those handlers and the
existing hook closes the provider after run drain and MCP teardown. SIGHUP
keeps the provider fallback. AIO acquire now refuses once shutdown began.
* docs(sandbox): document Gateway-owned sandbox shutdown and AIO admission fence
* test(gateway): inspect the lifespan body now that lifespan wraps it
* docs(changelog): reference #7560 for the sandbox SIGTERM drain-order fix
* ci: retrigger sandbox image smoke after a registry pull timeout
* fix(sandbox): derive the AIO sandbox ID off the event loop in async acquire
_acquire_internal_async offloads every acquire layer except the sandbox ID
derivation, which ran inline. _sandbox_id_for_thread checks the thread's
skills_view directory with exists(), and resolving it goes through
Paths.base_dir, so each async thread acquire made filesystem syscalls on the
event loop. The strict blocking-I/O gate raises on it.
Run the derivation through the same owned-worker helper as the reuse and
warm-pool layers around it, and anchor it in the blocking-I/O suite for both
the default and the policy-scoped ID.
* docs(changelog): reference #7507 for the AIO async acquire sandbox ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): reject write_file payloads that are the elision placeholder
The tool-output budget middleware replaces a superseded write_file payload
with a deterministic placeholder in the model-bound request only. That
placeholder is text the model sees in its own history, so it can imitate it
and submit it back as real write_file content: #7319 reports a delivered
file that was 194 bytes of placeholder.
write_file now rejects a payload that is the placeholder (whole payload,
allowing only surrounding whitespace) with an actionable error telling the
model to read_file the target path and write the real content; nothing is
written. The match is whole-payload on purpose: a substring guard would
block legitimate edits to this repository, whose constants.py and
persistence docs quote the template verbatim.
The placeholder wording moves to one shared constant,
WRITE_ELISION_PLACEHOLDER_TEMPLATE in deerflow.constants, which the
middleware renders from and the guard matches on, so the two cannot drift.
Regression tests fail on base (7 failed, 6 passed) and pass with the fix;
near-miss payloads that merely quote the placeholder still write.
Refs #7319
* fix(sandbox): match complete write elision placeholders
* docs: clarify write placeholder validation scope
* chore: re-trigger CLA status check
* fix(sandbox): preflight async writes and trim guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): keep document bytes when write_file translates paths
LocalSandbox.write_file rewrote container paths inside every file it stored,
so a dictated README.md ended up holding a host path while read_file masked
the rewrite for agent-written files: the read-back showed the virtual path,
and the agent confirmed content the file did not contain (#7461).
Translation now applies only to executable content, classified with the
shared is_code_file rule from deerflow.skills.package_files that the skill
installer, the export guard, and skill_manage_tool already use. Code suffixes
and extensionless files that open with a shebang keep the host-path rewrite
they need to run under a host interpreter (PR #1935); documents, data files,
and configs are stored verbatim and read back as stored.
* fix(sandbox): translate paths for interpreter scripts and stop reading pipes
The classifier added for #7461 asked only the skill-package rule, which
does not list `.awk` (nor `.sed`, `.lua`, `.r`, `.tcl`), so a written AWK
program kept container paths that the host awk cannot open when it runs
`awk -f`. Those suffixes are now the sandbox's own set: the shared rule
also decides what SkillScan and the installer treat as package code, and
widening it there is a different call.
Classification no longer touches the destination unless the path is
extensionless. A suffix answers the question by itself, so an append to
`events.py` no longer opens the file first, and the extensionless
shebang sniff goes through `os.open(O_NONBLOCK)` plus an `fstat` regular
file check, which is what keeps a FIFO append from blocking.
* test(sandbox): pin the non-blocking sniff with a suffixless pipe append
* fix(sandbox): close pipe write fds once when Popen fails
The command runners closed the pipe write descriptors twice on the
spawn-failure path: the except block released all four pipe fds, then
the finally block closed the two write fds again. In a busy gateway
another thread can claim a freed descriptor number between the two
closes, so the second os.close silently closed an unrelated descriptor
(a concurrent command's capture pipe or an HTTP socket). Guard the
parent's write-end close on a successful spawn.
* fix(sandbox): release pipe fds on BaseException and scope the Windows runner test
Review follow-up on #7394:
- Catch BaseException in both command runners' spawn-failure cleanup so a
KeyboardInterrupt mid-spawn still releases each pipe fd exactly once;
the guarded finally keeps the no-double-close guarantee.
- Skip the _run_windows_command failure-path param on non-Windows
runners: it reads subprocess.CREATE_NEW_PROCESS_GROUP when building
the call, which raises AttributeError on POSIX and would fail Linux CI.
- Add a KeyboardInterrupt regression test (fd-per-fd close accounting).
---------
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
* fix(sandbox): retain process group id for timeout cleanup
* test(sandbox): cover exited leader during group cleanup
* fix(sandbox): capture process group id before drain threads
* style(sandbox): separate timeout regression test
* test(sandbox): cover missing process-group fallback
* style(tests): format sandbox timeout regressions
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): search skills across category mounts
* fix(sandbox): address virtual search mount review
Search explicitly configured category mounts regardless of ignored names,
while retaining descendant filters and host-parent exclusion. Warn with the
virtual mount path before re-raising broken-mount errors, defer ordinary
path resolution, and cover lookahead across mount boundaries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): surface local directory search permission errors
Permission failures during local list_dir and glob walks must reach the
caller instead of looking like an empty successful result. Transient
filesystem races keep using the existing walk race allowlist.
Fixes#7210
* fix(sandbox): keep denied path in ls/glob permission errors
Guard chmod-based permission tests for non-root POSIX, reverse-resolve
PermissionError.filename in LocalSandbox.list_dir/glob like grep, and
surface that path from ls_tool/glob_tool instead of only the requested root.
* test(sandbox): use workspace/tmp_path leak guards for ls/glob denials
* fix(sandbox): exempt awk field arithmetic from the shell-variable path guard
Since #6930 every `$NAME/` in a host-bash command was treated as a
variable-prefixed host path unless both sides were digits, so legitimate
awk programs such as `awk '{print $NF/2}' data.txt` or
`awk '{print $sum/$n}' data.txt` on files under /mnt/user-data were
rejected with "Unsafe absolute paths".
Exempt a `$` only inside the single-quoted inline program of awk, gawk,
mawk or nawk (after `-F`, `-v` and `--`), where bash expands nothing and
awk reads `$NAME` as a field reference. awk string literals, option
values, input operands, double-quoted programs, text re-read by `sh -c`
or `eval`, heredocs, awk functions/aliases, unknown options such as
gawk's `--assign`, `-f` programs, and programs whose `"` may sit in a
regex or comment keep the strict scan.
Fixesbytedance/deer-flow#7021
* fix(sandbox): read awk string literals and comments before judging quotes
Review follow-up: the quote-reliability check gave up the awk exemption
for any `/` not preceded by an identifier, number or `]`, including a `/`
inside a plain string literal, so `awk '{n=split($0,p,"/"); print
$1/p[n]}'` was still rejected. Walk the program once with the same
string-literal tracker that computes the code spans: `/` and `#` inside
a literal are text, `#` outside one starts a comment to end of line, and
only a `/` outside a literal that cannot be division (regex literal)
forfeits the exemption.
* fix(sandbox): scan the awk exemption the way the shell parses the command
Review follow-up for the awk field-arithmetic exemption.
Bypasses closed: a backslash-newline let `aw\<nl>k() { eval "$1"; }`
define an awk function past the literal blocker regex, so any
continuation now forfeits the exemption; the word scanner emitted `)`,
`$(` and `>|` as command separators, so an `awk` that was only an
argument (`echo $(true) awk '...'`) looked like a command; `$'...'`,
comments and quotes nested in `"$(...)"` could desynchronise quote
pairing; the alias/function blocker only matched literal spellings and
ignored eval, source, hash, PATH, autoload and zsh's
`functions[awk]=` / `aliases[awk]=` arrays.
The scanner now follows bash: single and double quotes with escapes,
`#` comments at a word start, fd redirections, the usual separators,
and `$(...)` as a nested command context with fresh quoting. Anything
it does not model (backslash-newline, backticks, `$'...'`, heredocs, an
unmatched `)`, unbalanced quotes) returns no exemption. The command
word is found like `_validate_local_bash_shell_tokens` finds it:
after assignments, prefix keywords and wrappers, reduced to its
basename, inside every `$(...)`. A command that could redefine awk or
re-read quoted text gets no exemption at all, and the docstring says
the enumeration is best-effort.
False positives removed: `($2+$3)/2` and `length($0)/2` (a `/` after
`)` is division unless the `)` closes an if/while/for condition, and
the absolute-path scan is now exempt inside awk code too),
`/usr/bin/awk`, `LC_ALL=C awk`, `echo "$(awk ...)"`, `"<<"` inside an
awk string. The awk spans are computed lazily, only when a `$NAME/` or
absolute-path match needs them, and the division check scans backward
instead of re-slicing the program.
* fix(sandbox): exempt only single-segment absolute-path matches in awk code
Review follow-up. Inside awk code spans the absolute-path scan now skips
only a match with no second `/` (`/2`, `/n`, `/NR`), so a `/a/b` shape
is still rejected even if a span were ever misjudged; previously
`awk '{print ($1)/etc/passwd}'` and `awk 'BEGIN{x = 1 /etc/shadow}'`
were accepted.
`enable` joins the commands that withhold the exemption, since
`enable -f` can load a builtin under any name. The division check reads
the preceding `)` through the backward scan instead of re-slicing the
program for every `/`.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): search long lines for literal grep patterns
* test(sandbox): cover literal long lines across providers
* fix(sandbox): bound case-insensitive literal grep work
Replace repeated-prefix regex searches with KMP over single-character Unicode case matches. Keep local long-line literal results and existing regex and preview limits. Add a subprocess-deadline regression for the review counterexample and case-semantics coverage.
* docs(sandbox): keep grep guidance within inherited size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): let local file tools read the /mnt/user-data root
The local provider maps the aggregate /mnt/user-data root so listing it
matches AIO, but the tool layer only accepted paths below it: ls, glob
and grep on the root answered "Permission denied" (with a trailing
slash, from the resolved-path "path traversal detected" check).
validate_local_tool_path now accepts the bare root for reads, and the
user-data resolver accepts the shared parent of workspace/uploads/
outputs itself when the caller opts in. Only ls and the shared read
path opt in; writes to the root and files placed directly in it stay
rejected.
* docs(changelog): reference #6991 for the /mnt/user-data root fix
* fix(sandbox): list the /mnt/user-data root through its allowed dirs only
The first version let glob, grep and ls start at the user-data root
after validating only the root itself. The walkers then descended into
everything under the host user-data/ directory, so a file or directory
stored directly under it was listed and searched although read_file
rejects it.
ls now lists the root one allowed directory at a time (workspace,
uploads and outputs, each one level deep) through the sandbox's own
list_dir, so the host root is never walked and list_dir's existing
per-directory containment check still filters symlinks. The resolver
is back to main's behavior, so glob, grep and read_file keep rejecting
the root.
* fix(sandbox): resolve every entry of colon-joined path lists in forward path translation
* fix(sandbox): keep the original text of colon-list entries when reassembling
* fix(sandbox): reject shell-variable path prefixes in the host-bash guard
The local path guard scans commands for absolute paths with a regex whose
lookbehind skips a `/` that follows a word character. `~/`, `${HOME}/` and
`"$HOME"/` were rejected, but the bare `$HOME/...` form (quoted or not)
was never matched, so `cat $HOME/.ssh/id_rsa` ran on the host when
`sandbox.allow_host_bash` was enabled.
Scan `$NAME/...` separately and reject it, since the variable's value is
unknown at validation time. Variables inside URLs and non-ASCII prose keep
the existing exemptions. `$PWD/...` stays allowed because it names the same
file as the equivalent relative path, which the guard already accepts,
unless the command rebinds `PWD` itself.
* docs(changelog): reference #6930 for the host-bash variable path fix
* fix(sandbox): cover positional parameters and CDPATH in the host-bash guard
Review follow-up. `cat $1/etc/passwd` slipped both scans: the new variable
scan only matched names, and the absolute-path regex skips a `/` after a
digit. `bash -c` gets no positionals, so `$1` expands to nothing and the
shell opens `/etc/passwd`. Match `$0`-`$9` as well, but keep awk-style
arithmetic such as `$1/1024` allowed, since it was accepted before and
standard host roots do not start with a digit.
`CDPATH=$HOME cd docs` moved a relative `cd` outside the workspace, taking
both `$PWD/x` and a plain relative `x` with it. Reject any command that
mentions CDPATH in the working-directory check, so the `$PWD` exemption's
equivalence with relative paths holds.
* fix(sandbox): cover named homes and non-ASCII variable paths in the host-bash guard
- Reject `~name/...`, which expands to that user's home and slips the
absolute-path scan the same way `$HOME/...` did.
- Stop applying the non-ASCII prose exemption to the variable scan:
`$HOME/文档/...` is a host path.
- Allow awk field division `$1/$2`, which main accepts, and capture the
remainder in a lookahead so skipping `$1/` does not hide a later
`$2/etc/passwd`.
* fix(sandbox): decode cmd.exe output with the ANSI code page under UTF-8 mode
The cmd.exe branch left its pipe decoder to the `encoding=None` default of
`_run_windows_command`, which follows Python text-mode semantics via
`locale.getpreferredencoding(False)`. Under Python UTF-8 mode
(`PYTHONUTF8=1`, `-X utf8`, the Windows installer's UTF-8 opt-in, and the
default once PEP 686 lands) that call reports `"utf-8"`, while cmd.exe and
the children it spawns keep writing the host ANSI code page - so every
non-ASCII byte in cmd output collapses to U+FFFD, with no fallback path.
Measured on a real zh-CN host under `PYTHONUTF8=1` (`getpreferredencoding`
-> `utf-8`, `getencoding` -> `cp936`):
- through the `encoding=None` default: `'��� �����Z
'`
- pinned to the ANSI code page: `'你好 日本語
'`
This branch pins the cmd.exe pipe decoder to `locale.getencoding()`
(3.11+, repo floor is 3.12), the same selection the MSYS and PowerShell
branches already use for their fallback (#6727, #6548).
## Scope note
The `encoding=None` default itself is deliberately unchanged: it mirrors
Python's own `text=True` semantics and is pinned by
`test_windows_capture_matches_subprocess_text_mode_encoding`. Only the
cmd.exe branch - whose output is known to be the ANSI code page - overrides
it, the same way the MSYS and PowerShell branches already pin their own
decoders.
Follow-up to willem-bd's review note on #6727 (inline comment 4247281255)
flagging this path.
## Testing
- new (POSIX-runnable, mocked): the cmd branch passes
`encoding=locale.getencoding()` to `_run_windows_command`, verified under
a simulated UTF-8-mode host (`getpreferredencoding` -> `utf-8`,
`getencoding` -> `cp936`).
- updated: `test_execute_command_uses_cmd_command_mode_on_windows` now also
pins the new `encoding` kwarg.
- new (Windows; skipped when the console output CP does not match the ANSI
page - sibling tests can change it process-wide): real `cmd.exe /c echo`
roundtrip through `execute_command` returns the text intact (passes on a
zh-CN host when run standalone).
- full `tests/test_local_sandbox_encoding.py`: 40 passed, 2 skipped under
both `PYTHONUTF8=0` and `PYTHONUTF8=1` (Windows 11, Python 3.13.12, repo
venv); the text-mode contract test stays green in all three parametrized
modes (`locale-code-page`, `PYTHONUTF8`, `-X-utf8`). Red run against
unmodified main: both new/updated cmd-branch tests fail (`KeyError:
'encoding'`).
- `ruff check` + `ruff format --check` clean on both files.
* fix(sandbox): use the cmd console output code page
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): decode PowerShell native-child output with the locale fallback
Fixes#6646
## What
The Windows PowerShell branch of `LocalSandbox.execute_command` passed `encoding="utf-8"` to `_run_windows_command` with no `fallback_encoding`. The guarded console-UTF-8 preamble covers PowerShell's *own* output, but native children it runs (`python.exe`, CLI shims) inherit the pipe and keep writing the host code page, so the pipe decoder's `errors="replace"` turned every such byte into U+FFFD. This passes `fallback_encoding=locale.getpreferredencoding(False)` there, mirroring what the MSYS branch above already applies (#6548).
## Why
On Windows hosts whose ANSI code page is not UTF-8 (GBK on zh-CN hosts, cp932, cp949, cp1251, ...), non-ASCII tool results printed by native children reach the model garbled (#6646). PowerShell's own output stays correct because the preamble pins it to UTF-8, so the fallback only engages when a buffer is not valid UTF-8 — exactly the case where a native child wrote host-code-page bytes. Pure-ASCII and PowerShell-emitted UTF-8 buffers never touch the fallback, matching the tradeoff the MSYS branch already accepts.
## How checked
Reproduced on a real zh-CN Windows host (ANSI code page 936):
- Red on `main` (3f7a0769): a native `python.exe` child printing CJK text through `powershell.exe` / `pwsh.exe` returned `'\ufffd\ufffd\ufffd \ufffd\u0571\ufffd\ufffdZ\n'` — the new regression test failed on both shells:
```
FAILED tests/test_local_sandbox_encoding.py::test_windows_powershell_native_child_cjk_roundtrip[powershell.exe]
FAILED tests/test_local_sandbox_encoding.py::test_windows_powershell_native_child_cjk_roundtrip[pwsh.exe]
```
- Green with the fix: both pass, and the child's text round-trips exactly (`'\u4f60\u597d \u65e5\u672c\u8a9e\n'`).
- `tests/test_local_sandbox_encoding.py`: 34 passed, 1 skipped (Windows 11, CPython 3.13, `uv sync --frozen` environment). The two pre-existing PowerShell unit tests were extended so their `_run_windows_command` fakes accept the new `fallback_encoding` kwarg and assert it is the locale code page.
- `ruff check` and `ruff format --check` pass on both changed files.
* fix(sandbox): decode fallback must survive Python UTF-8 mode and mid-character captures
Follow-up to review feedback on #6727.
- Use ``locale.getencoding()`` (3.11+, repo floor is 3.12) instead of
``locale.getpreferredencoding(False)`` for both fallback call sites
(MSYS and the native-child guard): ``getpreferredencoding`` reports
``"utf-8"`` whenever the Gateway runs under Python UTF-8 mode
(``PYTHONUTF8=1``, ``-X utf8``), silently disabling the fallback and
leaving #6646 unfixed on those hosts. ``getencoding()`` deliberately
ignores UTF-8 mode and returns the code page the child's CRT used.
- After a failed strict UTF-8 decode, retry once without a trailing
incomplete multi-byte sequence, so a capture cut mid-character (the
10MB capture-cap boundary, or a timeout kill interrupting the child
mid-write) cannot cascade the whole buffer to the locale fallback.
Complete non-UTF-8 buffers still fail the retried attempt and reach
the fallback untouched.
- Tests: the Windows roundtrip probes now run under ``PYTHONUTF8=1``
with ``locale.getencoding`` pinned, so a green run proves the fallback
source ignores UTF-8 mode; added POSIX-runnable unit tests for the
cap-split and timeout-kill partial tails.
Test results (Windows host, ACP=936, repo venv via ``uv sync --frozen``):
``tests/test_local_sandbox_encoding.py`` 36 passed, 1 skipped, both with
``PYTHONUTF8=0`` and ``PYTHONUTF8=1``; ``ruff check`` and
``ruff format --check`` clean on both files.
* fix(sandbox): trim only on cut captures, preserving complete locale tails
Willem-bd's follow-up review pinned a real P2: on a cp1252 host the MSYS
branch receives b'caf\xe9' (a complete e-acute), the trailing E9 is trimmed
as an incomplete UTF-8 lead, and the ASCII prefix is accepted as valid UTF-8
- so the locale fallback never runs and the byte is silently dropped. No
capture cap or timeout is needed to trigger it.
Fix: the strict-retry trim is now authorized only when the capture may
actually have been cut mid-character - a byte-cap cut (signalled by the
capture's own truncated state) or a timeout kill (new mark_split_tail(),
called from _run_windows_command right after the process tree is killed). A
complete buffer, even one ending in a byte shaped like a bare UTF-8 lead,
now goes to the fallback with every byte intact.
Tests:
- new: cp1252 complete-tail roundtrip (red on b62ef75: returned 'caf').
- new: the same partial-shaped tail without a cut signal must fall back
intact instead of being accepted as a truncated UTF-8 prefix.
- new: POSIX-runnable _run_windows_command timeout integration test - the
child writes a partial UTF-8 sequence, hangs, is killed at the 1s
timeout; the trimmed prefix survives (this goes red if the kill path
forgets to authorize the trim, since the whole buffer cascades to the
fallback).
- updated: the timeout-kill unit test now marks the capture explicitly,
mirroring the real call site.
Evidence: Windows 11, Python 3.13.12 (repo venv), pytest
tests/test_local_sandbox_encoding.py -> 39 passed, 1 skipped under both
PYTHONUTF8=0 and PYTHONUTF8=1; ruff check + ruff format --check clean.
Red run against b62ef75: the three new/updated tests fail exactly as
described above.
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes
Why this change:
Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker
capability probe collapsed "provisioner unreachable" into "broker not
configured": one 1.5s timeout cached a negative for 300s, so new sandboxes
were created without the broker sidecar while the per-user credential
directories were still bind-mounted into the container, and existing Pods
kept being reused without their actual mode ever being read. Second, an
implicit-shell command that timed out or returned an unknown status only
marked the current client object; warm-pool reclaim and rediscovery built a
fresh client that re-entered a container whose shell state could not be
proven clean, and nothing about that ambiguity survived a Gateway restart.
What changed:
- The capability cache stores confirmed observations only, keyed per
provisioner endpoint/auth identity; an unknown answer (timeout, malformed
payload) raises instead of selecting credential placement, while a
confirmed broker requirement is retained through outages. One in-flight
probe per identity shares success and failure, and a Pod-attested mode
contradicting the cache forces one shared refresh.
- Execution uses the mode attested for the actual sandbox:
SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod
observation into the client, so a probe outage delays drift detection but
never denies reuse of an attested runtime, and never trusts an unattested
one.
- The provisioner reports each Pod's actual broker mode, generation (Pod
UID) and shell capacity from one read, rejects lark requests conflicting
with its configuration (409/503 carrying a capability-refresh marker the
Gateway uses to drop the stale cache entry), rejects conflicting or
Terminating Pods on coordination creates while reporting Terminating as
data to status/get/list, and answers Pod presence independently of
Service readiness.
- A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by
scope/sandbox/generation) fences containers after uncertain implicit
shell outcomes across client rebuilds and restarts; in-process reuse,
warm reclaim, discovery, create responses and orphan reconciliation all
pass one policy gate, and replacement runs under the existing ownership
and teardown fences. A fenced sandbox still tracked after a failed
release-recycle is destroyed inline on the next acquire.
- Quarantine records retire only after confirmed absence, retrying partial
cleanup for the quarantined generation and releasing remembered ports
under the same fences; local stop failures now propagate instead of being
logged away, and remote destroy errors surface to callers.
Problem solved:
A provisioner blip could silently strip broker isolation from new lark
sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted
default shells persist across warm reuse, rediscovery and restarts. Docs
updated: README, aio_sandbox and skills AGENTS.md.
* docs(changelog): record sandbox attestation and quarantine fix (#6436)
* fix(sandbox): preserve active leases and fail closed on unattested broker modes
Concurrent uploads could recycle a healthy AIO runtime while another holder
was creating or cleaning up a command session. Older provisioner responses
could also trigger destructive replacement before failing admission.
- Distinguish pending session creation from ambiguous outcomes and reserve
idle teardown until execution and upload holders finish scope cleanup.
- Reject stale fork lookups and reconcile cancelled acquires without
releasing a sandbox still borrowed by another holder.
- Default Lark broker mode to unverified and preserve unattested Pods until
the provisioner is upgraded.
- Recognize Docker 29 absence diagnostics while matching resource names exactly.
- Preserve Gateway upload errors and cover admission and lifecycle regressions.
* fix(sandbox): scope the unverified-broker refuse path to attested providers
The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox
without a lark_cli_broker attribute as unverified, but only AioSandbox
carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no
broker sidecar, no credential mounts, and no provisioner to upgrade, so
every lark-cli command on those providers failed with an unactionable
error.
Declare the tri-state on the Sandbox base class (defaulting to False,
the only mode brokerless providers have) so the None fail-closed path
stays scoped to AioSandbox's deliberate unverified default.
- Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring
the persistent_shell_sessions tri-state pattern
- Add a regression test pinning the non-broker overlay for a provider
without the attribute
* fix(sandbox): close review gaps in broker attestation and teardown lifecycle
Address the latest #6436 review round:
- The provisioner's post-build broker-mode 409 now carries the
capability-refresh marker, so an incompatible-mode conflict surfaced by
idempotent re-create, attestation drift or post-create verification drops
the Gateway's cached capability observation instead of repeating the same
409 for the full negative-cache TTL.
- AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False
when teardown is deferred because an execution/upload holder or a peer
teardown is still active, so callers can no longer mistake a deferred
destroy for a completed one.
- A live replacement under the same logical ID proves the fenced old
generations are absent; retire_replaced_generations prunes only those
stale records so they cannot strand future creates, while the live
generation's fence and unversioned crash fences are never touched.
- SandboxInfo.from_dict keeps a missing lark_cli_broker attestation
unverified (None) instead of silently rehydrating as attested non-broker.
- The restricted-network inspect path shares the strict
_is_no_such_network_error parser, closing the substring loophole for
context-name collisions and wrong-target errors.
* docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate
The three lease notes added in 26c307f3 pushed the effective guidance chain
to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance
(AG002). Fold them into two lines carrying the same rules.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): number remote read_file line ranges like LocalSandbox
E2B, BoxLite, Tenki and OpenSandbox sliced ranged reads with str.splitlines(), which also ends lines at a bare "\r", \f, \v, \x1c-\x1e, \x85, \u2028 and \u2029. LocalSandbox.read_file, count_file_lines and the read_file truncation marker end lines only at "\n", so on those providers a ranged read of such a file returned different lines, and following a truncation marker's start_line went back into text already shown. Add split_file_lines next to count_file_lines and use it in the four providers.
* test(sandbox): separate remote and host read paths
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
When Sandbox.connect() fails with a trusted 'sandbox not found' error,
_reconnect_live_client() used to release the deployment-level slot but
still re-raise. _evict_oldest_warm() then treated that confirmed-gone
result like an unknown-state failure: it kept the eviction tombstone and
never called _end_transition_locked(), so the local transitioning slot
leaked on every retry against an already-reaped VM. Once the leaked slots
reach the configured replicas, every acquire rejects or times out with
SANDBOX_CAPACITY_EXCEEDED while active=warm=reserved=0.
Return None from _reconnect_live_client() for trusted gone errors so all
callers (eviction, warm reclaim, discovery, reconciliation) take their
existing already-gone cleanup paths, keeping local bookkeeping consistent
with the deployment-level release on the same signal. Unknown-state
errors (network, auth) still retain the slot for a later retry.
Also report the transitioning count in SandboxCapacityExceededError
details and the wait-timeout message so hidden slot usage is visible
without a restart.
Fixes#6540
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): decode Git Bash output as UTF-8 so CJK tools are not garbled
Git Bash/MSYS always writes UTF-8 to its pipes, but the MSYS branch of
LocalSandbox.execute_command called _run_windows_command without an
encoding, so the capture fell back to locale.getpreferredencoding(). On a
non-UTF-8 host (GBK on zh-CN Windows) that decodes UTF-8 bytes with the host
code page and mangles every non-ASCII tool result. The pipe decoder replaces
instead of raising, so the corruption is silent. The PowerShell branch right
next to it has pinned UTF-8 since #5440.
Decode the MSYS branch as UTF-8 too, and pin the behaviour in
tests/test_local_sandbox_encoding.py: a unit test for the decoder choice and
a live round-trip through a real Git Bash (skipped when it is absent).
* fix(sandbox): keep locale fallback for native-child output under Git Bash
Git Bash/MSYS writes its own output as UTF-8, but Windows-native children
(python.exe, CLI shims) inherit the pipe and keep writing the host code
page. Forcing UTF-8 with errors=replace turned that output into U+FFFD.
Decode the captured buffer as UTF-8 first and retry with the locale when it
is not valid UTF-8, so both MSYS UTF-8 output and native-child code-page
output survive. The PowerShell branch is unchanged (its own output is pinned
to UTF-8 by the preamble); the fallback is only wired on the MSYS path.
Adds native-Python CJK regressions through execute_command.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): honor sandbox.environment on the local provider
LocalSandboxProvider ignored the documented sandbox.environment config
field (only the AIO provider consumed it), leaving public generation
skills (video-generation / image-generation) with no way to receive
provider API keys on the default local sandbox: env inheritance is
scrubbed by policy (#3861), the skills declare no required-secrets, and
dashboard runs cannot supply config.context.secrets. Fixes#6461.
- LocalSandbox accepts an operator environment; execute_command layers
it through the existing authorized injection channel
(build_sandbox_env), so injected entries survive the secret scrub
while platform credentials still never leak.
- Request-scoped env (declared required-secrets) wins on key collision.
- LocalSandboxProvider resolves $VAR refs from the gateway process env,
mirroring the AIO provider, and passes the mapping to every sandbox it
yields; missing/unloadable config degrades to injecting nothing.
* fix(sandbox): address review on local sandbox.environment
- P1: operator-injected values join the bash-tool output redaction set
(request-scoped values keep precedence on name collision), so an
echoed sandbox.environment credential no longer flows back into the
model-visible tool result.
- P2: consume resolved SandboxConfig.environment values verbatim. The
file-backed AppConfig loader already substitutes $VAR references;
re-resolving corrupted credentials whose value legitimately starts
with '$'. Drop the now-unneeded AIO-style resolver.
- Validate operator environment keys at LocalSandbox construction (a
typo'd key fails at startup, not on first command execution).
- Tests: verbatim passthrough, happy-path config loading, acquire()
wiring (generic + thread), key validation, and a P1 redaction
regression through the real bash tool path.
- Docs: SandboxConfig docstring and sandbox/AGENTS.md now describe the
operator injection channel and its output masking.
* fix(sandbox): redact effective local credentials in bash results
* fix(ci): keep local sandbox guidance within inherited budget
---------
Co-authored-by: vicfei <4058491+vicfei@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scanner): cache decoded text atomically
_cache_text_file keyed the cache on the virtual path and returned the path, and
the caller serves that path as the file's content, so a write that failed
part-way left a truncated entry that every later read handed back as the preview.
Publish through a temporary file plus os.replace.
* test(scanner): pin the cache entry against a failed publish
The helper-level case cannot distinguish the call site, so this drives the real
entry point with a failing publish and asserts the cached entry is untouched.
* docs(sandbox): keep agent guidance within its size limit
* fix(scanner): preserve publish errors during cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): drain direct release across cancellation
* test(sandbox): retain direct release across cancellation
* fix(sandbox): drain rollback release across cancellation
* test(sandbox): retain rollback release across cancellation
* test(sandbox): format rollback cancellation regression
* fix(sandbox): stop pinning a stale agent skill view once host bash is allowed
A restricted Agent run leaves a thread skill view behind. SandboxMiddleware
refreshes that view only while the provider can enforce skill isolation;
with host bash allowed LocalSandboxProvider cannot, so the middleware skips
it and restricted runs are rejected. The provider still mounted the view
whenever it existed, so later unrestricted runs on the thread kept the old
allowlist and could not read other enabled or newly added skills.
Look up the thread view only while isolation is supported, so those runs
get the shared projection the middleware expects. The view stays on disk
and is enforced again once host bash is turned off.
* docs(changelog): reference #6344 for the stale skill view fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): keep line endings in local sandbox reads, writes and str_replace
LocalSandbox opened files in text mode with the default newline=None.
Reads translated CRLF to LF, so str_replace on a CRLF file rewrote every
line as LF and the read-before-write hash missed newline-only changes;
on Windows, writes turned LF content into CRLF (breaking `bash run.sh`).
Read with newline="\n" and write with newline="", keeping line endings
as stored like the remote providers; reads still end lines only at "\n",
the rule count_file_lines and the truncation marker number lines by.
str_replace spells an LF-written old_str/new_str with CRLF when the file
uses it, so multi-line edits still match CRLF files on any provider.
The tool-output budget middleware wrote externalized output the same
way: on Windows the file no longer matched its stamped blob ref and the
next model call deleted it when no blob store was configured. It now
writes with newline="" too.
* docs(changelog): reference #6343 for the line-ending fix
* fix(sandbox): end local grep lines where read_file does
find_grep_matches still opened files with universal newlines, so a bare
"\r" split a line for grep but not for LocalSandbox.read_file, and a hit's
line_number stopped matching the start_line that reads it back. Read with
newline="\n" like read_file, and fold a CRLF terminator back to "\n" so
$-anchored patterns keep matching CRLF lines as before.
* fix(sandbox): scope ignore patterns to the search root in glob/grep
Rebuilt on top of current main so the changelog entry lands in the
Unreleased (2.2.0) Fixed section rather than the now-released 2.1.0 block.
* test(sandbox): cover the include_dirs glob branch at an ignored search root
The file.list_path branch only ran against a non-ignored tree, so a
regression there would go unnoticed. Two cases now exercise it with an
ignored root: the requested root keeps its contents, and ignored
descendants stay hidden.
* fix(middlewares): include file line count in read-before-write block message and clarify read_file range error
* fix(middlewares): address review feedback on line count pluralization and readability
* fix(middlewares): address round-2 review feedback for PR #6020
Stamping bug (_attach_read_mark, ggnnggez item 1):
Originally planned as a standalone issue; folded here since it is the exact mechanism behind the #6019 gate bypass. _attach_read_mark now returns early when message.content is in READ_FILE_NO_CONTENT_RESULTS - {READ_FILE_EMPTY}. In-band errors (invalid range, start_line exceeds file length, etc.) are returned with status='success' by LangChain so the existing message.status == 'error' guard did not catch them.
HTTP code collision [P3] (willem-bd):
Pre-stamp TOOL_META_KEY on the blocked ToolMessage before normalize_tool_result so files with 401, 403, 404, or 500 lines are not misclassified as auth/server errors by _classify_error_text. normalize_tool_message returns early when an existing TOOL_META_KEY entry is found.
Line-count parity (ggnnggez, inline L317):
Replace splitlines() with count_file_lines() in read_file_contract.py using count('\n'), matching LocalSandbox text-mode iteration and the _truncate_read_file_output formula. splitlines() over-counts for files with \f or \u2028 separators (e.g. pdftotext output).
Read hint (ggnnggez, inline _BLOCK_MESSAGE):
For empty files the ranged-read suggestion is omitted. For non-empty files the hint gives a concrete start_line=max(1, N-29), end_line=N.
Constant rename (ggnnggez, read_file_contract.py):
READ_FILE_INVALID_RANGE is the new canonical name; READ_FILE_EMPTY_RANGE kept as a backward-compat alias. _LEGACY_READ_FILE_EMPTY_RANGE removed (unreachable after the #6019 contract update).
Tests:
- TestBlockMessageLineCount parametrized with \f and \u2028 parity cases
- TestStampingGateBugFix: 4 no-content variants, gate-closed integration probe, empty-file-still-stamps assertion
- test_gate_block_on_numeric_line_count_is_recoverable: 401/403/404/500
- test_contract_constants_membership replaced with test_contract_constants_parity_with_localsandbox
---
NOT INCLUDED: ggnnggez item 2 (file length in read_file range error strings). READ_FILE_NO_CONTENT_RESULTS uses exact string membership and is also consumed by build_skill_usage for skill-read detection; making those strings dynamic requires a dedicated contract refactor. Deferred to a follow-up PR.
Note: test_web_fetch_relative_links failures in the full suite are pre-existing on upstream/main and unrelated to this change.
* test: clarify blank ranged reads in the write gate
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>