* fix(sandbox): surface missing-file AIO reads as actionable errors
When read_file targets a nonexistent file, the AIO server returns an error
envelope with data: {}, which the agent-sandbox SDK cannot parse into its
required FileReadResult fields. The resulting pydantic ValidationError text
became the model-visible tool result ("Error: 2 validation errors for
ResponseFileReadResult ..."), so the agent could not tell the file simply
does not exist and kept retrying or "fixing" nonexistent problems.
Catch ValidationError in AioSandbox.read_file and return an explicit
"Error: file not found or unreadable: <path>" instead. The branch is logged
at debug level: a missing file is an expected outcome, since
ReadBeforeWriteMiddleware probes every new-file write before allowing it, and
the previous error-level logging produced noisy gateway logs on that benign
path. Other exceptions keep the error log and the generic error string.
Fixesbytedance/deer-flow#6833
* docs(aio): align regression test reference style
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes
Why this change:
Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker
capability probe collapsed "provisioner unreachable" into "broker not
configured": one 1.5s timeout cached a negative for 300s, so new sandboxes
were created without the broker sidecar while the per-user credential
directories were still bind-mounted into the container, and existing Pods
kept being reused without their actual mode ever being read. Second, an
implicit-shell command that timed out or returned an unknown status only
marked the current client object; warm-pool reclaim and rediscovery built a
fresh client that re-entered a container whose shell state could not be
proven clean, and nothing about that ambiguity survived a Gateway restart.
What changed:
- The capability cache stores confirmed observations only, keyed per
provisioner endpoint/auth identity; an unknown answer (timeout, malformed
payload) raises instead of selecting credential placement, while a
confirmed broker requirement is retained through outages. One in-flight
probe per identity shares success and failure, and a Pod-attested mode
contradicting the cache forces one shared refresh.
- Execution uses the mode attested for the actual sandbox:
SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod
observation into the client, so a probe outage delays drift detection but
never denies reuse of an attested runtime, and never trusts an unattested
one.
- The provisioner reports each Pod's actual broker mode, generation (Pod
UID) and shell capacity from one read, rejects lark requests conflicting
with its configuration (409/503 carrying a capability-refresh marker the
Gateway uses to drop the stale cache entry), rejects conflicting or
Terminating Pods on coordination creates while reporting Terminating as
data to status/get/list, and answers Pod presence independently of
Service readiness.
- A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by
scope/sandbox/generation) fences containers after uncertain implicit
shell outcomes across client rebuilds and restarts; in-process reuse,
warm reclaim, discovery, create responses and orphan reconciliation all
pass one policy gate, and replacement runs under the existing ownership
and teardown fences. A fenced sandbox still tracked after a failed
release-recycle is destroyed inline on the next acquire.
- Quarantine records retire only after confirmed absence, retrying partial
cleanup for the quarantined generation and releasing remembered ports
under the same fences; local stop failures now propagate instead of being
logged away, and remote destroy errors surface to callers.
Problem solved:
A provisioner blip could silently strip broker isolation from new lark
sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted
default shells persist across warm reuse, rediscovery and restarts. Docs
updated: README, aio_sandbox and skills AGENTS.md.
* docs(changelog): record sandbox attestation and quarantine fix (#6436)
* fix(sandbox): preserve active leases and fail closed on unattested broker modes
Concurrent uploads could recycle a healthy AIO runtime while another holder
was creating or cleaning up a command session. Older provisioner responses
could also trigger destructive replacement before failing admission.
- Distinguish pending session creation from ambiguous outcomes and reserve
idle teardown until execution and upload holders finish scope cleanup.
- Reject stale fork lookups and reconcile cancelled acquires without
releasing a sandbox still borrowed by another holder.
- Default Lark broker mode to unverified and preserve unattested Pods until
the provisioner is upgraded.
- Recognize Docker 29 absence diagnostics while matching resource names exactly.
- Preserve Gateway upload errors and cover admission and lifecycle regressions.
* fix(sandbox): scope the unverified-broker refuse path to attested providers
The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox
without a lark_cli_broker attribute as unverified, but only AioSandbox
carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no
broker sidecar, no credential mounts, and no provisioner to upgrade, so
every lark-cli command on those providers failed with an unactionable
error.
Declare the tri-state on the Sandbox base class (defaulting to False,
the only mode brokerless providers have) so the None fail-closed path
stays scoped to AioSandbox's deliberate unverified default.
- Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring
the persistent_shell_sessions tri-state pattern
- Add a regression test pinning the non-broker overlay for a provider
without the attribute
* fix(sandbox): close review gaps in broker attestation and teardown lifecycle
Address the latest #6436 review round:
- The provisioner's post-build broker-mode 409 now carries the
capability-refresh marker, so an incompatible-mode conflict surfaced by
idempotent re-create, attestation drift or post-create verification drops
the Gateway's cached capability observation instead of repeating the same
409 for the full negative-cache TTL.
- AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False
when teardown is deferred because an execution/upload holder or a peer
teardown is still active, so callers can no longer mistake a deferred
destroy for a completed one.
- A live replacement under the same logical ID proves the fenced old
generations are absent; retire_replaced_generations prunes only those
stale records so they cannot strand future creates, while the live
generation's fence and unversioned crash fences are never touched.
- SandboxInfo.from_dict keeps a missing lark_cli_broker attestation
unverified (None) instead of silently rehydrating as attested non-broker.
- The restricted-network inspect path shares the strict
_is_no_such_network_error parser, closing the substring loophole for
context-name collisions and wrong-target errors.
* docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate
The three lease notes added in 26c307f3 pushed the effective guidance chain
to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance
(AG002). Fold them into two lines carrying the same rules.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Why this change:
In Lark broker mode the shim forwards piped stdin to the broker only
after EOF. AIO's /v1/bash transport keeps a subprocess stdin pipe open
for writes, so a bare `lark-cli ...` never saw EOF and the shim blocked
forever on input that would never arrive. An earlier brace-group/eval
approach fixed the hang but changed top-level parsing (aliases/extglob)
or let a trailing backslash consume the appended delimiter.
What changed:
- _run_bash_exec prefixes the original command with `exec < /dev/null`
on its own line, giving the fresh, per-invocation session immediate
EOF on default stdin. Explicit pipes/heredocs/files still override
fd0. The plain prefix keeps top-level parsing and appends no
delimiter a trailing backslash can consume.
- The persistent /v1/shell PTY transport keeps commands unchanged; the
shim already treats TTY stdin as no input.
- The shim drains non-TTY stdin under grace/tail idle budgets
(DEERFLOW_LARK_BROKER_STDIN_{GRACE,TAIL}_SECONDS, default 2s, valid
(0, 600]); an idle pipe exits 124 without contacting the broker, and
read errors fail loudly instead of becoming empty input.
- Broker logs each exec as argc/exit/elapsed only (never argument
values) and warns on concurrency-cap rejections.
Problem solved:
Lark commands without explicit stdin hung indefinitely in broker mode,
stalled producers could execute partial writes, and broker logs risked
capturing argument values. Docs updated: README, harness AGENTS.md,
docker/lark-cli-broker README.
* fix(sandbox): bound remaining shell session cleanup requests
release_command_scope(), close()'s scoped-session drain, and close()'s recovery-session cleanup called shell.cleanup_session() without a request budget, so a stalled cleanup could hold the thread-key serializer, scoped.lock, or the sandbox lock for the SDK's default transport budget. Pass the existing _bounded_cleanup_request_options() (5s, max_retries=0) at those three sites; _cleanup_session_best_effort() and its swallow-and-log contract are unchanged, as are create_session and list_dir lifetimes.
* fix(sandbox): bound AIO list_dir with a directory deadline
Give AioSandbox.list_dir its own 60s directory deadline with a 65s no-retry
host envelope, independent of bash_command_timeout. Preserve #5634's
shell-generation selection: list_dir runs on the current recovery session once
the implicit shell is fenced, and an ambiguous list_dir outcome - transport
timeout or an ambiguous returned status - fences whichever generation actually
executed the request, dropping local recovery ownership and attempting bounded
best-effort cleanup instead of leaving that session reusable. hard_timeout
remains definite termination and keeps the targeted session reusable. Only
completed/None results are parsed, so a partial find is never returned as a
complete listing.
Session creation RPC lifetime remains out of scope.
* fix(sandbox): recover sessions after transport failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): enforce AIO shell command timeouts
Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results.
* fix(sandbox): contain ambiguous AIO command outcomes
Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative.
* fix(sandbox): wire AIO command timeout from provider config
Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working.
* docs(sandbox): document AIO command timeout semantics
Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget.
* docs(sandbox): scope the wedge-resistance claim to command requests
willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up.
* fix(sandbox): make list_dir honor the shell generation fence
list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.
* fix(sandbox): stop list_dir from reporting failures as empty
Remote providers swallowed find/client errors as [] and 2>/dev/null
missing paths as empty stdout. ls_tool then told the agent the
directory was (empty). Raise OSError/FileNotFoundError instead so
the tool returns Error.
* fix(sandbox): list_dir raises on missing local paths and uses find -H
Empty stdout is not a missing path when find's start point is a
symlink (E2B /mnt/acp-workspace). Dereference only the start point
with find -H. LocalSandbox now raises FileNotFoundError for a
non-directory root, matching remote providers. AIO maps a missing
result.data to OSError rather than FileNotFoundError.
* fix(sandbox): group AIO list_dir find type predicates
Without parentheses, find PATH -maxdepth N -type f -o -type d applies
-type d without maxdepth and can drop files from the listing.
* fix(sandbox): distinguish list_dir command failure from missing path
Tenki, Boxlite, and OpenSandbox treated any empty find stdout as
FileNotFoundError, so a missing find binary (exit 127) or SDK error
looked like a missing directory. Raise OSError when find status is
outside (0, 1); keep FileNotFoundError for the find-ran-but-empty case.
* fix(sandbox): apply list_dir exit-status contract to AIO and E2B
Same gap as Tenki/Boxlite/OpenSandbox: empty find stdout with exit 127
was FileNotFoundError. Raise OSError when the status is outside (0, 1).
* fix(sandbox): classify list_dir by find status not head status
find | head under sh -lc reports head's exit code, so a missing find
binary (127) became FileNotFoundError. Record find's own status after
the bounded listing, treat SIGPIPE 141 as truncation success, and add
a shell-level regression test.
* test(auth): include projects permissions in /me contract pins
#5265 added projects:read/write/delete to the registered route set.
The /auth/me tests still pinned the pre-projects list, so CI failed
after merging main.
* fix(sandbox): do not treat missing list_dir marker as success
The generated script ended on `rm -f`, so process status was 0/1 even
when find's marker never landed. Both codes are in _FIND_OK, and the
parser fallback then classified an empty listing as FileNotFoundError —
the 127 misclassification this helper was meant to close.
Exit with find's status (126 if unknown). A missing marker is now
OSError unless the process status is already a non-OK failure.
* test(sandbox): emit list_dir status marker in provider fixtures
Parser now requires __DF_FIND_STATUS__ and refuses marker-less stdout.
Update AIO/Boxlite/E2B stubs and OpenSandbox/Tenki find fakes so listings
carry :0 and missing paths carry :1 with matching exit codes.
* style(sandbox): format list dir test fixture
* style(sandbox): format remote list dir helper
* docs(sandbox): keep guidance within the tested size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): stop stripping filenames when parsing find output in remote providers
The list_dir and glob parsers in the e2b, OpenSandbox, AIO, Tenki, and
BoxLite providers called .strip() on every line of find output. A
filename that legitimately ends (or begins) in whitespace was corrupted,
so the listed path never resolved on any follow-up file API call, and
the remote providers diverged from LocalSandbox, which preserves such
names via pathlib.
splitlines() already removes the line terminators, so filter empty lines
only and keep each entry verbatim. Same class of bug as the e2b
_sync_outputs_to_host fix (#4861), applied to the search parsers.
Adds a trailing-space regression test per provider at the seam each
suite already uses.
* fix(sandbox): split find output on \n only, and rename the tenki test
Review follow-ups from willem-bd:
- aio_sandbox.list_dir used str.splitlines(), which also breaks records on
\v, \f, \x1c-\x1e and \x85 - all legal inside a Linux filename, and all
contrary to this PR's own rule that the newline is the only delimiter.
find emits \n and nothing else, so split("\n") is the correct parse.
- Renamed test_search_preserves_trailing_space_in_filename to
test_list_dir_and_glob_preserve_trailing_space_in_filename, matching the
sibling tests in test_opensandbox_provider.py and test_boxlite_provider.py.
The body covers list_dir and glob; it never touches grep.
* feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2)
PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a
subagent completes, so objectively checkable requirements can never be
silently passed by a self-report.
- subagents/acceptance_checks.py: deterministic leaf families —
file:<path> exists|non-empty and file_written:<path> read through
read_current_file_content scoped to the shared thread workspace; the
read uses the sandbox-native virtual path form (the local read
validator and provider mount tables resolve /mnt/user-data/... paths,
not host paths); the scope decision canonicalizes with realpath on the
local sandbox so workspace symlinks cannot escape into uploads; a
remote provider's "Error: ..." return string is normalized to a
failed check (provider-typed via is_local_sandbox); a
UnicodeDecodeError marks a binary deliverable as existing and
non-empty; out-of-scope paths degrade to UNVERIFIED.
tests_passed:<command> anchors to a matching recorded bash execution
with status=success and a test-summary shape; matching is
shell-structure aware with control-flow attribution (span must end at
the last segment with provable execution), negating-option values are
ineligible evidence and a target negated anywhere in the command
degrades the match, extra flags must be selection-preserving, extra
positionals widen only after a path-scoped criterion, truncated
commands degrade via command_truncated, the summary shape is read
only from output attributable to the matched segment (preceding
segments provably silent by invocation form), and pass shapes require
a nonzero passed count. Criterion text is neutralized with
neutralize_untrusted_tags before storage/rendering. Anything else
renders UNVERIFIED, never silently passed.
- executor: accumulate bounded bash command/output evidence per streamed
chunk (merged by tool_call_id, newest-capped) so subagent
summarization compacting earlier messages cannot erase a recorded
execution; the recorded status is the actual shell exit status parsed
from the output's exit marker (signed codes included; the remote
Command exited with code N form is accepted only as the whole trimmed
output), falling back to deerflow_tool_meta only when no marker
exists.
- sandbox providers: e2b/opensandbox/tenki/boxlite append the
LocalSandbox-style "Exit Code: N" marker on nonzero exit even with
non-empty output; aio propagates the SDK's structured exit_code on
both exec paths the same way; local timeouts append Exit Code: 124;
and _truncate_bash_output always preserves a trailing exit marker
(signed included) inside its budget, with a 32-char floor raising any
smaller configured limit, so the actual shell outcome always survives
in the output text.
- task_tool: run the checklist offloaded (asyncio.to_thread) on the
completed branch, failure-isolated; stamp the verdict into result
metadata and render the per-criterion section into the model-visible
result text.
- status contract: additive subagent_acceptance_verdict transport with
read-side structural validation.
- delegation ledger: entry carries the verdict and renders a compact
acceptance segment; gateway strips caller-forged verdicts from both
ledger entries and message metadata, like the citation verdict.
- blocking-IO anchor pins the offload (teeth proven red->green); leaf
read errors catch only OSError/SandboxError so unexpected errors reach
the task-tool-level isolation instead of being mislabeled.
* fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4)
- negating options: overlap with a matched criterion target is now
checked by path/nodeid prefix, not exact token equality — excluding a
sub-path of the criterion's selection (pytest tests --deselect
tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds
- output attribution: any redirection token in the matched final segment
makes the recorded tail non-attributable (> / >> / 2> are word
characters to the parser, so redirection was invisible to the matcher)
- silent-source allowlist narrowed from any *activate suffix to the
*/bin/activate shape
- status_contract docstring: restore the shared-fixture sentence and
note subagent_acceptance_verdict is deliberately outside the fixture
- executor: update_bash_executions publishes [] (stream carried no
bash-family calls) instead of collapsing it into None, mirroring
update_tool_receipts
* fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4)
- tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error"
and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass
shape when the exit status is swallowed (|| true) or absent; zero-error
counts stay clean.
- file leaves: bound the deliverable read — a "wc -c" shell size probe
answers files above 50k bytes without loading ~2x their size, honoring
the host-bash kill switch and falling back to the full read on any
non-integer rendering, so verdicts never get less sound.
- executor: record the exit marker text as status_marker on harvested bash
evidence; the leaf detail now reports the marker actually seen instead of
asserting a failure indistinguishable from the command's own trailing text.
- extend the blocking-IO anchor to drive the probe branch inside the
offload; teeth re-verified red->green.
* fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4)
- file leaves: never read unbounded — size is established first (os.stat on
the validated local host path, so the host-bash-disabled configuration
needs no shell; a guarded wc -c on remote providers that renders
missing/unreadable in its own words). Above the 50k cap the leaf answers
from the size alone, at/below it the full read runs, and an
unestablishable size degrades to UNVERIFIED instead of an unlimited
fallback read.
- output attribution: source/. prefixes are never provably silent — a
crafted */bin/activate path shape says nothing about what the script
prints, so sourced segments can no longer lend a passing summary.
- executable identity: an explicitly path-spelled criterion now requires
the same normalized executable path; the basename rule stays only for
deliberately bare criterion commands.
* fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4)
- remote probe no longer runs in the sandbox's persistent shell: a fresh
env -i /bin/sh with absolute-path stat/realpath (poisoned functions,
aliases, PATH, exported functions, IFS, locale cannot steer it), plus a
marker env routing AIO onto a fresh per-call bash.exec session.
- metadata-only: stat never opens content, so a FIFO deliverable cannot
block the parent for the provider's idle timeout; non-regular files
(fifo/dir/symlink) degrade to UNVERIFIED.
- containment canonicalized against the literal mount root: a
final-component symlink or a swapped parent directory (root included)
cannot redirect the check outside shared storage; unprovable layouts
degrade to UNVERIFIED.
* fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4)
Literal-root equality made every remote file leaf permanently UNVERIFIED
on e2b and Tenki, which realize /mnt/user-data as a symlink to the home
dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink).
Containment now compares the file's realpath against the mount root's
realpath — exactly what the provider's own read path resolves, so probe
and read-back stay consistent; final-component symlinks stay rejected by
the non-dereferencing stat, and an intermediate dir-link escape under a
sane root still lands ESCAPED. The inner script is a module constant and
the suite now executes the composed probe for real against on-disk
layouts (real dir, symlinked prefix, final symlink, fifo, missing,
dir-link escape), which the canned-output stub could not see.
* fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4)
- matching: a criterion with no positional selection target (bare pytest,
make test) stands for the runner's default selection, so ANY negating
option (--ignore/--deselect/...) makes the recorded run a different
selection — unprovable. The overlap guard only sees consumed criterion
tokens, which a bare criterion does not have; scoped criteria keep the
unrelated-exclusion behavior.
- attribution: cd is no longer blanket-silent — CDPATH makes cd print the
resolved (subagent-chosen) destination and the pass shapes match as
substrings, so one mkdir 'all tests passed' plus an export minted a pass
for any quiet command. A cd argument or CDPATH= value (export or leading
assignment) carrying any summary shape makes the segment non-silent;
shape-free cd dir wrappers keep matching.
- docs: _truncate_bash_output states the effective 32-char floor (the
guarantee previously read as an unconditional max_chars bound).
* fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4)
Self-audit in the shape of the last review rounds — channels the matcher
classified as accounted-for that can change what runs, narrow the
selection, or lend the summary text:
- env assignments are no longer blanket-stripped: only an allowlist of
inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a
matched span, and a non-allowlisted assignment in any preceding segment
(pure-assignment or export NAME=) is state pollution — PATH redirects
the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code,
PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs,
BASH_ENV runs arbitrary shell startup. All degrade to unprovable.
- runtime expansions: any span token carrying /$( )/backticks, any
negating-option value carrying an expansion or glob (unknown excluded
set), and any extra executed token carrying glob metacharacters
(crafted option-looking filenames narrow invisibly) are unprovable.
Criterion-side globs stay self-consistent (literal match).
- cd: an argument carrying a runtime expansion or glob is non-silent
(unknown destination, unknown print); CDPATH= assignments are now
handled as state pollution at the match layer, subsuming the
value-shape special case.
* fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4)
- tests_passed: on a persistent-shell provider (new
Sandbox.persistent_shell_sessions capability, set by AioSandbox) every
leaf degrades to UNVERIFIED — any earlier call in the shared session
could have mutated the state the clean-looking run executed in, and
only a fresh controlled session (RFC section 6 verifier) can prove
otherwise. The flag is read from the provider registry without
acquiring a sandbox.
- env assignments: the allowlist is gone — no variable is provably inert
across repositories (CI/DEBUG are routinely read by tests). The span's
assignment prefix must equal the criterion's exactly (values included,
order-insensitive); any assignment or export NAME= in a preceding
segment is state pollution.
- scoping: positional targets are now read by option arity, so a path
embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never
counts as a selection target and an extra positional after such a
criterion narrows the default selection it denotes.
* fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4)
* fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4)
* fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4)
* fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4)
* fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4)
* fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4)
* fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4)
* fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4)
* fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4)
Images older than all-in-one-sandbox 1.9.x have no /v1/bash/* routes, so
every env-bearing command (skills declaring required-secrets) surfaced a
raw nginx 404 that the model kept retrying. Detect the 404, remember the
capability gap per sandbox instance, and return an actionable error that
names the minimum image version and the sandbox.image remediation.
No fallback through the legacy shell path on purpose: /v1/shell/exec has
no env parameter, and every workaround puts the secret values back into
the command string or on disk — the exact leak surfaces the
request-scoped secrets design closed.
Closes#3921
* fix(sandbox): create shell session before retrying on a fresh id
The AIO sandbox recovery path generated a UUID and passed it straight to
exec_command(id=...). The sandbox image only auto-creates a session when
exec_command is called with *no* id; an exec carrying an unknown id returns
HTTP 404 "Session not found". So every ErrorObservation recovery itself
404'd, turning a transient session lapse into an unrecoverable tool error
that looped the run up to the LangGraph recursion limit.
Explicitly create_session(id=fresh_id) before targeting that id on retry.
create_session is idempotent (returns the existing session if the id already
exists), so this is safe under the serializing lock.
Updated the regression test to assert the retry targets exactly the
created session id rather than a fabricated, uncreated one.
* fix(sandbox): release the one-shot recovery session after retry
The fresh session created on the ErrorObservation recovery path is used for
exactly one command -- the next execute_command runs with no id and returns
to the default session. Under persistent session corruption every command
would create another session that is never reused or released, accumulating
sessions on the container.
Release it best-effort with cleanup_session() in a finally, swallowing any
cleanup error so it never masks a successful retry.
Addresses review feedback on #3577.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): close AioSandbox HTTP client during provider teardown (#2872)
AioSandbox allocates a host-side agent_sandbox client (wrapping an
httpx.Client) in __init__, but AioSandboxProvider.release/destroy/shutdown
only popped provider state and tore down the backend container — the
client/transport owned by each cached AioSandbox was never explicitly
closed, accumulating unreclaimed sockets in long-running services.
- Add AioSandbox.close(): best-effort, idempotent close of the wrapped
httpx_client (falls back to top-level client.close()); errors are
logged but never raised so backend cleanup is never blocked.
- AioSandboxProvider.release()/destroy() now close the cached AioSandbox
before dropping it; shutdown() inherits this via destroy().
* fix(sandbox): close the real httpx.Client owned by AioSandbox (#2872)
The previous close() only walked one level (wrapper.httpx_client), which resolves to the Fern-generated HttpClient wrapper that has no close(). The real socket-owning httpx.Client lives one level deeper at _client_wrapper.httpx_client.httpx_client, so the close path never fired and host-side sockets still leaked.
Resolve the real httpx.Client with graceful degradation; clear self._client under the lock for use-after-close and concurrent double-close safety; mark provider release()/destroy() try/except as defense-in-depth; rewrite TestClose against the real nested structure to lock down the original no-op bug.
* fix(sandbox): pass no_change_timeout to exec_command to prevent 120s premature termination
The agent_sandbox library's shell API defaults no_change_timeout to 120
seconds. When AioSandbox.execute_command() called exec_command() without
this parameter, commands producing no output for 120s would return with
NO_CHANGE_TIMEOUT status even though the script was still running.
Pass no_change_timeout=600 to all exec_command calls (matching the
client-level HTTP timeout) so long-running commands are not cut short.
Fixes#2668
* test(sandbox): add assertions for no_change_timeout in execute_command and list_dir
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/2f37bc72-0826-4443-a6ba-e5b78c22fb5a
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: prevent concurrent subagent file write conflicts
Serialize same-path str_replace operations in sandbox tools
Guard AioSandbox write_file/update_file with the existing sandbox lock
Add regression tests for concurrent str_replace and append races
Verify with backend full tests and ruff lint checks
* fix(sandbox): Fix the concurrency issue of file operations on the same path in isolated sandboxes.
Ensure that different sandbox instances use independent locks for file operations on the same virtual path to avoid concurrency conflicts. Change the lock key from a single path to a composite key of (sandbox.id, path), and add tests to verify the concurrent safety of isolated sandboxes.
* feat(sandbox): Extract file operation lock logic to standalone module and fix concurrency issues
Extract file operation lock related logic from tools.py into a separate file_operation_lock.py module.
Fix data race issues during concurrent str_replace and write_file operations.
* fix(sandbox): serialize concurrent exec_command calls in AioSandbox
The AIO sandbox container maintains a single persistent shell session
that corrupts when multiple exec_command requests arrive concurrently
(e.g. when ToolNode issues parallel tool_calls). The corrupted session
returns 'ErrorObservation' strings as output, cascading into subsequent
commands.
Add a threading.Lock to AioSandbox to serialize shell commands. As a
secondary defense, detect ErrorObservation in output and retry with a
fresh session ID.
Fixes#1433
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(sandbox): address Copilot review findings
- Fix shell injection in list_dir: use shlex.quote(path) to escape
user-provided paths in the find command
- Narrow ErrorObservation retry condition from broad substring match
to the specific corruption signature to prevent false retries
- Improve test_lock_prevents_concurrent_execution: use threading.Barrier
to ensure all workers contend for the lock simultaneously
- Improve test_list_dir_uses_lock: assert lock.locked() is True during
exec_command to verify lock acquisition
* style: auto-format with ruff
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
