71 Commits

Author SHA1 Message Date
YxinMiracle
b6ba739297
fix(sandbox): scope restored references to authenticated threads (#5736)
* fix(sandbox): scope restored references to authenticated threads

* docs(sandbox): trim inherited guidance

* fix(sandbox): reject restored references without thread scope

---------

Co-authored-by: YxinMiracle <“939157765@qq.com”>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-23 09:53:49 +08:00
Totoro
0efdf8e7d8
fix(sandbox): cut read_file output at a line boundary and name the next start_line (#5474)
* fix(sandbox): cut read_file output at a line boundary and name the next start_line

read_file head-truncates at a character offset and its marker told the model
to continue with start_line/end_line while reporting only character counts,
so the cut usually fell mid-line and the model had to guess which line to
continue from. The cut now lands on the last line boundary the budget allows,
and the marker reports lines shown of lines total, keeps the character
counts, and names the exact next start_line. When the line at the cut is
longer than 4,096 characters (minified sources, one-line JSON) the cut stays
at the character limit and the marker names the line it fell inside, so a
re-read of that line is the continuation. Reads under the limit are unchanged.

* fix(sandbox): make the read_file continuation hold for ranged reads and long lines

Line numbers in the truncation marker are now file line numbers: read_file_tool
passes start_line - 1 as the line offset, so a ranged read that is itself
truncated names the right next line instead of one relative to its slice. A
ranged read is a provider slice joined with newlines, so the tool also says
so and a trailing newline there counts as an empty last line.

The long-line fallback now names a continuation only when it makes progress:
a read from the cut line when the whole line fits such a read, a single-line
read (start_line = end_line) when only the line alone fits max_chars, and bash
when even that cannot return it; the single-line form names no further line
after the last line of the read. The budget reserves one extra character so a
newline sitting exactly at the limit still counts as a complete line, the
"fits a fresh read" check uses a pessimistic estimate of the follow-up read's
marker, and a budget too small for any marker still returns a marker instead
of a bare prefix.

Adds unit cases for the ranged-read offset, the newline-at-budget edge, the
single-line-read and bash forms, tiny budgets and empty last lines, plus
end-to-end tests that drive read_file_tool with a LocalSandbox and follow the
markers across reads, asserting the kept segments reproduce the file without
gap or overlap.

* fix(sandbox): keep naming the next line after a bounded read's last line

A ranged read with an end_line below the file's length is a slice that stops
mid-file, so the single-line-read continuation must still name the line after
the slice's last line; only a read that reached the end of the file names
nothing further. The tool passes whether the read was bounded by an end_line
separately from the joined-lines hint, because a start_line-only read also
runs to the end of the file.

A blank line and a line past the end both read back as an empty slice; the
tool now tells them apart with a two-line probe, so a continuation named by a
marker that lands on a blank line answers "(empty)" rather than
"(start_line exceeds file length)".

* test(sandbox): adapt upstream continuation checks after rebase

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-16 23:57:38 +08:00
xiaodu55
49f2197ba1
fix(sandbox): report the line a read_file truncation lands on (#5478)
* fix(sandbox): report the line a read_file truncation lands on

The read_file tool head-truncates at a character offset and tells the
model to continue with start_line/end_line, but the marker reported only
character counts ("showing first N of M chars"), so the model had no way
to know which line the cut fell on — the cut almost always lands mid-line
and read_file output carries no line numbers (#5475).

The marker now also reports the 1-indexed line holding the first hidden
character and the file's total line count, and names the exact resume
point: "... [truncated: showing first N of M chars (cut lands in line L
of T). Use start_line=L — optionally with end_line — to continue without
a gap] ...". Resuming at the reported line is gap-free whether the cut
lands mid-line or exactly after a newline.

The marker length budget accounts for the new fields, so the
len(result) <= max_chars contract still holds.

* fix(sandbox): report absolute lines in ranged-read truncation markers

Review follow-up on #5478: read_file_tool runs the same truncation on
ranged reads (start_line/end_line), where the slice's line 1 is the
requested start_line, not the file's first line. The marker's reported
lines were slice-relative while the model reasons in absolute file lines,
so the resume hint could re-issue the identical start_line forever
(repro: resume at 831 -> "cut lands in line 831 of 5170" -> start_line=831).

_truncate_read_file_output gains a line_offset parameter (the 0-based
absolute line of the slice's first line) and reports absolute lines for
both the cut position and the range end; read_file_tool threads
effective_start - 1 through. Full reads pass the default offset 0 and are
byte-identical.

Regression tests pin the absolute coordinates and that the resume point
strictly advances past the slice start.
2026-09-16 22:24:01 +08:00
Hyeonsang Cho
ed986a10ef
fix(sandbox): report truncated remote glob and grep results (#5427)
* fix(sandbox): report truncated remote glob and grep results

BoxLite, Tenki, E2B, and OpenSandbox run find/grep in the sandbox, cap
the raw output with `| head`, and then filter those lines in Python:
ignored directories such as node_modules are dropped and grep's glob
scope is applied. They reported truncated only when max_results matches
survived the filter. When the capped lines were mostly filtered out, a
search with real matches past the cap came back short or empty with
truncated=False, and glob_tool/grep_tool rendered it as "No files
matched" / "No matches found". With the default max_results=200 and
1,200 files under node_modules, glob("**/*.py") reported no matches for
a workspace that has src/app.py.

remote_search_command now lets one line past its limit through, and
parse_remote_search_output(..., limit=) returns RemoteSearchOutput(text,
truncated): the first `limit` lines and whether the extra line arrived.
Exactly `limit` lines stays a complete result. Each provider passes the
cap it already computed to both calls and returns that truncated from
glob and grep when fewer than max_results results survive filtering.

The glob and grep tools now describe an empty truncated result as
incomplete instead of reporting no matches, which also covers AIO grep's
forwarded truncated flag. Sandbox.glob/grep document truncated as "the
matches may be incomplete".

* docs(changelog): reference #5427 in the remote search truncation entry

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-14 15:38:34 +08:00
Shxiao
d5ae3882b6
fix(sandbox): reverse-resolve forward-slash spellings of Windows host paths (#5373)
* fix(sandbox): reverse-resolve forward-slash spellings of Windows host paths

Forward resolution deliberately spells resolved paths with forward
slashes in commands and file content (#3869: backslashes break bash
escapes), but the reverse scanner anchored its matches on the native
backslash base, so on Windows every forward-resolved path that came back
in command output or agent-written files leaked the raw host path
instead of mapping to its container path. Match separator-agnostically
in LocalSandbox like sandbox.tools already does, align the two
regex-cache tests with the documented spellings, and refresh the
path_patterns rationale comments that described the old asymmetry.

* test(sandbox): pin the reverse mask to separator-agnostic matching

The flag is the entire Windows fix but is invisible on POSIX CI, so
assert the routing kwargs in the direct-helper wiring test — the same
pin test_tools_mask_patterns_route_through_the_helper already applies to
the sandbox.tools copy. A revert to separator-exact matching now fails
on every platform instead of silently reintroducing the host-path
leak on Windows.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-14 07:30:34 +08:00
Shxiao
fa89a12526
fix(sandbox): mask output tails into POSIX-style virtual paths (#5247)
* fix(sandbox): mask output tails into POSIX-style virtual paths

The output maskers slice the matched path tail from the original
output. With separator-agnostic matching, a Windows-spelled nested
tail kept its backslashes and was spliced into the POSIX-style virtual
path, so glob results and masked read output showed mixed paths like
/mnt/user-data/workspace/pkg\util.py or
/mnt/skills/integrations/lark-cli\lark-doc\SKILL.md. Virtual paths are
always POSIX-style, so normalize nested tails to forward slashes the
same way depth-1 tails already end up. Depth-1 tails and the callable
replacer (LocalSandbox._reverse_resolve_path) were unaffected.

Pin the nested-tail contract in test_sandbox_path_patterns; the
previously failing glob-tool and skills-masking regressions now pass
on Windows hosts.

* refactor(sandbox): share the mask tail-splicing rule; guard it on Linux CI

Review follow-up for #5247:

- hoist the tail-splicing rule (slice off the base, strip leading
  separators, normalize the rest to "/") into
  path_patterns.normalize_mask_tail and import it at both call sites,
  so the two maskers can only drift in their matching logic, not in
  the splice;
- add test_mask_local_paths_normalizes_windows_spelled_skill_tails,
  which spells the skills host root and the output with Windows-style
  strings so the nested tail keeps backslashes on every platform.
  Reverting the mask_local_paths_in_output-side normalization now goes
  red on Linux CI too, not only on Windows hosts.
2026-09-09 10:09:28 +08:00
gus
47f43f79f4
fix: improve local environment detection guidance (#5111)
Co-authored-by: angus-guo <217034332+angus-guo@users.noreply.github.com>
2026-09-02 21:35:33 +08:00
Aari
9e0fbd60fa
fix(sandbox): isolate concurrent subagent shell sessions (#5134)
* fix(sandbox): isolate concurrent subagent shell sessions

* fix(sandbox): make execution acquire idempotent

* fix(sandbox): close execution lifecycle gaps

* fix(sandbox): serialize retained client lifecycle

* fix(sandbox): close remaining client lifecycle gaps

* fix(sandbox): unwind failed client lookup

* fix(sandbox): protect internal lease identities

* fix(sandbox): make cancellation reconciliation durable

* fix(sandbox): fence cancelled workers and IM uploads
2026-09-02 21:05:23 +08:00
Zeren Wang
a06a6fed7e
feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2) (#5109)
* feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2)

PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a
subagent completes, so objectively checkable requirements can never be
silently passed by a self-report.

- subagents/acceptance_checks.py: deterministic leaf families —
  file:<path> exists|non-empty and file_written:<path> read through
  read_current_file_content scoped to the shared thread workspace; the
  read uses the sandbox-native virtual path form (the local read
  validator and provider mount tables resolve /mnt/user-data/... paths,
  not host paths); the scope decision canonicalizes with realpath on the
  local sandbox so workspace symlinks cannot escape into uploads; a
  remote provider's "Error: ..." return string is normalized to a
  failed check (provider-typed via is_local_sandbox); a
  UnicodeDecodeError marks a binary deliverable as existing and
  non-empty; out-of-scope paths degrade to UNVERIFIED.
  tests_passed:<command> anchors to a matching recorded bash execution
  with status=success and a test-summary shape; matching is
  shell-structure aware with control-flow attribution (span must end at
  the last segment with provable execution), negating-option values are
  ineligible evidence and a target negated anywhere in the command
  degrades the match, extra flags must be selection-preserving, extra
  positionals widen only after a path-scoped criterion, truncated
  commands degrade via command_truncated, the summary shape is read
  only from output attributable to the matched segment (preceding
  segments provably silent by invocation form), and pass shapes require
  a nonzero passed count. Criterion text is neutralized with
  neutralize_untrusted_tags before storage/rendering. Anything else
  renders UNVERIFIED, never silently passed.
- executor: accumulate bounded bash command/output evidence per streamed
  chunk (merged by tool_call_id, newest-capped) so subagent
  summarization compacting earlier messages cannot erase a recorded
  execution; the recorded status is the actual shell exit status parsed
  from the output's exit marker (signed codes included; the remote
  Command exited with code N form is accepted only as the whole trimmed
  output), falling back to deerflow_tool_meta only when no marker
  exists.
- sandbox providers: e2b/opensandbox/tenki/boxlite append the
  LocalSandbox-style "Exit Code: N" marker on nonzero exit even with
  non-empty output; aio propagates the SDK's structured exit_code on
  both exec paths the same way; local timeouts append Exit Code: 124;
  and _truncate_bash_output always preserves a trailing exit marker
  (signed included) inside its budget, with a 32-char floor raising any
  smaller configured limit, so the actual shell outcome always survives
  in the output text.
- task_tool: run the checklist offloaded (asyncio.to_thread) on the
  completed branch, failure-isolated; stamp the verdict into result
  metadata and render the per-criterion section into the model-visible
  result text.
- status contract: additive subagent_acceptance_verdict transport with
  read-side structural validation.
- delegation ledger: entry carries the verdict and renders a compact
  acceptance segment; gateway strips caller-forged verdicts from both
  ledger entries and message metadata, like the citation verdict.
- blocking-IO anchor pins the offload (teeth proven red->green); leaf
  read errors catch only OSError/SandboxError so unexpected errors reach
  the task-tool-level isolation instead of being mislabeled.

* fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4)

- negating options: overlap with a matched criterion target is now
  checked by path/nodeid prefix, not exact token equality — excluding a
  sub-path of the criterion's selection (pytest tests --deselect
  tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds
- output attribution: any redirection token in the matched final segment
  makes the recorded tail non-attributable (> / >> / 2> are word
  characters to the parser, so redirection was invisible to the matcher)
- silent-source allowlist narrowed from any *activate suffix to the
  */bin/activate shape
- status_contract docstring: restore the shared-fixture sentence and
  note subagent_acceptance_verdict is deliberately outside the fixture
- executor: update_bash_executions publishes [] (stream carried no
  bash-family calls) instead of collapsing it into None, mirroring
  update_tool_receipts

* fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4)

- tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error"
  and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass
  shape when the exit status is swallowed (|| true) or absent; zero-error
  counts stay clean.
- file leaves: bound the deliverable read — a "wc -c" shell size probe
  answers files above 50k bytes without loading ~2x their size, honoring
  the host-bash kill switch and falling back to the full read on any
  non-integer rendering, so verdicts never get less sound.
- executor: record the exit marker text as status_marker on harvested bash
  evidence; the leaf detail now reports the marker actually seen instead of
  asserting a failure indistinguishable from the command's own trailing text.
- extend the blocking-IO anchor to drive the probe branch inside the
  offload; teeth re-verified red->green.

* fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4)

- file leaves: never read unbounded — size is established first (os.stat on
  the validated local host path, so the host-bash-disabled configuration
  needs no shell; a guarded wc -c on remote providers that renders
  missing/unreadable in its own words). Above the 50k cap the leaf answers
  from the size alone, at/below it the full read runs, and an
  unestablishable size degrades to UNVERIFIED instead of an unlimited
  fallback read.
- output attribution: source/. prefixes are never provably silent — a
  crafted */bin/activate path shape says nothing about what the script
  prints, so sourced segments can no longer lend a passing summary.
- executable identity: an explicitly path-spelled criterion now requires
  the same normalized executable path; the basename rule stays only for
  deliberately bare criterion commands.

* fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4)

- remote probe no longer runs in the sandbox's persistent shell: a fresh
  env -i /bin/sh with absolute-path stat/realpath (poisoned functions,
  aliases, PATH, exported functions, IFS, locale cannot steer it), plus a
  marker env routing AIO onto a fresh per-call bash.exec session.
- metadata-only: stat never opens content, so a FIFO deliverable cannot
  block the parent for the provider's idle timeout; non-regular files
  (fifo/dir/symlink) degrade to UNVERIFIED.
- containment canonicalized against the literal mount root: a
  final-component symlink or a swapped parent directory (root included)
  cannot redirect the check outside shared storage; unprovable layouts
  degrade to UNVERIFIED.

* fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4)

Literal-root equality made every remote file leaf permanently UNVERIFIED
on e2b and Tenki, which realize /mnt/user-data as a symlink to the home
dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink).
Containment now compares the file's realpath against the mount root's
realpath — exactly what the provider's own read path resolves, so probe
and read-back stay consistent; final-component symlinks stay rejected by
the non-dereferencing stat, and an intermediate dir-link escape under a
sane root still lands ESCAPED. The inner script is a module constant and
the suite now executes the composed probe for real against on-disk
layouts (real dir, symlinked prefix, final symlink, fifo, missing,
dir-link escape), which the canned-output stub could not see.

* fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4)

- matching: a criterion with no positional selection target (bare pytest,
  make test) stands for the runner's default selection, so ANY negating
  option (--ignore/--deselect/...) makes the recorded run a different
  selection — unprovable. The overlap guard only sees consumed criterion
  tokens, which a bare criterion does not have; scoped criteria keep the
  unrelated-exclusion behavior.
- attribution: cd is no longer blanket-silent — CDPATH makes cd print the
  resolved (subagent-chosen) destination and the pass shapes match as
  substrings, so one mkdir 'all tests passed' plus an export minted a pass
  for any quiet command. A cd argument or CDPATH= value (export or leading
  assignment) carrying any summary shape makes the segment non-silent;
  shape-free cd dir wrappers keep matching.
- docs: _truncate_bash_output states the effective 32-char floor (the
  guarantee previously read as an unconditional max_chars bound).

* fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4)

Self-audit in the shape of the last review rounds — channels the matcher
classified as accounted-for that can change what runs, narrow the
selection, or lend the summary text:

- env assignments are no longer blanket-stripped: only an allowlist of
  inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a
  matched span, and a non-allowlisted assignment in any preceding segment
  (pure-assignment or export NAME=) is state pollution — PATH redirects
  the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code,
  PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs,
  BASH_ENV runs arbitrary shell startup. All degrade to unprovable.
- runtime expansions: any span token carrying /$( )/backticks, any
  negating-option value carrying an expansion or glob (unknown excluded
  set), and any extra executed token carrying glob metacharacters
  (crafted option-looking filenames narrow invisibly) are unprovable.
  Criterion-side globs stay self-consistent (literal match).
- cd: an argument carrying a runtime expansion or glob is non-silent
  (unknown destination, unknown print); CDPATH= assignments are now
  handled as state pollution at the match layer, subsuming the
  value-shape special case.

* fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4)

- tests_passed: on a persistent-shell provider (new
  Sandbox.persistent_shell_sessions capability, set by AioSandbox) every
  leaf degrades to UNVERIFIED — any earlier call in the shared session
  could have mutated the state the clean-looking run executed in, and
  only a fresh controlled session (RFC section 6 verifier) can prove
  otherwise. The flag is read from the provider registry without
  acquiring a sandbox.
- env assignments: the allowlist is gone — no variable is provably inert
  across repositories (CI/DEBUG are routinely read by tests). The span's
  assignment prefix must equal the criterion's exactly (values included,
  order-insensitive); any assignment or export NAME= in a preceding
  segment is state pollution.
- scoping: positional targets are now read by option arity, so a path
  embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never
  counts as a selection target and an extra positional after such a
  criterion narrows the default selection it denotes.

* fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4)

* fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4)

* fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4)

* fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4)

* fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4)

* fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4)

* fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4)

* fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4)

* fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4)
2026-09-01 16:13:41 +08:00
Aari
cdc886ae85
fix(sandbox): make tool descriptions optional (#4878)
* fix(sandbox): make tool descriptions optional

* fix(sandbox): address optional description review

* test(sandbox): pin optional description contracts
2026-09-01 14:15:26 +08:00
Janlay
45adb8fbb5
perf(runtime): bound gateway memory after terminal runs (#5112)
* fix(runtime): clean up terminal run records

* perf(sandbox): bound local path caches

* perf(runtime): release terminal run cycles

* fix(runtime): address terminal cleanup review

* fix(runtime): clean up after end publish failure

* fix(runtime): guard terminal cleanup from cancellation

* fix(runtime): discard fenced journal buffers

* fix(runtime): harden abort and teardown paths
2026-09-01 10:56:43 +08:00
PeaceMaker-best
137a3cb60d
fix(authz): recheck policy before sandbox reuse (#5006)
* fix(authz): recheck policy before sandbox reuse

* fix(authz): avoid duplicate async sandbox checks

* fix(authz): scope sandbox decision across middleware

* fix(authz): construct async providers on the event loop

* test(authz): avoid cold imports under Blockbuster

---------

Co-authored-by: 嗜鵼 <hy2010hy2010@qq.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-08-30 15:29:51 +08:00
hataa
cc6a2657e7
feat(authz): enforce sandbox:execute authorization at sandbox acquisition (#4063 Phase 3) (#4911)
Sandbox is an execution environment, not a named resource: multiple tools
(bash, read_file, write_file, glob, grep, ...) depend on it, all funneled
through ensure_sandbox_initialized / ensure_sandbox_initialized_async. Gate
the single acquisition entry point (single source of truth) instead of
maintaining a sandbox-tool-name set in middleware:

- authorize_sandbox_execution helper (authz/sandbox_authz.py) checks
  authorize("sandbox", "execute", target="*") — a binary judgment
  (can this role use the sandbox at all); RBAC allow:"*"/true permits,
  allow:[]/false denies.
- lazy path: ensure_sandbox_initialized (+ async) calls the gate before
  provider.acquire.
- eager path: SandboxMiddleware.before_agent / abefore_agent call the gate
  before _acquire_sandbox.
- deny raises SandboxAuthorizationError (SandboxError subclass) which
  propagates through tool execution as a friendly ToolMessage (RFC §9:
  'not a crash').
- authorization.enabled: false is a no-op everywhere; provider errors
  follow fail_closed (deny) / fail_open (allow).

12 tests in tests/test_sandbox_authorization.py cover disabled/allow/deny/
deny-via-bool/no-policy-unrestricted/provider-error-fail-closed/open/
internal-caller + ensure_sandbox_initialized deny (never acquires) and
allow (acquires) integration paths.
2026-08-24 16:32:06 +08:00
Xinmin Zeng
42fd5aa0b5
fix(sandbox): keep skill reads on provider mappings (#4792) 2026-08-13 21:20:49 +08:00
Minh Vu
904cee4a72
fix(sandbox): push read_file ranges into sandbox reads (#3824)
* fix read_file range validation and passthrough

* chore: format test_aio_sandbox.py

* test: cover e2b sandbox read_file line-range behavior

* fix(boxlite): support ranged file reads

* fix(sandbox): reject non-positive start lines

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-30 07:53:10 +08:00
Yufeng He
8eb3be59bd
fix(sandbox): unwrap Overwrite-wrapped state in ensure_sandbox_initialized (#4429)
* fix(sandbox): unwrap Overwrite-wrapped state in ensure_sandbox_initialized

The same fork-restored wrapper that crashed after_agent also reaches the sandbox init path, where sandbox_state.get() on the Overwrite object raises AttributeError. Share the unwrap helper from #4381's follow-up module deerflow/sandbox/overwrite.py and apply it at both init sites.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

* fix(sandbox): note why discarding fork_restored at the reuse sites is safe

* fix(sandbox): unify the Overwrite unwrap helper and pin the fall-through

- middleware.py now imports unwrap_sandbox from overwrite.py instead of
  keeping a second local copy whose docstring had already drifted; the
  shared helper covers both crash forms (subscript TypeError and the
  .get()-form AttributeError)
- test the acquire fall-through: when the fork-restored id is gone from
  the provider, a fresh sandbox is acquired and the stale wrapped state
  is replaced by the plain acquired dict
- the reuse-path test now also asserts runtime.state["sandbox"] stays
  wrapped, pinning the don't-treat-as-owned contract after_agent relies on

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

* fix(sandbox): unwrap Overwrite state in the sibling sandbox readers

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

---------

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-29 07:04:26 +08:00
Ryker_Feng
aacb99cfd2
feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B) (#4501)
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B)

Removes the plaintext Lark credential mounts (appSecret + OAuth tokens)
from the sandbox container. A long-running broker sidecar owns lark-cli
and the per-user config/data dirs and serves the command surface over
Pod loopback; the sandbox gets only a forwarding shim on PATH, so the
raw credential files never exist in the sandbox filesystem.

- lark_broker.py: stdlib-only loopback broker (argv passthrough with
  shell=False, server-injected credential env, bounded I/O) + shim
  script constant + install-shim mode.
- docker/lark-cli-broker: init(install-shim) + serve image.
- provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker →
  shim init container + lark-cli-broker sidecar (config/data mounted
  sidecar-only); credentials dropped from the sandbox container;
  /api/capabilities reports lark_cli_broker_image. Broker supersedes
  the Pattern A init-container binary when both are configured.
- gateway: lark_cli_env_overlay(broker=True) omits config/data env;
  sandbox_lark_broker_active() TTL-cached mode resolver; broker added
  to sandbox_runtime_mode / readiness and the settings UI.

Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change).

Closes #4338

* fix(lark): address Pattern B broker review findings (#4501)

Follow-up to the sidecar credential broker addressing the PR #4501 review:

- shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body
  so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC
  when the sandbox image ships no python3; interpreter pinnable via
  DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since
  $0 is the bare command name when run off PATH.
- broker: drop the dead cwd payload field (broker can't see the sandbox FS) and
  document the command-surface-only / no-file-IO limitation.
- broker: return a structured 500 JSON on unexpected exec errors so the shim
  gets a meaningful message, not an opaque transport failure; set a handler
  socket timeout to bound slow/stuck connections.
- broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that
  refuses secret-dumping subcommands before spawning the binary, forwarded from
  the provisioner sidecar.
- gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache
  negatives longer (300s) so non-broker remote-provisioner users don't pay a
  latency hit; guard the mode cache with a lock; drop the dead
  _probe_provisioner_lark_cli_init_image wrapper.
- docs: remove the broken design-doc link from the broker README.

Adds tests for launcher python resolution, cwd omission, denylist enforcement,
500-on-error, hot-path probe timeout + negative caching, and provisioner
denylist-env wiring.
2026-07-28 22:54:44 +08:00
Aari
d455a1815e
fix(sandbox): allow grep to search a single file (#4512) 2026-07-28 07:49:13 +08:00
Ryker_Feng
7aa314b4c1
feat: add Lark CLI integration (#3971)
* feat: add lark cli integration

* fix: polish lark integration actions

* feat: support lark incremental permissions

* fix: detect lark authorization completion

* fix: harden lark integration install

* feat: expand lark auth scopes and reuse host auth in sandbox

Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.

Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.

* style: fix lint issues from ruff and prettier

Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.

* fix(lark): address managed integration review feedback

* fix(frontend): stabilize integrations settings e2e

* test(sandbox): isolate remote backend legacy visibility check

* test: fix backend unit failures after merge

* Harden Lark integration review fixes

* Format Lark integration E2E test

* fix(lark): harden sandbox credential exposure and status disclosure

Address willem_bd's security review on PR #3971:

- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
  into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
  GET /lark/status and the config/auth complete responses for non-admin
  callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
  because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
  pointing at the sidecar-broker follow-up (#4338).

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-26 08:09:17 +08:00
Andrew Chen
ae510cb2e8
fix(sandbox): make an empty old_str a no-op in str_replace on any file (#4256)
str_replace guards the replacement with `if old_str not in content`, which
cannot reject an empty old_str -- `"" in content` is always true. So an
empty old_str reached `str.replace("", new_str)`, which inserts new_str at
every character boundary, and the tool rewrote the file while still
returning "OK":

    old_str='', new_str='# H\n'          -> OK, file silently prepended
    old_str='', new_str='X', replace_all -> OK, 'XdXeXfX XmXaXiXnX(X)X:X\nX...'

The empty-file branch above it already handles this case (`if not content:
if not old_str: return "OK"`), and the existing test states the intent
directly: "An empty old_str is a no-op edit and remains a benign OK". That
contract just never held once the file had content.

The tool is registered by default (config.example.yaml) and its schema
declares old_str as a plain string with no minLength, so a model can emit
"" legitimately; read-before-write only compares a hash and lets it past.

Check old_str first so the no-op holds whatever the file contains. The
empty-file case folds into the same not-found branch, which keeps its
message and behaviour.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:20:54 +08:00
hataa
10890e10a8
feat(authz): propagate trusted authorization principal context (#4203) 2026-07-17 14:49:51 +08:00
Aari
bcee5a9061
refactor(sandbox): give the host→virtual output-mask regex a single owner (#4108)
* refactor(sandbox): give the host→virtual output-mask regex a single owner

Two call sites rewrite host paths back to their virtual form in text that
reaches the model — LocalSandbox._reverse_output_patterns (bash output) and
sandbox.tools._compiled_mask_patterns (glob/grep/ls results) — and each built
the same `escape(base) + boundary + tail` rule from its own copy.

That duplication has already produced two bugs: #4035 added the segment
boundary to the reverse patterns and missed the masking patterns, and #4053
had to add the same boundary to the other copy. Extract the rule into
sandbox/path_patterns.py so a third copy cannot silently disagree.

The extraction is not a pure move: the two sites disagree on the base. tools.py
derives bases from _path_variants (which yields Windows spellings) and matches
them against output whose separators it does not control, so it relaxes the
separators inside the base; LocalSandbox resolves its bases from the running
platform and must not be widened. That difference is now an explicit
`separator_agnostic` parameter rather than an accident of two implementations.
The boundary and tail constants are private: build_output_mask_pattern is the
only supported spelling, so a third site cannot import the pieces and hand-roll
a variant.

Behavior is unchanged at both sites — pinned by tests that reproduce each
pre-extraction expression byte-for-byte.

* test(sandbox): pin the base the helper must not normalize

Review notes on #4108.

The committed snapshot compares the helper against hand-copied literals of the
pre-extraction expressions, so its red-ness rests on those literals, not on the
length of _BASES -- both sides compute the same expression, and 5k fuzzed bases
produce zero byte-differences. Mutating the helper one clause at a time (12
mutations over the boundary, the tail and the escape/replace) shows the seven
committed bases catch 11: the miss is a helper that normalizes its input by
rstripping a trailing separator. Only a trailing-slash base or a Windows drive
root catches that, and Path.resolve() / str(Path(...)) strip trailing slashes,
so neither call site can produce the former. C:\ survives resolve() with its
separator intact, so that is the one base worth adding.

Also point local_sandbox's comment at path_patterns, the owner, instead of
citing _content_pattern as the class reference, and drop the rationale it now
duplicates from the owner's docstring -- a second copy of the explanation drifts
the same way the second copy of the regex did. The site-specific half stays.

Comments and test data only; no behavior change.
2026-07-13 18:34:41 +08:00
Aari
c82fba41d9
fix(sandbox): guard the command path-translation regex with a segment boundary (#4110)
replace_virtual_paths_in_command matches the virtual root with no
segment-boundary lookahead:

    re.compile(rf"{re.escape(VIRTUAL_PATH_PREFIX)}(/[^\s\"';&|<>()]*)?")

The trailing group needs a "/" to consume anything, so when the character after
/mnt/user-data is "-", ".", "_", a digit or a letter, the group matches empty
and the bare root still matches. The substitution then rewrites it to the
thread's host user-data directory and the rest of the sibling name rides along,
so a command naming a prefix sibling of the mount root is pointed at a real host
directory outside the mount contract:

    cat /mnt/user-data-backup/secret.txt  ->  cat <host>/user-data-backup/secret.txt

which reads the host file. This is the same defect as #4035 (reverse patterns)
and #4053 (masking patterns), mirrored into the virtual->host direction; it is
the last unguarded member of that family.

The boundary class mirrors LocalSandbox._content_pattern's rather than
_command_pattern's: a virtual root can legitimately be followed by ":"
(PATH-style concatenation) or ",", which the shell-oriented class rejects, so
narrowing to it would stop translating paths that translate today. "$" covers a
command ending exactly at the root.
2026-07-12 23:42:58 +08:00
明年我18
0542d3c5f3
fix(sandbox): allow bash after cwd setup failure (#4051) 2026-07-12 23:40:52 +08:00
黄云龙
d2ab5bb819
fix(sandbox): str_replace on empty file only returns OK when old_str is empty (#4079)
str_replace returned "OK" whenever the target file was empty, silently
reporting success even when the model asked to replace a non-empty string
that could not possibly be present. Only short-circuit to "OK" when old_str
is also empty; otherwise return the standard not-found error.
2026-07-12 23:04:59 +08:00
Aari
97e2268d52
fix(sandbox): stop glob/grep/ls from surfacing disabled skills' files (#4096)
* fix(sandbox): stop glob/grep/ls from surfacing disabled skills' files

The disabled-skill gate checks the path a tool is given, but ls, glob and
grep all descend from it and return other paths, so a root above a disabled
skill still serves its files. glob and grep never called the gate at all;
ls called it only on its own argument and still leaked from a category root.

Add the entry gate to glob/grep, and filter what all three return through
the existing fail-closed _is_disabled_skill_path. The verdict is memoized per
skill because ExtensionsConfig.from_file() is uncached, so a per-match check
would turn a 100-match grep into 100 config reads.

* fix(sandbox): normalize trailing slashes in the disabled-skill path check

Review follow-ups on the disabled-skill gate:

- _extract_skill_name_from_skills_path returned "" instead of None for a
  category directory carrying a trailing slash. LocalSandbox.list_dir appends
  "/" to directories, so `ls /mnt/skills` yields "/mnt/skills/public/", giving
  parts ["public", ""]. The empty name skipped the `skill_name is None`
  short-circuit and fell through to a config read, landing on the right outcome
  only because unknown skills default to enabled. Drop empty segments so a
  trailing-slash category root takes the existing category-root branch.

- ls_tool resolved the runtime user id twice per call; hoist it into a local,
  matching glob_tool/grep_tool.

- Cover the CUSTOM path: custom/legacy skills resolve their enabled state
  through the per-user _skill_states.json, a different store from the public
  skills' extensions_config.json, and no automated test exercised it.
2026-07-12 11:32:41 +08:00
Aari
1df9abc924
fix(sandbox): guard the output-masking regex with a segment boundary (#4053)
* fix(sandbox): guard the output-masking regex with a segment boundary

`_compiled_mask_patterns` builds the same class of host→virtual matcher as
`LocalSandbox._reverse_output_patterns`, but without the segment-boundary
lookahead that one carries. The trailing group needs a separator to consume
anything, so when the character after a host base is `-`, `.`, `_`, a digit or
a letter, the group matches empty and the regex still matches the bare base.

`replace_match` then takes its `matched_path == base` branch and rewrites the
sibling: with `/mnt/skills` mounted at `.../skills`, output naming a sibling
`.../skills-extra/data.txt` is handed to the model as `/mnt/skills-extra/data.txt`
— a container path forward resolution explicitly refuses to map back, so
reading it raises FileNotFoundError.

This is the sibling site of #4035, which fixed the identical bug in
`local_sandbox.py`. That PR's scope argument enumerated the prefix matchers in
that file and missed this one; `mask_local_paths_in_output` runs on every
glob/grep match and on local bash output.

The boundary class mirrors `_content_pattern`'s, not `_command_pattern`'s: this
runs over arbitrary command output, where a base can legitimately be followed
by `,`, `:` or `\`, all of which the shell-oriented class rejects.

* test(sandbox): anchor the sibling boundary at the ACP source too

The sibling-rejection cases only fed the skills source. `_compiled_mask_patterns`
builds every source's matcher in one loop, so the ACP workspace carried the same
defect: nothing maps its parent, and `/mnt/acp-workspace-backup/hello.py` is
unresolvable in both directions.

User-data is the exception and is now pinned as such: `_thread_virtual_to_actual_mappings`
also maps the virtual root `/mnt/user-data` to the three dirs' common parent, so a
sibling of `outputs` is still inside a mount and has a real virtual path — the output
is byte-identical with and without the boundary. That test is green on main; it guards
the boundary from being narrowed into one that stops translating a mapped path.

Reverting only `boundary` turns the 5 skills + 4 ACP cases red and leaves the 3
user-data cases green.
2026-07-11 23:32:11 +08:00
黄云龙
08fdf61516
fix(sandbox): handle one-sided line ranges in read_file (#4078)
* fix(sandbox): honor single-bound line ranges in read_file

read_file only sliced content when BOTH start_line and end_line were
provided, so read_file(path, start_line=100) or read_file(path,
end_line=50) silently returned the whole file. Handle one-sided ranges:
default the missing bound (start->1, end->EOF), clamp start to 1, and
return clear messages when start_line exceeds the file length or start_line
> end_line.

* address review: guard one-sided end_line <= 0 in read_file

Add symmetric end_line guard and run the inverted-range check regardless
of which bounds are explicit, so a lone end_line<=0 returns a clean
error instead of a negative-index slice. Per @willem-bd review on #4078.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-11 23:20:53 +08:00
Zheng Feng
dcb2e687d5
feat(channels): add GitHub as a webhook-driven channel (#3754)
* feat(channels): add GitHub event-driven agents (#3754)

Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation.

* fix(llm-middleware): classify bare IndexError as transient

Upstream chat providers occasionally return 200 OK with an empty
generations list (observed against Volces "coding" on
ark.cn-beijing.volces.com). When that happens,
langchain_core.language_models.chat_models.ainvoke raises
``IndexError: list index out of range`` at
``llm_result.generations[0][0].message`` and kills the run.

Treat a bare IndexError reaching the middleware as a transient
upstream-payload glitch and route it through the existing
retry/backoff path instead of failing the whole agent run. The
retry budget and backoff schedule are unchanged.

Adds three regression tests covering the classifier and both the
recover-on-retry and exhausted-retries paths.

* fix(runtime): ignore stale LLM fallback markers from prior runs

When a run on a thread ends with the LLM-error-handling middleware emitting
a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError
empty-generations classification fix lands), that message is persisted to
the thread's checkpoint as part of the messages channel. LangGraph replays
the full message history in `stream_mode="values"` chunks, so every
subsequent run on the same thread re-streams the stale fallback marker —
and the worker's chunk scanner faithfully picks it up, flipping
`RunStatus.success` to `RunStatus.error` for runs that themselves had
no LLM failure at all.

Snapshot the set of pre-existing message ids from the pre-run checkpoint
and thread it through `_extract_llm_error_fallback_message` /
`_try_extract_from_message` as a filter. Markers on history messages are
ignored; markers on fresh messages produced during this run still trip
the error path. Falls back to an empty set when the checkpointer is
absent or the snapshot can't be captured, preserving the prior behavior
on first-run / no-state paths.

Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`)
plus an integration test exercising the full `run_agent` path with a stale
history checkpointer.

* fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs

GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed
the langgraph_sdk default 300s read deadline. The manager's runs.wait
call kept an HTTP stream open for the entire run lifetime, so the long
run blew up with httpx.ReadTimeout and the outer except branch then
released the dedupe key and emitted a false 'internal error' outbound.

The GitHub channel's outbound send is log-only by design: agents post to
the issue/PR via the gh CLI in the sandbox when they choose to comment
or create a PR. There is nothing for the manager to ferry back, so the
long-poll was pure overhead.

This change adds ChannelRunPolicy.fire_and_forget (default False) and
sets it True for the github channel. When fire_and_forget is True,
_handle_chat dispatches via client.runs.create (short POST, returns
once the run is pending) instead of client.runs.wait, and skips the
response-extraction + outbound-publish block. ConflictError on a busy
thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on
the busy case is preserved for any future non-github fire-and-forget
channel.

Other (non-github) channels are unchanged: their policy defaults
fire_and_forget=False and they continue to dispatch via runs.wait.

Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget:
- Default ChannelRunPolicy.fire_and_forget is False.
- The github policy registers fire_and_forget=True.
- github inbound calls runs.create, not runs.wait, with the right kwargs.
- github inbound publishes no outbound on success.
- ConflictError from runs.create still emits THREAD_BUSY_MESSAGE.
- Non-github channels (slack) still dispatch via runs.wait.

* test(lead-agent): accept user_id kwarg in skill-policy test stubs

The two GitHub-channel tests added in #3754 stubbed
_load_enabled_skills_for_tool_policy with a lambda that only accepted
`available_skills` and `app_config`, but the real function (and its call
site in agent.py) also passes `user_id`. This raised TypeError on every
run, failing backend-unit-tests.

Add `user_id=None` to match the three sibling stubs in the same file.

* refactor(gateway): disambiguate context-key set names

The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS
shared a confusable "CONTEXT_ONLY" token in different orders, and the
first broke the _CONTEXT_<X>_KEYS pattern of its sibling
_CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit:

  _CONTEXT_INTERNAL_CALLER_KEYS  - WHO: internal callers (scheduler) only
  _CONTEXT_RUNTIME_ONLY_KEYS     - WHERE: runtime context only, never configurable

Pure rename, no behavior change.
2026-07-04 22:56:24 +08:00
Xinmin Zeng
576577bd32
feat(channels): expose IM channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID (#3926)
* feat(channels): expose channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID

IM-channel skills need the sender's platform identity (Feishu open_id,
Slack Uxxx, ...). The channel manager already writes channel_user_id into
body.context, but the Gateway whitelist dropped it. Forward it into the
runtime context only (never configurable, which is checkpointed), and have
bash_tool export it as a fixed env var through a shell-quoted command
prefix.

The identity deliberately does not ride execute_command(env=...): that
channel is reserved for request-scoped secrets, and a non-empty env
switches AioSandbox onto the bash.exec path (fresh session per call,
image >= 1.9.3 required), which would have broken every IM bash command
on older sandbox images and abandoned persistent-shell semantics on new
ones. A command-string export keeps the legacy path, stays visible in
audit logs (it is an identifier, not a secret), and gives per-call
correctness in group chats where one thread and sandbox are shared by
senders with different platform ids.

Skipped on the Windows local sandbox, whose PowerShell/cmd.exe fallback
has no POSIX export.

Part of #3914

* feat(channels): propagate channel_user_id to subagents; cap value length

Review findings from the pre-PR verification pass:

- Subagent delegation dropped the sender identity: task_tool now captures
  channel_user_id from the parent runtime context and the executor
  forwards it into the subagent's context, mirroring the guardrail
  attribution fields (user_role/oauth_*/run_id). Without this, bash
  commands delegated via task lost the group-chat sender's id.
- body.context is client-writable on web requests, so values over 256
  chars are ignored instead of bloating every command string sent to the
  sandbox.

* fix(channels): set-or-unset channel_user_id so identity is per-call regardless of AIO session persistence

Review (willem-bd): the identity export could leak across senders in a
shared group-chat AIO sandbox. The AIO no-env path reuses a persistent
shell session (the class-lock reason, #1433), and the 256-char/type guard
made some commands carry no prefix — so a dropped-id command could resolve
the id a previous sender exported.

Make per-call correctness independent of session semantics: an IM-channel
command (channel_user_id present in context) now always carries an explicit
prefix — export VAR=<quoted> for a valid id, or unset VAR for an unusable
one (empty / non-str / over the cap). Non-IM runs (no key) are untouched.
A prefix unset has none of the '& ; unset' suffix hazard raised earlier.

Verified on a real AIO 1.11.0 container: the no-id shell path auto-creates
a session per call (does not persist today), but an explicit shared session
DOES persist (export stale-A -> readback [stale-A]); the unset prefix
clears it (-> []). So the fix holds even on an image whose no-id path
persists. Regression tests cover the dropped-id group-chat window and the
non-IM passthrough.

Part of #3914

* test(channels): align channel_user_id task test with new Command return shape

The merge from main changed task_tool to return a Command(update=...)
instead of a plain string; update the assertion to extract the tool
message via the existing _task_tool_message helper, matching the sibling
tests. Fixes the CI backend-unit-tests failure introduced by the merge.
2026-07-04 21:18:11 +08:00
Zhipeng Zheng
53a80d3ad1
feat(skills): per-user custom skill isolation with sandbox mounting (#3889)
* feat(skills): per-user skill isolation (#2905)

Implement user-scoped skill storage that isolates custom skills between
users while sharing public skills globally.

Key changes:
- Add UserScopedSkillStorage class for per-user custom skill directories
- Introduce get_or_new_user_skill_storage() factory with user_id context
- Auth middleware sets effective_user_id for request-scoped storage
- Agent/prompt/middleware now use user-scoped storage and prompt cache
- Sandbox mounts user-scoped skill directories for search/read tools
- Add validate_skill_file_path() to SkillStorage for path security
- Migration script supports --all-users bulk migration
- Frontend: add editable field to Skill type, error check in enableSkill
- All skill categories can be toggled (custom skills default to enabled)
- Update skill-creator SKILL.md with isolation-aware instructions

Tests:
- Add test_user_scoped_skill_storage.py (new)
- Update all existing skill tests for user-scoped storage
- Update sandbox, client, and router tests

* fix(skills): address second-round PR review feedback (#3889)

- P1-1: restrict legacy skill mount to users without custom skills
- P1-2: fail-closed for _is_disabled_skill_path (OSError → return True)
- P2-1: AND-merge global extensions_config skill disabled state
- P2-2: atomic write for _skill_states.json (mkstemp + replace)
- P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary
- P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256)
- P2-5: clear global prompt cache on PUBLIC skill toggle
- P2-6: invalidate skill caches on client.update_skill

* fix(tests): correct tool policy test after merge

* fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage

The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers
test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module
on CI. Migrate the default to the existing deerflow.constants constant,
matching the pattern already used by LocalSkillStorage, SkillStorage, and
the durable/tool_error middlewares.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-04 13:54:04 +08:00
AochenShen99
66b9e7f212
feat: emit structured runtime metadata (follow-up#3887) (#3906)
* feat: emit structured runtime metadata

* fix: avoid subagent import cycle in replay gateway

* fix: preserve legacy subtask result parsing

* refactor: tighten runtime metadata contracts

* fix(middleware): keep recovery hint on task exception wrapper content

The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.

* fix(subagents): require lowercase hex for result_sha256 reader

Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-04 11:27:19 +08:00
Nan Gao
b81334ccfe
feat(middlewares): deterministic read-before-write version gate for file tools (#3911) (#3912)
* docs(specs): read-before-write gate design for issue #3857 output layer

* docs(plans): read-before-write gate implementation plan (#3857)

* refactor(sandbox): extract read_current_file_content helper (#3857)

* feat(middlewares): read-before-write version gate for file tools (#3857)

* test(middlewares): pin async read-before-write gate paths (#3857)

* feat(config): wire ReadBeforeWriteMiddleware into runtime chain, default on (#3857)

* docs(sandbox): document read-before-write gate in tool docstrings and AGENTS.md (#3857)

* docs(plans): align plan doc with landed config_version (17) and drop machine-specific paths

Addresses Copilot review comments on #3912.

* fix(middlewares): read-before-write gate — error-string sandboxes fail open; serialize gate+execution per path (#3912 review)

- AIO/E2B read_file reports failures (incl. missing files) as 'Error: ...'
  strings instead of raising; the gate treated that string as existing file
  content and blocked first-write creation. Error-string reads now count as
  uninspectable: gate fails open, no mark is stamped.
- LangGraph runs one AIMessage's tool calls concurrently, so two same-turn
  writes could both pass on one stale mark before either mutation landed
  (and a read mark could hash a version the model never saw). Gate check +
  tool execution (and read + mark stamping) now share a per-(thread, path)
  critical section, separate from the tool-internal file_operation_lock.
2026-07-03 17:21:04 +08:00
bjtolo
cd982d6751
fix(sandbox): normalize Windows backslash paths to forward slashes in bash commands (#3869)
* fix(sandbox): normalize Windows backslash paths to forward slashes in bash commands

On Windows, `replace_virtual_paths_in_command()` and
`LocalSandbox._resolve_paths_in_command()` resolve virtual paths
(/mnt/skills, /mnt/user-data, /mnt/acp-workspace, custom mounts) to
host paths with backslashes (C:\Users\admin\...). Bash interprets
\U, \a, \d, \s, \n, \t as escape sequences, mangling the path.

Fix: add `.replace("\\", "/")` to all path resolution callbacks so
resolved paths use forward slashes, which bash handles correctly on
all platforms and Windows APIs/Python's open() accept natively.

Fixes #3865

* test(sandbox): regression tests for Windows backslash path normalization

Covers replace_virtual_paths_in_command and LocalSandbox._resolve_paths_in_command
with Windows-style backslash paths, asserting no backslashes survive in output.

Closes #3865

* test(sandbox): fix ACP test failure, keep 6 passing regression tests

Removed test_acp_workspace_no_backslash which triggered path traversal
validation. Remaining 6 tests cover user-data (3), skills (1), and
LocalSandbox custom-mount (2) paths — all with Windows backslash paths.

* test(sandbox): restore ACP Windows path normalization coverage
2026-07-03 11:09:26 +08:00
Xinmin Zeng
09988caf95
feat(skills): request-scoped secrets for skills (closes #3861) (#3871)
* feat(sandbox): per-call env injection + platform-secret scrubbing for skills

Add an env parameter to Sandbox.execute_command (abstract + local + AIO) so request-scoped secrets can be injected into skill subprocesses, and scrub platform credentials (*KEY*/*SECRET*/*TOKEN*/*PASSWORD*/*CREDENTIAL*) from the inherited environment by default so scoped injection is not security theatre. LocalSandbox always passes an explicit scrubbed env; AioSandbox routes env-bearing commands through bash.exec(env=) on a fresh session and leaves the legacy persistent-shell path unchanged. Part of #3861. BEHAVIOR CHANGE: execute_command no longer inherits the full os.environ; Windows encoding tests updated to assert the scrubbed dict.

* feat(skills): parse required-secrets frontmatter declaration

Add SecretRequirement and Skill.required_secrets, and parse the required-secrets SKILL.md frontmatter field (a string list or {name, optional} mappings), dropping malformed entries with a warning so one bad declaration does not invalidate the skill. The declared name is both the context.secrets key and the env var injected at activation. Part of #3861.

* feat(runtime): request-scoped secret carrier (context.secrets)

Add SECRETS_CONTEXT_KEY + extract_request_secrets, centralising the context.secrets carrier contract. The existing context passthrough (build_run_config -> _build_runtime_context) already carries the sub-key to runtime.context without mirroring it into configurable; characterization tests lock that behaviour. Part of #3861.

* feat(skills): inject declared secrets at slash-activation into bash env

Binding point A: when a skill is slash-activated, SkillActivationMiddleware resolves its declared required-secrets against the request's context.secrets and writes the per-run injection set to runtime.context. The bash tool forwards that set to execute_command(env=). A skill cannot harvest a host platform credential (is_host_platform_secret guard, cf. GHSA-rhgp-j443-p4rf), and injected values are redacted from bash output (mask_secret_values) so an echoed secret never re-enters the prompt/trace. Part of #3861.

* test(skills): lock the five secret leak surfaces + add trace redaction helper

Regression tests assert the secret value is absent from all five surfaces: prompt (activation message), checkpoint (graph state vs context separation), audit (journal records names only), trace (metadata builder never copies context; never mirrored to configurable), and stdout (mask_secret_values). Add redact_secret_context_keys as a defensive helper for any context serialization. Part of #3861.

* docs(backend): document request-scoped secrets for skills

Add Request-Scoped Secrets subsection (Skills) + env policy note (Sandbox) and the execute_command(env=) signature change, per the doc-sync policy. Part of #3861.

* fix(skills): close gaps found by end-to-end verification of request-scoped secrets

Real-gateway e2e + independent review of #3861 surfaced three defects, now fixed:

1. Slash activation never fired in the live chain. InputSanitizationMiddleware
   wraps user input in BEGIN/END markers before SkillActivationMiddleware sees it,
   and the original text was only preserved when an upload or IM channel set it.
   For a plain text message the slash command became undetectable, so no secret
   was ever resolved. Fix: the sanitizer now setdefaults the pre-wrap text into
   ORIGINAL_USER_CONTENT_KEY (additive; sanitization behaviour unchanged), so
   slash activation works for all messages. Pre-existing latent bug surfaced here.

2. The raw request config (with context.secrets) was persisted to runs.kwargs_json
   and echoed by the run API (RunResponse.kwargs). Fix: redact_config_secrets()
   strips secret-bearing context keys from the persisted/echoed copy in start_run;
   the live config that drives the run keeps them. build_run_config now also sets
   configurable.thread_id on the context path (the checkpointer requires it).

3. Connection-string credentials (DATABASE_URL, REDIS_URL, SENTRY_DSN, GH_PAT, ...)
   were not scrubbed from the inherited sandbox env. Fix: env_policy adds a *DSN*
   pattern plus an explicit connection-string denylist (no blanket *URL* — benign
   service URLs stay readable).

Verified end-to-end via a real gateway run (real LLM + skill activation + bash):
the secret reaches the sandbox subprocess and appears in NONE of prompt, trace,
checkpoint, audit, stdout, runs.kwargs_json, or the run API. Part of #3861.

* docs(backend): document the env scrub, persistence redaction, and sanitizer interaction

Sync the Request-Scoped Secrets section with the verification-driven fixes: inherited-env scrub (incl. connection-string denylist), run-record/run-API redaction as the 6th sealed leak surface, and the sanitizer preserving original content so slash activation fires. Part of #3861.

* fix(skills): inject caller secret over scrubbed host value; drop redundant host-name guard

A real-world demo (a skill calling a third-party cloud API with a request-scoped
key) exposed that the is_host_platform_secret guard was both wrong and harmful:
it refused to inject a caller-supplied secret whenever a same-named variable
existed in the Gateway env — which is exactly the #3861 use case (a per-user key
overriding a shared platform key). The guard was also redundant: build_sandbox_env
already scrubs secret-looking names from the inherited env before injection, so a
skill can never read a host credential — it only ever receives the caller's value.

Remove the guard; the injected (caller) value simply wins over the scrubbed host
value. Verified end-to-end: the agent called the real cloud API successfully with
the caller's key, the host's same-named key was scrubbed and never used, and the
caller's key leaked to none of the surfaces. Part of #3861.

* fix(skills): address review on request-scoped secrets (#3861)

Review fixes from PR #3871:

- E2BSandbox.execute_command now accepts env/timeout and routes them to
  commands.run(envs=, timeout=). The bash tool passes env= unconditionally,
  so the prior signature (command only) raised TypeError on every e2b bash
  call and broke e2b deployments entirely. env=None stays backward-compatible.
- SkillActivationMiddleware clears the active-secret set before resolving each
  activation, so a later skill in the same run never inherits an earlier
  skill's injection set (the #3861 contract: a skill only receives what the
  caller supplied AND that skill declared).
- AioSandbox env path uses a dedicated _DEFAULT_HARD_TIMEOUT — bash.exec exposes
  no idle/no-change timeout, so the prior reuse of the legacy idle constant
  conflated wall-clock vs idle semantics. The env path also retries on the
  ErrorObservation signature now, sharing the legacy persistent-shell recovery
  contract.
- mask_secret_values skips values below a minimum length floor so a short
  declared secret (e.g. "42") cannot shred unrelated bytes (exit codes,
  timestamps, sizes) of tool output. The secret is still injected into the
  subprocess; only the output mask skips it.

session_id reuse on the env path is intentionally NOT added: a shared session
could let request-scoped secrets ride the session env into later commands,
which the SDK does not contractually forbid. The fresh-session choice matches
the LocalSandbox model (each call is a fresh subprocess); the trade-off
(consecutive env-bearing calls do not share cwd/venv/exports) is documented on
_execute_with_env.
2026-07-03 07:51:22 +08:00
DanielWalnut
fe82552023
fix(sandbox): stop blocking bash commands (e.g. servers) from hanging the turn (#3864)
* fix(sandbox): stop blocking bash commands from hanging the turn

Starting a server through the host bash tool (e.g. `python -m http.server`)
could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command`
used `subprocess.run(capture_output=True)`, whose captured pipes are inherited
by any process the command spawns — so a backgrounded long-lived process
(`server &`) keeps the read end open and blocks `communicate()` until the
timeout fires, even though the foreground command already returned. Commands
that read stdin blocked the same way, and on timeout only the direct child was
killed, leaving orphaned process groups.

Rework the POSIX path to capture stdout/stderr via temp files instead of pipes,
take stdin from /dev/null, and run the command in its own session/process group:

- Backgrounded long-lived processes (servers) now return immediately while the
  process keeps running.
- A command reading stdin gets immediate EOF instead of blocking.
- A genuinely blocking foreground command is bounded by a configurable
  wall-clock timeout; on timeout the whole process group is killed and the agent
  gets an explanatory notice telling it to background long-lived processes.

The timeout is configurable via `sandbox.bash_command_timeout` (default 600).
The Windows path is unchanged. Adds focused regression tests and updates docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): instruct the agent to background long-lived processes

The code fix bounds a foreground server with a timeout, but the turn still
waits the full timeout before the run continues. Add the prompt-side half:
the bash tool description now tells the model to ALWAYS start long-lived
processes (e.g. web servers) in the background with output redirected, so the
tool returns immediately. The timeout notice points at the same readable
workspace log path. Pins the guidance with a test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): make fallback-kill exception explicit and observable

Address automated review: the inner `except OSError: pass` in
_terminate_process_group silently swallowed the case where the direct-child
fallback kill found the process already gone. Make the intent explicit with a
comment and a debug log instead of a bare pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): address bash timeout review feedback

* fix(sandbox): document fd cleanup races

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 14:55:31 +08:00
Eilen Shin
67dd75db62
chore: remove unused private helper functions (dead code) (#3749)
These three module-private helpers have no callers anywhere in the repo
(verified by a repo-wide identifier scan + direct greps; each appears only
at its own definition):

- agents/memory/updater.py: _create_empty_memory (a no-caller wrapper around
  storage.create_empty_memory, which is used directly elsewhere)
- runtime/runs/worker.py: _extract_human_message (also drops the now-unused
  HumanMessage TYPE_CHECKING import and TYPE_CHECKING from typing)
- sandbox/tools.py: _looks_like_unsafe_cwd_target (cwd safety is enforced via
  _is_allowed_local_bash_absolute_path)

Underscore-prefixed = module-internal, so zero references is conclusive.
Route handlers and @compiles-registered functions were excluded as false
positives. Public-surface candidates are noted in the issue, not touched here.

Fixes #3748.
2026-06-24 09:21:50 +08:00
Huixin615
4f192cb469
fix: align sandbox artifact mounts with channel user (#3729)
* fix: align sandbox artifact mounts with channel user

* fix: clean up sandbox user-scoped ids
2026-06-23 23:13:12 +08:00
Eilen Shin
f7f2a500d1
perf(sandbox): cache local-path masking patterns instead of recompiling per match (#3713)
mask_local_paths_in_output runs once per glob/grep match (tools.py:1540,
:1625) and once per bash/ls output, and each call rebuilt every skills /
ACP / user-data masking regex from scratch — Path.resolve() syscalls +
re.escape + re.compile. A grep returning up to 100 matches recompiled the
same patterns 100x.

Compile the host->virtual patterns once per ordered source set via
functools.lru_cache (keyed on the config-stable + per-thread (host,
virtual) pairs, so it self-invalidates when they change) and collapse the
three byte-identical replacer blocks into one applier. Behavior is
unchanged: same patterns, same application order, same per-thread mappings.

Verified behavior-preserving with an equivalence harness (original vs
refactored logic over 54 inputs incl. slash-style variants, base-only
matches, subpaths, overlapping mappings, missing skills/ACP, thread_data
None) -> 0 mismatches. Added cache/consistency tests alongside the
existing masking tests.

Fixes #3712.
2026-06-23 08:32:51 +08:00
Xinmin Zeng
97dd9ecf73
fix(sandbox): stop flagging string-literal path fragments as unsafe absolute paths (#3623)
* fix(sandbox): stop flagging string-literal path fragments as unsafe paths

The host-bash absolute-path guard scans the raw command string, so /segment
sequences inside string literals, f-strings, and templates were treated as
absolute path arguments and rejected — e.g. python -c "print(f'/端口{port}')"
or a REST template /devices/{id}/port. Whether a fragment tripped the guard
depended on the character right before the slash (a word char suppressed the
match), so the same literal could pass or fail unpredictably, pushing the model
into retry loops that bloat context and wall-clock time.

Exempt matches carrying non-ASCII characters or format braces: real host paths
a command would open contain neither, so these are text, not paths. The guard
is best-effort (not a security boundary), and plain ASCII host paths like
/etc/passwd — including ones written inside a code string such as
open('/etc/passwd') — stay rejected.

* fix(sandbox): only exempt identifier-template braces, not bash brace expansion

The literal-fragment exemption exempted any path fragment containing { or },
which let bash brace expansion (cat /etc/{passwd,shadow}) and ${VAR} expansion
reconstitute real host paths past validate_local_bash_command_paths. Tighten
the brace branch to only exempt fragments where every {...} block is a single
identifier-like placeholder (/devices/{id}/port, f-string /{port}); reject
${VAR} shell-variable expansion. Add parametrized regression tests for the
brace-expansion and shell-var bypasses.
2026-06-18 09:27:05 +08:00
Xinmin Zeng
6a4a30fa2b
fix(sandbox): return actionable hint when read_file hits a binary file (#3624)
read_file decodes with UTF-8. Binary uploads (.xlsx, images, ...) raise
UnicodeDecodeError in the sandbox layer. UnicodeDecodeError is a ValueError
subclass, not an OSError, so it bypassed the typed handlers and fell through
to the generic except, surfacing a vague "Unexpected error reading file"
message. The model could not tell the file was binary, so it retried
read_file instead of switching to bash + pandas/openpyxl, burning LLM
round-trips and bloating context with repeated failures.

Add a dedicated UnicodeDecodeError handler that tells the model the file is
binary and to use bash with a suitable library (or view_image for images).
2026-06-17 21:11:44 +08:00
Huixin615
88e36d9686
fix(#3189): prevent write_file streaming timeout on long reports (#3195)
* fix(#3189): prevent write_file streaming timeout on long reports

Adds a layered defense against StreamChunkTimeoutError caused by oversized
single-shot write_file tool calls:

- factory: default stream_chunk_timeout to 240s for OpenAI-compatible
  clients (overridable via ModelConfig.stream_chunk_timeout in config.yaml)
- sandbox/tools: server-side 80 KB length guard on non-append write_file
  calls (configurable via DEERFLOW_WRITE_FILE_MAX_BYTES env var, 0 disables);
  rejects oversized payloads with a structured error pointing the model at
  str_replace or append=True
- middleware: classify StreamChunkTimeoutError as transient but cap retries
  at 1 via per-exception _RETRY_BUDGET_OVERRIDES (same-payload retry on a
  chunk-gap timeout buffers the same way upstream; full 3-attempt loop
  would stack 6-12 min of dead air)
- middleware: surface an actionable user-facing message for stream-drop
  exceptions instead of leaking the raw langchain stack
- prompts: add a routing-style File Editing Workflow hint to both lead_agent
  and general_purpose subagent prompts, pointing the model at str_replace
  for incremental edits (mirrors Claude Code's Edit / Codex's apply_patch)
- tests: behavioural coverage for size guard, retry budget override,
  stream-drop user message, factory default injection

Refs #3189

* fix(#3189): drop stream_chunk_timeout for non-OpenAI providers

Address CR feedback on PR #3195:

- factory: pop `stream_chunk_timeout` from kwargs for any model_use_path other than `langchain_openai:ChatOpenAI` instead of returning early. `ModelConfig.stream_chunk_timeout` is part of the shared schema, so a user-supplied value on a non-OpenAI provider would otherwise be forwarded to its constructor and raise `TypeError: unexpected keyword argument`.

- factory: rewrite docstring to describe the actual `exclude_none=True` behaviour (explicit null is excluded and falls back to the default) instead of the misleading "None falling out via exclude_none=True keeps its value".

- tests: add regression coverage asserting the kwarg is stripped before reaching a non-OpenAI provider's constructor.

Refs: bytedance#3189

* fix(#3189): restrict stream-drop user copy to StreamChunkTimeoutError only

Per CR on #3195: narrow _STREAM_DROP_EXCEPTIONS to StreamChunkTimeoutError. Generic httpx RemoteProtocolError / ReadError fall back to the standard 'temporarily unavailable' copy, since they routinely fire on transient network blips where the 'split the output' guidance is misleading. Retry/backoff classification is unchanged — both remain transient/retriable. Tests updated to reflect new copy, plus a symmetric regression test for ReadError.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-06-07 17:47:11 +08:00
Lawrance_YXLiao
1c5c585741
fix(runtime): bound write_file execution-failure observations (#3133)
* fix(runtime): bound write_file execution-failure observations

* fix(runtime): preserve write_file error prefixes

* test(runtime): trim write_file prefix assertions

* refactor(runtime): drop redundant exception suffix for permission/directory write errors

Address Copilot review on #3133: the PermissionError and IsADirectoryError
branches now return self-contained, non-redundant messages (e.g.
"Error: Permission denied writing to file: /mnt/...") via direct
truncation, instead of going through _format_write_file_error which
appended a duplicate ": PermissionError: permission denied" suffix.

OSError, SandboxError and the generic Exception branches keep the
unified "Failed to write file '{path}': {ExceptionType}: {detail}"
format so the model still sees a stable, machine-readable error class.

Removes the now-unused message= parameter from _format_write_file_error,
keeping a single code path. Truncation contract (<= 2000 chars) and
host-path sanitization unchanged.

* fix(runtime): handle write_file sandbox init errors

Initialize the requested path before sandbox setup so early sandbox failures can still return a bounded write_file error.

Add a regression test for sandbox initialization failures.

* style(test): format sandbox security tests
2026-05-21 20:35:46 +08:00
AochenShen99
8b697245eb
fix(sandbox): avoid blocking sandbox readiness polling (#2822)
* fix(sandbox): offload async sandbox acquisition

Run blocking sandbox provider acquisition through the async provider hook so eager sandbox setup does not stall the event loop.

* fix(sandbox): add async readiness polling

Introduce an async sandbox readiness poller using httpx and asyncio.sleep while preserving the existing synchronous API.

* test(sandbox): cover async readiness polling

Lock in non-blocking readiness behavior so the async helper does not regress to requests.get or time.sleep.

* fix(sandbox): allow anonymous backend creation

* fix(sandbox): use async readiness in provider acquisition

* fix(sandbox): use async acquisition for lazy tools

* test(sandbox): cover anonymous remote creation

* fix(sandbox): clamp async readiness timeout budget

* fix(sandbox): offload async lock file handling

* fix(sandbox): delegate async middleware fallthrough

* docs(sandbox): document async acquisition path

* fix(sandbox): offload async sandbox release

* docs(sandbox): mention async release hook

* fix(sandbox): address async lock review

Reduce duplicate sync/async sandbox acquisition state handling and move async thread-lock waits onto a dedicated executor with cancellation-safe cleanup.

* chore: retrigger ci

Retrigger GitHub Actions after upstream main fixed the stale PR merge lint failure.

* test(sandbox): sync backend unit fixtures

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-21 14:44:34 +08:00
Xinmin Zeng
380255f722
fix(sandbox): uphold /mnt/user-data contract at Sandbox API boundary (#2873) (#2881)
* fix(sandbox): uphold /mnt/user-data contract at Sandbox API boundary (#2873)

LocalSandboxProvider used a process-wide singleton with no /mnt/user-data
mapping, forcing every caller to translate virtual paths via tools.py
before invoking the public Sandbox API. AIO already exposes /mnt/user-data
natively (per-thread bind mounts), so the same code path behaved
differently across implementations — and direct callers like
uploads.py:282 / feishu.py:389 only worked thanks to the
`uses_thread_data_mounts` workaround flag.

Switch the provider to a dual-track cache: keep the `"local"` singleton
for legacy acquire(None) callers (backward-compat for existing tests and
scripts), and create a per-thread LocalSandbox with id `"local:{tid}"`
for acquire(thread_id). Each per-thread instance carries PathMapping
entries for /mnt/user-data, its three subdirs, and /mnt/acp-workspace,
mirroring how AioSandboxProvider mounts those paths into its container.

is_local_sandbox() now recognises both id formats. `_agent_written_paths`
becomes per-thread (it was a process-wide set that leaked across
threads — a latent isolation bug also fixed by this change).

Verified via TDD: a new contract test suite hits the public Sandbox API
directly (write/read/list/exec/glob/grep/update + per-thread isolation +
lifecycle). 3212 backend tests still pass, ruff is clean.

* fix(sandbox): address Copilot review on #2881

Three follow-ups from Copilot's review of the LocalSandboxProvider refactor:

1. Synchronisation: ``acquire`` / ``get`` / ``reset`` mutated the cache without
   any lock, so concurrent acquire of the same ``thread_id`` could create two
   ``LocalSandbox`` instances and lose one's ``_agent_written_paths`` state.
   Add a provider-wide ``threading.Lock`` (matching ``AioSandboxProvider``) and
   build per-thread mappings outside the lock to avoid holding it during the
   ``ensure_thread_dirs`` filesystem touch.

2. Memory bound: ``_thread_sandboxes`` grew monotonically. Replace the plain
   dict with an ``OrderedDict`` LRU capped at
   ``DEFAULT_MAX_CACHED_THREAD_SANDBOXES`` (256, configurable per provider
   instance). ``get`` promotes touched threads to the MRU end so an active
   thread isn't evicted under load. Eviction is graceful: the next ``acquire``
   rebuilds a fresh sandbox; only ``_agent_written_paths`` (reverse-resolve
   hint) is lost.

3. Docs: update ``CLAUDE.md`` to reflect the new per-thread architecture, the
   LRU cap, and that ``is_local_sandbox`` recognises both id formats.

New regression tests:
- Concurrent ``acquire("alpha")`` from 8 threads yields a single instance
  (slow-init injection forces the race window wide open).
- Concurrent ``acquire`` of distinct thread_ids yields distinct instances.
- The cache evicts the least-recently-used thread once the cap is exceeded.
- ``get`` promotes recency so a polled thread survives a later acquire-storm.
2026-05-17 08:26:04 +08:00
Maz Benoscar
30a5846219
fix(tools): make write_file append discoverable in model-facing schema (#2843)
* fix: make tool argument behavior discoverable

The write_file tool already supported append=false by default with append=true for end-of-file writes, but the parsed docstring did not describe append in the model-facing schema. This records the overwrite default and append path in the tool description, adds resilient schema regression coverage, and keeps backend sandbox docs aligned.

The regression now also checks that every public parameter in the existing tool schema test matrix has a description. Enabling docstring parsing on setup_agent and update_agent fills the two existing gaps with their existing Args docs instead of duplicating descriptions elsewhere.

Constraint: Issue #2831 asks for a small docstring/schema discoverability fix without changing runtime file-writing behavior
Rejected: Changing write_file defaults | would alter existing overwrite semantics and broaden the fix beyond schema discoverability
Rejected: Exact phrase assertions | too brittle for future docstring rewording while testing the same behavior
Confidence: high
Scope-risk: narrow
Directive: Keep model-facing tool parameters documented through parsed docstrings or equivalent schema descriptions
Tested: cd backend && uv run pytest tests/test_setup_agent_tool.py tests/test_update_agent_tool.py tests/test_tool_args_schema_no_pydantic_warning.py tests/test_sandbox_tools_security.py::test_str_replace_and_append_on_same_path_should_preserve_both_updates -q
Tested: cd backend && uv run ruff check packages/harness/deerflow/sandbox/tools.py packages/harness/deerflow/tools/builtins/setup_agent_tool.py packages/harness/deerflow/tools/builtins/update_agent_tool.py tests/test_tool_args_schema_no_pydantic_warning.py
Not-tested: Full backend test suite
Co-authored-by: OmX <omx@oh-my-codex.dev>

* Fix the lint error

---------

Co-authored-by: OmX <omx@oh-my-codex.dev>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-10 23:09:03 +08:00
He Wang
7de9b5828b
fix(tools): introduce Runtime type alias to eliminate Pydantic serialization warning (#2774)
* fix(tools): introduce Runtime type alias to eliminate Pydantic serialization warning

Add deerflow/tools/types.py with:

    Runtime = ToolRuntime[dict[str, Any], ThreadState]

Replace every runtime: ToolRuntime[ContextT, ThreadState] and
runtime: ToolRuntime[dict[str, Any], ThreadState] annotation in
sandbox/tools.py, present_file_tool.py, task_tool.py, view_image_tool.py,
and skill_manage_tool.py with the new Runtime alias.

The unbound ContextT TypeVar (default None) caused
PydanticSerializationUnexpectedValue warnings on every tool call because
LangChain's BaseTool._parse_input calls model_dump() on the auto-generated
args_schema while DeerFlow passes a dict as runtime context.
Binding the context to dict[str, Any] aligns Pydantic's serialization
expectations with reality and removes the noise from all run modes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tools): extend Runtime alias to setup_agent and update_agent tools

Replace bare ToolRuntime annotations in setup_agent_tool.py and
update_agent_tool.py with the shared Runtime alias introduced in the
previous commit, and add both tools to the Pydantic serialization
warning regression test (13 cases total).

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(tools): loosen Pydantic warning filter to avoid version-specific format

Replace the brittle "field_name='context'" substring check with a looser
"context" match so the assertion stays valid if Pydantic changes its
internal warning format across versions.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(tools): simplify warning filter and clean up docstring

Remove the "context" substring condition from the Pydantic warning
filter — asserting that no PydanticSerializationUnexpectedValue fires
at all is both simpler and more comprehensive, since the test payload
contains only the tool's own args plus runtime.

Also update the module docstring to remove the version-specific warning
format example that was inconsistent with the looser filter.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-08 14:50:33 +08:00
Willem Jiang
844ad8e528
Merge branch 'main' into release/2.0-rc 2026-04-28 15:44:02 +08:00
DanielWalnut
6bd88fe14c
fix(sandbox): block host bash traversal escapes (#2560)
* fix(sandbox): block host bash traversal escapes

Fixes #2535

* fix(sandbox): harden local bash path guards

* fix(sandbox): avoid bash cd argument false positives

* Fix the lint error

Add function to resolve and validate user data path.

* Fix the lint error

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-28 12:18:41 +08:00
DanielWalnut
af8c0cfb78
fix(harness): constrain view_image to thread data paths (#2557)
* fix(harness): constrain view_image to thread data paths

Fixes #2530

* fix(harness): address view_image review findings

* style(harness): format view_image changes

* fix(harness): address view_image review comments
2026-04-28 11:13:17 +08:00