* fix(sandbox): project enabled skills into sandbox views
* fix(skills): keep projection mutations consistent
* fix(skills): fail closed on projection errors
* fix(skills): isolate per-scope failures during boot projection rebuild
rebuild_all_skill_projections() propagated any exception from the public
rebuild or from a single user's rebuild straight out of the gateway
lifespan startup, uncaught. A single broken user directory (bad
permissions, corrupted _skill_states.json, unreadable content) would
therefore abort gateway boot for every user, not just that one -
_rebuild_*_locked already fails closed internally (clears the view and
re-raises), so the boot loop only needed to stop treating that re-raise
as fatal.
Each scope's rebuild now fails closed independently and boot continues;
a scope left empty by a boot failure self-heals on the next sandbox
acquire via ensure_skill_projections().
Also patches deerflow.skills.projection.rebuild_all_skill_projections in
the memory-flush lifespan test fixture, matching the two sibling
fixtures in the same file — this call is now on the lifespan startup
path and the fixture's minimal SimpleNamespace config predates it.
* test(skills): update authz test for the projection-aware public toggle
_persist_shared_skill_state (introduced earlier in this branch) reads
the shared extensions_config.json fresh from disk under the projection
lock instead of through the cached get_extensions_config() singleton -
that's the whole point of the fix (stale worker caches must not clobber
another worker's concurrent update). The name no longer exists on the
skills router module, so the test's monkeypatch of it started raising
AttributeError instead of exercising the endpoint.
The mock storage in this test isn't a real LocalSkillStorage instance,
so _persist_shared_skill_state's projection-mutation branch is already
skipped (nullcontext) and it falls back to a fresh ExtensionsConfig()
for the nonexistent tmp config_path - no replacement monkeypatch needed.
* fix(sandbox): make skill projection ensure best-effort in acquire
acquire() called _ensure_skills_projection() directly, outside any
try/except, in both LocalSandboxProvider and AioSandboxProvider. Every
other skill-mount setup path in these providers has always caught
exceptions and logged a warning rather than failing sandbox acquire
outright (e.g. when config.yaml can't be resolved) - these two new call
sites broke that contract, so any projection failure (including simply
not having a config.yaml, as in CI's test environment) now failed
acquire() itself instead of just leaving skill mounts off.
_ensure_skills_projection now catches its own exceptions and returns
None; both providers' callers already tolerate that (a None projection
skips the skill-specific mounts, matching the existing degrade path)
after making _append_public_skill_mapping and the custom/legacy mount
block in LocalSandboxProvider explicitly None-safe.
Caught by running the full suite with config.yaml removed, matching
CI's environment - not caught locally because a real config.yaml was
present, masking the failure.
* fix(sandbox): make E2B skill projection mounts best-effort
_skill_projection_mounts called ensure_skill_projections with no guard,
unlike Local/AIO's _ensure_skills_projection. A raise propagated out of
_apply_mounts before the configured-mounts loop ran, so a skills
projection failure dropped the operator's own configured mounts too -
only caught by create()'s outer warning, with nothing applied at all.
Swallow here and return an empty mount list on failure, matching the
Local/AIO pattern: still fail-closed for skills, but no longer widens
the blast radius to unrelated configured mounts.
Review feedback from PR #4178.
* docs(skills): document projection trade-offs flagged in review
- _update_tree_digest: note the metadata-only (not content) hashing
trade-off and why runtime writes through this codebase are still
covered regardless (rebuild-under-lock + rename always changes inode).
- LocalSandboxProvider.acquire: note the acquire-time self-heal cost
(cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift).
- skill_projection_mutation: drop the no-op except-Exception-then-raise;
a raise from the mutation already propagates past the yield with the
view left cleared, no explicit re-raise needed.
- provisioner README: spell out that hostPath skills volumes require
the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node
or shared storage), and that the custom/legacy volumes' hostPath type
Directory (not DirectoryOrCreate) makes a violation of that assumption
a visible Pod-creation failure instead of a silent empty mount.
Review feedback from PR #4178.
* fix(skills): lazily repair user projections
* fix(skills): close projection review gaps
* fix(skills): refresh user projection enable state
* fix(skills): close projection review follow-ups
* fix(skills): preserve state across projection writes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): unwrap Overwrite-wrapped state in ensure_sandbox_initialized
The same fork-restored wrapper that crashed after_agent also reaches the sandbox init path, where sandbox_state.get() on the Overwrite object raises AttributeError. Share the unwrap helper from #4381's follow-up module deerflow/sandbox/overwrite.py and apply it at both init sites.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(sandbox): note why discarding fork_restored at the reuse sites is safe
* fix(sandbox): unify the Overwrite unwrap helper and pin the fall-through
- middleware.py now imports unwrap_sandbox from overwrite.py instead of
keeping a second local copy whose docstring had already drifted; the
shared helper covers both crash forms (subscript TypeError and the
.get()-form AttributeError)
- test the acquire fall-through: when the fork-restored id is gone from
the provider, a fresh sandbox is acquired and the stale wrapped state
is replaced by the plain acquired dict
- the reuse-path test now also asserts runtime.state["sandbox"] stays
wrapped, pinning the don't-treat-as-owned contract after_agent relies on
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(sandbox): unwrap Overwrite state in the sibling sandbox readers
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B)
Removes the plaintext Lark credential mounts (appSecret + OAuth tokens)
from the sandbox container. A long-running broker sidecar owns lark-cli
and the per-user config/data dirs and serves the command surface over
Pod loopback; the sandbox gets only a forwarding shim on PATH, so the
raw credential files never exist in the sandbox filesystem.
- lark_broker.py: stdlib-only loopback broker (argv passthrough with
shell=False, server-injected credential env, bounded I/O) + shim
script constant + install-shim mode.
- docker/lark-cli-broker: init(install-shim) + serve image.
- provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker →
shim init container + lark-cli-broker sidecar (config/data mounted
sidecar-only); credentials dropped from the sandbox container;
/api/capabilities reports lark_cli_broker_image. Broker supersedes
the Pattern A init-container binary when both are configured.
- gateway: lark_cli_env_overlay(broker=True) omits config/data env;
sandbox_lark_broker_active() TTL-cached mode resolver; broker added
to sandbox_runtime_mode / readiness and the settings UI.
Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change).
Closes#4338
* fix(lark): address Pattern B broker review findings (#4501)
Follow-up to the sidecar credential broker addressing the PR #4501 review:
- shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body
so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC
when the sandbox image ships no python3; interpreter pinnable via
DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since
$0 is the bare command name when run off PATH.
- broker: drop the dead cwd payload field (broker can't see the sandbox FS) and
document the command-surface-only / no-file-IO limitation.
- broker: return a structured 500 JSON on unexpected exec errors so the shim
gets a meaningful message, not an opaque transport failure; set a handler
socket timeout to bound slow/stuck connections.
- broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that
refuses secret-dumping subcommands before spawning the binary, forwarded from
the provisioner sidecar.
- gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache
negatives longer (300s) so non-broker remote-provisioner users don't pay a
latency hit; guard the mode cache with a lock; drop the dead
_probe_provisioner_lark_cli_init_image wrapper.
- docs: remove the broken design-doc link from the broker README.
Adds tests for launcher python resolution, cwd omission, denylist enforcement,
500-on-error, hot-path probe timeout + negative caching, and provisioner
denylist-env wiring.
* fix(sandbox): unwrap Overwrite-wrapped sandbox state in after_agent
Fork-restored checkpoints can deliver the sandbox channel still wrapped
in langgraph.types.Overwrite: the rollback restore applies replace-style
writes through a state-mutation graph in delta checkpoint mode, and
after_agent/aafter_agent then crash subscripting the wrapper
("TypeError: 'Overwrite' object is not subscriptable") on the next
sandbox tool run in the forked conversation. Unwrap before reading the
sandbox id, and pin both hooks against an Overwrite-wrapped state.
Refs #4380 (bug 1 of 2; the history-loss half is a separate display path)
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(sandbox): don't release fork-restored parent sandboxes
The Overwrite-wrapped value replays the parent thread's sandbox state, so releasing it from the forked run would evict the parent's warm sandbox. _unwrap_sandbox now reports the wrapped form, and both after_agent hooks skip the release for it while keeping the normal path unchanged.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add lark cli integration
* fix: polish lark integration actions
* feat: support lark incremental permissions
* fix: detect lark authorization completion
* fix: harden lark integration install
* feat: expand lark auth scopes and reuse host auth in sandbox
Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.
Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.
* style: fix lint issues from ruff and prettier
Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.
* fix(lark): address managed integration review feedback
* fix(frontend): stabilize integrations settings e2e
* test(sandbox): isolate remote backend legacy visibility check
* test: fix backend unit failures after merge
* Harden Lark integration review fixes
* Format Lark integration E2E test
* fix(lark): harden sandbox credential exposure and status disclosure
Address willem_bd's security review on PR #3971:
- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
GET /lark/status and the config/auth complete responses for non-admin
callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
pointing at the sidecar-broker follow-up (#4338).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
str_replace guards the replacement with `if old_str not in content`, which
cannot reject an empty old_str -- `"" in content` is always true. So an
empty old_str reached `str.replace("", new_str)`, which inserts new_str at
every character boundary, and the tool rewrote the file while still
returning "OK":
old_str='', new_str='# H\n' -> OK, file silently prepended
old_str='', new_str='X', replace_all -> OK, 'XdXeXfX XmXaXiXnX(X)X:X\nX...'
The empty-file branch above it already handles this case (`if not content:
if not old_str: return "OK"`), and the existing test states the intent
directly: "An empty old_str is a no-op edit and remains a benign OK". That
contract just never held once the file had content.
The tool is registered by default (config.example.yaml) and its schema
declares old_str as a plain string with no minLength, so a model can emit
"" legitimately; read-before-write only compares a hash and lets it past.
Check old_str first so the no-op holds whatever the file contains. The
empty-file case folds into the same not-found branch, which keeps its
message and behaviour.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(sandbox): give the host→virtual output-mask regex a single owner
Two call sites rewrite host paths back to their virtual form in text that
reaches the model — LocalSandbox._reverse_output_patterns (bash output) and
sandbox.tools._compiled_mask_patterns (glob/grep/ls results) — and each built
the same `escape(base) + boundary + tail` rule from its own copy.
That duplication has already produced two bugs: #4035 added the segment
boundary to the reverse patterns and missed the masking patterns, and #4053
had to add the same boundary to the other copy. Extract the rule into
sandbox/path_patterns.py so a third copy cannot silently disagree.
The extraction is not a pure move: the two sites disagree on the base. tools.py
derives bases from _path_variants (which yields Windows spellings) and matches
them against output whose separators it does not control, so it relaxes the
separators inside the base; LocalSandbox resolves its bases from the running
platform and must not be widened. That difference is now an explicit
`separator_agnostic` parameter rather than an accident of two implementations.
The boundary and tail constants are private: build_output_mask_pattern is the
only supported spelling, so a third site cannot import the pieces and hand-roll
a variant.
Behavior is unchanged at both sites — pinned by tests that reproduce each
pre-extraction expression byte-for-byte.
* test(sandbox): pin the base the helper must not normalize
Review notes on #4108.
The committed snapshot compares the helper against hand-copied literals of the
pre-extraction expressions, so its red-ness rests on those literals, not on the
length of _BASES -- both sides compute the same expression, and 5k fuzzed bases
produce zero byte-differences. Mutating the helper one clause at a time (12
mutations over the boundary, the tail and the escape/replace) shows the seven
committed bases catch 11: the miss is a helper that normalizes its input by
rstripping a trailing separator. Only a trailing-slash base or a Windows drive
root catches that, and Path.resolve() / str(Path(...)) strip trailing slashes,
so neither call site can produce the former. C:\ survives resolve() with its
separator intact, so that is the one base worth adding.
Also point local_sandbox's comment at path_patterns, the owner, instead of
citing _content_pattern as the class reference, and drop the rationale it now
duplicates from the owner's docstring -- a second copy of the explanation drifts
the same way the second copy of the regex did. The site-specific half stays.
Comments and test data only; no behavior change.
* fix(sandbox): use os.sep in reverse-resolve containment check on Windows
Path.resolve() always renders with the native separator (backslash on
Windows), but _reverse_resolve_path's containment check hardcoded a
"/" suffix when testing whether a resolved path is nested under a
mapping's local root. Only the exact-root case (no separator needed)
ever matched; every nested path fell through to the "no mapping
found" branch and returned the raw host path -- leaking the real
username and full directory tree into list_dir/glob/grep results and
bash output masking instead of the virtual /mnt/user-data/... path.
_is_read_only_path already does the equivalent check correctly via
os.sep, so this aligns _reverse_resolve_path with that pattern: the
containment check now compares with os.sep, and the extracted
relative portion is normalized to forward slashes before being
spliced into the (always POSIX-style) container path.
Also fixes a same-file cosmetic bug in list_dir's virtual
sub-directory overlay: it compared a bare child name (e.g.
"workspace") against a set of full container paths, so the
already-listed guard never matched and a mount whose subdirectory the
underlying scan already found (the common case for
/mnt/user-data/workspace, uploads, outputs) was appended a second
time.
Continues the same separator-bug class already fixed in this file by
#3869 (forward-direction command resolution) and #4035 (reverse
regex-boundary matching); neither touched this containment check.
* test(sandbox): add host-OS-independent regression test for the os.sep containment fix
_reverse_resolve_path's os.sep containment check (and the paired
lstrip(os.sep).replace(os.sep, "/") extraction) has no test that would
fail if reverted: backend CI runs only on ubuntu-latest, where
os.sep == "/" makes the pre-fix hardcoded "/" and the current os.sep
form observationally identical, so a plain POSIX-path test can't
discriminate between them.
Add a test that forces the Windows code path independent of host OS by
monkeypatching os.sep to "\" and stubbing both the module's Path name
and the sandbox's cached _resolved_local_paths to return
backslash-joined strings, mirroring what real WindowsPath.resolve()
produces -- without touching the filesystem or requiring an actual
Windows host. Verified this fails with the raw host path leaking
through when the os.sep fix is reverted to the hardcoded "/" form, and
passes with the fix in place.
replace_virtual_paths_in_command matches the virtual root with no
segment-boundary lookahead:
re.compile(rf"{re.escape(VIRTUAL_PATH_PREFIX)}(/[^\s\"';&|<>()]*)?")
The trailing group needs a "/" to consume anything, so when the character after
/mnt/user-data is "-", ".", "_", a digit or a letter, the group matches empty
and the bare root still matches. The substitution then rewrites it to the
thread's host user-data directory and the rest of the sibling name rides along,
so a command naming a prefix sibling of the mount root is pointed at a real host
directory outside the mount contract:
cat /mnt/user-data-backup/secret.txt -> cat <host>/user-data-backup/secret.txt
which reads the host file. This is the same defect as #4035 (reverse patterns)
and #4053 (masking patterns), mirrored into the virtual->host direction; it is
the last unguarded member of that family.
The boundary class mirrors LocalSandbox._content_pattern's rather than
_command_pattern's: a virtual root can legitimately be followed by ":"
(PATH-style concatenation) or ",", which the shell-oriented class rejects, so
narrowing to it would stop translating paths that translate today. "$" covers a
command ending exactly at the root.
str_replace returned "OK" whenever the target file was empty, silently
reporting success even when the model asked to replace a non-empty string
that could not possibly be present. Only short-circuit to "OK" when old_str
is also empty; otherwise return the standard not-found error.
* fix(sandbox): stop glob/grep/ls from surfacing disabled skills' files
The disabled-skill gate checks the path a tool is given, but ls, glob and
grep all descend from it and return other paths, so a root above a disabled
skill still serves its files. glob and grep never called the gate at all;
ls called it only on its own argument and still leaked from a category root.
Add the entry gate to glob/grep, and filter what all three return through
the existing fail-closed _is_disabled_skill_path. The verdict is memoized per
skill because ExtensionsConfig.from_file() is uncached, so a per-match check
would turn a 100-match grep into 100 config reads.
* fix(sandbox): normalize trailing slashes in the disabled-skill path check
Review follow-ups on the disabled-skill gate:
- _extract_skill_name_from_skills_path returned "" instead of None for a
category directory carrying a trailing slash. LocalSandbox.list_dir appends
"/" to directories, so `ls /mnt/skills` yields "/mnt/skills/public/", giving
parts ["public", ""]. The empty name skipped the `skill_name is None`
short-circuit and fell through to a config read, landing on the right outcome
only because unknown skills default to enabled. Drop empty segments so a
trailing-slash category root takes the existing category-root branch.
- ls_tool resolved the runtime user id twice per call; hoist it into a local,
matching glob_tool/grep_tool.
- Cover the CUSTOM path: custom/legacy skills resolve their enabled state
through the per-user _skill_states.json, a different store from the public
skills' extensions_config.json, and no automated test exercised it.
* fix(sandbox): guard the output-masking regex with a segment boundary
`_compiled_mask_patterns` builds the same class of host→virtual matcher as
`LocalSandbox._reverse_output_patterns`, but without the segment-boundary
lookahead that one carries. The trailing group needs a separator to consume
anything, so when the character after a host base is `-`, `.`, `_`, a digit or
a letter, the group matches empty and the regex still matches the bare base.
`replace_match` then takes its `matched_path == base` branch and rewrites the
sibling: with `/mnt/skills` mounted at `.../skills`, output naming a sibling
`.../skills-extra/data.txt` is handed to the model as `/mnt/skills-extra/data.txt`
— a container path forward resolution explicitly refuses to map back, so
reading it raises FileNotFoundError.
This is the sibling site of #4035, which fixed the identical bug in
`local_sandbox.py`. That PR's scope argument enumerated the prefix matchers in
that file and missed this one; `mask_local_paths_in_output` runs on every
glob/grep match and on local bash output.
The boundary class mirrors `_content_pattern`'s, not `_command_pattern`'s: this
runs over arbitrary command output, where a base can legitimately be followed
by `,`, `:` or `\`, all of which the shell-oriented class rejects.
* test(sandbox): anchor the sibling boundary at the ACP source too
The sibling-rejection cases only fed the skills source. `_compiled_mask_patterns`
builds every source's matcher in one loop, so the ACP workspace carried the same
defect: nothing maps its parent, and `/mnt/acp-workspace-backup/hello.py` is
unresolvable in both directions.
User-data is the exception and is now pinned as such: `_thread_virtual_to_actual_mappings`
also maps the virtual root `/mnt/user-data` to the three dirs' common parent, so a
sibling of `outputs` is still inside a mount and has a real virtual path — the output
is byte-identical with and without the boundary. That test is green on main; it guards
the boundary from being narrowed into one that stops translating a mapped path.
Reverting only `boundary` turns the 5 skills + 4 ACP cases red and leaves the 3
user-data cases green.
* fix(sandbox): honor single-bound line ranges in read_file
read_file only sliced content when BOTH start_line and end_line were
provided, so read_file(path, start_line=100) or read_file(path,
end_line=50) silently returned the whole file. Handle one-sided ranges:
default the missing bound (start->1, end->EOF), clamp start to 1, and
return clear messages when start_line exceeds the file length or start_line
> end_line.
* address review: guard one-sided end_line <= 0 in read_file
Add symmetric end_line guard and run the inverted-range check regardless
of which bounds are explicit, so a lone end_line<=0 returns a clean
error instead of a negative-index slice. Per @willem-bd review on #4078.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(sandbox): guard the reverse path-translation regex with a segment boundary
_reverse_output_patterns matches a mount's resolved local root with no
segment-boundary lookahead, unlike every other prefix match in this file: both
forward regexes carry one, and the three string-prefix resolvers compare against
`root + "/"`. Its optional trailing group needs a `/` or `\` to consume more, so
against a sibling that merely shares the prefix (".../skills-extra/data.txt") the
group matches empty and the regex still yields the bare root.
That extracted text then *equals* the mount root, which satisfies
_reverse_resolve_path's own `+ "/"` guard -- so the sibling is rewritten to
"/mnt/skills-extra/data.txt". Forward resolution refuses to map that back, so the
model is handed an absolute-looking container path it can never read. This runs
over every bash stdout/stderr and over read_file content.
Mirror _content_pattern's boundary class rather than _command_pattern's: this
regex sees arbitrary command output, where a root can legitimately be followed by
"," ":" or "\" -- all of which the shell-oriented class would reject, silently
dropping translations that work today. The trailing group keeps [/\\] so Windows
paths still match.
* test(sandbox): pin the end-of-output boundary that stops a host-path leak
The lookahead's `$` alternative had no test: deleting it left all 6866
backend tests green. It is not cosmetic. Output ending exactly at a mount
root -- printf '%s' "$PWD", a stripped last line, a capture truncated at
the buffer limit -- matches neither `/` nor `[^\w./-]`, so the pattern
fails and _reverse_resolve_paths_in_output emits the raw host path to the
model. main's unbounded pattern has no end-of-string problem; the
narrowing added here is what introduced the exposure.
Pin it: three prefixes, asserting both the container path and that the
host root is absent from the output. Deleting only `$` turns exactly
those three red.
Lift the boundary class and trailing group into named locals. The
compiled pattern is byte-identical; correctness hinges entirely on the
boundary class, so it should not sit at column 120 of a comprehension.
* fix(sandbox): scrub abbreviated *_PASS vars and Postgres PGPASSFILE from skill env
env_policy blocks the full PASSWORD/PASSWD spellings but not the ubiquitous
abbreviated _PASS form (DB_PASS, SMTP_PASS, MYSQL_PASS, ...), whose value is the
plaintext password itself, so every skill subprocess inherited it. Postgres's
file-based credential sources PGPASSFILE (.pgpass locator) and PGSERVICEFILE
(pg_service.conf, may carry a password) were likewise missed -- the psql analog
of the MYSQL_PWD/REDISCLI_AUTH no-flag sources #4018 added.
Collapse *PASSWORD*/*PASSWD* into a single *PASS* pattern (which subsumes both,
covers the abbreviated family, and catches PGPASSFILE) and add PGSERVICEFILE to
the exact-name denylist. *PASS* deliberately does not touch PWD/OLDPWD, which
carry no PASS substring. Capability-handle vars (KUBECONFIG, DOCKER_CONFIG,
GIT_ASKPASS, ...) are a broader, non-value-bearing class and are left out of
this fix.
test_secret_like_names_are_blocked already pins the contract; the new names are
added to its parametrize list (red before, green after) and the benign-names
test guards against over-scrubbing PWD/OLDPWD.
* test(sandbox): pin the *_ASKPASS over-scrub that *PASS* introduces
*PASS* also matches the credential-helper vars GIT_ASKPASS, SSH_ASKPASS and
SUDO_ASKPASS, which main inherits. Each names a program whose purpose is to hand
the caller a credential, so inheriting the pointer is the same leak class as
inheriting the value -- a skill can simply invoke it. Scrubbing them is intended;
pin all three in test_secret_like_names_are_blocked (red on main) so the
behaviour is a decision rather than a side effect of the pattern's shape.
test_benign_names_are_allowed could not have caught this: none of its 14 names
carries a PASS substring, so it passes identically before and after the
broadening and asserts nothing about it. Record that in its docstring -- the
absence of a benign PASS-bearing name is a decision, and PWD/OLDPWD are the
boundary the list does pin.
Also note the incidental COMPASS_*/BYPASS_* over-scrub in the module comment
(over-scrubbing is this module's fail-safe direction; required-secrets is the
escape hatch), and list PGPASSFILE alongside PGSERVICEFILE in AGENTS.md.
* fix(sandbox): scrub MYSQL_PWD and REDISCLI_AUTH from the inherited skill env
env_policy blocks the MySQL and Redis connection strings (MYSQL_URL, REDIS_URL)
because they embed a password, but not the password variables the same clients
read directly: mysql/libmysqlclient takes MYSQL_PWD and redis-cli takes
REDISCLI_AUTH with no further configuration. Both names carry no
KEY/SECRET/TOKEN/PASSWORD/PASSWD substring, so they were inherited by every
skill subprocess -- the leak surface this module exists to close (#3861).
They need exact entries rather than a pattern: *PWD* would also strip PWD and
OLDPWD. REDIS_AUTH is included as the common non-canonical variant.
* Apply suggestions from code review
- Distinguish REDIS_AUTH (defensive, non-canonical) from MYSQL_PWD and
REDISCLI_AUTH (documented no-flag credential sources) in the comment.
- Pin that request-scoped injection still overrides the host value for the
newly blocked names, keeping required-secrets a working escape hatch.
* fix(provisioner): gate legacy skills mount by user visibility
* fix(test): aio sandbox provider
* fix: use shared legacy skill visibility helper for sandbox mounts
* feat(channels): add GitHub event-driven agents (#3754)
Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation.
* fix(llm-middleware): classify bare IndexError as transient
Upstream chat providers occasionally return 200 OK with an empty
generations list (observed against Volces "coding" on
ark.cn-beijing.volces.com). When that happens,
langchain_core.language_models.chat_models.ainvoke raises
``IndexError: list index out of range`` at
``llm_result.generations[0][0].message`` and kills the run.
Treat a bare IndexError reaching the middleware as a transient
upstream-payload glitch and route it through the existing
retry/backoff path instead of failing the whole agent run. The
retry budget and backoff schedule are unchanged.
Adds three regression tests covering the classifier and both the
recover-on-retry and exhausted-retries paths.
* fix(runtime): ignore stale LLM fallback markers from prior runs
When a run on a thread ends with the LLM-error-handling middleware emitting
a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError
empty-generations classification fix lands), that message is persisted to
the thread's checkpoint as part of the messages channel. LangGraph replays
the full message history in `stream_mode="values"` chunks, so every
subsequent run on the same thread re-streams the stale fallback marker —
and the worker's chunk scanner faithfully picks it up, flipping
`RunStatus.success` to `RunStatus.error` for runs that themselves had
no LLM failure at all.
Snapshot the set of pre-existing message ids from the pre-run checkpoint
and thread it through `_extract_llm_error_fallback_message` /
`_try_extract_from_message` as a filter. Markers on history messages are
ignored; markers on fresh messages produced during this run still trip
the error path. Falls back to an empty set when the checkpointer is
absent or the snapshot can't be captured, preserving the prior behavior
on first-run / no-state paths.
Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`)
plus an integration test exercising the full `run_agent` path with a stale
history checkpointer.
* fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs
GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed
the langgraph_sdk default 300s read deadline. The manager's runs.wait
call kept an HTTP stream open for the entire run lifetime, so the long
run blew up with httpx.ReadTimeout and the outer except branch then
released the dedupe key and emitted a false 'internal error' outbound.
The GitHub channel's outbound send is log-only by design: agents post to
the issue/PR via the gh CLI in the sandbox when they choose to comment
or create a PR. There is nothing for the manager to ferry back, so the
long-poll was pure overhead.
This change adds ChannelRunPolicy.fire_and_forget (default False) and
sets it True for the github channel. When fire_and_forget is True,
_handle_chat dispatches via client.runs.create (short POST, returns
once the run is pending) instead of client.runs.wait, and skips the
response-extraction + outbound-publish block. ConflictError on a busy
thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on
the busy case is preserved for any future non-github fire-and-forget
channel.
Other (non-github) channels are unchanged: their policy defaults
fire_and_forget=False and they continue to dispatch via runs.wait.
Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget:
- Default ChannelRunPolicy.fire_and_forget is False.
- The github policy registers fire_and_forget=True.
- github inbound calls runs.create, not runs.wait, with the right kwargs.
- github inbound publishes no outbound on success.
- ConflictError from runs.create still emits THREAD_BUSY_MESSAGE.
- Non-github channels (slack) still dispatch via runs.wait.
* test(lead-agent): accept user_id kwarg in skill-policy test stubs
The two GitHub-channel tests added in #3754 stubbed
_load_enabled_skills_for_tool_policy with a lambda that only accepted
`available_skills` and `app_config`, but the real function (and its call
site in agent.py) also passes `user_id`. This raised TypeError on every
run, failing backend-unit-tests.
Add `user_id=None` to match the three sibling stubs in the same file.
* refactor(gateway): disambiguate context-key set names
The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS
shared a confusable "CONTEXT_ONLY" token in different orders, and the
first broke the _CONTEXT_<X>_KEYS pattern of its sibling
_CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit:
_CONTEXT_INTERNAL_CALLER_KEYS - WHO: internal callers (scheduler) only
_CONTEXT_RUNTIME_ONLY_KEYS - WHERE: runtime context only, never configurable
Pure rename, no behavior change.
* feat(channels): expose channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID
IM-channel skills need the sender's platform identity (Feishu open_id,
Slack Uxxx, ...). The channel manager already writes channel_user_id into
body.context, but the Gateway whitelist dropped it. Forward it into the
runtime context only (never configurable, which is checkpointed), and have
bash_tool export it as a fixed env var through a shell-quoted command
prefix.
The identity deliberately does not ride execute_command(env=...): that
channel is reserved for request-scoped secrets, and a non-empty env
switches AioSandbox onto the bash.exec path (fresh session per call,
image >= 1.9.3 required), which would have broken every IM bash command
on older sandbox images and abandoned persistent-shell semantics on new
ones. A command-string export keeps the legacy path, stays visible in
audit logs (it is an identifier, not a secret), and gives per-call
correctness in group chats where one thread and sandbox are shared by
senders with different platform ids.
Skipped on the Windows local sandbox, whose PowerShell/cmd.exe fallback
has no POSIX export.
Part of #3914
* feat(channels): propagate channel_user_id to subagents; cap value length
Review findings from the pre-PR verification pass:
- Subagent delegation dropped the sender identity: task_tool now captures
channel_user_id from the parent runtime context and the executor
forwards it into the subagent's context, mirroring the guardrail
attribution fields (user_role/oauth_*/run_id). Without this, bash
commands delegated via task lost the group-chat sender's id.
- body.context is client-writable on web requests, so values over 256
chars are ignored instead of bloating every command string sent to the
sandbox.
* fix(channels): set-or-unset channel_user_id so identity is per-call regardless of AIO session persistence
Review (willem-bd): the identity export could leak across senders in a
shared group-chat AIO sandbox. The AIO no-env path reuses a persistent
shell session (the class-lock reason, #1433), and the 256-char/type guard
made some commands carry no prefix — so a dropped-id command could resolve
the id a previous sender exported.
Make per-call correctness independent of session semantics: an IM-channel
command (channel_user_id present in context) now always carries an explicit
prefix — export VAR=<quoted> for a valid id, or unset VAR for an unusable
one (empty / non-str / over the cap). Non-IM runs (no key) are untouched.
A prefix unset has none of the '& ; unset' suffix hazard raised earlier.
Verified on a real AIO 1.11.0 container: the no-id shell path auto-creates
a session per call (does not persist today), but an explicit shared session
DOES persist (export stale-A -> readback [stale-A]); the unset prefix
clears it (-> []). So the fix holds even on an image whose no-id path
persists. Regression tests cover the dropped-id group-chat window and the
non-IM passthrough.
Part of #3914
* test(channels): align channel_user_id task test with new Command return shape
The merge from main changed task_tool to return a Command(update=...)
instead of a plain string; update the assertion to extract the tool
message via the existing _task_tool_message helper, matching the sibling
tests. Fixes the CI backend-unit-tests failure introduced by the merge.
* feat(skills): per-user skill isolation (#2905)
Implement user-scoped skill storage that isolates custom skills between
users while sharing public skills globally.
Key changes:
- Add UserScopedSkillStorage class for per-user custom skill directories
- Introduce get_or_new_user_skill_storage() factory with user_id context
- Auth middleware sets effective_user_id for request-scoped storage
- Agent/prompt/middleware now use user-scoped storage and prompt cache
- Sandbox mounts user-scoped skill directories for search/read tools
- Add validate_skill_file_path() to SkillStorage for path security
- Migration script supports --all-users bulk migration
- Frontend: add editable field to Skill type, error check in enableSkill
- All skill categories can be toggled (custom skills default to enabled)
- Update skill-creator SKILL.md with isolation-aware instructions
Tests:
- Add test_user_scoped_skill_storage.py (new)
- Update all existing skill tests for user-scoped storage
- Update sandbox, client, and router tests
* fix(skills): address second-round PR review feedback (#3889)
- P1-1: restrict legacy skill mount to users without custom skills
- P1-2: fail-closed for _is_disabled_skill_path (OSError → return True)
- P2-1: AND-merge global extensions_config skill disabled state
- P2-2: atomic write for _skill_states.json (mkstemp + replace)
- P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary
- P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256)
- P2-5: clear global prompt cache on PUBLIC skill toggle
- P2-6: invalidate skill caches on client.update_skill
* fix(tests): correct tool policy test after merge
* fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage
The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers
test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module
on CI. Migrate the default to the existing deerflow.constants constant,
matching the pattern already used by LocalSkillStorage, SkillStorage, and
the durable/tool_error middlewares.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: emit structured runtime metadata
* fix: avoid subagent import cycle in replay gateway
* fix: preserve legacy subtask result parsing
* refactor: tighten runtime metadata contracts
* fix(middleware): keep recovery hint on task exception wrapper content
The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.
* fix(subagents): require lowercase hex for result_sha256 reader
Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(specs): read-before-write gate design for issue #3857 output layer
* docs(plans): read-before-write gate implementation plan (#3857)
* refactor(sandbox): extract read_current_file_content helper (#3857)
* feat(middlewares): read-before-write version gate for file tools (#3857)
* test(middlewares): pin async read-before-write gate paths (#3857)
* feat(config): wire ReadBeforeWriteMiddleware into runtime chain, default on (#3857)
* docs(sandbox): document read-before-write gate in tool docstrings and AGENTS.md (#3857)
* docs(plans): align plan doc with landed config_version (17) and drop machine-specific paths
Addresses Copilot review comments on #3912.
* fix(middlewares): read-before-write gate — error-string sandboxes fail open; serialize gate+execution per path (#3912 review)
- AIO/E2B read_file reports failures (incl. missing files) as 'Error: ...'
strings instead of raising; the gate treated that string as existing file
content and blocked first-write creation. Error-string reads now count as
uninspectable: gate fails open, no mark is stamped.
- LangGraph runs one AIMessage's tool calls concurrently, so two same-turn
writes could both pass on one stale mark before either mutation landed
(and a read mark could hash a version the model never saw). Gate check +
tool execution (and read + mark stamping) now share a per-(thread, path)
critical section, separate from the tool-internal file_operation_lock.
* fix(sandbox): normalize Windows backslash paths to forward slashes in bash commands
On Windows, `replace_virtual_paths_in_command()` and
`LocalSandbox._resolve_paths_in_command()` resolve virtual paths
(/mnt/skills, /mnt/user-data, /mnt/acp-workspace, custom mounts) to
host paths with backslashes (C:\Users\admin\...). Bash interprets
\U, \a, \d, \s, \n, \t as escape sequences, mangling the path.
Fix: add `.replace("\\", "/")` to all path resolution callbacks so
resolved paths use forward slashes, which bash handles correctly on
all platforms and Windows APIs/Python's open() accept natively.
Fixes#3865
* test(sandbox): regression tests for Windows backslash path normalization
Covers replace_virtual_paths_in_command and LocalSandbox._resolve_paths_in_command
with Windows-style backslash paths, asserting no backslashes survive in output.
Closes#3865
* test(sandbox): fix ACP test failure, keep 6 passing regression tests
Removed test_acp_workspace_no_backslash which triggered path traversal
validation. Remaining 6 tests cover user-data (3), skills (1), and
LocalSandbox custom-mount (2) paths — all with Windows backslash paths.
* test(sandbox): restore ACP Windows path normalization coverage
* feat(sandbox): per-call env injection + platform-secret scrubbing for skills
Add an env parameter to Sandbox.execute_command (abstract + local + AIO) so request-scoped secrets can be injected into skill subprocesses, and scrub platform credentials (*KEY*/*SECRET*/*TOKEN*/*PASSWORD*/*CREDENTIAL*) from the inherited environment by default so scoped injection is not security theatre. LocalSandbox always passes an explicit scrubbed env; AioSandbox routes env-bearing commands through bash.exec(env=) on a fresh session and leaves the legacy persistent-shell path unchanged. Part of #3861. BEHAVIOR CHANGE: execute_command no longer inherits the full os.environ; Windows encoding tests updated to assert the scrubbed dict.
* feat(skills): parse required-secrets frontmatter declaration
Add SecretRequirement and Skill.required_secrets, and parse the required-secrets SKILL.md frontmatter field (a string list or {name, optional} mappings), dropping malformed entries with a warning so one bad declaration does not invalidate the skill. The declared name is both the context.secrets key and the env var injected at activation. Part of #3861.
* feat(runtime): request-scoped secret carrier (context.secrets)
Add SECRETS_CONTEXT_KEY + extract_request_secrets, centralising the context.secrets carrier contract. The existing context passthrough (build_run_config -> _build_runtime_context) already carries the sub-key to runtime.context without mirroring it into configurable; characterization tests lock that behaviour. Part of #3861.
* feat(skills): inject declared secrets at slash-activation into bash env
Binding point A: when a skill is slash-activated, SkillActivationMiddleware resolves its declared required-secrets against the request's context.secrets and writes the per-run injection set to runtime.context. The bash tool forwards that set to execute_command(env=). A skill cannot harvest a host platform credential (is_host_platform_secret guard, cf. GHSA-rhgp-j443-p4rf), and injected values are redacted from bash output (mask_secret_values) so an echoed secret never re-enters the prompt/trace. Part of #3861.
* test(skills): lock the five secret leak surfaces + add trace redaction helper
Regression tests assert the secret value is absent from all five surfaces: prompt (activation message), checkpoint (graph state vs context separation), audit (journal records names only), trace (metadata builder never copies context; never mirrored to configurable), and stdout (mask_secret_values). Add redact_secret_context_keys as a defensive helper for any context serialization. Part of #3861.
* docs(backend): document request-scoped secrets for skills
Add Request-Scoped Secrets subsection (Skills) + env policy note (Sandbox) and the execute_command(env=) signature change, per the doc-sync policy. Part of #3861.
* fix(skills): close gaps found by end-to-end verification of request-scoped secrets
Real-gateway e2e + independent review of #3861 surfaced three defects, now fixed:
1. Slash activation never fired in the live chain. InputSanitizationMiddleware
wraps user input in BEGIN/END markers before SkillActivationMiddleware sees it,
and the original text was only preserved when an upload or IM channel set it.
For a plain text message the slash command became undetectable, so no secret
was ever resolved. Fix: the sanitizer now setdefaults the pre-wrap text into
ORIGINAL_USER_CONTENT_KEY (additive; sanitization behaviour unchanged), so
slash activation works for all messages. Pre-existing latent bug surfaced here.
2. The raw request config (with context.secrets) was persisted to runs.kwargs_json
and echoed by the run API (RunResponse.kwargs). Fix: redact_config_secrets()
strips secret-bearing context keys from the persisted/echoed copy in start_run;
the live config that drives the run keeps them. build_run_config now also sets
configurable.thread_id on the context path (the checkpointer requires it).
3. Connection-string credentials (DATABASE_URL, REDIS_URL, SENTRY_DSN, GH_PAT, ...)
were not scrubbed from the inherited sandbox env. Fix: env_policy adds a *DSN*
pattern plus an explicit connection-string denylist (no blanket *URL* — benign
service URLs stay readable).
Verified end-to-end via a real gateway run (real LLM + skill activation + bash):
the secret reaches the sandbox subprocess and appears in NONE of prompt, trace,
checkpoint, audit, stdout, runs.kwargs_json, or the run API. Part of #3861.
* docs(backend): document the env scrub, persistence redaction, and sanitizer interaction
Sync the Request-Scoped Secrets section with the verification-driven fixes: inherited-env scrub (incl. connection-string denylist), run-record/run-API redaction as the 6th sealed leak surface, and the sanitizer preserving original content so slash activation fires. Part of #3861.
* fix(skills): inject caller secret over scrubbed host value; drop redundant host-name guard
A real-world demo (a skill calling a third-party cloud API with a request-scoped
key) exposed that the is_host_platform_secret guard was both wrong and harmful:
it refused to inject a caller-supplied secret whenever a same-named variable
existed in the Gateway env — which is exactly the #3861 use case (a per-user key
overriding a shared platform key). The guard was also redundant: build_sandbox_env
already scrubs secret-looking names from the inherited env before injection, so a
skill can never read a host credential — it only ever receives the caller's value.
Remove the guard; the injected (caller) value simply wins over the scrubbed host
value. Verified end-to-end: the agent called the real cloud API successfully with
the caller's key, the host's same-named key was scrubbed and never used, and the
caller's key leaked to none of the surfaces. Part of #3861.
* fix(skills): address review on request-scoped secrets (#3861)
Review fixes from PR #3871:
- E2BSandbox.execute_command now accepts env/timeout and routes them to
commands.run(envs=, timeout=). The bash tool passes env= unconditionally,
so the prior signature (command only) raised TypeError on every e2b bash
call and broke e2b deployments entirely. env=None stays backward-compatible.
- SkillActivationMiddleware clears the active-secret set before resolving each
activation, so a later skill in the same run never inherits an earlier
skill's injection set (the #3861 contract: a skill only receives what the
caller supplied AND that skill declared).
- AioSandbox env path uses a dedicated _DEFAULT_HARD_TIMEOUT — bash.exec exposes
no idle/no-change timeout, so the prior reuse of the legacy idle constant
conflated wall-clock vs idle semantics. The env path also retries on the
ErrorObservation signature now, sharing the legacy persistent-shell recovery
contract.
- mask_secret_values skips values below a minimum length floor so a short
declared secret (e.g. "42") cannot shred unrelated bytes (exit codes,
timestamps, sizes) of tool output. The secret is still injected into the
subprocess; only the output mask skips it.
session_id reuse on the env path is intentionally NOT added: a shared session
could let request-scoped secrets ride the session env into later commands,
which the SDK does not contractually forbid. The fresh-session choice matches
the LocalSandbox model (each call is a fresh subprocess); the trade-off
(consecutive env-bearing calls do not share cwd/venv/exports) is documented on
_execute_with_env.
* fix(sandbox): stop blocking bash commands from hanging the turn
Starting a server through the host bash tool (e.g. `python -m http.server`)
could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command`
used `subprocess.run(capture_output=True)`, whose captured pipes are inherited
by any process the command spawns — so a backgrounded long-lived process
(`server &`) keeps the read end open and blocks `communicate()` until the
timeout fires, even though the foreground command already returned. Commands
that read stdin blocked the same way, and on timeout only the direct child was
killed, leaving orphaned process groups.
Rework the POSIX path to capture stdout/stderr via temp files instead of pipes,
take stdin from /dev/null, and run the command in its own session/process group:
- Backgrounded long-lived processes (servers) now return immediately while the
process keeps running.
- A command reading stdin gets immediate EOF instead of blocking.
- A genuinely blocking foreground command is bounded by a configurable
wall-clock timeout; on timeout the whole process group is killed and the agent
gets an explanatory notice telling it to background long-lived processes.
The timeout is configurable via `sandbox.bash_command_timeout` (default 600).
The Windows path is unchanged. Adds focused regression tests and updates docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): instruct the agent to background long-lived processes
The code fix bounds a foreground server with a timeout, but the turn still
waits the full timeout before the run continues. Add the prompt-side half:
the bash tool description now tells the model to ALWAYS start long-lived
processes (e.g. web servers) in the background with output redirected, so the
tool returns immediately. The timeout notice points at the same readable
workspace log path. Pins the guidance with a test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): make fallback-kill exception explicit and observable
Address automated review: the inner `except OSError: pass` in
_terminate_process_group silently swallowed the case where the direct-child
fallback kill found the process already gone. Make the intent explicit with a
comment and a debug log instead of a bare pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): address bash timeout review feedback
* fix(sandbox): document fd cleanup races
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): synchronize sandbox provider singleton lifecycle
get_sandbox_provider() used an unsynchronized check-then-create, so two OS
threads (e.g. the main event loop and the Feishu channel thread, which runs
its own loop) could double-initialize the provider. With AioSandboxProvider
the overwritten instance leaks its idle-checker thread, since the only code
that joins it (shutdown()) is reachable only through the reference that was
overwritten.
reset_sandbox_provider(), shutdown_sandbox_provider() and set_sandbox_provider()
also touched the global without a lock, so a reset/shutdown racing an in-flight
create could clear it mid-construction or tear down an instance another thread
was about to return.
Guard all four lifecycle sites with a single module-level threading.Lock and
use double-checked locking in the getter, mirroring get_memory_storage().
* test(sandbox): add concurrent regression tests for provider singleton
- 8 threads racing on cold start, synchronized with a threading.Barrier so
the check-then-create race fires deterministically; asserts exactly one
provider instance is created.
- reset racing concurrent gets: asserts every returned value is a fully
constructed provider (never None / half-built).
* fix(sandbox): lock get read path, run provider callbacks outside the lock
Addresses the three review findings on #3730:
1. get_sandbox_provider()'s read+return ran outside _provider_lock, so a
concurrent reset/shutdown/set could null or tear the global between the
check and the return, handing callers None / a torn instance. The hot read
and the install reconciliation now both happen under the lock.
2. The non-reentrant _provider_lock was held across plugin-supplied callbacks
(resolve_class import + provider __init__ in get; provider.reset()/shutdown()
in reset/shutdown). A custom provider that re-entered these lifecycle
functions would self-deadlock, and a slow teardown blocked every concurrent
get(). resolve_class + construction now run outside the lock; reset/shutdown
detach the reference under the lock and invoke the callback outside it.
Tradeoff: racing cold-start callers may each construct a candidate. Exactly
one is installed and returned to everyone; the losers (e.g. an AioSandbox
instance that already started an idle-checker thread) are shut down so they
do not leak the orphan thread #3721 is about. set_sandbox_provider() documents
that it replaces but does not shut down the prior instance.
3. test_reset_racing_get reset the singleton to None *before* the barrier, so
the racing reset was a no-op and never exercised reset-of-a-live-provider.
It now populates the singleton up front so the reset tears down a live
instance while getters read it.
Tests: rename the cold-start test to assert "one installed singleton, observed
by all" (construction is no longer single under the new design); add
shutdown-vs-get, set-vs-get, and a losing-racer-shuts-down-its-orphan case.
All five pass; the existing sandbox/middleware/mounts/uploads suites are green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These three module-private helpers have no callers anywhere in the repo
(verified by a repo-wide identifier scan + direct greps; each appears only
at its own definition):
- agents/memory/updater.py: _create_empty_memory (a no-caller wrapper around
storage.create_empty_memory, which is used directly elsewhere)
- runtime/runs/worker.py: _extract_human_message (also drops the now-unused
HumanMessage TYPE_CHECKING import and TYPE_CHECKING from typing)
- sandbox/tools.py: _looks_like_unsafe_cwd_target (cwd safety is enforced via
_is_allowed_local_bash_absolute_path)
Underscore-prefixed = module-internal, so zero references is conclusive.
Route handlers and @compiles-registered functions were excluded as false
positives. Public-surface candidates are noted in the issue, not touched here.
Fixes#3748.
mask_local_paths_in_output runs once per glob/grep match (tools.py:1540,
:1625) and once per bash/ls output, and each call rebuilt every skills /
ACP / user-data masking regex from scratch — Path.resolve() syscalls +
re.escape + re.compile. A grep returning up to 100 matches recompiled the
same patterns 100x.
Compile the host->virtual patterns once per ordered source set via
functools.lru_cache (keyed on the config-stable + per-thread (host,
virtual) pairs, so it self-invalidates when they change) and collapse the
three byte-identical replacer blocks into one applier. Behavior is
unchanged: same patterns, same application order, same per-thread mappings.
Verified behavior-preserving with an equivalence harness (original vs
refactored logic over 54 inputs incl. slash-style variants, base-only
matches, subpaths, overlapping mappings, missing skills/ACP, thread_data
None) -> 0 mismatches. Added cache/consistency tests alongside the
existing masking tests.
Fixes#3712.
should_ignore_name runs once per directory entry during glob/grep tree
walks and looped over ~57 IGNORE_PATTERNS calling fnmatch per pattern.
Precompute the literal names into a frozenset (O(1)) and the few glob
patterns into one combined compiled regex via fnmatch.translate; per name
it's now one normcase + a set lookup + at most one regex match.
os.path.normcase preserves fnmatch's platform case behavior, so results
are identical (covered by an equivalence test against the old loop).
Fixes#3655.
* perf(sandbox): cache LocalSandbox path-rewrite regexes per instance
LocalSandbox re-sorted path_mappings and re-compiled its path-rewrite
regex on every tool call: _resolve_paths_in_command (every bash),
_resolve_paths_in_content (every write_file), and
_reverse_resolve_paths_in_output (every bash output / read_file). The
inputs are derived solely from self.path_mappings, which is assigned once
in __init__ and never mutated, so the work is identical every call.
Compile the patterns once per sandbox via functools.cached_property and
reuse them; hoist 'import re' to module scope. Behavior is unchanged —
only the per-call sort+escape+compile on the agent's hot path is removed.
Fixes#3647. Adds tests covering caching identity, unchanged rewriting,
path-segment boundary matching, and the empty-mappings pass-through.
* perf(sandbox): also cache resolved local paths and sorted mapping views
Beyond the regex compilation, _find_path_mapping, _is_read_only_path,
_resolve_path_with_mapping and _reverse_resolve_path re-sorted
path_mappings and re-ran Path(local_path).resolve() (a filesystem
syscall) on every call. Since path_mappings is immutable, cache the
resolved local root per mapping and the two sorted views via
cached_property and reuse them. Behavior is unchanged; the reverse-output
pattern builder now reuses the same resolved-path cache.
* fix(sandbox): stop flagging string-literal path fragments as unsafe paths
The host-bash absolute-path guard scans the raw command string, so /segment
sequences inside string literals, f-strings, and templates were treated as
absolute path arguments and rejected — e.g. python -c "print(f'/端口{port}')"
or a REST template /devices/{id}/port. Whether a fragment tripped the guard
depended on the character right before the slash (a word char suppressed the
match), so the same literal could pass or fail unpredictably, pushing the model
into retry loops that bloat context and wall-clock time.
Exempt matches carrying non-ASCII characters or format braces: real host paths
a command would open contain neither, so these are text, not paths. The guard
is best-effort (not a security boundary), and plain ASCII host paths like
/etc/passwd — including ones written inside a code string such as
open('/etc/passwd') — stay rejected.
* fix(sandbox): only exempt identifier-template braces, not bash brace expansion
The literal-fragment exemption exempted any path fragment containing { or },
which let bash brace expansion (cat /etc/{passwd,shadow}) and ${VAR} expansion
reconstitute real host paths past validate_local_bash_command_paths. Tighten
the brace branch to only exempt fragments where every {...} block is a single
identifier-like placeholder (/devices/{id}/port, f-string /{port}); reject
${VAR} shell-variable expansion. Add parametrized regression tests for the
brace-expansion and shell-var bypasses.
read_file decodes with UTF-8. Binary uploads (.xlsx, images, ...) raise
UnicodeDecodeError in the sandbox layer. UnicodeDecodeError is a ValueError
subclass, not an OSError, so it bypassed the typed handlers and fell through
to the generic except, surfacing a vague "Unexpected error reading file"
message. The model could not tell the file was binary, so it retried
read_file instead of switching to bash + pandas/openpyxl, burning LLM
round-trips and bloating context with repeated failures.
Add a dedicated UnicodeDecodeError handler that tells the model the file is
binary and to use bash with a suitable library (or view_image for images).
* fix(sandbox): persist lazily-acquired sandbox state via Command
ensure_sandbox_initialized mutates runtime.state in place, which is local
to the current tool invocation and is not picked up by LangGraph's channel
reducer. Subsequent graph steps and downstream consumers (such as
ToolOutputBudgetMiddleware and the sub-agent task_tool) therefore cannot
observe the sandbox id from state.
Wrap tool calls in SandboxMiddleware (wrap_tool_call / awrap_tool_call) to
detect fresh lazy initialization by diffing runtime.state before and after
the handler, and emit a proper state update via Command(update=...):
- ToolMessage results are wrapped into Command(update={sandbox, messages})
- Command results with a dict update are merged on the sandbox key while
preserving messages / goto / graph / resume
- Command results with non-dict updates are left untouched to avoid silent
data loss on unknown update shapes
Tests:
- 7 new unit tests cover lazy-init emit, passthrough, dict-update merge,
non-dict-update passthrough (sync and async)
- Refresh replay golden write_read_file.ultra.events.json: SSE 'values'
events now correctly carry the 'sandbox' key in their keys list, which
is the direct evidence that the fix is effective
Closes#3463
* refactor(sandbox): use dataclasses.replace to preserve Command fields
Address Copilot review on #3464: replace manual field-copy with
dataclasses.replace so any current or future Command fields are
preserved automatically when merging sandbox_update.
Also add a regression test that constructs a Command with non-None
graph/goto/resume to lock this behavior in.
In Docker production deployments, LocalSandboxProvider runs inside the
deer-flow-gateway container, so any `sandbox.mounts[].host_path` from
config.yaml is resolved against the gateway container's filesystem — not
the host machine. When the path isn't also bind-mounted into the gateway
service, the mount was silently dropped with only a WARNING log, leaving
agents reading an empty directory in production while the same config
worked under `make dev`.
Escalate the missing-host_path branch to logger.error with explicit
guidance about Docker bind mounts and docker-compose, so the failure is
hard to miss in default log configurations. Skip behaviour is preserved
to avoid breaking existing deployments.
Also clarify the misleading `VolumeMountConfig.host_path` field
description so it documents reality for both providers:
- LocalSandboxProvider checks host_path from inside the gateway process
(host in `make dev`, container in `make up`).
- AioSandboxProvider (DooD) passes host_path straight to `docker -v`
for the sandbox container, where the host Docker daemon resolves it
from the host machine's perspective.
config.example.yaml's `sandbox.mounts` comment gets a Note: block
pointing operators at the docker-compose bind-mount requirement so the
Docker-mode gotcha is discoverable from the canonical template.
Adds a regression test that:
- confirms missing host_path is still skipped (no behaviour break);
- asserts an ERROR record is emitted referencing the offending paths;
- asserts the message contains actionable Docker/gateway/docker-compose
keywords so future refactors can't quietly downgrade it.
Refs: https://github.com/bytedance/deer-flow/issues/3244
* fix(#3189): prevent write_file streaming timeout on long reports
Adds a layered defense against StreamChunkTimeoutError caused by oversized
single-shot write_file tool calls:
- factory: default stream_chunk_timeout to 240s for OpenAI-compatible
clients (overridable via ModelConfig.stream_chunk_timeout in config.yaml)
- sandbox/tools: server-side 80 KB length guard on non-append write_file
calls (configurable via DEERFLOW_WRITE_FILE_MAX_BYTES env var, 0 disables);
rejects oversized payloads with a structured error pointing the model at
str_replace or append=True
- middleware: classify StreamChunkTimeoutError as transient but cap retries
at 1 via per-exception _RETRY_BUDGET_OVERRIDES (same-payload retry on a
chunk-gap timeout buffers the same way upstream; full 3-attempt loop
would stack 6-12 min of dead air)
- middleware: surface an actionable user-facing message for stream-drop
exceptions instead of leaking the raw langchain stack
- prompts: add a routing-style File Editing Workflow hint to both lead_agent
and general_purpose subagent prompts, pointing the model at str_replace
for incremental edits (mirrors Claude Code's Edit / Codex's apply_patch)
- tests: behavioural coverage for size guard, retry budget override,
stream-drop user message, factory default injection
Refs #3189
* fix(#3189): drop stream_chunk_timeout for non-OpenAI providers
Address CR feedback on PR #3195:
- factory: pop `stream_chunk_timeout` from kwargs for any model_use_path other than `langchain_openai:ChatOpenAI` instead of returning early. `ModelConfig.stream_chunk_timeout` is part of the shared schema, so a user-supplied value on a non-OpenAI provider would otherwise be forwarded to its constructor and raise `TypeError: unexpected keyword argument`.
- factory: rewrite docstring to describe the actual `exclude_none=True` behaviour (explicit null is excluded and falls back to the default) instead of the misleading "None falling out via exclude_none=True keeps its value".
- tests: add regression coverage asserting the kwarg is stripped before reaching a non-OpenAI provider's constructor.
Refs: bytedance#3189
* fix(#3189): restrict stream-drop user copy to StreamChunkTimeoutError only
Per CR on #3195: narrow _STREAM_DROP_EXCEPTIONS to StreamChunkTimeoutError. Generic httpx RemoteProtocolError / ReadError fall back to the standard 'temporarily unavailable' copy, since they routinely fire on transient network blips where the 'split the output' guidance is misleading. Retry/backoff classification is unchanged — both remain transient/retriable. Tests updated to reflect new copy, plus a symmetric regression test for ReadError.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): add group/other read permissions to uploaded files for Docker sandbox (#3127)
When using AIO sandbox with LocalContainerBackend, uploaded files are
created with 0o600 (owner-only) permissions by the gateway process
running as root. The sandbox process inside the Docker container runs
as a non-root user and cannot read these bind-mounted files, causing
a "Permission denied" error on read_file.
Add `needs_upload_permission_adjustment` attribute to SandboxProvider
(default True) to indicate that uploaded files need chmod adjustment.
LocalSandboxProvider opts out (same user). A new `_make_file_sandbox_readable`
function adds S_IRGRP | S_IROTH bits after files are written, changing
permissions from 0o600 to 0o644 so the sandbox can read the uploads.
fixes#3127
* fix(uploads): unconditionally adjust file permissions for sandbox access
The conditional check meant uploaded files retained 0o600
permissions in some Docker sandbox configurations, preventing the
sandbox process (UID 1000) from reading them. Always add group/other
read bits so every sandbox setup can access uploaded content. Also add
read bits to the sync-path writable helper as defense in depth.
* fix(runtime): bound write_file execution-failure observations
* fix(runtime): preserve write_file error prefixes
* test(runtime): trim write_file prefix assertions
* refactor(runtime): drop redundant exception suffix for permission/directory write errors
Address Copilot review on #3133: the PermissionError and IsADirectoryError
branches now return self-contained, non-redundant messages (e.g.
"Error: Permission denied writing to file: /mnt/...") via direct
truncation, instead of going through _format_write_file_error which
appended a duplicate ": PermissionError: permission denied" suffix.
OSError, SandboxError and the generic Exception branches keep the
unified "Failed to write file '{path}': {ExceptionType}: {detail}"
format so the model still sees a stable, machine-readable error class.
Removes the now-unused message= parameter from _format_write_file_error,
keeping a single code path. Truncation contract (<= 2000 chars) and
host-path sanitization unchanged.
* fix(runtime): handle write_file sandbox init errors
Initialize the requested path before sandbox setup so early sandbox failures can still return a bounded write_file error.
Add a regression test for sandbox initialization failures.
* style(test): format sandbox security tests
* fix(sandbox): uphold /mnt/user-data contract at Sandbox API boundary (#2873)
LocalSandboxProvider used a process-wide singleton with no /mnt/user-data
mapping, forcing every caller to translate virtual paths via tools.py
before invoking the public Sandbox API. AIO already exposes /mnt/user-data
natively (per-thread bind mounts), so the same code path behaved
differently across implementations — and direct callers like
uploads.py:282 / feishu.py:389 only worked thanks to the
`uses_thread_data_mounts` workaround flag.
Switch the provider to a dual-track cache: keep the `"local"` singleton
for legacy acquire(None) callers (backward-compat for existing tests and
scripts), and create a per-thread LocalSandbox with id `"local:{tid}"`
for acquire(thread_id). Each per-thread instance carries PathMapping
entries for /mnt/user-data, its three subdirs, and /mnt/acp-workspace,
mirroring how AioSandboxProvider mounts those paths into its container.
is_local_sandbox() now recognises both id formats. `_agent_written_paths`
becomes per-thread (it was a process-wide set that leaked across
threads — a latent isolation bug also fixed by this change).
Verified via TDD: a new contract test suite hits the public Sandbox API
directly (write/read/list/exec/glob/grep/update + per-thread isolation +
lifecycle). 3212 backend tests still pass, ruff is clean.
* fix(sandbox): address Copilot review on #2881
Three follow-ups from Copilot's review of the LocalSandboxProvider refactor:
1. Synchronisation: ``acquire`` / ``get`` / ``reset`` mutated the cache without
any lock, so concurrent acquire of the same ``thread_id`` could create two
``LocalSandbox`` instances and lose one's ``_agent_written_paths`` state.
Add a provider-wide ``threading.Lock`` (matching ``AioSandboxProvider``) and
build per-thread mappings outside the lock to avoid holding it during the
``ensure_thread_dirs`` filesystem touch.
2. Memory bound: ``_thread_sandboxes`` grew monotonically. Replace the plain
dict with an ``OrderedDict`` LRU capped at
``DEFAULT_MAX_CACHED_THREAD_SANDBOXES`` (256, configurable per provider
instance). ``get`` promotes touched threads to the MRU end so an active
thread isn't evicted under load. Eviction is graceful: the next ``acquire``
rebuilds a fresh sandbox; only ``_agent_written_paths`` (reverse-resolve
hint) is lost.
3. Docs: update ``CLAUDE.md`` to reflect the new per-thread architecture, the
LRU cap, and that ``is_local_sandbox`` recognises both id formats.
New regression tests:
- Concurrent ``acquire("alpha")`` from 8 threads yields a single instance
(slow-init injection forces the race window wide open).
- Concurrent ``acquire`` of distinct thread_ids yields distinct instances.
- The cache evicts the least-recently-used thread once the cap is exceeded.
- ``get`` promotes recency so a polled thread survives a later acquire-storm.
* Fix local sandbox singleton reset on provider lifecycle
* Fix local sandbox singleton reset on provider reset
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: make tool argument behavior discoverable
The write_file tool already supported append=false by default with append=true for end-of-file writes, but the parsed docstring did not describe append in the model-facing schema. This records the overwrite default and append path in the tool description, adds resilient schema regression coverage, and keeps backend sandbox docs aligned.
The regression now also checks that every public parameter in the existing tool schema test matrix has a description. Enabling docstring parsing on setup_agent and update_agent fills the two existing gaps with their existing Args docs instead of duplicating descriptions elsewhere.
Constraint: Issue #2831 asks for a small docstring/schema discoverability fix without changing runtime file-writing behavior
Rejected: Changing write_file defaults | would alter existing overwrite semantics and broaden the fix beyond schema discoverability
Rejected: Exact phrase assertions | too brittle for future docstring rewording while testing the same behavior
Confidence: high
Scope-risk: narrow
Directive: Keep model-facing tool parameters documented through parsed docstrings or equivalent schema descriptions
Tested: cd backend && uv run pytest tests/test_setup_agent_tool.py tests/test_update_agent_tool.py tests/test_tool_args_schema_no_pydantic_warning.py tests/test_sandbox_tools_security.py::test_str_replace_and_append_on_same_path_should_preserve_both_updates -q
Tested: cd backend && uv run ruff check packages/harness/deerflow/sandbox/tools.py packages/harness/deerflow/tools/builtins/setup_agent_tool.py packages/harness/deerflow/tools/builtins/update_agent_tool.py tests/test_tool_args_schema_no_pydantic_warning.py
Not-tested: Full backend test suite
Co-authored-by: OmX <omx@oh-my-codex.dev>
* Fix the lint error
---------
Co-authored-by: OmX <omx@oh-my-codex.dev>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
