The acceptance test compiles a fixture crate from a cold target directory in
each of its six parametrizations. On a loaded runner the toolchain startup and
compile together can exceed the 60s command budget, and the case then fails on
"Exit Code: 124" instead of a test summary — main run 39566441443 went red on
the [ignored] case for exactly that reason while the previous five runs passed.
Give the command a budget that covers a cold compile. The assertion itself is
unchanged: the test still requires cargo's own summary in the captured output.
Co-authored-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(reflection): reject malformed variable paths before diagnosing them
resolve_variable splits on the last colon and assumed both halves were
usable. Two realistic typos fell through into misleading diagnoses:
- "json:" reached getattr(module, "") and reported
"Module json does not define a attribute/class" -- a blank name in the
middle of the sentence.
- "json:loads:extra" left the extra colon inside the module part, so the
failed import was reported as Missing dependency and the user was told to
run uv add json:loads, an instruction that cannot work.
Both are path mistakes, so they now get the same format guidance as a path
with no colon at all. Every provider, extension and model class path in the
repo flows through here, so the message is what an operator sees first.
Tests cover the two typos and assert the missing-dependency hint is not
offered for them.
* fix(reflection): reject relative module paths too, and pin the empty-module half
Review follow-ups on the malformed-path check:
- A leading-dot path (.foo:bar) is a relative import, and import_module
raises TypeError rather than ImportError for it. Every loader built on this
function catches ImportError, so the call crashed instead of reporting the
path. These paths are absolute by construction, so reject the shape.
- ":attr" was already covered by the new condition but had no test, which
left the empty-module half unpinned. Added one.
Both were mutation-checked: dropping each condition turns exactly its own
test red.
* fix(reflection): reject whitespace and trailing dots in variable paths too
Review follow-up on the shapes that stay inside the ImportError contract but
land in the missing-dependency branch:
- "json.:attr" advertises a distribution named `json.`
- "json :loads" and " json:attr" advertise `uv add json ` / `uv add json`,
where the space that IS the typo is invisible in the rendered hint
- "json:attr\n" reaches getattr with a name the caller never wrote
A dotted module path has no trailing dot and neither half carries
whitespace, so these get the same format guidance as the other four shapes.
Whitespace is checked inside both halves rather than at the ends, since
"json :loads" hides the typo in the middle.
Each new condition half is mutation-checked: removing the trailing-dot test
fails only the trailing-dot case, and removing both whitespace tests fails
exactly the two whitespace cases.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`test_config_version.py` guards its shell-script cases with
`@pytest.mark.skipif(SCRIPT_BASH is None, ...)`, except for three `version_45`
migration tests added later. On a host without Git Bash on PATH those tests
pass `None` as the executable and fail with
`TypeError: expected str, bytes or os.PathLike object, not NoneType` instead
of skipping like their siblings, so the module reads as broken rather than
inapplicable.
Verified with `PATH` reduced to `%SystemRoot%\System32`:
- before: the three tests failed with that TypeError
- after: the three tests skip like the rest of the shell-script cases
* feat(community): migrate GroundRoute tools to async HTTP
The GroundRoute provider sent its search and fetch requests through a sync
httpx.Client. A sync LangChain tool invoked through ainvoke runs on the
default executor, so the request held one of that pool's threads for the
whole network round-trip, and several providers queried in parallel queued
behind each other.
Make both tools coroutines over httpx.AsyncClient, resolve the API key and
the configured max_results in a single config read, and keep that read off
the event loop with asyncio.to_thread.
* refactor(community): drop the dead GroundRoute API key wrapper
The async migration resolved the key and max_results through
_resolve_tool_settings, which left _get_api_key as a second seam around
the same config read: only TestGetApiKey still exercised it. web_fetch
now reads its key through the same resolver, and the tests point at
_resolve_tool_settings.
Also record which httpx client factory the provider reaches for in the
runtime anchor. MockTransport runs its handler on the calling thread
without real blocking I/O, so a provider that stayed async but sent the
request through a sync client would satisfy the thread assertion; the
recorded factory is what catches that partial regression.
---------
Co-authored-by: treasuresyq <309743290+treasuresyq@users.noreply.github.com>
* fix(skills): create a missing extensions_config.json in the project root
When no extensions config resolved, the skill toggle created one at
Path.cwd().parent while the MCP writers used project_root(). Unless the
Gateway runs from backend/ with the project root at its parent, the
toggle's file landed outside the resolver's search path: the toggle was
lost on reload and the next MCP edit created a second file.
Add ExtensionsConfig.default_config_path() (the project root, the first
location resolve_config_path searches) and use it from the skill toggle
and both MCP create paths, so a created file is always the one the next
resolution reads back.
* docs(changelog): reference #7048 for the extensions config create path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): find Git Bash under a PortableGit mingw64 layout
`scripts/run-with-git-bash.cmd` derives `bash.exe` from `git.exe`'s parent,
which only holds for the Git for Windows `<root>/cmd` and `<root>/bin`
layouts. PortableGit and GitHub Desktop ship `git.exe` in
`<root>/mingw64/bin`, so the wrapper looked for `<root>/mingw64/bin/bash.exe`
and reported "Could not locate Git for Windows Bash" even though
`<root>/bin/bash.exe` exists. Every Makefile recipe that runs a repository
shell script failed on those hosts.
Resolve the shell under the Git installation root instead, and mirror the rule
in `backend/tests/support/shell.py` (documented as mirroring the wrapper) so
the shell-script suites stop skipping on the same hosts.
* test(backend): pin both Git layouts in one shell-discovery suite
Review follow-up for the PortableGit probe:
- `_git_roots` now probes `git.exe`'s parent and grandparent unconditionally —
exactly the two directories the Makefile wrapper checks — so the batch probe
and the Python mirror cannot drift; the docstring records that the first
`mingw64` candidate normally misses and is kept only to mirror it.
- fold the candidate-layout cases and the Windows-only wrapper case into
`tests/test_support_shell.py`, which already owns this module's unit tests,
instead of a second file with a duplicate `_touch` helper.
- report the wrapper's stdout in the failure message: its diagnostic goes to
stdout, so `stderr` was empty on the discovery-failure path.
* fix(scheduler): treat only busy-thread 409s as retryable
start_run exposes the retryable run-manager admission conflict as BusyThreadConflict (an HTTP 409), but also uses plain 409s for permanent idempotency-key and thread/assistant mismatches. The scheduler treated every HTTP 409 as an overlap conflict and requeued the occurrence, so a permanent launch error was retried until scheduler.queue_timeout_seconds elapsed. Other launch errors are documented to fail immediately, and the MCP notification launcher already distinguishes these types.
Match ConflictError and BusyThreadConflict only; a plain 409 now follows the normal failure path. The type is imported lazily so the scheduler module does not take a load-time dependency on app.gateway.services.
Tests: a plain idempotency 409 terminalizes the occurrence as failed, while BusyThreadConflict remains queued; existing ConflictError coverage is unchanged.
* fix(scheduler): document the busy-thread import boundary and pin the failure detail
Keep the BusyThreadConflict import local to the classifier and document why it is guaranteed to resolve: the scheduler is wired by the Gateway, where app.gateway.services is already loaded, and this must not become an optional dependency. Import the same type locally in the test so collection does not initialize gateway services for the rest of the scheduler suite. Also pin the persisted failure detail so the 409 reason stays operator-visible.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(harness): complete Lark device auth via a single-flight login run
The web authorization polling loop reused the request wait budget as the
lark-cli auth login subprocess lifetime: each /lark/auth/complete poll ran
the CLI with subprocess timeout=wait_timeout_seconds (8s from the frontend)
and killed it on timeout. The login transaction is not atomic — poll, redeem
the single-use device code, fetch user info, then persist credentials — so a
kill after redemption but before persistence stranded the flow, and the next
poll with the same code failed with 'device_code is invalid'.
Own the login process by the flow generation instead: the first poll spawns
one background run per (user_id, generation) and later polls attach to it;
wait_timeout_seconds only bounds the caller's wait (unchanged 504 contract)
and never terminates the CLI. Advancing the generation or re-issuing a device
code abandons and kills the orphaned process, and a finished run's outcome
stays attachable for 120s so late polls return the same result instead of
re-consuming the redeemed code.
* fix(lark): tolerate vanished credential-tree entries and bound owner-alive grace
Publication could kill the login worker with an unhandled FileNotFoundError
when a concurrently-written temp file vanished between iterdir() and lstat(),
leaving the marker live but no result file ever published; every foreign poll
then blocked 504 until the 600 s attempt bound, and could respawn an
already-redeemed code. Also, a bare-pid owner_alive check trusted for the
full 600 s let a recycled-pid or crashed-thread marker outlive any process
that could still publish.
- _credential_tree_path_kind returns "gone" on FileNotFoundError so the
walk skips vanished entries instead of aborting; the same tolerance is
applied to iterdir().
- _write_lark_login_result_once treats the re-harden as best-effort: a
filesystem hiccup is logged and publication continues so the marker sweep
still runs.
- The marker now records owner_pid_started_at and
_read_live_lark_login_marker_locked only keeps an owner-alive marker
within a 10 s publication grace, so a stale marker is swept and the flow
respawns instead of wedging every foreign poll.
Regression: test_walk_and_harden_tolerates_a_vanished_entry,
test_write_result_once_continues_when_hardening_fails,
test_abandoned_marker_with_dead_owner_is_swept_within_grace.
Focused suites: 139 passed, 12 skipped.
* fix(lark): harden surviving credentials through filesystem churn
Atomic login publication can remove a descendant after lstat but before
chmod. Catch FileNotFoundError around the entire per-entry walk so that
lstat, chmod and directory enumeration tolerate those disappearances.
A vanished root, permission error or symlink remains a hard failure.
Exercise real removals at each boundary and concurrent result publication;
assert that injection ran and surviving directories/files reach 0700/0600.
The regressions detect a skipped walk, swallowed permission failures and
loss of directory-churn tolerance in isolated mutation runs.
* fix(lark): retain login ownership through durable completion
Browser approval can take longer than the publication grace. Start that
grace at the first observed CLI exit, then reject abandoned attempts
without retrying a device code that may already have been redeemed.
Serialize atomic result publication under the credential lock. Preserve
published results through cleanup failures and retain terminal failures
on the marker when result storage fails. Never bypass credential hardening.
Reserve storage before spawning and roll back failed marker, process or
thread startup by reaping the CLI, clearing the registry and waking waiters.
Add process/filesystem failure regressions, a long-login two-worker case,
and strict unhandled-thread gates. Fix invocation logging in both workers
and reap forked test CLIs after the parent observes their launch. Document
the authorization lifecycle and required regression assertions.
Validated the reproduced failures and ten isolated mutations; final
focused, blocking-I/O, full-suite and independent review details are in PR #7182.
* fix(browser): preserve Live keyboard modifiers
* test(browser): cover Meta modifier on named keys
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Allow ordinary Agent tasks the same 16,000 JSON bytes already accepted for
file-reference tasks, keeping the 12,000-byte output cap and bounded rejection.
Cover ASCII, escaped Unicode, maximum output and pre-admission rejection.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`get_readme` and `get_file_content` answered every failure with a string --
`[README not found: HTTP 404]`, `[File not found: HTTP 404]` -- while `main()`
prints any string result on stdout and exits 0. A rate-limited run, a missing
repository and a real README therefore arrived on the same channel with the same
exit code, so a caller reading stdout could not tell a failure from a body.
Both readers now raise `GitHubAPIError` naming the operation, the target and the
underlying HTTP error. The CLI's existing handler already writes an exception to
stderr and exits 1, so a failure leaves the success channel empty and the status
stays visible; a successful fetch is unchanged.
Pins both transports (requests and the urllib fallback), 404 against 403 so a
rate limit cannot read as a missing repository, the file reader's path, and the
CLI contract: non-zero exit, empty stdout, status on stderr.
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): pin JSON state encoding across platforms
* test(channels): check encoding pins at each state I/O call
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(deploy): select production Compose env files
Use an operator-selected dotenv file consistently for interpolation, container environment, secret probes and build extras. Preserve default selection and shutdown after a profile is removed.
Co-Authored-By: Codex <noreply@openai.com>
* fix(deploy): address production env selection review
Clarify Docker launcher entry points and render real Compose through the launcher to verify both service environments.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): cover unreadable env selectors
Reject unreadable explicit selectors before Docker is invoked, using POSIX permission checks and restoring the fixture mode in finally.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): scope dotenv assertions to selected secrets
---------
Co-authored-by: Codex <noreply@openai.com>
* chore(doc):sync the chinese changelog info
* chore(doc):sync the changelog with 2.2.0 milestone
* chroe(doc):sync the Chinese changelog with the english one
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
* fix(skills): keep load error text out of user-facing skill read 500s
GET /api/skills, /api/skills/custom and /api/skills/{name} are open to
every signed-in user, but their 500 detail echoed str(e). A malformed or
unreadable extensions_config.json (re-read on every load by the
user-scoped storage) made all three return the file's absolute host path,
plus the [Errno 13] text for a permission error.
Return a fixed detail like the diagnostics and reload routes already do,
chain the cause, and keep the full error in the Gateway log.
* docs(changelog): reference #7155 for the skill read 500 detail fix
* test(skills): pin capability skill discovery 500 through the real Gateway app
GET /api/capabilities/installations/skills reads the same user-scoped
storage as the skill routes but has no handler of its own. Its 500 body
is TraceMiddleware's fixed fallback, so drive create_app() with a
malformed extensions_config.json and assert no host path reaches the
body on any of the four reads.
* fix(sandbox): measure warm-pool idle age on a monotonic clock
Warm-pool expiry measures a duration, but every provider stamped and reaped
entries using the wall clock. A forward correction could therefore destroy
fresh sandboxes, while a backward correction could retain expired ones until
the clock caught up.
Move every warm-pool writer and reader to time.monotonic(), including the
Tenki and OpenSandbox reset paths missed by the earlier closed attempt. Keep
absolute SandboxInfo.created_at and reconciliation grace timestamps on the
wall clock. Convert AIO's sibling active-idle timestamps as one clock domain,
and update finite clock mocks for the additional duration read.
Fixes#6970
* test(sandbox): keep active-idle race fixtures in the monotonic domain
The production AIO activity clock moved to monotonic time, but the shared
orphan-reconciliation helper still stamped active fixtures with wall time.
CI then computed a negative idle age, so two teardown race tests never reached
the ownership gate. Use the same monotonic domain as production.
* test(sandbox): read warm-pool idle fixtures on the monotonic clock
The idle-age conversion left several fixtures seeding the fields the reaper
now reads with values from the clock it no longer reads:
- `test_idle_reaping_forces_a_new_remote` froze `warm_pool_lifecycle.time.time`,
so the benchmark's reaper compared against real elapsed monotonic seconds —
zero over the test's lifetime. Nothing expired, the entry survived, and the
second turn came back as a warm hit.
- Three orphan-reconciliation fixtures seeded `_last_activity` with `0.0` and
two aio fixtures with `time.time()`, both of which are far below a running
monotonic clock: the computed idle age went negative, so the reaper never
reached the ownership claim these tests are built to exercise.
Seed them from `time.monotonic()` instead, and keep `created_at` (a wall-clock
field) on `time.time()`.
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* fix(channels): keep staging cleanup from replacing the save error
The channel stores publish state through a staged temporary file and, if the
rename fails, close and unlink that file inside the handler before re-raising.
Neither step was guarded, so a failing close skipped the unlink and a failing
unlink replaced the error that actually caused the save to fail -- leaving the
staged copy, which for the runtime-config and WeChat stores holds an 0600 IM
token, behind with nothing reporting it.
Make both steps independently best-effort, name the leaked file in a WARNING,
and suppress the diagnostic itself so a broken log handler cannot become the
reported failure. This is the same contract PR #6871 establishes for browser
captures, applied to the three JSON publishers in app/channels.
* fix(channels): address staging cleanup review follow-ups
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): preserve Markdown table cells in data-analysis exports
* docs(skills): document Markdown cell escaping contract
Clarify handling of Markdown exports and escaping rules.
* docs: describe data-analysis Markdown export escaping
Clarify data-analysis skill features and behavior in README.
* test(skills): cover Markdown export headers and multiline cells
Add a test for markdown export functionality in data analysis.
* feat(decisions): add an OpenAI Decisions backend for the decision slots
Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.
Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.
* fix(decisions): isolate endpoint identities and correct diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): quarantine unreadable memory.json in the read-error branch too
MarkdownMemoryStorage._load_memory_file quarantined summaries that failed
to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError)
branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without
quarantining. The leftover file was then silently overwritten by the next
save, destroying the user's memories with no backup.
Call the existing best-effort self._quarantine_unreadable(path) there as
well, matching the documented quarantine contract, and add a regression
test asserting the original bytes survive a subsequent load+save.
Fixes#7072
* fix(memory): preserve summaries on transient read failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
