* fix(sandbox): require a path start before translating virtual and host roots
Path translation only checked the right-hand segment boundary (#4035,
#4053), so a root embedded in a longer path or URL was rewritten too.
DeerFlow's own artifact URLs contain /mnt/user-data, so
curl .../artifacts/mnt/user-data/outputs/x.html had the thread's host
directory spliced into the URL, and a mount at /data turned an
unrelated /var/data/app.log into /var/mnt/data/app.log in model-visible
output.
Add a shared left-hand PATH_START_BOUNDARY: a root must not follow a
word character, '.', '~' or '-'. '/' is still allowed so
file:///mnt/user-data keeps mapping. It applies to the bash command
rewrite, LocalSandbox command/content resolution, and both reverse
masking paths.
* refactor(sandbox): derive the path-start lookbehind from one class
Build PATH_START_BOUNDARY from _PATH_START_BLOCKERS so the regex and
scanner paths cannot drift, and document the attached short-option
value (-o/data/x) as a known false negative of the start rule.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(checkpoints): reject malformed Redis history write records
Validate pending-write tuples before optional Redis history entries reach
delta materialization. Treat the whole malformed entry as a miss so the
authoritative saver rebuilds it without dropping or inventing state.
Cover actual checkpoint reads, whole-entry replacement, valid tuple/list
records, neighbor preservation and warm-cache rereads.
* fix(checkpoints): bind cached history writes to their channel
* fix(checkpoints): keep cache warning failures best-effort
Guard warning handlers across decode, ingestion, serialization, Redis outages and purge. Preserve authoritative rebuilds and valid neighboring writes, with native graph regression coverage.
* docs(gateway): keep guidance below the hard size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(streaming): make Redis stream END publication idempotent
* fix(streaming): align Redis test fake with Lua END checks
Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis.
* fix(streaming): gate legacy END scans after writer rollout
* fix(helm): synchronize embedded config version with schema
* test(runtime): use real stream bridge config in close coverage
Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(community): resolve async tool config off the event loop
Agent runs execute on the Gateway event loop, and the async community
tools resolved their tool config through get_app_config() inline. That
call stats and hashes config.yaml and extensions_config.json on every
call and reparses them after an edit, so each tool call held the loop
for it, up to three times per call: Tavily, SearXNG and Firecrawl read
the config again to build their client, InfoQuest read it three times,
and the browser tools read the launch config for every action.
Resolve it through asyncio.to_thread once per call, as image_search and
groundroute already do, and build the client from that same read. This
covers jina_ai, crawl4ai, browserless, tavily, searxng, firecrawl,
infoquest, ragflow, lightrag and browser_automation.
* docs(changelog): reference #7580 for the off-loop community tool config reads
* docs(changelog): use the existing community-tools label in the zh entry for #7580
* fix(community): resolve the jsonify dataset_rows config off the event loop
dataset_rows_tool is async and called _load_settings() inline, so every
call ran get_app_config()'s stat + sha256 revalidation on the loop.
Read it through asyncio.to_thread like the other async community tools,
and add the tool to the off-loop anchors.
* docs(changelog): include Jsonify in #7580 and drop trailing whitespace
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(i18n): survive a malformed locale cookie
getLocaleFromCookie decoded the raw cookie value without a guard, so a
malformed percent-encoding threw URIError during locale bootstrap and in
formatTimeAgo. It also split the cookie on every "=", truncating values
that contain one. Decode defensively and split on the first "=" only.
* fix(i18n): normalize the cookie locale in formatTimeAgo
The second caller of getLocaleFromCookie cast the raw value to Locale, so a
malformed-but-known-prefixed cookie skipped normalization. Normalize it like
useI18n does, and cover the malformed and valid cookie paths.
---------
Co-authored-by: 一片一片 <192845873+gdut4140@users.noreply.github.com>
* fix(setup-agent): keep a failed discard from replacing the publish outcome
* test(setup-agent): pin the post-publish discard and assert the real temp name
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): drain final cleanup across caller cancellation
* fix(subagents): preserve stop outcomes during cancellation
Follow up on PR #7546 discussion_r4374235193 and Issue #7535.
Capture the unwinding cancellation before the drained cleanup tail and use it
only while the result remains non-terminal. Preserve completed, failed and
timed-out decisions. Keep cancellation propagation and cleanup ownership.
Share the synchronous result holder and publish its existing FAILED outcome
before requesting cancellation. This necessary timeout boundary overlaps #7483;
no active-cancel terminal write is added to the cleanup tail.
Validation: 12 targeted regressions pass on Windows and Linux; make test
37876 passed / 808 skipped, make test-blocking-io 292 passed / 1 skipped.
Format, lint, diff checks and four-file incremental OCR delegated review pass.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): read the live config off the event loop on public auth routes
/register, /setup-status, /providers, /oauth/{provider} and
/callback/{provider} read the live auth config through get_app_config()
inline. That call stats and hashes config.yaml and extensions_config.json
on every request and reparses them after an edit, and these routes are
unauthenticated, so any client could keep that file IO on the event
loop. /login/local already reads its throttle policy in a worker.
Read the registration gate through asyncio.to_thread, and the SSO
routes' auth.oidc section through a shared _oidc_config() helper that
does the same.
* docs(changelog): reference #7576 for the off-loop auth route config reads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): keep a required secret when an optional duplicate precedes it
parse_required_secrets dropped every declaration of a name after the first,
so an earlier 'optional: true' entry made a later required declaration
unreachable and the skill ran without the secret. Duplicates now keep the
first position but a required declaration upgrades the entry, and the export
manifest applies the same rule.
* test(skills): cover duplicate secret order and whitespace
Port the three parameterized ordering and whitespace cases from PR #7570 while retaining #7569's full-YAML, optional-only, and export regressions.
* refactor(skills): keep the export optional read fail-closed
export.py read `optional` with `bool(secret.get("optional", False))`, so a
truthy string would have become optional=True — the opposite of
parse_required_secrets, which treats a non-boolean value as required
(#5738). validate_skill_frontmatter_text already refuses such a package
before the dedup loop (installation applies the same rule), so that
coercion is unreachable today; read the flag as `is True` anyway so the
two surfaces cannot drift if the gate ever moves.
Pin the parity from both sides: the secrets suite asserts that the same
document is rejected by validation and read as required by the parser, and
the export suite asserts a manifest with skill_export_invalid_frontmatter
and no requirements instead of a coerced optional=True.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): leave SIGTERM/SIGINT teardown to the Gateway lifespan
AIO and E2B providers built on the event-loop thread installed SIGTERM/SIGINT
handlers that force-destroyed every sandbox synchronously, before uvicorn
started the run drain. The Gateway lifespan now holds sandbox shutdown
ownership from its first step, so providers skip those handlers and the
existing hook closes the provider after run drain and MCP teardown. SIGHUP
keeps the provider fallback. AIO acquire now refuses once shutdown began.
* docs(sandbox): document Gateway-owned sandbox shutdown and AIO admission fence
* test(gateway): inspect the lifespan body now that lifespan wraps it
* docs(changelog): reference #7560 for the sandbox SIGTERM drain-order fix
* ci: retrigger sandbox image smoke after a registry pull timeout
* fix(frontend): return SSO sign-ins to their next path
The (auth) layout redirected every signed-in request to /workspace. The
Gateway sets the session cookie on the same redirect that lands on
/auth/callback?next=..., so the callback page never rendered and every SSO
sign-in landed on the workspace instead of the page that asked for it. A
signed-in visit to /login?next=... lost its target the same way.
Layouts cannot read searchParams, so the layout now renders the page for a
signed-in session and each page sends the user on: the callback and login to
the validated next path, setup to the workspace. The login page shows a
short wait message instead of the form while it redirects, and uses
router.replace so Back does not return to it.
* docs(changelog): reference #7467 for the SSO next-path fix
* fix(auth): reject control characters in the next redirect path
Browsers delete tab, LF and CR from a URL before parsing it, so a next of
"/\n//evil.example" passed both validators' "//" check and then navigated
to //evil.example. The login page's post-sign-in push already followed it,
and the signed-in login and SSO callback redirects reach it too.
validateAuthNextPath and validate_next_param now reject every C0 control
character and DEL. Percent-encoded forms are left alone: URL parsers do
not decode them, so they stay on the local origin.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): keep LF bytes in skill and agent text writes
Text mode translates a newline into os.linesep on write, so on Windows every
caller-supplied LF reached disk as CRLF while readers use universal newlines or
decode bytes explicitly. The difference stayed invisible: the API returned LF,
the file held CRLF, and a skill history record's prev_content described a
revision no API view could return (#7174).
Pass newline="" at the ten writers that persist or judge caller text: the local
and user-scoped skill writers, the skill history JSONL, the user skill-states
JSON, the SOUL.md/config.yaml staging in FileAgentStore, the managed-subagent
store, setup_agent's SOUL.md publish, the USER.md route, and the two temp copies
the frontmatter validator and the static scanner judge.
The tests emulate the Windows translation on io.open, so the regression runs on
the Linux CI shards instead of only on a Windows host: 10 failed before this
commit, 10 pass after.
* fix(skills): preserve candidate newlines during static scan
---------
Co-authored-by: Felix Shi <204130665+shi-felix@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read the route authorization config off the event loop
AuthMiddleware resolves route permissions for every authenticated
request, and resolve_route_permissions read the hot-reloaded
authorization config inline. get_app_config() stats and hashes
config.yaml and extensions_config.json on every call and reparses them
after an edit, so that file IO ran on the event loop per request.
Read it through asyncio.to_thread, like the plugin authorization path
already does. The context copy keeps runtime config overrides visible.
* docs(changelog): reference #7557 for the off-loop route authorization config read
* docs(config): note the get_app_config read cost where it is documented
Move the off-loop note from gateway/AGENTS.md, which is 12 bytes under
its hard limit, to the Config Caching section that documents what each
get_app_config() read costs.
* fix(utils): construct one-shot models off the event loop
* docs(utils): document all one-shot callers and offload scope
Project-file summaries use the shared one-shot helper as well as input
polishing and follow-up suggestions. Include that third caller in the
README and utility guidance, and distinguish the separate skill scanner
and goal evaluator constructors.
Runtime and tests are unchanged. The existing exact-head strict and caller
regressions pass together: 80 passed on Windows with Python 3.12.14.
* fix(memory): flush pre-compaction memory off the event loop
The async compaction path fired the before-summarization hooks inline.
memory_flush_hook hands the summarized messages to add_nowait, which is
sync HTTP for mem0 and Honcho (inherited default -> add) and OpenViking
(its own override), so every automatic compaction with an HTTP memory
backend blocked the Gateway event loop for the whole request.
acompact_state now runs the hooks through asyncio.to_thread, as
MemoryMiddleware.aafter_agent already does for its writes. Ordering,
failure isolation, summary-first timing and context propagation are
unchanged; the sync compact_state path is untouched.
* docs(changelog): reference #7530 for the pre-compaction memory flush fix
* test(memory): anchor the OpenViking pre-compaction flush off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(runtime): honor the web UI language in replies and DeerMem updates
Carry Settings > Appearance > Language as request-only run data: the
Gateway validates context.locale against a finite allowlist, the lead
agent prefers it for replies, and bundled DeerMem writes new facts and
summary prose in it. Explicit user language requests still take
precedence; checkpoint configurable, static prompts, and memory backend
APIs are unchanged.
Refs #7287
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime): guard language context and pin locale maps
* test(runtime): pin UI locales across frontend and Gateway
Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(channels): honor the configured recursion limit in IM runs
Every IM channel run asked the Gateway for recursion_limit 100, because
the channel manager seeded each run config with it. The Gateway honors a
request value, so the top-level recursion_limit never reached IM runs,
and a budget of 100 could stop a run after a handful of tool-calling
turns (six in one measurement with the default lead-agent middleware;
the count changes with the middleware chain).
IM runs now send no recursion_limit of their own and get the Gateway's
configured default, clamped by max_recursion_limit, like Gateway API
runs. A session config value still takes precedence, and GitHub channel
runs keep their floor of 250. The README and config.example.yaml IM
examples no longer pin session.config.recursion_limit to 100.
* docs(changelog): reference #7521 for the IM recursion-limit fix
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
A cross-process stream bridge (RedisStreamBridge) returns False from
start_producer_lease only after a transient backend error whose partial
lease registration was safely rolled back, leaving nothing tracked. The
admission guard treated that as fatal and raised, surfacing a 500 for an
otherwise healthy run whenever Redis had a momentary blip.
Degrade instead of rejecting: admit the run and log a warning. An
untracked producer falls back to the historical conservative recovery
path (wait for the real END or the retained-stream TTL), so the run is
no less safe than it was before producer leases existed. The lease is
never transferred or finished because none was taken; the worker
re-attempts its own lease independently.
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Move checkpoint modes, replay, module maps and terminal-ownership details intact into the existing runtime reference. Keep concise invariants and explicit links in AGENTS.md, reducing the effective chain from 99,004 to 80,817 bytes. The existing guidance regression now passes without changing budget thresholds.
* fix(projects): delete read-only documents when purging trash
A permanently deleted project document could never be purged on Windows:
the read-only file attribute makes `Path.unlink` fail with `PermissionError`
(`WinError 5`), where POSIX needs only a writable parent. A restored archive
or a mode-preserving sync sets that attribute, and because the purge re-raises,
the row rolls back and the trashed document keeps coming back with the same
error on every retry.
Clear the attribute and retry the unlink once, mirroring the retry that
`config/paths.py::_clear_read_only_tree` already performs for thread
directories. Any other unlink error still propagates, so the row stays
retryable.
* fix(projects): harden trash purge permission retries
---------
Co-authored-by: Aling33 <along308@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): count code points when previewing table cells and memory facts
artifact-table-preview cut a cell at 120 UTF-16 code units and
memory-settings-page's truncateFactPreview measured and cut fact previews
at 140, so an emoji spanning two units could be split at the cut — the
lone half renders as U+FFFD — and a preview that already fit the
documented code-point budget was still shortened. Both previews now share
truncateToCodePoints in core/utils/text, which measures and cuts by code
points, the same rule briefReason in core/scheduled-tasks/format.ts
already follows.
Add unit tests for the helper and a DOM regression test covering both the
fits-exactly and the cut-inside-an-emoji cases.
* fix(frontend): count code points in the artifact-cell expand gate too
The expandable gate still measured value.length (UTF-16 units), so a
120-code-point cell ending in an astral character opened the expand
control even though the inline preview already showed the whole cell —
the same units-vs-code-points mismatch the preview cut fixed.
The gate now counts Array.from(value), and the boundary test asserts
the fits cell renders as a plain cell with no expand control while the
overflows cell keeps its button and whole-emoji cut.
Suggested-by: gdut4140 (port of the #7509 expandable gate)
* fix(sandbox): derive the AIO sandbox ID off the event loop in async acquire
_acquire_internal_async offloads every acquire layer except the sandbox ID
derivation, which ran inline. _sandbox_id_for_thread checks the thread's
skills_view directory with exists(), and resolving it goes through
Paths.base_dir, so each async thread acquire made filesystem syscalls on the
event loop. The strict blocking-I/O gate raises on it.
Run the derivation through the same owned-worker helper as the reuse and
warm-pool layers around it, and anchor it in the blocking-I/O suite for both
the default and the policy-scoped ID.
* docs(changelog): reference #7507 for the AIO async acquire sandbox ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): backfill run.terminal on confirmed recovery takeover
* fix(runtime): isolate recovery terminal publication failures
* fix(runtime): fence recovery terminal publication lifecycle
* fix(runtime): unify recovery receipt publication
* docs(runtime): document lifecycle guard lock constraints
* fix(runtime): reuse event transaction for recovery guards
Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): make read_file_chunks cancellation-safe for worker handoffs
Cancelling a project document read had three unsafe windows around the
dedicated file-I/O worker: (1) a cancellation landing between the open
worker completing and the result reaching the coroutine leaked the
handle (the cleanup path never saw it); (2) the cleanup close could
start while an admitted read was still running; (3) a repeated
cancellation let the caller escape before the close worker settled
(#7491).
Gate every admitted operation with the existing await_drained
primitive: cancellation is delivered only after the operation settles,
the open result is retained in a list cell the moment the worker
returns it (so the cleanup close always has the handle), and waiting
for the next chunk via the yield stays immediately cancellable.
Regression tests gate the real executor with asyncio events at the
worker boundary (real files, no mocks): all three windows are red on
unmodified main and green with the fix; the happy path is unchanged.
The dispatch-spy promotion tests that import app.gateway.routers need
the full backend dependency set and could not run in the minimal local
env; they fail identically on unmodified main (verified).
Fixes#7491
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: fix _gated_offload return annotation to the unpacked tuple
Review nit: the helper returns (calls, gated) and every call site
unpacks it as a pair, but the annotation claimed list[str]. Type it as
the tuple it actually returns.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: pin missing-path FileNotFoundError in read_file_chunks
Convergence item from the duplicate-pair verdict (#7493): the
open-failure path must surface FileNotFoundError as-is — the cleanup
finally skips the close because retained is empty (no handle was ever
opened), rather than raising AttributeError from closing None.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
---------
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
Two Gateways booting the same missing database both reach CREATE DATABASE and
only one wins; the loser aborted startup with duplicate_database (42P04) even
though the winner's catalog row was already committed and the target was
usable. Follow the SQLSTATE, like the missing-database check beside it does, so
the loser falls through to its retry. Every other failure -- permission denied
included -- still propagates untouched.
* fix(scheduler): keep manual-trial launch origins stable across retries
A manual trial's synthetic queued row omitted scheduled_for, and _launch_provenance used the current now for manual triggers. A retry at a later time therefore produced a different origin, which is part of start_run's per-occurrence idempotent input: the retry either conflicted with the first attempt or submitted a different request. Include the admitted scheduled_for in the synthetic row and use the occurrence time for every trigger, falling back to now only when a legacy caller supplies none.
Tests: a manual queued row retried at two later clocks produces the same origin and keeps the admitted scheduled_for; existing manual/scheduled provenance and title tests stay green. Full scheduler suite: 1108 passed / 130 skipped; guidance 0 errors / 0 warnings; ruff clean.
* test(scheduler): drive manual retry through requeue and drain
The unit provenance test pins the stable-origin function directly. Add a driven case that admits a manual occurrence, hits a busy-thread ConflictError so the row requeues, then drains it at a later clock and asserts both launches carry the admitted scheduled_for and the same full origin.
* test(scheduler): pin the immediate queue row in the retry test
Capture the queued mappings passed to _attempt_queued_run so the driven test asserts both halves explicitly: the synthetic immediate-launch row and the durable row drained later both retain the admitted scheduled_for.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome
* fix(gateway): adopt the family helper so a published edit stays silent
* fix(frontend): allow non-ASCII local parts in auth password forms (#7488)
The sign-in, register, admin-setup and change-password forms relied on the
browser's type="email" constraint validation, which rejects any non-ASCII
local part before the submit handler runs. The backend identity schema
deliberately accepts such addresses (see emailSchema in @/core/auth/types),
so an account created via the API, an administrator or OIDC could never sign
in through the password form.
Forms are now noValidate, and the permissive isValidEmail helper (one @,
non-empty local part and domain, no whitespace) runs in the handlers instead,
staying strictly weaker than what the backend accepts.
* fix(auth): normalize emails and restore form password checks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): include markdown modification time in ownership records
* test(uploads): reject same-size edited companions on Windows
* fix(uploads): distinguish legacy companion records
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
