* fix(auth): read the live config off the event loop on public auth routes
/register, /setup-status, /providers, /oauth/{provider} and
/callback/{provider} read the live auth config through get_app_config()
inline. That call stats and hashes config.yaml and extensions_config.json
on every request and reparses them after an edit, and these routes are
unauthenticated, so any client could keep that file IO on the event
loop. /login/local already reads its throttle policy in a worker.
Read the registration gate through asyncio.to_thread, and the SSO
routes' auth.oidc section through a shared _oidc_config() helper that
does the same.
* docs(changelog): reference #7576 for the off-loop auth route config reads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): keep a required secret when an optional duplicate precedes it
parse_required_secrets dropped every declaration of a name after the first,
so an earlier 'optional: true' entry made a later required declaration
unreachable and the skill ran without the secret. Duplicates now keep the
first position but a required declaration upgrades the entry, and the export
manifest applies the same rule.
* test(skills): cover duplicate secret order and whitespace
Port the three parameterized ordering and whitespace cases from PR #7570 while retaining #7569's full-YAML, optional-only, and export regressions.
* refactor(skills): keep the export optional read fail-closed
export.py read `optional` with `bool(secret.get("optional", False))`, so a
truthy string would have become optional=True — the opposite of
parse_required_secrets, which treats a non-boolean value as required
(#5738). validate_skill_frontmatter_text already refuses such a package
before the dedup loop (installation applies the same rule), so that
coercion is unreachable today; read the flag as `is True` anyway so the
two surfaces cannot drift if the gate ever moves.
Pin the parity from both sides: the secrets suite asserts that the same
document is rejected by validation and read as required by the parser, and
the export suite asserts a manifest with skill_export_invalid_frontmatter
and no requirements instead of a coerced optional=True.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): leave SIGTERM/SIGINT teardown to the Gateway lifespan
AIO and E2B providers built on the event-loop thread installed SIGTERM/SIGINT
handlers that force-destroyed every sandbox synchronously, before uvicorn
started the run drain. The Gateway lifespan now holds sandbox shutdown
ownership from its first step, so providers skip those handlers and the
existing hook closes the provider after run drain and MCP teardown. SIGHUP
keeps the provider fallback. AIO acquire now refuses once shutdown began.
* docs(sandbox): document Gateway-owned sandbox shutdown and AIO admission fence
* test(gateway): inspect the lifespan body now that lifespan wraps it
* docs(changelog): reference #7560 for the sandbox SIGTERM drain-order fix
* ci: retrigger sandbox image smoke after a registry pull timeout
* fix(frontend): return SSO sign-ins to their next path
The (auth) layout redirected every signed-in request to /workspace. The
Gateway sets the session cookie on the same redirect that lands on
/auth/callback?next=..., so the callback page never rendered and every SSO
sign-in landed on the workspace instead of the page that asked for it. A
signed-in visit to /login?next=... lost its target the same way.
Layouts cannot read searchParams, so the layout now renders the page for a
signed-in session and each page sends the user on: the callback and login to
the validated next path, setup to the workspace. The login page shows a
short wait message instead of the form while it redirects, and uses
router.replace so Back does not return to it.
* docs(changelog): reference #7467 for the SSO next-path fix
* fix(auth): reject control characters in the next redirect path
Browsers delete tab, LF and CR from a URL before parsing it, so a next of
"/\n//evil.example" passed both validators' "//" check and then navigated
to //evil.example. The login page's post-sign-in push already followed it,
and the signed-in login and SSO callback redirects reach it too.
validateAuthNextPath and validate_next_param now reject every C0 control
character and DEL. Percent-encoded forms are left alone: URL parsers do
not decode them, so they stay on the local origin.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): keep LF bytes in skill and agent text writes
Text mode translates a newline into os.linesep on write, so on Windows every
caller-supplied LF reached disk as CRLF while readers use universal newlines or
decode bytes explicitly. The difference stayed invisible: the API returned LF,
the file held CRLF, and a skill history record's prev_content described a
revision no API view could return (#7174).
Pass newline="" at the ten writers that persist or judge caller text: the local
and user-scoped skill writers, the skill history JSONL, the user skill-states
JSON, the SOUL.md/config.yaml staging in FileAgentStore, the managed-subagent
store, setup_agent's SOUL.md publish, the USER.md route, and the two temp copies
the frontmatter validator and the static scanner judge.
The tests emulate the Windows translation on io.open, so the regression runs on
the Linux CI shards instead of only on a Windows host: 10 failed before this
commit, 10 pass after.
* fix(skills): preserve candidate newlines during static scan
---------
Co-authored-by: Felix Shi <204130665+shi-felix@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read the route authorization config off the event loop
AuthMiddleware resolves route permissions for every authenticated
request, and resolve_route_permissions read the hot-reloaded
authorization config inline. get_app_config() stats and hashes
config.yaml and extensions_config.json on every call and reparses them
after an edit, so that file IO ran on the event loop per request.
Read it through asyncio.to_thread, like the plugin authorization path
already does. The context copy keeps runtime config overrides visible.
* docs(changelog): reference #7557 for the off-loop route authorization config read
* docs(config): note the get_app_config read cost where it is documented
Move the off-loop note from gateway/AGENTS.md, which is 12 bytes under
its hard limit, to the Config Caching section that documents what each
get_app_config() read costs.
* fix(utils): construct one-shot models off the event loop
* docs(utils): document all one-shot callers and offload scope
Project-file summaries use the shared one-shot helper as well as input
polishing and follow-up suggestions. Include that third caller in the
README and utility guidance, and distinguish the separate skill scanner
and goal evaluator constructors.
Runtime and tests are unchanged. The existing exact-head strict and caller
regressions pass together: 80 passed on Windows with Python 3.12.14.
* fix(memory): flush pre-compaction memory off the event loop
The async compaction path fired the before-summarization hooks inline.
memory_flush_hook hands the summarized messages to add_nowait, which is
sync HTTP for mem0 and Honcho (inherited default -> add) and OpenViking
(its own override), so every automatic compaction with an HTTP memory
backend blocked the Gateway event loop for the whole request.
acompact_state now runs the hooks through asyncio.to_thread, as
MemoryMiddleware.aafter_agent already does for its writes. Ordering,
failure isolation, summary-first timing and context propagation are
unchanged; the sync compact_state path is untouched.
* docs(changelog): reference #7530 for the pre-compaction memory flush fix
* test(memory): anchor the OpenViking pre-compaction flush off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(runtime): honor the web UI language in replies and DeerMem updates
Carry Settings > Appearance > Language as request-only run data: the
Gateway validates context.locale against a finite allowlist, the lead
agent prefers it for replies, and bundled DeerMem writes new facts and
summary prose in it. Explicit user language requests still take
precedence; checkpoint configurable, static prompts, and memory backend
APIs are unchanged.
Refs #7287
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime): guard language context and pin locale maps
* test(runtime): pin UI locales across frontend and Gateway
Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(channels): honor the configured recursion limit in IM runs
Every IM channel run asked the Gateway for recursion_limit 100, because
the channel manager seeded each run config with it. The Gateway honors a
request value, so the top-level recursion_limit never reached IM runs,
and a budget of 100 could stop a run after a handful of tool-calling
turns (six in one measurement with the default lead-agent middleware;
the count changes with the middleware chain).
IM runs now send no recursion_limit of their own and get the Gateway's
configured default, clamped by max_recursion_limit, like Gateway API
runs. A session config value still takes precedence, and GitHub channel
runs keep their floor of 250. The README and config.example.yaml IM
examples no longer pin session.config.recursion_limit to 100.
* docs(changelog): reference #7521 for the IM recursion-limit fix
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
A cross-process stream bridge (RedisStreamBridge) returns False from
start_producer_lease only after a transient backend error whose partial
lease registration was safely rolled back, leaving nothing tracked. The
admission guard treated that as fatal and raised, surfacing a 500 for an
otherwise healthy run whenever Redis had a momentary blip.
Degrade instead of rejecting: admit the run and log a warning. An
untracked producer falls back to the historical conservative recovery
path (wait for the real END or the retained-stream TTL), so the run is
no less safe than it was before producer leases existed. The lease is
never transferred or finished because none was taken; the worker
re-attempts its own lease independently.
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Move checkpoint modes, replay, module maps and terminal-ownership details intact into the existing runtime reference. Keep concise invariants and explicit links in AGENTS.md, reducing the effective chain from 99,004 to 80,817 bytes. The existing guidance regression now passes without changing budget thresholds.
* fix(projects): delete read-only documents when purging trash
A permanently deleted project document could never be purged on Windows:
the read-only file attribute makes `Path.unlink` fail with `PermissionError`
(`WinError 5`), where POSIX needs only a writable parent. A restored archive
or a mode-preserving sync sets that attribute, and because the purge re-raises,
the row rolls back and the trashed document keeps coming back with the same
error on every retry.
Clear the attribute and retry the unlink once, mirroring the retry that
`config/paths.py::_clear_read_only_tree` already performs for thread
directories. Any other unlink error still propagates, so the row stays
retryable.
* fix(projects): harden trash purge permission retries
---------
Co-authored-by: Aling33 <along308@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): count code points when previewing table cells and memory facts
artifact-table-preview cut a cell at 120 UTF-16 code units and
memory-settings-page's truncateFactPreview measured and cut fact previews
at 140, so an emoji spanning two units could be split at the cut — the
lone half renders as U+FFFD — and a preview that already fit the
documented code-point budget was still shortened. Both previews now share
truncateToCodePoints in core/utils/text, which measures and cuts by code
points, the same rule briefReason in core/scheduled-tasks/format.ts
already follows.
Add unit tests for the helper and a DOM regression test covering both the
fits-exactly and the cut-inside-an-emoji cases.
* fix(frontend): count code points in the artifact-cell expand gate too
The expandable gate still measured value.length (UTF-16 units), so a
120-code-point cell ending in an astral character opened the expand
control even though the inline preview already showed the whole cell —
the same units-vs-code-points mismatch the preview cut fixed.
The gate now counts Array.from(value), and the boundary test asserts
the fits cell renders as a plain cell with no expand control while the
overflows cell keeps its button and whole-emoji cut.
Suggested-by: gdut4140 (port of the #7509 expandable gate)
* fix(sandbox): derive the AIO sandbox ID off the event loop in async acquire
_acquire_internal_async offloads every acquire layer except the sandbox ID
derivation, which ran inline. _sandbox_id_for_thread checks the thread's
skills_view directory with exists(), and resolving it goes through
Paths.base_dir, so each async thread acquire made filesystem syscalls on the
event loop. The strict blocking-I/O gate raises on it.
Run the derivation through the same owned-worker helper as the reuse and
warm-pool layers around it, and anchor it in the blocking-I/O suite for both
the default and the policy-scoped ID.
* docs(changelog): reference #7507 for the AIO async acquire sandbox ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): backfill run.terminal on confirmed recovery takeover
* fix(runtime): isolate recovery terminal publication failures
* fix(runtime): fence recovery terminal publication lifecycle
* fix(runtime): unify recovery receipt publication
* docs(runtime): document lifecycle guard lock constraints
* fix(runtime): reuse event transaction for recovery guards
Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): make read_file_chunks cancellation-safe for worker handoffs
Cancelling a project document read had three unsafe windows around the
dedicated file-I/O worker: (1) a cancellation landing between the open
worker completing and the result reaching the coroutine leaked the
handle (the cleanup path never saw it); (2) the cleanup close could
start while an admitted read was still running; (3) a repeated
cancellation let the caller escape before the close worker settled
(#7491).
Gate every admitted operation with the existing await_drained
primitive: cancellation is delivered only after the operation settles,
the open result is retained in a list cell the moment the worker
returns it (so the cleanup close always has the handle), and waiting
for the next chunk via the yield stays immediately cancellable.
Regression tests gate the real executor with asyncio events at the
worker boundary (real files, no mocks): all three windows are red on
unmodified main and green with the fix; the happy path is unchanged.
The dispatch-spy promotion tests that import app.gateway.routers need
the full backend dependency set and could not run in the minimal local
env; they fail identically on unmodified main (verified).
Fixes#7491
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: fix _gated_offload return annotation to the unpacked tuple
Review nit: the helper returns (calls, gated) and every call site
unpacks it as a pair, but the annotation claimed list[str]. Type it as
the tuple it actually returns.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: pin missing-path FileNotFoundError in read_file_chunks
Convergence item from the duplicate-pair verdict (#7493): the
open-failure path must surface FileNotFoundError as-is — the cleanup
finally skips the close because retained is empty (no handle was ever
opened), rather than raising AttributeError from closing None.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
---------
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
Two Gateways booting the same missing database both reach CREATE DATABASE and
only one wins; the loser aborted startup with duplicate_database (42P04) even
though the winner's catalog row was already committed and the target was
usable. Follow the SQLSTATE, like the missing-database check beside it does, so
the loser falls through to its retry. Every other failure -- permission denied
included -- still propagates untouched.
* fix(scheduler): keep manual-trial launch origins stable across retries
A manual trial's synthetic queued row omitted scheduled_for, and _launch_provenance used the current now for manual triggers. A retry at a later time therefore produced a different origin, which is part of start_run's per-occurrence idempotent input: the retry either conflicted with the first attempt or submitted a different request. Include the admitted scheduled_for in the synthetic row and use the occurrence time for every trigger, falling back to now only when a legacy caller supplies none.
Tests: a manual queued row retried at two later clocks produces the same origin and keeps the admitted scheduled_for; existing manual/scheduled provenance and title tests stay green. Full scheduler suite: 1108 passed / 130 skipped; guidance 0 errors / 0 warnings; ruff clean.
* test(scheduler): drive manual retry through requeue and drain
The unit provenance test pins the stable-origin function directly. Add a driven case that admits a manual occurrence, hits a busy-thread ConflictError so the row requeues, then drains it at a later clock and asserts both launches carry the admitted scheduled_for and the same full origin.
* test(scheduler): pin the immediate queue row in the retry test
Capture the queued mappings passed to _attempt_queued_run so the driven test asserts both halves explicitly: the synthetic immediate-launch row and the durable row drained later both retain the admitted scheduled_for.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome
* fix(gateway): adopt the family helper so a published edit stays silent
* fix(frontend): allow non-ASCII local parts in auth password forms (#7488)
The sign-in, register, admin-setup and change-password forms relied on the
browser's type="email" constraint validation, which rejects any non-ASCII
local part before the submit handler runs. The backend identity schema
deliberately accepts such addresses (see emailSchema in @/core/auth/types),
so an account created via the API, an administrator or OIDC could never sign
in through the password form.
Forms are now noValidate, and the permissive isValidEmail helper (one @,
non-empty local part and domain, no whitespace) runs in the handlers instead,
staying strictly weaker than what the backend accepts.
* fix(auth): normalize emails and restore form password checks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): include markdown modification time in ownership records
* test(uploads): reject same-size edited companions on Windows
* fix(uploads): distinguish legacy companion records
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(checkpoints): isolate cached history payloads from state readers
Copy nested snapshot seeds and pending-write payloads at both cache boundaries.
Keep user copy callbacks outside the registry lock so readers cannot poison
later checkpoint materialization or deadlock by re-entering the cache.
Cover sync/async state accessor reads, nested message payloads, disabled-cache
controls and callback lock ownership.
* docs(checkpoints): document synchronous history copy costs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): make docs search work on desktop and mobile
Docs search on deerflow.tech was broken twice over: no build step ever
generated the Pagefind index that Nextra's <Search> loads, and the custom
docs header that replaces Nextra's navbar never rendered a search box.
- `pnpm build` now runs `search:index` (pagefind 1.5.2, pinned) to write
public/_pagefind/ (gitignored).
- The landing Header and its mobile sheet take an optional `search` slot;
only the docs layout fills it, with localized labels and an aria-label.
- Each docs page carries a Pagefind `lang` filter and queries are scoped to
it, since the root <html lang> is fixed and would mix en/zh results.
- globals.css restyles the results panel with app tokens because Nextra's
<Head> variables are unset, lifts it above the mobile sheet, and keeps it
tappable; the sheet no longer closes before a result navigates and closes
after navigation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(frontend): add DESIGN.md describing the shipped design system
Records the existing tokens, typography, components, docs (Nextra)
overrides, motion, accessibility and copy rules, plus the landing-only
decorative exceptions, so UI changes have something to be checked
against. Linked from frontend/AGENTS.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(frontend): address Codex review of docs search
- Chinese search: the root <html lang> is fixed, so Pagefind indexed zh
pages with the English tokenizer and multi-character words (e.g. 偏好)
never matched. scripts/build-search-index.mjs now sets each page's lang
from its route before indexing, producing separate en-US and zh-CN
indexes, and DocsHtmlLang syncs <html lang> at runtime (resetting
Pagefind on a language switch) so queries hit the right index.
- Mobile menu: reset the open state when the route changes instead of
comparing against the saved path, so Back no longer reopens the sheet.
- Shortcut: stop mounting a second <Search> in Nextra's unused mobile menu,
which captured Cmd/Ctrl+K and focused an offscreen input at 640-767px.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(frontend): keep guidance within size budget
* fix(frontend): address docs search review feedback
Keep Nextra results within the mobile sheet scroll boundary and dismiss same-hash selections by pointer or keyboard. Rebuild Pagefind output without stale fragments and replace UI source-string checks with behavioral coverage.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): reject write_file payloads that are the elision placeholder
The tool-output budget middleware replaces a superseded write_file payload
with a deterministic placeholder in the model-bound request only. That
placeholder is text the model sees in its own history, so it can imitate it
and submit it back as real write_file content: #7319 reports a delivered
file that was 194 bytes of placeholder.
write_file now rejects a payload that is the placeholder (whole payload,
allowing only surrounding whitespace) with an actionable error telling the
model to read_file the target path and write the real content; nothing is
written. The match is whole-payload on purpose: a substring guard would
block legitimate edits to this repository, whose constants.py and
persistence docs quote the template verbatim.
The placeholder wording moves to one shared constant,
WRITE_ELISION_PLACEHOLDER_TEMPLATE in deerflow.constants, which the
middleware renders from and the guard matches on, so the two cannot drift.
Regression tests fail on base (7 failed, 6 passed) and pass with the fix;
near-miss payloads that merely quote the placeholder still write.
Refs #7319
* fix(sandbox): match complete write elision placeholders
* docs: clarify write placeholder validation scope
* chore: re-trigger CLA status check
* fix(sandbox): preflight async writes and trim guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): preserve Moonshot cache_write_tokens in usage metadata
langchain-openai 1.2.1's _create_usage_metadata reads
prompt_tokens_details.cached_tokens but not cache_write_tokens, so the
PatchedChatDeepSeek conversion dropped the cache-creation count that Kimi K3.1
reports, and DeerFlow's usage journal never saw it (#7426).
PatchedChatDeepSeek now overrides _create_chat_result to restore the raw SDK
cache_write_tokens as input_token_details["cache_creation"].
Offline regression: a real OpenAI ChatCompletion carrying
prompt_tokens_details {cached_tokens: 100, cache_write_tokens: 256} converts to
input_token_details {cache_read: 100, cache_creation: 256} (red before, green
after); the existing 14 patched_deepseek tests and the managed-provider suites
stay green.
* fix(models): restore cache_write_tokens on the streaming path too
Streamed runs (web UI messages-tuple) build usage metadata in
_convert_chunk_to_generation_chunk -> _create_usage_metadata, which ignores
cache_write_tokens, so the non-streaming-only override still lost the count on
the path streamed runs actually take (review follow-up on #7428).
_convert_chunk_to_generation_chunk now mirrors _create_chat_result's restore
for the final usage chunk. Note chunk["usage"] is JSON null on non-final
chunks, so the lookup uses `or {}` instead of a .get default (a None usage
would otherwise raise AttributeError inside every chunk and break
test_model's connection probe).
* fix(harness): keep timestamped output-name fallbacks distinct
`output_name_candidates()` sampled `datetime.now(UTC)` inside the yield
loop, so its five timestamped fallbacks were only distinct when the host
clock advanced between two generator steps. On a coarse clock (Windows
advances in sub-millisecond steps) all five produced the same name, and
the capture tools lost retries before failing with
`could not reserve an output filename`.
Sample the clock once when the first fallback is reached and offset each
attempt by one microsecond, so distinctness no longer depends on the host
clock. Add a regression test that freezes the clock, and update the
exhaustion test to hold all five distinct stamps.
* docs: describe the coarse clock accurately
---------
Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
