223 Commits

Author SHA1 Message Date
spud
8f08df9b48
fix(sandbox): fence ambiguous AIO session creation ownership (#5711)
* fix(sandbox): quarantine ambiguous shell session creation

* fix(sandbox): quarantine ambiguous bash session creation

* fix(sandbox): recycle sandboxes with ambiguous session creates

* fix(sandbox): keep quarantined release teardown non-raising

* docs(sandbox): keep guidance within instruction budget

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-22 20:48:22 +08:00
spud
74f006951c
fix(sandbox): bound AIO list_dir with a directory deadline (#5662)
* fix(sandbox): bound remaining shell session cleanup requests

release_command_scope(), close()'s scoped-session drain, and close()'s recovery-session cleanup called shell.cleanup_session() without a request budget, so a stalled cleanup could hold the thread-key serializer, scoped.lock, or the sandbox lock for the SDK's default transport budget. Pass the existing _bounded_cleanup_request_options() (5s, max_retries=0) at those three sites; _cleanup_session_best_effort() and its swallow-and-log contract are unchanged, as are create_session and list_dir lifetimes.

* fix(sandbox): bound AIO list_dir with a directory deadline

Give AioSandbox.list_dir its own 60s directory deadline with a 65s no-retry
host envelope, independent of bash_command_timeout. Preserve #5634's
shell-generation selection: list_dir runs on the current recovery session once
the implicit shell is fenced, and an ambiguous list_dir outcome - transport
timeout or an ambiguous returned status - fences whichever generation actually
executed the request, dropping local recovery ownership and attempting bounded
best-effort cleanup instead of leaving that session reusable. hard_timeout
remains definite termination and keeps the targeted session reusable. Only
completed/None results are parsed, so a partial find is never returned as a
complete listing.

Session creation RPC lifetime remains out of scope.

* fix(sandbox): recover sessions after transport failures

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-22 19:49:00 +08:00
Keswyk Cheung
ef21855b8d
fix: discard stale todo reminders during context compaction (#5614)
* fix: rebuild todo reminders after context compaction

* chore: remove implementation plan from PR

* refactor: share todo reminder message name

* docs: keep agent guidance within CI size budgets

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-22 17:45:01 +08:00
Hyeonsang Cho
90f0866fa8
fix(uploads): stop deleting a converted companion we cannot prove we wrote (#5673)
* fix(uploads): stop deleting a converted companion we cannot prove we wrote

Conversion names a document's Markdown companion after the document's
stem and falls back to a _N suffix when that name is already taken, so
the .md beside a document may belong to another document sharing the
stem, or to the user. Delete removed it anyway: uploading a.docx and
a.pdf produces a.md and a_1.md, and deleting a.pdf destroyed a.docx's
companion while orphaning a.pdf's own.

Delete now removes only the file it was asked to remove. The companion
stays listed and can be deleted on its own. Orphans are the cost of not
guessing; issue #5672 covers giving companions a provable owner, which
is what a safe cleanup needs, along with the two related readers that
still guess (the outline injected for a document and the agent's file
listing).

convertible_extensions loses its last use and is dropped from the
signature and both call sites. The gateway router keeps importing
CONVERTIBLE_EXTENSIONS for the ingestion bridge and now declares it in
__all__, where that module documents its re-exports.

* docs(changelog): note that delete keeps the converted markdown (#5673)
2026-09-22 14:50:11 +08:00
spud
61a99a1e8a
fix(sandbox): enforce bash command timeouts in AIO sandbox (#5634)
* fix(sandbox): enforce AIO shell command timeouts

Map Sandbox.execute_command(timeout=T) onto the AIO legacy shell as a server-side hard_timeout=T plus a bounded host request (ceil(T+5)s, max_retries=0), preserve and render the upstream status (hard_timeout -> terminated + Exit Code: 124; no_change_timeout -> may still be running), keep no_change_timeout from preempting hard_timeout, and restrict ErrorObservation replay to completed results.

* fix(sandbox): contain ambiguous AIO command outcomes

Treat a transport timeout and statuses terminated/no_change_timeout/unknown as ambiguous: never replay the current command, fence the scoped or recovery session generation with bounded best-effort cleanup, and never adopt a replacement session for those statuses. hard_timeout and completed keep the session reusable. Align the env-bearing bash.exec path with the same contract and make its retry result authoritative.

* fix(sandbox): wire AIO command timeout from provider config

Restore bash_tool's non-local behaviour (other providers keep their own defaults) and let AioSandboxProvider read sandbox.bash_command_timeout as the sandbox's default_command_timeout at every construction site, with explicit per-call timeouts still winning. Widen the config field to float with allow_inf_nan=False and read the provider value with the module fallback so partial-config providers keep working.

* docs(sandbox): document AIO command timeout semantics

Document the provider-scoped deadline (LocalSandbox/AioSandbox/OpenSandbox wire it; other providers keep their defaults), the semver-vs-frozen-:latest image behaviour, the no-replay guarantees, and trim the sandbox guidance back under its size budget.

* docs(sandbox): scope the wedge-resistance claim to command requests

willem-bd's review on #5634 reproduced that list_dir and both create_session call sites still make unbounded SDK requests while holding the same sandbox lock, so the guide must not read as if every AIO request is bounded. State that the bounded T+5s host wait applies to command requests and name session-creation and file/list RPCs as not covered; bounding those remains a separate operation-deadline/control-plane follow-up.

* fix(sandbox): make list_dir honor the shell generation fence

list_dir previously always targeted the implicit persistent shell, so after an ambiguous command outcome fenced that generation (#5634) a listing could re-enter it. Route default-shell target selection through _ensure_default_shell_session_id so both execute_command and list_dir reuse the explicit recovery session, creating one only when the implicit shell is fenced. No list_dir deadline, hard_timeout, request budget, retry, or status semantics are added; those remain scoped to #5644.
2026-09-22 11:29:38 +08:00
asts
0ff2e9ddcf
fix(mcp): honor configured stdio working directories (#5643)
* fix(mcp): honor configured stdio working directories

* fix(mcp): preserve defaults for empty working directories

Treat empty stdio cwd values as omitted, including unresolved environment references. Add real subprocess regressions for discovery and pooled-call defaults, plus direct connection-dictionary coverage.
2026-09-22 11:15:06 +08:00
Hyeonsang Cho
29d285731b
fix(uploads): convert the bytes we wrote, not the name they landed under (#5611)
* fix(uploads): convert the bytes we wrote, not the name they landed under

Document conversion re-opened the upload by name after it was already
visible in the thread's uploads directory: the Gateway converted the
committed file_path, and DeerFlowClient converted the copy it had just
placed there. That directory is writable from local and AIO sandboxes,
so a process watching it can replace the name with a symlink in the
window between the upload landing and the converter opening it. The
converter then reads whatever host file the link points at and writes
that content back into the thread as the .md companion, which the
sandbox can read. Reproduced end to end on both paths with a real xlsx:
the companion came back holding the host file's rows.

The Gateway now duplicates the descriptor of the staged file before the
link-commit, copies those bytes into a private directory outside the
uploads tree, and converts there. A descriptor cannot be redirected by
replacing a name, so the conversion input is the content this request
wrote. The client converts the caller's own source file instead of the
copy in uploads; the source is the file the caller handed in, which the
sandbox cannot reach.

Both already wrote the companion without following a symlink, so only
the read side changes. The uploads directory still receives exactly the
same files.

* docs(changelog): note upload conversion source fix (#5611)

* fix(uploads): close the conversion descriptor when staging its copy fails

Review follow-up. The private directory for the conversion copy was
created before the try that owns the duplicated descriptor, so a failure
there — a full or unwritable temporary filesystem — propagated without
closing it. The upload's own cleanup only unlinks the committed name and
releases the sandbox lease, so the descriptor stayed open for the life of
the process and kept the unlinked staged bytes allocated with it; repeated
failures accumulated both.

Directory creation now happens inside that try, and the finally removes
the directory only once it exists.

* fix(uploads): keep the conversion descriptor owned across cancellation

Review follow-up. run_file_io cannot interrupt its worker, so cancelling
the await around os.dup only abandoned the result: the duplicate was
created moments later with nothing left to close it, and it pinned the
staged bytes of an upload whose name the cleanup had already unlinked.
Cancellation after the duplication was just as leaky, because the
commit-path handler caught Exception and CancelledError is not one.

The duplication now runs as its own task, shielded from the caller's
cancellation, and closes its own result when the caller is gone by the
time the worker finishes. The commit path catches BaseException, closing
the descriptor it already owns before re-raising.

Both windows are pinned: one test stalls the duplication worker after it
allocates and cancels ingestion, the other stalls the commit so the
cancellation lands while the descriptor is owned.

* fix(uploads): drain the conversion copy so its descriptor always closes

Review follow-up. The copy worker owns the duplicated descriptor and
closes it in its own finally, but a bare await let a cancellation cancel
the executor job while it was still queued: the worker never ran, so that
finally never ran either, and the enclosing scope had already handed
ownership away and saw None. Draining also keeps a late worker from
writing into a private directory this scope has since removed.

The copy now goes through await_drained, the shield-and-drain helper the
Gateway already uses for offloads that must not be abandoned mid-flight.
Pinned by a test that holds the copy job queued, cancels ingestion, then
releases it and requires the descriptor to come back closed.
2026-09-21 07:34:09 +08:00
Eilen Shin
bd995a6a26
fix(agent): align unattended prompt with tool policy (#4919)
* fix(agent): align autonomous interaction guidance

* fix(agent): harden interaction policy selection

* fix(gateway): protect legacy interaction flags

* fix(channels): honor explicit interaction mode

* docs(agent): reduce inherited guidance size

* fix(agent): honor unattended policy across approval paths

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-20 18:57:49 +08:00
Yeager0204
f2463e6d4a
docs(sandbox): fix Apple Container verification instructions (#5605) 2026-09-20 17:56:30 +08:00
Hyeonsang Cho
2b6254f76d
fix(client): stop embedded uploads from writing through symlinks (#5578)
* fix(client): stop embedded uploads from writing through symlinks

DeerFlowClient.upload_files copied each file with shutil.copy2 and let
convert_file_to_markdown write the companion straight into the uploads
directory. Local and AIO sandboxes can write to that directory, so a
symlink planted at an upload name or at the companion's name was
followed: the upload's bytes and the converted Markdown landed in
whatever host file the link pointed to, and the call reported success.
The Gateway refuses symlinked destinations and the IM channels write
through write_upload_file_no_symlink; the embedded client never adopted
either.

Uploads now go through copy_upload_file_no_symlink, a new helper next
to write_upload_file_no_symlink. It keeps copy2's content, permission
bits and timestamps, so files stay readable to Docker sandboxes, but
applies them to the descriptor opened with O_NOFOLLOW and opens the
source first so a missing source cannot truncate an existing upload.
As in the Gateway, a file with an unsafe destination is skipped and
listed in skipped_files, success turns false, and the message says how
many were skipped. The companion is converted inside a private temporary
directory and then written with write_upload_file_no_symlink; one whose
name is unsafe is left out like a failed conversion, and the original
upload is kept.

* docs(changelog): note embedded upload symlink fix (#5578)

* fix(client): keep copy2's same-file guard and companion permissions

Review follow-up. Two regressions in the previous commit.

copy_upload_file_no_symlink opened the destination before comparing it
with the source, and that open truncates. Passing a file that already
sits in the thread's uploads directory therefore copied an emptied file
over itself: the upload reported success with size 0 and the original
bytes were gone, where copy2 raised SameFileError and left the file
alone. The destination is now compared with the source through
os.path.samestat before anything is opened, so identity — including a
hardlink or another spelling of the same path — raises SameFileError as
before.

The Markdown companion was published with write_upload_file_no_symlink,
which creates a new file as 0600 and ignores the converted file's mode.
Under umask 022 the companion became 0600 while its own document stayed
0644, so a bind-mounted sandbox running as another uid could read the
upload but not the Markdown the response advertises. It now goes
through the same copy helper as the upload, which preserves the
converter's permission bits.
2026-09-20 16:22:24 +08:00
Hyeonsang Cho
f9f3127dc1
fix(uploads): delete the requested upload, not a symlink's target (#5547)
* fix(uploads): delete the requested upload, not a symlink's target

delete_file_safe resolved the requested path before unlinking it. The
uploads directory is writable from local and AIO sandboxes, so a
symlink planted under an upload name was followed: deleting alias.pdf
removed the victim.pdf it pointed to, and the companion cleanup then
removed victim.md, while the link itself survived and the call reported
"Deleted alias.pdf". A link resolving outside the directory was already
refused by the traversal check, so the damage stayed inside the
thread's uploads.

The function now checks and unlinks the requested entry itself and
treats a symlink as not found, the same way list_files_in_dir already
hides it. unlink() never follows the final component, so a file swapped
for a link between the check and the unlink removes only the link.
Tests cover the helper, the Gateway DELETE route, and
DeerFlowClient.delete_upload.

* docs(changelog): note upload delete symlink fix (#5547)
2026-09-18 19:56:09 +08:00
zhangwei-way
b6503e9a35
feat(knowledge): add per-message RAGFlow retrieval scope (#5238)
* feat(knowledge): integrate RAGFlow retrieval and management

* test(knowledge): cover merged listing tool

* feat(knowledge): add per-message retrieval scope

* chore(docs): remove unrelated document

* docs(knowledge): add interaction screenshots

* feat(knowledge): simplify scope selector trigger

* docs(knowledge): refresh selector screenshot

* feat(knowledge): defer standalone management

* docs(knowledge): show chat-only scope UI

* fix(knowledge): honor scope on clarification replies

* fix(knowledge): harden scoped replay validation

* docs(knowledge): clarify replay scope precedence

* fix(knowledge): keep provider settings on tools

* fix(config): preserve tools-only knowledge settings

* fix(knowledge): submit custom assistant identity

* refactor(knowledge): trim PR scope changes

* fix(knowledge): sanitize document scope display

* feat(knowledge): enable scope selection in main chat

* fix(knowledge): emphasize active scope icon without button frame

* fix(knowledge): close context scrubbing and refresh e2e checks

* fix(knowledge): preserve idempotent canonical retries

* fix(knowledge): accept promptless conversation runs

* style(knowledge): format backend regression tests

* chore(knowledge): trim PR scope and fix frontend format

* fix(knowledge): remove shared-scope notice

* fix(knowledge): remove scope persistence notice

* docs(knowledge): include main chat in catalog scope

* fix(knowledge): preserve scope recovery and upgrades

* fix(config): preserve LightRAG knowledge upgrades

---------

Co-authored-by: foreleven <for-eleven@hotmail.com>
2026-09-18 16:59:31 +08:00
zeng-bohan
ce3e64242b
feat(gateway): checkpoint retention service on the #4189 deletion contract (#5308)
* feat(gateway): thread checkpoint retention service on the #4189 deletion contract

Implements exactly the two contract-proven deletion shapes (trailing
duration-only leaves, opt-in leaf sibling branches) with head-chain
protection, explicit id protection, a strict pending-writes guard, and
joint writes-row cleanup. Head resolution uses LangGraph's time-ordered
checkpoint ids; storage deletion mirrors the contract's per-backend data
model. Ships without a production trigger by design. Validated against
the contract suite (12 passed) plus 14 service scenarios across memory
and SQLite; Postgres paths are gated on TEST_POSTGRES_URI.

Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>

* fix(gateway): survivor-reachability blob GC and memory blob stats in retention service

Aligns the deletion service with the review-hardened contract: blob rows
are garbage-collected in a whole-thread pass against surviving
checkpoints' channel_versions (a real duration-only leaf shares its
parent's versions, so per-checkpoint version deletion would corrupt the
surviving state), the memory branch of the stats helper counts
saver.blobs and returns the full normalized shape, and per-node channel
versions are collected during the graph pass that already exists.

Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>

* fix(gateway): address review findings on checkpoint retention service

Resolves the review at a479cfe (willem-bd):

- Untested savers now fail fast: an explicit isinstance allowlist
  (InMemorySaver / AsyncSqliteSaver / AsyncPostgresSaver) raises
  NotImplementedError before any row is read or deleted, so a shallow or
  third-party saver can never issue partial DELETEs.
- The chain walk ends (break) instead of raising KeyError when the head's
  ancestor row is missing, matching the deletable loop's tolerance for
  missing parents.
- enforce_thread_retention takes an optional per-thread lock and documents
  the concurrency requirement: classification and deletion are two separate
  passes, so callers must serialize per-thread mutation (runtime
  _checkpoint_thread_lock) or guarantee quiescence.
- Dropped the dead mid-run guard: CheckpointTuple has no `next` field in
  langgraph-checkpoint 4.1.1, and pending_writes is populated for committed
  writes too (verified on the list path), so neither is a usable mid-run
  signal; the caller-held thread lock is the actual protection.
- Removed the write-only _node_step/_Node.step and fixed the head-selection
  docstring (newest by checkpoint id, not (step, checkpoint_id)).
- Documented the E1 leaf / history fast-path interaction in the contract doc
  and module docstring: the wiring PR must sequence retention away from
  history reads or adopt a policy that spares cache-carrying leaves.
- Added regression tests: unsupported saver, missing ancestor row, thread
  lock parameter.

Validation: test_checkpoint_retention_service 18 passed / 8 postgres-gated
skipped; contract + lineage suites 18 passed / 6 skipped; ruff check and
format clean.

* fix(retention): count non-empty writes dicts on memory saver

- _checkpoint_ids_with_writes now requires a non-empty writes dict on
  InMemorySaver: the empty phantom entry for checkpoints whose task wrote
  nothing no longer counts as "owns writes rows", so the default E1 pruning
  reaches the memory backend again (it was a silent no-op there).
- test_runtime_duration_leaf_pruned_by_default runs the shipping default
  (strict_pending_write_guard=True) and proves E1 is reachable out of the
  box on every backend; the stale override and its wrong SQLite premise
  are dropped.
- document that _checkpoint_thread_lock is non-reentrant: a caller already
  holding it must not pass it in, or retention self-deadlocks.

* test(checkpoint-retention): fix stray duplicated def token in test_duration_link_protected_after_next_run

The previous push left `async def def test_...` at line 244, which made the
module unimportable and failed collection of the whole suite (and ruff
format --check). Local copy was already correct; this commit re-pushes the
clean file. 18 passed / 8 postgres-skipped verified from a head worktree.

* fix(gateway): make retention correct on Postgres and fail closed on a bad cap

* validate max_delete_per_run before any store read: a negative cap used to
  widen the batch (Python slicing) instead of being rejected;
* report identical before/after stats for an empty thread instead of returning
  before stats_after is collected;
* protect each namespace's resume head and ancestor chain, so a persistent
  subgraph's latest checkpoint is no longer treated as a sibling leaf;
* read Postgres columns through a row-factory-agnostic helper (the PG savers
  open cursors with dict_row, where positional access raises KeyError: 0);
* classify the duration-only leaf without relying on metadata["writes"], which
  the Postgres saver strips via get_serializable_checkpoint_metadata.

Verified locally on memory, SQLite and a real Postgres 16 instance (62 passed,
0 skipped): the E1 shape now fires on Postgres, which no backend test covered
before CI ran the Postgres lig.

Signed-off-by: zeng-bohan <zengbh1@gmail.com>

* test(gateway): pin the Postgres-shape duration classifier; report per-namespace heads

- Deterministic regression for _mark_duration_leaves_without_the_marker:
  hand-put the Postgres round-trip shape (writes marker popped, source=
  update + accumulated run_durations + channel_versions identical to the
  parent) and assert the shipping default prunes it; a control that bumps
  one channel version (the client update_state shape) with otherwise
  identical metadata stays protected. Both legs run on memory and SQLite,
  so the class cannot silently re-widen (a resumable head losing head
  protection) or re-narrow (E1 never firing on Postgres) without a
  locally-executing test failing.
- RetentionReport.protected_head_id -> protected_head_ids: heads are now
  selected per namespace, so the report carries every namespace's head
  (root key = what an unsaved aget_tuple resolves) instead of only the
  global max - reshape it before the wiring PR starts consuming reports
  for audit/aggregation.

---------

Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Signed-off-by: zeng-bohan <zengbh1@gmail.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-18 16:46:19 +08:00
Weng Qiang
a23dbdd837
feat(tavily): support configured search domain filters (#5513) 2026-09-18 07:25:49 +08:00
George Pickett
6b0ebe6702
feat(mcp): identify DeerFlow in Parallel request headers (#5501) 2026-09-17 11:35:13 +08:00
Dan Caldr
f2857ad3ff
fix(sandbox): default to loopback bind on Docker Desktop for DooD sandboxes (#5446)
* fix(sandbox): default to loopback bind on Docker Desktop for DooD sandboxes (#5445)

* fix(sandbox): memoize desktop detection and clarify bind host docstring (#5445)

* fix(sandbox): latch desktop detection on success only to permit retry on transient failure (#5445)

* fix(sandbox): restrict desktop loopback bind to local DooD hostnames (#5445)

* fix(sandbox): add Desktop legacy aliases and parametrize DooD host tests (#5445)
2026-09-17 07:55:22 +08:00
Onefly
d8d110c637
fix(sandbox): prevent AIO subagent session eviction (#5178)
* fix(sandbox): prevent AIO subagent session eviction

* fix(sandbox): address PR 5178 review issues

* fix(sandbox): handle transient session and metadata failures

* fix(sandbox): fence capacity upgrades and validate reused limits

* docs(sandbox): restore list indentation and trim guidance

* fix(ci): stabilize Buzz persistence test and trim sandbox guidance

---------

Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-17 07:45:19 +08:00
Zeren Wang
a58ab484a6
feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443)
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash

Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.

Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
  admission pinning (both server-owned sets + worker hoist); latest-only
  request-scoped <project> block via DynamicContextMiddleware
  wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
  prefix + marker + provenance, never persisted); journal audit
  fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
  locked check-and-set; hash-qualified immutable shelf storage with
  Paths helpers; upload/list/content/delete-to-trash routes; project
  delete trashes the shelf in-transaction; request-scoped bounded
  <documents> index with honest count/shown + actionable overflow note;
  list_project_documents/read_project_document tools registered only on
  pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
  parity); POST from-thread with provenance; attach-to-thread with
  lock-staged copy (archived source allowed); read-only thread-files
  view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
  file moves), purge (continuous row lock across unlink/delete/commit,
  retryable on FS errors), retention sweep (lazy + startup, 24h orphan
  guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
  archived banner, content-missing rows), /workspace/trash route,
  sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
  + specs.

Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
  artifact + content responses; unified unsandboxed-iframe PDF preview
  (fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
  claims + os.link commit with suffix retry; same-name re-upload now
  unique-names instead of replacing); hidden staging only, no visible
  placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
  revalidation; drain locked filesystem work on cancellation; preserve
  bytes when an insert's commit state is uncertain (including trashed
  rows)
- original-integrity checks before serving text or cached conversions;
  content_missing surfaced in list responses (UI reads the flag, no
  409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
  declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
  rejects fractional/infinite values; composer counts staged
  attachments; pending attachments persist until submission or removal;
  in-flight instruction/rename edits survive save refetches; shelf and
  trash pagination; conversation-file and thread-files pages stay
  subscribed to refetches

Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.

Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
  bounded: the indexed expiry purge still runs on every trigger
  (GET /api/trash/documents, POST /api/trash/purge) while the
  O(all rows + all files) reconciliation is throttled to one run per
  user per 15 minutes (process-local, per-user window). The startup
  sweep now runs as a background task instead of blocking gateway
  readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
  like the render path, so a pasted, fenced <project>/<documents> snippet
  survives markdown export while real injected blocks (never fenced) are
  still removed. Fence regexes moved to a dependency-free leaf module to
  avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
  attribute (the upstream e2e contract locates it via :not([title])), and
  the upstream artifact-preview spec now pins the new contract: PDFs
  render unsandboxed, images keep sandbox="".

* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title

- Shutdown cancelled only the shield around the background startup sweep,
  so an all-users reconciliation that outlived the 5s budget kept walking
  rows and files while the document repo and DB engine were disposed
  underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
  which cancels the task and drains it before worker exit: the shield
  keeps the wait bounded, the cancel makes it final (CancelledError lands
  at the sweep's next await, and `_run_startup_trash_sweep` only catches
  `Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
  previous fix round, leaving the PDF frame without an accessible name
  while its siblings keep theirs. Restore it (WCAG frame titles), assert
  it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
  instead of `iframe:not([title])`.

* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel

`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.

* fix(projects): round-4 review — make Empty trash delete what it confirms

`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.

Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.

Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
2026-09-16 18:46:18 +08:00
Totoro
5b591a9039
feat(gateway): accept conversation references in run context and report the capability (#5463)
* feat(gateway): accept conversation references in run context and report the capability

LangGraph SDK clients build a fixed run body and drop unknown top-level
fields, so they cannot send the conversation_references field from #5399.
RunCreateRequest now lifts context.conversation_references into the
top-level field before validation, so it keeps the same bounds and error
locations, and drops it from context, so it never reaches the merged run
context or the checkpointed configurable. Sending both is a 422.

GET /api/features reports conversation_references {enabled, max_references}
with the same "tool is configured" predicate as run admission, so a client
can hide an entry point on deployments without the tool.

Related to #5398.

* fix(gateway): report the field type error for a malformed top-level reference list

A malformed top-level conversation_references sent alongside a context
list now fails with the field's own type error instead of the conflict
message. The tool-configured predicate reads tool.use directly, and the
features test doubles carry that attribute like every real ToolConfig.

* fix(gateway): treat every list-like top-level reference value as a conflict

Pydantic's lax mode coerces tuples, sets, frozensets and deques into the
list[str] field, so a direct Python caller passing one of those together
with context.conversation_references now reports the conflict instead of
slipping both grants through. Unreachable over HTTP, where JSON has no
such types.

* fix(gateway): ask pydantic whether a top-level reference value is list-like

Enumerating list-like types cannot track pydantic's lax acceptance set
(generators, UserList, dict key views also coerce into list[str]). The
conflict guard now validates the top-level value with a TypeAdapter for
list[Any]: whatever pydantic would coerce reports the conflict when it is
non-empty, and whatever it rejects still surfaces the field's own type
error. Regression tests cover deque, UserList, dict keys and a generator,
plus rejected scalars.

* fix(gateway): probe top-level references with the field's own annotation

The conflict guard now validates the top-level value with the exact item
annotation the field uses, so its acceptance set is the field's rather than
a superset: an item the field rejects (an empty string, a non-string, the
ints of a range or dict view) surfaces the field's own item error instead
of a conflict. The annotation is shared through one alias so the two cannot
drift.

* fix(gateway): materialise a one-shot iterator before probing top-level references

The item-validating probe could consume a generator while collecting an
item error, after which the field re-validated the exhausted iterator,
coerced it to [] and let the request through with the key still in
context. Iterators are now read once into a list that both the probe and
the field validate, so a bad item is reported at its index and a valid
generator is kept.

* fix(gateway): materialise every once-walkable iterable before probing references

Pydantic coerces any iterable into the list field, and an object whose
__iter__ hands out a generator once is not an Iterator instance, so the
previous gate let it reach the probe and be consumed. The lift now reads
every iterable except lists, tuples and the shapes the field rejects as a
whole (str, bytes, dict) into a list first, so the probe and the field
always validate the same items.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-09-16 15:37:57 +08:00
PeaceMaker-best
14c9d44440
feat(runtime): persist tool-progress phase transitions (#5214)
* feat(runtime): persist tool-progress phase transitions

Record bounded warn, block, and recover decisions for lead and task subagent runs while preserving event-loop isolation, fail-open behavior, and concurrent transition order.

* fix(runtime): trust server-owned tool progress attribution

* fix(runtime): centralize trusted audit attribution

* fix(runtime): preserve complete tool progress audit state

* docs: trim tool progress guidance to pass size check

* fix(runtime): fence subagent audit recorder loop

* docs(readme): sync tool-progress event coverage

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

---------

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: 嗜鵼 <hy2010hy2010@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-15 08:45:08 +08:00
Totoro
4ad55f598f
feat(conversation): continue reading a cut message by offset (#5434)
* feat(conversation): continue reading a cut message by offset

A referenced message longer than 4,000 characters was cut, and its
suffix could not be read back. Cut messages now carry a continuation
(message_seq, offset). read_conversation(thread_id, message_seq, offset)
returns the next part of that one message, sized to the same
tool-output budget as pages. The read scans only the requested row
under the existing visibility rules and rechecks ownership. Offsets
follow the source's current text; an offset past the end is rejected.

Related to #5398.

* docs(conversation): say continuations ignore limit

A continuation always returns one part of one message, so limit does not apply there. The tool schema now says so instead of discarding it silently.

Related to #5398.

* fix(conversation): stop instead of looping when no text fits the budget

With a read_conversation tool-output budget below the envelope size, the fitted text was empty and the continuation repeated the requested offset, so an agent would repeat the identical call forever. Page and continuation reads now return output_budget_too_small with no continuation.

Related to #5398.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-09-15 08:25:08 +08:00
yang rui
3dc895df4d
feat(models): pace shared RPM budgets before dispatch (#5432)
* feat(models): add shared RPM admission queues

* fix(models): address admission pacing review feedback

* docs: simplify request admission quick start guidance

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-15 07:14:35 +08:00
Totoro
6177b07c06
fix(conversation): clarify reference semantics and keep reader pages inline (#5421)
Document that read permission expiry and source deletion do not erase
text already copied into the destination conversation, and that reads
follow the source's current visible history. Truncated results now tell
the agent to acknowledge the omission and ask for the missing material
before claiming every requirement is covered.

Pages were filled to 20,000 text characters by cutting the last message
that did not fit, and that suffix could never be paged back. They could
also exceed the default 12,000-character tool-output budget, which
externalized the page to a file. Pages are now sized by their serialized
length against the read_conversation tool-output budget; a message that
does not fit starts the next page intact, so only a message over 4,000
characters (or one whose escaped JSON alone exceeds the budget) is cut.

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 15:05:30 +08:00
Totoro
533e30e7f2
[feat] add opt-in conversation reads for Gateway runs (#5399)
* feat: add scoped conversation reads to gateway runs

* refactor: share conversation tool path and reuse parsed text

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-09-14 10:51:01 +08:00
Hyeonsang Cho
1dd48d14d2
fix(models): reuse the Claude Code OAuth token read from a file descriptor (#5411)
* fix(models): reuse the Claude Code OAuth token read from a file descriptor

ClaudeChatModel accepts a Claude Code OAuth token through
CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR, but that handoff can be drained
only once: a pipe returns EOF after the first read and a file descriptor
keeps its advanced offset. _read_secret_from_file_descriptor read it again
on every call and kept nothing. Every ClaudeChatModel instance loads
credentials in model_post_init, and create_chat_model builds fresh
instances per run, so with a descriptor-only handoff the first model
authenticated and every model after it -- including the title model of
the very first run -- had no credential. The Anthropic SDK then raised
"Could not resolve authentication method" before sending a request.

A secret read from a descriptor is now kept for the life of the process,
keyed by (env_var, fd), so a different descriptor is still read fresh.
The read happens under a lock so two threads building their first model
concurrently cannot race one of them to EOF. Empty reads and OSError are
not cached and behave as before; lookup order, config keys, and log
messages are unchanged.

* docs(changelog): reference #5411 in the Claude Code OAuth descriptor fix entry

* test(models): pin that a closed descriptor handoff keeps its token

Review follow-up on #5411: the descriptor secret cache is keyed on the
fd number, which the OS recycles. Folding os.fstat identity into the key
would break the property the cache exists for -- once the handoff fd is
closed after the first read, fstat raises EBADF and every later model
would lose the token again -- and it would still miss a regular file
rewritten in place, which keeps its st_dev/st_ino.

The handoff is fixed at process start, so keep the number as the key and
state the invariant instead: a closed handoff keeps serving its token, a
secret placed on a recycled number is not re-read, and anything handing
over a new secret in-process must clear the cache. A new test pins the
closed-handoff behavior; an fstat-fingerprinted key fails exactly that
test.
2026-09-14 09:23:12 +08:00
yang rui
7513f16e0e
feat(settings): persist account preferences across browsers (#5397)
* feat(settings): persist account preferences across browsers

* docs(settings): scope preference guidance to user persistence

* fix(settings): preserve SSR and fence custom-agent defaults

* test: include user persistence in scoped guidance inventory

* fix(settings): sync explicit edits and preserve local tab updates
2026-09-14 07:25:41 +08:00
yang rui
dfe9a520b9
fix(mcp): isolate pooled sessions by owning event loop (#5396)
* fix(mcp): isolate pooled sessions by owning event loop

* refactor(mcp): remove obsolete eviction cancellation plumbing
2026-09-13 17:13:45 +08:00
Xuehao Xu
56540fab01
fix(gateway): make recursion limit configurable (#5390)
* fix(gateway): make recursion limit configurable

* docs: keep backend guidance within inherited size budget

* fix(gateway): address recursion limit review feedback

* docs(gateway): clarify recursion default scope

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-13 17:08:33 +08:00
Hyeonsang Cho
806a5bd427
fix(gateway): serve XML artifacts as attachments to block same-origin script (#5353)
* fix(gateway): serve XML artifacts as attachments to block same-origin script

GET /api/threads/{id}/artifacts/{path} forced only text/html,
application/xhtml+xml and image/svg+xml to download. Every other XML
document was served inline from the application origin: `.xml` guesses
to text/xml or application/xml depending on the host's mime.types, and
both fell through to the inline text branches. Browsers render any XML
MIME type as a document and run an XHTML-namespaced <script> inside it,
so a report.xml written by a prompt-injected agent and opened from a
chat link executed with the viewer's session: the HttpOnly access_token
rides same-origin fetches, and the double-submit csrf_token cookie is
JS-readable, so state-changing calls are reachable as well.

Treat HTML plus every WHATWG XML MIME type (text/xml, application/xml,
any +xml subtype) and text/xsl, which Blink also renders as XML, as
active content. A single helper owns the rule for both the regular-file
and the .skill-archive-member branches. The artifacts panel already
previews .xml as code through a ranged fetch, so preview and editing
keep working against the attachment response.

* docs(frontend): name XML among the artifacts the Gateway downloads

Review follow-up on #5353: resolveArtifactOpenURL's comment still named
only HTML/SVG as the active content the Gateway serves as a download.
XML documents now join that bucket, so the frontend note matches the
Gateway rule. Comment-only; no behavior change.
2026-09-12 09:15:51 +08:00
wutongyuonce
9f17bbeec7
feat(tools): filter list_uploaded_files by name and extension (#5341)
* feat(tools): filter list_uploaded_files by name and extension

Add optional query and extensions so historical upload discovery can
find older matching files instead of dropping them behind the default
20-item mtime cap.

Fixes #5339

* fix(tools): strip glob stars from list_uploaded_files extensions

Model-supplied tokens like *.pdf were prefixed to .*.pdf and never
matched Path.suffix. Also run ruff format so the backend format gate
passes.
2026-09-12 07:25:19 +08:00
Ryker_Feng
f52818fe5e
feat(skills): export custom skill packages with revision-bound preview (#5332)
* feat(skills): export custom skill packages with revision preview

* docs(gateway): keep export guidance within size budget

* ci: retry checks after transient uv setup download failure

* docs: focus skill export agent guidance on maintenance invariants

* fix(skills): handle export disconnects and bound archive transfers

* docs(gateway): remove redundant export guidance to fit merged budget

* fix(skills): reset export idle deadline after transfer progress
2026-09-11 16:21:23 +08:00
PeaceMaker-best
36ce7590b7
fix(agents): isolate loop detection state by run (#5344)
* fix(agents): scope loop detection state per run

* fix(agents): harden loop scope fallback

* docs: move loop lifecycle detail out of inherited guidance

---------

Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-11 13:45:52 +08:00
wutongyuonce
48a8978b7b
feat(scheduler): add interval schedule type (#5291)
* feat(scheduler): add interval schedule type

Allow scheduled tasks to fire every N seconds from last dispatch, not
only wall-clock cron or a single run_at. Cadence is UTC now+N with no
missed-beat catch-up, bounded by min_once_delay_seconds and 30 days.

* fix(scheduler): let interval tasks create, edit, and keep next run

Create/edit now keep every_seconds. Unchanged interval spec no longer
resets next_run_at, including timezone-only PATCH.

* fix(scheduler): keep non-minute intervals on edit

Stop rounding every_seconds to whole minutes in the form. Values that
are not whole minutes or hours now use a seconds unit so edit/duplicate
round-trips the stored cadence instead of rewriting it and resetting
next_run_at. Document that min_once_delay_seconds is also the interval
floor.

* fix(scheduler): clamp interval seconds to the default 60s floor

The new seconds unit allowed 1–59, which the API rejects under the
default min_once_delay_seconds. Clamp the form to >= 60 and show the
floor next to the preview. Also mention interval in the scheduler
field_doc, matching config.example.yaml.

* fix(scheduler): do not clamp interval amount while typing

Keystroke clamp made 90 become 9 -> 60, then 600, and backspace could
not leave 60. Keep the raw field text and apply the 60s floor on blur
and emit only.

* test(scheduler): cover interval input editing

* fix(frontend): preserve saved interval cadence until edited

* style(tests): format scheduled task router tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-10 16:46:01 +08:00
wutongyuonce
69f0f483eb
feat(scheduler): let scheduled tasks pin a custom agent (#5288)
* feat(scheduler): let scheduled tasks pin a custom agent

Create and update accept optional assistant_id, defaulting to lead_agent.
Custom names are normalized and must already exist for the task owner.
The workspace form exposes the same choice, and duplicate copies it.

Fixes #5286

* fix(scheduler): keep assistant-id PR free of interval tests

Drop the six interval tests that belonged to the interval schedule PR
and fail here because this tree still only accepts once/cron.

Treat lead_agent case-insensitively so LEAD_AGENT / lead-agent store
as the default. Omit unchanged assistant_id on edit so a deleted custom
agent does not 422 unrelated PATCH (rename, reschedule).

* fix(scheduler): format task page and browser tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-10 15:36:49 +08:00
zeng-bohan
a2808e8292
test(checkpoint): retention deletion contract + growth baseline (#4189 item 3) (#5255)
* test(checkpoint): retention deletion contract + growth baseline

Six contract scenarios x memory/sqlite/postgres pin what retention deletions
must never break (branch ancestors, explicit resume targets, pending writes,
duration-only chain links), prove the two safe shapes (leaf sibling branches,
trailing duration leaves), record the full-vs-delta growth baseline in the
normalized bench shape, and add an item 4 probe showing the default
ToolOutputBudgetMiddleware already externalizes oversized tool results.

Refs #4189

* test(checkpoint): make the retention contract load-bearing per review

Review findings from willem-bd and Ricky-7-Yan:

- scenario D pins its own row: before/after stats delta plus a serde
  round-trip of the stored write, instead of an always-true > 0 check
- _delete_checkpoint now performs the joint delete the doc mandates
  (checkpoint row + writes rows + blobs unreachable from surviving
  checkpoints), so E1/E2 exercise the shape they prescribe
- E1 builds the real runtime duration shape via persist_run_durations
  (parent dict clone, fresh id/ts, real metadata), which surfaces the
  shared-version case: the leaf's blobs are the surviving parent's rows
- contract doc: blob reachability must be computed from surviving
  checkpoints in a whole-thread pass; shared-version/duration-only
  hazard called out explicitly; memory data model includes saver.blobs
- _stats counts memory blob rows and returns the full normalized shape
  (logical byte totals included)
- probe: drops the unused middleware/outputs_dir graph parameters and
  discloses the manual-harness scope limit in the module docstring
- E1/E2 assert default head resolution (protected set item 5); unused
  graph_for helper and DURATION_ONLY_METADATA stand-in removed

Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>

* fix(checkpoint): scope probe cleanup to owned dirs, key report by backend

Second-round review findings on #5255:

- [P1] bench_tool_result_probe.py removed the whole user-supplied
  --outputs-dir (and the shared .probe-tmp) in its finally block, so
  pre-existing files were deleted on success and failure alike. The run
  now writes into (and removes) a fresh owned probe-run-* child beneath
  the requested directory, and SQLite databases live in a unique
  mkdtemp'd temp directory that is removed with the run. Regression
  tests pin that unrelated pre-existing files survive both a successful
  and a simulated failing run.
- [P2] the optional retention report keyed every backend's measurements
  under one shared name, so a multi-backend invocation kept only the
  last backend's numbers. _report() now takes the parameterized backend
  explicitly (saver_env.kind); regression pins that memory and sqlite
  entries coexist in one report file.

Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>

---------

Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
2026-09-08 19:21:57 +08:00
Sami Belhareth
f8f6cde23f
fix: preserve assistant/tool history in compaction summaries (#5248)
* fix: preserve bounded assistant and tool input during compaction

* fix: retain recent fallback summary input and clarify budget

* fix: preserve recent content in mixed-history summary fallback

* docs: trim middleware guidance to pass size check

---------

Co-authored-by: Sami Belhareth <6599699+belharethsami@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-08 16:00:45 +08:00
wutongyuonce
9ad79baf97
feat(gateway): support idempotent thread runs (#5258)
* feat(gateway): support idempotent thread runs

Accept Idempotency-Key on thread-scoped create, stream, and wait endpoints, scoped by owner and thread before durable admission.\n\nRefs #5257.

* fix(gateway): handle idempotent run reuse on wait and stream

Reused store-only records have no local task. /wait now waits on the
bridge when it can observe the stream, and otherwise returns durable
status instead of a stale checkpoint. A reused terminal stream that has
been evicted emits gap/reload_durable_state. Replay is bound to the
original input and assistant_id.

* fix(gateway): 409 reused in-flight streams on this worker

A store-only running record on a process-local bridge has no owner
stream. POST /runs/stream used to subscribe anyway, which created an
empty log and waited forever. Match join: 409 unless the run is already
terminal, so missing-stream retries can still emit gap.

* fix(gateway): keep observer joins off the idempotent stream-gap path

sse_consumer keyed missing-stream gap on the sticky
idempotency_reused flag, so a later join of a terminal run inherited
it. Gate that branch on apply_on_disconnect, which already separates
creating streams from joins. Document the retry outcomes clients have
to handle.

* fix(gateway): gate missing-stream gap on creating retry

Reuse apply_on_disconnect to pick gap vs end changed sse_consumer default path, so a missing stream started returning gap for default callers and for out-of-scope POST /api/runs/stream. Keep that branch behind emit_gap_on_missing_stream and pass it only from thread-scoped /runs/stream on this request reuse.

* fix(gateway): keep wait reuse off later checkpoints

Direct handler calls were crashing because FastAPI Header() leaked in as the Python default. Bind Idempotency-Key with Annotated so the default is None, and ignore non-str keys.

A reused completed /wait was still reading the latest thread checkpoint. After a later run on the same thread that is the later run's result. Return durable status instead of claiming the head as this run's output.

* fix(gateway): snapshot wait reuse and refresh store status

idempotency_reused lives on the shared cached record. Capture it before awaiting completion so an overlapping retry cannot suppress the original creating /wait checkpoint.

A store-only peer record still holds admission-time status after the owner publishes END. Refresh durable status/error before returning them.
2026-09-08 14:40:05 +08:00
Yusuf Gürdoğan
23bd76046a
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider

Add a community provider backed by Sofya (https://sofya.co). Its search
endpoint returns the content of the result pages, not only their snippets,
and its fetch endpoint returns a page as markdown. Both are plain JSON over
HTTP, so this needs no extra Python package (uses httpx, already a
dependency).

Changes:
- backend/packages/harness/deerflow/community/sofya/__init__.py
- backend/packages/harness/deerflow/community/sofya/tools.py
  Implements web_search_tool and web_fetch_tool using httpx.
  API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY
  env var. Follows the same interface and output shape as the existing
  ddg_search and serper providers, including the max_results parameter with
  config override and the structured "No results found" error.
- backend/tests/test_sofya_tools.py
  Unit tests covering API key resolution, config overrides, result mapping,
  time range, HTTP errors, empty results, and fetch failures.
- config.example.yaml: add commented-out Sofya web_search and web_fetch
  examples alongside the other providers
- .env.example: add SOFYA_API_KEY placeholder
- backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and
  the environment variables

* fix(sofya): honor caller max_results, validate search_depth, join time_range contract test

- Caller-supplied max_results now wins; config is used only when the
  argument is omitted, matching GroundRoute.
- search_depth is clamped to basic/snippets; an unsupported value logs a
  warning and falls back to basic.
- Sofya added to the shared time_range schema contract test.

* fix(sofya): cap per-result content so a search stays inline

An unbounded search payload (up to 20 read pages) crossed the tool output
budget middleware's externalize_min_chars threshold, which replaces the
result list with a file reference. Cap each result's content at
contents_max_characters (default 2000, 0 disables), matching Exa's config
key. Five capped results stay under the 12000 char threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): list Sofya in the recency contract, coerce non-string content

_clip subscripted its input, so a non-string content or description from
the API raised TypeError instead of degrading. Coerce to text first, the
way _sofya_post and _response_results guard the shapes around it. Also add
Sofya to the Web Search Recency section in backend/AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog

web_fetch sliced its content the same way web_search did before the last
push: a truthy non-string from the API passed the falsiness guard and then
raised TypeError. Reuse _clip, keeping the `or ""` so empty content still
reports "No content found".

Also add sofya to the community provider inventory in
packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs

The ZH web_fetch tab list named five providers where EN names seven. Both
tabs mirror their EN counterparts, so the two locales list the same
web_fetch providers again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-08 10:50:03 +08:00
George Pickett
9035934432
feat(mcp): add optional Parallel Search server (#5028)
* feat(mcp): add optional Parallel Search server

* docs(mcp): document Parallel Search opt-in and data sharing

* docs(mcp): address Parallel Search review feedback
2026-09-06 22:54:32 +08:00
Ryker_Feng
98b8e4657e
feat(chats): add archive and restore (#5236)
* feat(chats): add archive and restore

* test(chats): observe archive search requests in pagination e2e

* docs(gateway): move thread lifecycle details out of inherited guidance

* docs(chats): add concise archive and restore RFC

* docs(chats): move archive RFC discussion to issue 5237
2026-09-06 22:30:26 +08:00
PeaceMaker-best
e3df6ea4a8
feat(channels): select custom agents per conversation (#5168)
* feat(channels): select custom agents per conversation

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* fix(channels): reserve agent slash command across clients

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* fix(tui): hide reserved slash commands from skills

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* fix(channels): preserve selected agent across clients

---------

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-09-06 16:46:51 +08:00
hataa
aec7d73890
feat(knowledge): add read-only LightRAG retrieval, fixes #5208 (#5209) 2026-09-06 10:21:58 +08:00
PeaceMaker-best
3c36217a51
feat(observability): persist deferred tool promotions (#5183)
* feat(observability): persist deferred tool promotions

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* fix(ci): trim agent guidance chain

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

---------

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-09-05 14:03:00 +08:00
Aari
0f7d8709d3
feat(sandbox): add controlled egress with approvals (#5152)
* feat(sandbox): add controlled egress approvals

* Apply batched suggestions from code review

* fix(sandbox): harden restricted network policy

* fix(sandbox): harden denied egress handling

* fix(sandbox): isolate network proxy sidecar

* chore: retry sandbox image smoke

* fix(sandbox): close remaining network policy gaps

* fix(sandbox): harden relay token rejection

* fix(sandbox): fence incompatible policy replacement

* fix(sandbox): replace containers across network modes

* fix(sandbox): close remaining lifecycle gaps

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-04 23:46:57 +08:00
Jun
83cb6767b3
fix(sandbox): add FOWNER for AIO 1.11 startup (#5163)
* fix(sandbox): add FOWNER for AIO 1.11 startup

* test(sandbox): cover FOWNER startup capability

* docs(sandbox): document FOWNER capability

* test(sandbox): pin FOWNER regression smoke

* ci(sandbox): allow pinning FOWNER smoke image

* style(sandbox): format FOWNER smoke test

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-04 00:03:49 +08:00
Tu Naichao
ae82f426bf
fix(summarization): stop fraction triggers from crashing the agent build (#4901)
* fix(summarization): resolve fraction triggers from declared context_window, degrade instead of crashing the agent build

A fraction trigger/keep clause requires profile["max_input_tokens"], which any
third-party OpenAI-compatible model lacks, so SummarizationMiddleware
construction raised ValueError out of create_summarization_middleware and failed
the whole agent build (#3103).

- factory: translate a declared model context_window into the langchain
  profile (metadata-only, never reaches the provider payload); explicit
  caller/override profiles win
- summarization factory: drop unusable fraction trigger clauses (absolute
  clauses survive), fall a fraction keep back to the messages default, and
  disable compaction with an actionable warning only when no usable trigger
  clause remains — the agent build never dies from summarization config
- docs: config.example.yaml, ModelConfig.context_window, summarization.md

* refactor(summarization): share the default keep constant with the fraction fallback

The fraction-keep degradation fallback hardcoded ("messages", 20),
duplicating SummarizationConfig.keep's default_factory literal. Move the
value to a shared DEFAULT_KEEP constant so the two cannot drift apart.

* fix(summarization): keep trigger-null + fraction-keep constructing after degradation

A trigger of None with a fraction keep hit the all-clauses-dropped branch
(has_usable_trigger=False) and disabled compaction, and the accompanying
warning claimed configured triggers were all fraction-based when none were
configured. Only report nothing-usable when trigger clauses actually
existed; trigger:null keeps constructing the never-firing middleware with
the degraded keep, matching its behavior outside the degradation path.

* fix(summarization): address review — keep manual compaction, validate ContextSize, pin wiring

Review follow-ups on #4901:

- When every configured trigger is a dropped fraction clause, keep
  constructing the never-firing middleware (trigger=None) instead of
  returning None: manual /compact runs with force=True and never consults
  trigger clauses, so it must keep working for a profile-less model
  rather than reporting 'compaction is disabled'. The warning now says
  auto-compaction will not fire while manual compaction remains.
- ContextSize gains a config-load validator: fraction values must be in
  (0,1] (a percent-style 80 instead of 0.8 previously produced a threshold
  the context could never reach — a silently inert trigger), absolute
  values must be positive.
- New un-monkeypatched integration test pins the shipped wiring
  (context_window declared -> real factory attaches profile -> fraction
  clause survives -> middleware constructs), which the stubbed
  middleware-side tests and kwarg-capturing factory-side tests each
  stopped short of.
- Docs (summarization.md + config.example.yaml) clarify that the fraction
  resolves against the summary/anchor model's context_window
  (summarization.model_name when set, else the run model), including the
  mismatch caveat for a larger-window summary model.

* fix(summarization): reject non-finite ContextSize values at config load

YAML .nan / .inf pass pydantic's float parsing, and nan <= 0 is False,
so the positivity check alone let them through as dead thresholds
(count >= nan is always False) — the same silent-inert-trigger class the
range validator was added to close. Guard with math.isfinite first,
consistent with the existing non-finite guards on mem0 timeout_seconds
and poll_after_seconds.

* fix(summarization): merge context_window into inferred profile, require whole message counts

- construct the model first, then merge max_input_tokens into the
  provider-inferred langchain profile: passing profile= to the
  constructor replaced the whole inferred metadata (tool_calling,
  structured_output, io capabilities, output limits) with the single
  key. An explicitly configured profile is still never clobbered.
- reject non-integral ContextSize values for type=messages at config
  load: langchain slices the message list with them, so a float index
  raised TypeError mid-compaction.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-03 17:05:09 +08:00
hataa
08b27aef73
feat(auth): make login rate-limit parameters configurable, fixes #5108 (#5110)
* feat(auth): make login rate-limit parameters configurable, fixes #5108

Add auth.local.max_login_attempts (default 5) and auth.local.lockout_seconds
(default 300) so operators can tune the per-IP login throttle: raise the
ceiling for shared-egress-IP offices behind proxies/NAT, or tighten it for
stricter posture. Policy is live-read per call (matching the
_local_registration_enabled precedent), so a config reload applies without a
Gateway restart; raising the threshold mid-lockout immediately unblocks
affected IPs.

Review feedback addressed (willem-bd):
- Only FileNotFoundError falls back to the hardcoded defaults; a malformed
  config propagates, mirroring _local_registration_enabled, so an operator
  who tightened the policy never silently gets the more permissive defaults.
- _check_rate_limit looks up the record before resolving the policy, so a
  clean IP pays zero config reads (get_app_config re-hashes config.yaml per
  call and login_local is an unauthenticated async endpoint).

Bumps config_version to 39 in config.example.yaml and the Helm chart
(values.yaml + README example) so the chart drift check stays green.

* fix(auth): reject max_login_attempts=1 and honor live lockout_seconds for active lockouts

* fix(auth): close live-policy state gaps in login throttle (resurrection, count reset, broken-config verification)

* fix(auth): commit evaluated lockout duration on decreases too, preventing raise-resurrection

* test(auth): pin broken-config fail-closed sequence through the login route

* fix(auth): sweep expired locks by stored sentence and keep policy reads off the event loop

* fix(auth): re-read throttle record after the policy-resolution yield point

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-02 19:16:56 +08:00
AoHanBei
9b32b5d841
feat(observability): persist loop detection events (#5127)
* feat(observability): persist loop detection events

* fix(observability): persist subagent loop events

* fix(observability): narrow subagent loop event bridge

* fix(observability): attribute subagent loop events

* fix(tests): isolate subagent executor imports
2026-09-02 10:25:37 +08:00
Aari
340bff1107
feat(mcp): manage servers from Settings (#5022)
* feat(mcp): manage servers from settings

* fix(mcp): make settings updates targeted

* fix(mcp): reject ambiguous masked array edits

* fix(mcp): honor targeted server field deletions

* fix(mcp): preserve OAuth extension secrets

* fix(mcp): validate config before persistence

* fix(mcp): preserve environment placeholders

* fix(mcp): harden targeted configuration routes

* docs: keep gateway guidance within budget

* fix(mcp): protect per-tool override secrets

* fix(mcp): keep disabled edits structurally safe
2026-09-01 23:24:49 +08:00
Terminator666666
c17aa8b98f
fix(mcp): reject credentials that cannot travel as HTTP header values (#5066)
* fix(mcp): reject credentials that cannot travel as HTTP header values

A request-scoped secret or user_auth credential with a trailing newline
(the usual result of reading a token from a file, or a CRLF env-file),
CR/LF, surrounding whitespace, or characters outside Latin-1 sailed
through the credential interceptors into the HTTP client, where httpx/h11
reject it with an exception that echoes the full value:

    LocalProtocolError: Illegal header value b'Bearer sk-...\n'

ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage, so the secret landed in the prompt, the checkpoint, and
traces - everywhere headers_from_context promises it never goes.

Add illegal_header_value_reason to mcp/headers.py, mirroring the
transport's own rules (Latin-1 encodable; h11's field_vchar is [^\x00\s]
with SP/HTAB legal only between visible characters), and fail closed in
both interceptors before the value can reach the client. The denial names
only the secret key (plus the reason) and never repeats the value.

Illegal values are denied regardless of on_missing: the key is present,
so a passthrough fallback would silently run the call under the shared
discovery credential - the exact authority confusion the deny default
exists to prevent.

Values the transport accepts are not rejected: embedded SP/HTAB
('Bearer <token>'), Latin-1 high bytes, and DEL all still pass, pinned
by tests against h11's observed behaviour.

* fix(mcp): tighten header value validation to httpx's ASCII boundary

The validator mirrored h11's Latin-1 boundary, but the transport rejects
more than h11 does: build_server_params hands dict[str, str] headers
through the MCP SDK's create_mcp_http_client into httpx.AsyncClient, and
httpx (pinned 0.28.1) encodes str header values as ASCII - so a Latin-1
high byte like 'Bearer caf\xe9' passed validation here only to raise
UnicodeEncodeError inside httpx before h11 ever ran, with the exception
message repeating the offending value.

Validate str values against ASCII instead, flip the tests that pinned
Latin-1 high bytes as transportable, and pin the boundary against the
real client: create_mcp_http_client must reject what the validator
flags and construct cleanly for what it accepts (embedded SP/HTAB and
DEL still pass).

Addresses review feedback on the ASCII vs Latin-1 boundary.

* fix(mcp): validate OAuth and static header values at the same boundary

The validator added for headers_from_context and user_auth left two paths
uncovered. A token endpoint returning an access_token or token_type with a
newline reached httpx/h11, which raise with the full token in the message, and
ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage -- the leak this PR set out to close. The operator's static headers
had the same hole.

OAuthTokenManager.get_authorization_header now renders the Authorization value
through one checked helper, so the tool interceptor, the initial discovery
headers and the durable task path are all covered by a single guard. The
rendered value is what gets checked rather than the two fields separately,
because that is what the transport sees: an access_token with leading
whitespace is legal once it follows "Bearer ".

build_server_params applies the same check to statically configured headers.
build_servers_config already isolates a per-server failure, so a bad value
drops that one server and logs the reason instead of the value.

* docs(mcp): correct which transport echoes the full header value

The rationale claimed httpx and h11 both render the full value into their
exception message. Only h11 does, on the line break and surrounding whitespace
cases. httpx's ASCII failure is a UnicodeEncodeError naming the offending
character and its position, not the credential, so at most one character
escapes there; refusing the value up front buys an actionable error rather than
an encode failure raised from inside the client.

Corrected in headers.py and in every copy of the claim: context_headers.py,
user_scoped_auth.py, oauth.py, client.py, mcp/AGENTS.md, docs/MCP_SERVER.md,
the frontend mcp.mdx, and the test comments carrying the same wording. No
behavior change.

---------

Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
2026-08-31 15:07:30 +08:00