* fix(sandbox): leave SIGTERM/SIGINT teardown to the Gateway lifespan
AIO and E2B providers built on the event-loop thread installed SIGTERM/SIGINT
handlers that force-destroyed every sandbox synchronously, before uvicorn
started the run drain. The Gateway lifespan now holds sandbox shutdown
ownership from its first step, so providers skip those handlers and the
existing hook closes the provider after run drain and MCP teardown. SIGHUP
keeps the provider fallback. AIO acquire now refuses once shutdown began.
* docs(sandbox): document Gateway-owned sandbox shutdown and AIO admission fence
* test(gateway): inspect the lifespan body now that lifespan wraps it
* docs(changelog): reference #7560 for the sandbox SIGTERM drain-order fix
* ci: retrigger sandbox image smoke after a registry pull timeout
* fix(persistence): keep LF bytes in skill and agent text writes
Text mode translates a newline into os.linesep on write, so on Windows every
caller-supplied LF reached disk as CRLF while readers use universal newlines or
decode bytes explicitly. The difference stayed invisible: the API returned LF,
the file held CRLF, and a skill history record's prev_content described a
revision no API view could return (#7174).
Pass newline="" at the ten writers that persist or judge caller text: the local
and user-scoped skill writers, the skill history JSONL, the user skill-states
JSON, the SOUL.md/config.yaml staging in FileAgentStore, the managed-subagent
store, setup_agent's SOUL.md publish, the USER.md route, and the two temp copies
the frontmatter validator and the static scanner judge.
The tests emulate the Windows translation on io.open, so the regression runs on
the Linux CI shards instead of only on a Windows host: 10 failed before this
commit, 10 pass after.
* fix(skills): preserve candidate newlines during static scan
---------
Co-authored-by: Felix Shi <204130665+shi-felix@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read the route authorization config off the event loop
AuthMiddleware resolves route permissions for every authenticated
request, and resolve_route_permissions read the hot-reloaded
authorization config inline. get_app_config() stats and hashes
config.yaml and extensions_config.json on every call and reparses them
after an edit, so that file IO ran on the event loop per request.
Read it through asyncio.to_thread, like the plugin authorization path
already does. The context copy keeps runtime config overrides visible.
* docs(changelog): reference #7557 for the off-loop route authorization config read
* docs(config): note the get_app_config read cost where it is documented
Move the off-loop note from gateway/AGENTS.md, which is 12 bytes under
its hard limit, to the Config Caching section that documents what each
get_app_config() read costs.
* fix(utils): construct one-shot models off the event loop
* docs(utils): document all one-shot callers and offload scope
Project-file summaries use the shared one-shot helper as well as input
polishing and follow-up suggestions. Include that third caller in the
README and utility guidance, and distinguish the separate skill scanner
and goal evaluator constructors.
Runtime and tests are unchanged. The existing exact-head strict and caller
regressions pass together: 80 passed on Windows with Python 3.12.14.
* fix(memory): flush pre-compaction memory off the event loop
The async compaction path fired the before-summarization hooks inline.
memory_flush_hook hands the summarized messages to add_nowait, which is
sync HTTP for mem0 and Honcho (inherited default -> add) and OpenViking
(its own override), so every automatic compaction with an HTTP memory
backend blocked the Gateway event loop for the whole request.
acompact_state now runs the hooks through asyncio.to_thread, as
MemoryMiddleware.aafter_agent already does for its writes. Ordering,
failure isolation, summary-first timing and context propagation are
unchanged; the sync compact_state path is untouched.
* docs(changelog): reference #7530 for the pre-compaction memory flush fix
* test(memory): anchor the OpenViking pre-compaction flush off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(runtime): honor the web UI language in replies and DeerMem updates
Carry Settings > Appearance > Language as request-only run data: the
Gateway validates context.locale against a finite allowlist, the lead
agent prefers it for replies, and bundled DeerMem writes new facts and
summary prose in it. Explicit user language requests still take
precedence; checkpoint configurable, static prompts, and memory backend
APIs are unchanged.
Refs #7287
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime): guard language context and pin locale maps
* test(runtime): pin UI locales across frontend and Gateway
Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Move checkpoint modes, replay, module maps and terminal-ownership details intact into the existing runtime reference. Keep concise invariants and explicit links in AGENTS.md, reducing the effective chain from 99,004 to 80,817 bytes. The existing guidance regression now passes without changing budget thresholds.
* fix(projects): delete read-only documents when purging trash
A permanently deleted project document could never be purged on Windows:
the read-only file attribute makes `Path.unlink` fail with `PermissionError`
(`WinError 5`), where POSIX needs only a writable parent. A restored archive
or a mode-preserving sync sets that attribute, and because the purge re-raises,
the row rolls back and the trashed document keeps coming back with the same
error on every retry.
Clear the attribute and retry the unlink once, mirroring the retry that
`config/paths.py::_clear_read_only_tree` already performs for thread
directories. Any other unlink error still propagates, so the row stays
retryable.
* fix(projects): harden trash purge permission retries
---------
Co-authored-by: Aling33 <along308@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): derive the AIO sandbox ID off the event loop in async acquire
_acquire_internal_async offloads every acquire layer except the sandbox ID
derivation, which ran inline. _sandbox_id_for_thread checks the thread's
skills_view directory with exists(), and resolving it goes through
Paths.base_dir, so each async thread acquire made filesystem syscalls on the
event loop. The strict blocking-I/O gate raises on it.
Run the derivation through the same owned-worker helper as the reuse and
warm-pool layers around it, and anchor it in the blocking-I/O suite for both
the default and the policy-scoped ID.
* docs(changelog): reference #7507 for the AIO async acquire sandbox ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): backfill run.terminal on confirmed recovery takeover
* fix(runtime): isolate recovery terminal publication failures
* fix(runtime): fence recovery terminal publication lifecycle
* fix(runtime): unify recovery receipt publication
* docs(runtime): document lifecycle guard lock constraints
* fix(runtime): reuse event transaction for recovery guards
Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): make read_file_chunks cancellation-safe for worker handoffs
Cancelling a project document read had three unsafe windows around the
dedicated file-I/O worker: (1) a cancellation landing between the open
worker completing and the result reaching the coroutine leaked the
handle (the cleanup path never saw it); (2) the cleanup close could
start while an admitted read was still running; (3) a repeated
cancellation let the caller escape before the close worker settled
(#7491).
Gate every admitted operation with the existing await_drained
primitive: cancellation is delivered only after the operation settles,
the open result is retained in a list cell the moment the worker
returns it (so the cleanup close always has the handle), and waiting
for the next chunk via the yield stays immediately cancellable.
Regression tests gate the real executor with asyncio events at the
worker boundary (real files, no mocks): all three windows are red on
unmodified main and green with the fix; the happy path is unchanged.
The dispatch-spy promotion tests that import app.gateway.routers need
the full backend dependency set and could not run in the minimal local
env; they fail identically on unmodified main (verified).
Fixes#7491
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: fix _gated_offload return annotation to the unpacked tuple
Review nit: the helper returns (calls, gated) and every call site
unpacks it as a pair, but the annotation claimed list[str]. Type it as
the tuple it actually returns.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: pin missing-path FileNotFoundError in read_file_chunks
Convergence item from the duplicate-pair verdict (#7493): the
open-failure path must surface FileNotFoundError as-is — the cleanup
finally skips the close because retained is empty (no handle was ever
opened), rather than raising AttributeError from closing None.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
---------
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
Two Gateways booting the same missing database both reach CREATE DATABASE and
only one wins; the loser aborted startup with duplicate_database (42P04) even
though the winner's catalog row was already committed and the target was
usable. Follow the SQLSTATE, like the missing-database check beside it does, so
the loser falls through to its retry. Every other failure -- permission denied
included -- still propagates untouched.
* fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome
* fix(gateway): adopt the family helper so a published edit stays silent
* fix(uploads): include markdown modification time in ownership records
* test(uploads): reject same-size edited companions on Windows
* fix(uploads): distinguish legacy companion records
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(checkpoints): isolate cached history payloads from state readers
Copy nested snapshot seeds and pending-write payloads at both cache boundaries.
Keep user copy callbacks outside the registry lock so readers cannot poison
later checkpoint materialization or deadlock by re-entering the cache.
Cover sync/async state accessor reads, nested message payloads, disabled-cache
controls and callback lock ownership.
* docs(checkpoints): document synchronous history copy costs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): reject write_file payloads that are the elision placeholder
The tool-output budget middleware replaces a superseded write_file payload
with a deterministic placeholder in the model-bound request only. That
placeholder is text the model sees in its own history, so it can imitate it
and submit it back as real write_file content: #7319 reports a delivered
file that was 194 bytes of placeholder.
write_file now rejects a payload that is the placeholder (whole payload,
allowing only surrounding whitespace) with an actionable error telling the
model to read_file the target path and write the real content; nothing is
written. The match is whole-payload on purpose: a substring guard would
block legitimate edits to this repository, whose constants.py and
persistence docs quote the template verbatim.
The placeholder wording moves to one shared constant,
WRITE_ELISION_PLACEHOLDER_TEMPLATE in deerflow.constants, which the
middleware renders from and the guard matches on, so the two cannot drift.
Regression tests fail on base (7 failed, 6 passed) and pass with the fix;
near-miss payloads that merely quote the placeholder still write.
Refs #7319
* fix(sandbox): match complete write elision placeholders
* docs: clarify write placeholder validation scope
* chore: re-trigger CLA status check
* fix(sandbox): preflight async writes and trim guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): preserve Moonshot cache_write_tokens in usage metadata
langchain-openai 1.2.1's _create_usage_metadata reads
prompt_tokens_details.cached_tokens but not cache_write_tokens, so the
PatchedChatDeepSeek conversion dropped the cache-creation count that Kimi K3.1
reports, and DeerFlow's usage journal never saw it (#7426).
PatchedChatDeepSeek now overrides _create_chat_result to restore the raw SDK
cache_write_tokens as input_token_details["cache_creation"].
Offline regression: a real OpenAI ChatCompletion carrying
prompt_tokens_details {cached_tokens: 100, cache_write_tokens: 256} converts to
input_token_details {cache_read: 100, cache_creation: 256} (red before, green
after); the existing 14 patched_deepseek tests and the managed-provider suites
stay green.
* fix(models): restore cache_write_tokens on the streaming path too
Streamed runs (web UI messages-tuple) build usage metadata in
_convert_chunk_to_generation_chunk -> _create_usage_metadata, which ignores
cache_write_tokens, so the non-streaming-only override still lost the count on
the path streamed runs actually take (review follow-up on #7428).
_convert_chunk_to_generation_chunk now mirrors _create_chat_result's restore
for the final usage chunk. Note chunk["usage"] is JSON null on non-final
chunks, so the lookup uses `or {}` instead of a .get default (a None usage
would otherwise raise AttributeError inside every chunk and break
test_model's connection probe).
* fix(harness): keep timestamped output-name fallbacks distinct
`output_name_candidates()` sampled `datetime.now(UTC)` inside the yield
loop, so its five timestamped fallbacks were only distinct when the host
clock advanced between two generator steps. On a coarse clock (Windows
advances in sub-millisecond steps) all five produced the same name, and
the capture tools lost retries before failing with
`could not reserve an output filename`.
Sample the clock once when the first fallback is reached and offset each
attempt by one microsecond, so distinctness no longer depends on the host
clock. Add a regression test that freezes the clock, and update the
exhaustion test to hold all five distinct stamps.
* docs: describe the coarse clock accurately
---------
Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
* fix(memory): drop unusable numbers in stale extensions and consolidation confidence
* fix(memory): log defaulted consolidation confidence and pin the int passthrough
* fix(skills): honor skills.container_path in user-scoped skill storage
The user-scoped factory never passed container_path, so
UserScopedSkillStorage kept the default /mnt/skills while public
storage honored the configured root, and consumers reading the root
from the storage (container registry, skill activation) built paths
that do not exist once an operator moves the mount root.
Seed container_path from the resolved app config, matching the
public-storage factory; an explicit caller override still wins.
Fixes#6767
* fix(skills): seed container_path in the legacy-visibility kwargs branch
willem-bd review nit on #6770: user_should_see_legacy_skills constructs
a one-off UserScopedSkillStorage when callers pass kwargs, and that
branch skipped the app_config container_path seeding, so a future caller
combining kwargs with a custom skills root would read the default
/mnt/skills instead. Seed it the same way the factory does; an explicit
caller override still wins.
* fix(sandbox): keep document bytes when write_file translates paths
LocalSandbox.write_file rewrote container paths inside every file it stored,
so a dictated README.md ended up holding a host path while read_file masked
the rewrite for agent-written files: the read-back showed the virtual path,
and the agent confirmed content the file did not contain (#7461).
Translation now applies only to executable content, classified with the
shared is_code_file rule from deerflow.skills.package_files that the skill
installer, the export guard, and skill_manage_tool already use. Code suffixes
and extensionless files that open with a shebang keep the host-path rewrite
they need to run under a host interpreter (PR #1935); documents, data files,
and configs are stored verbatim and read back as stored.
* fix(sandbox): translate paths for interpreter scripts and stop reading pipes
The classifier added for #7461 asked only the skill-package rule, which
does not list `.awk` (nor `.sed`, `.lua`, `.r`, `.tcl`), so a written AWK
program kept container paths that the host awk cannot open when it runs
`awk -f`. Those suffixes are now the sandbox's own set: the shared rule
also decides what SkillScan and the installer treat as package code, and
widening it there is a different call.
Classification no longer touches the destination unless the path is
extensionless. A suffix answers the question by itself, so an append to
`events.py` no longer opens the file first, and the extensionless
shebang sniff goes through `os.open(O_NONBLOCK)` plus an `fstat` regular
file check, which is what keeps a FIFO append from blocking.
* test(sandbox): pin the non-blocking sniff with a suffixless pipe append
* fix(memory): rank capacity eviction per record, not per fact_id
select_facts_for_capacity keyed every score by fact_id, so an imported
batch that repeats an id collapsed into a single dict entry: the last
record's score won, and both the ranking pass and the evicted audit
read that one value for every position. A 0.9-confidence record could
therefore be evicted while its own 0.1-confidence duplicate survived.
The import path keeps caller-supplied ids without checking them for
uniqueness, and any over-cap import lands here.
Score by position instead: a parallel list next to the indexed facts,
so ranking and the audit read each record's own score. The public
``scores`` dict stays keyed by fact_id with last-write-wins
semantics, keeping the DeerMem benchmark contract and the callers
that never read it unchanged.
Red on main: test_confidence_policy_scores_duplicate_fact_ids_per_record
fails with the highest-confidence record evicted; green on this branch.
* fix(utils): keep share-less UNC roots out of the extended-length prefix
extended_length_path prefixed every absolute UNC path with
\\?\UNC\, but Win32 only accepts \\?\UNC\server\share\... — a
share-less root (\\server, \\server\, or a bare \) is not a valid
device path, so every Win32 call handed the prefixed spelling fails.
Only prefix when both the server and the share are present;
otherwise return the ordinary spelling unchanged.
Red on main: the \\server, \\server\, //, and //server cases of
test_windows_extended_spelling assert the forged prefix; green on
this branch.
* fix(sandbox): normalize mount container_path trailing slashes at config load
A container_path declared with a trailing slash (e.g. /mnt/reports/) was
accepted by VolumeMountConfig and normalized by LocalSandboxProvider when
building path mappings, but the sandbox tool path gates compare the raw
configured value, so every ls/read/write/glob/grep/bash access to the
mount failed with a confusing PermissionError. Normalize trailing
slashes in the config model so both layers agree, and reject the
container root '/' which the gates cannot express.
* fix(sandbox): normalize skills.container_path and pin empty-value rejection
Review follow-up on #7393:
- Share the container-path normalizer with SkillsConfig, whose prefix
feeds the same as-is comparison in _is_skills_path and had the same
trailing-slash dead-path failure mode.
- Pin the empty-string rejection: on main an empty container_path made
the gates match every absolute path, applying one mount's read_only
flag to all paths.
- Cover both config fields in CONFIGURATION.md.
---------
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
* fix(channels): retry the runtime config file lock instead of failing after ~10s on Windows
The runtime channel config sidecar lock (#6844) used Windows' blocking msvcrt LK_LOCK mode, which gives up after roughly ten seconds and surfaces cross-worker contention as Permission denied instead of serializing the read-modify-write cycles it exists to serialize, while fcntl.flock(LOCK_EX) waits on POSIX. Retry the non-blocking acquisition (LK_NBLCK + 0.2s) on the Windows fallback, mirroring the extensions manager's lock (#6020), and bind fcntl/msvcrt at module level so the fallback branch is testable.
Tests: new backend/tests/test_channel_runtime_config_store_lock.py (5 tests: retry cadence via fake msvcrt, sidecar byte creation, POSIX flock preservation, direct Windows regression through the sys.modules seam, real cross-thread contention). 3 of them fail on the unmodified base (one with OSError 36 raised by the blocking call itself) and all pass with the fix. Related suites: store peers + connections router 86 passed / 1 skipped; blocking_io + encoding + channels 498 passed / 2 skipped. Ruff format and check clean on both files. Windows 11, Python 3.12 (uv-managed venv).
* fix(channels): share contention retries and diagnose prolonged lock waits
* docs(channels): shorten guidance to satisfy lint budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`Paths.delete_thread_dir()` called `shutil.rmtree()` directly. Windows
refuses to unlink a file that carries the read-only attribute
(`WinError 5`) where POSIX only needs a writable parent, so one read-only
file under `user-data/` made the whole thread undeletable: `DELETE
/api/threads/{id}` answered 500 `Failed to delete local thread data`,
`rmtree` had already removed everything it walked before that entry, and
every later retry failed on the same file.
Retry at the call site: clear the read-only attribute tree-wide and call
`rmtree()` once more. Only `PermissionError` is retried, the retry still
raises when it cannot help, so a file another process holds open stays a
real error instead of becoming a silent skip. A `onexc` handler cannot do
this portably: POSIX `_rmtree_safe_fd()` catches the handler's own
`PermissionError`, re-dispatches it as a `scandir` error and leaves the
deletion to fail later as `ENOTEMPTY`.
Co-authored-by: Aling33 <along308@users.noreply.github.com>
* fix(subagents): preserve test summaries before evidence truncation
* fix(subagents): honor Node counters and cache harvested summaries
* fix(subagents): retain warning counts and Go case evidence
* fix(subagents): retain long pytest sibling summaries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(readability): preserve <a> links when Readability.js silently degrades on Windows
On Windows, simple_json_from_html_string(use_readability=True) invokes the
Readability.js subprocess, which succeeds without error but returns the
unextracted document instead of the extracted article. The pure-Python
fallback (use_readability=False) also drops every <a> destination. The net
effect is that web_fetch returns link-free Markdown on Windows.
Fix: after extraction, check whether the output retained the <a> elements
that the resolved HTML carried. When they are all gone, fall back to
converting the resolved HTML directly with markdownify — which preserves
<a> as [text](href) — so links survive regardless of the Readability.js
outcome.
Measured: 17 of 22 test_web_fetch_relative_links.py failures resolved;
43/48 pass on Windows (5 remaining are pre-existing malformed-markup
extraction tests, unrelated to this change).
Closes#6459
* test(readability): add degraded-extraction and scope-pinning tests
* test(readability): detect unwanted article fallback
* test(readability): classify HTML body in authority-tag census
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): pin explicit app config path across get_app_config
Fixes#7086. reload_app_config(config_path) now records an
_app_config_source pin (mirroring _extensions_config_source) so
get_app_config and DeerFlowClient(config_path=...) keep following the
explicit file instead of ambient default resolution.
* fix(config): synchronize app config cache and absolute pins
Serialize singleton reads, reloads, resets and injections so a concurrent call cannot publish a config under another file's pin. Keep explicit paths absolute across working-directory changes while preserving ambient discovery and failed-reload behavior.
Add deterministic concurrency and relative-path regressions. Validation: 33,875 offline tests and 262 blocking-I/O tests passed; frontend check, backend formatting and agent guidance passed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): make companion record removal best-effort
* style(uploads): format the new test and log the markdown-reference removal
* fix(uploads): report the orphaned record through the module that owns it
* fix(persistence): auto-create the Postgres database only when it is missing
init_engine retried bootstrap through CREATE DATABASE whenever the error
text contained "does not exist". Postgres words missing relations,
columns, functions and roles the same way, so a bootstrap that failed
inside an existing database (for example a migration reading a dropped
column) went on to CREATE DATABASE. Startup then failed with
DuplicateDatabaseError, or with "permission denied to create database"
for a role without CREATEDB, and the original error appeared nowhere in
the traceback.
Recognize a missing database by SQLSTATE 3D000 (invalid_catalog_name)
instead, following __cause__ for a wrapped copy, and re-raise every
other bootstrap error unchanged. A live PostgreSQL test covers both the
missing-database path and an existing database whose migration fails.
* docs(changelog): reference #7410 for the Postgres auto-create fix
* fix(persistence): walk orig, __cause__ and __context__ for the missing-database SQLSTATE
Match _is_unique_violation's chain walk, with its cycle guard, so a wrapper
that keeps the driver error only on `orig` or `__context__` still triggers
auto-create. Drop the live test's English message assertion; the SQLSTATE
check already proves the original error surfaced.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): never truncate an existing upload on same-name collision
Same-name uploads silently destroyed the earlier file on disk: the shared
write path (open_upload_file_no_symlink) opened the destination with
O_WRONLY|O_CREAT (no O_EXCL) and ftruncate(0), so re-uploading report.pdf
opened and emptied the report.pdf already stored for the thread, and two
concurrent same-name uploads could race on one inode. The Gateway
ingestion pipeline already refuses overwrites (staged no-overwrite link
commit), but the embedded client upload path (copy_upload_file_no_symlink)
and the channel ingress paths (write_upload_file_no_symlink) still went
through the truncating open.
Open every upload destination with O_EXCL. On FileExistsError, claim the
next free _N suffix against the directory's current contents via the
existing claim_unique_filename helper and retry — the same deterministic
behavior the Gateway pipeline documents (name_1.ext). The existing file is
never opened, so its bytes survive intact under any future naming policy.
Embedded-client and channel inbound-file responses now report the name the
bytes actually landed under instead of the requested name.
* test(uploads): cover concurrent same-name writers
* fix(uploads): skip known copy collisions before staging
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): reconcile running occurrences from terminal durable runs
A completion hook can fail after the durable run has already committed its terminal status (transient DB, observer, or notification error). Single-instance run_once only recovered expired launching claims, so the occurrence stayed running: a recurring schedule stopped firing and a once task never reached its terminal parent state until a restart.
Add a non-destructive ScheduledTaskRepository.reconcile_terminal_runs pass. It only selects running occurrences whose linked runs row is already terminal and whose scheduled metadata matches, then finalizes through the existing finalize_occurrence path with the run's status, error, and goal verdict. Live runs are never taken over or interrupted; multi-instance reconciliation already handles terminal runs through lease ownership. Single-instance run_once calls it after expired-launch recovery. No schema change.
Tests: a terminal success finalizes, a live run is left alone, error/timeout/interrupted map correctly, and a success with an unsatisfied goal verdict becomes unmet. Full scheduler suite: 1041 passed / 130 skipped; guidance 0 errors / 0 warnings; ruff clean.
* fix(scheduler): isolate terminal-run reconciliation failures per occurrence
A poisoned occurrence (the same observer or database failure that made the sweep necessary) ran in the same transaction as every other candidate, so one raised exception rolled back the whole pass and escaped run_once before admission and queue draining. Give each candidate its own session/transaction: a failure rolls back only that occurrence, logs its id, and the remaining candidates still commit. The service also catches a remaining pass-level failure so the rest of the poller runs.
Extend coverage: scheduled-trigger finalization (the motivating case), a second reconcile returning 0 with no state change, once-task completed projection, and a poisoned-observer candidate ordered before a healthy one whose success still commits. Full scheduler suite 1044 passed / 130 skipped.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
The quote-delimiter tables only knew ASCII quotes, backticks and curly
quotes, so `已生成「/mnt/user-data/outputs/季度 报告.pdf」。` captured the
handle `/mnt/user-data/outputs/季度` (the path pattern stops at the space)
and `已生成『…』。` kept the closer inside `real_ref`; either handle is dead
on a later read.
Register `「」` and `『』` as wrappers: a paired closer stays inside a
filename, an unpaired one is trimmed from a bare token, and corner-trailed
URLs and refs are no longer cut.
The persistence regression (capture -> registry round-trip -> resolution ->
real file read with the tool result gone) is ported from #7376 by
@yangjj-iso, per review.
* feat(projects): search shelf documents by keyword
Add search_project_documents: two-tier name/summary then bounded content scan, case-insensitive substring matching with original-text offsets. Never converts on the fly; reports stop_reason, content_scanned, skipped reasons, and capped unscanned/truncated ids.
* fix(projects): derive unscanned count without paging after budget trip
Review feedback: once the content budget trips, stop paging list_active to enumerate unscanned ids — cap ids at the trip page and derive the honest total from one count_active query (tier 1 is exhaustive, so active - tier1 hits - processed = unscanned). Also add a tool-level isolation test pinning that search never crosses projects or users.
* fix(projects): scan the shelf over one stable snapshot, not offset pages
Review feedback: offset paging re-read the live shelf per page, so a concurrent upload/trash between pages could duplicate a match or silently skip one while still claiming complete coverage. Tier 1 now materializes the whole shelf in a single SELECT (one read snapshot) and tier 2 iterates those same rows; the budget-trip unscanned count is derived exactly from the snapshot with no extra query. Mid-scan upload/trash regression tests pin both failure modes.
* fix(sandbox): close pipe write fds once when Popen fails
The command runners closed the pipe write descriptors twice on the
spawn-failure path: the except block released all four pipe fds, then
the finally block closed the two write fds again. In a busy gateway
another thread can claim a freed descriptor number between the two
closes, so the second os.close silently closed an unrelated descriptor
(a concurrent command's capture pipe or an HTTP socket). Guard the
parent's write-end close on a successful spawn.
* fix(sandbox): release pipe fds on BaseException and scope the Windows runner test
Review follow-up on #7394:
- Catch BaseException in both command runners' spawn-failure cleanup so a
KeyboardInterrupt mid-spawn still releases each pipe fd exactly once;
the guarded finally keeps the no-double-close guarantee.
- Skip the _run_windows_command failure-path param on non-Windows
runners: it reads subprocess.CREATE_NEW_PROCESS_GROUP when building
the call, which raises AttributeError on POSIX and would fail Linux CI.
- Add a KeyboardInterrupt regression test (fd-per-fd close accounting).
---------
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
The two-Gateway harness checklist in CONFIGURATION.md said a crashed
leader hands its providers over within lease_seconds + grace_seconds; a
peer only tries to acquire on its fixed lease_seconds // 3 tick, so the
worst case is one renewal interval more (50 s with the defaults), as the
leader-lease section of the same file already says. The grace_seconds
field description had the same shorter bound.
The channels.leader_lease comment in config.example.yaml still said web
UI credential changes reach the holder only when it next starts the
channel; since #6988 the holder applies them within one renewal interval.
The harness `check` footer still listed the IM channel gates as waiting
for leader election, which landed in #6843; it now points at the manual
channel item in CONFIGURATION.md, which also gains the runtime-config step.
* fix(stream): tolerate ~~~-fenced code and \-prefixed closer when decoding escaped newlines
_decode_escaped_newlines_outside_fences only recognised ``` fences, so replies
that used ~~~ fences (e.g. code blocks rendered by the model with
~~~python ...) kept literal backslash-n sequences escaped instead of decoding
the ones outside fences, and a fence whose closer was split as backslash+tilde
across chunks was misread mid-stream. Fixes#7262.
- detect the opening fence with (?:```|~~~) and remember which delimiter the
fence was opened with, so closing requires the same delimiter
- while inside a fence, hold back a closer that begins with a backslash so a
fence ending in backslash+tilde mid-stream is not misread; the held tail is
revisited on the next push and normalised on flush
- keep backtick hold semantics unchanged
* fix(mindie): preserve unclosed fences during newline decoding
Match one-shot decoding to streaming by choosing the earliest backtick or tilde opener and preserving the remainder when its matching closer is absent. Correct the string-return docstring and cover mixed delimiters across every two-chunk split and character-by-character streaming.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
