* fix(browserless): resolve the token through the shared credential rule
Browserless read BROWSERLESS_TOKEN and the configured token raw, so a value
from a file kept its trailing newline and became the auth query parameter;
every request then failed authentication. A whitespace-only value was sent as
a token instead of counting as unset. Use provider_credentials.resolve_api_key
like the other search and fetch providers, with the blank-config fallback to
the environment.
* fix(browserless): keep an explicit blank token opt-out
An explicitly configured token ("" or null) is authoritative: a self-hosted
capture endpoint uses it to opt out of authentication, and falling back to
BROWSERLESS_TOKEN sent the cloud credential to that endpoint. Only an omitted
token key falls back to the environment. Regression tests cover the client
token and the outgoing capture query.
* docs(changelog): record the browserless token normalization
Match the sibling #7981 entry in both locales; the fix closes out the
credential-rule family for the bundled search and fetch providers.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(typesafe): build the per-evaluation HTTPX client off the event loop
TypeSafeClient.aask constructs httpx.AsyncClient on the running loop, and
that constructor loads the certificate store synchronously
(create_ssl_context -> ssl.create_default_context(cafile=certifi.where())).
aask is the path GuardrailMiddleware.awrap_tool_call awaits, plus both
memory paths, so every guarded tool call stalls the Gateway loop for the
duration of the trust-store load.
Move the construction into asyncio.to_thread, the shape backend/AGENTS.md
records for the other HTTPX clients and the one the merged Crawl4AI anchor
tests. Requests and cleanup stay on-loop, so the deadline scope, retry
loop, bounded body read and the error taxonomy are untouched.
* fix(typesafe): close clients built during cancellation
Co-authored-by: Niranjan Bajgai <2053809+bajgai@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Niranjan Bajgai <2053809+bajgai@users.noreply.github.com>
* fix(skills): read undecodable skill state as missing
`_read_manifest` and `_read_skill_states` document that an unreadable file
means "no manifest" / "no overrides", and both catch OSError and
JSONDecodeError. A file that is not valid UTF-8 raises UnicodeDecodeError,
which is a ValueError and matches neither, so the projection and the skill
state read raise instead of falling back.
Catch it alongside the existing exceptions.
* fix(skills): keep the reason when profile state cannot be decoded
Review feedback on #7963: the state reader dropped the exception, so a decode
failure that silently discards the user's disabled set left nothing to diagnose,
and the next toggle overwrites those bytes. Log the exception, and log a debug
line when an unreadable projection manifest forces a rebuild.
* docs(skills): state that undecodable skill state fails open
_read_skill_states already degraded a malformed file to {} before this change;
making undecodable bytes take the same path made that explicit, so say so next to
the sandbox check that documents the opposite verdict for a single named skill.
---------
Co-authored-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
* fix(memory): select FTS5 matches by composite score
Score every SQL-filtered match before selecting the final composite top-k.
Keep candidate retention bounded with a streaming heap and preserve the
existing scoring formula and adapter-level filtering behavior.
Add regressions for confidence and time-decay reversals, category filters,
result prefixes, ties, and the public DeerMem storage path.
Fixes#8003
* fix(memory): report matched and retained FTS5 rows
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
remain from its review:
- `stored_next_run_at` caught the ValueErrors a dropped zone or a newer cron
parser raises, but a stored `schedule_spec` that is JSON null, a list or a
string fails on `.get` with AttributeError. The launch-failure handler
recomputes the schedule, so that escaped `run_once`'s claimed-task loop and
skipped the rest of the batch. Catch it too; the task is paused with the
reason like the other unusable schedules.
- `run_once` dispatched its claimed batch with a bare loop, so any per-record
raise in `dispatch_task` re-aborted the batch. Isolate each task's dispatch
and keep claim-level store failures loud above the loop.
Tests: `test_scheduler_poll_batch_isolation.py` (non-object specs, end-to-end
pause, per-task isolation). Red on `91f20a7`, green at head.
Refs #7998
* fix(threads): evict in-memory runs when a thread is deleted
RunManager.get() and list_by_thread() answer from the process-local _runs
registry before consulting the RunStore. DELETE /api/threads/{id} only
removed the durable rows, so a worker that had admitted the thread's runs
kept serving them: after the delete returned 200 and the thread itself
answered 404, GET /api/threads/{id}/runs still listed the run and
GET /api/threads/{id}/runs/{run_id} still returned 200 with the original
prompt intact in kwargs. The residue disappeared only on restart, which
is what made it look like a cache rather than a leak.
Add RunManager.discard_thread_runs() to drop this worker's copies for the
thread and call it from the delete path, next to the durable run
deletion. Internal thread-operation rows are left alone: the reservation
protecting the deletion is one of them and its own release path owns it.
Historical runs, the thread row, run events and checkpoints are unchanged.
* fix(threads): keep a live finalize tail attached when evicting runs
discard_thread_runs() popped every record it found, but worker.py clears
finalizing before the tail that publishes the terminal frame, closes the
journal and releases the sandbox lease. Popping mid-tail turns the tail's
manager-mediated writes into warnings on a record that no longer exists.
Skip records whose task is still running, mirroring what cleanup() already
does through shield(), and pin the guard with a unit test that seeds both
a finished run and a delete reservation. Document that a DELETE served by
one Gateway bounds, rather than removes, a peer's copies: they survive
until that peer's deferred cleanup(run_id) fires.
* fix(threads): hide a deleted run from reads during its finalize tail
A DELETE that arrives after worker.py clears `finalizing` but before the
worker task finishes cannot drop the record: the tail still publishes the
terminal frame, closes the journal and releases the sandbox lease through it,
so popping turns those writes into `set_status called for unknown run`.
Leaving the record readable instead keeps the run and its prompt reachable
through `GET /threads/{id}/runs` until `cleanup()`'s 300s delay fires, which
is exactly the window #8034 reports.
Records that a deletion reaches mid-tail are now flagged instead of skipped.
`get()`, `list_by_thread()`, `get_many_by_thread()`, the regenerate-source and
edit-replay reads skip a flagged record (and never rebuild it from a store row,
since the durable rows were deleted first), and the tail's own completion
evicts it — no second delete request and no timer.
* fix(threads): keep the tail eviction task referenced until it settles
The done-callback eviction is a separate task because it cannot await the
registry lock inline. The loop holds only weak references to tasks, so an
unreferenced eviction could be collected before it drops the record and the
deleted run would come back within `cleanup()`'s 300s window. Track it in a
manager-owned set and discard on completion.
* fix(threads): keep hidden runs hidden on every read path
The deletion flag set by discard_thread_runs is authoritative for "this run
is deleted", but two reads still treated it as a hint.
get() narrowed its store-less guard to a missing record, so a flagged
record on a store-less RunManager fell through to self._store.get() on
None; the AttributeError was swallowed by the broad except below while
every read in the flag window logged a hydration warning with a full
traceback. The record now settles the read before any store call, which
also keeps the read side-effect free as the comment intends. aget()
shares the path.
get_many_by_thread() rebuilt flagged runs from store rows. list_by_thread
already refuses that, because the durable row is deleted before the flag
is set and anything left is a stale or partial read; the batch read now
applies the same guard so the invariant is uniform.
---------
Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
* fix(subagents): keep tests_passed evidence per call when a provider reuses its id
Tool-call ids are unique per response, not per conversation. Some providers
number calls per response, so two turns can both issue call_0.
The bash evidence for tests_passed acceptance assumed one call per id in two
places:
- _harvest_bash_executions built one id -> command map over the whole
conversation, so every result under a reused id was labelled with the
last call's command. A passing subset run (pytest tests/unit) became
evidence for the full suite when a later call_0 ran pytest tests and had
no result yet.
- update_bash_executions merged values chunks by tool_call_id. A dict keeps
a key's first position, so a later run under a reused id overwrote the
earlier entry in place. With call_0 failing, call_1 passing and call_0
failing again, the merged list ended on the stale pass and the leaf held.
Pair results through pair_tool_call_results, which answers a call only from
its own response, and merge chunks by result message id, keeping the call id
for entries without one. Each run stays a separate entry in execution order,
so the latest run decides the leaf again.
* fix(subagents): key the summary cache per run and document id-less calls
- Cache full-output test summaries by result message id (call id only
without one), so reused call ids keep one cache slot per run.
- Update the bash_executions field doc and the subagents guide to the
result-id merge, and state that calls without an id record no
evidence (the leaf stays UNVERIFIED).
- Tests pin both: one classification per run under a reused id, and no
evidence for a call with a missing or empty id.
* chore(subagents): drop the stray empty deerflow/executor.py
The previous commit added a 0-byte module at the package root by mistake.
Nothing imports deerflow.executor; the real module is deerflow/subagents/executor.py.
* fix(skills): give a bracketed literal in the userinfo slot no host on every Python
`_http_host` answered from `urlsplit(url).hostname`, which decides the
ambiguous authority `http://[::1]@Example.COM/api` differently across
interpreters: 3.12 reads the bracketed part as userinfo and returns the host
`example.com`, 3.13 rejects the authority and `urlsplit` raises, so the helper
returns None. The same URL got two different answers depending on the runtime.
The answer is not cosmetic: `_is_outbound_url` compares it against
`_LOCAL_HTTP_HOSTS`, so whether a URL counts as local or outbound moved with the
interpreter. The test table added with the surrounding refactor (#7950) already
states the intended answer, None, which is why `tests/test_skillscan_native.py`
fails on 3.12 and passes on 3.13.
The bracketed-literal case is now rejected before `hostname` is read, so the
answer no longer depends on that URL-parsing change, and the shape keeps the
treatment the other malformed authorities get: no host, therefore outbound.
The test table also gains a bracketed literal with an IPv6 host and a port
(`http://[2001:DB8::1]@example.com:8443/api`), so a fix that only special-cased
the exact URL from the failure cannot pass.
On 3.12: `tests/test_skillscan_native.py` goes from 1 failed / 387 passed to 388
passed, and the skillscan suite (834 tests) passes. Reverting only the
orchestrator change puts both bracketed cases back to failing.
* fix(skills): reject raw brackets in URL userinfo
---------
Co-authored-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): drop empty strings from rebuilt user content
* fix(agents): normalize empty content before sanitizer early returns
* docs(agents): clarify input sanitization provider limits
* fix(channels): keep Gateway calls off the environment proxy
* fix(channels): keep the aio_sandbox policy import lazy
Importing deerflow.community.aio_sandbox.backend at module scope executes
the package __init__, which eagerly imports AioSandbox, AioSandboxProvider
and both backends. That makes every ChannelManager import load the entire
sandbox provider stack even when a different sandbox is configured.
Mirror app/gateway/health.py::_make_provisioner_client: move the import into
a small _make_gateway_client helper and route both Gateway call sites through
it, so the URL policy stays lazy.
* fix(channels): keep the run client off the environment proxy
langgraph_sdk.get_client() builds httpx.AsyncHTTPTransport(retries=5) on an
httpx.AsyncClient that leaves trust_env at its default True and exposes no
transport/trust_env override, so ChannelManager._get_client() still sent
runs.wait submissions to a loopback Gateway through HTTP_PROXY. That is the hop
#7876 reports: the control-plane calls are fixed, the run submission is not.
Route the SDK client through _make_langgraph_client: loopback and private
Gateways get an explicit trust_env=False httpx.AsyncClient carrying the SDK's
own retries, timeout and User-Agent and handed to LangGraphClient, while
external Gateways keep the stock get_client() path.
Also move the URL policy out of the aio_sandbox package into
deerflow.utils.http_policy (control_plane_trust_env). Keeping the import lazy
did not help on its own: aio_sandbox/__init__ eagerly imports the provider
stack, so importing aio_sandbox.backend still loaded all of it. The neutral
module has no httpx or sandbox dependency, so manager.py and gateway/health.py
import it at module scope, and aio_sandbox.backend re-exports it as
sandbox_http_trust_env for the existing sandbox call sites.
* fix(channels): keep AGENTS.md under budget and align legacy CSRF test with new client
* style(sandbox): remove unused urlparse import
---------
Co-authored-by: jack12178 <jack12178@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(middleware): scan PII emails in linear time
The email detector's pattern restarts at every position inside a
whitespace-free run and consumes the rest of the run before failing to
find '@', so a scan is quadratic in the run length. PiiRedactionMiddleware
re-redacts every user message on every model call, on the event loop, so
one pasted 100 KB hex dump stalled the Gateway for ~6 s per model call.
Scan from run starts only (a local part must reach the end of its run),
and match anchored where the previous match ended so glued addresses
such as 'a@b.cc+c@d.org' keep both matches, exactly as before.
* docs(changelog): reference #8006 for the linear PII email scan
* fix(middleware): group the embedded email pattern and pin every detector's linearity
Wrap the pattern behind the run-start lookbehind in a group so a future
top-level alternative in _EMAIL_PATTERN cannot escape the lookbehind, and
add one whitespace-free payload per other detector to the linearity test
so the AGENTS.md rule is enforced for all of them.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(agent): ground runtime decisions in structured evidence
Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.
Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.
* test(sandbox): expect read_file initialization error callback
* fix(agent): address evidence review follow-ups
* fix(agent): preserve retry budgets for OpenAI transport errors
* fix(agent): expose tool execution status in model requests
Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): reject Windows reserved device names in managed subagent creation
* Reject Windows reserved device names (CON, PRN, AUX, NUL, COM1-9, LPT1-9)
in normalize_managed_subagent_name when for_creation=True via
windows_incompatible_segment.
* Validate subagent names on creation in FileManagedSubagentStore.create
and /api/subagents POST gateway router before attempting disk writes.
* Keep read, update, and delete paths permissive for backward compatibility.
* Add unit tests for store validation and router endpoints.
* Update CHANGELOG.md under Fixed.
* fix(subagents): reject reserved names in SQL managed-subagent create
Apply the same for_creation check used by the file store so both
backends refuse Windows device names. Pin mixed-case Nul in tests and
import the shared normalizer at module level.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): keep HarnessGuard from rewriting the result that ends a loop
The alternating-loop check fired when three of the last four tool calls
failed, without checking the current call. After A and B failed in turn,
a successful B therefore received the OSCILLATION DETECTED warning: the
model was told to stop just as the loop ended, and structured output such
as JSON was corrupted by the appended text.
Only a failing call now continues the pattern, so successful results reach
the model unchanged. Failing alternation still warns.
* fix(agents): only warn on an alternating loop when the whole window failed
The oscillation check still counted 3 errors out of the last 4 calls, so a
window that contained a success could warn on the next failure, e.g.
A err, B err, A ok, B err. A success ends the loop, so require all four
calls in the window to have failed, and pin the break shapes with tests.
---------
Co-authored-by: Totoro <msy626836554@gmail.com>
* fix(agents): keep todo reminders in Goal turns after an earlier provider stop
TodoMiddleware skipped completion reminders whenever runtime.context held
safety_capped or model_length_capped. The worker keeps that key across the
run's hidden Goal continuation turns so the run record reports the first
cap, and an unmet Goal continues after these provider stops (it stands down
only on loop and token caps). After one capped response, every continuation
turn therefore lost its todo reminders and could stop with unfinished items.
Loop and token stops stay run-level and are still read from the context.
Provider safety and length stops now apply to the response they ended,
identified by the safety_termination / model_length_termination marker that
those guards stamp on it. The check still runs before the parent hook, so a
capped response cannot be resumed by the parallel-write retry.
* refactor(agents): import provider-stop markers from the guards that stamp them
TodoMiddleware matched the safety and length markers by string literals that
duplicated the guards' own. Each guard now exports its marker and stop reason
as module constants, used at its stamp and check sites and imported by
TodoMiddleware, so a rename in a guard cannot silently disable suppression.
Comment that completion_reminder_hard_stop_reasons is the inventory surface
and only its run-level entries are read from runtime.context, and run the
worker-level Goal continuation test for the safety guard too, which
overwrites stop_reason instead of keeping the first one.
---------
Co-authored-by: Totoro <msy626836554@gmail.com>
* fix(memory): skip text-less turns when formatting the memory prompt
A content-less message rendered as the literal "None", and a block list
without text blocks rendered as its Python repr, including any base64
data URL. Normalize through the same rules as the sibling content-to-text
helpers and skip turns that carry no text.
* test(memory): cover whitespace-only assistant turns
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): pause a task whose schedule no longer computes
Admission validates every schedule, but a stored one can stop computing
later: a zone the installed tz database no longer has, or a cron
expression a newer croniter rejects. The scheduler recomputes next_run_at
inside the shared poll, and that raise escaped:
- dispatch_task recomputed it in its except block after a successful
launch, so run_once aborted the claimed batch and left the occurrence
in "launching" with no last_error;
- once the launch lease expired, recover_expired_launch_claims found the
durable run and raised again for every poll, before claim_due_tasks,
so no other scheduled task was admitted for the rest of the process.
Runtime cursor writes (launch, launch repair, queue timeout, launch
failure, finalization) now use stored_next_run_at, which returns the
reason instead of raising. A live recurring task is paused with
"schedule can no longer be computed: ..." as its last_error and no next
run; a once task keeps its normal outcome. Resume re-validates the
schedule, so it reports the problem until the schedule is fixed.
run_once also isolates the lease-recovery pass, like the terminal-run
reconciliation next to it, so a failed pass no longer skips admission.
* docs(changelog): reference #7993 for the unschedulable task fix
* fix(sandbox): exempt expression division after closers from the host-bash guard
A single-segment absolute-path match directly after ')', ']' or '}' is
division in an inline expression (python3 -c, perl, node -e, $((...)) ),
not a host path; #7023 exempted this shape only inside single-quoted awk
programs. Per-closer rules keep unknown-value path builds rejected:
digits only after ')' and '}' ($1/1024 rationale), an identifier after
')' only when the paren closes a name(...) call, any segment after ']'.
Literal tokens ('ls /2') are unchanged.
Fixes#7444
* fix(sandbox): scope division exemptions to parsed expressions
* docs(sandbox): fit inherited guidance within the hard limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): reject Windows device names when creating a skill
A skill name is a `custom/<name>` directory component, but the hyphen-case
grammar admits Windows reserved device names (`nul`, `con`, `com1`, ...).
`skill_manage` create and `.skill` archive install accepted them: on Windows
`mkdir("nul")` reports success without producing a directory and the first
write inside it fails with FileNotFoundError; on Linux the install succeeds
but the tree can never be copied, mounted or exported to a Windows host.
Creation now rejects those names through `windows_incompatible_segment()`, the
check uploads, sandbox writes, skill support files, thread IDs and agent names
already use. Archive extraction also rejects member segments that cannot be
created on every host. Reads and deletes stay permissive so a skill an older
release created on a Linux host can still be inspected and removed.
Fixes#7945
* test(skills): pin the extraction guard and the tool-level create wiring
Address review findings on the skill-name portability change:
- Isolate the extraction guard: an archive with a portable folder and a
portable frontmatter name but a reserved support-file segment
(`my-skill/references/nul.md`) now fails with the member-path error, so
deleting the guard turns this suite red again.
- Pin the `skill_manage` wiring: `create` of a device name raises through the
tool, while `edit`/`delete` keep reaching the existing "not found" path.
- Reword the `safe_extract_skill_archive` docstring: members before the
offending one are already written into staging, so what the check
guarantees is that the install never commits under `custom/`.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(community): strip provider API keys before they reach a header
GroundRoute (from GROUNDROUTE_API_KEY), InfoQuest, Tavily, Exa, Jina,
Firecrawl, fastCRW and Crawl4AI put the configured or environment key into
an HTTP header as read. A key read from a file keeps its trailing newline
(a Kubernetes Secret created with --from-file and mounted through envFrom),
and the transport refuses it with an error that quotes the whole header:
h11's "Illegal header value b'Bearer <key>\n'" or requests' "... in header
value: '<key>\n'". The providers returned that message as the tool result,
so the key reached the model and the checkpointed conversation, and
GroundRoute, InfoQuest, Jina and Crawl4AI also wrote it to the error log.
Resolve each key through provider_credentials.resolve_api_key, as the other
bundled providers already strip theirs. Tavily, Exa and Firecrawl now read
their environment fallback themselves instead of leaving it to the SDK,
which would read the raw value again. A blank configured key now falls back
to the environment, as it already does for GroundRoute and the stripping
providers.
* docs(changelog): reference #7981 for the provider API key fix
* fix(community): keep fastCRW from sending the Firecrawl key
fastCRW builds the Firecrawl SDK client, and the SDK reads
FIRECRAWL_API_KEY whenever api_key is None. Once a blank configured key
resolved to None, a fastCRW config with api_key: "" and no CRW_API_KEY sent
the Firecrawl key to the fastCRW endpoint. With no api_key configured at
all, main already did the same.
Pass "" when fastCRW resolves no key of its own; the SDK falls back only on
None and sends no Authorization header for an empty key. A loopback test
pins the empty, blank and absent cases with FIRECRAWL_API_KEY set.
* fix(community): keep a blank environment key away from the SDKs
Tavily, Exa and Firecrawl resolved their environment fallback themselves,
but passed None when it was blank, and each SDK then re-read the same
variable unstripped. Tavily and Firecrawl now pass "", which the SDKs
treat as a missing key; Exa sends "" as is, so it fails before any
request instead.
The fastCRW comment records that only the Firecrawl SDK's v2 client
honours "", and the fastCRW regression test now covers web_fetch as well
as web_search, so a move to the v1 client would fail it.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agent): stop todo reminders after run hard limits
Respect loop_capped and token_capped before scheduling completion reminders, and include the behavior in the assembly policy fingerprint.
Cover sync and async agent graphs, blocked post-stop tools, preserved Todo state, and continuation from the same thread checkpoint. Validation: 316 targeted tests passed.
* docs(agent): explain todo behavior after hard stops
Document that loop and token hard stops suppress Todo completion reminders while retaining unfinished tasks for later runs.
* docs(changelog): record todo hard-stop fix
Add English and Chinese Unreleased entries referencing issue #7953.
* fix(agent): mark safety stops with partial responses
* fix(agent): honor provider stops before todo retries
* docs(agent): clarify todo behavior after provider stops
* test(agent): isolate todo safety-chain runtime dependencies
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
* fix(subagents): build the chat model off the event loop
`SubagentExecutor._create_agent` is awaited from `_aexecute` on the shared
event loop, and it constructed the chat model inline. Provider construction can
read local credentials -- the Codex provider loads the Codex CLI `auth.json`
while it is built -- so every delegated task did synchronous file IO on the loop
before its first model call, which stalls every concurrent turn sharing that
loop.
Build the model through `run_assembly`, the dedicated assembly pool the other
provider-construction sites already use, and leave the rest of the middleware
and graph assembly where it is. The helper preserves request context, and a
cancelled construction still never starts an inference.
`tests/blocking_io/test_subagent_model_offloop.py` drives the real
`_create_agent` with a real `CodexChatModel` under the strict Blockbuster gate:
on `main` the credential load fails with `BlockingError(os.stat)`, and reverting
the `run_assembly` call reproduces the same failure.
* docs(subagents): name the isolated subagent loop in the off-loop notes
The blocked loop is the persistent isolated subagent loop every concurrent
delegated task shares, not the Gateway request loop; say so in the boundary
audit, the regression docstring, and the assertion message.
---------
Co-authored-by: Totoro <msy626836554@gmail.com>
* fix(sandbox): build the readiness client off the event loop
`wait_for_sandbox_ready_async` exists so sandbox startup waits do not block the
runtime event loop, yet it constructed `httpx.AsyncClient` inline: the
constructor resolves the system trust store synchronously, so the loop stalled
for the whole construction before the first poll was sent (~320 ms in a local
probe on this host).
Move the construction into a worker thread and own it as a task, the way
backend/AGENTS.md now requires for HTTPX clients: a cancellation that lands
while the worker is still inside the constructor waits for it and closes the
client it builds, so the pool is not abandoned. Requests, polling cadence,
timeouts and cleanup stay on the event loop.
The blocking-I/O regression pins both halves. Against the previous code the
trust-store assertion fires on the loop thread; with a plain `asyncio.to_thread`
instead of the owned task, the cancellation test reports the leaked client.
* fix(sandbox): preserve readiness cancellation during cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
compact_thread_context built the summarization middleware inline on the
Gateway event loop, and with it the run-profile model and any separately
configured summary model. When the provider reads local credential files
(the Codex provider reads the Codex CLI auth.json while it is constructed),
each POST /api/threads/{thread_id}/compact did synchronous file IO on the
shared loop, once for every model it built, so other requests and streams
waited for it. The route passes no app_config, so the call also read the
live config on the loop first, which stats and hashes config.yaml on every
call. #7767 listed this site in the utils guidance as not covered.
Build the middleware through run_assembly, as #7484 and #7767 do for the
one-shot helpers, skill moderation and the goal evaluator, and read the live
config in that pool too when no app_config is passed. The lead agent already
builds the same middleware there through run_agent's agent factory call.
Model resolution, the memory-flush policy, the disabled, failed and
pending-work outcomes and the written checkpoint are unchanged, and
cancellation during construction neither starts the summary call nor writes
a checkpoint.
tests/blocking_io/test_compaction_model_offloop.py builds real
CodexChatModels under the strict Blockbuster gate. On main, compaction
reported itself disabled because every model build hit BlockingError
(os.stat), and the /compact handler answered 500 because the gate rejected
its config read first.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): restore facts from older backups
Use the current snapshot revision for existing facts during backup import so saved contents can be restored after later edits. Preserve manifest and fact preconditions, scope isolation, and caller-owned input data.
* refactor(memory): share import revision preconditions
* docs(memory): fit import guidance within chain budget
* fix(memory): validate backup revisions before restoration
---------
Co-authored-by: liyinlong <173714821+lilong555@users.noreply.github.com>
* fix(parsing): keep literal think tags inside JSON string values
Structured parsers ran strip_think_blocks over the whole response before
json.loads. A model answer that quotes the tag -- "Explain <think>example
</think>" -- lost the quoted text, and an unclosed literal tag inside a
string cut the JSON document mid-string, so suggestions returned None and
goal evaluation failed to parse.
Strip reasoning only outside JSON string literals: leading blocks are still
removed whole (a JSON example inside reasoning is never taken as the
answer), and blocks after the payload starts are removed only outside
quotes. Suggestions and the goal evaluator both use the shared helper.
* fix(parsing): keep think-tag stripping complete outside the JSON payload
The string-aware walker only covered the payload itself: a reasoning block
after prose and before the payload was copied verbatim, so brackets inside
it became the parsers' find/rfind targets, and an unclosed block in
trailing prose was kept where strip_think_blocks used to truncate.
Find the payload start while stepping over think blocks, strip the head with
strip_think_blocks, keep strip/truncate semantics after the top-level
payload closes, and truncate at an unclosed block outside a string. Adds a
CHANGELOG entry for the user-visible parsing change.
Tool-call ids are unique per assistant response, not per conversation.
extract_skills built a single id -> path map over the whole message
list, so a later read that reused an id overwrote the earlier one: the
earlier skill silently dropped out of the ledger and its correct
ToolMessage logged "mismatched skill read metadata".
Rebuild the map from each AIMessage's own calls and resolve a
ToolMessage against the nearest preceding response, matching the
pairing rule tool_call_args.pair_tool_call_results already applies.
Fixes#7858
Signed-off-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
Co-authored-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
* fix(community): initialize InfoQuest TLS clients off the event loop
InfoQuest's fetch, web search and image search constructed
httpx.AsyncClient inline in their coroutines, so the default transport
loaded the TLS trust store synchronously on the agent event loop
(~125 ms per call in a local probe). RAGFlow, LightRAG, You.com,
GroundRoute and Webz already construct their clients with
asyncio.to_thread; InfoQuest was the missed sibling.
Move the three constructions to asyncio.to_thread and cover all three
methods in the blocking-I/O TLS test. Requests, cleanup, timeouts and
responses stay on the event loop.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* docs: qualify the off-loop HTTPX provider list
The list is not exhaustive; several community clients still construct
their client inline. Use non-exhaustive wording until the follow-up
sweep lands.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* docs: correct the off-loop provider list after merging main
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): degrade a blob without mimeType instead of failing the tool call
A resource link or embedded resource without mimeType made the blob guard
raise, so the whole tool call failed instead of degrading the attachment.
Keep the guard for genuinely unusable payloads and let a missing media type
fall through to the caller's default.
* Modify test for text blocks and mime type validation
Update test to check for text blocks and their mime types.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* test(mcp): remove MIME assertion from text content block
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(browserless): construct the fetch and screenshot clients off the event loop
* fix(browserless): close the client a cancelled construction still builds
Keeping the constructor off the loop is right, but awaiting
`asyncio.to_thread(httpx.AsyncClient, ...)` directly is not cancellation-safe:
the await can be cancelled while the worker thread is still inside the
constructor. The await is abandoned, the thread runs on and hands back a live
client, and execution never enters the `async with` that would have closed it,
so `is_closed` stays False and the connection pool leaks on that cancellation.
Own the construction as a task instead. When the caller is cancelled, wait for
the task with the shield loop this package already uses for cleanup, close
whatever client it produced, and only then let the cancellation propagate. A
failure while closing is logged rather than raised, so it cannot mask the
cancellation the caller asked for.
The new regression test gates the constructor on a thread event, cancels the
operation while it is parked there, and releases the worker afterwards. Against
the previous commit it fails for both fetch_html_with_status and
capture_screenshot with "the client built after the cancellation was leaked".
---------
Co-authored-by: inchang-ing <197932532+inchang-ing@users.noreply.github.com>
* fix(sandbox): defer fresh AIO skill-policy acquisition
Prepare the authorized thread skill view before the model runs without
acquiring an AIO container for a fresh lazy run. Keep existing sandbox
bindings and upload-based providers on the eager rebind/sync path.
Add sync/async regressions for first-tool authorization, strict rebinding,
filtered read-only AIO mounts, and policy-scoped warm reuse.
AI assistance: Codex; the account owner confirmed human review of the
complete seven-file patch and responsibility for the change.
* docs(sandbox): keep lazy-isolation guidance within inherited budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): continue native batches and honor item dependencies
* docs: link native batch follow-up changelog entries
* test(migrations): pin isolated history-seed upgrade to 0050
* fix(subagents): preserve follow-up scope, blockers and fair scheduling
* test(memory): freeze capacity evaluation time
* fix(subagents): guard follow-up admission and retain run controls
* fix(subagents): enforce runs:cancel on tool cancellation of follow-up batches
* style(tests): format test_batch_followup_host with ruff
* fix(subagents): drop the admission checkpoint check and keep SDK cancel compatible
The checkpoint check inside run admission waited on the checkpoint saver's
lock while holding the SQLite write lock, which could make another chat's
checkpoint write fail. Remove it; dispatch still checks the conversation
before admission.
A follow-up waiting for an unanswered question now keeps its claim until the
lease expires instead of rereading the conversation every poll. Cancel passes
the caller's run authority only to submitters that take it, so existing SDK
submitters keep the two-argument call. Document that retrying a prerequisite
by hand does not requeue its blocked items, and list 0052 as the migration head.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
