* fix(agents): do not hide invalid config with file fallback
* test(agents): cover invalid on-disk config fallback
* fix(agents): resolve stores off the event loop
* fix(agents): distinguish missing nested config from main config
* fix(agents): reject missing explicit config path
* test(agents): isolate config fallback test
* test(agents): isolate router blocking IO coverage
* test(agents): pin malformed config.yaml parse-error propagation
An unparseable config.yaml used to be swallowed by the broad
except Exception and silently downgrade to FileAgentStore. The
narrowed except FileNotFoundError already propagates
yaml.ParserError/ScannerError; pin that contract with a real
on-disk config instead of monkeypatched get_app_config.
* feat(ci): split backend unit tests into parallel CI shards
Split the single offline backend `make test` job into four GitHub Actions
matrix shards (SPLITS=4, GROUP=1..4) via pytest-split, so the ~12k-test suite
runs in parallel instead of in one 15-minute job. Each shard runs on its own
runner with its own Postgres/Redis services; fail-fast: false lets a failing
shard report its owned tests without cancelling its peers.
`make test` stays the canonical full-suite entry point; CI now calls the new
`make test-shard SPLITS=4 GROUP=N`. tests/blocking_io remains owned solely by
the dedicated blocking-I/O workflow (excluded via --ignore), extending #5105.
Fixes#5088
* test(ci): make backend test shards duration-aware and pin the contract
Make `make test-shard` an explicit least_duration split that READS
backend/.test_durations (read-only for shards, so concurrent CI jobs never
race writes on it), and add `make test-shard-durations` to regenerate that
file from the full offline suite. Update the CI unit-test workflow contract to
call `make test-shard SPLITS=4 GROUP=<n>` and assert the shard command carries
--splits 4, --group 2, -m "not live", --ignore=tests/blocking_io and
--splitting-algorithm least_duration. Verified on the real 13,140-test normal
suite that the four shards are pairwise disjoint and their union equals the
unsplit suite.
Refs #5088
* test(ci): fail fast when the duration baseline is missing
`make test-shard` now requires backend/.test_durations and exits with a clear
error instead of letting pytest-split silently degrade to an even (count-based)
split. Harden the CI contract test to pin `--durations-path=.test_durations` and
to assert the repo ships the committed duration baseline.
Refs #5088
* docs: trim backend/AGENTS.md within guidance budget
* test(ci): add backend test duration baseline
Add the duration baseline generated by a full offline backend run on a
GitHub-hosted ubuntu-latest runner (the same runner type the shards use), so
`make test-shard` balances the four matrix shards by real wall-clock cost.
Refs #5088
* test(ci): make the duration writer honor DURATIONS_FILE
`test-shard-durations` now writes `--durations-path=$(DURATIONS_FILE)` instead of a
hard-coded .test_durations, so the reader and writer stay consistent when the
path is overridden.
Refs #5088
* test(ci): address sharding review feedback
* test: isolate subagent execution capacity state
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): stop stripping filenames when parsing find output in remote providers
The list_dir and glob parsers in the e2b, OpenSandbox, AIO, Tenki, and
BoxLite providers called .strip() on every line of find output. A
filename that legitimately ends (or begins) in whitespace was corrupted,
so the listed path never resolved on any follow-up file API call, and
the remote providers diverged from LocalSandbox, which preserves such
names via pathlib.
splitlines() already removes the line terminators, so filter empty lines
only and keep each entry verbatim. Same class of bug as the e2b
_sync_outputs_to_host fix (#4861), applied to the search parsers.
Adds a trailing-space regression test per provider at the seam each
suite already uses.
* fix(sandbox): split find output on \n only, and rename the tenki test
Review follow-ups from willem-bd:
- aio_sandbox.list_dir used str.splitlines(), which also breaks records on
\v, \f, \x1c-\x1e and \x85 - all legal inside a Linux filename, and all
contrary to this PR's own rule that the newline is the only delimiter.
find emits \n and nothing else, so split("\n") is the correct parse.
- Renamed test_search_preserves_trailing_space_in_filename to
test_list_dir_and_glob_preserve_trailing_space_in_filename, matching the
sibling tests in test_opensandbox_provider.py and test_boxlite_provider.py.
The body covers list_dir and glob; it never touches grep.
* fix(skills): reject a blank SKILL.md description at the write gate
_validate_skill_frontmatter only applied its description rules when the
value was truthy, so a blank or whitespace-only description passed the
gate. The loader rejects it, so the file was written to disk and then
disappeared from every consumer.
On the PUT edit endpoint that meant the write committed first and the
response was a 404 -- the previously working skill was gone with no
rollback. Same shape via the .skill install path and the agent-facing
skill_manage tool, which reported success for a skill that never loads.
Empty names were already rejected; description was the only field where
the write gate and the loader disagreed.
* test(skills): pin rollback rejection of a blank-description history entry
Add a regression test for the rollback path: restoring a history entry
whose stored content has an empty description must return 400
("Description cannot be empty") and leave the on-disk SKILL.md untouched,
rather than the previous destructive path that wrote the unloadable
content and then 404'd. On main this returns 404, so the test also pins
the intended status-code change on this path.
* fix(frontend): truncate long subtask card titles to a single line
The subtask card header rendered task.description without any width
constraint; when a provider omits the optional description, the full
task prompt becomes the title and overflows the card.
Wrap the title in a truncating span (full text remains available via
the title tooltip and the expanded card body), give the step min-w-0
flex-1, and pin the status cluster with shrink-0 so overflow resolves
at the title.
Add an e2e test asserting a long prompt renders with the truncate
class, a real ellipsis (scrollWidth > clientWidth), and single-line
height.
* fix(frontend): keep subtask card status cluster shrinkable on narrow viewports
The shrink-0 status cluster could not shrink below its max-content (model
label + usage + status pill, up to ~456px with a long tool-call
description), so on narrow viewports it overflowed the header row while the
title collapsed to zero. Drop shrink-0 and add min-w-0 to both the cluster
and the pill (the pill's min-content is the status text's longest
unbreakable word, so one min-w-0 was not enough), and floor the title at
min-w-24 so it stays visible.
Also extend the e2e spec per review: an in_progress shimmer truncation test
(held-open SSE stream keeps the card running), a 375px no-overflow
assertion for both the resting and running card, and a pixel-budget
single-line check instead of parseFloat(lineHeight) which NaNs on the
'normal' keyword.
* test(frontend): honest fixture text and explicit visibility timeout in subtask spec
Review nits: the long-title fixture lifted the stopped test's human turn
whose text narrates the stop scenario; give it its own LONG_TASK_USER_TEXT
and override content alongside id and tool_calls. Add the missing 15s
timeout on the running-375px title visibility assertion so a future
reorder doesn't turn the 5s default into a cold-start flake.
* fix(frontend): default to Webpack over Turbopack in dev to avoid PostCSS worker leak on macOS
On macOS arm64, Turbopack + Next.js 16.2.11 + Tailwind CSS v4 causes an
unbounded spawn of PostCSS evaluator processes that consume high CPU and
memory and never return a response. Webpack is unaffected.
Change the no-override default in getDevBundler() from platform-dependent
Turbopack (all non-Windows) to Webpack. DEER_FLOW_DEV_BUNDLER=turbo
continues to work as an explicit opt-in for local diagnosis.
Fixes#5132
* docs(frontend): address webpack default review feedback
* docs(frontend): clarify webpack default rationale
* fix(history): stop dropping user messages that fall outside the loaded page window
Two independent paths made a user's own message disappear from a long thread
(#4666, #4508, #4363). Both are reproduced by a real two-round run: once the
thread passes the 50-row `/messages/page` window AND context compaction fires,
the two sources of truth stop overlapping at the head.
1. Middleware-answered tool results never reached the event store. A middleware
that short-circuits a tool call (e.g. ReadBeforeWriteMiddleware's blocked
write) returns a user-visible ToolMessage, but LangChain never emits
`on_tool_end`, so RunJournal never persisted it — the user saw it during the
run and it vanished on reload. RunJournal already reconciles final-output
tool messages, but only for an `ask_clarification` allowlist. The allowlist
is removed; scope stays bounded by the three conditions that actually matter
(visible, this run's lead agent, not already persisted), so subagent results
still stay in their own step feed.
2. mergeMessages discarded the checkpoint prefix before the first shared anchor.
#4065 correctly established that a summarization-rescued early message must
not be appended to the tail, and suppressed it instead. That suppression is
what deletes the message when the first history page no longer reaches back
to it. It is now woven in before the first shared anchor — the one position
both the checkpoint and seq-sorted history agree on — so #4065's invariant
(never the tail) still holds. A collapsed unloaded gap is recoverable by
paging; a dropped message is not.
Verified against real captured payloads from the reproducing run: the first user
message returns to the transcript. Its exact position is still approximate —
after compaction the live window carries too few anchors to place it precisely,
which only seq-based ordering can close.
Backend: 10809 passed (baseline 10808; same 15 pre-existing failures in
browser/crawler community tools). Frontend: 986 passed, typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(events): look up a persisted message's seq by identity
Groundwork for placing checkpoint messages in the seq-ordered thread feed
(#4666). A checkpoint carries no seq of its own and loses messages to
summarization, so once the feed's 50-row page window no longer reaches back to a
surviving old message, a client has nothing to place it by. The seq already
exists in run_events keyed by the message id — this exposes it without paging
the whole feed.
`message_identity` is the backend half of the identity rule the frontend applies
in `hooks.ts::messageIdentity`: a ToolMessage is keyed by `tool_call_id`, and
DynamicContextMiddleware's `X` / `X__user` human copies collapse to one identity.
The two halves must stay in sync — a mismatch is silent, degrading placement
rather than raising.
`get_message_seqs` is implemented for all three stores. Misses are absent from
the result rather than an error, so callers degrade to their own placement rule;
the earliest seq wins when one identity resolves to several rows, so a
re-persisted message keeps the position it first occupied. The DB store decodes
rows in Python because `content` is a TEXT column holding a JSON string, not a
JSON column — the identity fields cannot be projected in SQL.
Nothing consumes this yet; no behavior change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(runtime): carry each persisted message's feed seq on values frames
Attaches `additional_kwargs.deerflow_seq` to messages in a root `values` frame
that the thread feed already holds, so a client can place a message the
checkpoint kept but its loaded history page window no longer reaches (#4666).
Nothing is written back to the checkpoint: the seq is added when the frame is
serialized and belongs to that frame only.
Cost is bounded to frames introducing identities the run has not resolved yet.
Messages this run produces are not in the feed while streaming, so they are
looked up once, recorded as misses, and never retried — in a real run the only
frame that pays for a query is the one where compaction brings older messages
back into view. Measured on a reproducing two-round run: 1 lookup across 25
values frames.
The stamper is built once per run rather than per `_stream_once`, or a goal
continuation would discard the resolved seqs. Subgraph frames are not stamped:
a subagent's snapshot is not part of this thread's feed ordering. A lookup
failure logs and leaves the frame unstamped rather than failing it — placement
is an enhancement and clients fall back to their own ordering rule.
Frontend does not read the field yet; no behavior change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(gateway): strip the server-owned message seq from untrusted input
`deerflow_seq` is display metadata the Gateway attaches when it serializes a
values frame. A client replaying messages (regenerate / edit-and-rerun) would
otherwise write it into the checkpoint, where it becomes wrong the moment the
thread is forked — a branch re-seeds its feed and reassigns seq (#4380).
Joins the existing server-owned key set, so it follows the same trusted-internal
rule as the dynamic-context and view-image markers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(frontend): place a checkpoint message by its feed seq, not its nearest anchor
Completes #4666. Weaving a compaction-rescued message before the first shared
anchor keeps it in the transcript, but not in the right place: after compaction
the live window carries too few anchors, and the nearest one can sit deep inside
the loaded page window — measured at row 25 of 50 on a reproducing run, which is
why the first user turn rendered mid-transcript instead of at the head.
Both sides now carry the backend's thread-global seq. `buildVisibleHistoryMessages`
copies each row's `seq` onto the message (same shape as the existing `run_id`),
and the Gateway stamps it onto `values` frame messages it has already persisted.
A live message whose seq is below the loaded window's lower bound is placed ahead
of everything on screen rather than before the nearest anchor. A message with no
seq — still streaming, so not in the feed yet — keeps the weaving path, since the
tail is already its correct position.
Verified against the captured payloads of the reproducing run: the first user
message goes from absent, to #13 (behind the second question), to #0.
Frontend: 988 passed, typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(frontend): place a pre-window checkpoint message even when no anchor is shared
Also #4666. Placing a compaction-rescued message by its feed seq was gated on
reaching a shared anchor, because the split ran inside the anchor walk. When the
loaded page and the live checkpoint share no identity at all, that walk never
runs and the message fell through to `[...canonical, ...live]` — appended after
the entire window, the one arrangement #4065 proved wrong, with its seq known
the whole time.
That is not a corner case. Open an old, already-summarized conversation and send
a message: the page on screen is the newest rows from before that turn, while
the checkpoint holds the rescued first user turn plus steps of the new run that
are not in the feed yet. On a reproducing run the two sides shared zero anchors
and the user's own first question rendered at row 50 of 50 — the reported
"first message jumps to the bottom".
Split `beforeWindow` out of `live` before walking anchors, walk `liveInWindow`,
and use it for the no-anchor branch as well, so a message routed ahead of the
window is not re-appended at the tail by dedup.
Measured on captured payloads of a reproducing run (real gateway, real
compaction), first user message position:
no shared anchor: row 50 -> row 0, seq order monotonic again
shared anchors: row 0 -> row 0 (unchanged)
paged to the top: row 0 -> row 0 (unchanged)
Regression test verified red-green: reverting the fix fails it with the message
rendered after the window.
Frontend: 989 passed, eslint + tsc clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(gateway): stamp the message feed seq on checkpoint reads, not only on stream frames
Completes #4666. `_MessageSeqStamper` sits on the streaming publish path, so a
client that joins a live run learns where a summarization-rescued turn belongs
while a client that merely opens the conversation does not — and opening is the
common case. `GET /threads/{id}/state` and `POST /threads/{id}/history` returned
the checkpoint with no seq at all, so the merge fell back to the nearest shared
anchor, which after summarization sits deep inside the loaded page.
Reproduced in a browser against a real gateway, on a thread that had already
compacted: the user's first question rendered at row 320 of 389, behind the
newest question instead of at the head. Both reads showed 0 of 13 messages
carrying a seq. That is the reported symptom, still present after the streaming
fix.
Add `stamp_messages_with_seq`, the request-scoped counterpart of the stamper:
everything a checkpoint still holds is already persisted, so one batched lookup
resolves the whole list and there is nothing to retry later. Resolve the store
through `_optional_run_event_store` rather than `get_run_event_store`, because
seq is placement metadata — a deployment without a feed must still be able to
read a thread.
After the fix, on the same thread in the same browser: 13 of 13 messages carry a
seq and the first question renders at the head, ahead of the newest one.
Backend: ruff clean, 326 passed across the touched suites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(harness): move the injected-user-id suffix helpers to utils.messages to break an import cycle
message_identity imported strip_injected_user_message_id_suffix from the
dynamic-context middleware, closing a cycle (middleware -> deerflow.runtime
-> worker -> events -> middleware) that only stayed hidden while an earlier
import happened to break it. Define INJECTED_USER_MESSAGE_ID_SUFFIX and the
strip helper in deerflow.utils.messages and re-export them from the
middleware so existing importers keep working.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(docs): improve formatting and clarity in AGENTS.md and message-merge.test.ts
* perf(events): stop the seq scan once every wanted identity is resolved
Rows past the last wanted seq can only be re-persisted copies that
already lose the earliest-seq-wins tiebreak, so all three stores now
break out of the scan (and the db store out of its per-row JSON
decoding) once found covers wanted. Matters most for /state and
/history reads of long threads, where this lookup runs with no run
cache and a typically tiny wanted set.
Raised by review on #4696.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(events): share the seq-stamping expression between the two stampers
The walrus-plus-merge expression was duplicated verbatim between
stamp_messages_with_seq and _MessageSeqStamper.stamp — two counterparts
of one rule where silent divergence is the likely failure mode if only
one side is edited. Both now call attach_message_seq next to
MESSAGE_SEQ_KEY in message_identity.py. The trailing
isinstance(message, Mapping) guard was unreachable (a non-Mapping entry
already got identity = None) and is gone with the extraction.
Raised by review on #4696.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(events): seq stamping survives launch paths without user context
The db store's get_message_seqs defaults to user_id=AUTO, which raises
when no user is in the contextvar — the first strict-AUTO read ever
called from the worker context. On a launch path that never inherits
the auth context (e.g. a null-owner scheduled task), stamp()'s except
clause swallowed that into a per-frame warning and silently disabled
seq stamping for exactly the background runs that need it.
The stamper now soft-resolves the user id once at build time — the
same rule as the worker's write paths beside it (unset -> no filter)
— and passes it explicitly. jsonl/memory stores gain the same
user_id kwarg the base list_messages contract already carries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(events): SQL-prefilter the message seq lookup's candidate rows
get_message_seqs scanned and JSON-decoded every message row of the
thread: the early exit never fires when a wanted identity is absent
from the feed (a message still streaming, or checkpoint-only), and
/state / /history reads want the newest messages, so the ascending
scan traversed essentially the whole feed — with the content column
carrying full tool outputs, that is heavy I/O plus N JSON parses on
exactly the long threads this lookup exists for.
A LIKE prefilter now keeps that cost in SQL: only rows containing a
wanted raw id as a substring are fetched and decoded. False positives
are re-checked by message_identity; LIKE wildcards are escaped; an id
json.dumps would escape (breaking the verbatim-substring guarantee)
falls the whole set back to the full scan rather than silently
missing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(agents): sink runtime mechanism docs below the gateway guidance budget
Merging main pushed backend/app/gateway/AGENTS.md past its 40KB soft
budget (main had left 81 bytes of headroom). Per the nearest-file rule,
move the mechanism detail of the message-seq stamping and run-delivery
receipt sections — both owned by runtime/ code — into
packages/harness/deerflow/runtime/AGENTS.md, leaving the gateway file
the REST-surface summary and a pointer. The seq section also documents
the stamper's build-time soft user-id resolution and the db store's SQL
prefilter from the review follow-ups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(agents): sink durable-MCP task detail below the backend guidance budget
Merging main pushed backend/AGENTS.md past its 24KB module soft budget
(main itself is at 24762 after #4848 — this branch adds zero net bytes
to the file). Per the nearest-file rule, move the two durable-MCP task
runtime bullets' mechanism detail into
packages/harness/deerflow/mcp/AGENTS.md, leaving summaries and
pointers; this also restores ~2KB of headroom so the next merge does
not trip the same wire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(events): re-ask a message-seq miss once the feed advances
The run-scoped stamper cached lookup misses for the whole run. A message
this run produces reaches a values frame before RunJournal flushes it, so
its first lookup legitimately misses — and the journal persists it moments
later, giving it a feed seq the stamper never asks for again. A long run
that afterwards rolls past the history page and compacts then carries that
message unstamped, back to the approximate anchor placement this stamper
exists to replace (#4666). A transient store error had the same permanent
effect, since the except clause degrades to an empty result.
A miss is now provisional while a hit stays final: RunJournal counts its
successful event-store writes as `feed_generation`, and the stamper re-asks
a missed identity only once that counter moves. Retrying is therefore
bounded by feed writes rather than by frames — the per-frame query the
run-scoped cache was built to avoid — and a failed lookup costs one
generation instead of the run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(gateway): issue request trace ids unconditionally
The request trace id was gated behind logging.enhance.enabled at every
entry point, so downstream code had to keep asking whether one existed:
a header-provenance flag in its own ContextVar, a precedence resolver,
and three-level carrier fallbacks at each consumer.
Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP;
ensure_trace_context covers the entry points that never touch ASGI --
scheduled occurrences, MCP task notification runs, IM channel messages,
and the embedded client -- each scoped to one unit of work so a
long-lived worker task cannot leak one occurrence's id into the next.
The ContextVar becomes the only source; the response header, runtime
context, run metadata and log records are derived outputs.
Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and
drop their presence guards. Removed: resolve_deerflow_trace_id, the
header-provenance flag and its three helpers, set/reset_current_trace_id,
is_trace_correlation_enabled and its gateway alias.
BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and
it cannot be turned off; logging.enhance.enabled controls log output
only. Installations on the default enabled: false will start seeing the
header. No config keys were added or removed.
* fix(gateway): stop persisting a caller-supplied trace id on the run record
body.metadata forks two ways: through build_run_config into the live run
config, which the run worker restamps, and through create_or_reject into
the run record that the runs API echoes verbatim. Only the first was
covered, so a client sending metadata.deerflow_trace_id made the most
durable and most visible surface of a run disagree with the X-Trace-Id
and the log lines the same request produced -- a correlation id that
does not match the logs is worse than none.
Stamp the server-issued id once at the trust boundary so both forks
receive it, preserving the caller's own metadata keys. Close the same
gap on config.context, which reaches the runtime context by a separate
path: _build_runtime_context no longer merges server-owned keys from the
caller, and _install_runtime_context assigns rather than setdefaults.
A thread's metadata is no longer seeded with the run-scoped id of
whichever run created it -- one thread spans many runs and as many
trace ids.
Found by driving a real run through the Gateway and reading the run back
from the runs API; every unit test built its metadata by hand and so
could not see it.
* fix(gateway): expose X-Trace-Id to split-origin browser clients
X-Trace-Id is not on the CORS safelist, so a browser client served from
a separate origin could not read it -- and those are exactly the clients
that cannot read the Gateway's logs either, leaving them with nothing to
quote in a bug report. Same-origin nginx deployments were unaffected,
which is why this stayed hidden.
Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing
TRACE_ID_HEADER rather than repeating the literal.
* fix(gateway): keep X-Trace-Id on unhandled-exception 500s
Starlette's ServerErrorMiddleware sits outside every user middleware and
emits unhandled-exception 500s through the raw send, so those responses
never pass TraceMiddleware's header-writing wrapper. The 500 for a server
bug is exactly the response a user most needs to correlate with a log line,
and it was the one response that shipped without the id.
TraceMiddleware now tracks whether http.response.start has been sent. On an
exception with no response started it emits its own plain 500 carrying the
header, then re-raises: the outer ServerErrorMiddleware sees the response
already started and only re-raises too, so the server's exception logging is
untouched. An exception mid-stream keeps propagating unchanged — a second
response start cannot be sent, and the already-written header stands.
The trace id is printable ASCII by construction (normalize_trace_id /
generate_trace_id), which is what makes the raw latin-1 header encoding
safe.
* fix(gateway): strip the forged trace id from the persisted request echo
The run-record fix stopped a forged metadata.deerflow_trace_id on the
authoritative metadata surface, but the raw request echo still carried one:
create_or_reject persists body.config verbatim as runs.kwargs_json, which
the runs API serves back. A client posting config.context.deerflow_trace_id
therefore still got its forged value stored and echoed on one API surface
while the header, logs, run metadata, and checkpoint all carried the real
id — the id is ignored as input there, so echoing it back only manufactures
disagreement.
Two changes close it. redact_config_secrets — already the shared scrub for
that echo, applied at admission and again at serve time, so historical
records are covered too — now also drops deerflow_trace_id from
config.metadata and config.context. And build_run_config now merges run
metadata onto a copy of the caller's config["metadata"] instead of updating
it in place: the nested values of the request config are reference copies,
so the in-place merge was writing the server-stamped key through into
body.config, contaminating the "what the client sent" record before it was
persisted (and incidentally masking the forged-value echo on the metadata
container).
The regression test posts a forged id through body.metadata,
config.metadata, and config.context at once and reads the kwargs echo back
off the run record, failing if either leak returns.
* docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence
The trace section of the harness AGENTS.md now covers the two fixes that
close the derived-output rule (the kwargs-echo scrub in
redact_config_secrets plus build_run_config's copy merge, and
TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains
their Fixed entries.
It also writes down the one accepted divergence: a crash-recovered
scheduled launch reuses the durable run through its idempotency key, and
start_run returns early on idempotency_reused without restamping — so the
run record keeps the first attempt's deerflow_trace_id while the retry's
own log lines carry the freshly minted id of its ensure_trace_context
binding. The divergence is confined to the crash-recovery window and is
accepted rather than fixed: restamping on reuse would rewrite a persisted
record for a run that already exists, which is worse than two ids that each
correlate their own attempt's logs. Written down so the next reader of the
scheduler recovery path does not diagnose it as a bug.
* docs(config): align the logging.enhance schema note with the unconditional trace id
The config-module AGENTS.md still described logging.enhance as the gate for
the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is
gone: ids are issued unconditionally and this block decides log output only.
Left as-is, the stale wording invites an agent to "restore" a header gate it
believes was lost. Reworded to match the sibling AGENTS.md files and
config.example.yaml, with a pointer to the Request Trace Context section
that owns the full model.
* docs(changelog): link the trace entries to #5119
The five new entries pointed at the ([#XXXX]) placeholder with no reference
definition, rendering as literal text instead of a link — and RELEASING.md
step 2 relies on those references when the section becomes release notes.
All five now point at #5119, with the definition appended to the reference
block.
* refactor(harness): rename _stream_without_trace_context to _stream_turn
The name asserted the opposite of what the method now does. It was accurate
while logging.enhance.enabled could route stream() around the trace scope;
with the gate gone it is the only stream implementation left, and it binds
the id itself via ensure_trace_id(). Private, so the rename touches only the
definition and the one stream() call site.
* docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget
The expanded Request Trace Context section pushed the effective AGENTS.md
chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit
scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the
section from 7,359 to 4592 bytes with no facts removed: the entry-point
table, the derived-output rule and its enforcement points, the accepted
scheduled-retry divergence, the two resolution helpers, the stream()
binding rationale, the log-output-only gate, the CORS listing, the 500
fallback, and the test map all remain.
Sized against the merge, not just the branch: current main grew the same
chain by ~724 bytes, so the check was verified on the merged tree as well
(97,772 bytes; branch tree 97,048).
* fix(gateway): declare content-length on the fallback 500
The pre-response 500 declared content-type but no content-length, leaving
the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on
HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response
it replaces, which sends content-length: 21. The explicit header keeps the
fallback byte-identical to what clients saw before.
* docs(readme): drop the trace-correlation condition from the translations
The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id
matches X-Trace-Id "when request trace correlation is enabled". The id now
always matches and that condition no longer exists, so each bullet states
the unconditional match and that logging.enhance.enabled only controls
whether the id is printed into logs — the one piece of the feature a user
can still configure.
* test(gateway): pin TraceMiddleware wiring through create_app()
Every X-Trace-Id test exercised a hand-built four-route app, so the real
stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting
it — or short-circuiting above it — passed CI while silently dropping both
the response header and the ambient id the run-record stamp and enhanced log
records derive from. One case now drives /health through create_app() and
asserts the inbound id round-trips; mutation-checked by removing the wiring
line, which fails exactly this test.
* docs(gateway): note the fallback 500 is CORS-opaque
The pre-response 500 is emitted outside CORSMiddleware — the exception has
already unwound past it — so it carries no Access-Control-Allow-Origin and
a split-origin browser client cannot read the id on this one response,
unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the
class and in the CHANGELOG entry rather than fixed: replicating the origin
allowlist outside CORSMiddleware would let the two policies drift.
* fix(harness): keep abandoned-stream cleanup inside the trace binding
stream() binds the turn's id around each next(inner) and resets it before
yielding, but the finally's inner.close() ran after that binding was gone.
Abandoning the stream therefore drove the inner LangGraph generator's
GeneratorExit/finally path with no trace id — or an unrelated ambient one
from whichever context ran the close — so cancellation and finalization
logs and callbacks did not correlate with the turn they belong to.
inner.close() is now wrapped in a local bind/reset of the same turn id. The
token is set and reset in the same frame, never across a yield, so the
per-step cross-context safety is preserved even when GC closes the
generator from another Context — pinned by the existing copy_context close
test, which now exercises this path. The regression test records the id
from the inner generator's finally and fails without the binding.
* test(harness): teach the worker-trace fake about RunManager.cleanup
Upstream #5112 (bound gateway memory after terminal runs) added a
run_manager.cleanup(run_id) call to run_agent's finalization, so the
merge-commit CI run failed all five worker-trace-binding tests with
AttributeError on this PR's _FakeRunManager. The fake gains the same no-op
shape as its other methods.
* docs(gateway): bring the gateway AGENTS.md back under its soft budget
Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over
the 40,960 soft budget that
test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes
enforces — its Unit Tests run on main was cancelled by push concurrency, so
main is currently red on that test and every PR merge-run inherits the
failure. Two whitespace/wording trims in the row #5092 touched (a doubled
space, and "its configured `context_window`" → "its `context_window`")
bring the file to 40,953 with no content change.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2)
PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a
subagent completes, so objectively checkable requirements can never be
silently passed by a self-report.
- subagents/acceptance_checks.py: deterministic leaf families —
file:<path> exists|non-empty and file_written:<path> read through
read_current_file_content scoped to the shared thread workspace; the
read uses the sandbox-native virtual path form (the local read
validator and provider mount tables resolve /mnt/user-data/... paths,
not host paths); the scope decision canonicalizes with realpath on the
local sandbox so workspace symlinks cannot escape into uploads; a
remote provider's "Error: ..." return string is normalized to a
failed check (provider-typed via is_local_sandbox); a
UnicodeDecodeError marks a binary deliverable as existing and
non-empty; out-of-scope paths degrade to UNVERIFIED.
tests_passed:<command> anchors to a matching recorded bash execution
with status=success and a test-summary shape; matching is
shell-structure aware with control-flow attribution (span must end at
the last segment with provable execution), negating-option values are
ineligible evidence and a target negated anywhere in the command
degrades the match, extra flags must be selection-preserving, extra
positionals widen only after a path-scoped criterion, truncated
commands degrade via command_truncated, the summary shape is read
only from output attributable to the matched segment (preceding
segments provably silent by invocation form), and pass shapes require
a nonzero passed count. Criterion text is neutralized with
neutralize_untrusted_tags before storage/rendering. Anything else
renders UNVERIFIED, never silently passed.
- executor: accumulate bounded bash command/output evidence per streamed
chunk (merged by tool_call_id, newest-capped) so subagent
summarization compacting earlier messages cannot erase a recorded
execution; the recorded status is the actual shell exit status parsed
from the output's exit marker (signed codes included; the remote
Command exited with code N form is accepted only as the whole trimmed
output), falling back to deerflow_tool_meta only when no marker
exists.
- sandbox providers: e2b/opensandbox/tenki/boxlite append the
LocalSandbox-style "Exit Code: N" marker on nonzero exit even with
non-empty output; aio propagates the SDK's structured exit_code on
both exec paths the same way; local timeouts append Exit Code: 124;
and _truncate_bash_output always preserves a trailing exit marker
(signed included) inside its budget, with a 32-char floor raising any
smaller configured limit, so the actual shell outcome always survives
in the output text.
- task_tool: run the checklist offloaded (asyncio.to_thread) on the
completed branch, failure-isolated; stamp the verdict into result
metadata and render the per-criterion section into the model-visible
result text.
- status contract: additive subagent_acceptance_verdict transport with
read-side structural validation.
- delegation ledger: entry carries the verdict and renders a compact
acceptance segment; gateway strips caller-forged verdicts from both
ledger entries and message metadata, like the citation verdict.
- blocking-IO anchor pins the offload (teeth proven red->green); leaf
read errors catch only OSError/SandboxError so unexpected errors reach
the task-tool-level isolation instead of being mislabeled.
* fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4)
- negating options: overlap with a matched criterion target is now
checked by path/nodeid prefix, not exact token equality — excluding a
sub-path of the criterion's selection (pytest tests --deselect
tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds
- output attribution: any redirection token in the matched final segment
makes the recorded tail non-attributable (> / >> / 2> are word
characters to the parser, so redirection was invisible to the matcher)
- silent-source allowlist narrowed from any *activate suffix to the
*/bin/activate shape
- status_contract docstring: restore the shared-fixture sentence and
note subagent_acceptance_verdict is deliberately outside the fixture
- executor: update_bash_executions publishes [] (stream carried no
bash-family calls) instead of collapsing it into None, mirroring
update_tool_receipts
* fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4)
- tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error"
and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass
shape when the exit status is swallowed (|| true) or absent; zero-error
counts stay clean.
- file leaves: bound the deliverable read — a "wc -c" shell size probe
answers files above 50k bytes without loading ~2x their size, honoring
the host-bash kill switch and falling back to the full read on any
non-integer rendering, so verdicts never get less sound.
- executor: record the exit marker text as status_marker on harvested bash
evidence; the leaf detail now reports the marker actually seen instead of
asserting a failure indistinguishable from the command's own trailing text.
- extend the blocking-IO anchor to drive the probe branch inside the
offload; teeth re-verified red->green.
* fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4)
- file leaves: never read unbounded — size is established first (os.stat on
the validated local host path, so the host-bash-disabled configuration
needs no shell; a guarded wc -c on remote providers that renders
missing/unreadable in its own words). Above the 50k cap the leaf answers
from the size alone, at/below it the full read runs, and an
unestablishable size degrades to UNVERIFIED instead of an unlimited
fallback read.
- output attribution: source/. prefixes are never provably silent — a
crafted */bin/activate path shape says nothing about what the script
prints, so sourced segments can no longer lend a passing summary.
- executable identity: an explicitly path-spelled criterion now requires
the same normalized executable path; the basename rule stays only for
deliberately bare criterion commands.
* fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4)
- remote probe no longer runs in the sandbox's persistent shell: a fresh
env -i /bin/sh with absolute-path stat/realpath (poisoned functions,
aliases, PATH, exported functions, IFS, locale cannot steer it), plus a
marker env routing AIO onto a fresh per-call bash.exec session.
- metadata-only: stat never opens content, so a FIFO deliverable cannot
block the parent for the provider's idle timeout; non-regular files
(fifo/dir/symlink) degrade to UNVERIFIED.
- containment canonicalized against the literal mount root: a
final-component symlink or a swapped parent directory (root included)
cannot redirect the check outside shared storage; unprovable layouts
degrade to UNVERIFIED.
* fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4)
Literal-root equality made every remote file leaf permanently UNVERIFIED
on e2b and Tenki, which realize /mnt/user-data as a symlink to the home
dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink).
Containment now compares the file's realpath against the mount root's
realpath — exactly what the provider's own read path resolves, so probe
and read-back stay consistent; final-component symlinks stay rejected by
the non-dereferencing stat, and an intermediate dir-link escape under a
sane root still lands ESCAPED. The inner script is a module constant and
the suite now executes the composed probe for real against on-disk
layouts (real dir, symlinked prefix, final symlink, fifo, missing,
dir-link escape), which the canned-output stub could not see.
* fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4)
- matching: a criterion with no positional selection target (bare pytest,
make test) stands for the runner's default selection, so ANY negating
option (--ignore/--deselect/...) makes the recorded run a different
selection — unprovable. The overlap guard only sees consumed criterion
tokens, which a bare criterion does not have; scoped criteria keep the
unrelated-exclusion behavior.
- attribution: cd is no longer blanket-silent — CDPATH makes cd print the
resolved (subagent-chosen) destination and the pass shapes match as
substrings, so one mkdir 'all tests passed' plus an export minted a pass
for any quiet command. A cd argument or CDPATH= value (export or leading
assignment) carrying any summary shape makes the segment non-silent;
shape-free cd dir wrappers keep matching.
- docs: _truncate_bash_output states the effective 32-char floor (the
guarantee previously read as an unconditional max_chars bound).
* fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4)
Self-audit in the shape of the last review rounds — channels the matcher
classified as accounted-for that can change what runs, narrow the
selection, or lend the summary text:
- env assignments are no longer blanket-stripped: only an allowlist of
inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a
matched span, and a non-allowlisted assignment in any preceding segment
(pure-assignment or export NAME=) is state pollution — PATH redirects
the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code,
PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs,
BASH_ENV runs arbitrary shell startup. All degrade to unprovable.
- runtime expansions: any span token carrying /$( )/backticks, any
negating-option value carrying an expansion or glob (unknown excluded
set), and any extra executed token carrying glob metacharacters
(crafted option-looking filenames narrow invisibly) are unprovable.
Criterion-side globs stay self-consistent (literal match).
- cd: an argument carrying a runtime expansion or glob is non-silent
(unknown destination, unknown print); CDPATH= assignments are now
handled as state pollution at the match layer, subsuming the
value-shape special case.
* fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4)
- tests_passed: on a persistent-shell provider (new
Sandbox.persistent_shell_sessions capability, set by AioSandbox) every
leaf degrades to UNVERIFIED — any earlier call in the shared session
could have mutated the state the clean-looking run executed in, and
only a fresh controlled session (RFC section 6 verifier) can prove
otherwise. The flag is read from the provider registry without
acquiring a sandbox.
- env assignments: the allowlist is gone — no variable is provably inert
across repositories (CI/DEBUG are routinely read by tests). The span's
assignment prefix must equal the criterion's exactly (values included,
order-insensitive); any assignment or export NAME= in a preceding
segment is state pollution.
- scoping: positional targets are now read by option arity, so a path
embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never
counts as a selection target and an extra positional after such a
criterion narrows the default selection it denotes.
* fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4)
* fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4)
* fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4)
* fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4)
* fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4)
* fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4)
* fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4)
* fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4)
* fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4)
* fix(runs): reject cancel actions on GET stream joins
stream_existing_run is registered for both GET and POST, and its
?action=interrupt|rollback branch cancels the run. The CSRF middleware
exempts GET, so a session-authenticated browser could be forced
cross-site (img/script/top-level navigation) into
GET /api/threads/{id}/runs/{run_id}/stream?action=interrupt|rollback —
a state-changing GET that bypasses the CSRF protection guarding the
POST variant. Introduced with the dual registration in #1403.
The handler's docstring already documents cancel-then-stream as
POST-only (the LangGraph SDK's joinStream/useStream stop button uses
POST); enforce it: GET with an action answers 405, action-less GET
joins and POST cancel-then-stream are unchanged.
Regression drives the real router: GET+action is 405 with the run left
running, plain GET join still streams, POST+action still cancels.
* fix(runs): scope the 405 detail to the action requirement
"GET is a read-only stream join" overstates the current main: on a
locally-owned run with the default on_disconnect=cancel, a GET join's
disconnect can still trigger cancellation. That observer-disconnect
vector is closed by #5041; the detail here should only claim what this
guard enforces.
* fix(runs): harden GET stream action rejection
* fix(runs): align stream schema with method contract
* test(runs): pin GET stream action 405 through the production stack
Review follow-up (defence-in-depth): the GET-action suite drove bare
FastAPI() apps, so nothing pinned that a session-authenticated
cross-site GET reaches the route gate at all once CSRF exempts the
safe method. test_pat_auth.py already assembles the production
middleware order (AuthMiddleware inner, CSRFMiddleware outer), so its
mirror app now registers the real _reject_get_stream_action
dependency on a GET join route.
The new case pins the end-to-end premise: an authenticated GET
?action=interrupt is answered 405 + Allow: POST by the production
route dependency, while the same unauthenticated GET dies at
AuthMiddleware's 401 before any route logic runs.
Validation: focused suites (test_pat_auth, test_stream_get_action,
test_csrf_middleware) — 62 passed; ruff check + format clean; the new
case errors on the pre-fix baseline (guard absent), confirming the
pin.
* fix(messages): drop legacy <uploaded_files> tag handling (#4212)
PR #4174 unified upload-context injection on <current_uploads> (IM and web
both flow through UploadsMiddleware), and #4632 documented the current
path. This removes the remaining backward-compat parsing of the
pre-#4174 <uploaded_files> tag, the final cleanup item tracked by the
issue:
- deermem: only <current_uploads> is stripped from human turns before
memory persistence, and the upload-sentence scrubber drops the legacy
tag alternative.
- mem0: the mirrored message filter recognises only <current_uploads>.
- InputSanitizationMiddleware: remove the legacy tag from the blocked-tag
denylist (it existed only because deermem parsed the old tag).
- frontend: stripUploadedFilesTag / stripInternalMarkers /
parseUploadedFiles and the message-list fallback parse only
<current_uploads>; demo thread fixtures are migrated to the current tag.
Scope decision: a <uploaded_files> block in pre-#4174 history is now
treated as ordinary user content (pinned by tests in both layers) instead
of being silently dropped or stripped.
* style: apply prettier formatting to stripUploadedFilesTag
* fix(uploads): keep legacy <uploaded_files> stripping for display/export only
Addresses review feedback on #4826: removing the legacy tag from the
frontend display layer made pre-#4174 threads render raw <uploaded_files>
XML (with server-side upload paths) in chat, copy data, and JSON exports.
The backend cleanup stands — memory pipelines and the sanitization denylist
treat only <current_uploads> as an internal marker. The frontend keeps the
legacy spelling in its display/export-only utilities
(stripUploadedFilesTag / INTERNAL_MARKER_TAGS / parseUploadedFiles and the
message-list fallback) so old history renders cleanly without leaking
internal paths, while the memory/sanitization scope-decision tests remain
unchanged.
Frontend tests now pin both spellings: <current_uploads> and legacy
<uploaded_files> are stripped from copy data, markdown leak-stripping, and
JSON exports.
* docs(ui): record accepted display-spoof tradeoff for legacy upload tag
Review note (willem-bd): since <uploaded_files> is off the sanitization
denylist, a live user can type the legacy spelling and fabricate file
chips / hide their own message text in display. Display-only and
self-inflicted with no backend semantics, so it is accepted for now;
documented at both the message-list fallback and stripUploadedFilesTag.
Age-gating the legacy spelling remains a possible follow-up.
---------
Co-authored-by: betterkite <313258397+betterkite@users.noreply.github.com>
* fix(security): sanitize MCP-sourced tool results through the same trust boundary
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(security): sync the trust-boundary docs with tag coverage and pin the untagged branch
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(subagents): clean up background task entry on unexpected poller exit
* fix(subagents): pin deferred cleanup to the persistent subagent loop
The non-terminal fallback scheduled the deferred registry cleanup with
asyncio.create_task on the poller's own loop. Under synchronous tool
invocation the sync wrapper runs the tool coroutine through
asyncio.run(), which cancels caller-loop tasks at teardown, so the
cleanup died before executing and the _background_tasks entry leaked —
the same lifecycle leak the terminal path already fixed.
Schedule the deferred cleaner on the process-owned persistent subagent
loop instead, via the new public executor helper
run_on_isolated_subagent_loop (asyncio.run_coroutine_threadsafe). The
cleaner only touches thread-safe registry helpers, so it is
loop-agnostic. A caller-loop fallback remains for the unreachable case
where the persistent loop cannot be obtained, so scheduling never
raises out of an unwind path that is already handling an error. The
polling-timeout return path, which shares the scheduler, is fixed the
same way.
Tests no longer stub the scheduler: the non-terminal fallback test
drives the real scheduling wrapper on an equivalent long-lived loop and
asserts cleanup runs after asyncio.run() tears the caller loop down,
and run_on_isolated_subagent_loop itself is covered against the real
persistent loop with the caller loop closed underneath.
* fix(subagents): harden interrupted finalization against failing status path
Three edge cases from review on the unexpected-exit unwind:
1. Finalization no longer depends on the failing status accessor.
_peek_subagent_result distinguishes a gone entry from an unreadable
one instead of letting the accessor's exception abort the unwind;
_finalize_interrupted_subagent never raises (so the original poller
exception is preserved) and attaches the deferred cleaner, whose
last resort force-removes a persistently unreadable entry via the
new executor force_cleanup_background_task.
2. The generic-error unwind waits only a short grace period
(_UNEXPECTED_EXIT_GRACE_SECONDS) instead of the full execution
timeout before re-raising; the remaining lifecycle stays with the
deferred cleaner on the persistent subagent loop.
3. The deferred cleaner reports the subagent's final usage (deltas
since the unwind snapshot included, via final=True bypassing
usage_reported; the journal dedupes by source_run_id) before
removing the terminal entry. The report is transferred in a plain
worker thread so the RunJournal's loop-bound progress flush is
skipped rather than scheduled on a foreign loop.
* fix(subagents): pin deferred final usage delivery to the parent run loop
_report_deferred_final_usage ran record_external_llm_usage_records in a
to_thread worker, making it the first cross-thread RunJournal writer: the
unlocked accumulators can lose token updates and _tokens_by_model mutations
race get_completion_data() iteration on the parent loop. Capture the parent
loop at unwind time (it is alive in every path that continues the run) and
deliver the final report onto it with call_soon_threadsafe, serialized with
all other journal access; when that loop is already closed (asyncio.run
teardown) the report is dropped on purpose — the run has persisted and
nothing reads the counters back.
The live-loop test exercises the real recorder path (journal captures the
running loop of every call) instead of stubbing _report_subagent_usage, so a
cross-thread report would surface as a wrong-loop entry. Also gates the two
teardown tests on the caller loop actually closing (the deferred cleaner
could otherwise legitimately deliver while that loop is still winding down),
and adds a task_tool-level regression test for execute_async submit failure
leaving no registry residue (rolled back inside execute_async since #5086).
* docs(subagents): document the reverse loop boundary; observability + test fixes
Extend subagents/AGENTS.md's Isolated-loop callback boundary with the
reverse-direction contract from #5069: deferred registry cleanup is pinned
to the persistent subagent loop via run_on_isolated_subagent_loop, and the
final usage report is handed back onto the parent run's loop captured at
unwind time — never invoked from the persistent loop or a worker thread,
which would silently reintroduce the journal accumulator/iteration race.
Dropping the final report (closed parent loop) is the one path where a
subagent's tail usage goes permanently unaccounted, so both drop branches
now log at info with the execution id and the unaccounted record count.
Restore the retention assert in test_deferred_cleanup_task_retained_and_
survives_gc: the bounded wait observes the production done-callback discard,
the assert (not a manual discard) is what fails if that callback is
deleted.
* fix(subagents): honour grace-wait cancellation and shrink deferred-cleaner captures
Two review follow-ups on the unwind:
- The shared unwind absorbs CancelledError (never-raise contract), so a
graph-node cancellation landing inside the generic-error grace wait was
swallowed and the node ended as a failed tool call instead of an
interrupted run. The generic branch now re-checks task.cancelling() after
the unwind and re-raises CancelledError; the absorb site documents why the
cancellation path needs it and where the discrimination lives.
- The deferred cleaner captured the whole run runtime, pinning the parent
run's journal and event store for up to a full poll budget (~31 min) via
the strongly-held task handle — worst on the polling-timeout path, where a
stuck subagent pinned its run's journal for a second full timeout after
the tool returned. The recorder is now resolved on the unwind path and is
the only capture (plus ids and the report loop); a None recorder skips
reporting entirely.
Both behaviours are regression-tested (red on the previous head, green
after): a cancellation parked inside the grace wait surfaces as
CancelledError, and the runtime is collectable while the cleaner still
polls. The closed-loop drop test now uses a real recorder via
runtime.callbacks so the drop it pins is loop-based, not recorder-absence.
* fix(mcp): reject credentials that cannot travel as HTTP header values
A request-scoped secret or user_auth credential with a trailing newline
(the usual result of reading a token from a file, or a CRLF env-file),
CR/LF, surrounding whitespace, or characters outside Latin-1 sailed
through the credential interceptors into the HTTP client, where httpx/h11
reject it with an exception that echoes the full value:
LocalProtocolError: Illegal header value b'Bearer sk-...\n'
ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage, so the secret landed in the prompt, the checkpoint, and
traces - everywhere headers_from_context promises it never goes.
Add illegal_header_value_reason to mcp/headers.py, mirroring the
transport's own rules (Latin-1 encodable; h11's field_vchar is [^\x00\s]
with SP/HTAB legal only between visible characters), and fail closed in
both interceptors before the value can reach the client. The denial names
only the secret key (plus the reason) and never repeats the value.
Illegal values are denied regardless of on_missing: the key is present,
so a passthrough fallback would silently run the call under the shared
discovery credential - the exact authority confusion the deny default
exists to prevent.
Values the transport accepts are not rejected: embedded SP/HTAB
('Bearer <token>'), Latin-1 high bytes, and DEL all still pass, pinned
by tests against h11's observed behaviour.
* fix(mcp): tighten header value validation to httpx's ASCII boundary
The validator mirrored h11's Latin-1 boundary, but the transport rejects
more than h11 does: build_server_params hands dict[str, str] headers
through the MCP SDK's create_mcp_http_client into httpx.AsyncClient, and
httpx (pinned 0.28.1) encodes str header values as ASCII - so a Latin-1
high byte like 'Bearer caf\xe9' passed validation here only to raise
UnicodeEncodeError inside httpx before h11 ever ran, with the exception
message repeating the offending value.
Validate str values against ASCII instead, flip the tests that pinned
Latin-1 high bytes as transportable, and pin the boundary against the
real client: create_mcp_http_client must reject what the validator
flags and construct cleanly for what it accepts (embedded SP/HTAB and
DEL still pass).
Addresses review feedback on the ASCII vs Latin-1 boundary.
* fix(mcp): validate OAuth and static header values at the same boundary
The validator added for headers_from_context and user_auth left two paths
uncovered. A token endpoint returning an access_token or token_type with a
newline reached httpx/h11, which raise with the full token in the message, and
ToolErrorHandlingMiddleware copies that message into a model-visible
ToolMessage -- the leak this PR set out to close. The operator's static headers
had the same hole.
OAuthTokenManager.get_authorization_header now renders the Authorization value
through one checked helper, so the tool interceptor, the initial discovery
headers and the durable task path are all covered by a single guard. The
rendered value is what gets checked rather than the two fields separately,
because that is what the transport sees: an access_token with leading
whitespace is legal once it follows "Bearer ".
build_server_params applies the same check to statically configured headers.
build_servers_config already isolates a per-server failure, so a bad value
drops that one server and logs the reason instead of the value.
* docs(mcp): correct which transport echoes the full header value
The rationale claimed httpx and h11 both render the full value into their
exception message. Only h11 does, on the line break and surrounding whitespace
cases. httpx's ASCII failure is a UnicodeEncodeError naming the offending
character and its position, not the credential, so at most one character
escapes there; refusing the value up front buys an actionable error rather than
an encode failure raised from inside the client.
Corrected in headers.py and in every copy of the claim: context_headers.py,
user_scoped_auth.py, oauth.py, client.py, mcp/AGENTS.md, docs/MCP_SERVER.md,
the frontend mcp.mdx, and the test comments carrying the same wording. No
behavior change.
---------
Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
* test: isolate Docker bridge gateway fallback from host DNS
Force the fallback-path test to bypass host DNS resolution.
Production bind-host behavior is unchanged.
Refs #5106
* test: document Docker fallback isolation
Explain why the fallback test must replace host DNS resolution.
Refs #5106
* fix(test): exclude blocking I/O suite from make test
Keep make test-blocking-io as the dedicated suite owner.
Add regression coverage for the Makefile contract.
Refs #5088
* test: pin blocking I/O workflow ownership
Document both targets required for full offline validation.
Keep the dedicated workflow and Makefile target under contract coverage.
Refs #5088
* docs(test): align blocking-I/O test guidance
* fix(agents): normalize Command-wrapped tool results
Command-wrapped ToolMessages skipped result metadata and progress
tracking, so error receipts could be recorded as success.
* fix(agents): stamp error meta from subagent_status failures
Delegated task Commands leave ToolMessage.status at success and do not
use an Error: content prefix, so normalize_tool_message was labeling
failed/cancelled/timed_out results as success. Honor structured
subagent_status before content heuristics and cover the four statuses.
* style: ruff-format tool_result_meta tests
---------
Co-authored-by: Yuzhong Zhang <BetterAndBetterII@users.noreply.github.com>
* perf(frontend): cache settled copy-data derivation across streaming chunks
Every SSE values chunk re-renders MessageList, and the re-render re-derived
copy/toolbar text for every settled row: getAssistantTurnCopyData re-ran the
O(turn bytes) content extraction per settled group, and MessageListItem's
toolbar recomputed getMessageCopyData per message. Settled group arrays keep
their identity across chunks (deriveStableMessageGroups), so both derivations
now cache on that stable reference: a WeakMap keyed on the messages array for
turn copy data, and a useMemo on message identity for the toolbar copy text.
Fixes#5094
* fix(frontend): gate row copy-data memo and correct cache win claim
Address review: derive one memoized copy value only when
isHuman || (!isLoading && showCopyButton) and reuse it for both editing
and the toolbar, so settled assistant rows (whose toolbar never renders)
skip the derivation and human rows derive once, not twice; correct the
assistantTurnCopyDataCache comment — the regex/trim split is already
cached per message, the cache's win is the traversal/allocations for
string turns and the uncached O(bytes) map/join/trim for array-content
turns (benchmarked: 5.1x / 17.2x per settled history sweep).
* style(frontend): expand single-line messages array for Prettier
* feat(harness): subagent report contract and delegation acceptance criteria (RFC #4651 PR3)
Layer 1 receipt verification is inert unless subagents actually cite their
execution record. This lands the prompt layer that closes the adoption gap:
- New subagents/report_contract.py owns the model-facing contract text,
derived from the single-owner citation format (format_citation /
receipt_id) so prompts can never drift from the verifier. The executor
injects <report_contract> into every subagent system prompt — built-in
and custom alike — requiring [rN tool_name] citations for action claims,
verifiable handles (absolute path, URL, ID, HTTP status) for
deliverables, and explicit failure reporting; the citation clause
follows verification.receipts_enabled.
- The task tool gains an optional keyword-only acceptance_criteria
parameter, handed to the SubagentExecutor constructor and rendered into
the subagent's SystemMessage (stripped, capped 20 items x 500 chars) —
deliberately never the task HumanMessage, which InputSanitizationMiddleware
classes as genuine user input and would HTML-escape into untrusted-input
framing. The docstring frames subagent results as self-reports, states
the citation cross-check's evidence boundary (resolved = the call
happened, not that the claim is correct), and documents when to attach
criteria with the canonical leaf forms. Deterministic leaf checking
remains a separate layer.
- The lead delegation workflow now instructs reading the ledger citation
line as execution evidence only and spot-checking verifiable handles
before synthesizing.
- report_contract / acceptance_criteria are registered as blocked
framework-authority tags in input sanitization so untrusted input
cannot forge the verification contract.
* fix(harness): neutralize acceptance criteria before system-channel injection
render_acceptance_criteria_section interpolated lead-model-supplied acceptance_criteria verbatim into the subagent SystemMessage after only stripping/capping. A criterion such as '</acceptance_criteria><system>...</system>' could close the wrapper and open a framework authority tag, bypassing InputSanitizationMiddleware.
Route each criterion through neutralize_untrusted_tags (the shared prompt-injection primitive) so blocked authority tags are HTML-escaped before interpolation. Add regression tests at the renderer and the executor _build_initial_state path.
* fix(harness): keep model-supplied criteria off the system channel
- Move acceptance_criteria values into the task HumanMessage — the
untrusted channel InputSanitizationMiddleware escapes and
boundary-frames. The subagent SystemMessage now carries only a
framework-owned <acceptance_criteria> pointer note (no criterion
text), so natural-language injection inside a criterion keeps
task-data priority and cannot override framework instructions
(PR #5090 review, willem-bd P1).
- Condition the lead delegation workflow's citation verification
guidance on verification.receipts_enabled and qualify the task
tool's result-reading text with the enabled state, so a
receipts-disabled configuration no longer tells the lead to
require citation evidence that cannot exist (P2).
* fix(harness): drop execution-record promise from report contract when receipts are disabled
The <report_contract> opening was emitted unconditionally, so a
verification.receipts_enabled=false subagent was told its report would
be cross-checked against an execution record that cannot exist in that
mode (terminal_receipts() returns None; no verdict, no ledger citation
line). The opening now follows receipts_enabled: enabled keeps the
cross-check language, disabled describes the handle-only review mode
(PR #5090 review, willem-bd P2).
* docs: record the prompt-layer trust-boundary self-check
Generalizes the PR #5090 review outcome: before adding prompt text, ask
of every data source in it what trust level it has and which channel it
should ride — model/user-influenceable values ride the untrusted
sanitized data channel, never framework-owned system text. Added to the
PR template (Agents/LangGraph surface) and agents/AGENTS.md.
* fix(mcp): keep session owner teardown safe across cancellation paths
* fix(mcp): gate pooled-session publication on the commit inside the owner task
The owner resolved `ready` as soon as initialize() finished, but the
session only became pool property when the creator promoted it into
_entries in Phase 4. A concurrent get_session() could join the in-flight
creation and receive that session from Phase 2b while the creator was
still parked in the Phase-2 eviction teardown; cancelling the creator
then ran the Phase-2 unwind, which unconditionally shut the owner down —
closing the session underneath the joiner (#5008 review).
Move the commit into the owner task: initialize() success now pops the
in-flight record, registers the session in _entries, and resolves ready
with the session in one atomic critical section, so 'ready resolved with
a result' is exactly 'session registered and pool-owned'. Joiners can
therefore only ever receive a committed session, and both creator unwind
paths (Phase 2 and Phase 3) skip teardown when the creation already
committed, leaving the pooled session to LRU eviction / close_*. When the
record was removed before the commit (close_* or creator unwind), the
owner aborts and ready carries the same cancellation the old Phase-4
not-still-ours path raised, so joiners fail with the creation's outcome
instead of hanging or holding an unmanaged session.
test_cancelled_creator_does_not_close_session_held_by_joiner reproduces
the review's scenario deterministically (MAX_SESSIONS=1, hung LRU victim,
gated initialize, second caller receives the session, creator cancelled):
red on the previous commit, green now.
test_joiner_follows_creation_outcome_when_creator_is_cancelled pins the
joiner outcome-gating semantics as a drift guard.
* feat(e2b-sandbox): make mount upload deadline configurable
Replace the hardcoded 120-second mount upload deadline with a
configurable `mount_upload_deadline_seconds` key read from
SandboxConfig (extra=allow). The value is validated: zero and
negative inputs are clamped to 1 second. Omitting the key
preserves the existing 120-second default.
This addresses the follow-up from PR #4842 review: operators
with large mounts or slow networks can now size the deadline to
their deployment without changing code.
* fix(e2b-sandbox): address review feedback on configurable deadline
- Remove import-time default capture from _mount_deadline_reason()
and _MountUploadBudget.deadline_seconds to prevent silent drift.
- Add warning log when mount_upload_deadline_seconds is clamped to 1
(was silent before).
- Update AGENTS.md E2B Mount Uploads section: deadline is now
configurable, not fixed 120.
- Add mount_upload_deadline_seconds to YAML examples in provider
docstring and __init__.py.
- Add config-path test that exercises SandboxConfig -> _load_config ->
_apply_mounts end-to-end.
* fix(e2b-sandbox-provider): handle non-numeric mount_upload_deadline_seconds
Guard _resolve_mount_upload_deadline against None, non-numeric strings,
and other invalid values. None returns the default; non-numeric strings
like '120s' or 'abc' log a warning and fall back to the 120-second
default instead of crashing provider init with TypeError/ValueError.
Extend the parametrized clamp test with None, suffix, and alpha cases,
and add a warning assertion. Update CONFIGURATION.md with the new
mount_upload_deadline_seconds key and its behavior.
* fix(sandbox): handle infinite mount deadline
* fix(subagents): harden background-task registry and capacity snapshot edge cases
- execute_async drops the just-registered background entry when submitting
to the isolated loop fails. The caller sees the exception and never
polls, and cleanup_background_task refuses non-terminal entries, so the
entry would otherwise stay as a PENDING zombie forever.
- SubagentExecutionCapacity.snapshot derives queued from len(_waiters)
instead of iterating it. snapshot is read from non-loop threads (e.g.
configure_subagent_execution_capacity) while the loop thread mutates the
deque, so iteration can raise 'deque mutated during iteration'. The raw
length may count a waiter that just timed out but has not removed itself
yet, which only makes the busy-check more conservative.
* fix(subagents): close failure-path gaps around background submit
Address both review findings on the background submission lifecycle:
- execute_async() copies the isolated-loop context before registering
the _background_tasks entry, so a context-copy failure (callback-
manager copy or loop-bound handler filtering) can no longer strand a
permanent PENDING entry the caller will never poll.
- _submit_to_isolated_loop_in_context() resolves the loop before
calling the coroutine factory. As direct run_coroutine_threadsafe
arguments the coroutine was created first, so a loop-startup failure
stranded a never-awaited coroutine (RuntimeWarning + retained
captures until collection). Both call sites share the fix.
New tests verified red on the previous implementation, green after:
- context-copy failure leaves no registry residue
- the real submit helper (only the loop getter patched) never invokes
the coroutine factory when loop startup fails
* fix(subagents): close the coroutine when scheduling rejects it
run_coroutine_threadsafe can itself raise once the coroutine exists (e.g.
the loop closes between the lookup and the internal call_soon_threadsafe).
Wrap the call, close the rejected coroutine, and re-raise; a focused test
patches only run_coroutine_threadsafe and asserts the created coroutine
reaches CORO_CLOSED.
