* fix(gateway): scope per-run message and event reads to the run's data identity
start_run stamps run-event rows with the raw trusted owner id, but
GET /runs/{rid}/messages and /runs/{rid}/events called the event store with
user_id=AUTO, which resolves to the internal caller's make_safe_user_id form.
On the database event store, owners whose id contains characters outside
[A-Za-z0-9_-] (IM channel ids such as "feishu:owner-777") therefore got an
empty list. The regenerate source-run scan had the same mismatch and fell
back or returned 409.
Pass _run_scope_user_id() explicitly, as the thread message routes already
do (#5448), and add user_id to RunEventStore.list_messages_by_run() so every
backend accepts it like list_messages()/list_events(). Browser and API
sessions keep their per-user filter.
* fix(gateway): scope workspace-changes and archive reads to the run's data identity
The same raw-owner mismatch reached two more run-scoped routes. On the SQL
stores, GET /runs/{rid}/workspace-changes reported no changes because its
event lookup used user_id=AUTO, and GET /runs/{rid}/artifacts/archive
answered 404 because both its run-row lookup and its run.delivery event
lookup did.
Pass _run_scope_user_id() to both, and let get_workspace_changes_response()
forward an explicit user_id to the event store.
* docs(changelog): reference #6282 in the run-scope read fix entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): keep renewing the lease while a staged terminal run finalizes
With an event store, run_agent stages the terminal status in memory and
commits it to the run store only after the journal flush, delivery receipt,
workspace scan and duration checkpoint. _renew_leases selected runs by local
status, so it stopped renewing as soon as the status was staged. A
finalization longer than lease_seconds + grace_seconds let a peer, or this
worker's own reconciler, claim the still-active row and record the
successful run as an orphan error.
The worker now marks RunRecord.terminal_commit_pending until the deferred
commit is attempted, and lease renewal treats it like the existing
scheduled-goal barrier: keep renewing, release on a confirmed own commit,
and fence when a peer claims the row. A fenced run stops renewing.
* docs(changelog): reference #6263 for the staged terminal lease renewal fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(runtime): add P0 per-LLM-call telemetry and summarization no-op counter
Measurement only; no runtime behavior changes.
- RunJournal enriches llm.ai.response / llm.error metadata with
langchain_run_id, caller_category, provider, model, stop_reason, status,
token counts and rendered request size (chars). Unavailable values are None.
- Summarization middleware emits a middleware:summarize event per summarizer
call and counts outputs identical to the prior summary_text. The call is
never skipped.
- All telemetry fails soft.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(runtime): register summarize middleware tag, document telemetry contract
- Add MIDDLEWARE_SUMMARIZE_TAG to the middleware event catalog and the
contract's known_tags; emit the middleware:summarize event with it.
- Document the optional per-call telemetry metadata on llm.ai.response and
llm.error in contracts/run_event_stream_contract.json.
- Measure request size by walking string leaves instead of json.dumps on the
event-loop thread.
- Initialise summarization call/no-op counters in __init__.
- Move RunJournal telemetry docs to the runtime guide.
- Add tests: llm.error without a start, replayed on_llm_end, and a DB-store
metadata round trip.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(runtime): refresh token counters on usage replay
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add host-bound Agent run control
* fix(extensions): preserve internal launches and scope run control
* docs(extensions): use a plugin-local run operation key
* feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3)
Phase 3 / Skills — the second of three resource-type PRs (Models, Skills,
Sandbox). The RBAC provider already maps "skill" → config key "skills"
(rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed.
Layer 1 (assembly-time, mirrors Phase 1B):
- filter_available_skills_by_authorization() in skill_filter.py filters the
skill-name allowlist (set[str] | None) by the provider's "skill" policy.
In lead agent, runs after _available_skill_names() so the catalog
(describe_skill) and SkillActivationMiddleware share one filtered set.
In subagent executor _load_skills(), filters config.skills before disk load.
- filter_resources_by_authorization() in enforcement.py — generic version of
filter_tools_by_authorization for any resource with a name attribute.
Layer 2 (slash-activation):
- No new middleware code needed. SkillActivationMiddleware._resolve_activation
already checks "reference.name not in self._available_skills". Since Layer 1
filters the allowlist, denied skills cannot be slash-activated.
authorization.enabled: false is a complete no-op. available_skills=None (no
agent allowlist) + enabled: resolves all skill names from config and filters.
12 new tests + 243 existing authz/skills tests pass.
* test(subagents): wait on the chained future instead of polling done() in caller-loop teardown test
* feat(authz): enforce action-scoped skill:activate at runtime and reuse the catalog loader for filter candidates
* feat(authz): arm skill:activate on the subagent chain, unify the client skill-surface user, and record the Layer 2 reversal in the design notes
* chore(authz): align the subagent filter's fallback user with the chain's bucket convention
* feat(authz): gate autonomous skill loading on skill:activate, close the bootstrap assembly leak, and reuse one provider per subagent assembly
Review round 4 (willem-bd, CHANGES_REQUESTED):
- [P1] Autonomous load path: describe_skill filters its match set on the
action-scoped skill:activate decision (all-denied is indistinguishable
from "no match"), and ToolErrorHandlingMiddleware stamps skill_context
entries only when the action allows it — a denied SKILL.md read gets a
skill_context_denied marker instead, so durable context, allowed-tools
policy, and autonomous secret bindings never activate a denied skill.
File visibility itself stays Layer 1 (sandbox-projection semantics).
- [P1] Bootstrap assembly: build_middlewares/apply_prompt_template now
receive the authorization-filtered set instead of the raw
_BOOTSTRAP_SKILL_NAMES, and an empty skill_names index is preserved
when deferred discovery is on rather than round-tripping through None
into the legacy full-metadata prompt that re-advertises a denied
skill (deferred-off keeps the legacy rendering; the filtered empty
allowlist blanks it on denial).
- [P2] SubagentExecutor resolves ResolvedSkillAuthorization once per
assembly (_resolve_skill_authorization, memoized per executor) and
shares it across the Layer 1 filter (authorization=), the activation
middleware, and both new gates — uncached provider factories can no
longer hand the layers different instances/policy snapshots.
All gates share one helper (skill_filter.skill_activation_allowed) so
slash, describe, and skill-file-load cannot drift; provider errors keep
the configured fail-closed/fail-open policy. skill_authorization is
threaded through _build_runtime_middlewares into both runtime builders.
Tests: describe gating (deny/fail-closed/fail-open/disabled), read
stamping (denied marker vs entry) + extract_skills skip without warning,
bootstrap three-way param (allow/deny x deferred on/off), lead + subagent
chain wiring for the stamp gate, and the distinct-instance provider
factory repro. Each fix is mutation-verified (reverting it fails its
test). Full A/B suite run: no new failures vs HEAD.
* feat(authz): re-authorize persisted skill_context before reuse and route async paths through aauthorize
Review round 5 (willem-bd):
- [P1] Persisted entries: skill_context survives across runs, so both
consumers now re-check the action-scoped skill:activate decision before
reuse. SkillToolPolicyMiddleware._active_skills_for_paths skips a
denied skill (same skip as a disabled/unallowlisted one; all-denied
keeps the existing fail-closed-to-builtins semantics) and
_in_context_secret_sources binds no secrets for a denied entry.
skill_authorization is threaded into SkillToolPolicyMiddleware on both
chains.
- [P2] Async provider API: skill_activation_allowed_async mirrors the sync
helper through await provider.aauthorize() (same pattern as
authorize_sandbox_execution_async). Async hooks precompute an
activation_decisions map on the event loop (slash target + persisted
entry names, one batch of awaits) and hand it to the worker-thread
handler, which consults the map instead of the sync API:
SkillActivationMiddleware.awrap_model_call (activation + secret
binding), ToolErrorHandlingMiddleware.awrap_tool_call (_amaybe_stamp
awaits the decision before stamping), and SkillToolPolicyMiddleware's
awrap_model_call/awrap_tool_call (entry names + slash source name).
describe_skill becomes a StructuredTool with both a sync function
(invoke -> authorize) and a coroutine (ainvoke -> aauthorize), so async
agent execution never routes a loop-affine provider through the sync
API while sync callers keep their semantics.
Tests: allow-read-then-deny-next-run regressions for secret binding and
allowed-tools (partial deny does not poison, all-denied fails closed to
builtins), and an _AsyncOnlyProvider (sync authorize always fails,
aauthorize works) driving four async-path assertions (activation, stamp
allow/deny, describe, tool policy). All six fixes are mutation-verified;
full A/B suite run against HEAD shows no new failures.
* fix(authz): authorize skill activation by declared Skill.name, not directory basename
ShenAC-SAC's P2: bundled skills may declare a name that differs from their
directory (skills/public/vercel-deploy-claimable declares name: vercel-deploy),
but the runtime gates derived authorization targets from the container path.
- New deerflow/skills/container_registry.py: build_container_path_registry()
+ canonical_skill_name() — the one path-to-Skill.name resolution shared by
the activation, tool-policy, and read-stamping middlewares.
- ToolErrorHandlingMiddleware resolves the read path through the (user-scoped,
via new user_id kwarg threaded from both runtime builders) registry before
the skill:activate decision, sync and async (registry lookup in a worker
thread, aauthorize() on the loop). RBAC allow by the declared name now
activates; allow by the directory name does not.
- SkillToolPolicyMiddleware async hooks canonicalize the policy paths first
(three-phase: thread registry -> loop aauthorize -> thread filter reusing
the registry; the cached per-model-step decision still short-circuits),
so the decision map is keyed by the names _active_skills_for_paths checks —
no more map miss falling back to the sync provider API from the thread.
_entry_names removed: persisted entry["name"] is path-derived by
construction; entries are resolved by path instead.
- SkillActivationMiddleware collects slash names (already canonical) plus
persisted entry *paths*, canonicalized off-loop via
_canonical_names_for_paths().
Regressions (test_skills_authorization.py 45->50): sync stamp target, RBAC
allow declared-name/deny directory-name both directions, async stamp via
_AsyncOnlyProvider, composed async slash -> tool-policy keeps the skill's
tools with zero sync fallback, persisted-entry secret binding survives the
name mismatch. All three mutations (stamp target, policy map key, entry
canonicalization) fail their tests.
* fix(authz): close the sync-fallback class on skill activation paths
willem-bd's two P2s on the canonical-name fix, plus a fourth instance of
the same class found while sweeping the membership:
- SkillToolPolicyMiddleware: a transient prepass registry failure is now
preserved (_REGISTRY_LOAD_FAILED marker) — the worker-side filter applies
its own fail-closed treatment instead of silently retrying storage,
which recovered skills absent from the decision map and fell back to
the synchronous provider API from the thread (fail-open retaining a
denied skill's allowed-tools).
- SkillActivationMiddleware: an empty entry-path list no longer pays a
full skill-tree scan on ordinary authorization-enabled async model
steps (previously a passive path with zero registry I/O).
- 4th instance (reproduced with the PR fixtures, then fixed): the entry
prepass failure on the secret-binding path let the consumer-side
_load_skill_registry_by_path silently recover and resolve entry names
outside the decision map — the same sync fallback, fail-open binding
the aauthorize-denied skill's declared secret. The fix goes one step
further than failure propagation: the prepass hands down the registry
*snapshot* the decision map was keyed by, and entry sources resolve
against that snapshot, so a map miss is impossible by construction
(this also closes the narrower window where both loads succeed but
disagree because storage changed mid-step). The slash source is
deliberately unaffected: it does not consult the decision map
(validated once at activation) and keeps its fresh per-call load.
- Class membership swept: skill_activation_allowed() sync call sites are
exactly four (stamping sync path and describe's sync implementation are
legitimately sync); the only two async-reachable fallback points (the
_activation_allowed helpers) are both construction-guarded now.
Structural hardening (typed async-batch decision object where a map
miss is fail-closed by construction) recorded in the plan notes as a
follow-up, anchored at those two helpers.
- _RegistryArg type alias replaces the degenerate dict|object|None union
in the policy middleware signatures.
Tests (test_skills_authorization.py 50->54): transient prepass failure ->
builtins-only with exactly one storage attempt and zero provider calls;
ordinary async step -> zero registry scans; two-call scenario -> entry
binding zeroed while the slash binding survives; entries resolve against
the prepass snapshot under a mid-step rename. Mutations verified red:
marker reverted to None, over-broad slash kill, snapshot bypass.
* fix(authz): correct secret-binding reference points and gate the rendered skill reminder
Review round 9 (willem-bd) plus construction-audit follow-ups on the same
class — every async divergence between two loads of the same data now has
an explicit, pinned reference point:
- Slash secret source resolves from a FRESH post-activation registry: the
prepass snapshot is taken before the aauthorize awaits, so binding the
slash source from it would activate NEW_KEY content while injecting
OLD_KEY when a skill's declared secrets change in that window. The
'one scan per step' bonus that motivated sharing the snapshot is
withdrawn — consistency wins. The two secret sources resolve
independently: a transient fresh-load failure zeroes only the slash
binding; entries keep binding from their own snapshot.
- Same-skill dominance: when a skill is both the slash source (fresh) and
a persisted entry (snapshot), only the activation-era declaration binds
— unioning both eras injected a secret the skill no longer declares.
Aligns with the tool-policy rule (slash activation dominates for the
run); different-skill entries still stack additively.
- The rendered 'Active skills' reminder (DurableContextMiddleware) no
longer advertises skills the provider denies — surface consistency with
describe_skill. Publication/consumption split: the activation
middleware publishes per-step path-keyed skill:activate decisions into
the run context (async: reused from the prepass snapshot, zero extra
I/O; sync: one scan + authorize() only when entries exist — passive
steps stay scan-free), the renderer consumes them and filters its
render copy (state untouched; unpublished = permissive, matching the
other run-context carriers). New carrier key listed in
REDACTED_CONTEXT_KEYS with a redaction regression.
- Wiring regressions on both chains for everything added above (user_id
to the stamp gate, skill_authorization + token pairing + activation-
before-durable ordering to the durable renderer): each mutation
verified red (M4-M13 across this change set).
- Tests 54->65: slash post-activation lookup, source independence,
dominance, sync-chain sync-API path, render filter (async/sync/
disabled with load-count assertions), chain wiring x2, redaction
listing. Plan notes record rounds 9-2..6 including the reference-point
post-mortem and the audit axes exhausted.
* fix(authz): anchor same-skill dominance on the slash identity, not binding results
Review round 10 (willem-bd) plus a self-audit extension the new checklist
items caught in the first fix itself:
- R10 (P2): the same-skill exclusion was derived from successfully BOUND
slash sources. When the prepass snapshots required-secrets [OLD_KEY]
and the operator strips the declaration during the await window, the
slash source binds nothing (required_secrets gate) — the exclusion set
stayed empty and the stale entry view kept injecting OLD_KEY for a
skill that now declares no secrets. The exclusion now derives from the
AUTHENTICATED slash activation identity (run-context source path,
owner-token checked): the identity's declared name resolves from the
fresh registry first, then the entry snapshot, and joins the exclusion
set regardless of binding success.
- Self-audit (value->gate->downstream on the fix itself): the declared
name can also change mid-run (same path, foo->bar). A name-anchored
exclusion then holds only the new name while the snapshot entry
resolves the same path to the old one — both eras bound again. The
PATH is the stable identity shared by the activation and every earlier
read: entries at the slash identity's path are now excluded before
registry resolution (exclude_paths), independent of what either
version calls the skill; the name-based set remains for cross-path
same-name shadowing.
Tests 65->67: no-secrets-era dominance (ACTIVE secrets None under the
OLD_KEY-to-empty interleaving), path-anchored dominance across a mid-run
rename (only the activation-era key binds). Mutations verified red:
identity-derived exclusion disabled (M14), path exclusion disabled
(M16).
* test(authz): pass explicit config to lead middleware wiring test
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agent): drop script guidance from the lead prompt when no bash tool is bound
The workspace section told the agent how to write scripts and commands
("prefer relative paths", "avoid hardcoding /mnt/user-data/... inside
generated scripts") even when no tool could run them. With the default
LocalSandboxProvider host bash is off and the bash tool is removed, so
agents wrote helper scripts nothing could execute, then stalled or
computed by hand.
apply_prompt_template() takes bash_available (default True, so other
callers keep today's text). The lead agent (both assembly branches) and
the embedded client pass whether their authorized tools, bound or
deferred, include bash. Without it:
- the two script bullets become one line: no bash tool is bound, work
out results directly and write them with write_file;
- the subagent section no longer shows direct bash examples, and marks
the bash subagent unavailable, since it inherits the lead's tool groups;
- the ACP hint no longer suggests `bash cp`.
* fix(goal): show the goal evaluator the user's answer to a Human Input Card
The web UI sends a Human Input Card answer as a hidden human message
(hide_from_ui with a human_input_response) and shows it on the card.
format_visible_conversation() skipped every hidden message, so the
evaluator saw the question but never the answer. Following the user's
answer then read as the assistant guessing, and the goal stood down with
needs_user_input although the user had answered.
A hidden human message whose human_input_response passes
read_human_input_response() now adds a line
"User (Human Input Card answer): <value>" from the structured value; the
question is already on the card's own line. Other hidden messages, such
as goal continuations, stay out. Over the 12,000-char cap the latest
answer the cut drops is kept at the top next to the latest request, and
is never taken for the request itself. Without card answers the cut is
unchanged.
* fix(agent): keep the subagent examples and bash hint in line with the bound tools
Review follow-up for the lead prompt without bash.
- The last delegation example follows the tools. With a bash subagent on
offer it is unchanged. A lead with bash and no bash subagent keeps
"Run a routine test, build, or git command directly." without the Bash
subagent sentence. A lead without bash gets a file-work example instead.
- The unavailable-bash description no longer points at the sandbox
provider. Since #2253 it renders only when the bash subagent is
registered and the lead binds no bash, where the sandbox is not the cause.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(goal): record a failed evaluator call as a stand-down
When the goal evaluator call raised (a provider error, or an answer that
was not valid JSON), _prepare_goal_continuation_input() logged a warning
and returned without writing anything. The goal stayed active with no
last_evaluation, or with an earlier run's verdict and run id, unlike the
other paths that stop the goal loop.
Record it like the no_durable_end_of_turn stand-down: blocker run_failed,
stand_down_reason evaluator_failed, the current run id, and a reason that
names only the exception type. Nothing is recorded if the run was aborted
meanwhile. A failure before the evaluator is called keeps its behavior and
now logs "Could not prepare goal evaluation".
* docs(gateway): fit the evaluator-failure note in the AGENTS.md budget
Main grew backend/app/gateway/AGENTS.md to 49,006 bytes, and the longer
note pushed the merged file past the 49,152-byte hard limit that
tests/test_agent_guidance_check.py enforces. Say it in one clause.
* docs(gateway): keep main's AGENTS.md line after the merge
The merge from main kept this branch's version of the goal paragraph,
which restored the Human Input Card wording #6068 replaced and put the
file at 49,194 bytes, over the 49,152-byte limit the agent-guidance
check enforces. Take main's line. It already says every non-satisfied
evaluation is persisted with last_evaluation, which this fix makes true
for a failed evaluator call.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(goal): show the goal evaluator the tool calls and results
The evaluator read only the text of user and assistant messages and was told
to fail closed without explicit evidence, so it stood down with
missing_evidence on most completed file tasks: the tool calls and tool
results that the web UI shows never reached it.
Include the assistant's tool calls and the tools' results inside the
existing message window, shortened (200 characters per argument value, 600
per step, line breaks kept), and tell the evaluator to treat tool results as
data, that a successful tool call alone does not meet a goal, and that a
guessed input means needs_user_input. The no-progress and final-clear
signatures are unchanged.
* fix(goal): tighten the evaluator's tool evidence after review
- Pair each tool result with the latest call of its id before it. Some
providers reuse ids such as call_0 across turns, and the window-wide
map labelled an earlier read with a later write's name.
- Keep model-supplied tool names on one line in calls and result labels,
and escape U+0085, U+2028 and U+2029 in arguments and results, which
json.dumps(ensure_ascii=False) leaves raw, so no tool step can start a
new evidence line.
- Leave out the results of a hidden assistant message's calls. The web UI
renders a result only under its call, so neither shows there either.
A clarification prompt stays: the UI shows it as its own card.
- Over the 12,000-char cap, keep whole lines from the end instead of
slicing mid-line. Keep the latest user message that the cut drops at
the top, whole when it and the lines after it fit, else shortened to
2,000 chars. If the next assistant message back does not fit whole,
keep its end in the room left. A marker counts the lines left out.
- Docs: the Gateway AGENTS.md and a worker docstring no longer say the
evaluator never sees tool messages.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(events): read a non-filename run ID as an unknown run on JSONL
The run-scoped routes (events, messages, workspace-changes) pass the URL's
run_id straight to the event store. With run_events.backend: jsonl, an ID
that cannot be a filename (run.1, a%20b, %2e%2e) raised ValueError and
surfaced as a 500, while the memory and database stores return an empty
result.
JSONL reads and deletes now treat such an ID as an unknown run through one
helper, _existing_run_file(), which returns before building any path.
Writes still reject it.
* docs(changelog): reference #6070 for the JSONL run ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): reject SQLite file: URIs instead of opening them as filenames
resolve_sqlite_conn_str passed a file: connection string through
unchanged, but LangGraph's SQLite checkpointer and Store factories
(SqliteSaver, SqliteStore, AsyncSqliteSaver, AsyncSqliteStore) connect
without uri=True, so SQLite treated the whole URI as a literal filename:
file:checkpoints.db?mode=rwc created a file with that exact name in the
working directory, file::memory:?cache=shared persisted to disk, and a
file:/// URI failed to open. The readiness probe did open URIs with
uri=True, so it checked a different file than the runtime used and
reported in-memory URIs as not_configured.
Reject file: URIs in resolve_sqlite_conn_str, the one helper all four
factories go through, with an error that names
checkpointer.connection_string. The probe drops its URI parsing and
reports a rejected connection string as unreachable.
* docs(changelog): reference #6069
* test(persistence): pin the case-sensitive file: URI check
_ensure_postgres_schema_with_pool runs inside the drained pool context but
acquired its connection with a raw 'async with pool.connection()'. Caller
cancellation arriving while the connection's __aexit__ is blocked interrupts
that exit, so the connection is never returned to the pool and the pool's own
drained teardown closes with a checked-out connection — the backend-ownership
invariant documented in runtime/AGENTS.md (same class as the provider-context
fix that introduced drained_async_context).
Use drained_async_context for the schema-setup connection so the connection
returns to the pool before the pool context unwinds.
* chore(checkpoint): bump to checkpoint 4.2.0 and drop the stale saver patch
langgraph-checkpoint 4.2.0 fixes the write dropped after a full -> delta
migration (upstream langchain-ai/langgraph#8526) and keeps its own
InMemorySaver override, so the compatibility patch in checkpoint_patches.py
could neither detect the fix nor stand down: it read langgraph's version
while InMemorySaver ships in langgraph-checkpoint, and it would have kept
shadowing the upstream implementation on 4.2.0. The postgres release 3.1.2
locates plain-value delta seeds (upstream #8535).
- floor langgraph-checkpoint>=4.2.0,<5.0 in the harness and raise the postgres
extra to >=3.1.2,<3.2; langgraph's own >=4.1.0 would otherwise let a
resolver reintroduce the bug now that the patch is gone
- delete the patch, its version guard, and the three implementation-assertion
tests; test_full_to_delta_migration_replays_on_same_thread stays as the gate
and fails on 4.1.1 without the patch (verified by a one-off unpatch run)
- langgraph stays 1.2.9 and langgraph-checkpoint-sqlite stays 3.1.1
Verification: make lint; delta checkpointer/cache/worker-resume suites green
with the migration gate passing on memory and sqlite (postgres params skip
without TEST_POSTGRES_URI). The full make test run before the packaging-pin
fix below was 13 failed / 18523 passed; 12 of those failures reproduce on main
with the same environment (git stash baseline, identical failure sets), 7 of
them because the local config.yaml runs checkpoint_channel_mode: delta. The
remaining one, the postgres-extra string pin in tests/test_checkpointer.py, is
updated here.
Upstream status for the surrounding gaps: cache parity across off/cold/warm
states, postgres pagination (langchain-ai/langgraph#8448) and parallel-superstep
replay order (#8382) are still open upstream, as is the abandoned-branch fork
fix (#8548) that the delta resume linearization in runtime/runs/worker.py works
around, so that path stays.
* docs(changelog): keep checkpoint upgrade in Unreleased
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): share one change position across an atomic thread operation
MemoryRunStore.create_thread_operation_atomic marked each row it touched
separately, so one interrupt-and-replace consumed two positions in the
(change_seq, run_id) cursor that list_changed consumers page with. The SQL
store allocates a single position for the whole set, and runtime/AGENTS.md
documents that as the contract for an atomic thread operation.
The memory store is the backend an install runs whenever no durable database
is configured, so a cursor reader could observe the interrupted row and its
replacement under two positions instead of one, ordered by run_id.
Allocate one position after the raise-only scan -- so a rejected operation
still consumes nothing -- and reuse it for the claimed rows and the new row.
* test(runtime): pin that a rejected thread operation consumes no position
create_thread_operation_atomic allocates its change position only after the
raise-only candidate scan, so a ConflictError consumes nothing -- the memory
store's counterpart of the SQL store rolling back and leaving its clock
untouched. Nothing asserted that ordering.
A consumed-but-unused position leaves no trace in the rows themselves, so a
test comparing list_changed output passes even with the allocation hoisted
above the scan. Assert instead on the position the next accepted operation
lands on: that surfaces a hoist as a gap.
* feat(mcp): re-scope to MCP task claim lifecycle only
Keep PR #4966 a small, closed MCP lease/cancellation state-machine change and
move RunJournal and Run lifecycle work into dedicated follow-ups. This branch
contains only the MCP task claim lifecycle:
- mcp task release/snapshot fencing by owner + per-claim lease token
- phase-level single-flight poll/cancel/notification owners with retained handoff
- routine cancellation no longer persisted as a task failure diagnostic
- bounded ordinary release ownership retention past the drain deadline
- 0018_mcp_task_lease_tokens migration + migration/bootstrap head assertions
- wait_for_task_until helper (MCP uses it); worker-specific capture helper moved
to the run-finalization follow-up
RunJournal (journal.py + test_run_journal.py) and run lifecycle
(manager/worker/store/run sql + run tests) are preserved on
backup/cancellation-safety-full and will be raised as separate follow-ups.
* fix(mcp): unblock claims after ambiguous handoff resolves
A phase-level single-flight owner only guards an ambiguous claim outcome. Once
the claim resolves, the phase owner is released immediately; the handoff may
continue releasing returned rows as bounded, service-owned background work
(transferred to _compensation_tasks on timeout). Per-claim token fencing rejects
a late release against a newer claim generation, so a stuck release no longer
locks the whole phase until process restart.
- README: drop the stale progress-snapshot sentence from the bounded ordinary
release description.
- service: pop the identity-checked phase owner as soon as the claim outcome is
known, then release returned rows with the bounded path; carry the release in
_compensation_tasks if it exceeds the drain deadline.
- mcp/AGENTS.md: document that only an unresolved claim outcome (not the handoff)
blocks later phase scans, and that returned-row releases may continue in the
background once the owner is released.
- tests: pin that the phase owner is released before a stuck release finishes
while the release stays service strong-owned.
* refactor(mcp): remove unused single-record claim wrappers
_poll_one, _cancel_one, and _notify_one are unreachable in production: the
worker always processes claimed records through _run_claimed_batch, so these
wrappers preserved a second, dead single-record lifecycle (state is None)
whose only observable behavior was a wrapper-specific cancellation release.
Remove the three wrappers and migrate the regressions that guarded their
cancel/release invariants to exercise the production _run_claimed_batch path
(operation=_*_one_claimed, release=_release_*_after_cancellation). The single
wrapper-only "state is None" contract (test_poll_release_hang_without_batch)
is deleted; all 11 remaining invariants (CancelledError preservation, repeated
cancellation, poll-only token-fenced lease release, notification claimed vs
dispatched phase release, hung compensation -> service ownership, and
background compensation exactly-once observation) are now covered through the
real batch lifecycle.
* fix(mcp): fence claim-owned mutations against stale generations
The per-claim token check in the ORM release/apply paths was only in the
SELECT; the final write went out by primary key. On SQLite (where
with_for_update() is a no-op) a mutation from an older claim generation
could therefore clear a claim that a newer generation had reclaimed after lease
expiry — the exact distributed lease-fencing failure the per-claim token was
meant to prevent.
Make every claim-owned mutation a single atomic conditional UPDATE with the
owner and per-claim token in the WHERE clause (rowcount 0 => stale, return
False, no mutation):
- release_claim: atomic fence; record the poll-failure event after the fence
wins (same transaction, holding the write lock).
- apply_snapshot / apply_cancel_snapshot: atomic fence; record the event after.
- finish_notification_run: atomic fence; use a CASE on event_version >>
dispatch_version to keep a newer event pending for redelivery instead of
swallowing it as delivered.
Add one regression per path: a stale generation's release/apply/finish after a
same-worker reclaim is rejected and never clears the newer claim.
* test(mcp): pin the migration chain head to the lease-token revision
0026_mcp_task_lease_tokens becomes the alembic head, so the chain-head pin in the 0025 repair test had to move on. Follow the 0023 precedent there (single head plus expected predecessor) instead of pinning a literal head, and give the new revision its own migration test, which owns the pin and covers the nullable claim-token columns on upgrade and their removal on downgrade.
* refactor(mcp): close cancellation cleanup leftovers
* fix(mcp): retain cancelled release diagnostics
* test(mcp): remove obsolete settled compensation case
* test(mcp): cover interleaved lease reclaim races
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(events): serialize DB deletion with thread writers
* fix(runs): delete thread history without dropping reservations
* fix(feedback): support owner-scoped thread cleanup
* fix(threads): clean persisted records on deletion
* fix(threads): correct the feedback cleanup rationale
* test(runs): drop the wall-clock probe from the in-flight delete test
* docs: record the thread-delete and event-store fence contracts
* fix(threads): preserve legacy event-store delete compatibility
* fix(gateway): preserve owner isolation when thread metadata is missing
Follow-up to the #5448 review P1 (post-merge finding): owner_check=True
also authorizes threads whose meta row is missing (legacy compatibility)
or NULL-owner (shared/pre-auth data). _run_scope_user_id returned None
for every trusted internal caller, which dropped the only remaining
per-user filter on those threads and let an internal caller acting for
owner A read owner B's persisted runs.
_run_scope_user_id now takes the thread_id and consults the thread meta
store: when an existing meta row establishes ownership, the authorized
thread's runs are still read unfiltered (merged #5448 semantics,
including owner-header-less internal callers); when the meta row is
missing or NULL-owner, the filter falls back to the acting owner's raw
stamp (the exact value start_run writes) — or the synthetic "default"
identity without an owner header — so cross-user runs stay hidden.
Isolation coverage uses the real MemoryThreadMetaStore with no metadata
row (and a NULL-owner row) plus another user's persisted run: /runs and
/runs/page must be empty and /runs/{run_id} must 404 for internal
callers, while an established-ownership thread keeps the unfiltered
read.
* fix(gateway): gate run-scoped sub-resource reads for internal callers
Review follow-up on #5484: the P1 owner-isolation class remained
reachable through run-scoped sibling reads that apply no per-user filter
at all — /runs/{run_id}/messages, /events, /join, /stream and
/workspace-changes query by (thread_id, run_id) directly, so on
missing/NULL-owner threads an internal caller acting for owner A could
still read owner B's run content by id (verified 200 at the previous
head).
- Extract _thread_ownership_established (shared meta-row check) and add
_require_run_visible_to_scope: for internal callers on threads without
established ownership, the run's own user_id stamp must match the
acting owner's raw value (or the legacy "default" stamp) or the read
404s. Established-ownership threads and every non-internal caller keep
their existing thread-scoped semantics.
- Wire the gate into join, stream, messages, events and
workspace-changes; reword the now-stale messages comment to track the
new scoping semantics.
Regression tests: sub-resource reads 404 for a mismatched internal
owner while the matching owner reads them normally, and the owner-less
fallback branch (synthetic "default" filter on missing-meta threads) is
pinned. Red confirmed against the pre-gate head.
* fix(gateway): gate cancel and artifact archive for internal callers
Review follow-up on #5484 round 2: POST /cancel resolved runs unscoped
(require_existing=True only closes the missing-meta case — NULL-owner
meta rows still pass), so an internal caller acting for a different
owner could interrupt another owner's active run on a shared thread
while /join and /stream were already gated. The archive manifest and
download pair likewise leaked the other owner's delivered-file count
and a 200-vs-409 delivery oracle on NULL-owner threads (missing-meta
threads were already denied by require_existing=True).
All three routes now call _require_run_visible_to_scope; its docstring
records the extended coverage. NULL-owner-thread regression tests pin:
a mismatched internal owner gets 404 from cancel, manifest and archive
download, while the acting owner reaches the real conflict path (409 on
a terminal run) and reads the manifest (file_count 2).
* fix(gateway): tolerate state-less request stand-ins in the scope helpers
The new owner-isolation gate and _run_scope_user_id read request.state
directly, which crashed the FakeRequest-based unit suites for the run
events, workspace-changes and scope endpoints (backend-unit-tests shards
1/2/4 on #5484). Read the state object defensively first: a request
without state is simply not an internal caller, so those paths keep
their pre-gate semantics.
* fix(gateway): scope the thread token-usage aggregate by owner
Review follow-up on #5484 round 4: GET /{thread_id}/token-usage called
aggregate_tokens_by_thread(thread_id) with no user filter at all, so on
missing/NULL-owner threads an internal caller acting for owner A read
owner B's spend, model names, run count and (with include_active=true)
live activity; the NULL-owner variant reached browser sessions too.
build_context_usage's latest-model lookup was unfiltered as well.
aggregate_tokens_by_thread gains an optional user_id (mirroring
list_by_thread: explicit None = unfiltered, AUTO resolves the contextvar)
in the memory store, the SQL repository and the store base;
build_context_usage/_resolve_thread_model_name thread the scope through
the latest-run lookup; the token-usage endpoint passes
_run_scope_user_id's value. Established-ownership threads aggregate
unfiltered as before; shared/missing-meta threads narrow to the acting
identity. Stale helper-test comment reworded after the #5482 merge
adaptation.
* test(gateway): pin the unfiltered aggregate on established-ownership threads
Review follow-up on #5484 round 5: the established-ownership branch of
the token-usage scoping (store receives user_id=None) was the only
unpinned half of the contract — the round-4 call-assertions never set
app.state.thread_store, so their None came from the user-less stand-in
path. test_token_usage_unfiltered_on_established_ownership_for_
internal_callers seeds an established meta row plus runs stamped by two
different identities and asserts the totals fold (166 = 111 + 55);
together with the isolation tests it now catches both failure modes
(always-stamp narrowing and always-None leak).
* fix(runtime): keep store-less run history through cleanup
run_agent schedules cleanup for every terminal run, but cleanup evicted
the in-memory record unconditionally. With a durable RunStore that is
safe: the store fallback in get()/list_by_thread() still serves the run.
Without one there is no fallback, so an embedded consumer that builds
RunManager() with its default store=None lost completed runs from
history entirely.
Gate eviction on a backing store, matching the retain-forever behaviour
documented for memory-only mode, and cover it: the two index/cleanup
tests now use a store, plus a regression asserting a store-less cleanup
keeps the record readable.
* test(runtime): terminalize the store-backed cleanup record first
Mirror the production sequence — run_agent only schedules cleanup once
the run is terminal and its store row is finalized — and pin that a
terminal store row hydrates with its status intact.
* fix(goal): wait for the user when a turn ends on an unanswered question
ask_clarification and the sandbox network prompt put their question in a
ToolMessage and end the graph. The goal evaluator only reads human and AI text,
so it never saw the question, judged the goal not met, and the worker queued a
hidden continuation telling the agent to keep going while the card was still
open. The agent could then act on a guess before the user answered.
Stand the goal down with blocker needs_user_input, without calling the
evaluator, when the trailing tool results include a human input request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(goal): cover resuming after answered clarification
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): expose incremental run evidence reader
* fix(extensions): address run evidence review feedback
* docs(extensions): clarify run deletion reconciliation
* docs(migrations): align current head documentation
* fix(extensions): isolate run evidence event reads
* test: avoid pinning run change migration to latest head
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
* fix(goal): stand the goal down once the run has hit its token budget
Since #5410 goal continuations share the run's token budget, so a
continuation queued after the budget's hard stop only spends one more
model call before its tool calls are stripped. Pass the run's stop_reason
into the goal loop and stand the goal down with "token_capped" in that
case. The evaluator still runs first, so a goal the capped run satisfied
is cleared as before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(goal): list the token-budget stop among goal-loop preconditions
Also pin that a satisfied goal is cleared, not stood down, when the run
hit its token budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(worker): don't fail the parent run on a subagent error carried by task_running
The worker scans root frames for the LLM error-fallback marker to decide the
parent run's status, and custom frames are root frames. task_running events
carry each delegated subagent message, including a subagent's marked error
fallback, so a subagent that failed after retries marked the whole parent run
as error (and an edit-and-rerun rolled back) even though the executor already
reported task_failed and the lead answered. Skip custom frames in both the
single-mode and multi-mode stream paths.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(worker): cover the single-mode stream loop for task_running fallbacks
Run the task_running regression test with stream_modes=["custom"] too,
which takes the worker's single-mode loop. Without the single-mode
guard only that case fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(runtime): resolve omitted run owners before idempotent reuse
HTTP run admissions do not pass a user_id. The SQL run store stamps the
request user from the contextvar onto the row, but RunManager kept None
on its process-local RunRecord, so the two disagreed about who owns the
run. A keyed retry that reached a peer worker, or the owning worker after
cleanup() released its local record, hydrated the stamped row, compared
its owner with None, and raised "Run idempotency key resolved to a
different thread or user", which start_run surfaced as a 500.
MemoryRunStore stored None on both sides and never hit the check.
The same mismatch hid HTTP runs from owner-scoped history reads, which
filter local records by the current user, and skipped the worker's MCP
background_tasks projection, which only runs for records with an owner.
create() and _admit_thread_operation() now resolve an omitted owner from
the current user before building the record, and keep None when no user
is in context instead of falling back to the default bucket, so the
local record and every store agree on the owner. HTTP runs now receive
the background_tasks projection, so the replay golden's values frames
gain that key.
* docs(changelog): reference #5401 in the keyed retry owner fix entry
* test(runtime): close the SQL engine when peer-reuse test setup fails
Review follow-up on #5401: the sql case of the two-worker start_run test
initialized the engine above the try whose finally calls close_engine().
init_engine() assigns the module-global engine and session factory before
bootstrapping the schema, so a failure there skipped the teardown and left
a stale engine for later tests in the same process.
Store setup and the RunManager workers now live inside the try, so the
teardown runs whether setup or the test body fails.
* fix(runtime): stop idempotent reuse from blocking the thread on a peer worker
When an idempotent run admission lands on a worker that does not own the
run, RunManager hydrates the stored row and returns it as the reused
record. It also registered that row in the worker's local run map, but
only the owning worker's task lifecycle finalizes and cleans up local
records, so the copy kept its admission-time pending/running status for
the life of the process. On that worker every later reject-strategy
admission for the thread returned 409 until a restart, run reads kept
reporting the stale status, orphan reconciliation skipped the run as
locally live if the owner crashed, and a cancel took the local-owner
path and marked the owner's still-running row interrupted.
Return the hydrated row as a detached store-only handle instead of
registering it. A local record for the key is already returned before
the store insert, so the removed lookup of an existing local record was
unreachable. get(), cancel() and reconciliation now read the durable row
on the peer, matching the documented non-owner contract.
* test(runtime): pin keyed retries of a terminal reused run on the SQL store
Review follow-up on #5393: the post-cleanup release relied on a keyed
retry resolving through the terminal row's idempotency conflict, but
only MemoryRunStore pinned that path, and no test retried on the owner
after its local record was cleaned up.
The SQL repository test now retries the key on the peer and on the
owner once the run is terminal and cleaned up, asserting both get the
same run back as a store-only reused handle before the keyless
follow-up is admitted.
* feat: add opt-in task notes and compacted history recall
* fix: validate task continuity state and preserve user answers
Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence.
* fix: align Helm config version with task continuity schema
* fix: preserve mixed task history and declare continuity policies
* fix: recover malformed history and evict archives atomically
* fix(tools): run tool assembly off-loop at async entry points
get_available_tools() may block on MCP cache initialization while it is
called on async agent-assembly paths (task_tool, durable batch execution),
stalling the calling event loop for the full discovery duration.
Dispatch the (unchanged, synchronous) assembly call to a worker thread via
asyncio.to_thread at the two async entry points so the loop keeps processing
requests, SSE frames, cancellations, and timers.
Fixes#5172
* fix(tools): offload lead-agent assembly off-loop and pin with blocking-io anchors
Review follow-up for #5224:
- run_agent now dispatches agent_factory(...) through asyncio.to_thread, so
lead-agent assembly (including both get_available_tools call sites in
_assemble_lead_agent) runs off the event loop — the Gateway headline
scenario from issue #5172.
- _ensure_sync_invocable_tool takes a double-checked threading.Lock, making
the in-place tool.func wrap on the shared tool singletons explicitly
single-shot now that assembly can run concurrently on worker threads.
- Add backend/tests/blocking_io/test_tool_assembly_offloop.py: blocking-probe
anchors for task_tool and SubagentBatchService._execute_item under the
strict Blockbuster gate, plus a meta-check proving the gate trips on the
exact syscall class (ExtensionsConfig.from_file on the loop). Verified the
anchor goes red when the offload is flattened back to a plain call.
* fix(gateway): build checkpoint state accessor off-loop; anchor run_agent offload
Review follow-up for #5224:
- Add abuild_checkpoint_state_accessor (asyncio.to_thread around the
unchanged sync builder) and switch every async call site to it: the
stateless_wait route, thread_runs, both threads call sites, and the
build_thread_checkpoint_state_accessor boundary. The agent-factory
assembly re-enters get_available_tools() and may block on MCP cache
initialization; repeat calls hit _state_accessor_graph_cache and only
pay the thread hop.
- Add a third blocking-io anchor driving the real run_agent with minimal
RunManager/bridge stubs; the factory performs a real production blocking
read (ExtensionsConfig.from_file()) and the test asserts assembly never
runs on the main thread. Verified the anchor goes red when the run_agent
offload is flattened back to a plain call.
- Adapt the test_threads_router checkpoint-builder patch sites to the new
async name.
* refactor(tools): carry assembly offloads on a dedicated bounded pool
Review follow-up for #5224:
- Add utils/assembly_io.py: a dedicated ThreadPoolExecutor (default 8
workers, DEER_FLOW_ASSEMBLY_WORKERS-overridable, mirroring
utils/file_io.py and tools/sync.py) with run_assembly(), which copies
contextvars explicitly. A hung stdio MCP server parks its worker for
the full MCP timeout; carrying assembly hops on the loop's default
executor would let a few parked assemblies queue every other
to_thread/run_in_executor(None, ...) caller behind them.
- Switch all four offloads (run_agent, task_tool, batch _execute_item,
abuild_checkpoint_state_accessor) to run_assembly().
- State the cold-path behavior in the accessor docstring: the graph
cache validates factory identity, so non-identity-stable factories may
duplicate lead-agent assembly across concurrent readers (MCP discovery
stays process-wide single-flight); the pool bounds the duplicates.
- Add a fourth blocking-io anchor driving build_thread_checkpoint_state_
accessor with a per-resolution fresh factory (always a cache miss) and
the real production blocking read; enumerate all four offloads in the
gate's module docstring. Verified the anchor goes red when
abuild_checkpoint_state_accessor is flattened back to a plain call.
* fix(subagents): revalidate batch item before launch; make assembly pool observable
Review follow-up for #5224:
- _execute_item() revalidates the durable state right after assembly and
before executor.execute_async(): renew_item_lease() returns valid=False
when cancel_batch() terminalized the item or the lease was lost while
assembly was parked, and the launch is skipped (the canceller already
finalized the item). Previously the launch was unconditional and the
poll loop's cancellation checks only started after execution began.
- Regression test driving the real SQLite repository: a blocking assembly
probe parks _execute_item, cancel_batch() lands, and the launch is
skipped with the item staying cancelled. Verified the test goes red
when the revalidation is removed.
- run_assembly() tracks pending assemblies and logs a throttled WARNING
once the pending count exceeds the worker count, so assembly starvation
(workers parked on a hung MCP server) is distinguishable from idle.
- The run_agent blocking-io anchor now binds a sentinel extension
snapshot via ctx.extensions and asserts the factory observed it through
get_agent_build_extensions(), pinning run_assembly()'s ContextVar
propagation. Verified red when ctx.run is dropped.
- Document the assembly pool in backend/AGENTS.md.
* fix(utils): decrement the assembly pending count on the pool thread
The pending-assembly counter behind the starvation warning decremented
from the asyncio future's done callback, which never fires once the
submitting loop is closed while its worker is still running: the count
ratcheted up permanently and eventually fired the starvation warning
with no starvation behind it (reproduced at 97dc9bec by review).
Decrement instead from the dispatched work item: run_assembly() wraps
func so a finally drops the count under the pending lock on the pool
thread, and the done callback is gone.
Pin the counter with tests/test_assembly_io.py: a healthy call returns
the count to zero, and an abandoned loop (stopped while the worker is
parked) does not wedge it — the abandoned case goes red against the old
done-callback decrement.
* docs(utils): fix the pending-counter comment after the decrement move
The comment still described the removed done-callback decrement,
contradicting _work()'s own comment; state the actual mechanism
(increment on the loop before dispatch, decrement from the dispatched
work item's finally on a pool thread).
* test(gateway): retarget checkpoint-accessor stubs to the services seam
thread_runs and runs now call abuild_checkpoint_state_accessor, so the
upstream wait-reader, regenerate-prepare, and idempotency tests must stub
the sync builder where abuild resolves it (app.gateway.services); stubbing
the removed router re-exports fails with AttributeError at setup. The
async seam semantics are unchanged: run_assembly invokes the stubbed
sync builder off-loop and propagates its return values and exceptions.
Move the agent/tool assembly off-load note from backend/AGENTS.md to
deerflow/utils/AGENTS.md (next to assembly_io.py) so the effective
instruction chain for agents/middlewares no longer grows past the AG002
hard limit.
* fix(runtime): serialize same-key accessor assembly and release queued-cancel slots
Address the three review follow-ups on the assembly off-load:
- assembly_io: a job cancelled while still queued never runs its work
item, so the dispatched finally never fired and _pending_assemblies
stayed elevated until a false starvation warning. Exactly-once cleanup
now rides the concurrent future's cancelled() state — cancel() only
succeeds before the executor starts the item, so cancelled() is true
precisely when the finally will never run — plus a submit-failure
release; the one-worker queued-cancellation case is pinned red/green.
- services: overlapping cold readers sharing one cache key could both
run full agent assembly. _state_accessor_graph now serializes per key
through a thread-side KeyedLockTable (pool threads, no running loop)
and re-validates factory/app-config identity under the lock, so the
factory runs exactly once while identity changes still rebuild. Cache
dict access is lock-guarded now that construction runs off-loop.
- guidance inventory: register deerflow/utils/AGENTS.md in
EXPECTED_GUIDANCE_PATHS so test_repository_has_the_approved_scoped_
guidance_shape matches the relocated assembly note (CI shard 4).
* test(keyed-lock): pin KeyedLockTable reclamation and waiter bypass directly
Thread-side counterparts of the async table's own tests: overlapping
hold() calls serialize (a late arrival joins the live entry instead of
creating a second lock that bypasses a queued waiter), the last check-in
pops the entry, and many unique keys leave the registry empty. Both
regressions verified red — popping unconditionally trips the late-arrival
test, never reclaiming trips the many-keys test.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>