168 Commits

Author SHA1 Message Date
一片一片
fdb9deab6d
fix(runtime): report a failed scheduled-goal cleanup on the run stream (#8322)
* fix(runtime): report a failed scheduled-goal cleanup on the run stream

The finalizer flips a finished run to `error` when it cannot remove the
occurrence's scheduled goal, but it published no `error` event. The stream
then ended as a success while the record and `/wait` reported the failure:
the web UI raises its run-failure toast and streaming IM channels mark a
turn failed only on that event.

Set the same deferred `terminal_error_frame` the unpersisted delivery
receipt uses, so both finalizer-side terminal errors end the stream the
way the record does.

* fix(runtime): publish a staged error frame only for an error outcome

An interrupt accepted while scheduled-goal cleanup runs keeps the run slot
active. If cleanup then raises, the error is staged, but the cancellation
branch persists `interrupted`; the stale frame made a cancelled run show a
failure toast while the record and `/wait` reported interruption.

Publish `terminal_error_frame` only when the committed outcome is the
error it describes, which also covers a cancellation that wins after the
delivery receipt staged its frame.

The delta cases now set the worker's checkpoint mode, and a new test
cancels the run from inside a failing cleanup.

* fix(runtime): publish a staged error frame only for the committed error

A rollback also commits `RunStatus.error`, so the status check alone still
sent the staged cleanup message while the row and `/wait` said `Rolled
back by user`. Require the frame's message to equal the committed error.

The cancellation test now covers interrupt and rollback over both run
stores and both checkpoint modes; the rollback cases fail without the
message check.
2026-11-18 14:25:53 +08:00
Bas Cancrinus
3751eb8daf
feat(extensions): pass a task's final status, error, goal verdict and run metadata to on_task_stop (#8304)
* feat(extensions): pass a task's final status, error, goal verdict and run metadata to on_task_stop

A lead run's final status is decided after the graph has ended: a model
that fails after retries ends the graph normally, so callback-based
tracing reports the run as OK. on_task_stop runs after the status is
persisted but only received completed | failed | aborted.

Additive, so existing contributors keep working:
- TaskInfo.metadata: a read-only copy of the run's metadata (scheduled
  runs carry scheduled_task_id, scheduled_task_run_id, scheduled_trigger);
  empty for subagents.
- TaskInfo.stop: a new TaskStopDetail (status, error, goal_verdict), set
  only on the info passed to on_task_stop.
- deerflow-extension-api 0.2.13.

Refs #8303

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(subagents): compare stop identity without the stop detail; assert its status

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extensions): keep TaskInfo identity and serialization; flag unconfirmed stop status

Review follow-up for #8304:
- Exclude TaskInfo.stop and .metadata from equality and hashing
  (compare=False), so the stop info finds state keyed by the start info.
- Replace MappingProxyType with ReadOnlyDict, a dict subclass that rejects
  mutation and freezes nested mappings (lists become tuples). asdict(),
  deepcopy, pickle and json.dumps work again; observers sharing one
  snapshot can no longer change nested values for the next observer.
- TaskStopDetail.persisted: False when the run store did not confirm the
  terminal write or the worker lost ownership, so a staged status is not
  reported as final.
- Document waiting_for_input as a resumable pause (outcome failed) and the
  persisted flag (en/zh observers and reference).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nan Gao <thunderggnn@gmail.com>
2026-11-17 19:57:40 +08:00
一片一片
ea1833bd7a
fix(runtime): publish a failed artifact delivery on the run stream (#8252)
* fix(runtime): publish the delivery failure on the run stream

* docs(api): note the delivery-failure stream event

* test(runtime): cover the deferred delivery-failure frame

* docs(runtime): log the delivery fix and explain the fallback exemption

* test(e2e): present the recorded output in the replay scenario

The write/read-file fixture predates the delivery check, so the replayed
run has been ending as an error without presenting note.txt. This PR makes
that failure visible on the stream, which broke the replay golden and the
render e2e. Add the present_files step the current scenario produces and
regenerate the golden.

* docs: drop the CHANGELOG entry

CONTRIBUTING.md asks contributors to leave CHANGELOG.md and CHANGELOG_zh.md
to the maintainers at merge time, so the Unreleased ordering and the
English/Chinese pair stay in sync.
2026-11-16 18:39:06 +08:00
Totoro
b0bca9dcb4
feat(mcp): bind HTTP tool credentials to approved custom agents (#8242)
* feat(mcp): bind agent credentials to approved definitions

* fix(mcp): enforce live auth policy and pin task destinations

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-11-14 22:10:33 +08:00
2SinceJuly
04a9833763
fix(subagents): keep a steered subagent report in the parent snapshot (#8167)
* fix(subagents): keep a steered subagent report in the parent snapshot

A Steer frees the parent's turn before its subagent finishes: wait_tool
returns a 'result is pending' task result and the real report arrives
later as a hidden subagent-result HumanMessage.

ParentContextSnapshot only kept genuine user messages, so it dropped that
report and kept the pending placeholder. A later context_mode=snapshot
delegation therefore inherited a pending result and could not use quotes
the parent had already received.

Keep the hidden subagent-result message as historical conversation data.

* fix(subagents): share the delayed result message name

---------

Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 22:30:48 +08:00
Evan
b4410f64d4
fix(agents): keep the delegation ledger exact across HITL approval and reused call ids (#8128)
* fix(agents): count each delegation once when a provider reuses its call id

Tool-call ids are unique per response, not per conversation. Some providers
number every response's calls from call_0, so two task calls in one run can
share an id.

The delegation ledger used (run_id, tool_call_id) as a delegation's identity:

- extract_delegations built one id -> entry map over the whole run, so a
  later call_0 replaced the earlier delegation's description and took its
  result while it was still running.
- merge_delegations replaced the same (run_id, id), keeping one entry.
- SubagentLimitMiddleware counted distinct ids, so the per-run cap
  (subagents.max_total_per_run) never advanced past the first call_0 and a
  run could launch any number of subagents one response at a time.

Each entry now also records dispatch_message_id, the AIMessage that issued the
call, and results pair through pair_tool_call_results, which answers a call
only from its own response. The reducer, DurableContext's change filter and
the cap use the extended identity. An entry without a dispatch id still
updates its run's entry for that call id, and a dispatch-scoped entry adopts a
ledger entry written before the field existed, so runs in flight across the
upgrade are not counted twice. The steer path names the dispatching AIMessage
from the tool request's state, so its delegation update and the graph's own
capture stay one entry.

* fix(agents): close an earlier run's dangling dispatch by its own response

_close_delegations_left_by_earlier_runs decided "answered" by
(run_id, tool_call_id). When a stopped run had dispatched call_0 twice,
the first reply kept the second, dangling entry in_progress, so the
lead kept hearing "already delegated" for work nothing would finish.

Entries that carry dispatch_message_id are now decided by pairing that
response's calls through pair_tool_call_results. Legacy entries and
dispatches no longer in history keep the id-based inference.

Also pin that two runs reusing call_0 stay two ledger entries, and
cross-reference the three delegation identity helpers.

* fix(agents): settle an approval-gated task delegation in the ledger

With `task` in `hitl_approval.tools`, the paused run records the dispatch
as in_progress and the resume run executes it. The resume run opens with
no HumanMessage, so its capture window starts after the pre-existing
messages and never contains the dispatching AIMessage: the result was
dropped and the entry stayed in_progress with no result_brief, forever.

DurableContext now pairs results the resumed window receives with the
last pre-existing dispatch and settles the issuing run's entry, keeping
its run_id instead of re-tagging it. HITL denials of `task` carry
`cancelled` subagent status, so a rejected or denied delegation settles
too.

* fix(agents): address review on the delegation ledger

- Steer path: hold each wait's dispatch AIMessage locally and pass it to
  the observer, so a later response reusing the call id while the first
  call is still observed cannot take over its args or dispatch id.
- New-turn close: a dispatch whose reply carries subagent status settles
  from that reply instead of staying in_progress, covering a crash
  between the tools checkpoint and the next capture.
- Reuse thread_state.delegation_key instead of a duplicate identity
  helper in DurableContext.
- CHANGELOG: reference #8128.

* docs(changelog): drop the #8128 entry per review
2026-11-12 19:46:58 +08:00
Wenchao An
3d826c6d95
feat(approvals): add Ask, Auto and Full tool permission modes (#8033)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls

* fix(hitl): address approval review findings

* docs(hitl): keep agent guidance under size limit

* fix(hitl): guard approval pause recovery

* docs(hitl): shrink inherited guidance

* fix(hitl): harden approval admission and recovery

* fix(approvals): address review safety and reconciliation gaps

* ci: retry approval checks after runner timeout

---------

Co-authored-by: wangzeren <1004695029@qq.com>
2026-11-11 20:59:05 +08:00
Hyeonsang Cho
8850f8eb7e
fix(runtime): keep assembly evidence from reviving a run a peer reclaimed (#8046)
* fix(runtime): keep assembly evidence from reviving a run a peer reclaimed

In heartbeat mode every run-row write is conditional on this worker's
live lease except assembly evidence, which went through RunStore.put().
That upsert rewrote status, owner, lease, error and stop_reason from the
local record. When a peer had already reclaimed the run, it put the row
back to running and erased the takeover error. If the owner's local lease
was still live, its renewal then succeeded, the peer never reclaimed it
again, and the run finished as success after the peer had published its
terminal recovery event. If the write landed late, the row came back with
an expired lease and was reclaimed a second time.

Add RunStore.merge_owned_metadata(), which merges metadata keys only for
the owner holding a live lease on a pending/running row and writes no
other field. SQL follows the existing clock -> row lock -> conditional
UPDATE order; MemoryRunStore mirrors it. With heartbeat enabled,
record_assembly_evidence() uses it, refuses a locally fenced run before
any write, fences the run when the store rejects the merge, and fails
closed on stores without the method instead of falling back to put().
Single-worker mode keeps the full snapshot.

* docs(changelog): reference #8046 for the assembly evidence takeover fix

* test(runtime): scope the simulated lease expiry to the run's thread

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 07:56:53 +08:00
liyinlong
99e395aa9c
fix(runtime): recover completed thread reservations on admission (#8068)
* fix(runtime): recover completed thread reservations on admission

* Address thread reservation recovery review

* Place reservation recovery note below agent fixes

---------

Co-authored-by: liyinlong <173714821+lilong555@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-10 15:40:34 +08:00
Evan
226d1bc5ce
fix(runtime): persist a rejected HITL call in the resume run's history (#8073)
* fix(runtime): persist a rejected HITL call in the resume run's history

An approval pause and its resolution are two Runs. The paused Run dispatches
the gated call; the resume Run produces its result. An approved call executes
and reaches on_tool_end, but a rejection is written by HITL after_model with no
tool callback. Journal reconciliation only persists such middleware-written
results for calls dispatched by the current Run, so the rejection stayed in
checkpoint state and never reached the run-event history: /messages and the
resume run's messages had no llm.tool.result for it.

The worker already reads the durable interrupt payload to recover the gated
tool names. It now also returns each paused tool_call_id with its name, and the
resume journal adopts those calls, so the existing reconciliation persists the
rejection under the resume Run. Approved results keep their tool-callback event
and are deduplicated by message id; results from other earlier Runs are still
not re-attributed.

* fix(runtime): keep HITL replay names independent of call adoption

Derive the replay-gate tool names and the adopted call ids separately
from the matched pause's actions, so an action without a usable
tool_call_id still gates replay. Pin that the all-reject escape paths
(approval disabled, gated tools changed or cleared while paused) reach
the history feed under the resume run, check resume-run attribution in
the mixed-decision test, and add a CHANGELOG entry.

* refactor(runtime): drop unused _pending_hitl_resume_calls wrapper

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-10 15:11:09 +08:00
2SinceJuly
4e09e63a68
fix(threads): evict in-memory runs when a thread is deleted (#8036)
* fix(threads): evict in-memory runs when a thread is deleted

RunManager.get() and list_by_thread() answer from the process-local _runs
registry before consulting the RunStore. DELETE /api/threads/{id} only
removed the durable rows, so a worker that had admitted the thread's runs
kept serving them: after the delete returned 200 and the thread itself
answered 404, GET /api/threads/{id}/runs still listed the run and
GET /api/threads/{id}/runs/{run_id} still returned 200 with the original
prompt intact in kwargs. The residue disappeared only on restart, which
is what made it look like a cache rather than a leak.

Add RunManager.discard_thread_runs() to drop this worker's copies for the
thread and call it from the delete path, next to the durable run
deletion. Internal thread-operation rows are left alone: the reservation
protecting the deletion is one of them and its own release path owns it.
Historical runs, the thread row, run events and checkpoints are unchanged.

* fix(threads): keep a live finalize tail attached when evicting runs

discard_thread_runs() popped every record it found, but worker.py clears
finalizing before the tail that publishes the terminal frame, closes the
journal and releases the sandbox lease. Popping mid-tail turns the tail's
manager-mediated writes into warnings on a record that no longer exists.

Skip records whose task is still running, mirroring what cleanup() already
does through shield(), and pin the guard with a unit test that seeds both
a finished run and a delete reservation. Document that a DELETE served by
one Gateway bounds, rather than removes, a peer's copies: they survive
until that peer's deferred cleanup(run_id) fires.

* fix(threads): hide a deleted run from reads during its finalize tail

A DELETE that arrives after worker.py clears `finalizing` but before the
worker task finishes cannot drop the record: the tail still publishes the
terminal frame, closes the journal and releases the sandbox lease through it,
so popping turns those writes into `set_status called for unknown run`.
Leaving the record readable instead keeps the run and its prompt reachable
through `GET /threads/{id}/runs` until `cleanup()`'s 300s delay fires, which
is exactly the window #8034 reports.

Records that a deletion reaches mid-tail are now flagged instead of skipped.
`get()`, `list_by_thread()`, `get_many_by_thread()`, the regenerate-source and
edit-replay reads skip a flagged record (and never rebuild it from a store row,
since the durable rows were deleted first), and the tail's own completion
evicts it — no second delete request and no timer.

* fix(threads): keep the tail eviction task referenced until it settles

The done-callback eviction is a separate task because it cannot await the
registry lock inline. The loop holds only weak references to tasks, so an
unreferenced eviction could be collected before it drops the record and the
deleted run would come back within `cleanup()`'s 300s window. Track it in a
manager-owned set and discard on completion.

* fix(threads): keep hidden runs hidden on every read path

The deletion flag set by discard_thread_runs is authoritative for "this run
is deleted", but two reads still treated it as a hint.

get() narrowed its store-less guard to a missing record, so a flagged
record on a store-less RunManager fell through to self._store.get() on
None; the AttributeError was swallowed by the broad except below while
every read in the flag window logged a hydration warning with a full
traceback. The record now settles the read before any store call, which
also keeps the read side-effect free as the comment intends. aget()
shares the path.

get_many_by_thread() rebuilt flagged runs from store rows. list_by_thread
already refuses that, because the durable row is deleted before the flag
is set and anything left is a stale or partial read; the batch read now
applies the same guard so the invariant is uniform.

---------

Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
2026-11-09 20:21:07 +08:00
Zeren Wang
0d04322d41
feat(hitl): add human-in-the-loop approval for high-risk tool calls (#7979)
* feat(hitl): add human-in-the-loop approval for high-risk tool calls

* fix(hitl): address approval review findings

* docs(hitl): keep agent guidance under size limit

* fix(hitl): guard approval pause recovery

* docs(hitl): shrink inherited guidance

* fix(hitl): harden approval admission and recovery
2026-11-09 09:52:35 +08:00
Totoro
3f9a78e02e
fix(goal,artifacts): verify completion and artifact handoff (#7961)
* fix(goal,artifacts): verify completion and artifact handoff

* fix(goal,artifacts): harden shared artifact handoff

* docs(agents): shrink inherited guidance to fit chain budget

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-11-07 18:21:49 +08:00
Totoro
593ba49458
feat(subagents): continue requested batch work and schedule explicit prerequisites (#7839)
* feat(subagents): continue native batches and honor item dependencies

* docs: link native batch follow-up changelog entries

* test(migrations): pin isolated history-seed upgrade to 0050

* fix(subagents): preserve follow-up scope, blockers and fair scheduling

* test(memory): freeze capacity evaluation time

* fix(subagents): guard follow-up admission and retain run controls

* fix(subagents): enforce runs:cancel on tool cancellation of follow-up batches

* style(tests): format test_batch_followup_host with ruff

* fix(subagents): drop the admission checkpoint check and keep SDK cancel compatible

The checkpoint check inside run admission waited on the checkpoint saver's
lock while holding the SQLite write lock, which could make another chat's
checkpoint write fail. Remove it; dispatch still checks the conversation
before admission.

A follow-up waiting for an unanswered question now keeps its claim until the
lease expires instead of rereading the conversation every poll. Cancel passes
the caller's run authority only to submitters that take it, so existing SDK
submitters keep the two-argument call. Document that retrying a prerequisite
by hand does not requeue its blocked items, and list 0052 as the migration head.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-05 15:10:21 +08:00
Wenchao An
b6ed47778d
feat(extensions): add versioned team file handoffs (#7817)
* feat(extensions): add versioned team file handoffs

* fix(extensions): fence team artifact execution and snapshots

* fix(agent-teams): retain resumed keys and bound artifact transfers

* ci: reclaim runner space before sandbox image smoke

* ci: budget sandbox image preparation separately

* fix(extensions): version artifact contracts and document snapshots
2026-11-05 14:50:09 +08:00
golemon
5371343f69
fix(gateway): continue dispatching queued inputs after cancellation (#7746)
* fix(gateway): continue queued inputs after cancellation

* docs(runtime): clarify queued input cancellation handling

* fix(gateway): clarify cancelled-input error and log race-window skips

* docs(changelog): document queued input cancellation fix

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-04 10:39:20 +08:00
Nan Gao
1dd6d05627
fix(extensions): hand compaction observers the live task store and name the task in CompactionEvent (#7831)
* feat(extension-api): add task identity fields to CompactionEvent (0.2.10)

CompactionEvent carried the transform, its hashes and its counts, but
nothing that names the task, run or thread the compaction happened in.
An observer that wants to relate a compaction to the run that produced it
had to match hashes across sessions, which is unreliable.

Append three optional fields, task_id, run_id and thread_id (str | None,
default None). Existing keyword and positional construction is unchanged,
so already-released extensions keep working. The host fills them in the
next commit.

Bump deerflow-extension-api to 0.2.10 and the harness's exact pin with it.

* fix(extensions): hand compaction observers the live task store and its identity

The summarization middleware already recovers the live task store from
the runtime to observe its summary system call, and the registry
allocates that store precisely because a compaction observer is
registered. notify_context_compacted() then discarded it and handed every
observer a detached store, with a docstring claiming there was no live
task.

- The lead run worker and the subagent executor now seed every task store
  they allocate with the task's TaskInfo, not only when a lifecycle
  contributor is registered; the lifecycle notification reuses the seeded
  object. A subagent without a parent run_id still gets no TaskInfo.
- compact_state() and acompact_state() pass the store from
  task_store_from_runtime() to _record_compaction().
- notify_context_compacted() takes an optional task_store. A live store
  goes to the observers and fills the event's task_id/run_id/thread_id
  from its TaskInfo; without one (manual /compact runs outside any run)
  observers still receive a detached store and the ids stay None.

For a background subagent the store keeps its public scope id (the
provider tool_call_id) while TaskInfo.task_id stays the execution id, as
lifecycle contributors already see it; the event follows TaskInfo.

* docs(extensions): document compaction event identity and seeded TaskInfo

The extensions guide and the developer manual still said compaction
observers always receive a detached store because no task is live at the
call site. Describe what the host does now: lead runs and subagents with a
parent run seed every task store with TaskInfo, compaction observers get
the live store, and CompactionEvent's task_id/run_id/thread_id come from
it; only manual /compact falls back to a detached store with None ids.
Adds the 0.2.10 row to the contract version table (PR number pending).

* docs(changelog): note compaction observers receive the live task identity

* docs(extensions): list compaction observers among task-store triggers

* docs: reference #7831 for the compaction event identity change

* docs(extensions): backfill contract history rows 0.2.5 to 0.2.9

The version table jumped from 0.2.4 to 0.2.10 although 0.2.5-0.2.9
shipped. Add those rows, attributing names merged without a bump to
the first release that carried them (#5796 in 0.2.5, #6920 and #5645
in 0.2.8, #7729 in 0.2.10). The page intro now says it documents
0.2.4 plus the 0.2.10 CompactionEvent fields, and points to the
history for the rest.
2026-11-03 15:06:52 +08:00
spud
21175ba46f
fix(runtime): persistently fence stale run event producers (#7619)
* fix(runtime): fence revoked run event producers

* fix(runtime): reduce off-mode producer guard locking

* docs: keep inherited guidance within size budget

* chore(runtime): remove development phase labels

* fix(persistence): normalize libpq SSL options for asyncpg connections

* docs(runtime): clarify producer fencing behavior in off mode

* test(persistence): align migration TLS probes with channel-binding rejection

* fix(runtime): authorize fenced Gateway history seeding

* fix(scheduler): preserve takeover identity in recovery publication

* fix(scheduler): report unconfirmed recovery publication

* fix(runtime): preserve history messages before the first human turn

* fix(runtime): clarify run creation failure diagnostic

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-02 15:26:42 +08:00
Tanishq
bf9b031127
fix(skills,goal): build the skill-moderation and goal-evaluator models off the event loop (#7767)
scan_skill_content and the goal evaluator built their chat model inline on
the Gateway event loop. When the provider reads local credential files (the
Codex provider reads the Codex CLI auth.json while it is constructed), each
scanned skill edit, rollback, install or agent-managed skill write, and the
evaluator construction in each goal run, did synchronous file IO on the
shared loop, so other requests and streams waited for it. #7484 removed the
same stall for the one-shot helpers and listed these two constructors in the
utils guidance as not covered.

Build both providers through run_assembly, as the one-shot helpers do: the
moderation model in scan_skill_content, the evaluator in
evaluate_goal_completion when no model is injected, and the run worker's
cached evaluator factory. A scan without app_config (the in-graph
skill_manage call) also reads the live config in that pool, and its
fail-closed fallback reuses that config instead of reading config.yaml again
on the loop; an unreadable config still fails closed. Verdicts, the
fail-open/fail-closed policy, error propagation and tracing are unchanged,
and cancellation during construction never starts inference.

tests/blocking_io/test_system_model_offloop.py builds a real CodexChatModel
under the strict Blockbuster gate. On main the scan fell back to its
fail-closed "unavailable" block, the evaluator raised BlockingError
(os.stat), and the worker stood the goal down as evaluator_failed.
2026-11-02 09:24:31 +08:00
Evan
02df37c5de
fix(runtime): charge a steered subagent's usage to the token budget (#7776)
* fix(runtime): charge a steered subagent's usage to the token budget

A task that finishes after a Steer reports through a receipt HumanMessage, so its usage never reached the dispatching AIMessage the token budget reads. Carry the usage over when the receipt is delivered.

* fix(runtime): report deferred usage with no AIMessage to charge

Log the rare case instead of dropping the usage silently, and scan for AIMessages once per delivery.
2026-11-01 23:28:04 +08:00
Evan
be39a8a879
fix(runtime): resume children paused for a withdrawn steer (#7753)
* fix(runtime): resume children paused for a withdrawn steer

wait_tool pauses every child as soon as a steer is pending, but only a
delivered steer sets steering_active, the flag finish_graph needs before it
resumes them. Removing the steer before delivery left the children parked
and the run waiting on them.

Release the children when deliver finds no input and no steer is in
progress. They stay paused during later model turns of a delivered steer.

* fix(runtime): release children paused by a steer withdrawn mid-turn

The steering_active guard skipped the release whenever another steer was
being handled. A child started during that turn is outside its
interrupted_children snapshot, so nothing resumed it after a second steer
was withdrawn and finish_graph waited forever.

Release every child a delivered steer does not hold. Each deliver re-takes
that snapshot, so a paused child outside it can only belong to a withdrawn
steer.

* fix(runtime): keep a replacement reusing a call ID out of the held steer

interrupted_children stores call IDs, and child() can replace a closed gate under the same ID. The replacement then looked held by an earlier delivered steer, so a withdrawn steer left it paused forever in a replaced turn. Drop the ID when its gate is replaced.
2026-11-01 23:18:49 +08:00
Totoro
a17e5694cc
fix(sandbox): stop owned command processes when a run is cancelled (#7772)
* fix(sandbox): stop owned command processes when a run is cancelled

* fix(sandbox): handle incomplete AIO output responses

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-01 23:09:25 +08:00
Evan
c6a3993695
fix(runtime): keep each parallel task's gate across a steer (#7755)
wait_tool reused the name gate for its pause loop, so every parallel task
returned the last child's result id and its done callback closed that
child's gate. The message reducer then dropped all but one pending result,
and the model was told that a running subagent had been interrupted.

Rename the loop variable so each call keeps its own gate.
2026-10-31 21:40:59 +08:00
Totoro
44e372dd7f
feat(runtime): support Queue and Steer during active Web runs (#7729)
* feat(runtime): add Queue and Steer for active Web runs

* style(frontend): format Queue and Steer controls

* fix(runtime): retain user Steer controls and order active inputs

* docs: add Chinese changelog entry for Queue and Steer

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-31 11:35:36 +08:00
LingYi
30aa601f6a
fix(runtime): isolate run admission from unrelated heartbeats (#7664)
Signed-off-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
2026-10-29 22:19:10 +08:00
SumEn
c58c71f3fa
fix(runs): move thread index membership on snapshot replacement (#7622) 2026-10-28 20:47:01 +08:00
Totoro
c48447a4ae
fix(runtime): present outputs before a loop wrap-up and explain step-limit stops (#7588)
* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops

Loop-detection warnings now ask the agent to present the files it already
created in /mnt/user-data/outputs before its final answer, when
present_files is available.

A run that exhausts recursion_limit now ends with stop_reason
recursion_capped and a plain message that lists outputs it created but
did not present. An active goal stands down without an evaluator call.
The /wait error envelope keeps the GraphRecursionError name, and IM
channels reply with the step-limit message instead of a generic failure.

* docs(changelog): note run-end delivery and step-limit stops (#7588)

* test(runtime): stop pinning LangGraph's step count in the step-limit test

The produced paths and the message already follow the observed model
calls, so the exact count only pinned LangGraph's per-step accounting.
Also note on the step-limit goal branch that its caller passes no
evaluator factory, so it has to stay ahead of the evaluator section.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-27 10:34:34 +08:00
haotianqi666
4f6ee59535
fix(gateway): preserve null input for static continuation (#7578)
* fix(gateway): preserve null input for static continuation

* fix(runtime): preserve continuation inputs during MCP projection

---------

Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
2026-10-26 21:32:25 +08:00
spud
33f7b43400
fix(runtime): backfill run.terminal after confirmed recovery takeover (#7500)
* fix(runtime): backfill run.terminal on confirmed recovery takeover

* fix(runtime): isolate recovery terminal publication failures

* fix(runtime): fence recovery terminal publication lifecycle

* fix(runtime): unify recovery receipt publication

* docs(runtime): document lifecycle guard lock constraints

* fix(runtime): reuse event transaction for recovery guards

Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-25 23:22:02 +08:00
lilei
ccb5fbfa8c
fix(runtime): reject delta resumes that discard pending tools (#7441)
* fix(runtime): reject delta resumes that discard pending tools

* fix(runtime): preserve pending head work on rejected delta replay

* fix(runtime): preserve non-Send checkpoint tasks on resume

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-25 14:44:32 +08:00
spud
e9074b7126
feat(runtime): publish owner-confirmed run.terminal evidence (#7134)
* feat(runtime): publish authoritative run terminal events

* fix(runtime): preserve terminal attribution diagnostics

* docs(runtime): consolidate terminal event docs and trim agent guidance

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-25 00:12:26 +08:00
Xuehao Xu
cfe2bec371
feat(workflows): add multi-Gateway coordination and session-authorized failover (#7138)
* feat(workflows): add leased multi-Gateway coordination and fenced dispatch

* test(workflows): validate deployed failover and canonicalize config revisions

* fix(workflows): share namespace across registration and lease fencing

* fix(workflows): recover interrupted cancellation without admission caps

* fix(workflows): retain execution lease through child cleanup

* docs: trim guidance padding to fit merged instruction budget

* fix(workflows): release rejected admission slots and normalize gateway routing

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-23 23:31:43 +08:00
Totoro
fc4f719664
fix(goal): retain supplied evidence and retry one malformed evaluator verdict (#7345)
* fix(goal): use spare evidence budget for long user input

* fix(goal): retry one malformed evaluator verdict

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-23 20:29:35 +08:00
Totoro
2f0cd407fc
fix(goal): base no-progress counting on successful operations (#7271)
* fix(goal): track successful operations for no-progress checks

* fix(goal): reject failed exits and narration-only progress

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-23 00:14:32 +08:00
TAI-YUAN GUO
f5907405a0
fix(workspace): don't infer created or deleted files from truncated snapshots (#7172)
* fix(workspace): don't infer created or deleted files from truncated snapshots

A bounded workspace scan stops at max_scanned_files and marks the snapshot
truncated. Absence from such a snapshot is unknown, so creation is only
inferred from a complete before snapshot and deletion only from a complete
after snapshot. get_changed_paths() is the single filter for the summary,
text capture and output-delivery detection.

The scanner now walks subdirectories in sorted order so the truncation
boundary is reproducible across the before and after scans, and the docs
note that the budget is consumed root by root (workspace before outputs).

Supersedes #5231.

Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(runtime): record truncated delivery verification skips

Record verification_skipped=truncated_pre_run_snapshot in run.delivery
receipts when a truncated pre-run scan leaves no provable output changes.
Carry the snapshot into both normal and exceptional receipt finalization.

Keep observed-output verification and terminal statuses unchanged. Cover
success, failure, cancellation and capture failures through the real worker,
and document the per-run receipt marker.

---------

Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-22 16:52:53 +08:00
PeaceMaker-best
60940ae9c5
fix(runtime): retain interrupted assistant output (#7153)
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-22 11:47:53 +08:00
Stella-xixi
5cef1a11ef
fix(checkpoints): run conservative retention after completion (#6051)
* fix(checkpoints): wire conservative retention after runs

* fix(checkpoints): fence retention before checkpoint reads

* fix(checkpoints): address retention review feedback

* fix(checkpoints): retry delta reads after retention race

* fix(checkpoints): bound admission retention to the root tail

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-20 22:23:47 +08:00
Totoro
059758f321
fix(runtime): stop Goal continuations after loop hard stops (#7167)
* fix(runtime): stop Goal continuations after loop hard stops

* chore: remove redundant SVG sources from Goal fix

* test(goal): remove PR-only screenshot generation

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-20 16:55:13 +08:00
Nan Gao
085924a3d0
fix(channels): run each IM message once across instances, restarts and late redeliveries (#7104)
* fix(channels): run each IM message once across instances, restarts and late redeliveries

Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.

- Discord messages are keyed (discord, guild_id or "dm", posting channel,
  message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
  before a side effect; Discord claims it on the Gateway loop before
  _create_thread, marks the published message dedupe_claimed, and gives the
  claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
  the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
  attempt made every keyed retry a 409). A reused run's status-only /wait or
  stream gap is resolved to this message's own turn via threads.get_state;
  409s about the message's own keyed run keep the key instead of reading as
  a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
  (claim_thread_id on channel_conversations and channel_thread_bindings);
  the loser adopts the winner and deletes the empty thread it minted.
  /new and /agent use still rebind.

Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).

* fix(channels): give back Discord dedupe claims for messages never processed

Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:

- The hand-off task releases the claim on cancellation too (stop() drains it
  while it resolves the connection identity), via await_drained so the
  cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
  message that shutdown drops from the queue, after the grace period and on
  a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
  because stop() already closed cross-loop intake, or a hand-off cancelled
  before it ran, leaves the claim tracked, and stop() releases it last.

* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation

Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.

- Keys from trusted internal callers (the IM channel manager) are scoped by
  owner only (`internal-run:` prefix); browser and API keys keep the
  owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
  RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
  see the same type, and start_run answers it with 409 "Idempotency-Key
  already used with a different request" instead of a 500. The channel
  manager already treats that 409 as the message's admitted run.

Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 22:57:19 +08:00
RongJie G
bf2cb0e1df
fix(runtime): recover terminal Redis streams after producer loss (#7093)
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 15:53:54 +08:00
spud
850b3b2547
fix(runtime): attribute terminal commits and isolate lease renewals (#6919)
* fix(runtime): confirm terminal commits and isolate lease renewal

* fix(runtime): diagnose unsupported terminal store contracts

* fix(runtime): attribute atomic claims and close lease lifecycle gaps

* fix(runtime): require explicit terminal completion capability

* fix(runtime): retry renewals that race rollback admission
2026-10-19 15:06:35 +08:00
Ryker_Feng
d1d0958bea
feat(skills): add guarded evolution host APIs (#5645)
* feat(skills): add guarded evolution host APIs

* docs(agents): avoid growing saturated guidance chains

* fix(skills): address mutation host review findings

* fix(skills): order name fence before projection lock

* fix(skills): preserve composite toggle lock order

* fix(skills): address catalog and pagination review feedback

* docs: keep subagent guidance within merged instruction budget

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-18 21:17:02 +08:00
TAI-YUAN GUO
a1349a1fe1
fix(runtime): persist the effective model name for every run (#6980)
run_agent() only refreshed runs.model_name when the request named a model,
and it read the effective model from agent.metadata, which compiled
LangGraph graphs do not have. The default model (no model_name requested)
and fallback models were therefore never persisted. Use the effective
model that lead-agent assembly already resolves, and write only when it
differs from the stored value.

Supersedes #3110. Fixes #3109.

Co-authored-by: Yi Tang <6054101+yitang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 11:10:38 +08:00
Totoro
16d36ccd21
feat(sandbox): add bounded AIO prewarming and lifecycle benchmarks (#6998)
* fix(benchmark): validate sandbox lifecycle and concurrent measurements

* feat(sandbox): prewarm likely AIO tool turns

(cherry picked from commit 3a1017ed032ed9331987bc3dc61f23dcdac4e905)

* fix(sandbox): bind bounded prewarming to execution leases

* docs: trim frontend guidance below its existing size limit

* fix(benchmark): separate failed lifecycle stage timings

* test(sandbox): measure prewarm overlap and unused resources through agent graphs

* fix(sandbox): finish lease and sampler cleanup after cancellation or errors

* docs(benchmark): distinguish initialization wait from model overlap

* docs(helm): align the configuration example with version 61

* fix(sandbox): preserve synchronous agent graphs when prewarming is enabled

* test(cargo): isolate native fixtures from inherited make jobserver flags

* fix(sandbox): require managed cleanup before prewarming

* fix(sandbox): log denied prewarming as an expected skip

* fix(benchmark): filter provider summary input records

---------

Co-authored-by: nothingyuancando <1579035512@qq.com>
2026-10-18 08:10:56 +08:00
Xuehao Xu
169002a4c9
feat(workflows): add Phase 2 approvals, immutable artifacts and configuration evidence (#6920)
* feat(workflows): add versioned approvals and run configuration evidence

* docs: trim repeated setup guidance to respect inherited size limit

* docs: leave room in inherited agent guidance

* test(channels): align Buzz capability expectation with upstream registry

* fix(workflows): fence file consumers and preserve review access

* fix(runtime): reuse assembly evidence key at run admission
2026-10-17 09:41:15 +08:00
万家良
4f1c848744
fix(runtime): defer host cancellation through the whole terminal finalization (#6914)
* fix(runtime): defer host cancellation through the whole terminal finalization

Cancelling run_agent() during an early terminal-finalization await (e.g.
the delivery-receipt write) escaped the surrounding `except Exception`
guards, because asyncio.CancelledError derives from BaseException. The
cancellation then skipped the remaining terminal tail: terminal status
persistence, the completion hook, task-stop notification, and the stream
END marker.

Add `_await_despite_host_cancellation`, which shields each terminal step
so the started awaitable keeps running under host cancellation, defers the
interruption (uncancelling the task), and re-awaits the shield until the
step completes. Apply it to the durable finalization stages; the original
interruption is still re-raised after publish_end.

* fix(runtime): bound terminal deferral and shield abort-path cancellation

* fix(runtime): retain owned finalization through cancellation

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-16 07:25:19 +08:00
Totoro
c994826d06
feat(channels): generalize group notifications and add opt-in task/goal IM notices (#6903)
* feat(channels): generalize IM group targets and add task/goal run notifications

- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.

* fix(notifications): address review feedback and group/runtime notification edge cases

* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests

* fix(channels): preserve group binding names and legacy authorization

Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-15 10:30:55 +08:00
Totoro
5e67d41c47
fix(goal): validate goal text and preserve the current goal on message replay (#6685)
* fix(goal): validate goal text and preserve current goal on message replay

* fix(goal): reject malformed continuation counters before admission

* fix(goal): reject external reducer overwrite markers

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-11 18:39:54 +08:00
ordinary being
a7dd4849eb
fix(runtime): mark cancel takeovers with the orphan-recovery stop reason (#6674)
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason

A cross-worker cancel that takes over an expired-lease run writes error but
left stop_reason unset. `orphan_recovered` is the only signal that lets a
store-only SSE/`wait` consumer synthesize END after a heartbeat, because the
producer is known to be gone. Without it, a subscriber that outlives the
crashed owner waits forever for an END no surviving worker publishes; periodic
reconciliation ignores the row because it is already terminal.

Mark the takeover like periodic orphan recovery: claim_for_takeover already
accepts the stop reason and the reconcile path already passes it. The new
tests pin the takeover row's marker and prove a real takeover wakes a waiting
consumer; three existing takeover test doubles now forward the store argument.

* docs(gateway): keep guidance within the 48KB budget

The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment.

* fix(gateway): complete streams after scheduler orphan recovery

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-11 18:30:09 +08:00
RongJie G
b708145001
fix(runtime): require explicit resume identity capability (#6571)
* fix(runtime): require explicit resume identity capability

Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>

* fix(runtime): guard key-only idempotency admission

---------

Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-11 11:55:20 +08:00