* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops
Loop-detection warnings now ask the agent to present the files it already
created in /mnt/user-data/outputs before its final answer, when
present_files is available.
A run that exhausts recursion_limit now ends with stop_reason
recursion_capped and a plain message that lists outputs it created but
did not present. An active goal stands down without an evaluator call.
The /wait error envelope keeps the GraphRecursionError name, and IM
channels reply with the step-limit message instead of a generic failure.
* docs(changelog): note run-end delivery and step-limit stops (#7588)
* test(runtime): stop pinning LangGraph's step count in the step-limit test
The produced paths and the message already follow the observed model
calls, so the exact count only pinned LangGraph's per-step accounting.
Also note on the step-limit goal branch that its caller passes no
evaluator factory, so it has to stay ahead of the evaluator section.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(checkpoints): reject malformed Redis history write records
Validate pending-write tuples before optional Redis history entries reach
delta materialization. Treat the whole malformed entry as a miss so the
authoritative saver rebuilds it without dropping or inventing state.
Cover actual checkpoint reads, whole-entry replacement, valid tuple/list
records, neighbor preservation and warm-cache rereads.
* fix(checkpoints): bind cached history writes to their channel
* fix(checkpoints): keep cache warning failures best-effort
Guard warning handlers across decode, ingestion, serialization, Redis outages and purge. Preserve authoritative rebuilds and valid neighboring writes, with native graph regression coverage.
* docs(gateway): keep guidance below the hard size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(streaming): make Redis stream END publication idempotent
* fix(streaming): align Redis test fake with Lua END checks
Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis.
* fix(streaming): gate legacy END scans after writer rollout
* fix(helm): synchronize embedded config version with schema
* test(runtime): use real stream bridge config in close coverage
Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Move checkpoint modes, replay, module maps and terminal-ownership details intact into the existing runtime reference. Keep concise invariants and explicit links in AGENTS.md, reducing the effective chain from 99,004 to 80,817 bytes. The existing guidance regression now passes without changing budget thresholds.
* fix(runtime): backfill run.terminal on confirmed recovery takeover
* fix(runtime): isolate recovery terminal publication failures
* fix(runtime): fence recovery terminal publication lifecycle
* fix(runtime): unify recovery receipt publication
* docs(runtime): document lifecycle guard lock constraints
* fix(runtime): reuse event transaction for recovery guards
Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(checkpoints): isolate cached history payloads from state readers
Copy nested snapshot seeds and pending-write payloads at both cache boundaries.
Keep user copy callbacks outside the registry lock so readers cannot poison
later checkpoint materialization or deadlock by re-entering the cache.
Cover sync/async state accessor reads, nested message payloads, disabled-cache
controls and callback lock ownership.
* docs(checkpoints): document synchronous history copy costs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: honor checkpoint object in thread state updates
Prefer the checkpoint object over the deprecated checkpoint_id, validate
its thread_id, and preserve checkpoint namespace when updating state.
Add regression coverage for checkpoint-object forks and thread mismatch.
* fix: preserve client context handling in checkpoint PR
* fix: support state updates in checkpoint namespaces
* fix: reject unknown checkpoint namespaces
* fix(gateway): isolate subgraph titles from thread metadata
Only synchronize state titles into conversation metadata for the root checkpoint namespace. Cover persisted child updates with and without checkpoint IDs in full and delta modes, and retain root title synchronization and branch sequence cleanup.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(goal): use spare evidence budget for long user input
* fix(goal): retry one malformed evaluator verdict
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(workspace): don't infer created or deleted files from truncated snapshots
A bounded workspace scan stops at max_scanned_files and marks the snapshot
truncated. Absence from such a snapshot is unknown, so creation is only
inferred from a complete before snapshot and deletion only from a complete
after snapshot. get_changed_paths() is the single filter for the summary,
text capture and output-delivery detection.
The scanner now walks subdirectories in sorted order so the truncation
boundary is reproducible across the before and after scans, and the docs
note that the budget is consumed root by root (workspace before outputs).
Supersedes #5231.
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(runtime): record truncated delivery verification skips
Record verification_skipped=truncated_pre_run_snapshot in run.delivery
receipts when a truncated pre-run scan leaves no provable output changes.
Carry the snapshot into both normal and exceptional receipt finalization.
Keep observed-output verification and terminal statuses unchanged. Cover
success, failure, cancellation and capture failures through the real worker,
and document the per-run receipt marker.
---------
Co-authored-by: liutian <91778998+ImcLiuQian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* fix(channels): run each IM message once across instances, restarts and late redeliveries
Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.
- Discord messages are keyed (discord, guild_id or "dm", posting channel,
message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
before a side effect; Discord claims it on the Gateway loop before
_create_thread, marks the published message dedupe_claimed, and gives the
claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
attempt made every keyed retry a 409). A reused run's status-only /wait or
stream gap is resolved to this message's own turn via threads.get_state;
409s about the message's own keyed run keep the key instead of reading as
a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
(claim_thread_id on channel_conversations and channel_thread_bindings);
the loser adopts the winner and deletes the empty thread it minted.
/new and /agent use still rebind.
Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).
* fix(channels): give back Discord dedupe claims for messages never processed
Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:
- The hand-off task releases the claim on cancellation too (stop() drains it
while it resolves the connection identity), via await_drained so the
cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
message that shutdown drops from the queue, after the grace period and on
a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
because stop() already closed cross-loop intake, or a hand-off cancelled
before it ran, leaves the claim tracked, and stop() releases it last.
* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation
Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.
- Keys from trusted internal callers (the IM channel manager) are scoped by
owner only (`internal-run:` prefix); browser and API keys keep the
owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
see the same type, and start_run answers it with 409 "Idempotency-Key
already used with a different request" instead of a 500. The channel
manager already treats that 409 as the message's admitted run.
Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: support client context capabilities
(cherry picked from commit b7c223f9415dcd545cbe05a5f5a6e47073a42d68)
[Totoro-qaq: resolved onto main. The persisted-reminder middleware hunks and their tests are dropped (delivery is redesigned in a follow-up); the RunCreateRequest description moved to app/gateway/run_models.py.]
* fix: address client context review comments
(cherry picked from commit bce6da6b819228eda58e887872720b6f9b5fab69)
[Totoro-qaq: middleware hunks dropped with the redesigned delivery.]
* feat(agents): deliver client context as a request-scoped data block
The cherry-picked #3106 put context.client into the persisted first-turn
reminder, with change detection and a "not provided" clear marker. Main
has since moved the date reminder to a SystemMessage, made request-scoped
context (the project block, #5443) request-only, and agents/AGENTS.md now
requires user-influenced values to ride the HumanMessage data channel,
never system text.
Render runtime.context["client"] on every model call in
DynamicContextMiddleware.wrap_model_call/awrap_model_call as one hidden
HumanMessage right after the leading SystemMessages, where DurableContext
puts its data. It is never a state update and never flagged
dynamic_context_reminder, so nothing is checkpointed, summarized or
replayed, and a run without a client gets no block. An unchanged client
keeps the previous request as the next request's prefix.
The renderer sanitizes again (direct graph callers bypass the Gateway),
html-escapes values and frames them with frame_untrusted_text: the same
--- BEGIN/END USER INPUT --- boundary the user's own turn and the MCP task
notification get. The lead prompt treats everything outside those markers
as framework data, so unframed values would rank above the user's own
text; framed, context.client can say nothing the same caller could not
already say in input.messages. client_context joins _BLOCKED_TAG_NAMES so
a forged block in user input or a tool result is escaped, and one static,
value-free sentence in the lead prompt names the block as untrusted hints,
never instructions, permissions or identity. The now unused empty-context
marker is removed.
* fix(gateway): log dropped context.client fields without values
A run request is never rejected over context.client: unknown fields,
nested values and malformed entries are silently dropped, the same
leniency as the other context keys. That left no signal when a custom
client sent something the Gateway ignored (review on #3106).
Route both the config.context.client and the body context.client paths
through one helper that emits a debug line whenever sanitizing changed
the value. Like the malformed client_timezone branch, the line carries no
values; unlike client_timezone, client.name and preference strings are
free caller-controlled text, so they are never logged, not even
sanitized. A clean client, an explicit null or an absent key logs
nothing.
Tests also pin the config-only path (sanitized before the early return)
and the end-to-end start_run admission: the sanitized client reaches
config["context"] and never configurable.
* fix(runtime): bound numeric client context preferences
String preferences are capped at 200 characters, but integers were kept
at any size. A JSON body may carry integers of up to 4300 digits, so 32 of
them rendered a block of about 138,000 characters, re-sent on every model
call of the run. In-process callers could also pass an int past the str()
digit limit, which failed the model call, or a number subclass with its
own __str__.
Keep integers within JavaScript's Number.MAX_SAFE_INTEGER, the range a
browser client can send exactly, and coerce numbers to plain int and
float.
* docs(api): describe client context as advisory request-scoped hints
The cherry-picked section described a hidden reminder that no longer
exists. Document what a custom client can rely on: the allow-list and its
limits (strings, the integer range, and only the first 32 entries of each
map), debug-level logging of dropped fields, runtime-only forwarding, and
delivery as a per-request user-role block, framed like user input, rather
than system text or thread history.
Only top-level context.client is documented. An explicit
config.context.client is sanitized for the run as well, but the run record
keeps the caller's config as sent, so it is not offered as an alternative.
It also answers the #3106 review: capabilities and preferences are
advisory hints, not enforcement, which belongs outside the model; name is
informational and never for authorization; the same object should be
sent on every run to keep the cached prefix stable; subagents and
embedded DeerFlowClient runs do not receive it, and server-started runs
(schedules, IM channels, GitHub agents, MCP notifications, extensions
unless the plugin passes one) carry none.
* fix(context): declare client delivery and keep client out of configurable
The request-only <client_context> block is model-visible, so
DynamicContextMiddleware now declares its delivery and the sanitizer's
limits in release_policy_parameters(); runs before and after this change
no longer share a policy fingerprint.
build_run_config copies body.config['configurable'] verbatim and
configurable is persisted with checkpoints, so a client placed there is
dropped instead of being stored raw. Client context stays runtime-only.
Document context.client in the README and in the middleware guide.
* fix(context): apply the PII redaction policy to the client block
DynamicContextMiddleware adds the <client_context> message inside the
outer PiiRedactionMiddleware, and the hidden message is not re-sanitized,
so with pii_redaction enabled a preference such as an email address
reached the model verbatim while the same address in the user turn was
redacted. Pass the rendered block through the shared redact_text seam
with this middleware's app_config.pii_redaction, and declare the setting
in release_policy_parameters().
* fix(context): keep client hints out of stored run config
---------
Co-authored-by: Blair Leng <25675774+BlairLeng@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
run_agent() only refreshed runs.model_name when the request named a model,
and it read the effective model from agent.metadata, which compiled
LangGraph graphs do not have. The default model (no model_name requested)
and fallback models were therefore never persisted. Use the effective
model that lead-agent assembly already resolves, and write only when it
differs from the stored value.
Supersedes #3110. Fixes#3109.
Co-authored-by: Yi Tang <6054101+yitang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): retire memory buffers and wake existing subscribers
* fix(streaming): preserve end for fully consumed retired streams
* docs: keep frontend guidance within its hard budget
* fix(streaming): report gaps when retirement wins initial attachment
Cancel event and progress writes before force-close waits, bound progress settlement, and supervise cancellation-resistant reporters until their late result is observed.
Fixes#6986
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
* fix(compaction): reject manual compaction of threads with pending execution
A thread paused at a static breakpoint (interrupt_before=["tools"]) keeps its
pending node send in the checkpoint's __pregel_tasks channel. The compaction
write goes through the state-mutation graph, which schedules no tasks, so
LangGraph drops the pending send ('Ignoring unknown node name tools in pending
sends') and a resumed run finishes success without ever executing the tool.
Fail closed: detect a non-empty __pregel_tasks in the raw checkpoint tuple
inside compact_thread_context and raise ContextCompactionPendingWork, mapped
to HTTP 409 by the compact route. The reservation (reserve_checkpoint_write)
already serializes this check against runs and other checkpoint writers, and
the write still forks from the exact head that was read.
Adds dual-mode (full/delta) regression tests: the rejected compaction leaves
checkpoint id, next and pending task ids untouched, and resuming afterwards
executes the pending tool exactly once; idle threads still compact.
* fix(compaction): reject unavailable checkpoints before summarizing
* docs: keep compaction guidance within merged size budget
* fix(runtime): defer host cancellation through the whole terminal finalization
Cancelling run_agent() during an early terminal-finalization await (e.g.
the delivery-receipt write) escaped the surrounding `except Exception`
guards, because asyncio.CancelledError derives from BaseException. The
cancellation then skipped the remaining terminal tail: terminal status
persistence, the completion hook, task-stop notification, and the stream
END marker.
Add `_await_despite_host_cancellation`, which shields each terminal step
so the started awaitable keeps running under host cancellation, defers the
interruption (uncancelling the task), and re-awaits the shield until the
step completes. Apply it to the durable finalization stages; the original
interruption is still re-raised after publish_end.
* fix(runtime): bound terminal deferral and shield abort-path cancellation
* fix(runtime): retain owned finalization through cancellation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(runtime): cover the driver suffix in sync and store DSNs
* fix(runtime): strip the SQLAlchemy driver suffix from sync and store DSNs
With `postgres_url: postgresql+asyncpg://...` and the default empty
`postgres_schema`, the sync checkpointer, the sync store and the async
store passed the DSN to psycopg's `from_conn_string` unchanged, and
libpq cannot parse the `+asyncpg` scheme. `dsn_with_search_path` only
drops the suffix when it adds a schema, so a non-empty schema hid the
bug. The async checkpointer already calls `normalize_libpq_dsn` first.
Call `normalize_libpq_dsn` at these three call sites as well.
* refactor(persistence): drop the driver suffix inside dsn_with_search_path
dsn_with_search_path dropped a SQLAlchemy `+driver` suffix only when it
added a search_path, so every libpq caller had to wrap the DSN in
normalize_libpq_dsn to cover an empty schema, and flattening that
nesting later would bring the bug back.
Normalize at the top of dsn_with_search_path so both paths drop the
suffix and check the scheme in one place, and remove the outer call
from its five callers. The sync checkpointer and both stores are back
to their code on main.
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): preserve canonical delta-history order in cache
CachedHistorySaver composed a parent checkpoint's channel writes from
CheckpointTuple.pending_writes in storage order, but the saver's
authoritative get_delta_channel_history orders writes by (task_id, idx).
On the in-memory backend pending_writes follows task-arrival order, so a
fan-out super-step (several parallel tasks writing one channel in a single
checkpoint) was materialized in a different order with the cache enabled
than with the bare saver, breaking the cache's transparency invariant.
Sort a checkpoint's channel writes by task_id in _channel_writes, restoring
the canonical (task_id, idx) order while preserving each task's internal idx
sequence. The sort is a no-op on sqlite and postgres, whose pending_writes
already arrive ORDER BY task_id, idx.
Add async and sync regression tests over a real InMemorySaver seeded with a
fan-out checkpoint whose writes arrive out of task_id order; both diverge
from the inner saver's native walk without the fix and match with it.
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
* docs(runtime): clarify cached saver ordering contract
---------
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason
A cross-worker cancel that takes over an expired-lease run writes error but
left stop_reason unset. `orphan_recovered` is the only signal that lets a
store-only SSE/`wait` consumer synthesize END after a heartbeat, because the
producer is known to be gone. Without it, a subscriber that outlives the
crashed owner waits forever for an END no surviving worker publishes; periodic
reconciliation ignores the row because it is already terminal.
Mark the takeover like periodic orphan recovery: claim_for_takeover already
accepts the stop reason and the reconcile path already passes it. The new
tests pin the takeover row's marker and prove a real takeover wakes a waiting
consumer; three existing takeover test doubles now forward the store argument.
* docs(gateway): keep guidance within the 48KB budget
The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment.
* fix(gateway): complete streams after scheduler orphan recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): drain staged-terminal runs on shutdown before checkpointer teardown
RunManager.shutdown() selected only pending/running records, but with an event
store a run stages its terminal status in memory and keeps finalizing: journal
flush, delivery receipt, workspace scan, the duration checkpoint write and the
deferred terminal commit, all while terminal_commit_pending is set. A Gateway
shutdown landing in that window did not await the run, so langgraph_runtime
could close the checkpointer before the duration write landed - the #3373
class the drain exists to prevent.
Drain those records too, using the existing _awaits_terminal_commit state that
_renew_leases already relies on; await them without cancelling (a cancel would
skip their terminal tail) and never overwrite their staged terminal status
with interrupted.
* fix(runtime): surface shutdown terminal finalizer failures
Cover timed-out staged terminal runs for both commit barriers and terminal outcomes. Log failed finalizers with their run ID, staged status, and traceback while preserving status and completing the drain.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
