* fix(hitl): never pause on an approval call without a usable id
A provider can emit a tool call with no id (langchain_core keeps `id=None`),
an empty id, or the same id twice in one response. `HitlApprovalMiddleware`
copied that id into the interrupt payload, but the Gateway resume validator
requires unique non-empty strings: every resume answered 422 and every new run
stayed 409 `approval_pending`, so the thread was wedged permanently — the
all-reject escape is validated after that check. Denial paths also built
`ToolMessage(tool_call_id=None)`, which raises inside `after_model`, so a
non-interactive run failed instead of denying the call.
Treat such calls as unaddressable: strip them from the model response with a
visible rejection note instead of pausing, and deny by removal whenever a
`ToolMessage` could not pair. The tool never executes and the thread stays
usable.
Fixes#8069
* docs(hitl): state the empty-id rule accurately and track legacy recovery
Review follow-up on the unaddressable-approval guard:
- An empty tool-call id was technically resumable end to end (langchain-core
accepts `ToolMessage("")`, and the Gateway validator accepts `""`), so only a
null or duplicated id can wedge a pause. The rule still denies empty ids
because strict providers reject them when the conversation is replayed; the
docstring, CHANGELOG and interaction policy now say exactly that, and
`_process_responses` uses the same `_addressable` predicate as the pause
guard so the two checks cannot drift.
- Add an operator note for pauses stored before this fix: they are not cleared
here, so continue in a new thread; in-place all-reject recovery is tracked in
#8071.
* fix(hitl): drop a colliding call id from every serialized surface
Review follow-up: `_strip_calls` removed the unaddressable approval calls from
`tool_calls`, but `clone_ai_message_with_tool_calls` filters the raw
`additional_kwargs["tool_calls"]` payload and content tool-call blocks by id.
A non-target call sharing a removed id kept that id in `kept_ids`, so the
removed calls stayed in the raw provider payload and the next request would
advertise calls that have no result.
Drop the whole colliding id group: an id shared with a denied approval call
cannot be answered unambiguously either. The regression covers a mixed response
(duplicate approval ids plus an ordinary call with the same id) and asserts the
raw payload, content blocks, execution and visible note.
* fix(hitl): reject an id shared with an ordinary call before pausing
Review follow-up: `_addressable(targets)` only asked whether a `ToolMessage`
can answer the targeted calls. A single approval-required call whose id is
shared by an ordinary call in the same response still paused, even though
approving or rejecting it would also answer the ordinary call by id.
Add `_unambiguous_ids`: every targeted id must also be unique across the whole
response. A collision denies the colliding group (strip + visible note) before
the pause, and the resume path applies the same check to a legacy pause. New
regression: one approval-required call and one ordinary call sharing an id.
* fix(hitl): strip a deny target whose id an ordinary call shares
`_deny_all` routed only on `_addressable`, which checks uniqueness among the
targeted calls. A target whose id an ordinary non-target call also carried was
therefore answered with a deny `ToolMessage` while the ordinary call stayed
serialized, so the next request advertised the id twice with two results.
`_deny_all` now requires `_unambiguous_ids` and strips the whole colliding id
group, matching the pause path. The tool still never executes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): persist a rejected HITL call in the resume run's history
An approval pause and its resolution are two Runs. The paused Run dispatches
the gated call; the resume Run produces its result. An approved call executes
and reaches on_tool_end, but a rejection is written by HITL after_model with no
tool callback. Journal reconciliation only persists such middleware-written
results for calls dispatched by the current Run, so the rejection stayed in
checkpoint state and never reached the run-event history: /messages and the
resume run's messages had no llm.tool.result for it.
The worker already reads the durable interrupt payload to recover the gated
tool names. It now also returns each paused tool_call_id with its name, and the
resume journal adopts those calls, so the existing reconciliation persists the
rejection under the resume Run. Approved results keep their tool-callback event
and are deduplicated by message id; results from other earlier Runs are still
not re-attributed.
* fix(runtime): keep HITL replay names independent of call adoption
Derive the replay-gate tool names and the adopted call ids separately
from the matched pause's actions, so an action without a usable
tool_call_id still gates replay. Pin that the all-reject escape paths
(approval disabled, gated tools changed or cleared while paused) reach
the history feed under the resume run, check resume-run attribution in
the mixed-decision test, and add a CHANGELOG entry.
* refactor(runtime): drop unused _pending_hitl_resume_calls wrapper
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(threads): keep the owner row when deleting checkpoints fails
DELETE /api/threads/{id} treated the checkpoint delete as best-effort: a
failure was logged at DEBUG, the threads_meta row was removed anyway and
the request returned success. Read routes allow a thread without a meta
row as an untracked legacy thread, so the surviving checkpoints (title,
messages, goal) became readable by every signed-in user through
GET /threads/{id}, /state, /history and /goal, and the owner could not
retry because DELETE requires the row.
A failed checkpoint delete now logs the exception and returns a generic
500 before the meta row is touched, so the thread stays owned and the
owner can retry. The remaining cleanup steps stay best-effort; their
reads are owner-scoped.
* docs(changelog): reference #7992 for the thread delete checkpoint fix
* fix(threads): require checkpoint deletion before dropping ownership
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(agent): ground runtime decisions in structured evidence
Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.
Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.
* test(sandbox): expect read_file initialization error callback
* fix(agent): address evidence review follow-ups
* fix(agent): preserve retry budgets for OpenAI transport errors
* fix(agent): expose tool execution status in model requests
Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep Feishu cards within the card size limit
Card text was unbounded and ASCII-escaped, so an answer of about 4,400 CJK
characters made the card request body exceed Feishu's 30 KB limit (230025).
The final patch and its reply fallback were both rejected, the answer was
lost and the running card stayed on its last streamed text.
Cards are now serialized without ASCII escapes and budgeted by the bytes
their text adds to the request body. A streamed update shows what fits; a
longer final answer continues in up to 10 cards replied to the same source
message, split at line breaks, and a retry resumes after the cards already
delivered.
* docs(changelog): reference #7798 for the Feishu card size fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): key WeCom group conversations by chatid, not the sender
Group messages used the sender's userid as chat_id and topic_id, the same
key as the sender's private chat. Wherever no group connection kept them
apart (auth-disabled mode, require_bound_identity: false, legacy group
bindings), a member's private chat and every group they wrote in ran in one
DeerFlow thread. Groups are now keyed by chatid with one thread per member.
A group-scoped connection carries its sender-keyed thread over once; a
legacy binding, which spans every group of the bot, does not.
* docs(changelog): reference #7794 for the WeCom group conversation fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): continue native batches and honor item dependencies
* docs: link native batch follow-up changelog entries
* test(migrations): pin isolated history-seed upgrade to 0050
* fix(subagents): preserve follow-up scope, blockers and fair scheduling
* test(memory): freeze capacity evaluation time
* fix(subagents): guard follow-up admission and retain run controls
* fix(subagents): enforce runs:cancel on tool cancellation of follow-up batches
* style(tests): format test_batch_followup_host with ruff
* fix(subagents): drop the admission checkpoint check and keep SDK cancel compatible
The checkpoint check inside run admission waited on the checkpoint saver's
lock while holding the SQLite write lock, which could make another chat's
checkpoint write fail. Remove it; dispatch still checks the conversation
before admission.
A follow-up waiting for an unanswered question now keeps its claim until the
lease expires instead of rereading the conversation every poll. Cancel passes
the caller's run authority only to submitters that take it, so existing SDK
submitters keep the two-argument call. Document that retrying a prerequisite
by hand does not requeue its blocked items, and list 0052 as the migration head.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): include nginx logs in dev_multi_instance.sh logs all
`logs all` selected only the two Gateway logs, the same as running
`logs` with no argument, so nginx's error and access logs never showed
up although the help lists all next to nginx. `all` now adds the nginx
logs, and the nginx selection also covers nginx.log, where the start
command's own output goes and which no option printed before.
* docs(changelog): reference #7723 for the dev_multi_instance logs fix
* test(scripts): pin dev_multi_instance logs when nginx never started
Cover the layout start_nginx leaves behind when it bails on a missing
binary or a busy port: only gateway-*.log exists, so `logs all` prints
the two Gateway files and `logs nginx` prints nothing, both exiting 0.
* fix(tui): show picker labels as written instead of parsing markup
SelectScreen passed thread titles and model display names to Textual as
plain str, which Textual parses as content markup. A thread title such as
'Fix [/bold] in parser' raised MarkupError and exited the app whenever
/threads opened, and '[a]' vanished from the list. Wrap the title and
option labels in Rich Text so they render literally.
* docs(changelog): reference #7631 for the TUI picker markup fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Declare required thinking, native reasoning effort and image support for both
K3.1 Pro IDs. Normalize fixed parameters at the final Moonshot request boundary
and cover real SDK serialization with offline regression tests.
Fixes#7425
* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops
Loop-detection warnings now ask the agent to present the files it already
created in /mnt/user-data/outputs before its final answer, when
present_files is available.
A run that exhausts recursion_limit now ends with stop_reason
recursion_capped and a plain message that lists outputs it created but
did not present. An active goal stands down without an evaluator call.
The /wait error envelope keeps the GraphRecursionError name, and IM
channels reply with the step-limit message instead of a generic failure.
* docs(changelog): note run-end delivery and step-limit stops (#7588)
* test(runtime): stop pinning LangGraph's step count in the step-limit test
The produced paths and the message already follow the observed model
calls, so the exact count only pinned LangGraph's per-step accounting.
Also note on the step-limit goal branch that its caller passes no
evaluator factory, so it has to stay ahead of the evaluator section.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM binding codes on screen and report goal state in replies
Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a
dialog that keeps the /connect command on screen with a copy button, its
expiry time and the binding status, instead of a toast that disappears.
The bots' /connect replies and the IM /goal replies now use the owner's
web UI language (else channel_connections.notification_locale), and /goal
reports the goal's state as the web goal bar does: in progress, why it
stopped, or that it was met.
* docs(changelog): note the IM binding dialog and localized IM replies (#7591)
* test(channels): check IM reply tables against every notification locale
The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal
reply table had none. Both now match NOTIFICATION_LOCALES with the same
keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale
fails here instead of raising KeyError at reply time.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(streaming): make Redis stream END publication idempotent
* fix(streaming): align Redis test fake with Lua END checks
Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis.
* fix(streaming): gate legacy END scans after writer rollout
* fix(helm): synchronize embedded config version with schema
* test(runtime): use real stream bridge config in close coverage
Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(runtime): honor the web UI language in replies and DeerMem updates
Carry Settings > Appearance > Language as request-only run data: the
Gateway validates context.locale against a finite allowlist, the lead
agent prefers it for replies, and bundled DeerMem writes new facts and
summary prose in it. Explicit user language requests still take
precedence; checkpoint configurable, static prompts, and memory backend
APIs are unchanged.
Refs #7287
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime): guard language context and pin locale maps
* test(runtime): pin UI locales across frontend and Gateway
Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(channels): honor the configured recursion limit in IM runs
Every IM channel run asked the Gateway for recursion_limit 100, because
the channel manager seeded each run config with it. The Gateway honors a
request value, so the top-level recursion_limit never reached IM runs,
and a budget of 100 could stop a run after a handful of tool-calling
turns (six in one measurement with the default lead-agent middleware;
the count changes with the middleware chain).
IM runs now send no recursion_limit of their own and get the Gateway's
configured default, clamped by max_recursion_limit, like Gateway API
runs. A session config value still takes precedence, and GitHub channel
runs keep their floor of 250. The README and config.example.yaml IM
examples no longer pin session.config.recursion_limit to 100.
* docs(changelog): reference #7521 for the IM recursion-limit fix
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Move checkpoint modes, replay, module maps and terminal-ownership details intact into the existing runtime reference. Keep concise invariants and explicit links in AGENTS.md, reducing the effective chain from 99,004 to 80,817 bytes. The existing guidance regression now passes without changing budget thresholds.
* fix(runtime): backfill run.terminal on confirmed recovery takeover
* fix(runtime): isolate recovery terminal publication failures
* fix(runtime): fence recovery terminal publication lifecycle
* fix(runtime): unify recovery receipt publication
* docs(runtime): document lifecycle guard lock constraints
* fix(runtime): reuse event transaction for recovery guards
Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(threads): reject an assistant_id that names no agent at thread creation
POST /api/threads stored assistant_id unchecked. Every state read then built
a run config from it, and build_run_config's ValueError answered 500 on GET
thread, state and history until the thread was deleted.
Validate it at creation with the rule run creation already applies, now
shared as custom_agent_name(), and answer 422. Records written before this
change are read with the lead agent's schema instead of failing.
* docs(changelog): reference #7445 for the thread-creation assistant_id fix
* fix(threads): clear invalid legacy assistant ids in branches
* docs(changelog): note the branch assistant_id behavior for #7445
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): normalize mount container_path trailing slashes at config load
A container_path declared with a trailing slash (e.g. /mnt/reports/) was
accepted by VolumeMountConfig and normalized by LocalSandboxProvider when
building path mappings, but the sandbox tool path gates compare the raw
configured value, so every ls/read/write/glob/grep/bash access to the
mount failed with a confusing PermissionError. Normalize trailing
slashes in the config model so both layers agree, and reject the
container root '/' which the gates cannot express.
* fix(sandbox): normalize skills.container_path and pin empty-value rejection
Review follow-up on #7393:
- Share the container-path normalizer with SkillsConfig, whose prefix
feeds the same as-is comparison in _is_skills_path and had the same
trailing-slash dead-path failure mode.
- Pin the empty-string rejection: on main an empty container_path made
the gates match every absolute path, applying one mount's read_only
flag to all paths.
- Cover both config fields in CONFIGURATION.md.
---------
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>
The two-Gateway harness checklist in CONFIGURATION.md said a crashed
leader hands its providers over within lease_seconds + grace_seconds; a
peer only tries to acquire on its fixed lease_seconds // 3 tick, so the
worst case is one renewal interval more (50 s with the defaults), as the
leader-lease section of the same file already says. The grace_seconds
field description had the same shorter bound.
The channels.leader_lease comment in config.example.yaml still said web
UI credential changes reach the holder only when it next starts the
channel; since #6988 the holder applies them within one renewal interval.
The harness `check` footer still listed the IM channel gates as waiting
for leader election, which landed in #6843; it now points at the manual
channel item in CONFIGURATION.md, which also gains the runtime-config step.
* fix(stream): tolerate ~~~-fenced code and \-prefixed closer when decoding escaped newlines
_decode_escaped_newlines_outside_fences only recognised ``` fences, so replies
that used ~~~ fences (e.g. code blocks rendered by the model with
~~~python ...) kept literal backslash-n sequences escaped instead of decoding
the ones outside fences, and a fence whose closer was split as backslash+tilde
across chunks was misread mid-stream. Fixes#7262.
- detect the opening fence with (?:```|~~~) and remember which delimiter the
fence was opened with, so closing requires the same delimiter
- while inside a fence, hold back a closer that begins with a backslash so a
fence ending in backslash+tilde mid-stream is not misread; the held tail is
revisited on the next push and normalised on flush
- keep backtick hold semantics unchanged
* fix(mindie): preserve unclosed fences during newline decoding
Match one-shot decoding to streaming by choosing the earliest backtick or tilde opener and preserving the remainder when its matching closer is absent. Correct the string-return docstring and cover mixed delimiters across every two-chunk split and character-by-character streaming.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): answer malformed run bodies with 4xx instead of 500
RunCreateRequest types config, context and input as free-form dicts, so
some bodies it accepts still failed during run admission with an
exception nothing mapped, and the run routes answered 500:
- a non-object config.configurable (build_run_config's dict.update, and
.get in the stateless thread selector)
- a non-object config.context (build_run_config's own ValueError)
- an assistant_id that does not normalize to letters, digits and hyphens
- a non-string agent_name in config.configurable, config.context or
context (_load_scope_agent_config)
- an AI message whose tool_calls is a string, object or list of strings
(AttributeError from LangChain's validator)
The request model now requires config.configurable and config.context to
be objects or null (422 invalid_run_config_section), and
build_run_config checks configurable as it already checked context.
start_run maps build_run_config's ValueError to 422, as it does for
validate_thread_id. Message conversion also maps AttributeError to its
existing 400, and a non-string agent_name gets the existing 422 for an
unresolvable agent. All of these fail before admission, so no run is
created.
* docs(changelog): reference #7362 for the malformed run body fix
* docs(changelog): note the configurable shapes #7362 now rejects
* fix(tui): let quit skip an interrupted run's unfinished step
After Ctrl+C the synchronous agent stream keeps running until its next
event. The run was a Textual thread worker, i.e. a thread on asyncio's
default executor, which App.run() joins on shutdown (up to 300 s on
3.12) and interpreter exit joins again with no timeout. Quitting while
a long step was still running therefore hung silently after the
terminal had been restored, and further Ctrl+C presses only printed a
traceback.
Run workers on a daemon thread instead. Quit now waits visibly for
running workers (an interrupted step still checkpoints, a finished run
still saves its title) and refuses sends meanwhile; a second quit
request exits at once. Report unexpected worker exceptions as an error
row and end the run, since Textual no longer catches them.
* docs(changelog): reference #7326 for the TUI quit-after-interrupt fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Exa web_search and web_fetch accept a per-tool base_url, falling back to
EXA_BASE_URL and then the SDK default. Overrides are validated like the
Serper base URL. web_fetch screens a configured backend with the shared
delegated-backend guard.
Co-authored-by: lachy3433 <lachlanallen.co@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
