* fix(harness): keep a cancelled batch cancelled when retrying a failed item
retry_item only kept a paused batch as it was, so retrying a failed item
of a cancelled batch reset the batch to queued and the worker claimed and
ran the item. A cancelled batch now rejects the retry (409), re-checked
under the write lock because SQLite ignores FOR UPDATE, and the workspace
no longer offers Retry there.
* docs(changelog): reference #7792 for the cancelled batch retry fix
* fix(agents): store an explicit null description as empty
PUT /api/agents/{name} accepted "description": null and wrote it into the
agent's config, but AgentConfig.description is a str. Every later load of
that agent failed validation: the PUT itself, GET and repair PUTs returned
500, the file backend dropped the agent from the list, and the db backend
failed the owner's whole list.
Store an explicit null as the empty default, and have parse_agent_config
read a stored null description (already written by this route, or a bare
YAML key) as empty so affected agents load again.
* docs(changelog): reference #7638 for the agent null description fix
* docs(agents): document that a null description resets it to empty
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runs): trim event_types filter entries before querying
* style(runs): compute the trimmed event type once per entry
Addresses the review nit on #7758: use the walrus pattern already present
in this file (lines 657 and 1658) instead of stripping each entry twice.
No behavior change.
---------
Co-authored-by: gouyu-hou <gouyu-hou@users.noreply.github.com>
Co-authored-by: gouyu-hou <213296172+gouyu-hou@users.noreply.github.com>
* fix(console): tolerate malformed usage counters and pricing
* docs(routers): record console counter tolerance in the router contract
* fix(console): reject stored counters beyond the float range
A JSON integer is not bounded by the float range: 10**400 parses as an
exact int, so _token_count passed it through and _token_cost then raised
OverflowError on the division back to float, taking /usage, /stats and
/runs down with a 500 - the same failure mode the rest of this PR
removes for every other junk shape.
Reject counts whose float conversion is not finite inside the existing
try, so they coerce to 0 and the run falls back to its legacy totals.
10**30 stays usable.
---------
Co-authored-by: gouyu-hou <gouyu-hou@users.noreply.github.com>
Co-authored-by: gouyu-hou <213296172+gouyu-hou@users.noreply.github.com>
* feat(runtime): add Queue and Steer for active Web runs
* style(frontend): format Queue and Steer controls
* fix(runtime): retain user Steer controls and order active inputs
* docs: add Chinese changelog entry for Queue and Steer
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): mask backend secrets in the memory config responses
GET /api/memory/config and /api/memory/status returned
memory.backend_config as loaded. Config loading has already resolved
$VAR references by then, so a DeerMem extraction model api_key or a
Honcho api_key reached every caller with memory:read in plaintext,
which with authorization disabled (the default) is every signed-in
user.
MemoryConfigResponse now masks credential-named values (api_key,
*_token, password, authorization, ...) at any depth as "***". The
key is matched on its end, so knobs such as token_counting and
guaranteed_token_budget and the api_key_env variable name stay
readable, and unset values stay null or empty. The loaded config the
backend uses is not changed.
* docs(changelog): reference #7665 for the memory config secret masking fix
* refactor(gateway): share config secret masking between MCP and memory
The memory config masking forked the MCP router's key normalization
and recursive masker. Both now use app/gateway/secret_masking.py, and
each router keeps its own key rule: MCP matches credential terms
anywhere in the key, memory matches them at the end and keeps unset
values. MCP masking is unchanged.
The memory rule now also covers plural and cookie keys (api_keys,
secrets, passwords, cookies). Plural tokens stays out because
max_injection_tokens is a DeerMem knob. A secret-named container is
still masked whole, since its keys can be arbitrary names; the helper
docstring and a test now say so.
* docs(changelog): move the #7665 entry to Unreleased
* feat(subagents): add thinking_enabled and reasoning_effort config for sub-agents (#4875)
(cherry picked from commit b1083728b23504cb0f1d1a463f8db70c5ec1c5a7)
* fix: resolve CI failures for subagent thinking config PR
- Fix UnboundLocalError in task_tool.py: initialize parent_thinking_enabled
and parent_reasoning_effort before the if runtime block
- Remove unused imports from test_subagent_thinking_config.py
(SimpleNamespace, pytest, resolve_subagent_model_name)
- Fix import sorting in subagents_config.py and test file
- Add reasoning_effort: None to executor test assertion
(cherry picked from commit 187851e307b871937e48bb67ae1b34b8b69bf92d)
* fix: address review comments - remove parent inheritance, supports_thinking clamp, Literal type
- Remove parent_thinking_enabled/parent_reasoning_effort from metadata
extraction in task_tool.py (backward compatible: subagents only think
when explicitly configured)
- Add supports_thinking clamp in _create_agent to prevent cross-model
mismatch errors
- Change SubagentConfig.reasoning_effort to Literal type
- Fix sandbox_state assignment that was accidentally removed
- Fix syntax errors in subagents_config.py (missing closing parens)
- Update all docstrings/comments from "inherit" to "no override"
- Simplify test helper and remove parent inheritance tests
(cherry picked from commit 74f9c659ef4a98bd1756109929ab0748fab82317)
* feat(subagents): adapt reasoning settings to current native workers
* fix(subagents): retain model context on reasoning rejection
* fix(subagents): preserve model-factory reasoning diagnostics
---------
Co-authored-by: Neon549 <neon549@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops
Loop-detection warnings now ask the agent to present the files it already
created in /mnt/user-data/outputs before its final answer, when
present_files is available.
A run that exhausts recursion_limit now ends with stop_reason
recursion_capped and a plain message that lists outputs it created but
did not present. An active goal stands down without an evaluator call.
The /wait error envelope keeps the GraphRecursionError name, and IM
channels reply with the step-limit message instead of a generic failure.
* docs(changelog): note run-end delivery and step-limit stops (#7588)
* test(runtime): stop pinning LangGraph's step count in the step-limit test
The produced paths and the message already follow the observed model
calls, so the exact count only pinned LangGraph's per-step accounting.
Also note on the step-limit goal branch that its caller passes no
evaluator factory, so it has to stay ahead of the evaluator section.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM binding codes on screen and report goal state in replies
Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a
dialog that keeps the /connect command on screen with a copy button, its
expiry time and the binding status, instead of a toast that disappears.
The bots' /connect replies and the IM /goal replies now use the owner's
web UI language (else channel_connections.notification_locale), and /goal
reports the goal's state as the web goal bar does: in progress, why it
stopped, or that it was met.
* docs(changelog): note the IM binding dialog and localized IM replies (#7591)
* test(channels): check IM reply tables against every notification locale
The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal
reply table had none. Both now match NOTIFICATION_LOCALES with the same
keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale
fails here instead of raising KeyError at reply time.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(auth): read the live config off the event loop on public auth routes
/register, /setup-status, /providers, /oauth/{provider} and
/callback/{provider} read the live auth config through get_app_config()
inline. That call stats and hashes config.yaml and extensions_config.json
on every request and reparses them after an edit, and these routes are
unauthenticated, so any client could keep that file IO on the event
loop. /login/local already reads its throttle policy in a worker.
Read the registration gate through asyncio.to_thread, and the SSO
routes' auth.oidc section through a shared _oidc_config() helper that
does the same.
* docs(changelog): reference #7576 for the off-loop auth route config reads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): return SSO sign-ins to their next path
The (auth) layout redirected every signed-in request to /workspace. The
Gateway sets the session cookie on the same redirect that lands on
/auth/callback?next=..., so the callback page never rendered and every SSO
sign-in landed on the workspace instead of the page that asked for it. A
signed-in visit to /login?next=... lost its target the same way.
Layouts cannot read searchParams, so the layout now renders the page for a
signed-in session and each page sends the user on: the callback and login to
the validated next path, setup to the workspace. The login page shows a
short wait message instead of the form while it redirects, and uses
router.replace so Back does not return to it.
* docs(changelog): reference #7467 for the SSO next-path fix
* fix(auth): reject control characters in the next redirect path
Browsers delete tab, LF and CR from a URL before parsing it, so a next of
"/\n//evil.example" passed both validators' "//" check and then navigated
to //evil.example. The login page's post-sign-in push already followed it,
and the signed-in login and SSO callback redirects reach it too.
validateAuthNextPath and validate_next_param now reject every C0 control
character and DEL. Percent-encoded forms are left alone: URL parsers do
not decode them, so they stay on the local origin.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): keep LF bytes in skill and agent text writes
Text mode translates a newline into os.linesep on write, so on Windows every
caller-supplied LF reached disk as CRLF while readers use universal newlines or
decode bytes explicitly. The difference stayed invisible: the API returned LF,
the file held CRLF, and a skill history record's prev_content described a
revision no API view could return (#7174).
Pass newline="" at the ten writers that persist or judge caller text: the local
and user-scoped skill writers, the skill history JSONL, the user skill-states
JSON, the SOUL.md/config.yaml staging in FileAgentStore, the managed-subagent
store, setup_agent's SOUL.md publish, the USER.md route, and the two temp copies
the frontmatter validator and the static scanner judge.
The tests emulate the Windows translation on io.open, so the regression runs on
the Linux CI shards instead of only on a Windows host: 10 failed before this
commit, 10 pass after.
* fix(skills): preserve candidate newlines during static scan
---------
Co-authored-by: Felix Shi <204130665+shi-felix@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome
* fix(gateway): adopt the family helper so a published edit stays silent
* fix(threads): reject an assistant_id that names no agent at thread creation
POST /api/threads stored assistant_id unchecked. Every state read then built
a run config from it, and build_run_config's ValueError answered 500 on GET
thread, state and history until the thread was deleted.
Validate it at creation with the rule run creation already applies, now
shared as custom_agent_name(), and answer 422. Records written before this
change are read with the lead agent's schema instead of failing.
* docs(changelog): reference #7445 for the thread-creation assistant_id fix
* fix(threads): clear invalid legacy assistant ids in branches
* docs(changelog): note the branch assistant_id behavior for #7445
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): let ordinary users save bundled personal connections
The personal MCP policy read `server.cwd`, which McpServerConfigResponse
does not declare. A bundled launcher definition (DingTalk, WeCom,
HubSpot) never carries `cwd`, so for a non-administrator the check
raised AttributeError and every create, edit or enable of one returned
500. The web UI installs catalog connections with scope "user", so an
ordinary user could not install any of the three.
Read a supplied `cwd` from the model's extras instead. A bundled
launcher with a working directory still requires an administrator.
* docs(changelog): reference #7400 for the bundled personal MCP fix
* docs(changelog): say #7400 makes bundled personal installs work, not work again
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): bound run event type filters
* test(gateway): cover run event filter cardinality
* style(gateway): group run event filter limit with module constants
Unsafe filenames and unnamed parts were dropped silently: the response kept
`success: true` with an empty `files` list, so the #7122 frontend gate never
blocked a chat send whose attachment had been rejected (#7160).
Both drop arms now record the rejected name in `skipped_files`, and `success`
follows `bool(uploaded_files)`: an all-rejected batch fails, a mixed batch
succeeds and lists what was dropped.
* fix(extensions): keep atomic-write cleanup from replacing the save error
Extension snapshot rollback, the managed plugins rewrite, and skill-archive
upload copies now use cleanup_failed_atomic_write so a failed unlink or
warning log cannot mask the original publication error.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
* fix(skills): keep upload and storage cleanup from replacing the save error
Route the skill-archive upload discard, custom-skill file writes, and
per-user skill-state writes through cleanup_failed_atomic_write. Close the
extension rollback descriptor on fdopen failure, and record the plugins
temporary path before the write so a short write is still removed.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
* fix(skills): keep form close and snapshot delete from masking the save error
Discard the uploaded skill archive before closing the multipart form, and
suppress close errors so a flush failure cannot replace the install outcome.
Snapshot rollback of a missing original now uses cleanup_failed_atomic_write,
and skill-state writes clean up on BaseException including KeyboardInterrupt.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
* test(extensions): assert logging cleanup cannot replace the delete-branch error
The log-handler check only ran for the unlink parameter, where the handler
is never patched. Move it under the logging failure so a handler exception
must stay out of the surfaced install error.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
---------
Co-authored-by: wannT-1 <200890097+wannT-1@users.noreply.github.com>
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix: honor checkpoint object in thread state updates
Prefer the checkpoint object over the deprecated checkpoint_id, validate
its thread_id, and preserve checkpoint namespace when updating state.
Add regression coverage for checkpoint-object forks and thread mismatch.
* fix: preserve client context handling in checkpoint PR
* fix: support state updates in checkpoint namespaces
* fix: reject unknown checkpoint namespaces
* fix(gateway): isolate subgraph titles from thread metadata
Only synchronize state titles into conversation metadata for the root checkpoint namespace. Cover persisted child updates with and without checkpoint IDs in full and delta modes, and retain root title synchronization and branch sequence cleanup.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): keep host paths out of the MCP config mutation 500 details
The five MCP config mutation routes answered an unexpected failure with
`f"Failed to …: {str(e)}"`. Each one runs a read-modify-write of
`extensions_config.json` (`_apply_mcp_config_update`,
`_apply_mcp_servers_create`, `_apply_mcp_server_config_update`,
`_apply_mcp_server_delete`, `_apply_mcp_server_state_update`), so the error the
handler catches is usually an `OSError` from `atomic_write_extensions_config`
that names the absolute host path — the `DEER_FLOW_HOME` layout of the server.
The custom-agent, user-profile and skills-read routes already return fixed text
for this reason (#7255; `routers/AGENTS.md`: "agent config parse errors and
`OSError`s carry host paths and stay in the server log"). These five were the
remaining MCP mutation 500s on the same file, so they now return the fixed
detail and keep the exception in the existing log line.
Red first, on the unmodified `main` — all five put the planted host path in
`detail`:
$ python -m pytest tests/test_mcp_config_error_details.py -q
5 failed in 6.37s
After the change, 5 passed; the wider MCP suites (246 tests across
`blocking_io/test_mcp_router.py`, `test_mcp_config_secrets.py`,
`test_mcp_lifecycle*.py`) stay green.
* fix(mcp): normalize personal config mutation errors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): keep OSError text out of upload, delete and attach 500s
Thread upload, upload delete and shelf attach-to-thread put str(exc) in
their 500 detail. A staging or unlink OSError names its file, so a
permission error, or a disk too full for mkstemp, returned the absolute
host path of the thread's uploads directory, user and thread IDs included.
Return a fixed detail that still names the caller's file. The existing
error log lines keep the full message for operators.
* docs(changelog): reference #7217 for the upload 500 detail fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): keep host paths out of agent and user-profile error details
The custom-agent and user-profile routes put str(e) in their 500
detail. Agent config parse errors and filesystem OSErrors name the
absolute host path under users/<user_id>/, so a caller saw it in the
response. Return the fixed "Failed to <action>" detail the import and
export routes already use and keep the full error in the existing log
lines. PUT also handles a parse error from its initial config load.
The scheduled-task assistant resolver echoed the same parse error as
its invalid_assistant 422 message, which reached the UI and the
scheduler tool result. It now logs the error and returns a fixed
message under the same code.
* docs(changelog): reference #7255 for the agents error detail fix
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(skills): create a missing extensions_config.json in the project root
When no extensions config resolved, the skill toggle created one at
Path.cwd().parent while the MCP writers used project_root(). Unless the
Gateway runs from backend/ with the project root at its parent, the
toggle's file landed outside the resolver's search path: the toggle was
lost on reload and the next MCP edit created a second file.
Add ExtensionsConfig.default_config_path() (the project root, the first
location resolve_config_path searches) and use it from the skill toggle
and both MCP create paths, so a created file is always the one the next
resolution reads back.
* docs(changelog): reference #7048 for the extensions config create path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): keep load error text out of user-facing skill read 500s
GET /api/skills, /api/skills/custom and /api/skills/{name} are open to
every signed-in user, but their 500 detail echoed str(e). A malformed or
unreadable extensions_config.json (re-read on every load by the
user-scoped storage) made all three return the file's absolute host path,
plus the [Errno 13] text for a permission error.
Return a fixed detail like the diagnostics and reload routes already do,
chain the cause, and keep the full error in the Gateway log.
* docs(changelog): reference #7155 for the skill read 500 detail fix
* test(skills): pin capability skill discovery 500 through the real Gateway app
GET /api/capabilities/installations/skills reads the same user-scoped
storage as the skill routes but has no handler of its own. Its 500 body
is TraceMiddleware's fixed fallback, so drive create_app() with a
malformed extensions_config.json and assert no host path reaches the
body on any of the four reads.
* fix(skills): answer invalid custom-skill history names with 400
GET /api/skills/custom/{skill_name}/history let the ValueError from
validate_skill_name fall through to its catch-all, so a malformed name
returned 500 and logged a traceback on every request, while delete and
rollback answer 400.
Validate the name up front instead of adding a handler-wide
except ValueError: a corrupt history line raises JSONDecodeError, also a
ValueError, and must stay a server error. Both outcomes are pinned.
* fix(skills): keep corrupt history a 500 on rollback, validate delete names early
Sibling sweep of the history fix. Rollback's handler-wide except ValueError
also caught json.JSONDecodeError, so a corrupt history record answered 400
with the parser message and no log; it is now a logged 500 like history.
Delete only failed an invalid name inside the drained mutation tail, after
taking the owner's mutation lock, and logged it as a tail error although
_drain_skill_mutation keeps caller 4xx unlogged. It now rejects the name
before any storage work. Each guard is pinned by a mutation-checked test.
* docs(changelog): reference #7123 for the skill history status fix
* fix(skills): treat non-UTF-8 skill history as a server fault on rollback
UnicodeDecodeError is a ValueError too, so a non-UTF-8 history file still
reached rollback's ValueError -> 400 mapping while GET .../history answered
the same file with 500. Catch it alongside JSONDecodeError and pin both
corruption shapes on both routes.
Also drop the whitespace-only separator lines the changelog merge added
between other entries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): run each IM message once across instances, restarts and late redeliveries
Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.
- Discord messages are keyed (discord, guild_id or "dm", posting channel,
message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
before a side effect; Discord claims it on the Gateway loop before
_create_thread, marks the published message dedupe_claimed, and gives the
claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
attempt made every keyed retry a 409). A reused run's status-only /wait or
stream gap is resolved to this message's own turn via threads.get_state;
409s about the message's own keyed run keep the key instead of reading as
a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
(claim_thread_id on channel_conversations and channel_thread_bindings);
the loser adopts the winner and deletes the empty thread it minted.
/new and /agent use still rebind.
Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).
* fix(channels): give back Discord dedupe claims for messages never processed
Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:
- The hand-off task releases the claim on cancellation too (stop() drains it
while it resolves the connection identity), via await_drained so the
cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
message that shutdown drops from the queue, after the grace period and on
a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
because stop() already closed cross-loop intake, or a hand-off cancelled
before it ran, leaves the claim tracked, and stop() releases it last.
* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation
Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.
- Keys from trusted internal callers (the IM channel manager) are scoped by
owner only (`internal-run:` prefix); browser and API keys keep the
owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
see the same type, and start_run answers it with 409 "Idempotency-Key
already used with a different request" instead of a 500. The channel
manager already treats that 409 as the message's admitted run.
Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): check the project before staging shelf uploads
The shelf upload and save-from-thread routes staged the body under
users/{user_id}/projects/{project_id}/ before checking the project. A
missing, foreign, or archived id left an empty {project_id}/documents/
directory that the retention sweep keeps forever, and an id longer than
one path component failed with ENAMETOOLONG (500).
Both routes now require an owned, active project first and answer the
same 404 without writing. The shelf insert still re-checks under the
project row lock.
* test(projects): pin the shelf insert's re-check behind the early project check
With the early check in place, no route test reached the insert's
row-lock re-check any more. Archive the project between the check and
the insert and assert both routes still answer 404 and leave no staging
file.
* docs(changelog): reference #7040 for the shelf upload project check
* docs(changelog): drop the whitespace-only line after the zh #7040 entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): share encrypted runtime channel config across Gateway instances
IM bot credentials entered in the web UI lived in each host's plaintext
channels/runtime-config.json, and the router cached the merged config on
app.state for the life of the process. With several Gateway instances a
save or disconnect served by an instance that does not hold the
provider's leader lease never reached the transport holder: it kept the
old credentials until it next started the channel, and a disconnect
could leave the old bot connected.
- New shared table channel_runtime_configs (migration 0044): one row per
provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY,
a plaintext disconnect tombstone, and a version bumped by every save.
- SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres
(memory keeps the file); it is synchronous on app_sync_sqlalchemy_url
because its callers already run in worker threads.
- The router re-reads the store on every request instead of caching the
merged config, and refuses (503) to save credentials without a key.
- The leader-lease supervisor compares the versions on each renewal tick
and restarts or stops the holder's transport when a provider's config
changed; a takeover reads a disconnect instead of reviving cached
credentials.
- The first start imports an existing runtime-config.json (rows already
in the table win) and deletes the plaintext file.
* docs(changelog): reference #6988 for the shared runtime channel config
* ci: rerun checks after #6999 fixed the frontend guidance budget on main
* fix(channels): start a held transport that is down when its runtime config changes
When the lease holder's transport was down (a start that failed on the
old credentials), applying a peer-saved config only updated the cached
config, and the transport started on the next lease round: two renewal
ticks instead of the one the docs promise. The apply step now starts it
on the held lease right away, with the lease round's backoff bookkeeping.
* fix(compaction): reject manual compaction of threads with pending execution
A thread paused at a static breakpoint (interrupt_before=["tools"]) keeps its
pending node send in the checkpoint's __pregel_tasks channel. The compaction
write goes through the state-mutation graph, which schedules no tasks, so
LangGraph drops the pending send ('Ignoring unknown node name tools in pending
sends') and a resumed run finishes success without ever executing the tool.
Fail closed: detect a non-empty __pregel_tasks in the raw checkpoint tuple
inside compact_thread_context and raise ContextCompactionPendingWork, mapped
to HTTP 409 by the compact route. The reservation (reserve_checkpoint_write)
already serializes this check against runs and other checkpoint writers, and
the write still forks from the exact head that was read.
Adds dual-mode (full/delta) regression tests: the rejected compaction leaves
checkpoint id, next and pending task ids untouched, and resuming afterwards
executes the pending tool exactly once; idle threads still compact.
* fix(compaction): reject unavailable checkpoints before summarizing
* docs: keep compaction guidance within merged size budget
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
