* fix(uploads): keep OSError text out of upload, delete and attach 500s
Thread upload, upload delete and shelf attach-to-thread put str(exc) in
their 500 detail. A staging or unlink OSError names its file, so a
permission error, or a disk too full for mkstemp, returned the absolute
host path of the thread's uploads directory, user and thread IDs included.
Return a fixed detail that still names the caller's file. The existing
error log lines keep the full message for operators.
* docs(changelog): reference #7217 for the upload 500 detail fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): keep host paths out of agent and user-profile error details
The custom-agent and user-profile routes put str(e) in their 500
detail. Agent config parse errors and filesystem OSErrors name the
absolute host path under users/<user_id>/, so a caller saw it in the
response. Return the fixed "Failed to <action>" detail the import and
export routes already use and keep the full error in the existing log
lines. PUT also handles a parse error from its initial config load.
The scheduled-task assistant resolver echoed the same parse error as
its invalid_assistant 422 message, which reached the UI and the
scheduler tool result. It now logs the error and returns a fixed
message under the same code.
* docs(changelog): reference #7255 for the agents error detail fix
* test: stabilize reporter evidence and guard missing chat nodes
* feat(models): scope Web model choices to tasks, turns and conversations
* test(e2e): mock model-control preview for ordinary chat submissions
* fix(models): harden scoped choices and stale Agent submissions
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(skills): create a missing extensions_config.json in the project root
When no extensions config resolved, the skill toggle created one at
Path.cwd().parent while the MCP writers used project_root(). Unless the
Gateway runs from backend/ with the project root at its parent, the
toggle's file landed outside the resolver's search path: the toggle was
lost on reload and the next MCP edit created a second file.
Add ExtensionsConfig.default_config_path() (the project root, the first
location resolve_config_path searches) and use it from the skill toggle
and both MCP create paths, so a created file is always the one the next
resolution reads back.
* docs(changelog): reference #7048 for the extensions config create path
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): treat only busy-thread 409s as retryable
start_run exposes the retryable run-manager admission conflict as BusyThreadConflict (an HTTP 409), but also uses plain 409s for permanent idempotency-key and thread/assistant mismatches. The scheduler treated every HTTP 409 as an overlap conflict and requeued the occurrence, so a permanent launch error was retried until scheduler.queue_timeout_seconds elapsed. Other launch errors are documented to fail immediately, and the MCP notification launcher already distinguishes these types.
Match ConflictError and BusyThreadConflict only; a plain 409 now follows the normal failure path. The type is imported lazily so the scheduler module does not take a load-time dependency on app.gateway.services.
Tests: a plain idempotency 409 terminalizes the occurrence as failed, while BusyThreadConflict remains queued; existing ConflictError coverage is unchanged.
* fix(scheduler): document the busy-thread import boundary and pin the failure detail
Keep the BusyThreadConflict import local to the classifier and document why it is guaranteed to resolve: the scheduler is wired by the Gateway, where app.gateway.services is already loaded, and this must not become an optional dependency. Import the same type locally in the test so collection does not initialize gateway services for the rest of the scheduler suite. Also pin the persisted failure detail so the 409 reason stays operator-visible.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): pin JSON state encoding across platforms
* test(channels): check encoding pins at each state I/O call
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): keep load error text out of user-facing skill read 500s
GET /api/skills, /api/skills/custom and /api/skills/{name} are open to
every signed-in user, but their 500 detail echoed str(e). A malformed or
unreadable extensions_config.json (re-read on every load by the
user-scoped storage) made all three return the file's absolute host path,
plus the [Errno 13] text for a permission error.
Return a fixed detail like the diagnostics and reload routes already do,
chain the cause, and keep the full error in the Gateway log.
* docs(changelog): reference #7155 for the skill read 500 detail fix
* test(skills): pin capability skill discovery 500 through the real Gateway app
GET /api/capabilities/installations/skills reads the same user-scoped
storage as the skill routes but has no handler of its own. Its 500 body
is TraceMiddleware's fixed fallback, so drive create_app() with a
malformed extensions_config.json and assert no host path reaches the
body on any of the four reads.
* fix(channels): keep staging cleanup from replacing the save error
The channel stores publish state through a staged temporary file and, if the
rename fails, close and unlink that file inside the handler before re-raising.
Neither step was guarded, so a failing close skipped the unlink and a failing
unlink replaced the error that actually caused the save to fail -- leaving the
staged copy, which for the runtime-config and WeChat stores holds an 0600 IM
token, behind with nothing reporting it.
Make both steps independently best-effort, name the leaked file in a WARNING,
and suppress the diagnostic itself so a broken log handler cannot become the
reported failure. This is the same contract PR #6871 establishes for browser
captures, applied to the three JSON publishers in app/channels.
* fix(channels): address staging cleanup review follow-ups
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): answer invalid custom-skill history names with 400
GET /api/skills/custom/{skill_name}/history let the ValueError from
validate_skill_name fall through to its catch-all, so a malformed name
returned 500 and logged a traceback on every request, while delete and
rollback answer 400.
Validate the name up front instead of adding a handler-wide
except ValueError: a corrupt history line raises JSONDecodeError, also a
ValueError, and must stay a server error. Both outcomes are pinned.
* fix(skills): keep corrupt history a 500 on rollback, validate delete names early
Sibling sweep of the history fix. Rollback's handler-wide except ValueError
also caught json.JSONDecodeError, so a corrupt history record answered 400
with the parser message and no log; it is now a logged 500 like history.
Delete only failed an invalid name inside the drained mutation tail, after
taking the owner's mutation lock, and logged it as a tail error although
_drain_skill_mutation keeps caller 4xx unlogged. It now rejects the name
before any storage work. Each guard is pinned by a mutation-checked test.
* docs(changelog): reference #7123 for the skill history status fix
* fix(skills): treat non-UTF-8 skill history as a server fault on rollback
UnicodeDecodeError is a ValueError too, so a non-UTF-8 history file still
reached rollback's ValueError -> 400 mapping while GET .../history answered
the same file with 500. Catch it alongside JSONDecodeError and pin both
corruption shapes on both routes.
Also drop the whitespace-only separator lines the changelog merge added
between other entries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): run each IM message once across instances, restarts and late redeliveries
Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.
- Discord messages are keyed (discord, guild_id or "dm", posting channel,
message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
before a side effect; Discord claims it on the Gateway loop before
_create_thread, marks the published message dedupe_claimed, and gives the
claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
attempt made every keyed retry a 409). A reused run's status-only /wait or
stream gap is resolved to this message's own turn via threads.get_state;
409s about the message's own keyed run keep the key instead of reading as
a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
(claim_thread_id on channel_conversations and channel_thread_bindings);
the loser adopts the winner and deletes the empty thread it minted.
/new and /agent use still rebind.
Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).
* fix(channels): give back Discord dedupe claims for messages never processed
Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:
- The hand-off task releases the claim on cancellation too (stop() drains it
while it resolves the connection identity), via await_drained so the
cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
message that shutdown drops from the queue, after the grace period and on
a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
because stop() already closed cross-loop intake, or a hand-off cancelled
before it ran, leaves the claim tracked, and stop() releases it last.
* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation
Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.
- Keys from trusted internal callers (the IM channel manager) are scoped by
owner only (`internal-run:` prefix); browser and API keys keep the
owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
see the same type, and start_run answers it with 409 "Idempotency-Key
already used with a different request" instead of a 500. The channel
manager already treats that 409 as the message's admitted run.
Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): check the project before staging shelf uploads
The shelf upload and save-from-thread routes staged the body under
users/{user_id}/projects/{project_id}/ before checking the project. A
missing, foreign, or archived id left an empty {project_id}/documents/
directory that the retention sweep keeps forever, and an id longer than
one path component failed with ENAMETOOLONG (500).
Both routes now require an owned, active project first and answer the
same 404 without writing. The shelf insert still re-checks under the
project row lock.
* test(projects): pin the shelf insert's re-check behind the early project check
With the early check in place, no route test reached the insert's
row-lock re-check any more. Archive the project between the check and
the insert and assert both routes still answer 404 and leave no staging
file.
* docs(changelog): reference #7040 for the shelf upload project check
* docs(changelog): drop the whitespace-only line after the zh #7040 entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): fail durable tasks whose connection no longer resolves
A personal MCP runtime name is derived from the server's complete definition, and a deployment task server belongs to the Gateway startup snapshot. Editing, disabling, or deleting a personal server while a durable task is in flight therefore makes the persisted server name unresolvable. McpTaskToolCaller raised LookupError, which the service classifies as transient: status polls and cancellation retried forever, leaving the task stuck in working and impossible to cancel.
Raise a dedicated McpTaskConnectionUnavailableError instead, and terminalize it: status polls persist failed with the connection error and stop; a pending cancellation persists failed locally because the remote task can no longer be reached. Transient status/cancel errors keep their existing capped-backoff retry policy.
Tests: new service tests pin both terminal paths, and caller tests pin the exception for missing deployment and stale personal server names.
* fix(mcp): preserve LookupError compatibility for connection errors
McpTaskConnectionUnavailableError now subclasses LookupError instead of RuntimeError. The four existing personal-MCP assertions in test_personal_mcp.py still treat a missing or stale connection as a lookup failure; keeping that base preserves their behavior and avoids breaking callers that caught the previous type, while the task service still catches this exact type for permanent-failure classification.
Add a compatibility test and rerun the personal-MCP and task suites.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* feat: support client context capabilities
(cherry picked from commit b7c223f9415dcd545cbe05a5f5a6e47073a42d68)
[Totoro-qaq: resolved onto main. The persisted-reminder middleware hunks and their tests are dropped (delivery is redesigned in a follow-up); the RunCreateRequest description moved to app/gateway/run_models.py.]
* fix: address client context review comments
(cherry picked from commit bce6da6b819228eda58e887872720b6f9b5fab69)
[Totoro-qaq: middleware hunks dropped with the redesigned delivery.]
* feat(agents): deliver client context as a request-scoped data block
The cherry-picked #3106 put context.client into the persisted first-turn
reminder, with change detection and a "not provided" clear marker. Main
has since moved the date reminder to a SystemMessage, made request-scoped
context (the project block, #5443) request-only, and agents/AGENTS.md now
requires user-influenced values to ride the HumanMessage data channel,
never system text.
Render runtime.context["client"] on every model call in
DynamicContextMiddleware.wrap_model_call/awrap_model_call as one hidden
HumanMessage right after the leading SystemMessages, where DurableContext
puts its data. It is never a state update and never flagged
dynamic_context_reminder, so nothing is checkpointed, summarized or
replayed, and a run without a client gets no block. An unchanged client
keeps the previous request as the next request's prefix.
The renderer sanitizes again (direct graph callers bypass the Gateway),
html-escapes values and frames them with frame_untrusted_text: the same
--- BEGIN/END USER INPUT --- boundary the user's own turn and the MCP task
notification get. The lead prompt treats everything outside those markers
as framework data, so unframed values would rank above the user's own
text; framed, context.client can say nothing the same caller could not
already say in input.messages. client_context joins _BLOCKED_TAG_NAMES so
a forged block in user input or a tool result is escaped, and one static,
value-free sentence in the lead prompt names the block as untrusted hints,
never instructions, permissions or identity. The now unused empty-context
marker is removed.
* fix(gateway): log dropped context.client fields without values
A run request is never rejected over context.client: unknown fields,
nested values and malformed entries are silently dropped, the same
leniency as the other context keys. That left no signal when a custom
client sent something the Gateway ignored (review on #3106).
Route both the config.context.client and the body context.client paths
through one helper that emits a debug line whenever sanitizing changed
the value. Like the malformed client_timezone branch, the line carries no
values; unlike client_timezone, client.name and preference strings are
free caller-controlled text, so they are never logged, not even
sanitized. A clean client, an explicit null or an absent key logs
nothing.
Tests also pin the config-only path (sanitized before the early return)
and the end-to-end start_run admission: the sanitized client reaches
config["context"] and never configurable.
* fix(runtime): bound numeric client context preferences
String preferences are capped at 200 characters, but integers were kept
at any size. A JSON body may carry integers of up to 4300 digits, so 32 of
them rendered a block of about 138,000 characters, re-sent on every model
call of the run. In-process callers could also pass an int past the str()
digit limit, which failed the model call, or a number subclass with its
own __str__.
Keep integers within JavaScript's Number.MAX_SAFE_INTEGER, the range a
browser client can send exactly, and coerce numbers to plain int and
float.
* docs(api): describe client context as advisory request-scoped hints
The cherry-picked section described a hidden reminder that no longer
exists. Document what a custom client can rely on: the allow-list and its
limits (strings, the integer range, and only the first 32 entries of each
map), debug-level logging of dropped fields, runtime-only forwarding, and
delivery as a per-request user-role block, framed like user input, rather
than system text or thread history.
Only top-level context.client is documented. An explicit
config.context.client is sanitized for the run as well, but the run record
keeps the caller's config as sent, so it is not offered as an alternative.
It also answers the #3106 review: capabilities and preferences are
advisory hints, not enforcement, which belongs outside the model; name is
informational and never for authorization; the same object should be
sent on every run to keep the cached prefix stable; subagents and
embedded DeerFlowClient runs do not receive it, and server-started runs
(schedules, IM channels, GitHub agents, MCP notifications, extensions
unless the plugin passes one) carry none.
* fix(context): declare client delivery and keep client out of configurable
The request-only <client_context> block is model-visible, so
DynamicContextMiddleware now declares its delivery and the sanitizer's
limits in release_policy_parameters(); runs before and after this change
no longer share a policy fingerprint.
build_run_config copies body.config['configurable'] verbatim and
configurable is persisted with checkpoints, so a client placed there is
dropped instead of being stored raw. Client context stays runtime-only.
Document context.client in the README and in the middleware guide.
* fix(context): apply the PII redaction policy to the client block
DynamicContextMiddleware adds the <client_context> message inside the
outer PiiRedactionMiddleware, and the hidden message is not re-sanitized,
so with pii_redaction enabled a preference such as an email address
reached the model verbatim while the same address in the user turn was
redacted. Pass the rendered block through the shared redact_text seam
with this middleware's app_config.pii_redaction, and declare the setting
in release_policy_parameters().
* fix(context): keep client hints out of stored run config
---------
Co-authored-by: Blair Leng <25675774+BlairLeng@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(channels): share encrypted runtime channel config across Gateway instances
IM bot credentials entered in the web UI lived in each host's plaintext
channels/runtime-config.json, and the router cached the merged config on
app.state for the life of the process. With several Gateway instances a
save or disconnect served by an instance that does not hold the
provider's leader lease never reached the transport holder: it kept the
old credentials until it next started the channel, and a disconnect
could leave the old bot connected.
- New shared table channel_runtime_configs (migration 0044): one row per
provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY,
a plaintext disconnect tombstone, and a version bumped by every save.
- SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres
(memory keeps the file); it is synchronous on app_sync_sqlalchemy_url
because its callers already run in worker threads.
- The router re-reads the store on every request instead of caching the
merged config, and refuses (503) to save credentials without a key.
- The leader-lease supervisor compares the versions on each renewal tick
and restarts or stops the holder's transport when a provider's config
changed; a takeover reads a disconnect instead of reviving cached
credentials.
- The first start imports an existing runtime-config.json (rows already
in the table win) and deletes the plaintext file.
* docs(changelog): reference #6988 for the shared runtime channel config
* ci: rerun checks after #6999 fixed the frontend guidance budget on main
* fix(channels): start a held transport that is down when its runtime config changes
When the lease holder's transport was down (a start that failed on the
old credentials), applying a peer-saved config only updated the cached
config, and the transport started on the next lease round: two renewal
ticks instead of the one the docs promise. The apply step now starts it
on the held lease right away, with the lease round's backoff bookkeeping.
* fix(compaction): reject manual compaction of threads with pending execution
A thread paused at a static breakpoint (interrupt_before=["tools"]) keeps its
pending node send in the checkpoint's __pregel_tasks channel. The compaction
write goes through the state-mutation graph, which schedules no tasks, so
LangGraph drops the pending send ('Ignoring unknown node name tools in pending
sends') and a resumed run finishes success without ever executing the tool.
Fail closed: detect a non-empty __pregel_tasks in the raw checkpoint tuple
inside compact_thread_context and raise ContextCompactionPendingWork, mapped
to HTTP 409 by the compact route. The reservation (reserve_checkpoint_write)
already serializes this check against runs and other checkpoint writers, and
the write still forks from the exact head that was read.
Adds dual-mode (full/delta) regression tests: the rejected compaction leaves
checkpoint id, next and pending task ids untouched, and resuming afterwards
executes the pending tool exactly once; idle threads still compact.
* fix(compaction): reject unavailable checkpoints before summarizing
* docs: keep compaction guidance within merged size budget
* test(runtime): cover the driver suffix in sync and store DSNs
* fix(runtime): strip the SQLAlchemy driver suffix from sync and store DSNs
With `postgres_url: postgresql+asyncpg://...` and the default empty
`postgres_schema`, the sync checkpointer, the sync store and the async
store passed the DSN to psycopg's `from_conn_string` unchanged, and
libpq cannot parse the `+asyncpg` scheme. `dsn_with_search_path` only
drops the suffix when it adds a schema, so a non-empty schema hid the
bug. The async checkpointer already calls `normalize_libpq_dsn` first.
Call `normalize_libpq_dsn` at these three call sites as well.
* refactor(persistence): drop the driver suffix inside dsn_with_search_path
dsn_with_search_path dropped a SQLAlchemy `+driver` suffix only when it
added a search_path, so every libpq caller had to wrap the DSN in
normalize_libpq_dsn to cover an empty schema, and flattening that
nesting later would bring the bug back.
Normalize at the top of dsn_with_search_path so both paths drop the
suffix and check the scheme in one place, and remove the outer call
from its five callers. The sync checkpointer and both stores are back
to their code on main.
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(slack): accept file_share events and download inbound files
Slack was dropping every Events API message with a subtype, including
user uploads (subtype=file_share). Allowlist file_share, attach event
files to the inbound message, and download private URLs in receive_file
with the bot Bearer token (bytes via INBOUND_FILE_CONTENT_KEY).
Fixes#6853
* fix(slack): tighten inbound file download guards
Address review on #6855: allowlist https files.slack.com before Bearer
fetch, cap aggregate in-memory bytes per file_share message, report
Content-Encoding refusals as download failed, and share one
_web_client_for_connection helper for inbound/outbound.
* fix(slack): satisfy ruff format and AGENTS.md size gate
Reformat test_slack_file_share.py and drop the AGENTS.md file_share
note so channels/AGENTS.md stays under the 49152-byte hard limit
(file was already near the ceiling without it).
* fix(discord): accept inbound attachments and file-only messages
Empty content no longer drops messages that have attachments. Capture
CDN descriptors on inbound, download in receive_file via Bot auth, and
hand bytes through INBOUND_FILE_CONTENT_KEY like Slack/Telegram.
Closes#6856
* fix(discord): gate bot auth and share inbound file helpers
Address review on #6857: attach Bot Authorization only for Discord CDN
hosts, report Content-Encoding refusals as download failed, and hoist
filename/size helpers into app.channels.inbound_files (Telegram delegates
to the same module).
* fix(discord): sort telegram imports for ruff I001
CI lint-backend failed on unsorted imports in telegram.py after the
shared inbound_files helper was introduced.
* style: ruff-format discord inbound attachments test
* fix(discord): show the enforced attachment download limit
---------
Co-authored-by: Li-john1021 <Li-john1021@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM runs alive when the creating request disconnects
The channel manager holds a `runs.wait` / `runs.stream` request open for
the whole run, and that request dies with the Gateway instance that
received the IM message: a leader hand-off, a rolling restart, a crash.
The Gateway cancels a run whose creating request disconnects unless the
request asked for `on_disconnect: continue`, so every IM run in flight
was cancelled on failover.
Channel runs now ask for `continue`, like scheduled-task and MCP-task
runs already do. The run finishes on the Gateway executing it and its
thread state is persisted, so the answer stays readable in the web UI
even though the IM reply is lost with the old transport. The key is set
after the fire-and-forget branch because the SDK's `runs.create` has no
`on_disconnect` parameter.
Docs: CONFIGURATION.md and the Helm/deployment guides no longer list
"failover cancels IM runs" as a known limit; IM_CHANNEL_CONNECTIONS.md
and the channels AGENTS.md describe the policy.
* docs(changelog): link #6876
* fix(channels): keep the inbound dedupe key once a run may be executing
With `on_disconnect: continue` a run outlives the manager worker that
requested it, but the worker loop released the inbound dedupe key on
cancellation, so a provider redelivery on the replacement instance
admitted the same event again and executed the turn twice (review on
#6876).
Each inbound message now carries an admission record: `requested`
flips before the run request leaves the manager, `confirmed` once the
Gateway proved it admitted the run (a run id, a stream frame, a
completed response). The generic release paths (worker cancellation,
worker/handler exceptions, a broken stream) keep the key when the run
may continue: after confirmation, on cancellation after the request
was sent, and on post-request transport failures (read timeout, read
error, protocol error). Pre-request failures, Gateway rejections
(busy thread, stale pin, status errors, connection failures) and the
stream's terminal `error` frame still release it so the redelivery
can retry.
* fix(buzz): page a saturated history instead of skipping the oldest
buzz-relay serves at most RELAY_HISTORY_LIMIT (2000) stored events per
subscription, NEWEST-FIRST, even when the filter carries a `since`. A channel
that banked more than the cap across one disconnect therefore had its OLDEST
unread messages neither sent nor marked: the live page delivered the newest
2000, the watermark advanced past them, and the rest were lost -- the one
remaining path in this connector that could skip a message outright.
A chat/backlog page's EOSE now drives a descending `until` walk. When a page
comes back AT the cap it may have older matches held back, so the connector
re-asks with `until` set to that page's oldest `created_at` -- on a SEPARATE
`buzz-backlog-<uuid>` subscription id, never the live one -- and repeats until
a page comes back under the cap or reaches the `since` the live REQ was sent
with, bounded by MAX_BACKLOG_PAGES (50, warned when hit). Delivery is charged
to the subscription that carried it before any gate, so a page's saturation is
measured against what the relay sent, not what the allowlist/mention gates kept.
The walk is strictly additive: the live subscription, its filter, and the
watermark are never touched; a CLOSED for a backlog page only ends that walk;
re-opening the live subscription closes any superseded page rather than
orphaning it; and every failure simply leaves the pre-existing behaviour for the
next reconnect to retry.
* fix(buzz): skip the backlog walk when the channel has no read floor
Reviewer willem-bd noted the descending-until walk has no semantic stop when
`floor is None`: that is exactly a channel's first connect (no stored
watermark) and again after a MAX_CACHED_CHANNELS eviction of its cursor. On
those connects the walk would fetch up to MAX_BACKLOG_PAGES * RELAY_HISTORY_LIMIT
events per channel instead of the newest 2000 the connector used to take.
Require a floor before opening any backlog page. The backlog-loss case this walk
exists for always has a watermark (the cursor survives reconnects and restarts),
so it always has a floor; first connect and an evicted cursor keep the previous
behaviour. Covers the skip with a new test, retargets the page-bound test to a
real walk above its floor, and documents the floor requirement.
* fix(buzz): stop orphaned and stalled backlog walks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): control IM notifications and add WeChat/QQ delivery
* fix(scheduler): skip notification preference locks without recipients
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): parse Slack and DingTalk allowed_users fail-closed
Slack and DingTalk still built their allowlists with their own
_normalize_allowed_users: every entry went through str() without
stripping, only list/tuple/set counted as collections, and the gate was
`if self._allowed_users and ...`. A configured list whose entries are all
blank, such as `allowed_users: [$SLACK_USER_ID]` with the variable set to
an empty string, became an empty set and let every sender through.
" U123 " never matched U123, and a frozenset was stringified into one ID.
Parse both with the shared parse_allowed_users helper that Telegram,
WeChat and QQ use, and gate on `is not None`: unset, null, empty
collections and a blank string allow everyone; a scalar is one entry;
strings are stripped and integers keep matching their string form, as
before; booleans, floats, None and mappings are rejected (a mapping as a
whole); a configured value with no valid ID denies every sender and logs
an error. A scalar containing commas or whitespace warns that it stays
one literal ID.
* refactor(channels): share the Slack and DingTalk allowlist coercer and separator warning
Slack and DingTalk had byte-identical ID coercers and their own copies of
the scalar-separator warning. Move the coercer into allowed_users.py as
parse_string_or_int_user_id and opt both channels into
warn_on_scalar_separators, as QQ and WeChat do. Log messages are unchanged.
Pin that a separator-only string such as "," stays one literal ID
(denying everyone else) and still warns.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): merge runtime channel config saves across Gateway workers
Each Gateway worker builds its own ChannelRuntimeConfigStore, loads
runtime-config.json once, and rewrote the whole file from that copy on
every save. With two workers, a bot saved on one worker was dropped when
another bot was saved on the other, and a disconnected bot came back.
Every save, disconnect and removal now rereads the file while holding the
thread lock and an advisory sidecar file lock shared across processes,
changes only its own provider, and saves. The served snapshot is
refreshed only after a successful save, and an unreadable file still
falls back to this process's copy as before.
* docs(changelog): reference #6844 for the runtime channel config fix
* fix(channels): normalize the runtime config lock path like extensions_config
The sidecar lock was derived from the store path as given, so a worker
configured with a symlinked spelling of runtime-config.json took a
different lock file and no longer excluded peers using the real path.
Resolve a symlinked path, and make any other path absolute, before
deriving the sidecar, as extensions_config_file_lock already does.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(buzz): treat a scalar allowed_users entry as one member pubkey
BuzzChannel built its allowlist by iterating config.get("allowed_users") directly. A scalar string -- what an allowed_users: $BUZZ_OWNER_PUBKEY reference resolves to -- was iterated character by character, feeding each character to parse_pubkey, whose first non-hex character raised and aborted channel construction; a numeric scalar raised TypeError the same way. The sibling QQ (#6730) and Discord (#6776) connectors already treat a scalar as a single entry.
Parse the value with the shared parse_allowed_users helper via a parse_pubkey coercer: a scalar is one entry, list entries keep their per-entry validation, unusable entries (bad hex, numeric refs, mappings) are dropped with the shared warning instead of crashing, and the deny-by-default contract is unchanged -- an empty or blank value still yields an empty allowlist with start()'s existing deny-all warning.
config.example.yaml states the accepted shape next to the key, matching the mention_free_channels note added by #6776.
* docs(config): split buzz allowed_users note into aligned comment lines
The two-line note on the buzz example entry was emitted with a literal
`/n` where a newline was intended, so the second note never became its
own comment line and the `#`-column alignment shared with the
DENY-BY-DEFAULT lines below was broken. Split it to match the
mention_free_channels sibling shape (review comment on #6811).
* fix(channels): make Feishu and DingTalk transports stoppable in-process
FeishuChannel.stop() and DingTalkChannel.stop() could not end their SDK
threads. Both SDKs run on a thread-private event loop that only returned
on a fatal error: lark-oapi's ws.Client.start() parks forever in
run_until_complete(_select()) and has no public stop, and the channel
never kept the client handle; dingtalk-stream's start_forever() is a
while-True loop that also catches CancelledError, and the disconnect()
the channel called does not exist on the SDK. So every stop joined for
5s, raised ChannelStopTimeout, and ChannelService retained the instance
while the platform connection stayed alive on the old process. That
burns the full budget on shutdown and on POST /api/channels/{name}/restart,
and it blocks the per-provider leader lease: a demoted leader could not
release the transport before a peer took over.
Both channels now publish the SDK thread's loop and client under a lock;
stop() schedules a shutdown on that loop and then joins. Feishu disarms
lark's _auto_reconnect before closing the socket and stopping the loop,
because start() otherwise swallows the interrupted run_until_complete in
its reconnect branch and dials again. DingTalk drives its own copy of the
SDK connect loop that re-checks the running flag before every dial, runs
the blocking ticket request off the loop, cancels keepalive with the
socket, and logs the endpoint rather than the ticket-bearing payload. A
shared shutdown_private_loop cancels the SDK's leftover tasks with a
bounded drain and closes the loop; a stop requested before the client
exists never dials. ChannelStopTimeout now only fires for a thread that
genuinely wedges inside an SDK call. Telegram already stopped this way.
Tests: a fake lark ws.Client with the SDK's start() shape (including the
reconnect branch that swallows a bare loop stop), a real local websockets
server that observes the DingTalk close, the stop-before-client race for
both, and a blocking-io anchor that stop() ends a live private loop
without blocking the Gateway loop. All eight were red on main.
* docs(changelog): link the stoppable-transports entry to #6798
* fix(channels): release the socket on a stalled close and bound the DingTalk ticket request
Review follow-ups on #6798 (Totoro-qaq):
Feishu: the bounded _disconnect() wait cancelled websockets' close before
its own close-timeout abort ran, and lark clears _conn in its finally, so
a peer that never answers the close frame left the socket to garbage
collection. Worse, a stop landing while _connect() still held lark's
_lock snapshotted _conn as None. _shutdown_ws now settles the in-flight
dial by taking that lock under one shared close deadline, snapshots the
connection, runs _disconnect() under the same deadline, and aborts the
transport itself (abort_connection_transport in base.py) when the
handshake did not finish, yielding once so connection_lost runs before
loop.stop(). DingTalk's websocket.close() path gets the same abort.
DingTalk: the SDK's open_connection() calls requests.post with no
timeout, and running it in an executor let a stalled ticket endpoint
block interpreter exit (executor workers are joined at shutdown). The
channel now sends the same request through httpx on the stream loop
under a 30s deadline, so stop() cancels it and no worker thread lingers.
Tests drive the real SDK clients against loopback peers: a WebSocket
peer that completes the handshake and never ACKs the close (both
channels assert the connection's socket reports fileno() == -1 right
after stop(); red with the abort disabled), a real websockets server for
the clean path, and a loopback ticket server (ok / 500 / never responds)
with a no-lingering-non-daemon-thread check and a request-shape check.
* fix(buzz): accept a single mention-free channel id
`_mention_free` iterated the configured value directly, so one unquoted
UUID became a set of its characters and a numeric or boolean scalar
raised TypeError while the channel was being constructed. Both shapes are
documented as valid for the sibling Discord connector.
Parse the key through a dedicated helper that follows the Discord
convention: a scalar is a single entry, list entries are kept verbatim,
and an unusable entry never matches so `require_mention` stays in force.
* fix(channels): share scalar-or-list channel ID parsing
* docs(channels): keep guidance within CI size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
