* fix(mcp): fail durable tasks whose connection no longer resolves
A personal MCP runtime name is derived from the server's complete definition, and a deployment task server belongs to the Gateway startup snapshot. Editing, disabling, or deleting a personal server while a durable task is in flight therefore makes the persisted server name unresolvable. McpTaskToolCaller raised LookupError, which the service classifies as transient: status polls and cancellation retried forever, leaving the task stuck in working and impossible to cancel.
Raise a dedicated McpTaskConnectionUnavailableError instead, and terminalize it: status polls persist failed with the connection error and stop; a pending cancellation persists failed locally because the remote task can no longer be reached. Transient status/cancel errors keep their existing capped-backoff retry policy.
Tests: new service tests pin both terminal paths, and caller tests pin the exception for missing deployment and stale personal server names.
* fix(mcp): preserve LookupError compatibility for connection errors
McpTaskConnectionUnavailableError now subclasses LookupError instead of RuntimeError. The four existing personal-MCP assertions in test_personal_mcp.py still treat a missing or stale connection as a lookup failure; keeping that base preserves their behavior and avoids breaking callers that caught the previous type, while the task service still catches this exact type for permanent-failure classification.
Add a compatibility test and rerun the personal-MCP and task suites.
---------
Co-authored-by: Tong-bit-art <Tong-bit-art@users.noreply.github.com>
* feat: support client context capabilities
(cherry picked from commit b7c223f9415dcd545cbe05a5f5a6e47073a42d68)
[Totoro-qaq: resolved onto main. The persisted-reminder middleware hunks and their tests are dropped (delivery is redesigned in a follow-up); the RunCreateRequest description moved to app/gateway/run_models.py.]
* fix: address client context review comments
(cherry picked from commit bce6da6b819228eda58e887872720b6f9b5fab69)
[Totoro-qaq: middleware hunks dropped with the redesigned delivery.]
* feat(agents): deliver client context as a request-scoped data block
The cherry-picked #3106 put context.client into the persisted first-turn
reminder, with change detection and a "not provided" clear marker. Main
has since moved the date reminder to a SystemMessage, made request-scoped
context (the project block, #5443) request-only, and agents/AGENTS.md now
requires user-influenced values to ride the HumanMessage data channel,
never system text.
Render runtime.context["client"] on every model call in
DynamicContextMiddleware.wrap_model_call/awrap_model_call as one hidden
HumanMessage right after the leading SystemMessages, where DurableContext
puts its data. It is never a state update and never flagged
dynamic_context_reminder, so nothing is checkpointed, summarized or
replayed, and a run without a client gets no block. An unchanged client
keeps the previous request as the next request's prefix.
The renderer sanitizes again (direct graph callers bypass the Gateway),
html-escapes values and frames them with frame_untrusted_text: the same
--- BEGIN/END USER INPUT --- boundary the user's own turn and the MCP task
notification get. The lead prompt treats everything outside those markers
as framework data, so unframed values would rank above the user's own
text; framed, context.client can say nothing the same caller could not
already say in input.messages. client_context joins _BLOCKED_TAG_NAMES so
a forged block in user input or a tool result is escaped, and one static,
value-free sentence in the lead prompt names the block as untrusted hints,
never instructions, permissions or identity. The now unused empty-context
marker is removed.
* fix(gateway): log dropped context.client fields without values
A run request is never rejected over context.client: unknown fields,
nested values and malformed entries are silently dropped, the same
leniency as the other context keys. That left no signal when a custom
client sent something the Gateway ignored (review on #3106).
Route both the config.context.client and the body context.client paths
through one helper that emits a debug line whenever sanitizing changed
the value. Like the malformed client_timezone branch, the line carries no
values; unlike client_timezone, client.name and preference strings are
free caller-controlled text, so they are never logged, not even
sanitized. A clean client, an explicit null or an absent key logs
nothing.
Tests also pin the config-only path (sanitized before the early return)
and the end-to-end start_run admission: the sanitized client reaches
config["context"] and never configurable.
* fix(runtime): bound numeric client context preferences
String preferences are capped at 200 characters, but integers were kept
at any size. A JSON body may carry integers of up to 4300 digits, so 32 of
them rendered a block of about 138,000 characters, re-sent on every model
call of the run. In-process callers could also pass an int past the str()
digit limit, which failed the model call, or a number subclass with its
own __str__.
Keep integers within JavaScript's Number.MAX_SAFE_INTEGER, the range a
browser client can send exactly, and coerce numbers to plain int and
float.
* docs(api): describe client context as advisory request-scoped hints
The cherry-picked section described a hidden reminder that no longer
exists. Document what a custom client can rely on: the allow-list and its
limits (strings, the integer range, and only the first 32 entries of each
map), debug-level logging of dropped fields, runtime-only forwarding, and
delivery as a per-request user-role block, framed like user input, rather
than system text or thread history.
Only top-level context.client is documented. An explicit
config.context.client is sanitized for the run as well, but the run record
keeps the caller's config as sent, so it is not offered as an alternative.
It also answers the #3106 review: capabilities and preferences are
advisory hints, not enforcement, which belongs outside the model; name is
informational and never for authorization; the same object should be
sent on every run to keep the cached prefix stable; subagents and
embedded DeerFlowClient runs do not receive it, and server-started runs
(schedules, IM channels, GitHub agents, MCP notifications, extensions
unless the plugin passes one) carry none.
* fix(context): declare client delivery and keep client out of configurable
The request-only <client_context> block is model-visible, so
DynamicContextMiddleware now declares its delivery and the sanitizer's
limits in release_policy_parameters(); runs before and after this change
no longer share a policy fingerprint.
build_run_config copies body.config['configurable'] verbatim and
configurable is persisted with checkpoints, so a client placed there is
dropped instead of being stored raw. Client context stays runtime-only.
Document context.client in the README and in the middleware guide.
* fix(context): apply the PII redaction policy to the client block
DynamicContextMiddleware adds the <client_context> message inside the
outer PiiRedactionMiddleware, and the hidden message is not re-sanitized,
so with pii_redaction enabled a preference such as an email address
reached the model verbatim while the same address in the user turn was
redacted. Pass the rendered block through the shared redact_text seam
with this middleware's app_config.pii_redaction, and declare the setting
in release_policy_parameters().
* fix(context): keep client hints out of stored run config
---------
Co-authored-by: Blair Leng <25675774+BlairLeng@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(channels): share encrypted runtime channel config across Gateway instances
IM bot credentials entered in the web UI lived in each host's plaintext
channels/runtime-config.json, and the router cached the merged config on
app.state for the life of the process. With several Gateway instances a
save or disconnect served by an instance that does not hold the
provider's leader lease never reached the transport holder: it kept the
old credentials until it next started the channel, and a disconnect
could leave the old bot connected.
- New shared table channel_runtime_configs (migration 0044): one row per
provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY,
a plaintext disconnect tombstone, and a version bumped by every save.
- SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres
(memory keeps the file); it is synchronous on app_sync_sqlalchemy_url
because its callers already run in worker threads.
- The router re-reads the store on every request instead of caching the
merged config, and refuses (503) to save credentials without a key.
- The leader-lease supervisor compares the versions on each renewal tick
and restarts or stops the holder's transport when a provider's config
changed; a takeover reads a disconnect instead of reviving cached
credentials.
- The first start imports an existing runtime-config.json (rows already
in the table win) and deletes the plaintext file.
* docs(changelog): reference #6988 for the shared runtime channel config
* ci: rerun checks after #6999 fixed the frontend guidance budget on main
* fix(channels): start a held transport that is down when its runtime config changes
When the lease holder's transport was down (a start that failed on the
old credentials), applying a peer-saved config only updated the cached
config, and the transport started on the next lease round: two renewal
ticks instead of the one the docs promise. The apply step now starts it
on the held lease right away, with the lease round's backoff bookkeeping.
* fix(compaction): reject manual compaction of threads with pending execution
A thread paused at a static breakpoint (interrupt_before=["tools"]) keeps its
pending node send in the checkpoint's __pregel_tasks channel. The compaction
write goes through the state-mutation graph, which schedules no tasks, so
LangGraph drops the pending send ('Ignoring unknown node name tools in pending
sends') and a resumed run finishes success without ever executing the tool.
Fail closed: detect a non-empty __pregel_tasks in the raw checkpoint tuple
inside compact_thread_context and raise ContextCompactionPendingWork, mapped
to HTTP 409 by the compact route. The reservation (reserve_checkpoint_write)
already serializes this check against runs and other checkpoint writers, and
the write still forks from the exact head that was read.
Adds dual-mode (full/delta) regression tests: the rejected compaction leaves
checkpoint id, next and pending task ids untouched, and resuming afterwards
executes the pending tool exactly once; idle threads still compact.
* fix(compaction): reject unavailable checkpoints before summarizing
* docs: keep compaction guidance within merged size budget
* test(runtime): cover the driver suffix in sync and store DSNs
* fix(runtime): strip the SQLAlchemy driver suffix from sync and store DSNs
With `postgres_url: postgresql+asyncpg://...` and the default empty
`postgres_schema`, the sync checkpointer, the sync store and the async
store passed the DSN to psycopg's `from_conn_string` unchanged, and
libpq cannot parse the `+asyncpg` scheme. `dsn_with_search_path` only
drops the suffix when it adds a schema, so a non-empty schema hid the
bug. The async checkpointer already calls `normalize_libpq_dsn` first.
Call `normalize_libpq_dsn` at these three call sites as well.
* refactor(persistence): drop the driver suffix inside dsn_with_search_path
dsn_with_search_path dropped a SQLAlchemy `+driver` suffix only when it
added a search_path, so every libpq caller had to wrap the DSN in
normalize_libpq_dsn to cover an empty schema, and flattening that
nesting later would bring the bug back.
Normalize at the top of dsn_with_search_path so both paths drop the
suffix and check the scheme in one place, and remove the outer call
from its five callers. The sync checkpointer and both stores are back
to their code on main.
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(slack): accept file_share events and download inbound files
Slack was dropping every Events API message with a subtype, including
user uploads (subtype=file_share). Allowlist file_share, attach event
files to the inbound message, and download private URLs in receive_file
with the bot Bearer token (bytes via INBOUND_FILE_CONTENT_KEY).
Fixes#6853
* fix(slack): tighten inbound file download guards
Address review on #6855: allowlist https files.slack.com before Bearer
fetch, cap aggregate in-memory bytes per file_share message, report
Content-Encoding refusals as download failed, and share one
_web_client_for_connection helper for inbound/outbound.
* fix(slack): satisfy ruff format and AGENTS.md size gate
Reformat test_slack_file_share.py and drop the AGENTS.md file_share
note so channels/AGENTS.md stays under the 49152-byte hard limit
(file was already near the ceiling without it).
* fix(discord): accept inbound attachments and file-only messages
Empty content no longer drops messages that have attachments. Capture
CDN descriptors on inbound, download in receive_file via Bot auth, and
hand bytes through INBOUND_FILE_CONTENT_KEY like Slack/Telegram.
Closes#6856
* fix(discord): gate bot auth and share inbound file helpers
Address review on #6857: attach Bot Authorization only for Discord CDN
hosts, report Content-Encoding refusals as download failed, and hoist
filename/size helpers into app.channels.inbound_files (Telegram delegates
to the same module).
* fix(discord): sort telegram imports for ruff I001
CI lint-backend failed on unsorted imports in telegram.py after the
shared inbound_files helper was introduced.
* style: ruff-format discord inbound attachments test
* fix(discord): show the enforced attachment download limit
---------
Co-authored-by: Li-john1021 <Li-john1021@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM runs alive when the creating request disconnects
The channel manager holds a `runs.wait` / `runs.stream` request open for
the whole run, and that request dies with the Gateway instance that
received the IM message: a leader hand-off, a rolling restart, a crash.
The Gateway cancels a run whose creating request disconnects unless the
request asked for `on_disconnect: continue`, so every IM run in flight
was cancelled on failover.
Channel runs now ask for `continue`, like scheduled-task and MCP-task
runs already do. The run finishes on the Gateway executing it and its
thread state is persisted, so the answer stays readable in the web UI
even though the IM reply is lost with the old transport. The key is set
after the fire-and-forget branch because the SDK's `runs.create` has no
`on_disconnect` parameter.
Docs: CONFIGURATION.md and the Helm/deployment guides no longer list
"failover cancels IM runs" as a known limit; IM_CHANNEL_CONNECTIONS.md
and the channels AGENTS.md describe the policy.
* docs(changelog): link #6876
* fix(channels): keep the inbound dedupe key once a run may be executing
With `on_disconnect: continue` a run outlives the manager worker that
requested it, but the worker loop released the inbound dedupe key on
cancellation, so a provider redelivery on the replacement instance
admitted the same event again and executed the turn twice (review on
#6876).
Each inbound message now carries an admission record: `requested`
flips before the run request leaves the manager, `confirmed` once the
Gateway proved it admitted the run (a run id, a stream frame, a
completed response). The generic release paths (worker cancellation,
worker/handler exceptions, a broken stream) keep the key when the run
may continue: after confirmation, on cancellation after the request
was sent, and on post-request transport failures (read timeout, read
error, protocol error). Pre-request failures, Gateway rejections
(busy thread, stale pin, status errors, connection failures) and the
stream's terminal `error` frame still release it so the redelivery
can retry.
* fix(buzz): page a saturated history instead of skipping the oldest
buzz-relay serves at most RELAY_HISTORY_LIMIT (2000) stored events per
subscription, NEWEST-FIRST, even when the filter carries a `since`. A channel
that banked more than the cap across one disconnect therefore had its OLDEST
unread messages neither sent nor marked: the live page delivered the newest
2000, the watermark advanced past them, and the rest were lost -- the one
remaining path in this connector that could skip a message outright.
A chat/backlog page's EOSE now drives a descending `until` walk. When a page
comes back AT the cap it may have older matches held back, so the connector
re-asks with `until` set to that page's oldest `created_at` -- on a SEPARATE
`buzz-backlog-<uuid>` subscription id, never the live one -- and repeats until
a page comes back under the cap or reaches the `since` the live REQ was sent
with, bounded by MAX_BACKLOG_PAGES (50, warned when hit). Delivery is charged
to the subscription that carried it before any gate, so a page's saturation is
measured against what the relay sent, not what the allowlist/mention gates kept.
The walk is strictly additive: the live subscription, its filter, and the
watermark are never touched; a CLOSED for a backlog page only ends that walk;
re-opening the live subscription closes any superseded page rather than
orphaning it; and every failure simply leaves the pre-existing behaviour for the
next reconnect to retry.
* fix(buzz): skip the backlog walk when the channel has no read floor
Reviewer willem-bd noted the descending-until walk has no semantic stop when
`floor is None`: that is exactly a channel's first connect (no stored
watermark) and again after a MAX_CACHED_CHANNELS eviction of its cursor. On
those connects the walk would fetch up to MAX_BACKLOG_PAGES * RELAY_HISTORY_LIMIT
events per channel instead of the newest 2000 the connector used to take.
Require a floor before opening any backlog page. The backlog-loss case this walk
exists for always has a watermark (the cursor survives reconnects and restarts),
so it always has a floor; first connect and an evicted cursor keep the previous
behaviour. Covers the skip with a new test, retargets the page-bound test to a
real walk above its floor, and documents the floor requirement.
* fix(buzz): stop orphaned and stalled backlog walks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): control IM notifications and add WeChat/QQ delivery
* fix(scheduler): skip notification preference locks without recipients
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): parse Slack and DingTalk allowed_users fail-closed
Slack and DingTalk still built their allowlists with their own
_normalize_allowed_users: every entry went through str() without
stripping, only list/tuple/set counted as collections, and the gate was
`if self._allowed_users and ...`. A configured list whose entries are all
blank, such as `allowed_users: [$SLACK_USER_ID]` with the variable set to
an empty string, became an empty set and let every sender through.
" U123 " never matched U123, and a frozenset was stringified into one ID.
Parse both with the shared parse_allowed_users helper that Telegram,
WeChat and QQ use, and gate on `is not None`: unset, null, empty
collections and a blank string allow everyone; a scalar is one entry;
strings are stripped and integers keep matching their string form, as
before; booleans, floats, None and mappings are rejected (a mapping as a
whole); a configured value with no valid ID denies every sender and logs
an error. A scalar containing commas or whitespace warns that it stays
one literal ID.
* refactor(channels): share the Slack and DingTalk allowlist coercer and separator warning
Slack and DingTalk had byte-identical ID coercers and their own copies of
the scalar-separator warning. Move the coercer into allowed_users.py as
parse_string_or_int_user_id and opt both channels into
warn_on_scalar_separators, as QQ and WeChat do. Log messages are unchanged.
Pin that a separator-only string such as "," stays one literal ID
(denying everyone else) and still warns.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): merge runtime channel config saves across Gateway workers
Each Gateway worker builds its own ChannelRuntimeConfigStore, loads
runtime-config.json once, and rewrote the whole file from that copy on
every save. With two workers, a bot saved on one worker was dropped when
another bot was saved on the other, and a disconnected bot came back.
Every save, disconnect and removal now rereads the file while holding the
thread lock and an advisory sidecar file lock shared across processes,
changes only its own provider, and saves. The served snapshot is
refreshed only after a successful save, and an unreadable file still
falls back to this process's copy as before.
* docs(changelog): reference #6844 for the runtime channel config fix
* fix(channels): normalize the runtime config lock path like extensions_config
The sidecar lock was derived from the store path as given, so a worker
configured with a symlinked spelling of runtime-config.json took a
different lock file and no longer excluded peers using the real path.
Resolve a symlinked path, and make any other path absolute, before
deriving the sidecar, as extensions_config_file_lock already does.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(buzz): treat a scalar allowed_users entry as one member pubkey
BuzzChannel built its allowlist by iterating config.get("allowed_users") directly. A scalar string -- what an allowed_users: $BUZZ_OWNER_PUBKEY reference resolves to -- was iterated character by character, feeding each character to parse_pubkey, whose first non-hex character raised and aborted channel construction; a numeric scalar raised TypeError the same way. The sibling QQ (#6730) and Discord (#6776) connectors already treat a scalar as a single entry.
Parse the value with the shared parse_allowed_users helper via a parse_pubkey coercer: a scalar is one entry, list entries keep their per-entry validation, unusable entries (bad hex, numeric refs, mappings) are dropped with the shared warning instead of crashing, and the deny-by-default contract is unchanged -- an empty or blank value still yields an empty allowlist with start()'s existing deny-all warning.
config.example.yaml states the accepted shape next to the key, matching the mention_free_channels note added by #6776.
* docs(config): split buzz allowed_users note into aligned comment lines
The two-line note on the buzz example entry was emitted with a literal
`/n` where a newline was intended, so the second note never became its
own comment line and the `#`-column alignment shared with the
DENY-BY-DEFAULT lines below was broken. Split it to match the
mention_free_channels sibling shape (review comment on #6811).
* fix(channels): make Feishu and DingTalk transports stoppable in-process
FeishuChannel.stop() and DingTalkChannel.stop() could not end their SDK
threads. Both SDKs run on a thread-private event loop that only returned
on a fatal error: lark-oapi's ws.Client.start() parks forever in
run_until_complete(_select()) and has no public stop, and the channel
never kept the client handle; dingtalk-stream's start_forever() is a
while-True loop that also catches CancelledError, and the disconnect()
the channel called does not exist on the SDK. So every stop joined for
5s, raised ChannelStopTimeout, and ChannelService retained the instance
while the platform connection stayed alive on the old process. That
burns the full budget on shutdown and on POST /api/channels/{name}/restart,
and it blocks the per-provider leader lease: a demoted leader could not
release the transport before a peer took over.
Both channels now publish the SDK thread's loop and client under a lock;
stop() schedules a shutdown on that loop and then joins. Feishu disarms
lark's _auto_reconnect before closing the socket and stopping the loop,
because start() otherwise swallows the interrupted run_until_complete in
its reconnect branch and dials again. DingTalk drives its own copy of the
SDK connect loop that re-checks the running flag before every dial, runs
the blocking ticket request off the loop, cancels keepalive with the
socket, and logs the endpoint rather than the ticket-bearing payload. A
shared shutdown_private_loop cancels the SDK's leftover tasks with a
bounded drain and closes the loop; a stop requested before the client
exists never dials. ChannelStopTimeout now only fires for a thread that
genuinely wedges inside an SDK call. Telegram already stopped this way.
Tests: a fake lark ws.Client with the SDK's start() shape (including the
reconnect branch that swallows a bare loop stop), a real local websockets
server that observes the DingTalk close, the stop-before-client race for
both, and a blocking-io anchor that stop() ends a live private loop
without blocking the Gateway loop. All eight were red on main.
* docs(changelog): link the stoppable-transports entry to #6798
* fix(channels): release the socket on a stalled close and bound the DingTalk ticket request
Review follow-ups on #6798 (Totoro-qaq):
Feishu: the bounded _disconnect() wait cancelled websockets' close before
its own close-timeout abort ran, and lark clears _conn in its finally, so
a peer that never answers the close frame left the socket to garbage
collection. Worse, a stop landing while _connect() still held lark's
_lock snapshotted _conn as None. _shutdown_ws now settles the in-flight
dial by taking that lock under one shared close deadline, snapshots the
connection, runs _disconnect() under the same deadline, and aborts the
transport itself (abort_connection_transport in base.py) when the
handshake did not finish, yielding once so connection_lost runs before
loop.stop(). DingTalk's websocket.close() path gets the same abort.
DingTalk: the SDK's open_connection() calls requests.post with no
timeout, and running it in an executor let a stalled ticket endpoint
block interpreter exit (executor workers are joined at shutdown). The
channel now sends the same request through httpx on the stream loop
under a 30s deadline, so stop() cancels it and no worker thread lingers.
Tests drive the real SDK clients against loopback peers: a WebSocket
peer that completes the handshake and never ACKs the close (both
channels assert the connection's socket reports fileno() == -1 right
after stop(); red with the abort disabled), a real websockets server for
the clean path, and a loopback ticket server (ok / 500 / never responds)
with a no-lingering-non-daemon-thread check and a request-shape check.
* fix(buzz): accept a single mention-free channel id
`_mention_free` iterated the configured value directly, so one unquoted
UUID became a set of its characters and a numeric or boolean scalar
raised TypeError while the channel was being constructed. Both shapes are
documented as valid for the sibling Discord connector.
Parse the key through a dedicated helper that follows the Discord
convention: a scalar is a single entry, list entries are kept verbatim,
and an unusable entry never matches so `require_mention` stays in force.
* fix(channels): share scalar-or-list channel ID parsing
* docs(channels): keep guidance within CI size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(wechat): cover allowed_file_extensions coercion boundaries
* fix(wechat): never widen a configured allowed_file_extensions back to defaults
* fix(wechat): warn when a scalar allowed_file_extensions looks like a list
A comma- or space-separated scalar string is still one literal extension, so
`allowed_file_extensions: ".pdf, .zip"` blocks every file with nothing in the
startup log pointing at the config. Mirror the `_parse_allowed_users` guard.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(agents): trim restated rationale to restore AG002 chain headroom
The agents/middlewares AGENTS.md chain crossed the 96 KiB AG002 hard
limit at 3f7a0769 (98441 > 98304 bytes); that push produced no lint-check
run, and later pushes compare equal to their base, so main stayed green
while a full-worktree check_agent_guidance run reports an error. Trim
restated rationale:
- backend/AGENTS.md: the Feishu button-card sentence restated semantics
app/channels/AGENTS.md already owns (originator-validated, one-time,
fail-closed clicks); keep the flag visibility and point at the module
guide.
- agents/middlewares/AGENTS.md: drop the membership-probe rationale
duplicated by the model_response.py paragraph, the ToolOutputBudget
ordering narration, and the lead-only append note restated by the
lead-only middlewares heading.
Chain: 98441 -> 98198 bytes (106 under the hard limit).
* docs(channels): own the Feishu card fail-closed invariant in the module guide
Review follow-up on #6759: the backend/AGENTS.md trim dropped two things
app/channels/AGENTS.md item 7 does not own - the fail-closed-across-workers
guarantee and the owning file's name - so the invariant left every
instruction chain and the pointer dead-ended one level above the
implementation. State both in item 7, paid by dropping the therefore-clause
restated by the single-active-owner invariant above it (the partial unique
index and the losing-writer retry already imply deterministic resolution).
channels/AGENTS.md: 49134 -> 49129 bytes (AG001 headroom 18 -> 23).
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(qq): parse allowed_users with the shared fail-closed allowlist
QQChannel built its allowlist with set(config.get("allowed_users") or []).
A single OpenID, which is what an `allowed_users: $QQ_ALLOWED_USERS`
reference resolves to, became a set of its characters, so the configured
user was silently dropped. A mapping authorized every key, including one
mapped to false, and a numeric scalar raised TypeError while the channel
was being constructed.
Parse it with the shared parse_allowed_users helper that Telegram and
WeChat already use: unset, null, empty collections and a blank string
allow everyone; a scalar OpenID is one entry; entries that are not
non-blank strings (such as numeric QQ account numbers) are dropped with a
warning, and a mapping is rejected as a whole; a configured value with no
valid OpenID denies every sender and logs an error. A scalar containing commas or whitespace warns
that it stays one literal ID. Binding codes still precede the allowlist.
* test(qq): gate group messages by member OpenID in allowlist tests
Every allowlist test sent a C2C frame, and the shared event() helper puts
the sender in both user_openid and member_openid, so a group-path
regression in sender resolution would pass unnoticed. admits() now takes
a kind; group frames carry only member_openid and a group_openid that
differs from the sender. The listed-OpenID test runs for both kinds, and a
new test shows that listing the group's OpenID does not admit its members.
Document that a blank or whitespace-only allowed_users string allows every
sender, like an unset value or an empty list.
* refactor(channels): share scalar allowlist separator warnings
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(dingtalk): preserve Markdown table cell boundaries
DingTalk's sampleMarkdown fallback re-splits table rows on every pipe, so an
escaped pipe (\|) inside a cell is read as a column separator and every value
after it shifts one column to the left. The strip("|") wrapper also removes an
empty leading or trailing cell, dropping that value's column label.
Split rows on unescaped pipes only, and drop at most one outer border cell per
side, so escaped pipes and empty edge cells stay under the right label.
Adds six parameterized regressions plus one full fallback-adapter regression;
all seven fail on the base commit and pass after the fix.
* docs(dingtalk): document escaped-backslash boundary in table row splitter
Document the residual boundary case flagged in review: the
single-character lookbehind in _split_markdown_table_row cannot
distinguish an escaped pipe (\|) from an escaped backslash followed
by a real separator (\\|), so a cell ending in a literal backslash
absorbs the next cell. Full GFM backslash counting is out of scope
for this helper.
* fix(gateway): read subagent batch exports in small pages
The results.jsonl export loaded 500 full rows per page before streaming
them. A row can hold max_result_chars of report plus as much evidence
JSON (result_artifact), so with the default limit one request peaked
around 280 MB and decoding a page's JSON columns stalled the event loop
for about half a second; at the 1,000,000 ceiling a page was roughly
1.7 GB.
Read 10 rows per page instead. Output and ordering are unchanged; a
full export of 1,000 default-sized rows now peaks at 6 MB with a 46 ms
worst stall and takes the same time.
* docs(changelog): reference #6706
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): render Feishu option clarifications as button cards
When the agent's ask_clarification carries options, the Feishu channel
now renders the question as an interactive card with one button per
option instead of plain text. This is the outbound half: card
construction plus a process-local pending registry that a later
card-action callback handler will validate clicks against.
- New feishu_card_interaction module owns request extraction, button
card rendering, and the pending registry (30-minute TTL, matching the
typed-reply clarification window)
- Manager attaches the clarification's structured human_input payload to
the final outbound's metadata so channels can render richer
affordances without touching the Gateway protocol
- Feishu channel renders the button card only when
card_interaction_enabled is set (default off); free-text, form, and
over-four-option clarifications keep the plain-text path
- Button cards bypass the markdown wrapper via a raw_card_content flag
on the card send/patch helpers; plain-text call signatures unchanged
* feat(channels): close the Feishu button-card click loop
The inbound half of interactive clarification cards: a click on an
option button validates, resolves the clarification, and marks the
card in place.
- Register the card-action callback only when card_interaction_enabled
is set; clicks arrive as WS EVENT frames through the SDK's callback
processor map, and the synchronous return value becomes the toast ack
- consume_if_originator atomically validates operator identity against
the pending entry: non-originators get a refusal toast and the entry
survives; unknown or expired requests are toast-only and dropped
- A validated click consumes once, updates the card to a terminal
resolved state, and injects a hidden inbound carrying the
human_input_response so the run resumes on the originating thread
with Web-card parity
- Manager translates the inbound's human_input_response metadata into
the human message's additional_kwargs with hide_from_ui set
* docs(channels): document Feishu interactive clarification cards
- README / README_zh: optional card_interaction_enabled setup, calling
out that card.action.trigger lives under the console's callback
configuration tab (separate from event subscriptions) and requires
publishing a new app version
- config.example.yaml: commented flag with the same console caveat
- backend/AGENTS.md: module ownership note for feishu_card_interaction
* fix(channels): close Feishu card-click routing and ack gaps from review
Address review on #6670:
- Carry the routing identity (topic_id, connection_id, owner_user_id)
on the pending entry and the synthetic inbound, so clicks resolve to
the clarified thread in topic-scoped chats and pass bound-identity
admission instead of opening a new thread or being rejected
- Validate option_value against the registered options at consume time:
an unknown value is treated like a stale card and the entry is kept
- Return a warning toast when a click cannot be dispatched, and flip
the card to its resolved state only after the inbound is durably in
the bus — never display a choice that was not recorded
- Drop the dead mark_sent/consume registry APIs and pass the payload's
own option ids through to the buttons
- Add an end-to-end test through the manager's thread resolution
pinning that a card answer reuses the clarified thread
* fix(channels): carry workspace_id on the Feishu card-click inbound
The connections-enabled admission path re-reads the binding with the
message's workspace_id, and Feishu connections are bound with
workspace_id = chat_id. The synthetic inbound set connection_id and
owner_user_id but never workspace_id, so the re-read missed and every
click was rejected as unbound after the entry was consumed.
- Set inbound.workspace_id from the entry's chat_id (Feishu's
chat-as-workspace binding convention)
- Extend the card-click e2e to require_bound_identity=True, driven
through the channel's real _resolve_card_click production
* fix(channels): stabilize Feishu clarification delivery and routing
Option cards could fail through the real reply SDK, resume another conversation,
or leave a stale text hint after the answer was accepted.
Unify raw-card delivery and persist trusted thread aliases before showing buttons.
Keep protocol request IDs separate from bounded, one-time callback tokens; validate
originators and option pairs, and retire only the accepted card's text/button state.
Treat valid hidden human answers as turn boundaries so old replies and artifacts
do not leak into the next answer. Cover consecutive clarifications and /new routing
through real SDK request models, stores, and the run journal.
Validation: 147 focused tests and 29,421 offline tests passed (405 skipped);
strict blocking-I/O passed 245 tests (1 skipped). Ruff and guidance checks passed.
* fix(channels): preserve clarification context in Feishu cards
Button cards replaced the full clarification text but omitted its background,
leaving users without the context needed to choose an option.
Carry optional context from request metadata and render its original Markdown
before the question. Keep question-only cards for absent, blank, or invalid context.
Cover rendering and all real SDK delivery paths, and update both user guides.
Validation: 156 focused tests, 245 strict blocking-I/O tests, Ruff, and guidance
checks passed. The offline run recorded 29,381 passed, 49 failed, and 405 skipped;
48 import failures reproduced before the fix. All 49 cases passed on both versions
in isolated environments, and the Feishu-to-export order check passed 13 tests.
* fix(scheduler): preserve unchanged once schedules near execution
* docs: keep schedule guidance within the size limit
* style(scheduler): use English for the once-edit comment
* docs(scheduler): keep guidance within directory budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason
A cross-worker cancel that takes over an expired-lease run writes error but
left stop_reason unset. `orphan_recovered` is the only signal that lets a
store-only SSE/`wait` consumer synthesize END after a heartbeat, because the
producer is known to be gone. Without it, a subscriber that outlives the
crashed owner waits forever for an END no surviving worker publishes; periodic
reconciliation ignores the row because it is already terminal.
Mark the takeover like periodic orphan recovery: claim_for_takeover already
accepts the stop reason and the reconcile path already passes it. The new
tests pin the takeover row's marker and prove a real takeover wakes a waiting
consumer; three existing takeover test doubles now forward the store argument.
* docs(gateway): keep guidance within the 48KB budget
The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment.
* fix(gateway): complete streams after scheduler orphan recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): preserve server binding history across gateway workers
A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied.
Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones.
* fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying
An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call.
Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer.
* refactor(mcp): share stdio fingerprinting across lifecycle paths
Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper.
Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed.
* docs(mcp): condense lifecycle guidance under the AGENTS chain limit
The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract.
* docs: trim inherited agent guidance below CI size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(authz): account disable/enable across every authentication surface (#4063 gap 3)
Vertical slice 1 of the user-lifecycle work: an operator can suspend an
account and every authentication surface rejects it, while the row —
credentials, role, OAuth linkage — is retained so re-enabling restores
exactly what was suspended.
- users.disabled (NOT NULL, server default 0) via migration
0036_user_disabled; User/UserResponse carry and expose it.
- Enforcement at all four surfaces: password login (no credential
comparison happens), JWT session resolution (401 ACCOUNT_DISABLED —
sessions die at the next request), PAT resolution (the generic 401, no
oracle), OAuth provisioning (403 with a contact-admin message; the
link is kept so re-enable restores access).
- SQLiteUserRepository.set_disabled: single-column, serialized write
(same idiom as update_system_role) with the last-ACTIVE-admin guard —
disabled admins cannot authenticate, so they cannot restore management
access; re-enabling is always allowed. Field-scoping completed both
ways: the credential writer mirrors and never writes disabled, so
a stale password-change snapshot cannot re-enable a suspended account.
- PATCH /api/v1/admin/users/{id} becomes a partial account update
(system_role and/or disabled), each field routed to its field-scoped
writer; empty bodies 422.
Tests 11->18: route (disable/re-enable, last-active-admin 409, empty
422, combined update), real-repo (field isolation both directions,
disabled-admin-not-active, guard inside the serialized write, re-enable
always), login rejection with the CORRECT password (only the disabled
gate can reject) plus resolver-visible flag. Mutations verified red:
guard removed (M19), login gate removed (M20 — after hardening the test
with a real hash so the gate is load-bearing).
* fix(authz): portable boolean default, alternate-authenticator suspension, active-admin role guard
Review round 1 on #6566 — three P1s:
- [PG default] sa.text("0") is a raw integer expression PostgreSQL
rejects as a BOOLEAN default, breaking startup migration and fresh
create_all bootstraps alike. Both the migration and the ORM model now
use sa.false()/false() — dialect-rendered, portable.
- [alternate authenticators] WebSockets bypass AuthMiddleware, so a
disabled user's still-valid cookie could open a NEW browser/stream
connection and drive the retained session; the standalone LangGraph
authenticate callback accepted the same suspended JWT. Both now route
through a shared post-lookup validator (validate_resolved_session_user
in deps.py — token_version + disabled in one place) that the HTTP
dependency also uses, so a lifecycle change lands everywhere at once.
- [active-admin role guard] update_system_role counted every admin row,
so 'disable B, then demote self' — or a combined {system_role, disabled}
self-update — could strand the deployment with zero administrators able
to authenticate or re-enable anyone. The demotion guard now counts
ACTIVE admins only (disabled admins cannot authenticate), under the same
serialization boundary.
Tests 18->22: shared-validator verdicts, WS admission after suspension
(real token, real repo), LangGraph-callback admission after suspension,
disable-then-demote-self + combined self-update both rejected (fake-repo
route level and real-repo update_system_role level). Mutations verified
red: helper's disabled check removed (M21 — three admission tests fail),
active-admin predicate reverted to all-admins (M22 — repo-level strand
scenario fails). 366 tests green on the affected surface.
* fix(migrations): chain 0036_user_disabled after upstream 0037
The maintainer main-merge brought 0037_project_document_summaries,
which branched from 0035 independently of this PR's 0036 — alembic now
sees two heads and every engine bootstrap fails. Chain 0036 after 0037
(single head; revision numbers are labels, the graph is what matters).
* fix(migrations): renumber user_disabled to 0038 and follow the chain-test convention
Upstream landed its own 0036 (run_idempotency_request) with 0037 chained
after it while this PR was in review — my 0036_user_disabled collided on
the number and my interim chain-fix (0036 revises 0037) left the graph
in a state upstream's chain tests reject (heads == [0037] hardcoded).
- 0036_user_disabled -> 0038_user_disabled, revises
0037_project_document_summaries: single head 0038.
- Chain tests follow the repo convention (each migration PR appends the
new head): 0032/0035 gain the DOC_SUMMARIES link and CURRENT_HEAD =
0038; 0037's single-head assertion moves to CURRENT_HEAD with its
own-revision link check unchanged.
* fix(migrations): pin the remaining chain tests to the 0038_user_disabled head
The 0027/0028, 0029 and 0034 chain tests still pinned the single head to
0037_project_document_summaries (and 0034 asserted the head's parent was
0036_run_idempotency_request), so the renumbered lifecycle migration broke
two unit-test shards. Follow the chain-test convention: every pinned head
moves to 0038_user_disabled and 0034 walks the chain through 0037.
* fix(authz): enforce account suspension on the internal-auth dispatch surface
Round-3 review finding: the internal-auth principal is synthesized from
trusted headers without a users-row lookup, so a suspended user with an
active channel binding kept driving the agent, and scheduler-owned runs
(their fabricated in-process requests skip AuthMiddleware entirely) kept
launching. Two gates now enforce the lifecycle there:
- AuthMiddleware rejects owner-bound internal calls whose users row is
disabled (401 account_disabled) at the single gate every internal HTTP
call passes through. Auth-disabled mode is exempt (no suspendable
accounts, and its owner header carries the synthetic identity), and
owner ids without a users row stay allowed — a nonexistent account
cannot be suspended.
- start_run re-asserts it at run admission after the owner is resolved
for attribution: the scheduler records the rejection as the
occurrence's launch error and no run record is created.
Also addresses the remaining round-3 nits: drop the inline token_version
check duplicated by validate_resolved_session_user, correct the
local_provider comment (login never surfaces ACCOUNT_DISABLED), refresh
the stale 0038 docstring header, and restore the deps module caches in
the two suspension tests instead of leaking a closed engine.
* fix(auth): carry the disabled field on the frontend synthetic users
UserResponse now always includes disabled, so the auth-disabled contract
e2e (gateway /auth/me deep-equals the frontend synthetic user) failed
without the field on AUTH_DISABLED_USER. Add disabled to both synthetic
users and declare it optional on the zod schema — absent still parses as
not disabled for pre-gap-3 backends.
* fix(authz): log when the internal-owner suspension check is skipped
The middleware gate's users-lookup failure path silently degraded to
allow; keep the fail-open behavior but leave a WARNING with the
exception, so a store that cannot be read is observable in production
logs instead of invisible.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): share chat-to-thread bindings through the database
ChannelManager kept its IM chat-to-thread bindings
(channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file,
channels/store.json, loaded once and rewritten whole on every change. With
several Gateway processes a binding created on one replica was invisible to
the others (the next message for the same chat opened a second thread) and
concurrent writers clobbered each other's file.
- New channel_thread_bindings table (migration 0036_channel_thread_bindings)
and harness package deerflow.persistence.channel_thread_bindings: key PK
(the legacy composite string), channel_name, chat_id, nullable topic_id,
thread_id, user_id, epoch created_at/updated_at; index (channel_name,
chat_id). Dialect upsert keeps created_at; prefix remove and batched
ON CONFLICT DO NOTHING import.
- app.channels.store.ChannelStore is an async protocol: SqlChannelStore for
sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for
database.backend: memory. resolve_channel_store picks at ChannelService
construction; ChannelService.start() imports an existing store.json once
into an empty table and renames it store.json.migrated (a concurrently
importing peer neither duplicates nor clobbers; a populated table leaves
the file alone).
- Callers updated: manager, connection_identity.lookup_thread_id, Feishu
(send() awaits the mapping writer; the synchronous lark callback bridges
its two store calls to the Gateway loop with a bounded wait), Discord
(channels_dir).
- Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at
the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md,
README) and CHANGELOG updated.
* docs(changelog): link the channel thread bindings entry to #6558
* fix(channels): reject only the Gateway loop in the Feishu store bridge
lark-oapi 1.5.5 invokes the message callback synchronously from its async
frame handler, on the private event loop that FeishuChannel._run_ws starts
for the SDK thread. The bridge's guard treated any running loop as unsafe,
so every real inbound message skipped the stored-topic lookup and the
pending-mapping restore: a reply whose parent card was mapped but whose
root was not fell back to the unknown root, and could consume an
unrelated pending clarification.
Only a wait issued from the Gateway loop itself (the loop that executes
the submitted coroutine) is refused now; the SDK's own loop may block for
the bounded wait, as that thread blocked on the JSON rewrite before.
Tests drive _on_message from a worker thread running its own loop, the
production shape, for both the lookup and the write path, and pin the
Gateway-loop refusal.
* fix(channels): defer unresolved Feishu lookups and harden the store.json import
Review follow-ups on the chat-to-thread binding store (#6558):
- Import: a legacy entry whose key, channel_name, chat_id, topic_id,
thread_id or user_id exceeds its column width is skipped and counted
with the malformed ones instead of failing the whole batch on
PostgreSQL (SQLite silently accepted the same row, so the backends
diverged on one file). The corrupt-file warning now says what each
caller does: the JSON store starts fresh, the import is skipped and
the file kept.
- Feishu bridge: store calls run on the Gateway loop under a 3s
asyncio.wait_for (the SDK loop also services ws keepalives; the old
10s bound could stall it per message), and the lark-thread wait is
that budget plus 1s. A deadline or store error is StoreLookupUnresolved,
never a miss: _on_message no longer routes such a reply onto its
unknown root (which created a second thread and later overwrote the
parent alias). The message joins a per-chat FIFO drained on the
Gateway loop with backoff for up to 30s, then is dropped with a
warning. _on_message is split into parse / resolve / dispatch so the
deferred path dispatches the same way; later messages for the chat
queue behind the deferred one.
* fix(tests): expect follow_redirects in the Jina client kwargs after #5808#5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and
updated test_jina_client.py, but test_jina_response_limit.py and
test_jina_retry_after.py still pin the exact client kwargs to proxy and
trust_env, so every backend-unit-tests shard that lands on them fails on
main (the backend unit-test workflow only runs on pull requests, which is
why the push to main did not surface it). Update the two expectations to
include the new keyword.
(cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050)
* Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808"
This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7.
* fix(channels): bound deferred Feishu routing by the inbound queue capacity
Review follow-up on #6558: a store outage let every subsequent message for
a chat join the deferred queue with no capacity check (the bus reservation
was only taken after routing succeeded), and each queued message started a
fresh 30s retry budget when it reached the head, so a burst could be
retained for N x 30s and grow without bound.
- Intake is reserved BEFORE routing for every store-backed message
(bus.reserve_inbound is thread-safe and already used from SDK threads),
so deferred and live messages share the one configured bound,
channels.inbound_queue_maxsize; a message that cannot be admitted is
dropped at once with a warning, never queued or routed as a miss.
- The reservation travels with the message into dispatch (committed by
_prepare_inbound), is released on the batch path (the flush reserves at
publish time), on drop after the budget, on a failed deferral, on
dispatch failure and for every leftover when the drain task is cancelled
by stop().
- The retry budget runs from the message's arrival: a queued message whose
budget is spent when it reaches the head is dropped without a lookup.
- The bounded resolver returns the unresolved outcome instead of raising
it through the tracked task, which logged an ERROR for an expected,
handled store timeout.
- Upstream's new blocking-io test constructs the store by its old class
name; it now uses JsonChannelStore.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): reconcile committed Gateway config mutations
Install selective MCP binding fences inside the Gateway config commit boundary and finish teardown after releasing config locks. Reject frozen durable-task server changes before persistence.
Preserve drained cancellation semantics and cover delete/re-add, durable-only bindings, multi-server diffs, shared writers, and post-write reconciliation failure.
* test(mcp): pin committed handoff boundaries
Prove the fallback fence is installed before releasing the config lock, a committed writer cannot swallow an earlier shared reset, delete/re-add advances the applied baseline and cache generation, and retries after a post-write 500 have explicit conflict or missing semantics.
* fix(mcp): fence committed config after reload failure
Treat a post-write reload failure as a committed handoff failure so local MCP state is detached before releasing the config lock and teardown remains outside it.
* fix(mcp): always finish committed pool teardown
Run finish_mcp_reconciliation from a post-lock try/finally so response or logging failures cannot strand a retired full-reset pool.
* test(mcp): fence cold-start binding race
Pin the in-flight discovery path that keeps the committed fast path from admitting a late old-fingerprint binding.
* docs(mcp): document committed teardown guarantee
* fix(mcp): sanitize committed response errors
Keep post-write response conversion failures behind a fixed HTTP detail while preserving the retired-pool teardown guarantee.
* fix(mcp): avoid duplicate committed reconciliation error logs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
