26 Commits

Author SHA1 Message Date
llb555
3f1438d4a0
fix(channels): publish Discord and WeChat JSON state atomically (#7595)
* fix(channels): publish Discord and WeChat JSON state atomically

* fix(channels): adopt the shared atomic publisher for both state writers

* test(channels): pin the cleanup-failure direction too

* test(channels): keep a platform-independent guard on the publish call shape

* style(tests): apply ruff format to the channel encoding test
2026-10-30 14:54:41 +08:00
Totoro
ab8cd4e10f
fix(channels): keep IM binding codes on screen and report goal state in replies (#7591)
* fix(channels): keep IM binding codes on screen and report goal state in replies

Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a
dialog that keeps the /connect command on screen with a copy button, its
expiry time and the binding status, instead of a toast that disappears.

The bots' /connect replies and the IM /goal replies now use the owner's
web UI language (else channel_connections.notification_locale), and /goal
reports the goal's state as the web goal bar does: in progress, why it
stopped, or that it was met.

* docs(changelog): note the IM binding dialog and localized IM replies (#7591)

* test(channels): check IM reply tables against every notification locale

The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal
reply table had none. Both now match NOTIFICATION_LOCALES with the same
keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale
fails here instead of raising KeyError at reply time.

---------

Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-10-27 09:46:58 +08:00
huyengiang101086-spec
1ad8147947
fix(channels): pin JSON state encoding across platforms (#7228)
* fix(channels): pin JSON state encoding across platforms

* test(channels): check encoding pins at each state I/O call

---------

Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-21 18:16:22 +08:00
Nan Gao
085924a3d0
fix(channels): run each IM message once across instances, restarts and late redeliveries (#7104)
* fix(channels): run each IM message once across instances, restarts and late redeliveries

Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.

- Discord messages are keyed (discord, guild_id or "dm", posting channel,
  message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
  before a side effect; Discord claims it on the Gateway loop before
  _create_thread, marks the published message dedupe_claimed, and gives the
  claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
  the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
  attempt made every keyed retry a 409). A reused run's status-only /wait or
  stream gap is resolved to this message's own turn via threads.get_state;
  409s about the message's own keyed run keep the key instead of reading as
  a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
  (claim_thread_id on channel_conversations and channel_thread_bindings);
  the loser adopts the winner and deletes the empty thread it minted.
  /new and /agent use still rebind.

Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).

* fix(channels): give back Discord dedupe claims for messages never processed

Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:

- The hand-off task releases the claim on cancellation too (stop() drains it
  while it resolves the connection identity), via await_drained so the
  cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
  message that shutdown drops from the queue, after the grace period and on
  a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
  because stop() already closed cross-loop intake, or a hand-off cancelled
  before it ran, leaves the claim tracked, and stop() releases it last.

* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation

Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.

- Keys from trusted internal callers (the IM channel manager) are scoped by
  owner only (`internal-run:` prefix); browser and API keys keep the
  owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
  RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
  see the same type, and start_run answers it with 409 "Idempotency-Key
  already used with a different request" instead of a 500. The channel
  manager already treats that 409 as the message's admitted run.

Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 22:57:19 +08:00
Li-john1021
32d7dd5fe8
fix(discord): accept inbound attachments and file-only messages (#6857)
* fix(discord): accept inbound attachments and file-only messages

Empty content no longer drops messages that have attachments. Capture
CDN descriptors on inbound, download in receive_file via Bot auth, and
hand bytes through INBOUND_FILE_CONTENT_KEY like Slack/Telegram.

Closes #6856

* fix(discord): gate bot auth and share inbound file helpers

Address review on #6857: attach Bot Authorization only for Discord CDN
hosts, report Content-Encoding refusals as download failed, and hoist
filename/size helpers into app.channels.inbound_files (Telegram delegates
to the same module).

* fix(discord): sort telegram imports for ruff I001

CI lint-backend failed on unsorted imports in telegram.py after the
shared inbound_files helper was introduced.

* style: ruff-format discord inbound attachments test

* fix(discord): show the enforced attachment download limit

---------

Co-authored-by: Li-john1021 <Li-john1021@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-14 23:01:02 +08:00
Grapette.L
9b2151a3a8
fix(buzz): a single mention_free_channels ID was split into characters, a numeric one stopped the connector (#6776)
* fix(buzz): accept a single mention-free channel id

`_mention_free` iterated the configured value directly, so one unquoted
UUID became a set of its characters and a numeric or boolean scalar
raised TypeError while the channel was being constructed. Both shapes are
documented as valid for the sibling Discord connector.

Parse the key through a dedicated helper that follows the Discord
convention: a scalar is a single entry, list entries are kept verbatim,
and an unusable entry never matches so `require_mention` stays in force.

* fix(channels): share scalar-or-list channel ID parsing

* docs(channels): keep guidance within CI size budget

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-13 14:55:49 +08:00
Nan Gao
6c4c74a504
fix(channels): share chat-to-thread bindings through the database (#6558)
* fix(channels): share chat-to-thread bindings through the database

ChannelManager kept its IM chat-to-thread bindings
(channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file,
channels/store.json, loaded once and rewritten whole on every change. With
several Gateway processes a binding created on one replica was invisible to
the others (the next message for the same chat opened a second thread) and
concurrent writers clobbered each other's file.

- New channel_thread_bindings table (migration 0036_channel_thread_bindings)
  and harness package deerflow.persistence.channel_thread_bindings: key PK
  (the legacy composite string), channel_name, chat_id, nullable topic_id,
  thread_id, user_id, epoch created_at/updated_at; index (channel_name,
  chat_id). Dialect upsert keeps created_at; prefix remove and batched
  ON CONFLICT DO NOTHING import.
- app.channels.store.ChannelStore is an async protocol: SqlChannelStore for
  sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for
  database.backend: memory. resolve_channel_store picks at ChannelService
  construction; ChannelService.start() imports an existing store.json once
  into an empty table and renames it store.json.migrated (a concurrently
  importing peer neither duplicates nor clobbers; a populated table leaves
  the file alone).
- Callers updated: manager, connection_identity.lookup_thread_id, Feishu
  (send() awaits the mapping writer; the synchronous lark callback bridges
  its two store calls to the Gateway loop with a bounded wait), Discord
  (channels_dir).
- Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at
  the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md,
  README) and CHANGELOG updated.

* docs(changelog): link the channel thread bindings entry to #6558

* fix(channels): reject only the Gateway loop in the Feishu store bridge

lark-oapi 1.5.5 invokes the message callback synchronously from its async
frame handler, on the private event loop that FeishuChannel._run_ws starts
for the SDK thread. The bridge's guard treated any running loop as unsafe,
so every real inbound message skipped the stored-topic lookup and the
pending-mapping restore: a reply whose parent card was mapped but whose
root was not fell back to the unknown root, and could consume an
unrelated pending clarification.

Only a wait issued from the Gateway loop itself (the loop that executes
the submitted coroutine) is refused now; the SDK's own loop may block for
the bounded wait, as that thread blocked on the JSON rewrite before.
Tests drive _on_message from a worker thread running its own loop, the
production shape, for both the lookup and the write path, and pin the
Gateway-loop refusal.

* fix(channels): defer unresolved Feishu lookups and harden the store.json import

Review follow-ups on the chat-to-thread binding store (#6558):

- Import: a legacy entry whose key, channel_name, chat_id, topic_id,
  thread_id or user_id exceeds its column width is skipped and counted
  with the malformed ones instead of failing the whole batch on
  PostgreSQL (SQLite silently accepted the same row, so the backends
  diverged on one file). The corrupt-file warning now says what each
  caller does: the JSON store starts fresh, the import is skipped and
  the file kept.
- Feishu bridge: store calls run on the Gateway loop under a 3s
  asyncio.wait_for (the SDK loop also services ws keepalives; the old
  10s bound could stall it per message), and the lark-thread wait is
  that budget plus 1s. A deadline or store error is StoreLookupUnresolved,
  never a miss: _on_message no longer routes such a reply onto its
  unknown root (which created a second thread and later overwrote the
  parent alias). The message joins a per-chat FIFO drained on the
  Gateway loop with backoff for up to 30s, then is dropped with a
  warning. _on_message is split into parse / resolve / dispatch so the
  deferred path dispatches the same way; later messages for the chat
  queue behind the deferred one.

* fix(tests): expect follow_redirects in the Jina client kwargs after #5808

#5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and
updated test_jina_client.py, but test_jina_response_limit.py and
test_jina_retry_after.py still pin the exact client kwargs to proxy and
trust_env, so every backend-unit-tests shard that lands on them fails on
main (the backend unit-test workflow only runs on pull requests, which is
why the push to main did not surface it). Update the two expectations to
include the new keyword.

(cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050)

* Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808"

This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7.

* fix(channels): bound deferred Feishu routing by the inbound queue capacity

Review follow-up on #6558: a store outage let every subsequent message for
a chat join the deferred queue with no capacity check (the bus reservation
was only taken after routing succeeded), and each queued message started a
fresh 30s retry budget when it reached the head, so a burst could be
retained for N x 30s and grow without bound.

- Intake is reserved BEFORE routing for every store-backed message
  (bus.reserve_inbound is thread-safe and already used from SDK threads),
  so deferred and live messages share the one configured bound,
  channels.inbound_queue_maxsize; a message that cannot be admitted is
  dropped at once with a warning, never queued or routed as a miss.
- The reservation travels with the message into dispatch (committed by
  _prepare_inbound), is released on the batch path (the flush reserves at
  publish time), on drop after the budget, on a failed deferral, on
  dispatch failure and for every leftover when the drain task is cancelled
  by stop().
- The retry budget runs from the message's arrival: a queued message whose
  budget is spent when it reaches the head is dropped without a lookup.
- The bounded resolver returns the unresolved outcome instead of raising
  it through the tracked task, which logged an ERROR for an expected,
  handled store timeout.
- Upstream's new blocking-io test constructs the store by its old class
  name; it now uses JsonChannelStore.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-10 14:00:00 +08:00
NanPan
ca1cae6185
fix(discord): drain thread mapping persistence across cancellation (#6356)
* fix(discord): drain thread mapping persistence across cancellation

* test(discord): retain mapping flush across cancellation
2026-10-06 15:49:27 +08:00
Grapette.L
2523be658b
fix(discord): read allowed_guilds as IDs and deny every guild on an unreadable allowlist (#6338)
* test(discord): expose allowlist parsing failures instead of opening the bot

config.example.yaml documents channels.discord.allowed_guilds as a value that
"can also be a single guild ID", but the constructor iterates it: a scalar
string becomes one entry per character, and an unquoted YAML int raises
TypeError. Whatever remains is gated by truthiness, so an allowlist with no
parseable entry leaves the set empty and admits every guild.

* fix(discord): read allowed_guilds as IDs, and fail closed when unreadable

The allowlist was built by iterating the configured value, so the documented
single-guild shorthand ("can also be a single guild ID" in
config.example.yaml) became one entry per character: allowed_guilds: "321"
admitted guilds 1, 2 and 3 while blocking 321, and an unquoted YAML int raised
TypeError in the constructor. The empty result then read as "no allowlist", so
a restriction the operator configured could open the bot to every guild the bot
joined.

Parse both ID lists through helpers that treat a scalar as a one-entry list and
drop unreadable entries with a warning, following the telegram channel: an
allowlist with no parseable entry now denies every guild and logs at ERROR.
Mention exemptions keep their IDs verbatim, where an unreadable entry only
leaves mention_only in force.

* docs(discord): explain allowlist parsing and deny-all behavior

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-06 07:56:10 +08:00
Hyeonsang Cho
aeeee2994f
fix(channels): run Discord connection-repository work on the Gateway loop (#6214)
* fix(channels): run Discord connection-repository work on the Gateway loop

discord.py delivers messages on a private loop in the client thread, and
the Discord adapter awaited the channel-connection repository there even
though its SQLAlchemy engine and pool belong to the Gateway loop. On
PostgreSQL the first Discord message after the Gateway used the pool
failed with "got Future attached to a different loop" and was dropped; on
SQLite a pool-exhausting burst failed with "Queue is bound to a different
event loop" and left the pool's wait queue bound to the Discord loop, so
the Gateway's own queries then failed the same way.

Resolve the identity together with the intake commit, and run /connect
binding, on the Gateway loop through _submit_threadsafe_coroutine, as
Telegram, Feishu and DingTalk already do. Bind replies return through
_run_on_discord_loop, and start()/stop() now open and drain the
cross-thread submissions.

* docs(changelog): reference #6214 for the Discord connection-loop fix

* fix(channels): correct the unscheduled Discord bind comment and pin it

When _submit_threadsafe_coroutine refuses the bind, base.py closes the
coroutine, so consume_oauth_state never runs and the one-time code stays
valid; only the Discord message is swallowed. Say that instead of
claiming the code is consumed, and pin both halves with a test: the
attempt is handled (never a chat turn) and the code still binds later.

* fix(channels): undo Discord typing when the Gateway hand-off fails

Since identity resolution moved to the Gateway loop, both inbound paths
started typing and the ack reaction as soon as the hand-off was
scheduled. A lookup that then raised (database outage, pool-checkout
timeout) dropped the message but left an endless typing loop and a
success reaction behind.

The hand-off now returns its completion future (new
Channel._submit_threadsafe_coroutine_future, which the bool helper wraps)
and the Discord side waits on it through asyncio.shield. Typing still
registers before the hand-off settles, so a fast reply's _stop_typing
cannot run first; the ack reaction waits for the commit. On failure, or
when stop() drains the hand-off, the indicator this message started is
stopped (never one another in-flight message owns) and no ack is added.

* fix(channels): keep shared Discord typing alive for dependent messages

Two messages to the same target share one typing task: the first starts
it, a later one reuses it. If the creator's hand-off failed after a
follower had committed, or while the follower was still pending, the
creator stopped the indicator the follower relied on, and nothing
restored it while the follower's request ran.

Count the hand-offs that depend on each typing task. A pending or
committed hand-off holds the task (a commit keeps its hold until a reply
stops the indicator), a failed one releases its hold, and the indicator
is stopped only when the failed message started it and no hold remains.
The count is dropped when the task ends or the channel cancels typing.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-03 17:14:19 +08:00
Hyeonsang Cho
9a97cb9420
fix(channels): send Discord typing indicators through the discord.py 2.x API (#6138)
* fix(channels): send Discord typing indicators through the discord.py 2.x API

_start_typing() called channel.trigger_typing(), which discord.py removed
in 2.0; the project requires discord.py>=2.7.0. The typing loop swallowed
every exception, so each tick raised AttributeError and no indicator was
ever sent, and the tests never noticed because they mocked the removed
method on a SimpleNamespace.

Await channel.typing() instead (awaiting the returned Typing sends one
indicator), and log a failed tick at DEBUG rather than discarding it. The
existing typing-lifecycle tests now mock typing(), and new tests drive
_start_typing() against a fake shaped like the 2.x Messageable, which has
no trigger_typing.

* docs(changelog): reference #6138

* fix(channels): warn on a Discord typing loop's first failed indicator

Logging every failed typing tick at DEBUG left a permanently dead
indicator invisible at the default INFO level, the same silence that hid
the trigger_typing AttributeError. A bot that lost Send Messages in a
channel (403 on every tick) or is being rate limited could not be told
apart from a working one without enabling DEBUG.

Log the first failure of each typing loop at WARNING with the traceback,
and later ticks of that loop at DEBUG so a retrying loop cannot flood the
log. The level is per loop, so the next reply's loop warns again.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-02 22:49:28 +08:00
huyengiang101086-spec
baae72d876
fix(channels): point the Discord missing-dependency hint at the discord extra (#6054)
* test(scripts): cover discord extra auto-detection

The discord mapping in scripts/detect_uv_extras.py worked but was neither
listed in the "currently maps" docstring nor covered by a test, unlike the
buzz mapping that grew up beside it.

* fix(channels): point the Discord missing-dependency hint at the discord extra

discord.py is the only IM transport that is not a core dependency — it ships in
the optional `discord` extra, which detect_uv_extras.py, backend/Dockerfile and
deploy.sh already expand. Its ImportError hint told users to run `uv add
discord.py` instead, which rewrites pyproject.toml/uv.lock in a tree every
restart syncs with `--locked`. Mirror the buzz hint wording and note the extra
in config.example.yaml next to the buzz block.

---------

Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
2026-10-01 10:51:37 +08:00
Hyeonsang Cho
fe22e3899a
fix(channels): keep blocking SDK teardown off the event loop in stop() (#6134)
* fix(channels): keep blocking SDK teardown off the event loop in stop()

SlackChannel.stop() called SocketModeClient.close() inline. close() joins
the SDK's message-processor thread (~0.7s every call, measured) and waits
for in-flight listeners, whose blocking Web API calls can run up to the
WebClient timeout, so every run, stream and channel on the Gateway stalled
during shutdown and POST /api/channels/{name}/restart.

Feishu and DingTalk had the same shape: they joined their SDK threads
inline with a 5s timeout, and since those threads only return on a fatal
error, the join normally froze the loop for the full 5s.

The teardown now runs through asyncio.to_thread. Slack detaches the client
first so a still-queued connect skips it, and shields and tracks the close
task so a cancelled stop() leaves it running and a retried stop() awaits
the same close instead of closing twice.

tests/blocking_io/test_channel_stop_offloop.py anchors all three stop()
paths to the strict Blockbuster gate, driving a real slack-sdk client.

* docs(changelog): reference #6134

* fix(channels): join the Discord client thread off the event loop in stop()

DiscordChannel.stop() still joined its client thread inline with a 10s
timeout. The thread usually exits right after the cross-loop client close,
but a timed-out close or a slow _run_client() drain keeps it alive for the
whole timeout, stalling the shared Gateway loop the same way the Slack,
Feishu and DingTalk teardown did. Run the join through asyncio.to_thread so
stop() matches the channels/AGENTS.md teardown rule, and add Discord to the
strict Blockbuster anchor.
2026-10-01 10:00:53 +08:00
FanouZeng-TT
4006ca072f
fix(channels): keep the delimiter newline when splitting long Discord replies (#6062)
* fix(channels): preserve Discord split boundaries

* test(channels): cover Discord split at leading remainder newline

Signed-off-by: Willem Jiang <willem.jiang@gmail.com>

---------

Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-30 10:17:46 +08:00
Kris
1d8b6ead0b
chore(discord): best-effort flush thread mappings on stop() + restart regression tests (#5461)
* fix(discord): flush thread mappings on shutdown to survive restart (#2897)

The Discord channel already persists channel->thread mappings, but a mapping created right before a hard shutdown (process killed between thread creation and its background persistence write) could still be lost. Flush in-memory mappings in stop() as a best-effort safety net.

Add regression tests covering the persist/load round-trip across a simulated restart and the stop() flush path.

Refs #2897

* fix(discord): gate stop() flush on _thread_store_loaded (#5461 review)

Address review feedback:
- Only flush thread mappings from stop() once _load_active_threads() has
  run, so a stop() taken before the load (start() bailed on a missing
  bot_token / discord import error) cannot overwrite the store with {}.
- Fix isort order in the test module (ruff I001).
- Make the restart test a plain sync test (it never awaits).
- Add a regression test proving stop() does not clobber the store before load.

* fix(discord): gate stop() flush on _thread_store_loaded (#5461 review)

Address review feedback:
- Only flush thread mappings from stop() once _load_active_threads() has
  run, so a stop() taken before the load (start() bailed on a missing
  bot_token / discord import error) cannot overwrite the store with {}.
- Fix isort order in the test module (ruff I001).
- Make the restart test a plain sync test (it never awaits).
- Add a regression test proving stop() does not clobber the store before load.

---------

Co-authored-by: wcy12378 <wcy12378@users.noreply.github.com>
2026-09-16 15:29:14 +08:00
hataa
37b03a3811
fix(channels): bound Discord outbound cross-loop awaits and restart dead clients, fixes #5226 (#5227)
* fix(channels): bound Discord outbound cross-loop awaits and restart dead clients, fixes #5226

* docs: reduce inherited agent guidance size

* fix(channels): stop half-started channels before discarding them

_start_channel now tears the instance down (stop + untrack) whenever start()
raises or the channel never reaches is_running, so an outbound listener
subscribed before the transport was confirmed cannot outlive its channel.
Addresses the review on #5227.

* fix(channels): retain half-started channels until failed-start cleanup completes

Ownership in _stop_and_discard_channel now mirrors ChannelService.stop(): the instance is dropped only after its stop() completes. A cancellation arriving mid-cleanup (or a stop() that raises) leaves it tracked, so a retried readiness attempt stops it again before replacing it and service shutdown can still reach it — untracking first orphaned resources nobody could clean up. Addresses the round-3 review on #5227.

* fix(channels): defer replacement when a retained channel fails to stop

The pre-retry stop in ensure_channel_ready popped unconditionally, so a
retained instance whose second stop() raised was untracked with its
outbound listener still subscribed — the same orphan one hop later.
restart_channel (del after failed stop) and remove_channel (pop before
stop) had the same shape. All three now route through
_stop_and_discard_channel and decline the operation for that round when
the instance is retained, so _start_channel can never overwrite a
still-listening channel. Addresses the review on #5227.

* fix(channels): enforce the retention guarantee inside the readiness attempt loop

A failed attempt whose cleanup retained the instance used to let the next
attempt (attempts=2 is the production default) construct a fresh instance
and overwrite the retained one via _start_channel's unconditional
assignment — orphaning the first instance's subscribed listener one hop
earlier than the cross-round guard covers. The guard now lives at the
mechanism: _start_channel refuses to install while the name is still
tracked, and ensure_channel_ready ends the loop on retention. The shared
discard helper's log message is path-neutral. Addresses the review on
#5227.

* fix(channels): make the retained-instance guard message path-neutral

The guard can fire for any still-tracked instance, not only failed
cleanup, so the message must not assume the cause.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-10 14:36:55 +08:00
Sunshine
9600414002
fix(channels): retain strong reference to Discord ack-reaction tasks (#5049)
* fix(channels): retain strong reference to Discord ack-reaction tasks

The event loop keeps only weak references to scheduled tasks, so the bare
asyncio.create_task(self._add_reaction(message)) at both message-acceptance
sites could be garbage-collected mid-flight and silently drop the ✅
acknowledgment. The module-level retention set holds each task until
completion and logs failures, mirroring the deferred subagent cleanup fix
in #4928 and the stream bridge cleanup fix in #4931.

* fix(channels): drain in-flight ack reactions on Discord channel shutdown

Adversarial follow-up to the retention fix: stop() cancelled typing tasks
but left in-flight ack-reaction tasks in the module retention set when the
channel stopped mid-HTTP-call, pinning the channel instance and the discord
Message object graph across restart cycles. _cancel_ephemeral_tasks /
_discard_ephemeral_tasks now cover both families at every stop() cleanup
point and the _run_client finally. Also tests the done-callback error-log
branch and documents the GC test as a contract pin.

* test(channels): pin Discord ack-reaction drain wiring and sharpen wording

Second adversarial pass found the drain test cancelled a never-started
task (no yield between schedule and cancel), nothing exercised the real
stop() cross-loop wiring, and the cleanup log messages still said
"typing-task" while now covering both families. The drain test now yields
so the task genuinely suspends first, a new test drives stop() through a
real background loop and fails if the wiring reverts to typing-only
cleanup, wording covers ephemeral tasks, and the discard path snapshots
the set before iterating.

* refactor(channels): scope Discord ack-reaction retention to the channel instance

The retention set was module-level, so one DiscordChannel's stop() would
cancel every other instance's in-flight ack reactions. Instance-level
matches the typing-task bookkeeping and keeps shutdown scoped; pinned by
a two-channel isolation test.

* test(channels): drive two-channel ack isolation through full stop()

Independent review of the instance-scoping refactor confirmed the P2 fix
but suggested matching the review's letter exactly: the isolation test now
shuts the first channel down via stop() instead of the drain helper, and
one stale docstring reference to the former module-level set is corrected.
2026-08-27 17:56:56 +08:00
Eilen Shin
ce4ef1bb2f
fix(channels): bound inbound intake and worker lifecycle (#4800)
* fix(channels): bound inbound intake and worker lifecycle

* fix(channels): harden overload retry and shutdown draining

* fix(channels): retain shutdown task ownership
2026-08-14 12:35:24 +08:00
AoHanBei
df01102dfc
fix(discord): prevent typing tasks after stop (#4752)
* fix(discord): prevent typing tasks after stop

* fix(discord): serialize typing cleanup on event loop

* fix(discord): harden typing cleanup on loop exit
2026-08-11 21:54:19 +08:00
hataa
e9161ff148
fix(channels): offload blocking filesystem IO in Discord channel (#3927)
DiscordChannel ran synchronous filesystem IO on the event loop: thread-mapping
persistence/restore and outbound attachment reads. Offload all of it via
asyncio.to_thread:
- start() -> _load_active_threads (restore mappings on startup)
- _on_message -> _persist_thread_mappings (flush mappings to disk)
- send_file -> _read_attachment_bytes (read bytes; handed to discord.File as
  an in-memory BytesIO buffer)

Thread-mapping state is split to avoid a race surfaced in review (#3927):
_record_thread_mapping updates the in-memory _active_threads dict and
_active_thread_ids set synchronously on the event loop, so a follow-up message
in a newly created thread is recognized immediately — before the offloaded
persistence write completes. Deferring that update into the worker thread
opened a window where _on_message's membership check misclassified the message
as orphaned and created a duplicate thread.

__init__ only computes paths, so construction stays IO-free.

Blockbuster regression tests cover the IO-free constructor, the
record-then-persist split (memory visible before persistence), discard of a
replaced thread id, and the load path.

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-04 16:16:09 +08:00
Nan Gao
68ba4198b8
fix(channels): make channel connect flow deterministic (#3582)
* fix(channels): make channel connect flow deterministic

* make format

* fix(channels): apply connect-code before allowed_users on telegram and wechat

The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and
WeChat still gate _check_user/allowed_users before connect-code dispatch, so
a newly allowlisted-but-unbound user is silently rejected when binding via the
browser deep-link / connect-code flow — the same deadlock the PR fixes.

- telegram: consume the /start deep-link token before the allowed_users gate.
- wechat: handle the /connect code before the allowed_users gate, and defer
  inbound file extraction + context-token tracking past the gate so blocked
  senders no longer trigger CDN downloads or token bookkeeping.

Adds regression tests for both adapters mirroring the slack/dingtalk coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(channels): enforce single-active-owner invariant at the DB layer

_revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row
lock or constraint covering active rows. Under READ COMMITTED, two concurrent
connect-code consumes for the same (provider, external_account_id, workspace_id)
from different owners could each observe "no other active owner" and both commit
a connected row, leaving find_connection_by_external_identity nondeterministic.

- Add a partial unique index on (provider, external_account_id, workspace_id)
  WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the
  database guarantees at most one non-revoked row per external identity.
- Reorder upsert_connection to revoke other owners' active rows before the new
  connected row is flushed (so the index is satisfied at commit), wrapped in a
  bounded rollback-and-retry loop. A losing concurrent writer now retries
  against the now-visible state instead of committing a duplicate.

Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(channels): harden connect-status polling primitive

pollChannelConnectionUntilResolved was a free-floating recursive setTimeout
started from onSuccess with no cancellation, no per-provider dedup, a redundant
second endpoint per tick, and an unbounded loop on a non-finite expires_in.

- Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls
  only listChannelConnections() and invalidates the providers query once when the
  bind resolves, instead of fetching both endpoints every tick.
- Guard expires_in with a finite check + default window so undefined/NaN can no
  longer produce a poll loop that runs until the page closes.
- Track one active poll handle per provider in useConnectChannelProvider via a
  ref Map: a new connect cancels the prior poll for that provider, and a useEffect
  cleanup cancels all polls on unmount.

Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics

Moving the allowed_users gate past _extract_text meant the parsed-message INFO
log (text=%r, first 100 chars) fired for senders that allowed_users would have
rejected, defeating the filter's noise/privacy role. Move that log to after the
allowed_users gate so blocked senders' message text never reaches INFO logs.

Also document the two operator-relevant semantic changes in backend/CLAUDE.md:
connect-code dispatch runs before allowed_users (so allowed_users is no longer a
bind-time defense; the model relies on code confidentiality + 600s TTL + one-time
consumption), and the single-active-owner-per-external-identity transfer semantics
now backed by the partial unique index.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide

Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md:
connect codes are consumed before allowed_users (so a not-yet-allowlisted user can
still complete a first bind, and allowed_users is not a bind-time defense), and an
external identity has at most one active owner with last-bind-wins transfer enforced
at the DB layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(channels): lift connect-code dispatch into Channel base class

Each adapter duplicated the ordering-sensitive boilerplate of extracting a
/connect code and guarding on the connection repo before its allowed_users gate.
The duplication is what let telegram/wechat drift and keep the gate ahead of the
bind. Centralize it:

- Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments).
- Add Channel._pending_connect_code(text), which guards on the repo and extracts
  the code, documenting that adapters MUST consult it before authorization so a
  browser-initiated bind can bootstrap a not-yet-authorized identity.
- Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper.
  This also fixes a latent inconsistency where slack dispatched a bind even when
  no connection repo was configured.

Pure refactor — the full channel suite stays green; adds a direct unit test for
the base helper's contract.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* make format

* fix(channels): redact DingTalk parsed-message INFO log content

Log text_len instead of the first 100 chars of message text, so message
content never reaches INFO logs (the after-gate move already keeps blocked
senders out entirely). This takes over the redaction from #3584 so only this
PR touches dingtalk.py, letting the two PRs merge in any order conflict-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 10:15:31 +08:00
hataa
1783da42f4
fix(channels): close Discord file handle after upload (#3561)
send_file opened the attachment with a bare open() and never closed it,
leaking a file descriptor on every Discord file delivery. The handle was
also leaked on the failure path: when target.send raised, the except
branch logged and returned without closing fp. The "# noqa: SIM115"
suppressed the lint warning instead of fixing it.

Wrap open() in a with statement that stays open for the full upload —
the discord client reads fp while target.send runs on _discord_loop, and
once that future resolves the bytes are consumed, so closing here is
safe. This closes the handle on both the success and exception paths and
matches how telegram and feishu already handle their file uploads.

Adds regression tests asserting the handle is closed after send_file on
both the success and failure paths.

Refs #3544
2026-06-13 23:27:17 +08:00
DanielWalnut
aa015462a7
feat(im): Add user-owned IM channel connections (#3487)
* Add user-owned IM channel connections

* Fix dev startup and channel connect popup

* Use async channel connect flow

* Harden dev service daemon startup

* Support local IM channel connections

* Align IM connections with local channels

* Fix safe user id digest algorithm

* Address Copilot IM channel feedback

* Address IM channel review comments

* Support all integrated IM channel connections

* Format additional channel connection tests

* Keep unavailable channel connect buttons clickable

* Fix IM channel provider icons

* Add runtime setup for enabled IM channels

* Guard global shortcut key handling

* Keep configured IM channels editable

* Avoid password autofill for channel secrets

* Make channel threads visible to connection owners

* Persist IM runtime config locally

* Allow disconnecting runtime IM channels

* Route no-auth channel sessions to local user

* Use default user for auth-disabled local mode

* Show IM channel source on threads

* Prefill IM channel runtime config

* Reflect IM channel runtime health

* Ignore Feishu message read events

* Ignore Feishu non-content message events

* Let setup wizard enable IM channels

* Fix frontend formatting after merge

* Stabilize backend tests without local config

* Isolate channel runtime config tests

* Address channel connection review comments

* Use sha256 user buckets with legacy migration

* Ensure runtime IM channels are ready after restart

* Persist disconnected IM channel state

* Address channel connection review comments

* Address channel connection review findings

Frontend connect flow:
- Open the runtime-config dialog only when a provider still needs
  credentials; configured providers go straight to the connect flow, so
  the binding-code/deep-link path is reachable from the UI again.
- After saving credentials, continue into the connect flow when a user
  binding is still required (multi-user mode) instead of stopping at a
  "Connected" toast.
- Extract shared provider-state helpers to core/channels/provider-state
  and add unit + e2e coverage for the direct-connect and
  configure-then-connect paths.

Provider status semantics:
- Report connection_status from the user's newest connection row;
  with no binding it is not_connected, except in auth-disabled local
  mode where a configured running channel is effectively connected.

Concurrency and event-loop correctness:
- Offload ChannelRuntimeConfigStore construction and writes, channel
  service construction, and Slack connection replies to threads; add a
  tests/blocking_io/ anchor for the runtime-config handlers.
- Consume binding codes with a conditional UPDATE so a code can only be
  used once under concurrent workers; retry upsert_connection as an
  update when a concurrent insert wins the unique constraint.
- Serialize ensure_channel_ready per channel so concurrent provider
  polls cannot double-start a channel worker.

Config and migration hardening:
- Stop mutating the get_app_config()-cached Telegram provider config;
  the runtime store now owns the UI-entered bot username.
- Register channel_connections in STARTUP_ONLY_FIELDS with the
  standardized startup-only Field description.
- Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name
  so another user's same-prefix bucket can never be migrated.
- Remove the unused Telegram process_webhook_update path and document
  src/core/channels in the frontend docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address PR review comments on authz scoping and channel runtime

Security (review feedback from ShenAC-SAC):
- Scope internal-token callers to the connection owner carried in
  X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright,
  in both require_permission(owner_check=True) and the stateless run
  endpoints. Internal callers keep access to their own and
  shared/legacy threads, and may claim a default-owned channel thread
  for its real owner, but a leaked internal token no longer grants
  cross-user thread access.
- Require admin privileges for POST/DELETE /api/channels/{provider}/
  runtime-config: runtime credentials and channel workers are
  instance-wide shared state (same model as the MCP config API).
  Read-only provider listing stays available to all users.

Performance (review feedback from willem-bd):
- Skip the redundant thread channel-metadata PATCH after the first
  successful backfill per thread.
- Reuse the per-connection Slack WebClient until its token changes
  instead of constructing one per outbound message.
- Reconcile channel readiness for all providers concurrently in
  GET /api/channels/providers.

Also resolve the code-quality unused-import flag in the blocking-io
anchor by pre-importing the channel service via importlib.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix prettier formatting in provider-state test

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Reconcile UI runtime channel config with config reload on restart

Main now reloads a channel's config.yaml entry on restart_channel()
(#3514, issue #3497). Adapt the user-owned connection flow to coexist:

- configure_channel() restarts with reload_config=False — the caller
  just supplied the authoritative config (browser-entered credentials
  that are never written to config.yaml), so a file reload must not
  clobber it with the stale on-disk entry.
- _load_channel_config() re-applies the UI runtime-store overlay used
  at startup, so an operator-triggered restart keeps browser-entered
  credentials for channels without a config.yaml entry and does not
  resurrect a channel disconnected from the UI.
- Offload the reload's disk IO (config.yaml + runtime store) with
  asyncio.to_thread, matching the blocking-IO policy on this branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 15:24:58 +08:00
DanielWalnut
16391e35ab
fix(skills): harden slash skill activation across chat channels (#3466)
* support slash skill activation

* format slash skill activation

* Preserve slash skill activation with uploads

* Address slash skill review feedback

* Address slash skill follow-up review

* Fix lazy slash skill storage resolution

* Keep slash skill activation out of system prompt

* Address slash skill review issues

* fix: harden slash skill command handling

* feat(frontend): add slash skill autocomplete

* fix: address slash skill review feedback

* fix: preserve slash skill text for IM uploads
2026-06-09 23:07:17 +08:00
Yi Tang
48e038f752
feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators (#2842)
* feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators

Add mention_only config to only respond when bot is mentioned, with
allowed_channels override. Add thread_mode for Hermes-style auto-thread
creation. Add periodic typing indicators while bot is processing.

* fix(discord): include allowed_channels in mention_only skip condition (line 274)

* docs: fix Discord config example to match boolean thread_mode implementation

* style: format with ruff

* fix(discord): apply Copilot review fixes and resolve lint errors

- Remove unused Optional import
- Fix thread_ts type hints to str | None
- Fix has_mention logic for None values
- Implement thread_mode fallback to channel replies on thread creation failure
- Fix thread_mode docstring alignment
- Fix allowed_channels comment formatting in config.example.yaml

* fix(discord): reset context for orphaned threads in mention_only mode

When a message arrives in a thread not tracked by _active_threads,
clear thread_id and typing_target so the message falls through to
the standard channel handling pipeline, which creates a fresh thread
instead of incorrectly routing to the stale thread.

* fix(discord): create new thread on @ when channel has existing tracked thread

When mention_only is enabled and a user @-s the bot in a channel
that already has a tracked thread, create a new thread instead of
incorrectly routing to the old one.

* fix(discord): allow no-@ thread replies while skipping no-@ channel messages

The skip block for no-@ messages was too aggressive — it blocked
continuation replies within tracked threads AND incorrectly routed
no-@ channel messages to the existing thread.

Now:
- Thread message, no @ → routed to existing tracked thread
- Channel message, no @ → skipped
- Channel message, with @ → creates new thread

* feat(discord): add checkmark reaction to acknowledge received messages

* Move discord.py to optional dependency and auto-detect from config.yaml

- Add discord extra to [project.optional-dependencies] in pyproject.toml
- Update detect_uv_extras.py to map channels.discord.enabled: true -> --extra discord
- Set UV_EXTRAS=discord in docker-compose-dev.yaml gateway env

* fix(discord): persist thread-channel mappings to store for recovery after restart

Discord's _active_threads dict was purely in-memory, so all channel-to-thread
mappings were lost on server restart. This fix bridges ChannelStore into
DiscordChannel:

- Save thread mappings to store.json after every thread creation
- Restore active threads from store on DiscordChannel startup
- Pass channel_store to all channels via service.py config injection

Store keys follow the pattern: discord:<channel_id>:<thread_id>

* fix(discord): address Copilot review — fix types, typing targets, cross-thread safety, and config comments

* fix(tests): add multitask_strategy param to mock for clarification follow-up test

* fix(tests): explicitly set model_name=None for title middleware test isolation

* fix(discord): use trigger_typing() instead of typing() for typing indicators

discord.py 2.x TextChannel.typing() and Thread.typing() are async context
managers, not one-shot coroutines. Use trigger_typing() for periodic
typing indicator pings.

* fix(discord): cancel typing tasks on channel shutdown

Prevents 'Task was destroyed but it is pending' warnings when the
Discord client stops while typing indicator loops are still running.

* fix(scripts): detect nested YAML config for discord extra

section_value() only matched top-level YAML sections. Added
nested_section_value() that handles two-level nesting (e.g.,
channels.discord.enabled), so auto-detection of the discord
extra works when config uses the standard nested format.

* fix(docker): remove hard-coded UV_EXTRAS=discord from dev compose

Relies on auto-detection via detect_uv_extras.py instead of forcing
discord.py install even when channels.discord.enabled is false.
Matches production docker-compose.yaml behavior (UV_EXTRAS:-).

* refactor(nginx): move proxy_buffering/proxy_cache to server level

DRY cleanup — these directives were repeated in 14 location blocks.
Set at server level once, reducing duplication and risk of drift.

* fix(discord): use dedicated JSON file for thread persistence

Replace ChannelStore usage for Discord thread-ID persistence with a
dedicated discord_threads.json file. ChannelStore is designed to map
IM conversations to DeerFlow thread IDs — using it to persist Discord
thread IDs was semantically wrong and confusing.

Changes:
- _save_thread() now reads/writes a simple {channel_id: thread_id} JSON dict
- _load_active_threads() reads directly from the JSON file
- File path derived from ChannelStore directory (when available) or
  defaults to ~/.deer-flow/channels/discord_threads.json
- Removed unused ChannelStore import

* fix(discord): address WillemJiang's code review comments on PR #2842

1. Remove semantically incorrect message_in_thread variable. At this code
   point (after the Thread case is handled above), we're guaranteed to be in
   a channel, not a thread. Always apply mention_only check here.

2. Add _active_thread_ids reverse-lookup set for O(1) thread ID membership
   checks instead of O(n) scan of _active_threads.values(). Keep the set
   in sync with _active_threads in _load_active_threads() and _save_thread().

3. Add _thread_store_lock (threading.Lock) to protect _active_threads and
   the JSON file from concurrent access between the Discord loop thread
   (_run_client) and the main thread (_load_active_threads, _save_thread).
2026-05-15 22:30:05 +08:00
Matt Van Horn
c4d273a68a
feat(channels): add Discord channel integration (#1806)
* feat(channels): add Discord channel integration

Add a Discord bot channel following the existing Telegram/Slack pattern.
The bot listens for messages, creates conversation threads, and relays
responses back to Discord with 2000-char message splitting.

- DiscordChannel extends Channel base class
- Lazy imports discord.py with install hint
- Thread-based conversations (each Discord thread maps to a DeerFlow thread)
- Allowed guilds filter for access control
- File attachment support via discord.File
- Registered in service.py and manager.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(channels): address Copilot review suggestions for Discord integration

- Disable @everyone/@here mentions via AllowedMentions.none()
- Add 10s timeout to client close to prevent shutdown hangs
- Log publish_inbound errors via future callback instead of silently dropping
- Open file handle on caller thread to avoid cross-thread ownership issues
- Notify user in channel when thread creation fails

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(discord): resolve lint errors in Discord channel

- Replace asyncio.TimeoutError with builtin TimeoutError (UP041)
- Remove extraneous f-string prefix (F541)
- Apply ruff format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tests): remove fake langgraph_sdk shim from test_discord_channel

The module-level sys.modules.setdefault shim installed a fake
langgraph_sdk.errors.ConflictError during pytest collection. Because
pytest imports all test modules before running them, test_channels.py
then imported the fake ConflictError instead of the real one.

In test_handle_feishu_stream_conflict_sends_busy_message, the test
constructs ConflictError(message, response=..., body=...). The fake
only subclasses Exception (which takes no kwargs), so the construction
raised TypeError. The manager's _is_thread_busy_error check then saw a
TypeError instead of a ConflictError and fell through to the generic
'An error occurred' message.

langgraph_sdk is a real dependency, so the shim is unnecessary.
Removing it makes both test files import the same real ConflictError
and the full suite pass (1773 passed, 15 skipped).

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-11 17:48:04 +08:00