mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-16 00:10:42 +00:00
26 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3f1438d4a0
|
fix(channels): publish Discord and WeChat JSON state atomically (#7595)
* fix(channels): publish Discord and WeChat JSON state atomically * fix(channels): adopt the shared atomic publisher for both state writers * test(channels): pin the cleanup-failure direction too * test(channels): keep a platform-independent guard on the publish call shape * style(tests): apply ruff format to the channel encoding test |
||
|
|
ab8cd4e10f
|
fix(channels): keep IM binding codes on screen and report goal state in replies (#7591)
* fix(channels): keep IM binding codes on screen and report goal state in replies Connecting Slack, Discord, Feishu, DingTalk, WeCom, Buzz or QQ now opens a dialog that keeps the /connect command on screen with a copy button, its expiry time and the binding status, instead of a toast that disappears. The bots' /connect replies and the IM /goal replies now use the owner's web UI language (else channel_connections.notification_locale), and /goal reports the goal's state as the web goal bar does: in progress, why it stopped, or that it was met. * docs(changelog): note the IM binding dialog and localized IM replies (#7591) * test(channels): check IM reply tables against every notification locale The connect-reply check compared a fixed en-US/zh-CN pair, and the /goal reply table had none. Both now match NOTIFICATION_LOCALES with the same keys in each, like the existing _GOAL_REASON_TEXT check, so a new locale fails here instead of raising KeyError at reply time. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
|
1ad8147947
|
fix(channels): pin JSON state encoding across platforms (#7228)
* fix(channels): pin JSON state encoding across platforms * test(channels): check encoding pins at each state I/O call --------- Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
085924a3d0
|
fix(channels): run each IM message once across instances, restarts and late redeliveries (#7104)
* fix(channels): run each IM message once across instances, restarts and late redeliveries Inbound IM messages were deduplicated only by the 10-minute inbound dedupe store (#7103). Discord direct messages had no key at all, a Discord thread reply got an instance-dependent key, a Discord thread was created before dedupe ran, a redelivery past the window started a second run, and two instances racing on a conversation's first messages bound it to two threads. - Discord messages are keyed (discord, guild_id or "dm", posting channel, message_id); the posting channel is the same on every instance. - MessageBus.claim_inbound lets an adapter record the manager's dedupe key before a side effect; Discord claims it on the Gateway loop before _create_thread, marks the published message dedupe_claimed, and gives the claim back when the message is never committed. - runs.wait / runs.stream / runs.create send an Idempotency-Key derived from the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per attempt made every keyed retry a 409). A reused run's status-only /wait or stream gap is resolved to this message's own turn via threads.get_state; 409s about the message's own keyed run keep the key instead of reading as a busy thread; an ended reused GitHub run gets no follow-up watcher. - A conversation's first thread binding is claimed insert-if-absent (claim_thread_id on channel_conversations and channel_thread_bindings); the loser adopts the winner and deletes the empty thread it minted. /new and /agent use still rebind. Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md, CHANGELOG (en/zh). * fix(channels): give back Discord dedupe claims for messages never processed Review round 1 on #7104. A Discord claim records the manager's dedupe key before the message is queued, so every path that drops the message unrun must release it, or a peer refuses the redelivery for the rest of the TTL: - The hand-off task releases the claim on cancellation too (stop() drains it while it resolves the connection identity), via await_drained so the cancellation still propagates. - ChannelManager._discard_pending_inbound releases the key of a claimed message that shutdown drops from the queue, after the grace period and on a cancelled shutdown. - The adapter tracks claims until they are committed. A release refused because stop() already closed cross-loop intake, or a hand-off cancelled before it ran, leaves the claim tracked, and stop() releases it last. * fix(gateway): keep IM run idempotency stable when /new rebinds the conversation Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner and thread. /new rebinds an IM conversation to a fresh thread, so a late redelivery of a message admitted on the old thread was requested on the new one, got a different scoped key and started a second run. - Keys from trusted internal callers (the IM channel manager) are scoped by owner only (`internal-run:` prefix); browser and API keys keep the owner-and-thread scope (`http-run:`). - A key that resolves to a run of another thread or user raises RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers see the same type, and start_run answers it with 409 "Idempotency-Key already used with a different request" instead of a 500. The channel manager already treats that 409 as the message's admitted run. Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay under its budget), CHANGELOG (en/zh). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
32d7dd5fe8
|
fix(discord): accept inbound attachments and file-only messages (#6857)
* fix(discord): accept inbound attachments and file-only messages Empty content no longer drops messages that have attachments. Capture CDN descriptors on inbound, download in receive_file via Bot auth, and hand bytes through INBOUND_FILE_CONTENT_KEY like Slack/Telegram. Closes #6856 * fix(discord): gate bot auth and share inbound file helpers Address review on #6857: attach Bot Authorization only for Discord CDN hosts, report Content-Encoding refusals as download failed, and hoist filename/size helpers into app.channels.inbound_files (Telegram delegates to the same module). * fix(discord): sort telegram imports for ruff I001 CI lint-backend failed on unsorted imports in telegram.py after the shared inbound_files helper was introduced. * style: ruff-format discord inbound attachments test * fix(discord): show the enforced attachment download limit --------- Co-authored-by: Li-john1021 <Li-john1021@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
9b2151a3a8
|
fix(buzz): a single mention_free_channels ID was split into characters, a numeric one stopped the connector (#6776)
* fix(buzz): accept a single mention-free channel id `_mention_free` iterated the configured value directly, so one unquoted UUID became a set of its characters and a numeric or boolean scalar raised TypeError while the channel was being constructed. Both shapes are documented as valid for the sibling Discord connector. Parse the key through a dedicated helper that follows the Discord convention: a scalar is a single entry, list entries are kept verbatim, and an unusable entry never matches so `require_mention` stays in force. * fix(channels): share scalar-or-list channel ID parsing * docs(channels): keep guidance within CI size budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
6c4c74a504
|
fix(channels): share chat-to-thread bindings through the database (#6558)
* fix(channels): share chat-to-thread bindings through the database ChannelManager kept its IM chat-to-thread bindings (channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file, channels/store.json, loaded once and rewritten whole on every change. With several Gateway processes a binding created on one replica was invisible to the others (the next message for the same chat opened a second thread) and concurrent writers clobbered each other's file. - New channel_thread_bindings table (migration 0036_channel_thread_bindings) and harness package deerflow.persistence.channel_thread_bindings: key PK (the legacy composite string), channel_name, chat_id, nullable topic_id, thread_id, user_id, epoch created_at/updated_at; index (channel_name, chat_id). Dialect upsert keeps created_at; prefix remove and batched ON CONFLICT DO NOTHING import. - app.channels.store.ChannelStore is an async protocol: SqlChannelStore for sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for database.backend: memory. resolve_channel_store picks at ChannelService construction; ChannelService.start() imports an existing store.json once into an empty table and renames it store.json.migrated (a concurrently importing peer neither duplicates nor clobbers; a populated table leaves the file alone). - Callers updated: manager, connection_identity.lookup_thread_id, Feishu (send() awaits the mapping writer; the synchronous lark callback bridges its two store calls to the Gateway loop with a bounded wait), Discord (channels_dir). - Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md, README) and CHANGELOG updated. * docs(changelog): link the channel thread bindings entry to #6558 * fix(channels): reject only the Gateway loop in the Feishu store bridge lark-oapi 1.5.5 invokes the message callback synchronously from its async frame handler, on the private event loop that FeishuChannel._run_ws starts for the SDK thread. The bridge's guard treated any running loop as unsafe, so every real inbound message skipped the stored-topic lookup and the pending-mapping restore: a reply whose parent card was mapped but whose root was not fell back to the unknown root, and could consume an unrelated pending clarification. Only a wait issued from the Gateway loop itself (the loop that executes the submitted coroutine) is refused now; the SDK's own loop may block for the bounded wait, as that thread blocked on the JSON rewrite before. Tests drive _on_message from a worker thread running its own loop, the production shape, for both the lookup and the write path, and pin the Gateway-loop refusal. * fix(channels): defer unresolved Feishu lookups and harden the store.json import Review follow-ups on the chat-to-thread binding store (#6558): - Import: a legacy entry whose key, channel_name, chat_id, topic_id, thread_id or user_id exceeds its column width is skipped and counted with the malformed ones instead of failing the whole batch on PostgreSQL (SQLite silently accepted the same row, so the backends diverged on one file). The corrupt-file warning now says what each caller does: the JSON store starts fresh, the import is skipped and the file kept. - Feishu bridge: store calls run on the Gateway loop under a 3s asyncio.wait_for (the SDK loop also services ws keepalives; the old 10s bound could stall it per message), and the lark-thread wait is that budget plus 1s. A deadline or store error is StoreLookupUnresolved, never a miss: _on_message no longer routes such a reply onto its unknown root (which created a second thread and later overwrote the parent alias). The message joins a per-chat FIFO drained on the Gateway loop with backoff for up to 30s, then is dropped with a warning. _on_message is split into parse / resolve / dispatch so the deferred path dispatches the same way; later messages for the chat queue behind the deferred one. * fix(tests): expect follow_redirects in the Jina client kwargs after #5808 #5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and updated test_jina_client.py, but test_jina_response_limit.py and test_jina_retry_after.py still pin the exact client kwargs to proxy and trust_env, so every backend-unit-tests shard that lands on them fails on main (the backend unit-test workflow only runs on pull requests, which is why the push to main did not surface it). Update the two expectations to include the new keyword. (cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050) * Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808" This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7. * fix(channels): bound deferred Feishu routing by the inbound queue capacity Review follow-up on #6558: a store outage let every subsequent message for a chat join the deferred queue with no capacity check (the bus reservation was only taken after routing succeeded), and each queued message started a fresh 30s retry budget when it reached the head, so a burst could be retained for N x 30s and grow without bound. - Intake is reserved BEFORE routing for every store-backed message (bus.reserve_inbound is thread-safe and already used from SDK threads), so deferred and live messages share the one configured bound, channels.inbound_queue_maxsize; a message that cannot be admitted is dropped at once with a warning, never queued or routed as a miss. - The reservation travels with the message into dispatch (committed by _prepare_inbound), is released on the batch path (the flush reserves at publish time), on drop after the budget, on a failed deferral, on dispatch failure and for every leftover when the drain task is cancelled by stop(). - The retry budget runs from the message's arrival: a queued message whose budget is spent when it reaches the head is dropped without a lookup. - The bounded resolver returns the unresolved outcome instead of raising it through the tracked task, which logged an ERROR for an expected, handled store timeout. - Upstream's new blocking-io test constructs the store by its old class name; it now uses JsonChannelStore. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
ca1cae6185
|
fix(discord): drain thread mapping persistence across cancellation (#6356)
* fix(discord): drain thread mapping persistence across cancellation * test(discord): retain mapping flush across cancellation |
||
|
|
2523be658b
|
fix(discord): read allowed_guilds as IDs and deny every guild on an unreadable allowlist (#6338)
* test(discord): expose allowlist parsing failures instead of opening the bot
config.example.yaml documents channels.discord.allowed_guilds as a value that
"can also be a single guild ID", but the constructor iterates it: a scalar
string becomes one entry per character, and an unquoted YAML int raises
TypeError. Whatever remains is gated by truthiness, so an allowlist with no
parseable entry leaves the set empty and admits every guild.
* fix(discord): read allowed_guilds as IDs, and fail closed when unreadable
The allowlist was built by iterating the configured value, so the documented
single-guild shorthand ("can also be a single guild ID" in
config.example.yaml) became one entry per character: allowed_guilds: "321"
admitted guilds 1, 2 and 3 while blocking 321, and an unquoted YAML int raised
TypeError in the constructor. The empty result then read as "no allowlist", so
a restriction the operator configured could open the bot to every guild the bot
joined.
Parse both ID lists through helpers that treat a scalar as a one-entry list and
drop unreadable entries with a warning, following the telegram channel: an
allowlist with no parseable entry now denies every guild and logs at ERROR.
Mention exemptions keep their IDs verbatim, where an unreadable entry only
leaves mention_only in force.
* docs(discord): explain allowlist parsing and deny-all behavior
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
aeeee2994f
|
fix(channels): run Discord connection-repository work on the Gateway loop (#6214)
* fix(channels): run Discord connection-repository work on the Gateway loop discord.py delivers messages on a private loop in the client thread, and the Discord adapter awaited the channel-connection repository there even though its SQLAlchemy engine and pool belong to the Gateway loop. On PostgreSQL the first Discord message after the Gateway used the pool failed with "got Future attached to a different loop" and was dropped; on SQLite a pool-exhausting burst failed with "Queue is bound to a different event loop" and left the pool's wait queue bound to the Discord loop, so the Gateway's own queries then failed the same way. Resolve the identity together with the intake commit, and run /connect binding, on the Gateway loop through _submit_threadsafe_coroutine, as Telegram, Feishu and DingTalk already do. Bind replies return through _run_on_discord_loop, and start()/stop() now open and drain the cross-thread submissions. * docs(changelog): reference #6214 for the Discord connection-loop fix * fix(channels): correct the unscheduled Discord bind comment and pin it When _submit_threadsafe_coroutine refuses the bind, base.py closes the coroutine, so consume_oauth_state never runs and the one-time code stays valid; only the Discord message is swallowed. Say that instead of claiming the code is consumed, and pin both halves with a test: the attempt is handled (never a chat turn) and the code still binds later. * fix(channels): undo Discord typing when the Gateway hand-off fails Since identity resolution moved to the Gateway loop, both inbound paths started typing and the ack reaction as soon as the hand-off was scheduled. A lookup that then raised (database outage, pool-checkout timeout) dropped the message but left an endless typing loop and a success reaction behind. The hand-off now returns its completion future (new Channel._submit_threadsafe_coroutine_future, which the bool helper wraps) and the Discord side waits on it through asyncio.shield. Typing still registers before the hand-off settles, so a fast reply's _stop_typing cannot run first; the ack reaction waits for the commit. On failure, or when stop() drains the hand-off, the indicator this message started is stopped (never one another in-flight message owns) and no ack is added. * fix(channels): keep shared Discord typing alive for dependent messages Two messages to the same target share one typing task: the first starts it, a later one reuses it. If the creator's hand-off failed after a follower had committed, or while the follower was still pending, the creator stopped the indicator the follower relied on, and nothing restored it while the follower's request ran. Count the hand-offs that depend on each typing task. A pending or committed hand-off holds the task (a commit keeps its hold until a reply stops the indicator), a failed one releases its hold, and the indicator is stopped only when the failed message started it and no hold remains. The count is dropped when the task ends or the channel cancels typing. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
9a97cb9420
|
fix(channels): send Discord typing indicators through the discord.py 2.x API (#6138)
* fix(channels): send Discord typing indicators through the discord.py 2.x API _start_typing() called channel.trigger_typing(), which discord.py removed in 2.0; the project requires discord.py>=2.7.0. The typing loop swallowed every exception, so each tick raised AttributeError and no indicator was ever sent, and the tests never noticed because they mocked the removed method on a SimpleNamespace. Await channel.typing() instead (awaiting the returned Typing sends one indicator), and log a failed tick at DEBUG rather than discarding it. The existing typing-lifecycle tests now mock typing(), and new tests drive _start_typing() against a fake shaped like the 2.x Messageable, which has no trigger_typing. * docs(changelog): reference #6138 * fix(channels): warn on a Discord typing loop's first failed indicator Logging every failed typing tick at DEBUG left a permanently dead indicator invisible at the default INFO level, the same silence that hid the trigger_typing AttributeError. A bot that lost Send Messages in a channel (403 on every tick) or is being rate limited could not be told apart from a working one without enabling DEBUG. Log the first failure of each typing loop at WARNING with the traceback, and later ticks of that loop at DEBUG so a retrying loop cannot flood the log. The level is per loop, so the next reply's loop warns again. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
baae72d876
|
fix(channels): point the Discord missing-dependency hint at the discord extra (#6054)
* test(scripts): cover discord extra auto-detection The discord mapping in scripts/detect_uv_extras.py worked but was neither listed in the "currently maps" docstring nor covered by a test, unlike the buzz mapping that grew up beside it. * fix(channels): point the Discord missing-dependency hint at the discord extra discord.py is the only IM transport that is not a core dependency — it ships in the optional `discord` extra, which detect_uv_extras.py, backend/Dockerfile and deploy.sh already expand. Its ImportError hint told users to run `uv add discord.py` instead, which rewrites pyproject.toml/uv.lock in a tree every restart syncs with `--locked`. Mirror the buzz hint wording and note the extra in config.example.yaml next to the buzz block. --------- Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com> |
||
|
|
fe22e3899a
|
fix(channels): keep blocking SDK teardown off the event loop in stop() (#6134)
* fix(channels): keep blocking SDK teardown off the event loop in stop()
SlackChannel.stop() called SocketModeClient.close() inline. close() joins
the SDK's message-processor thread (~0.7s every call, measured) and waits
for in-flight listeners, whose blocking Web API calls can run up to the
WebClient timeout, so every run, stream and channel on the Gateway stalled
during shutdown and POST /api/channels/{name}/restart.
Feishu and DingTalk had the same shape: they joined their SDK threads
inline with a 5s timeout, and since those threads only return on a fatal
error, the join normally froze the loop for the full 5s.
The teardown now runs through asyncio.to_thread. Slack detaches the client
first so a still-queued connect skips it, and shields and tracks the close
task so a cancelled stop() leaves it running and a retried stop() awaits
the same close instead of closing twice.
tests/blocking_io/test_channel_stop_offloop.py anchors all three stop()
paths to the strict Blockbuster gate, driving a real slack-sdk client.
* docs(changelog): reference #6134
* fix(channels): join the Discord client thread off the event loop in stop()
DiscordChannel.stop() still joined its client thread inline with a 10s
timeout. The thread usually exits right after the cross-loop client close,
but a timed-out close or a slow _run_client() drain keeps it alive for the
whole timeout, stalling the shared Gateway loop the same way the Slack,
Feishu and DingTalk teardown did. Run the join through asyncio.to_thread so
stop() matches the channels/AGENTS.md teardown rule, and add Discord to the
strict Blockbuster anchor.
|
||
|
|
4006ca072f
|
fix(channels): keep the delimiter newline when splitting long Discord replies (#6062)
* fix(channels): preserve Discord split boundaries * test(channels): cover Discord split at leading remainder newline Signed-off-by: Willem Jiang <willem.jiang@gmail.com> --------- Signed-off-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
1d8b6ead0b
|
chore(discord): best-effort flush thread mappings on stop() + restart regression tests (#5461)
* fix(discord): flush thread mappings on shutdown to survive restart (#2897) The Discord channel already persists channel->thread mappings, but a mapping created right before a hard shutdown (process killed between thread creation and its background persistence write) could still be lost. Flush in-memory mappings in stop() as a best-effort safety net. Add regression tests covering the persist/load round-trip across a simulated restart and the stop() flush path. Refs #2897 * fix(discord): gate stop() flush on _thread_store_loaded (#5461 review) Address review feedback: - Only flush thread mappings from stop() once _load_active_threads() has run, so a stop() taken before the load (start() bailed on a missing bot_token / discord import error) cannot overwrite the store with {}. - Fix isort order in the test module (ruff I001). - Make the restart test a plain sync test (it never awaits). - Add a regression test proving stop() does not clobber the store before load. * fix(discord): gate stop() flush on _thread_store_loaded (#5461 review) Address review feedback: - Only flush thread mappings from stop() once _load_active_threads() has run, so a stop() taken before the load (start() bailed on a missing bot_token / discord import error) cannot overwrite the store with {}. - Fix isort order in the test module (ruff I001). - Make the restart test a plain sync test (it never awaits). - Add a regression test proving stop() does not clobber the store before load. --------- Co-authored-by: wcy12378 <wcy12378@users.noreply.github.com> |
||
|
|
37b03a3811
|
fix(channels): bound Discord outbound cross-loop awaits and restart dead clients, fixes #5226 (#5227)
* fix(channels): bound Discord outbound cross-loop awaits and restart dead clients, fixes #5226 * docs: reduce inherited agent guidance size * fix(channels): stop half-started channels before discarding them _start_channel now tears the instance down (stop + untrack) whenever start() raises or the channel never reaches is_running, so an outbound listener subscribed before the transport was confirmed cannot outlive its channel. Addresses the review on #5227. * fix(channels): retain half-started channels until failed-start cleanup completes Ownership in _stop_and_discard_channel now mirrors ChannelService.stop(): the instance is dropped only after its stop() completes. A cancellation arriving mid-cleanup (or a stop() that raises) leaves it tracked, so a retried readiness attempt stops it again before replacing it and service shutdown can still reach it — untracking first orphaned resources nobody could clean up. Addresses the round-3 review on #5227. * fix(channels): defer replacement when a retained channel fails to stop The pre-retry stop in ensure_channel_ready popped unconditionally, so a retained instance whose second stop() raised was untracked with its outbound listener still subscribed — the same orphan one hop later. restart_channel (del after failed stop) and remove_channel (pop before stop) had the same shape. All three now route through _stop_and_discard_channel and decline the operation for that round when the instance is retained, so _start_channel can never overwrite a still-listening channel. Addresses the review on #5227. * fix(channels): enforce the retention guarantee inside the readiness attempt loop A failed attempt whose cleanup retained the instance used to let the next attempt (attempts=2 is the production default) construct a fresh instance and overwrite the retained one via _start_channel's unconditional assignment — orphaning the first instance's subscribed listener one hop earlier than the cross-round guard covers. The guard now lives at the mechanism: _start_channel refuses to install while the name is still tracked, and ensure_channel_ready ends the loop on retention. The shared discard helper's log message is path-neutral. Addresses the review on #5227. * fix(channels): make the retained-instance guard message path-neutral The guard can fire for any still-tracked instance, not only failed cleanup, so the message must not assume the cause. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
9600414002
|
fix(channels): retain strong reference to Discord ack-reaction tasks (#5049)
* fix(channels): retain strong reference to Discord ack-reaction tasks The event loop keeps only weak references to scheduled tasks, so the bare asyncio.create_task(self._add_reaction(message)) at both message-acceptance sites could be garbage-collected mid-flight and silently drop the ✅ acknowledgment. The module-level retention set holds each task until completion and logs failures, mirroring the deferred subagent cleanup fix in #4928 and the stream bridge cleanup fix in #4931. * fix(channels): drain in-flight ack reactions on Discord channel shutdown Adversarial follow-up to the retention fix: stop() cancelled typing tasks but left in-flight ack-reaction tasks in the module retention set when the channel stopped mid-HTTP-call, pinning the channel instance and the discord Message object graph across restart cycles. _cancel_ephemeral_tasks / _discard_ephemeral_tasks now cover both families at every stop() cleanup point and the _run_client finally. Also tests the done-callback error-log branch and documents the GC test as a contract pin. * test(channels): pin Discord ack-reaction drain wiring and sharpen wording Second adversarial pass found the drain test cancelled a never-started task (no yield between schedule and cancel), nothing exercised the real stop() cross-loop wiring, and the cleanup log messages still said "typing-task" while now covering both families. The drain test now yields so the task genuinely suspends first, a new test drives stop() through a real background loop and fails if the wiring reverts to typing-only cleanup, wording covers ephemeral tasks, and the discard path snapshots the set before iterating. * refactor(channels): scope Discord ack-reaction retention to the channel instance The retention set was module-level, so one DiscordChannel's stop() would cancel every other instance's in-flight ack reactions. Instance-level matches the typing-task bookkeeping and keeps shutdown scoped; pinned by a two-channel isolation test. * test(channels): drive two-channel ack isolation through full stop() Independent review of the instance-scoping refactor confirmed the P2 fix but suggested matching the review's letter exactly: the isolation test now shuts the first channel down via stop() instead of the drain helper, and one stale docstring reference to the former module-level set is corrected. |
||
|
|
ce4ef1bb2f
|
fix(channels): bound inbound intake and worker lifecycle (#4800)
* fix(channels): bound inbound intake and worker lifecycle * fix(channels): harden overload retry and shutdown draining * fix(channels): retain shutdown task ownership |
||
|
|
df01102dfc
|
fix(discord): prevent typing tasks after stop (#4752)
* fix(discord): prevent typing tasks after stop * fix(discord): serialize typing cleanup on event loop * fix(discord): harden typing cleanup on loop exit |
||
|
|
e9161ff148
|
fix(channels): offload blocking filesystem IO in Discord channel (#3927)
DiscordChannel ran synchronous filesystem IO on the event loop: thread-mapping persistence/restore and outbound attachment reads. Offload all of it via asyncio.to_thread: - start() -> _load_active_threads (restore mappings on startup) - _on_message -> _persist_thread_mappings (flush mappings to disk) - send_file -> _read_attachment_bytes (read bytes; handed to discord.File as an in-memory BytesIO buffer) Thread-mapping state is split to avoid a race surfaced in review (#3927): _record_thread_mapping updates the in-memory _active_threads dict and _active_thread_ids set synchronously on the event loop, so a follow-up message in a newly created thread is recognized immediately — before the offloaded persistence write completes. Deferring that update into the worker thread opened a window where _on_message's membership check misclassified the message as orphaned and created a duplicate thread. __init__ only computes paths, so construction stays IO-free. Blockbuster regression tests cover the IO-free constructor, the record-then-persist split (memory visible before persistence), discard of a replaced thread id, and the load path. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
68ba4198b8
|
fix(channels): make channel connect flow deterministic (#3582)
* fix(channels): make channel connect flow deterministic * make format * fix(channels): apply connect-code before allowed_users on telegram and wechat The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and WeChat still gate _check_user/allowed_users before connect-code dispatch, so a newly allowlisted-but-unbound user is silently rejected when binding via the browser deep-link / connect-code flow — the same deadlock the PR fixes. - telegram: consume the /start deep-link token before the allowed_users gate. - wechat: handle the /connect code before the allowed_users gate, and defer inbound file extraction + context-token tracking past the gate so blocked senders no longer trigger CDN downloads or token bookkeeping. Adds regression tests for both adapters mirroring the slack/dingtalk coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(channels): enforce single-active-owner invariant at the DB layer _revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row lock or constraint covering active rows. Under READ COMMITTED, two concurrent connect-code consumes for the same (provider, external_account_id, workspace_id) from different owners could each observe "no other active owner" and both commit a connected row, leaving find_connection_by_external_identity nondeterministic. - Add a partial unique index on (provider, external_account_id, workspace_id) WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the database guarantees at most one non-revoked row per external identity. - Reorder upsert_connection to revoke other owners' active rows before the new connected row is flushed (so the index is satisfied at commit), wrapped in a bounded rollback-and-retry loop. A losing concurrent writer now retries against the now-visible state instead of committing a duplicate. Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(channels): harden connect-status polling primitive pollChannelConnectionUntilResolved was a free-floating recursive setTimeout started from onSuccess with no cancellation, no per-provider dedup, a redundant second endpoint per tick, and an unbounded loop on a non-finite expires_in. - Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls only listChannelConnections() and invalidates the providers query once when the bind resolves, instead of fetching both endpoints every tick. - Guard expires_in with a finite check + default window so undefined/NaN can no longer produce a poll loop that runs until the page closes. - Track one active poll handle per provider in useConnectChannelProvider via a ref Map: a new connect cancels the prior poll for that provider, and a useEffect cleanup cancels all polls on unmount. Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics Moving the allowed_users gate past _extract_text meant the parsed-message INFO log (text=%r, first 100 chars) fired for senders that allowed_users would have rejected, defeating the filter's noise/privacy role. Move that log to after the allowed_users gate so blocked senders' message text never reaches INFO logs. Also document the two operator-relevant semantic changes in backend/CLAUDE.md: connect-code dispatch runs before allowed_users (so allowed_users is no longer a bind-time defense; the model relies on code confidentiality + 600s TTL + one-time consumption), and the single-active-owner-per-external-identity transfer semantics now backed by the partial unique index. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md: connect codes are consumed before allowed_users (so a not-yet-allowlisted user can still complete a first bind, and allowed_users is not a bind-time defense), and an external identity has at most one active owner with last-bind-wins transfer enforced at the DB layer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(channels): lift connect-code dispatch into Channel base class Each adapter duplicated the ordering-sensitive boilerplate of extracting a /connect code and guarding on the connection repo before its allowed_users gate. The duplication is what let telegram/wechat drift and keep the gate ahead of the bind. Centralize it: - Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments). - Add Channel._pending_connect_code(text), which guards on the repo and extracts the code, documenting that adapters MUST consult it before authorization so a browser-initiated bind can bootstrap a not-yet-authorized identity. - Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper. This also fixes a latent inconsistency where slack dispatched a bind even when no connection repo was configured. Pure refactor — the full channel suite stays green; adds a direct unit test for the base helper's contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * make format * fix(channels): redact DingTalk parsed-message INFO log content Log text_len instead of the first 100 chars of message text, so message content never reaches INFO logs (the after-gate move already keeps blocked senders out entirely). This takes over the redaction from #3584 so only this PR touches dingtalk.py, letting the two PRs merge in any order conflict-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
1783da42f4
|
fix(channels): close Discord file handle after upload (#3561)
send_file opened the attachment with a bare open() and never closed it, leaking a file descriptor on every Discord file delivery. The handle was also leaked on the failure path: when target.send raised, the except branch logged and returned without closing fp. The "# noqa: SIM115" suppressed the lint warning instead of fixing it. Wrap open() in a with statement that stays open for the full upload — the discord client reads fp while target.send runs on _discord_loop, and once that future resolves the bytes are consumed, so closing here is safe. This closes the handle on both the success and exception paths and matches how telegram and feishu already handle their file uploads. Adds regression tests asserting the handle is closed after send_file on both the success and failure paths. Refs #3544 |
||
|
|
aa015462a7
|
feat(im): Add user-owned IM channel connections (#3487)
* Add user-owned IM channel connections
* Fix dev startup and channel connect popup
* Use async channel connect flow
* Harden dev service daemon startup
* Support local IM channel connections
* Align IM connections with local channels
* Fix safe user id digest algorithm
* Address Copilot IM channel feedback
* Address IM channel review comments
* Support all integrated IM channel connections
* Format additional channel connection tests
* Keep unavailable channel connect buttons clickable
* Fix IM channel provider icons
* Add runtime setup for enabled IM channels
* Guard global shortcut key handling
* Keep configured IM channels editable
* Avoid password autofill for channel secrets
* Make channel threads visible to connection owners
* Persist IM runtime config locally
* Allow disconnecting runtime IM channels
* Route no-auth channel sessions to local user
* Use default user for auth-disabled local mode
* Show IM channel source on threads
* Prefill IM channel runtime config
* Reflect IM channel runtime health
* Ignore Feishu message read events
* Ignore Feishu non-content message events
* Let setup wizard enable IM channels
* Fix frontend formatting after merge
* Stabilize backend tests without local config
* Isolate channel runtime config tests
* Address channel connection review comments
* Use sha256 user buckets with legacy migration
* Ensure runtime IM channels are ready after restart
* Persist disconnected IM channel state
* Address channel connection review comments
* Address channel connection review findings
Frontend connect flow:
- Open the runtime-config dialog only when a provider still needs
credentials; configured providers go straight to the connect flow, so
the binding-code/deep-link path is reachable from the UI again.
- After saving credentials, continue into the connect flow when a user
binding is still required (multi-user mode) instead of stopping at a
"Connected" toast.
- Extract shared provider-state helpers to core/channels/provider-state
and add unit + e2e coverage for the direct-connect and
configure-then-connect paths.
Provider status semantics:
- Report connection_status from the user's newest connection row;
with no binding it is not_connected, except in auth-disabled local
mode where a configured running channel is effectively connected.
Concurrency and event-loop correctness:
- Offload ChannelRuntimeConfigStore construction and writes, channel
service construction, and Slack connection replies to threads; add a
tests/blocking_io/ anchor for the runtime-config handlers.
- Consume binding codes with a conditional UPDATE so a code can only be
used once under concurrent workers; retry upsert_connection as an
update when a concurrent insert wins the unique constraint.
- Serialize ensure_channel_ready per channel so concurrent provider
polls cannot double-start a channel worker.
Config and migration hardening:
- Stop mutating the get_app_config()-cached Telegram provider config;
the runtime store now owns the UI-entered bot username.
- Register channel_connections in STARTUP_ONLY_FIELDS with the
standardized startup-only Field description.
- Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name
so another user's same-prefix bucket can never be migrated.
- Remove the unused Telegram process_webhook_update path and document
src/core/channels in the frontend docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address PR review comments on authz scoping and channel runtime
Security (review feedback from ShenAC-SAC):
- Scope internal-token callers to the connection owner carried in
X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright,
in both require_permission(owner_check=True) and the stateless run
endpoints. Internal callers keep access to their own and
shared/legacy threads, and may claim a default-owned channel thread
for its real owner, but a leaked internal token no longer grants
cross-user thread access.
- Require admin privileges for POST/DELETE /api/channels/{provider}/
runtime-config: runtime credentials and channel workers are
instance-wide shared state (same model as the MCP config API).
Read-only provider listing stays available to all users.
Performance (review feedback from willem-bd):
- Skip the redundant thread channel-metadata PATCH after the first
successful backfill per thread.
- Reuse the per-connection Slack WebClient until its token changes
instead of constructing one per outbound message.
- Reconcile channel readiness for all providers concurrently in
GET /api/channels/providers.
Also resolve the code-quality unused-import flag in the blocking-io
anchor by pre-importing the channel service via importlib.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix prettier formatting in provider-state test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reconcile UI runtime channel config with config reload on restart
Main now reloads a channel's config.yaml entry on restart_channel()
(#3514, issue #3497). Adapt the user-owned connection flow to coexist:
- configure_channel() restarts with reload_config=False — the caller
just supplied the authoritative config (browser-entered credentials
that are never written to config.yaml), so a file reload must not
clobber it with the stale on-disk entry.
- _load_channel_config() re-applies the UI runtime-store overlay used
at startup, so an operator-triggered restart keeps browser-entered
credentials for channels without a config.yaml entry and does not
resurrect a channel disconnected from the UI.
- Offload the reload's disk IO (config.yaml + runtime store) with
asyncio.to_thread, matching the blocking-IO policy on this branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
16391e35ab
|
fix(skills): harden slash skill activation across chat channels (#3466)
* support slash skill activation * format slash skill activation * Preserve slash skill activation with uploads * Address slash skill review feedback * Address slash skill follow-up review * Fix lazy slash skill storage resolution * Keep slash skill activation out of system prompt * Address slash skill review issues * fix: harden slash skill command handling * feat(frontend): add slash skill autocomplete * fix: address slash skill review feedback * fix: preserve slash skill text for IM uploads |
||
|
|
48e038f752
|
feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators (#2842)
* feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators
Add mention_only config to only respond when bot is mentioned, with
allowed_channels override. Add thread_mode for Hermes-style auto-thread
creation. Add periodic typing indicators while bot is processing.
* fix(discord): include allowed_channels in mention_only skip condition (line 274)
* docs: fix Discord config example to match boolean thread_mode implementation
* style: format with ruff
* fix(discord): apply Copilot review fixes and resolve lint errors
- Remove unused Optional import
- Fix thread_ts type hints to str | None
- Fix has_mention logic for None values
- Implement thread_mode fallback to channel replies on thread creation failure
- Fix thread_mode docstring alignment
- Fix allowed_channels comment formatting in config.example.yaml
* fix(discord): reset context for orphaned threads in mention_only mode
When a message arrives in a thread not tracked by _active_threads,
clear thread_id and typing_target so the message falls through to
the standard channel handling pipeline, which creates a fresh thread
instead of incorrectly routing to the stale thread.
* fix(discord): create new thread on @ when channel has existing tracked thread
When mention_only is enabled and a user @-s the bot in a channel
that already has a tracked thread, create a new thread instead of
incorrectly routing to the old one.
* fix(discord): allow no-@ thread replies while skipping no-@ channel messages
The skip block for no-@ messages was too aggressive — it blocked
continuation replies within tracked threads AND incorrectly routed
no-@ channel messages to the existing thread.
Now:
- Thread message, no @ → routed to existing tracked thread
- Channel message, no @ → skipped
- Channel message, with @ → creates new thread
* feat(discord): add checkmark reaction to acknowledge received messages
* Move discord.py to optional dependency and auto-detect from config.yaml
- Add discord extra to [project.optional-dependencies] in pyproject.toml
- Update detect_uv_extras.py to map channels.discord.enabled: true -> --extra discord
- Set UV_EXTRAS=discord in docker-compose-dev.yaml gateway env
* fix(discord): persist thread-channel mappings to store for recovery after restart
Discord's _active_threads dict was purely in-memory, so all channel-to-thread
mappings were lost on server restart. This fix bridges ChannelStore into
DiscordChannel:
- Save thread mappings to store.json after every thread creation
- Restore active threads from store on DiscordChannel startup
- Pass channel_store to all channels via service.py config injection
Store keys follow the pattern: discord:<channel_id>:<thread_id>
* fix(discord): address Copilot review — fix types, typing targets, cross-thread safety, and config comments
* fix(tests): add multitask_strategy param to mock for clarification follow-up test
* fix(tests): explicitly set model_name=None for title middleware test isolation
* fix(discord): use trigger_typing() instead of typing() for typing indicators
discord.py 2.x TextChannel.typing() and Thread.typing() are async context
managers, not one-shot coroutines. Use trigger_typing() for periodic
typing indicator pings.
* fix(discord): cancel typing tasks on channel shutdown
Prevents 'Task was destroyed but it is pending' warnings when the
Discord client stops while typing indicator loops are still running.
* fix(scripts): detect nested YAML config for discord extra
section_value() only matched top-level YAML sections. Added
nested_section_value() that handles two-level nesting (e.g.,
channels.discord.enabled), so auto-detection of the discord
extra works when config uses the standard nested format.
* fix(docker): remove hard-coded UV_EXTRAS=discord from dev compose
Relies on auto-detection via detect_uv_extras.py instead of forcing
discord.py install even when channels.discord.enabled is false.
Matches production docker-compose.yaml behavior (UV_EXTRAS:-).
* refactor(nginx): move proxy_buffering/proxy_cache to server level
DRY cleanup — these directives were repeated in 14 location blocks.
Set at server level once, reducing duplication and risk of drift.
* fix(discord): use dedicated JSON file for thread persistence
Replace ChannelStore usage for Discord thread-ID persistence with a
dedicated discord_threads.json file. ChannelStore is designed to map
IM conversations to DeerFlow thread IDs — using it to persist Discord
thread IDs was semantically wrong and confusing.
Changes:
- _save_thread() now reads/writes a simple {channel_id: thread_id} JSON dict
- _load_active_threads() reads directly from the JSON file
- File path derived from ChannelStore directory (when available) or
defaults to ~/.deer-flow/channels/discord_threads.json
- Removed unused ChannelStore import
* fix(discord): address WillemJiang's code review comments on PR #2842
1. Remove semantically incorrect message_in_thread variable. At this code
point (after the Thread case is handled above), we're guaranteed to be in
a channel, not a thread. Always apply mention_only check here.
2. Add _active_thread_ids reverse-lookup set for O(1) thread ID membership
checks instead of O(n) scan of _active_threads.values(). Keep the set
in sync with _active_threads in _load_active_threads() and _save_thread().
3. Add _thread_store_lock (threading.Lock) to protect _active_threads and
the JSON file from concurrent access between the Discord loop thread
(_run_client) and the main thread (_load_active_threads, _save_thread).
|
||
|
|
c4d273a68a
|
feat(channels): add Discord channel integration (#1806)
* feat(channels): add Discord channel integration Add a Discord bot channel following the existing Telegram/Slack pattern. The bot listens for messages, creates conversation threads, and relays responses back to Discord with 2000-char message splitting. - DiscordChannel extends Channel base class - Lazy imports discord.py with install hint - Thread-based conversations (each Discord thread maps to a DeerFlow thread) - Allowed guilds filter for access control - File attachment support via discord.File - Registered in service.py and manager.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(channels): address Copilot review suggestions for Discord integration - Disable @everyone/@here mentions via AllowedMentions.none() - Add 10s timeout to client close to prevent shutdown hangs - Log publish_inbound errors via future callback instead of silently dropping - Open file handle on caller thread to avoid cross-thread ownership issues - Notify user in channel when thread creation fails Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(discord): resolve lint errors in Discord channel - Replace asyncio.TimeoutError with builtin TimeoutError (UP041) - Remove extraneous f-string prefix (F541) - Apply ruff format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tests): remove fake langgraph_sdk shim from test_discord_channel The module-level sys.modules.setdefault shim installed a fake langgraph_sdk.errors.ConflictError during pytest collection. Because pytest imports all test modules before running them, test_channels.py then imported the fake ConflictError instead of the real one. In test_handle_feishu_stream_conflict_sends_busy_message, the test constructs ConflictError(message, response=..., body=...). The fake only subclasses Exception (which takes no kwargs), so the construction raised TypeError. The manager's _is_thread_busy_error check then saw a TypeError instead of a ConflictError and fell through to the generic 'An error occurred' message. langgraph_sdk is a real dependency, so the shim is unnecessary. Removing it makes both test files import the same real ConflictError and the full suite pass (1773 passed, 15 skipped). --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |