Nan Gao 085924a3d0
fix(channels): run each IM message once across instances, restarts and late redeliveries (#7104)
* fix(channels): run each IM message once across instances, restarts and late redeliveries

Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.

- Discord messages are keyed (discord, guild_id or "dm", posting channel,
  message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
  before a side effect; Discord claims it on the Gateway loop before
  _create_thread, marks the published message dedupe_claimed, and gives the
  claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
  the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
  attempt made every keyed retry a 409). A reused run's status-only /wait or
  stream gap is resolved to this message's own turn via threads.get_state;
  409s about the message's own keyed run keep the key instead of reading as
  a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
  (claim_thread_id on channel_conversations and channel_thread_bindings);
  the loser adopts the winner and deletes the empty thread it minted.
  /new and /agent use still rebind.

Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).

* fix(channels): give back Discord dedupe claims for messages never processed

Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:

- The hand-off task releases the claim on cancellation too (stop() drains it
  while it resolves the connection identity), via await_drained so the
  cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
  message that shutdown drops from the queue, after the grace period and on
  a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
  because stop() already closed cross-loop intake, or a hand-off cancelled
  before it ran, leaves the claim tracked, and stop() releases it last.

* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation

Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.

- Keys from trusted internal callers (the IM channel manager) are scoped by
  owner only (`internal-run:` prefix); browser and API keys keep the
  owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
  RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
  see the same type, and start_run answers it with 409 "Idempotency-Key
  already used with a different request" instead of a 500. The channel
  manager already treats that 409 as the message's admitted run.

Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 22:57:19 +08:00
..

Documentation

This directory contains detailed documentation for the DeerFlow backend.

Document Description
ARCHITECTURE.md System architecture overview
API.md Complete API reference
AUTH_DESIGN.md User authentication, CSRF, platform-trust (IM / Internal Auth), and per-user isolation
SSO.md OIDC / SSO single sign-on
IM_CHANNEL_CONNECTIONS.md IM channel user binding (channel_connections)
CONFIGURATION.md Configuration options
SETUP.md Quick setup guide

Feature Documentation

Document Description
STREAMING.md Token-level streaming design: Gateway vs DeerFlowClient paths, stream_mode semantics, per-id dedup
RUN_EVENT_STREAM.md Persisted run event stream contract: envelope, producers, consumers, and known gaps
FILE_UPLOAD.md File upload functionality
PATH_EXAMPLES.md Path types and usage examples
SANDBOX_MEMORY_PROFILING.md Sandbox memory baseline and runtime comparison guide
summarization.md Context summarization feature
plan_mode_usage.md Plan mode with TodoList
AUTO_TITLE_GENERATION.md Automatic title generation

Development

Document Description
TODO.md Planned features and known issues

Getting Started

  1. New to DeerFlow? Start with SETUP.md for quick installation
  2. Configuring the system? See CONFIGURATION.md
  3. Understanding the architecture? Read ARCHITECTURE.md
  4. Building integrations? Check API.md for API reference

Document Organization

docs/
├── README.md                  # This file
├── ARCHITECTURE.md            # System architecture
├── API.md                     # API reference
├── AUTH_DESIGN.md             # User authentication and isolation design
├── CONFIGURATION.md           # Configuration guide
├── SETUP.md                   # Setup instructions
├── FILE_UPLOAD.md             # File upload feature
├── PATH_EXAMPLES.md           # Path usage examples
├── summarization.md           # Summarization feature
├── plan_mode_usage.md         # Plan mode feature
├── STREAMING.md               # Token-level streaming design
├── RUN_EVENT_STREAM.md        # Persisted run event stream contract
├── AUTO_TITLE_GENERATION.md   # Title generation
├── TITLE_GENERATION_IMPLEMENTATION.md  # Title implementation details
└── TODO.md                    # Roadmap and issues