feat(channels): arbitrate IM transports with a per-provider leader lease (#6843)

* feat(channels): arbitrate IM transports with a per-provider leader lease

Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.

- channel_leases table (migration 0041): one row per provider. Acquire is
  one dialect-native upsert decided against the database clock and bumps
  a fencing epoch; renew extends only a live lease at the holder's epoch;
  release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
  (send time of the last confirmed renewal + lease_seconds); holds() turns
  false one renewal interval early, so the transport stops at least
  grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
  start path; fixed-rate supervisor renews/acquires and runs transport
  starts/stops as per-provider tasks; takeover reloads the provider's
  saved config; failed starts back off; shutdown releases only leases
  whose transport stopped. get_status reports leader/leader_here and the
  holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
  true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.

* docs(changelog): link the channel leader lease entry to #6843

* fix(channels): see peer-held leases for providers absent from local config

Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.

Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

CHANGELOG.md

@@ -946,6 +946,27 @@ This release closes that milestone with **621 merged pull requests**.
 
 #### Channels
 
+- **channels:** IM channels can run on several Gateway instances. With
+  `channels.leader_lease` active (`auto`, the default, follows the
+  multi-process signal), each provider's platform connection belongs to the
+  one instance holding its row in the new `channel_leases` table (migration
+  `0042_channel_leases`): a single upsert decides ownership against the
+  database clock, renewals carry a fencing epoch, and the holder stops its
+  transport on its own monotonic deadline before a peer may connect, so a
+  stalled database cannot leave two Telegram pollers or two Discord sessions
+  running. A crashed holder is replaced within `lease_seconds + grace_seconds`
+  plus one renewal interval; a cleanly stopped one releases and hands over on
+  the peer's next tick, and a transport whose stop timed out keeps its lease
+  until it expires. GitHub webhooks, the message bus and the manager stay on
+  every instance. `GET /api/channels/` reports each provider's `leader`, and
+  `running` reflects the holder's transport on every instance, so the web UI's
+  provider list and connect flow work wherever the load balancer sends them;
+  scheduled-task IM notifications are claimed only by the provider's holder.
+  The startup gate now recommends `auto`/`true` when it refuses `enabled:
+  false`, and refuses an explicit `enabled: true` without a SQL database.
+  Credentials saved from the web UI still reach the holder only when it next
+  starts the channel, and a failover cancels the IM replies in flight. ([#6843])
+
 - **channels:** New `channels.leader_lease` section (`enabled: auto|true|false`,
   `lease_seconds`, `grace_seconds`; startup-only), reserved for the per-provider
   leader lease that will give one Gateway instance each IM platform connection
@@ -11665,5 +11686,6 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
 [#6781]: https://github.com/bytedance/deer-flow/pull/6781
 [#6785]: https://github.com/bytedance/deer-flow/pull/6785
 [#6798]: https://github.com/bytedance/deer-flow/pull/6798
+[#6843]: https://github.com/bytedance/deer-flow/pull/6843
 [#6844]: https://github.com/bytedance/deer-flow/pull/6844
 [#6852]: https://github.com/bytedance/deer-flow/pull/6852

CHANGELOG_zh.md

@@ -729,6 +729,21 @@
 
 #### 渠道
 
+- **channels:** IM 通道可以在多个 Gateway 实例上运行。`channels.leader_lease`
+  生效时（默认 `auto` 跟随多进程信号），每个 provider 的平台连接只属于在新表
+  `channel_leases`（迁移 `0042_channel_leases`）中持有该行的那个实例：归属由一条
+  upsert 依据数据库时钟判定，续约携带防护 epoch，持有者按自己的单调时钟在对端
+  可以连接之前先停传输，因此数据库卡住也不会出现两个 Telegram 轮询者或两个
+  Discord 会话。持有者崩溃后，对端在 `lease_seconds + grace_seconds` 再加一个续约
+  周期内接管；正常停止的持有者会释放租约，对端在下一次心跳接手；停止超时的传输保留
+  租约直到过期。GitHub webhook、消息总线和 manager 仍在每个实例上运行。
+  `GET /api/channels/` 会报告每个 provider 的 `leader`，`running` 在每个实例上都反映
+  持有者的传输状态，所以无论负载均衡把请求发到哪个实例，Web UI 的 provider 列表和
+  连接流程都能正常工作；定时任务的 IM 通知只由该 provider 的持有者认领。启动门禁在
+  拒绝 `enabled: false` 时改为推荐 `auto`/`true`，并拒绝没有 SQL 数据库的显式
+  `enabled: true`。从 Web UI 保存的凭据仍要等持有者下次启动该通道才生效，故障切换会
+  取消进行中的 IM 回复。 ([#6843])
+
 - **channels:** 新增 `channels.leader_lease` 配置段（`enabled: auto|true|false`、
   `lease_seconds`、`grace_seconds`；仅启动时读取），为按 provider 的 leader 租约预留，
   落地后每个 IM 平台连接只由一个 Gateway 实例持有（传输归属本身另行提交，在此之前
@@ -9319,5 +9334,6 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
 [#6746]: https://github.com/bytedance/deer-flow/pull/6746
 [#6781]: https://github.com/bytedance/deer-flow/pull/6781
 [#6785]: https://github.com/bytedance/deer-flow/pull/6785
+[#6843]: https://github.com/bytedance/deer-flow/pull/6843
 [#6844]: https://github.com/bytedance/deer-flow/pull/6844
 [#6852]: https://github.com/bytedance/deer-flow/pull/6852

README.md

@@ -552,7 +552,7 @@ Administrators can suspend and restore accounts from **Settings → Users** (adm
 DeerFlow still uses `Forwarded` / `X-Forwarded-*` headers to recover the browser-facing scheme and origin behind a proxy. The bundled nginx sets `X-Forwarded-Proto`, but preserves an upstream HTTPS value and does not overwrite every forwarded header. Configure the outer trusted proxy to replace or strip client-supplied forwarding headers before traffic reaches DeerFlow.
 
 > [!IMPORTANT]
-> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). Multi-worker deployments require Postgres, the Redis stream bridge (`stream_bridge.type: redis`), `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; process-local memory/JSONL event stores cannot enforce singleton delivery receipts across workers. Kubernetes replicas run one worker per Pod, which the worker count cannot see: declare them with `deployment.multi_instance: true` (or `DEER_FLOW_MULTI_INSTANCE=1`, which deploy tooling such as a Helm chart can set from its replica count) so the startup gate enforces the same prerequisites instead of staying inert. Declared instances that store credentials (`channel_connections`) must also share one at-rest `DEER_FLOW_CREDENTIALS_KEY` (the Helm chart and `make up` generate it; back it up, see [backend/docs/CONFIGURATION.md](backend/docs/CONFIGURATION.md#credentials-encryption-key-deer_flow_credentials_key)). The bridge shares SSE delivery and bounded `Last-Event-ID` replay across workers. When a valid reconnect cursor has been trimmed, or a subscriber that already established an empty-stream wait falls behind before its first delivery, Memory and Redis emit a machine-readable SSE `gap` event instead of silently returning a partial replay; the Web UI reloads durable thread/event state and resumes from the retained tail. Lease reconciliation marks runs from dead workers as errors, persists their delivery receipts, publishes the terminal stream marker, schedules retained-stream cleanup, and updates the affected thread status. SSE, `/wait`, and internal stream consumers use `stream_bridge.heartbeat_interval_seconds` (default `15`) for idle liveness checks; changing it requires a Gateway restart. Malformed Redis reconnect IDs live-tail new events instead of replaying the retained buffer, and the rolling retained-buffer TTL (`stream_ttl_seconds`) remains a cleanup safety net rather than a run timeout. Failed-login counters and lockouts for `POST /api/v1/auth/login/local` (`auth.local.max_login_attempts` / `lockout_seconds`) are kept in the shared `login_throttle` table whenever the application database is SQLite or Postgres (`auth.local.throttle_storage: auto`, the default), so every replica enforces one lockout per client IP; `memory` keeps the historical per-process counter, which under N replicas hands an attacker N × `max_login_attempts` guesses and logs a startup warning. IM chat-to-thread bindings live in the shared `channel_thread_bindings` table whenever the application database is SQLite or Postgres (an existing `channels/store.json` is imported once at startup and renamed `store.json.migrated`); each instance's platform connections still need one owner per provider — `channels.leader_lease` (default `auto`) is reserved for that per-provider leader lease, and until it ships the setting only drives the startup gate, which refuses `enabled: false` while an IM channel other than GitHub is enabled; keep one instance while channels are enabled (see [IM channel leader lease](backend/docs/CONFIGURATION.md#im-channel-leader-lease-channelsleader_lease)); follow-up buffers still need their own multi-worker coordination. To try this topology on one machine, `make dev-multi` starts two Gateways on shared Postgres, Redis and `DEER_FLOW_HOME`, and `make dev-multi-check` runs the cross-instance checks; see the local two-Gateway harness in [backend/docs/CONFIGURATION.md](backend/docs/CONFIGURATION.md#local-two-gateway-harness).
+> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). Multi-worker deployments require Postgres, the Redis stream bridge (`stream_bridge.type: redis`), `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; process-local memory/JSONL event stores cannot enforce singleton delivery receipts across workers. Kubernetes replicas run one worker per Pod, which the worker count cannot see: declare them with `deployment.multi_instance: true` (or `DEER_FLOW_MULTI_INSTANCE=1`, which deploy tooling such as a Helm chart can set from its replica count) so the startup gate enforces the same prerequisites instead of staying inert. Declared instances that store credentials (`channel_connections`) must also share one at-rest `DEER_FLOW_CREDENTIALS_KEY` (the Helm chart and `make up` generate it; back it up, see [backend/docs/CONFIGURATION.md](backend/docs/CONFIGURATION.md#credentials-encryption-key-deer_flow_credentials_key)). The bridge shares SSE delivery and bounded `Last-Event-ID` replay across workers. When a valid reconnect cursor has been trimmed, or a subscriber that already established an empty-stream wait falls behind before its first delivery, Memory and Redis emit a machine-readable SSE `gap` event instead of silently returning a partial replay; the Web UI reloads durable thread/event state and resumes from the retained tail. Lease reconciliation marks runs from dead workers as errors, persists their delivery receipts, publishes the terminal stream marker, schedules retained-stream cleanup, and updates the affected thread status. SSE, `/wait`, and internal stream consumers use `stream_bridge.heartbeat_interval_seconds` (default `15`) for idle liveness checks; changing it requires a Gateway restart. Malformed Redis reconnect IDs live-tail new events instead of replaying the retained buffer, and the rolling retained-buffer TTL (`stream_ttl_seconds`) remains a cleanup safety net rather than a run timeout. Failed-login counters and lockouts for `POST /api/v1/auth/login/local` (`auth.local.max_login_attempts` / `lockout_seconds`) are kept in the shared `login_throttle` table whenever the application database is SQLite or Postgres (`auth.local.throttle_storage: auto`, the default), so every replica enforces one lockout per client IP; `memory` keeps the historical per-process counter, which under N replicas hands an attacker N × `max_login_attempts` guesses and logs a startup warning. IM chat-to-thread bindings live in the shared `channel_thread_bindings` table whenever the application database is SQLite or Postgres (an existing `channels/store.json` is imported once at startup and renamed `store.json.migrated`); each provider's platform connection is held by one instance through the shared `channel_leases` table — `channels.leader_lease` (default `auto`, on wherever the multi-process signal is) hands it to another instance when the holder dies, and the startup gate refuses `enabled: false` while an IM channel other than GitHub is enabled (see [IM channel leader lease](backend/docs/CONFIGURATION.md#im-channel-leader-lease-channelsleader_lease)); follow-up buffers still need their own multi-worker coordination. To try this topology on one machine, `make dev-multi` starts two Gateways on shared Postgres, Redis and `DEER_FLOW_HOME`, and `make dev-multi-check` runs the cross-instance checks; see the local two-Gateway harness in [backend/docs/CONFIGURATION.md](backend/docs/CONFIGURATION.md#local-two-gateway-harness).
 >
 > The local two-Gateway harness disables IM providers under both `channels` and `channel_connections` by default. `DEERFLOW_MI_KEEP_CHANNELS=1` preserves them; `channels.leader_lease` stays as configured.
 >

backend/app/channels/leadership.py

@@ -0,0 +1,225 @@
+"""Per-provider leader lease bookkeeping for one Gateway instance (``channels.leader_lease``).
+
+The shared ``channel_leases`` table (``deerflow.persistence.channel_leases``)
+decides, against the database clock, which instance may hold a provider's
+platform connection. This module keeps the holder's side of that contract:
+
+- **Local deadline.** A lease granted or renewed by a statement sent at local
+  monotonic time ``t`` cannot be taken by a peer before ``t + lease_seconds +
+  grace_seconds`` (the database stamps the expiry no earlier than ``t`` and a
+  peer must wait ``grace_seconds`` past it). The holder therefore treats its
+  lease as safe only while ``now + renew_interval < t + lease_seconds``, where
+  ``t`` is the send time of the last *confirmed* renewal. Offsets between host
+  clocks do not matter; only the monotonic clock's rate does.
+- **Fail closed.** :meth:`ChannelLeadership.holds` turns false one renewal
+  interval before that deadline whether or not the database answered, and the
+  caller (``ChannelService``) stops the transport when it does. A renewal
+  that the database rejects (another owner or a newer epoch) is a loss and the
+  transport stops at once; a renewal that errors or times out is merely
+  unconfirmed and the deadline keeps running.
+- **Bounded calls.** Every repository call is bounded by
+  ``operation_timeout``, half the renewal interval, so a stalled database
+  cannot hold a renewal past the deadline it protects. A call that times out
+  after the statement committed only ever errs toward "not held".
+
+Peers' leases are cached from one bounded read of the whole table per
+supervision round (:meth:`ChannelLeadership.observe_all`) and from failed
+acquisitions, so ``get_status`` (sync) and the notification claim filter know
+who leads *every* provider, including one this instance has no config for
+(a provider connected from the web UI on a peer never enters this instance's
+channel config).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import enum
+import logging
+import time
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass
+from typing import Protocol, TypeVar
+
+from deerflow.persistence.channel_leases import ChannelLeaseGrant, ChannelLeaseState
+
+logger = logging.getLogger(__name__)
+
+_T = TypeVar("_T")
+
+
+class ChannelLeaseRepository(Protocol):
+    """What :class:`ChannelLeadership` needs from ``SqlChannelLeaseRepository``."""
+
+    async def acquire(self, provider: str, owner_id: str, *, ttl_seconds: float, grace_seconds: float) -> ChannelLeaseGrant | None: ...
+
+    async def renew(self, provider: str, owner_id: str, epoch: int, *, ttl_seconds: float, transport_running: bool) -> bool: ...
+
+    async def release(self, provider: str, owner_id: str, epoch: int) -> bool: ...
+
+    async def get(self, provider: str) -> ChannelLeaseState | None: ...
+
+    async def list_all(self) -> list[ChannelLeaseState]: ...
+
+
+class RenewOutcome(enum.Enum):
+    RENEWED = "renewed"
+    #: The database rejected the renewal: another owner or a newer epoch holds the row.
+    LOST = "lost"
+    #: The renewal errored or timed out; the lease may or may not still be ours.
+    UNCONFIRMED = "unconfirmed"
+
+
+@dataclass(frozen=True)
+class _Held:
+    epoch: int
+    #: Local monotonic time by which a peer may be able to take the lease.
+    deadline: float
+
+
+class ChannelLeadership:
+    """The lease state of one Gateway instance across every arbitrated provider."""
+
+    def __init__(
+        self,
+        repo: ChannelLeaseRepository,
+        *,
+        owner_id: str,
+        lease_seconds: int,
+        grace_seconds: int,
+        clock: Callable[[], float] = time.monotonic,
+    ) -> None:
+        self._repo = repo
+        self.owner_id = owner_id
+        self.lease_seconds = float(lease_seconds)
+        self.grace_seconds = float(grace_seconds)
+        # ``channels.leader_lease.lease_seconds`` documents renewal every
+        # ``lease_seconds // 3`` seconds; the config floor of 3 keeps it >= 1.
+        self.renew_interval = float(max(1, int(lease_seconds) // 3))
+        self.operation_timeout = max(0.5, self.renew_interval / 2)
+        self._clock = clock
+        self._held: dict[str, _Held] = {}
+        self._observed: dict[str, ChannelLeaseState] = {}
+        self._locks: dict[str, asyncio.Lock] = {}
+
+    # -- local view -----------------------------------------------------
+
+    def now(self) -> float:
+        return self._clock()
+
+    def holds(self, provider: str) -> bool:
+        """True while a transport for ``provider`` may keep running until the next renewal."""
+        held = self._held.get(provider)
+        return held is not None and self._clock() + self.renew_interval < held.deadline
+
+    def has_record(self, provider: str) -> bool:
+        """True while this instance believes it holds the lease row, even near its deadline."""
+        return provider in self._held
+
+    def held_providers(self) -> list[str]:
+        return sorted(self._held)
+
+    def leader_of(self, provider: str) -> str | None:
+        """The owner of a live lease as last seen by this instance, or ``None``."""
+        if provider in self._held:
+            return self.owner_id
+        observed = self._observed.get(provider)
+        return observed.owner_id if observed is not None and observed.live else None
+
+    def peer_leads(self, provider: str) -> bool:
+        """Whether a live lease held by another instance was last seen for ``provider``."""
+        if provider in self._held:
+            return False
+        observed = self._observed.get(provider)
+        return bool(observed is not None and observed.live and observed.owner_id != self.owner_id)
+
+    def peer_transport_running(self, provider: str) -> bool:
+        """Whether a live peer leader reported its transport running at its last renewal."""
+        if provider in self._held:
+            return False
+        observed = self._observed.get(provider)
+        return bool(observed is not None and observed.live and observed.transport_running)
+
+    def forget(self, provider: str) -> None:
+        """Drop the local hold without touching the row (the lease is lost or past its deadline)."""
+        self._held.pop(provider, None)
+
+    # -- database operations ---------------------------------------------
+
+    async def acquire(self, provider: str) -> bool:
+        """Hold ``provider``'s lease, taking it if free; ``False`` while a peer holds it or on error."""
+        async with self._lock(provider):
+            if self.holds(provider):
+                return True
+            sent = self._clock()
+            try:
+                grant = await self._bounded(self._repo.acquire(provider, self.owner_id, ttl_seconds=self.lease_seconds, grace_seconds=self.grace_seconds))
+            except Exception:
+                logger.warning("Could not acquire the %s channel leader lease; retrying on the next renewal tick", provider, exc_info=True)
+                return False
+            if grant is None:
+                # A peer holds the row, so any local record is stale.
+                self._held.pop(provider, None)
+                await self._observe(provider)
+                return False
+            if provider not in self._held:
+                logger.info("Acquired the %s channel leader lease (epoch %d)", provider, grant.epoch)
+            self._held[provider] = _Held(epoch=grant.epoch, deadline=sent + self.lease_seconds)
+            self._observed.pop(provider, None)
+            return True
+
+    async def renew(self, provider: str, *, transport_running: bool) -> RenewOutcome:
+        async with self._lock(provider):
+            held = self._held.get(provider)
+            if held is None:
+                return RenewOutcome.LOST
+            sent = self._clock()
+            try:
+                renewed = await self._bounded(self._repo.renew(provider, self.owner_id, held.epoch, ttl_seconds=self.lease_seconds, transport_running=transport_running))
+            except Exception:
+                logger.warning("Could not confirm the %s channel leader lease renewal (epoch %d)", provider, held.epoch, exc_info=True)
+                return RenewOutcome.UNCONFIRMED
+            if not renewed:
+                logger.warning("Lost the %s channel leader lease (epoch %d): it expired or another instance took it", provider, held.epoch)
+                self._held.pop(provider, None)
+                return RenewOutcome.LOST
+            self._held[provider] = _Held(epoch=held.epoch, deadline=sent + self.lease_seconds)
+            return RenewOutcome.RENEWED
+
+    async def release(self, provider: str) -> None:
+        """Hand the lease back so a peer can take it at once (best effort; it expires anyway)."""
+        async with self._lock(provider):
+            held = self._held.pop(provider, None)
+            if held is None:
+                return
+            try:
+                await self._bounded(self._repo.release(provider, self.owner_id, held.epoch))
+            except Exception:
+                logger.warning("Could not release the %s channel leader lease (epoch %d); it expires on its own", provider, held.epoch, exc_info=True)
+                return
+            logger.info("Released the %s channel leader lease (epoch %d)", provider, held.epoch)
+
+    async def observe_all(self) -> None:
+        """Refresh the cached view of every provider's lease (best effort, one bounded read)."""
+        try:
+            states = await self._bounded(self._repo.list_all())
+        except Exception:
+            logger.debug("Could not read the channel leader lease table", exc_info=True)
+            return
+        self._observed = {state.provider: state for state in states if state.provider not in self._held}
+
+    async def _observe(self, provider: str) -> None:
+        try:
+            state = await self._bounded(self._repo.get(provider))
+        except Exception:
+            logger.debug("Could not read the %s channel leader lease", provider, exc_info=True)
+            return
+        if state is None:
+            self._observed.pop(provider, None)
+        else:
+            self._observed[provider] = state
+
+    async def _bounded(self, call: Awaitable[_T]) -> _T:
+        return await asyncio.wait_for(call, timeout=self.operation_timeout)
+
+    def _lock(self, provider: str) -> asyncio.Lock:
+        return self._locks.setdefault(provider, asyncio.Lock())

backend/app/channels/service.py

@@ -6,16 +6,19 @@ import asyncio
 import logging
 import math
 import os
-from collections.abc import Callable, Mapping
+import socket
+import uuid
+from collections.abc import Awaitable, Callable, Mapping
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
 from app.channels.base import Channel, ChannelStopTimeout
+from app.channels.leadership import ChannelLeadership, RenewOutcome
 from app.channels.manager import DEFAULT_CHANNEL_MAX_CONCURRENCY, DEFAULT_CHANNEL_SHUTDOWN_GRACE_PERIOD_SECONDS, DEFAULT_GATEWAY_URL, DEFAULT_LANGGRAPH_URL, ChannelManager
 from app.channels.message_bus import DEFAULT_INBOUND_QUEUE_MAXSIZE, MessageBus
 from app.channels.runtime_config_store import merge_runtime_channel_configs
 from app.channels.store import ChannelStore, SqlChannelStore, resolve_channel_store
-from deerflow.config.channel_leader_lease_config import CHANNEL_LEADER_LEASE_KEY, ChannelLeaderLeaseConfig, channel_leader_lease_config
+from deerflow.config.channel_leader_lease_config import CHANNEL_LEADER_LEASE_FIELD, CHANNEL_LEADER_LEASE_KEY, ChannelLeaderLeaseConfig, channel_leader_lease_config
 from deerflow.utils.file_io import await_drained
 
 logger = logging.getLogger(__name__)
@@ -52,6 +55,15 @@ _CHANNEL_CREDENTIAL_KEYS: dict[str, list[str]] = {
     "wechat": ["bot_token"],
 }
 
+# Providers the leader lease never arbitrates: GitHub receives webhooks pushed
+# to whichever instance the load balancer picks and opens no platform
+# connection of its own, so every instance keeps it.
+_UNARBITRATED_PROVIDERS = frozenset({"github"})
+
+# Ceiling for the supervisor's retry backoff when a leader cannot start its
+# transport (bad credentials, platform outage).
+_LEASE_START_BACKOFF_MAX_SECONDS = 300.0
+
 _CHANNELS_LANGGRAPH_URL_ENV = "DEER_FLOW_CHANNELS_LANGGRAPH_URL"
 _CHANNELS_GATEWAY_URL_ENV = "DEER_FLOW_CHANNELS_GATEWAY_URL"
 
@@ -86,7 +98,7 @@ def enabled_transport_providers(app_config: Any) -> list[str]:
             provider = getattr(connections, name, None)
             if provider is not None and getattr(provider, "enabled", False):
                 names.add(name)
-    names.discard("github")
+    names.difference_update(_UNARBITRATED_PROVIDERS)
     return sorted(names)
 
 
@@ -144,6 +156,32 @@ def _make_connection_repo(connection_config: ChannelConnectionsConfig | None, *,
     return ChannelConnectionRepository(session_factory, cipher=credentials_cipher)
 
 
+def _default_lease_owner_id() -> str:
+    """Same shape as the RunManager worker id the Gateway passes in (``hostname:uuid``)."""
+    return f"{socket.gethostname()}:{uuid.uuid4().hex}"
+
+
+def _make_leadership(lease_config: ChannelLeaderLeaseConfig, *, owner_id: str | None) -> ChannelLeadership:
+    """Build the lease bookkeeping over the shared application database.
+
+    The lease needs the shared table; an active lease without a SQL database
+    is a configuration the startup gate refuses, so reaching it here is an
+    error rather than a silent fallback to every instance connecting.
+    """
+    from deerflow.persistence.channel_leases import SqlChannelLeaseRepository
+    from deerflow.persistence.engine import get_session_factory
+
+    session_factory = get_session_factory()
+    if session_factory is None:
+        raise RuntimeError(f"{CHANNEL_LEADER_LEASE_FIELD} is active but the application database is not available; set database.backend to sqlite or postgres")
+    return ChannelLeadership(
+        SqlChannelLeaseRepository(session_factory),
+        owner_id=owner_id or _default_lease_owner_id(),
+        lease_seconds=lease_config.lease_seconds,
+        grace_seconds=lease_config.grace_seconds,
+    )
+
+
 class ChannelService:
     """Manages the lifecycle of all configured IM channels.
 
@@ -160,6 +198,7 @@ class ChannelService:
         app_config: AppConfig | None = None,
         get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
         store: ChannelStore | None = None,
+        leadership: ChannelLeadership | None = None,
     ) -> None:
         config = dict(channels_config or {})
         # ``channels.leader_lease`` is a settings block, not a provider: take it
@@ -206,6 +245,15 @@ class ChannelService:
         self._manager_start_lock = asyncio.Lock()
         self._readiness_locks: dict[str, asyncio.Lock] = {}
         self._config_epochs: dict[str, int] = {}
+        # ``None`` unless ``channels.leader_lease`` is active for this process:
+        # then only the lease holder of a provider may start its transport.
+        self._leadership = leadership
+        self._supervisor_task: asyncio.Task | None = None
+        # One transport action (start / stop) in flight per provider, so a
+        # slow platform never delays another provider's renewal.
+        self._lease_actions: dict[str, asyncio.Task] = {}
+        # provider -> (consecutive failed starts, monotonic time of next retry)
+        self._lease_start_backoff: dict[str, tuple[int, float]] = {}
 
     @classmethod
     def from_app_config(
@@ -214,6 +262,8 @@ class ChannelService:
         *,
         get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
         credentials_cipher: Any | None = None,
+        multi_process: bool = False,
+        lease_owner_id: str | None = None,
     ) -> ChannelService:
         """Create a ChannelService from the application config.
 
@@ -222,7 +272,10 @@ class ChannelService:
         callers (including most tests) that don't need follow-up-buffer
         auto-draining can omit it. ``credentials_cipher`` is the Gateway's
         ``DEER_FLOW_CREDENTIALS_KEY`` cipher for per-connection credentials;
-        without it stored credentials stay unavailable.
+        without it stored credentials stay unavailable. ``multi_process`` is
+        the Gateway's multi-process signal, which ``channels.leader_lease:
+        auto`` follows; ``lease_owner_id`` names this instance in the lease
+        table (the Gateway passes its RunManager worker id).
         """
         if app_config is None:
             from deerflow.config.app_config import get_app_config
@@ -237,12 +290,15 @@ class ChannelService:
         connection_config = getattr(app_config, "channel_connections", None)
         connections_enabled = connection_config is not None and getattr(connection_config, "enabled", False)
         require_bound_identity = bool(connections_enabled and getattr(connection_config, "require_bound_identity", True))
+        lease_config = channel_leader_lease_config(channels_config)
+        leadership = _make_leadership(lease_config, owner_id=lease_owner_id) if lease_config.resolve(multi_process=multi_process) else None
         return cls(
             channels_config=channels_config,
             connection_repo=_make_connection_repo(connection_config, credentials_cipher=credentials_cipher),
             require_bound_identity=require_bound_identity,
             app_config=app_config,
             get_stream_bridge=get_stream_bridge,
+            leadership=leadership,
         )
 
     async def start(self) -> None:
@@ -277,7 +333,24 @@ class ChannelService:
 
         ready_status = await self.ensure_ready_channels(attempts=2)
         ready_count = sum(1 for ready in ready_status.values() if ready)
-        logger.info("ChannelService started with %d/%d ready channels", ready_count, len(ready_status))
+        if self._leadership is None:
+            logger.info("ChannelService started with %d/%d ready channels", ready_count, len(ready_status))
+            return
+        led_elsewhere = sorted(name for name in ready_status if self._arbitrated(name) and not self._leadership.has_record(name))
+        logger.info(
+            "ChannelService started with %d/%d ready channels; leader lease active as %s (led by another instance: %s)",
+            ready_count,
+            len(ready_status),
+            self._leadership.owner_id,
+            ", ".join(led_elsewhere) or "none",
+        )
+        if self._running and generation == self._shutdown_generation:
+            # Learn peers' leases before the first tick, so status and
+            # notification claims are right from the start.
+            await self._leadership.observe_all()
+        # Re-check after the read: a stop() during it must not be followed by a supervisor.
+        if self._running and generation == self._shutdown_generation:
+            self._start_supervisor()
 
     async def ensure_ready_channels(self, *, attempts: int = 1) -> dict[str, bool]:
         """Start or restart enabled configured channels that are not ready."""
@@ -351,6 +424,11 @@ class ChannelService:
                     logger.warning("Readiness retry deferred: previous %s channel failed to stop and remains tracked", name)
                     return False
 
+            if not await self._claim_transport(name):
+                # Another instance holds the lease: its transport is the one
+                # that serves this provider, so there is nothing to start here.
+                return False
+
             max_attempts = max(1, attempts)
             for attempt in range(max_attempts):
                 if not self._running:
@@ -373,6 +451,9 @@ class ChannelService:
         self._shutdown_generation += 1
         self._stopping = True
         self._running = False
+        # The supervisor starts and stops transports on its own; it must be
+        # gone before shutdown snapshots the transports it is about to stop.
+        await self._stop_supervisor()
         # Reject new provider work first. Existing workers keep draining during
         # manager.stop(), and channel transports remain alive until that drain
         # completes so an already-sent "Working on it..." can still receive its
@@ -398,6 +479,13 @@ class ChannelService:
                     self._channels.pop(name, None)
                 logger.info("Channel stopped")
 
+        # Hand back the leases of providers whose transport is gone, so a peer
+        # connects on its next tick instead of after lease + grace. A retained
+        # transport (bounded stop timed out) may still hold its socket: keep
+        # its lease to expire on its own rather than invite a second connection.
+        if self._leadership is not None:
+            await asyncio.gather(*(self._leadership.release(name) for name in self._leadership.held_providers() if name not in self._channels))
+
         if stop_errors:
             raise ExceptionGroup("one or more channels failed to stop", stop_errors)
 
@@ -443,12 +531,33 @@ class ChannelService:
         return self._config_epochs[name]
 
     async def restart_channel(self, name: str, *, reload_config: bool = True) -> bool:
-        """Restart a specific channel. Returns True if successful."""
+        """Restart a specific channel. Returns True if successful.
+
+        Under the leader lease a restart on an instance that does not hold the
+        provider's lease returns False; :meth:`leader_of` names the holder.
+        """
         async with self._channel_lock(name):
             return await self._restart_channel_locked(name, reload_config=reload_config)
 
-    async def _restart_channel_locked(self, name: str, *, reload_config: bool = True) -> bool:
-        """Restart a specific channel. The caller must already hold ``_channel_lock(name)``."""
+    async def _take_over_channel(self, name: str) -> bool:
+        """Start a provider whose lease this instance just took, from its freshly loaded config.
+
+        Unlike :meth:`restart_channel` it leaves a transport that is already
+        running alone: a readiness request may have started it between the
+        acquisition and this call.
+        """
+        async with self._channel_lock(name):
+            if self._transport_running(name):
+                return True
+            return await self._restart_channel_locked(name, reload_config=True)
+
+    async def _restart_channel_locked(self, name: str, *, reload_config: bool = True, accept_peer_leader: bool = False) -> bool:
+        """Restart a specific channel. The caller must already hold ``_channel_lock(name)``.
+
+        ``accept_peer_leader`` reports success when another instance holds the
+        provider's lease: :meth:`configure_channel` accepted the config, and
+        the transport serving the provider is the holder's.
+        """
         if name in self._channels:
             channel = self._channels[name]
             # Same ownership rule as readiness retries: retain an instance
@@ -476,8 +585,18 @@ class ChannelService:
 
         if not config.get("enabled", False):
             logger.info("Channel %s is disabled, skipping restart", name)
+            await self._release_idle_lease(name)
             return True
 
+        if not await self._claim_transport(name):
+            leader = self.leader_of(name)
+            if accept_peer_leader and leader is not None:
+                # Runtime config does not reach the holder yet: it keeps the
+                # config it started with until it starts the channel again.
+                logger.warning("Channel %s is connected by Gateway instance %s, which keeps its current configuration until it next starts the channel", name, leader)
+                return True
+            return False
+
         return await self._start_channel(name, config)
 
     async def configure_channel(self, name: str, config: dict[str, Any]) -> bool:
@@ -491,7 +610,7 @@ class ChannelService:
                 # The caller just supplied the authoritative config (e.g. credentials
                 # entered in the browser that are never written to config.yaml) — a
                 # file reload here would clobber it with the stale on-disk entry.
-                return await self._restart_channel_locked(name, reload_config=False)
+                return await self._restart_channel_locked(name, reload_config=False, accept_peer_leader=True)
             finally:
                 self._bump_config_epoch(name)
 
@@ -503,6 +622,7 @@ class ChannelService:
                 self._config.pop(name, None)
                 channel = self._channels.get(name)
                 if channel is None:
+                    await self._release_idle_lease(name)
                     return True
                 # Stop-then-drop with the shared ownership rule: a channel whose
                 # stop() fails stays tracked (and returns False) instead of being
@@ -511,6 +631,7 @@ class ChannelService:
                 if self._channels.get(name) is channel:
                     logger.warning("Removal incomplete: %s channel failed to stop and remains tracked", name)
                     return False
+                await self._release_idle_lease(name)
                 logger.info("Channel stopped and removed")
                 return True
             finally:
@@ -583,6 +704,15 @@ class ChannelService:
             logger.warning("Refusing to start %s: another channel instance is still tracked under this name (previous cleanup incomplete, or the instance is still running)", name)
             return False
 
+        # The lease fence at the mechanism itself: every start path (startup,
+        # readiness, restart, runtime config, supervisor) ends here, and only
+        # the holder of a provider's leader lease may open its platform
+        # connection. Callers acquire first (_claim_transport); this check
+        # makes no database call.
+        if self._arbitrated(name) and not self._leadership.holds(name):
+            logger.warning("Refusing to start %s: this Gateway instance does not hold its leader lease", name)
+            return False
+
         try:
             from deerflow.reflection import resolve_class
 
@@ -640,8 +770,219 @@ class ChannelService:
                 self._channels.pop(name, None)
             return False
 
+    # -- leader lease ---------------------------------------------------------
+    #
+    # With ``channels.leader_lease`` active, every arbitrated provider (all but
+    # GitHub) has one transport across the deployment: the instance holding
+    # its ``channel_leases`` row. The manager, bus and GitHub webhooks keep
+    # running on every instance. ``_supervise`` renews held leases every
+    # ``renew_interval`` seconds, takes free ones, and stops a transport
+    # *before* giving its lease up for lost (see ``app.channels.leadership``).
+
+    def _arbitrated(self, name: str) -> bool:
+        """Whether ``name``'s transport is subject to the leader lease in this process."""
+        return self._leadership is not None and name in _CHANNEL_REGISTRY and name not in _UNARBITRATED_PROVIDERS
+
+    def _wants_transport(self, name: str) -> bool:
+        config = self._config.get(name)
+        return isinstance(config, dict) and bool(config.get("enabled", False))
+
+    def _transport_running(self, name: str) -> bool:
+        channel = self._channels.get(name)
+        return channel is not None and bool(channel.is_running)
+
+    async def _claim_transport(self, name: str) -> bool:
+        """True when this instance may start ``name``'s transport, taking its lease if it is free."""
+        if not self._arbitrated(name):
+            return True
+        return await self._leadership.acquire(name)
+
+    async def _release_idle_lease(self, name: str) -> None:
+        """Hand back a held lease once the provider is disabled here and its transport is gone."""
+        if self._arbitrated(name) and self._leadership.has_record(name) and name not in self._channels and not self._wants_transport(name):
+            await self._leadership.release(name)
+
+    @property
+    def lease_owner_id(self) -> str | None:
+        """This instance's name in the lease table, or ``None`` without the leader lease."""
+        return self._leadership.owner_id if self._leadership is not None else None
+
+    def leader_of(self, name: str) -> str | None:
+        """The instance holding ``name``'s lease as last seen here (``None`` without the lease or a holder)."""
+        if not self._arbitrated(name):
+            return None
+        return self._leadership.leader_of(name)
+
+    def notification_excluded_providers(self) -> frozenset[str] | None:
+        """Providers whose scheduled-run notifications this instance must not claim.
+
+        ``None`` without the leader lease. Otherwise the arbitrated providers
+        whose lease this instance does not hold and that either are enabled
+        here or were last seen held by a live peer: their holder delivers over
+        its own transport, and a claim here would only park the row for the
+        channel-down interval. The peer check covers a provider connected from
+        the web UI on another instance, which never enters this instance's
+        config. Providers no instance runs stay claimable, so their rows still
+        settle.
+        """
+        if self._leadership is None:
+            return None
+        lead = self._leadership
+        return frozenset(name for name in _CHANNEL_REGISTRY if self._arbitrated(name) and not lead.has_record(name) and (self._wants_transport(name) or lead.peer_leads(name)))
+
+    def _start_supervisor(self) -> None:
+        if self._leadership is None or self._supervisor_task is not None:
+            return
+        self._supervisor_task = asyncio.create_task(self._supervise(), name="channel-leader-lease")
+
+    async def _stop_supervisor(self) -> None:
+        tasks = [task for task in (self._supervisor_task, *self._lease_actions.values()) if task is not None]
+        self._supervisor_task = None
+        for task in tasks:
+            task.cancel()
+        if tasks:
+            await asyncio.gather(*tasks, return_exceptions=True)
+
+    async def _supervise(self) -> None:
+        lead = self._leadership
+        loop = asyncio.get_running_loop()
+        next_tick = loop.time()
+        while self._running:
+            # Fixed rate rather than sleep-after-round: a slow round must not
+            # stretch the gap between renewals the deadline is computed from.
+            # Missed ticks are skipped, not replayed as a burst.
+            now = loop.time()
+            next_tick = max(next_tick + lead.renew_interval, now)
+            await asyncio.sleep(next_tick - now)
+            if not self._running:
+                return
+            try:
+                await self._supervise_once()
+            except Exception:
+                logger.exception("Channel leader lease round failed; retrying on the next tick")
+
+    async def _supervise_once(self) -> None:
+        names = set(self._leadership.held_providers())
+        names.update(name for name in self._config if self._arbitrated(name) and self._wants_transport(name))
+        names.update(name for name in self._channels if self._arbitrated(name))
+        ordered = sorted(names)
+        # Lease calls are bounded and run concurrently; transport work is
+        # handed to per-provider tasks so it never delays a renewal. The
+        # whole-table read keeps status and notification claims aware of
+        # leases this instance has no config for.
+        observed, *decisions = await asyncio.gather(self._leadership.observe_all(), *(self._lease_round(name) for name in ordered), return_exceptions=True)
+        if isinstance(observed, BaseException):
+            logger.error("Channel leader lease table read failed", exc_info=observed)
+        for name, decision in zip(ordered, decisions):
+            if isinstance(decision, BaseException):
+                logger.error("Channel %s leader lease round failed", name, exc_info=decision)
+            elif decision is not None and self._running:
+                action, preempt = decision
+                self._schedule_lease_action(name, action, preempt=preempt)
+
+    async def _lease_round(self, name: str) -> tuple[Callable[[], Awaitable[None]], bool] | None:
+        """Renew or acquire ``name``'s lease and return the transport action it calls for.
+
+        The second element is ``preempt``: a fail-closed stop cancels a start
+        still in flight instead of waiting behind it.
+        """
+        lead = self._leadership
+        if lead.has_record(name):
+            outcome = await lead.renew(name, transport_running=self._transport_running(name))
+            if outcome is RenewOutcome.LOST:
+                return (lambda: self._stop_transport(name, release=False)), True
+            if outcome is RenewOutcome.UNCONFIRMED:
+                if lead.holds(name):
+                    return None
+                # Fail closed: a peer may take the lease once it has been
+                # expired for grace_seconds, so the transport stops now, while
+                # that is still at least grace_seconds away.
+                logger.error("Stopping the %s transport: its leader lease renewal could not be confirmed before the local deadline", name)
+                lead.forget(name)
+                return (lambda: self._stop_transport(name, release=False)), True
+            if not self._wants_transport(name):
+                return (lambda: self._stop_transport(name, release=True)), False
+            if not self._transport_running(name) and self._lease_start_due(name):
+                return (lambda: self._start_led_transport(name, reload_config=False)), False
+            return None
+        if name in self._channels:
+            # A transport left without a lease (its stop failed after the lease
+            # ended): keep trying to stop it.
+            return (lambda: self._stop_transport(name, release=False)), False
+        if self._wants_transport(name) and await lead.acquire(name):
+            # Taking over from another instance: reload the provider's config
+            # so credentials saved since this instance started are used.
+            return (lambda: self._start_led_transport(name, reload_config=True)), False
+        return None
+
+    def _schedule_lease_action(self, name: str, action: Callable[[], Awaitable[None]], *, preempt: bool) -> None:
+        current = self._lease_actions.get(name)
+        if current is not None and not current.done():
+            if not preempt:
+                return
+            current.cancel()
+
+        async def run() -> None:
+            if current is not None:
+                await asyncio.gather(current, return_exceptions=True)
+            await action()
+
+        task = asyncio.create_task(run(), name=f"channel-leader-lease-{name}")
+        self._lease_actions[name] = task
+        task.add_done_callback(lambda done, name=name: self._lease_action_done(name, done))
+
+    def _lease_action_done(self, name: str, task: asyncio.Task) -> None:
+        if self._lease_actions.get(name) is task:
+            self._lease_actions.pop(name, None)
+        if not task.cancelled() and task.exception() is not None:
+            logger.error("Channel %s leader lease action failed", name, exc_info=task.exception())
+
+    async def _stop_transport(self, name: str, *, release: bool) -> None:
+        """Stop ``name``'s transport; with ``release``, then hand its (still held) lease back."""
+        async with self._channel_lock(name):
+            channel = self._channels.get(name)
+            if channel is not None:
+                await self._stop_and_discard_channel(name, channel)
+                if self._channels.get(name) is channel:
+                    if release:
+                        logger.warning("Keeping the %s leader lease: its transport failed to stop and may still be connected", name)
+                    else:
+                        logger.error("The %s transport failed to stop after its leader lease ended; another instance may connect concurrently until it stops", name)
+                    return
+                logger.info("Stopped the %s transport (%s)", name, "provider disabled here" if release else "leader lease ended")
+            if release:
+                await self._release_idle_lease(name)
+
+    async def _start_led_transport(self, name: str, *, reload_config: bool) -> None:
+        if not self._running:
+            return
+        if reload_config:
+            await self._take_over_channel(name)
+        else:
+            await self.ensure_channel_ready(name)
+        self._record_lease_start(name, ok=self._transport_running(name) or not self._wants_transport(name))
+
+    def _lease_start_due(self, name: str) -> bool:
+        backoff = self._lease_start_backoff.get(name)
+        return backoff is None or self._leadership.now() >= backoff[1]
+
+    def _record_lease_start(self, name: str, *, ok: bool) -> None:
+        if ok:
+            self._lease_start_backoff.pop(name, None)
+            return
+        failures = self._lease_start_backoff.get(name, (0, 0.0))[0] + 1
+        delay = min(_LEASE_START_BACKOFF_MAX_SECONDS, self._leadership.renew_interval * 2 ** (failures - 1))
+        self._lease_start_backoff[name] = (failures, self._leadership.now() + delay)
+        logger.warning("The %s transport did not start on its leader; retrying in %.0fs (attempt %d)", name, delay, failures)
+
     def get_status(self) -> dict[str, Any]:
-        """Return status information for all channels."""
+        """Return status information for all channels.
+
+        Under the leader lease each arbitrated provider also reports
+        ``leader`` (the holder's instance id, or ``None``) and ``leader_here``,
+        and ``running`` covers a peer holder whose transport was running at its
+        last renewal, so every instance answers the same way.
+        """
         channels_status = {}
         for name in _CHANNEL_REGISTRY:
             config = self._config.get(name, {})
@@ -651,6 +992,10 @@ class ChannelService:
                 "enabled": enabled,
                 "running": running,
             }
+            if self._arbitrated(name):
+                channels_status[name]["running"] = running or self._leadership.peer_transport_running(name)
+                channels_status[name]["leader"] = self._leadership.leader_of(name)
+                channels_status[name]["leader_here"] = self._leadership.has_record(name)
         return {
             "service_running": self._running,
             "channels": channels_status,
@@ -710,6 +1055,8 @@ async def start_channel_service(
     *,
     get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
     credentials_cipher: Any | None = None,
+    multi_process: bool = False,
+    lease_owner_id: str | None = None,
 ) -> ChannelService:
     """Create and start the global ChannelService from app config.
 
@@ -719,6 +1066,8 @@ async def start_channel_service(
     a run's completion and auto-drain buffered follow-ups. ``app.py``'s
     lifespan passes a closure over ``app.state.stream_bridge`` here, the same
     pattern it already uses for ``ScheduledTaskService``'s ``launch_run``.
+    ``multi_process`` and ``lease_owner_id`` select and name this instance's
+    ``channels.leader_lease`` participation (see ``from_app_config``).
     """
     global _channel_service
     if _channel_service is not None:
@@ -726,7 +1075,14 @@ async def start_channel_service(
     # from_app_config resolves the channel store path (realpath) and reads the
     # runtime config files; keep that disk IO off the event loop.
     # asyncio.to_thread forwards both args and kwargs to the target callable.
-    service = await asyncio.to_thread(ChannelService.from_app_config, app_config, get_stream_bridge=get_stream_bridge, credentials_cipher=credentials_cipher)
+    service = await asyncio.to_thread(
+        ChannelService.from_app_config,
+        app_config,
+        get_stream_bridge=get_stream_bridge,
+        credentials_cipher=credentials_cipher,
+        multi_process=multi_process,
+        lease_owner_id=lease_owner_id,
+    )
     _channel_service = service
 
     async def rollback_failed_start() -> None:

backend/app/gateway/app.py

@@ -18,7 +18,7 @@ from app.gateway.auth_middleware import AuthMiddleware
 from app.gateway.browser_capability import ensure_browser_runtime_available
 from app.gateway.config import get_gateway_config
 from app.gateway.csrf_middleware import CORS_EXPOSED_HEADERS, CSRFMiddleware, get_configured_cors_origins
-from app.gateway.deps import langgraph_runtime
+from app.gateway.deps import langgraph_runtime, multi_process_deployment
 from app.gateway.health import READINESS_CHECKPOINTER_CONFIG_ATTR, READINESS_PROVISIONER_URL_ATTR, readiness_payload
 from app.gateway.routers import (
     agents,
@@ -523,6 +523,9 @@ async def _start_scheduled_task_notification_delivery(app: FastAPI, startup_conf
             # Both repositories come from the same session factory, so the
             # connection repository is present whenever the outbox one is.
             resolve_connections=connection_repo.list_connections,
+            # Under channels.leader_lease, rows for a provider whose transport
+            # another instance holds are left to that instance.
+            excluded_providers=getattr(channel_service, "notification_excluded_providers", None),
             resolve_notification_preferences=ImNotificationPreferencesRepository(get_session_factory()).get,
             poll_interval_seconds=startup_config.scheduler.poll_interval_seconds,
             # Language of notices whose owner has no UI language preference.
@@ -768,6 +771,11 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
                 startup_config,
                 get_stream_bridge=lambda: getattr(app.state, "stream_bridge", None),
                 credentials_cipher=credentials_cipher,
+                # channels.leader_lease: auto follows the same multi-process
+                # signal the startup gate used; the lease owner is this
+                # instance's run worker id, so lease and run logs line up.
+                multi_process=multi_process_deployment(startup_config),
+                lease_owner_id=getattr(getattr(app.state, "run_manager", None), "worker_id", None),
             )
             logger.info("Channel service started: %s", channel_service.get_status())
         except Exception:

backend/app/gateway/deps.py

@@ -165,11 +165,13 @@ def _enforce_postgres_for_multi_worker(config: AppConfig) -> None:
        channel other than GitHub is enabled: every instance would open the
        platform connection with the same bot credentials (Telegram rejects
        the second poller with 409, Discord answers every message twice).
-       ``auto`` (the default) and ``true`` pass the clause, but they are reserved
-       configuration for the lease-backed transport ownership that ships
-       separately: in this version they do not stop the duplicate connections,
-       so the refusal offers only the single-instance and channel-disable
-       remedies.
+       ``auto`` (the default) and ``true`` hand each provider's connection to
+       the one instance holding its ``channel_leases`` row.
+
+    On a single instance only one lease rule applies: an explicit
+    ``channels.leader_lease.enabled: true`` needs a SQL database for its lease
+    table, so ``database.backend: memory`` is refused while a channel the lease
+    arbitrates is enabled.
 
     This gate runs once at startup before any persistence engine is
     initialised so the error message is clear and the process exits
@@ -197,6 +199,13 @@ def _enforce_postgres_for_multi_worker(config: AppConfig) -> None:
 
     signal = _multi_process_signal(config)
     if signal is None:
+        if leader_lease.enabled is True and backend not in ("sqlite", "postgres"):
+            providers = _enabled_channel_transport_providers(config)
+            if providers:
+                raise SystemExit(
+                    f"{CHANNEL_LEADER_LEASE_FIELD}.enabled=true needs the shared application database for its lease table, but database.backend is '{backend}' "
+                    f"while IM channels {', '.join(providers)} are enabled. Set database.backend to sqlite or postgres, or set {CHANNEL_LEADER_LEASE_FIELD}.enabled to auto."
+                )
         return
     reason, rollback = signal
 
@@ -250,12 +259,20 @@ def _enforce_postgres_for_multi_worker(config: AppConfig) -> None:
                 f"{reason} cannot run with {CHANNEL_LEADER_LEASE_FIELD}.enabled=false while IM channels {', '.join(providers)} are enabled: "
                 "every instance would open the platform connection with the same bot credentials (Telegram rejects the second poller "
                 "with 409 Conflict, Discord answers every message twice, and per-process channel state diverges). "
-                f"{rollback}, or disable those channels (channels.<provider>.enabled=false and channel_connections.<provider>.enabled=false). "
-                f"{CHANNEL_LEADER_LEASE_FIELD}.enabled=auto|true is reserved for the per-provider leader lease that ships separately and does not "
-                "yet stop these duplicate connections, so it is not a remedy here."
+                f"Set {CHANNEL_LEADER_LEASE_FIELD}.enabled to auto (the default) or true so one instance holds each provider's connection; "
+                f"alternatively {rollback[0].lower()}{rollback[1:]}, or disable those channels (channels.<provider>.enabled=false and channel_connections.<provider>.enabled=false)."
             )
 
 
+def multi_process_deployment(config: AppConfig) -> bool:
+    """Whether this process must assume peer Gateway processes (the gate's multi-process signal).
+
+    ``channels.leader_lease: auto`` follows exactly this signal, so the lease
+    turns on wherever the gate enforced the multi-process prerequisites.
+    """
+    return _multi_process_signal(config) is not None
+
+
 def _channel_leader_lease(config: AppConfig) -> ChannelLeaderLeaseConfig:
     """Parse ``channels.leader_lease``; a malformed block exits with the dotted path in the message."""
     try:

backend/app/gateway/routers/channels.py

@@ -53,6 +53,11 @@ async def restart_channel(name: str, request: Request) -> ChannelRestartResponse
     if success:
         logger.info("Channel %s restarted successfully", name)
         return ChannelRestartResponse(success=True, message=f"Channel {name} restarted successfully")
-    else:
-        logger.warning("Failed to restart channel %s", name)
-        return ChannelRestartResponse(success=False, message=f"Failed to restart channel {name}")
+    leader = service.leader_of(name) if hasattr(service, "leader_of") else None
+    if isinstance(leader, str) and leader != getattr(service, "lease_owner_id", None):
+        # channels.leader_lease: only the instance holding the provider's lease
+        # runs its transport, and this request reached a different instance.
+        logger.info("Channel %s restart declined: Gateway instance %s holds its leader lease", name, leader)
+        return ChannelRestartResponse(success=False, message=f"Channel {name} is connected by another Gateway instance ({leader}); it can only be restarted there")
+    logger.warning("Failed to restart channel %s", name)
+    return ChannelRestartResponse(success=False, message=f"Failed to restart channel {name}")

backend/app/scheduler/notification_delivery.py

@@ -45,6 +45,12 @@ SummaryResolver = Callable[[str, str | None], Any]
 # it to confirm, at send time, that the delivery's target is still bound.
 ConnectionResolver = Callable[[str], Any]
 
+# Returns the providers this instance must not claim rows for, or None for no
+# restriction. Under ``channels.leader_lease`` (``ChannelService
+# .notification_excluded_providers``) these are the providers another instance
+# holds the transport of: claiming their rows here would only park them.
+ExcludedProvidersResolver = Callable[[], Any]
+
 # A claimed row stuck in "sending" longer than this is considered orphaned
 # (process died between claim and the final status write) and flipped back
 # to pending. Generous on purpose: a healthy send plus status write finishes
@@ -69,6 +75,7 @@ class NotificationDeliveryWorker:
         batch_size: int = 10,
         resolve_run_summary: SummaryResolver | None = None,
         resolve_connections: ConnectionResolver | None = None,
+        excluded_providers: ExcludedProvidersResolver | None = None,
         resolve_notification_preferences: Callable[[str], Any] | None = None,
         stale_sending_timeout_seconds: int = _STALE_SENDING_TIMEOUT_SECONDS,
         stop_timeout_seconds: float = _STOP_TIMEOUT_SECONDS,
@@ -78,6 +85,7 @@ class NotificationDeliveryWorker:
         self._resolve_channel = resolve_channel
         self._resolve_run_summary = resolve_run_summary
         self._resolve_connections = resolve_connections
+        self._excluded_providers = excluded_providers
         self._resolve_notification_preferences = resolve_notification_preferences
         self._poll_interval_seconds = poll_interval_seconds
         self._batch_size = batch_size
@@ -122,7 +130,11 @@ class NotificationDeliveryWorker:
 
     async def run_once(self, *, now: datetime) -> None:
         await self._recover_stale_sending(now)
-        rows = await self._delivery_repo.claim_due_deliveries(now=now, limit=self._batch_size)
+        excluded = self._current_exclusions()
+        if excluded is None:
+            rows = await self._delivery_repo.claim_due_deliveries(now=now, limit=self._batch_size)
+        else:
+            rows = await self._delivery_repo.claim_due_deliveries(now=now, limit=self._batch_size, exclude_providers=excluded)
         for row in rows:
             try:
                 await self._deliver(row)
@@ -137,6 +149,15 @@ class NotificationDeliveryWorker:
                 except Exception:
                     logger.warning("Could not mark crashed delivery %s as failed; stale reset will recover it", row.get("id"), exc_info=True)
 
+    def _current_exclusions(self) -> frozenset[str] | None:
+        """Providers to leave to the instance holding their transport (read fresh every poll)."""
+        if self._excluded_providers is None:
+            return None
+        excluded = self._excluded_providers()
+        if not isinstance(excluded, (set, frozenset, list, tuple)) or not excluded:
+            return None
+        return frozenset(str(provider) for provider in excluded)
+
     async def _recover_stale_sending(self, now: datetime) -> None:
         """Lease-style reconciliation, run before every claim (the first poll
         after startup is covered too). Mirrors how the scheduler recovers

backend/docs/CONFIGURATION.md

@@ -755,7 +755,7 @@ Notes:
 - The declaration also drives the `agent_storage.backend: file` divergence warning, the inbound webhook dedupe warning, and the WeChat QR-login guard, which otherwise only look at the worker count.
 - With the default DeerMem backend, the declaration also warns when the derived SQLite retrieval index sits inside the shared `storage_path` (the default `{storage_path}/.retrieval`). Set `memory.backend_config.retrieval_index_path` to an instance-local directory (relative values resolve against `storage_path`): SQLite WAL is unsupported on network filesystems, and the index is rebuilt from the Markdown facts at startup and re-synced per user scope after a peer writes.
 - A declared multi-instance deployment that enables a feature storing credentials (today `channel_connections.enabled: true`) also needs one shared `DEER_FLOW_CREDENTIALS_KEY`; see [Credentials encryption key](#credentials-encryption-key-deer_flow_credentials_key).
-- IM channels: the gate refuses `channels.leader_lease.enabled: false` while a provider other than GitHub is enabled (under `channels.*`, or connectable from the web UI through `channel_connections.<provider>.enabled`), because every instance would otherwise open the platform connection with the same bot credentials. `auto`/`true` pass the clause but are reserved for the lease-backed transport ownership that ships separately, so keep one instance while channels are enabled; see [IM channel leader lease](#im-channel-leader-lease-channelsleader_lease).
+- IM channels: the gate refuses `channels.leader_lease.enabled: false` while a provider other than GitHub is enabled (under `channels.*`, or connectable from the web UI through `channel_connections.<provider>.enabled`), because every instance would otherwise open the platform connection with the same bot credentials. With `auto` (the default) or `true` the instances share each provider's connection through the `channel_leases` table; see [IM channel leader lease](#im-channel-leader-lease-channelsleader_lease).
 - Restart-required: the gate runs once at startup. Restart all Gateway instances together after changing it.
 
 #### Local two-Gateway harness
@@ -763,7 +763,7 @@ Notes:
 `scripts/dev_multi_instance.sh` (also `make dev-multi`, `make dev-multi-check`, `make dev-multi-down`) runs this topology on one machine, so cross-instance behavior can be checked without a cluster:
 
 - Throwaway `postgres:17-alpine` and `redis:7-alpine` containers (`deerflow-mi-postgres`, `deerflow-mi-redis`) on `127.0.0.1:55432` and `127.0.0.1:56379`, Gateway A on `127.0.0.1:8001` and Gateway B on `127.0.0.1:8011` (8002 stays free for the provisioner). When nginx is installed, `http://localhost:2027` round-robins `/api` over both Gateways, names the Gateway that answered in an `X-DeerFlow-Upstream` response header, and forwards `/` to a frontend on port 3000 if you start one. Ports, container names and images are overridable; see `--help`.
-- The generated `config.yaml` is your own `config.yaml` (otherwise `config.example.yaml`; `DEERFLOW_MI_BASE_CONFIG` picks another file) plus `deployment.multi_instance: true`, `database.backend: postgres`, `run_events.backend: db`, `run_ownership.heartbeat_enabled: true` and `stream_bridge.type: redis`. Settings the startup gate refuses are changed and reported: process-local browser tools are removed and an enabled scheduler gets `scheduler.multi_instance: true`. Enabled IM providers under both `channels.*` and the UI-connectable `channel_connections.*` are disabled because both Gateways would connect the same bot; `DEERFLOW_MI_KEEP_CHANNELS=1` keeps them. The `channels.leader_lease` block and connection metadata stay unchanged.
+- The generated `config.yaml` is your own `config.yaml` (otherwise `config.example.yaml`; `DEERFLOW_MI_BASE_CONFIG` picks another file) plus `deployment.multi_instance: true`, `database.backend: postgres`, `run_events.backend: db`, `run_ownership.heartbeat_enabled: true` and `stream_bridge.type: redis`. Settings the startup gate refuses are changed and reported: process-local browser tools are removed and an enabled scheduler gets `scheduler.multi_instance: true`. Enabled IM providers under both `channels.*` and the UI-connectable `channel_connections.*` are disabled so the harness never connects your bots unasked; `DEERFLOW_MI_KEEP_CHANNELS=1` keeps them, and the leader lease then gives each platform connection to one Gateway. The `channels.leader_lease` block and connection metadata stay unchanged.
 - Everything the pair writes stays in the harness, never in your own deployment: any `checkpointer` section is removed (it would win over `database` for checkpoints and the Store), an explicit `sandbox.ownership` and a Redis `database.checkpoint_cache` point at the harness Redis (ownership timing settings are kept), a DeerMem `storage_path` (either spelling) and `blob_storage.backend_config.root` are dropped so their data lands in the harness home, and a local AIO sandbox gets a `container_prefix` derived from the state dir, so the harness never adopts or reaps sandbox containers it did not start. A non-DeerMem memory backend is used as configured, and the harness says so.
 - Both processes share one `DEER_FLOW_HOME`, one copy of `extensions_config.json` (MCP and skill edits never reach your own file), and generated `AUTH_JWT_SECRET`, `DEER_FLOW_INTERNAL_AUTH_TOKEN` and `DEER_FLOW_CREDENTIALS_KEY` values. Each process keeps its own DeerMem `retrieval_index_path`, which the shared config reads from `$DEERFLOW_MI_RETRIEVAL_INDEX_PATH`.
 - Gateway B starts only after A answers `/health`. Alembic migrations are serialized by an advisory lock, but the LangGraph checkpointer and store `setup()` that runs next is not, so two cold starts against an empty database could race.
@@ -792,7 +792,7 @@ These still need a manual pass:
 - cancel a run on B from A: it takes effect within about one heartbeat (~10 s; a lower `run_ownership.lease_seconds` makes the boundary easier to observe);
 - change an MCP filesystem server's args on A: B's local-bash allowlist follows;
 - edit a skill on A: B's system-prompt skills section changes within 30 s;
-- the IM channel gates (one leader per platform, failover after killing the leader, runtime-config changes), once channel leader election lands; run them with `DEERFLOW_MI_KEEP_CHANNELS=1`.
+- with `DEERFLOW_MI_KEEP_CHANNELS=1` and test bots configured: only one Gateway connects each platform (`GET /api/channels/` reports the same `leader` on both, Telegram logs no `409`, Discord answers once), `stop <leader> --kill` hands each provider to the other Gateway within `lease_seconds + grace_seconds`, and a plain `stop <leader>` hands them over on the other Gateway's next tick.
 
 ### IM channel leader lease (`channels.leader_lease`)
 
@@ -804,13 +804,17 @@ channels:
     grace_seconds: 10
 ```
 
-Every IM channel except GitHub connects outbound to its platform (Telegram long polling, Discord / Slack / Feishu / DingTalk WebSockets). With more than one Gateway instance each one would open that connection with the same bot credentials: Telegram rejects the second poller with `409 Conflict`, Discord delivers every message to every session so the user is answered twice, and the per-process `ChannelManager` state (the `/model` cache, GitHub follow-up buffers) diverges. The leader lease will arbitrate this: one row per provider in the shared database, held by one instance and renewed on a heartbeat, so only the holder starts that provider's transport. GitHub is webhook-driven and needs no lease. **In this version the section only drives the startup gate**: the transport ownership ships separately, so `auto`/`true` are reserved settings, not an ownership guarantee, and a deployment with channels enabled must still run one instance.
+Every IM channel except GitHub connects outbound to its platform (Telegram long polling, Discord / Slack / Feishu / DingTalk WebSockets). With more than one Gateway instance each one would open that connection with the same bot credentials: Telegram rejects the second poller with `409 Conflict`, Discord delivers every message to every session so the user is answered twice, and the per-process `ChannelManager` state (the `/model` cache, GitHub follow-up buffers) diverges. The leader lease arbitrates this: the `channel_leases` table (migration `0042_channel_leases`) holds one row per provider, and only the instance holding a provider's row opens that provider's platform connection. The `ChannelManager`, the message bus and GitHub webhooks keep running on every instance.
 
-- `enabled: auto` (default) follows the multi-process signal: on with `GATEWAY_WORKERS` / `WEB_CONCURRENCY > 1`, `deployment.multi_instance: true` or `DEER_FLOW_MULTI_INSTANCE=1`, off on a single instance, which then never writes lease rows and behaves as before. `true` always turns it on; `false` never does, and the multi-process startup gate refuses `false` while an IM channel other than GitHub is enabled. The refusal names the providers and offers only the remedies that work today (a single instance, or disabling those channels); it does not recommend `auto`/`true`, which would merely bypass the clause until the lease ships.
-- `lease_seconds` (≥ 3, default 30) is how long a held lease stays valid without renewal; the holder renews every `lease_seconds // 3` seconds. `grace_seconds` (≥ 0, default 10) is the extra time after expiry before a peer may take the lease, absorbing a slow renewal. Worst-case failover after a crash is `lease_seconds + grace_seconds`; a cleanly stopped holder releases immediately, so a peer takes over within one heartbeat.
+- `enabled: auto` (default) follows the multi-process signal: on with `GATEWAY_WORKERS` / `WEB_CONCURRENCY > 1`, `deployment.multi_instance: true` or `DEER_FLOW_MULTI_INSTANCE=1`, off on a single instance, which then never writes lease rows and behaves as before. `true` always turns it on and needs `database.backend: sqlite` or `postgres` (a single instance with `memory` and an arbitrated channel enabled refuses to start); `false` never does, and the multi-process startup gate refuses `false` while an IM channel other than GitHub is enabled.
+- `lease_seconds` (≥ 3, default 30) is how long a held lease stays valid without renewal; the holder renews every `lease_seconds // 3` seconds. `grace_seconds` (≥ 0, default 10) is the extra time after expiry before a peer may take the lease, absorbing a slow renewal. Worst-case failover after a crash is `lease_seconds + grace_seconds` plus one renewal interval; a cleanly stopped holder releases its leases, so a peer takes over on its next tick.
+- Ownership is decided against the **database clock** inside each SQL statement (an expired lease changes hands only once it has been expired for `grace_seconds`, and every acquisition bumps a fencing `epoch`), so clock skew between hosts cannot hand a provider to two instances. The holder fails closed on its own monotonic clock: when a renewal is rejected, or cannot be confirmed for long enough that the next renewal would come too late, it stops the transport first — at least `grace_seconds` before a peer is allowed to connect.
+- Every instance tries to take every free lease of a provider it has enabled, so providers spread across instances by whoever starts first. A holder whose transport fails to start keeps the lease and retries with backoff (up to 5 minutes); a transport that cannot be stopped within its bounded teardown keeps its lease until it expires rather than invite a second connection.
+- `GET /api/channels/` reports, per arbitrated provider, `leader` (the holder's instance id, the RunManager worker id `hostname:uuid`) and `leader_here`, and `running` reflects the holder's transport on every instance, so `GET /api/channels/providers` and `POST /api/channels/{provider}/connect` behave the same wherever the load balancer sends them. `POST /api/channels/{name}/restart` on a non-holder answers `success: false` and names the holder. Scheduled-task IM notifications are claimed only by the holder of their provider.
+- Runtime channel configuration does not cross instances yet: `POST /api/channels/{provider}/runtime-config` saves the values (in the shared `runtime-config.json`) and applies them only where the provider's transport runs — a non-holder accepts them without restarting the holder, which keeps its current credentials until it next starts the channel (an instance taking a lease over reloads the provider's saved config first). Make credential changes on a single instance or restart the holder afterwards.
 - The block is validated on every topology: an unknown key, a non-mapping value, `enabled` outside `auto | true | false`, `lease_seconds < 3` or `grace_seconds < 0` stop startup with the dotted path in the message.
 - Restart-required (`channels.leader_lease` in the reload-boundary registry): the gate evaluates it once and `ChannelService` captures it at construction. Change it on every instance and restart them together.
-- The lease-backed transport ownership itself (the `channel_leases` table, the supervising loop and the leader reported by `GET /api/channels/providers`) ships separately; until it lands, `auto`/`true` do not stop every instance from connecting, so keep one instance while channels are enabled. This section fixes the configuration contract and the gate so that rollout needs no further config change.
+- Known limits: a leader change cancels the IM runs in flight on the old holder (their `on_disconnect` is `cancel`), and transport-local state such as Feishu running cards is lost on failover.
 
 ### Credentials encryption key (`DEER_FLOW_CREDENTIALS_KEY`)
 

backend/docs/IM_CHANNEL_CONNECTIONS.md

@@ -625,6 +625,15 @@ the worker last loaded or wrote. WeChat QR login auth state
 follows the same local-runtime model and may persist a QR-derived bot token in
 the channel state directory.
 
+With several Gateway instances, `channels.leader_lease` gives each provider's
+platform connection to one instance: the holder of the provider's row in
+`channel_leases`. Bindings, connect codes and conversations are shared SQL
+state, so the connect flow works on any instance; `GET /api/channels/` names
+each provider's `leader`. Runtime credentials saved on an instance that does
+not hold the provider apply on the holder only when it next starts the
+channel. See
+[IM channel leader lease](CONFIGURATION.md#im-channel-leader-lease-channelsleader_lease).
+
 ## Security Notes
 
 - Browser APIs remain authenticated and CSRF-protected.

backend/packages/harness/deerflow/config/channel_leader_lease_config.py

@@ -7,12 +7,11 @@ Telegram rejects the second poller with ``409 Conflict``, Discord delivers
 every message to every session so the user is answered twice, and the
 per-process ``ChannelManager`` state diverges between instances.
 
-The leader lease will be the arbitration for that: a row per provider in the
+The leader lease arbitrates that: a ``channel_leases`` row per provider in the
 shared database, held by one instance and renewed on a heartbeat, so only the
-holder starts the provider's transport. This module owns the configuration
-section and the startup gate consumes it; the transport arbitration itself
-ships separately with ``ChannelService``, so ``auto`` / ``true`` are reserved
-settings until then and must not be presented as an ownership guarantee.
+holder starts the provider's transport (``app.channels.leadership`` and
+``ChannelService``). This module owns the configuration section; the startup
+gate and ``ChannelService.from_app_config`` consume it.
 
 ``channels`` is not part of the ``AppConfig`` Pydantic schema (its provider
 blocks carry raw credentials), so this model is validated from the
@@ -44,9 +43,9 @@ class ChannelLeaderLeaseConfig(BaseModel):
         default="auto",
         description=(
             "auto (default) turns the lease on whenever the Gateway sees a multi-process signal (GATEWAY_WORKERS / WEB_CONCURRENCY > 1, "
-            "deployment.multi_instance: true or DEER_FLOW_MULTI_INSTANCE=1) and leaves a single instance untouched; true always turns it on; "
-            "false never does. A multi-process deployment with IM channels enabled refuses to start when this is false. Until the lease-backed "
-            "transport ownership ships, auto and true are reserved settings: they do not yet stop every instance from connecting."
+            "deployment.multi_instance: true or DEER_FLOW_MULTI_INSTANCE=1) and leaves a single instance untouched; true always turns it on "
+            "(it needs a sqlite or postgres database); false never does. While it is on, only the instance holding a provider's lease opens that "
+            "platform connection (GitHub webhooks stay on every instance). A multi-process deployment with IM channels enabled refuses to start when this is false."
         ),
     )
     lease_seconds: int = Field(

backend/packages/harness/deerflow/persistence/channel_leases/__init__.py

@@ -0,0 +1,20 @@
+"""Per-provider IM transport leader lease shared by every Gateway instance (``channels.leader_lease``)."""
+
+from deerflow.persistence.channel_leases.model import (
+    CHANNEL_LEASE_OWNER_LENGTH,
+    CHANNEL_LEASE_PROVIDER_LENGTH,
+    ChannelLeaseGrant,
+    ChannelLeaseRow,
+    ChannelLeaseState,
+)
+from deerflow.persistence.channel_leases.sql import SqlChannelLeaseRepository, db_now
+
+__all__ = [
+    "CHANNEL_LEASE_OWNER_LENGTH",
+    "CHANNEL_LEASE_PROVIDER_LENGTH",
+    "ChannelLeaseGrant",
+    "ChannelLeaseRow",
+    "ChannelLeaseState",
+    "SqlChannelLeaseRepository",
+    "db_now",
+]

backend/packages/harness/deerflow/persistence/channel_leases/model.py

@@ -0,0 +1,67 @@
+"""ORM model for the per-provider IM transport leader lease (``channel_leases``).
+
+One row per IM provider (``telegram``, ``discord``, ...). The Gateway instance
+named by ``owner_id`` holds the provider's platform connection until
+``lease_expires_at``; it renews on a heartbeat, and a peer may take the row
+over only once the lease has been expired for ``channels.leader_lease
+.grace_seconds``. ``epoch`` grows by one on every acquisition, so a renewal or
+release that carries an older epoch cannot touch a newer holder's lease.
+
+Both timestamps are epoch seconds read from the **database** clock (see
+:mod:`deerflow.persistence.channel_leases.sql`), never from an instance's own
+wall clock: every comparison that decides who may connect runs inside one SQL
+statement against one clock, so skew between hosts cannot hand the same
+provider to two instances.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from sqlalchemy import BigInteger, Boolean, Double, PrimaryKeyConstraint, String
+from sqlalchemy.orm import Mapped, mapped_column
+
+from deerflow.persistence.base import Base
+
+#: Longest provider key (registry names are short; this leaves headroom).
+CHANNEL_LEASE_PROVIDER_LENGTH = 64
+#: Longest owner id. Matches ``runs.owner_worker_id``: the Gateway uses the
+#: RunManager worker id (``hostname:uuid``) as the lease owner.
+CHANNEL_LEASE_OWNER_LENGTH = 128
+
+
+class ChannelLeaseRow(Base):
+    __tablename__ = "channel_leases"
+
+    provider: Mapped[str] = mapped_column(String(CHANNEL_LEASE_PROVIDER_LENGTH), nullable=False)
+    owner_id: Mapped[str] = mapped_column(String(CHANNEL_LEASE_OWNER_LENGTH), nullable=False)
+    epoch: Mapped[int] = mapped_column(BigInteger, nullable=False)
+    # Database-clock epoch seconds; see the module docstring.
+    lease_expires_at: Mapped[float] = mapped_column(Double, nullable=False)
+    # Whether the holder's transport was running at its last renewal, so a
+    # peer's status endpoint can report "configured but not running".
+    transport_running: Mapped[bool] = mapped_column(Boolean, nullable=False)
+    updated_at: Mapped[float] = mapped_column(Double, nullable=False)
+
+    __table_args__ = (PrimaryKeyConstraint("provider", name="pk_channel_leases"),)
+
+
+@dataclass(frozen=True)
+class ChannelLeaseGrant:
+    """A lease this instance now holds: ``epoch`` fences every later renewal and release."""
+
+    provider: str
+    epoch: int
+    lease_expires_at: float
+
+
+@dataclass(frozen=True)
+class ChannelLeaseState:
+    """A provider's lease row as of one read, judged against the database clock of that read."""
+
+    provider: str
+    owner_id: str
+    epoch: int
+    lease_expires_at: float
+    transport_running: bool
+    live: bool

backend/packages/harness/deerflow/persistence/channel_leases/sql.py

@@ -0,0 +1,165 @@
+"""SQL repository for ``channel_leases`` (SQLite or PostgreSQL).
+
+Every write is one statement whose WHERE clause decides ownership against the
+database clock of that statement (``statement_timestamp()`` on PostgreSQL,
+``julianday('now')`` on SQLite, both as epoch seconds), so the decision never
+depends on an instance's wall clock:
+
+- ``acquire`` is one dialect-native upsert: ``INSERT ... ON CONFLICT
+  (provider) DO UPDATE ... WHERE lease_expires_at + grace < now OR owner_id =
+  :me``. A missing row is created at epoch 1; an existing row changes hands
+  only when its lease has been expired for ``grace`` seconds (or this owner
+  already holds it), and every acquisition bumps ``epoch``. Two instances
+  racing on the same provider serialize on the row: PostgreSQL evaluates the
+  ``DO UPDATE`` predicate on the row version the winner committed, and SQLite
+  serializes writers, so exactly one statement changes a row. The winner reads
+  its new epoch back in the same transaction (its own write, still locked).
+- ``renew`` extends only a live lease: ``UPDATE ... WHERE owner_id = :me AND
+  epoch = :epoch AND lease_expires_at > now``. A lease that already expired
+  must be re-acquired (bumping the epoch) instead of silently revived, so a
+  holder whose renewals lapsed cannot keep a lease a peer is about to take.
+- ``release`` sets ``lease_expires_at`` to 0 under the same owner/epoch fence,
+  so a peer may acquire on its next attempt instead of waiting out the TTL.
+  The row stays (its epoch keeps growing across holders).
+
+Outcomes are decided by the affected-row count, never by ``RETURNING``
+values: through SQLAlchemy's aiosqlite adapter, an ``UPDATE ... RETURNING``
+that runs while another connection reads the table has been observed to hand
+back one of the statement's own bind values instead of the column (the stored
+row is correct). A lease must not mistake such a row for a confirmed renewal,
+so writes report ``rowcount`` and the epoch is read with a plain ``SELECT``.
+
+The holder's own safety deadline is local and monotonic, and lives in
+``app.channels.leadership``: the lease row only says who may connect, the
+holder must stop its transport before a peer is allowed to.
+
+Database errors propagate; the caller treats them as "renewal not confirmed".
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from sqlalchemy import Double, cast, extract, func, or_, select, update
+from sqlalchemy.dialects.postgresql import insert as pg_insert
+from sqlalchemy.dialects.sqlite import insert as sqlite_insert
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from deerflow.persistence.channel_leases.model import ChannelLeaseGrant, ChannelLeaseRow, ChannelLeaseState
+
+#: Julian day number of the Unix epoch (1970-01-01T00:00:00Z).
+_UNIX_EPOCH_JULIAN_DAY = 2440587.5
+_SECONDS_PER_DAY = 86400.0
+
+
+def _dialect(session: AsyncSession) -> str:
+    name = session.get_bind().dialect.name
+    if name not in ("postgresql", "sqlite"):
+        raise ValueError(f"Unsupported channel lease database dialect: {name}")
+    return name
+
+
+def db_now(dialect: str) -> Any:
+    """The current time of the statement, in epoch seconds, from the database clock.
+
+    SQLite evaluates ``'now'`` once per statement step, PostgreSQL's
+    ``statement_timestamp()`` is fixed for the statement, so every reference in
+    one statement sees the same instant.
+    """
+    if dialect == "postgresql":
+        return cast(extract("epoch", func.statement_timestamp()), Double)
+    return (func.julianday("now") - _UNIX_EPOCH_JULIAN_DAY) * _SECONDS_PER_DAY
+
+
+class SqlChannelLeaseRepository:
+    """Persistence facade for ``channel_leases``."""
+
+    def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
+        self._sf = session_factory
+
+    async def acquire(self, provider: str, owner_id: str, *, ttl_seconds: float, grace_seconds: float) -> ChannelLeaseGrant | None:
+        """Take the provider's lease, or return ``None`` while another owner holds it."""
+        async with self._sf() as session:
+            dialect = _dialect(session)
+            now = db_now(dialect)
+            insert = pg_insert if dialect == "postgresql" else sqlite_insert
+            stmt = insert(ChannelLeaseRow).values(provider=provider, owner_id=owner_id, epoch=1, lease_expires_at=now + ttl_seconds, transport_running=False, updated_at=now)
+            stmt = stmt.on_conflict_do_update(
+                index_elements=[ChannelLeaseRow.provider],
+                set_={
+                    "owner_id": owner_id,
+                    "epoch": ChannelLeaseRow.epoch + 1,
+                    "lease_expires_at": now + ttl_seconds,
+                    "transport_running": False,
+                    "updated_at": now,
+                },
+                where=or_(ChannelLeaseRow.lease_expires_at + grace_seconds < now, ChannelLeaseRow.owner_id == owner_id),
+            )
+            if (await session.execute(stmt)).rowcount != 1:
+                await session.rollback()
+                return None
+            # Our own write, read inside the transaction that still holds it.
+            row = (await session.execute(select(ChannelLeaseRow.epoch, ChannelLeaseRow.lease_expires_at).where(ChannelLeaseRow.provider == provider, ChannelLeaseRow.owner_id == owner_id))).one()
+            await session.commit()
+        return ChannelLeaseGrant(provider=provider, epoch=int(row.epoch), lease_expires_at=float(row.lease_expires_at))
+
+    async def renew(self, provider: str, owner_id: str, epoch: int, *, ttl_seconds: float, transport_running: bool) -> bool:
+        """Extend a live lease this owner holds at ``epoch``; ``False`` means it is no longer held."""
+        async with self._sf() as session:
+            now = db_now(_dialect(session))
+            stmt = (
+                update(ChannelLeaseRow)
+                .where(
+                    ChannelLeaseRow.provider == provider,
+                    ChannelLeaseRow.owner_id == owner_id,
+                    ChannelLeaseRow.epoch == epoch,
+                    ChannelLeaseRow.lease_expires_at > now,
+                )
+                .values(lease_expires_at=now + ttl_seconds, transport_running=transport_running, updated_at=now)
+            )
+            renewed = (await session.execute(stmt)).rowcount == 1
+            await session.commit()
+        return renewed
+
+    async def release(self, provider: str, owner_id: str, epoch: int) -> bool:
+        """Give up the lease held at ``epoch`` so a peer can take it at once; ``False`` if it was not ours."""
+        async with self._sf() as session:
+            now = db_now(_dialect(session))
+            stmt = update(ChannelLeaseRow).where(ChannelLeaseRow.provider == provider, ChannelLeaseRow.owner_id == owner_id, ChannelLeaseRow.epoch == epoch).values(lease_expires_at=0.0, transport_running=False, updated_at=now)
+            released = (await session.execute(stmt)).rowcount == 1
+            await session.commit()
+        return released
+
+    async def get(self, provider: str) -> ChannelLeaseState | None:
+        """Read one provider's lease; ``live`` is judged against the database clock of the read."""
+        states = await self._read(ChannelLeaseRow.provider == provider)
+        return states[0] if states else None
+
+    async def list_all(self) -> list[ChannelLeaseState]:
+        return await self._read(None)
+
+    async def _read(self, predicate: Any) -> list[ChannelLeaseState]:
+        async with self._sf() as session:
+            now = db_now(_dialect(session))
+            stmt = select(
+                ChannelLeaseRow.provider,
+                ChannelLeaseRow.owner_id,
+                ChannelLeaseRow.epoch,
+                ChannelLeaseRow.lease_expires_at,
+                ChannelLeaseRow.transport_running,
+                (ChannelLeaseRow.lease_expires_at > now).label("live"),
+            ).order_by(ChannelLeaseRow.provider)
+            if predicate is not None:
+                stmt = stmt.where(predicate)
+            rows = (await session.execute(stmt)).all()
+        return [
+            ChannelLeaseState(
+                provider=row.provider,
+                owner_id=row.owner_id,
+                epoch=int(row.epoch),
+                lease_expires_at=float(row.lease_expires_at),
+                transport_running=bool(row.transport_running),
+                live=bool(row.live),
+            )
+            for row in rows
+        ]

backend/packages/harness/deerflow/persistence/migrations/AGENTS.md

@@ -31,7 +31,8 @@ The empty-DB path keeps using `create_all` because `Base.metadata` is the only a
 `0032_activity_and_task_events` → `0033_batch_result_artifact` → `0034_run_event_seq_watermark` →
 `0035_login_throttle` → `0036_run_idempotency_request` → `0037_project_document_summaries` →
 `0038_channel_thread_bindings` → `0039_user_disabled` →
-`0040_scheduled_group_notice` → `0041_im_notice_preferences` (current head). The preference
+`0040_scheduled_group_notice` → `0041_im_notice_preferences` →
+`0042_channel_leases` (current head). The preference
 revision adds a separate owner/key table with a cascading users foreign key and
 does not alter users; the project-documents revision adds a new owner-scoped
 shelf table, and the MCP lease-token revision adds two nullable token columns to
@@ -184,6 +185,7 @@ on installs that never enabled it. The convention is:
 - `migrations/versions/0036_run_idempotency_request.py` — adds nullable `runs.idempotency_request_json` for versioned keyed-resume identities. The field is private persistence state, never part of `kwargs_json` or a run response; this keeps new-writer/old-reader rolling upgrades from exposing resume digests or synthetic input. Legacy rows remain NULL; new readers accept only the exact old-writer identity shape from `kwargs`, while truly identity-less resume fails closed. The bootstrap forward-compat floor is unchanged. Chains after `0035_login_throttle`. Test: `tests/test_migration_0036_run_idempotency_request.py`
 - `migrations/versions/0037_project_document_summaries.py` — adds nullable `project_documents.summary` (best-effort LLM one-line description, off by default; no backfill, so legacy rows stay NULL and the bootstrap forward-compat floor is unchanged); chains after `0036_run_idempotency_request`. Test: `tests/test_migration_0037_project_document_summaries.py`
 - `migrations/versions/0038_channel_thread_bindings.py` — creates `channel_thread_bindings` (`key` PK = the legacy `channel_name:chat_id[:topic_id]` string, parsed `channel_name`/`chat_id`/nullable `topic_id`, `thread_id`, `user_id` (`""` when unknown, never NULL), epoch `Double` `created_at`/`updated_at` + `ix_channel_thread_bindings_channel_chat`), the shared successor of the per-process `channels/store.json` that `ChannelManager` used for unbound IM conversations (`persistence/channel_thread_bindings/`: `SqlChannelThreadBindingRepository` upserts with a dialect-native `ON CONFLICT (key) DO UPDATE` that keeps `created_at`, deletes a chat's base + topic rows by key prefix with `startswith(autoescape=True)`, and imports with batched `ON CONFLICT DO NOTHING` so two replicas importing the same file concurrently neither duplicate nor clobber a row; `app/channels/store.py::SqlChannelStore.import_legacy_json` runs that once from `ChannelService.start()` and renames the file to `store.json.migrated`). The shape is 1:1 with the JSON entries so the import is lossless; `split_binding_key` parses legacy keys on the first two colons exactly as the old `list_entries()` did (lookups always match the full `key`). Table create/drop and the index are guarded with a fresh inspector in both directions. New table, so the bootstrap forward-compat floor is unchanged; chains after `0037_project_document_summaries`. Test: `tests/test_migration_0038_channel_thread_bindings.py`
+- `migrations/versions/0042_channel_leases.py` — creates `channel_leases` (`provider` PK, `owner_id`, fencing `epoch` BIGINT, database-clock epoch `Double` `lease_expires_at`/`updated_at`, `transport_running`), the per-provider IM leader lease behind `channels.leader_lease` (`persistence/channel_leases/`: acquire is one dialect-native upsert whose `DO UPDATE ... WHERE lease_expires_at + grace < now OR owner_id = :me` bumps the epoch; renew extends only a live lease at the holder's epoch; release sets the expiry to 0; every `now` is the database clock, never an instance's). Table create/drop is guarded with a fresh inspector in both directions. New table, so the bootstrap forward-compat floor is unchanged; chains after `0041_im_notice_preferences`. Test: `tests/test_migration_0042_channel_leases.py`
 - `persistence/bootstrap.py` — `bootstrap_schema(engine, backend=...)`, the three-branch provisioning decision, locked revision validation, and the narrow 0019 forward-compatibility exception
 - `extensions/loader.py::load_extensions` — registers each spec's `table_prefix` with `register_extension_table_prefix()`
 - Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter, including extension-owned tables), `tests/test_extension_loader.py::TestTablePrefixRegistration` (spec-to-filter wiring), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor), `tests/test_migration_0004_run_ownership_dedupe.py` + `tests/test_migration_0007_scheduled_run_active_dedupe.py` (dedupe-before-unique-index pre-steps), `tests/test_migration_0025_repair_run_change_seq.py` (issue #5516 skipped-revision heal)

backend/packages/harness/deerflow/persistence/migrations/versions/0042_channel_leases.py

@@ -0,0 +1,56 @@
+"""Per-provider IM transport leader lease.
+
+Revision ID: 0042_channel_leases
+Revises: 0041_im_notice_preferences
+
+Creates ``channel_leases``: one row per IM provider naming the Gateway
+instance that holds its platform connection (``owner_id``), a fencing
+``epoch`` bumped on every acquisition, the database-clock epoch seconds the
+lease expires at, whether the holder's transport was running at its last
+renewal, and ``updated_at``. Only instances with ``channels.leader_lease``
+active write it; a single-instance deployment leaves it empty.
+
+``_helpers.py`` only guards columns, so the table is guarded with a fresh
+inspector in both directions: an upgrade that finds the table (an interrupted
+earlier attempt, or an empty-database bootstrap that already ran
+``create_all``) completes, and a downgrade after the table is gone still
+moves the version row.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0042_channel_leases"
+down_revision: str | Sequence[str] | None = "0041_im_notice_preferences"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+_TABLE = "channel_leases"
+
+
+def _has_table() -> bool:
+    return sa.inspect(op.get_bind()).has_table(_TABLE)
+
+
+def upgrade() -> None:
+    if _has_table():
+        return
+    op.create_table(
+        _TABLE,
+        sa.Column("provider", sa.String(length=64), nullable=False),
+        sa.Column("owner_id", sa.String(length=128), nullable=False),
+        sa.Column("epoch", sa.BigInteger(), nullable=False),
+        sa.Column("lease_expires_at", sa.Double(), nullable=False),
+        sa.Column("transport_running", sa.Boolean(), nullable=False),
+        sa.Column("updated_at", sa.Double(), nullable=False),
+        sa.PrimaryKeyConstraint("provider", name="pk_channel_leases"),
+    )
+
+
+def downgrade() -> None:
+    if _has_table():
+        op.drop_table(_TABLE)

backend/packages/harness/deerflow/persistence/models/__init__.py

@@ -21,6 +21,7 @@ from deerflow.persistence.channel_connections.model import (
     ChannelCredentialRow,
     ChannelOAuthStateRow,
 )
+from deerflow.persistence.channel_leases.model import ChannelLeaseRow
 from deerflow.persistence.channel_thread_bindings.model import ChannelThreadBindingRow
 from deerflow.persistence.feedback.model import FeedbackRow
 from deerflow.persistence.login_throttle.model import LoginThrottleRow
@@ -46,6 +47,7 @@ __all__ = [
     "ChannelConnectionRow",
     "ChannelConversationRow",
     "ChannelCredentialRow",
+    "ChannelLeaseRow",
     "ChannelOAuthStateRow",
     "ChannelThreadBindingRow",
     "FeedbackRow",

backend/packages/harness/deerflow/persistence/notification_deliveries/sql.py

@@ -14,6 +14,7 @@ from __future__ import annotations
 
 import logging
 import uuid
+from collections.abc import Collection
 from datetime import UTC, datetime, timedelta
 from typing import Any
 
@@ -185,9 +186,13 @@ class NotificationDeliveryRepository:
             )
             return result.scalar_one_or_none()
 
-    async def claim_due_deliveries(self, *, now: datetime, limit: int) -> list[dict[str, Any]]:
+    async def claim_due_deliveries(self, *, now: datetime, limit: int, exclude_providers: Collection[str] = ()) -> list[dict[str, Any]]:
         """Atomically claim due pending rows by flipping them to ``sending``.
 
+        ``exclude_providers`` leaves rows for those providers pending for
+        another worker: under the IM leader lease, the instance that holds a
+        provider's transport delivers its rows.
+
         The UPDATE is guarded by ``status = 'pending'`` and returns its own
         result set via ``RETURNING`` (SQLite >= 3.35, Postgres), so the rows
         handed back are exactly the rows THIS transaction flipped. A separate
@@ -207,6 +212,7 @@ class NotificationDeliveryRepository:
                         .where(
                             NotificationDeliveryRow.status == "pending",
                             NotificationDeliveryRow.available_at <= now,
+                            *((NotificationDeliveryRow.provider.not_in(sorted(exclude_providers)),) if exclude_providers else ()),
                         )
                         .order_by(NotificationDeliveryRow.available_at, NotificationDeliveryRow.created_at)
                         .limit(limit)

backend/tests/blocking_io/test_channel_leader_lease.py

@@ -0,0 +1,98 @@
+"""Regression anchors: the IM leader lease must not block the event loop.
+
+``ChannelService`` acquires, renews and releases ``channel_leases`` rows on the
+Gateway event loop: at startup, on every supervision tick (every
+``lease_seconds // 3`` seconds per provider), from readiness requests and at
+shutdown. Every statement goes through the async session (aiosqlite /
+asyncpg) and every ``wait_for`` bound is a loop timer. If any of it regresses
+onto the loop — a synchronous driver call, a blocking clock or lock — the
+strict Blockbuster gate raises ``BlockingError`` here.
+
+The engine and schema are built off the loop: that is fixture work, not the
+production path under test.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncIterator
+from pathlib import Path
+
+import pytest
+import pytest_asyncio
+import sqlalchemy as sa
+from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, create_async_engine
+
+# Pre-import so lazy imports inside the service are cached no-ops under the gate.
+import deerflow.persistence.models  # noqa: F401
+from app.channels.leadership import ChannelLeadership, RenewOutcome
+from app.channels.service import ChannelService
+from app.channels.store import JsonChannelStore
+from deerflow import reflection
+from deerflow.persistence.base import Base
+from deerflow.persistence.channel_leases import ChannelLeaseRow, SqlChannelLeaseRepository
+
+pytestmark = pytest.mark.asyncio
+
+
+def _build_sqlite_engine(path: Path) -> AsyncEngine:
+    sync_engine = sa.create_engine(f"sqlite:///{path.as_posix()}")
+    try:
+        Base.metadata.create_all(sync_engine, tables=[ChannelLeaseRow.__table__])
+    finally:
+        sync_engine.dispose()
+    return create_async_engine(f"sqlite+aiosqlite:///{path.as_posix()}")
+
+
+@pytest_asyncio.fixture
+async def repo(tmp_path) -> AsyncIterator[SqlChannelLeaseRepository]:
+    engine = await asyncio.to_thread(_build_sqlite_engine, tmp_path / "leases.db")
+    try:
+        yield SqlChannelLeaseRepository(async_sessionmaker(engine, expire_on_commit=False))
+    finally:
+        await engine.dispose()
+
+
+class _Transport:
+    def __init__(self, *, bus: object, config: dict) -> None:
+        self.is_running = False
+
+    async def start(self) -> None:
+        self.is_running = True
+
+    async def stop(self) -> None:
+        self.is_running = False
+
+
+async def test_lease_round_trip_does_not_block_loop(repo) -> None:
+    holder = ChannelLeadership(repo, owner_id="A", lease_seconds=30, grace_seconds=10)
+    peer = ChannelLeadership(repo, owner_id="B", lease_seconds=30, grace_seconds=10)
+    assert await holder.acquire("telegram") is True
+    assert await peer.acquire("telegram") is False  # refusal plus the observation read
+    assert await holder.renew("telegram", transport_running=True) is RenewOutcome.RENEWED
+    await holder.release("telegram")
+    assert await peer.acquire("telegram") is True
+    assert [state.owner_id for state in await repo.list_all()] == ["B"]
+
+
+async def test_service_supervision_and_shutdown_do_not_block_loop(repo, tmp_path, monkeypatch) -> None:
+    """The production caller: startup acquisition, a supervision round, and the releasing shutdown."""
+    monkeypatch.setattr(reflection, "resolve_class", lambda *_args, **_kwargs: _Transport)
+    store_path = await asyncio.to_thread(lambda: tmp_path / "store.json")
+    service = ChannelService(
+        channels_config={"telegram": {"enabled": True}, "github": {"enabled": True}},
+        store=JsonChannelStore(store_path),
+        leadership=ChannelLeadership(repo, owner_id="A", lease_seconds=30, grace_seconds=10),
+    )
+
+    async def noop() -> None:
+        return None
+
+    service.manager.start = noop
+    service.manager.stop = noop
+    service._start_supervisor = lambda: None
+    await service.start()
+    await service._supervise_once()
+    assert service.get_status()["channels"]["telegram"]["leader"] == "A"
+    await service.stop()
+    assert (await repo.get("telegram")).live is False

backend/tests/test_channel_credentials_wiring.py

@@ -170,7 +170,7 @@ def test_start_channel_service_threads_the_cipher_through(monkeypatch: pytest.Mo
         async def start(self):
             return None
 
-    def fake_from_app_config(app_config=None, *, get_stream_bridge=None, credentials_cipher=None):
+    def fake_from_app_config(app_config=None, *, get_stream_bridge=None, credentials_cipher=None, **_lease):
         captured["credentials_cipher"] = credentials_cipher
         return _FakeService()
 

backend/tests/test_channel_leadership.py

@@ -0,0 +1,242 @@
+"""``ChannelLeadership``: the holder's side of the per-provider IM leader lease.
+
+The repository decides ownership against the database clock; this class keeps
+the local, monotonic deadline that tells the holder when it must stop its
+transport. The fake repository below shares the test clock, so the safety
+margin (a peer can take over only ``lease + grace`` after the last confirmed
+renewal was *sent*) is checked exactly.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+
+from app.channels.leadership import ChannelLeadership, RenewOutcome
+from deerflow.persistence.channel_leases import ChannelLeaseGrant, ChannelLeaseState
+
+pytestmark = pytest.mark.asyncio
+
+
+class Clock:
+    def __init__(self) -> None:
+        self.now = 1000.0
+
+    def __call__(self) -> float:
+        return self.now
+
+
+class FakeRepo:
+    """One lease table, judged against the shared test clock like the SQL repository."""
+
+    def __init__(self, clock: Clock) -> None:
+        self.clock = clock
+        self.rows: dict[str, dict] = {}
+        self.fail: set[str] = set()
+        self.call_latency = 0.0  # simulated statement time on the test clock
+        self.calls: list[tuple] = []
+
+    def _maybe_fail(self, op: str) -> None:
+        self.calls.append((op,))
+        if op in self.fail:
+            raise ConnectionError(f"{op} failed")
+
+    async def acquire(self, provider, owner_id, *, ttl_seconds, grace_seconds):
+        self._maybe_fail("acquire")
+        self.clock.now += self.call_latency
+        row = self.rows.get(provider)
+        now = self.clock.now
+        if row is None or row["expires"] + grace_seconds < now or row["owner"] == owner_id:
+            epoch = (row["epoch"] + 1) if row else 1
+            self.rows[provider] = {"owner": owner_id, "epoch": epoch, "expires": now + ttl_seconds, "running": False}
+            return ChannelLeaseGrant(provider=provider, epoch=epoch, lease_expires_at=now + ttl_seconds)
+        return None
+
+    async def renew(self, provider, owner_id, epoch, *, ttl_seconds, transport_running):
+        self._maybe_fail("renew")
+        self.clock.now += self.call_latency
+        row = self.rows.get(provider)
+        now = self.clock.now
+        if row is None or row["owner"] != owner_id or row["epoch"] != epoch or row["expires"] <= now:
+            return False
+        row.update(expires=now + ttl_seconds, running=transport_running)
+        return True
+
+    async def release(self, provider, owner_id, epoch):
+        self._maybe_fail("release")
+        row = self.rows.get(provider)
+        if row is None or row["owner"] != owner_id or row["epoch"] != epoch:
+            return False
+        row.update(expires=0.0, running=False)
+        return True
+
+    async def get(self, provider):
+        self._maybe_fail("get")
+        row = self.rows.get(provider)
+        if row is None:
+            return None
+        return ChannelLeaseState(provider=provider, owner_id=row["owner"], epoch=row["epoch"], lease_expires_at=row["expires"], transport_running=row["running"], live=row["expires"] > self.clock.now)
+
+    async def list_all(self):
+        self._maybe_fail("list_all")
+        return [await self.get(provider) for provider in sorted(self.rows)]
+
+
+def _leadership(repo: FakeRepo, clock: Clock, owner: str = "A", *, lease_seconds: int = 30, grace_seconds: int = 10) -> ChannelLeadership:
+    return ChannelLeadership(repo, owner_id=owner, lease_seconds=lease_seconds, grace_seconds=grace_seconds, clock=clock)
+
+
+async def test_intervals_follow_the_documented_lease_fraction():
+    clock = Clock()
+    lead = _leadership(FakeRepo(clock), clock, lease_seconds=30)
+    assert (lead.renew_interval, lead.operation_timeout) == (10.0, 5.0)
+    small = _leadership(FakeRepo(clock), clock, lease_seconds=3)
+    assert (small.renew_interval, small.operation_timeout) == (1.0, 0.5)
+
+
+async def test_holds_stops_one_renewal_interval_before_the_local_deadline():
+    clock = Clock()
+    lead = _leadership(FakeRepo(clock), clock)
+    assert await lead.acquire("telegram") is True
+
+    clock.now += 19.9  # deadline 1030, interval 10
+    assert lead.holds("telegram") is True
+    clock.now += 0.1
+    assert lead.holds("telegram") is False
+    assert lead.has_record("telegram") is True  # the row may still be ours; the transport may not keep running
+
+
+async def test_deadline_counts_from_when_the_renewal_was_sent():
+    """A slow statement must not stretch the deadline: the database stamped the expiry no earlier than the send."""
+    clock = Clock()
+    repo = FakeRepo(clock)
+    lead = _leadership(repo, clock)
+    await lead.acquire("telegram")
+
+    clock.now += 10
+    sent_at = clock.now
+    repo.call_latency = 4.0
+    assert await lead.renew("telegram", transport_running=True) is RenewOutcome.RENEWED
+
+    peer_can_take_at = repo.rows["telegram"]["expires"] + lead.grace_seconds
+    assert peer_can_take_at >= sent_at + lead.lease_seconds + lead.grace_seconds
+    clock.now = sent_at + lead.lease_seconds - lead.renew_interval
+    assert lead.holds("telegram") is False  # judged from the send time, not the (later) reply
+
+
+async def test_a_rejected_renewal_is_a_loss():
+    clock = Clock()
+    repo = FakeRepo(clock)
+    lead = _leadership(repo, clock)
+    await lead.acquire("telegram")
+    repo.rows["telegram"]["epoch"] += 1  # a peer took the row over
+
+    assert await lead.renew("telegram", transport_running=True) is RenewOutcome.LOST
+    assert lead.has_record("telegram") is False
+    assert lead.holds("telegram") is False
+
+
+async def test_an_unconfirmed_renewal_keeps_the_deadline_running():
+    clock = Clock()
+    repo = FakeRepo(clock)
+    lead = _leadership(repo, clock)
+    await lead.acquire("telegram")
+    repo.fail.add("renew")
+
+    clock.now += 10
+    assert await lead.renew("telegram", transport_running=True) is RenewOutcome.UNCONFIRMED
+    assert lead.holds("telegram") is True  # 1010 + 10 < 1030
+    clock.now += 10
+    assert await lead.renew("telegram", transport_running=True) is RenewOutcome.UNCONFIRMED
+    assert lead.holds("telegram") is False  # 1020 + 10 is not before 1030: stop now, grace_seconds before a peer may connect
+    assert lead.has_record("telegram") is True
+
+
+async def test_a_stalled_database_call_is_bounded_by_the_operation_timeout():
+    class StalledRepo(FakeRepo):
+        async def renew(self, *args, **kwargs):
+            await asyncio.sleep(30)
+
+    clock = Clock()
+    repo = StalledRepo(clock)
+    lead = ChannelLeadership(repo, owner_id="A", lease_seconds=3, grace_seconds=1, clock=clock)
+    await lead.acquire("telegram")
+
+    outcome = await asyncio.wait_for(lead.renew("telegram", transport_running=True), timeout=5)
+    assert outcome is RenewOutcome.UNCONFIRMED
+
+
+async def test_a_refused_acquire_records_the_peer_for_status():
+    clock = Clock()
+    repo = FakeRepo(clock)
+    peer = _leadership(repo, clock, owner="B")
+    await peer.acquire("telegram")
+    await peer.renew("telegram", transport_running=True)
+
+    lead = _leadership(repo, clock, owner="A")
+    assert await lead.acquire("telegram") is False
+    assert lead.leader_of("telegram") == "B"
+    assert lead.peer_transport_running("telegram") is True
+    assert lead.leader_of("discord") is None
+
+    clock.now += 100  # the peer's lease lapsed
+    assert await lead.acquire("telegram") is True
+    assert lead.leader_of("telegram") == "A"
+    assert lead.peer_transport_running("telegram") is False
+
+
+async def test_acquire_while_holding_is_a_no_op_and_a_lost_record_is_dropped():
+    clock = Clock()
+    repo = FakeRepo(clock)
+    lead = _leadership(repo, clock)
+    await lead.acquire("telegram")
+    calls = len(repo.calls)
+    assert await lead.acquire("telegram") is True
+    assert len(repo.calls) == calls  # no statement while the hold is safe
+
+    clock.now += 25  # near the deadline: the next acquire re-takes our own row
+    repo.rows["telegram"].update(owner="B", expires=clock.now + 30)
+    assert await lead.acquire("telegram") is False
+    assert lead.has_record("telegram") is False
+
+
+async def test_release_is_fenced_by_epoch_and_best_effort():
+    clock = Clock()
+    repo = FakeRepo(clock)
+    lead = _leadership(repo, clock)
+    await lead.acquire("telegram")
+    await lead.release("telegram")
+    assert repo.rows["telegram"]["expires"] == 0.0
+    assert lead.has_record("telegram") is False
+
+    await lead.acquire("discord")
+    repo.fail.add("release")
+    await lead.release("discord")  # the error is logged, not raised; the row expires on its own
+    assert lead.has_record("discord") is False
+    await lead.release("slack")  # nothing held: no call
+    assert ("release",) in repo.calls
+
+
+async def test_observe_all_sees_peer_leases_this_instance_never_tried():
+    """A provider connected on a peer is in no local config, so no failed acquire ever observes it."""
+    clock = Clock()
+    repo = FakeRepo(clock)
+    peer = _leadership(repo, clock, owner="B")
+    await peer.acquire("telegram")
+    await peer.renew("telegram", transport_running=True)
+    lead = _leadership(repo, clock, owner="A")
+    await lead.acquire("discord")
+
+    assert lead.leader_of("telegram") is None
+    await lead.observe_all()
+    assert (lead.leader_of("telegram"), lead.peer_leads("telegram"), lead.peer_transport_running("telegram")) == ("B", True, True)
+    assert (lead.leader_of("discord"), lead.peer_leads("discord")) == ("A", False)  # our own row never reads as a peer's
+
+    clock.now += 100  # the peer's lease lapses
+    await lead.observe_all()
+    assert (lead.leader_of("telegram"), lead.peer_leads("telegram")) == (None, False)
+
+    repo.fail.add("list_all")
+    await lead.observe_all()  # logged, not raised; the last view stays
+    assert lead.leader_of("telegram") is None

backend/tests/test_channel_lease_repository.py

@@ -0,0 +1,212 @@
+"""``channel_leases`` repository: the per-provider IM leader lease shared by every Gateway instance.
+
+Each case runs on SQLite and, when ``TEST_POSTGRES_URI`` is set, on PostgreSQL
+in a throwaway schema. Expiry is staged by writing ``lease_expires_at``
+relative to the database clock (``db_now``) instead of sleeping, so the cases
+are deterministic and exercise the same clock the repository decides with.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+import time
+import uuid
+from collections.abc import AsyncIterator
+from dataclasses import dataclass
+
+import pytest
+import pytest_asyncio
+import sqlalchemy as sa
+from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, create_async_engine
+from support.postgres import asyncpg_test_url
+
+from deerflow.persistence.base import Base
+from deerflow.persistence.channel_leases import ChannelLeaseRow, SqlChannelLeaseRepository, db_now
+from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
+
+pytestmark = pytest.mark.asyncio
+
+TTL = 30.0
+GRACE = 10.0
+
+
+@dataclass
+class Database:
+    dialect: str
+    engines: list[AsyncEngine]
+
+    def repo(self, index: int = 0) -> SqlChannelLeaseRepository:
+        return SqlChannelLeaseRepository(async_sessionmaker(self.engines[index], expire_on_commit=False))
+
+    async def set_expiry(self, provider: str, *, seconds_from_now: float) -> None:
+        """Move the row's expiry relative to the database clock."""
+        async with self.engines[0].begin() as conn:
+            await conn.execute(sa.update(ChannelLeaseRow).where(ChannelLeaseRow.provider == provider).values(lease_expires_at=db_now(self.dialect) + seconds_from_now))
+
+
+@pytest_asyncio.fixture(params=["sqlite", "postgres"])
+async def db(request, tmp_path) -> AsyncIterator[Database]:
+    """Two engines over one database: two Gateway instances sharing the table."""
+    if request.param == "sqlite":
+        url = f"sqlite+aiosqlite:///{(tmp_path / 'leases.db').as_posix()}"
+        engines = [create_async_engine(url) for _ in range(2)]
+        async with engines[0].begin() as conn:
+            await conn.run_sync(Base.metadata.create_all, tables=[ChannelLeaseRow.__table__])
+        try:
+            yield Database("sqlite", engines)
+        finally:
+            for engine in engines:
+                await engine.dispose()
+        return
+
+    uri = os.environ.get("TEST_POSTGRES_URI")
+    if not uri:
+        pytest.skip("requires TEST_POSTGRES_URI (channel_leases on PostgreSQL)")
+    schema = f"channel_leases_{uuid.uuid4().hex}"
+    engines = [create_async_engine(asyncpg_test_url(uri), connect_args=build_asyncpg_connect_args(schema)) for _ in range(2)]
+    try:
+        async with engines[0].begin() as conn:
+            await conn.execute(sa.text(f'CREATE SCHEMA "{schema}"'))
+            await conn.run_sync(Base.metadata.create_all, tables=[ChannelLeaseRow.__table__])
+        yield Database("postgresql", engines)
+    finally:
+        async with engines[0].begin() as conn:
+            await conn.execute(sa.text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
+        for engine in engines:
+            await engine.dispose()
+
+
+async def test_db_now_reads_epoch_seconds(db):
+    """The clock expression is in seconds since the epoch, not days or a timestamp string."""
+    async with db.engines[0].connect() as conn:
+        value = (await conn.execute(sa.select(db_now(db.dialect)))).scalar_one()
+    assert abs(float(value) - time.time()) < 60
+
+
+async def test_first_acquire_creates_the_row_and_a_peer_is_refused_while_it_is_live(db):
+    a, b = db.repo(0), db.repo(1)
+
+    grant = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+    assert grant is not None and grant.epoch == 1
+    assert await b.acquire("telegram", "B", ttl_seconds=TTL, grace_seconds=GRACE) is None
+
+    state = await b.get("telegram")
+    assert (state.owner_id, state.epoch, state.live, state.transport_running) == ("A", 1, True, False)
+    # Providers are independent rows.
+    assert (await b.acquire("discord", "B", ttl_seconds=TTL, grace_seconds=GRACE)).epoch == 1
+    assert [s.provider for s in await a.list_all()] == ["discord", "telegram"]
+
+
+async def test_renew_extends_only_the_holders_epoch_and_reports_the_transport(db):
+    a = db.repo(0)
+    grant = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+
+    assert await a.renew("telegram", "A", grant.epoch, ttl_seconds=TTL, transport_running=True) is True
+    assert (await a.get("telegram")).transport_running is True
+    # Another owner, or the right owner at a stale epoch, cannot renew.
+    assert await a.renew("telegram", "B", grant.epoch, ttl_seconds=TTL, transport_running=True) is False
+    assert await a.renew("telegram", "A", grant.epoch + 1, ttl_seconds=TTL, transport_running=True) is False
+    assert await a.renew("slack", "A", 1, ttl_seconds=TTL, transport_running=True) is False
+
+
+async def test_an_expired_lease_is_not_revived_by_renewal(db):
+    """Renewal only extends a live lease; a lapsed holder must re-acquire (and bump the epoch)."""
+    a = db.repo(0)
+    grant = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+    await db.set_expiry("telegram", seconds_from_now=-1)
+
+    assert await a.renew("telegram", "A", grant.epoch, ttl_seconds=TTL, transport_running=True) is False
+    again = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+    assert again is not None and again.epoch == grant.epoch + 1
+
+
+async def test_a_peer_takes_an_expired_lease_only_after_the_grace_period(db):
+    a, b = db.repo(0), db.repo(1)
+    first = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+
+    await db.set_expiry("telegram", seconds_from_now=-(GRACE - 2))  # expired, still inside the grace window
+    assert await b.acquire("telegram", "B", ttl_seconds=TTL, grace_seconds=GRACE) is None
+    assert (await b.get("telegram")).live is False
+
+    await db.set_expiry("telegram", seconds_from_now=-(GRACE + 2))
+    taken = await b.acquire("telegram", "B", ttl_seconds=TTL, grace_seconds=GRACE)
+    assert taken is not None and taken.epoch == first.epoch + 1
+    # The previous holder is fenced out of the new epoch.
+    assert await a.renew("telegram", "A", first.epoch, ttl_seconds=TTL, transport_running=True) is False
+    assert await a.release("telegram", "A", first.epoch) is False
+    assert (await a.get("telegram")).owner_id == "B"
+
+
+async def test_the_holder_reacquiring_bumps_the_epoch_and_fences_its_old_one(db):
+    a = db.repo(0)
+    first = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+    second = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+
+    assert second.epoch == first.epoch + 1
+    assert await a.renew("telegram", "A", first.epoch, ttl_seconds=TTL, transport_running=False) is False
+    assert await a.renew("telegram", "A", second.epoch, ttl_seconds=TTL, transport_running=False) is True
+
+
+async def test_release_hands_the_lease_over_at_once(db):
+    a, b = db.repo(0), db.repo(1)
+    grant = await a.acquire("telegram", "A", ttl_seconds=TTL, grace_seconds=GRACE)
+    await a.renew("telegram", "A", grant.epoch, ttl_seconds=TTL, transport_running=True)
+
+    assert await a.release("telegram", "A", grant.epoch) is True
+    state = await b.get("telegram")
+    assert (state.live, state.transport_running) == (False, False)
+    # No waiting out lease + grace after a release.
+    taken = await b.acquire("telegram", "B", ttl_seconds=TTL, grace_seconds=GRACE)
+    assert taken is not None and taken.epoch == grant.epoch + 1
+    # A second release at the old epoch cannot undo the new holder's lease.
+    assert await a.release("telegram", "A", grant.epoch) is False
+    assert (await b.get("telegram")).live is True
+
+
+async def test_racing_instances_get_exactly_one_grant(db):
+    """Concurrent first acquisitions (an empty table) and takeovers both have a single winner."""
+    repos = [db.repo(i % 2) for i in range(8)]
+
+    grants = await asyncio.gather(*(repo.acquire("discord", f"owner-{i}", ttl_seconds=TTL, grace_seconds=GRACE) for i, repo in enumerate(repos)))
+    winners = [i for i, grant in enumerate(grants) if grant is not None]
+    assert len(winners) == 1
+    assert (await repos[0].get("discord")).owner_id == f"owner-{winners[0]}"
+
+    await db.set_expiry("discord", seconds_from_now=-(GRACE + 5))
+    grants = await asyncio.gather(*(repo.acquire("discord", f"next-{i}", ttl_seconds=TTL, grace_seconds=GRACE) for i, repo in enumerate(repos)))
+    assert len([grant for grant in grants if grant is not None]) == 1
+    assert (await repos[0].get("discord")).epoch == 2
+
+
+async def test_writes_do_not_trust_returning_rows_under_a_concurrent_read(tmp_path):
+    """Review follow-up on #6843: the supervisor reads the whole table while it renews.
+
+    Through SQLAlchemy's aiosqlite adapter, an ``UPDATE ... RETURNING`` running
+    while another connection reads the table intermittently handed back one of
+    its own bind values (the provider name) instead of ``lease_expires_at``, on
+    fresh engines in roughly a third of the rounds. Renew, release and acquire
+    therefore decide by affected-row count; every round here must confirm both
+    renewals and leave real numbers in the row.
+    """
+    for scenario in range(12):
+        url = f"sqlite+aiosqlite:///{(tmp_path / f'leases-{scenario}.db').as_posix()}"
+        engine = create_async_engine(url)
+        try:
+            async with engine.begin() as conn:
+                await conn.run_sync(Base.metadata.create_all, tables=[ChannelLeaseRow.__table__])
+            repo = SqlChannelLeaseRepository(async_sessionmaker(engine, expire_on_commit=False))
+            grants = {provider: await repo.acquire(provider, "A", ttl_seconds=TTL, grace_seconds=GRACE) for provider in ("discord", "telegram")}
+            for _ in range(2):
+                read, *renewed = await asyncio.gather(
+                    repo.list_all(),
+                    repo.renew("discord", "A", grants["discord"].epoch, ttl_seconds=TTL, transport_running=False),
+                    repo.renew("telegram", "A", grants["telegram"].epoch, ttl_seconds=TTL, transport_running=True),
+                )
+                assert renewed == [True, True]
+                assert {state.provider for state in read} == {"discord", "telegram"}
+            async with engine.connect() as conn:
+                kinds = (await conn.execute(sa.text("SELECT typeof(lease_expires_at), typeof(updated_at) FROM channel_leases"))).all()
+            assert set(kinds) == {("real", "real")}
+        finally:
+            await engine.dispose()

backend/tests/test_channel_service_leader_lease.py

@@ -0,0 +1,532 @@
+"""``ChannelService`` under ``channels.leader_lease``: one transport per provider across instances.
+
+Two services share one SQLite ``channel_leases`` table, as two Gateway
+instances share the application database. Transports are fakes; supervision
+rounds are driven by hand (``_supervise_once`` + the scheduled per-provider
+actions) except for the end-to-end failover case, which runs the real loop
+with a 3-second lease.
+
+On ``main`` before the lease existed both services started every enabled
+provider: that duplicate Telegram poller / Discord session is what
+``test_only_the_lease_holder_starts_an_arbitrated_transport`` pins.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from collections.abc import AsyncIterator
+from pathlib import Path
+
+import pytest
+import pytest_asyncio
+import sqlalchemy as sa
+from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmaker, create_async_engine
+
+from app.channels.base import ChannelStopTimeout
+from app.channels.leadership import ChannelLeadership
+from app.channels.service import ChannelService
+from app.channels.store import JsonChannelStore
+from deerflow import reflection
+from deerflow.persistence.base import Base
+from deerflow.persistence.channel_leases import ChannelLeaseRow, SqlChannelLeaseRepository, db_now
+
+pytestmark = pytest.mark.asyncio
+
+
+class FakeTransport:
+    """A platform connection: ``started`` counts real starts across the test."""
+
+    started: list[FakeTransport] = []
+    start_gate: asyncio.Event | None = None
+    fail_start = False
+    stop_error: Exception | None = None
+
+    def __init__(self, *, bus: object, config: dict) -> None:
+        self.config = config
+        self.is_running = False
+
+    async def start(self) -> None:
+        if FakeTransport.start_gate is not None:
+            await FakeTransport.start_gate.wait()
+        if FakeTransport.fail_start:
+            return
+        self.is_running = True
+        FakeTransport.started.append(self)
+
+    async def stop(self) -> None:
+        if FakeTransport.stop_error is not None:
+            raise FakeTransport.stop_error
+        self.is_running = False
+
+
+class Clock:
+    def __init__(self) -> None:
+        self.now = 1000.0
+
+    def __call__(self) -> float:
+        return self.now
+
+
+@pytest.fixture(autouse=True)
+def fake_transports(monkeypatch):
+    FakeTransport.started = []
+    FakeTransport.start_gate = None
+    FakeTransport.fail_start = False
+    FakeTransport.stop_error = None
+    monkeypatch.setattr(reflection, "resolve_class", lambda *_args, **_kwargs: FakeTransport)
+    yield
+
+
+@pytest_asyncio.fixture
+async def engine(tmp_path) -> AsyncIterator[AsyncEngine]:
+    engine = create_async_engine(f"sqlite+aiosqlite:///{(tmp_path / 'leases.db').as_posix()}")
+    async with engine.begin() as conn:
+        await conn.run_sync(Base.metadata.create_all, tables=[ChannelLeaseRow.__table__])
+    try:
+        yield engine
+    finally:
+        await engine.dispose()
+
+
+def _repo(engine: AsyncEngine) -> SqlChannelLeaseRepository:
+    return SqlChannelLeaseRepository(async_sessionmaker(engine, expire_on_commit=False))
+
+
+CHANNELS = {"telegram": {"enabled": True, "bot_token": "t-1"}, "discord": {"enabled": True, "bot_token": "d-1"}, "github": {"enabled": True}}
+
+
+def _service(engine, owner: str, tmp_path: Path, *, clock=None, channels=None, repo=None, lease_seconds: int = 30, grace_seconds: int = 10, supervise: bool = False) -> ChannelService:
+    leadership = ChannelLeadership(repo or _repo(engine), owner_id=owner, lease_seconds=lease_seconds, grace_seconds=grace_seconds, clock=clock or time.monotonic)
+    service = ChannelService(channels_config={name: dict(block) for name, block in (channels or CHANNELS).items()}, store=JsonChannelStore(tmp_path / owner / "store.json"), leadership=leadership)
+
+    async def noop() -> None:
+        return None
+
+    service.manager.start = noop
+    service.manager.stop = noop
+    if not supervise:
+        service._start_supervisor = lambda: None
+    # A takeover reloads the provider's config from disk; keep the cached one.
+    service._load_channel_config = lambda name: None
+    return service
+
+
+async def _round(service: ChannelService) -> None:
+    """One supervision round, then wait for the transport actions it scheduled."""
+    await service._supervise_once()
+    if service._lease_actions:
+        await asyncio.gather(*list(service._lease_actions.values()))
+
+
+def _running(service: ChannelService) -> set[str]:
+    return {name for name, channel in service._channels.items() if channel.is_running}
+
+
+async def _lease(engine, provider: str) -> ChannelLeaseRow | None:
+    async with async_sessionmaker(engine)() as session:
+        return await session.get(ChannelLeaseRow, provider)
+
+
+async def test_only_the_lease_holder_starts_an_arbitrated_transport(engine, tmp_path):
+    a, b = _service(engine, "A", tmp_path), _service(engine, "B", tmp_path)
+    await a.start()
+    await b.start()
+    try:
+        assert _running(a) == {"telegram", "discord", "github"}
+        # GitHub is webhook-driven and stays on every instance; the rest have one transport.
+        assert _running(b) == {"github"}
+        assert len([t for t in FakeTransport.started if "bot_token" in t.config]) == 2
+
+        await _round(a)  # A's renewal reports its transports running
+        await _round(b)  # B's refused acquisitions observe A
+        for service in (a, b):
+            status = service.get_status()["channels"]
+            assert status["telegram"]["running"] is True
+            assert status["telegram"]["leader"] == "A"
+            assert status["github"]["running"] is True and "leader" not in status["github"]
+        assert a.get_status()["channels"]["telegram"]["leader_here"] is True
+        assert b.get_status()["channels"]["telegram"]["leader_here"] is False
+        assert b.leader_of("telegram") == "A" and b.leader_of("github") is None
+    finally:
+        await a.stop()
+        await b.stop()
+
+
+async def test_a_stopping_holder_releases_and_the_peer_takes_over_on_its_next_round(engine, tmp_path):
+    a, b = _service(engine, "A", tmp_path), _service(engine, "B", tmp_path)
+    await a.start()
+    await b.start()
+    try:
+        await a.stop()
+        row = await _lease(engine, "telegram")
+        assert (row.owner_id, row.lease_expires_at) == ("A", 0.0)  # released, not left to expire
+
+        # The new holder starts from its freshly loaded config (credentials saved since it started).
+        b._load_channel_config = lambda name: {"enabled": True, "bot_token": f"{name}-rotated"}
+        await _round(b)
+        assert _running(b) == {"telegram", "discord", "github"}
+        assert b._channels["telegram"].config["bot_token"] == "telegram-rotated"
+        assert (await _lease(engine, "telegram")).owner_id == "B"
+    finally:
+        await b.stop()
+
+
+async def test_a_lost_lease_stops_the_transport(engine, tmp_path):
+    a = _service(engine, "A", tmp_path)
+    await a.start()
+    try:
+        async with engine.begin() as conn:  # a peer took the row over (newer epoch)
+            await conn.execute(sa.update(ChannelLeaseRow).where(ChannelLeaseRow.provider == "telegram").values(owner_id="B", epoch=ChannelLeaseRow.epoch + 1))
+        await _round(a)
+        assert "telegram" not in a._channels
+        assert _running(a) == {"discord", "github"}
+        # The next round only observes the peer; it does not reconnect.
+        await _round(a)
+        assert "telegram" not in a._channels
+        assert a.get_status()["channels"]["telegram"]["leader"] == "B"
+    finally:
+        await a.stop()
+    assert (await _lease(engine, "telegram")).owner_id == "B"  # shutdown never releases someone else's lease
+
+
+async def test_unconfirmed_renewals_stop_the_transport_before_a_peer_may_connect(engine, tmp_path):
+    clock = Clock()
+    repo = _repo(engine)
+    a = _service(engine, "A", tmp_path, clock=clock, repo=repo)
+    await a.start()
+    try:
+
+        async def unreachable(*args, **kwargs):
+            raise ConnectionError("database unreachable")
+
+        repo.renew = unreachable
+        clock.now += 10
+        await _round(a)
+        assert _running(a) == {"telegram", "discord", "github"}  # one missed renewal is tolerated
+
+        clock.now += 10  # the next tick would be at the local deadline (1000 + 30)
+        await _round(a)
+        assert _running(a) == {"github"}
+        # A peer may connect only lease + grace after the last confirmed renewal was sent:
+        # stopping at 1020 leaves grace_seconds + one interval of margin.
+        assert clock.now + a._leadership.renew_interval + a._leadership.grace_seconds <= 1000 + 30 + 10
+        # The row is not released: the database may be the one that is unreachable.
+        assert (await _lease(engine, "telegram")).owner_id == "A"
+    finally:
+        await a.stop()
+
+
+async def test_paths_on_a_non_holder_never_start_the_transport(engine, tmp_path):
+    a, b = _service(engine, "A", tmp_path), _service(engine, "B", tmp_path)
+    await a.start()
+    await b.start()
+    try:
+        assert await b.ensure_channel_ready("telegram") is False
+        assert await b.restart_channel("telegram", reload_config=False) is False
+        assert b.leader_of("telegram") == "A"
+        # Runtime config is accepted (the holder's transport serves the provider) but starts nothing here.
+        assert await b.configure_channel("telegram", {"enabled": True, "bot_token": "t-2"}) is True
+        assert b._config["telegram"]["bot_token"] == "t-2"
+        assert await b._start_channel("telegram", {"enabled": True}) is False  # the fence itself
+        assert "telegram" not in b._channels
+        assert len([t for t in FakeTransport.started if t.config.get("bot_token", "").startswith("t-")]) == 1
+    finally:
+        await a.stop()
+        await b.stop()
+
+
+async def test_a_transport_that_fails_to_stop_keeps_its_lease_on_shutdown(engine, tmp_path):
+    a, b = _service(engine, "A", tmp_path, channels={"telegram": {"enabled": True}}), _service(engine, "B", tmp_path, channels={"telegram": {"enabled": True}})
+    await a.start()
+    FakeTransport.stop_error = ChannelStopTimeout("telegram polling thread still alive")
+    with pytest.raises(ExceptionGroup):
+        await a.stop()
+    FakeTransport.stop_error = None
+    row = await _lease(engine, "telegram")
+    assert row.owner_id == "A" and row.lease_expires_at > 0  # left to expire: the old socket may still be open
+    await b.start()
+    try:
+        assert "telegram" not in b._channels
+    finally:
+        await b.stop()
+
+
+async def test_removing_a_provider_releases_its_lease_for_a_peer(engine, tmp_path):
+    a, b = _service(engine, "A", tmp_path), _service(engine, "B", tmp_path)
+    await a.start()
+    await b.start()
+    try:
+        assert await a.remove_channel("telegram") is True
+        assert (await _lease(engine, "telegram")).lease_expires_at == 0.0
+        await _round(b)
+        assert "telegram" in _running(b)
+    finally:
+        await a.stop()
+        await b.stop()
+
+
+async def test_a_provider_disabled_on_the_holder_is_stopped_and_released_by_supervision(engine, tmp_path):
+    a = _service(engine, "A", tmp_path)
+    await a.start()
+    try:
+        a._config["discord"] = {"enabled": False}
+        await _round(a)
+        assert "discord" not in a._channels
+        assert (await _lease(engine, "discord")).lease_expires_at == 0.0
+    finally:
+        await a.stop()
+
+
+async def test_notification_claims_are_left_to_the_holder(engine, tmp_path):
+    a, b = _service(engine, "A", tmp_path), _service(engine, "B", tmp_path)
+    await a.start()
+    await b.start()
+    try:
+        assert a.notification_excluded_providers() == frozenset()
+        assert b.notification_excluded_providers() == frozenset({"telegram", "discord"})
+    finally:
+        await a.stop()
+        await b.stop()
+
+
+async def test_a_provider_connected_on_a_peer_is_left_to_it_although_absent_from_local_config(engine, tmp_path):
+    """Review on #6843: a provider set up from the web UI on A never enters B's config.
+
+    B must still leave its notification rows to A (instead of claiming and
+    parking them) and report A as its leader, which it learns from the lease
+    table rather than from its own config.
+    """
+    a = _service(engine, "A", tmp_path, channels={"github": {"enabled": True}})
+    b = _service(engine, "B", tmp_path, channels={"github": {"enabled": True}})
+    await a.start()
+    await b.start()
+    try:
+        assert await a.configure_channel("telegram", {"enabled": True, "bot_token": "ui-token"}) is True
+        assert "telegram" in _running(a) and "telegram" not in b._config
+        await _round(a)  # A's renewal reports its transport running
+        assert "telegram" not in b.notification_excluded_providers()  # before B has seen the lease table
+
+        await _round(b)
+        assert b.notification_excluded_providers() == frozenset({"telegram"})
+        status = b.get_status()["channels"]["telegram"]
+        assert (status["leader"], status["leader_here"], status["running"]) == ("A", False, True)
+        assert "telegram" not in b._channels  # observing never starts anything
+
+        assert await a.remove_channel("telegram") is True  # released: no instance runs it now
+        await _round(b)
+        assert b.notification_excluded_providers() == frozenset()  # rows settle again (park / fail)
+        assert b.get_status()["channels"]["telegram"]["leader"] is None
+    finally:
+        await a.stop()
+        await b.stop()
+
+
+async def test_a_slow_start_never_delays_another_providers_renewal(engine, tmp_path):
+    a = _service(engine, "A", tmp_path, channels={"telegram": {"enabled": True}, "discord": {"enabled": True}})
+    await a.start()
+    try:
+        a._channels.pop("discord")  # the transport died; the next round restarts it
+        gate = asyncio.Event()
+        FakeTransport.start_gate = gate
+        await a._supervise_once()  # schedules the (blocked) discord start and returns
+        assert "discord" in a._lease_actions
+        before = (await _lease(engine, "telegram")).updated_at
+        await asyncio.sleep(0.01)
+        await a._supervise_once()  # telegram renews while discord's start is still pending
+        assert (await _lease(engine, "telegram")).updated_at > before
+        gate.set()
+        await asyncio.gather(*list(a._lease_actions.values()))
+        assert "discord" in _running(a)
+    finally:
+        FakeTransport.start_gate = None
+        await a.stop()
+
+
+async def test_a_holder_that_cannot_start_backs_off(engine, tmp_path):
+    clock = Clock()
+    a = _service(engine, "A", tmp_path, clock=clock, channels={"telegram": {"enabled": True}})
+    FakeTransport.fail_start = True
+    await a.start()
+    try:
+        assert "telegram" not in _running(a)
+        attempts = []
+        original = a.ensure_channel_ready
+
+        async def counting(name, *args, **kwargs):
+            attempts.append(name)
+            return await original(name, *args, **kwargs)
+
+        a.ensure_channel_ready = counting
+        await _round(a)
+        await _round(a)  # inside the backoff window: no new attempt
+        assert attempts == ["telegram"]
+        clock.now += a._leadership.renew_interval * 2
+        await _round(a)
+        assert attempts == ["telegram", "telegram"]
+        FakeTransport.fail_start = False
+        clock.now += a._leadership.renew_interval * 4
+        await _round(a)
+        assert "telegram" in _running(a)
+        assert "telegram" not in a._lease_start_backoff
+    finally:
+        await a.stop()
+
+
+async def test_without_the_lease_every_instance_behaves_as_before(tmp_path):
+    service = ChannelService(channels_config={name: dict(block) for name, block in CHANNELS.items()}, store=JsonChannelStore(tmp_path / "store.json"))
+
+    async def noop() -> None:
+        return None
+
+    service.manager.start = noop
+    service.manager.stop = noop
+    await service.start()
+    try:
+        assert _running(service) == {"telegram", "discord", "github"}
+        assert service.get_status()["channels"]["telegram"] == {"enabled": True, "running": True}
+        assert service.notification_excluded_providers() is None
+        assert service.lease_owner_id is None and service.leader_of("telegram") is None
+        assert service._supervisor_task is None
+    finally:
+        await service.stop()
+
+
+async def test_from_app_config_turns_the_lease_on_from_the_multi_process_signal(monkeypatch, tmp_path):
+    """``auto`` follows the Gateway's multi-process signal; ``false`` never builds the lease."""
+    from types import SimpleNamespace
+
+    import app.channels.service as service_module
+
+    built = []
+    monkeypatch.setattr(service_module, "_make_leadership", lambda config, *, owner_id: built.append((config.lease_seconds, owner_id)) or None)
+    monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path))
+    config = SimpleNamespace(model_extra={"channels": {"leader_lease": {"enabled": "auto", "lease_seconds": 12}}})
+
+    ChannelService.from_app_config(config, multi_process=False)
+    assert built == []
+    ChannelService.from_app_config(config, multi_process=True, lease_owner_id="host:worker")
+    assert built == [(12, "host:worker")]
+    config.model_extra["channels"]["leader_lease"]["enabled"] = False
+    ChannelService.from_app_config(config, multi_process=True)
+    assert built == [(12, "host:worker")]
+
+
+async def test_an_active_lease_without_a_database_refuses_to_build(monkeypatch):
+    import deerflow.persistence.engine as engine_module
+    from app.channels.service import _make_leadership
+    from deerflow.config.channel_leader_lease_config import ChannelLeaderLeaseConfig
+
+    monkeypatch.setattr(engine_module, "get_session_factory", lambda: None)
+    with pytest.raises(RuntimeError, match="channels.leader_lease is active"):
+        _make_leadership(ChannelLeaderLeaseConfig(enabled=True), owner_id=None)
+
+
+async def test_a_crashed_holder_is_replaced_within_one_lease_period(engine, tmp_path):
+    """End to end with the real supervision loop: the holder dies without releasing (kill -9)."""
+    a = _service(engine, "A", tmp_path, channels={"telegram": {"enabled": True}}, lease_seconds=3, grace_seconds=1, supervise=True)
+    b = _service(engine, "B", tmp_path, channels={"telegram": {"enabled": True}}, lease_seconds=3, grace_seconds=1, supervise=True)
+    await a.start()
+    await b.start()
+    try:
+        assert "telegram" in _running(a) and "telegram" not in b._channels
+        await asyncio.sleep(1.5)  # several renewals: B must not take a live lease
+        assert "telegram" not in b._channels
+
+        # Crash A: its loop stops renewing and nothing is released.
+        await a._stop_supervisor()
+        crashed_at = time.monotonic()
+        deadline = crashed_at + 3 + 1 + 3  # lease + grace + slack for B's tick and the SQLite round trips
+        while "telegram" not in _running(b) and time.monotonic() < deadline:
+            await asyncio.sleep(0.1)
+        assert "telegram" in _running(b)
+        async with engine.connect() as conn:
+            # The takeover respected the grace period on the database clock.
+            row = (await conn.execute(sa.select(ChannelLeaseRow.owner_id, ChannelLeaseRow.epoch, db_now("sqlite")).where(ChannelLeaseRow.provider == "telegram"))).one()
+        assert (row.owner_id, row.epoch) == ("B", 2)
+        assert time.monotonic() - crashed_at >= 1.0
+    finally:
+        a._running = False
+        a._channels.clear()
+        await b.stop()
+
+
+async def test_start_channel_service_threads_the_lease_settings(monkeypatch):
+    import app.channels.service as service_module
+
+    captured = {}
+
+    class _Started:
+        async def start(self):
+            return None
+
+    def fake_from_app_config(app_config=None, **kwargs):
+        captured.update(kwargs)
+        return _Started()
+
+    monkeypatch.setattr(service_module.ChannelService, "from_app_config", staticmethod(fake_from_app_config))
+    monkeypatch.setattr(service_module, "_channel_service", None)
+    await service_module.start_channel_service(multi_process=True, lease_owner_id="host:worker-1")
+    assert (captured["multi_process"], captured["lease_owner_id"]) == (True, "host:worker-1")
+
+
+async def test_gateway_lifespan_passes_the_gate_signal_and_the_run_worker_id():
+    """``auto`` follows the gate's multi-process signal; the lease owner is the run worker id."""
+    from contextlib import asynccontextmanager
+    from types import SimpleNamespace
+    from unittest.mock import AsyncMock, MagicMock, patch
+
+    from fastapi import FastAPI
+
+    from app.gateway.app import lifespan
+
+    @asynccontextmanager
+    async def runtime(app, _startup_config):
+        app.state.run_manager = SimpleNamespace(worker_id="host:worker-1")
+        yield
+
+    startup_config = MagicMock()
+    startup_config.log_level = "INFO"
+    startup_config.memory.enabled = False
+    startup_config.memory.shutdown_flush_timeout_seconds = 5.0
+    captured = {}
+
+    async def fake_start(_startup_config, **kwargs):
+        captured.update(kwargs)
+        service = MagicMock()
+        service.get_status.return_value = {}
+        return service
+
+    with (
+        patch("app.gateway.app.get_app_config", return_value=startup_config),
+        patch("app.gateway.app.get_gateway_config", return_value=MagicMock(host="x", port=0)),
+        patch("app.gateway.app.langgraph_runtime", runtime),
+        patch("app.gateway.app.multi_process_deployment", return_value=True) as signal,
+        # The real configure_logging would raise the root logger to INFO for every later test.
+        patch("app.gateway.app.configure_logging"),
+        patch("deerflow.skills.projection.ensure_public_skill_projection"),
+        patch("app.gateway.app.auth.close_oidc_service", AsyncMock()),
+        patch("app.channels.service.start_channel_service", side_effect=fake_start),
+        patch("app.channels.service.stop_channel_service", AsyncMock()),
+        patch("deerflow.agents.memory.get_memory_manager", return_value=MagicMock()),
+    ):
+        async with lifespan(FastAPI()):
+            pass
+
+    signal.assert_called_once_with(startup_config)
+    assert (captured["multi_process"], captured["lease_owner_id"]) == (True, "host:worker-1")
+
+
+def test_multi_process_deployment_is_the_gate_signal(monkeypatch):
+    from types import SimpleNamespace
+
+    from app.gateway.deps import multi_process_deployment
+
+    monkeypatch.delenv("DEER_FLOW_MULTI_INSTANCE", raising=False)
+    monkeypatch.delenv("WEB_CONCURRENCY", raising=False)
+    monkeypatch.setenv("GATEWAY_WORKERS", "1")
+    single = SimpleNamespace(deployment=SimpleNamespace(multi_instance=False))
+    assert multi_process_deployment(single) is False
+    assert multi_process_deployment(SimpleNamespace(deployment=SimpleNamespace(multi_instance=True))) is True
+    monkeypatch.setenv("GATEWAY_WORKERS", "2")
+    assert multi_process_deployment(single) is True

backend/tests/test_channels.py

@@ -6513,7 +6513,7 @@ class TestGithubFollowupBuffer:
             def get_status(self):
                 return {}
 
-        def fake_from_app_config(app_config=None, *, get_stream_bridge=None, credentials_cipher=None):
+        def fake_from_app_config(app_config=None, *, get_stream_bridge=None, credentials_cipher=None, **_lease):
             captured["get_stream_bridge"] = get_stream_bridge
             return _FakeService()
 

backend/tests/test_channels_router.py

@@ -84,3 +84,29 @@ def test_get_channels_status_remains_read_only(monkeypatch):
     assert response.status_code == 200
     assert response.json()["service_running"] is True
     assert response.json()["channels"]["slack"]["running"] is True
+
+
+def test_restart_on_a_non_holder_names_the_lease_holder(monkeypatch):
+    """channels.leader_lease: a restart that reaches an instance without the provider's lease says where it runs."""
+    service = SimpleNamespace(restart_channel=AsyncMock(return_value=False), leader_of=lambda name: "pod-b:worker", lease_owner_id="pod-a:worker")
+    monkeypatch.setattr("app.channels.service.get_channel_service", lambda: service)
+    app = make_authed_test_app(user_factory=_admin_user)
+    app.include_router(channels.router)
+
+    with TestClient(app) as client:
+        response = client.post("/api/channels/telegram/restart")
+
+    assert response.status_code == 200
+    assert response.json() == {"success": False, "message": "Channel telegram is connected by another Gateway instance (pod-b:worker); it can only be restarted there"}
+
+
+def test_a_failed_restart_on_the_holder_keeps_the_generic_message(monkeypatch):
+    service = SimpleNamespace(restart_channel=AsyncMock(return_value=False), leader_of=lambda name: "pod-a:worker", lease_owner_id="pod-a:worker")
+    monkeypatch.setattr("app.channels.service.get_channel_service", lambda: service)
+    app = make_authed_test_app(user_factory=_admin_user)
+    app.include_router(channels.router)
+
+    with TestClient(app) as client:
+        response = client.post("/api/channels/telegram/restart")
+
+    assert response.json() == {"success": False, "message": "Failed to restart channel telegram"}

backend/tests/test_migration_0027_0028_notification_outbox.py

@@ -23,7 +23,7 @@ OUTBOX = "0027_notification_deliveries"
 PARKED = "0028_parked_attempts"
 SCHEDULER = "0029_scheduler_agent_tasks"
 CLAIM_TOKENS = "0030_notification_claim_tokens"
-HEAD = "0041_im_notice_preferences"
+HEAD = "0042_channel_leases"
 PREVIOUS = "0026_mcp_task_lease_tokens"
 TABLE = "notification_deliveries"
 

backend/tests/test_migration_0029_scheduler_agent_tasks.py

@@ -23,7 +23,7 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
 from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
 
 REVISION = "0029_scheduler_agent_tasks"
-CURRENT_HEAD = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
 NEXT = "0030_notification_claim_tokens"
 PREVIOUS = "0028_parked_attempts"
 TASK_FIELDS = {"origin_thread_id", "goal_objective", "max_runs", "end_at", "standing_notes"}

backend/tests/test_migration_0032_activity_and_task_events.py

@@ -24,7 +24,7 @@ IDEMPOTENCY = "0036_run_idempotency_request"
 DOCUMENT_SUMMARIES = "0037_project_document_summaries"
 CHANNEL_BINDINGS = "0038_channel_thread_bindings"
 USER_DISABLED = "0039_user_disabled"
-CURRENT_HEAD = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
 PREVIOUS = "0031_scheduled_streak_boundary"
 TABLES = {"thread_read_markers", "thread_read_versions", "scheduled_task_events"}
 INDEXES = {

backend/tests/test_migration_0034_run_event_seq_watermark.py

@@ -24,7 +24,7 @@ IDEMPOTENCY = "0036_run_idempotency_request"
 DOCUMENT_SUMMARIES = "0037_project_document_summaries"
 CHANNEL_BINDINGS = "0038_channel_thread_bindings"
 USER_DISABLED = "0039_user_disabled"
-CURRENT_HEAD = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
 TABLE = "run_event_thread_seq"
 pytestmark = pytest.mark.asyncio
 

backend/tests/test_migration_0035_login_throttle.py

@@ -25,7 +25,7 @@ IDEMPOTENCY = "0036_run_idempotency_request"
 DOCUMENT_SUMMARIES = "0037_project_document_summaries"
 CHANNEL_BINDINGS = "0038_channel_thread_bindings"
 USER_DISABLED = "0039_user_disabled"
-CURRENT_HEAD = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
 PREVIOUS = "0034_run_event_seq_watermark"
 TABLE = "login_throttle"
 COLUMNS = {"ip", "fail_count", "locked_at", "lock_duration_seconds", "updated_at"}

backend/tests/test_migration_0037_project_document_summaries.py

@@ -21,7 +21,8 @@ PREVIOUS = "0036_run_idempotency_request"
 CHANNEL_BINDINGS = "0038_channel_thread_bindings"
 USER_DISABLED = "0039_user_disabled"
 GROUP_NOTICE = "0040_scheduled_group_notice"
-CURRENT_HEAD = "0041_im_notice_preferences"
+IM_NOTICE_PREFERENCES = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
 pytestmark = pytest.mark.asyncio
 
 
@@ -31,7 +32,8 @@ async def test_0037_chains_after_0036_and_is_single_head():
     assert script.get_revision(CHANNEL_BINDINGS).down_revision == REVISION
     assert script.get_revision(USER_DISABLED).down_revision == CHANNEL_BINDINGS
     assert script.get_revision(GROUP_NOTICE).down_revision == USER_DISABLED
-    assert script.get_revision(CURRENT_HEAD).down_revision == GROUP_NOTICE
+    assert script.get_revision(IM_NOTICE_PREFERENCES).down_revision == GROUP_NOTICE
+    assert script.get_revision(CURRENT_HEAD).down_revision == IM_NOTICE_PREFERENCES
     assert script.get_revision(REVISION).down_revision == PREVIOUS
     # alembic_version.version_num is VARCHAR(32).
     assert len(REVISION) <= 32

backend/tests/test_migration_0038_channel_thread_bindings.py

@@ -22,7 +22,7 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
 
 REVISION = "0038_channel_thread_bindings"
 PREVIOUS = "0037_project_document_summaries"
-CURRENT_HEAD = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
 TABLE = "channel_thread_bindings"
 COLUMNS = {"key", "channel_name", "chat_id", "topic_id", "thread_id", "user_id", "created_at", "updated_at"}
 INDEX = "ix_channel_thread_bindings_channel_chat"

backend/tests/test_migration_0040_scheduled_group_notice.py

@@ -16,12 +16,13 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
 
 REVISION = "0040_scheduled_group_notice"
 PREVIOUS = "0039_user_disabled"
+CURRENT_HEAD = "0042_channel_leases"
 COLUMN = "notification_group_connection_id"
 
 
 def test_group_notification_revision_has_one_head_after_0039():
     script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
-    assert script.get_heads() == ["0041_im_notice_preferences"]
+    assert script.get_heads() == [CURRENT_HEAD]
     assert script.get_revision("0041_im_notice_preferences").down_revision == REVISION
     assert script.get_revision(REVISION).down_revision == PREVIOUS
     assert len(REVISION) <= 32

backend/tests/test_migration_0042_channel_leases.py

@@ -0,0 +1,160 @@
+"""Migration 0042: the shared ``channel_leases`` table (per-provider IM transport leader lease)."""
+
+from __future__ import annotations
+
+import asyncio
+import os
+import uuid
+
+import pytest
+import sqlalchemy as sa
+from alembic import command
+from alembic.autogenerate import compare_metadata
+from alembic.migration import MigrationContext
+from alembic.script import ScriptDirectory
+from sqlalchemy.ext.asyncio import create_async_engine
+from support.postgres import asyncpg_test_url
+
+from deerflow.persistence import bootstrap
+from deerflow.persistence.base import Base
+from deerflow.persistence.channel_leases import ChannelLeaseRow
+from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
+
+REVISION = "0042_channel_leases"
+PREVIOUS = "0041_im_notice_preferences"
+CURRENT_HEAD = "0042_channel_leases"
+TABLE = "channel_leases"
+COLUMNS = {"provider", "owner_id", "epoch", "lease_expires_at", "transport_running", "updated_at"}
+pytestmark = pytest.mark.asyncio
+
+
+async def test_0042_remains_in_the_single_migration_chain():
+    script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
+    assert script.get_heads() == [CURRENT_HEAD]
+    assert script.get_revision(REVISION).down_revision == PREVIOUS
+    # alembic_version.version_num is VARCHAR(32).
+    assert len(REVISION) <= 32
+
+
+def _engine(tmp_path, backend):
+    if backend == "sqlite":
+        return create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'migration.db'}"), None
+    uri = os.environ.get("TEST_POSTGRES_URI")
+    if not uri:
+        pytest.skip("requires TEST_POSTGRES_URI (real Postgres migration 0042)")
+    schema = f"channel_leases_{uuid.uuid4().hex}"
+    return create_async_engine(asyncpg_test_url(uri), connect_args=build_asyncpg_connect_args(schema)), schema
+
+
+async def _shape(engine) -> dict:
+    async with engine.connect() as conn:
+
+        def read(sync):
+            inspector = sa.inspect(sync)
+            tables = set(inspector.get_table_names())
+            if TABLE not in tables:
+                return {"tables": tables}
+            return {
+                "tables": tables,
+                "columns": {col["name"]: col for col in inspector.get_columns(TABLE)},
+                "pk": inspector.get_pk_constraint(TABLE),
+            }
+
+        return await conn.run_sync(read)
+
+
+async def _orm_diff(engine) -> list:
+    """Alembic's own drift check, restricted to the table this revision owns."""
+
+    def only_leases(obj, name, type_, reflected, compare_to):
+        if type_ == "table":
+            return name == TABLE
+        return getattr(getattr(obj, "table", None), "name", TABLE) == TABLE
+
+    async with engine.connect() as conn:
+
+        def diff(sync):
+            context = MigrationContext.configure(sync, opts={"include_object": only_leases, "compare_type": True, "compare_server_default": True})
+            return compare_metadata(context, Base.metadata)
+
+        return await conn.run_sync(diff)
+
+
+async def _drop_schema(engine, schema):
+    if schema:
+        async with engine.begin() as conn:
+            await conn.execute(sa.text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
+
+
+@pytest.mark.parametrize("backend", ["sqlite", "postgres"])
+async def test_0042_creates_the_table_matching_the_orm_and_downgrades_cleanly(tmp_path, backend):
+    engine, schema = _engine(tmp_path, backend)
+    cfg = bootstrap._get_alembic_config(engine, postgres_schema=schema or "")
+    try:
+        if schema:
+            async with engine.begin() as conn:
+                await conn.execute(sa.text(f'CREATE SCHEMA "{schema}"'))
+        await asyncio.to_thread(command.upgrade, cfg, PREVIOUS)
+        assert TABLE not in (await _shape(engine))["tables"]
+
+        await asyncio.to_thread(command.upgrade, cfg, REVISION)
+        shape = await _shape(engine)
+        assert set(shape["columns"]) == COLUMNS
+        assert shape["pk"]["constrained_columns"] == ["provider"]
+        assert shape["pk"]["name"] in ("pk_channel_leases", None)  # SQLite does not report constraint names
+        assert all(column["nullable"] is False for column in shape["columns"].values())
+        # ``make migrate-rev`` would propose nothing: the revision matches the ORM model.
+        assert await _orm_diff(engine) == []
+
+        async with engine.begin() as conn:
+            await conn.execute(sa.insert(ChannelLeaseRow).values(provider="telegram", owner_id="host:worker", epoch=1, lease_expires_at=1_700_000_030.0, transport_running=True, updated_at=1_700_000_000.0))
+        with pytest.raises(sa.exc.IntegrityError):
+            async with engine.begin() as conn:
+                await conn.execute(sa.insert(ChannelLeaseRow).values(provider="telegram", owner_id="other", epoch=1, lease_expires_at=1.0, transport_running=False, updated_at=1.0))
+
+        await asyncio.to_thread(command.downgrade, cfg, PREVIOUS)
+        assert TABLE not in (await _shape(engine))["tables"]
+        await asyncio.to_thread(command.upgrade, cfg, REVISION)
+        assert TABLE in (await _shape(engine))["tables"]
+    finally:
+        await _drop_schema(engine, schema)
+        await engine.dispose()
+
+
+@pytest.mark.parametrize("backend", ["sqlite", "postgres"])
+async def test_0042_completes_after_create_all_and_downgrades_after_a_manual_drop(tmp_path, backend):
+    """The table already exists (an interrupted earlier attempt): upgrade still lands, and vice versa."""
+    engine, schema = _engine(tmp_path, backend)
+    cfg = bootstrap._get_alembic_config(engine, postgres_schema=schema or "")
+    try:
+        if schema:
+            async with engine.begin() as conn:
+                await conn.execute(sa.text(f'CREATE SCHEMA "{schema}"'))
+        await asyncio.to_thread(command.upgrade, cfg, PREVIOUS)
+        async with engine.begin() as conn:
+            await conn.run_sync(Base.metadata.create_all, tables=[ChannelLeaseRow.__table__])
+
+        await asyncio.to_thread(command.upgrade, cfg, REVISION)
+        async with engine.connect() as conn:
+            assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == REVISION
+
+        async with engine.begin() as conn:
+            await conn.execute(sa.text(f"DROP TABLE {TABLE}"))
+        await asyncio.to_thread(command.downgrade, cfg, PREVIOUS)
+        async with engine.connect() as conn:
+            assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == PREVIOUS
+    finally:
+        await _drop_schema(engine, schema)
+        await engine.dispose()
+
+
+async def test_bootstrap_provisions_the_table_on_an_empty_database(tmp_path):
+    engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'bootstrap.db'}")
+    try:
+        await bootstrap.bootstrap_schema(engine, backend="sqlite")
+        await bootstrap.bootstrap_schema(engine, backend="sqlite")
+        assert set((await _shape(engine))["columns"]) == COLUMNS
+        async with engine.connect() as conn:
+            assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == CURRENT_HEAD
+    finally:
+        await engine.dispose()

backend/tests/test_multi_worker_postgres_gate.py

@@ -988,8 +988,8 @@ def test_multi_worker_rejects_disabled_leader_lease_with_an_enabled_channel(monk
     assert "discord, telegram" in msg  # sorted, only the enabled ones
     assert "slack" not in msg
     assert "409" in msg
-    assert "Set GATEWAY_WORKERS=1" in msg
-    _assert_lease_is_not_offered_as_a_remedy(msg)
+    assert "set GATEWAY_WORKERS=1" in msg
+    _assert_lease_is_offered_as_the_remedy(msg)
 
 
 @pytest.mark.parametrize("leader_lease", [None, {}, {"enabled": "auto"}, {"enabled": True}, {"enabled": True, "lease_seconds": 15, "grace_seconds": 5}])
@@ -1023,22 +1023,21 @@ def test_declared_multi_instance_rejects_disabled_leader_lease_for_ui_connectabl
     assert "IM channels telegram are enabled" in msg
     assert "deployment.multi_instance=false and run a single Gateway instance" in msg
     assert "channel_connections.<provider>.enabled=false" in msg
-    _assert_lease_is_not_offered_as_a_remedy(msg)
+    _assert_lease_is_offered_as_the_remedy(msg)
 
 
-def _assert_lease_is_not_offered_as_a_remedy(msg: str) -> None:
-    """Transport arbitration ships separately: until then auto/true only bypass the refusal.
+def _assert_lease_is_offered_as_the_remedy(msg: str) -> None:
+    """The leader lease now arbitrates each provider's transport, so it is the first remedy.
 
-    The remedies the refusal may offer are the ones that work today -- a single
-    instance or disabled channels. Recommending ``auto``/``true`` would send an
-    operator straight back into the duplicate connections the clause exists to
-    prevent (review on #6746).
+    While the arbitration had not shipped, ``auto``/``true`` only bypassed the
+    refusal and the message had to say so (review on #6746). With
+    ``channel_leases`` in place the refusal leads with the setting that keeps
+    every instance running, then the single-instance and channel-disable fallbacks.
     """
-    assert "Set channels.leader_lease.enabled to auto" not in msg
-    assert "so one instance owns" not in msg
-    assert "reserved" in msg and "not a remedy" in msg
     remedies = msg.split("diverges). ", 1)[1]
-    assert remedies.startswith(("Set GATEWAY_WORKERS=1", "Set deployment.multi_instance=false", "Unset DEER_FLOW_MULTI_INSTANCE")), remedies
+    assert remedies.startswith("Set channels.leader_lease.enabled to auto (the default) or true so one instance holds each provider's connection"), remedies
+    assert "reserved" not in msg and "not a remedy" not in msg
+    assert "alternatively " in remedies and "or disable those channels" in remedies
 
 
 def test_disabled_channel_connections_master_switch_does_not_count_providers():
@@ -1051,6 +1050,39 @@ def test_single_instance_ignores_a_disabled_leader_lease():
     _enforce_postgres_for_multi_worker(_channels_cluster(leader_lease={"enabled": False}, channels={"telegram": {"enabled": True}}))
 
 
+def test_single_instance_explicit_leader_lease_needs_a_sql_database():
+    """``enabled: true`` turns the lease on without peers; its table needs sqlite or postgres."""
+    config = _channels_cluster(leader_lease={"enabled": True}, channels={"telegram": {"enabled": True}})
+    config.database = DatabaseConfig(backend="memory")
+    with pytest.raises(SystemExit) as exc_info:
+        _enforce_postgres_for_multi_worker(config)
+    msg = str(exc_info.value)
+    assert "channels.leader_lease.enabled=true" in msg
+    assert "database.backend is 'memory'" in msg
+    assert "IM channels telegram are enabled" in msg
+
+
+@pytest.mark.parametrize("backend", ["sqlite", "postgres"])
+def test_single_instance_explicit_leader_lease_runs_on_a_sql_database(backend):
+    config = _channels_cluster(leader_lease={"enabled": True}, channels={"telegram": {"enabled": True}})
+    config.database = DatabaseConfig(backend=backend)
+    _enforce_postgres_for_multi_worker(config)
+
+
+@pytest.mark.parametrize(
+    ("leader_lease", "channels"),
+    [
+        ({"enabled": "auto"}, {"telegram": {"enabled": True}}),  # auto stays off on a single instance
+        ({"enabled": True}, {"telegram": {"enabled": False}}),  # nothing for the lease to arbitrate
+        ({"enabled": True}, {"github": {"enabled": True}}),  # webhooks are never arbitrated
+    ],
+)
+def test_single_instance_memory_database_passes_when_the_lease_has_nothing_to_hold(leader_lease, channels):
+    config = _channels_cluster(leader_lease=leader_lease, channels=channels)
+    config.database = DatabaseConfig(backend="memory")
+    _enforce_postgres_for_multi_worker(config)
+
+
 @pytest.mark.parametrize("block", [{"enabled": "maybe"}, {"lease_seconds": 0}, {"grace_seconds": -1}, {"enable": False}, "auto"])
 def test_malformed_leader_lease_is_refused_on_every_topology(block):
     with pytest.raises(SystemExit) as exc_info:

backend/tests/test_notification_deliveries_repository.py

@@ -96,6 +96,18 @@ class TestNotificationDeliveryRepository:
         assert len(first) == 1
         assert second == []
 
+    @pytest.mark.anyio
+    async def test_claim_leaves_excluded_providers_pending_for_their_lease_holder(self, repo):
+        # Under the IM leader lease an instance that does not hold a provider's
+        # transport leaves its rows to the holder instead of claiming and parking them.
+        await repo.enqueue(**_enqueue_kwargs(provider="wecom"))
+        await repo.enqueue(**_enqueue_kwargs(provider="feishu", target="ou_owner"))
+
+        claimed = await repo.claim_due_deliveries(now=datetime.now(UTC), limit=10, exclude_providers=frozenset({"wecom"}))
+        assert [row["provider"] for row in claimed] == ["feishu"]
+        holder = await repo.claim_due_deliveries(now=datetime.now(UTC), limit=10)
+        assert [(row["provider"], row["status"]) for row in holder] == [("wecom", "sending")]
+
     @pytest.mark.anyio
     async def test_claim_respects_limit(self, repo):
         for index in range(3):

backend/tests/test_notification_delivery_worker.py

@@ -28,14 +28,18 @@ class FakeDeliveryRepo:
         self.terminal_failures = []
         self.resets = []
         self.events = []
+        self.exclusions = []
 
     async def reset_stale_sending_rows(self, *, now, timeout):
         self.resets.append((now, timeout))
         self.events.append("reset")
         return 0
 
-    async def claim_due_deliveries(self, *, now, limit):
-        claimed, self.rows = self.rows[:limit], self.rows[limit:]
+    async def claim_due_deliveries(self, *, now, limit, **kwargs):
+        excluded = kwargs.get("exclude_providers") or frozenset()
+        self.exclusions.append(kwargs.get("exclude_providers"))
+        claimed = [row for row in self.rows if row["provider"] not in excluded][:limit]
+        self.rows = [row for row in self.rows if row not in claimed]
         self.claims.append((now, limit))
         self.events.append("claim")
         return claimed
@@ -1022,3 +1026,34 @@ async def test_binding_check_reads_the_real_connection_repository_shape(tmp_path
         assert await worker._target_still_connected(_delivery_row()) is False
     finally:
         await close_engine()
+
+
+@pytest.mark.asyncio
+async def test_worker_leaves_rows_of_providers_another_instance_leads():
+    """``excluded_providers`` is read on every poll; ``None`` or an empty set claims everything."""
+    repo = FakeDeliveryRepo([_delivery_row(delivery_id="d-wecom"), _delivery_row(delivery_id="d-feishu", provider="feishu")])
+    channel = FakeChannel()
+    led_elsewhere = {"value": frozenset({"feishu"})}
+    worker = NotificationDeliveryWorker(delivery_repo=repo, resolve_channel=lambda provider: channel if provider == "wecom" else None, excluded_providers=lambda: led_elsewhere["value"])
+
+    await worker.run_once(now=datetime.now(UTC))
+    assert repo.exclusions == [frozenset({"feishu"})]
+    assert repo.sent == ["d-wecom"]
+    assert [row["id"] for row in repo.rows] == ["d-feishu"]  # left pending, not parked
+
+    led_elsewhere["value"] = frozenset()
+    await worker.run_once(now=datetime.now(UTC))
+    assert repo.exclusions[-1] is None  # no restriction: the plain claim
+    assert repo.rows == []
+
+
+@pytest.mark.asyncio
+async def test_worker_ignores_a_non_collection_exclusion_result():
+    """A test double or a misbehaving resolver must not narrow the claim to nothing."""
+    from unittest.mock import MagicMock
+
+    repo = FakeDeliveryRepo([_delivery_row()])
+    worker = NotificationDeliveryWorker(delivery_repo=repo, resolve_channel=lambda provider: FakeChannel(), excluded_providers=MagicMock())
+    await worker.run_once(now=datetime.now(UTC))
+    assert repo.exclusions == [None]
+    assert repo.sent == ["delivery-1"]

config.example.yaml

@@ -3048,20 +3048,20 @@ run_ownership:
 #   #
 #   # Leader lease: with more than one Gateway instance, only one instance may
 #   # hold a provider's platform connection (Telegram rejects a second poller
-#   # with 409, Discord answers every message twice). The lease will be a row
-#   # per provider in the shared database, held by one instance and renewed on
-#   # a heartbeat; a crashed holder is replaced after lease_seconds +
-#   # grace_seconds, a cleanly stopped one within a heartbeat. The transport
-#   # ownership itself ships separately: in this version the section only drives
-#   # the startup gate, so keep ONE instance while channels are enabled.
-#   # Restart-required.
+#   # with 409, Discord answers every message twice). Each provider gets a row
+#   # in the shared channel_leases table, held by one instance and renewed on a
+#   # heartbeat; only the holder connects (GitHub webhooks stay on every
+#   # instance). A crashed holder is replaced within lease_seconds +
+#   # grace_seconds plus one renewal interval, a cleanly stopped one on the
+#   # next heartbeat. Runtime credential changes from the web UI reach the
+#   # holder only when it next starts the channel. Restart-required.
 #   leader_lease:
 #     # auto (default): on whenever the Gateway sees a multi-process signal
 #     # (GATEWAY_WORKERS / WEB_CONCURRENCY > 1, deployment.multi_instance: true
 #     # or DEER_FLOW_MULTI_INSTANCE=1), off on a single instance. true: always
-#     # on. false: never -- a multi-process deployment with an IM channel other
-#     # than GitHub enabled then refuses to start. auto/true are reserved until
-#     # the lease ships; they do not yet stop every instance from connecting.
+#     # on (needs database.backend sqlite or postgres). false: never -- a
+#     # multi-process deployment with an IM channel other than GitHub enabled
+#     # then refuses to start.
 #     enabled: auto
 #     lease_seconds: 30   # validity of a held lease; renewed every lease_seconds // 3
 #     grace_seconds: 10   # extra time before a peer may take an expired lease

deploy/helm/deer-flow/README.md

@@ -300,16 +300,18 @@ kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002
   PodDisruptionBudget (`minAvailable: 1`) is rendered immediately and blocks
   every voluntary eviction and node drain while only one Pod exists, so set
   `gateway.podDisruptionBudget.enabled: false` for that interim if you need
-  to drain nodes first. `gateway.replicas: 1` stays the
-  default because the following are still single-instance: **IM channels**
-  (every Pod would connect to every platform — Telegram polling conflicts and
-  Discord double-processes; keep 1 replica while channels are enabled).
-  `channels.leader_lease` in `config` (default `auto`) is reserved for the
-  per-provider leader lease that will give one Pod each platform connection;
-  until that ships the setting only drives the startup gate, which refuses
-  `enabled: false` while channels are enabled, and `auto`/`true` do not yet
-  make several replicas safe. Also single-instance: the WeChat QR login and
-  browser tools. Before raising replicas, also set
+  to drain nodes first. **IM channels** run on several replicas through
+  `channels.leader_lease` in `config` (default `auto`, which turns on with the
+  multi-instance declaration): each platform connection is held by the one Pod
+  owning that provider's row in the `channel_leases` table, a crashed holder is
+  replaced within `lease_seconds + grace_seconds` plus one renewal interval
+  (50 s by default) and a Pod stopped cleanly hands its providers over on the
+  next heartbeat. The startup gate refuses `enabled: false` while channels are
+  enabled. Credentials saved from the web UI reach the holding Pod only when it
+  next starts the channel, so restart it after changing them, and a failover
+  cancels the IM replies in flight on the old holder. `gateway.replicas: 1`
+  stays the default because the WeChat QR login and browser tools are still
+  single-instance. Before raising replicas, also set
   `persistence.home.accessMode: ReadWriteMany` on multi-node clusters (thread
   uploads, outputs, memory and `extensions_config.json` live on that volume)
   and `agent_storage.backend: db` so custom agents are visible on every Pod.

deploy/helm/deer-flow/values.yaml

@@ -32,13 +32,12 @@ gateway:
   # Redis stream bridge (sandbox ownership leases are inferred from it). The
   # chart also exports DEER_FLOW_MULTI_INSTANCE=true when replicas > 1 so the
   # Gateway's startup gate enforces those prerequisites instead of staying
-  # inert (its worker count only sees one Pod). Still single-instance: IM
-  # channels (every Pod connects to every platform -- keep replicas at 1
-  # while channels are enabled), the WeChat QR login, and browser tools.
-  # `channels.leader_lease` in `config` (default `auto`) is reserved for the
-  # per-provider leader lease that will give one Pod each platform connection;
-  # until it ships the setting only drives the startup gate, which refuses
-  # `enabled: false` while channels are enabled (see the chart README).
+  # inert (its worker count only sees one Pod). IM channels follow
+  # `channels.leader_lease` in `config` (default `auto`, on with replicas > 1):
+  # one Pod holds each platform connection and another takes over when it
+  # dies; the startup gate refuses `enabled: false` while channels are enabled
+  # (see the chart README). Still single-instance: the WeChat QR login and
+  # browser tools.
   # Multi-node clusters also need `persistence.home.accessMode: ReadWriteMany`,
   # and `agent_storage.backend: db` keeps custom agents visible on every Pod.
   # Memory stays on that shared volume, but DeerMem's SQLite retrieval index is

frontend/src/content/en/application/deployment-guide.mdx

@@ -156,7 +156,7 @@ The Helm chart ships `gateway.replicas: 1`. Raising it is supported for the web
 
 The chart exports `DEER_FLOW_MULTI_INSTANCE=true` when `gateway.replicas > 1`, and the Gateway refuses to start if any of these is missing. If you scale with `kubectl scale` or an HPA instead of `gateway.replicas`, set `gateway.multiInstance: true` first so the declaration, the PodDisruptionBudget and the required shared `AUTH_JWT_SECRET` do not keep their single-replica rendering (the PodDisruptionBudget then blocks node drains of the single Pod until the scale-up happens; disable it with `gateway.podDisruptionBudget.enabled: false` for that interim if needed). On multi-node clusters also set `persistence.home.accessMode: ReadWriteMany` (uploads, outputs, memory and `extensions_config.json` live on that volume; MCP and skill changes made through one Pod are then visible to the others on their next request) and `agent_storage.backend: db` so custom agents are visible on every Pod.
 
-Still single-instance: IM channels (every Pod would connect to every platform), the WeChat QR login, and browser tools. Keep one replica while those are enabled. `channels.leader_lease` in `config` (default `auto`) is reserved for the per-provider leader lease that will give one Pod each platform connection; until it ships the setting only drives the startup gate, which refuses `enabled: false` while channels are enabled, and `auto`/`true` do not yet make several replicas safe.
+IM channels work on several replicas through `channels.leader_lease` in `config` (default `auto`): one Pod holds each platform connection, and when it dies another Pod takes over within `lease_seconds + grace_seconds` plus one renewal interval (50 seconds by default). The startup gate refuses `enabled: false` while channels are enabled. Channel credentials saved from the web UI reach the Pod holding that platform only when it next starts the channel, so restart that Pod after changing them; a takeover cancels the IM replies the old Pod was still generating. Still single-instance: the WeChat QR login and browser tools. Keep one replica while those are enabled.
 
 ### K8s Provisioner setup
 

frontend/src/content/zh/application/deployment-guide.mdx

@@ -153,7 +153,7 @@ Helm chart 默认 `gateway.replicas: 1`。当所有副本共享同一个 Postgre
 
 当 `gateway.replicas > 1` 时，chart 会导出 `DEER_FLOW_MULTI_INSTANCE=true`，缺少上述任一项时 Gateway 会拒绝启动。如果你通过 `kubectl scale` 或 HPA 而不是 `gateway.replicas` 扩容，请先设置 `gateway.multiInstance: true`：它们在不执行 Helm upgrade 的情况下改变副本数，否则该声明、PodDisruptionBudget 和必需的共享 `AUTH_JWT_SECRET` 都会停留在单副本的渲染结果上（在真正扩容之前，PodDisruptionBudget 会阻止驱逐唯一的 Pod 和排空节点；如有需要，可在此期间设置 `gateway.podDisruptionBudget.enabled: false`）。多节点集群还需要把 `persistence.home.accessMode` 设为 `ReadWriteMany`（上传、产物、记忆与 `extensions_config.json` 都在该卷上；在一个 Pod 上修改的 MCP 与技能配置会在其他 Pod 的下一次请求中生效），并设置 `agent_storage.backend: db` 让自定义 agent 在每个 Pod 上可见。
 
-仍然只能单实例运行的部分：IM 通道（每个 Pod 都会连接所有平台）、WeChat 扫码登录、浏览器工具。启用这些功能时请保持 1 个副本。`config` 中的 `channels.leader_lease`（默认 `auto`）是为按 provider 的 leader 租约预留的配置，落地后每个平台的连接只由一个 Pod 持有；在此之前它只驱动启动门禁（启用通道时拒绝 `enabled: false`），`auto`/`true` 目前并不能让多副本安全运行。
+IM 通道可以通过 `config` 中的 `channels.leader_lease`（默认 `auto`）在多副本下运行：每个平台的连接只由一个 Pod 持有，该 Pod 宕机后，另一个 Pod 会在 `lease_seconds + grace_seconds` 再加一个续约周期内接管（默认 50 秒）。启用通道时启动门禁会拒绝 `enabled: false`。在 Web UI 中保存的通道凭据要等持有该平台的 Pod 下次启动通道时才会生效，修改后请重启该 Pod；接管会取消旧 Pod 上仍在生成的 IM 回复。仍然只能单实例运行的部分：WeChat 扫码登录、浏览器工具。启用这些功能时请保持 1 个副本。
 
 ### K8s Provisioner 配置
 

scripts/AGENTS.md

@@ -176,8 +176,8 @@ Only "no config anywhere" creates `<checkout>/config.yaml` from the example.
 multi-instance gate and every endpoint/data root inside its state dir
 (`backend/tests/test_dev_multi_instance_script.py`).
 Unless `keep_channels` is requested, disable provider flags in both `channels`
-and `channel_connections`; UI credentials can otherwise start a duplicate
-transport later. Preserve `channels.leader_lease` and connection metadata.
+and `channel_connections`; UI credentials can otherwise connect a bot
+later. Preserve `channels.leader_lease` and connection metadata.
 
 ## Shell Script Invocation Contract
 

scripts/dev_multi_instance.py

@@ -170,7 +170,7 @@ def build_multi_instance_config(
             for name in disabled:
                 channels[name]["enabled"] = False
             if disabled:
-                notes.append(f"disabled IM channels under {section_name}: {', '.join(disabled)}: both Gateways would connect the same bot (set DEERFLOW_MI_KEEP_CHANNELS=1 to keep them)")
+                notes.append(f"disabled IM channels under {section_name}: {', '.join(disabled)}: the harness keeps your bots offline unless asked (set DEERFLOW_MI_KEEP_CHANNELS=1 to keep them; the leader lease then gives each bot to one Gateway)")
 
     memory = _section(config, "memory")
     manager_class = memory.get("manager_class")

scripts/dev_multi_instance.sh

@@ -80,7 +80,7 @@ Environment overrides:
   DEERFLOW_MI_FRONTEND_PORT, DEERFLOW_MI_POSTGRES_PORT, DEERFLOW_MI_REDIS_PORT
   DEERFLOW_MI_POSTGRES_CONTAINER, DEERFLOW_MI_REDIS_CONTAINER,
   DEERFLOW_MI_POSTGRES_IMAGE, DEERFLOW_MI_REDIS_IMAGE
-  DEERFLOW_MI_KEEP_CHANNELS=1 keep IM channels enabled (both Gateways then connect every bot)
+  DEERFLOW_MI_KEEP_CHANNELS=1 keep IM channels enabled (the leader lease gives each bot to one Gateway)
   DEERFLOW_MI_HEALTH_TIMEOUT  seconds to wait for each Gateway's /health (default 180)
 
 'make stop' and 'make dev' reclaim port 8001 and this checkout's Gateways, so