mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-18 09:40:44 +00:00
* fix(channels): share chat-to-thread bindings through the database ChannelManager kept its IM chat-to-thread bindings (channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file, channels/store.json, loaded once and rewritten whole on every change. With several Gateway processes a binding created on one replica was invisible to the others (the next message for the same chat opened a second thread) and concurrent writers clobbered each other's file. - New channel_thread_bindings table (migration 0036_channel_thread_bindings) and harness package deerflow.persistence.channel_thread_bindings: key PK (the legacy composite string), channel_name, chat_id, nullable topic_id, thread_id, user_id, epoch created_at/updated_at; index (channel_name, chat_id). Dialect upsert keeps created_at; prefix remove and batched ON CONFLICT DO NOTHING import. - app.channels.store.ChannelStore is an async protocol: SqlChannelStore for sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for database.backend: memory. resolve_channel_store picks at ChannelService construction; ChannelService.start() imports an existing store.json once into an empty table and renames it store.json.migrated (a concurrently importing peer neither duplicates nor clobbers; a populated table leaves the file alone). - Callers updated: manager, connection_identity.lookup_thread_id, Feishu (send() awaits the mapping writer; the synchronous lark callback bridges its two store calls to the Gateway loop with a bounded wait), Discord (channels_dir). - Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md, README) and CHANGELOG updated. * docs(changelog): link the channel thread bindings entry to #6558 * fix(channels): reject only the Gateway loop in the Feishu store bridge lark-oapi 1.5.5 invokes the message callback synchronously from its async frame handler, on the private event loop that FeishuChannel._run_ws starts for the SDK thread. The bridge's guard treated any running loop as unsafe, so every real inbound message skipped the stored-topic lookup and the pending-mapping restore: a reply whose parent card was mapped but whose root was not fell back to the unknown root, and could consume an unrelated pending clarification. Only a wait issued from the Gateway loop itself (the loop that executes the submitted coroutine) is refused now; the SDK's own loop may block for the bounded wait, as that thread blocked on the JSON rewrite before. Tests drive _on_message from a worker thread running its own loop, the production shape, for both the lookup and the write path, and pin the Gateway-loop refusal. * fix(channels): defer unresolved Feishu lookups and harden the store.json import Review follow-ups on the chat-to-thread binding store (#6558): - Import: a legacy entry whose key, channel_name, chat_id, topic_id, thread_id or user_id exceeds its column width is skipped and counted with the malformed ones instead of failing the whole batch on PostgreSQL (SQLite silently accepted the same row, so the backends diverged on one file). The corrupt-file warning now says what each caller does: the JSON store starts fresh, the import is skipped and the file kept. - Feishu bridge: store calls run on the Gateway loop under a 3s asyncio.wait_for (the SDK loop also services ws keepalives; the old 10s bound could stall it per message), and the lark-thread wait is that budget plus 1s. A deadline or store error is StoreLookupUnresolved, never a miss: _on_message no longer routes such a reply onto its unknown root (which created a second thread and later overwrote the parent alias). The message joins a per-chat FIFO drained on the Gateway loop with backoff for up to 30s, then is dropped with a warning. _on_message is split into parse / resolve / dispatch so the deferred path dispatches the same way; later messages for the chat queue behind the deferred one. * fix(tests): expect follow_redirects in the Jina client kwargs after #5808 #5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and updated test_jina_client.py, but test_jina_response_limit.py and test_jina_retry_after.py still pin the exact client kwargs to proxy and trust_env, so every backend-unit-tests shard that lands on them fails on main (the backend unit-test workflow only runs on pull requests, which is why the push to main did not surface it). Update the two expectations to include the new keyword. (cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050) * Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808" This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7. * fix(channels): bound deferred Feishu routing by the inbound queue capacity Review follow-up on #6558: a store outage let every subsequent message for a chat join the deferred queue with no capacity check (the bus reservation was only taken after routing succeeded), and each queued message started a fresh 30s retry budget when it reached the head, so a burst could be retained for N x 30s and grow without bound. - Intake is reserved BEFORE routing for every store-backed message (bus.reserve_inbound is thread-safe and already used from SDK threads), so deferred and live messages share the one configured bound, channels.inbound_queue_maxsize; a message that cannot be admitted is dropped at once with a warning, never queued or routed as a miss. - The reservation travels with the message into dispatch (committed by _prepare_inbound), is released on the batch path (the flush reserves at publish time), on drop after the budget, on a failed deferral, on dispatch failure and for every leftover when the drain task is cancelled by stop(). - The retry budget runs from the message's arrival: a queued message whose budget is spent when it reaches the head is dropped without a lookup. - The bounded resolver returns the unresolved outcome instead of raising it through the tracked task, which logged an ERROR for an expected, handled store timeout. - Upstream's new blocking-io test constructs the store by its old class name; it now uses JsonChannelStore. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
140 lines
6.4 KiB
Python
140 lines
6.4 KiB
Python
"""SQL repository for ``channel_thread_bindings`` (SQLite or PostgreSQL).
|
|
|
|
Concurrency contract, shared with ``app.channels.store.SqlChannelStore``:
|
|
|
|
- ``upsert`` is one dialect-native ``INSERT ... ON CONFLICT (key) DO UPDATE``
|
|
that replaces ``thread_id`` / ``user_id`` / ``updated_at`` and keeps the
|
|
existing ``created_at`` — the overwrite semantics the JSON store had — so two
|
|
replicas recording the same conversation never produce two rows and never
|
|
lose the first stamp.
|
|
- ``insert_missing`` is the one-time import primitive: ``INSERT ... ON CONFLICT
|
|
DO NOTHING`` in bounded batches, returning only the rows it actually inserted,
|
|
so two replicas importing the same ``store.json`` concurrently can neither
|
|
duplicate nor clobber a binding that already exists (including one a peer
|
|
wrote live between the emptiness check and the insert).
|
|
- ``delete_prefix`` removes ``key == prefix`` and every ``key`` starting with
|
|
``prefix + ":"`` (``startswith(autoescape=True)``, so ``%`` / ``_`` in a chat
|
|
id match literally) — exact parity with the JSON ``remove()`` that cleared a
|
|
chat's base mapping and all its topics by key prefix, independent of how a
|
|
legacy key was split into components.
|
|
|
|
Database errors propagate: the manager cannot route an IM message without the
|
|
database anyway, and failing loudly beats silently splitting a conversation
|
|
across two threads.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Iterable
|
|
|
|
from sqlalchemy import delete, func, or_, select
|
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
|
|
|
from deerflow.persistence.channel_thread_bindings.model import ChannelThreadBinding, ChannelThreadBindingRow
|
|
|
|
#: Rows per ``INSERT`` statement during the import; keeps bind-parameter counts small on both dialects.
|
|
IMPORT_BATCH_SIZE = 500
|
|
|
|
_COLUMNS = (
|
|
ChannelThreadBindingRow.key,
|
|
ChannelThreadBindingRow.channel_name,
|
|
ChannelThreadBindingRow.chat_id,
|
|
ChannelThreadBindingRow.topic_id,
|
|
ChannelThreadBindingRow.thread_id,
|
|
ChannelThreadBindingRow.user_id,
|
|
ChannelThreadBindingRow.created_at,
|
|
ChannelThreadBindingRow.updated_at,
|
|
)
|
|
|
|
|
|
def _insert_for(session: AsyncSession):
|
|
dialect = session.get_bind().dialect.name
|
|
if dialect == "postgresql":
|
|
return pg_insert
|
|
if dialect == "sqlite":
|
|
return sqlite_insert
|
|
raise ValueError(f"Unsupported channel thread binding database dialect: {dialect}")
|
|
|
|
|
|
def _values(binding: ChannelThreadBinding) -> dict[str, object]:
|
|
return {
|
|
"key": binding.key,
|
|
"channel_name": binding.channel_name,
|
|
"chat_id": binding.chat_id,
|
|
"topic_id": binding.topic_id,
|
|
"thread_id": binding.thread_id,
|
|
"user_id": binding.user_id,
|
|
"created_at": binding.created_at,
|
|
"updated_at": binding.updated_at,
|
|
}
|
|
|
|
|
|
def _binding(row) -> ChannelThreadBinding:
|
|
key, channel_name, chat_id, topic_id, thread_id, user_id, created_at, updated_at = row
|
|
return ChannelThreadBinding(key, channel_name, chat_id, topic_id, thread_id, user_id or "", float(created_at), float(updated_at))
|
|
|
|
|
|
class SqlChannelThreadBindingRepository:
|
|
"""Persistence facade for ``channel_thread_bindings``."""
|
|
|
|
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
|
self._sf = session_factory
|
|
|
|
async def get_thread_id(self, key: str) -> str | None:
|
|
async with self._sf() as session:
|
|
return (await session.execute(select(ChannelThreadBindingRow.thread_id).where(ChannelThreadBindingRow.key == key))).scalar_one_or_none()
|
|
|
|
async def upsert(self, binding: ChannelThreadBinding) -> None:
|
|
"""Create or overwrite the binding; an existing row keeps its ``created_at``."""
|
|
async with self._sf() as session:
|
|
stmt = _insert_for(session)(ChannelThreadBindingRow).values(**_values(binding))
|
|
stmt = stmt.on_conflict_do_update(
|
|
index_elements=[ChannelThreadBindingRow.key],
|
|
set_={"thread_id": stmt.excluded.thread_id, "user_id": stmt.excluded.user_id, "updated_at": stmt.excluded.updated_at},
|
|
)
|
|
await session.execute(stmt)
|
|
await session.commit()
|
|
|
|
async def delete(self, key: str) -> bool:
|
|
async with self._sf() as session:
|
|
result = await session.execute(delete(ChannelThreadBindingRow).where(ChannelThreadBindingRow.key == key))
|
|
await session.commit()
|
|
return result.rowcount > 0
|
|
|
|
async def delete_prefix(self, prefix: str) -> int:
|
|
"""Delete ``prefix`` itself and every ``prefix:*`` key; returns the number of rows removed."""
|
|
async with self._sf() as session:
|
|
predicate = or_(ChannelThreadBindingRow.key == prefix, ChannelThreadBindingRow.key.startswith(prefix + ":", autoescape=True))
|
|
result = await session.execute(delete(ChannelThreadBindingRow).where(predicate))
|
|
await session.commit()
|
|
return int(result.rowcount or 0)
|
|
|
|
async def list(self, channel_name: str | None = None) -> list[ChannelThreadBinding]:
|
|
stmt = select(*_COLUMNS).order_by(ChannelThreadBindingRow.created_at, ChannelThreadBindingRow.key)
|
|
if channel_name:
|
|
stmt = stmt.where(ChannelThreadBindingRow.channel_name == channel_name)
|
|
async with self._sf() as session:
|
|
rows = (await session.execute(stmt)).all()
|
|
return [_binding(row) for row in rows]
|
|
|
|
async def count(self) -> int:
|
|
async with self._sf() as session:
|
|
return int((await session.execute(select(func.count()).select_from(ChannelThreadBindingRow))).scalar_one())
|
|
|
|
async def insert_missing(self, bindings: Iterable[ChannelThreadBinding]) -> int:
|
|
"""``INSERT ... ON CONFLICT DO NOTHING`` in batches; returns how many rows this call inserted."""
|
|
rows = [_values(binding) for binding in bindings]
|
|
if not rows:
|
|
return 0
|
|
inserted = 0
|
|
async with self._sf() as session:
|
|
insert = _insert_for(session)
|
|
for start in range(0, len(rows), IMPORT_BATCH_SIZE):
|
|
batch = rows[start : start + IMPORT_BATCH_SIZE]
|
|
result = await session.execute(insert(ChannelThreadBindingRow).values(batch).on_conflict_do_nothing(index_elements=[ChannelThreadBindingRow.key]))
|
|
inserted += int(result.rowcount or 0)
|
|
await session.commit()
|
|
return inserted
|