diff --git a/README.md b/README.md index 0539310d5..d13e2ee50 100644 --- a/README.md +++ b/README.md @@ -427,7 +427,7 @@ cleanup when migrating from legacy metadata credentials. DeerFlow supports receiving tasks from messaging apps. Channels auto-start when configured — no public IP required for any of them. -DeerFlow can also expose user-owned IM channel connections in the workspace UI. When `channel_connections` is enabled, logged-in users can bind Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, or WeCom from the sidebar / Settings > Channels. It reuses the existing outbound `channels.*` transports, so no public IP or provider callback URL is required. Incoming IM messages then run under the connected DeerFlow user account. See [IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md) for setup and security notes. +DeerFlow can also expose user-owned IM channel connections in the workspace UI. When `channel_connections` is enabled, logged-in users can bind Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, WeCom, or Buzz from the sidebar / Settings > Channels. It reuses the existing outbound `channels.*` transports, so no public IP or provider callback URL is required. Incoming IM messages then run under the connected DeerFlow user account. See [IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md) for setup and security notes. | Channel | Transport | Difficulty | |---------|-----------|------------| @@ -437,6 +437,7 @@ DeerFlow can also expose user-owned IM channel connections in the workspace UI. | WeChat | Tencent iLink (long-polling) | Moderate | | WeCom | WebSocket | Moderate | | DingTalk | Stream Push (WebSocket) | Moderate | +| Buzz | Nostr relay (WebSocket, NIP-42) | Moderate | **Configuration in `config.yaml`:** diff --git a/backend/AGENTS.md b/backend/AGENTS.md index badc625e5..1b7c996da 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -910,9 +910,11 @@ Bridges external messaging platforms (Feishu, Slack, Telegram, Discord, DingTalk - `store.py` - JSON-file persistence mapping `channel_name:chat_id[:topic_id]` → `thread_id` (keys are `channel:chat` for root conversations and `channel:chat:topic` for threaded conversations) - `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands including `/goal` (setting a goal persists it through Gateway and then routes the objective as a chat turn), keeps Slack/Discord on `client.runs.wait()`, uses `client.runs.stream(["messages-tuple", "values"])` for Feishu/Telegram incremental outbound updates, serializes same-thread Feishu turns in-manager when the channel's `ChannelRunPolicy.serialize_thread_runs=True` so rapid follow-ups queue instead of tripping the runtime busy reply, and switches to `client.runs.create()` (fire-and-forget, returns once the run is `pending`) for channels whose `ChannelRunPolicy.fire_and_forget=True` so long autonomous runs do not hit the SDK default 300s `httpx.ReadTimeout` A swallowed streaming failure publishes its final outbound before releasing the inbound dedupe key, so a provider redelivery can retry without overtaking the terminal reply. + **What may be published from the stream is an allowlist, not a denylist** (`_accumulate_stream_text`): only assistant message types — LangChain serializes `AIMessage.type` as `"ai"` and `AIMessageChunk.type` as `"AIMessageChunk"`, plus the OpenAI-style `"assistant"` spelling for foreign runtimes — become displayable text. The previous rule rejected only payloads whose `type` contained `"tool"` and therefore published everything else, which leaked DeerFlow's hidden model context to every streaming IM channel: `DynamicContextMiddleware` injects recalled memory as a hidden `HumanMessage` (`type == "human"`) and rewrites the user's own turn into a new `HumanMessage`, `DurableContextMiddleware` injects a hidden `` `HumanMessage`, and LangGraph fans those state writes out on the `messages-tuple` stream. Proved live on a Buzz relay, which published a `` fact block and, in another run, a verbatim echo of the user's own message as the assistant's reply. Matching is by prefix (`ai` / `assistant`), never substring, because ordinary words contain `"ai"` (`chain`, `domain`). The message type is resolved by `_stream_payload_type`, which handles both the `model_dump()` shape DeerFlow's own gateway emits and LangChain's `to_json()` constructor shape (whose top-level `type` is the literal `"constructor"`, with the class name at the tail of the `id` path). A bare `str` payload is no longer accepted at all: it carries no type information, so it cannot be attributed to the assistant, and nothing in DeerFlow produces one (`runtime/serialization.py::serialize_messages_tuple` always emits `[message_dict, metadata]`). - `base.py` - Abstract `Channel` base class (start/stop/send lifecycle) - `service.py` - Manages lifecycle of all configured channels from `config.yaml` - `slack.py` / `feishu.py` / `telegram.py` / `discord.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `telegram.py` accepts inbound text/photos/documents, preserves media captions, hands token-free attachment bytes to the shared upload pipeline, edits the "Working on it..." stream target in place via `editMessageText`, and can optionally send final Markdown replies as Rich Messages through `channels.telegram.rich_messages`; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured, and overrides `receive_file` to download inbound images (`picture`/`richText`) and documents (`file`) by `downloadCode` into the thread uploads bucket, mirroring `feishu.py`) +- `buzz.py` - Buzz (Nostr relay) implementation: one NIP-42-authenticated websocket, pubkey-allowlist + mention/DM/thread-follow gating, streaming replies via in-place kind-40003 edits; requires the `buzz` dependency extra. **Subscription model** (operator-facing version in [IM_CHANNEL_CONNECTIONS.md](docs/IM_CHANNEL_CONNECTIONS.md#buzz-subscription-model)): the relay fans kind-9 chat events out **only** to channel-scoped subscriptions, proved against a live relay — `REQ {"kinds":[9]}` is accepted and answered with `EOSE` but never receives an event (the connector authenticated and then sat silent forever), `REQ {"kinds":[9],"#h":[uuid]}` works, and a multi-value `#h` matches nothing, so it is strictly one REQ per channel (the same shape as Buzz's own `buzz-acp` harness). Every connection therefore rebuilds three kinds of subscription after NIP-42 auth: `buzz-discovery` (`{"kinds":[39000]}`, a historical query returning exactly the channels this identity belongs to, one stored event each then `EOSE` — adding `#p` returns zero, do not "narrow" it), `buzz-membership` (`{"kinds":[44100,44101],"#p":[us],"since":}`, the relay-signed member-added/member-removed notifications whose `p` tag names the affected member and `h` tag the channel), and one `buzz-chat-` per discovered channel. Chat subscriptions open as each kind-39000 arrives; the discovery `EOSE` is the completeness barrier that retries any that failed and warns when discovery found nothing. A kind-44100 for our pubkey subscribes to the new channel **live** (then re-issues discovery so its name/type reach the DM-detection cache, but only when that channel's metadata is actually missing — an unconditional refresh is how a burst of 44100s multiplied into one discovery pass each); a kind-44101 issues `buzz_nostr.close_frame` for exactly that channel's subscription and drops its metadata. **The membership subscription is scoped to LIVE events** and this is load-bearing: buzz-relay *stores* 44100/44101 and serves history newest-first (default limit 2000), so an unscoped filter replayed the whole membership history on every connect — every stored add read as live, re-running discovery once each (M+1 discovery passes × N stored kind-39000 events per connect, observed live as two `channel discovery complete` lines and one channel logged `` because the 44100 path subscribed before its metadata arrived), re-subscribing channels we have since been removed from, and letting a stored 44101 transiently unsubscribe a channel we are still in. `since` is anchored at the moment the socket opened (`_session_started_at`) minus `MEMBERSHIP_LOOKBACK_SECONDS` (60) of slack, which covers both relay clock skew and a membership change published *during* the connect/auth handshake; the slack can only cost an idempotent replay of the last minute. **A relay `CLOSED` frame is recovered, not merely forgotten** — every subscription on the socket fails silently when dropped, so `_handle_closed` re-issues it, bounded by `MAX_RESUBSCRIBE_ATTEMPTS` (3) per subscription id per connection *and per auth epoch* (`_resubscribe_attempts` is reset on session start, on re-auth, and by `stop()`, so pre-auth rejections — which the auth branch already recovers wholesale — never spend the authenticated session's budget). The first retry is immediate (the common case is a one-off hiccup); later ones back off 1s then 2s, awaited inline in the read loop rather than in a background task that could outlive its own socket. **`auth-required:` before this socket has completed its NIP-42 handshake is the expected bootstrap sequence, not a refusal**, and `_handle_closed` short-circuits it ahead of every recovery path: the connector opens its control REQs immediately in case the relay serves unauthenticated reads, a closed relay answers `auth-required:` plus an `AUTH` challenge, and the auth branch re-opens everything. That case is logged at DEBUG and consumes neither the permanent-refusal branch nor the retry budget (a chat subscription is still dropped from `_chat_subscriptions`, since it genuinely is not subscribed and discovery is what re-opens it). Treating it as permanent produced an operator-facing warning claiming discovery/membership tracking was DOWN in the same run where discovery then completed, every channel was subscribed, and a brand-new channel's kind-44100 was picked up one second later. The boundary is the per-socket `_auth_completed` flag, set once the signed AUTH event has been sent and cleared on session entry, session exit, and `stop()`; the same reason *after* that stays loud, and says the subscription is down until the relay's next AUTH challenge or the next reconnect rather than borrowing the non-auth wording. Then `_is_transient_close` decides whether to retry at all: NIP-01/NIP-42 `auth-required:`/`restricted:`/`blocked:`/`mute:`/`invalid:`/`pow:` prefixes and buzz-relay's own removal/revocation prose are permanent (do not fight the relay over a channel that is no longer ours; a post-auth `auth-required:` is recovered by the AUTH branch, not by re-issuing), `rate-limited:`/`error:`/no-reason-at-all and anything unrecognized are transient — the default resolves toward "keep listening" because going silently deaf is the failure this exists to remove, and the attempt budget bounds a wrong guess. A chat `CLOSED` is only ever recovered for a channel already in `_chat_subscriptions`: a `CLOSED` is relay-supplied, so acting on an unknown one would let a relay induce a subscription just by naming a channel. Every subscription that goes unlistened is logged at WARNING, never INFO. Subscription ids are deterministic per channel precisely so one can be replaced or closed without disturbing the others on the socket, and `_chat_subscriptions` is per-socket state cleared on session end, on re-auth (a pre-auth REQ may have been rejected), and by `stop()`. Three bounds on remote-fed state: `MAX_CACHED_CHANNELS` (512) caps the kind-39000 metadata cache, the watermark map, and the resubscribe-attempt map; `MAX_CHANNEL_SUBSCRIPTIONS` (256, well under buzz-relay's own 1024-per-connection ceiling) caps live chat subscriptions — at the cap new channels are refused and named in a warning rather than evicting a working subscription. **Known bound (documented, not fixed):** the relay caps historical delivery at 2000 events per subscription, newest-first, even with a `since`, so >2000 unread messages in a *single* channel across a disconnect loses the oldest — the relay never sends them and the watermark advances past them. That is the one remaining path that can skip; everything else fails toward replay. **Trust model** (operator-facing version in [IM_CHANNEL_CONNECTIONS.md](docs/IM_CHANNEL_CONNECTIONS.md#buzz-trust-model)): every inbound `EVENT` is authenticated at the single `handle_relay_frame` choke point — the NIP-01 id is recomputed from the delivered payload and the BIP-340 Schnorr signature verified against the claimed `pubkey` (`buzz_nostr.verify_event`, pure and total: malformed input returns `False`, never raises) — so `ev["pubkey"]`, the authorization principal for both the allowlist and the `/connect` bind, cannot be forged by a relay the DeerFlow operator does not run. What remains trusted is the *authorship* of kind-39000 channel metadata: any member can sign one, and because per-channel subscriptions are now driven by discovery, a forged kind-39000 has two effects rather than one — it can mark a channel `type: "dm"` (relaxing `require_mention` for that channel) **and** it can induce a chat subscription for a channel of the forger's choosing, since the channels we listen to are exactly the channels we hold metadata for. Neither makes anything be *acted on*: `allowed_users` and per-event signature verification are independent gates, so an induced subscription only means the relay reads its own traffic back to a subscriber that drops it, bounded by `MAX_CHANNEL_SUBSCRIPTIONS` (which refuses rather than evicts, so it cannot displace a real channel). Same for a forged kind-44100, except its `p` tag is re-checked locally so it must at least name us. Closing this needs a configured trusted relay pubkey, which `relay_url` is not. `allowed_users` is deny-by-default (empty = nobody, unlike siblings' empty = everyone), so `start()` logs a WARNING when it is empty and each drop logs at DEBUG. The resubscribe cursor (`since`) is **per channel**, advances only for events that were actually processed, and never past `now + MAX_FUTURE_SKEW_SECONDS`, because it is peer-supplied (`created_at`) and a single future-dated event otherwise made the connector permanently deaf. Per channel rather than global is the safety-critical half: subscriptions are per channel, so one shared cursor is the newest event seen in *any* channel and a busy channel would drag it past a quiet channel's unread messages, skipping them on the next reconnect — measured on a live relay, three channels of one identity sat ~28h apart. Per-channel cursors can only ever cost duplicate delivery (absorbed by the manager's `event_id` dedupe), and an evicted cursor degrades to "no `since`", i.e. the relay's default backlog — both fail toward replay, never toward a miss. Streaming tracks every oversize chunk index (`_stream_targets` for chunk 0, `_stream_tails` for the rest), since the manager republishes cumulative text and reposting `chunks[1:]` per update flooded the channel; all of it is per-connection state cleared by `stop()`, and the remote-fed kind-39000 cache is capped at `MAX_CACHED_CHANNELS`. **`send()` refuses outright to publish text carrying a hidden model-context wrapper** (``, ``, `` — `_HIDDEN_CONTEXT_MARKERS`), logging at ERROR and clearing the stream bookkeeping on a blocked `is_final`. This is defense in depth behind the manager's allowlist, and it lives here rather than in a sibling connector because on Buzz a leak is permanent: every streaming update is an immutable public Nostr event, so a corrective edit only changes what clients render while the original leaked event stays on the relay. Matching is on the literal opening tag, so a reply that merely talks about memory is still published - `github.py` - Webhook-driven GitHub channel. Inbound messages come from `POST /api/webhooks/github`; outbound is log-only because GitHub agents post explicitly with `gh` from their sandbox when they choose to comment or create a PR - `app/gateway/routers/channel_connections.py` - Browser-facing user connection and disconnect APIs - `deerflow.persistence.channel_connections` - SQL-backed user-owned connection, optional credential, connect state, and conversation store @@ -944,7 +946,7 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_ - `langgraph_url` - LangGraph-compatible Gateway API base URL (default: `http://localhost:8001/api`) - `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`) - In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://gateway:8001/api` for `langgraph_url` and `http://gateway:8001` for `gateway_url`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`. -- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token, optional `rich_messages` for final Markdown Rich Messages), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming), `github` (operator kill-switch `enabled`, plus `default_mention_login` for mention-required GitHub triggers) +- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token, optional `rich_messages` for final Markdown Rich Messages), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming), `github` (operator kill-switch `enabled`, plus `default_mention_login` for mention-required GitHub triggers), `buzz` (relay_url, private_key) **User-owned channel connections** (`config.yaml` -> `channel_connections`): - Disabled by default. It is a user-binding layer on top of the existing `channels.*` runtime config, not a replacement for provider bot credentials. diff --git a/backend/app/channels/buzz.py b/backend/app/channels/buzz.py new file mode 100644 index 000000000..212e12fbb --- /dev/null +++ b/backend/app/channels/buzz.py @@ -0,0 +1,1413 @@ +"""Buzz (Nostr) channel: DeerFlow as a member of a Buzz workspace relay. + +One NIP-42-authenticated WebSocket to ``relay_url``. Inbound kind-9 chat events are +gated (pubkey allowlist, then mention/DM/thread-follow) and published to the bus; +outbound replies post one kind-9 message and then stream via kind-40003 in-place edits. + +Subscriptions are CHANNEL-SCOPED, which is the shape the relay actually serves: + +- ``REQ {"kinds":[9]}`` (global) is accepted and answered with EOSE, but the relay + never fans a chat event out to it -- a connector subscribed that way authenticates + successfully and then receives nothing, forever. Proved against a live relay. +- ``REQ {"kinds":[9], "#h":[uuid]}`` works, and a multi-value ``#h`` does NOT, so + there is exactly one chat subscription per channel (matching Buzz's own agent + harness, whose ``subscribe_channel_from`` is likewise per channel). + +So each connection: authenticate, discover the channels this identity belongs to +with a historical ``kinds:[39000]`` REQ, open one ``#h`` chat subscription per +discovered channel, and keep a live ``kinds:[44100,44101] #p=`` subscription so +channels we are added to (or removed from) later are picked up without a reconnect. + +Every one of those subscriptions can be killed by a single relay ``CLOSED`` frame, +and each one dying is a *silent* outage (a dead chat subscription deafens one +channel; a dead ``buzz-membership`` stops us ever learning we were added to or +removed from a channel; a dead ``buzz-discovery`` kills the completeness sweep). +So a ``CLOSED`` is recovered rather than merely forgotten -- bounded by +``MAX_RESUBSCRIBE_ATTEMPTS`` per subscription per connection, and only when the +relay's stated reason suggests re-issuing the same REQ could work at all (see +``_is_transient_close``). Every subscription that goes unlistened is named at +WARNING, because "listening to nothing" must never again be indistinguishable from +"nothing is being said". + +KNOWN BOUND (documented, not fixed): the relay caps historical delivery at 2000 +events per subscription and serves them NEWEST-FIRST, even with a ``since``. So a +channel that accumulated more than 2000 unread messages across one disconnect +loses the oldest of them: the relay never sends them, and the watermark advances +past them as the newer ones are processed. This is the one bounded skip path that +remains, it needs a disconnect plus >2000 messages in a single channel to trigger, +and closing it would require paging the backlog with descending ``until`` queries. +""" + +from __future__ import annotations + +import asyncio +import json +import logging +import random +import time +from typing import Any +from urllib.parse import urlparse + +from app.channels import buzz_nostr +from app.channels.base import Channel +from app.channels.commands import is_known_channel_command +from app.channels.connection_identity import attach_connection_identity +from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage + +logger = logging.getLogger(__name__) + +# Headroom under the relay's 64KB edit-content cap (kind-40003 events). +EDIT_MAX_BYTES = 60_000 + +# Ceiling for how far ahead of our own clock a peer-supplied ``created_at`` may be +# and still advance the resubscribe watermark (see ``_advance_watermark``). +MAX_FUTURE_SKEW_SECONDS = 60 + +# Cap on the kind-39000 metadata cache. Any relay member can publish channel +# metadata, so this map is remote-fed and would otherwise grow without bound for +# the process lifetime; evicting the oldest entry only costs us the DM mention +# exemption for that channel until its metadata is seen again (``_is_dm`` fails +# closed on a cache miss). +MAX_CACHED_CHANNELS = 512 + +# Cap on how many per-channel chat subscriptions one connection may hold. The +# channel list is remote-fed (kind-39000 metadata and kind-44100 membership +# notifications both come off the wire), and each entry is a real REQ on the socket, +# so this is bounded twice over: well under buzz-relay's own per-connection ceiling +# (``MAX_SUBSCRIPTIONS = 1024`` in its REQ handler) and far above any plausible +# workspace. At the cap we REFUSE new subscriptions and log, rather than evicting an +# existing one: eviction would silently deafen a channel that is currently working, +# whereas refusing leaves every established channel intact and names the one that +# did not fit. +MAX_CHANNEL_SUBSCRIPTIONS = 256 + +# Subscription ids. Deterministic so a subscription can be replaced or closed +# individually (``CLOSE``) without disturbing the others on the same socket. +DISCOVERY_SUB_ID = "buzz-discovery" +MEMBERSHIP_SUB_ID = "buzz-membership" +CHAT_SUB_PREFIX = "buzz-chat-" + +# How many times ONE subscription may be re-opened after the relay ``CLOSED`` it, +# per connection and per auth epoch. Re-issuing a closed REQ immediately and +# unconditionally is a tight loop against a relay that keeps closing it (each +# CLOSED provokes a REQ which provokes a CLOSED), so the budget is what makes +# recovery safe rather than the backoff: at the cap we stop, say so loudly, and +# leave it to the next reconnect -- which rebuilds every subscription anyway. +MAX_RESUBSCRIBE_ATTEMPTS = 3 + +# Backoff between re-subscribe attempts. The FIRST retry is immediate: a one-off +# relay hiccup is the common case and should be repaired without a stall. Later +# retries back off (1s, then 2s), which is what a persistently-closing relay gets. +# The delay is awaited inline in the relay read loop rather than handed to a +# background task -- 3s of worst-case delayed frame processing per subscription +# per connection is cheaper than a task that can outlive its own socket and fire a +# REQ into a connection that no longer exists. +RESUBSCRIBE_BASE_DELAY_SECONDS = 1.0 + +# How far BEFORE the socket opened the live membership subscription starts. +# +# Without any ``since`` the relay replays its whole stored 44100/44101 history on +# every connection: each historical "you were added" reads as live (re-subscribing +# channels we have since been removed from and re-running discovery once per +# event), and each historical "you were removed" transiently unsubscribes a channel +# we are still in. Anchoring ``since`` at connection time fixes that, but a bare +# ``now`` would open a hole: the relay stamps ``created_at`` with ITS clock, and a +# membership change published during our connect/auth handshake is genuinely live +# yet already in the past by the time the REQ goes out. This slack covers both -- +# it can only ever cost the replay of the last minute of membership changes, which +# is idempotent (``_ensure_chat_subscription`` no-ops on a channel already +# subscribed), whereas being one second short costs a channel we never hear from. +MEMBERSHIP_LOOKBACK_SECONDS = 60 + +# Bound on how long ``stop()`` waits for the cancelled relay loop to finish. +STOP_TIMEOUT_SECONDS = 5.0 + +# NIP-42's machine-readable ``CLOSED`` prefix for "authenticate first". +# +# BEFORE this connection has completed its NIP-42 handshake this is not a refusal +# at all -- it is the expected bootstrap sequence. ``_session`` deliberately opens +# the control subscriptions immediately, in case the relay serves unauthenticated +# reads; a closed relay answers ``auth-required:`` and an ``AUTH`` challenge, and +# the auth branch then re-opens every subscription. Treating that as a permanent +# refusal produced an operator-facing warning claiming discovery/membership +# tracking was DOWN at the exact moment it was coming up -- observed live, in the +# same run where discovery then completed, every channel was subscribed, and a +# brand-new channel's kind-44100 was picked up one second later. +# +# AFTER the handshake the same reason means the relay stopped accepting our +# authenticated session, which is a genuine outage and stays loud. +# +# "Completed its NIP-42 handshake" means the relay has ACKNOWLEDGED our AUTH +# event (an ``OK ... true`` for it, or -- if the relay never sends one -- +# ``_confirm_auth_if_pending``'s fallback), never merely that we SENT it. A live +# relay was observed still processing an AUTH we had already sent when it closed +# the just-reopened control subscriptions with this same reason, so a flag set at +# send-time misclassified that as a post-auth refusal instead of the bootstrap +# race it still was. +_AUTH_REQUIRED_CLOSE_PREFIX = "auth-required:" + +# NIP-01 asks relays to give ``CLOSED`` a machine-readable prefix. These are the +# ones where re-issuing the IDENTICAL REQ cannot succeed, so retrying is just +# fighting the relay: +# auth-required: -- the relay wants a NIP-42 AUTH, and it sends an ``AUTH`` +# challenge to say so; ``_session``'s auth branch re-opens every +# subscription once that completes, which is the real recovery. +# restricted: / blocked: / mute: -- this identity is not permitted to read this. +# invalid: / pow: -- the filter itself was rejected; the same filter will be again. +_PERMANENT_CLOSE_PREFIXES = (_AUTH_REQUIRED_CLOSE_PREFIX, "restricted:", "blocked:", "mute:", "invalid:", "pow:") + +# buzz-relay does not always use a NIP-01 prefix -- it closes a channel's +# subscription with prose when access is revoked (e.g. the channel was archived, or +# we were removed from it) -- and NIP-01's generic categories (``error:``, +# ``rate-limited:``) leave the actual condition in the human-readable remainder, so +# ``error: channel not found`` is a permanent close wearing a retryable category. +# The prose is therefore matched regardless of prefix. Matching is deliberately +# narrow and everything unmatched DEFAULTS TO TRANSIENT: the failure this whole +# change exists to remove is going silently deaf, so an unrecognised reason resolves +# toward "keep listening", and ``MAX_RESUBSCRIBE_ATTEMPTS`` bounds the cost of +# guessing wrong in that direction. +_PERMANENT_CLOSE_MARKERS = ("revoke", "not a member", "not a channel member", "no longer a member", "access denied", "unauthorized", "forbidden", "archiv", "not found", "does not exist") + +# Wording mirrors every sibling adapter's `/connect` reply style (discord.py's +# `_send_connection_reply`, slack.py's `_post_connection_reply`, and the same +# templates in wecom.py / feishu.py / dingtalk.py / wechat.py), substituting +# "Buzz" for the platform name. +_CONNECT_REPLY_TEXT = { + "success": "Buzz connected to DeerFlow.", + "invalid": "Buzz connection code is invalid or expired.", + "error": "Buzz connection could not be completed from this message.", +} + + +def _chunk_text(text: str, limit: int = EDIT_MAX_BYTES) -> list[str]: + """Split *text* into chunks whose UTF-8 ENCODED byte length never exceeds *limit*. + + Operates character-by-character (not on raw encoded bytes), so a multi-byte + UTF-8 character is always appended to a chunk whole -- it is measured before + being added and, if it would push the running byte total over *limit*, the + chunk is flushed first and the character starts the next one. This makes a + split-mid-character corruption structurally impossible, regardless of + whether *limit* happens to be a multiple of any character's byte width. + """ + chunks, current, size = [], [], 0 + for ch in text: + b = len(ch.encode()) + if size + b > limit and current: + chunks.append("".join(current)) + current, size = [], 0 + current.append(ch) + size += b + if current: + chunks.append("".join(current)) + return chunks or [""] + + +def _is_auth_required_close(reason: str) -> bool: + """Does the relay's ``CLOSED`` reason say "authenticate first" (NIP-42)?""" + return (reason or "").strip().lower().startswith(_AUTH_REQUIRED_CLOSE_PREFIX) + + +# DeerFlow's hidden model-context wrappers. These are literal tags this codebase +# injects into the model's input and never into an assistant reply: +# ``DynamicContextMiddleware`` wraps recalled memory in ```` and the date +# reminder in ````; ``DurableContextMiddleware`` wraps the +# conversation summary, delegation ledger and active skills in +# ````. Matching the opening tag (not the words) is what +# keeps an ordinary reply that merely talks about memory publishable. +_HIDDEN_CONTEXT_MARKERS = ("", "", "") + + +def _hidden_context_marker(text: str) -> str | None: + """Return the hidden-context wrapper *text* carries, if any.""" + lowered = (text or "").lower() + return next((marker for marker in _HIDDEN_CONTEXT_MARKERS if marker in lowered), None) + + +def _is_transient_close(reason: str) -> bool: + """Could re-issuing the identical REQ plausibly succeed? + + This is the whole distinction between "recover from a relay hiccup" and "fight + the relay over a channel we are no longer in". It reads only the relay's stated + reason, so it is exactly as good as what the relay chose to say -- which is why + the two directions are weighted differently: + + - A recognised *permanent* reason (a NIP-01/NIP-42 category prefix, or + revocation/removal prose anywhere in the message) is believed and NOT retried. + Re-subscribing to a channel we have been thrown out of would achieve nothing + and look like an attack from the relay's side. + - Anything else -- a generic ``rate-limited:``/``error:`` category, or a + ``CLOSED`` with no reason at all, which NIP-01 permits -- is treated as + transient and retried. The connector's worst failure mode is going silently + deaf, so an unknown reason resolves toward "keep listening"; + ``MAX_RESUBSCRIBE_ATTEMPTS`` is what stops that from becoming a loop when the + guess is wrong. + """ + text = (reason or "").strip().lower() + if text.startswith(_PERMANENT_CLOSE_PREFIXES): + return False + return not any(marker in text for marker in _PERMANENT_CLOSE_MARKERS) + + +class BuzzChannel(Channel): + _connect: Any = None # test seam: async callable returning an async-context-manager transport + + def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None: + super().__init__(name="buzz", bus=bus, config=config) + self._relay_url = str(config.get("relay_url", "")).strip() + if not self._relay_url.startswith(("ws://", "wss://")): + raise ValueError("channels.buzz.relay_url must be a ws:// or wss:// URL") + # One community per relay URL (see the design's multi-community note), so the + # relay host is this channel's workspace: it scopes inbound dedupe, the + # persisted connection row written by `/connect`, and the lookup that resolves + # that row back on the inbound path. Computed once here so those three uses + # can never drift apart. + self._workspace_id = urlparse(self._relay_url).netloc + self._private_key_raw = str(config.get("private_key", "")) + self._keys: buzz_nostr.NostrKeys | None = None # parsed in start() so coincurve stays lazy + self._allowed_users = {buzz_nostr.parse_pubkey(v) for v in config.get("allowed_users", []) or []} + self._require_mention = bool(config.get("require_mention", True)) + self._mention_free = {str(c) for c in config.get("mention_free_channels", []) or []} + self._channel_meta: dict[str, dict[str, Any]] = {} + self._stream_targets: dict[tuple[str, str | None], str] = {} + self._stream_tails: dict[tuple[str, str | None], list[str]] = {} # overflow chunk ids beyond chunk 0, per conversation + self._last_requester: dict[tuple[str, str | None], str] = {} + self._pending_auth_challenge: str | None = None # set from an AUTH relay frame; consumed by the NIP-42 flow in _session + self._seen_created_at: dict[str, int] = {} # channel id -> high-water mark of PROCESSED created_at (see _advance_watermark) + self._chat_subscriptions: set[str] = set() # channel ids with a live per-channel REQ on the CURRENT connection + self._resubscribe_attempts: dict[str, int] = {} # sub id -> CLOSED-recovery attempts spent on the current connection/auth epoch + self._auth_completed = False # has THIS socket's NIP-42 handshake been ACKNOWLEDGED by the relay? (see _handle_auth_ok, _AUTH_REQUIRED_CLOSE_PREFIX) + self._pending_auth_event_id: str | None = None # id of the AUTH event most recently sent, awaiting a matching OK (see _handle_auth_ok / _confirm_auth_if_pending) + self._session_started_at: int | None = None # wall clock at which the CURRENT socket opened; anchors the live membership filter + self._transport: Any = None + self._task: asyncio.Task | None = None + self._publish = self.bus.publish_inbound # test seam (discord.py idiom) + + @property + def supports_streaming(self) -> bool: + return True + + # -- lifecycle --------------------------------------------------------- + + async def start(self) -> None: + if self._running: + return + self._keys = buzz_nostr.parse_private_key(self._private_key_raw) + if not self._allowed_users: + # Deny-by-default is deliberate (unlike siblings, where an empty + # allowlist means "allow all"), so this cannot be a hard failure -- + # but a configured-and-connected channel that silently drops every + # message looks exactly like a broken relay to the operator. Say so + # once, loudly, at the only point where it is actionable. + logger.warning("[buzz] channels.buzz.allowed_users is empty: EVERY inbound chat message will be dropped (Buzz denies by default). Add member pubkeys (hex or npub) to enable the channel.") + self.bus.subscribe_outbound(self._on_outbound) + self._spawn_connection() + self._running = True + logger.info("[buzz] channel started (relay=%s pubkey=%s allowed_users=%d)", self._relay_url, self._keys.pubkey_hex, len(self._allowed_users)) + + def _spawn_connection(self) -> None: + self._task = asyncio.create_task(self._run_loop(), name="buzz-relay-loop") + + async def stop(self) -> None: + """Cancel the relay loop and drop every piece of per-connection state. + + Teardown is *bounded* and *coherent*, in that order: + + - Bounded: the cancelled task is awaited via ``asyncio.wait`` rather than + ``asyncio.wait_for``. ``wait_for`` guarantees the awaited task is + finished before it raises ``TimeoutError``, so a task that swallows + ``CancelledError`` (a bug, but the exact case a timeout exists for) + would hang ``stop()`` forever instead of the intended 5s. + ``asyncio.wait`` returns after the timeout regardless and cancels + nothing further, so this always returns. + - Coherent: the reviewer's finding was that a timed-out ``_task`` was + dropped while it might still own ``_transport``, leaving an abandoned + task posting on a socket the channel believed it no longer held. The + ``finally`` now clears ``_transport`` (so ``_post_event`` fails fast + rather than writing through a socket nobody owns) alongside ``_task``, + and the timeout is logged instead of passing silently. + + Per-connection bookkeeping (stream placeholders, overflow tails, last + requester, half-consumed auth challenge, remote-fed channel metadata, + per-channel chat subscriptions) is also cleared: those maps key on relay + event ids and subscription ids from the session being torn down, so a + later ``start()`` must not resume editing placeholders -- or believe it is + still subscribed to anything -- from a previous process lifetime. The + per-channel replay cursors (``_seen_created_at``) deliberately survive, so + a restart resumes where it left off instead of replaying every channel. + """ + if not self._running: + return + self._running = False + self.bus.unsubscribe_outbound(self._on_outbound) + if self._task is not None: + task = self._task + task.cancel() + try: + _, pending = await asyncio.wait({task}, timeout=STOP_TIMEOUT_SECONDS) + if pending: + logger.warning("[buzz] relay loop did not finish within %ss of cancellation; abandoning it", STOP_TIMEOUT_SECONDS) + elif not task.cancelled() and task.exception() is not None: + # _run_loop is designed to never end on an ordinary connection error (it + # backs off and retries instead -- see its docstring), but stop() must + # still complete cleanly rather than re-raise whatever a genuine bug in + # the relay loop task ended with. + logger.error("[buzz] relay loop task ended with an error during stop: %s", task.exception()) + except asyncio.CancelledError: + raise + except Exception: + logger.exception("[buzz] failed to await the relay loop task during stop") + finally: + self._task = None + self._transport = None + self._stream_targets.clear() + self._stream_tails.clear() + self._last_requester.clear() + self._channel_meta.clear() + self._chat_subscriptions.clear() + self._resubscribe_attempts.clear() + self._auth_completed = False + self._pending_auth_event_id = None + self._session_started_at = None + self._pending_auth_challenge = None + logger.info("[buzz] channel stopped") + + # -- subscriptions ------------------------------------------------------ + + @staticmethod + def _chat_sub_id(channel_id: str) -> str: + return f"{CHAT_SUB_PREFIX}{channel_id}" + + def _chat_filter(self, channel_id: str) -> dict: + """The NIP-01 filter for ONE channel's chat subscription. + + ``#h`` is not an optimization -- it is the only shape the relay fans kind-9 + events out to (see the module docstring), and it may carry exactly one value. + + ``since`` rides along only once we have a watermark for THIS channel + (``_seen_created_at``, advanced by ``_advance_watermark``): a channel we + have not processed anything in yet gets no ``since``, so the relay's default + backlog applies, and afterwards it is the newest ``created_at`` we have + already processed in that channel -- so re-subscribing after a drop neither + replays the whole channel nor silently skips what was published while we + were disconnected. + + "Processed" means accepted-and-published (or a fully handled ``/connect``), + never merely received. That is strictly safer for the skip direction: the + cursor can only ever be lower than it would have been under an + advance-on-everything rule, so any event that rule would have covered is + still covered. What it costs is replay -- events we already dropped come + back after a reconnect and are dropped again by the same gates, which is + idempotent -- plus one guaranteed redelivery of the boundary event itself, + since NIP-01 ``since`` is inclusive; the manager's inbound dedupe (keyed on + our ``event_id`` metadata within this workspace) is what keeps that from + starting a second run. + + KNOWN BOUND: ``since`` narrows the query but does not lift the relay's cap on + how much history one subscription may be served -- buzz-relay answers with at + most 2000 stored events, NEWEST-FIRST. So if a single channel accumulated more + than 2000 messages while we were disconnected, the relay simply never sends + the oldest of them, and the watermark advances past them as the newer ones are + processed. That is the one bounded skip path that remains after this fix + (everything else fails toward replay); closing it would mean paging the + backlog with descending ``until`` queries rather than one REQ per channel. + """ + chat_filter: dict = {"kinds": [buzz_nostr.KIND_CHAT], "#h": [channel_id]} + since = self._seen_created_at.get(channel_id) + if since: + chat_filter["since"] = since + return chat_filter + + def _discovery_filter(self) -> dict: + """Historical kind-39000 query: which channels is this identity a member of? + + Deliberately unfiltered beyond the kind. The relay scopes kind-39000 by + membership itself, returning one stored event per channel we belong to + followed by EOSE -- adding ``#p`` (an obvious-looking narrowing) matches + nothing and returns zero channels, verified against a live relay. + """ + return {"kinds": [buzz_nostr.KIND_CHANNEL_META]} + + def _membership_filter(self) -> dict: + """Live membership notifications addressed to us: added to / removed from a channel. + + ``since`` is not an optimization, it is the difference between "live" and + "the entire history of this identity's membership, replayed as news". + buzz-relay STORES 44100/44101 events and serves history newest-first with a + default limit of 2000, so an unscoped filter meant every connection: + + - logged "added to channel ...; subscribing" once per historical add, each + re-issuing the discovery REQ, so one connect produced M+1 discovery passes + over N stored kind-39000 events (the two "channel discovery complete" + lines seen live, and the channel that logged as ```` because the + 44100 path subscribed before its metadata had arrived); + - re-subscribed every channel we were EVER added to, including ones we have + since been removed from, burning subscription-cap slots; and + - let a historical 44101 unsubscribe (and ``_channel_meta.pop``) a channel we + are still in -- recovered only incidentally, because newest-first ordering + happens to deliver the oldest add last. + + The window this must not lose is a membership change published DURING the + connect/auth handshake: it is genuinely live, but the relay stamps it with + its own clock and it is already in the past by the time the post-auth REQ + goes out. ``MEMBERSHIP_LOOKBACK_SECONDS`` of slack behind the moment the + socket opened covers both that window and relay/DeerFlow clock skew. Cost of + the slack: at most the last minute of membership changes replayed, which is + idempotent (``_ensure_chat_subscription`` no-ops on an already-subscribed + channel, and ``_handle_membership_event`` only re-runs discovery when the + channel's metadata is actually missing). Cost of not having it: a channel we + were added to during the handshake is never heard from until a reconnect. + """ + assert self._keys is not None + started_at = self._session_started_at if self._session_started_at is not None else int(time.time()) + since = max(0, started_at - MEMBERSHIP_LOOKBACK_SECONDS) + return {"kinds": [buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED], "#p": [self._keys.pubkey_hex], "since": since} + + async def _open_control_subscriptions(self, ws) -> None: + """Channel discovery plus membership notifications: the two per-connection subscriptions.""" + await ws.send(buzz_nostr.req_frame(DISCOVERY_SUB_ID, self._discovery_filter())) + await ws.send(buzz_nostr.req_frame(MEMBERSHIP_SUB_ID, self._membership_filter())) + + async def _ensure_chat_subscription(self, channel_id: str) -> None: + """Open this channel's chat subscription unless it is already open on this connection. + + Never raises: it runs from the relay read loop, where a failed REQ must not + take down the connection. A failure simply leaves the channel unsubscribed + and out of ``_chat_subscriptions``, so the discovery EOSE sweep (or the next + kind-39000 for it, or a reconnect) retries. + """ + if not channel_id or channel_id in self._chat_subscriptions: + return + transport = self._transport + if transport is None: + return # no connection: subscriptions are per-socket and are rebuilt by _session + if len(self._chat_subscriptions) >= MAX_CHANNEL_SUBSCRIPTIONS: + logger.warning("[buzz] per-channel subscription limit reached (%d); not listening to channel %s", MAX_CHANNEL_SUBSCRIPTIONS, channel_id) + return + try: + await transport.send(buzz_nostr.req_frame(self._chat_sub_id(channel_id), self._chat_filter(channel_id))) + except Exception: + logger.warning("[buzz] failed to subscribe to channel %s; will retry", channel_id, exc_info=True) + return + self._chat_subscriptions.add(channel_id) + logger.info("[buzz] listening to channel %s (%s)", self._channel_meta.get(channel_id, {}).get("name") or "", channel_id) + + async def _close_chat_subscription(self, channel_id: str) -> None: + """Stop listening to one channel, leaving every other subscription on this socket alone.""" + if channel_id not in self._chat_subscriptions: + return + self._chat_subscriptions.discard(channel_id) + transport = self._transport + if transport is None: + return + try: + await transport.send(buzz_nostr.close_frame(self._chat_sub_id(channel_id))) + except Exception: + logger.warning("[buzz] failed to unsubscribe from channel %s", channel_id, exc_info=True) + + def _claim_resubscribe_attempt(self, sub_id: str) -> int | None: + """Spend one retry from *sub_id*'s budget for this connection; ``None`` once spent. + + The counter deliberately does NOT reset when a re-subscribe succeeds. A relay + that accepts the REQ and then closes it again is the exact loop the budget + exists to bound, and "it worked for a moment" is not evidence that it is + working. The whole map is reset when the socket (or the auth epoch) changes, + which is the only event that genuinely makes the past irrelevant. + + Keys are our own subscription ids and only ever added for subscriptions we + actually held, but a long-lived connection can cycle through many channels, + so the map is FIFO-capped like the other per-channel maps. Eviction only ever + hands a channel a fresh retry budget. + """ + used = self._resubscribe_attempts.get(sub_id, 0) + if used >= MAX_RESUBSCRIBE_ATTEMPTS: + return None + self._resubscribe_attempts[sub_id] = used + 1 + while len(self._resubscribe_attempts) > MAX_CACHED_CHANNELS: + self._resubscribe_attempts.pop(next(iter(self._resubscribe_attempts))) + return used + 1 + + async def _resubscribe_backoff(self, attempt: int) -> None: + """Pause before re-issuing a closed REQ; the first attempt is immediate.""" + if attempt <= 1: + return + await asyncio.sleep(RESUBSCRIBE_BASE_DELAY_SECONDS * 2 ** (attempt - 2)) + + async def _handle_closed(self, sub_id: str, reason: str) -> None: + """Route a relay ``CLOSED`` frame to the recovery its subscription needs. + + A ``CLOSED`` is the relay dropping a subscription, and EVERY subscription on + this socket fails silently when that happens: a chat subscription deafens one + channel, ``buzz-membership`` stops us learning about channels we are added to + or removed from, ``buzz-discovery`` kills the completeness sweep and the + "no channels" warning. Forgetting the subscription is therefore only half a + response -- something has to re-open it, or the connector runs the rest of + that socket's life quietly missing traffic it believes it is receiving. + + The one close that is NOT an outage is ``auth-required:`` before this + socket has completed its NIP-42 handshake: that is the bootstrap sequence + working as designed (opportunistic pre-auth REQs, refused by a closed + relay, re-opened wholesale by ``_session``'s auth branch). It is handled + here, ahead of every recovery path, so it consumes neither the + permanent-refusal branch nor the retry budget and says nothing alarming -- + the previous behaviour warned that discovery/membership tracking was down + at the exact moment it was coming up. A chat subscription is still dropped + from the live set, because it genuinely is not subscribed from this moment + and the auth branch/discovery sweep is what re-opens it. + """ + if _is_auth_required_close(reason) and not self._auth_completed: + if sub_id.startswith(CHAT_SUB_PREFIX): + self._chat_subscriptions.discard(sub_id[len(CHAT_SUB_PREFIX) :]) + logger.debug("[buzz] relay closed %s pending NIP-42 auth (%s); the auth handshake re-opens it", sub_id, reason or "") + return + if sub_id == DISCOVERY_SUB_ID or sub_id == MEMBERSHIP_SUB_ID: + await self._recover_control_subscription(sub_id, reason) + elif sub_id.startswith(CHAT_SUB_PREFIX): + await self._recover_chat_subscription(sub_id[len(CHAT_SUB_PREFIX) :], reason) + else: + logger.info("[buzz] relay closed unrecognized subscription %s: %s", sub_id, reason or "") + + async def _recover_control_subscription(self, sub_id: str, reason: str) -> None: + """Re-issue a closed ``buzz-discovery`` / ``buzz-membership`` REQ, bounded. + + These two are opened ONLY by ``_open_control_subscriptions`` -- at session + start and in the auth branch -- so before this existed a post-auth ``CLOSED`` + for either one was terminal for the connection: nothing else in the process + would ever send that REQ again. + + A permanent reason is not retried here, and for ``auth-required:`` that is + not a gap: the relay pairs it with an ``AUTH`` challenge, and ``_session``'s + auth branch re-opens both control subscriptions once the NIP-42 handshake + completes. Re-issuing here as well would only race that. (The *pre*-auth + ``auth-required:`` case never reaches this function at all -- see + ``_handle_closed`` -- because there nothing is down.) + + The two permanent branches are logged separately because they are not the + same outage. A relay demanding re-authentication mid-session is recovered + by the next ``AUTH`` challenge; a ``restricted:``/``blocked:``/``invalid:`` + refusal is not recovered until the connection is rebuilt. Saying "down + until the next reconnect or NIP-42 re-auth" for both was what made the + bootstrap warning read as an outage report. + """ + logger.warning("[buzz] relay closed control subscription %s: %s", sub_id, reason or "") + if not _is_transient_close(reason): + if _is_auth_required_close(reason): + logger.warning("[buzz] relay demanded re-authentication for %s AFTER this connection completed NIP-42 auth. It stays down until the relay's next AUTH challenge re-opens it, or until the next reconnect.", sub_id) + else: + logger.warning("[buzz] not re-opening %s: the relay refused it. Channel discovery/membership tracking is down until the next reconnect.", sub_id) + return + attempt = self._claim_resubscribe_attempt(sub_id) + if attempt is None: + logger.warning("[buzz] gave up re-opening control subscription %s after %d attempts; it stays down until the next reconnect", sub_id, MAX_RESUBSCRIBE_ATTEMPTS) + return + await self._resubscribe_backoff(attempt) + transport = self._transport + if transport is None: + return # the socket went away while backing off; _session rebuilds everything + sub_filter = self._discovery_filter() if sub_id == DISCOVERY_SUB_ID else self._membership_filter() + try: + await transport.send(buzz_nostr.req_frame(sub_id, sub_filter)) + except Exception: + logger.warning("[buzz] failed to re-open control subscription %s", sub_id, exc_info=True) + return + logger.info("[buzz] re-opened control subscription %s (attempt %d/%d)", sub_id, attempt, MAX_RESUBSCRIBE_ATTEMPTS) + + async def _recover_chat_subscription(self, channel_id: str, reason: str) -> None: + """Re-open one channel's chat subscription after the relay closed it, bounded. + + Only for a subscription we actually held. A ``CLOSED`` frame is relay-supplied + input, so treating one for an unknown channel as a recovery would let any + relay induce a chat subscription to a channel of its choosing simply by naming + it -- and would also let it grow ``_resubscribe_attempts`` without bound. + + The channel is dropped from ``_chat_subscriptions`` first, unconditionally: it + is genuinely not subscribed from this moment, and leaving it there would make + the retry (and any later kind-44100 or discovery sweep) a no-op -- the original + reason ``_forget_subscription`` existed. + """ + if channel_id not in self._chat_subscriptions: + logger.info("[buzz] relay closed chat subscription for channel %s, which we were not subscribed to: %s", channel_id, reason or "") + return + self._chat_subscriptions.discard(channel_id) + name = self._channel_meta.get(channel_id, {}).get("name") or "" + logger.warning("[buzz] relay closed the chat subscription for channel %s (%s), so it is now UNLISTENED: %s", name, channel_id, reason or "") + if not _is_transient_close(reason): + # Removed from the channel, or not authorized to read it: re-subscribing + # would be fighting the relay over a channel that is no longer ours. The + # metadata entry is deliberately left alone -- the next discovery pass is + # authoritative about membership, and ``_on_discovery_complete`` already + # documents the one-REQ-per-stale-channel cost of a stale entry. + logger.warning("[buzz] not re-subscribing to channel %s: the relay's reason says the subscription is no longer ours", channel_id) + return + attempt = self._claim_resubscribe_attempt(self._chat_sub_id(channel_id)) + if attempt is None: + logger.warning("[buzz] gave up re-subscribing to channel %s (%s) after %d attempts; it stays unlistened until the next reconnect", name, channel_id, MAX_RESUBSCRIBE_ATTEMPTS) + return + await self._resubscribe_backoff(attempt) + await self._ensure_chat_subscription(channel_id) + + def _confirm_auth_if_pending(self) -> None: + """Fallback NIP-42 confirmation for a relay that never sends an ``OK`` for AUTH. + + NIP-42 says a relay SHOULD reply ``OK`` to an AUTH event, but "should" is + not "does", and this connector must not depend on relay behaviour it + cannot control to ever leave its pre-auth-confirmed state. Without this + fallback, a relay that silently accepts the AUTH but never sends an ``OK`` + would leave ``_auth_completed`` false for the rest of the connection, and + EVERY later ``auth-required:`` close would misclassify as bootstrap noise + -- the same lie this whole fix removes, just permanently instead of for + one race window. + + Chosen over a bounded sleep/timeout: this is driven by the relay's own + response rather than the wall clock, so it needs no sleep, cannot fire + early, and adds no unbounded wait to the read loop. Reaching the discovery + subscription's EOSE is a safe signal because it only happens on a REQ the + relay actually served: an auth-rejecting relay CLOSEs the just-reopened + control subscriptions instead (routed to ``_handle_closed``, never here), + so seeing EOSE without a CLOSED first is itself proof the AUTH we sent was + accepted, explicit ``OK`` or not. + + Scoped to "we sent an AUTH and have not yet seen an ``OK`` for it" + (``_pending_auth_event_id`` set) so it can never fire before an AUTH was + even attempted -- where "not yet authenticated" is already the correct + read and this must stay a no-op. + """ + if self._pending_auth_event_id is None: + return + self._pending_auth_event_id = None + self._auth_completed = True + logger.info("[buzz] treating successful discovery as implicit NIP-42 confirmation (relay sent no explicit OK for our AUTH event)") + + async def _on_discovery_complete(self) -> None: + """EOSE for the discovery subscription: every channel we belong to has now been sent. + + Chat subscriptions are opened as each kind-39000 arrives, so this is normally + a no-op sweep. It exists because that is the point where the result can be + relied on: a channel whose REQ failed mid-discovery is retried here, and a + connector that discovered nothing is called out rather than sitting silent. + + The sweep runs over the whole metadata cache rather than only what this + connection discovered, which can re-subscribe to a channel we were removed + from while disconnected (kind-44101 prunes the cache, but only if we were + connected to receive it). That is self-correcting and deliberately not + engineered around: the relay answers with ``CLOSED``, whose reason names a + removal or a revocation, so ``_recover_chat_subscription`` classifies it as + permanent, drops the subscription, and does NOT retry -- the cost is one REQ + per stale channel per reconnect and never a wrong-channel message, since the + allowlist and signature gates are what decide whether anything is acted on. + + Also the (fallback) point where ``_confirm_auth_if_pending`` is given its + chance to run: reaching this EOSE at all is only possible on a REQ the + relay actually served, which is evidence the AUTH we sent was accepted even + if the relay never sent an explicit ``OK`` for it. See that method's + docstring for why this is a safe signal and why a bounded sleep was not + used instead. + """ + self._confirm_auth_if_pending() + for channel_id in list(self._channel_meta): + await self._ensure_chat_subscription(channel_id) + if self._chat_subscriptions: + logger.info("[buzz] channel discovery complete: listening to %d channel(s)", len(self._chat_subscriptions)) + else: + logger.warning("[buzz] channel discovery returned no channels: this identity is not a member of any channel on %s (add it with `buzz channels add-member`)", self._relay_url) + + def _handle_auth_ok(self, event_id: str, accepted: bool, message: str) -> None: + """React to a relay ``OK`` acknowledgment, specifically for our outstanding NIP-42 AUTH event. + + NIP-01 sends ``["OK", , , ]`` for every event + this connector publishes -- a chat post, an edit, an AUTH -- not only AUTH, + and this connector does not otherwise track its own published event ids to + correlate them against. So anything that is not the one outstanding AUTH + event id is silently ignored here: a rejected chat publish is a delivery + problem for ``send()``'s own retry path, not this session's auth + bookkeeping. + + This -- the relay's own affirmative reply for the EXACT event id we sent + -- is what is allowed to flip ``_auth_completed``, never the act of + sending the AUTH event itself. Sending only means we tried: a live relay + was observed still processing the AUTH it would eventually accept while it + closed the just-reopened control subscriptions with ``auth-required:``, and + a flag set at send-time misread that ordinary bootstrap tail as a genuine + post-auth refusal (see ``_AUTH_REQUIRED_CLOSE_PREFIX`` and + ``_handle_closed``). A relay that never sends an ``OK`` at all is covered + by ``_confirm_auth_if_pending``'s fallback instead. + + ``OK ... false`` is the other half of this fix: the relay REJECTED our + AUTH, which is a real, actionable problem (a bad key, clock skew outside + the relay's tolerance, a relay-side policy) and is logged loudly here + rather than silently leaving the connection merely "not yet + authenticated". It must never set ``_auth_completed`` -- the session stays + in its pre-auth-confirmed state, quiet-recoverable the normal way if a + fresh ``AUTH`` challenge arrives. + """ + if event_id != self._pending_auth_event_id: + return + self._pending_auth_event_id = None + if accepted: + self._auth_completed = True + logger.info("[buzz] relay acknowledged our NIP-42 AUTH event") + else: + logger.warning("[buzz] relay REJECTED our NIP-42 AUTH event: %s", message or "") + + async def _session(self, ws) -> None: + """Run one relay connection's lifetime: authenticate, discover, subscribe, read frames. + + ``self._transport`` is set for the duration of this connection so ``send()`` + / ``_post_event()`` can post outbound events on it, and is unconditionally + cleared in ``finally`` on the way out -- including on error -- so a stale + reference can never survive past this connection (``_post_event`` reads it + fresh on every call and raises when it is ``None``; see its docstring). + ``_chat_subscriptions`` is per-socket bookkeeping and is cleared alongside + it, so the next connection re-sends every REQ instead of believing + subscriptions from a dead socket are still live. + + NIP-42 auth is opportunistic, not upfront: discovery/membership REQs are + sent immediately in case the relay allows unauthenticated reads, but + ``handle_relay_frame`` records any ``AUTH`` challenge the relay sends onto + ``self._pending_auth_challenge``, and this loop reacts to it the next time + around by signing and sending the AUTH event and then RE-RUNNING DISCOVERY + -- our relay is closed, so the pre-auth REQs may have been rejected, and + without redoing them we would end up authenticated but listening to nothing. + The per-channel subscription set is dropped at the same moment for the same + reason: a chat REQ issued pre-auth may have been rejected too, so it must + not be remembered as live. A fresh challenge is per-connection (the relay + mints a new one on every new socket), so all of this runs again from + scratch on every reconnect. + + ``_resubscribe_attempts`` (the per-subscription ``CLOSED``-recovery budget) + is reset at exactly those two points, and for the same reason: a pre-auth + REQ that the relay closed because we had not authenticated yet is recovered + wholesale by the auth branch, and must not spend the budget that protects the + authenticated session from a relay that keeps closing a live subscription. + + ``_auth_completed`` is the same boundary expressed as a flag, and it is + strictly per socket (cleared on entry AND in ``finally``): it is what lets + ``_handle_closed`` tell the expected bootstrap ``auth-required:`` refusal + apart from a relay that stops honouring an already-authenticated session. + + It is set ONLY once the relay has ACKNOWLEDGED our AUTH event with a + matching ``OK ... true`` (``_handle_auth_ok``) -- never merely because we + sent one. A live relay was observed still processing the AUTH we had + already sent when it closed the just-reopened control subscriptions with + this same ``auth-required:`` reason, so a flag set at send-time + misclassified the tail of the ordinary bootstrap race as a post-auth + refusal (the operator-facing warning that started this fix). Sending AUTH + here only records ``_pending_auth_event_id`` so the eventual ``OK`` can be + matched back to it; a relay that never sends one at all is covered by + ``_confirm_auth_if_pending``'s fallback (see its docstring). A relay that + REJECTS the AUTH (``OK ... false``) is a real, actionable problem and is + logged loudly by ``_handle_auth_ok`` -- without ever setting this flag, so + the connection is never mistaken for authenticated. + """ + self._transport = ws + self._session_started_at = int(time.time()) + self._chat_subscriptions.clear() + self._resubscribe_attempts.clear() + self._auth_completed = False + self._pending_auth_event_id = None + try: + await self._open_control_subscriptions(ws) + async for raw in ws: + await self.handle_relay_frame(raw) + if self._pending_auth_challenge is not None: + assert self._keys is not None + challenge = self._pending_auth_challenge + self._pending_auth_challenge = None + auth = buzz_nostr.build_auth_event(self._keys, self._relay_url, challenge, created_at=int(time.time())) + self._pending_auth_event_id = auth["id"] + await ws.send(json.dumps(["AUTH", auth], separators=(",", ":"))) + self._chat_subscriptions.clear() + self._resubscribe_attempts.clear() + await self._open_control_subscriptions(ws) + finally: + self._transport = None + self._chat_subscriptions.clear() + self._resubscribe_attempts.clear() + self._auth_completed = False + self._pending_auth_event_id = None + self._session_started_at = None + + async def _run_loop(self) -> None: + """Own the relay connection for the channel's lifetime: connect, run one session, retry forever. + + Runs as the asyncio task ``_spawn_connection`` starts; ``stop()`` cancels it. + Every sibling connector (wecom.py, discord.py, feishu.py) delegates + reconnection to its vendor SDK -- Buzz has no SDK, so this loop owns + reconnect/backoff itself: exponential backoff capped at 60s with jitter + (so a thundering herd of Buzz-connected agents restarting together does not + all retry in lockstep), reset to 0 after any connection that actually got + established (so a long stable run doesn't leave a stale attempt count that + punishes the next transient blip with a large initial delay). + + A clean stream end -- ``_session`` returning normally because the relay + closed the socket on us -- is treated the same as a connection error: both + fall through to the backoff-and-retry below rather than ending the loop, + because for a relay client "the peer hung up" is exactly the situation + reconnection exists for. + + ``asyncio.CancelledError`` is re-raised immediately rather than reaching the + generic ``except Exception`` below -- it isn't an ``Exception`` subclass in + the supported Python version anyway, but the explicit clause documents the + contract: this is how the guarded ``stop()`` (Task 3) actually stops this + otherwise-infinite loop, and it must never be swallowed here. + + Untrusted relay input is guarded one layer down: ``handle_relay_frame`` + already logs-and-drops a non-JSON payload or a malformed ``EVENT`` instead + of raising, so a single bad frame reaches neither this loop's + ``except Exception`` (reconnect) nor kills the read loop -- it is simply + skipped and iteration continues. + """ + attempt = 0 + while True: + try: + if self._connect is not None: + ws = await self._connect() + else: + import websockets + + ws = await websockets.connect(self._relay_url) + async with ws: + attempt = 0 # reset only once a connection is actually established + await self._session(ws) + # Clean stream end (relay closed on us): reconnect with backoff too. + attempt += 1 + except asyncio.CancelledError: + raise + except Exception as exc: + attempt += 1 + logger.warning("[buzz] relay connection error (attempt %d): %s", attempt, exc) + # Cap the counter itself, not just the delay: 2**10 already exceeds the + # 60s ceiling below, so clamping here keeps the exponent computation + # cheap indefinitely instead of growing the attempt count (and the + # bigint math behind 2**attempt) without bound across a very long + # outage. + attempt = min(attempt, 10) + delay = min(60, 2**attempt) + random.uniform(0, 1) + await asyncio.sleep(delay) + + # -- inbound ------------------------------------------------------------- + + async def handle_relay_frame(self, raw: str) -> None: + """Route one raw relay WebSocket frame (Task 6 calls this per frame received). + + Relay input is untrusted: a non-JSON payload, an unexpected frame shape, or + an ``EVENT`` payload whose tags/timestamp are malformed is logged and dropped + rather than raised, so one bad frame can never crash the read loop. + + Every ``EVENT`` is signature-verified (``buzz_nostr.verify_event``) here, + at the single entry point, BEFORE either handler runs. The relay operator + is not necessarily the DeerFlow operator on a team-run Buzz relay, and + ``ev["pubkey"]`` is just a field in a relay-supplied JSON object -- without + this check a malicious or compromised relay could name any allowlisted + author it liked and trigger tool-executing runs, or bind a victim's pubkey + to an attacker's DeerFlow account through ``/connect``. Both handlers make + authorization decisions from the event (the chat gate uses ``pubkey`` as + the principal; kind-39000 metadata can relax the mention requirement), so + verifying at the choke point rather than inside each one leaves no path + where an unverified event reaches a decision. + + Chosen over verifying later, behind the cheap self/kind/channel gates, on + purpose: a Schnorr verify is tens of microseconds against human-rate chat + traffic, so the throughput saved by gating first is worth less than the + guarantee that no future edit can reorder a gate ahead of the check. + """ + try: + frame = json.loads(raw) + except json.JSONDecodeError: + logger.warning("[buzz] non-JSON relay frame ignored") + return + if not isinstance(frame, list) or not frame: + return + kind = frame[0] + if kind == "AUTH" and len(frame) >= 2: + self._pending_auth_challenge = str(frame[1]) + elif kind == "OK" and len(frame) >= 3: + self._handle_auth_ok(str(frame[1]), frame[2] is True, str(frame[3]) if len(frame) >= 4 else "") + elif kind == "EOSE" and len(frame) >= 2 and frame[1] == DISCOVERY_SUB_ID: + await self._on_discovery_complete() + elif kind == "CLOSED" and len(frame) >= 2: + await self._handle_closed(str(frame[1]), str(frame[2]) if len(frame) >= 3 else "") + elif kind == "EVENT" and len(frame) >= 3 and isinstance(frame[2], dict): + ev = frame[2] + if not buzz_nostr.verify_event(ev): + logger.warning("[buzz] dropped relay event with an invalid id/signature (claimed pubkey=%s kind=%s)", ev.get("pubkey"), ev.get("kind")) + return + try: + ev_kind = ev.get("kind") + if ev_kind == buzz_nostr.KIND_CHANNEL_META: + channel_id = self._handle_meta_event(ev) + if channel_id: + await self._ensure_chat_subscription(channel_id) + elif ev_kind == buzz_nostr.KIND_CHAT: + await self._handle_chat_event(ev) + elif ev_kind in (buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED): + await self._handle_membership_event(ev) + except Exception: + # Defense in depth against malformed tags/timestamps inside an + # otherwise well-shaped EVENT frame (e.g. a non-integer created_at, + # or a "tags" field that isn't a list of [name, value, ...] lists). + logger.warning("[buzz] malformed relay event ignored", exc_info=True) + # other EOSE / NOTICE frames need no action + + def _handle_meta_event(self, ev: dict) -> str | None: + """Cache kind-39000 channel metadata: ``d`` = channel id, ``t`` = type, ``name``. + + Returns the channel id it cached (or ``None``), because kind-39000 is also + the channel-DISCOVERY event: the caller turns each one into that channel's + chat subscription. The two uses share one handler on purpose -- the set of + channels we hold metadata for is exactly the set we are a member of. + + Trust assumption (authorship, not authenticity): ``handle_relay_frame`` + has already proved this event really was signed by the pubkey it names, + but this does not check WHICH pubkey that is. In the Buzz protocol, + kind-39000 channel-discovery events are expected to be published by the + relay's own keypair, not by ordinary members. So any relay member able to + publish an event can still legitimately sign one and mark an arbitrary + channel ``type: "dm"``, which relaxes the mention gate for that channel + (see ``_is_dm``) — it does NOT bypass the independent pubkey allowlist + gate in ``_handle_chat_event``. Closing this gap needs a trusted relay + pubkey to check the author against, and nothing already configured + identifies one: ``relay_url`` is a network address, not a signing key. + Deliberately left as a follow-up rather than inventing a new required + config key (e.g. ``relay_pubkey``) for it here. + + The cache is remote-fed, so it is capped (``MAX_CACHED_CHANNELS``) on a + first-in-first-out basis: an unbounded map would let any member grow this + process's memory one forged ``d`` tag at a time. Eviction is safe by + construction — a channel we hold no metadata for is treated as a non-DM + (``_is_dm`` fails closed), and the next kind-39000 event for it repopulates + the entry. + """ + d_values = buzz_nostr.tag_values(ev, "d") + if not d_values: + return None + names = buzz_nostr.tag_values(ev, "name") + types = buzz_nostr.tag_values(ev, "t") + channel_id = d_values[0] + meta = {"type": types[0] if types else "stream", "name": names[0] if names else ""} + self._channel_meta[channel_id] = meta + while len(self._channel_meta) > MAX_CACHED_CHANNELS: + self._channel_meta.pop(next(iter(self._channel_meta))) + logger.debug("[buzz] channel metadata cached: %s (%s) type=%s", meta["name"] or "", channel_id, meta["type"]) + return channel_id + + async def _handle_membership_event(self, ev: dict) -> None: + """React to a relay-signed kind-44100 / kind-44101 membership notification. + + This is what makes a newly added channel work WITHOUT a reconnect: the relay + publishes one of these (``p`` = the affected member, ``h`` = the channel + uuid) on the membership subscription opened for every connection, so being + added to a channel is a live event rather than something we would only + notice the next time the socket happened to drop. + + The ``#p`` filter on the subscription is the relay's claim, not a proof, so + the ``p`` tag is re-checked here: another member's membership change must + never make us subscribe to a channel we do not belong to. (The event's + signature is already verified at ``handle_relay_frame``; what is trusted, as + for kind-39000, is that the signer is the relay -- see the class docstring's + trust model. A member who forges one can at most make us open a subscription + the relay will refuse or answer with nothing.) + + Removal drops the cached metadata too: the entry would otherwise keep the + channel in the discovery-EOSE sweep and make every later reconnect + re-subscribe to a channel we have been thrown out of. + """ + assert self._keys is not None + if self._keys.pubkey_hex not in buzz_nostr.tag_values(ev, "p"): + return # someone else's membership change + channel_ids = buzz_nostr.tag_values(ev, "h") + if not channel_ids: + return + channel_id = channel_ids[0] + if ev.get("kind") == buzz_nostr.KIND_MEMBER_ADDED: + known = channel_id in self._channel_meta + logger.info("[buzz] added to channel %s; subscribing", channel_id) + # Subscribe first so no message is missed while metadata catches up, then + # re-run discovery: the new channel's name and type (which drives the DM + # mention exemption) are only carried by its kind-39000 event. + await self._ensure_chat_subscription(channel_id) + if not known: + # ... but only when there is something to learn. Discovery re-issues + # the REQ on its own subscription id, so the relay answers with every + # stored kind-39000 and a fresh EOSE -- i.e. a whole extra discovery + # pass. A 44100 for a channel whose metadata we already hold tells us + # nothing new, and re-running discovery for it is how a burst of + # membership notifications multiplied into M+1 discovery passes per + # connect. ``since`` (see ``_membership_filter``) is what stops that + # burst arriving at all; this is the second, independent stop. + await self._refresh_channel_discovery() + else: + logger.info("[buzz] removed from channel %s; unsubscribing", channel_id) + await self._close_chat_subscription(channel_id) + self._channel_meta.pop(channel_id, None) + + async def _refresh_channel_discovery(self) -> None: + """Re-issue the discovery REQ on its existing subscription id. + + Replaces the subscription in place, so the relay re-sends every kind-39000 + it holds for us: metadata for the new channel arrives, and each event runs + through ``_ensure_chat_subscription``, which is a no-op for channels already + subscribed. Cheaper to reason about than a one-off narrowed query, and it + reconciles anything else that changed while we were connected. + """ + transport = self._transport + if transport is None: + return + try: + await transport.send(buzz_nostr.req_frame(DISCOVERY_SUB_ID, self._discovery_filter())) + except Exception: + logger.warning("[buzz] failed to refresh channel discovery", exc_info=True) + + def _is_dm(self, channel_id: str) -> bool: + """True only when metadata was positively cached as ``type == "dm"``. + + Fails closed: a channel we have no kind-39000 metadata for yet is never + treated as a DM, even though ``dict.get(..., {})`` would otherwise make an + absent entry look indistinguishable from an unset (non-DM) type. + """ + return self._channel_meta.get(channel_id, {}).get("type") == "dm" + + def _thread_root(self, ev: dict) -> str | None: + e_tags = buzz_nostr.tag_values(ev, "e") + return e_tags[0] if e_tags else None + + def _strip_own_mention(self, text: str) -> str: + """Strip a single, unambiguous leading ``@mention`` token. + + Nostr chat events carry no verified mapping from the free-text "@Name" a + client rendered into the message body to the pubkeys in the event's ``p`` + tags — the caller only confirms *some* p-tagged mention exists (see + ``mentioned`` in ``_handle_chat_event``), never that the specific leading + token names *us*. When a second ``@token`` immediately follows the first + (e.g. "@Alice, @DeerFlow help"), guessing that the first one is ours risks + silently discarding a different member's mention while leaving ours + untouched, so the conservative choice is to leave the text completely + alone rather than guess. The common single-mention case ("@DeerFlow + hello") remains unambiguous and is still stripped. + """ + stripped = text.lstrip() + if not stripped.startswith("@"): + return text.strip() + _, sep, rest = stripped.partition(" ") + if not sep: + return stripped # "@DeerFlow" alone: nothing to strip without losing the whole message + if rest.lstrip().startswith("@"): + return text.strip() # ambiguous multi-mention prefix: don't guess which one is ours + return rest.strip() or stripped + + async def _bind_connection(self, code: str, author: str, channel_id: str) -> None: + """Consume a ``/connect `` bind code for *author* (a Nostr pubkey hex). + + Always fully handles the request — valid code, invalid/expired/already-used + code, or a connection-repo error — so the caller (``_handle_chat_event``) can + unconditionally return right after awaiting this without ever falling through + to ``_make_inbound``/``_publish``. Mirrors discord.py's / slack.py's + ``_bind_connection_from_connect_code``. + + Bind success/failure is fully determined and logged by the try/except below + BEFORE ``_reply_to_connect`` is ever invoked, so a failure to *send* the + confirmation/error reply can never be attributed back to (or logged as) a + bind failure — see ``_reply_to_connect``. This ordering is deliberate: an + earlier review of this method flagged that sending from inside the same + try/except that decides bind success would let a relay-send hiccup on an + otherwise-successful bind get reported as "failed to bind". + """ + if self._connection_repo is None: + return # unreachable in practice: _pending_connect_code already requires this + try: + state = await self._connection_repo.consume_oauth_state(provider="buzz", state=code) + if state is None: + logger.info("[buzz] /connect code invalid, expired, or already used (pubkey=%s)", author) + outcome = "invalid" + else: + await self._connection_repo.upsert_connection( + owner_user_id=state["owner_user_id"], + provider="buzz", + external_account_id=author, + workspace_id=self._workspace_id, + metadata={"pubkey": author}, + status="connected", + ) + logger.info("[buzz] connected pubkey=%s to owner_user_id=%s", author, state["owner_user_id"]) + outcome = "success" + except Exception: + # A repo/DB error binding the code must not propagate: handle_relay_frame's + # outer guard would also catch it, but catching here keeps the log specific + # to the bind failure instead of a generic "malformed relay event ignored". + logger.exception("[buzz] failed to bind /connect code for pubkey=%s", author) + outcome = "error" + + # Bind success/failure is already fully decided and logged above; sending + # the reply is a separate, best-effort concern from here on. + await self._reply_to_connect(channel_id, author, _CONNECT_REPLY_TEXT[outcome]) + + async def _reply_to_connect(self, channel_id: str, author: str, text: str) -> None: + """Best-effort confirmation/error reply for a ``/connect`` attempt. + + The caller (``_bind_connection``) has already fully decided and logged the + bind outcome before this runs. A failure here is a relay-send problem, not + a bind problem: mirroring discord.py's ``_send_connection_reply`` / slack.py's + ``_post_connection_reply``, it never raises and logs its own, + distinctly-worded warning, so it can never be mistaken for (or logged as) a + failed bind. Uses a single attempt (no retry/backoff): this is a courtesy + notification, not the delivery-critical agent-response path ``send()`` + serves, so a transient relay hiccup here should not add retry latency to + the inbound relay read loop. + """ + assert self._keys is not None + try: + event = buzz_nostr.build_chat_event(self._keys, channel_id, text, created_at=int(time.time()), mentions=(author,)) + await self._send_with_retry(lambda: self._post_event(event), max_retries=1, operation_name="connect-reply") + except Exception: + logger.warning("[buzz] failed to send /connect reply to pubkey=%s", author) + + def _advance_watermark(self, channel_id: str, created_at: int) -> None: + """Move THIS CHANNEL's resubscribe cursor to *created_at*, refusing future timestamps. + + The cursor is per channel because subscriptions are: a single global + watermark is the newest event processed in ANY channel, so a busy channel + would keep dragging it forward and a quiet channel's next REQ would ask for + events newer than traffic that never belonged to it -- silently skipping + everything published in the quiet channel while we were disconnected. That + is the one direction this cursor must never fail in. Per channel costs one + integer per channel and cannot skip. + + ``created_at`` is chosen by the event's author, and ``_chat_filter`` replays + it as ``since`` on every reconnect. Accepting it unchecked was a remote + denial of service: one event stamped year-5138 pinned the cursor there, so + every subsequent REQ asked for events newer than that and the connector went + permanently deaf with no log and no recovery short of a process restart. + + The cursor therefore never moves beyond ``now + MAX_FUTURE_SKEW_SECONDS``. + An out-of-range timestamp is IGNORED rather than clamped down to the + ceiling: clamping would still hand an attacker a blind window of exactly + the slack, while ignoring leaves the cursor where the last plausible event + put it. A legitimately fast-clocked member merely fails to advance the + cursor, which costs replay (drops we re-apply) and never a miss. + + Channel ids arrive in remote ``h`` tags, so the map is remote-fed and capped + the same way ``_channel_meta`` is. Eviction is safe by construction: a + channel with no cursor simply re-subscribes without ``since`` and gets the + relay's default backlog, i.e. eviction can only ever cost replay. + """ + if created_at <= 0 or not channel_id: + return + ceiling = int(time.time()) + MAX_FUTURE_SKEW_SECONDS + if created_at > ceiling: + logger.debug("[buzz] ignoring future-dated created_at=%d for the resubscribe cursor (ceiling=%d)", created_at, ceiling) + return + self._seen_created_at[channel_id] = max(self._seen_created_at.get(channel_id, 0), created_at) + while len(self._seen_created_at) > MAX_CACHED_CHANNELS: + self._seen_created_at.pop(next(iter(self._seen_created_at))) + + async def _attach_connection_identity(self, inbound: InboundMessage) -> InboundMessage: + """Resolve a persisted ``/connect`` binding for this pubkey, exactly as every sibling does. + + Without this the bind is write-only: ``connection_id`` / ``owner_user_id`` + stay ``None``, so ``ChannelManager`` runs the turn under a synthetic + pubkey-derived user with its own memory and file buckets instead of the + bound DeerFlow account, and revoking the connection has no runtime effect. + + ``fallback_without_workspace`` stays off (unlike discord/dingtalk/wecom, + whose workspace is legitimately absent for DMs): ``_bind_connection`` + always writes ``workspace_id=``, which ``__init__`` guarantees + is non-empty, so every Buzz row is reachable by the primary lookup. Adding + the ``None`` candidate could only ever match a row this connector did not + write, and matching it would resolve a pubkey bound on some other relay — + the one thing the workspace scoping exists to prevent. + """ + return await attach_connection_identity(inbound, repo=self._connection_repo, provider="buzz", workspace_id=self._workspace_id) + + async def _handle_chat_event(self, ev: dict) -> None: + assert self._keys is not None + author = str(ev.get("pubkey", "")) + channel_id_values = buzz_nostr.tag_values(ev, "h") + if not channel_id_values or author == self._keys.pubkey_hex: + return # no channel tag, or our own event (no self-reply loops) + channel_id = channel_id_values[0] + created_at = int(ev.get("created_at", 0)) + text = str(ev.get("content", "")) + + # /connect must be consulted before the allowlist gate (framework + # ordering rule — see Channel._pending_connect_code) so a not-yet-bound + # user can bootstrap a binding even though they aren't allowlisted yet. + # Unlike every other gate below, a /connect message is never published: + # _bind_connection fully handles it (valid, invalid, or erroring code) + # and this always returns immediately after, matching every sibling + # adapter's _bind_connection_from_connect_code (discord.py, slack.py, + # wecom.py, dingtalk.py, wechat.py, feishu.py). Falling through to the + # mention/allowlist gates and publishing this as an ordinary chat message + # would let any pubkey trigger a real agent run just by prefixing a + # message with "/connect" — Buzz's run policy sets + # requires_bound_identity=False, so the manager has no independent + # bound-identity check to catch that. + code = self._pending_connect_code(text) + if code is not None: + await self._bind_connection(code, author, channel_id) + # A bind attempt (valid or not) is fully processed, so it may advance the + # cursor: leaving it behind would replay this /connect on every reconnect + # and answer each replay with a spurious "code invalid or expired" reply. + self._advance_watermark(channel_id, created_at) + return + if author not in self._allowed_users: + # Deny-by-default is intentional (see start()'s empty-allowlist warning), + # but a silent drop is indistinguishable from a broken relay when an + # operator forgets to allowlist someone. Debug level: an open relay can + # carry plenty of chatter from members we never intend to serve. + logger.debug("[buzz] dropped chat event from non-allowlisted pubkey=%s in channel %s (%s)", author, self._channel_meta.get(channel_id, {}).get("name") or "", channel_id) + return + + # code is always None here: a non-None code was already fully handled + # and returned above, so this gate only ever sees ordinary chat text. + thread_root = self._thread_root(ev) + mentioned = self._keys.pubkey_hex in buzz_nostr.tag_values(ev, "p") + store = self.config.get("channel_store") + engaged_thread = bool(thread_root and store is not None and store.get_thread_id(self.name, channel_id, topic_id=thread_root)) + allowed_without_mention = (not self._require_mention) or channel_id in self._mention_free or self._is_dm(channel_id) or engaged_thread + if not mentioned and not allowed_without_mention: + return + + if mentioned: + text = self._strip_own_mention(text) + + msg_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT + inbound: InboundMessage = self._make_inbound(chat_id=channel_id, user_id=author, text=text, msg_type=msg_type, thread_ts=thread_root, metadata={"event_id": str(ev.get("id", ""))}) + inbound.topic_id = thread_root + inbound.workspace_id = self._workspace_id + inbound = await self._attach_connection_identity(inbound) + self._last_requester[(channel_id, thread_root)] = author + await self._publish(inbound) + # Only a fully accepted-and-published event advances the cursor, and only + # after the publish actually succeeded. A dropped event must never move it + # (that was the DoS), and a failed publish must leave it replayable. + self._advance_watermark(channel_id, created_at) + + # -- outbound -------------------------------------------------------------- + + async def _post_event(self, event: dict) -> None: + """Sign-and-post is already done by the caller; this only delivers the frame. + + Reads ``self._transport`` at call time (never cached): Task 6's relay loop + sets it to a live WebSocket for the duration of a connection and back to + ``None`` on disconnect, so a stale reference here would keep "succeeding" + against a socket that is no longer attached to anything. + """ + if self._transport is None: + raise RuntimeError("[buzz] relay connection not established") + await self._transport.send(buzz_nostr.event_frame(event)) + + async def _edit_or_repost(self, msg: OutboundMessage, target: str, content: str, now: int, *, label: str) -> str: + """Edit *target* in place; if that fails after retries, post a fresh message instead. + + Returns the event id later updates for this slot must target -- ``target`` + itself on success, the replacement's id after a degrade. Degrading rather + than raising is the "never lose content" rule the placeholder path has + always had; no mention rides the replacement because the requester was + already notified by the original post and re-mentioning on every degraded + edit would spam notifications. + """ + assert self._keys is not None + edit = buzz_nostr.build_edit_event(self._keys, msg.chat_id, target, content, created_at=now) + try: + await self._send_with_retry(lambda: self._post_event(edit), max_retries=3, operation_name=f"edit{label}") + return target + except Exception: + fresh = buzz_nostr.build_chat_event(self._keys, msg.chat_id, content, created_at=now, reply_to=msg.thread_ts) + await self._send_with_retry(lambda: self._post_event(fresh), max_retries=3, operation_name=f"post-degraded{label}") + return fresh["id"] + + async def send(self, msg: OutboundMessage) -> None: + """Post one placeholder chat message, then stream via in-place edits. + + The first message for a given ``(chat_id, thread_ts)`` is a kind-9 chat + event (the only place a mention can ride, since kind-40003 edits carry + only ``h``/``e`` tags — see ``buzz_nostr.build_edit_event``). Every + subsequent update for the same key edits that placeholder in place via a + kind-40003 event targeting its id. ``is_final`` clears the tracked target + so the next run in this conversation starts a fresh placeholder instead of + editing a stale, already-answered message. + + Oversize text is split by ``_chunk_text`` into <= ``EDIT_MAX_BYTES``-byte + chunks: the first chunk rides the placeholder/edit, any remaining chunks + ride follow-up kind-9 messages threaded to the same thread root, tracked + per conversation in ``_stream_targets`` / ``_stream_tails`` respectively. + + Every chunk index is tracked, not just chunk 0, because the manager + publishes CUMULATIVE text on each streaming update: an oversize reply + therefore re-splits into >= 2 chunks on EVERY update, and posting + ``chunks[1:]`` fresh each time flooded the channel with a near-duplicate + tail per update (a ~100KB answer at ~1 update/sec produced dozens). A + follow-up message is now posted only when the chunk count actually GROWS; + an index we have already posted is edited in place, exactly like chunk 0. + + Raises whatever the underlying send raised (including ``_post_event``'s + ``RuntimeError`` when ``_transport`` is ``None``) after retries are + exhausted, so the framework's outer retry/error-logging path in + ``Channel._on_outbound`` observes the failure instead of a reply being + silently dropped. The ``is_final`` bookkeeping below runs in a + ``finally`` block precisely so that a raised exception still propagates + to the caller *and* still clears the stale target — see the ``finally`` + comment for why both matter. + + DEFENSE IN DEPTH: text carrying one of DeerFlow's hidden model-context + wrappers is refused outright. The real fix is one layer up + (``manager._accumulate_stream_text`` now allowlists assistant message + types instead of denylisting tool ones), but this connector is the one + where a leak cannot be taken back: every streaming update is an immutable + public Nostr event, so a corrective edit only changes what clients render + while the original leaked event stays on the relay. Posting nothing is + therefore the right failure direction here, and it is deliberately not + replicated into the sibling connectors, whose channels can be edited or + deleted. See ``_HIDDEN_CONTEXT_MARKERS`` for why this cannot fire on an + ordinary reply that merely talks about memory. + """ + assert self._keys is not None + key = (msg.chat_id, msg.thread_ts) + now = int(time.time()) + if marker := _hidden_context_marker(msg.text): + logger.error("[buzz] REFUSED to publish a reply carrying hidden model context (%s) to channel %s; this is a leak upstream of the connector, not a relay problem", marker, msg.chat_id) + if msg.is_final: + self._stream_targets.pop(key, None) + self._stream_tails.pop(key, None) + return + chunks = _chunk_text(msg.text) + target = self._stream_targets.get(key) + + try: + if target is None: + requester = self._last_requester.get(key) + mentions = (requester,) if requester else () + first = buzz_nostr.build_chat_event(self._keys, msg.chat_id, chunks[0], created_at=now, reply_to=msg.thread_ts, mentions=mentions) + await self._send_with_retry(lambda: self._post_event(first), max_retries=3, operation_name="post") + self._stream_targets[key] = first["id"] + else: + self._stream_targets[key] = await self._edit_or_repost(msg, target, chunks[0], now, label="") + + tails = self._stream_tails.get(key) + for index, extra in enumerate(chunks[1:]): + if tails is not None and index < len(tails): + tails[index] = await self._edit_or_repost(msg, tails[index], extra, now, label="-overflow") + continue + follow = buzz_nostr.build_chat_event(self._keys, msg.chat_id, extra, created_at=now, reply_to=msg.thread_ts) + await self._send_with_retry(lambda ev=follow: self._post_event(ev), max_retries=3, operation_name="post-overflow") + if tails is None: + tails = self._stream_tails.setdefault(key, []) + tails.append(follow["id"]) + finally: + # Reviewer finding: the degrade-to-fresh-post branch above and the + # overflow-chunk loop can both raise after their own retries are + # exhausted, which used to propagate out of send() *before* reaching + # an unconditional pop at the end of the function -- leaking a stale + # (or half-updated) _stream_targets[key] entry whenever the failing + # call had is_final=True. Once the relay recovered, the next send() + # for this conversation would then EDIT that abandoned placeholder + # instead of starting a fresh message, contradicting this method's + # own contract. A `finally` clears the bookkeeping on every path -- + # success or failure -- while still letting the exception propagate + # (a `finally` block never suppresses an in-flight exception unless + # it itself returns/raises), so `Channel._on_outbound` still logs + # the failure exactly as before. The overflow-tail map added for the + # message-flood fix is cleared on exactly the same terms, so a final + # send can never leave the next run editing this run's tail messages. + if msg.is_final: + self._stream_targets.pop(key, None) + self._stream_tails.pop(key, None) diff --git a/backend/app/channels/buzz_nostr.py b/backend/app/channels/buzz_nostr.py new file mode 100644 index 000000000..6c1769f1f --- /dev/null +++ b/backend/app/channels/buzz_nostr.py @@ -0,0 +1,201 @@ +"""Pure Nostr (NIP-01) helpers for the Buzz channel connector. + +No I/O, no wall-clock: callers supply ``created_at``. BIP-340 signing is done via +``coincurve``, which ships in the optional ``buzz`` dependency extra and is imported +lazily so the rest of the app never requires it. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass +from typing import Any + +_BECH32_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l" + +COINCURVE_INSTALL_HINT = "The Buzz channel requires the 'buzz' extra: run `uv sync --extra buzz` (installs coincurve for BIP-340 signing)." + + +def _require_coincurve(): + try: + import coincurve + except ImportError as exc: # pragma: no cover - exercised via BuzzChannel.start + raise RuntimeError(COINCURVE_INSTALL_HINT) from exc + return coincurve + + +@dataclass(frozen=True) +class NostrKeys: + secret: bytes + pubkey_hex: str + + +def _bech32_polymod(values: list[int]) -> int: + gen = [0x3B6A57B2, 0x26508E6D, 0x1EA119FA, 0x3D4233DD, 0x2A1462B3] + chk = 1 + for v in values: + b = chk >> 25 + chk = (chk & 0x1FFFFFF) << 5 ^ v + for i in range(5): + chk ^= gen[i] if ((b >> i) & 1) else 0 + return chk + + +def _bech32_decode(expected_hrp: str, value: str) -> bytes: + if "1" not in value: + raise ValueError(f"not bech32: {value!r}") + hrp, data_part = value.rsplit("1", 1) + if hrp != expected_hrp: + raise ValueError(f"expected {expected_hrp!r} bech32, got {hrp!r}") + try: + data = [_BECH32_CHARSET.index(c) for c in data_part] + except ValueError as exc: + raise ValueError(f"invalid bech32 character in {value!r}") from exc + hrp_expanded = [ord(c) >> 5 for c in hrp] + [0] + [ord(c) & 31 for c in hrp] + if _bech32_polymod(hrp_expanded + data) != 1: + raise ValueError(f"bad bech32 checksum in {value!r}") + acc = bits = 0 + out = bytearray() + for v in data[:-6]: + acc = (acc << 5) | v + bits += 5 + if bits >= 8: + bits -= 8 + out.append((acc >> bits) & 0xFF) + if len(out) != 32: + raise ValueError(f"expected 32-byte payload in {value!r}") + return bytes(out) + + +def _parse_32_bytes(value: str, bech_hrp: str) -> bytes: + value = value.strip() + if value.lower().startswith(f"{bech_hrp}1"): + return _bech32_decode(bech_hrp, value.lower()) + try: + raw = bytes.fromhex(value) + except ValueError as exc: + raise ValueError(f"expected 64-hex or {bech_hrp}1... value") from exc + if len(raw) != 32: + raise ValueError("expected exactly 32 bytes") + return raw + + +def parse_private_key(value: str) -> NostrKeys: + secret = _parse_32_bytes(value, "nsec") + coincurve = _require_coincurve() + pubkey = coincurve.PrivateKey(secret).public_key.format(compressed=True)[1:] + return NostrKeys(secret=secret, pubkey_hex=pubkey.hex()) + + +def parse_pubkey(value: str) -> str: + return _parse_32_bytes(value, "npub").hex() + + +def event_id(pubkey_hex: str, created_at: int, kind: int, tags: list[list[str]], content: str) -> str: + payload = json.dumps([0, pubkey_hex, created_at, kind, tags, content], separators=(",", ":"), ensure_ascii=False) + return hashlib.sha256(payload.encode()).hexdigest() + + +def sign_event(keys: NostrKeys, kind: int, tags: list[list[str]], content: str, created_at: int) -> dict: + coincurve = _require_coincurve() + eid = event_id(keys.pubkey_hex, created_at, kind, tags, content) + sig = coincurve.PrivateKey(keys.secret).sign_schnorr(bytes.fromhex(eid)) + return {"id": eid, "pubkey": keys.pubkey_hex, "created_at": created_at, "kind": kind, "tags": tags, "content": content, "sig": sig.hex()} + + +KIND_CHAT = 9 +KIND_EDIT = 40003 +KIND_AUTH = 22242 +KIND_CHANNEL_META = 39000 +# Relay-signed membership notifications (buzz-core's KIND_MEMBER_ADDED_NOTIFICATION / +# KIND_MEMBER_REMOVED_NOTIFICATION). Each carries ``p`` = the affected member's pubkey +# and ``h`` = the channel uuid, which is how a connected client learns it was added to +# (or removed from) a channel without reconnecting. +KIND_MEMBER_ADDED = 44100 +KIND_MEMBER_REMOVED = 44101 + + +def build_auth_event(keys: NostrKeys, relay_url: str, challenge: str, created_at: int) -> dict: + return sign_event(keys, KIND_AUTH, [["relay", relay_url], ["challenge", challenge]], "", created_at) + + +def build_chat_event(keys: NostrKeys, channel_id: str, content: str, created_at: int, reply_to: str | None = None, mentions: tuple[str, ...] = ()) -> dict: + tags: list[list[str]] = [["h", channel_id]] + if reply_to: + tags.append(["e", reply_to]) + tags.extend(["p", m] for m in mentions) + return sign_event(keys, KIND_CHAT, tags, content, created_at) + + +def build_edit_event(keys: NostrKeys, channel_id: str, target_event_id: str, content: str, created_at: int) -> dict: + return sign_event(keys, KIND_EDIT, [["h", channel_id], ["e", target_event_id]], content, created_at) + + +def verify_event(event: Any) -> bool: + """True only when *event* carries a self-consistent id and a valid BIP-340 signature. + + Two independent checks, both required: + + 1. The NIP-01 event id is RECOMPUTED from the event's own + ``pubkey``/``created_at``/``kind``/``tags``/``content`` and must equal the + ``id`` the sender claims -- so ``id`` cannot be borrowed from a different + (legitimately signed) event while the payload is swapped. + 2. The Schnorr signature must verify against that id under the claimed + ``pubkey``, which is what actually binds the payload to its author. + + Relay input is untrusted, so this NEVER raises: any missing, mistyped, + non-hex, or wrong-length field -- or a payload that is not even a mapping -- + is simply an event that fails to verify, and callers must be able to treat + "malformed" and "forged" identically without a try/except at every call site. + A missing ``coincurve`` (the optional ``buzz`` extra) also lands here and + fails closed; it is unreachable in practice because ``BuzzChannel.start()`` + already parses its private key through ``coincurve`` and would have failed + with :data:`COINCURVE_INSTALL_HINT` long before any event arrived. + """ + try: + if not isinstance(event, dict): + return False + pubkey = event.get("pubkey") + sig = event.get("sig") + claimed_id = event.get("id") + content = event.get("content") + created_at = event.get("created_at") + kind = event.get("kind") + tags = event.get("tags") + # bool is an int subclass; a JSON `true` in either numeric field would + # otherwise serialize as "true" and silently change the canonical form. + if not isinstance(pubkey, str) or not isinstance(sig, str) or not isinstance(claimed_id, str) or not isinstance(content, str) or not isinstance(tags, list): + return False + if not isinstance(created_at, int) or isinstance(created_at, bool) or not isinstance(kind, int) or isinstance(kind, bool): + return False + if event_id(pubkey, created_at, kind, tags, content) != claimed_id: + return False + coincurve = _require_coincurve() + return bool(coincurve.PublicKeyXOnly(bytes.fromhex(pubkey)).verify(bytes.fromhex(sig), bytes.fromhex(claimed_id))) + except Exception: + return False + + +def req_frame(sub_id: str, *filters: dict) -> str: + return json.dumps(["REQ", sub_id, *filters], separators=(",", ":")) + + +def event_frame(event: dict) -> str: + return json.dumps(["EVENT", event], separators=(",", ":")) + + +def close_frame(sub_id: str) -> str: + """NIP-01 ``CLOSE``: stop an individual subscription without dropping the socket. + + Needed because chat subscriptions are per channel (the relay only fans kind-9 + events out to ``#h``-scoped subscriptions), so being removed from a channel has + to unsubscribe exactly that one -- the other channels' subscriptions, the + discovery subscription, and the membership subscription all ride the same + connection and must survive. + """ + return json.dumps(["CLOSE", sub_id], separators=(",", ":")) + + +def tag_values(event: dict, name: str) -> list[str]: + return [t[1] for t in event.get("tags", []) if len(t) >= 2 and t[0] == name] diff --git a/backend/app/channels/buzz_run_policy.py b/backend/app/channels/buzz_run_policy.py new file mode 100644 index 000000000..d858a3d45 --- /dev/null +++ b/backend/app/channels/buzz_run_policy.py @@ -0,0 +1,12 @@ +"""Run-policy registration for the Buzz channel (imported for side effect from manager.py).""" + +from app.channels.run_policy import CHANNEL_RUN_POLICY, ChannelRunPolicy + + +def register_policy() -> None: + # Same-thread follow-ups queue instead of tripping the busy reply (Feishu precedent); + # the adapter-level pubkey allowlist is the identity gate, so no bound identity needed. + CHANNEL_RUN_POLICY["buzz"] = ChannelRunPolicy(serialize_thread_runs=True, requires_bound_identity=False) + + +register_policy() diff --git a/backend/app/channels/manager.py b/backend/app/channels/manager.py index 72be91337..d7982006f 100644 --- a/backend/app/channels/manager.py +++ b/backend/app/channels/manager.py @@ -18,6 +18,7 @@ from urllib.parse import quote import httpx from langgraph_sdk.errors import ConflictError +from app.channels import buzz_run_policy as _buzz_run_policy # noqa: F401 from app.channels import feishu_run_policy as _feishu_run_policy # noqa: F401 from app.channels.commands import KNOWN_CHANNEL_COMMANDS from app.channels.dedupe_store import InboundDedupeStore, MemoryInboundDedupeStore @@ -125,6 +126,7 @@ INBOUND_DEDUPE_METADATA_KEYS = ("event_id", "message_id", "msg_id") CHAT_SCOPED_WORKSPACE_CHANNELS = frozenset({"telegram", "feishu", "wechat"}) CHANNEL_CAPABILITIES = { + "buzz": {"supports_streaming": True}, "dingtalk": {"supports_streaming": False}, "discord": {"supports_streaming": False}, "feishu": {"supports_streaming": True}, @@ -515,12 +517,93 @@ def _extract_stream_message_id(payload: Any, metadata: Any) -> str | None: return None +def _stream_payload_type(payload: Mapping[str, Any]) -> str: + """Resolve the message ``type`` of one ``messages-tuple`` payload. + + Two payload shapes reach this function and they name the message type in + different places: + + * The shape DeerFlow's own gateway emits (``runtime/serialization.py`` + calls ``model_dump()``): ``type`` is the LangChain literal directly -- + ``"ai"`` / ``"AIMessageChunk"`` / ``"human"`` / ``"tool"`` / ``"system"``. + * LangChain's ``to_json()`` constructor shape, which + ``_extract_stream_message_id`` and the content extraction below already + accommodate: the wrapper's own ``type`` is the literal string + ``"constructor"`` and the real class name is the last element of the + ``id`` path (``["langchain", "schema", "messages", "AIMessageChunk"]``), + with the constructor kwargs under ``kwargs``. + + Reading only the top level would classify every constructor-shaped payload + as ``"constructor"``, which an allowlist rejects (safe) but which would + also mean hidden context and assistant output are treated identically -- + so the class name is resolved properly instead of guessed. + """ + raw_type = payload.get("type") + if isinstance(raw_type, str) and raw_type and raw_type != "constructor": + return raw_type + kwargs = payload.get("kwargs") + if isinstance(kwargs, Mapping): + nested = kwargs.get("type") + if isinstance(nested, str) and nested: + return nested + lc_path = payload.get("id") + if isinstance(lc_path, (list, tuple)) and lc_path: + tail = lc_path[-1] + if isinstance(tail, str) and tail: + return tail + return raw_type if isinstance(raw_type, str) else "" + + +def _is_assistant_stream_type(payload_type: str) -> bool: + """Is this message type assistant output, i.e. displayable in an IM channel? + + An ALLOWLIST, deliberately. The previous denylist ("reject anything whose + type contains 'tool'") published every other message type, and DeerFlow + writes hidden model context into the ``messages`` channel as ordinary + messages: ``DynamicContextMiddleware`` injects the ```` block as a + hidden ``HumanMessage`` (``type == "human"``) and rewrites the user's own + turn into a new ``HumanMessage``, and ``DurableContextMiddleware`` injects a + hidden ```` ``HumanMessage``. LangGraph fans state + writes out on the ``messages-tuple`` stream, so all of those reached the + channel as if they were the assistant's reply -- proved live on a Buzz + relay, where each streaming update is an immutable public Nostr event and a + later corrective edit cannot unpublish the leaked one. + + The accepted spellings are the ones assistant output actually carries: + LangChain serializes ``AIMessage.type`` as ``"ai"`` and + ``AIMessageChunk.type`` as ``"AIMessageChunk"``; ``"assistant"`` is the + OpenAI-style spelling a foreign runtime may use. Matching is by prefix + rather than substring because a substring test is not safe here -- ordinary + English words contain "ai" ("chain", "domain"), so ``"ai" in type`` would + admit a future/foreign type name by accident, which is exactly the class of + mistake this allowlist exists to prevent. No LangChain message type other + than the AI ones begins with "ai" or "assistant". + """ + normalized = payload_type.strip().lower() + return normalized.startswith(("ai", "assistant")) + + def _accumulate_stream_text( buffers: dict[str, str], current_message_id: str | None, event_data: Any, ) -> tuple[str | None, str | None]: - """Convert a ``messages-tuple`` event into the latest displayable AI text.""" + """Convert a ``messages-tuple`` event into the latest displayable AI text. + + Only assistant output is displayable. Hidden human/system context (memory + facts, durable context, the middleware-rewritten echo of the user's own + message) and tool traffic must never be published to an IM channel; see + :func:`_is_assistant_stream_type`. + + A bare ``str`` payload -- previously accepted here and buffered under the + current message id -- carries no type information at all, so it cannot be + attributed to the assistant. Nothing in DeerFlow produces it (the gateway + always serializes a ``messages-tuple`` chunk as ``[message_dict, metadata]`` + via ``runtime/serialization.py::serialize_messages_tuple``), and a runtime + that did emit raw text deltas would emit hidden context the same way, with + no way to tell them apart. Under an allowlist an unattributable payload is + dropped rather than published. + """ payload = event_data metadata: Any = None if isinstance(event_data, (list, tuple)): @@ -529,16 +612,10 @@ def _accumulate_stream_text( if len(event_data) > 1: metadata = event_data[1] - if isinstance(payload, str): - message_id = current_message_id or "__default__" - buffers[message_id] = _merge_stream_text(buffers.get(message_id, ""), payload) - return buffers[message_id], message_id - if not isinstance(payload, Mapping): return None, current_message_id - payload_type = str(payload.get("type", "")).lower() - if "tool" in payload_type: + if not _is_assistant_stream_type(_stream_payload_type(payload)): return None, current_message_id text = _extract_text_content(payload.get("content")) diff --git a/backend/app/channels/service.py b/backend/app/channels/service.py index f46724beb..972bdc0ee 100644 --- a/backend/app/channels/service.py +++ b/backend/app/channels/service.py @@ -23,6 +23,7 @@ if TYPE_CHECKING: # Channel name → import path for lazy loading _CHANNEL_REGISTRY: dict[str, str] = { + "buzz": "app.channels.buzz:BuzzChannel", "dingtalk": "app.channels.dingtalk:DingTalkChannel", "discord": "app.channels.discord:DiscordChannel", "feishu": "app.channels.feishu:FeishuChannel", @@ -35,6 +36,7 @@ _CHANNEL_REGISTRY: dict[str, str] = { # Keys that indicate a user has configured credentials for a channel. _CHANNEL_CREDENTIAL_KEYS: dict[str, list[str]] = { + "buzz": ["private_key"], "dingtalk": ["client_id", "client_secret"], "discord": ["bot_token"], "feishu": ["app_id", "app_secret"], diff --git a/backend/app/gateway/routers/channel_connections.py b/backend/app/gateway/routers/channel_connections.py index 5fbb62b7d..a435f7532 100644 --- a/backend/app/gateway/routers/channel_connections.py +++ b/backend/app/gateway/routers/channel_connections.py @@ -92,6 +92,7 @@ _PROVIDER_META: dict[str, dict[str, str]] = { "dingtalk": {"display_name": "DingTalk", "auth_mode": "binding_code"}, "wechat": {"display_name": "WeChat", "auth_mode": "binding_code"}, "wecom": {"display_name": "WeCom", "auth_mode": "binding_code"}, + "buzz": {"display_name": "Buzz", "auth_mode": "binding_code"}, } _CREDENTIAL_FIELDS: dict[str, tuple[dict[str, str], ...]] = { @@ -117,6 +118,10 @@ _CREDENTIAL_FIELDS: dict[str, tuple[dict[str, str], ...]] = { {"name": "bot_id", "label": "Bot ID", "type": "text"}, {"name": "bot_secret", "label": "Bot secret", "type": "password"}, ), + "buzz": ( + {"name": "relay_url", "label": "Relay URL", "type": "text"}, + {"name": "private_key", "label": "Private key (hex or nsec)", "type": "password"}, + ), } _RUNTIME_REQUIREMENTS: dict[str, tuple[str, ...]] = { @@ -127,6 +132,7 @@ _RUNTIME_REQUIREMENTS: dict[str, tuple[str, ...]] = { "dingtalk": ("client_id", "client_secret"), "wechat": ("bot_token",), "wecom": ("bot_id", "bot_secret"), + "buzz": ("relay_url", "private_key"), } diff --git a/backend/docs/IM_CHANNEL_CONNECTIONS.md b/backend/docs/IM_CHANNEL_CONNECTIONS.md index c49803894..a7c64ad6e 100644 --- a/backend/docs/IM_CHANNEL_CONNECTIONS.md +++ b/backend/docs/IM_CHANNEL_CONNECTIONS.md @@ -1,6 +1,6 @@ # IM Channel Connections -DeerFlow supports user-owned IM channel bindings for Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, and WeCom. The feature reuses the existing `channels.*` runtime configuration, so it works in local and private deployments with the same outbound transports already supported by DeerFlow. +DeerFlow supports user-owned IM channel bindings for Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, WeCom, and Buzz. The feature reuses the existing `channels.*` runtime configuration, so it works in local and private deployments with the same outbound transports already supported by DeerFlow. No public IP, OAuth callback URL, or provider webhook is required in this implementation. @@ -293,6 +293,11 @@ channels: enabled: true bot_id: $WECOM_BOT_ID bot_secret: $WECOM_BOT_SECRET + + buzz: + enabled: true + relay_url: wss://buzz.example.com + private_key: $BUZZ_PRIVATE_KEY # hex or nsec1… ``` Then enable user bindings in `channel_connections`: @@ -327,6 +332,9 @@ channel_connections: wecom: enabled: true + + buzz: + enabled: true ``` `channel_connections` does not duplicate provider secrets. It only controls the browser-facing connect UI and stores per-user binding records. Telegram needs `bot_username` only so the frontend can open a deep link. @@ -361,6 +369,57 @@ Feishu/Lark, DingTalk, WeChat, and WeCom: - The UI shows `Send /connect to the DeerFlow bot.` - The already-running long-connection or polling worker receives the message and binds the platform user/workspace identity to the current DeerFlow user. +Buzz: + +- Unlike the bot/app credentials above, Buzz has no separate developer console: DeerFlow joins the relay as an ordinary member identity. Generate a Nostr keypair for that identity — everything below refers to its **hex public key**. +- **Onboarding takes two separate steps, and both are required.** Relay membership and channel membership are different things, and doing only the first produces a connector that connects and authenticates cleanly while receiving nothing: + + 1. **Register the pubkey as a relay member** — `buzz-admin add-member --pubkey `. This is what lets the identity authenticate (NIP-42) and publish at all. + 2. **Add it to each channel it should participate in** — `buzz channels add-member --channel --pubkey --role bot`. Chat events are only delivered to a channel's members, and the relay additionally **rejects** any message that `p`-mentions a non-member with `mentioned pubkeys are not channel members` — so without this step the connector can neither hear a mention nor answer one. + + See the [Buzz project](https://github.com/block/buzz) for the admin tooling. +- **Channels are auto-discovered — you do not list them in `config.yaml`.** On every connection the connector asks the relay which channels this identity belongs to and subscribes to each one individually. Adding it to a new channel later takes effect **live**, without a restart or reconnect: the relay sends a membership notification and the connector starts listening immediately (and stops listening when it is removed). If you see `channel discovery returned no channels` in the logs, step 2 above has not been done. +- Configure `relay_url` and `private_key` (hex or `nsec1…`) under `channels.buzz`, then enable `channel_connections.buzz`. +- The frontend creates a short one-time code. +- The UI shows `Send /connect to the DeerFlow Buzz bot.` +- The already-running Buzz relay-loop worker receives the message — sent as a DM or an @mention in a channel both parties belong to — and binds the sender's Nostr pubkey to the current DeerFlow user. +- Requires the `buzz` dependency extra (`uv sync --extra buzz`) for the `coincurve` library. `scripts/detect_uv_extras.py` (and Docker/production builds via `backend/Dockerfile`) auto-detect and preserve this extra when `channels.buzz.enabled: true` in `config.yaml`, the same way the `browser` extra is auto-detected for `browser_navigate`. + +### Buzz subscription model + +Buzz's relay only delivers chat events to **channel-scoped** subscriptions, which is why the connector's subscriptions look the way they do. A global `REQ {"kinds":[9]}` is accepted and answered with `EOSE`, but no chat event is ever fanned out to it, and a single subscription cannot cover several channels either (a multi-value `#h` matches nothing). So, on **every** connection, after NIP-42 auth completes: + +| Subscription | Filter | Purpose | +|---|---|---| +| `buzz-discovery` | `{"kinds":[39000]}` | Historical query listing exactly the channels this identity is a member of (one stored event each, then `EOSE`). Supplies each channel's name and type, which is also what the DM mention-exemption reads. Do **not** narrow it with `#p` — that matches nothing. | +| `buzz-membership` | `{"kinds":[44100,44101], "#p":[""], "since": …}` | **Live** membership notifications. `44100` (added) subscribes to the new channel immediately; `44101` (removed) closes that channel's subscription. This is what makes a newly added channel work without a restart. | +| `buzz-chat-` | `{"kinds":[9], "#h":[""], "since": …}` | One per discovered channel — the only shape that actually receives messages. | + +Consequences worth knowing operationally: + +- **Replay is tracked per channel.** Each channel carries its own `since` watermark, advanced only by events DeerFlow actually processed. A single shared watermark would let a busy channel drag the cursor past a quiet channel's unread messages and skip them after a reconnect; per-channel cursors can only ever cost duplicate delivery (which the manager's inbound dedupe absorbs), never a miss. +- **Membership is scoped to live events.** The relay *stores* 44100/44101 events, so without a `since` every connection replayed the whole membership history as if it had just happened — re-running channel discovery once per stored add (you would see several `channel discovery complete` lines for one connect, and channels logged as ``), re-subscribing channels you have since been removed from, and briefly unsubscribing channels you are still in. The subscription is therefore anchored at the moment the socket opened, minus 60s of slack so a membership change made *during* the connect/auth handshake — or a small relay clock skew — is still picked up. +- **The number of channel subscriptions is capped** (256). The channel list comes off the wire, so it is bounded like any other remote-fed state. At the cap, new channels are refused and named in a `per-channel subscription limit reached` warning rather than an existing, working subscription being evicted. +- **A subscription the relay closes is re-opened, up to 3 times per connection.** Every subscription on the socket fails *silently* when the relay drops it: a chat subscription deafens one channel, `buzz-membership` stops DeerFlow ever learning it was added to or removed from a channel, `buzz-discovery` kills the completeness sweep. So a `CLOSED` frame is recovered, not just noted, and the subscription that went quiet is always named at WARNING level. Recovery is skipped when the relay's stated reason says the subscription is not ours any more — a NIP-01/NIP-42 `auth-required:` / `restricted:` / `blocked:` / `invalid:` prefix, or buzz-relay's own revocation wording — because re-issuing the same REQ then just fights the relay. Any other reason (including a `CLOSED` with no reason at all) is treated as a hiccup and retried, with the 3-attempt budget as the backstop; after that it stays down until the next reconnect, which rebuilds everything from scratch. +- **Known bound: more than 2000 unread messages in one channel across a disconnect loses the oldest of them.** The relay caps historical delivery at 2000 events per subscription and serves them newest-first, even with a `since`. DeerFlow processes what it receives and the channel's watermark advances past the rest, so those older messages are never delivered and never retried. Every other gap in the design fails toward duplicate delivery (which is absorbed by inbound dedupe); this is the one remaining case that can skip, and it needs both a disconnect and a >2000-message backlog in a *single* channel to occur. + +### Buzz trust model + +On a team-run Buzz relay the relay operator is not necessarily the DeerFlow operator, so be precise about what the connector proves and what it takes on trust: + +**Verified (cryptographically, on every inbound event):** DeerFlow recomputes each event's NIP-01 id from the delivered payload and verifies its BIP-340 Schnorr signature against the claimed `pubkey` before the event can influence anything. A relay therefore cannot rewrite a member's message, replay one author's signature onto another payload, or claim an allowlisted author it does not hold the key for. This applies to `/connect` binds as well as ordinary chat, so a relay cannot bind someone else's pubkey to an attacker's DeerFlow account. Events that fail verification are dropped with a warning. + +**Trusted (not verified):** the *authorship* of kind-39000 channel metadata. Buzz publishes channel discovery events from the relay's own keypair, but nothing already configured identifies that key (`relay_url` is a network address, not a signing key), so DeerFlow only proves such an event was signed by *some* member. Because channel discovery and subscription are now driven by exactly these events, a forged kind-39000 has two effects, not one: + +1. It can mark a channel `type: "dm"`, which relaxes the `require_mention` requirement for that channel. +2. It can make DeerFlow **open a chat subscription** for a channel of the forger's choosing, since the set of channels DeerFlow listens to is the set it holds metadata for. + +Neither can make anything be *acted on*. The `allowed_users` allowlist and per-event signature verification are independent gates: an author who is not allowlisted is dropped regardless of channel type or how the subscription was opened. The blast radius of (2) is a relay reading its own traffic back to a subscriber that ignores it, bounded by the 256-subscription cap (which refuses new subscriptions rather than evicting working ones, so an induced subscription cannot displace a real channel). The same applies to a forged kind-44100 membership notification, except that its `p` tag is re-checked locally, so it must at least name this identity. If you need the mention requirement to be unforgeable on a relay whose members you do not all trust, keep those channels out of `mention_free_channels` and treat DM detection as convenience rather than a boundary. + +**Deny-by-default allowlist:** unlike other providers (where an empty `allowed_users` means "allow everyone"), `channels.buzz.allowed_users` is deliberately deny-by-default — an empty list means *nobody* can trigger a run, and DeerFlow logs a startup warning saying so. Add each member pubkey (hex or `npub1…`) that should be able to reach the agent. Individual drops are logged at DEBUG level. + +**Bound identity:** once a pubkey completes `/connect`, its inbound messages resolve to that connection and run under the bound DeerFlow user (memory, files, and artifacts land in that user's buckets). Bindings are scoped to the relay host, so the same pubkey on a different relay is a different identity and must bind separately. + Codes use 128 bits of randomness, expire after 10 minutes, and are single-use. For providers with an `allowed_users` allowlist (Telegram, Slack, DingTalk, WeChat, …), a valid `/connect ` (or Telegram `/start `) is consumed **before** the allowlist is checked. This is intentional: a user who is not yet on the allowlist — and whose platform identity the bot has therefore never seen — can still complete their first browser-initiated bind. After binding, `allowed_users` continues to gate ordinary (non-bind) messages as before. diff --git a/backend/packages/harness/deerflow/config/channel_connections_config.py b/backend/packages/harness/deerflow/config/channel_connections_config.py index 7e740b7af..164e463f5 100644 --- a/backend/packages/harness/deerflow/config/channel_connections_config.py +++ b/backend/packages/harness/deerflow/config/channel_connections_config.py @@ -50,6 +50,7 @@ class ChannelConnectionsConfig(BaseModel): dingtalk: BindingCodeChannelConnectionConfig = Field(default_factory=BindingCodeChannelConnectionConfig) wechat: BindingCodeChannelConnectionConfig = Field(default_factory=BindingCodeChannelConnectionConfig) wecom: BindingCodeChannelConnectionConfig = Field(default_factory=BindingCodeChannelConnectionConfig) + buzz: BindingCodeChannelConnectionConfig = Field(default_factory=BindingCodeChannelConnectionConfig) def provider_status(self, provider: str) -> dict[str, bool]: config = getattr(self, provider, None) diff --git a/backend/pyproject.toml b/backend/pyproject.toml index e47f41813..991ac85af 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -28,6 +28,7 @@ dependencies = [ postgres = ["deerflow-harness[postgres]"] redis = ["deerflow-harness[redis]"] discord = ["discord.py>=2.7.0"] +buzz = ["coincurve>=20.0.0"] monocle = ["deerflow-harness[monocle]"] browser = ["deerflow-harness[browser]"] memory-zh = ["deerflow-harness[memory-zh]"] @@ -69,9 +70,13 @@ index-url = "https://pypi.org/simple" # breaking change is requiring Python >=3.10 (we require >=3.12), the sdk # only imports `websockets.asyncio.client`/`websockets.exceptions` (both # 16-compatible), and DeerFlow never uses the sdk's WebSocket transport -# (httpx/SSE only). Pin the exact pre-upgrade 16.0 for the IM channel -# integrations (dingtalk-stream, python-telegram-bot, etc.) that ran on it -# before. Remove once langgraph-sdk relaxes the pin upstream. Note: enabling +# (httpx/SSE only). DeerFlow does have one direct consumer of its own: the Buzz +# channel (`app/channels/buzz.py`) imports the top-level `websockets` package and +# calls `websockets.connect()`, which in 16.0 is the same asyncio client the sdk +# uses, re-exported at the package root -- so it is covered by the same +# compatibility argument. Pin the exact pre-upgrade 16.0 for that plus the IM +# channel integrations (dingtalk-stream, python-telegram-bot, etc.) that ran on +# it before. Remove once langgraph-sdk relaxes the pin upstream. Note: enabling # the `openai[realtime]` or `slack-sdk[optional]` extras would conflict (they # also cap websockets<16). override-dependencies = ["websockets==16.0"] diff --git a/backend/tests/test_buzz_channel.py b/backend/tests/test_buzz_channel.py new file mode 100644 index 000000000..d00f01c58 --- /dev/null +++ b/backend/tests/test_buzz_channel.py @@ -0,0 +1,2015 @@ +"""Tests for the Buzz (Nostr) channel connector.""" + +import asyncio +import json +import logging +import time +from unittest.mock import AsyncMock + +import pytest + +pytest.importorskip("coincurve") + +from app.channels import buzz_nostr +from app.channels.base import Channel +from app.channels.buzz import EDIT_MAX_BYTES, MAX_CACHED_CHANNELS, MAX_CHANNEL_SUBSCRIPTIONS, MAX_RESUBSCRIBE_ATTEMPTS, MEMBERSHIP_LOOKBACK_SECONDS, BuzzChannel, _chunk_text +from app.channels.manager import CHANNEL_CAPABILITIES +from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage +from app.channels.run_policy import CHANNEL_RUN_POLICY +from app.channels.service import _CHANNEL_CREDENTIAL_KEYS, _CHANNEL_REGISTRY + +SK3_HEX = "0000000000000000000000000000000000000000000000000000000000000003" +PK3_HEX = "f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9" + +# Review FINDING 4 made inbound events signature-verified, so test authors can no +# longer be arbitrary hex strings ("dd" * 32 has no private key and therefore no +# signature): every fixture author is now a real keypair whose events are signed +# exactly the way a relay member's client would sign them. +SK_OWNER = "0000000000000000000000000000000000000000000000000000000000000005" +SK_OUTSIDER = "0000000000000000000000000000000000000000000000000000000000000007" +SK_NEWCOMER = "0000000000000000000000000000000000000000000000000000000000000009" +# The relay's own keypair: it signs kind-39000 channel discovery events and the +# kind-44100/44101 membership notifications (buzz-relay's `state.relay_keypair`). +SK_RELAY = "000000000000000000000000000000000000000000000000000000000000000b" +OWNER = buzz_nostr.parse_private_key(SK_OWNER).pubkey_hex # the one allowlisted author +OUTSIDER = buzz_nostr.parse_private_key(SK_OUTSIDER).pubkey_hex # relay member, not allowlisted +NEWCOMER = buzz_nostr.parse_private_key(SK_NEWCOMER).pubkey_hex # not allowlisted; binds via /connect +CHANNEL = "136852ee-63e1-49c2-8927-413b5ee8e5f7" +CHANNEL_B = "5e0209b6-2d67-5a2e-894c-9ea597f17202" + + +def _channel(**overrides) -> BuzzChannel: + config = {"relay_url": "wss://buzz.example.com", "private_key": SK3_HEX, "allowed_users": [OWNER], **overrides} + return BuzzChannel(bus=MessageBus(), config=config) + + +def test_registered_in_framework_registries(): + assert _CHANNEL_REGISTRY["buzz"] == "app.channels.buzz:BuzzChannel" + assert _CHANNEL_CREDENTIAL_KEYS["buzz"] == ["private_key"] + assert CHANNEL_CAPABILITIES["buzz"] == {"supports_streaming": True} + policy = CHANNEL_RUN_POLICY["buzz"] + assert policy.serialize_thread_runs is True and policy.requires_bound_identity is False + + +def test_is_a_channel_named_buzz_with_streaming(): + ch = _channel() + assert isinstance(ch, Channel) and ch.name == "buzz" and ch.supports_streaming is True + + +def test_config_parsing_normalizes_allowlist_and_defaults(): + ch = _channel(allowed_users=[OWNER.upper()], mention_free_channels=[CHANNEL]) + assert ch._allowed_users == {OWNER} + assert ch._require_mention is True and ch._mention_free == {CHANNEL} + assert ch._relay_url == "wss://buzz.example.com" + + +def test_config_rejects_non_websocket_relay_url(): + with pytest.raises(ValueError): + _channel(relay_url="https://buzz.example.com") + + +def test_start_and_stop_manage_outbound_subscription(): + async def run(): + ch = _channel() + ch._spawn_connection = lambda: None # skeleton: no real socket in tests + await ch.start() + assert ch.is_running and ch.bus._outbound_listeners == [ch._on_outbound] + await ch.stop() + assert not ch.is_running and ch.bus._outbound_listeners == [] + + asyncio.run(run()) + + +def test_start_is_idempotent_against_double_start(): + """A second start() while already running must not double-subscribe or re-spawn. + + Reproduces the review finding that calling start() twice appended + `_on_outbound` to `bus._outbound_listeners` twice and spawned a second + concurrent relay-loop task, since the original skeleton had no + re-entrancy guard (unlike github.py / discord.py's `if self._running: + return`). + """ + + async def run(): + ch = _channel() + spawn_calls = 0 + + def fake_spawn() -> None: + nonlocal spawn_calls + spawn_calls += 1 + + ch._spawn_connection = fake_spawn + + await ch.start() + await ch.start() # must be a no-op: already running + + assert ch.bus._outbound_listeners == [ch._on_outbound] + assert spawn_calls == 1 + + asyncio.run(run()) + + +def test_stop_is_safe_after_the_relay_task_already_crashed(): + """stop() must not re-raise a stored exception from an already-finished task. + + Reproduces the review finding that a later stop() awaited the finished relay + task and only caught CancelledError/TimeoutError, so any other stored + exception propagated out of stop() -- leaving `_task` non-None and skipping + the "stopped" log. + + Updated for Task 6: this originally reproduced the crash via the Task-3 + skeleton's `_run_loop` stub, which raised `NotImplementedError` on its very + first statement. Task 6 replaces that stub with a real reconnect-forever loop + that, by design, does NOT crash on an ordinary connection error -- it backs + off and retries instead (see test_run_loop_reconnects_after_connection_failure). + Real network I/O is also not an option here: with no `_connect` seam configured, + the real `_run_loop` would call `websockets.connect()` against this test's fake + `wss://buzz.example.com` relay URL, which is real (slow, sandboxing-dependent) + network I/O that must never run inside a unit test. So the "already crashed" + scenario is now reproduced by monkeypatching `_run_loop` itself to simulate a + hypothetical future bug there, while still exercising the REAL + (non-monkeypatched) `_spawn_connection` -> real asyncio task path this test + is about. + """ + + async def run(): + ch = _channel() + + async def crashing_run_loop() -> None: + raise RuntimeError("simulated _run_loop bug") + + ch._run_loop = crashing_run_loop + await ch.start() # real _spawn_connection: creates a real asyncio task + + # Give the event loop a chance to run the (monkeypatched) _run_loop to + # completion (it raises on its very first statement, so one or two + # scheduling turns are enough). + for _ in range(10): + if ch._task is not None and ch._task.done(): + break + await asyncio.sleep(0) + assert ch._task is not None and ch._task.done() + + await ch.stop() # must not raise the task's stored RuntimeError + + assert not ch.is_running + assert ch._task is None + + asyncio.run(run()) + + +def _event(*, sk=SK_OWNER, kind=9, content="@DeerFlow hello", channel=CHANNEL, mentions=(PK3_HEX,), reply_to=None, created_at=1700000100): + """Build a REAL relay event: signed by *sk*, with the id the relay would see. + + Since FINDING 4 the connector verifies both, so a hand-built dict with a made-up + id and no `sig` is now indistinguishable from a relay-forged event -- and is + dropped as one. Fixtures must therefore be authentic.""" + tags = [["h", channel]] + if reply_to: + tags.append(["e", reply_to]) + tags.extend(["p", m] for m in mentions) + return buzz_nostr.sign_event(buzz_nostr.parse_private_key(sk), kind, tags, content, created_at) + + +def _meta_event(channel=CHANNEL, *, name="general", channel_type="stream", sk=SK_RELAY, created_at=1700000060): + """A real kind-39000 channel-discovery event, shaped exactly like buzz-relay's. + + Verified against the live relay: `d` = channel uuid, `name` = channel name, + `t` = channel type (`stream` / `dm`), signed by the relay keypair.""" + return buzz_nostr.sign_event(buzz_nostr.parse_private_key(sk), buzz_nostr.KIND_CHANNEL_META, [["d", channel], ["name", name], ["t", channel_type]], "", created_at) + + +def _membership_event(kind, channel=CHANNEL, *, target=PK3_HEX, sk=SK_RELAY, created_at=1700000070): + """A real kind-44100/44101 membership notification, shaped like buzz-relay's. + + Verified against the live relay: `p` = the affected member's pubkey, `h` = the + channel uuid, content = a JSON blob naming the actor. Relay-signed.""" + event_type = "member_added" if kind == buzz_nostr.KIND_MEMBER_ADDED else "member_removed" + content = json.dumps({"type": event_type, "channel_id": channel, "actor": OWNER}) + return buzz_nostr.sign_event(buzz_nostr.parse_private_key(sk), kind, [["p", target], ["h", channel]], content, created_at) + + +def _started(**overrides): + ch = _channel(**overrides) + ch._keys = buzz_nostr.parse_private_key(SK3_HEX) + captured = [] + + async def publish(msg): + captured.append(msg) + + ch._publish = publish + return ch, captured + + +def _dispatch(ch, ev): + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "sub1", ev]))) + + +def test_mentioned_allowed_author_is_published(): + ch, captured = _started() + ev = _event() + _dispatch(ch, ev) + assert len(captured) == 1 + msg = captured[0] + assert msg.channel_name == "buzz" and msg.chat_id == CHANNEL and msg.user_id == OWNER + assert msg.text == "hello" # own leading @mention stripped + assert msg.metadata["event_id"] == ev["id"] and msg.workspace_id == "buzz.example.com" + assert msg.msg_type == InboundMessageType.CHAT + + +def test_disallowed_author_is_dropped(): + ch, captured = _started() + _dispatch(ch, _event(sk=SK_OUTSIDER)) + assert captured == [] + + +def test_own_events_are_ignored(): + ch, captured = _started() + _dispatch(ch, _event(sk=SK3_HEX)) + assert captured == [] + + +def test_unmentioned_channel_message_is_dropped_but_dm_passes(): + ch, captured = _started() + _dispatch(ch, _event(content="no mention here", mentions=())) + assert captured == [] + ch._handle_meta_event({"kind": 39000, "tags": [["d", CHANNEL], ["t", "dm"], ["name", "DM"]]}) + _dispatch(ch, _event(content="dm without mention", mentions=())) + assert len(captured) == 1 and captured[0].text == "dm without mention" + + +def test_mention_free_channel_and_thread_follow_pass_without_mention(): + ch, captured = _started(mention_free_channels=[CHANNEL]) + _dispatch(ch, _event(content="open channel", mentions=())) + assert len(captured) == 1 + + class FakeStore: + def get_thread_id(self, channel_name, chat_id, topic_id=None): + return "thread-1" if topic_id == "aa" * 32 else None + + ch2, captured2 = _started() + ch2.config["channel_store"] = FakeStore() + _dispatch(ch2, _event(content="follow-up", mentions=(), reply_to="aa" * 32)) + assert len(captured2) == 1 and captured2[0].topic_id == "aa" * 32 + + +def test_thread_replies_map_topic_and_requester_and_watermark(): + ch, captured = _started() + _dispatch(ch, _event(reply_to="aa" * 32, created_at=1700000200)) + assert captured[0].topic_id == "aa" * 32 and captured[0].thread_ts == "aa" * 32 + assert ch._last_requester[(CHANNEL, "aa" * 32)] == OWNER + assert ch._seen_created_at[CHANNEL] == 1700000200 + + +def test_auth_frame_records_challenge_and_meta_defaults_fail_closed(): + ch, captured = _started() + asyncio.run(ch.handle_relay_frame('["AUTH","challenge-xyz"]')) + assert ch._pending_auth_challenge == "challenge-xyz" + # unknown channel type == not a DM -> unmentioned message still dropped + _dispatch(ch, _event(channel="99999999-9999-4999-8999-999999999999", content="x", mentions=())) + assert captured == [] + + +# -- Task 4 review fixes ----------------------------------------------------- + + +class FakeConnectionRepo: + """Minimal test double for ChannelConnectionRepository: consume, upsert, and lookup. + + The lookup half was added for the review finding that the `/connect` bind was + write-only, so it stores what `upsert_connection` wrote under the same + (provider, external_account_id, workspace_id) identity the real repository keys + on -- a lookup with the wrong workspace therefore misses, exactly as it would + against SQL. + """ + + def __init__(self, *, states=None, raise_on_consume=False): + self._states = dict(states or {}) + self._raise_on_consume = raise_on_consume + self._connections: dict[tuple, dict] = {} + self.upserts = [] + self.lookups = [] + + async def consume_oauth_state(self, *, provider, state, now=None): + if self._raise_on_consume: + raise RuntimeError("boom") + owner_user_id = self._states.pop(state, None) + if owner_user_id is None: + return None + return {"owner_user_id": owner_user_id, "provider": provider, "requested_scopes": [], "metadata": {}, "redirect_after": None} + + async def upsert_connection(self, **kwargs): + self.upserts.append(kwargs) + connection = {"id": f"conn-{len(self.upserts)}", **kwargs} + self._connections[(kwargs["provider"], kwargs["external_account_id"], kwargs.get("workspace_id"))] = connection + return connection + + async def find_connection_by_external_identity(self, *, provider, external_account_id, workspace_id=None): + self.lookups.append({"provider": provider, "external_account_id": external_account_id, "workspace_id": workspace_id}) + return self._connections.get((provider, external_account_id, workspace_id)) + + +def test_connect_code_binds_and_never_publishes_even_for_unauthorized_author(): + """FINDING 1 (Critical): a valid /connect code from a non-allowlisted pubkey + must bind via the connection repo and never reach _publish as a chat message.""" + repo = FakeConnectionRepo(states={"tok-1": "owner-xyz"}) + ch, captured = _started(connection_repo=repo) + _dispatch(ch, _event(sk=SK_OUTSIDER, content="/connect tok-1", mentions=())) + assert captured == [] + assert len(repo.upserts) == 1 + assert repo.upserts[0]["owner_user_id"] == "owner-xyz" + assert repo.upserts[0]["external_account_id"] == OUTSIDER + assert repo.upserts[0]["provider"] == "buzz" + + +def test_connect_code_invalid_never_publishes_and_does_not_bind(): + """An unrecognized/expired code must still never publish, and must not upsert.""" + repo = FakeConnectionRepo() + ch, captured = _started(connection_repo=repo) + _dispatch(ch, _event(sk=SK_OUTSIDER, content="/connect not-a-real-code", mentions=())) + assert captured == [] + assert repo.upserts == [] + + +def test_connect_code_repo_error_never_publishes_and_does_not_crash(): + """A connection-repo failure while binding must be swallowed, not crash the read loop, + and must still never fall through to publish.""" + repo = FakeConnectionRepo(raise_on_consume=True) + ch, captured = _started(connection_repo=repo) + _dispatch(ch, _event(sk=SK_OUTSIDER, content="/connect tok-1", mentions=())) + assert captured == [] + + +def test_connect_code_never_publishes_even_for_already_allowed_and_mentioned_author(): + """Ruling: a /connect message must NEVER reach _publish, valid code or not -- + even when the author is already allowlisted and mentioned.""" + repo = FakeConnectionRepo(states={"tok-2": "owner-abc"}) + ch, captured = _started(connection_repo=repo) + _dispatch(ch, _event(sk=SK_OWNER, content="/connect tok-2", mentions=(PK3_HEX,))) + assert captured == [] + assert len(repo.upserts) == 1 and repo.upserts[0]["external_account_id"] == OWNER + + +def test_known_command_classifies_as_command_but_plain_text_stays_chat(): + """FINDING 3: is_known_channel_command classification had no direct coverage.""" + ch, captured = _started() + _dispatch(ch, _event(content="@DeerFlow /goal ship it", mentions=(PK3_HEX,))) + assert len(captured) == 1 + assert captured[0].msg_type == InboundMessageType.COMMAND + assert captured[0].text == "/goal ship it" + + _dispatch(ch, _event(content="@DeerFlow just chatting", mentions=(PK3_HEX,))) + assert len(captured) == 2 + assert captured[1].msg_type == InboundMessageType.CHAT + + +def test_strip_own_mention_leaves_ambiguous_multi_mention_text_untouched(): + """FINDING 4: "@Alice, @DeerFlow help" must not have Alice's mention dropped + just because our own mention also appears in the message.""" + ch, captured = _started() + _dispatch(ch, _event(content="@Alice, @DeerFlow help", mentions=(PK3_HEX,))) + assert len(captured) == 1 + assert captured[0].text == "@Alice, @DeerFlow help" + + +# -- Task 5: outbound — placeholder post, streaming edits, final, oversize split --- + + +class FakeTransport: + def __init__(self): + self.sent = [] + + async def send(self, text): + self.sent.append(json.loads(text)) + + +def _outbound(ch, text, *, is_final, thread_ts=None): + return OutboundMessage(channel_name="buzz", chat_id=CHANNEL, thread_id="t1", text=text, is_final=is_final, thread_ts=thread_ts) + + +def _events_of(transport): + return [f[1] for f in transport.sent if f[0] == "EVENT"] + + +def test_streaming_posts_placeholder_then_edits_then_final(): + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + ch._last_requester[(CHANNEL, None)] = OWNER + asyncio.run(ch.send(_outbound(ch, "Working…", is_final=False))) + asyncio.run(ch.send(_outbound(ch, "Working… more", is_final=False))) + asyncio.run(ch.send(_outbound(ch, "Final answer", is_final=True))) + events = _events_of(transport) + assert [e["kind"] for e in events] == [9, 40003, 40003] + placeholder = events[0] + assert ["p", OWNER] in placeholder["tags"] # requester notified on the initial post + assert all(["e", placeholder["id"]] in e["tags"] for e in events[1:]) + assert events[-1]["content"] == "Final answer" + assert (CHANNEL, None) not in ch._stream_targets # final clears the target + + +def test_thread_reply_targets_thread_root(): + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + root = "aa" * 32 + asyncio.run(ch.send(_outbound(ch, "reply", is_final=True, thread_ts=root))) + (ev,) = _events_of(transport) + assert ev["kind"] == 9 and ["e", root] in ev["tags"] + + +def test_oversized_final_splits_into_followup_posts(): + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + big = "x" * 130_000 # > 2 * EDIT_MAX_BYTES + asyncio.run(ch.send(_outbound(ch, big, is_final=True))) + events = _events_of(transport) + assert events[0]["kind"] == 9 and all(e["kind"] == 9 for e in events[1:]) + assert "".join(e["content"] for e in events) == big + assert all(len(e["content"].encode()) <= 60_000 for e in events) + + +def test_send_without_transport_raises_for_retry(monkeypatch): + # Correction vs. the brief's literal snippet: _send_with_retry sleeps 2**attempt + # seconds between its 3 attempts on a real failure, which would otherwise burn + # ~3s of wall-clock time here for no benefit (this is a pure failure-path test). + # Patching the shared retry helper's sleep call keeps it instant without + # touching BuzzChannel/_send_with_retry itself. + monkeypatch.setattr("app.channels.base.asyncio.sleep", AsyncMock()) + ch, _ = _started() + with pytest.raises(RuntimeError): + asyncio.run(ch.send(_outbound(ch, "hi", is_final=True))) + + +def test_chunk_text_never_splits_a_multibyte_character_across_chunks(): + """Correctness requirement beyond the brief's (ASCII-only) oversize test: + splitting is by ENCODED BYTE LENGTH and must never cut a multi-byte UTF-8 + character in half. Uses a 4-byte-wide character (an emoji outside the BMP) + with a limit that is deliberately NOT a multiple of 4, so a naive + text.encode()[:limit]-style byte slice would corrupt a character; the + real chunker must not.""" + text = "\U0001f600" * 50 # grinning-face emoji: 4 bytes each in UTF-8 + limit = 61 + chunks = _chunk_text(text, limit=limit) + assert "".join(chunks) == text + assert all(len(c.encode()) <= limit for c in chunks) + # A split-mid-character chunk could never have a byte length that is an + # exact multiple of the (uniform) 4-byte character width. + assert all(len(c.encode()) % 4 == 0 for c in chunks) + assert chunks[0] == "\U0001f600" * 15 # 15*4=60 <= 61 bytes; a 16th char would make 64 > 61 + + +def test_edit_failure_degrades_to_fresh_post_and_retargets_stream(monkeypatch): + """Correctness requirement beyond the brief's literal tests: if an edit fails + after retries, BuzzChannel must degrade by posting a fresh message rather than + losing the content, and must retarget _stream_targets at the new message so + later edits in the same conversation land on it instead of the abandoned one.""" + monkeypatch.setattr("app.channels.base.asyncio.sleep", AsyncMock()) + ch, _ = _started() + + class FailingEditTransport: + def __init__(self): + self.sent = [] + + async def send(self, text): + frame = json.loads(text) + if frame[0] == "EVENT" and frame[1]["kind"] == 40003: + raise RuntimeError("relay rejected edit") + self.sent.append(frame) + + transport = FailingEditTransport() + ch._transport = transport + + asyncio.run(ch.send(_outbound(ch, "placeholder", is_final=False))) + asyncio.run(ch.send(_outbound(ch, "update that fails to edit", is_final=False))) + + events = [f[1] for f in transport.sent if f[0] == "EVENT"] + assert [e["kind"] for e in events] == [9, 9] # placeholder + degraded fresh post, no successful edit + assert events[1]["content"] == "update that fails to edit" + assert ch._stream_targets[(CHANNEL, None)] == events[1]["id"] # retargeted to the new message + + +# -- Review finding: _stream_targets must not leak when a FINAL send() raises ----- + + +def test_stream_target_cleared_even_when_final_send_fails(monkeypatch): + """FINDING (Important/spec): the edit->degrade path can raise (both the edit + AND the degraded fresh-post attempts exhaust their retries), which used to + propagate out of send() BEFORE reaching the `if msg.is_final: pop` at the end + -- leaking a stale _stream_targets entry. This must hold regardless of + success/failure: (a) send() must still raise (the framework's retry/error + path in Channel._on_outbound must still see the failure), and (b) the stale + target must not survive, so the NEXT send() for this conversation starts a + fresh placeholder (kind 9) rather than editing the abandoned one (kind + 40003) once the relay recovers.""" + monkeypatch.setattr("app.channels.base.asyncio.sleep", AsyncMock()) + ch, _ = _started() + + class AlwaysFailingTransport: + async def send(self, text): + raise RuntimeError("relay down") + + ch._transport = AlwaysFailingTransport() + + # Seed an existing placeholder target, as a prior successful non-final + # send() would have, so this final call takes the edit (not "first post") + # branch -- the exact branch the finding calls out. + key = (CHANNEL, None) + ch._stream_targets[key] = "ff" * 32 + + with pytest.raises(RuntimeError): + asyncio.run(ch.send(_outbound(ch, "final answer", is_final=True))) + + assert key not in ch._stream_targets # no stale/partial target survives a raised final send + + # Once the relay recovers, the next send() for the same conversation must + # start a fresh placeholder, not an edit targeting the abandoned message. + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.send(_outbound(ch, "retry after recovery", is_final=True))) + (ev,) = _events_of(transport) + assert ev["kind"] == 9 + + +def test_stream_target_cleared_when_overflow_chunk_send_fails(monkeypatch): + """Second FINDING regression case, pinning the other raise site named in the + review: the overflow-chunk loop can also raise (a later chunk fails after + retries even though the first chunk already succeeded). _stream_targets must + still be cleared for this key -- not left pointing at the successfully-sent + first chunk -- and the exception must still propagate.""" + monkeypatch.setattr("app.channels.base.asyncio.sleep", AsyncMock()) + ch, _ = _started() + + class FailsAfterFirstFrame: + def __init__(self): + self.sent = [] + + async def send(self, text): + frame = json.loads(text) + if self.sent: # the first frame succeeds; every one after raises + raise RuntimeError("relay dropped mid-stream") + self.sent.append(frame) + + transport = FailsAfterFirstFrame() + ch._transport = transport + key = (CHANNEL, None) + big = "x" * 130_000 # forces at least one follow-up overflow chunk + + with pytest.raises(RuntimeError): + asyncio.run(ch.send(_outbound(ch, big, is_final=True))) + + assert key not in ch._stream_targets + + +# -- Task 4 carry-forward: /connect must reply, and a failed reply send must ------ +# -- never be reported as a failed bind ------------------------------------------- + + +def test_connect_success_sends_confirmation_reply(): + repo = FakeConnectionRepo(states={"tok-conf": "owner-conf"}) + ch, captured = _started(connection_repo=repo) + transport = FakeTransport() + ch._transport = transport + _dispatch(ch, _event(sk=SK_NEWCOMER, content="/connect tok-conf", mentions=())) + events = _events_of(transport) + assert len(events) == 1 + assert events[0]["kind"] == 9 + assert events[0]["content"] == "Buzz connected to DeerFlow." + assert ["p", NEWCOMER] in events[0]["tags"] + assert captured == [] # still never published as a chat message + + +def test_connect_invalid_code_sends_error_reply(): + repo = FakeConnectionRepo() + ch, captured = _started(connection_repo=repo) + transport = FakeTransport() + ch._transport = transport + _dispatch(ch, _event(sk=SK_NEWCOMER, content="/connect not-a-real-code", mentions=())) + (event,) = _events_of(transport) + assert event["content"] == "Buzz connection code is invalid or expired." + + +def test_connect_repo_error_sends_error_reply_and_does_not_crash(): + repo = FakeConnectionRepo(raise_on_consume=True) + ch, captured = _started(connection_repo=repo) + transport = FakeTransport() + ch._transport = transport + _dispatch(ch, _event(sk=SK_NEWCOMER, content="/connect tok-1", mentions=())) + (event,) = _events_of(transport) + assert event["content"] == "Buzz connection could not be completed from this message." + + +def test_connect_success_survives_a_failed_confirmation_send(caplog): + """CRITICAL (Task 4 review carry-forward pitfall): a failure to SEND the + confirmation for an otherwise-successful bind must never be reported or + logged as a bind failure. ch._transport is left unset (None) so the reply + attempt raises, but the bind itself (consume_oauth_state + upsert_connection) + must still go through, and only a distinctly-worded send-failure warning may + be logged -- never "failed to bind".""" + repo = FakeConnectionRepo(states={"tok-fail-send": "owner-fail-send"}) + ch, captured = _started(connection_repo=repo) + with caplog.at_level(logging.INFO, logger="app.channels.buzz"): + _dispatch(ch, _event(sk=SK_NEWCOMER, content="/connect tok-fail-send", mentions=())) + + assert len(repo.upserts) == 1 # the bind itself succeeded despite the failed reply + assert repo.upserts[0]["owner_user_id"] == "owner-fail-send" + assert captured == [] + assert "failed to bind" not in caplog.text + assert "failed to send" in caplog.text + + +# -- Task 6: relay connection loop -- connect, NIP-42 auth, subscribe, reconnect -- + + +class ScriptedWS: + """Async-iterable fake websocket: yields scripted frames, records sends.""" + + def __init__(self, frames): + self.frames = list(frames) + self.sent = [] + + async def send(self, text): + self.sent.append(json.loads(text)) + + def __aiter__(self): + return self + + async def __anext__(self): + if not self.frames: + raise StopAsyncIteration + return self.frames.pop(0) + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + +def test_session_authenticates_then_subscribes_per_discovered_channel_with_since_cursor(): + """Rewritten for the live-test finding (see the per-channel section below). + + The original version of this test asserted the buggy shape: a GLOBAL + `{"kinds":[9], "since": ...}` filter with no `#h`. The relay accepts that REQ + and answers EOSE, but never fans a single chat event out to it, so the + connector connected, authenticated, and stayed permanently silent. The + assertions are otherwise unchanged in strength -- NIP-42 auth still happens, + a kind-39000 discovery subscription is still opened, and the `since` + watermark still rides the chat filter -- but that chat filter must now be + `#h`-scoped to a channel we actually discovered. + """ + ch, _ = _started() + ch._seen_created_at[CHANNEL] = 1700000500 + ws = ScriptedWS(['["AUTH","challenge-1"]', json.dumps(["EVENT", "buzz-discovery", _meta_event()]), '["EOSE","buzz-discovery"]']) + + asyncio.run(ch._session(ws)) + auth_frames = [f for f in ws.sent if f[0] == "AUTH"] + req_frames = [f for f in ws.sent if f[0] == "REQ"] + assert len(auth_frames) == 1 and auth_frames[0][1]["kind"] == 22242 + assert ["challenge", "challenge-1"] in auth_frames[0][1]["tags"] + assert req_frames, "expected a REQ subscription" + filters = [f0 for f in req_frames for f0 in f[2:] if isinstance(f0, dict)] + assert any(39000 in f.get("kinds", []) for f in filters) + assert any(9 in f.get("kinds", []) and f.get("#h") == [CHANNEL] and f.get("since") == 1700000500 for f in filters) + # ... and never the global kind-9 filter the relay silently ignores. + assert not [f for f in filters if 9 in f.get("kinds", []) and not f.get("#h")] + + +def test_session_routes_events_through_handle_relay_frame(): + ch, captured = _started() + ws = ScriptedWS([json.dumps(["EVENT", "s", _event()])]) + asyncio.run(ch._session(ws)) + assert len(captured) == 1 + + +def test_run_loop_reconnects_after_connection_failure(): + """After a connect failure, _run_loop must back off and retry rather than giving up. + + Correction vs. the brief's literal mock setup: patching `app.channels.buzz.asyncio.sleep` + with a bare `unittest.mock.AsyncMock()` patches the *shared* `asyncio` module object (since + `buzz.py` does `import asyncio`, not `from asyncio import sleep`), so it also silently + replaces the `asyncio.sleep(0)` this test's own polling loop relies on to yield control + back to the event loop. A bare AsyncMock's returned coroutine has no real suspension point, + so awaiting it never actually hands control back to the scheduler -- verified empirically + (see task report) two ways: (a) with the polling loop's own `asyncio.sleep(0)` calls also + silently mocked out, the `_run_loop` task never gets scheduled even once, so `attempts` + stays empty and the test fails outright; (b) if only the polling loop is protected (e.g. by + capturing a `real_sleep` reference before patching) while `_run_loop`'s internal backoff + `await asyncio.sleep(delay)` remains a non-yielding mock, `_run_loop` can retry in a genuine + infinite tight loop with zero suspension points anywhere in its call chain (mocked connect, + trivial ScriptedWS stubs, non-yielding sleep) -- this reproducibly hung the interpreter at + 100% CPU in manual verification and had to be killed. The fix keeps the mock's call-count + bookkeeping (`slept.await_count`) but gives its `side_effect` a genuine zero-duration + `asyncio.sleep(0)` (captured before patching, so it cannot recursively call itself), + so every backoff still really yields to the loop -- never a real multi-second delay, but + never a non-yielding busy spin either. `asyncio.wait_for(..., timeout=10)` is an outer, + real-wall-clock safety bound so a future regression here fails fast instead of hanging CI. + """ + ch, _ = _started() + attempts = [] + + def make_connect(): + async def connect(): + attempts.append(1) + if len(attempts) == 1: + raise ConnectionError("boom") + return ScriptedWS([json.dumps(["EVENT", "s", _event()])]) + + return connect + + ch._connect = make_connect() + + async def run(): + import unittest.mock + + real_sleep = asyncio.sleep # captured before patching: used by the mock's side_effect + + async def instant_yield(*_args, **_kwargs): + await real_sleep(0) # a genuine, zero-duration event-loop tick -- never real seconds + + with unittest.mock.patch("app.channels.buzz.asyncio.sleep", new=unittest.mock.AsyncMock(side_effect=instant_yield)) as slept: + task = asyncio.get_running_loop().create_task(ch._run_loop()) + for _ in range(200): + await asyncio.sleep(0) + if len(attempts) >= 2 and task.done() is False and not ch._task: + break + if len(attempts) >= 2: + break + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + assert slept.await_count >= 1 # backed off after the failure + + asyncio.run(asyncio.wait_for(run(), timeout=10)) + assert len(attempts) >= 2 + + +def test_spawn_connection_cannot_fail_start_even_if_connect_immediately_errors(): + """Carried-forward invariant (from an earlier review, resolved in this task): in start(), + `_running = True` is set AFTER `subscribe_outbound()` and `_spawn_connection()`. That was + only safe while `_spawn_connection` was a bare `create_task(...)` call that could not itself + raise. Task 6 gives `_run_loop` real, potentially-failing connect logic, so this pins that + the invariant still holds: `_spawn_connection` remains non-fallible because it still only + calls `asyncio.create_task(self._run_loop(), ...)`, which schedules the coroutine and + returns without running any of its body -- a connect failure happens later, inside the + spawned task, never synchronously inside start(). So even a connect that fails on its very + first attempt cannot leave start() partially applied (outbound listener subscribed but + `_running` still False, which would make the guarded stop() silently no-op and leak both). + """ + + async def run(): + import unittest.mock + + ch = _channel() + + async def immediately_failing_connect(): + raise RuntimeError("boom-on-first-connect") + + ch._connect = immediately_failing_connect + + with unittest.mock.patch("app.channels.buzz.asyncio.sleep", new=unittest.mock.AsyncMock()): + await ch.start() # must fully commit even though the spawned relay loop will + # immediately hit immediately_failing_connect the first time it gets scheduled + assert ch.is_running + assert ch.bus._outbound_listeners == [ch._on_outbound] + assert ch._task is not None + + await ch.stop() # must cleanly unwind: no leaked listener/task either + + assert not ch.is_running + assert ch.bus._outbound_listeners == [] + assert ch._task is None + + asyncio.run(run()) + + +# -- Task 7: channel_connections config + browser provider wiring ----------------- + + +def test_channel_connections_config_knows_buzz(): + """Correction vs. the brief's literal expected value: `provider_status()` + computes `configured = enabled and bool(config.configured)`, so a disabled + provider is always reported as `configured: False` -- confirmed by every + sibling provider in test_channel_connections_config.py:: + test_provider_status_reports_disabled_and_unknown_providers (all + {"enabled": False, "configured": False} on a default/disabled config). + Buzz's `BindingCodeChannelConnectionConfig` (always-True `configured` + property, same as discord/feishu/dingtalk/wechat/wecom) is therefore + indistinguishable from its siblings here until it is also enabled.""" + from deerflow.config.channel_connections_config import ChannelConnectionsConfig + + cfg = ChannelConnectionsConfig() + assert cfg.provider_status("buzz") == {"enabled": False, "configured": False} + + enabled_cfg = ChannelConnectionsConfig.model_validate({"enabled": True, "buzz": {"enabled": True}}) + assert enabled_cfg.provider_status("buzz") == {"enabled": True, "configured": True} + + +def test_browser_provider_wiring_for_buzz(): + from app.gateway.routers import channel_connections as cc + + assert cc._PROVIDER_META["buzz"] == {"display_name": "Buzz", "auth_mode": "binding_code"} + assert {f["name"] for f in cc._CREDENTIAL_FIELDS["buzz"]} == {"relay_url", "private_key"} + assert cc._RUNTIME_REQUIREMENTS["buzz"] == ("relay_url", "private_key") + + +# -- FINAL REVIEW FINDING 1 (Critical): the resubscribe cursor is peer-controlled -- + + +def test_future_dated_event_from_a_dropped_author_never_moves_the_cursor(): + """Reproduces the remote DoS: any relay member -- allowlisted or not -- publishes + one kind-9 event stamped year-5138. The event is correctly dropped by the + allowlist, but the watermark used to be advanced BEFORE that gate, so every + later REQ carried `since=99999999999999` and the connector went permanently + deaf on this and every future connection, with no log and no recovery short of + a process restart.""" + ch, captured = _started() + _dispatch(ch, _event(sk=SK_OUTSIDER, created_at=99999999999999)) + assert captured == [] + assert ch._seen_created_at == {} + assert "since" not in ch._chat_filter(CHANNEL) + + +def test_future_dated_event_from_an_allowlisted_author_is_delivered_but_capped(): + """The clamp is independent of the allowlist: an authorized member with a badly + skewed (or deliberately absurd) clock still gets their message through, but must + not be able to blind the connector's next reconnect either.""" + ch, captured = _started() + _dispatch(ch, _event(created_at=99999999999999)) + assert len(captured) == 1 + assert ch._seen_created_at == {} + + +def test_dropped_event_with_a_plausible_timestamp_also_leaves_the_cursor_alone(): + """Advance-on-accept, not advance-on-receive: a non-allowlisted member's ordinary + message must not silently move the cursor past events we would have accepted.""" + ch, captured = _started() + _dispatch(ch, _event(sk=SK_OUTSIDER, created_at=1700000400)) + assert captured == [] + assert ch._seen_created_at == {} + + +def test_accepted_event_advances_the_cursor_and_rides_the_next_subscription(): + ch, captured = _started() + _dispatch(ch, _event(created_at=1700000300)) + assert len(captured) == 1 + assert ch._seen_created_at[CHANNEL] == 1700000300 + assert ch._chat_filter(CHANNEL)["since"] == 1700000300 + + +def test_a_handled_connect_advances_the_cursor_but_a_forged_one_does_not(): + """A fully processed /connect may advance the cursor (otherwise every reconnect + replays it and answers with a spurious "code invalid or expired"), while an event + that never gets processed at all must not.""" + repo = FakeConnectionRepo(states={"tok-cursor": "owner-cursor"}) + ch, _ = _started(connection_repo=repo) + ch._transport = FakeTransport() + _dispatch(ch, _event(sk=SK_NEWCOMER, content="/connect tok-cursor", mentions=(), created_at=1700000250)) + assert ch._seen_created_at[CHANNEL] == 1700000250 + + forged = _event(sk=SK_OUTSIDER, content="/connect tok-cursor", mentions=(), created_at=1700000600) + forged["content"] = "/connect tok-cursor-tampered" # breaks the signature + _dispatch(ch, forged) + assert ch._seen_created_at[CHANNEL] == 1700000250 + + +# -- FINAL REVIEW FINDING 2 (Important): /connect binds were never resolved inbound -- + + +def test_bound_pubkey_inbound_message_carries_the_connection_identity(): + """Without `attach_connection_identity` the whole browser-connections feature was + inert for Buzz: `connection_id`/`owner_user_id` stayed None, so the manager ran the + turn under a synthetic pubkey-derived user (its own memory + file buckets) instead + of the bound DeerFlow account, and DELETE /api/channels/connections/{id} had no + runtime effect.""" + repo = FakeConnectionRepo(states={"tok-bind": "owner-bound"}) + ch, captured = _started(connection_repo=repo) + ch._transport = FakeTransport() + + _dispatch(ch, _event(sk=SK_OWNER, content="/connect tok-bind", mentions=())) + assert len(repo.upserts) == 1 + + _dispatch(ch, _event(sk=SK_OWNER)) + assert len(captured) == 1 + assert captured[0].connection_id == "conn-1" + assert captured[0].owner_user_id == "owner-bound" + assert captured[0].workspace_id == "buzz.example.com" + # Scoped to this relay: a bind written for buzz.example.com must be looked up the + # same way, and never through a workspace-less fallback that would resolve a + # pubkey bound on some *other* relay. + assert repo.lookups[-1] == {"provider": "buzz", "external_account_id": OWNER, "workspace_id": "buzz.example.com"} + + +def test_unbound_pubkey_inbound_message_carries_no_connection_identity(): + repo = FakeConnectionRepo() + ch, captured = _started(connection_repo=repo) + _dispatch(ch, _event()) + assert len(captured) == 1 + assert captured[0].connection_id is None and captured[0].owner_user_id is None + assert captured[0].workspace_id == "buzz.example.com" + assert [lookup["workspace_id"] for lookup in repo.lookups] == ["buzz.example.com"] + + +# -- FINAL REVIEW FINDING 3 (Important): oversize streaming re-posted the tail ------ + + +def _visible_conversation(transport): + """Replay posts + edits the way a Buzz client renders them: id -> current text.""" + visible, order = {}, [] + for ev in _events_of(transport): + if ev["kind"] == 9: + visible[ev["id"]] = ev["content"] + order.append(ev["id"]) + else: + target = next(t[1] for t in ev["tags"] if t[0] == "e") + visible[target] = ev["content"] + return [visible[eid] for eid in order] + + +def test_successive_oversize_updates_edit_the_tail_instead_of_reposting_it(): + """The manager publishes CUMULATIVE text on every streaming update, so an oversize + reply re-splits into >= 2 chunks on EVERY update. `chunks[1:]` used to be posted as + brand-new, untracked kind-9 messages each time -- a realistic 100KB answer at + ~1 update/sec flooded the channel with dozens of near-duplicate tails. Posts must + now be bounded by the number of distinct chunk INDEXES ever needed.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + updates = ["x" * (EDIT_MAX_BYTES + 1_000), "y" * (EDIT_MAX_BYTES + 20_000), "z" * (2 * EDIT_MAX_BYTES + 1_000)] + assert [len(_chunk_text(t)) for t in updates] == [2, 2, 3] # the shape this test is about + for index, text in enumerate(updates): + asyncio.run(ch.send(_outbound(ch, text, is_final=index == len(updates) - 1))) + + events = _events_of(transport) + posts = [e for e in events if e["kind"] == 9] + edits = [e for e in events if e["kind"] == 40003] + assert len(posts) == 3 # one per distinct chunk index (was 5: a fresh tail per update) + assert len(edits) == 4 # chunk 0 twice, tail 0 twice + assert "".join(_visible_conversation(transport)) == updates[-1] + assert (CHANNEL, None) not in ch._stream_tails # final clears the tail bookkeeping too + + +def test_repeated_oversize_updates_of_a_stable_size_post_nothing_new(): + """The flood's worst case: N updates that never grow past two chunks must produce + exactly two messages in the channel, no matter how many updates arrive.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + def update(i): + return f"u{i}" * (EDIT_MAX_BYTES // 2 + 500) # 61_000 bytes: always exactly 2 chunks + + for i in range(6): + asyncio.run(ch.send(_outbound(ch, update(i), is_final=i == 5))) + + posts = [e for e in _events_of(transport) if e["kind"] == 9] + assert len(posts) == 2 # was 7: one placeholder plus a fresh tail on every update + assert "".join(_visible_conversation(transport)) == update(5) + + +def test_tail_bookkeeping_is_cleared_even_when_a_final_oversize_send_fails(monkeypatch): + """Same invariant the placeholder already had: a raised FINAL send must not leave + tail ids behind for the next run to edit.""" + monkeypatch.setattr("app.channels.base.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + key = (CHANNEL, None) + asyncio.run(ch.send(_outbound(ch, "a" * (EDIT_MAX_BYTES + 1_000), is_final=False))) + assert len(ch._stream_tails[key]) == 1 + + class AlwaysFailingTransport: + async def send(self, text): + raise RuntimeError("relay down") + + ch._transport = AlwaysFailingTransport() + with pytest.raises(RuntimeError): + asyncio.run(ch.send(_outbound(ch, "b" * (EDIT_MAX_BYTES + 1_000), is_final=True))) + assert key not in ch._stream_tails and key not in ch._stream_targets + + +# -- FINAL REVIEW FINDING 4 (Important): events were never signature-verified ------- + + +def test_event_with_a_tampered_payload_never_reaches_publish(): + ch, captured = _started() + ev = _event() + ev["content"] = "@DeerFlow rm -rf /mnt/user-data" # rewritten in flight by the relay + _dispatch(ch, ev) + assert captured == [] + + +def test_relay_cannot_forge_an_allowlisted_author(): + """`ev["pubkey"]` is the authorization principal, and on a team-run relay the relay + operator is not necessarily the DeerFlow operator: an unverified pubkey field let a + malicious relay name an allowlisted author and trigger tool-executing runs.""" + ch, captured = _started() + ev = _event(sk=SK_OUTSIDER) # genuinely signed by a non-allowlisted member + ev["pubkey"] = OWNER # ... but delivered claiming the allowlisted one + _dispatch(ch, ev) + assert captured == [] + + +def test_relay_cannot_forge_a_connect_event_to_bind_another_members_pubkey(): + """The bind path consumes `pubkey` too: forging it would bind a victim's identity to + the attacker's DeerFlow account, so verification has to happen before /connect.""" + repo = FakeConnectionRepo(states={"tok-forge": "attacker"}) + ch, captured = _started(connection_repo=repo) + ev = _event(sk=SK_OUTSIDER, content="/connect tok-forge", mentions=()) + ev["pubkey"] = OWNER + _dispatch(ch, ev) + assert repo.upserts == [] and captured == [] + + +def test_unsigned_channel_metadata_is_rejected_before_it_can_relax_the_mention_gate(): + ch, captured = _started() + fake_meta = {"id": "aa" * 32, "pubkey": OUTSIDER, "created_at": 1700000000, "kind": 39000, "tags": [["d", CHANNEL], ["t", "dm"]], "content": "", "sig": "00" * 64} + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "s", fake_meta]))) + assert ch._channel_meta == {} + _dispatch(ch, _event(content="dm without mention", mentions=())) + assert captured == [] + + +def test_signed_channel_metadata_is_still_cached_and_still_grants_the_dm_exemption(): + """Verification must not break the legitimate path it guards.""" + ch, captured = _started() + meta = buzz_nostr.sign_event(buzz_nostr.parse_private_key(SK_OUTSIDER), 39000, [["d", CHANNEL], ["t", "dm"], ["name", "Ops DM"]], "", 1700000050) + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "s", meta]))) + assert ch._channel_meta[CHANNEL] == {"type": "dm", "name": "Ops DM"} + _dispatch(ch, _event(content="dm without mention", mentions=())) + assert len(captured) == 1 + + +# -- FINAL REVIEW FINDING 5: operability + bounded remote-fed state ---------------- + + +def test_empty_allowlist_warns_at_startup(caplog): + """Buzz keeps deny-by-default semantics (siblings treat empty as allow-all), so a + misconfigured operator must not be left with a silently dead channel.""" + + async def run(): + ch = _channel(allowed_users=[]) + ch._spawn_connection = lambda: None + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + await ch.start() + await ch.stop() + + asyncio.run(run()) + assert "allowed_users is empty" in caplog.text + + +def test_allowlist_drop_is_logged_at_debug(caplog): + ch, captured = _started() + with caplog.at_level(logging.DEBUG, logger="app.channels.buzz"): + _dispatch(ch, _event(sk=SK_OUTSIDER)) + assert captured == [] + assert "non-allowlisted" in caplog.text + + +def test_stop_is_bounded_and_coherent_when_the_relay_loop_ignores_cancellation(monkeypatch, caplog): + """`asyncio.wait_for` waits for the cancelled task to actually finish, so a loop that + swallows CancelledError hung stop() forever instead of timing out; and the timeout + path dropped `_task` while the task might still own `_transport`, leaving an + abandoned task posting on a socket the channel believed it had released.""" + monkeypatch.setattr("app.channels.buzz.STOP_TIMEOUT_SECONDS", 0.01) + + async def run(): + ch = _channel() + started = asyncio.Event() + + async def wedged_run_loop(): + ch._transport = FakeTransport() + started.set() + for _ in range(20): # bounded (~0.2s) so the test can never wedge the suite + try: + await asyncio.sleep(0.01) + except asyncio.CancelledError: + pass # deliberately swallow the cancel stop() sends + + ch._run_loop = wedged_run_loop + await ch.start() + task = ch._task + await asyncio.wait_for(started.wait(), timeout=5) + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + await asyncio.wait_for(ch.stop(), timeout=5) + + assert ch._task is None and ch._transport is None + await asyncio.wait_for(task, timeout=5) # let the abandoned loop retire cleanly + + asyncio.run(run()) + assert "did not finish" in caplog.text + + +def test_stop_clears_per_connection_state_so_a_restart_cannot_edit_stale_placeholders(): + async def run(): + ch = _channel() + ch._spawn_connection = lambda: None + await ch.start() + ch._stream_targets[(CHANNEL, None)] = "aa" * 32 + ch._stream_tails[(CHANNEL, None)] = ["bb" * 32] + ch._last_requester[(CHANNEL, None)] = OWNER + ch._channel_meta[CHANNEL] = {"type": "dm", "name": "x"} + ch._pending_auth_challenge = "challenge" + ch._pending_auth_event_id = "auth-event-id" + + await ch.stop() + + assert not ch._stream_targets and not ch._stream_tails and not ch._last_requester + assert not ch._channel_meta and ch._pending_auth_challenge is None + assert ch._pending_auth_event_id is None + + asyncio.run(run()) + + +def test_channel_metadata_cache_is_bounded_against_remote_feeding(): + """Any relay member can publish kind-39000 events, so the cache is remote-fed and + must not grow for the process lifetime one forged `d` tag at a time.""" + ch, _ = _started() + for i in range(MAX_CACHED_CHANNELS + 25): + ch._handle_meta_event({"kind": 39000, "tags": [["d", f"chan-{i}"], ["t", "stream"]]}) + assert len(ch._channel_meta) == MAX_CACHED_CHANNELS + assert "chan-0" not in ch._channel_meta # oldest evicted first + assert f"chan-{MAX_CACHED_CHANNELS + 24}" in ch._channel_meta + + +# -- LIVE-TEST FINDING: the relay only fans out to channel-scoped subscriptions ---- +# +# Proved against a real Buzz relay (wss://buzz.atg.one), with a second identity +# publishing into a channel the connector is a member of, while both subscription +# shapes were open on the same authenticated socket: +# +# REQ {"kinds":[9]} -> accepted, EOSE, and ZERO live events +# REQ {"kinds":[9], "#h":[uuid]} -> the event arrives +# REQ {"kinds":[9], "#h":[uuid-a,uuid-b]}-> ZERO events (so one REQ per channel) +# +# The connector therefore has to discover its channels (historical kind-39000 REQ) +# and open ONE chat subscription per channel, and learn about channels it is added +# to afterwards from the relay-signed kind-44100/44101 membership notifications. + + +def _req_filters(transport): + """Every filter object across every REQ frame the connector sent.""" + return [f for frame in transport.sent if frame[0] == "REQ" for f in frame[2:] if isinstance(f, dict)] + + +def _reqs_by_sub(transport): + return {frame[1]: [f for f in frame[2:] if isinstance(f, dict)] for frame in transport.sent if frame[0] == "REQ"} + + +def _closed_subs(transport): + return [frame[1] for frame in transport.sent if frame[0] == "CLOSE"] + + +def test_chat_subscription_is_opened_per_channel_and_never_globally(): + """THE BUG: one global `{"kinds":[9]}` filter matched nothing the relay fans out. + + Fails against the pre-fix connector, which sent exactly one un-scoped kind-9 + filter for the whole connection and no per-channel REQ at all.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL, name="home-network")]))) + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL_B, name="general")]))) + + by_sub = _reqs_by_sub(transport) + assert by_sub[f"buzz-chat-{CHANNEL}"] == [{"kinds": [9], "#h": [CHANNEL]}] + assert by_sub[f"buzz-chat-{CHANNEL_B}"] == [{"kinds": [9], "#h": [CHANNEL_B]}] + # No global kind-9 filter, and no multi-value `#h` (the relay drops both). + for f in _req_filters(transport): + if 9 in f.get("kinds", []): + assert len(f.get("#h", [])) == 1, f + + +def test_discovery_events_populate_metadata_and_drive_chat_subscriptions(): + """Discovery does double duty: it is the DM-detection cache AND the channel list.""" + ch, captured = _started() + transport = FakeTransport() + ch._transport = transport + + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL, name="Ops DM", channel_type="dm")]))) + + assert ch._channel_meta[CHANNEL] == {"type": "dm", "name": "Ops DM"} + assert f"buzz-chat-{CHANNEL}" in _reqs_by_sub(transport) + # ... and the cached type still relaxes the mention gate, as before. + _dispatch(ch, _event(content="dm without mention", mentions=())) + assert len(captured) == 1 + + +def test_repeated_metadata_for_a_known_channel_does_not_resubscribe(): + """kind-39000 is addressable and re-emitted on every channel edit; one REQ is enough.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + for _ in range(4): + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + assert len([frame for frame in transport.sent if frame[0] == "REQ"]) == 1 + + +def test_discovery_eose_resubscribes_any_channel_whose_req_failed(): + """The discovery EOSE is the completeness barrier: by then every discovered + channel must have a live chat subscription, including one whose REQ lost a race + with a flaky socket.""" + ch, _ = _started() + + class DropsFirstSend: + def __init__(self): + self.sent = [] + self.failed = False + + async def send(self, text): + if not self.failed: + self.failed = True + raise RuntimeError("relay hiccup") + self.sent.append(json.loads(text)) + + transport = DropsFirstSend() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + assert _reqs_by_sub(transport) == {} # the REQ was lost, and must not be remembered as open + + asyncio.run(ch.handle_relay_frame('["EOSE","buzz-discovery"]')) + assert f"buzz-chat-{CHANNEL}" in _reqs_by_sub(transport) + + +def test_membership_notification_subscribes_to_a_new_channel_without_a_reconnect(): + """kind-44100 for OUR pubkey is how the connector learns it was added to a channel + mid-connection. It must subscribe immediately -- a channel that only starts + working after the next reconnect is the same silent failure in slow motion.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-membership", _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL_B)]))) + + by_sub = _reqs_by_sub(transport) + assert by_sub[f"buzz-chat-{CHANNEL_B}"] == [{"kinds": [9], "#h": [CHANNEL_B]}] + # ... and the discovery subscription is re-issued so the new channel's name/type + # (which drives DM detection) is refreshed rather than staying unknown. + assert by_sub["buzz-discovery"] == [{"kinds": [39000]}] + + +def test_membership_notification_for_another_member_is_ignored(): + """The membership subscription is `#p`-filtered, but the filter is the relay's + claim; someone else's add must not make us subscribe to their channel.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-membership", _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL_B, target=OUTSIDER)]))) + assert transport.sent == [] + + +def test_member_removed_closes_only_that_channels_subscription(): + """kind-44101: stop listening to that channel, keep every other subscription on + the same socket (this is what `close_frame` exists for).""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL_B, name="general")]))) + + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-membership", _membership_event(buzz_nostr.KIND_MEMBER_REMOVED, CHANNEL)]))) + + assert _closed_subs(transport) == [f"buzz-chat-{CHANNEL}"] + assert ch._chat_subscriptions == {CHANNEL_B} + assert CHANNEL not in ch._channel_meta # stale metadata must not survive the removal + + # Being re-added later must work on the same connection. + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-membership", _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL)]))) + assert ch._chat_subscriptions == {CHANNEL, CHANNEL_B} + + +def test_relay_closing_a_chat_subscription_is_forgotten_so_it_can_be_reopened(): + """buzz-relay CLOSEs a channel's subscription when access is revoked (e.g. the + channel is archived). Remembering it as live would make the later 44100 + resubscribe a no-op.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + assert ch._chat_subscriptions == {CHANNEL} + + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", f"buzz-chat-{CHANNEL}", "channel access revoked"]))) + assert ch._chat_subscriptions == set() + + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-membership", _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL)]))) + assert ch._chat_subscriptions == {CHANNEL} + + +def test_watermark_is_per_channel_so_a_quiet_channel_is_never_skipped(): + """A single global `since` is the unsafe direction here: it is the newest event + processed in ANY channel, so a quiet channel's REQ would ask for events newer + than a busy channel's traffic and silently skip everything published in the + quiet one while we were disconnected.""" + ch, captured = _started(mention_free_channels=[CHANNEL, CHANNEL_B]) + _dispatch(ch, _event(channel=CHANNEL, content="busy", mentions=(), created_at=1700000900)) + _dispatch(ch, _event(channel=CHANNEL_B, content="quiet", mentions=(), created_at=1700000100)) + assert len(captured) == 2 + + assert ch._chat_filter(CHANNEL)["since"] == 1700000900 + assert ch._chat_filter(CHANNEL_B)["since"] == 1700000100 # not dragged forward by the busy channel + + +def test_watermark_map_is_bounded_against_remote_feeding(): + """Channel ids come from remote `h` tags, so the cursor map is remote-fed too. + Eviction only ever costs replay (the relay's default backlog), never a skip.""" + ch, _ = _started() + for i in range(MAX_CACHED_CHANNELS + 25): + ch._advance_watermark(f"chan-{i}", 1700000000 + i) + assert len(ch._seen_created_at) == MAX_CACHED_CHANNELS + assert "chan-0" not in ch._seen_created_at + assert f"chan-{MAX_CACHED_CHANNELS + 24}" in ch._seen_created_at + + +def test_chat_subscription_count_is_bounded(caplog): + """One REQ per channel means the subscription count is driven by remote-fed + channel metadata; it must be capped rather than tracking it without limit.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + for i in range(MAX_CHANNEL_SUBSCRIPTIONS + 5): + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(f"chan-{i}")]))) + assert len(ch._chat_subscriptions) == MAX_CHANNEL_SUBSCRIPTIONS + assert "chan-0" in ch._chat_subscriptions # a working subscription is never evicted for a new one + assert "subscription limit" in caplog.text + + +def test_session_reestablishes_auth_discovery_and_per_channel_subscriptions_on_reconnect(): + """Every subscription is per-connection state: a reconnect must redo NIP-42 auth, + re-run discovery, and reopen a chat subscription per channel. Before this fix the + reconnect faithfully restored a subscription that received nothing.""" + ch, _ = _started() + + frames = ['["AUTH","challenge-{n}"]', None, '["EOSE","buzz-discovery"]'] + sockets = [] + + def make_socket(n): + ws = ScriptedWS([frames[0].format(n=n), json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]), frames[2]]) + sockets.append(ws) + return ws + + connects = [] + + async def connect(): + connects.append(1) + if len(connects) == 1: + return make_socket(1) + if len(connects) == 2: + return make_socket(2) + raise asyncio.CancelledError() + + ch._connect = connect + + async def run(): + import unittest.mock + + real_sleep = asyncio.sleep + + async def instant_yield(*_a, **_kw): + await real_sleep(0) + + with unittest.mock.patch("app.channels.buzz.asyncio.sleep", new=unittest.mock.AsyncMock(side_effect=instant_yield)): + task = asyncio.get_running_loop().create_task(ch._run_loop()) + for _ in range(500): + await real_sleep(0) + if len(connects) >= 3: + break + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + + started_at = int(time.time()) + asyncio.run(asyncio.wait_for(run(), timeout=10)) + + assert len(sockets) == 2, "expected the relay loop to reconnect" + for ws in sockets: + by_sub = _reqs_by_sub(ws) + assert [f for f in ws.sent if f[0] == "AUTH"], "expected NIP-42 auth on every connection" + assert by_sub["buzz-discovery"] == [{"kinds": [39000]}] + assert by_sub[f"buzz-chat-{CHANNEL}"] == [{"kinds": [9], "#h": [CHANNEL]}] + # Membership is LIVE-only: the kinds/#p shape is unchanged, plus a `since` + # anchored at this connection so the relay's stored membership history is + # never replayed as if it had just happened (resilience FINDING 3). + (membership_filter,) = by_sub["buzz-membership"] + assert membership_filter["kinds"] == [44100, 44101] and membership_filter["#p"] == [PK3_HEX] + assert membership_filter["since"] >= started_at - MEMBERSHIP_LOOKBACK_SECONDS + + +def test_session_end_drops_chat_subscription_bookkeeping(): + """Subscriptions do not survive their socket; remembering them would make the + next connection skip the REQs it must re-send.""" + ch, _ = _started() + ws = ScriptedWS([json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)])]) + asyncio.run(ch._session(ws)) + assert ch._chat_subscriptions == set() + assert ch._transport is None + + +# -- RESILIENCE REVIEW: a CLOSED subscription must not deafen the connector -------- +# +# FINDING 1: a post-auth `CLOSED` for a control subscription (`buzz-discovery` / +# `buzz-membership`) was logged at INFO and never re-issued, so the connector +# silently stopped learning about channels it is added to or removed from for the +# life of that socket. +# FINDING 2: a `CLOSED` for a chat subscription was forgotten with no retry, so a +# transient relay CLOSE deafened exactly one channel for the rest of the +# connection -- invisibly. Nothing re-opened it until a 44100 for that channel or +# a full reconnect. +# FINDING 3: the membership filter carried no `since`, so every connection replayed +# the entire stored membership history: each historical 44100 logged "added to +# channel ...; subscribing" as if it were live and re-ran discovery, producing +# M+1 discovery passes per connect (the live log's TWO "channel discovery +# complete" lines), re-subscribing channels we have since been removed from, and +# letting a historical 44101 transiently drop a channel we ARE still in. + + +def _reqs_for(transport, sub_id): + return [f for f in transport.sent if f[0] == "REQ" and f[1] == sub_id] + + +def test_post_auth_close_of_a_control_subscription_is_reopened(monkeypatch, caplog): + """FINDING 1: the control subscriptions are opened ONLY by + `_open_control_subscriptions`, which runs at session start and in the auth + branch. A relay hiccup that CLOSEs `buzz-membership` after auth therefore ended + membership tracking for the life of the socket -- the connector stops learning + about channels it is added to or removed from, and nothing says so. Same for + `buzz-discovery`, whose death also kills the EOSE sweep and the "no channels" + warning.""" + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-membership", "error: subscription dropped"]))) + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-discovery", ""]))) + + by_sub = _reqs_by_sub(transport) + assert by_sub["buzz-discovery"] == [{"kinds": [39000]}] + (membership_filter,) = by_sub["buzz-membership"] + assert membership_filter["kinds"] == [buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED] + assert membership_filter["#p"] == [PK3_HEX] + # A dead control subscription is an outage, not an INFO-level curiosity. + assert "buzz-membership" in caplog.text and "buzz-discovery" in caplog.text + + +def test_control_resubscription_is_bounded_so_a_closing_relay_is_never_fought_forever(monkeypatch, caplog): + """Re-issuing immediately is a tight loop if the relay keeps closing it, so the + retries are bounded per connection and the exhaustion is loud.""" + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + for _ in range(MAX_RESUBSCRIBE_ATTEMPTS + 4): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-membership", "error: try again"]))) + + assert len(_reqs_for(transport, "buzz-membership")) == MAX_RESUBSCRIBE_ATTEMPTS + assert "gave up" in caplog.text + + +def test_re_auth_restores_the_control_resubscribe_budget(): + """The budget is per connection AND per auth epoch: pre-auth CLOSEDs (which the + auth branch already recovers from wholesale) must not spend the budget that + protects the authenticated session.""" + ch, _ = _started() + ch._resubscribe_attempts["buzz-membership"] = MAX_RESUBSCRIBE_ATTEMPTS + ws = ScriptedWS(['["AUTH","challenge-1"]']) + asyncio.run(ch._session(ws)) + assert ch._resubscribe_attempts == {} + + +def test_transient_close_of_a_chat_subscription_reopens_that_channel(monkeypatch, caplog): + """FINDING 2: the precise failure class this whole fix exists to eliminate -- one + channel goes deaf for the rest of the connection and nothing says so.""" + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + assert ch._chat_subscriptions == {CHANNEL} + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", f"buzz-chat-{CHANNEL}", "error: relay hiccup"]))) + + assert ch._chat_subscriptions == {CHANNEL} # re-opened, not silently dropped + assert len(_reqs_for(transport, f"buzz-chat-{CHANNEL}")) == 2 + assert CHANNEL in caplog.text # the channel going unlistened is named at WARNING + + +@pytest.mark.parametrize( + "reason", + [ + "channel access revoked", + "restricted: you are not a member of this channel", + "auth-required: we can only serve channel members", + "blocked: pubkey is not allowed here", + "invalid: unknown channel", + # A permanent condition wearing a generic, retryable-looking category: the + # NIP-01 prefix is only the category, the actual reason is the remainder. + "error: channel not found", + ], +) +def test_a_legitimate_close_of_a_chat_subscription_is_never_retried(monkeypatch, reason): + """Do not fight the relay: a close that says we were removed, are unauthorized, + or sent an unacceptable filter cannot be fixed by re-issuing the same REQ.""" + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", f"buzz-chat-{CHANNEL}", reason]))) + + assert ch._chat_subscriptions == set() + assert len(_reqs_for(transport, f"buzz-chat-{CHANNEL}")) == 1 # never re-issued + + +def test_chat_resubscription_is_bounded_per_connection(monkeypatch, caplog): + ch, _ = _started() + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + for _ in range(MAX_RESUBSCRIBE_ATTEMPTS + 4): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", f"buzz-chat-{CHANNEL}", "error: flapping"]))) + + # 1 original REQ + exactly MAX_RESUBSCRIBE_ATTEMPTS retries, then it stops. + assert len(_reqs_for(transport, f"buzz-chat-{CHANNEL}")) == 1 + MAX_RESUBSCRIBE_ATTEMPTS + assert ch._chat_subscriptions == set() + assert "gave up" in caplog.text + + +def test_a_close_for_a_channel_we_never_subscribed_to_never_induces_a_subscription(monkeypatch): + """A CLOSED frame is relay-supplied. Recovering one we never opened would let any + relay induce a chat subscription to a channel of its choosing just by naming it.""" + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", f"buzz-chat-{CHANNEL_B}", "error: hiccup"]))) + + assert transport.sent == [] + assert ch._chat_subscriptions == set() + + +def test_membership_filter_is_scoped_to_live_events_so_history_is_never_replayed(): + """FINDING 3: buzz-relay STORES 44100/44101 events and serves history + newest-first (default limit 2000). Without a `since`, every connection replayed + the whole membership history: each stored 44100 logged "added to channel ...; + subscribing" as if it were live, re-subscribed channels we have since been + removed from, and a stored 44101 transiently dropped a channel we ARE still in.""" + ch, _ = _started() + before = int(time.time()) + ws = ScriptedWS([]) + asyncio.run(ch._session(ws)) + after = int(time.time()) + + (frame,) = _reqs_for(ws, "buzz-membership") + (membership_filter,) = [f for f in frame[2:] if isinstance(f, dict)] + assert membership_filter["kinds"] == [buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED] + assert membership_filter["#p"] == [PK3_HEX] + since = membership_filter["since"] + # Anchored at connection time (minus a bounded slack for relay clock skew and + # the handshake window), never at the epoch. + assert before - MEMBERSHIP_LOOKBACK_SECONDS <= since <= after + # ... so nothing stored before this connection -- every fixture event included -- + # can come back as if it were live. + assert since > 1700000070 + + +CHANNEL_STALE = "9f1c4d3a-77b2-4e10-9a55-2c6d8e0b1f34" # we were added to it once, and removed since + + +class StoringRelayWS: + """A fake relay that STORES events and honours `since`, the way buzz-relay does. + + This is what makes the historical-replay symptom reproducible without a live + relay: whether the connector's stored membership history comes back is decided + by the connector's OWN filter, so an unscoped `buzz-membership` REQ replays it + and a `since`-scoped one does not. It is also closed pre-auth (REQs are answered + with `auth-required:` until the NIP-42 handshake completes), which is why the + connector re-runs discovery after authenticating. + """ + + def __init__(self, stored, *, challenge="challenge-1"): + self.stored = list(stored) + self.sent = [] + self.authenticated = False + self._pending = [json.dumps(["AUTH", challenge])] + + @staticmethod + def _matches(ev, filt): + if ev["kind"] not in filt.get("kinds", []): + return False + if "since" in filt and ev["created_at"] < filt["since"]: + return False + return all(set(buzz_nostr.tag_values(ev, key[1:])) & set(wanted) for key, wanted in filt.items() if key.startswith("#")) + + async def send(self, text): + frame = json.loads(text) + self.sent.append(frame) + if frame[0] == "AUTH": + self.authenticated = True + return + if frame[0] != "REQ": + return + sub_id, filters = frame[1], [f for f in frame[2:] if isinstance(f, dict)] + if not self.authenticated: + self._pending.append(json.dumps(["CLOSED", sub_id, "auth-required: we only serve authenticated members"])) + return + self._pending.extend(json.dumps(["EVENT", sub_id, ev]) for ev in self.stored if any(self._matches(ev, f) for f in filters)) + self._pending.append(json.dumps(["EOSE", sub_id])) + + def __aiter__(self): + return self + + async def __anext__(self): + if not self._pending: + raise StopAsyncIteration + return self._pending.pop(0) + + +def test_one_connect_performs_exactly_one_discovery_pass_and_replays_no_membership_history(caplog): + """Directly pins the duplicate-discovery symptom seen live. + + buzz-relay stores 44100/44101 events, so an unscoped `buzz-membership` filter + replayed the whole membership history on every connect. Each stored 44100 read + as live: it logged "added to channel ...; subscribing", re-subscribed even to + channels we have since been removed from, and re-issued the discovery REQ -- so + one connect produced M+1 discovery passes over N stored kind-39000 events (the + two "channel discovery complete" lines in the live log; the `` channel + came from the 44100 path subscribing before that channel's metadata arrived). + + The relay double here honours `since` exactly as the real one does, so the + connector's own filter is what decides the outcome.""" + ch, _ = _started() + ws = StoringRelayWS( + [ + _meta_event(CHANNEL, name="general"), + _meta_event(CHANNEL_B, name="ops"), + # Stored membership history: one add for a channel we are still in, one + # for a channel we have since been removed from. Both long past. + _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL, created_at=1600000000), + _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL_STALE, created_at=1600000001), + ] + ) + + with caplog.at_level(logging.INFO, logger="app.channels.buzz"): + asyncio.run(ch._session(ws)) + + assert caplog.text.count("channel discovery complete") == 1 + # The pre-auth REQ (which this relay answers with `auth-required:`) and its + # post-auth re-issue -- and no third one from a replayed 44100. + assert len(_reqs_for(ws, "buzz-discovery")) == 2 + # (`_chat_subscriptions` is per-socket and cleared by `_session`'s finally, so the + # wire is what proves each discovered channel was subscribed exactly once.) + for channel_id in (CHANNEL, CHANNEL_B): + assert _reqs_for(ws, f"buzz-chat-{channel_id}") == [["REQ", f"buzz-chat-{channel_id}", {"kinds": [9], "#h": [channel_id]}]] + # A channel we were removed from is never resurrected by its stored add. + assert _reqs_for(ws, f"buzz-chat-{CHANNEL_STALE}") == [] + assert "added to channel" not in caplog.text + assert "" not in caplog.text + + +def test_membership_add_for_an_already_known_channel_does_not_re_run_discovery(): + """Belt and braces for the same symptom: discovery is re-issued only when the new + channel's name/type are actually missing, so a duplicate (or replayed) 44100 + cannot multiply discovery passes.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + baseline = len(_reqs_for(transport, "buzz-discovery")) + + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-membership", _membership_event(buzz_nostr.KIND_MEMBER_ADDED, CHANNEL)]))) + + assert len(_reqs_for(transport, "buzz-discovery")) == baseline + assert ch._chat_subscriptions == {CHANNEL} + + +# -- LIVE-TEST FINDING 2: `auth-required` during bootstrap is not a refusal -------- +# +# The connector opens its control subscriptions immediately (in case the relay +# serves unauthenticated reads), the relay CLOSEs them because NIP-42 auth has +# not happened yet, and the auth branch then legitimately re-opens both. Live: +# +# WARNING [buzz] relay closed control subscription buzz-discovery: auth-required: not authenticated +# WARNING [buzz] not re-opening buzz-discovery: the relay refused it. Channel +# discovery/membership tracking is down until the next reconnect or NIP-42 re-auth. +# ... discovery then completed, every channel was subscribed, and a live 44100 +# for a brand-new channel was picked up seconds later. +# +# So the operator-facing warning stated an outage that demonstrably was not +# happening. Pre-auth `auth-required` is now the expected bootstrap case (quiet, +# and it does not spend the permanent-refusal path or the retry budget); the same +# close AFTER a completed NIP-42 handshake is a genuine problem and stays loud. + + +def test_pre_auth_auth_required_close_is_the_expected_bootstrap_case_not_a_refusal(caplog): + """The exact live sequence: control REQs go out pre-auth, the relay closes + them with `auth-required:`, the AUTH handshake completes, and both control + subscriptions come back. No permanent-refusal warning may be emitted, because + nothing is down.""" + ch, _ = _started() + ws = ScriptedWS( + [ + json.dumps(["CLOSED", "buzz-discovery", "auth-required: not authenticated"]), + json.dumps(["CLOSED", "buzz-membership", "auth-required: not authenticated"]), + '["AUTH","challenge-1"]', + json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]), + '["EOSE","buzz-discovery"]', + ] + ) + + with caplog.at_level(logging.DEBUG, logger="app.channels.buzz"): + asyncio.run(ch._session(ws)) + + # Opened pre-auth, closed by the relay, re-opened wholesale by the auth branch. + assert len(_reqs_for(ws, "buzz-discovery")) == 2 + assert len(_reqs_for(ws, "buzz-membership")) == 2 + assert len(_reqs_for(ws, f"buzz-chat-{CHANNEL}")) == 1 # discovery ran and the channel was subscribed + assert ch._chat_subscriptions == set() # cleared on session end, but it WAS listening + # The lie: nothing was down, so nothing may say it was. + assert "not re-opening" not in caplog.text + assert "discovery/membership tracking is down" not in caplog.text + warnings = [r for r in caplog.records if r.levelno >= logging.WARNING] + assert warnings == [], f"pre-auth bootstrap must be quiet, got: {[r.getMessage() for r in warnings]}" + + +def test_pre_auth_auth_required_close_does_not_spend_the_resubscribe_budget(monkeypatch): + """The retry budget protects the AUTHENTICATED session from a relay that keeps + closing a live subscription. A bootstrap close is recovered wholesale by the + auth branch and must not consume it.""" + monkeypatch.setattr("app.channels.buzz.asyncio.sleep", AsyncMock()) + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + for _ in range(MAX_RESUBSCRIBE_ATTEMPTS + 2): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-discovery", "auth-required: not authenticated"]))) + + assert ch._resubscribe_attempts == {} + assert _reqs_for(transport, "buzz-discovery") == [] # the auth branch re-opens it, not this path + + +def test_post_auth_auth_required_close_is_still_treated_as_serious_and_logged_loudly(caplog): + """After the NIP-42 handshake completed, the relay demanding auth again is a + real problem: the subscription is genuinely down until a fresh AUTH challenge + or a reconnect, and the operator has to be told.""" + ch, _ = _started() + ch._auth_completed = True + transport = FakeTransport() + ch._transport = transport + + with caplog.at_level(logging.DEBUG, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-membership", "auth-required: not authenticated"]))) + + warnings = [r.getMessage() for r in caplog.records if r.levelno >= logging.WARNING] + assert warnings, "a post-auth auth-required close must be loud" + assert any("buzz-membership" in message for message in warnings) + assert _reqs_for(transport, "buzz-membership") == [] # still not fought over; re-auth is the recovery + + +def test_post_auth_permanent_refusal_still_says_tracking_is_down(caplog): + """The truthful case the old wording was borrowed from: a non-auth permanent + refusal really does leave discovery/membership tracking down until reconnect.""" + ch, _ = _started() + ch._auth_completed = True + ch._transport = FakeTransport() + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-discovery", "restricted: not permitted"]))) + + assert "not re-opening" in caplog.text + assert "down until the next reconnect" in caplog.text + + +def test_pre_auth_auth_required_close_of_a_chat_subscription_is_forgotten_so_auth_reopens_it(caplog): + """A chat REQ issued before auth can be closed the same way. It must be dropped + from the live set (so discovery re-opens it) without an UNLISTENED alarm.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.handle_relay_frame(json.dumps(["EVENT", "buzz-discovery", _meta_event(CHANNEL)]))) + assert ch._chat_subscriptions == {CHANNEL} + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", f"buzz-chat-{CHANNEL}", "auth-required: not authenticated"]))) + + assert ch._chat_subscriptions == set() + assert len(_reqs_for(transport, f"buzz-chat-{CHANNEL}")) == 1 # not retried here; auth/discovery re-opens it + assert "UNLISTENED" not in caplog.text + + +def test_auth_completed_flag_is_per_connection(): + """It anchors "pre-auth" to THIS socket: a fresh connection starts unauthenticated + again, and a torn-down channel must not remember it was ever authenticated.""" + + async def run(): + ch, _ = _started() + ch._auth_completed = True + await ch._session(ScriptedWS([])) + assert ch._auth_completed is False + ch._auth_completed = True + await ch._session(ScriptedWS(['["AUTH","challenge-1"]'])) + assert ch._auth_completed is False + + asyncio.run(run()) + + +def test_session_does_not_mark_auth_completed_merely_because_auth_was_sent(): + """Sending the AUTH event is only our half of the NIP-42 handshake. + + Replaces a previous version of this test that pinned the exact bug this fix + removes: it asserted `_auth_completed` became true the instant AUTH was sent, + with no `OK` anywhere in the script. A live relay was observed still + processing an AUTH it had already received -- and closing the just-reopened + control subscriptions with `auth-required:` -- well after the connector had + sent it, so `_auth_completed` must stay false until the relay actually + acknowledges the exact event id (see `test_ok_true_for_our_auth_event_marks_the_connection_authenticated`).""" + ch, _ = _started() + seen = [] + ws = ScriptedWS(['["AUTH","challenge-1"]']) + original = ch._open_control_subscriptions + + async def spy(transport): + seen.append(ch._auth_completed) + await original(transport) + + ch._open_control_subscriptions = spy + asyncio.run(ch._session(ws)) + assert seen == [False, False] # pre-auth open, then the post-AUTH-SENT re-open -- still not ack'd + assert ch._pending_auth_event_id is None # cleared by _session's finally on teardown, ack or not + + +def test_auth_required_close_between_auth_sent_and_relay_ok_is_still_bootstrap(caplog): + """THE LIVE DEFECT this fix removes. + + The relay is still processing our AUTH when it closes the control + subscription the auth branch just re-opened. Before the relay's OK for THIS + auth event arrives, that CLOSED must still read as the ordinary bootstrap + race (quiet, recovered) -- not a post-auth outage. Fails against the pre-fix + connector, which flipped `_auth_completed` true the moment AUTH was SENT, so + this exact sequence misclassified as a genuine post-auth refusal and logged + the live warning: "relay demanded re-authentication for buzz-membership AFTER + this connection completed NIP-42 auth".""" + ch, _ = _started() + ws = ScriptedWS( + [ + '["AUTH","challenge-1"]', + json.dumps(["CLOSED", "buzz-membership", "auth-required: not authenticated"]), + ] + ) + + with caplog.at_level(logging.DEBUG, logger="app.channels.buzz"): + asyncio.run(ch._session(ws)) + + warnings = [r.getMessage() for r in caplog.records if r.levelno >= logging.WARNING] + assert warnings == [], f"must stay quiet before the relay OKs our AUTH, got: {warnings}" + assert "pending NIP-42 auth" in caplog.text + + +def test_ok_true_for_our_auth_event_marks_the_connection_authenticated(): + ch, _ = _started() + ch._pending_auth_event_id = "auth-event-id-1" + asyncio.run(ch.handle_relay_frame(json.dumps(["OK", "auth-event-id-1", True, "welcome"]))) + assert ch._auth_completed is True + assert ch._pending_auth_event_id is None + + +def test_ok_false_for_our_auth_event_is_a_loud_failure_and_does_not_authenticate(caplog): + """A relay-rejected AUTH is a real, actionable problem -- surfaced loudly -- + and must never leave the connection looking authenticated.""" + ch, _ = _started() + ch._pending_auth_event_id = "auth-event-id-1" + + with caplog.at_level(logging.WARNING, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["OK", "auth-event-id-1", False, "bad signature"]))) + + assert ch._auth_completed is False + assert ch._pending_auth_event_id is None + assert "bad signature" in caplog.text + assert "REJECTED" in caplog.text + + +def test_ok_frame_for_an_unrelated_event_does_not_affect_auth_state(): + """OK is sent for every published event (chat posts, edits, AUTH), not only + AUTH. Anything that is not the one outstanding AUTH event id must be a no-op + here -- a rejected chat publish is send()'s own retry-path problem, not this + session's auth bookkeeping.""" + ch, _ = _started() + ch._pending_auth_event_id = "auth-event-id-1" + asyncio.run(ch.handle_relay_frame(json.dumps(["OK", "some-other-event-id", True, ""]))) + assert ch._auth_completed is False + assert ch._pending_auth_event_id == "auth-event-id-1" # untouched + + +def test_auth_required_close_after_relay_ok_true_stays_loud_via_the_real_ok_path(caplog): + """The complementary required case, driven through the real OK handler rather + than a manually-set flag: once the relay has genuinely acknowledged our AUTH, + the same `auth-required:` reason on a later close is a real problem again.""" + ch, _ = _started() + ch._pending_auth_event_id = "auth-event-id-1" + asyncio.run(ch.handle_relay_frame(json.dumps(["OK", "auth-event-id-1", True, ""]))) + assert ch._auth_completed is True + ch._transport = FakeTransport() + + with caplog.at_level(logging.DEBUG, logger="app.channels.buzz"): + asyncio.run(ch.handle_relay_frame(json.dumps(["CLOSED", "buzz-membership", "auth-required: not authenticated"]))) + + warnings = [r.getMessage() for r in caplog.records if r.levelno >= logging.WARNING] + assert any("buzz-membership" in m and "AFTER this connection completed NIP-42 auth" in m for m in warnings), warnings + + +def test_discovery_eose_implicitly_confirms_auth_when_relay_sends_no_ok(): + """Fallback for a relay that never sends an explicit `OK` for AUTH (NIP-42 says + it SHOULD, not that it MUST). Reaching discovery EOSE on a re-opened control + subscription is only possible if the relay actually accepted the session -- + an auth-rejecting relay CLOSEs it instead -- so this is treated as implicit + confirmation rather than leaving `_auth_completed` false (and every later + `auth-required:` close misclassified as bootstrap noise) for the rest of the + connection.""" + ch, _ = _started() + ch._pending_auth_event_id = "auth-event-id-1" + + asyncio.run(ch._on_discovery_complete()) + + assert ch._auth_completed is True + assert ch._pending_auth_event_id is None + + +def test_discovery_eose_without_a_pending_auth_send_does_not_fabricate_confirmation(): + """The fallback must only fire for an AUTH we actually sent and are awaiting an + ack for -- not on every ordinary discovery EOSE (e.g. the very first, pre-auth + one, if the relay happens to allow unauthenticated discovery reads).""" + ch, _ = _started() + assert ch._pending_auth_event_id is None + + asyncio.run(ch._on_discovery_complete()) + + assert ch._auth_completed is False + + +# -- FINDING 1 defense in depth: never publish DeerFlow's hidden model context ----- +# +# The manager-side allowlist (`_accumulate_stream_text`) is the fix. This is the +# second layer, and it lives here rather than in a sibling connector because on +# Buzz a leak is PERMANENT: every streaming update is an immutable public Nostr +# event, so a corrective edit only changes what clients render -- the original +# leaked event stays on the relay forever. + + +@pytest.mark.parametrize( + "leaked", + [ + "\nFacts:\n- [context | 0.70] User interacts with the assistant through the DeerFlow chat channel.\n", + "\n## Conversation summary so far\nprivate\n", + "Sure! Today is 2026-08-01", + ], +) +def test_hidden_context_is_never_posted_to_the_relay(leaked, caplog): + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + + with caplog.at_level(logging.ERROR, logger="app.channels.buzz"): + asyncio.run(ch.send(_outbound(ch, leaked, is_final=False))) + + assert transport.sent == [] + assert "hidden" in caplog.text.lower() + + +def test_a_blocked_final_still_clears_the_stream_bookkeeping(): + """Refusing to publish must not strand the placeholder, or the next run in this + conversation would edit an already-answered message.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.send(_outbound(ch, "working", is_final=False))) + assert (CHANNEL, None) in ch._stream_targets + + asyncio.run(ch.send(_outbound(ch, "leak", is_final=True))) + + assert ch._stream_targets == {} and ch._stream_tails == {} + + +def test_ordinary_replies_that_merely_mention_the_words_are_still_published(): + """The guard keys on DeerFlow's literal hidden-context wrappers, not on prose.""" + ch, _ = _started() + transport = FakeTransport() + ch._transport = transport + asyncio.run(ch.send(_outbound(ch, "I stored that in memory and in the durable context data.", is_final=True))) + assert len(_events_of(transport)) == 1 diff --git a/backend/tests/test_buzz_nostr.py b/backend/tests/test_buzz_nostr.py new file mode 100644 index 000000000..ce99f33e0 --- /dev/null +++ b/backend/tests/test_buzz_nostr.py @@ -0,0 +1,180 @@ +"""Tests for the pure Nostr helpers behind the Buzz channel connector.""" + +import json + +import pytest + +coincurve = pytest.importorskip("coincurve") + +from app.channels import buzz_nostr # noqa: E402 + +SK3_HEX = "0000000000000000000000000000000000000000000000000000000000000003" +SK3_NSEC = "nsec1qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqps52s3re" +PK3_HEX = "f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9" +PK3_NPUB = "npub1lycg5qvjtrp3qjf5f7zl382j9x6nrjz9sdhenvyxq8c3808qxmus6gq266" +CHANNEL = "136852ee-63e1-49c2-8927-413b5ee8e5f7" + + +def test_parse_private_key_hex_derives_xonly_pubkey(): + keys = buzz_nostr.parse_private_key(SK3_HEX) + assert keys.secret == bytes.fromhex(SK3_HEX) + assert keys.pubkey_hex == PK3_HEX + + +def test_parse_private_key_nsec_matches_hex(): + assert buzz_nostr.parse_private_key(SK3_NSEC) == buzz_nostr.parse_private_key(SK3_HEX) + + +@pytest.mark.parametrize("bad", ["", "zz" * 32, "nsec1invalid", "npub1lycg5qvjtrp3qjf5f7zl382j9x6nrjz9sdhenvyxq8c3808qxmus6gq266"]) +def test_parse_private_key_rejects_garbage(bad): + with pytest.raises(ValueError): + buzz_nostr.parse_private_key(bad) + + +def test_parse_pubkey_accepts_hex_and_npub(): + assert buzz_nostr.parse_pubkey(PK3_HEX.upper()) == PK3_HEX + assert buzz_nostr.parse_pubkey(PK3_NPUB) == PK3_HEX + + +def test_event_id_matches_nip01_reference_vector(): + eid = buzz_nostr.event_id(PK3_HEX, 1700000000, 9, [["h", CHANNEL]], "hello buzz") + assert eid == "6aa2ef0a72e39e52ac7c3680e6a76ed75c90340e684148da6221086b443d2089" + + +def test_sign_event_produces_valid_schnorr_signature(): + keys = buzz_nostr.parse_private_key(SK3_HEX) + ev = buzz_nostr.sign_event(keys, 9, [["h", CHANNEL]], "hello buzz", created_at=1700000000) + assert ev["id"] == "6aa2ef0a72e39e52ac7c3680e6a76ed75c90340e684148da6221086b443d2089" + assert ev["pubkey"] == PK3_HEX and ev["kind"] == 9 and ev["tags"] == [["h", CHANNEL]] + xonly = coincurve.PublicKeyXOnly(bytes.fromhex(PK3_HEX)) + assert xonly.verify(bytes.fromhex(ev["sig"]), bytes.fromhex(ev["id"])) + + +def _keys(): + return buzz_nostr.parse_private_key(SK3_HEX) + + +def test_build_auth_event_carries_relay_and_challenge_tags(): + ev = buzz_nostr.build_auth_event(_keys(), "wss://buzz.example.com", "abc123", created_at=1700000001) + assert ev["kind"] == 22242 + assert ["relay", "wss://buzz.example.com"] in ev["tags"] and ["challenge", "abc123"] in ev["tags"] + + +def test_build_chat_event_tags_channel_reply_and_mentions(): + ev = buzz_nostr.build_chat_event(_keys(), CHANNEL, "hi", created_at=1700000002, reply_to="ab" * 32, mentions=("cd" * 32,)) + assert ev["kind"] == 9 + assert ["h", CHANNEL] in ev["tags"] and ["e", "ab" * 32] in ev["tags"] and ["p", "cd" * 32] in ev["tags"] + + +def test_build_chat_event_minimal_has_only_channel_tag(): + ev = buzz_nostr.build_chat_event(_keys(), CHANNEL, "hi", created_at=1700000002) + assert ev["tags"] == [["h", CHANNEL]] + + +def test_build_edit_event_targets_existing_message(): + ev = buzz_nostr.build_edit_event(_keys(), CHANNEL, "ef" * 32, "new text", created_at=1700000003) + assert ev["kind"] == 40003 + assert ev["tags"] == [["h", CHANNEL], ["e", "ef" * 32]] and ev["content"] == "new text" + + +def test_frames_serialize_as_nostr_wire_arrays(): + req = json.loads(buzz_nostr.req_frame("sub1", {"kinds": [9]}, {"kinds": [39000]})) + assert req == ["REQ", "sub1", {"kinds": [9]}, {"kinds": [39000]}] + ev = buzz_nostr.build_chat_event(_keys(), CHANNEL, "x", created_at=1700000004) + assert json.loads(buzz_nostr.event_frame(ev)) == ["EVENT", ev] + assert json.loads(buzz_nostr.close_frame("sub1")) == ["CLOSE", "sub1"] + + +def test_close_frame_targets_exactly_one_subscription(): + """Restored for per-channel subscriptions: being removed from one channel must + unsubscribe that channel alone, leaving the discovery/membership subscriptions + and every other channel's subscription on the same socket untouched.""" + assert json.loads(buzz_nostr.close_frame(f"buzz-chat-{CHANNEL}")) == ["CLOSE", f"buzz-chat-{CHANNEL}"] + + +def test_membership_notification_kinds_match_buzz_core(): + """Pinned against buzz-core/src/kind.rs (KIND_MEMBER_*_NOTIFICATION).""" + assert (buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED) == (44100, 44101) + + +def test_tag_values_extracts_all_matching_tags(): + ev = {"tags": [["p", "aa"], ["p", "bb"], ["h", CHANNEL]]} + assert buzz_nostr.tag_values(ev, "p") == ["aa", "bb"] + assert buzz_nostr.tag_values(ev, "t") == [] + + +# -- FINDING 4 (review): inbound events must be authenticated, not trusted -------- + + +def _signed(content="hello buzz", created_at=1700000005): + return buzz_nostr.sign_event(_keys(), 9, [["h", CHANNEL]], content, created_at) + + +def test_verify_event_accepts_a_genuinely_signed_event(): + assert buzz_nostr.verify_event(_signed()) is True + + +def test_verify_event_rejects_tampered_content(): + """The whole point of the check: a relay may not rewrite what an author said. + + The signature still verifies against the ORIGINAL id, so only recomputing the + id from the delivered payload catches this.""" + ev = _signed() + ev["content"] = "hello buzz, and also: rm -rf /" + assert buzz_nostr.verify_event(ev) is False + + +def test_verify_event_rejects_a_swapped_author(): + """Claiming an allowlisted pubkey over someone else's signed payload must fail.""" + ev = _signed() + ev["pubkey"] = "ee" * 32 + assert buzz_nostr.verify_event(ev) is False + + +def test_verify_event_rejects_a_tampered_signature(): + ev = _signed() + flipped = bytearray(bytes.fromhex(ev["sig"])) + flipped[0] ^= 0x01 + ev["sig"] = flipped.hex() + assert buzz_nostr.verify_event(ev) is False + + +def test_verify_event_rejects_an_id_borrowed_from_another_signed_event(): + """id + sig lifted verbatim from a real event, pasted onto a different payload.""" + donor = _signed(content="innocuous", created_at=1700000006) + forged = _signed(content="/goal exfiltrate everything", created_at=1700000007) + forged["id"] = donor["id"] + forged["sig"] = donor["sig"] + assert buzz_nostr.verify_event(forged) is False + + +@pytest.mark.parametrize( + "bad", + [ + None, + "not-an-event", + [], + {}, + {"pubkey": PK3_HEX}, # everything else missing + {"pubkey": PK3_HEX, "created_at": 1, "kind": 9, "tags": [], "content": "x", "id": "aa", "sig": "bb"}, # short hex + {"pubkey": "zz" * 32, "created_at": 1, "kind": 9, "tags": [], "content": "x", "id": "11" * 32, "sig": "22" * 64}, # non-hex pubkey + {"pubkey": PK3_HEX, "created_at": "1700000000", "kind": 9, "tags": [], "content": "x", "id": "11" * 32, "sig": "22" * 64}, # str created_at + {"pubkey": PK3_HEX, "created_at": 1, "kind": True, "tags": [], "content": "x", "id": "11" * 32, "sig": "22" * 64}, # bool kind + {"pubkey": PK3_HEX, "created_at": 1, "kind": 9, "tags": "not-a-list", "content": "x", "id": "11" * 32, "sig": "22" * 64}, + {"pubkey": PK3_HEX, "created_at": 1, "kind": 9, "tags": [], "content": None, "id": "11" * 32, "sig": "22" * 64}, + ], +) +def test_verify_event_returns_false_for_malformed_input_without_raising(bad): + """Relay input is untrusted, so every malformed shape must be a plain False. + + A raise here would surface as `handle_relay_frame`'s generic "malformed relay + event" path at best -- and at worst tempt a caller into a bare except that also + swallows a real verification failure.""" + assert buzz_nostr.verify_event(bad) is False + + +def test_verify_event_rejects_an_unsigned_but_otherwise_valid_event(): + """The pre-fix test fixtures' shape: correct id, no signature at all.""" + ev = _signed() + ev.pop("sig") + assert buzz_nostr.verify_event(ev) is False diff --git a/backend/tests/test_channels.py b/backend/tests/test_channels.py index 453540029..e5f54dd3e 100644 --- a/backend/tests/test_channels.py +++ b/backend/tests/test_channels.py @@ -397,6 +397,7 @@ class TestChannelBase: assert "prepare_inbound failed for msg_id=m1: boom" in caplog.text def test_channel_capabilities_match_channel_defaults(self): + from app.channels.buzz import BuzzChannel from app.channels.dingtalk import DingTalkChannel from app.channels.discord import DiscordChannel from app.channels.feishu import FeishuChannel @@ -409,6 +410,7 @@ class TestChannelBase: bus = MessageBus() defaults = { + "buzz": BuzzChannel(bus=bus, config={"relay_url": "wss://buzz.example.com"}).supports_streaming, "dingtalk": DingTalkChannel(bus=bus, config={}).supports_streaming, "discord": DiscordChannel(bus=bus, config={}).supports_streaming, "feishu": FeishuChannel(bus=bus, config={}).supports_streaming, @@ -9649,3 +9651,254 @@ def test_merge_stream_text_newline_split(): def test_merge_stream_text_normal_append(): _merge = _get_merge_stream_text() assert _merge("Hello ", "world") == "Hello world" + + +# --------------------------------------------------------------------------- +# LIVE-TEST FINDING 1 (critical, data disclosure): _accumulate_stream_text +# decided what streamed payloads become displayable assistant text by REJECTING +# only payloads whose ``type`` contained "tool". DeerFlow injects hidden +# context -- memory facts (DynamicContextMiddleware) and durable context +# (DurableContextMiddleware) -- as hidden HumanMessages whose ``type`` is +# "human", and DynamicContextMiddleware also rewrites the user's own turn into +# a fresh HumanMessage. All of those are written to the messages channel, so +# LangGraph fans them out on the ``messages-tuple`` stream and the old denylist +# accumulated them and published them to the IM channel as the assistant's +# reply. Proved live on a Buzz relay: the connector published +# "\nFacts:\n- [context | 0.70] ...\n ▉" and, in another run, +# a verbatim echo of the user's own inbound message. +# +# The filter is now an ALLOWLIST of assistant message types. These tests pin +# both directions: hidden context never accumulates, and assistant streaming -- +# including multi-chunk merging across one message id -- is untouched. +# --------------------------------------------------------------------------- + + +def _get_accumulate_stream_text(): + from app.channels.manager import _accumulate_stream_text + + return _accumulate_stream_text + + +_MEMORY_LEAK_TEXT = "\nFacts:\n- [context | 0.70] User interacts with the assistant through the DeerFlow chat channel.\n" + + +def test_accumulate_stream_text_rejects_hidden_memory_human_message(): + """The exact live leak: a hidden HumanMessage carrying a block.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, message_id = _accumulate( + buffers, + None, + [ + { + "id": "run-1__memory", + "type": "human", + "content": _MEMORY_LEAK_TEXT, + "additional_kwargs": {"hide_from_ui": True}, + }, + {"langgraph_node": "model_request"}, + ], + ) + assert text is None + assert message_id is None + assert buffers == {} + + +def test_accumulate_stream_text_rejects_durable_context_human_message(): + """DurableContextMiddleware's hidden block.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, _ = _accumulate( + buffers, + None, + [ + {"id": "dc-1", "type": "human", "content": "\n## Conversation summary so far\nsecret\n"}, + {}, + ], + ) + assert text is None + assert buffers == {} + + +def test_accumulate_stream_text_rejects_echo_of_plain_human_message(): + """DynamicContextMiddleware re-writes the user's own turn as a new + HumanMessage; echoing it back to the channel as the assistant's reply is + the second half of the same live leak.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, _ = _accumulate(buffers, None, [{"id": "u-1__user", "type": "human", "content": "what is my deploy status?"}, {}]) + assert text is None + assert buffers == {} + + +def test_accumulate_stream_text_rejects_system_message(): + """The SystemMessage is hidden context too.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, _ = _accumulate(buffers, None, [{"id": "s-1", "type": "system", "content": "Today is 2026-08-01"}, {}]) + assert text is None + assert buffers == {} + + +def test_accumulate_stream_text_still_rejects_tool_payloads(): + """Pre-existing behavior: tool calls and tool results are never displayable.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + assert _accumulate(buffers, None, [{"id": "t-1", "type": "tool", "content": "bash output"}, {}]) == (None, None) + assert _accumulate(buffers, None, [{"id": "t-2", "type": "ToolMessageChunk", "content": "more output"}, {}]) == (None, None) + assert buffers == {} + + +def test_accumulate_stream_text_rejects_hidden_context_wrapped_in_kwargs_shape(): + """The LangChain ``to_json`` shape the function already reads content from: + the wrapper's own ``type`` is "constructor", so the real message type has to + be resolved from ``kwargs``/``id`` or hidden context walks straight through.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, _ = _accumulate( + buffers, + None, + [ + { + "lc": 1, + "type": "constructor", + "id": ["langchain", "schema", "messages", "HumanMessage"], + "kwargs": {"id": "m-1", "content": _MEMORY_LEAK_TEXT}, + }, + {}, + ], + ) + assert text is None + assert buffers == {} + + +def test_accumulate_stream_text_accepts_assistant_chunk_in_kwargs_shape(): + """...and the same shape must still stream an AIMessageChunk.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, message_id = _accumulate( + buffers, + None, + [ + { + "lc": 1, + "type": "constructor", + "id": ["langchain", "schema", "messages", "AIMessageChunk"], + "kwargs": {"id": "ai-9", "content": "Hello"}, + }, + {}, + ], + ) + assert text == "Hello" + assert message_id == "ai-9" + + +def test_accumulate_stream_text_rejects_untyped_bare_string_payload(): + """A bare ``str`` payload carries no type information at all, so it cannot be + attributed to the assistant. Under an allowlist an unattributable payload + must not be published -- hidden context arriving that way would be + indistinguishable from assistant output.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + assert _accumulate(buffers, None, "Hello") == (None, None) + assert _accumulate(buffers, "ai-1", ["raw text", {}]) == (None, "ai-1") + assert buffers == {} + + +@pytest.mark.parametrize("payload_type", ["ai", "AIMessageChunk", "AIMessage", "assistant"]) +def test_accumulate_stream_text_accepts_every_assistant_type_spelling(payload_type): + """The literal ``type`` values assistant output actually carries: LangChain + serializes AIMessage as "ai" and AIMessageChunk as "AIMessageChunk"; the + OpenAI-style "assistant" spelling is accepted for foreign runtimes.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + text, message_id = _accumulate(buffers, None, [{"id": "ai-1", "content": "Hi", "type": payload_type}, {"langgraph_node": "agent"}]) + assert text == "Hi" + assert message_id == "ai-1" + + +def test_accumulate_stream_text_merges_multi_chunk_assistant_stream_across_one_message_id(): + """The function's entire purpose. Pinned hard so the allowlist can never + silently kill streaming for Feishu / Telegram / WeCom / Buzz.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + current: str | None = None + seen = [] + for delta in ("Hello", " ", "world", "!"): + text, current = _accumulate(buffers, current, [{"id": "ai-1", "content": delta, "type": "AIMessageChunk"}, {"langgraph_node": "agent"}]) + seen.append(text) + assert seen == ["Hello", "Hello ", "Hello world", "Hello world!"] + assert current == "ai-1" + assert buffers == {"ai-1": "Hello world!"} + + +def test_accumulate_stream_text_keeps_separate_buffers_per_message_id(): + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + _accumulate(buffers, None, [{"id": "ai-1", "content": "first", "type": "AIMessageChunk"}, {}]) + _accumulate(buffers, "ai-1", [{"id": "ai-2", "content": "second", "type": "AIMessageChunk"}, {}]) + text, current = _accumulate(buffers, "ai-2", [{"id": "ai-1", "content": "-more", "type": "AIMessageChunk"}, {}]) + assert text == "first-more" + assert current == "ai-1" + assert buffers == {"ai-1": "first-more", "ai-2": "second"} + + +def test_accumulate_stream_text_hidden_context_between_assistant_chunks_never_enters_the_buffer(): + """Interleaving is the realistic live shape: the middleware writes its hidden + HumanMessage in the middle of a turn. It must neither be published nor + corrupt the assistant buffer it sits between.""" + _accumulate = _get_accumulate_stream_text() + buffers: dict[str, str] = {} + _, current = _accumulate(buffers, None, [{"id": "ai-1", "content": "Deploy ", "type": "AIMessageChunk"}, {}]) + leaked, current_after = _accumulate(buffers, current, [{"id": "mem-1", "type": "human", "content": _MEMORY_LEAK_TEXT}, {}]) + assert leaked is None + assert current_after == "ai-1" # the assistant message id is preserved + text, _ = _accumulate(buffers, current_after, [{"id": "ai-1", "content": "succeeded.", "type": "AIMessageChunk"}, {}]) + assert text == "Deploy succeeded." + assert buffers == {"ai-1": "Deploy succeeded."} + + +def test_streaming_chat_never_publishes_hidden_memory_context(monkeypatch): + """End-to-end through _handle_streaming_chat: the hidden HumanMessage + the live relay actually received must reach no outbound message at all.""" + from app.channels.manager import ChannelManager + + monkeypatch.setattr("app.channels.manager.STREAM_UPDATE_MIN_INTERVAL_SECONDS", 0.0) + + async def go(): + bus = MessageBus() + store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json") + manager = ChannelManager(bus=bus, store=store) + outbound_received: list[OutboundMessage] = [] + + async def capture_outbound(msg): + outbound_received.append(msg) + + bus.subscribe_outbound(capture_outbound) + + stream_events = [ + _make_stream_part("messages-tuple", [{"id": "u-1__user", "type": "human", "content": "what is my deploy status?"}, {}]), + _make_stream_part("messages-tuple", [{"id": "u-1__memory", "type": "human", "content": _MEMORY_LEAK_TEXT, "additional_kwargs": {"hide_from_ui": True}}, {}]), + _make_stream_part("messages-tuple", [{"id": "ai-1", "content": "All green.", "type": "AIMessageChunk"}, {"langgraph_node": "agent"}]), + _make_stream_part( + "values", + {"messages": [{"type": "human", "content": "what is my deploy status?"}, {"type": "ai", "content": "All green."}], "artifacts": []}, + ), + ] + + mock_client = _make_mock_langgraph_client() + mock_client.runs.stream = MagicMock(return_value=_make_async_iterator(stream_events)) + manager._client = mock_client + await manager.start() + + await bus.publish_inbound(InboundMessage(channel_name="buzz", chat_id="chan-1", user_id="pk-1", text="what is my deploy status?", thread_ts=None)) + await _wait_for(lambda: any(m.is_final for m in outbound_received)) + await manager.stop() + + assert outbound_received, "expected at least the final outbound" + for published in outbound_received: + assert "" not in published.text + assert "what is my deploy status?" not in published.text + assert [m.text for m in outbound_received] == ["All green. ▉", "All green."] + + _run(go()) diff --git a/backend/tests/test_detect_uv_extras.py b/backend/tests/test_detect_uv_extras.py index 0429ea871..eb32f260b 100644 --- a/backend/tests/test_detect_uv_extras.py +++ b/backend/tests/test_detect_uv_extras.py @@ -173,6 +173,35 @@ def test_detect_from_config_ignores_commented_browser_tool(tmp_path): assert detect.detect_from_config(cfg) == [] +def test_detect_from_config_buzz_via_channels_enabled(tmp_path): + cfg = tmp_path / "config.yaml" + cfg.write_text( + "channels:\n buzz:\n enabled: true\n relay_url: wss://buzz.example.com\n", + ) + assert detect.detect_from_config(cfg) == ["buzz"] + + +def test_detect_from_config_buzz_disabled_returns_no_extras(tmp_path): + cfg = tmp_path / "config.yaml" + cfg.write_text("channels:\n buzz:\n enabled: false\n") + assert detect.detect_from_config(cfg) == [] + + +def test_detect_from_config_no_channels_section_returns_no_buzz_extra(tmp_path): + cfg = tmp_path / "config.yaml" + cfg.write_text("database:\n backend: sqlite\n") + assert detect.detect_from_config(cfg) == [] + + +def test_detect_from_config_ignores_commented_buzz_block(tmp_path): + """Mirrors the fully-commented example block shipped in config.example.yaml.""" + cfg = tmp_path / "config.yaml" + cfg.write_text( + "# channels:\n# buzz:\n# enabled: true\n# relay_url: wss://buzz.example.com\ndatabase:\n backend: sqlite\n", + ) + assert detect.detect_from_config(cfg) == [] + + def test_detect_from_config_memory_stream_bridge_returns_no_extras(tmp_path): cfg = tmp_path / "config.yaml" cfg.write_text("stream_bridge:\n type: memory\n queue_maxsize: 256\n") @@ -193,6 +222,13 @@ def test_detect_from_config_dedupes_when_both_present(tmp_path): assert detect.detect_from_config(cfg) == ["postgres"] +def test_detect_from_config_combines_buzz_with_postgres(tmp_path): + cfg = tmp_path / "config.yaml" + cfg.write_text("database:\n backend: postgres\nchannels:\n buzz:\n enabled: true\n") + # Existing postgres/redis/browser detection is unaffected by the new rule. + assert detect.detect_from_config(cfg) == ["buzz", "postgres"] + + def test_detect_from_config_missing_file_returns_empty(tmp_path): assert detect.detect_from_config(tmp_path / "does-not-exist.yaml") == [] diff --git a/backend/uv.lock b/backend/uv.lock index f9465ab8b..06b4cfcbd 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -676,6 +676,34 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d5/e1/3714a2f371985215c219c2a70953d38e3eed81ef165aed061d21de0e998b/cobble-0.1.4-py3-none-any.whl", hash = "sha256:36c91b1655e599fd428e2b95fdd5f0da1ca2e9f1abb0bc871dec21a0e78a2b44", size = 3984, upload-time = "2024-06-01T18:11:07.911Z" }, ] +[[package]] +name = "coincurve" +version = "21.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6f/a2/f2a38eb05b747ed3e54e1be33be339d4a14c1f5cc6a6e2b342b5e8160d51/coincurve-21.0.0.tar.gz", hash = "sha256:8b37ce4265a82bebf0e796e21a769e56fdbf8420411ccbe3fafee4ed75b6a6e5", size = 128986, upload-time = "2025-03-08T15:31:24.266Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f3/61/a2d9e109f99b6f5e65e653ac998b0944c5b82c568ac142fcbb381a4803be/coincurve-21.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f60ad56113f08e8c540bb89f4f35f44d434311433195ffff22893ccfa335070c", size = 1391948, upload-time = "2025-03-08T15:30:32.899Z" }, + { url = "https://files.pythonhosted.org/packages/24/5a/2da75ee00a722ef1fa068ada3bc34c564595ead86fef573434e2f0cb0a5c/coincurve-21.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1cb1cd19fb0be22e68ecb60ad950b41f18b9b02eebeffaac9391dc31f74f08f2", size = 1384958, upload-time = "2025-03-08T15:30:34.705Z" }, + { url = "https://files.pythonhosted.org/packages/dc/50/6bf0bf7e8a9a9dd419ecc1e479dcb9fbfe657029276ad703806a25a2bef2/coincurve-21.0.0-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:05d7e255a697b3475d7ae7640d3bdef3d5bc98ce9ce08dd387f780696606c33b", size = 1606576, upload-time = "2025-03-08T15:30:36.796Z" }, + { url = "https://files.pythonhosted.org/packages/bd/ab/9e89908fdd09ad522938085587aaa821b022f4def16c286c5580cfc85811/coincurve-21.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5a366c314df7217e3357bb8c7d2cda540b0bce180705f7a0ce2d1d9e28f62ad4", size = 1613642, upload-time = "2025-03-08T15:30:38.416Z" }, + { url = "https://files.pythonhosted.org/packages/b7/75/050b6fd08978de85a7b480f0f220ab6a30967c0910119f3096a8dd40befc/coincurve-21.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1b04778b75339c6e46deb9ae3bcfc2250fbe48d1324153e4310fc4996e135715", size = 1616974, upload-time = "2025-03-08T15:30:39.939Z" }, + { url = "https://files.pythonhosted.org/packages/d7/62/2740ba0cafebf45708633635fecadcbe582d7a3ed1ce8b4637921feceaf8/coincurve-21.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8efcbdcd50cc219989a2662e6c6552f455efc000a15dd6ab3ebf4f9b187f41a3", size = 1644133, upload-time = "2025-03-08T15:30:41.733Z" }, + { url = "https://files.pythonhosted.org/packages/94/14/1f27c3048c4084fa85ef65f42a4ca631f2b184336e6d9446fecec20e0a7f/coincurve-21.0.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:6df44b4e3b7acdc1453ade52a52e3f8a5b53ecdd5a06bd200f1ec4b4e250f7d9", size = 1619918, upload-time = "2025-03-08T15:30:43.284Z" }, + { url = "https://files.pythonhosted.org/packages/ca/22/7ec3ec4c8e7764daa25767d6674cb5741ea2d9b39ff758e9918d22a4b49b/coincurve-21.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bcc0831f07cb75b91c35c13b1362e7b9dc76c376b27d01ff577bec52005e22a8", size = 1645797, upload-time = "2025-03-08T15:30:44.974Z" }, + { url = "https://files.pythonhosted.org/packages/fb/60/87982b7499943ab12605df7b14f6001fff331aca0881b260682461e2309d/coincurve-21.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:5dd7b66b83b143f3ad3861a68fc0279167a0bae44fe3931547400b7a200e90b1", size = 1329255, upload-time = "2025-03-08T15:30:46.4Z" }, + { url = "https://files.pythonhosted.org/packages/62/c0/65b60b371579570931daca8a3f67debfc1482908b8ed03432297274a27da/coincurve-21.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:78dbe439e8cb22389956a4f2f2312813b4bd0531a0b691d4f8e868c7b366555d", size = 1325973, upload-time = "2025-03-08T15:30:48.056Z" }, + { url = "https://files.pythonhosted.org/packages/b3/40/cce55adaec37a588eb24b67da8eb68926546458e12ed2c4c2a21deb93d4c/coincurve-21.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:9df5ceb5de603b9caf270629996710cf5ed1d43346887bc3895a11258644b65b", size = 1391762, upload-time = "2025-03-08T15:30:49.586Z" }, + { url = "https://files.pythonhosted.org/packages/ca/7a/628a30281d246ce98aea56592e0c8e79b03a93ee8b85d688db3388130c2d/coincurve-21.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:154467858d23c48f9e5ab380433bc2625027b50617400e2984cc16f5799ab601", size = 1384921, upload-time = "2025-03-08T15:30:51.103Z" }, + { url = "https://files.pythonhosted.org/packages/61/cc/719c5da31e6ba07e438abcf962f7a365eb69a06a0621ca4f2a484f344e09/coincurve-21.0.0-cp313-cp313-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f57f07c44d14d939bed289cdeaba4acb986bba9f729a796b6a341eab1661eedc", size = 1606559, upload-time = "2025-03-08T15:30:53.218Z" }, + { url = "https://files.pythonhosted.org/packages/b2/ee/dd14237013d732e7fc3248c0c33a1d36b88b5378dfa3e624a50a23fb6f19/coincurve-21.0.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3fb03e3a388a93d31ed56a442bdec7983ea404490e21e12af76fb1dbf097082a", size = 1613684, upload-time = "2025-03-08T15:30:55.087Z" }, + { url = "https://files.pythonhosted.org/packages/f0/05/eaa7f36a03376ced1c19e0cb563341cc83fe48f5734b2effe8f16d0ee0ab/coincurve-21.0.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d09ba4fd9d26b00b06645fcd768c5ad44832a1fa847ebe8fb44970d3204c3cb7", size = 1617001, upload-time = "2025-03-08T15:30:57.036Z" }, + { url = "https://files.pythonhosted.org/packages/39/32/fc75f1dd914ac95eb2704425c7ca1a9f509f982e15d05e0ca895b9e6ea9c/coincurve-21.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1a1e7ee73bc1b3bcf14c7b0d1f44e6485785d3b53ef7b16173c36d3cefa57f93", size = 1643924, upload-time = "2025-03-08T15:30:58.737Z" }, + { url = "https://files.pythonhosted.org/packages/1a/4b/8c6e65b5755e26fc02077803879747615c1c327047328d1784bccb4ff4c3/coincurve-21.0.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ad05952b6edc593a874df61f1bc79db99d716ec48ba4302d699e14a419fe6f51", size = 1619964, upload-time = "2025-03-08T15:31:00.275Z" }, + { url = "https://files.pythonhosted.org/packages/64/bc/d0a743305ff9fa26e72b4c77b534d5958ec8030b3772555a7172a0c134e5/coincurve-21.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4d2bf350ced38b73db9efa1ff8fd16a67a1cb35abb2dda50d89661b531f03fd3", size = 1645526, upload-time = "2025-03-08T15:31:01.952Z" }, + { url = "https://files.pythonhosted.org/packages/9d/44/ab082e2dc8c9a45774f1bb9961f58b43c0882b866f5c469ead932d45a35d/coincurve-21.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:54d9500c56d5499375e579c3917472ffcf804c3584dd79052a79974280985c74", size = 1329285, upload-time = "2025-03-08T15:31:03.591Z" }, + { url = "https://files.pythonhosted.org/packages/f3/94/407f6fc811310f15b1fc7255f436f6a9040854213beeb10093f56b5b7fd3/coincurve-21.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:773917f075ec4b94a7a742637d303a3a082616a115c36568eb6c873a8d950d18", size = 1326027, upload-time = "2025-03-08T15:31:05.318Z" }, +] + [[package]] name = "colorama" version = "0.4.6" @@ -801,6 +829,9 @@ dependencies = [ browser = [ { name = "deerflow-harness", extra = ["browser"] }, ] +buzz = [ + { name = "coincurve" }, +] discord = [ { name = "discord-py" }, ] @@ -834,6 +865,7 @@ dev = [ [package.metadata] requires-dist = [ { name = "bcrypt", specifier = ">=4.0.0" }, + { name = "coincurve", marker = "extra == 'buzz'", specifier = ">=20.0.0" }, { name = "deerflow-harness", editable = "packages/harness" }, { name = "deerflow-harness", extras = ["browser"], marker = "extra == 'browser'", editable = "packages/harness" }, { name = "deerflow-harness", extras = ["memory-zh"], marker = "extra == 'memory-zh'", editable = "packages/harness" }, @@ -857,7 +889,7 @@ requires-dist = [ { name = "uvicorn", extras = ["standard"], specifier = ">=0.34.0" }, { name = "wecom-aibot-python-sdk", specifier = ">=0.1.6" }, ] -provides-extras = ["postgres", "redis", "discord", "monocle", "browser", "memory-zh"] +provides-extras = ["postgres", "redis", "discord", "buzz", "monocle", "browser", "memory-zh"] [package.metadata.requires-dev] dev = [ diff --git a/config.example.yaml b/config.example.yaml index 8d7a54ed9..56af387d1 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -15,7 +15,7 @@ # ============================================================================ # Bump this number when the config schema changes. # Run `make config-upgrade` to merge new fields into your local config.yaml. -config_version: 32 +config_version: 33 # ============================================================================ # Logging @@ -2162,6 +2162,9 @@ run_ownership: # # wecom: # enabled: false +# +# buzz: +# enabled: false # ============================================================================ # IM Channels Configuration @@ -2279,6 +2282,20 @@ run_ownership: # mention_only: false # If true, only respond when the bot is mentioned # allowed_channels: [] # Optional: channel IDs exempt from mention_only (bot responds without mention) # thread_mode: false # If true, group a channel conversation into a thread +# +# # Buzz (https://github.com/block/buzz) — Nostr-relay team workspace. +# # DeerFlow joins as a member identity; generate a keypair and register it as +# # a relay member, then @mention or DM DeerFlow in Buzz. Requires the `buzz` +# # dependency extra (uv sync --extra buzz). +# buzz: +# enabled: false +# relay_url: wss://buzz.example.com +# private_key: $BUZZ_PRIVATE_KEY # hex or nsec1… +# allowed_users: [] # pubkeys (hex or npub) allowed to trigger runs. +# # DENY-BY-DEFAULT (unlike other channels): empty +# # means nobody, and DeerFlow logs a startup warning. +# require_mention: true # @mention needed in channels +# mention_free_channels: [] # channel UUIDs that respond to every message # ============================================================================ # Guardrails Configuration diff --git a/deploy/helm/deer-flow/README.md b/deploy/helm/deer-flow/README.md index e782ca122..b37c75746 100644 --- a/deploy/helm/deer-flow/README.md +++ b/deploy/helm/deer-flow/README.md @@ -124,7 +124,7 @@ they resolve from the `secrets` map): ```yaml config: | - config_version: 32 + config_version: 33 models: - name: gpt-4 use: langchain_openai:ChatOpenAI diff --git a/deploy/helm/deer-flow/values.yaml b/deploy/helm/deer-flow/values.yaml index 15ebd2ff2..6dafba56e 100644 --- a/deploy/helm/deer-flow/values.yaml +++ b/deploy/helm/deer-flow/values.yaml @@ -240,7 +240,7 @@ ingress: # -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never # inline literal secret values here. The default enables provisioner sandbox. config: | - config_version: 32 + config_version: 33 log_level: info models: [] diff --git a/scripts/detect_uv_extras.py b/scripts/detect_uv_extras.py index deb552008..9625e2d9f 100755 --- a/scripts/detect_uv_extras.py +++ b/scripts/detect_uv_extras.py @@ -13,6 +13,7 @@ Order of resolution: - stream_bridge.type == redis -> redis - tools[].name == browser_navigate -> browser - sandbox.ownership.type == redis -> redis + - channels.buzz.enabled == true -> buzz 3. Runtime environment toggles that enable optional backends: - DEER_FLOW_STREAM_BRIDGE_REDIS_URL -> redis - DEER_FLOW_SANDBOX_OWNERSHIP_REDIS_URL -> redis @@ -267,6 +268,8 @@ def detect_from_config(path: Path) -> list[str]: extras.add("redis") if (nested_section_value(lines, "channels.discord", "enabled") or "").lower() == "true": extras.add("discord") + if (nested_section_value(lines, "channels.buzz", "enabled") or "").lower() == "true": + extras.add("buzz") if tools_include_name(lines, "browser_navigate"): extras.add("browser") return sorted(extras)