ajayr d732b90dc3
feat(channels): add Buzz (Nostr) channel connector (#4649)
* feat(channels): add Buzz (Nostr) channel connector

Adds a Buzz (https://github.com/block/buzz) channel so DeerFlow can join a
Nostr-relay workspace as a member: it answers @mentions in channels, replies
to DMs, and streams answers by editing one message in place.

  * app/channels/buzz_nostr.py — pure NIP-01 helpers: canonical event ids,
    BIP-340 signing/verification, chat/edit/auth builders, relay frames.
  * app/channels/buzz.py — BuzzChannel: one NIP-42-authenticated websocket,
    channel discovery (kind 39000) with one subscription per channel, live
    membership tracking (44100/44101), per-channel replay watermarks, and
    replies posted once then edited in place (kind 40003).
  * app/channels/buzz_run_policy.py — same-thread serialization, mirroring
    the Feishu precedent.

Inbound is gated in order: signature verification, self-drop, /connect
bind-and-return, pubkey allowlist, then mention / DM / mention-free /
thread-follow. Off by default; needs the new optional `buzz` extra
(coincurve, lazily imported), which detect_uv_extras resolves from
channels.buzz.enabled the same way it already handles channels.discord.

Two relay behaviours drove the design and are worth knowing when reviewing:
a global {"kinds":[9]} subscription receives nothing from buzz-relay and a
multi-value "#h" filter receives nothing either, so one REQ per channel is
required; and a single global `since` cursor skips quiet channels, so
watermarks are per channel.

Signed-off-by: Ajay R <ajayr@formbuddy.com>

* fix(channels): only publish assistant messages from the IM stream

`_accumulate_stream_text` decided what streamed `messages-tuple` payloads
become displayable text by rejecting ONLY payloads whose `type` contained
"tool", so it published everything else. DeerFlow writes hidden model
context into the messages channel as ordinary messages -- memory recall and
the rewritten user turn as hidden HumanMessages (DynamicContextMiddleware),
the `<durable_context_data>` block as another (DurableContextMiddleware) --
and LangGraph fans those state writes out on the messages stream, so they
reached every streaming IM channel as the assistant's reply.

Proved live on a Buzz relay: the connector published a `<memory>` fact block
and, in another run, a verbatim echo of the user's own inbound message.
Affects Feishu, Telegram, WeCom and Buzz; worst on Buzz, where each update
is an immutable public Nostr event that a corrective edit cannot unpublish.

Invert the filter to an allowlist of assistant message types. Two new pure
helpers keep it testable:

- `_stream_payload_type` resolves the type from both shapes the function
  already handles: the `model_dump()` shape the gateway emits, and
  LangChain's `to_json()` constructor shape whose own `type` is the literal
  "constructor" and whose class name is the tail of the `id` path.
- `_is_assistant_stream_type` matches "ai"/"assistant" by PREFIX, not
  substring -- ordinary words contain "ai" ("chain", "domain"), and a
  substring test would admit a foreign type name by accident.

The bare-`str` branch is removed: an untyped payload cannot be attributed to
the assistant, nothing in DeerFlow produces one (serialize_messages_tuple
always emits `[message_dict, metadata]`), and a runtime that emitted raw text
deltas would emit hidden context the same way. Per-message-id buffering and
merging are unchanged.

Tests pin both directions, including multi-chunk merging across one message
id, so the allowlist cannot silently kill streaming, plus an end-to-end
`_handle_streaming_chat` test asserting the live payload never reaches an
outbound message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ajay R <ajayr@formbuddy.com>

* chore(helm): bump config_version to 33 in chart values and README

config.example.yaml moved to 33 for the buzz channel block; the chart's
embedded config example and its README copy track it (config_version only
drives the outdated-config warning, per scripts/check_config_version.sh).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ajay R <ajayr@formbuddy.com>

---------

Signed-off-by: Ajay R <ajayr@formbuddy.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 08:29:22 +08:00

1414 lines
83 KiB
Python

"""Buzz (Nostr) channel: DeerFlow as a member of a Buzz workspace relay.
One NIP-42-authenticated WebSocket to ``relay_url``. Inbound kind-9 chat events are
gated (pubkey allowlist, then mention/DM/thread-follow) and published to the bus;
outbound replies post one kind-9 message and then stream via kind-40003 in-place edits.
Subscriptions are CHANNEL-SCOPED, which is the shape the relay actually serves:
- ``REQ {"kinds":[9]}`` (global) is accepted and answered with EOSE, but the relay
never fans a chat event out to it -- a connector subscribed that way authenticates
successfully and then receives nothing, forever. Proved against a live relay.
- ``REQ {"kinds":[9], "#h":[uuid]}`` works, and a multi-value ``#h`` does NOT, so
there is exactly one chat subscription per channel (matching Buzz's own agent
harness, whose ``subscribe_channel_from`` is likewise per channel).
So each connection: authenticate, discover the channels this identity belongs to
with a historical ``kinds:[39000]`` REQ, open one ``#h`` chat subscription per
discovered channel, and keep a live ``kinds:[44100,44101] #p=<us>`` subscription so
channels we are added to (or removed from) later are picked up without a reconnect.
Every one of those subscriptions can be killed by a single relay ``CLOSED`` frame,
and each one dying is a *silent* outage (a dead chat subscription deafens one
channel; a dead ``buzz-membership`` stops us ever learning we were added to or
removed from a channel; a dead ``buzz-discovery`` kills the completeness sweep).
So a ``CLOSED`` is recovered rather than merely forgotten -- bounded by
``MAX_RESUBSCRIBE_ATTEMPTS`` per subscription per connection, and only when the
relay's stated reason suggests re-issuing the same REQ could work at all (see
``_is_transient_close``). Every subscription that goes unlistened is named at
WARNING, because "listening to nothing" must never again be indistinguishable from
"nothing is being said".
KNOWN BOUND (documented, not fixed): the relay caps historical delivery at 2000
events per subscription and serves them NEWEST-FIRST, even with a ``since``. So a
channel that accumulated more than 2000 unread messages across one disconnect
loses the oldest of them: the relay never sends them, and the watermark advances
past them as the newer ones are processed. This is the one bounded skip path that
remains, it needs a disconnect plus >2000 messages in a single channel to trigger,
and closing it would require paging the backlog with descending ``until`` queries.
"""
from __future__ import annotations
import asyncio
import json
import logging
import random
import time
from typing import Any
from urllib.parse import urlparse
from app.channels import buzz_nostr
from app.channels.base import Channel
from app.channels.commands import is_known_channel_command
from app.channels.connection_identity import attach_connection_identity
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage
logger = logging.getLogger(__name__)
# Headroom under the relay's 64KB edit-content cap (kind-40003 events).
EDIT_MAX_BYTES = 60_000
# Ceiling for how far ahead of our own clock a peer-supplied ``created_at`` may be
# and still advance the resubscribe watermark (see ``_advance_watermark``).
MAX_FUTURE_SKEW_SECONDS = 60
# Cap on the kind-39000 metadata cache. Any relay member can publish channel
# metadata, so this map is remote-fed and would otherwise grow without bound for
# the process lifetime; evicting the oldest entry only costs us the DM mention
# exemption for that channel until its metadata is seen again (``_is_dm`` fails
# closed on a cache miss).
MAX_CACHED_CHANNELS = 512
# Cap on how many per-channel chat subscriptions one connection may hold. The
# channel list is remote-fed (kind-39000 metadata and kind-44100 membership
# notifications both come off the wire), and each entry is a real REQ on the socket,
# so this is bounded twice over: well under buzz-relay's own per-connection ceiling
# (``MAX_SUBSCRIPTIONS = 1024`` in its REQ handler) and far above any plausible
# workspace. At the cap we REFUSE new subscriptions and log, rather than evicting an
# existing one: eviction would silently deafen a channel that is currently working,
# whereas refusing leaves every established channel intact and names the one that
# did not fit.
MAX_CHANNEL_SUBSCRIPTIONS = 256
# Subscription ids. Deterministic so a subscription can be replaced or closed
# individually (``CLOSE``) without disturbing the others on the same socket.
DISCOVERY_SUB_ID = "buzz-discovery"
MEMBERSHIP_SUB_ID = "buzz-membership"
CHAT_SUB_PREFIX = "buzz-chat-"
# How many times ONE subscription may be re-opened after the relay ``CLOSED`` it,
# per connection and per auth epoch. Re-issuing a closed REQ immediately and
# unconditionally is a tight loop against a relay that keeps closing it (each
# CLOSED provokes a REQ which provokes a CLOSED), so the budget is what makes
# recovery safe rather than the backoff: at the cap we stop, say so loudly, and
# leave it to the next reconnect -- which rebuilds every subscription anyway.
MAX_RESUBSCRIBE_ATTEMPTS = 3
# Backoff between re-subscribe attempts. The FIRST retry is immediate: a one-off
# relay hiccup is the common case and should be repaired without a stall. Later
# retries back off (1s, then 2s), which is what a persistently-closing relay gets.
# The delay is awaited inline in the relay read loop rather than handed to a
# background task -- 3s of worst-case delayed frame processing per subscription
# per connection is cheaper than a task that can outlive its own socket and fire a
# REQ into a connection that no longer exists.
RESUBSCRIBE_BASE_DELAY_SECONDS = 1.0
# How far BEFORE the socket opened the live membership subscription starts.
#
# Without any ``since`` the relay replays its whole stored 44100/44101 history on
# every connection: each historical "you were added" reads as live (re-subscribing
# channels we have since been removed from and re-running discovery once per
# event), and each historical "you were removed" transiently unsubscribes a channel
# we are still in. Anchoring ``since`` at connection time fixes that, but a bare
# ``now`` would open a hole: the relay stamps ``created_at`` with ITS clock, and a
# membership change published during our connect/auth handshake is genuinely live
# yet already in the past by the time the REQ goes out. This slack covers both --
# it can only ever cost the replay of the last minute of membership changes, which
# is idempotent (``_ensure_chat_subscription`` no-ops on a channel already
# subscribed), whereas being one second short costs a channel we never hear from.
MEMBERSHIP_LOOKBACK_SECONDS = 60
# Bound on how long ``stop()`` waits for the cancelled relay loop to finish.
STOP_TIMEOUT_SECONDS = 5.0
# NIP-42's machine-readable ``CLOSED`` prefix for "authenticate first".
#
# BEFORE this connection has completed its NIP-42 handshake this is not a refusal
# at all -- it is the expected bootstrap sequence. ``_session`` deliberately opens
# the control subscriptions immediately, in case the relay serves unauthenticated
# reads; a closed relay answers ``auth-required:`` and an ``AUTH`` challenge, and
# the auth branch then re-opens every subscription. Treating that as a permanent
# refusal produced an operator-facing warning claiming discovery/membership
# tracking was DOWN at the exact moment it was coming up -- observed live, in the
# same run where discovery then completed, every channel was subscribed, and a
# brand-new channel's kind-44100 was picked up one second later.
#
# AFTER the handshake the same reason means the relay stopped accepting our
# authenticated session, which is a genuine outage and stays loud.
#
# "Completed its NIP-42 handshake" means the relay has ACKNOWLEDGED our AUTH
# event (an ``OK ... true`` for it, or -- if the relay never sends one --
# ``_confirm_auth_if_pending``'s fallback), never merely that we SENT it. A live
# relay was observed still processing an AUTH we had already sent when it closed
# the just-reopened control subscriptions with this same reason, so a flag set at
# send-time misclassified that as a post-auth refusal instead of the bootstrap
# race it still was.
_AUTH_REQUIRED_CLOSE_PREFIX = "auth-required:"
# NIP-01 asks relays to give ``CLOSED`` a machine-readable prefix. These are the
# ones where re-issuing the IDENTICAL REQ cannot succeed, so retrying is just
# fighting the relay:
# auth-required: -- the relay wants a NIP-42 AUTH, and it sends an ``AUTH``
# challenge to say so; ``_session``'s auth branch re-opens every
# subscription once that completes, which is the real recovery.
# restricted: / blocked: / mute: -- this identity is not permitted to read this.
# invalid: / pow: -- the filter itself was rejected; the same filter will be again.
_PERMANENT_CLOSE_PREFIXES = (_AUTH_REQUIRED_CLOSE_PREFIX, "restricted:", "blocked:", "mute:", "invalid:", "pow:")
# buzz-relay does not always use a NIP-01 prefix -- it closes a channel's
# subscription with prose when access is revoked (e.g. the channel was archived, or
# we were removed from it) -- and NIP-01's generic categories (``error:``,
# ``rate-limited:``) leave the actual condition in the human-readable remainder, so
# ``error: channel not found`` is a permanent close wearing a retryable category.
# The prose is therefore matched regardless of prefix. Matching is deliberately
# narrow and everything unmatched DEFAULTS TO TRANSIENT: the failure this whole
# change exists to remove is going silently deaf, so an unrecognised reason resolves
# toward "keep listening", and ``MAX_RESUBSCRIBE_ATTEMPTS`` bounds the cost of
# guessing wrong in that direction.
_PERMANENT_CLOSE_MARKERS = ("revoke", "not a member", "not a channel member", "no longer a member", "access denied", "unauthorized", "forbidden", "archiv", "not found", "does not exist")
# Wording mirrors every sibling adapter's `/connect` reply style (discord.py's
# `_send_connection_reply`, slack.py's `_post_connection_reply`, and the same
# templates in wecom.py / feishu.py / dingtalk.py / wechat.py), substituting
# "Buzz" for the platform name.
_CONNECT_REPLY_TEXT = {
"success": "Buzz connected to DeerFlow.",
"invalid": "Buzz connection code is invalid or expired.",
"error": "Buzz connection could not be completed from this message.",
}
def _chunk_text(text: str, limit: int = EDIT_MAX_BYTES) -> list[str]:
"""Split *text* into chunks whose UTF-8 ENCODED byte length never exceeds *limit*.
Operates character-by-character (not on raw encoded bytes), so a multi-byte
UTF-8 character is always appended to a chunk whole -- it is measured before
being added and, if it would push the running byte total over *limit*, the
chunk is flushed first and the character starts the next one. This makes a
split-mid-character corruption structurally impossible, regardless of
whether *limit* happens to be a multiple of any character's byte width.
"""
chunks, current, size = [], [], 0
for ch in text:
b = len(ch.encode())
if size + b > limit and current:
chunks.append("".join(current))
current, size = [], 0
current.append(ch)
size += b
if current:
chunks.append("".join(current))
return chunks or [""]
def _is_auth_required_close(reason: str) -> bool:
"""Does the relay's ``CLOSED`` reason say "authenticate first" (NIP-42)?"""
return (reason or "").strip().lower().startswith(_AUTH_REQUIRED_CLOSE_PREFIX)
# DeerFlow's hidden model-context wrappers. These are literal tags this codebase
# injects into the model's input and never into an assistant reply:
# ``DynamicContextMiddleware`` wraps recalled memory in ``<memory>`` and the date
# reminder in ``<system-reminder>``; ``DurableContextMiddleware`` wraps the
# conversation summary, delegation ledger and active skills in
# ``<durable_context_data>``. Matching the opening tag (not the words) is what
# keeps an ordinary reply that merely talks about memory publishable.
_HIDDEN_CONTEXT_MARKERS = ("<memory>", "<durable_context_data>", "<system-reminder>")
def _hidden_context_marker(text: str) -> str | None:
"""Return the hidden-context wrapper *text* carries, if any."""
lowered = (text or "").lower()
return next((marker for marker in _HIDDEN_CONTEXT_MARKERS if marker in lowered), None)
def _is_transient_close(reason: str) -> bool:
"""Could re-issuing the identical REQ plausibly succeed?
This is the whole distinction between "recover from a relay hiccup" and "fight
the relay over a channel we are no longer in". It reads only the relay's stated
reason, so it is exactly as good as what the relay chose to say -- which is why
the two directions are weighted differently:
- A recognised *permanent* reason (a NIP-01/NIP-42 category prefix, or
revocation/removal prose anywhere in the message) is believed and NOT retried.
Re-subscribing to a channel we have been thrown out of would achieve nothing
and look like an attack from the relay's side.
- Anything else -- a generic ``rate-limited:``/``error:`` category, or a
``CLOSED`` with no reason at all, which NIP-01 permits -- is treated as
transient and retried. The connector's worst failure mode is going silently
deaf, so an unknown reason resolves toward "keep listening";
``MAX_RESUBSCRIBE_ATTEMPTS`` is what stops that from becoming a loop when the
guess is wrong.
"""
text = (reason or "").strip().lower()
if text.startswith(_PERMANENT_CLOSE_PREFIXES):
return False
return not any(marker in text for marker in _PERMANENT_CLOSE_MARKERS)
class BuzzChannel(Channel):
_connect: Any = None # test seam: async callable returning an async-context-manager transport
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="buzz", bus=bus, config=config)
self._relay_url = str(config.get("relay_url", "")).strip()
if not self._relay_url.startswith(("ws://", "wss://")):
raise ValueError("channels.buzz.relay_url must be a ws:// or wss:// URL")
# One community per relay URL (see the design's multi-community note), so the
# relay host is this channel's workspace: it scopes inbound dedupe, the
# persisted connection row written by `/connect`, and the lookup that resolves
# that row back on the inbound path. Computed once here so those three uses
# can never drift apart.
self._workspace_id = urlparse(self._relay_url).netloc
self._private_key_raw = str(config.get("private_key", ""))
self._keys: buzz_nostr.NostrKeys | None = None # parsed in start() so coincurve stays lazy
self._allowed_users = {buzz_nostr.parse_pubkey(v) for v in config.get("allowed_users", []) or []}
self._require_mention = bool(config.get("require_mention", True))
self._mention_free = {str(c) for c in config.get("mention_free_channels", []) or []}
self._channel_meta: dict[str, dict[str, Any]] = {}
self._stream_targets: dict[tuple[str, str | None], str] = {}
self._stream_tails: dict[tuple[str, str | None], list[str]] = {} # overflow chunk ids beyond chunk 0, per conversation
self._last_requester: dict[tuple[str, str | None], str] = {}
self._pending_auth_challenge: str | None = None # set from an AUTH relay frame; consumed by the NIP-42 flow in _session
self._seen_created_at: dict[str, int] = {} # channel id -> high-water mark of PROCESSED created_at (see _advance_watermark)
self._chat_subscriptions: set[str] = set() # channel ids with a live per-channel REQ on the CURRENT connection
self._resubscribe_attempts: dict[str, int] = {} # sub id -> CLOSED-recovery attempts spent on the current connection/auth epoch
self._auth_completed = False # has THIS socket's NIP-42 handshake been ACKNOWLEDGED by the relay? (see _handle_auth_ok, _AUTH_REQUIRED_CLOSE_PREFIX)
self._pending_auth_event_id: str | None = None # id of the AUTH event most recently sent, awaiting a matching OK (see _handle_auth_ok / _confirm_auth_if_pending)
self._session_started_at: int | None = None # wall clock at which the CURRENT socket opened; anchors the live membership filter
self._transport: Any = None
self._task: asyncio.Task | None = None
self._publish = self.bus.publish_inbound # test seam (discord.py idiom)
@property
def supports_streaming(self) -> bool:
return True
# -- lifecycle ---------------------------------------------------------
async def start(self) -> None:
if self._running:
return
self._keys = buzz_nostr.parse_private_key(self._private_key_raw)
if not self._allowed_users:
# Deny-by-default is deliberate (unlike siblings, where an empty
# allowlist means "allow all"), so this cannot be a hard failure --
# but a configured-and-connected channel that silently drops every
# message looks exactly like a broken relay to the operator. Say so
# once, loudly, at the only point where it is actionable.
logger.warning("[buzz] channels.buzz.allowed_users is empty: EVERY inbound chat message will be dropped (Buzz denies by default). Add member pubkeys (hex or npub) to enable the channel.")
self.bus.subscribe_outbound(self._on_outbound)
self._spawn_connection()
self._running = True
logger.info("[buzz] channel started (relay=%s pubkey=%s allowed_users=%d)", self._relay_url, self._keys.pubkey_hex, len(self._allowed_users))
def _spawn_connection(self) -> None:
self._task = asyncio.create_task(self._run_loop(), name="buzz-relay-loop")
async def stop(self) -> None:
"""Cancel the relay loop and drop every piece of per-connection state.
Teardown is *bounded* and *coherent*, in that order:
- Bounded: the cancelled task is awaited via ``asyncio.wait`` rather than
``asyncio.wait_for``. ``wait_for`` guarantees the awaited task is
finished before it raises ``TimeoutError``, so a task that swallows
``CancelledError`` (a bug, but the exact case a timeout exists for)
would hang ``stop()`` forever instead of the intended 5s.
``asyncio.wait`` returns after the timeout regardless and cancels
nothing further, so this always returns.
- Coherent: the reviewer's finding was that a timed-out ``_task`` was
dropped while it might still own ``_transport``, leaving an abandoned
task posting on a socket the channel believed it no longer held. The
``finally`` now clears ``_transport`` (so ``_post_event`` fails fast
rather than writing through a socket nobody owns) alongside ``_task``,
and the timeout is logged instead of passing silently.
Per-connection bookkeeping (stream placeholders, overflow tails, last
requester, half-consumed auth challenge, remote-fed channel metadata,
per-channel chat subscriptions) is also cleared: those maps key on relay
event ids and subscription ids from the session being torn down, so a
later ``start()`` must not resume editing placeholders -- or believe it is
still subscribed to anything -- from a previous process lifetime. The
per-channel replay cursors (``_seen_created_at``) deliberately survive, so
a restart resumes where it left off instead of replaying every channel.
"""
if not self._running:
return
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
if self._task is not None:
task = self._task
task.cancel()
try:
_, pending = await asyncio.wait({task}, timeout=STOP_TIMEOUT_SECONDS)
if pending:
logger.warning("[buzz] relay loop did not finish within %ss of cancellation; abandoning it", STOP_TIMEOUT_SECONDS)
elif not task.cancelled() and task.exception() is not None:
# _run_loop is designed to never end on an ordinary connection error (it
# backs off and retries instead -- see its docstring), but stop() must
# still complete cleanly rather than re-raise whatever a genuine bug in
# the relay loop task ended with.
logger.error("[buzz] relay loop task ended with an error during stop: %s", task.exception())
except asyncio.CancelledError:
raise
except Exception:
logger.exception("[buzz] failed to await the relay loop task during stop")
finally:
self._task = None
self._transport = None
self._stream_targets.clear()
self._stream_tails.clear()
self._last_requester.clear()
self._channel_meta.clear()
self._chat_subscriptions.clear()
self._resubscribe_attempts.clear()
self._auth_completed = False
self._pending_auth_event_id = None
self._session_started_at = None
self._pending_auth_challenge = None
logger.info("[buzz] channel stopped")
# -- subscriptions ------------------------------------------------------
@staticmethod
def _chat_sub_id(channel_id: str) -> str:
return f"{CHAT_SUB_PREFIX}{channel_id}"
def _chat_filter(self, channel_id: str) -> dict:
"""The NIP-01 filter for ONE channel's chat subscription.
``#h`` is not an optimization -- it is the only shape the relay fans kind-9
events out to (see the module docstring), and it may carry exactly one value.
``since`` rides along only once we have a watermark for THIS channel
(``_seen_created_at``, advanced by ``_advance_watermark``): a channel we
have not processed anything in yet gets no ``since``, so the relay's default
backlog applies, and afterwards it is the newest ``created_at`` we have
already processed in that channel -- so re-subscribing after a drop neither
replays the whole channel nor silently skips what was published while we
were disconnected.
"Processed" means accepted-and-published (or a fully handled ``/connect``),
never merely received. That is strictly safer for the skip direction: the
cursor can only ever be lower than it would have been under an
advance-on-everything rule, so any event that rule would have covered is
still covered. What it costs is replay -- events we already dropped come
back after a reconnect and are dropped again by the same gates, which is
idempotent -- plus one guaranteed redelivery of the boundary event itself,
since NIP-01 ``since`` is inclusive; the manager's inbound dedupe (keyed on
our ``event_id`` metadata within this workspace) is what keeps that from
starting a second run.
KNOWN BOUND: ``since`` narrows the query but does not lift the relay's cap on
how much history one subscription may be served -- buzz-relay answers with at
most 2000 stored events, NEWEST-FIRST. So if a single channel accumulated more
than 2000 messages while we were disconnected, the relay simply never sends
the oldest of them, and the watermark advances past them as the newer ones are
processed. That is the one bounded skip path that remains after this fix
(everything else fails toward replay); closing it would mean paging the
backlog with descending ``until`` queries rather than one REQ per channel.
"""
chat_filter: dict = {"kinds": [buzz_nostr.KIND_CHAT], "#h": [channel_id]}
since = self._seen_created_at.get(channel_id)
if since:
chat_filter["since"] = since
return chat_filter
def _discovery_filter(self) -> dict:
"""Historical kind-39000 query: which channels is this identity a member of?
Deliberately unfiltered beyond the kind. The relay scopes kind-39000 by
membership itself, returning one stored event per channel we belong to
followed by EOSE -- adding ``#p`` (an obvious-looking narrowing) matches
nothing and returns zero channels, verified against a live relay.
"""
return {"kinds": [buzz_nostr.KIND_CHANNEL_META]}
def _membership_filter(self) -> dict:
"""Live membership notifications addressed to us: added to / removed from a channel.
``since`` is not an optimization, it is the difference between "live" and
"the entire history of this identity's membership, replayed as news".
buzz-relay STORES 44100/44101 events and serves history newest-first with a
default limit of 2000, so an unscoped filter meant every connection:
- logged "added to channel ...; subscribing" once per historical add, each
re-issuing the discovery REQ, so one connect produced M+1 discovery passes
over N stored kind-39000 events (the two "channel discovery complete"
lines seen live, and the channel that logged as ``<unnamed>`` because the
44100 path subscribed before its metadata had arrived);
- re-subscribed every channel we were EVER added to, including ones we have
since been removed from, burning subscription-cap slots; and
- let a historical 44101 unsubscribe (and ``_channel_meta.pop``) a channel we
are still in -- recovered only incidentally, because newest-first ordering
happens to deliver the oldest add last.
The window this must not lose is a membership change published DURING the
connect/auth handshake: it is genuinely live, but the relay stamps it with
its own clock and it is already in the past by the time the post-auth REQ
goes out. ``MEMBERSHIP_LOOKBACK_SECONDS`` of slack behind the moment the
socket opened covers both that window and relay/DeerFlow clock skew. Cost of
the slack: at most the last minute of membership changes replayed, which is
idempotent (``_ensure_chat_subscription`` no-ops on an already-subscribed
channel, and ``_handle_membership_event`` only re-runs discovery when the
channel's metadata is actually missing). Cost of not having it: a channel we
were added to during the handshake is never heard from until a reconnect.
"""
assert self._keys is not None
started_at = self._session_started_at if self._session_started_at is not None else int(time.time())
since = max(0, started_at - MEMBERSHIP_LOOKBACK_SECONDS)
return {"kinds": [buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED], "#p": [self._keys.pubkey_hex], "since": since}
async def _open_control_subscriptions(self, ws) -> None:
"""Channel discovery plus membership notifications: the two per-connection subscriptions."""
await ws.send(buzz_nostr.req_frame(DISCOVERY_SUB_ID, self._discovery_filter()))
await ws.send(buzz_nostr.req_frame(MEMBERSHIP_SUB_ID, self._membership_filter()))
async def _ensure_chat_subscription(self, channel_id: str) -> None:
"""Open this channel's chat subscription unless it is already open on this connection.
Never raises: it runs from the relay read loop, where a failed REQ must not
take down the connection. A failure simply leaves the channel unsubscribed
and out of ``_chat_subscriptions``, so the discovery EOSE sweep (or the next
kind-39000 for it, or a reconnect) retries.
"""
if not channel_id or channel_id in self._chat_subscriptions:
return
transport = self._transport
if transport is None:
return # no connection: subscriptions are per-socket and are rebuilt by _session
if len(self._chat_subscriptions) >= MAX_CHANNEL_SUBSCRIPTIONS:
logger.warning("[buzz] per-channel subscription limit reached (%d); not listening to channel %s", MAX_CHANNEL_SUBSCRIPTIONS, channel_id)
return
try:
await transport.send(buzz_nostr.req_frame(self._chat_sub_id(channel_id), self._chat_filter(channel_id)))
except Exception:
logger.warning("[buzz] failed to subscribe to channel %s; will retry", channel_id, exc_info=True)
return
self._chat_subscriptions.add(channel_id)
logger.info("[buzz] listening to channel %s (%s)", self._channel_meta.get(channel_id, {}).get("name") or "<unnamed>", channel_id)
async def _close_chat_subscription(self, channel_id: str) -> None:
"""Stop listening to one channel, leaving every other subscription on this socket alone."""
if channel_id not in self._chat_subscriptions:
return
self._chat_subscriptions.discard(channel_id)
transport = self._transport
if transport is None:
return
try:
await transport.send(buzz_nostr.close_frame(self._chat_sub_id(channel_id)))
except Exception:
logger.warning("[buzz] failed to unsubscribe from channel %s", channel_id, exc_info=True)
def _claim_resubscribe_attempt(self, sub_id: str) -> int | None:
"""Spend one retry from *sub_id*'s budget for this connection; ``None`` once spent.
The counter deliberately does NOT reset when a re-subscribe succeeds. A relay
that accepts the REQ and then closes it again is the exact loop the budget
exists to bound, and "it worked for a moment" is not evidence that it is
working. The whole map is reset when the socket (or the auth epoch) changes,
which is the only event that genuinely makes the past irrelevant.
Keys are our own subscription ids and only ever added for subscriptions we
actually held, but a long-lived connection can cycle through many channels,
so the map is FIFO-capped like the other per-channel maps. Eviction only ever
hands a channel a fresh retry budget.
"""
used = self._resubscribe_attempts.get(sub_id, 0)
if used >= MAX_RESUBSCRIBE_ATTEMPTS:
return None
self._resubscribe_attempts[sub_id] = used + 1
while len(self._resubscribe_attempts) > MAX_CACHED_CHANNELS:
self._resubscribe_attempts.pop(next(iter(self._resubscribe_attempts)))
return used + 1
async def _resubscribe_backoff(self, attempt: int) -> None:
"""Pause before re-issuing a closed REQ; the first attempt is immediate."""
if attempt <= 1:
return
await asyncio.sleep(RESUBSCRIBE_BASE_DELAY_SECONDS * 2 ** (attempt - 2))
async def _handle_closed(self, sub_id: str, reason: str) -> None:
"""Route a relay ``CLOSED`` frame to the recovery its subscription needs.
A ``CLOSED`` is the relay dropping a subscription, and EVERY subscription on
this socket fails silently when that happens: a chat subscription deafens one
channel, ``buzz-membership`` stops us learning about channels we are added to
or removed from, ``buzz-discovery`` kills the completeness sweep and the
"no channels" warning. Forgetting the subscription is therefore only half a
response -- something has to re-open it, or the connector runs the rest of
that socket's life quietly missing traffic it believes it is receiving.
The one close that is NOT an outage is ``auth-required:`` before this
socket has completed its NIP-42 handshake: that is the bootstrap sequence
working as designed (opportunistic pre-auth REQs, refused by a closed
relay, re-opened wholesale by ``_session``'s auth branch). It is handled
here, ahead of every recovery path, so it consumes neither the
permanent-refusal branch nor the retry budget and says nothing alarming --
the previous behaviour warned that discovery/membership tracking was down
at the exact moment it was coming up. A chat subscription is still dropped
from the live set, because it genuinely is not subscribed from this moment
and the auth branch/discovery sweep is what re-opens it.
"""
if _is_auth_required_close(reason) and not self._auth_completed:
if sub_id.startswith(CHAT_SUB_PREFIX):
self._chat_subscriptions.discard(sub_id[len(CHAT_SUB_PREFIX) :])
logger.debug("[buzz] relay closed %s pending NIP-42 auth (%s); the auth handshake re-opens it", sub_id, reason or "<no reason>")
return
if sub_id == DISCOVERY_SUB_ID or sub_id == MEMBERSHIP_SUB_ID:
await self._recover_control_subscription(sub_id, reason)
elif sub_id.startswith(CHAT_SUB_PREFIX):
await self._recover_chat_subscription(sub_id[len(CHAT_SUB_PREFIX) :], reason)
else:
logger.info("[buzz] relay closed unrecognized subscription %s: %s", sub_id, reason or "<no reason>")
async def _recover_control_subscription(self, sub_id: str, reason: str) -> None:
"""Re-issue a closed ``buzz-discovery`` / ``buzz-membership`` REQ, bounded.
These two are opened ONLY by ``_open_control_subscriptions`` -- at session
start and in the auth branch -- so before this existed a post-auth ``CLOSED``
for either one was terminal for the connection: nothing else in the process
would ever send that REQ again.
A permanent reason is not retried here, and for ``auth-required:`` that is
not a gap: the relay pairs it with an ``AUTH`` challenge, and ``_session``'s
auth branch re-opens both control subscriptions once the NIP-42 handshake
completes. Re-issuing here as well would only race that. (The *pre*-auth
``auth-required:`` case never reaches this function at all -- see
``_handle_closed`` -- because there nothing is down.)
The two permanent branches are logged separately because they are not the
same outage. A relay demanding re-authentication mid-session is recovered
by the next ``AUTH`` challenge; a ``restricted:``/``blocked:``/``invalid:``
refusal is not recovered until the connection is rebuilt. Saying "down
until the next reconnect or NIP-42 re-auth" for both was what made the
bootstrap warning read as an outage report.
"""
logger.warning("[buzz] relay closed control subscription %s: %s", sub_id, reason or "<no reason>")
if not _is_transient_close(reason):
if _is_auth_required_close(reason):
logger.warning("[buzz] relay demanded re-authentication for %s AFTER this connection completed NIP-42 auth. It stays down until the relay's next AUTH challenge re-opens it, or until the next reconnect.", sub_id)
else:
logger.warning("[buzz] not re-opening %s: the relay refused it. Channel discovery/membership tracking is down until the next reconnect.", sub_id)
return
attempt = self._claim_resubscribe_attempt(sub_id)
if attempt is None:
logger.warning("[buzz] gave up re-opening control subscription %s after %d attempts; it stays down until the next reconnect", sub_id, MAX_RESUBSCRIBE_ATTEMPTS)
return
await self._resubscribe_backoff(attempt)
transport = self._transport
if transport is None:
return # the socket went away while backing off; _session rebuilds everything
sub_filter = self._discovery_filter() if sub_id == DISCOVERY_SUB_ID else self._membership_filter()
try:
await transport.send(buzz_nostr.req_frame(sub_id, sub_filter))
except Exception:
logger.warning("[buzz] failed to re-open control subscription %s", sub_id, exc_info=True)
return
logger.info("[buzz] re-opened control subscription %s (attempt %d/%d)", sub_id, attempt, MAX_RESUBSCRIBE_ATTEMPTS)
async def _recover_chat_subscription(self, channel_id: str, reason: str) -> None:
"""Re-open one channel's chat subscription after the relay closed it, bounded.
Only for a subscription we actually held. A ``CLOSED`` frame is relay-supplied
input, so treating one for an unknown channel as a recovery would let any
relay induce a chat subscription to a channel of its choosing simply by naming
it -- and would also let it grow ``_resubscribe_attempts`` without bound.
The channel is dropped from ``_chat_subscriptions`` first, unconditionally: it
is genuinely not subscribed from this moment, and leaving it there would make
the retry (and any later kind-44100 or discovery sweep) a no-op -- the original
reason ``_forget_subscription`` existed.
"""
if channel_id not in self._chat_subscriptions:
logger.info("[buzz] relay closed chat subscription for channel %s, which we were not subscribed to: %s", channel_id, reason or "<no reason>")
return
self._chat_subscriptions.discard(channel_id)
name = self._channel_meta.get(channel_id, {}).get("name") or "<unnamed>"
logger.warning("[buzz] relay closed the chat subscription for channel %s (%s), so it is now UNLISTENED: %s", name, channel_id, reason or "<no reason>")
if not _is_transient_close(reason):
# Removed from the channel, or not authorized to read it: re-subscribing
# would be fighting the relay over a channel that is no longer ours. The
# metadata entry is deliberately left alone -- the next discovery pass is
# authoritative about membership, and ``_on_discovery_complete`` already
# documents the one-REQ-per-stale-channel cost of a stale entry.
logger.warning("[buzz] not re-subscribing to channel %s: the relay's reason says the subscription is no longer ours", channel_id)
return
attempt = self._claim_resubscribe_attempt(self._chat_sub_id(channel_id))
if attempt is None:
logger.warning("[buzz] gave up re-subscribing to channel %s (%s) after %d attempts; it stays unlistened until the next reconnect", name, channel_id, MAX_RESUBSCRIBE_ATTEMPTS)
return
await self._resubscribe_backoff(attempt)
await self._ensure_chat_subscription(channel_id)
def _confirm_auth_if_pending(self) -> None:
"""Fallback NIP-42 confirmation for a relay that never sends an ``OK`` for AUTH.
NIP-42 says a relay SHOULD reply ``OK`` to an AUTH event, but "should" is
not "does", and this connector must not depend on relay behaviour it
cannot control to ever leave its pre-auth-confirmed state. Without this
fallback, a relay that silently accepts the AUTH but never sends an ``OK``
would leave ``_auth_completed`` false for the rest of the connection, and
EVERY later ``auth-required:`` close would misclassify as bootstrap noise
-- the same lie this whole fix removes, just permanently instead of for
one race window.
Chosen over a bounded sleep/timeout: this is driven by the relay's own
response rather than the wall clock, so it needs no sleep, cannot fire
early, and adds no unbounded wait to the read loop. Reaching the discovery
subscription's EOSE is a safe signal because it only happens on a REQ the
relay actually served: an auth-rejecting relay CLOSEs the just-reopened
control subscriptions instead (routed to ``_handle_closed``, never here),
so seeing EOSE without a CLOSED first is itself proof the AUTH we sent was
accepted, explicit ``OK`` or not.
Scoped to "we sent an AUTH and have not yet seen an ``OK`` for it"
(``_pending_auth_event_id`` set) so it can never fire before an AUTH was
even attempted -- where "not yet authenticated" is already the correct
read and this must stay a no-op.
"""
if self._pending_auth_event_id is None:
return
self._pending_auth_event_id = None
self._auth_completed = True
logger.info("[buzz] treating successful discovery as implicit NIP-42 confirmation (relay sent no explicit OK for our AUTH event)")
async def _on_discovery_complete(self) -> None:
"""EOSE for the discovery subscription: every channel we belong to has now been sent.
Chat subscriptions are opened as each kind-39000 arrives, so this is normally
a no-op sweep. It exists because that is the point where the result can be
relied on: a channel whose REQ failed mid-discovery is retried here, and a
connector that discovered nothing is called out rather than sitting silent.
The sweep runs over the whole metadata cache rather than only what this
connection discovered, which can re-subscribe to a channel we were removed
from while disconnected (kind-44101 prunes the cache, but only if we were
connected to receive it). That is self-correcting and deliberately not
engineered around: the relay answers with ``CLOSED``, whose reason names a
removal or a revocation, so ``_recover_chat_subscription`` classifies it as
permanent, drops the subscription, and does NOT retry -- the cost is one REQ
per stale channel per reconnect and never a wrong-channel message, since the
allowlist and signature gates are what decide whether anything is acted on.
Also the (fallback) point where ``_confirm_auth_if_pending`` is given its
chance to run: reaching this EOSE at all is only possible on a REQ the
relay actually served, which is evidence the AUTH we sent was accepted even
if the relay never sent an explicit ``OK`` for it. See that method's
docstring for why this is a safe signal and why a bounded sleep was not
used instead.
"""
self._confirm_auth_if_pending()
for channel_id in list(self._channel_meta):
await self._ensure_chat_subscription(channel_id)
if self._chat_subscriptions:
logger.info("[buzz] channel discovery complete: listening to %d channel(s)", len(self._chat_subscriptions))
else:
logger.warning("[buzz] channel discovery returned no channels: this identity is not a member of any channel on %s (add it with `buzz channels add-member`)", self._relay_url)
def _handle_auth_ok(self, event_id: str, accepted: bool, message: str) -> None:
"""React to a relay ``OK`` acknowledgment, specifically for our outstanding NIP-42 AUTH event.
NIP-01 sends ``["OK", <event_id>, <accepted>, <message>]`` for every event
this connector publishes -- a chat post, an edit, an AUTH -- not only AUTH,
and this connector does not otherwise track its own published event ids to
correlate them against. So anything that is not the one outstanding AUTH
event id is silently ignored here: a rejected chat publish is a delivery
problem for ``send()``'s own retry path, not this session's auth
bookkeeping.
This -- the relay's own affirmative reply for the EXACT event id we sent
-- is what is allowed to flip ``_auth_completed``, never the act of
sending the AUTH event itself. Sending only means we tried: a live relay
was observed still processing the AUTH it would eventually accept while it
closed the just-reopened control subscriptions with ``auth-required:``, and
a flag set at send-time misread that ordinary bootstrap tail as a genuine
post-auth refusal (see ``_AUTH_REQUIRED_CLOSE_PREFIX`` and
``_handle_closed``). A relay that never sends an ``OK`` at all is covered
by ``_confirm_auth_if_pending``'s fallback instead.
``OK ... false`` is the other half of this fix: the relay REJECTED our
AUTH, which is a real, actionable problem (a bad key, clock skew outside
the relay's tolerance, a relay-side policy) and is logged loudly here
rather than silently leaving the connection merely "not yet
authenticated". It must never set ``_auth_completed`` -- the session stays
in its pre-auth-confirmed state, quiet-recoverable the normal way if a
fresh ``AUTH`` challenge arrives.
"""
if event_id != self._pending_auth_event_id:
return
self._pending_auth_event_id = None
if accepted:
self._auth_completed = True
logger.info("[buzz] relay acknowledged our NIP-42 AUTH event")
else:
logger.warning("[buzz] relay REJECTED our NIP-42 AUTH event: %s", message or "<no reason given>")
async def _session(self, ws) -> None:
"""Run one relay connection's lifetime: authenticate, discover, subscribe, read frames.
``self._transport`` is set for the duration of this connection so ``send()``
/ ``_post_event()`` can post outbound events on it, and is unconditionally
cleared in ``finally`` on the way out -- including on error -- so a stale
reference can never survive past this connection (``_post_event`` reads it
fresh on every call and raises when it is ``None``; see its docstring).
``_chat_subscriptions`` is per-socket bookkeeping and is cleared alongside
it, so the next connection re-sends every REQ instead of believing
subscriptions from a dead socket are still live.
NIP-42 auth is opportunistic, not upfront: discovery/membership REQs are
sent immediately in case the relay allows unauthenticated reads, but
``handle_relay_frame`` records any ``AUTH`` challenge the relay sends onto
``self._pending_auth_challenge``, and this loop reacts to it the next time
around by signing and sending the AUTH event and then RE-RUNNING DISCOVERY
-- our relay is closed, so the pre-auth REQs may have been rejected, and
without redoing them we would end up authenticated but listening to nothing.
The per-channel subscription set is dropped at the same moment for the same
reason: a chat REQ issued pre-auth may have been rejected too, so it must
not be remembered as live. A fresh challenge is per-connection (the relay
mints a new one on every new socket), so all of this runs again from
scratch on every reconnect.
``_resubscribe_attempts`` (the per-subscription ``CLOSED``-recovery budget)
is reset at exactly those two points, and for the same reason: a pre-auth
REQ that the relay closed because we had not authenticated yet is recovered
wholesale by the auth branch, and must not spend the budget that protects the
authenticated session from a relay that keeps closing a live subscription.
``_auth_completed`` is the same boundary expressed as a flag, and it is
strictly per socket (cleared on entry AND in ``finally``): it is what lets
``_handle_closed`` tell the expected bootstrap ``auth-required:`` refusal
apart from a relay that stops honouring an already-authenticated session.
It is set ONLY once the relay has ACKNOWLEDGED our AUTH event with a
matching ``OK ... true`` (``_handle_auth_ok``) -- never merely because we
sent one. A live relay was observed still processing the AUTH we had
already sent when it closed the just-reopened control subscriptions with
this same ``auth-required:`` reason, so a flag set at send-time
misclassified the tail of the ordinary bootstrap race as a post-auth
refusal (the operator-facing warning that started this fix). Sending AUTH
here only records ``_pending_auth_event_id`` so the eventual ``OK`` can be
matched back to it; a relay that never sends one at all is covered by
``_confirm_auth_if_pending``'s fallback (see its docstring). A relay that
REJECTS the AUTH (``OK ... false``) is a real, actionable problem and is
logged loudly by ``_handle_auth_ok`` -- without ever setting this flag, so
the connection is never mistaken for authenticated.
"""
self._transport = ws
self._session_started_at = int(time.time())
self._chat_subscriptions.clear()
self._resubscribe_attempts.clear()
self._auth_completed = False
self._pending_auth_event_id = None
try:
await self._open_control_subscriptions(ws)
async for raw in ws:
await self.handle_relay_frame(raw)
if self._pending_auth_challenge is not None:
assert self._keys is not None
challenge = self._pending_auth_challenge
self._pending_auth_challenge = None
auth = buzz_nostr.build_auth_event(self._keys, self._relay_url, challenge, created_at=int(time.time()))
self._pending_auth_event_id = auth["id"]
await ws.send(json.dumps(["AUTH", auth], separators=(",", ":")))
self._chat_subscriptions.clear()
self._resubscribe_attempts.clear()
await self._open_control_subscriptions(ws)
finally:
self._transport = None
self._chat_subscriptions.clear()
self._resubscribe_attempts.clear()
self._auth_completed = False
self._pending_auth_event_id = None
self._session_started_at = None
async def _run_loop(self) -> None:
"""Own the relay connection for the channel's lifetime: connect, run one session, retry forever.
Runs as the asyncio task ``_spawn_connection`` starts; ``stop()`` cancels it.
Every sibling connector (wecom.py, discord.py, feishu.py) delegates
reconnection to its vendor SDK -- Buzz has no SDK, so this loop owns
reconnect/backoff itself: exponential backoff capped at 60s with jitter
(so a thundering herd of Buzz-connected agents restarting together does not
all retry in lockstep), reset to 0 after any connection that actually got
established (so a long stable run doesn't leave a stale attempt count that
punishes the next transient blip with a large initial delay).
A clean stream end -- ``_session`` returning normally because the relay
closed the socket on us -- is treated the same as a connection error: both
fall through to the backoff-and-retry below rather than ending the loop,
because for a relay client "the peer hung up" is exactly the situation
reconnection exists for.
``asyncio.CancelledError`` is re-raised immediately rather than reaching the
generic ``except Exception`` below -- it isn't an ``Exception`` subclass in
the supported Python version anyway, but the explicit clause documents the
contract: this is how the guarded ``stop()`` (Task 3) actually stops this
otherwise-infinite loop, and it must never be swallowed here.
Untrusted relay input is guarded one layer down: ``handle_relay_frame``
already logs-and-drops a non-JSON payload or a malformed ``EVENT`` instead
of raising, so a single bad frame reaches neither this loop's
``except Exception`` (reconnect) nor kills the read loop -- it is simply
skipped and iteration continues.
"""
attempt = 0
while True:
try:
if self._connect is not None:
ws = await self._connect()
else:
import websockets
ws = await websockets.connect(self._relay_url)
async with ws:
attempt = 0 # reset only once a connection is actually established
await self._session(ws)
# Clean stream end (relay closed on us): reconnect with backoff too.
attempt += 1
except asyncio.CancelledError:
raise
except Exception as exc:
attempt += 1
logger.warning("[buzz] relay connection error (attempt %d): %s", attempt, exc)
# Cap the counter itself, not just the delay: 2**10 already exceeds the
# 60s ceiling below, so clamping here keeps the exponent computation
# cheap indefinitely instead of growing the attempt count (and the
# bigint math behind 2**attempt) without bound across a very long
# outage.
attempt = min(attempt, 10)
delay = min(60, 2**attempt) + random.uniform(0, 1)
await asyncio.sleep(delay)
# -- inbound -------------------------------------------------------------
async def handle_relay_frame(self, raw: str) -> None:
"""Route one raw relay WebSocket frame (Task 6 calls this per frame received).
Relay input is untrusted: a non-JSON payload, an unexpected frame shape, or
an ``EVENT`` payload whose tags/timestamp are malformed is logged and dropped
rather than raised, so one bad frame can never crash the read loop.
Every ``EVENT`` is signature-verified (``buzz_nostr.verify_event``) here,
at the single entry point, BEFORE either handler runs. The relay operator
is not necessarily the DeerFlow operator on a team-run Buzz relay, and
``ev["pubkey"]`` is just a field in a relay-supplied JSON object -- without
this check a malicious or compromised relay could name any allowlisted
author it liked and trigger tool-executing runs, or bind a victim's pubkey
to an attacker's DeerFlow account through ``/connect``. Both handlers make
authorization decisions from the event (the chat gate uses ``pubkey`` as
the principal; kind-39000 metadata can relax the mention requirement), so
verifying at the choke point rather than inside each one leaves no path
where an unverified event reaches a decision.
Chosen over verifying later, behind the cheap self/kind/channel gates, on
purpose: a Schnorr verify is tens of microseconds against human-rate chat
traffic, so the throughput saved by gating first is worth less than the
guarantee that no future edit can reorder a gate ahead of the check.
"""
try:
frame = json.loads(raw)
except json.JSONDecodeError:
logger.warning("[buzz] non-JSON relay frame ignored")
return
if not isinstance(frame, list) or not frame:
return
kind = frame[0]
if kind == "AUTH" and len(frame) >= 2:
self._pending_auth_challenge = str(frame[1])
elif kind == "OK" and len(frame) >= 3:
self._handle_auth_ok(str(frame[1]), frame[2] is True, str(frame[3]) if len(frame) >= 4 else "")
elif kind == "EOSE" and len(frame) >= 2 and frame[1] == DISCOVERY_SUB_ID:
await self._on_discovery_complete()
elif kind == "CLOSED" and len(frame) >= 2:
await self._handle_closed(str(frame[1]), str(frame[2]) if len(frame) >= 3 else "")
elif kind == "EVENT" and len(frame) >= 3 and isinstance(frame[2], dict):
ev = frame[2]
if not buzz_nostr.verify_event(ev):
logger.warning("[buzz] dropped relay event with an invalid id/signature (claimed pubkey=%s kind=%s)", ev.get("pubkey"), ev.get("kind"))
return
try:
ev_kind = ev.get("kind")
if ev_kind == buzz_nostr.KIND_CHANNEL_META:
channel_id = self._handle_meta_event(ev)
if channel_id:
await self._ensure_chat_subscription(channel_id)
elif ev_kind == buzz_nostr.KIND_CHAT:
await self._handle_chat_event(ev)
elif ev_kind in (buzz_nostr.KIND_MEMBER_ADDED, buzz_nostr.KIND_MEMBER_REMOVED):
await self._handle_membership_event(ev)
except Exception:
# Defense in depth against malformed tags/timestamps inside an
# otherwise well-shaped EVENT frame (e.g. a non-integer created_at,
# or a "tags" field that isn't a list of [name, value, ...] lists).
logger.warning("[buzz] malformed relay event ignored", exc_info=True)
# other EOSE / NOTICE frames need no action
def _handle_meta_event(self, ev: dict) -> str | None:
"""Cache kind-39000 channel metadata: ``d`` = channel id, ``t`` = type, ``name``.
Returns the channel id it cached (or ``None``), because kind-39000 is also
the channel-DISCOVERY event: the caller turns each one into that channel's
chat subscription. The two uses share one handler on purpose -- the set of
channels we hold metadata for is exactly the set we are a member of.
Trust assumption (authorship, not authenticity): ``handle_relay_frame``
has already proved this event really was signed by the pubkey it names,
but this does not check WHICH pubkey that is. In the Buzz protocol,
kind-39000 channel-discovery events are expected to be published by the
relay's own keypair, not by ordinary members. So any relay member able to
publish an event can still legitimately sign one and mark an arbitrary
channel ``type: "dm"``, which relaxes the mention gate for that channel
(see ``_is_dm``) — it does NOT bypass the independent pubkey allowlist
gate in ``_handle_chat_event``. Closing this gap needs a trusted relay
pubkey to check the author against, and nothing already configured
identifies one: ``relay_url`` is a network address, not a signing key.
Deliberately left as a follow-up rather than inventing a new required
config key (e.g. ``relay_pubkey``) for it here.
The cache is remote-fed, so it is capped (``MAX_CACHED_CHANNELS``) on a
first-in-first-out basis: an unbounded map would let any member grow this
process's memory one forged ``d`` tag at a time. Eviction is safe by
construction — a channel we hold no metadata for is treated as a non-DM
(``_is_dm`` fails closed), and the next kind-39000 event for it repopulates
the entry.
"""
d_values = buzz_nostr.tag_values(ev, "d")
if not d_values:
return None
names = buzz_nostr.tag_values(ev, "name")
types = buzz_nostr.tag_values(ev, "t")
channel_id = d_values[0]
meta = {"type": types[0] if types else "stream", "name": names[0] if names else ""}
self._channel_meta[channel_id] = meta
while len(self._channel_meta) > MAX_CACHED_CHANNELS:
self._channel_meta.pop(next(iter(self._channel_meta)))
logger.debug("[buzz] channel metadata cached: %s (%s) type=%s", meta["name"] or "<unnamed>", channel_id, meta["type"])
return channel_id
async def _handle_membership_event(self, ev: dict) -> None:
"""React to a relay-signed kind-44100 / kind-44101 membership notification.
This is what makes a newly added channel work WITHOUT a reconnect: the relay
publishes one of these (``p`` = the affected member, ``h`` = the channel
uuid) on the membership subscription opened for every connection, so being
added to a channel is a live event rather than something we would only
notice the next time the socket happened to drop.
The ``#p`` filter on the subscription is the relay's claim, not a proof, so
the ``p`` tag is re-checked here: another member's membership change must
never make us subscribe to a channel we do not belong to. (The event's
signature is already verified at ``handle_relay_frame``; what is trusted, as
for kind-39000, is that the signer is the relay -- see the class docstring's
trust model. A member who forges one can at most make us open a subscription
the relay will refuse or answer with nothing.)
Removal drops the cached metadata too: the entry would otherwise keep the
channel in the discovery-EOSE sweep and make every later reconnect
re-subscribe to a channel we have been thrown out of.
"""
assert self._keys is not None
if self._keys.pubkey_hex not in buzz_nostr.tag_values(ev, "p"):
return # someone else's membership change
channel_ids = buzz_nostr.tag_values(ev, "h")
if not channel_ids:
return
channel_id = channel_ids[0]
if ev.get("kind") == buzz_nostr.KIND_MEMBER_ADDED:
known = channel_id in self._channel_meta
logger.info("[buzz] added to channel %s; subscribing", channel_id)
# Subscribe first so no message is missed while metadata catches up, then
# re-run discovery: the new channel's name and type (which drives the DM
# mention exemption) are only carried by its kind-39000 event.
await self._ensure_chat_subscription(channel_id)
if not known:
# ... but only when there is something to learn. Discovery re-issues
# the REQ on its own subscription id, so the relay answers with every
# stored kind-39000 and a fresh EOSE -- i.e. a whole extra discovery
# pass. A 44100 for a channel whose metadata we already hold tells us
# nothing new, and re-running discovery for it is how a burst of
# membership notifications multiplied into M+1 discovery passes per
# connect. ``since`` (see ``_membership_filter``) is what stops that
# burst arriving at all; this is the second, independent stop.
await self._refresh_channel_discovery()
else:
logger.info("[buzz] removed from channel %s; unsubscribing", channel_id)
await self._close_chat_subscription(channel_id)
self._channel_meta.pop(channel_id, None)
async def _refresh_channel_discovery(self) -> None:
"""Re-issue the discovery REQ on its existing subscription id.
Replaces the subscription in place, so the relay re-sends every kind-39000
it holds for us: metadata for the new channel arrives, and each event runs
through ``_ensure_chat_subscription``, which is a no-op for channels already
subscribed. Cheaper to reason about than a one-off narrowed query, and it
reconciles anything else that changed while we were connected.
"""
transport = self._transport
if transport is None:
return
try:
await transport.send(buzz_nostr.req_frame(DISCOVERY_SUB_ID, self._discovery_filter()))
except Exception:
logger.warning("[buzz] failed to refresh channel discovery", exc_info=True)
def _is_dm(self, channel_id: str) -> bool:
"""True only when metadata was positively cached as ``type == "dm"``.
Fails closed: a channel we have no kind-39000 metadata for yet is never
treated as a DM, even though ``dict.get(..., {})`` would otherwise make an
absent entry look indistinguishable from an unset (non-DM) type.
"""
return self._channel_meta.get(channel_id, {}).get("type") == "dm"
def _thread_root(self, ev: dict) -> str | None:
e_tags = buzz_nostr.tag_values(ev, "e")
return e_tags[0] if e_tags else None
def _strip_own_mention(self, text: str) -> str:
"""Strip a single, unambiguous leading ``@mention`` token.
Nostr chat events carry no verified mapping from the free-text "@Name" a
client rendered into the message body to the pubkeys in the event's ``p``
tags — the caller only confirms *some* p-tagged mention exists (see
``mentioned`` in ``_handle_chat_event``), never that the specific leading
token names *us*. When a second ``@token`` immediately follows the first
(e.g. "@Alice, @DeerFlow help"), guessing that the first one is ours risks
silently discarding a different member's mention while leaving ours
untouched, so the conservative choice is to leave the text completely
alone rather than guess. The common single-mention case ("@DeerFlow
hello") remains unambiguous and is still stripped.
"""
stripped = text.lstrip()
if not stripped.startswith("@"):
return text.strip()
_, sep, rest = stripped.partition(" ")
if not sep:
return stripped # "@DeerFlow" alone: nothing to strip without losing the whole message
if rest.lstrip().startswith("@"):
return text.strip() # ambiguous multi-mention prefix: don't guess which one is ours
return rest.strip() or stripped
async def _bind_connection(self, code: str, author: str, channel_id: str) -> None:
"""Consume a ``/connect <code>`` bind code for *author* (a Nostr pubkey hex).
Always fully handles the request — valid code, invalid/expired/already-used
code, or a connection-repo error — so the caller (``_handle_chat_event``) can
unconditionally return right after awaiting this without ever falling through
to ``_make_inbound``/``_publish``. Mirrors discord.py's / slack.py's
``_bind_connection_from_connect_code``.
Bind success/failure is fully determined and logged by the try/except below
BEFORE ``_reply_to_connect`` is ever invoked, so a failure to *send* the
confirmation/error reply can never be attributed back to (or logged as) a
bind failure — see ``_reply_to_connect``. This ordering is deliberate: an
earlier review of this method flagged that sending from inside the same
try/except that decides bind success would let a relay-send hiccup on an
otherwise-successful bind get reported as "failed to bind".
"""
if self._connection_repo is None:
return # unreachable in practice: _pending_connect_code already requires this
try:
state = await self._connection_repo.consume_oauth_state(provider="buzz", state=code)
if state is None:
logger.info("[buzz] /connect code invalid, expired, or already used (pubkey=%s)", author)
outcome = "invalid"
else:
await self._connection_repo.upsert_connection(
owner_user_id=state["owner_user_id"],
provider="buzz",
external_account_id=author,
workspace_id=self._workspace_id,
metadata={"pubkey": author},
status="connected",
)
logger.info("[buzz] connected pubkey=%s to owner_user_id=%s", author, state["owner_user_id"])
outcome = "success"
except Exception:
# A repo/DB error binding the code must not propagate: handle_relay_frame's
# outer guard would also catch it, but catching here keeps the log specific
# to the bind failure instead of a generic "malformed relay event ignored".
logger.exception("[buzz] failed to bind /connect code for pubkey=%s", author)
outcome = "error"
# Bind success/failure is already fully decided and logged above; sending
# the reply is a separate, best-effort concern from here on.
await self._reply_to_connect(channel_id, author, _CONNECT_REPLY_TEXT[outcome])
async def _reply_to_connect(self, channel_id: str, author: str, text: str) -> None:
"""Best-effort confirmation/error reply for a ``/connect`` attempt.
The caller (``_bind_connection``) has already fully decided and logged the
bind outcome before this runs. A failure here is a relay-send problem, not
a bind problem: mirroring discord.py's ``_send_connection_reply`` / slack.py's
``_post_connection_reply``, it never raises and logs its own,
distinctly-worded warning, so it can never be mistaken for (or logged as) a
failed bind. Uses a single attempt (no retry/backoff): this is a courtesy
notification, not the delivery-critical agent-response path ``send()``
serves, so a transient relay hiccup here should not add retry latency to
the inbound relay read loop.
"""
assert self._keys is not None
try:
event = buzz_nostr.build_chat_event(self._keys, channel_id, text, created_at=int(time.time()), mentions=(author,))
await self._send_with_retry(lambda: self._post_event(event), max_retries=1, operation_name="connect-reply")
except Exception:
logger.warning("[buzz] failed to send /connect reply to pubkey=%s", author)
def _advance_watermark(self, channel_id: str, created_at: int) -> None:
"""Move THIS CHANNEL's resubscribe cursor to *created_at*, refusing future timestamps.
The cursor is per channel because subscriptions are: a single global
watermark is the newest event processed in ANY channel, so a busy channel
would keep dragging it forward and a quiet channel's next REQ would ask for
events newer than traffic that never belonged to it -- silently skipping
everything published in the quiet channel while we were disconnected. That
is the one direction this cursor must never fail in. Per channel costs one
integer per channel and cannot skip.
``created_at`` is chosen by the event's author, and ``_chat_filter`` replays
it as ``since`` on every reconnect. Accepting it unchecked was a remote
denial of service: one event stamped year-5138 pinned the cursor there, so
every subsequent REQ asked for events newer than that and the connector went
permanently deaf with no log and no recovery short of a process restart.
The cursor therefore never moves beyond ``now + MAX_FUTURE_SKEW_SECONDS``.
An out-of-range timestamp is IGNORED rather than clamped down to the
ceiling: clamping would still hand an attacker a blind window of exactly
the slack, while ignoring leaves the cursor where the last plausible event
put it. A legitimately fast-clocked member merely fails to advance the
cursor, which costs replay (drops we re-apply) and never a miss.
Channel ids arrive in remote ``h`` tags, so the map is remote-fed and capped
the same way ``_channel_meta`` is. Eviction is safe by construction: a
channel with no cursor simply re-subscribes without ``since`` and gets the
relay's default backlog, i.e. eviction can only ever cost replay.
"""
if created_at <= 0 or not channel_id:
return
ceiling = int(time.time()) + MAX_FUTURE_SKEW_SECONDS
if created_at > ceiling:
logger.debug("[buzz] ignoring future-dated created_at=%d for the resubscribe cursor (ceiling=%d)", created_at, ceiling)
return
self._seen_created_at[channel_id] = max(self._seen_created_at.get(channel_id, 0), created_at)
while len(self._seen_created_at) > MAX_CACHED_CHANNELS:
self._seen_created_at.pop(next(iter(self._seen_created_at)))
async def _attach_connection_identity(self, inbound: InboundMessage) -> InboundMessage:
"""Resolve a persisted ``/connect`` binding for this pubkey, exactly as every sibling does.
Without this the bind is write-only: ``connection_id`` / ``owner_user_id``
stay ``None``, so ``ChannelManager`` runs the turn under a synthetic
pubkey-derived user with its own memory and file buckets instead of the
bound DeerFlow account, and revoking the connection has no runtime effect.
``fallback_without_workspace`` stays off (unlike discord/dingtalk/wecom,
whose workspace is legitimately absent for DMs): ``_bind_connection``
always writes ``workspace_id=<relay host>``, which ``__init__`` guarantees
is non-empty, so every Buzz row is reachable by the primary lookup. Adding
the ``None`` candidate could only ever match a row this connector did not
write, and matching it would resolve a pubkey bound on some other relay —
the one thing the workspace scoping exists to prevent.
"""
return await attach_connection_identity(inbound, repo=self._connection_repo, provider="buzz", workspace_id=self._workspace_id)
async def _handle_chat_event(self, ev: dict) -> None:
assert self._keys is not None
author = str(ev.get("pubkey", ""))
channel_id_values = buzz_nostr.tag_values(ev, "h")
if not channel_id_values or author == self._keys.pubkey_hex:
return # no channel tag, or our own event (no self-reply loops)
channel_id = channel_id_values[0]
created_at = int(ev.get("created_at", 0))
text = str(ev.get("content", ""))
# /connect <code> must be consulted before the allowlist gate (framework
# ordering rule — see Channel._pending_connect_code) so a not-yet-bound
# user can bootstrap a binding even though they aren't allowlisted yet.
# Unlike every other gate below, a /connect message is never published:
# _bind_connection fully handles it (valid, invalid, or erroring code)
# and this always returns immediately after, matching every sibling
# adapter's _bind_connection_from_connect_code (discord.py, slack.py,
# wecom.py, dingtalk.py, wechat.py, feishu.py). Falling through to the
# mention/allowlist gates and publishing this as an ordinary chat message
# would let any pubkey trigger a real agent run just by prefixing a
# message with "/connect" — Buzz's run policy sets
# requires_bound_identity=False, so the manager has no independent
# bound-identity check to catch that.
code = self._pending_connect_code(text)
if code is not None:
await self._bind_connection(code, author, channel_id)
# A bind attempt (valid or not) is fully processed, so it may advance the
# cursor: leaving it behind would replay this /connect on every reconnect
# and answer each replay with a spurious "code invalid or expired" reply.
self._advance_watermark(channel_id, created_at)
return
if author not in self._allowed_users:
# Deny-by-default is intentional (see start()'s empty-allowlist warning),
# but a silent drop is indistinguishable from a broken relay when an
# operator forgets to allowlist someone. Debug level: an open relay can
# carry plenty of chatter from members we never intend to serve.
logger.debug("[buzz] dropped chat event from non-allowlisted pubkey=%s in channel %s (%s)", author, self._channel_meta.get(channel_id, {}).get("name") or "<unnamed>", channel_id)
return
# code is always None here: a non-None code was already fully handled
# and returned above, so this gate only ever sees ordinary chat text.
thread_root = self._thread_root(ev)
mentioned = self._keys.pubkey_hex in buzz_nostr.tag_values(ev, "p")
store = self.config.get("channel_store")
engaged_thread = bool(thread_root and store is not None and store.get_thread_id(self.name, channel_id, topic_id=thread_root))
allowed_without_mention = (not self._require_mention) or channel_id in self._mention_free or self._is_dm(channel_id) or engaged_thread
if not mentioned and not allowed_without_mention:
return
if mentioned:
text = self._strip_own_mention(text)
msg_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
inbound: InboundMessage = self._make_inbound(chat_id=channel_id, user_id=author, text=text, msg_type=msg_type, thread_ts=thread_root, metadata={"event_id": str(ev.get("id", ""))})
inbound.topic_id = thread_root
inbound.workspace_id = self._workspace_id
inbound = await self._attach_connection_identity(inbound)
self._last_requester[(channel_id, thread_root)] = author
await self._publish(inbound)
# Only a fully accepted-and-published event advances the cursor, and only
# after the publish actually succeeded. A dropped event must never move it
# (that was the DoS), and a failed publish must leave it replayable.
self._advance_watermark(channel_id, created_at)
# -- outbound --------------------------------------------------------------
async def _post_event(self, event: dict) -> None:
"""Sign-and-post is already done by the caller; this only delivers the frame.
Reads ``self._transport`` at call time (never cached): Task 6's relay loop
sets it to a live WebSocket for the duration of a connection and back to
``None`` on disconnect, so a stale reference here would keep "succeeding"
against a socket that is no longer attached to anything.
"""
if self._transport is None:
raise RuntimeError("[buzz] relay connection not established")
await self._transport.send(buzz_nostr.event_frame(event))
async def _edit_or_repost(self, msg: OutboundMessage, target: str, content: str, now: int, *, label: str) -> str:
"""Edit *target* in place; if that fails after retries, post a fresh message instead.
Returns the event id later updates for this slot must target -- ``target``
itself on success, the replacement's id after a degrade. Degrading rather
than raising is the "never lose content" rule the placeholder path has
always had; no mention rides the replacement because the requester was
already notified by the original post and re-mentioning on every degraded
edit would spam notifications.
"""
assert self._keys is not None
edit = buzz_nostr.build_edit_event(self._keys, msg.chat_id, target, content, created_at=now)
try:
await self._send_with_retry(lambda: self._post_event(edit), max_retries=3, operation_name=f"edit{label}")
return target
except Exception:
fresh = buzz_nostr.build_chat_event(self._keys, msg.chat_id, content, created_at=now, reply_to=msg.thread_ts)
await self._send_with_retry(lambda: self._post_event(fresh), max_retries=3, operation_name=f"post-degraded{label}")
return fresh["id"]
async def send(self, msg: OutboundMessage) -> None:
"""Post one placeholder chat message, then stream via in-place edits.
The first message for a given ``(chat_id, thread_ts)`` is a kind-9 chat
event (the only place a mention can ride, since kind-40003 edits carry
only ``h``/``e`` tags — see ``buzz_nostr.build_edit_event``). Every
subsequent update for the same key edits that placeholder in place via a
kind-40003 event targeting its id. ``is_final`` clears the tracked target
so the next run in this conversation starts a fresh placeholder instead of
editing a stale, already-answered message.
Oversize text is split by ``_chunk_text`` into <= ``EDIT_MAX_BYTES``-byte
chunks: the first chunk rides the placeholder/edit, any remaining chunks
ride follow-up kind-9 messages threaded to the same thread root, tracked
per conversation in ``_stream_targets`` / ``_stream_tails`` respectively.
Every chunk index is tracked, not just chunk 0, because the manager
publishes CUMULATIVE text on each streaming update: an oversize reply
therefore re-splits into >= 2 chunks on EVERY update, and posting
``chunks[1:]`` fresh each time flooded the channel with a near-duplicate
tail per update (a ~100KB answer at ~1 update/sec produced dozens). A
follow-up message is now posted only when the chunk count actually GROWS;
an index we have already posted is edited in place, exactly like chunk 0.
Raises whatever the underlying send raised (including ``_post_event``'s
``RuntimeError`` when ``_transport`` is ``None``) after retries are
exhausted, so the framework's outer retry/error-logging path in
``Channel._on_outbound`` observes the failure instead of a reply being
silently dropped. The ``is_final`` bookkeeping below runs in a
``finally`` block precisely so that a raised exception still propagates
to the caller *and* still clears the stale target — see the ``finally``
comment for why both matter.
DEFENSE IN DEPTH: text carrying one of DeerFlow's hidden model-context
wrappers is refused outright. The real fix is one layer up
(``manager._accumulate_stream_text`` now allowlists assistant message
types instead of denylisting tool ones), but this connector is the one
where a leak cannot be taken back: every streaming update is an immutable
public Nostr event, so a corrective edit only changes what clients render
while the original leaked event stays on the relay. Posting nothing is
therefore the right failure direction here, and it is deliberately not
replicated into the sibling connectors, whose channels can be edited or
deleted. See ``_HIDDEN_CONTEXT_MARKERS`` for why this cannot fire on an
ordinary reply that merely talks about memory.
"""
assert self._keys is not None
key = (msg.chat_id, msg.thread_ts)
now = int(time.time())
if marker := _hidden_context_marker(msg.text):
logger.error("[buzz] REFUSED to publish a reply carrying hidden model context (%s) to channel %s; this is a leak upstream of the connector, not a relay problem", marker, msg.chat_id)
if msg.is_final:
self._stream_targets.pop(key, None)
self._stream_tails.pop(key, None)
return
chunks = _chunk_text(msg.text)
target = self._stream_targets.get(key)
try:
if target is None:
requester = self._last_requester.get(key)
mentions = (requester,) if requester else ()
first = buzz_nostr.build_chat_event(self._keys, msg.chat_id, chunks[0], created_at=now, reply_to=msg.thread_ts, mentions=mentions)
await self._send_with_retry(lambda: self._post_event(first), max_retries=3, operation_name="post")
self._stream_targets[key] = first["id"]
else:
self._stream_targets[key] = await self._edit_or_repost(msg, target, chunks[0], now, label="")
tails = self._stream_tails.get(key)
for index, extra in enumerate(chunks[1:]):
if tails is not None and index < len(tails):
tails[index] = await self._edit_or_repost(msg, tails[index], extra, now, label="-overflow")
continue
follow = buzz_nostr.build_chat_event(self._keys, msg.chat_id, extra, created_at=now, reply_to=msg.thread_ts)
await self._send_with_retry(lambda ev=follow: self._post_event(ev), max_retries=3, operation_name="post-overflow")
if tails is None:
tails = self._stream_tails.setdefault(key, [])
tails.append(follow["id"])
finally:
# Reviewer finding: the degrade-to-fresh-post branch above and the
# overflow-chunk loop can both raise after their own retries are
# exhausted, which used to propagate out of send() *before* reaching
# an unconditional pop at the end of the function -- leaking a stale
# (or half-updated) _stream_targets[key] entry whenever the failing
# call had is_final=True. Once the relay recovered, the next send()
# for this conversation would then EDIT that abandoned placeholder
# instead of starting a fresh message, contradicting this method's
# own contract. A `finally` clears the bookkeeping on every path --
# success or failure -- while still letting the exception propagate
# (a `finally` block never suppresses an in-flight exception unless
# it itself returns/raises), so `Channel._on_outbound` still logs
# the failure exactly as before. The overflow-tail map added for the
# message-flood fix is cleared on exactly the same terms, so a final
# send can never leave the next run editing this run's tail messages.
if msg.is_final:
self._stream_targets.pop(key, None)
self._stream_tails.pop(key, None)