FanouZeng-TT 064bb50602
fix(buzz): page a saturated history instead of skipping the oldest (#6792)
* fix(buzz): page a saturated history instead of skipping the oldest

buzz-relay serves at most RELAY_HISTORY_LIMIT (2000) stored events per
subscription, NEWEST-FIRST, even when the filter carries a `since`. A channel
that banked more than the cap across one disconnect therefore had its OLDEST
unread messages neither sent nor marked: the live page delivered the newest
2000, the watermark advanced past them, and the rest were lost -- the one
remaining path in this connector that could skip a message outright.

A chat/backlog page's EOSE now drives a descending `until` walk. When a page
comes back AT the cap it may have older matches held back, so the connector
re-asks with `until` set to that page's oldest `created_at` -- on a SEPARATE
`buzz-backlog-<uuid>` subscription id, never the live one -- and repeats until
a page comes back under the cap or reaches the `since` the live REQ was sent
with, bounded by MAX_BACKLOG_PAGES (50, warned when hit). Delivery is charged
to the subscription that carried it before any gate, so a page's saturation is
measured against what the relay sent, not what the allowlist/mention gates kept.

The walk is strictly additive: the live subscription, its filter, and the
watermark are never touched; a CLOSED for a backlog page only ends that walk;
re-opening the live subscription closes any superseded page rather than
orphaning it; and every failure simply leaves the pre-existing behaviour for the
next reconnect to retry.

* fix(buzz): skip the backlog walk when the channel has no read floor

Reviewer willem-bd noted the descending-until walk has no semantic stop when
`floor is None`: that is exactly a channel's first connect (no stored
watermark) and again after a MAX_CACHED_CHANNELS eviction of its cursor. On
those connects the walk would fetch up to MAX_BACKLOG_PAGES * RELAY_HISTORY_LIMIT
events per channel instead of the newest 2000 the connector used to take.

Require a floor before opening any backlog page. The backlog-loss case this walk
exists for always has a watermark (the cursor survives reconnects and restarts),
so it always has a floor; first connect and an evicted cursor keep the previous
behaviour. Covers the skip with a new test, retargets the page-bound test to a
real walk above its floor, and documents the floor requirement.

* fix(buzz): stop orphaned and stalled backlog walks

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-14 15:59:46 +08:00
..

Documentation

This directory contains detailed documentation for the DeerFlow backend.

Document Description
ARCHITECTURE.md System architecture overview
API.md Complete API reference
AUTH_DESIGN.md User authentication, CSRF, platform-trust (IM / Internal Auth), and per-user isolation
SSO.md OIDC / SSO single sign-on
IM_CHANNEL_CONNECTIONS.md IM channel user binding (channel_connections)
CONFIGURATION.md Configuration options
SETUP.md Quick setup guide

Feature Documentation

Document Description
STREAMING.md Token-level streaming design: Gateway vs DeerFlowClient paths, stream_mode semantics, per-id dedup
RUN_EVENT_STREAM.md Persisted run event stream contract: envelope, producers, consumers, and known gaps
FILE_UPLOAD.md File upload functionality
PATH_EXAMPLES.md Path types and usage examples
SANDBOX_MEMORY_PROFILING.md Sandbox memory baseline and runtime comparison guide
summarization.md Context summarization feature
plan_mode_usage.md Plan mode with TodoList
AUTO_TITLE_GENERATION.md Automatic title generation

Development

Document Description
TODO.md Planned features and known issues

Getting Started

  1. New to DeerFlow? Start with SETUP.md for quick installation
  2. Configuring the system? See CONFIGURATION.md
  3. Understanding the architecture? Read ARCHITECTURE.md
  4. Building integrations? Check API.md for API reference

Document Organization

docs/
├── README.md                  # This file
├── ARCHITECTURE.md            # System architecture
├── API.md                     # API reference
├── AUTH_DESIGN.md             # User authentication and isolation design
├── CONFIGURATION.md           # Configuration guide
├── SETUP.md                   # Setup instructions
├── FILE_UPLOAD.md             # File upload feature
├── PATH_EXAMPLES.md           # Path usage examples
├── summarization.md           # Summarization feature
├── plan_mode_usage.md         # Plan mode feature
├── STREAMING.md               # Token-level streaming design
├── RUN_EVENT_STREAM.md        # Persisted run event stream contract
├── AUTO_TITLE_GENERATION.md   # Title generation
├── TITLE_GENERATION_IMPLEMENTATION.md  # Title implementation details
└── TODO.md                    # Roadmap and issues