Zeren Wang 553f85e788
feat(projects): best-effort LLM summaries for shelf documents (#6581)
Opt-in per-document summary column rendered into the injected
<documents> index, list_project_documents JSON, and the documents API.

Pipeline (projects/summaries.py, behind the narrow enqueue_summary
interface):
- The two shelf write routes enqueue a job on created=True; dedup hits
  never regenerate. Off by default (projects.summaries_enabled).
- Bounded worker pool (summary_concurrency + bounded summary_queue_size;
  a full queue drops the job) fed by FIFO; shutdown drains queued jobs,
  delivers one sentinel per worker via blocking put (queue smaller than
  the worker pool cannot strand idle workers), and bounds only
  genuinely in-flight jobs with the grace period.
- Jobs carry (document_id, project_id, user_id) explicitly; the job's
  owner is bound into the user ContextVar around the model call so
  tracing/usage attribution never falls back to 'default'.
- Text source reuses read_text_serving_path (convertible-first rule,
  content validation); conversion has no deadline (await_drained), the
  model call has a caller-side wait limit — the worker slot is retained
  until the provider call actually finishes and late results are
  discarded.
- Output contract: strip leading <think> reasoning -> unwrap fence/
  quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on
  a character boundary; length-capped responses (provider-normalized
  detectors) are failures.
- set_summary pins updated_at (literal self-assignment) so a landing
  summary never reorders the shelf; owner-predicated UPDATE is a no-op
  on rows purged mid-generation.
- Metrics: eligible/success/failed/skipped/queue_dropped/
  uncapped_accepted.

Schema/config:
- project_documents.summary (migration 0036, nullable, no backfill;
  forward-compat floor unchanged).
- Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS
  (captured once at lifespan startup); config_version 57 -> 58.
- run_oneshot_llm gains an optional model_overrides passthrough (passed
  only when set, existing callers unchanged) and a detailed variant
  exposing the raw response message for termination-signal detection.

Docs: API.md response field, ARCHITECTURE.md pipeline paragraph,
migrations AGENTS.md chain head, root AGENTS.md records the scoped
all-core exception with the lift-out path.
2026-10-10 07:11:02 +08:00
..

Documentation

This directory contains detailed documentation for the DeerFlow backend.

Document Description
ARCHITECTURE.md System architecture overview
API.md Complete API reference
AUTH_DESIGN.md User authentication, CSRF, platform-trust (IM / Internal Auth), and per-user isolation
SSO.md OIDC / SSO single sign-on
IM_CHANNEL_CONNECTIONS.md IM channel user binding (channel_connections)
CONFIGURATION.md Configuration options
SETUP.md Quick setup guide

Feature Documentation

Document Description
STREAMING.md Token-level streaming design: Gateway vs DeerFlowClient paths, stream_mode semantics, per-id dedup
RUN_EVENT_STREAM.md Persisted run event stream contract: envelope, producers, consumers, and known gaps
FILE_UPLOAD.md File upload functionality
PATH_EXAMPLES.md Path types and usage examples
SANDBOX_MEMORY_PROFILING.md Sandbox memory baseline and runtime comparison guide
summarization.md Context summarization feature
plan_mode_usage.md Plan mode with TodoList
AUTO_TITLE_GENERATION.md Automatic title generation

Development

Document Description
TODO.md Planned features and known issues

Getting Started

  1. New to DeerFlow? Start with SETUP.md for quick installation
  2. Configuring the system? See CONFIGURATION.md
  3. Understanding the architecture? Read ARCHITECTURE.md
  4. Building integrations? Check API.md for API reference

Document Organization

docs/
├── README.md                  # This file
├── ARCHITECTURE.md            # System architecture
├── API.md                     # API reference
├── AUTH_DESIGN.md             # User authentication and isolation design
├── CONFIGURATION.md           # Configuration guide
├── SETUP.md                   # Setup instructions
├── FILE_UPLOAD.md             # File upload feature
├── PATH_EXAMPLES.md           # Path usage examples
├── summarization.md           # Summarization feature
├── plan_mode_usage.md         # Plan mode feature
├── STREAMING.md               # Token-level streaming design
├── RUN_EVENT_STREAM.md        # Persisted run event stream contract
├── AUTO_TITLE_GENERATION.md   # Title generation
├── TITLE_GENERATION_IMPLEMENTATION.md  # Title implementation details
└── TODO.md                    # Roadmap and issues