mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
c943d953cd
* docs(checkpoints): add a bilingual checkpoint storage manual Add the missing docs-site coverage for dual-mode checkpoint storage (`database.checkpoint_channel_mode: full | delta`) as a ten-chapter manual under `harness/checkpoints/` in both languages: concepts, quick start, channel modes, snapshot cadence, history cache, resume and rollback, operating checkpoints, observability, troubleshooting by symptom, and a reference of every configuration key, error message, HTTP status mapping, public symbol, load-bearing test, and pinned upstream defect. The index page is `asIndexPage`, so `/docs/harness/checkpoints` is the section itself and its `_meta.ts` lists only the nine child chapters. Discovery: `harness/_meta.ts` registers the section, the harness configuration module table gains a `database` row, and the thread persistence section of `agents-and-threads` links to the manual. Content is grounded in the code. Only the three measured values recorded in `backend/docs/runtime-guidance-details.md` are quoted; the upstream-derived 41x storage figure in the untracked delta-channel RFC is deliberately not cited. Pinned upstream defects are documented with their issue and PR numbers and the versions they are pinned at (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`). * docs(checkpoints): correct the history-cache restart requirement The history cache is built with the checkpointer at Gateway startup (`langgraph_runtime()` -> `make_checkpointer()` -> `make_checkpoint_cache()`), so editing `database.checkpoint_cache.*` does not change a running process. The manual previously marked the whole section "not restart-required", which conflated "not frozen for correctness" with "hot-reloadable" and could mislead an operator changing capacity, credentials, or retention. Distinguish the two properties everywhere the section is described: history-cache.mdx marks every field as needing a restart and adds a callout explaining the difference, reference.mdx redefines the Restart column and fixes the five rows and the effect list (adding the `asyncprov` evidence alias), troubleshooting.mdx replaces the "two keys need no restart" claim and gains a dedicated symptom entry, and observability.mdx notes that changed counters need a restart. The Chinese text is corrected in parallel. Verified: the docs link test and the docs unit suite pass, frontmatter parses for all 20 pages, and every in-page anchor resolves on the rendered English and Chinese history-cache, troubleshooting, reference, and observability pages. * docs(checkpoints): account for periodic snapshots in the storage-growth claim Every periodic snapshot stores the entire message list as of that step (`_DeltaSnapshot(ch.get())`, persisted as a checkpoint blob), so with a fixed snapshot frequency K the retained total after N turns is O(N²/K + N), not O(N). The manual claimed unconditional linear growth, which understated capacity planning — the more so because nothing in DeerFlow prunes those snapshots yet. Distinguish per-step delta writes from total retained storage in index.mdx (prose and table), modes.mdx (step-cost row), and operations.mdx; note in cadence.mdx that K is also the capacity knob that scales the snapshot bytes. The English and Chinese chapters are corrected in parallel. The reference chapter keeps the released #4292 wording as a verbatim quotation and now names the retained-storage bound beside it. Verified: `_DeltaSnapshot(ch.get())` and its blob serialization in the pinned langgraph (channels/delta.py:196-198, checkpoint/serde/types.py:19-25) confirm the premise; the docs link test and docs unit suite pass; all ten changed pages render with resolving anchors and no stale linear-growth wording. * docs(checkpoints): correct the run-error surfaces and repair two evidence pointers Review follow-ups on the checkpoint manual, both languages. Error-surface corrections: - Run creation is not always asynchronous: when a thread's run-event feed is empty but a checkpoint head exists, ensure_checkpoint_history_seeded reads that head inside start_run, where only ConflictError (409) and UnsupportedStrategyError (501) are mapped, so a mode mismatch returns a plain 500 before admission. Troubleshooting, observability and the reference route tables now scope the "does not pre-gate" claim. - The 503 symptom is retargeted. Only Gateway startup, make_lead_agent and DeerFlowClient.__init__ call the freeze helpers, so a config.yaml edit cannot make a state/history/thread route answer 503. A mode edit is injected over by the run worker and ignored until a restart; a cadence edit fails the next run with the restart-required message. The router's 503 mapping is marked defensive in modes and the reference. - The regenerate section title no longer groups the state endpoints, which map a mismatch to 409. - operations: the Textual TUI catches the mismatch through stream_actions(); only the headless --print/--json runners print a traceback. Evidence-pointer repairs: - reference: this branch's CHANGELOG insertion shifted the #5734 entry from 3032-3039 to 3040-3048; the citation is re-pointed. - reference: the #8548 row cited an untracked worktree plan file; replaced by the in-repo test anchor, and "mirror issue #8551" reworded to "related". - reference: new `agent` alias for the agent-factory freeze sites. - history-cache: the bare `deps:`/`asyncprov:` aliases are expanded to full paths; the alias legend lives only on the reference page. * docs(checkpoints): say a delta fork is accepted, not refused "Delta mode cannot fork" overstated the constraint. Nothing in the stack rejects the fork: LangGraph stores the second child, no saver checks for an existing sibling, and ensure_checkpoint_mode_compatible returns immediately when the process mode is delta. What breaks is the read — the walk replays the abandoned sibling's writes stored under the shared parent. Reproduced on the pinned stack (langgraph 1.2.9 / langgraph-checkpoint 4.2.0, AsyncSqliteSaver) by running this repo's own #4458 fixture without the linearization call: the forked resume returns ["h1", "a1", "h2", "a2", "a2-new"] instead of ["h1", "a1", "h2", "a2-new"], i.e. the superseded answer comes back. - resume-and-rollback: title/description/heading now say "cannot fork correctly"; new "Forks are accepted, not refused" subsection covering the unguarded write side. - troubleshooting: the "old answer reappears" cause states the same split. - reference: the POST /threads/{id}/state row notes that a client-supplied checkpoint_id older than the head is applied as a delta fork with no linearization — the write shape upstream tracks as #8551 (open), which the run-path linearization does not cover. * docs(checkpoints): scope the unguarded state-update fork to what the probe showed The previous commit said a mutation against an older checkpoint stores its writes under the shared parent and implied the sibling branch suffers for it. Probing the pinned stack (AsyncSqliteSaver, delta messages channel) shows: - the write does land under the shared parent and joins the write set the parent's other children walk (raw walk for a sibling includes the abandoned head's own write); - but reducer channels are written with Overwrite, which replaces the accumulated value, so the forked branch reads clean and the abandoned branch is unchanged in that shape — #8551 did not reproduce; state the masking mechanism and the upstream tracking issue, not a corruption claim. * docs(checkpoints): fix a frontmatter description that broke the docs build `resume-and-rollback.mdx` shipped `description: Delta mode cannot fork a thread correctly: a forked delta read ...` in the previous commit. The unquoted ": " turns the value into a nested mapping, so Nextra's YAML frontmatter parse failed and the e2e lane could not start its web server (YAMLParseError at line 2, column 14) — the exact failure class this manual warns docs authors about, introduced by me two commits ago. - Replace the colon with an em dash in the English description (the Chinese one uses a full-width colon and was unaffected). - Add tests/unit/content/docs-frontmatter.test.ts: every `src/content/**/*.mdx` frontmatter block must be YAML-parseable, implemented as the one rule that actually breaks it (an unquoted scalar containing ": "), because the unit suite never compiles MDX and only a build or a dev render catches it. Verified the guard fails on the broken line and passes on the fix. Verified with a clean `BETTER_AUTH_SECRET=local-dev-secret pnpm build` on this tree: all pages prerender, resume-and-rollback EN/ZH included.
DeerFlow Frontend
Like the original DeerFlow 1.0, we would love to give the community a minimalistic and easy-to-use web interface with a more modern and flexible architecture.
Tech Stack
- Framework: Next.js 16 with App Router
- UI: React 19, Tailwind CSS 4, Shadcn UI, MagicUI and React Bits
- AI Integration: LangGraph SDK and Vercel AI Elements
Quick Start
Prerequisites
- Node.js 22+
- pnpm 10.26.2+
Installation
# Install dependencies
pnpm install
# Copy environment variables
cp .env.example .env
# Edit .env with your configuration
Development
# Start development server
pnpm dev
# The app will be available at http://localhost:3000
Build & Test
# Type check
pnpm typecheck
# Check formatting
pnpm format
# Apply formatting
pnpm format:write
# Lint
pnpm lint
# Run unit tests
pnpm test
# One-time setup: install Playwright Chromium browser
pnpm exec playwright install chromium
# Run E2E tests (builds and starts production server automatically)
pnpm test:e2e
# Build for production
pnpm build
# Start production server
pnpm start
Site Map
├── / # Landing page
├── /chats # Chat list
├── /chats/new # New chat page
└── /chats/[thread_id] # A specific chat page
Configuration
Environment Variables
Key environment variables (see .env.example for full list):
# Backend API URL (optional, uses local Next.js/nginx proxy by default)
NEXT_PUBLIC_BACKEND_BASE_URL="http://localhost:8001"
# LangGraph-compatible API URL (optional, uses local Next.js/nginx proxy by default)
NEXT_PUBLIC_LANGGRAPH_BASE_URL="http://localhost:8001/api"
Project Structure
tests/
├── e2e/ # E2E tests (Playwright, Chromium, mocked backend)
└── unit/ # Unit tests (mirrors src/ layout)
src/
├── app/ # Next.js App Router pages
│ ├── api/ # API routes
│ ├── showcase/ # Allowlisted public read-only demos
│ ├── workspace/ # Main workspace pages
│ └── mock/ # Mock/demo pages
├── components/ # React components
│ ├── ui/ # Reusable UI components
│ ├── workspace/ # Workspace-specific components
│ ├── landing/ # Landing page components
│ └── ai-elements/ # AI-related UI elements
├── core/ # Core business logic
│ ├── api/ # API client & data fetching
│ ├── artifacts/ # Artifact management
│ ├── config/ # App configuration
│ ├── i18n/ # Internationalization
│ ├── mcp/ # MCP integration
│ ├── messages/ # Message handling
│ ├── models/ # Data models & types
│ ├── settings/ # User settings
│ ├── skills/ # Skills system
│ ├── threads/ # Thread management
│ ├── todos/ # Todo system
│ └── utils/ # Utility functions
├── hooks/ # Custom React hooks
├── lib/ # Shared libraries & utilities
├── server/ # Server-side code
│ └── better-auth/ # Authentication setup and session helpers
└── styles/ # Global styles
Scripts
| Command | Description |
|---|---|
pnpm dev |
Start development server with Webpack |
pnpm build |
Build for production |
pnpm start |
Start production server |
pnpm test |
Run unit tests with Rstest |
pnpm test:e2e |
Run E2E tests with Playwright |
pnpm format |
Check formatting with Prettier |
pnpm format:write |
Apply formatting with Prettier |
pnpm lint |
Run ESLint |
pnpm lint:fix |
Fix ESLint issues |
pnpm typecheck |
Run TypeScript type checking |
pnpm check |
Run both lint and typecheck |
Development Notes
- Uses pnpm workspaces (see
packageManagerin package.json) - Webpack is the default development bundler until the upstream Turbopack PostCSS worker leak is fixed in a stable Next.js release (#5132). Set
DEER_FLOW_DEV_BUNDLER=turboto opt in to Turbopack for local diagnosis, orDEER_FLOW_DEV_BUNDLER=webpackto select Webpack explicitly. Reconsider the default after the stable fix is verified on macOS arm64 and Linux. - Environment validation can be skipped with
SKIP_ENV_VALIDATION=1(useful for Docker) - Backend API URLs are optional; nginx proxy is used by default in development
License
MIT License. See LICENSE for details.