mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
4bab280250
* fix(skills): share skill cache invalidation across Gateway replicas SkillStorage.load_skills() rescans disk and re-reads extensions_config.json on every call, but the prompt layer in deerflow/agents/lead_agent/prompt.py caches the enabled-skills list, the per-(app_config, user_id) entries and the rendered <skill_system> section per process, and every skill mutation in routers/skills.py only refreshed the process that handled it. With several uvicorn workers or several Pods sharing one home volume, a skill installed, edited, deleted, rolled back or toggled through replica A kept being offered as the old skill set by replica B until B restarted, and POST /api/skills/reload had to be sent to every Pod separately. Reuse the MCP cache-reset mechanism. A new generic helper, deerflow.config.shared_reset_marker, owns a hidden JSON marker beside the resolved extensions config (.<config name>.<suffix>.json): publish() replaces it atomically with a random generation under extensions_config_write_lock + extensions_config_file_lock, chaining previous_generation and an optional user_id; SharedResetMarkerTracker polls it at most once per second (monotonic clock, never blocking behind a concurrent poller), adopts the current state silently on its first poll, and classifies a change as user-scoped only when exactly one publication carrying that user happened since the last poll. deerflow/mcp/cache.py now builds its .<config>.mcp-cache-reset.json marker through the same helper with no behavior change. prompt.py publishes .<config>.skills-cache-reset.json via publish_skills_cache_reset(user_id=...) and checks the marker on every cache lookup (global cache, per-config cache, prompt-section LRU, user scoped entries), calling _invalidate_enabled_skills_cache() or invalidate_user_skill_cache(). The publishing process remembers its own generation so it does not retire the caches it just rebuilt. Every mutation tail in routers/skills.py (install/upload, edit, delete, rollback, enable/disable, reload) refreshes locally and then publishes the marker off the event loop; /skills/reload answers scope=shared_config, or scope=process when no extensions config path resolves, and a marker publication failure is a 500 so a 200 never means only one process was refreshed. Docs: README, backend/docs/API.md, skills/config/mcp/gateway AGENTS.md, and frontend skills.mdx (en/zh; also fixes the reference to a non-existent skills/loader.py) describe the shared invalidation and drop the per-Pod guidance. CHANGELOG entry under Unreleased / Fixed. * docs(changelog): link the skills cache reset entry to #6495 * fix(skills): drain the reload tail and keep marker failures out of 404 Two review follow-ups on the shared skills cache reset (#6495). POST /skills/reload awaited the local refresh and the marker publication unshielded. CancelledError is a BaseException, so a caller that disconnected between the two slipped past the `except Exception -> 500` mapping, surfaced no error, and left every peer replica serving the old skill set -- the one outcome the operator hook for external mount writes exists to prevent. The refresh and the publication now run as one `await_drained(_drain_skill_mutation("reload", ...))` tail, the same discipline the install/edit/delete/rollback/toggle tails already use; the exception -> 500 mapping and the scope response are unchanged. Publication errors could surface as 404. resolve_shared_config_path() only swallows the resolution-time FileNotFoundError; a config directory that vanishes between resolution and the lock file / atomic replace still raised FileNotFoundError from the marker write, and the install, edit, delete and rollback handlers mapped that to HTTP 404 with the raw OS error text for a skill that had in fact been applied. publish_skills_cache_reset() now re-raises OSError (and a misconfigured project root's ValueError) as SkillCacheResetPublishError, a RuntimeError that no handler maps to 404/400, so every handler reports it through its generic 500 branch; the message names only the exception type, never a server path. The install handler gains an explicit clause that pins that contract beside its 404 mapping. Tests: cancelling reload after the refresh still publishes the marker; a FileNotFoundError from the marker write yields 500 (not 404) for install, edit, delete and rollback, 500 for the public toggle and the generic reload error; the wrapper type is not an OSError.
DeerFlow Frontend
Like the original DeerFlow 1.0, we would love to give the community a minimalistic and easy-to-use web interface with a more modern and flexible architecture.
Tech Stack
- Framework: Next.js 16 with App Router
- UI: React 19, Tailwind CSS 4, Shadcn UI, MagicUI and React Bits
- AI Integration: LangGraph SDK and Vercel AI Elements
Quick Start
Prerequisites
- Node.js 22+
- pnpm 10.26.2+
Installation
# Install dependencies
pnpm install
# Copy environment variables
cp .env.example .env
# Edit .env with your configuration
Development
# Start development server
pnpm dev
# The app will be available at http://localhost:3000
Build & Test
# Type check
pnpm typecheck
# Check formatting
pnpm format
# Apply formatting
pnpm format:write
# Lint
pnpm lint
# Run unit tests
pnpm test
# One-time setup: install Playwright Chromium browser
pnpm exec playwright install chromium
# Run E2E tests (builds and starts production server automatically)
pnpm test:e2e
# Build for production
pnpm build
# Start production server
pnpm start
Site Map
├── / # Landing page
├── /chats # Chat list
├── /chats/new # New chat page
└── /chats/[thread_id] # A specific chat page
Configuration
Environment Variables
Key environment variables (see .env.example for full list):
# Backend API URL (optional, uses local Next.js/nginx proxy by default)
NEXT_PUBLIC_BACKEND_BASE_URL="http://localhost:8001"
# LangGraph-compatible API URL (optional, uses local Next.js/nginx proxy by default)
NEXT_PUBLIC_LANGGRAPH_BASE_URL="http://localhost:8001/api"
Project Structure
tests/
├── e2e/ # E2E tests (Playwright, Chromium, mocked backend)
└── unit/ # Unit tests (mirrors src/ layout)
src/
├── app/ # Next.js App Router pages
│ ├── api/ # API routes
│ ├── showcase/ # Allowlisted public read-only demos
│ ├── workspace/ # Main workspace pages
│ └── mock/ # Mock/demo pages
├── components/ # React components
│ ├── ui/ # Reusable UI components
│ ├── workspace/ # Workspace-specific components
│ ├── landing/ # Landing page components
│ └── ai-elements/ # AI-related UI elements
├── core/ # Core business logic
│ ├── api/ # API client & data fetching
│ ├── artifacts/ # Artifact management
│ ├── config/ # App configuration
│ ├── i18n/ # Internationalization
│ ├── mcp/ # MCP integration
│ ├── messages/ # Message handling
│ ├── models/ # Data models & types
│ ├── settings/ # User settings
│ ├── skills/ # Skills system
│ ├── threads/ # Thread management
│ ├── todos/ # Todo system
│ └── utils/ # Utility functions
├── hooks/ # Custom React hooks
├── lib/ # Shared libraries & utilities
├── server/ # Server-side code
│ └── better-auth/ # Authentication setup and session helpers
└── styles/ # Global styles
Scripts
| Command | Description |
|---|---|
pnpm dev |
Start development server with Webpack |
pnpm build |
Build for production |
pnpm start |
Start production server |
pnpm test |
Run unit tests with Rstest |
pnpm test:e2e |
Run E2E tests with Playwright |
pnpm format |
Check formatting with Prettier |
pnpm format:write |
Apply formatting with Prettier |
pnpm lint |
Run ESLint |
pnpm lint:fix |
Fix ESLint issues |
pnpm typecheck |
Run TypeScript type checking |
pnpm check |
Run both lint and typecheck |
Development Notes
- Uses pnpm workspaces (see
packageManagerin package.json) - Webpack is the default development bundler until the upstream Turbopack PostCSS worker leak is fixed in a stable Next.js release (#5132). Set
DEER_FLOW_DEV_BUNDLER=turboto opt in to Turbopack for local diagnosis, orDEER_FLOW_DEV_BUNDLER=webpackto select Webpack explicitly. Reconsider the default after the stable fix is verified on macOS arm64 and Linux. - Environment validation can be skipped with
SKIP_ENV_VALIDATION=1(useful for Docker) - Backend API URLs are optional; nginx proxy is used by default in development
License
MIT License. See LICENSE for details.