mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-20 03:26:18 +00:00
5eb59cb130
* fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes Docker sandboxes are shared across gateway workers, but each worker kept its own in-memory warm pool. Startup reconciliation adopted every running container, so a peer idle reaper could destroy sandboxes another worker still owned and tool calls hit 502 / Connection refused. Add file-based ownership leases under sandbox-leases/, only adopt true orphans, refuse idle/replica/shutdown destroy while a foreign lease is live, and renew the lease on create/get/release/reclaim. Fixes #4206 * fix(sandbox): close lease fail-open, hot-path IO, and check→destroy race Address review of the multi-worker orphan lease (#4206): - read_lease returns None only for a genuinely-absent lease and raises (CorruptLeaseError/OSError) when a lease is unreadable or corrupt, so the ownership check fails closed instead of mistaking an unprovable peer lease for a free container. clear_lease still removes a stuck/corrupt file. - get() no longer renews the lease (blocking mkdir/fsync/os.replace on the event loop path used by ensure_sandbox_initialized_async); active leases are renewed off the event loop from the idle checker (_renew_active_leases). - The ownership check and container stop run under a per-sandbox flock guard (lease_ownership_guard); every lease write takes the same guard so a peer's touch cannot interleave with a destroy. Same-host multi-worker scope, not a multi-pod distributed lock. Also fixes the ruff format lint on the branch. Adds regression tests: corrupt and unreadable lease fail closed, a tests/blocking_io anchor keeping get() non-blocking on the event loop, and a peer-touch/destroy interleave test. * fix(sandbox): share container ownership across gateway instances Rework of the #4206 fix per review: ownership state is shared through a third-party service instead of being maintained per gateway instance, following the stream_bridge precedent (sandbox.ownership.type: memory | redis). The file lease and its same-host flock guard are deleted, not ported — they only covered workers on one host, while the deployment that hits #4206 is a load-balanced multi-instance gateway. A lease answers "who reaps this container", not "who may use it". Containers are deterministic per (user, thread), so consecutive turns legitimately land on different instances: take() transfers ownership on acquire, while claim() gates every adopt/reap path. Leases carry a state — own: or del: — so a takeover is refused against a teardown in progress. Without it an unconditional take() would overwrite a destroyer's claim and the peer's container stop would land on a sandbox the new owner had already handed to an agent. renew() distinguishes a lapsed lease from one a peer took; only the latter drops the sandbox. Collapsing them meant a Redis restart evicted every in-flight sandbox on every instance at once. Renewal runs on its own thread with a TTL derived from its interval, never from idle_timeout: renewal used to ride the idle checker, which does not start at idle_timeout: 0, so leases silently lapsed on a supported config. Ownership establishment is fail-closed: a sandbox whose ownership cannot be published is never handed out, and a just-created container is destroyed rather than leaked as an adoptable orphan. Every destroy path claims before untracking. The memory store is single-instance only and says so; the resolver reads app_config.stream_bridge and the env var in the bridge's own order, so deployments already using Redis get a redis ownership store without extra config. * fix(sandbox): wait out a recovery grace before adopting a keyless container An absent ownership lease meant two opposite things on two paths. Renewal reads it as LAPSED and re-establishes it: nobody took the lease, so the container is still ours. Reconciliation read the same absent key as "orphan" and adopted on sight. After the store loses its keys (a Redis restart without persistence, or eviction under maxmemory) every owner is alive and merely pre-renewal-tick. Whichever instance reconciled first therefore adopted every live container; each real owner's next renewal reported LOST and dropped a sandbox it was serving mid-turn, leaving it for the adopter to idle-destroy — #4206 through the back door, in the very case the LAPSED handling was added to make safe. Not limited to startup: an already-running instance hits the same window from the idle checker's periodic reconcile. _adoptable_after_grace requires an untracked container to be seen unowned across a full lease TTL before it can be adopted. That rebuilds the delay the state loss erased: a live owner republishes within one renewal interval, shorter than the TTL by construction, while a crashed owner never does, so its containers are still adopted one grace later rather than leaking. A republished lease resets the grace; a pausing-only timer would still expire over a live owner's lease. The peek is read-only — the atomic claim still gates adoption. The grace is skipped when the store cannot coordinate across processes: no peer can hold a lease such a store would show us, so single-instance deployments keep instant orphan cleanup, and a grace could not help a multi-worker gateway on memory anyway. * fix(sandbox): hold the teardown lease for as long as the container stop runs claim(..., for_destroy=True) wrote the del: marker with the ordinary lease TTL and nothing refreshed it. renew() extends only own: and deliberately reports a teardown as LOST, and the destroy paths drop the sandbox from the maps the renewal loop iterates — so a container stop that outlived the TTL let the marker lapse, a peer's take() succeeded against the still-running container, and the stop then landed on the turn that had just been handed it. That is the exact window the del: state exists to close, reopened by its own expiry. The two lease states alone never made the per-sandbox flock redundant, as I claimed when deleting it: a held lock cannot expire, a lease can. The exclusion has to be held deliberately rather than assumed to outlast the work it guards. _held_teardown_lease wraps both _backend.destroy() call sites and re-claims the marker every renewal_interval_seconds until the stop returns. No store change is needed: claim(for_destroy=True) already refreshes an existing del: marker on both backends. Reachable without an abnormal backend. The schema bounds only renewal_interval_seconds (> 0) and ttl_multiplier (>= 2), so a legal config puts the TTL below a normal container stop; and LocalContainerBackend._stop_container passes no timeout to subprocess.run, so a wedged daemon blocks unbounded even at the default 120s TTL. The TTL stays finite on purpose: the heartbeat dies with the process, so a destroyer that crashes mid-stop still releases the container one TTL later instead of marking it undestroyable forever. * fix(sandbox): hold the teardown lease on every del: stop, and pin the claims that had no test 90936b49 said `_held_teardown_lease` wrapped "both" `_backend.destroy()` call sites. There are three. `_drop_unhealthy_sandbox` marks `del:` and then blocks on the same unbounded stop, and it untracks *before* claiming, so `_renew_owned_leases` cannot see the id either — nothing refreshed the marker. Reproduced against a real redis: the peer's `take()` succeeds 1.0s into a 2.5s stop. Same window, third path. That miss came from the habit the rest of this commit addresses: a property asserted in prose, with no test that could falsify it. Auditing every load-bearing claim in this feature — AGENTS.md, the store docstrings, the provider's design comments — against the test that would go red turned up several more, each verified by mutating the code and watching the suite stay green. Tests that could not fail: - `test_reconcile_fails_closed_when_ownership_unknown` reached the grace gate, not the claim. A bare MagicMock answers `owner()` with a truthy mock, so the container read as peer-owned and deferred; `claim()` was never called. It stayed green with `_claim_ownership` failing open. Adding the grace ahead of the claim is what hollowed it out — inserting a gate can silently disarm the tests for the gate behind it. - `test_adoption_grace_restarts_when_a_live_owner_republishes` never distinguished reset from pause. Those diverge only on a *second* lapse, which it never drove, so it passed with the reset deleted. Claims with no test at all, each now pinned (mutation → red, per test): - `destroy()`, `_evict_oldest_warm`, `_reclaim_warm_pool_sandbox`, `_register_created_sandbox` and `shutdown()`'s warm loop were each the one untested sibling of an "every path does X" enumeration. `shutdown()` was never driven with a non-empty warm pool, so a loop bypassing the ownership claim — stopping a live peer's container on our exit — went unnoticed. - Renewal's unknown-is-not-lost rule, the single deliberate exception to fail-closed. Inverting it drops every active and warm sandbox on every instance the moment the store blinks. - Both hops of the stream-bridge redis inference. Deleting either left the suite green while every config.yaml-native multi-instance deployment silently fell back to memory — #4206 reopened on exactly the deployments the inference exists for. Claims narrowed instead, because they promised more than the code delivers: - "run against both backends ... cannot drift" — CI provisions no redis, so the merge gate runs the memory tier only and the Lua never executes there. - "Every destroy path claims before untracking" — `_drop_unhealthy_sandbox` untracks first, deliberately, under its `expected_info` TOCTOU guard. - "Atomic: concurrent claims from different instances cannot both succeed" — true via Lua on redis, vacuous on the single-instance memory store, and pinned by neither, since the contract suite drives sequential calls. A concurrency test against the memory store would make the claim look covered while the mechanism that carries it still never runs in CI. * fix(sandbox): release the teardown marker when a destroy() stop fails The three `del:`-marked stop paths disagreed on failure. `_destroy_warm_entry` releases on both outcomes and says why: the stop failed, so the container is probably still up, and a marker left behind refuses its own thread's `take()` until the TTL lapses. `_drop_unhealthy_sandbox` does the same. `destroy()` had no such guard — a raising backend propagated straight past `_release_ownership`, and the thread could not re-acquire for a full TTL. Fails safe rather than fatal: a stuck marker stops peers from touching the container, it is not the cross-instance kill. But the paths must agree, and this one is the odd one out. Release, then re-raise. Swallowing would be the easier symmetry with `_destroy_warm_entry`'s `return False`, but `destroy()` has no failure return and `shutdown()` logs per sandbox off the exception, so swallowing would silently narrow what callers can see. Found by comparing the three paths after @fancyboi999 asked for release to be handled "consistently with the other destroy paths" on the unhealthy path — which 0d2377b2 already does. This is the sibling that wasn't. * fix(deploy): bump chart config_version to 27 for sandbox.ownership config.example.yaml went to 27 with the new sandbox.ownership section, but the chart embeds its own copy and stayed at 26, so validate-chart failed. A bare bump: the chart already sets stream_bridge.type=redis, which is what resolve_ownership_config infers a redis ownership store from, so no field change is needed. * fix(sandbox): release the teardown lease from its heartbeat, not the caller `_held_teardown_lease` joined its heartbeat only briefly and the caller cleared the `del:` marker right after the stop. A refresh `claim` still in flight (`RedisOwnershipStore` had no socket timeout, so a round trip could block) could land *after* that release and rewrite `del:` on a container whose stop had already completed — refusing a fresh `take()` (or rolling back a fresh create) until the TTL. Move the release into the heartbeat's own `finally`, after its loop stops, so no refresh can run after it. The three destroy paths no longer release after the `with` (`destroy()`'s no-container branch still does, since no lease was held there). Bound every store round trip with a socket timeout so the in-flight refresh — and thus the deferred release — stays finite, and broaden the heartbeat's `except` so an unexpected error cannot strand the marker during a long stop. Also fold in the review follow-ups: stop re-resolving an already-resolved ownership config in the factory, document the Redis-outage-vs-TTL boundary in config.example.yaml, and add a tests/blocking_io anchor pinning that `release()`'s store round trip stays off the event loop. * fix(sandbox): refuse a non-destroy claim that would unwind our own teardown `claim(for_destroy=False)` against our own `del:` lease fell through and overwrote it with `own:`, cancelling a teardown that was already in flight. The container stop cannot be recalled, so downgrading the marker would let a `take()` hand out a container that is about to die -- #4206, self-inflicted. No caller does this today: the two non-destroy callers run against an absent key (the LAPSED re-claim) or an unowned one (post-grace reconcile). The contract has to forbid it rather than rely on that staying true. Fixed in both backends. The redis rule lives in Lua and the memory rule in Python, so fixing one only would let them drift silently -- and the shared contract suite is what is supposed to catch that drift, so it now covers this. Also adds a contention test for `claim`. The suite drove sequential calls only, so it pinned the exclusion predicate but not the atomicity that predicate depends on; eight instances now race for one container and exactly one must win. * fix(sandbox): bound the container stop so it cannot outlive its teardown lease `_stop_container` passed no `timeout` to `subprocess.run`, so a wedged container runtime blocks it forever. The `del:` marker is what keeps a peer from re-acquiring the container while the stop runs, but a marker is a lease and a lease can lapse: a store outage longer than the TTL frees it, a peer's `take()` succeeds against the still-running container, and the stop then lands on the turn that was just handed it -- the exact #4206 failure. The teardown heartbeat already covers the case where the store stays reachable. This bounds the worst case independently of the ownership layer, which is the point: it holds even when the ownership layer is the thing that failed. A timeout is not swallowed like a `CalledProcessError`. That error means the runtime answered "I could not stop it"; a timeout means we do not know, and the container is probably still running -- returning normally would let `_destroy_warm_entry` report a clean stop and drop the warm entry, leaking a running container nothing tracks. * fix(sandbox): exclude this instance's own reapers from its acquire path An ownership lease excludes peers and nothing else. `claim()` and `take()` both succeed against our own `own:` lease by design -- that is what lets a destroy path claim what it already owns -- so `del:` says nothing to this process's other threads. Meanwhile every reaper decides outside `_lock`, because a store round trip must not be held under the lock that guards every acquire. So each reaper acts on a decision its own acquire path may already have invalidated, and the store cannot see the difference. Six paths end in an irreversible act (a container stop, or closing a host-side client) on a decision made outside the lock. All six reproduce: _evict_oldest_warm re-checks warm membership, then releases the lock _reap_expired_warm no re-check at all _cleanup_idle_sandboxes re-verifies idle, then releases the lock _renew_owned_leases acts on a stale renew() -> LOST release() same staleness on its own refresh _drop_unhealthy_sandbox untracks before claiming, opening discovery Both warm reapers are a regression from the deferred pop this branch introduced: `WarmPoolLifecycleMixin` popped under the lock, so a reclaim's membership check failed and the race could not occur. Deferring the pop is still right (popping first loses the container on a refused claim), so the exclusion has to be made explicit instead. The idle path is pre-existing in shape, but this branch widened it from a few instructions to a network round trip by claiming ownership before untracking. Two guards, because the two directions want opposite answers: Reaping -- nothing may promote it. The reaper reserves the id, and every promote path refuses a reserved id exactly as it refuses a peer's `del:` (drop and cold-start). The "is this still reapable?" test travels with the reservation as a predicate and runs in the same critical section, because checking first and reserving second is the window, not a narrower version of it. Forgetting -- the peer legitimately wins, so the promote is what to detect. `_publish_ownership` bumps a per-id acquire epoch; the callers that decide from a store round trip snapshot it first, and the pop is skipped if it moved. Object identity cannot substitute: the reuse path re-publishes ownership while handing out the same tracked `AioSandbox`, so an identity check sees nothing and the pop closes a client mid-turn. `still_reapable` is required rather than defaulting to unconditional -- the safe default is the one that makes a new call site think about it. That diverges from the mixin hook, which is safe because this provider overrides both mixin callers, and loud rather than silent if those are ever dropped. Also closes a client leak on the discover path: "nothing to roll back" was true of the container but not of the HTTP client constructed before the publish, which the sibling create path already closes. The shared-store test view rebound `owner_id` outside the store's lock, so a concurrent claim could execute under the wrong id and read its own lease as a peer's. Serialized, so the heartbeat-hold tests stop flaking. * fix(sandbox): mark acquire intent before the ownership round trip A guard must become visible no later than the transition it guards. The acquire epoch cannot manage that for `take()`: the takeover is durable before `take()` returns -- redis has committed the SET while the reply is still in flight -- and the epoch can only be written afterwards. In that interval the store already says the container is ours while the epoch still reads as it did when a renewal decided `LOST`, so the stale forget walks through, drops the maps and closes the client the acquire is about to hand back. Acquire then returns an id the provider no longer tracks and `get()` answers `None` for the rest of the turn. `_publish_ownership` now publishes an intent mark under `_lock` before the round trip; the epoch keeps covering the other half, "an acquire completed since you decided". `_forget_lost_sandbox` honours the intent mark unconditionally rather than only when an epoch is supplied -- today's epoch-less callers cannot reach the window, but "no epoch" reading as "no guard" is how the next caller of a dangerous primitive gets written. The same invariant had four more instances, all reproduced: reuse returns a decision the forget already invalidated -- before the mark is set a `LOST` is both current and correct, so the forget legitimately runs and the entry reuse decided to hand out is gone. Re-check after publishing and fall through to discovery instead. reclaim installs an entry a reaper reserved after its check -- the warm entry is still visible during the stop, and the reaper's claim succeeds because reclaim's own take() just made the lease ours. Re-check likewise. the reservation was released before the entry was removed -- the pop belonged to the caller, leaving a gap where the container is stopped, the entry is still in `_warm_pool`, and nothing marks it. `_destroy_warm_entry` removes it itself, inside the reservation; the pop stays deferred relative to the stop, just not to the reservation. reconcile adopts a container this instance is tearing down -- adoption is a promote and needs the same reservation check as the others. Neither existing guard excludes it: the claim succeeds because the lease is ours, and on `memory` the recovery grace is skipped outright. The pre-round-trip checks in reuse and reclaim are kept as early-outs, since they skip a health check and a store round trip on a doomed entry, and are pinned to that job rather than to a correctness role they no longer hold. The teardown reservation predicate runs under `_lock`, so it must not touch the lock. Documented rather than engineered around: making the lock reentrant to tolerate it would trade a loud hang for a quiet class of re-entrancy bugs across the rest of the provider. * fix(sandbox): honor local teardown after ownership publish * fix(sandbox): clear a stale warm entry when an id becomes active Active and warm are exclusive states, and the two register paths were the only place that could hold both: they inserted into `_sandboxes` without popping `_warm_pool`, so one container ended up with two reapers. `_reap_expired_warm` judges an entry by its warm timestamp and never consults `_last_activity`, so it stops a container an agent is actively using while `_sandboxes` still hands out its client. Reachable because `_reconcile_orphans` adopts an untracked-but-running container into the warm pool inside the register's publish -> track window, and on the `memory` store it adopts on sight: `_adoptable_after_grace` short-circuits when `supports_cross_process` is False, so an id carrying this process's own lease reads as adoptable. That window is new to this branch -- on main the track was a single locked insert with nothing before it. Both register paths now pop the warm entry inside the same locked section that installs the active one. * fix(sandbox): harden ownership renewal teardown --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
DeerFlow Backend
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
Architecture
┌──────────────────────────────────────┐
│ Nginx (Port 2026) │
│ Unified reverse proxy │
└───────┬──────────────────┬───────────┘
│
/api/langgraph/* │ /api/* (other)
rewritten to /api/* │
▼
┌────────────────────────────────────────┐
│ Gateway API (8001) │
│ FastAPI REST + agent runtime │
│ │
│ Models, MCP, Skills, Memory, Uploads, │
│ Artifacts, Threads, Runs, Streaming │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Lead Agent │ │
│ │ Middleware Chain, Tools, Subagents │ │
│ └────────────────────────────────────┘ │
└────────────────────────────────────────┘
Request Routing (via Nginx):
/api/langgraph/*→ Gateway LangGraph-compatible API - agent interactions, threads, streaming/api/*(other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup/(non-API) → Frontend - Next.js web interface
Core Components
Lead Agent
The single LangGraph agent (lead_agent) is the runtime entry point, created via make_lead_agent(config). It combines:
- Dynamic model selection with thinking and vision support
- Middleware chain for cross-cutting concerns (9 middlewares)
- Tool system with sandbox, MCP, community, and built-in tools
- Subagent delegation for parallel task execution
- System prompt with skills injection, memory context, and working directory guidance
Middleware Chain
Middlewares execute in strict order, each handling a specific concern:
| # | Middleware | Purpose |
|---|---|---|
| 1 | ThreadDataMiddleware | Creates per-thread isolated directories (workspace, uploads, outputs) |
| 2 | UploadsMiddleware | Injects newly uploaded files into conversation context |
| 3 | SandboxMiddleware | Acquires sandbox environment for code execution |
| 4 | SummarizationMiddleware | Reduces context when approaching token limits (optional) |
| 5 | TodoListMiddleware | Tracks multi-step tasks in plan mode (optional) |
| 6 | TitleMiddleware | Auto-generates conversation titles after first exchange |
| 7 | MemoryMiddleware | Queues conversations for async memory extraction |
| 8 | ViewImageMiddleware | Injects image data for vision-capable models (conditional) |
| 9 | ClarificationMiddleware | Intercepts clarification requests and interrupts execution (must be last) |
Sandbox System
Per-thread isolated execution with virtual path translation:
- Abstract interface:
execute_command,read_file,write_file,list_dir - Providers:
LocalSandboxProvider(filesystem) andAioSandboxProvider(Docker, in community/). Async runtime paths use async sandbox lifecycle hooks so startup, readiness polling, and release do not block the event loop.AioSandboxProvidervalidates active-cache and warm-pool containers during acquire/reuse, dropping definitively dead entries so a thread can provision a fresh sandbox after an unexpected container exit while keepingget()as an in-memory lookup. Backend health-check failures are treated as unknown, not dead, and a container that cannot be verified during discovery is simply not adopted (acquire falls through to create instead of failing). - Virtual paths:
/mnt/user-data/{workspace,uploads,outputs}→ thread-specific physical directories - Skills path:
/mnt/skills→deer-flow/skills/directory - Skills loading: Recursively discovers nested
SKILL.mdfiles underskills/{public,custom}and preserves nested container paths - SkillScan: Native offline deterministic scanning runs before the LLM skill scanner on installs and agent-managed skill writes;
CRITICALfindings block and warning findings become LLM context - File-write safety:
str_replaceserializes read-modify-write per(sandbox.id, path)so isolated sandboxes keep concurrency even when virtual paths match - Tools:
bash,ls,read_file,write_file,str_replace(write_fileoverwrites by default and exposesappendfor end-of-file writes;bashis disabled by default when usingLocalSandboxProvider; useAioSandboxProviderfor isolated shell access)
Subagent System
Async task delegation with concurrent execution:
- Built-in agents:
general-purpose(full toolset) andbash(command specialist, exposed only when shell access is available) - Concurrency: Max 3 subagents per turn, 15-minute timeout
- Execution: Background thread pools with status tracking and SSE events
- Flow: Agent calls
task()tool → executor runs subagent in background → polls for completion → returns result
Memory System
LLM-powered persistent context retention across conversations:
- Automatic extraction: Analyzes conversations for user context, facts, and preferences
- Structured storage: User context (work, personal, top-of-mind), history, and confidence-scored facts
- Debounced updates: Batches updates to minimize LLM calls (configurable wait time)
- System prompt injection: Top facts + context injected into agent prompts
- Run-level memory identity:
GET /api/threads/{thread_id}/runs/{run_id}/events?event_types=context:memoryreturns the SHA-256 identity of the effective hidden memory block without copying memory text into the event store - Storage: JSON file with mtime-based cache invalidation
Tool Ecosystem
| Category | Tools |
|---|---|
| Sandbox | bash, ls, read_file, write_file, str_replace |
| Built-in | present_files, ask_clarification, view_image, task (subagent) |
| Community | Tavily (web search), Jina AI (web fetch), Crawl4AI (web fetch), Firecrawl (scraping), fastCRW (scraping), DuckDuckGo (image search) |
| MCP | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
| Skills | Domain-specific workflows injected via system prompt |
Gateway API
FastAPI application providing REST endpoints for frontend integration:
| Route | Purpose |
|---|---|
GET /api/models |
List available LLM models |
GET/PUT /api/mcp/config |
Manage MCP server configurations |
POST /api/mcp/cache/reset |
Reset cached MCP tools so they reload on next use |
GET/PUT /api/skills |
List and manage skills |
POST /api/skills/install |
Install skill from .skill archive |
GET /api/memory |
Retrieve memory data |
POST /api/memory/reload |
Force memory reload |
GET /api/memory/config |
Memory configuration |
GET /api/memory/status |
Combined config + data |
GET /api/threads/{id}/runs/{run_id}/events |
Debug/audit events for one run; filter event_types=context:memory for effective memory identity |
POST /api/threads/{id}/uploads |
Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths, auto-renames duplicate filenames in one request) |
GET /api/threads/{id}/uploads/list |
List uploaded files |
DELETE /api/threads/{id} |
Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
GET /api/threads/{id}/artifacts/{path} |
Serve generated artifacts |
IM Channels
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final runs.wait() response path, while Feishu now streams through runs.stream(["messages-tuple", "values"]), serializes rapid same-thread turns inside the channel manager, and updates a single in-thread card per source message in place.
For Feishu card updates, DeerFlow stores the running card's message_id per inbound message and patches that same card until the run finishes, preserving the existing OK / DONE reaction flow. When a follow-up arrives inside an existing Feishu topic while another turn is still running, the later message now waits on the mapped DeerFlow thread_id, receives a queued/running card on that exact source message, and keeps a compact source-message blockquote in subsequent patches so rapid consecutive questions remain distinguishable.
Quick Start
Prerequisites
- Python 3.12+
- uv package manager
- API keys for your chosen LLM provider
Installation
cd deer-flow
# Copy configuration files
cp config.example.yaml config.yaml
# Install backend dependencies
cd backend
make install
Configuration
Edit config.yaml in the project root:
models:
- name: gpt-4o
display_name: GPT-4o
use: langchain_openai:ChatOpenAI
model: gpt-4o
api_key: $OPENAI_API_KEY
supports_thinking: false
supports_vision: true
- name: gpt-5-responses
display_name: GPT-5 (Responses API)
use: langchain_openai:ChatOpenAI
model: gpt-5
api_key: $OPENAI_API_KEY
use_responses_api: true
output_version: responses/v1
supports_vision: true
Set your API keys:
export OPENAI_API_KEY="your-api-key-here"
Running
Full Application (from project root):
make dev # Starts Gateway + Frontend + Nginx
Access at: http://localhost:2026
Backend Only (from backend directory):
# Gateway API + embedded agent runtime
make dev
Direct access: Gateway at http://localhost:8001
Terminal Workbench (TUI) — a terminal-native UI over the embedded harness, no services required:
uv pip install 'deerflow-harness[tui]' # optional 'textual' dependency
deerflow # launch the TUI
deerflow --print "summarize this repo" # headless one-shot
Sessions opened in the TUI appear in the Web UI sidebar (it writes the shared
threads_meta store under the local default user). See docs/TUI.md.
Project Structure
backend/
├── packages/harness/ # deerflow-harness package (import: deerflow.*)
│ └── deerflow/
│ ├── agents/ # Agent system
│ │ ├── lead_agent/ # Main agent (factory, prompts)
│ │ ├── middlewares/ # Middleware components
│ │ ├── memory/ # Memory extraction & storage
│ │ └── thread_state.py # ThreadState schema
│ ├── sandbox/ # Sandbox execution
│ │ ├── local/ # Local filesystem provider
│ │ ├── sandbox.py # Abstract interface
│ │ ├── tools.py # bash, ls, read/write/str_replace
│ │ └── middleware.py # Sandbox lifecycle
│ ├── subagents/ # Subagent delegation
│ │ ├── builtins/ # general-purpose, bash agents
│ │ ├── executor.py # Background execution engine
│ │ └── registry.py # Agent registry
│ ├── tools/builtins/ # Built-in tools
│ ├── mcp/ # MCP protocol integration
│ ├── models/ # Model factory
│ ├── skills/ # Skill discovery & loading
│ ├── config/ # Configuration system
│ ├── runtime/ # Embedded run execution (RunManager, StreamBridge)
│ ├── persistence/ # Checkpointer/store engines & schema migrations
│ ├── guardrails/ # Pre-tool-call authorization providers
│ ├── tracing/ # Tracer factory & trace metadata
│ ├── uploads/ # Uploads manager
│ ├── tui/ # Terminal UI (`deerflow` console script)
│ ├── community/ # Community tools & providers
│ ├── reflection/ # Dynamic module loading
│ └── utils/ # Utilities
├── app/ # FastAPI Gateway + IM channels (import: app.*)
│ ├── gateway/ # Gateway API
│ │ ├── app.py # Application setup
│ │ └── routers/ # Route modules
│ └── channels/ # IM channel integrations
├── docs/ # Documentation
├── tests/ # Test suite
├── langgraph.json # LangGraph graph registry for tooling/Studio compatibility
├── pyproject.toml # Python dependencies
├── Makefile # Development commands
└── Dockerfile # Container build
langgraph.json is not the default service entrypoint. The scripts and Docker
deployments run the Gateway embedded runtime; the file is kept for LangGraph
tooling, Studio, or direct LangGraph Server compatibility.
Configuration
Main Configuration (config.yaml)
Place in project root. Config values starting with $ resolve as environment variables.
Key sections:
models- LLM configurations with class paths, API keys, thinking/vision flagstools- Tool definitions with module paths and groupstool_groups- Logical tool groupingssandbox- Execution environment providerskills- Skills directory pathstitle- Auto-title generation settingssummarization- Context summarization settingssubagents- Subagent system (enabled/disabled)memory- Memory system settings (enabled, storage, debounce, facts limits)
Provider note:
models[*].usereferences provider classes by module path (for examplelangchain_openai:ChatOpenAI).- If a provider module is missing, DeerFlow now returns an actionable error with install guidance (for example
uv add langchain-google-genai).
Extensions Configuration (extensions_config.json)
MCP servers and skill states in a single file:
{
"mcpServers": {
"github": {
"enabled": true,
"type": "stdio",
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"}
},
"secure-http": {
"enabled": true,
"type": "http",
"url": "https://api.example.com/mcp",
"oauth": {
"enabled": true,
"token_url": "https://auth.example.com/oauth/token",
"grant_type": "client_credentials",
"client_id": "$MCP_OAUTH_CLIENT_ID",
"client_secret": "$MCP_OAUTH_CLIENT_SECRET"
}
},
"postgres": {
"enabled": false,
"type": "stdio",
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"],
"description": "PostgreSQL database access",
"routing": {
"mode": "prefer",
"priority": 50,
"keywords": ["orders", "users", "SQL", "database", "table"]
},
"tools": {
"query": {
"routing": {
"priority": 100,
"keywords": ["query database", "orders table", "metrics"]
}
}
}
}
},
"skills": {
"pdf-processing": {"enabled": true}
}
}
routing adds soft MCP preference hints to the agent prompt. It helps the
model prefer a configured MCP tool for matching requests without forbidding
other tools. When tool_search.enabled=true defers MCP schemas, matching
routing metadata can auto-promote up to tool_search.auto_promote_top_k
deferred schemas before the model call.
Environment Variables
DEER_FLOW_CONFIG_PATH- Override config.yaml locationDEER_FLOW_EXTENSIONS_CONFIG_PATH- Override extensions_config.json location- Model API keys:
OPENAI_API_KEY,ANTHROPIC_API_KEY,DEEPSEEK_API_KEY, etc. - Tool API keys:
TAVILY_API_KEY,GITHUB_TOKEN, etc.
LangSmith Tracing
DeerFlow has built-in LangSmith integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
Setup:
- Sign up at smith.langchain.com and create a project.
- Add the following to your
.envfile in the project root:
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
LANGSMITH_PROJECT=xxx
Legacy variables: The LANGCHAIN_TRACING_V2, LANGCHAIN_API_KEY, LANGCHAIN_PROJECT, and LANGCHAIN_ENDPOINT variables are also supported for backward compatibility. LANGSMITH_* variables take precedence when both are set.
Langfuse Tracing
DeerFlow also supports Langfuse observability for LangChain-compatible runs.
Add the following to your .env file:
LANGFUSE_TRACING=true
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_BASE_URL=https://cloud.langfuse.com
If you are using a self-hosted Langfuse deployment, set LANGFUSE_BASE_URL to your Langfuse host.
Dual Provider Behavior
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
Docker: In docker-compose.yaml, tracing is disabled by default (LANGSMITH_TRACING=false). Set LANGSMITH_TRACING=true and/or LANGFUSE_TRACING=true in your .env, together with the required credentials, to enable tracing in containerized deployments.
Development
Commands
make install # Install dependencies
make dev # Run Gateway API + embedded agent runtime (port 8001)
make gateway # Run Gateway API without reload (port 8001)
make lint # Run linter (ruff)
make format # Format code (ruff)
make detect-blocking-io # Inventory blocking IO that may block the backend event loop
make migrate-rev MSG="..." # Autogenerate a new alembic revision against the live ORM models
Schema Migrations
DeerFlow's application tables (runs, threads_meta, feedback, users,
run_events, and the channel_* tables) are owned by alembic. The Gateway
runs alembic upgrade head automatically on startup via
bootstrap_schema(engine, backend=...), so operators do not run alembic
manually in production. Bootstrap is concurrency-safe (Postgres advisory lock
across processes; per-engine asyncio.Lock inside one SQLite process) and
idempotent against pre-existing schemas (empty / legacy / versioned).
When you add or change an ORM model, ship the change as a new revision under
packages/harness/deerflow/persistence/migrations/versions/:
make migrate-rev MSG="add foo column to runs"
The target invokes scripts/_autogen_revision.py, which builds a fresh temp
SQLite at head and diffs the live models against it — so a clean checkout
does not need a pre-existing ./data/deerflow.db. Review the generated file
and switch raw op.add_column / op.drop_column calls to the idempotent
helpers in migrations/_helpers.py before committing. There is no
make migrate / make migrate-stamp target on purpose — Gateway startup is
the only execution path, which keeps operational mistakes off the table. See
backend/CLAUDE.md (Schema Migrations) for the full design.
Code Style
- Linter/Formatter:
ruff - Line length: 240 characters
- Python: 3.12+ with type hints
- Quotes: Double quotes
- Indentation: 4 spaces
Testing
uv run pytest
make detect-blocking-io statically scans backend business code for blocking
IO that may run on the backend event loop and is not test-coverage-bound. It
prints a concise summary for human review and writes complete JSON findings to
.deer-flow/blocking-io-findings.json at the repository root (regardless of
whether the target is invoked from the repo root or from backend/). JSON
findings include both broad IO category and review-oriented fields such as
priority, location, blocking_call, event_loop_exposure, reason, and
code. priority is a deterministic review ordering from the operation type,
not proof of a bug. Bare-name same-file calls are resolved by function name,
so duplicate helper names in one file can conservatively over-report async
reachability.
Technology Stack
- LangGraph (1.0.6+) - Agent framework and multi-agent orchestration
- LangChain (1.2.3+) - LLM abstractions and tool system
- FastAPI (0.115.0+) - Gateway REST API
- langchain-mcp-adapters - Model Context Protocol support
- agent-sandbox - Sandboxed code execution
- markitdown - Multi-format document conversion
- tavily-python / firecrawl-py - Web search and scraping
Documentation
- Configuration Guide
- Architecture Details
- API Reference
- File Upload
- Path Examples
- Context Summarization
- Plan Mode
- Setup Guide
License
See the LICENSE file in the project root.
Contributing
See CONTRIBUTING.md for contribution guidelines.