github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-14 16:08:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/docs/ARCHITECTURE.md
Hyeonsang Cho 556975f284
fix(gateway): gate github_token and disable_clarification on internal callers (#5338) 
* fix(gateway): gate github_token and disable_clarification on internal callers

`non_interactive` is honored only for internally-authenticated callers
because it strips `ask_clarification` from the lead-agent toolset. The two
sibling run-context keys reproduced that effect without the gate.

`merge_run_context_overrides` forwarded `_CONTEXT_RUNTIME_ONLY_KEYS`
regardless of `internal`, and `strip_internal_context_keys` scrubbed only
`_CONTEXT_INTERNAL_CALLER_KEYS` -- so any session or PAT caller could set
`disable_clarification` through `body.context`, or through the free-form
`body.config` that `build_run_config` copies verbatim. That is not a milder
flag than `non_interactive`: ClarificationMiddleware answers every
clarification -- `risk_confirmation` included -- with "proceed without
asking" instead of interrupting, and SandboxMiddleware reads the two keys as
the same non-interactive signal. `github_token` rode the same path into
`runtime.context`, where the bash tool exports it as `GH_TOKEN`/`GITHUB_TOKEN`,
and a copy smuggled through `body.config['configurable']` reached the
checkpoint store the context-only rule exists to avoid.

Both keys are produced server-side by the channel run policies, which reach
the Gateway over the internally-authenticated request channel, so gate them
the same way: forward them only when `internal=True`, and scrub the union
`_INTERNAL_ONLY_CONTEXT_KEYS` from both config sections for every other
caller. Destination stays an orthogonal axis -- `_CONTEXT_RUNTIME_ONLY_KEYS`
still land in `context` alone, never in checkpoint-persisted `configurable`.

Regression coverage in tests/test_gateway_services.py pins both smuggling
surfaces and replays the real start_run assembly order for a session caller
and for an internal one, so the GitHub channel keeps carrying its minted
token.

* docs(changelog): record the internal-only run-context key gate (#5338)

* docs(agents): keep the run-context note inside the AGENTS.md budgets

The AG002 inherited-chain check failed at this head. The new backend section
and the root scheduled-task sentence added 993 B to the root and backend
guidance both the sandbox and middlewares chains inherit, pushing sandbox 6 B
over the 98304 B hard limit and growing the middlewares chain, which main
already exceeds by 155 B. An already-over chain is only tolerated while it does
not grow, so the shared ancestors had to come back to their base size.

Condensed the new material and removed prose the root file was duplicating:

- The trust-boundary section keeps both gated surfaces, both helpers, the
  trust-vs-destination split, and the disable_clarification note in half the
  space.
- The root scheduled-task bullet names all three internal-only keys and both
  smuggling surfaces while staying under its previous size.
- Dropped the root `scheduler.recursion_limit` bullet, which restated
  backend/AGENTS.md:18 almost verbatim; its one unique fact (a YAML edit needs
  no Gateway restart) moved to that bullet.
- Deduplicated the nginx routing sentence, which already deferred to the
  backend routing table, and tightened the waiver note's sequencing tail.

Root and backend guidance now sit 50 B under their combined base size, so the
sandbox chain returns to 97310 B and the middlewares chain no longer grows.
Every file stays under its AG001 soft budget.
2026-09-11 11:33:52 +08:00

11 KiB
Raw Blame History

DeerFlow Architecture

This document is the top-level architecture overview for DeerFlow. It explains the "big picture" — how the services, layers, and cross-cutting subsystems fit together — and points to the module-level guides that own the depth:

DeerFlow 2.0 is a ground-up rewrite of the original Deep Research framework (see README.md); it shares no code with v1.

1. What DeerFlow Is

DeerFlow (Deep Exploration and Efficient Research Flow) is an open-source super-agent harness built on LangGraph. A single "lead agent" orchestrates sub-agents, persistent memory, sandboxed code execution, and extensible skills/tools — all isolated per conversation thread. The frontend is a Next.js chat UI; external IM platforms (Feishu, Slack, Telegram, Discord, DingTalk) bridge into the same agent through the Gateway.

2. Service Topology

A single make dev (or Docker stack) runs four cooperating services; Nginx is the only public entry point.

Service Port Role
Nginx 2026 Unified reverse proxy — open this in the browser
Gateway API 8001 FastAPI REST API + embedded LangGraph-compatible agent runtime
Frontend 3000 Next.js web interface
Provisioner 8002 Optional — only when sandbox is in provisioner/K8s mode

Nginx routing (the key entry-point contract):

  • /api/langgraph/* → Gateway's LangGraph-compatible runtime (rewritten to native /api/*)
  • /api/* (other) → Gateway REST routers
  • /* (non-API) → Frontend

This lets standard LangGraph SDK clients talk to DeerFlow without a separate LangGraph server. Both compose files publish nginx as "${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026" — loopback by default; the Gateway's 8001 is never published. Any new published port must carry an explicit bind address (backend/tests/test_compose_default_bind_host.py pins this for every service in both compose files).

3. Backend: Harness / App Split

The backend is two layers with a strict one-way dependency:

  • Harness (backend/packages/harness/deerflow/, import prefix deerflow.*) — the publishable agent framework: orchestration, tools, sandbox, models, MCP, skills, memory, config. Everything needed to build and run agents.
  • App (backend/app/, import prefix app.*) — unpublished application code: the FastAPI Gateway and IM channel integrations.

Rule: App imports deerflow, but deerflow never imports app. This boundary is enforced in CI by backend/tests/test_harness_boundary.py. A thin third package, deerflow-extension-api (backend/packages/extension-api/), defines the host-independent extension contract that plugins implement.

There is also an embedded Python client (deerflow.client.DeerFlowClient) used by scheduled tasks and tests to drive the same run lifecycle programmatically.

Agent runtime path

All run modes (local make dev, Docker, prod) execute the agent through the Gateway via RunManager + run_agent() + StreamBridge (packages/harness/deerflow/runtime/). The agent is assembled by make_lead_agent() and wrapped in a middleware chain that runs before the model call:

  1. ThreadDataMiddleware — set up workspace/uploads/outputs paths
  2. UploadsMiddleware — inject uploaded file list
  3. SandboxMiddleware — acquire sandbox
  4. SummarizationMiddleware — context reduction (if enabled)
  5. TitleMiddleware — auto-generate conversation title
  6. TodoListMiddleware — task tracking (plan mode)
  7. ViewImageMiddleware — vision-model image handling
  8. ClarificationMiddleware — handle ask_clarification

SSE streaming carries both per-chunk messages and bounded values snapshots; with stream_subgraphs, delegated subagents publish namespaced SSE events (values|<ns>, LangGraph Platform style) rather than impersonating root frames, so SDK clients don't lose the parent thread view.

State, tools, sandbox

  • ThreadState extends LangGraph's AgentState with sandbox, artifacts, thread_data, title, todos, viewed_images. Each thread gets isolated data dirs under backend/.deer-flow/threads/{thread_id}/.
  • Tools come from three sources, merged by get_available_tools(): built-ins (present_files, ask_clarification, view_image, review_skill_package), configured tools (bash, read_file, write_file, str_replace, ls, web search/fetch), and MCP tools.
  • Sandbox is an abstract SandboxProvider with LocalSandboxProvider (dev, direct execution) and AioSandboxProvider (Docker, production isolation). Agent code executes inside sandbox boundaries with virtual path mapping (/mnt/user-data/...).

4. Frontend: Stateful Chat over LangGraph SDK

Next.js 16 / React 19 / TypeScript / Tailwind v4. Stack: LangGraph SDK (@langchain/langgraph-sdk) for orchestration + streaming, TanStack Query for server state. Requires Node 22+ and pnpm 10.26.2+.

The frontend is a stateful chat app: users create threads (conversations), send messages, set thread-scoped /goal completion conditions, and receive streamed responses. The backend may produce artifacts (files/code), todos, and goal-state updates.

Source layout (frontend/src/):

  • app/ — App Router routes: /workspace/chats/[thread_id] (authenticated chat), /workspace/agents/[agent_name] (custom agents), /showcase/[thread_id] (allowlisted public read-only demos), /api/* route handlers, (auth)/{login,setup,auth/callback}.
  • core/ — the business-logic heart. Domains: threads/ (creation, streaming, state), api/ (LangGraph client singleton), agents/, auth/, artifacts/, channels/, integrations/, memory/, skills/, mcp/, models/, tasks/, todos/, tools/, workspace-changes/, config/, i18n/ (en-US, zh-CN), and more.
  • components/ — workspace/ (chat), landing/, docs/; ui/ and ai-elements/ are registry-generated (Shadcn / Vercel AI SDK) and must not be hand-edited.
  • hooks/, lib/ (cn()), content/ (MDX), styles/.

Streaming data flow: core/threads/ subscribes to the LangGraph run stream via the core/api/ client singleton, normalizes SSE events (messages, values, task_*, artifact deltas) into TanStack-Query-managed thread state that components render. Subtask progress rides root-namespace task_* custom events (the web frontend does not request subgraph streaming).

By default the frontend connects through nginx: NEXT_PUBLIC_LANGGRAPH_BASE_URL=/api/langgraph and NEXT_PUBLIC_BACKEND_BASE_URL= (empty). Leave these unset for the standard make dev / Docker flow.

5. Cross-Cutting Subsystems

These span both layers and require reading multiple files to understand:

  • Config system — lives at repo root: config.yaml (models, tools, sandbox, summarization, scheduler) and extensions_config.json (MCP servers + skills). Both are gitignored, generated from the *.example.* templates, and editable at runtime via the Gateway API. Operator-controlled third-party plugins: live only in config.yaml (never the API-writable extensions_config.json) because that list causes code import.
  • Skills — skills/public/ (committed) and skills/custom/ (gitignored); managed integration packs are global at .deer-flow/integrations/skills/{provider}/. Skills are discovered/loaded lazily by the harness; skills/public/skill-reviewer/ is a read-only quality reviewer using the harness review_skill_package tool.
  • Sub-agents — background delegation via SubagentExecutor (server-side execution_id) correlated to provider tool_call_id for ToolMessage/SSE/lifecycle/persistence. Scheduled tasks reuse the same Gateway run lifecycle (scheduler decides when, not how).
  • Scheduled tasks — workspace page /workspace/scheduled-tasks + a background scheduler gated by config.yaml → scheduler.enabled; non-interactive runs drop ask_clarification and client-supplied non_interactive (see the run-context trust boundary in §6).
  • Long-running MCP — a durable McpTaskService (leased rows, DB as source of truth) keeps remote task IDs/polling out of the agent loop.
  • Version sources — a release version must match in backend/pyproject.toml, frontend/package.json, and deploy/helm/deer-flow/Chart.yaml (version + appVersion); pushing a v* tag triggers CI that runs scripts/verify_versions.sh and blocks all publishing on drift. See RELEASING.md.

6. Security & Isolation Model

  • Thread isolation: each conversation has separate data dirs; uploads are validated against path traversal and staged as .upload-*.part before atomic replace.
  • Sandbox isolation: production should use the Docker AioSandboxProvider; local sandbox is dev-only direct execution.
  • MCP isolation: each MCP server runs in its own process with runtime env-var resolution; servers toggle independently.
  • Run-context trust boundary: run context reaches the agent from two client-writable surfaces — body.context and the free-form body.config — so every server-produced key is gated on both. non_interactive, disable_clarification, and github_token are honored only for internally-authenticated callers (the scheduler and IM/webhook channel policies) and scrubbed from a non-internal caller's config; identity and sandbox lifecycle fields are cleared unconditionally and restamped from auth state.
  • Loopback-by-default ingress: nginx is the only published surface; the Gateway's 8001 is container-internal and never published. A bare "${PORT}:2026" bind (0.0.0.0) is rejected by convention and CI. See the Security Notice in README.md before any non-loopback deployment.

7. Where to Go Next