github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-25 14:06:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/AGENTS.md
Nan Gao 5d8a9492ee
docs(extensions v0.2.3): cover full-stack plugins and request-scoped run evidence (#5775) 
* docs(extensions): add an eleven-page extension developer manual

Add a harness/extensions/ section in English and Chinese documenting the
deerflow-extension-api 0.2.3 contract:

  index                when to write an extension vs a tool, MCP server or
                       skill; contribution kinds; loading; failure and
                       trust model; versioning
  quick-start          build, test, install and remove a working extension
  runtime              load sequence, API version rules, scopes and
                       ExtensionData, fail-open and cancellation, diagnostics
  middleware           placements and where each lands, scope, ordering,
                       observe-only wrap hooks, failure isolation
  observers            task lifecycle, system model calls, agent assembly
                       and fingerprint, context compaction
  services-and-routes  service lifecycle, router rejection rules, auth,
                       principals
  run-evidence         service and request-scoped readers (#5727), cursor
                       semantics, deletions, redaction, store differences
  plugins              experimental full-stack plugins, inline modules and
                       manifest-based browser assets (#5685)
  operations           extension manager CLI, sources, upgrade, rollback,
                       Docker, Helm, recovery
  troubleshooting      indexed by the exact log and error strings
  reference            every public name and the contract version history

Also fix stale descriptions: the contribution kinds listed in AGENTS.md
and deerflow/extensions/AGENTS.md (now eight, including plugins), the
run-evidence redaction claim (only auth_token is removed), and the plugin
mount return value in docs/full-stack-plugins.md ({ dispose }, not a
function). Changelog entries added in both languages.

* docs(agents): trim root AGENTS.md to keep inherited chains under the size limit

* docs(extensions): scope the manual to the 0.2.1 contract

Plugins and request-scoped run evidence are not in 2.1.x-dev yet; they
move to a stacked follow-up so this change can be cherry-picked.

* docs(extensions): cover full-stack plugins and request-scoped run evidence

Brings the extension manual up to deerflow-extension-api 0.2.3. Stacked on
the 0.2.1 manual so that part can be cherry-picked to 2.1.x-dev alone.

* docs(extensions): clarify PAT rejection on contributed routes

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-23 18:55:12 +08:00

240 lines
15 KiB
Markdown
Raw Blame History

# AGENTS.md
This file provides guidance to AI coding agents (Claude Code, Codex, and others) when working with code in this repository. It is the source of truth; the sibling `CLAUDE.md` imports it via `@AGENTS.md`.
It is the **monorepo orientation layer**: it maps the whole repo and points to the
module guides that own the depth. For anything inside a module, read that module's
guide rather than expecting full detail here:
- **[backend/AGENTS.md](backend/AGENTS.md)** — backend depth: harness/app split, agent &
middleware chain, sandbox, MCP, skills, memory, IM channels, persistence/migrations,
config system, test layout.
- **[frontend/AGENTS.md](frontend/AGENTS.md)** — frontend depth: Next.js App Router layout,
thread/streaming data flow, code style, commands.
## What is DeerFlow
DeerFlow is a LangGraph-based AI super-agent system with a full-stack architecture. The
backend runs a "super agent" with sandboxed execution, persistent memory, subagent
delegation, and extensible tools (built-in, MCP, community), all per-thread isolated. The
frontend is a Next.js chat UI. External IM platforms (Feishu, Slack, Telegram, Discord,
DingTalk) bridge into the same agent through the Gateway.
## Service Topology
A single `make dev` / Docker stack runs four cooperating services:
| Service | Port | Role |
| --------------- | ------ | ------------------------------------------------------------------- |
| **Nginx** | `2026` | Unified reverse-proxy entry point — open this in the browser |
| **Gateway API** | `8001` | FastAPI REST API + embedded LangGraph-compatible agent runtime |
| **Frontend** | `3000` | Next.js web interface |
| **Provisioner** | `8002` | Optional — only when sandbox is configured for provisioner/K8s mode |
Nginx is the single public entry: it proxies `/api/*` to the Gateway, rewriting
`/api/langgraph/*` onto the Gateway's native routes, and serves the frontend — see
[backend/AGENTS.md](backend/AGENTS.md) for the runtime and router detail. It compresses
HTML and configured textual assets, deliberately leaving SSE, fonts, images, audio, and
video uncompressed at the proxy layer.
Both compose files publish that entry as `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"`
— **loopback by default**, matching the README's documented deployment model; a bare
`"${PORT}:2026"` binds `0.0.0.0`, which does not. The root `PORT` value is Docker ingress
configuration only; local orchestration pins Next.js to `3000` so loading `.env` cannot
make `make dev` wait on the wrong port. Nginx listening `default_server` on IPv4+IPv6 and
the Gateway binding `0.0.0.0:8001` are container-internal on purpose: the published nginx
port is the entire external surface. Any new published port needs an explicit bind
address; `backend/tests/test_compose_default_bind_host.py` pins this for every service in
both compose files.
## Repository Map
```
deer-flow/
├── Makefile # Root orchestration: drives the full stack (dev/start/stop, docker, setup)
├── config.example.yaml # Template → copy to config.yaml (gitignored) at repo root
├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
├── backend/ # Python backend — see backend/AGENTS.md
│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
│ ├── extensions/sources/ # Deployable snapshots of locally installed Python extensions
│ ├── packages/extension-api/ # deerflow-extension-api package (import: deerflow_extension_api.*) — public extension contract
│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
├── frontend/ # Next.js frontend (pnpm) — see frontend/AGENTS.md
├── docker/ # docker-compose files, nginx config, provisioner
├── skills/ # Agent skills: public/ (committed), custom/ (gitignored)
│ # Managed integration skill packs are global at .deer-flow/integrations/skills/{provider}/
│ # Integration credentials and enabled state remain per-user
├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
├── examples/ # Extension examples: deerflow-extension-{example,bookmarks}
├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
└── docs/ # Cross-cutting docs, plans, and design notes
```
Third-party extensions are loaded from a top-level `plugins:` list in `config.yaml`
(operator-controlled on purpose — that list causes code to be imported, so it is deliberately
kept out of the API-writable `extensions_config.json`). Packaged extensions can contribute
middleware, lifecycle observers, Gateway services, FastAPI HTTP routers, and experimental
full-stack plugins. Manage them with `deerflow extensions install/upgrade/list/enable/disable/remove` or the root
`make extension-*` wrappers. Every mutation requires a Gateway restart, and both build
hooks and extension code execute with Gateway privileges, so only trusted operator sources
belong in this path. The manager transaction, accepted source forms, lock discipline, and
contribution contract live in
[the extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md); the user manual
is `frontend/src/content/{en,zh}/harness/extensions/`.
Runtime config lives at the **repo root**: copy `config.example.yaml` → `config.yaml`
(main app config) and `extensions_config.example.json` → `extensions_config.json` (MCP
servers + skills). Both real files are gitignored and may be edited at runtime via the
Gateway API. Config schema and resolution order are documented in
[backend/AGENTS.md](backend/AGENTS.md).
Skill quality review note:
- `skills/public/skill-reviewer/` is the built-in read-only skill quality reviewer.
It uses the harness-layer `review_skill_package` tool and contracts in
`contracts/skill_review/`. Model-visible review data is compact and
tag-neutralized; full raw payloads stay in tool artifacts. See
[backend/AGENTS.md](backend/AGENTS.md) for the non-activation, SkillScan, and
`skill-creator` ownership boundaries.
- CI waivers live in `.github/skill-review-waivers.v1.json` and are enforced by
`scripts/review_changed_public_skills.py`. Pull requests may validate waiver
edits from their head revision, but only the manifest from the trusted base
revision can suppress that run. Entries match one error finding exactly,
include the reviewed file's SHA-256 and an expiry date, remain visible in CI
output, and can never waive blocker findings. An entry may also preapprove
future full-file SHA-256 values, effective only once the manifest change lands
in the trusted base — so relying on a waiver takes two merges: the manifest
first, the skill change after, then promote the consumed hash to `file_sha256`
in a follow-up cleanup.
Scheduled-task note:
- The scheduled-task MVP adds a workspace page at `/workspace/scheduled-tasks` plus a background scheduler service gated by `config.yaml -> scheduler.enabled`.
- Scheduled background runs are intentionally non-interactive: the lead-agent toolset excludes `ask_clarification` when `context.non_interactive=true`. That key, `disable_clarification`, and `github_token` are honored only for internally-authenticated callers; client-supplied copies are dropped from both `body.context` and `body.config`.
- Busy scheduled occurrences are persisted as `queued`; `launching` is a short lease-fenced claim, `running` remains the normal Gateway run lifecycle, and `scheduler.queue_timeout_seconds` bounds the durable wait. Do not reintroduce skip-on-overlap or count waiting rows against `max_concurrent_runs`.
## Commands: Root vs. Module
**Root `make` targets drive the whole stack** (run from the repo root):
```bash
make setup # Interactive setup wizard (recommended for new users)
make doctor # Check configuration and system requirements
make support-bundle # Generate redacted troubleshooting summary, AI issue draft, and optional zip
make config # Generate local config files from the examples
make check # Check that required tools are installed
make install # Install all dependencies (frontend + backend + pre-commit hooks)
make extension-install SOURCE=... # Install and enable a trusted Python extension
make extension-upgrade SOURCE=... # Replace an installed extension and keep its config
make extension-list # List configured Python extensions
make extension-enable NAME=... # Enable an installed extension (restart required)
make extension-disable NAME=... # Disable without uninstalling (restart required)
make extension-remove NAME=... # Remove package and config entry (restart required)
make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
make start # Start all services in production mode (local, optimized); SKIP_FRONTEND_BUILD=1 reuses the last frontend build
make stop # Stop all running services
make up / down # Build/stop the production Docker stack (browser at localhost:2026)
make docker-start / docker-stop / docker-logs # Docker development environment
```
Production startup uses the image's pre-built Python environment with `uv run
--no-sync`, gives the Gateway a real `/health` probe, and makes `make up` wait
for that probe before printing its success banner. A readiness failure must
surface Compose status and recent Gateway logs instead of claiming the stack is
running.
Docker log and restart commands resolve `DEER_FLOW_ROOT` from the current
checkout before invoking Compose, matching the start and stop commands.
Run `make help` for the full list.
**Per-module commands drive a single module** (run inside that module):
```bash
# Backend (see backend/AGENTS.md for the full set)
cd backend && make dev # Gateway API with reload (port 8001)
cd backend && make test # Default backend suite; excludes live and blocking-I/O tests
cd backend && make test-blocking-io # Strict blocking-I/O suite
cd backend && make lint # ruff check
cd backend && make format # ruff format
# Frontend (see frontend/AGENTS.md for the full set)
cd frontend && pnpm dev # Dev server: Webpack by default (override with DEER_FLOW_DEV_BUNDLER=turbo)
cd frontend && pnpm check # Lint + type check (run before committing)
cd frontend && pnpm test # Unit tests
```
Rule of thumb: **root `make` = the full application**; **`backend/Makefile` and `frontend/`
(`pnpm`) = per-module work.**
Host pnpm calls use `scripts/pnpm.py`: native Windows tries `pnpm.cmd` before
`pnpm`; POSIX reverses the order. Its Corepack fallback applies the same ordering
to `corepack.cmd` and `corepack`. The runner operates from `frontend/` so
Corepack honors its pinned package-manager version.
### Prerequisites before `make dev`
`make dev` does **not** generate config files. First-time setup order:
```bash
make config # copy config.example.yaml -> config.yaml and extensions_config.example.json -> extensions_config.json (both gitignored)
make install # install frontend + backend deps and pre-commit hooks
make dev # then start everything
```
Without `config.yaml` present, services fail to boot. `config.yaml` / `extensions_config.json`
may be edited at runtime via the Gateway API but are gitignored, so never commit them.
### Run a single test
```bash
# Backend (pytest); run one file or one test function
cd backend && python -m pytest tests/test_compose_default_bind_host.py -q
cd backend && python -m pytest tests/path/to/test.py::test_func -q
# Frontend (rstest)
cd frontend && pnpm rstest run <pattern> # e.g. pnpm rstest run my-component
```
### Logs
- Docker stack: `make docker-logs` (or `docker compose -f docker/... logs -f <svc>`).
- Local `make dev`: each service logs to its own terminal pane. Frontend dev-server
errors surface in the browser console at `localhost:3000`; backend tracebacks appear
in the Gateway terminal.
## Where to Go Next
- Backend work → **[backend/AGENTS.md](backend/AGENTS.md)**
- Frontend work → **[frontend/AGENTS.md](frontend/AGENTS.md)**
- Setup & install → **[Install.md](Install.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**
- Project overview & usage → **[README.md](README.md)** (translations: `README_zh.md`,
`README_ja.md`, `README_fr.md`, `README_ru.md`)
- Security policy → **[SECURITY.md](SECURITY.md)**
- Changes → **[CHANGELOG.md](CHANGELOG.md)**
- Cutting a release → **[RELEASING.md](RELEASING.md)**
## Cross-Cutting Conventions
These apply repo-wide; module guides own the module-specific detail.
- **Documentation update policy** — keep docs in sync with code: update `README.md` for
user-facing changes and the relevant `AGENTS.md` for development/architecture changes in
the same change set.
- **Test-driven development** — features and bug fixes ship with tests. Backend tests live
in `backend/tests/` (TDD is mandatory there; see [backend/AGENTS.md](backend/AGENTS.md));
frontend tests live in `frontend/tests/`.
- **Format before pushing** — run `make format` (backend) / `pnpm check` (frontend). Backend
CI enforces `ruff format --check`, so formatting must be clean before a push.
- **Skill text encoding** — treat `SKILL.md` and other textual skill resources as UTF-8;
Python utilities that read or write them must pass `encoding="utf-8"` rather than
relying on the platform locale.
- **Version sources must stay in lockstep** — a release version must match identically in
`backend/pyproject.toml`, `frontend/package.json`, and `deploy/helm/deer-flow/Chart.yaml`
(`version` + `appVersion`). Pushing a `v*` git tag triggers CI that runs
`scripts/verify_versions.sh` and **blocks all publishing** if any source drifts. Before
bumping a version, run `scripts/bump_version.sh <ver>` (aligns all four at once) and
`scripts/verify_versions.sh <ver>` to catch drift early. See [RELEASING.md](RELEASING.md).
- **Don't edit `CLAUDE.md`** — it only contains `@AGENTS.md`. All agent guidance changes
belong here in `AGENTS.md`; `CLAUDE.md` is a thin import shim.
Reference in New Issue View Git Blame Copy Permalink