mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-19 02:56:17 +00:00

Otavio Rodrigues Santana 27cb73659d

fix(auth): correct OAuth conflict error message + validate multi-worker Postgres claim with real concurrency benchmark (#5026 )

* fix(auth): correct OAuth uniqueness error and index parity on Postgres

create_user() caught any IntegrityError on commit and always reported it
as a duplicate email. The email pre-check already rules out a real email
collision in the common case, so any IntegrityError reaching that handler
is actually idx_users_oauth_identity firing instead -- confirmed against
both backends: SQLite reports "UNIQUE constraint failed:
users.oauth_provider, users.oauth_id", Postgres reports a
UniqueViolationError naming the same index. The caller saw "Email already
registered" for an OAuth account conflict, which is wrong and would send
API consumers debugging the wrong field.

Distinguish the two cases via a substring check on the driver error text
(both backends name the oauth columns) and raise an accurate message for
each.

Also add postgresql_where to the same index, alongside the existing
sqlite_where. This is not a correctness fix -- verified empirically that
Postgres already enforces the same practical uniqueness without it
(NULL is never equal to NULL in either backends unique index, so real
duplicate (provider, id) pairs are already rejected and NULL/NULL rows
are already unconstrained). postgresql_where makes the index genuinely
partial on Postgres too, matching the stated intent in the surrounding
comment and keeping the index smaller as the common case (plain-password
accounts, both columns NULL) accumulates.

* test(bench): add multi-process SQLite vs Postgres concurrency benchmark

CONFIGURATION.md documents that multi-worker deployments must use Postgres
because "SQLite silently ignores row-level locks", but nothing in the repo
exercised that claim against real separate worker processes -- the existing
checkpoint benchmarks (scripts/benchmark/checkpoint/) measure single-process
read/write latency, and the existing Postgres tests
(test_pg_schema_integration.py, test_multi_worker_postgres_gate.py) cover
schema placement and config validation, not throughput or lock behavior
under concurrent load.

run_concurrency_bench.py spawns N real OS processes (subprocess.Popen, not
asyncio tasks or threads within one process) against the shared users
table, mixing reads (get_user_by_email) and writes (create_user) at a
configurable ratio, and reports throughput, error counts by exception
type, and p50/p95/p99/max latency per run.

Measured locally (2/4/8/16 workers, 100 ops/worker, 70/30 read/write):
SQLite completed all operations with zero errors at every worker count
(busy_timeout absorbs contention rather than raising), but total
throughput stayed flat around 28-34 ops/s regardless of worker count, and
p99 latency grew from ~400ms at 2 workers to ~5.9s at 16, with a 22s max.
Postgres throughput scaled with worker count (41 to 66 ops/s) and p99
stayed under 500ms at every worker count tested. Raw JSON output from
both runs is available on request; exact numbers will vary by machine and
are not asserted in the test suite.

test_bench_concurrency.py unit-tests the pure aggregation logic
(percentile math, error grouping, crashed-worker handling) the same way
test_bench_checkpoint_channels.py does for the existing benchmarks --
fast, no DB required, not the full multi-process sweep in CI.

* fix(auth): inspect the driver exception for OAuth conflict detection

str(exc) embeds the full failed INSERT statement, whose column list
always names oauth_provider/oauth_id, so a substring check on it
misclassified every commit-time IntegrityError on the users table as
an OAuth conflict (reproduced on SQLite: a duplicate primary key with
a different email raised "OAuth account already linked: None/None").

_is_oauth_identity_violation now inspects exc.orig instead: constraint_name
on Postgres, both violated column names present (not a bare "oauth"
substring) on SQLite.

Also ships the alembic revision idx_users_oauth_identity's postgresql_where
predicate needed: 0001_baseline created it as a full index on Postgres,
and ORM metadata changes only affect fresh create_all databases, never an
already-versioned deployment.

Addresses review feedback from willem-bd.

* fix(bench): run the concurrency benchmark in an isolated schema and derive paths from the checkout

--pg-url accepted an arbitrary database URL while the code pinned
postgres_schema="public" and unconditionally ran DELETE FROM users --
against any non-disposable database that permanently destroyed every
auth account. Each run now generates a unique throwaway schema
(bench_<uuid>), points both the seeder and every worker subprocess at
it via postgres_schema, and drops only that schema (DROP SCHEMA ...
CASCADE) once the full worker-count sweep finishes.

Also stopped hard-coding /opt/deer-flow/backend as the checkout path
and .venv/bin/python3 as the interpreter: BACKEND_DIR is now derived
from Path(__file__), and workers are spawned with sys.executable (the
orchestrator's own interpreter) instead, so the documented
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py
command works from any checkout.

Addresses review feedback from willem-bd.

* fix: shorten oauth-index revision id, repin migration-head assertions, fix bench read/write mix

- 0017_users_oauth_identity_partial_pg (36 chars) exceeded
  alembic_version.version_num's VARCHAR(32) limit, which would fail
  stamping/upgrading on both fresh and existing Postgres deployments.
  Renamed to 0017_oauth_identity_pg_partial (30 chars).
- Repinned every test asserting 0016_subagent_batches as the migration
  head (test_persistence_bootstrap[.py|_concurrency.py|_regression.py],
  test_migration_0004/0007/0015) to the new 0017 revision id.
- worker.py's `(i % 100) < int(read_ratio * 100)` assumed n_ops >= 100;
  at the documented default (50 ops/worker, 0.7 read ratio) it produced
  either all-reads or all-writes, never the claimed mixed workload.
  Replaced with read_count()/is_read_op(), which distribute an exact
  round(n_ops * read_ratio) reads evenly across the sequence via modular
  spacing, and added test_bench_worker.py covering the default values
  plus small op counts.

* fix(bench): establish a real physical connection before timing ops

async with sf(): pass entered an empty AsyncSession without checking out
a physical connection -- SQLAlchemy stays lazy until the first statement
executes. That pushed connection-establishment cost onto each worker's
first timed operation instead of conn_time_s, and at 16 workers those 16
cold first-ops (1% of a 1600-op sample) could skew the reported p99.
Execute a real `SELECT 1` before starting the timer instead.

Verified with a real end-to-end run (uv sync + sqlite backend, 2
workers/10 ops, 0 errors) plus the full auth/bench/migration-bootstrap
suites (135 tests) and ruff check/format, all clean.

* fix(bench): synchronize workers before timing, fix percentile off-by-one

Two remaining measurement issues from review:

- run_workers() started the wall clock before spawning any worker, so
  throughput/wall_time absorbed N processes' staggered Python-startup and
  connection-establishment cost, and early workers could run ahead of ones
  still starting. Workers now print READY right before their timed loop
  and block on stdin for a GO signal; the orchestrator waits for every
  READY, then starts the timer and releases all workers together.

- summarize()'s pct() used int(len(latencies) * p) directly as a
  zero-based index -- a one-based-rank-as-index bug that put p95 and p99
  at the same slot (the max) for any 20-or-fewer-sample run, and for the
  documented 100-sample default. Now delegates to
  checkpoint_bench_common.percentile(), the already-correct nearest-rank
  implementation used elsewhere in the same benchmark family, instead of
  a second, broken one.

Verified: 14/14 unit tests pass (2 new pinned-value regression tests for
the percentile bug, using the reviewer's own 20-sample repro), ruff
clean, and a real 2/4-worker SQLite multi-process smoke run completes
with distinct p95/p99/max latencies and no hang.

* fix(bench): absolute SQLite bench path, surface crash diagnostics, exit nonzero on failure; share OAuth index constant + cover Postgres branch

Three more findings from review at 5fd25a7:

- seed_baseline() cleaned an absolute .deer-flow/bench_data path but
  handed DatabaseConfig a relative one, which resolves against the
  CALLER's CWD -- not BACKEND_DIR. Invoking the documented command from
  anywhere other than backend/ silently pointed the seeder and the
  (cwd=BACKEND_DIR) workers at two different directories: workers crashed
  with 'unable to open database file' while the run still printed a
  well-formed summary and exited 0. Both seed_baseline() and worker.py's
  make_session_factory() now use the same absolute path.

- Crashed workers' stderr was captured then discarded, and main() always
  exited 0 -- an all-crashed sweep was indistinguishable from a real
  (uneventful) measurement to anything checking the exit code or
  --out. run_workers() now prints each crash immediately and tags it with
  the real worker_id (previously always None); summarize() exposes
  crashed_worker_errors alongside the existing crashed_workers count;
  main() exits 1 via the new summary_indicates_failure() whenever any
  sweep crashed or fell short of expected_total_ops.

- idx_users_oauth_identity was hardcoded separately in the ORM Index and
  in _is_oauth_identity_violation's Postgres branch, with no test to
  catch drift, and that branch had zero non-skipped coverage (its only
  guard needs a live Postgres CI never configures). Exported
  OAUTH_IDENTITY_INDEX_NAME from user/model.py as the shared source of
  truth (migrations intentionally keep their own frozen literal, matching
  every other revision in that package) and added stub-exception unit
  tests pinning both the asyncpg constraint_name path and the sqlite
  message-substring path, positive and negative.

Verified: 107 passed locally (auth + bench-unit suites), ruff clean, and
two real reproductions -- invoking run_concurrency_bench.py from a
scratch directory outside backend/ (the reviewer's exact repro) now
completes 8/8 ops with crashed_workers: 0 instead of crashing, and the
new crashed_worker_errors/exit-code logic is exercised directly by the
new unit tests against the real summarize()/summary_indicates_failure().

* fix(auth): attribute create_user IntegrityErrors to the right constraint

Two coupled review findings on the classification helpers:

P3 (fall-through) -- after ruling out the OAuth-identity index, create_user
raised "Email already registered: {email}" for every remaining
IntegrityError, including the duplicate-primary-key case the new
regression test exercises, whose address is not registered. Added
_is_email_violation() so the email message is used only for an actual
users.email collision that raced past the pre-check; anything else (in
practice a duplicate id) now raises a neutral
"User already exists (constraint: <name>)".

P2 (unreachable asyncpg branch) -- exc.orig is not the asyncpg error.
SQLAlchemy's asyncpg dialect re-raises its own DBAPI IntegrityError
(pgcode/sqlstate only) 'from' the real asyncpg error, so constraint_name
lives on exc.orig.__cause__. getattr(exc.orig, "constraint_name", None)
was always None on Postgres; the helpers only worked there by accident,
matching asyncpg's DETAIL line in the message fallback. Added
_driver_constraint_name() which walks orig then orig.__cause__, and the
stub tests now model that real shape (orig wrapper + __cause__) instead of
a constraint_name that no driver puts on orig directly.

Tests: 76 passed. New coverage for the email-race path, both new helpers
on each backend, the neutral fallback message, and the cause-chain walk.

* fix(bench): match app SQLite PRAGMAs in workers; fail a sweep on any op error

Two review follow-ups:

- worker.py opened its SQLite engine with only connect_args timeout=30.
  synchronous and foreign_keys are per-connection PRAGMAs, so workers ran
  at SQLite's synchronous=FULL / foreign_keys=OFF while a real Gateway
  worker runs synchronous=NORMAL (persistence/engine.py::_enable_sqlite_wal)
  -- an extra fsync per commit on the measured 30%-write path, overstating
  SQLite's cost in the direction that flatters the "use Postgres"
  conclusion. Added a connect listener applying the same four PRAGMAs, with
  a test asserting synchronous/foreign_keys/journal_mode on a real worker
  connection.

- summary_indicates_failure() only looked at crashes and the completed vs
  expected op counts, so a sweep where every op completed but raised
  (e.g. writes hitting OperationalError) passed as a clean measurement:
  completed_ops == expected, 0 crashes. Added an "errors > 0" clause; the
  error breakdown stays in the JSON, only the exit code changes. Test added.

test_bench_concurrency.py + test_bench_worker.py green (20), plus a real
2-worker sqlite smoke run (6/6 ops, 0 errors, exit 0).

* fix(auth): match the real email index name; only claim "exists" for uniqueness

Review follow-ups on the classification helpers:

- email is mapped_column(unique=True, index=True), which SQLAlchemy and
  0001_baseline realise as a single UNIQUE INDEX (ix_users_email), not a
  named UNIQUE constraint. _is_email_violation compared the driver
  constraint name against "users_email_key", which Postgres never emits,
  so that arm was dead on Postgres (SQLite matched via the message). Fixed
  to ix_users_email.

- the residual IntegrityError fallback raised "User already exists" for
  every remaining IntegrityError -- a NOT NULL / CHECK / foreign-key
  violation is not a "user already exists" condition and is not part of
  create_user's ValueError contract. Added _is_uniqueness_violation
  (sqlstate 23505, or the SQLite "UNIQUE/PRIMARY KEY constraint failed"
  message); only that raises the "already exists" ValueError, everything
  else propagates unchanged.

- documented scripts/benchmark/concurrency/ in backend/AGENTS.md alongside
  the other benchmark family.

Tests: 78 auth + 20 bench-unit pass, ruff clean. New coverage for
_is_uniqueness_violation on both backends and for a non-uniqueness
IntegrityError propagating out of create_user.

* fix(bench): don't pre-close worker stdin (breaks communicate); require --pg-url for postgres

* fix(bench): ruff format; time throughput on the op phase, not teardown

- lint-backend: ruff format the files touched in this PR.
- Throughput window (P2): the orchestrator sampled its wall clock after
  every worker's communicate() returned, so it also covered each worker's
  engine.dispose(), result serialization and stdout transfer. Each worker
  now times just its operation phase (GO -> last op) and reports
  ops_elapsed_s; summarize() uses max(ops_elapsed_s) over the workers -- all
  released by the same GO -- as the throughput window (ops_window_s).
- Exercise migration 0018 (P2): test_user_oauth_partial_index.py goes
  through bootstrap create_all(), which builds the partial index from ORM
  metadata and never runs 0018.upgrade(). New Postgres-gated
  test_migration_0018_oauth_identity_pg_partial.py alembic-upgrades to 0017
  (full index), then 0018 (asserts the predicate appears), then downgrades
  (asserts the full index is restored) and re-upgrades.

* docs(middlewares): tighten SandboxAudit and Clarification entries in AGENTS.md

PR #5134 grew agents/middlewares/AGENTS.md ~1.8 KB, pushing the effective
AGENTS.md chain for that directory over the 96 KiB hard limit once this
branch also documents scripts/benchmark/concurrency/ in backend/AGENTS.md.
Condense the two longest middleware entries (SandboxAuditMiddleware,
ClarificationMiddleware) without dropping any identifier, example, issue
reference, ordering constraint, or documented gap; chain back to ~96.8 KiB.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

2026-09-03 08:17:26 +08:00

27 KiB

Raw Blame History

AGENTS.md

Project Overview

DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.

Architecture:

Gateway API (port 8001): REST API plus embedded LangGraph-compatible agent runtime
Frontend (port 3000): Next.js web interface
Nginx (port 2026): Unified reverse proxy entry point
Provisioner (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode

Runtime:

make dev, Docker dev, and production all run the agent runtime in Gateway via RunManager + run_agent() + StreamBridge (packages/harness/deerflow/runtime/). Nginx exposes that runtime at /api/langgraph/* and rewrites it to Gateway's native /api/* routers.
Gateway streams write_file and str_replace argument deltas in bounded batches when clients also subscribe to values; messages-only consumers retain the original per-chunk contract, while values preserves the complete tool call.
With stream_subgraphs, subgraph frames keep their namespace in the SSE event name (values|<ns>, LangGraph Platform style) instead of impersonating root frames — a delegated subagent inherits the parent checkpoint namespace, so publishing its values snapshot as bare values replaces the whole thread view in SDK clients (#4399). Root-only consumers (file-tool chunk batcher, subagent event persistence, LLM error-fallback detection) ignore namespaced frames. The web frontend does not request subgraph streaming; subtask progress rides root-namespace task_* custom events.
Background subagent identity is deliberately split: the provider tool_call_id remains the correlation key for ToolMessage, task_* SSE events, persisted lifecycle events, frontend cards, and the public ExtensionData.scope_id contract (stored as SubagentResult.external_task_id), while SubagentExecutor.execute_async() generates a full server-side execution_id for SubagentResult.task_id, the process-wide registry, polling, cancellation, timeout handling, and cleanup. Provider IDs are not globally unique across parent runs, so they must never become registry ownership keys; scheduler closures retain their own SubagentResult rather than resolving ownership again through the mutable registry. Terminal subagent token usage travels in the current run's ToolMessage.additional_kwargs and is attributed from message state, never through a process-global provider-ID cache.
Scheduled-task executions must reuse that same Gateway run lifecycle. The scheduler may decide when work runs, but it must dispatch through the existing run path rather than introducing a parallel execution stack. Scheduled launches pass scheduler.recursion_limit (default 1000, matching the web UI's recursion_limit: 1000, clamped by max_recursion_limit) via launch_scheduled_thread_run; the value is read from get_app_config() at dispatch.
The background scheduler is single-instance by default. scheduler.multi_instance=true opts into lease-aware recovery across Gateway instances and requires shared Postgres, run_ownership.heartbeat_enabled=true, and run_events.backend=db; otherwise startup rejects the configuration. Live scheduled runs are preserved when a peer starts; expired launch claims return to the durable queue, expired run leases are atomically taken over, stale launch writes are fenced by lease ownership, and the Postgres advisory-locked budget makes max_concurrent_runs a shared global cap for launching/running rows.
Long-running MCP work uses a separate durable task runtime (McpTaskService + mcp_tasks, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and ThreadState receives only a bounded current-thread projection. Full contract (leases, cancellation fencing, delivery idempotency, management-tool exposure): packages/harness/deerflow/mcp/AGENTS.md.
MCP task notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 are part of that same contract — see packages/harness/deerflow/mcp/AGENTS.md.
Scheduled-task dispatch enforces at most one non-terminal occurrence per task through uq_scheduled_task_run_active (task_id WHERE status IN ('queued','launching','running')). queued is durable and survives restart; launching carries a short owner/expiry lease and is the only state that may call the normal Gateway launch path; running is associated with the durable run. Each occurrence also supplies a stable run-admission idempotency key, so a recovered launch retry reuses the same durable run. A reused-thread ConflictError moves launching back to queued, while non-conflict launch errors become terminal failed. Waiting rows do not consume max_concurrent_runs; the atomic queue claim enforces the budget. Repeated triggers coalesce on the one active row, and same-thread FIFO treats older queued, launching, and running rows as blockers. The task definition stays immutable for all three active states because queue admission, PATCH/resume, pause, and delete serialize on the parent task row before touching the occurrence row. Pause/delete atomically interrupt existing queued rows and reject launching/running rows; PATCH/resume reject every active state, and mutation errors advertise pause cancellation only for queued work. A manual trigger may queue and run while the parent schedule remains paused. Recovery and multi-instance reconciliation lock task/run pairs in deterministic task-id/run-id order and must reconstruct run_id, started_at, and the live error state before releasing the short launch claim. Launch/failure/timeout bookkeeping changes the occurrence and its parent task in one parent-first transaction so a peer cannot claim the released task between those writes. Queue timeout marks the occurrence failed and advances a scheduled occurrence so it cannot immediately requeue forever; repository write boundaries coerce serialized task timestamps before binding SQL DateTime fields.
extensions_config.json is written at runtime by the Gateway (PUT/PATCH /api/mcp/config, the MCP enable switch, skill updates), so the production compose mounts it read-write while config.yaml stays :ro; Helm copies its ConfigMap seed into a writable home-volume directory before Gateway starts. Every read-modify-write holds both extensions_config_write_lock and the sidecar advisory extensions_config_file_lock, because the process-local lock alone loses updates across workers. Docker mounts the compose file as its own mount point, and Linux refuses rename() over a mount point with EBUSY even when the mount is writable — so atomic_write_extensions_config keeps the temp-file-plus-rename path and falls back to an in-place overwrite only on EBUSY. That fallback is deliberately non-atomic (a crash mid-write truncates the file); it exists because the alternative is a write that can never succeed, and only its first occurrence per target is logged at warning level. Any other errno still propagates. Pinned by tests/test_compose_extensions_config_writable.py, tests/test_extensions_config_atomic_write.py, and tests/test_helm_extensions_config_writable.py.

Project Structure:

deer-flow/
├── Makefile                    # Root commands (check, install, dev, stop)
├── config.yaml                 # Main application configuration
├── extensions_config.json      # MCP servers and skills configuration
├── backend/                    # Backend application (this directory)
│   ├── Makefile               # Backend-only commands (dev, gateway, lint)
│   ├── langgraph.json         # LangGraph Studio graph configuration
│   ├── packages/
│   │   ├── extension-api/     # public, host-independent extension contracts (import: deerflow_extension_api.*)
│   │   └── harness/           # deerflow-harness package (import: deerflow.*)
│   │       ├── pyproject.toml
│   │       └── deerflow/
│   │           ├── agents/            # LangGraph agent system
│   │           │   ├── lead_agent/    # Main agent (factory + system prompt)
│   │           │   ├── middlewares/   # middleware components (see Middleware Chain section)
│   │           │   ├── memory/        # Memory extraction, queue, prompts
│   │           │   └── thread_state.py # ThreadState schema
│   │           ├── sandbox/           # Sandbox execution system
│   │           │   ├── local/         # Local filesystem provider
│   │           │   ├── sandbox.py     # Abstract Sandbox interface
│   │           │   ├── tools.py       # bash, ls, read/write/str_replace
│   │           │   └── middleware.py  # Sandbox lifecycle management
│   │           ├── subagents/         # Subagent delegation system
│   │           │   ├── builtins/      # general-purpose, bash agents
│   │           │   ├── executor.py    # Background execution engine
│   │           │   └── registry.py    # Agent registry
│   │           ├── tools/builtins/    # Built-in tools (present_files, ask_clarification, view_image, review_skill_package)
│   │           ├── mcp/               # MCP integration (tools, cache, client)
│   │           ├── integrations/      # Managed first-party integration installers (e.g. Lark CLI skill pack)
│   │           ├── extensions/        # Python plugin loader, registry, placement, and isolation
│   │           ├── models/            # Model factory with thinking/vision support
│   │           ├── skills/            # Skills discovery, loading, parsing
│   │           ├── config/            # Configuration system (app, model, sandbox, tool, etc.)
│   │           ├── community/         # Community tools (search/fetch/scrape, image search, AIO sandbox)
│   │           ├── reflection/        # Dynamic module loading (resolve_variable, resolve_class)
│   │           ├── utils/             # Utilities (network, readability)
│   │           └── client.py          # Embedded Python client (DeerFlowClient)
│   ├── app/                   # Application layer (import: app.*)
│   │   ├── gateway/           # FastAPI Gateway API
│   │   │   ├── app.py         # FastAPI application
│   │   │   └── routers/       # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
│   │   └── channels/          # IM platform integrations
│   ├── scripts/benchmark/       # Standalone reproducible backend benchmarks
│   ├── tests/                 # Test suite
│   └── docs/                  # Documentation
├── frontend/                   # Next.js frontend application
└── skills/                     # Agent skills directory
    ├── public/                # Public skills (committed)
    └── custom/                # Custom skills (gitignored)

Important Development Guidelines

Documentation Update Policy

CRITICAL: Always update README.md and AGENTS.md after every code change

When making code changes, you MUST update the relevant documentation:

Update README.md for user-facing changes (features, setup, usage instructions)
Update AGENTS.md for development changes (architecture, commands, workflows, internal systems). CLAUDE.md imports it via @AGENTS.md, so editing AGENTS.md updates both.
Keep documentation synchronized with the codebase at all times
Ensure accuracy and timeliness of all documentation

Backend Benchmarks

scripts/benchmark/ contains standalone, reproducible measurements and evaluations of production backend behavior. A benchmark may import the production function it measures, but it must not duplicate or introduce an alternative runtime implementation.

Pin every external dataset by immutable revision and SHA-256. Callers provide the local dataset path; evaluation commands must not silently download data.
Never commit upstream dataset text, credentials, complete provider requests, or response headers. Committed manifests may contain stable IDs and source locators. Synthetic cases must identify themselves as synthetic.
Read provider credentials and endpoints from named environment variables. Version model IDs, inference parameters, prompts, retry rules, clocks, and random seeds in the evaluation config.
Public raw results may contain case IDs, policy decisions, model hypotheses, grades, and non-secret response metadata. Keep dataset questions, reference answers, memory content, and full provider payloads in ignored local run directories.
Use fixed clocks and deterministic ordering for offline selection. Results must record the config, manifest, prompt, dataset, and git revisions used.

scripts/benchmark/deermem_eviction/ evaluates the production select_facts_for_capacity() implementation used by DeerMem. It compares only the historical confidence policy and PR #4789's opt-in hybrid-v1; do not add another eviction strategy to this evaluation. Run its offline checks from backend/:

PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate-contracts
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate --dataset "$LONGMEMEVAL_ORACLE_PATH"
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction run-policy \
  --dataset "$LONGMEMEVAL_ORACLE_PATH" \
  --output-dir /tmp/deermem-eviction-policy-run
PYTHONPATH=. uv run pytest tests/test_bench_deermem_eviction_*.py -q

The offline test suite must not require network access, provider credentials, or the LongMemEval dataset. Small LongMemEval-shaped fixtures must be synthetic and generated by tests.

scripts/benchmark/concurrency/ measures multi-process contention on the users table (N separate OS processes, not asyncio tasks) for SQLite vs Postgres -- the scenario CONFIGURATION.md requires Postgres for. worker.py connects directly via SQLAlchemy (skipping the ~8.5s Alembic bootstrap the orchestrator already ran once) and mirrors the app's per-connection SQLite PRAGMAs; run_concurrency_bench.py seeds a disposable per-run Postgres schema, synchronises workers on a READY/GO barrier before timing, and exits non-zero on any crash, short op count, or errors > 0. Postgres runs need a throwaway database via --pg-url; nothing here touches public. Run from backend/:

uv run python scripts/benchmark/concurrency/run_concurrency_bench.py \
  --backend sqlite --workers 2,4,8,16 --ops-per-worker 50 --read-ratio 0.7
uv run pytest tests/test_bench_concurrency.py tests/test_bench_worker.py -q

Commands

Root directory (for full application):

make check      # Check system requirements
make install    # Install all dependencies (frontend + backend)
make extension-install SOURCE=...  # Install and enable a trusted Python extension
make extension-list                # List configured Python extensions
make extension-enable NAME=...     # Enable an installed extension
make extension-disable NAME=...    # Disable an extension without uninstalling it
make extension-remove NAME=...     # Remove a managed extension
make detect-thread-boundaries  # Inventory backend executor/thread/event-loop boundaries
make dev        # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
make start      # Start production services locally
make stop       # Stop all services

Backend directory (for backend development only):

make install            # Install backend dependencies
make dev                # Gateway API, reload (port 8001)
make gateway            # Gateway API only (port 8001)
make test               # offline tests (no live/blocking-io)
make test-live          # live tests (real APIs)
make test-blocking-io   # strict Blockbuster gate on tests/blocking_io/
make test-shard SPLITS=4 GROUP=2  # one duration-aware shard
make test-shard-durations  # refresh baseline
make lint               # ruff lint
make format             # ruff format
make migrate-rev MSG="..."  # Autogenerate a new alembic revision (see Schema Migrations section)

The backend make dev target pre-creates and excludes DEER_FLOW_HOME (default: backend/.deer-flow) and backend/sandbox from Uvicorn's reload watcher. Do not replace it with a bare uvicorn --reload: agent tasks write Python and other runtime files below DEER_FLOW_HOME, which would otherwise restart the Gateway during an active run.

More specific AGENTS.md files in backend code directories contain the subsystem sections split from this file. Follow the nearest file in the directory tree.

Architecture

Harness / App Split

The backend is split into two layers with a strict dependency direction:

Harness (packages/harness/deerflow/): Publishable agent framework package (deerflow-harness). Import prefix: deerflow.*. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
App (app/): Unpublished application code. Import prefix: app.*. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).

Dependency rule: App imports deerflow, but deerflow never imports app. This boundary is enforced by tests/test_harness_boundary.py which runs in CI.

Import conventions:

# Harness internal
from deerflow.agents import make_lead_agent
from deerflow.models import create_chat_model

# App internal
from app.gateway.app import app
from app.channels.service import start_channel_service

# App → Harness (allowed)
from deerflow.config import get_app_config

# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
# from app.gateway.routers.uploads import ...  # ← will fail CI

Package import hygiene: the deerflow.agents and deerflow.subagents package roots expose heavyweight graph/executor entrypoints lazily. The deerflow.agents:make_lead_agent LangGraph Server entrypoint is a concrete thin module-level function because the server resolves graph factories directly from the module dictionary; the wrapper keeps the lead-agent and skill-cache imports inside the function so importing the package remains lightweight. Internal modules that only need lightweight types, config, or registries should import the concrete submodule instead of adding eager package-root imports that pull in the tool graph or subagent executor during state/schema imports.

ThreadMetaStore.search() keeps JSON filter semantics identical across memory, SQLite, and PostgreSQL: missing differs from null, bool differs from int, and float filters accept integer or real JSON numbers through json_value_matches.

Development Workflow

Test-Driven Development (TDD) — MANDATORY

Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.

Write tests in backend/tests/ following the existing naming convention test_<feature>.py
Run both offline targets before and after your change: make test and make test-blocking-io
Tests must pass before a feature is considered complete
For lightweight config/utility modules, prefer pure unit tests with no external dependencies
If a module causes circular import issues in tests, add a sys.modules mock in tests/conftest.py (see existing example for deerflow.subagents.executor)

# Run default offline tests
make test

# Run strict blocking-I/O tests
make test-blocking-io

# Explicit live integration tests (requires config.yaml and credentials;
# calls real APIs and may create local side effects)
make test-live

# Run a specific test file
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v

Direct pytest collection or execution of tests/test_client_live.py remains skipped unless DEER_FLOW_RUN_LIVE_TESTS=1 is set. Do not add that opt-in to default CI workflows.

Jina request-failure logging tests set a dummy API key so the separate once-per-process missing-key warning cannot make assertions depend on test order or shard placement. Missing-key behavior has its own tests in tests/test_jina_client.py.

Running the Full Application

From the project root directory:

make dev

This starts all services and makes the application available at http://localhost:2026.

All startup modes:

	Local Foreground	Local Daemon	Docker Dev	Docker Prod
Dev	`./scripts/serve.sh --dev` `make dev`	`./scripts/serve.sh --dev --daemon` `make dev-daemon`	`./scripts/docker.sh start` `make docker-start`	—
Prod	`./scripts/serve.sh --prod` `make start`	`./scripts/serve.sh --prod --daemon` `make start-daemon`	—	`./scripts/deploy.sh` `make up`

Action	Local	Docker Dev	Docker Prod
Stop	`./scripts/serve.sh --stop` `make stop`	`./scripts/docker.sh stop` `make docker-stop`	`./scripts/deploy.sh down` `make down`
Restart	`./scripts/serve.sh --restart [flags]`	`./scripts/docker.sh restart`	—

Nginx routing:

/api/langgraph/* → Gateway embedded runtime (8001), rewritten to /api/*
/api/* (other) → Gateway API (8001)
/ (non-API) → Frontend (3000)

Running Backend Services Separately

From the backend directory:

# Gateway API
make gateway

Direct access (without nginx):

Gateway: http://localhost:8001

Frontend Configuration

The frontend uses environment variables to connect to backend services:

NEXT_PUBLIC_LANGGRAPH_BASE_URL - Defaults to /api/langgraph (through nginx)
NEXT_PUBLIC_BACKEND_BASE_URL - Defaults to empty string (through nginx)

When using make dev from root, the frontend automatically connects through nginx.

Key Features

Web Search Recency

DDG, Brave, Tavily, and SearXNG web_search share optional time_range=day|week|month|year; omission preserves request shape. DDG maps to d|w|m|y, Brave to pd|pw|pm|py, and Tavily/SearXNG pass values unchanged. For recency, DDGS 9.14.1 uses only enabled Brave, DuckDuckGo, and Yahoo engines that honor timelimit: auto/all resolves to this set, incompatible configured engines are removed, and an empty set falls back to it. Re-check on DDGS upgrades.

File Upload

Multi-file upload with automatic document conversion:

Endpoint: POST /api/threads/{thread_id}/uploads
Supports: PDF, PPT, Excel, Word documents (converted via markitdown)
Rejects directory inputs before copying so uploads stay all-or-nothing
Reuses one conversion worker per request when called from an active event loop
Files stored in thread-isolated directories under the resolving user's bucket (users/{user_id}/threads/{thread_id}/user-data/uploads). For IM channels the owner is threaded explicitly via the user_id= kwarg (see IM Channels → Owner-scoped file storage); HTTP/embedded callers resolve it from get_effective_user_id()
Duplicate filenames in a single upload request are auto-renamed with _N suffixes so later files do not truncate earlier files
Gateway HTTP uploads stage bytes as .upload-*.part files and atomically replace the destination only after size validation. These staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools, and swept on Gateway startup if a hard crash leaves one behind.
Gateway HTTP upload/list/delete handlers offload filesystem work through deerflow.utils.file_io.run_file_io, a dedicated ContextVar-preserving file IO executor. Non-mounted sandbox uploads acquire sandboxes with SandboxProvider.acquire_async() and offload read_bytes() plus sandbox.update_file() together.
Mounted upload paths skip both sandbox acquisition and per-file synchronization. For AIO remote/provisioner deployments this requires an explicit, accurate sandbox.thread_data_mounts: true; omission preserves backend auto-detection.
Agent receives uploaded file list via UploadsMiddleware; title generation continues to use the original user request rather than the injected upload-context wrapper, with attachment-only messages falling back to New Conversation

See docs/FILE_UPLOAD.md for details.

Plan Mode

TodoList middleware for complex multi-step tasks:

Controlled via runtime config: config.configurable.is_plan_mode = True
Provides write_todos tool for task tracking
One task in_progress at a time, real-time updates

See docs/plan_mode_usage.md for details.

Context Summarization

Automatic conversation summarization when approaching token limits:

Configured in config.yaml under summarization key
Trigger types: tokens, messages, or fraction of max input
Keeps recent messages while summarizing older ones
Manual compaction uses POST /api/threads/{id}/compact, reuses the same DeerFlowSummarizationMiddleware, writes a new checkpoint with updated messages and summary_text, and bumps only those channel versions. The route uses the shared reserve_checkpoint_write() boundary (also used by manual state updates). Its short-lived checkpoint_write thread operation shares the durable active-thread uniqueness constraint with run admission, preventing either worker-local or cross-worker checkpoint-write races.

See docs/summarization.md for details.

Vision Support

For models with supports_vision: true:

ViewImageMiddleware processes images in conversation
view_image_tool added to agent's toolset
Images are converted to base64 and appended to the model request as a hidden message carrying both a reserved ID prefix and a server-owned metadata marker; Gateway strips that marker from untrusted input, and the middleware requires both identifiers to recognize its own message. The middleware injects inside wrap_model_call, so the payload never enters graph state: checkpoints retain only lightweight viewed_images metadata, while client-chosen IDs survive. It also sweeps its own message out of every request before rebuilding it, so a payload stranded in an older checkpoint by an interrupted run stops being resent

Code Style

Uses ruff for linting and formatting
Line length: 240 characters
Python 3.12+ with type hints
Double quotes, space indentation

Documentation

See docs/ directory for detailed documentation:

CONFIGURATION.md - Configuration options
ARCHITECTURE.md - Architecture details
API.md - API reference
SETUP.md - Setup guide
FILE_UPLOAD.md - File upload feature
PATH_EXAMPLES.md - Path types and usage
summarization.md - Context summarization
plan_mode_usage.md - Plan mode with TodoList

27 KiB Raw Blame History