mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-08-01 19:06:01 +00:00
docs(schedule): keep the module notes current with main
Rebases the schedule section onto main's newer AGENTS.md so the test-live and thread-boundary entries added meanwhile are not lost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
rayhpeng
parent
b936da4931
commit
849692aeac
@ -87,6 +87,7 @@ When making code changes, you MUST update the relevant documentation:
|
||||
```bash
|
||||
make check # Check system requirements
|
||||
make install # Install all dependencies (frontend + backend)
|
||||
make detect-thread-boundaries # Inventory backend executor/thread/event-loop boundaries
|
||||
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make start # Start production services locally
|
||||
make stop # Stop all services
|
||||
@ -97,13 +98,46 @@ make stop # Stop all services
|
||||
make install # Install backend dependencies
|
||||
make dev # Run Gateway API with reload (port 8001)
|
||||
make gateway # Run Gateway API only (port 8001)
|
||||
make test # Run all backend tests
|
||||
make test # Run offline backend tests (excludes live external-API tests)
|
||||
make test-live # Explicitly run live DeerFlowClient tests with real APIs
|
||||
make test-blocking-io # Run strict Blockbuster runtime gate on tests/blocking_io/
|
||||
make lint # Lint with ruff
|
||||
make format # Format code with ruff
|
||||
make migrate-rev MSG="..." # Autogenerate a new alembic revision (see Schema Migrations section)
|
||||
```
|
||||
|
||||
The root `detect-thread-boundaries` target statically inventories execution
|
||||
boundaries under `backend/app/` and `backend/packages/harness/deerflow/`. It
|
||||
prints a concise count by execution domain and writes the complete, versioned
|
||||
JSON payload to `.deer-flow/thread-boundary-inventory.json`. Every finding has
|
||||
a stable `boundary_kind`: `asyncio_default_executor`, `dedicated_executor`,
|
||||
`anyio_worker_thread`, `direct_event_loop_blocking`, `separate_event_loop`, or
|
||||
`unresolved_dynamic_boundary`.
|
||||
|
||||
The AST inventory covers `asyncio.to_thread`, default and explicit
|
||||
`run_in_executor` submissions, imported aliases, simple same-module helper
|
||||
wrappers (after pre-registering dedicated executor targets), `set_default_executor`,
|
||||
`ThreadPoolExecutor` construction/submission,
|
||||
additional event loops, synchronous LangChain tools, and direct
|
||||
`BaseChatModel` fallback inheritance. It remains read-only and does not alter
|
||||
executor routing or sizing.
|
||||
|
||||
To supplement the static scan with configured runtime types, run:
|
||||
|
||||
```bash
|
||||
python scripts/detect_thread_boundaries.py \
|
||||
--runtime-config config.yaml \
|
||||
--json-output .deer-flow/thread-boundary-inventory.json
|
||||
```
|
||||
|
||||
Runtime inspection imports configured tool objects and model classes so it can
|
||||
record concrete tool names/types/modules, sync functions, async coroutines,
|
||||
and `_agenerate`/`_astream` ownership. It does not invoke tools, instantiate
|
||||
models, or call external services; import failures remain in the JSON as
|
||||
`unresolved_dynamic_boundary` records. The detector implementation and focused
|
||||
coverage live in `tests/support/detectors/thread_boundaries.py` and
|
||||
`tests/test_detect_thread_boundaries.py`.
|
||||
|
||||
The `detect-blocking-io` target parses `app/`, `packages/harness/deerflow/`,
|
||||
and `scripts/` with AST. By default it reports only blocking IO candidates that
|
||||
are inside async code, reachable from async code in the same file, or reachable
|
||||
@ -452,8 +486,8 @@ Localhost persistence deliberately reads the direct request `Host` and ignores `
|
||||
|--------|-----------|
|
||||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||
| **Features** (`/api/features`) | `GET /` - report config-gated feature availability (`agents_api.enabled`, `browser_control.enabled`) for frontend UI gating |
|
||||
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
|
||||
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). All priced models must use one currency; mixed currencies disable cost reporting and leave cost/currency fields null instead of producing invalid aggregates. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - replace the full config with whole-payload stdio validation; `PATCH /config` - toggle one server while preserving the raw extensions config and validating only an enabled target; both writes reload config and reset the process-local MCP cache |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes |
|
||||
| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_ready` (whether `lark-cli` will actually be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
@ -650,6 +684,7 @@ that cannot tell sibling branches apart.
|
||||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||||
|
||||
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
|
||||
**Benefit-based routing policy**: Enabling subagents exposes delegation as an optimization, not a default response to complexity. The lead prompt defaults to direct execution and permits `task` only when parallel latency, specialist capability, or context-isolation benefit clearly exceeds startup, duplicate-discovery, synthesis, state-conflict, and side-effect costs. Inter-agent output dependencies and overlapping mutable state are hard vetoes for parallel dispatch, while duplicate discovery and a cheap direct path remain costs rather than categorical vetoes; a bounded sequential chain may run in one subagent when specialist or context-isolation benefit clearly wins. Parallel scopes must be independent and non-overlapping, the lead uses the fewest useful subagents, and every later batch is re-evaluated while retaining any within-batch parallel benefit. When the enforced per-response limit is 1, the rendered prompt removes parallel and multi-batch benefit guidance and permits delegation only for material specialist or context-isolation benefit. Keep this policy aligned across `lead_agent/prompt.py`, the `task` tool description, and both built-in role descriptions; routing regressions are pinned in `tests/test_subagent_routing_prompt.py`, `tests/test_subagent_prompt_security.py`, and `tests/test_lead_agent_prompt.py`.
|
||||
**User-scoped Skills**: Subagents resolve their configured skills through `get_or_new_user_skill_storage(user_id)` using the parent runtime identity, with `DEFAULT_USER_ID` only when no identity is available. This keeps custom-skill shadowing and visibility aligned with the lead agent instead of reading the global-only catalog.
|
||||
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
|
||||
**Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 1-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box)
|
||||
@ -718,7 +753,7 @@ E2B output sync records remote file versions and actual host file metadata in a
|
||||
add a parallel routing middleware for PR1-style preference hints.
|
||||
- **Stdio file outputs**: Persistent stdio sessions are scoped by `user_id:thread_id`. For stdio transports only, DeerFlow pins the subprocess default `cwd` to the thread workspace and `TMPDIR`/`TMP`/`TEMP` to `workspace/.mcp/tmp/`, unless the operator explicitly configured `cwd` or temp env values. SSE/HTTP transports skip this filesystem prep entirely.
|
||||
- **Stdio path translation**: MCP-returned local file references are not copied. If a `ResourceLink` or conservative free-text path resolves to an existing file inside the thread's mounted user-data tree, it is translated deterministically to `/mnt/user-data/...`; paths outside that tree remain unchanged.
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (the `PUT /api/mcp/config` reset only clears the cache in its own worker)
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (`PUT /api/mcp/config` keeps whole-payload validation, while `PATCH /api/mcp/config` changes only one server's `enabled` field, normalizes the same `type`/MCP-spec `transport` alias as the runtime config model, and validates the target only when enabling it; either endpoint's reset clears the cache only in its own worker). MCP, skill, and embedded-client writers share `atomic_write_extensions_config()`, which writes and fsyncs a same-directory temporary file before `os.replace()` and preserves an existing file's mode and symlink target; failed serialization or replacement leaves the prior config intact and cleans up the temporary file.
|
||||
|
||||
### Skills System (`packages/harness/deerflow/skills/`)
|
||||
|
||||
@ -1185,7 +1220,7 @@ Config is env-driven like the others — `MonocleTracingConfig`, built in `get_t
|
||||
- `skills` - Map of skill name → state (enabled)
|
||||
- `middlewares` - Zero-argument `AgentMiddleware` class paths for lead and subagent runtime extension. `config.yaml -> extensions` can override these fields after validation; overrides are replace-per-field, not list concatenation.
|
||||
|
||||
Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and skill state at runtime; `middlewares` remains an operator-controlled config-file extension point.
|
||||
Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and skill state at runtime; their `extensions_config.json` writes use the shared atomic replacement helper, while `middlewares` remains an operator-controlled config-file extension point.
|
||||
|
||||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||||
|
||||
@ -1220,7 +1255,13 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk
|
||||
|
||||
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
|
||||
|
||||
**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml)
|
||||
**Tests**: `tests/test_client.py` (offline unit tests including
|
||||
`TestGatewayConformance`), `tests/test_client_live.py` (live integration tests,
|
||||
requires a root `config.yaml`, valid API credentials, and explicit opt-in via
|
||||
`make test-live` or `DEER_FLOW_RUN_LIVE_TESTS=1`). The live suite calls real
|
||||
external APIs and may incur API costs or create local sandboxes, artifacts, and
|
||||
files. It is marked `live`, excluded from `make test`, and skipped in default
|
||||
CI.
|
||||
|
||||
**Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`.
|
||||
|
||||
@ -1231,19 +1272,27 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk
|
||||
**Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.**
|
||||
|
||||
- Write tests in `backend/tests/` following the existing naming convention `test_<feature>.py`
|
||||
- Run the full suite before and after your change: `make test`
|
||||
- Run the full offline suite before and after your change: `make test`
|
||||
- Tests must pass before a feature is considered complete
|
||||
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
# Run all offline tests
|
||||
make test
|
||||
|
||||
# Explicit live integration tests (requires config.yaml and credentials;
|
||||
# calls real APIs and may create local side effects)
|
||||
make test-live
|
||||
|
||||
# Run a specific test file
|
||||
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v
|
||||
```
|
||||
|
||||
Direct pytest collection or execution of `tests/test_client_live.py` remains
|
||||
skipped unless `DEER_FLOW_RUN_LIVE_TESTS=1` is set. Do not add that opt-in to
|
||||
default CI workflows.
|
||||
|
||||
### Running the Full Application
|
||||
|
||||
From the **project root** directory:
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user