From 849692aeac867f602bbab6201011788415b66ead Mon Sep 17 00:00:00 2001 From: rayhpeng Date: Fri, 31 Jul 2026 11:29:04 +0800 Subject: [PATCH] docs(schedule): keep the module notes current with main Rebases the schedule section onto main's newer AGENTS.md so the test-live and thread-boundary entries added meanwhile are not lost. Co-Authored-By: Claude Opus 5 --- backend/AGENTS.md | 65 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 57 insertions(+), 8 deletions(-) diff --git a/backend/AGENTS.md b/backend/AGENTS.md index ab346e9a8..70f94652c 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -87,6 +87,7 @@ When making code changes, you MUST update the relevant documentation: ```bash make check # Check system requirements make install # Install all dependencies (frontend + backend) +make detect-thread-boundaries # Inventory backend executor/thread/event-loop boundaries make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight make start # Start production services locally make stop # Stop all services @@ -97,13 +98,46 @@ make stop # Stop all services make install # Install backend dependencies make dev # Run Gateway API with reload (port 8001) make gateway # Run Gateway API only (port 8001) -make test # Run all backend tests +make test # Run offline backend tests (excludes live external-API tests) +make test-live # Explicitly run live DeerFlowClient tests with real APIs make test-blocking-io # Run strict Blockbuster runtime gate on tests/blocking_io/ make lint # Lint with ruff make format # Format code with ruff make migrate-rev MSG="..." # Autogenerate a new alembic revision (see Schema Migrations section) ``` +The root `detect-thread-boundaries` target statically inventories execution +boundaries under `backend/app/` and `backend/packages/harness/deerflow/`. It +prints a concise count by execution domain and writes the complete, versioned +JSON payload to `.deer-flow/thread-boundary-inventory.json`. Every finding has +a stable `boundary_kind`: `asyncio_default_executor`, `dedicated_executor`, +`anyio_worker_thread`, `direct_event_loop_blocking`, `separate_event_loop`, or +`unresolved_dynamic_boundary`. + +The AST inventory covers `asyncio.to_thread`, default and explicit +`run_in_executor` submissions, imported aliases, simple same-module helper +wrappers (after pre-registering dedicated executor targets), `set_default_executor`, +`ThreadPoolExecutor` construction/submission, +additional event loops, synchronous LangChain tools, and direct +`BaseChatModel` fallback inheritance. It remains read-only and does not alter +executor routing or sizing. + +To supplement the static scan with configured runtime types, run: + +```bash +python scripts/detect_thread_boundaries.py \ + --runtime-config config.yaml \ + --json-output .deer-flow/thread-boundary-inventory.json +``` + +Runtime inspection imports configured tool objects and model classes so it can +record concrete tool names/types/modules, sync functions, async coroutines, +and `_agenerate`/`_astream` ownership. It does not invoke tools, instantiate +models, or call external services; import failures remain in the JSON as +`unresolved_dynamic_boundary` records. The detector implementation and focused +coverage live in `tests/support/detectors/thread_boundaries.py` and +`tests/test_detect_thread_boundaries.py`. + The `detect-blocking-io` target parses `app/`, `packages/harness/deerflow/`, and `scripts/` with AST. By default it reports only blocking IO candidates that are inside async code, reachable from async code in the same file, or reachable @@ -452,8 +486,8 @@ Localhost persistence deliberately reads the direct request `Host` and ignores ` |--------|-----------| | **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details | | **Features** (`/api/features`) | `GET /` - report config-gated feature availability (`agents_api.enabled`, `browser_control.enabled`) for frontend UI gating | -| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured | -| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) | +| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). All priced models must use one currency; mixed currencies disable cost reporting and leave cost/currency fields null instead of producing invalid aggregates. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured | +| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - replace the full config with whole-payload stdio validation; `PATCH /config` - toggle one server while preserving the raw extensions config and validating only an enabled target; both writes reload config and reset the process-local MCP cache | | **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes | | **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_ready` (whether `lark-cli` will actually be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants | | **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data | @@ -650,6 +684,7 @@ that cannot tell sibling branches apart. ### Subagent System (`packages/harness/deerflow/subagents/`) **Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist) +**Benefit-based routing policy**: Enabling subagents exposes delegation as an optimization, not a default response to complexity. The lead prompt defaults to direct execution and permits `task` only when parallel latency, specialist capability, or context-isolation benefit clearly exceeds startup, duplicate-discovery, synthesis, state-conflict, and side-effect costs. Inter-agent output dependencies and overlapping mutable state are hard vetoes for parallel dispatch, while duplicate discovery and a cheap direct path remain costs rather than categorical vetoes; a bounded sequential chain may run in one subagent when specialist or context-isolation benefit clearly wins. Parallel scopes must be independent and non-overlapping, the lead uses the fewest useful subagents, and every later batch is re-evaluated while retaining any within-batch parallel benefit. When the enforced per-response limit is 1, the rendered prompt removes parallel and multi-batch benefit guidance and permits delegation only for material specialist or context-isolation benefit. Keep this policy aligned across `lead_agent/prompt.py`, the `task` tool description, and both built-in role descriptions; routing regressions are pinned in `tests/test_subagent_routing_prompt.py`, `tests/test_subagent_prompt_security.py`, and `tests/test_lead_agent_prompt.py`. **User-scoped Skills**: Subagents resolve their configured skills through `get_or_new_user_skill_storage(user_id)` using the parent runtime identity, with `DEFAULT_USER_ID` only when no identity is available. This keeps custom-skill shadowing and visibility aligned with the lead agent instead of reading the global-only catalog. **Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers) **Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 1-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box) @@ -718,7 +753,7 @@ E2B output sync records remote file versions and actual host file metadata in a add a parallel routing middleware for PR1-style preference hints. - **Stdio file outputs**: Persistent stdio sessions are scoped by `user_id:thread_id`. For stdio transports only, DeerFlow pins the subprocess default `cwd` to the thread workspace and `TMPDIR`/`TMP`/`TEMP` to `workspace/.mcp/tmp/`, unless the operator explicitly configured `cwd` or temp env values. SSE/HTTP transports skip this filesystem prep entirely. - **Stdio path translation**: MCP-returned local file references are not copied. If a `ResourceLink` or conservative free-text path resolves to an existing file inside the thread's mounted user-data tree, it is translated deterministically to `/mnt/user-data/...`; paths outside that tree remain unchanged. -- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (the `PUT /api/mcp/config` reset only clears the cache in its own worker) +- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (`PUT /api/mcp/config` keeps whole-payload validation, while `PATCH /api/mcp/config` changes only one server's `enabled` field, normalizes the same `type`/MCP-spec `transport` alias as the runtime config model, and validates the target only when enabling it; either endpoint's reset clears the cache only in its own worker). MCP, skill, and embedded-client writers share `atomic_write_extensions_config()`, which writes and fsyncs a same-directory temporary file before `os.replace()` and preserves an existing file's mode and symlink target; failed serialization or replacement leaves the prior config intact and cleans up the temporary file. ### Skills System (`packages/harness/deerflow/skills/`) @@ -1185,7 +1220,7 @@ Config is env-driven like the others — `MonocleTracingConfig`, built in `get_t - `skills` - Map of skill name → state (enabled) - `middlewares` - Zero-argument `AgentMiddleware` class paths for lead and subagent runtime extension. `config.yaml -> extensions` can override these fields after validation; overrides are replace-per-field, not list concatenation. -Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and skill state at runtime; `middlewares` remains an operator-controlled config-file extension point. +Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and skill state at runtime; their `extensions_config.json` writes use the shared atomic replacement helper, while `middlewares` remains an operator-controlled config-file extension point. ### Embedded Client (`packages/harness/deerflow/client.py`) @@ -1220,7 +1255,13 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk **Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent. -**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml) +**Tests**: `tests/test_client.py` (offline unit tests including +`TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, +requires a root `config.yaml`, valid API credentials, and explicit opt-in via +`make test-live` or `DEER_FLOW_RUN_LIVE_TESTS=1`). The live suite calls real +external APIs and may incur API costs or create local sandboxes, artifacts, and +files. It is marked `live`, excluded from `make test`, and skipped in default +CI. **Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`. @@ -1231,19 +1272,27 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk **Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.** - Write tests in `backend/tests/` following the existing naming convention `test_.py` -- Run the full suite before and after your change: `make test` +- Run the full offline suite before and after your change: `make test` - Tests must pass before a feature is considered complete - For lightweight config/utility modules, prefer pure unit tests with no external dependencies - If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`) ```bash -# Run all tests +# Run all offline tests make test +# Explicit live integration tests (requires config.yaml and credentials; +# calls real APIs and may create local side effects) +make test-live + # Run a specific test file PYTHONPATH=. uv run pytest tests/test_.py -v ``` +Direct pytest collection or execution of `tests/test_client_live.py` remains +skipped unless `DEER_FLOW_RUN_LIVE_TESTS=1` is set. Do not add that opt-in to +default CI workflows. + ### Running the Full Application From the **project root** directory: