deer-flow/backend/packages/harness/deerflow/mcp/AGENTS.md

MCP System (packages/harness/deerflow/mcp/)

• Uses langchain-mcp-adapters MultiServerMCPClient for multi-server management

• Long-running task foundation: mcp/tasks/ defines the protocol-neutral McpTaskDriver contract and normalized TaskSnapshot states (submitted, working, input_required, completed, failed, cancelled). A driver-supplied poll_after_seconds must be a finite positive number, validated at the TaskSnapshot boundary so every driver is held to the same invariant rather than each one guarding the consumer that turns the interval into a timedelta. persistence/mcp_tasks/ owns the durable remote-handle mapping, poll schedule, notification state, lease owner, and separate consecutive poll/delivery error counters; app/mcp_tasks/McpTaskService performs status, cancellation, and notification work outside the Agent/LLM loop. Notification retries keep their idempotency attempt separate from the delivery-failure count, use capped exponential backoff, and stop after five failures; strict existing-thread admission dead-letters a deleted/mismatched target immediately. A status result is applied only when the worker still owns an unexpired lease, so a stale result cannot be written after expiry even before another worker reclaims the row. Poll timestamps and retry schedules are based on the remote call's completion time rather than the scan start. If submission succeeds but persistence fails or the caller is cancelled while persistence is in flight, the service best-effort cancels the remote task and preserves the original error or cancellation if that compensation also fails. The exact uq_mcp_tasks_user_server_remote conflict is different: an existing durable row already owns the remote handle, so the conflict surfaces without cancelling that tracked task. Unexpected per-task poll failures are isolated from sibling claims and remain recoverable through lease expiry; Gateway shutdown cancels the poller so a hung external status call cannot block process exit. input_required and terminal states stop polling and become notification_status=pending for later Agent/UI delivery. Durable recovery requires a SQL database backend (sqlite or postgres); the in-memory backend leaves the repository/service unavailable. The runtime is startup-configured by mcp_tasks and disabled by default until a concrete driver is registered; this foundation does not alter ordinary MCP tool behavior on its own.

• Runtime availability boundary: the installed process-local submitter is the source of truth for durable task-management tool exposure. mcp_tasks is startup-only; changing it on disk does not alter the live toolset until the Gateway restarts.

• Long-running ordinary task driver: extensions_config.json -> mcpServers.<server>.task_toolsets binds exact raw submit/status/cancel names; one raw tool may occupy only one role across that server's groups. mcp/tools.py hides status/cancel and replaces submit with a wrapper that returns only the local task ID after persistence. ordinary.py reads only MCP structuredContent, maps remote running to working, and treats error_code=task_not_found or malformed structured output as permanent failure. A status call with isError=true is a retryable call failure: the first text content block is retained as a bounded diagnostic, while a permanent remote-task outcome must arrive in a normal result with structured status=failed. task_tool_caller.py restores the same (server_name, user_id:thread_id) stdio session scope; HTTP/SSE calls remain ephemeral, apply session_init_timeout to initialization and tool_call_timeout to task calls, and support server-level OAuth refresh outside an Agent run. McpTaskService exponentially backs off transient status/cancel errors without a maximum attempt count, derives API tracking_degraded from the consecutive-error threshold, keeps input_required on a slower poll, and caps finite positive remote poll hints at 24 hours. Task-enabled server runtime/binding configuration and mcpInterceptors are frozen to the Gateway startup snapshot; hot drift fails clearly before tool discovery can diverge from background calls, while presentation-only fields and non-task servers remain reloadable. Configured task toolsets fail startup when the runtime is disabled or persistence is memory. Users still cannot submit an answer back to an input_required remote task.

• Durable task payload bounds: persisted task errors are capped at 4,000 characters. input_required and result_artifact must each serialize as valid JSON within 64 KiB; an invalid or oversized payload becomes a permanent protocol failure rather than being truncated and changing its semantics. Remote task IDs/task names are limited to 255 characters and task-enabled server names to 128, matching the SQL schema; an oversized submitted remote ID is rejected only after the Service has the handle so compensation cancellation still runs. Oversized results retain the existing bounded preview/truncation/artifact behavior.

• Lazy initialization: Tools loaded on first use via get_cached_mcp_tools()

• Persistent stdio session capacity: MCPSessionPool.MAX_SESSIONS is a hard cap on the live LRU registry. Capacity is enforced both before creating a session and when an in-flight session is promoted, because different keys can finish initialization concurrently after all observing spare capacity. Promotion-time victims are signalled and drained by separately tracked teardown tasks on their owning loops; the newly promoted owner must never await a victim, so a blocked victim exit cannot prevent its own closure or disconnect recovery.

• Cache invalidation: Detects extensions-config changes by comparing the resolved config path and a (mtime, size, sha256) content signature against the values recorded at initialization, not a strict mtime > comparison. This catches same-second edits, mtime that stays put or moves backward (git checkout, cp -p / backup restore, tar / rsync, object-store / network mounts), and a switch to a different config file with an equal-or-older mtime. The signature helper (config/file_signature.py::get_config_signature) is shared with config/app_config.py::get_app_config() for the sibling runtime-editable config file, rather than each maintaining its own copy. ExtensionsConfig.resolve_config_path() raises FileNotFoundError for an explicit config_path/DEER_FLOW_EXTENSIONS_CONFIG_PATH that points at a missing file — an operator-asserted path going missing is a real misconfiguration, so this is intentionally loud for callers that load the config for actual use (e.g. from_file() via get_mcp_tools()); only the fallback search mode returns None. The MCP cache's own path resolution (mcp/cache.py::_resolve_config_path) is narrower: it catches that specific FileNotFoundError locally and treats it the same as "unconfigured", so this staleness check degrades to "not stale" instead of propagating an exception when a previously-valid explicit/env-var config disappears mid-run. If initialize_mcp_tools() itself observes a config-signature change between the pre-load and post-load snapshots, that discard branch must reset the tool cache through the same session-pool retirement path as normal stale invalidation before waiters retry; otherwise a stale load can leave (server_name, scope_key) sessions from the abandoned connection available to the next wrapper build.

• Transports: stdio (command-based), SSE, HTTP

• Per-server tool-name prefixing: mcpServers.<server>.tool_name_prefix defaults to true, preserving the collision-safe <server_name>_ prefix. Servers whose tools already carry a stable namespace may set it to false; discovery then calls langchain_mcp_adapters.tools.load_mcp_tools with that server's flag. Source routing and stdio session-pool wrapping are based on the producing server and transport, never on whether the visible tool name starts with the server prefix.

• OAuth (HTTP/SSE): Supports token endpoint flows (client_credentials, refresh_token) with automatic token refresh + Authorization header injection. The rendered <token_type> <access_token> passes mcp/headers.py::illegal_header_value_reason inside OAuthTokenManager.get_authorization_header — the one boundary the tool interceptor, the initial discovery headers and the durable task path all read their value from — so a token endpoint returning something the transport would refuse fails closed instead of letting h11 echo the token into a model-visible tool error. The rendered value is what gets checked, not the two fields separately, because that is what the transport sees. The operator's static headers get the same check in mcp/client.py::build_server_params, where build_servers_config already drops just that server and logs the reason.

• Per-user credentials (HTTP/SSE): mcpServers.<server>.user_auth maps DeerFlow user ids to credential header values ($ENV_VAR references supported). mcp/user_scoped_auth.py::build_user_scoped_auth_interceptor rewrites the configured header on every tool call from the authenticated runtime user (registered after OAuth in mcp/interceptors.py, so its per-user value wins the header for servers declaring both). Fail-closed: an unmapped user or an empty resolved credential raises a ToolException unless on_missing: "passthrough" is set; a resolved credential the transport would refuse as a header value (line break, surrounding whitespace, non-ASCII — mcp/headers.py::illegal_header_value_reason) is always denied without echoing the value, since h11 renders the full value into its exception message on the line break and whitespace cases and tool errors are model-visible (httpx catches the non-ASCII case earlier, naming only the offending character). The server's static headers serve startup tool discovery only. Gateway GET masks user_auth.users values; PUT round-trips masked values by preserving stored credentials.

• Per-request credentials (HTTP/SSE): mcpServers.<server>.headers_from_context maps HTTP header names to keys of the run request's config.context.secrets carrier, for credentials the caller chooses per request (multi-tenant gateways, per-run API keys) rather than per configured user. mcp/context_headers.py::build_context_headers_interceptor resolves the mapping on every tool call and rewrites those headers. Registered after user_auth in mcp/interceptors.py, so for a server declaring several sources the per-request value wins the final header: precedence is static headers < oauth < user_auth < headers_from_context. Fail-closed: a mapped key absent from the request secrets (or resolved empty) raises a ToolException naming only the missing key, unless on_missing: "passthrough" is set — a silent fallback would send one tenant's call under the discovery credential's authority. A resolved value the transport would refuse as a header value (line break, surrounding whitespace, non-ASCII) is always denied regardless of on_missing, without echoing the value: h11 renders the full value into its exception message on a line break or surrounding whitespace, and ToolErrorHandlingMiddleware copies tool errors into model-visible messages, so an unchecked bad credential would land the secret in the prompt, the checkpoint, and traces. sse/http only; a stdio server warns and is skipped, as with user_auth. The block stores names, never a credential, so the Gateway returns it unmasked and a PUT replaces the declared mapping verbatim; only extra="allow" keys inside it get sensitive-key masking, and those are restored from the stored block on a round-trip like every other masked extra. Durable task_toolsets calls split: submit is awaited inside the Agent run and carries the mapped headers (McpTaskToolCaller.call_tool(request_scoped_headers=True), set only by OrdinaryMcpTaskDriver.submit), while status and cancel run after that run ended and keep the server-level credentials — so on_missing: "deny" covers submit but not those polls, which is what the startup warning is about.

• Header names are case-insensitive (mcp/headers.py): every credential interceptor writes through apply_header_overrides, which drops a key differing only in case and emits the spelling the connection already uses. Without it a static authorization and an injected Authorization both reach httpx — the adapter merges connection and override headers with a plain {**static, **override} splat — and a server reading the field with a single-value accessor gets the static entry, silently inverting the precedence above. headers_from_context.headers also rejects two spellings of one header at config load.

• Reading the run context from an interceptor: use request.runtime (LangGraph's tool node injects a ToolRuntime into any tool parameter named runtime, which covers both the pooled stdio wrapper and langchain-mcp-adapters' own HTTP/SSE tool), falling back to ambient langgraph.runtime.get_runtime(). Do not use langgraph.config.get_config()["context"]: the run context rides the runtime, not the RunnableConfig propagated to child runnables, so that key is None inside a tool call. tests/test_mcp_context_headers.py::test_adapter_tool_receives_the_runtime_langgraph_injects pins the injection rule against an upstream rename by disabling the ambient fallback and driving a real adapter tool through a real graph.

• Routing hints: extensions_config.json -> mcpServers.<server>.routing and tools.<original_tool_name>.routing are soft preference metadata. The effective routing is resolved while mcp/tools.py::get_mcp_tools() still has both source_name and the original MCP tool name, then stored on tool.metadata under deerflow_mcp_routing. Prompt rendering uses tools/builtins/tool_search.py::get_mcp_routing_hints_prompt_section, which references tool_search when a hinted MCP tool is currently deferred; do not add a parallel routing middleware for PR1-style preference hints.

• Stdio file outputs: Persistent stdio sessions are scoped by user_id:thread_id. For stdio transports only, DeerFlow pins the subprocess default cwd to the thread workspace and TMPDIR/TMP/TEMP to workspace/.mcp/tmp/, unless the operator explicitly configured cwd or temp env values. .mcp is a DeerFlow-owned internal namespace: its temporary/debug files remain addressable when returned by a tool but are excluded from run workspace-change summaries — by directory name at any depth, consistent with the other reserved names in EXCLUDED_DIR_NAMES (.git, node_modules, …) and robust if a server ever creates a relative .mcp from a different cwd. Both launch paths pin it at the workspace root today. SSE/HTTP transports skip this filesystem prep entirely.

• Stdio disconnect recovery: Ordinary Agent tool calls and durable task submit/status/cancel calls that receive the MCP SDK's explicit Connection closed error or an AnyIO closed-stream error evict only that (server_name, user_id:thread_id) session when the registered entry is still the same ClientSession that failed. A late error from an old concurrent call cannot evict its replacement or a new in-flight creation. The failing call still surfaces its original error and is never replayed automatically; a later retry creates a fresh subprocess/session. Protocol timeouts, normal isError=true tool results, and interceptor failures do not evict a healthy stateful session.

• Stdio path translation: MCP-returned local file references are not copied. If a ResourceLink or conservative free-text path resolves to an existing file inside the thread's mounted user-data tree, it is translated deterministically to /mnt/user-data/...; paths outside that tree remain unchanged.

• Runtime updates: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (PUT /api/mcp/config keeps whole-payload validation, while PATCH /api/mcp/config changes only one server's enabled field, normalizes the same type/MCP-spec transport alias as the runtime config model, and validates the target only when enabling it; either endpoint's reset clears the cache only in its own worker). MCP, skill, and embedded-client writers hold the process-local extensions_config_write_lock plus the sidecar advisory extensions_config_file_lock for the complete read-modify-write/reload cycle, then share atomic_write_extensions_config(), which writes and fsyncs a same-directory temporary file before os.replace() and preserves an existing file's mode and symlink target; failed serialization or replacement leaves the prior config intact and cleans up the temporary file.

• Stdio launch policy at the HTTP boundary (routers/mcp.py::_validate_mcp_update_request, shared by PUT and the enable branch of PATCH): a config file may express anything, but the API is untrusted input, so an API-registered stdio server must (a) name a bare executable from the allowlist — _DEFAULT_MCP_STDIO_COMMAND_ALLOWLIST = {npx, uvx}, extended by DEER_FLOW_MCP_STDIO_COMMAND_ALLOWLIST, with path separators, whitespace, and shell metacharacters rejected in command; (b) carry no args flag in _ARBITRARY_EXEC_ARGS; and (c) set no env name in _CODE_INJECTING_ENV_VARS. Checks (b) and (c) exist because the command check alone names a binary without constraining what that binary runs. The env denylist applies to every allowlisted command, and both denylists match --flag=value as well as --flag value. The args denylist's scope depends on the command, because where a launcher stops parsing its own flags is what decides whether a token is an exec flag at all:

  • For a package launcher in _PACKAGE_LAUNCHERS ({npx, uvx}) only the launcher's own option region is screened. npx/uvx stop parsing their flags at the package name and hand every later token to the spawned server's argv, where -c is routinely "config" and -e "env" — screening those rejected ordinary third-party servers while covering nothing. A bare -- ends the region too: only the first token after it is the package name. Finding that boundary needs each launcher's option arity, since a value is not a positional — npx -p <pkg> -c '<command>' runs the command (-p is npm exec's --package, so <pkg> is its value and npm keeps parsing), so ending the region at the first non-flag token would walk straight past it. _NPX_BOOLEAN_ARGS is generated from @npmcli/config's definitions (npm 10.9.4) minus the -p exec override; _UVX_VALUE_ARGS comes from uvx --help (uv 0.11.1). Regenerate these against a newer launcher rather than hand-editing. The unknown-option default is deliberately opposite per launcher, following the exec set rather than symmetry: npx owns real exec flags (-c/--call), so an unknown option consumes a value and keeps the region open (npm errors on options it does not define, so this cannot reject a working invocation); uvx owns no string-eval flag at all, so its screen is a tripwire, an unknown option consumes nothing, and uv's large boolean surface cannot over-block. uvx's exec set also drops the short spellings, because -c is uv's --constraints and -p its --python.
  • Every other command is screened whole, with two extra rules, because it is an interpreter rather than a package runner: -p counts as an exec flag there (node's --print), and single-dash short-option clusters are decomposed letter by letter so node -pe cannot pass a check that only splits on =.

  Verdicts are pinned against the real launchers: for npx, every argument vector the validator rejects is one npx actually executes, and every vector it allows is one npx passes through to the server. env screening covers names that execute code unconditionally at process startup, e.g. PYTHONPATH/PYTHONHOME, which run a caller-controlled sitecustomize.py at interpreter startup under plain uvx. Caller-controlled search paths are a weaker, conditional class and are an accepted residual: LD_LIBRARY_PATH/DYLD_LIBRARY_PATH (conditional on the process loading a shadowable library, and legitimately set by native-dependency servers) and NODE_PATH (searched after the local node_modules chain, so it cannot shadow an installed dependency, and ignored entirely by ESM import — it can only supply a CJS module that would otherwise fail to resolve). Do not move a search path into the set: it would make the "unconditional" rule untrue, which is how a defense-in-depth list starts being mistaken for a boundary. Remote transports skip all three — they spawn nothing. This is defense in depth, not a trust boundary. npx/uvx exist to fetch and execute remote packages, so an admin can still point one at a package they published; the boundary is admin authentication plus network reachability. Do not add a check here on the assumption that it makes MCP registration safe for untrusted admins — it does not, and the fix for that is not a bigger denylist.

Durable MCP task runtime (mcp_tasks, McpTaskService; summarized in backend/AGENTS.md): Long-running MCP work uses a separate durable task runtime rather than keeping remote task IDs or status polling inside the Agent loop. Explicit task_toolsets bind raw submit/status/cancel names; only submit remains Agent-visible, and its wrapper persists the remote handle before returning a local ID. McpTaskService claims due rows with leases, resolves a protocol-specific McpTaskDriver, and writes normalized snapshots back to mcp_tasks; expired leases are the restart-recovery mechanism, and a result returned after expiry or after a cancel request must be discarded even when the owner token still matches. The first cancel request fences an in-flight poll lease, while repeats preserve an active cancellation lease so they cannot issue concurrent remote cancels; cancellation backoff starts when the remote attempt finishes, so a slow timeout cannot consume the retry delay. Cancellation, polling, and notification batches isolate per-task exceptions; an unexpected cancellation/poll failure leaves that record's lease to expire, while notification failures release only the affected lease for retry. Input-required and terminal event snapshots are delivered by idempotent Agent runs and marked delivered only after run success; the trusted notification instruction stays outside the input boundary while the serialized remote event is framed as untrusted data. A busy-thread conflict is normalized back to the service boundary so the queued snapshot coalesces to the latest task event. A missing dispatched run becomes a failed delivery attempt, while transient run-store hydration errors stay distinguishable and retry the same lookup. The database is the source of truth; ThreadState receives only a bounded current-thread projection, and display names are neutralized at that model-state boundary. The installed process-local submitter is the source of truth for management-tool exposure; hot mcp_tasks edits take effect only after restart, and active skills must explicitly declare the list/cancel business tools.

Task notification failure handling: MCP notification failures use a consecutive counter separate from the idempotency-key dispatch_attempt, capped exponential backoff, latest-event rebuilding before a run launches, and a five-attempt budget before dead_letter. A permanently missing/mismatched target thread is dead-lettered immediately instead of being recreated or reclaimed. HTTP and Agent cancellation requests return after the durable cancel fence; the background loop alone owns the potentially slow remote call and retry schedule. The HTTP cancel endpoint rejects requests with 503 when the loop is not running (mcp_tasks_available false, e.g. mcp_tasks.enabled=false with SQL persistence), so a cancellation is never acknowledged without a worker to perform it. The bounded notification error/count/status join poll and cancellation diagnostics in the task detail API and expanded card.