* feat(community): add Sofya web search provider Add a community provider backed by Sofya (https://sofya.co). Its search endpoint returns the content of the result pages, not only their snippets, and its fetch endpoint returns a page as markdown. Both are plain JSON over HTTP, so this needs no extra Python package (uses httpx, already a dependency). Changes: - backend/packages/harness/deerflow/community/sofya/__init__.py - backend/packages/harness/deerflow/community/sofya/tools.py Implements web_search_tool and web_fetch_tool using httpx. API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY env var. Follows the same interface and output shape as the existing ddg_search and serper providers, including the max_results parameter with config override and the structured "No results found" error. - backend/tests/test_sofya_tools.py Unit tests covering API key resolution, config overrides, result mapping, time range, HTTP errors, empty results, and fetch failures. - config.example.yaml: add commented-out Sofya web_search and web_fetch examples alongside the other providers - .env.example: add SOFYA_API_KEY placeholder - backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and the environment variables * fix(sofya): honor caller max_results, validate search_depth, join time_range contract test - Caller-supplied max_results now wins; config is used only when the argument is omitted, matching GroundRoute. - search_depth is clamped to basic/snippets; an unsupported value logs a warning and falls back to basic. - Sofya added to the shared time_range schema contract test. * fix(sofya): cap per-result content so a search stays inline An unbounded search payload (up to 20 read pages) crossed the tool output budget middleware's externalize_min_chars threshold, which replaces the result list with a file reference. Cap each result's content at contents_max_characters (default 2000, 0 disables), matching Exa's config key. Five capped results stay under the 12000 char threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): list Sofya in the recency contract, coerce non-string content _clip subscripted its input, so a non-string content or description from the API raised TypeError instead of degrading. Coerce to text first, the way _sofya_post and _response_results guard the shapes around it. Also add Sofya to the Web Search Recency section in backend/AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog web_fetch sliced its content the same way web_search did before the last push: a truthy non-string from the API passed the falsiness guard and then raised TypeError. Reuse _clip, keeping the `or ""` so empty content still reports "No content found". Also add sofya to the community provider inventory in packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs The ZH web_fetch tab list named five providers where EN names seven. Both tabs mirror their EN counterparts, so the two locales list the same web_fetch providers again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
10 KiB
Tool System (packages/harness/deerflow/tools/)
get_available_tools(groups, include_mcp, model_name, subagent_enabled) assembles:
- Config-defined tools - Resolved from
config.yamlviaresolve_variable() - MCP tools - From enabled MCP servers (lazy initialized, cached with resolved-path + content-signature invalidation)
- Built-in tools:
present_files- Make output files visible to user (only/mnt/user-data/outputs); virtual paths useresolve_runtime_user_id(runtime)so validation resolves the same user-scoped outputs directory established byThreadDataMiddlewareask_clarification- Request clarification (intercepted by ClarificationMiddleware, which preserves text fallback and addsartifact.human_inputfor Web UI Human Input Cards). Beyond free text and single choice, the request-side v2 protocol supportsfields(structured form card collecting several values at once; field types: text/textarea/number/select/multi_select/checkbox/date, validated and normalized server-side in the middleware — invalid entries are dropped, unknown types degrade totext; a standalone multi-select question is a one-field form). Replies stay on the v1 response protocol (text/option): the form card submits a readable text summaryview_image- Read image as base64 (added only if model supports vision)setup_agent- Bootstrap-only: persist a brand-new custom agent'sSOUL.mdandconfig.yaml. Bound only whenis_bootstrap=True.update_agent- Custom-agent-only: persist self-updates to the current agent'sSOUL.md/config.yamlfrom inside a normal chat (partial update + atomic write). Bound whenagent_nameis set andis_bootstrap=False.
- Subagent tool (if enabled):
task- Delegate to subagent (prompt,subagent_type, optionalacceptance_criteria, and an optional model-visibledescriptionused only as a short progress label). Execution never depends ondescription; lifecycle display falls back topromptwhen a provider omits it. Subagent reports are self-reports: the docstring directs the lead to expect[rN]receipt citations and verifiable handles whileverification.receipts_enabled(and explicitly qualifies that disabled receipts mean no citations and no citation verdict), to read the delegation ledger's citation cross-check as execution evidence only, and to attachacceptance_criteriafor objectively checkable outcomes (canonical formsfile:<path> exists|non-empty,file_written:<path>,tests_passed:<command>); criteria are handed to the executor and appended to the subagent's task message as untrusted data (seesubagents/report_contract.py). Polling safety timeouts carry the latest published tool receipts into the terminal task metadata before requesting background cancellation.batch_task,batch_status,cancel_batch- Explicit durable batch submission/progress/cancellation. Added only while the startup SQL-backed batch submitter is installed; large results stay in the owner-scoped API/JSONL export rather than the lead context.- Direct
create_deerflow_agentintegrations receive cloned tools bound to their explicitSubagentRuntime. The boundtaskforwards that runtime's exact execution controller and optional caller-ownedAppConfiginto registry/model/tool resolution andSubagentExecutor; bound batch tools use the same config snapshot and resolve only that runtime's submitter before falling back to no other application's active worker. Keep the original tool name/schema unchanged so model contracts and user-tool deduplication remain stable.
The ordinary task boundary carries one narrow parent-loop middleware recorder into the isolated subagent runtime under separate loop-detection and tool-promotion keys. It schedules only record_middleware calls back onto the loop that owns RunJournal, keeps an execution-local atomic promotion claim so parallel searches do not double-report one new schema, is fenced and drained once before task returns, and never exposes the journal or event store to the child loop. Durable batch tasks have no parent run journal and do not use this bridge.
Scheduled-task runtime note:
- Scheduled background runs set
context.non_interactive=trueand therefore excludeask_clarificationfrom the lead-agent tool list. This keeps scheduler-triggered runs from stalling on human confirmation mid-execution.non_interactiveis an internal-only context key: it is merged frombody.contextonly when the request authenticated as the process-internal user (the scheduler path), never from arbitrary HTTP/IM clients.
Durable MCP task-management tools are added only while the process-local task submitter is installed. They expose bounded local task fields, including whether cancellation was requested, but never the remote handle. Cancellation records that request durably and returns immediately; the background service owns the remote call and retries. These remain ordinary business tools under an active skill's allowed-tools policy and must be declared explicitly.
Community tools (packages/harness/deerflow/community/): optional integrations, each in its own subpackage and wired through config.yaml. Documented examples:
tavily/- Web search (5 results default) and web fetch (4KB limit)sofya/- Web search (5 results default, per-result content capped at 2000 chars) and web fetch (4KB limit)jina_ai/- Web fetch via Jina reader API with readability extractionfirecrawl/- Web scraping via Firecrawl APIimage_search/- Image search via DuckDuckGoaio_sandbox/- Docker-based isolation (AioSandboxProvider)browser_automation/- Agentic browser control (statefulnavigate → observe → click/typeloop) via Playwright, distinct from the read-onlyweb_fetch/web_capturetools. Tools:browser_navigate,browser_snapshot,browser_click,browser_type,browser_get_text,browser_back,browser_screenshot,browser_close(configgroup: browser). A process-localBrowserSessionManagerowns one private, loop-affine Playwright event-loop thread (same pattern as the BoxLite provider) so a per-thread browser session survives across turns regardless of the caller's loop (Gateway / TUI / test). Each action returns a fresh page snapshot whose interactive elements are addressed by a stable numeric[ref]index (stamped asdata-df-refduring snapshot), so the model acts on what it just observed instead of holding stale handles or guessing selectors. URLs are SSRF-screened via the sharedvalidate_public_http_url(opt-outallow_private_addressesonly for intentional internal targets). CDP attachment cannot install the request guard on an existing Chrome context, socdp_urlfails closed unless the operator explicitly setsallow_unguarded_cdp: truefor a trusted local browser. Browser REST/Live access also requires an exact non-NULL thread owner, rather than the general legacy shared-thread policy, because retained pages may contain authenticated state. Session admission is a hardmax_sessionscap: pinned Live/operation sessions are never evicted, and a new thread is rejected when no unpinned session can be closed; one Live viewer owns a session at a time. Optional dependency:cd backend && uv sync --extra browser && uv run playwright install chromium;scripts/detect_uv_extras.pypreserves the extra whenconfig.yamlenablesbrowser_navigate, and Gateway startup fails fast if configured browser control cannot import Playwright. Tests:tests/test_browser_automation.py(mocked tools + a real-Chromium integration test guarded byimportorskip);tests/manual_browser_live_check.pyis a manual DeepSeek-driven end-to-end check (not collected by pytest). Live UI input dispatch is kept independent from JPEG capture: non-move actions start a rate-limited background refresh loop, so pointer, wheel, or keyboard input stays responsive while continuous gestures still produce frames throughout the interaction.
Additional providers also live here (boxlite, brave, browserless, crawl4ai, ddg_search, e2b_sandbox, exa, fastcrw, groundroute, infoquest, searxng, serper, serply, sofya, tencent_wsa, tenki); see each subpackage for specifics. tencent_wsa uses Tencent Cloud Web Search's service API key endpoint (TENCENTCLOUD_WSA_APIKEY), not Tencent Cloud SecretId/SecretKey signing. Its max_results is capped at 50; requests above 10 use Tencent's optional Cnt parameter, which needs a Tencent Cloud plan that supports it. E2B bootstrap is required. If it fails, the provider kills and closes the unusable remote sandbox. New sandbox creation raises an error. Warm-pool reclaim and remote discovery discard the sandbox and continue acquisition. E2B mounts remain optional.
E2B output sync records remote file versions and actual host file metadata in a thread-local manifest. The manifest binds to the remote sandbox ID. A complete output listing removes entries for deleted files. This avoids repeat downloads when the host filesystem rounds modification times. A single release-time sync pass is bounded by aggregate ceilings (_MAX_SYNC_TOTAL_BYTES, _MAX_SYNC_FILES, _SYNC_DEADLINE_SECONDS) on top of the per-file _MAX_DOWNLOAD_SIZE cap, so a pathological outputs tree cannot make release download unboundedly; a truncated pass logs what it dropped and leaves the manifest un-pruned (only entries observed in that pass are reconciled), so files it never reached are retried on the next release rather than being forgotten.
ACP agent tools:
invoke_acp_agent- Invokes external ACP-compatible agents fromconfig.yaml- ACP launchers must be real ACP adapters. The standard
codexCLI is not ACP-compatible by itself; configure a wrapper such asnpx -y @zed-industries/codex-acpor an installedcodex-acpbinary - MiniMax Code speaks ACP directly: configure
command: mcodewithargs: ["acp"]. It receives DeerFlow's enabled MCP servers and uses the per-thread ACP workspace; the Gateway process must have an authenticatedmcodeexecutable onPATH - ACP results collect only
agent_message_chunktext. Thought chunks remain internal and must not be concatenated into the tool result - Missing ACP executables now return an actionable error message instead of a raw
[Errno 2] - Each ACP agent uses a per-thread workspace at
{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/. The workspace is accessible to the lead agent via the virtual path/mnt/acp-workspace/(read-only). In docker sandbox mode, the directory is volume-mounted into the container at/mnt/acp-workspace(read-only); in local sandbox mode, path translation is handled bytools.py