Yusuf Gürdoğan 23bd76046a
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider

Add a community provider backed by Sofya (https://sofya.co). Its search
endpoint returns the content of the result pages, not only their snippets,
and its fetch endpoint returns a page as markdown. Both are plain JSON over
HTTP, so this needs no extra Python package (uses httpx, already a
dependency).

Changes:
- backend/packages/harness/deerflow/community/sofya/__init__.py
- backend/packages/harness/deerflow/community/sofya/tools.py
  Implements web_search_tool and web_fetch_tool using httpx.
  API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY
  env var. Follows the same interface and output shape as the existing
  ddg_search and serper providers, including the max_results parameter with
  config override and the structured "No results found" error.
- backend/tests/test_sofya_tools.py
  Unit tests covering API key resolution, config overrides, result mapping,
  time range, HTTP errors, empty results, and fetch failures.
- config.example.yaml: add commented-out Sofya web_search and web_fetch
  examples alongside the other providers
- .env.example: add SOFYA_API_KEY placeholder
- backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and
  the environment variables

* fix(sofya): honor caller max_results, validate search_depth, join time_range contract test

- Caller-supplied max_results now wins; config is used only when the
  argument is omitted, matching GroundRoute.
- search_depth is clamped to basic/snippets; an unsupported value logs a
  warning and falls back to basic.
- Sofya added to the shared time_range schema contract test.

* fix(sofya): cap per-result content so a search stays inline

An unbounded search payload (up to 20 read pages) crossed the tool output
budget middleware's externalize_min_chars threshold, which replaces the
result list with a file reference. Cap each result's content at
contents_max_characters (default 2000, 0 disables), matching Exa's config
key. Five capped results stay under the 12000 char threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): list Sofya in the recency contract, coerce non-string content

_clip subscripted its input, so a non-string content or description from
the API raised TypeError instead of degrading. Coerce to text first, the
way _sofya_post and _response_results guard the shapes around it. Also add
Sofya to the Web Search Recency section in backend/AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog

web_fetch sliced its content the same way web_search did before the last
push: a truthy non-string from the API passed the falsiness guard and then
raised TypeError. Reuse _clip, keeping the `or ""` so empty content still
reports "No content found".

Also add sofya to the community provider inventory in
packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs

The ZH web_fetch tab list named five providers where EN names seven. Both
tabs mirror their EN counterparts, so the two locales list the same
web_fetch providers again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-08 10:50:03 +08:00

10 KiB

Tool System (packages/harness/deerflow/tools/)

get_available_tools(groups, include_mcp, model_name, subagent_enabled) assembles:

  1. Config-defined tools - Resolved from config.yaml via resolve_variable()
  2. MCP tools - From enabled MCP servers (lazy initialized, cached with resolved-path + content-signature invalidation)
  3. Built-in tools:
    • present_files - Make output files visible to user (only /mnt/user-data/outputs); virtual paths use resolve_runtime_user_id(runtime) so validation resolves the same user-scoped outputs directory established by ThreadDataMiddleware
    • ask_clarification - Request clarification (intercepted by ClarificationMiddleware, which preserves text fallback and adds artifact.human_input for Web UI Human Input Cards). Beyond free text and single choice, the request-side v2 protocol supports fields (structured form card collecting several values at once; field types: text/textarea/number/select/multi_select/checkbox/date, validated and normalized server-side in the middleware — invalid entries are dropped, unknown types degrade to text; a standalone multi-select question is a one-field form). Replies stay on the v1 response protocol (text/option): the form card submits a readable text summary
    • view_image - Read image as base64 (added only if model supports vision)
    • setup_agent - Bootstrap-only: persist a brand-new custom agent's SOUL.md and config.yaml. Bound only when is_bootstrap=True.
    • update_agent - Custom-agent-only: persist self-updates to the current agent's SOUL.md / config.yaml from inside a normal chat (partial update + atomic write). Bound when agent_name is set and is_bootstrap=False.
  4. Subagent tool (if enabled):
    • task - Delegate to subagent (prompt, subagent_type, optional acceptance_criteria, and an optional model-visible description used only as a short progress label). Execution never depends on description; lifecycle display falls back to prompt when a provider omits it. Subagent reports are self-reports: the docstring directs the lead to expect [rN] receipt citations and verifiable handles while verification.receipts_enabled (and explicitly qualifies that disabled receipts mean no citations and no citation verdict), to read the delegation ledger's citation cross-check as execution evidence only, and to attach acceptance_criteria for objectively checkable outcomes (canonical forms file:<path> exists|non-empty, file_written:<path>, tests_passed:<command>); criteria are handed to the executor and appended to the subagent's task message as untrusted data (see subagents/report_contract.py). Polling safety timeouts carry the latest published tool receipts into the terminal task metadata before requesting background cancellation.
    • batch_task, batch_status, cancel_batch - Explicit durable batch submission/progress/cancellation. Added only while the startup SQL-backed batch submitter is installed; large results stay in the owner-scoped API/JSONL export rather than the lead context.
    • Direct create_deerflow_agent integrations receive cloned tools bound to their explicit SubagentRuntime. The bound task forwards that runtime's exact execution controller and optional caller-owned AppConfig into registry/model/tool resolution and SubagentExecutor; bound batch tools use the same config snapshot and resolve only that runtime's submitter before falling back to no other application's active worker. Keep the original tool name/schema unchanged so model contracts and user-tool deduplication remain stable.

The ordinary task boundary carries one narrow parent-loop middleware recorder into the isolated subagent runtime under separate loop-detection and tool-promotion keys. It schedules only record_middleware calls back onto the loop that owns RunJournal, keeps an execution-local atomic promotion claim so parallel searches do not double-report one new schema, is fenced and drained once before task returns, and never exposes the journal or event store to the child loop. Durable batch tasks have no parent run journal and do not use this bridge.

Scheduled-task runtime note:

  • Scheduled background runs set context.non_interactive=true and therefore exclude ask_clarification from the lead-agent tool list. This keeps scheduler-triggered runs from stalling on human confirmation mid-execution. non_interactive is an internal-only context key: it is merged from body.context only when the request authenticated as the process-internal user (the scheduler path), never from arbitrary HTTP/IM clients.

Durable MCP task-management tools are added only while the process-local task submitter is installed. They expose bounded local task fields, including whether cancellation was requested, but never the remote handle. Cancellation records that request durably and returns immediately; the background service owns the remote call and retries. These remain ordinary business tools under an active skill's allowed-tools policy and must be declared explicitly.

Community tools (packages/harness/deerflow/community/): optional integrations, each in its own subpackage and wired through config.yaml. Documented examples:

  • tavily/ - Web search (5 results default) and web fetch (4KB limit)
  • sofya/ - Web search (5 results default, per-result content capped at 2000 chars) and web fetch (4KB limit)
  • jina_ai/ - Web fetch via Jina reader API with readability extraction
  • firecrawl/ - Web scraping via Firecrawl API
  • image_search/ - Image search via DuckDuckGo
  • aio_sandbox/ - Docker-based isolation (AioSandboxProvider)
  • browser_automation/ - Agentic browser control (stateful navigate → observe → click/type loop) via Playwright, distinct from the read-only web_fetch/web_capture tools. Tools: browser_navigate, browser_snapshot, browser_click, browser_type, browser_get_text, browser_back, browser_screenshot, browser_close (config group: browser). A process-local BrowserSessionManager owns one private, loop-affine Playwright event-loop thread (same pattern as the BoxLite provider) so a per-thread browser session survives across turns regardless of the caller's loop (Gateway / TUI / test). Each action returns a fresh page snapshot whose interactive elements are addressed by a stable numeric [ref] index (stamped as data-df-ref during snapshot), so the model acts on what it just observed instead of holding stale handles or guessing selectors. URLs are SSRF-screened via the shared validate_public_http_url (opt-out allow_private_addresses only for intentional internal targets). CDP attachment cannot install the request guard on an existing Chrome context, so cdp_url fails closed unless the operator explicitly sets allow_unguarded_cdp: true for a trusted local browser. Browser REST/Live access also requires an exact non-NULL thread owner, rather than the general legacy shared-thread policy, because retained pages may contain authenticated state. Session admission is a hard max_sessions cap: pinned Live/operation sessions are never evicted, and a new thread is rejected when no unpinned session can be closed; one Live viewer owns a session at a time. Optional dependency: cd backend && uv sync --extra browser && uv run playwright install chromium; scripts/detect_uv_extras.py preserves the extra when config.yaml enables browser_navigate, and Gateway startup fails fast if configured browser control cannot import Playwright. Tests: tests/test_browser_automation.py (mocked tools + a real-Chromium integration test guarded by importorskip); tests/manual_browser_live_check.py is a manual DeepSeek-driven end-to-end check (not collected by pytest). Live UI input dispatch is kept independent from JPEG capture: non-move actions start a rate-limited background refresh loop, so pointer, wheel, or keyboard input stays responsive while continuous gestures still produce frames throughout the interaction.

Additional providers also live here (boxlite, brave, browserless, crawl4ai, ddg_search, e2b_sandbox, exa, fastcrw, groundroute, infoquest, searxng, serper, serply, sofya, tencent_wsa, tenki); see each subpackage for specifics. tencent_wsa uses Tencent Cloud Web Search's service API key endpoint (TENCENTCLOUD_WSA_APIKEY), not Tencent Cloud SecretId/SecretKey signing. Its max_results is capped at 50; requests above 10 use Tencent's optional Cnt parameter, which needs a Tencent Cloud plan that supports it. E2B bootstrap is required. If it fails, the provider kills and closes the unusable remote sandbox. New sandbox creation raises an error. Warm-pool reclaim and remote discovery discard the sandbox and continue acquisition. E2B mounts remain optional.

E2B output sync records remote file versions and actual host file metadata in a thread-local manifest. The manifest binds to the remote sandbox ID. A complete output listing removes entries for deleted files. This avoids repeat downloads when the host filesystem rounds modification times. A single release-time sync pass is bounded by aggregate ceilings (_MAX_SYNC_TOTAL_BYTES, _MAX_SYNC_FILES, _SYNC_DEADLINE_SECONDS) on top of the per-file _MAX_DOWNLOAD_SIZE cap, so a pathological outputs tree cannot make release download unboundedly; a truncated pass logs what it dropped and leaves the manifest un-pruned (only entries observed in that pass are reconciled), so files it never reached are retried on the next release rather than being forgotten.

ACP agent tools:

  • invoke_acp_agent - Invokes external ACP-compatible agents from config.yaml
  • ACP launchers must be real ACP adapters. The standard codex CLI is not ACP-compatible by itself; configure a wrapper such as npx -y @zed-industries/codex-acp or an installed codex-acp binary
  • MiniMax Code speaks ACP directly: configure command: mcode with args: ["acp"]. It receives DeerFlow's enabled MCP servers and uses the per-thread ACP workspace; the Gateway process must have an authenticated mcode executable on PATH
  • ACP results collect only agent_message_chunk text. Thought chunks remain internal and must not be concatenated into the tool result
  • Missing ACP executables now return an actionable error message instead of a raw [Errno 2]
  • Each ACP agent uses a per-thread workspace at {base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/. The workspace is accessible to the lead agent via the virtual path /mnt/acp-workspace/ (read-only). In docker sandbox mode, the directory is volume-mounted into the container at /mnt/acp-workspace (read-only); in local sandbox mode, path translation is handled by tools.py