Jun 3a6e681dee
fix(view-image): read active sandbox images from sandbox (#5306)
* fix(view-image): read remote sandbox images from sandbox

* docs(tools): clarify view_image sandbox behavior

* fix(view-image): address sandbox lifecycle review

* fix(view-image): preserve image provenance across sandbox replacement

* fix(view-image): address provider recovery review

* fix(view-image): drain cancelled tool reads
2026-09-10 15:48:55 +08:00

11 KiB

Tool System (packages/harness/deerflow/tools/)

get_available_tools(groups, include_mcp, model_name, subagent_enabled) assembles:

  1. Config-defined tools - Resolved from config.yaml via resolve_variable()
  2. MCP tools - From enabled MCP servers (lazy initialized, cached with resolved-path + content-signature invalidation)
  3. Built-in tools:
    • present_files - Make output files visible to user (only /mnt/user-data/outputs); virtual paths use resolve_runtime_user_id(runtime) so validation resolves the same user-scoped outputs directory established by ThreadDataMiddleware
    • ask_clarification - Request clarification (intercepted by ClarificationMiddleware, which preserves text fallback and adds artifact.human_input for Web UI Human Input Cards). Beyond free text and single choice, the request-side v2 protocol supports fields (structured form card collecting several values at once; field types: text/textarea/number/select/multi_select/checkbox/date, validated and normalized server-side in the middleware — invalid entries are dropped, unknown types degrade to text; a standalone multi-select question is a one-field form). Replies stay on the v1 response protocol (text/option): the form card submits a readable text summary
    • view_image - Read image bytes for vision-capable models; live sandbox bytes win for the same sandbox generation, replacement-sandbox recovery uses only SHA-256-verified synchronized host bytes, and async tool invocation drains blocking reads before cancellation may release the sandbox lease
    • setup_agent - Bootstrap-only: persist a brand-new custom agent's SOUL.md and config.yaml. Bound only when is_bootstrap=True.
    • update_agent - Custom-agent-only: persist self-updates to the current agent's SOUL.md / config.yaml from inside a normal chat (partial update + atomic write). Bound when agent_name is set and is_bootstrap=False.
  4. Subagent tool (if enabled):
    • task - Delegate to subagent (prompt, subagent_type, optional acceptance_criteria, and an optional model-visible description used only as a short progress label). Execution never depends on description; lifecycle display falls back to prompt when a provider omits it. Subagent reports are self-reports: the docstring directs the lead to expect [rN] receipt citations and verifiable handles while verification.receipts_enabled (and explicitly qualifies that disabled receipts mean no citations and no citation verdict), to read the delegation ledger's citation cross-check as execution evidence only, and to attach acceptance_criteria for objectively checkable outcomes (canonical forms file:<path> exists|non-empty, file_written:<path>, tests_passed:<command>); criteria are handed to the executor and appended to the subagent's task message as untrusted data (see subagents/report_contract.py). Polling safety timeouts carry the latest published tool receipts into the terminal task metadata before requesting background cancellation.
    • batch_task, batch_status, cancel_batch - Explicit durable batch submission/progress/cancellation. Added only while the startup SQL-backed batch submitter is installed; large results stay in the owner-scoped API/JSONL export rather than the lead context. Items accept optional acceptance_criteria; item queries and exports expose the separate acceptance_verdict. Progress counts describe execution, not acceptance; unmet and UNVERIFIED conditions never trigger automatic retries.
    • Direct create_deerflow_agent integrations receive cloned tools bound to their explicit SubagentRuntime. The bound task forwards that runtime's exact execution controller and optional caller-owned AppConfig into registry/model/tool resolution and SubagentExecutor; bound batch tools use the same config snapshot and resolve only that runtime's submitter before falling back to no other application's active worker. Keep the original tool name/schema unchanged so model contracts and user-tool deduplication remain stable.

The ordinary task boundary carries one narrow parent-loop middleware recorder into the isolated subagent runtime under separate loop-detection and tool-promotion keys. It schedules only record_middleware calls back onto the loop that owns RunJournal, keeps an execution-local atomic promotion claim so parallel searches do not double-report one new schema, is fenced and drained once before task returns, and never exposes the journal or event store to the child loop. Durable batch tasks have no parent run journal and do not use this bridge.

Scheduled-task runtime note:

  • Scheduled background runs set context.non_interactive=true and therefore exclude ask_clarification from the lead-agent tool list. This keeps scheduler-triggered runs from stalling on human confirmation mid-execution. non_interactive is an internal-only context key: it is merged from body.context only when the request authenticated as the process-internal user (the scheduler path), never from arbitrary HTTP/IM clients.

Durable MCP task-management tools are added only while the process-local task submitter is installed. They expose bounded local task fields, including whether cancellation was requested, but never the remote handle. Cancellation records that request durably and returns immediately; the background service owns the remote call and retries. These remain ordinary business tools under an active skill's allowed-tools policy and must be declared explicitly.

Community tools (packages/harness/deerflow/community/): optional integrations, each in its own subpackage and wired through config.yaml. Documented examples:

  • tavily/ - Web search (5 results default) and web fetch (4KB limit)
  • sofya/ - Web search (5 results default, per-result content capped at 2000 chars) and web fetch (4KB limit)
  • jina_ai/ - Web fetch via Jina reader API with readability extraction
  • firecrawl/ - Web scraping via Firecrawl API
  • image_search/ - Image search via DuckDuckGo
  • aio_sandbox/ - Docker-based isolation (AioSandboxProvider)
  • browser_automation/ - Agentic browser control (stateful navigate → observe → click/type loop) via Playwright, distinct from the read-only web_fetch/web_capture tools. Tools: browser_navigate, browser_snapshot, browser_click, browser_type, browser_get_text, browser_back, browser_screenshot, browser_close (config group: browser). A process-local BrowserSessionManager owns one private, loop-affine Playwright event-loop thread (same pattern as the BoxLite provider) so a per-thread browser session survives across turns regardless of the caller's loop (Gateway / TUI / test). Each action returns a fresh page snapshot whose interactive elements are addressed by a stable numeric [ref] index (stamped as data-df-ref during snapshot), so the model acts on what it just observed instead of holding stale handles or guessing selectors. URLs are SSRF-screened via the shared validate_public_http_url (opt-out allow_private_addresses only for intentional internal targets). CDP attachment cannot install the request guard on an existing Chrome context, so cdp_url fails closed unless the operator explicitly sets allow_unguarded_cdp: true for a trusted local browser. Browser REST/Live access also requires an exact non-NULL thread owner, rather than the general legacy shared-thread policy, because retained pages may contain authenticated state. Session admission is a hard max_sessions cap: pinned Live/operation sessions are never evicted, and a new thread is rejected when no unpinned session can be closed; one Live viewer owns a session at a time. Optional dependency: cd backend && uv sync --extra browser && uv run playwright install chromium; scripts/detect_uv_extras.py preserves the extra when config.yaml enables browser_navigate, and Gateway startup fails fast if configured browser control cannot import Playwright. Tests: tests/test_browser_automation.py (mocked tools + a real-Chromium integration test guarded by importorskip); tests/manual_browser_live_check.py is a manual DeepSeek-driven end-to-end check (not collected by pytest). Live UI input dispatch is kept independent from JPEG capture: non-move actions start a rate-limited background refresh loop, so pointer, wheel, or keyboard input stays responsive while continuous gestures still produce frames throughout the interaction.

Additional providers also live here (boxlite, brave, browserless, crawl4ai, ddg_search, e2b_sandbox, exa, fastcrw, groundroute, infoquest, searxng, serper, serply, sofya, tencent_wsa, tenki); see each subpackage for specifics. tencent_wsa uses Tencent Cloud Web Search's service API key endpoint (TENCENTCLOUD_WSA_APIKEY), not Tencent Cloud SecretId/SecretKey signing. Its max_results is capped at 50; requests above 10 use Tencent's optional Cnt parameter, which needs a Tencent Cloud plan that supports it. E2B bootstrap is required. If it fails, the provider kills and closes the unusable remote sandbox. New sandbox creation raises an error. Warm-pool reclaim and remote discovery discard the sandbox and continue acquisition. E2B mounts remain optional.

E2B output sync records remote file versions and actual host file metadata in a thread-local manifest. The manifest binds to the remote sandbox ID. A complete output listing removes entries for deleted files. This avoids repeat downloads when the host filesystem rounds modification times. A single release-time sync pass is bounded by aggregate ceilings (_MAX_SYNC_TOTAL_BYTES, _MAX_SYNC_FILES, _SYNC_DEADLINE_SECONDS) on top of the per-file _MAX_DOWNLOAD_SIZE cap, so a pathological outputs tree cannot make release download unboundedly; a truncated pass logs what it dropped and leaves the manifest un-pruned (only entries observed in that pass are reconciled), so files it never reached are retried on the next release rather than being forgotten.

ACP agent tools:

  • invoke_acp_agent - Invokes external ACP-compatible agents from config.yaml
  • ACP launchers must be real ACP adapters. The standard codex CLI is not ACP-compatible by itself; configure a wrapper such as npx -y @zed-industries/codex-acp or an installed codex-acp binary
  • MiniMax Code speaks ACP directly: configure command: mcode with args: ["acp"]. It receives DeerFlow's enabled MCP servers and uses the per-thread ACP workspace; the Gateway process must have an authenticated mcode executable on PATH
  • ACP results collect only agent_message_chunk text. Thought chunks remain internal and must not be concatenated into the tool result
  • Missing ACP executables now return an actionable error message instead of a raw [Errno 2]
  • Each ACP agent uses a per-thread workspace at {base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/. The workspace is accessible to the lead agent via the virtual path /mnt/acp-workspace/ (read-only). In docker sandbox mode, the directory is volume-mounted into the container at /mnt/acp-workspace (read-only); in local sandbox mode, path translation is handled by tools.py