diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 391ea2d581..58cfde97c1 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -19,71 +19,44 @@ The backend runs a LangGraph-based super agent with sandbox execution, persisten - Run-history `status` filters are occurrence states, not task states. `ScheduledTaskRunStatus` in `persistence/scheduled_tasks/model.py` is the shared API/repository vocabulary and must match the active and terminal occurrence-status sets. Keep owner lookup before reading history, and apply SQL task/status predicates before pagination; omitted status preserves the existing response. - The background scheduler is single-instance by default. `scheduler.multi_instance=true` opts into lease-aware recovery across Gateway instances and requires shared Postgres, `run_ownership.heartbeat_enabled=true`, and `run_events.backend=db`; otherwise startup rejects the configuration. Live scheduled runs are preserved when a peer starts; expired launch claims return to the durable queue, expired run leases are atomically taken over, stale launch writes are fenced by lease ownership, and the Postgres advisory-locked budget makes `max_concurrent_runs` a shared global cap for `launching`/`running` rows. - The multi-process startup gate also fires on `deployment.multi_instance: true`; contract in `docs/CONFIGURATION.md`. -- Long-running MCP work uses a separate durable task runtime (`McpTaskService` + `mcp_tasks`, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and `ThreadState` receives only a bounded current-thread projection. Full contract (leases, cancellation fencing, delivery idempotency, management-tool exposure): [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md). -- MCP task notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 are part of that same contract — see [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md). +- Long-running MCP work uses a separate durable task runtime (`McpTaskService` + `mcp_tasks`, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and `ThreadState` receives only a bounded current-thread projection. Full contract — including notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 (leases, cancellation fencing, delivery idempotency, management-tool exposure): [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md). - Scheduled-task dispatch permits one active occurrence per task via `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). Durable `queued` rows survive restarts; only lease-fenced `launching` may call Gateway launch; `running` references the durable run. Stable admission idempotency keys reuse that run after recovery. Reused-thread `ConflictError` returns `launching` to `queued`; other launch errors become `failed`. Atomic queue claims enforce `max_concurrent_runs` and the per-owner cap, excluding waiting rows; the budget count and its UPDATE are separate statements, so writers must serialize before the count (Postgres advisory lock; SQLite `BEGIN IMMEDIATE`, whose deferred transaction otherwise reserves the writer only at the UPDATE) or claims on distinct rows overshoot the cap. Repeated triggers coalesce; same-thread FIFO blocks behind older active rows. Queue admission, PATCH/resume, pause and delete lock the parent before the occurrence, freezing active task definitions. Pause/delete atomically cancel `queued` work but reject `launching`/`running`; PATCH/resume reject all active states. Only queued conflicts offer pause cancellation. Manual triggers may queue/run while paused. - Scheduled-task lifecycle, recovery and queue timeout: [persistence guide](packages/harness/deerflow/persistence/AGENTS.md#scheduled-task-lifecycle). - `POST /api/scheduled-tasks/preview-cron` requires authenticated `threads:read`. Bounded cron previews call the shared scheduler calculator in `asyncio.to_thread`, preserving its DST semantics. Capture the optional aware reference once; return UTC and offset-bearing local occurrences without acquiring task/thread/run stores or dispatching work. This advisory API does not reserve execution. - Gateway MCP/skill updates write `extensions_config.json` at runtime. Production Compose mounts it read-write (`config.yaml` stays `:ro`); Helm seeds a writable home-volume copy. Hold both `extensions_config_write_lock` and the sidecar advisory `extensions_config_file_lock` during read-modify-write to prevent lost updates across workers. `atomic_write_extensions_config` uses temp-file replacement; only mount-point `EBUSY` permits a non-atomic in-place overwrite (crashes can truncate). Warn about this fallback once per target; propagate other errors. `get_extensions_config()` revalidates reads by file signature; see [config caching](packages/harness/deerflow/config/AGENTS.md). Tests: `tests/test_compose_extensions_config_writable.py`, `tests/test_extensions_config_atomic_write.py`, `tests/test_helm_extensions_config_writable.py`. - MCP cache reset scope and discovery fencing: [MCP guide](packages/harness/deerflow/mcp/AGENTS.md). -**Project Structure**: +**Project Structure** (backend/ subtree; the root guide maps the full repo): ``` -deer-flow/ -├── Makefile # Root commands (check, install, dev, stop) -├── config.yaml # Main application configuration -├── extensions_config.json # MCP servers and skills configuration -├── backend/ # Backend application (this directory) -│ ├── Makefile # Backend-only commands (dev, gateway, lint) -│ ├── langgraph.json # LangGraph Studio graph configuration -│ ├── packages/ -│ │ ├── extension-api/ # public, host-independent extension contracts (import: deerflow_extension_api.*) -│ │ └── harness/ # deerflow-harness package (import: deerflow.*) -│ │ ├── pyproject.toml -│ │ └── deerflow/ -│ │ ├── agents/ # LangGraph agent system -│ │ │ ├── lead_agent/ # Main agent (factory + system prompt) -│ │ │ ├── middlewares/ # middleware components (see Middleware Chain section) -│ │ │ ├── memory/ # Memory extraction, queue, prompts -│ │ │ └── thread_state.py # ThreadState schema -│ │ ├── sandbox/ # Sandbox execution system -│ │ │ ├── local/ # Local filesystem provider -│ │ │ ├── sandbox.py # Abstract Sandbox interface -│ │ │ ├── tools.py # bash, ls, read/write/str_replace -│ │ │ └── middleware.py # Sandbox lifecycle management -│ │ ├── subagents/ # Subagent delegation system -│ │ │ ├── builtins/ # general-purpose, bash agents -│ │ │ ├── executor.py # Background execution engine -│ │ │ └── registry.py # Agent registry -│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image, review_skill_package) -│ │ ├── mcp/ # MCP integration (tools, cache, client) -│ │ ├── integrations/ # Managed first-party integration installers (e.g. Lark CLI skill pack) -│ │ ├── extensions/ # Python plugin loader, registry, placement, and isolation -│ │ ├── models/ # Model factory with thinking/vision support -│ │ ├── skills/ # Skills discovery, loading, parsing -│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.) -│ │ ├── community/ # Community tools (search/fetch/scrape, image search, AIO sandbox) -│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class) -│ │ ├── utils/ # Utilities (network, readability) -│ │ └── client.py # Embedded Python client (DeerFlowClient) -│ ├── app/ # Application layer (import: app.*) -│ │ ├── gateway/ # FastAPI Gateway API -│ │ │ ├── app.py # FastAPI application -│ │ │ └── routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels) -│ │ └── channels/ # IM platform integrations -│ ├── scripts/benchmark/ # Standalone reproducible backend benchmarks -│ ├── tests/ # Test suite -│ └── docs/ # Documentation -├── frontend/ # Next.js frontend application -└── skills/ # Agent skills directory - ├── public/ # Public skills (committed) - └── custom/ # Custom skills (gitignored) +backend/ +├── Makefile # Backend-only commands (dev, gateway, lint) +├── langgraph.json # LangGraph Studio graph configuration +├── packages/ +│ ├── extension-api/ # public, host-independent extension contracts (import: deerflow_extension_api.*) +│ └── harness/ # deerflow-harness package (import: deerflow.*) +│ └── deerflow/ +│ ├── agents/ # LangGraph agent system (lead_agent/, middlewares/ — see Middleware Chain, memory/, thread_state.py) +│ ├── sandbox/ # Sandbox execution system +│ ├── subagents/ # Subagent delegation (builtins/, executor.py, registry.py) +│ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image, review_skill_package) +│ ├── mcp/ # MCP integration (tools, cache, client) +│ ├── integrations/ # Managed first-party integration installers (e.g. Lark CLI skill pack) +│ ├── extensions/ # Python plugin loader, registry, placement, and isolation +│ ├── models/ # Model factory with thinking/vision support +│ ├── skills/ # Skills discovery, loading, parsing +│ ├── config/ # Configuration system +│ ├── community/ # Community tools (search/fetch/scrape, image search, AIO sandbox) +│ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class) +│ ├── utils/ # Utilities (network, readability) +│ └── client.py # Embedded Python client (DeerFlowClient) +├── app/ # Application layer (import: app.*) +│ ├── gateway/ # FastAPI Gateway API (app.py, routers/) +│ └── channels/ # IM platform integrations +├── scripts/benchmark/ # Standalone reproducible backend benchmarks +├── tests/ # Test suite +└── docs/ # Documentation ``` -ATX outline closing markers use a linear suffix scan; do not use unanchored -whitespace regex searches on unbounded uploaded headings. The long-heading -regression exercises the production extractor under a generous process deadline. - ## Important Development Guidelines ### Documentation Update Policy @@ -275,10 +248,7 @@ PYTHONPATH=. uv run pytest tests/test_.py -v Keep live tests opt-in via `DEER_FLOW_RUN_LIVE_TESTS=1`; guard POSIX-only markers with `os.name` for Windows collection. -Jina logging tests use dummy keys (`tests/test_jina_client.py`). -The Jina API redirect regression uses an offline `httpx.MockTransport` and checks that temporary redirects keep the POST method. -Jina/Browserless/InfoQuest resolve URLs without rebuilding HTML. -InfoQuest connect/read timeout is 30s, separate from crawl timeouts (`tests/test_infoquest_http_timeout.py`). +Jina logging tests use dummy keys; the Jina redirect regression runs offline via `httpx.MockTransport` and keeps POST across temporary redirects; InfoQuest connect/read timeout is 30s, separate from crawl timeouts (`tests/test_jina_client.py`, `tests/test_infoquest_http_timeout.py`). ### Running the Full Application @@ -296,11 +266,6 @@ Run `make dev` from the repo root to start all services at `http://localhost:202 | **Stop** | `./scripts/serve.sh --stop`
`make stop` | `./scripts/docker.sh stop`
`make docker-stop` | `./scripts/deploy.sh down`
`make down` | | **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — | -**Nginx routing**: -- `/api/langgraph/*` → Gateway embedded runtime (8001), rewritten to `/api/*` -- `/api/*` (other) → Gateway API (8001) -- `/` (non-API) → Frontend (3000) - ### Running Backend Services Separately From the **backend** directory: @@ -339,18 +304,12 @@ Title fallback: result URL, then request URL. ### File Upload -Outlines use ATX syntax (1–6 hashes, space/tab separator, ≤3 leading spaces), strip closing hashes and skip fenced code. -- Endpoint: `POST /api/threads/{thread_id}/uploads` -- Supports: PDF, PPT, Excel, Word documents (converted via `markitdown`) -- Rejects directories before copying to keep uploads all-or-nothing -- One conversion worker per request when called from an active event loop -- Files stored in thread-isolated directories under the resolving user's bucket (`users/{user_id}/threads/{thread_id}/user-data/uploads`). For IM channels the owner is threaded explicitly via the `user_id=` kwarg (see IM Channels → Owner-scoped file storage); HTTP/embedded callers resolve it from `get_effective_user_id()` -- Per-thread `upload-companions/`: source mtime/ctime. -- Duplicate filenames within one request get `_N` suffixes to prevent overwrites. -- Gateway HTTP uploads stage `.upload-*.part` files, hidden from upload listings, agent context, and sandbox listings/searches. After size validation, publication is atomic; staged-name cleanup logs errors and leaves leftovers for startup sweep. The sweep keeps lone `.part` files under 24h: they may be in flight on another replica; `st_nlink > 1` means a published alias, removed at any age so the multi-link check can still replace the file. -- Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor. Non-mounted sandbox uploads acquire sandboxes with `SandboxProvider.acquire_async()` and offload `read_bytes()` plus `sandbox.update_file()` together. -- Mounted uploads skip sandbox acquire/sync. AIO remote/provisioner requires accurate `sandbox.thread_data_mounts: true`; omission keeps backend auto-detection. -- `UploadsMiddleware` caps outline titles at 200 characters and previews at 2000 including markers. Titles use `original_user_content`, not upload-prefixed content; attachment-only titles use a sanitized, bounded filename or count. +Outlines use ATX syntax (1–6 hashes, space/tab separator, ≤3 leading spaces), strip closing hashes and skip fenced code; closing markers use a linear suffix scan — never unanchored whitespace regex on unbounded headings. +- Endpoint: `POST /api/threads/{thread_id}/uploads`; PDF/PPT/Excel/Word via `markitdown`, one conversion worker per request; directories rejected, keeping uploads all-or-nothing; duplicate filenames in one request get `_N` suffixes. +- Files live in thread-isolated `users/{user_id}/threads/{thread_id}/user-data/uploads` (IM channels thread the owner via the `user_id=` kwarg — see IM Channels → Owner-scoped file storage; HTTP/embedded callers resolve `get_effective_user_id()`); per-thread `upload-companions/` holds source mtime/ctime. +- Gateway HTTP uploads stage `.upload-*.part` files hidden from upload listings, agent context, and sandbox listings/searches; publication is atomic after size validation. Startup sweep keeps lone `.part` files under 24h (possibly in flight on another replica); `st_nlink > 1` means a published alias, removed at any age so the multi-link check can still replace the file. +- Upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io` (ContextVar-preserving); non-mounted sandbox uploads use `SandboxProvider.acquire_async()` and offload `read_bytes()` + `sandbox.update_file()` together. Mounted uploads skip acquire/sync; AIO remote/provisioner requires accurate `sandbox.thread_data_mounts: true`, omission keeps backend auto-detection. +- `UploadsMiddleware` caps outline titles at 200 characters and previews at 2000 including markers. Titles use `original_user_content`; attachment-only titles use a sanitized, bounded filename or count. See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details. @@ -366,27 +325,17 @@ Interaction-sensitive changes must follow [policy](docs/RUN_INTERACTION_POLICY.m ### Context Summarization -Automatic conversation summarization when approaching token limits: -- Configured in `config.yaml` under `summarization` key -- Trigger types: tokens, messages, or fraction of max input -- Keeps recent messages while summarizing older ones -- Manual compaction uses `POST /api/threads/{id}/compact`, reuses the same - `DeerFlowSummarizationMiddleware`, writes a new checkpoint with updated - `messages` and `summary_text`, and bumps only those channel versions. - The route uses the shared `reserve_checkpoint_write()` boundary (also used by - manual state updates). Its short-lived `checkpoint_write` thread operation - shares the durable active-thread uniqueness constraint with run admission, - preventing either worker-local or cross-worker checkpoint-write races. +Automatic conversation summarization when approaching token limits, configured +under the `summarization` key in `config.yaml` (triggers: tokens, messages, or +fraction of max input); recent messages are kept while older ones are summarized. +- Manual compaction uses `POST /api/threads/{id}/compact`, reuses the same `DeerFlowSummarizationMiddleware`, writes a new checkpoint with updated `messages`/`summary_text`, and bumps only those channel versions. The route uses the shared `reserve_checkpoint_write()` boundary (also used by manual state updates); its short-lived `checkpoint_write` operation shares the durable active-thread uniqueness constraint with run admission, preventing worker-local and cross-worker checkpoint-write races. - Cache only first-candidate no-ops; never suppress primary retries with fallback results. See [reuse and telemetry](docs/summarization.md#reuse-and-telemetry) for cache and counter contracts. See [docs/summarization.md](docs/summarization.md) for details. ### Vision Support -For models with `supports_vision: true`: -- `ViewImageMiddleware` processes images in conversation -- `view_image_tool` added to agent's toolset -- Images are converted to base64 and appended to the model request as a hidden message carrying both a reserved ID prefix and a server-owned metadata marker; Gateway strips that marker from untrusted input, and the middleware requires both identifiers to recognize its own message. The middleware injects inside `wrap_model_call`, so the payload never enters graph state: checkpoints retain only lightweight `viewed_images` metadata, while client-chosen IDs survive. It also sweeps its own message out of every request before rebuilding it, so a payload stranded in an older checkpoint by an interrupted run stops being resent +For models with `supports_vision: true`: `ViewImageMiddleware` processes images and adds `view_image_tool`. Images reach the model as base64 in a hidden message carrying a reserved ID prefix plus a server-owned metadata marker (Gateway strips the marker from untrusted input; the middleware requires both to recognize its own message). The injection happens inside `wrap_model_call`, so payloads never enter graph state — checkpoints keep only lightweight `viewed_images` metadata, and client-chosen IDs survive; the middleware also sweeps its own message out of every request before rebuilding it, so a payload stranded in an older checkpoint by an interrupted run stops being resent. ## Code Style diff --git a/backend/packages/harness/deerflow/agents/factory.py b/backend/packages/harness/deerflow/agents/factory.py index 0d040878c3..6a5e80bd74 100644 --- a/backend/packages/harness/deerflow/agents/factory.py +++ b/backend/packages/harness/deerflow/agents/factory.py @@ -213,7 +213,7 @@ def _assemble_from_features( ) -> tuple[list[AgentMiddleware], list[BaseTool]]: """Build an ordered middleware chain + extra tools from *feat*. - Middleware order matches ``make_lead_agent`` (14 middlewares): + Middleware order matches ``make_lead_agent`` (18 middlewares): 0-2. Sandbox infrastructure (ThreadData → Uploads → Sandbox) 3. DanglingToolCallMiddleware (always) @@ -228,7 +228,9 @@ def _assemble_from_features( 10. ViewImageMiddleware (vision feature) 11. SubagentLimitMiddleware (subagent feature) 12. LoopDetectionMiddleware (loop_detection feature) - 13. ClarificationMiddleware (always last) + 13. TokenBudgetMiddleware (token_budget feature) + 13a. ModelContentCompatibilityMiddleware (always) + 14. ClarificationMiddleware (always last) Two-phase ordering: 1. Built-in chain — fixed sequential append. @@ -435,6 +437,19 @@ def _assemble_from_features( # ``enabled`` defaults to False for config.yaml; ``token_budget=True`` is the opt-in. chain.append(TokenBudgetMiddleware.from_config(TokenBudgetConfig(enabled=True))) + # --- [13a] ModelContentCompatibility (always) --- + # Read-time sanitizer: rewrites URL-sourced file/image blocks in list-form + # message content into text placeholders in the request view only, so + # threads poisoned before the conversion-layer fix heal on the next model + # call. Placement contract: after every built-in content transform so it + # sees the near-final request, and before the custom/extension injection + # point (_insert_extra) and the terminal tail — in the lead chain it is + # followed by configured extensions, terminal-response and finish-reason + # middlewares, and the always-last ClarificationMiddleware. + from deerflow.agents.middlewares.model_content_compatibility_middleware import ModelContentCompatibilityMiddleware + + chain.append(ModelContentCompatibilityMiddleware()) + # --- [14] Clarification (always last among built-ins) --- chain.append(ClarificationMiddleware()) extra_tools.append(ask_clarification_tool) diff --git a/backend/packages/harness/deerflow/agents/lead_agent/agent.py b/backend/packages/harness/deerflow/agents/lead_agent/agent.py index 701edadf99..65bdda1486 100644 --- a/backend/packages/harness/deerflow/agents/lead_agent/agent.py +++ b/backend/packages/harness/deerflow/agents/lead_agent/agent.py @@ -40,6 +40,7 @@ from deerflow.agents.middlewares.clarification_middleware import ClarificationMi from deerflow.agents.middlewares.configured_extensions import load_configured_extension_middlewares from deerflow.agents.middlewares.loop_detection_middleware import LoopDetectionMiddleware from deerflow.agents.middlewares.memory_middleware import MemoryMiddleware +from deerflow.agents.middlewares.model_content_compatibility_middleware import ModelContentCompatibilityMiddleware from deerflow.agents.middlewares.model_length_finish_reason_middleware import ModelLengthFinishReasonMiddleware from deerflow.agents.middlewares.safety_finish_reason_middleware import SafetyFinishReasonMiddleware from deerflow.agents.middlewares.subagent_limit_middleware import SubagentLimitMiddleware @@ -479,6 +480,7 @@ Being proactive with task management demonstrates thoroughness and ensures all r # MemoryMiddleware queues conversation for memory update (after TitleMiddleware) # ViewImageMiddleware should be before ClarificationMiddleware to inject image details before LLM # ToolErrorHandlingMiddleware should be before ClarificationMiddleware to convert tool exceptions to ToolMessages +# ModelContentCompatibilityMiddleware sanitizes model-incompatible URL content blocks in the request view; register near the end so it sees the near-final message list # ClarificationMiddleware should be last to intercept clarification requests after model calls def build_middlewares( config: RunnableConfig, @@ -752,6 +754,15 @@ def build_middlewares( if token_budget_config.enabled and resolved_app_config.token_usage.enabled: middlewares.append(CompletedSubagentUsageMiddleware()) + # Downgrade persisted URL-sourced file/image blocks (pre-fix MCP + # ResourceLink results) that the Chat Completions serializer rejects, so + # threads bricked before the conversion-layer fix heal at read time. + # Request view only: state and checkpoints are never written. Appended + # after every built-in content transform so it sanitizes the near-final + # message list, and outer of the MODEL_PHYSICAL extension anchor so + # extensions placed there keep their final-request guarantee. + middlewares.append(ModelContentCompatibilityMiddleware()) + # Inject custom middlewares before ClarificationMiddleware if custom_middlewares: middlewares.extend(custom_middlewares) diff --git a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md index 4f32b00e3a..e0df907e3e 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md @@ -36,6 +36,9 @@ SystemMessageCoalescing, ViewImage, SkillActivation. Summarization/Title use `SystemOperationKind.SUMMARIZATION`/`.TITLE`; summaries enter via DurableContext's `durable_context_data`, memory recall via DynamicContext's `dynamic_context_memory`. Memory only queues extraction. +Rewrites that only modify the model-bound request view and never enter state do +not require provenance stamps (precedents: InputSanitizationMiddleware, +DanglingToolCallMiddleware, ModelContentCompatibilityMiddleware). **Self-description.** Configurable middleware exposes JSON-serialisable `release_policy_parameters()` (`ReleasePolicyProvider`, duck typed). Update @@ -129,12 +132,13 @@ Before changing a later authorization phase, read the [authorization RFC](../../ [Loop detection lifecycle](../../../../../docs/LOOP_DETECTION.md) for fallback identity, cleanup/LRU/reset, severity ordering, and test invariants. 33. **TokenBudgetMiddleware** - `token_budget.enabled`: run-ID budgets survive continuations; missing/invalid IDs reset invocation state. With token usage tracking, place CompletedSubagentUsageMiddleware after the guard: reversed `after_model` hooks backfill terminal child usage before enforcement. Keep TokenUsageMiddleware earlier so current-step attribution reflects calls retained by all guards. -34. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail -35. **Configured extension middlewares** - `extensions.middlewares` in `config.yaml` or `extensions_config.json` optionally accepts `module.path:ClassName` strings or `{class, kwargs}` objects. `deerflow.reflection.resolve_class` loads `AgentMiddleware` classes; import, class, and constructor errors fail agent creation. `kwargs` must be JSON-compatible; YAML dates/timestamps become ISO strings. Order: built-ins/custom and loop/token guards → extensions → terminal-response/safety/clarification tail. Subagents share the list before their safety tail; separate lead/subagent lists are unsupported. Trusted operator config only: paths instantiate arbitrary code. Gateway skill/MCP toggles preserve it in raw JSON; adding an API write path requires explicit trust-boundary review. -36. **TerminalResponseMiddleware** - When a provider returns an empty terminal `AIMessage` after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success -37. **ModelLengthFinishReasonMiddleware** - Match stamps `stop_reason=model_length_capped` and ends tool loop. Suppresses calls, appends notice even with partial text, and stamps `model_length_termination` so downstream guards stand down. Preserves content blocks. -38. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first -39. **ClarificationMiddleware** - Intercepts `ask_clarification`, writes a readable `ToolMessage.content` fallback plus a structured `ToolMessage.artifact.human_input` payload, and interrupts via `Command(goto=END)` (must be last). `after_model` drops same-turn sibling tool calls so they cannot run before the user answers; a malformed `ask_clarification` parked on `invalid_tool_calls` is the same stop signal. `disable_clarification` runs keep the siblings. Payloads are versioned — legacy `free_text`/`choice_with_other` stay `version: 1`; the v2 `form` mode (from `fields`) is `version: 2` so older frontends reject it and fall back to plain text. Field normalization is deterministic and lives in the middleware (it short-circuits before tool execution, so tool-arg typing gives no runtime validation), and it is atomic: any structurally broken entry — non-dict, bad/duplicate name, a name colliding with a JS `Object.prototype` member (`__proto__`/`constructor`), or exceeding the caps (16 fields / 24 options per field / 200 chars per text / `MAX_FORM_SERIALIZED_BYTES` = 16KB UTF-8, the per-item caps alone admitting forms whose IM text fallback overruns channel limits) — degrades the whole form to the legacy option/free-text modes, so a card never renders "complete" while missing a field. Benign issues degrade locally (unknown types — incl. unhashable JSON like `type: []`, which must not raise from the membership probe — and option-less selects become `text`); options are trimmed/deduped with blanks dropped (form- and top-level) since the frontend rejects blank labels. XML-to-dict option payloads are recursively flattened from dict/list containers in source order, scalar leaves kept, residual XML tags stripped before that trimming. Checkboxes are booleans defaulting to "no"; `required` on one means consent semantics. The response protocol is unchanged (v1 `text`/`option`): form cards submit a text summary as `response_kind: "text"`, so journal persistence needs no new allowlist entries. `RunJournal` reconciles visible, unpersisted `ToolMessage`s for current-run lead-agent calls at the next lead-agent `on_chat_model_start` after consumption, or at successful root `on_chain_end`. This preserves any middleware short-circuit, including blocked writes, across later model errors and checkpoint compaction (#4666). `_remember_current_run_tool_calls` excludes subagents; their results stay in `subagent.step`. Human Input Card replies are `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden sources (currently `ask_clarification`) as `llm.human.input`. +34. **ModelContentCompatibilityMiddleware** - Read-time sanitizer (`wrap_model_call`/`awrap_model_call`, request view only; never writes state): rewrites URL-sourced `file` blocks (any role, gated on the `url` key; an http(s) URL is preserved when the selected model's serializer accepts it — OpenAI Responses API or Anthropic) and non-fetchable-scheme (non-`http`/`https`/`data`) `image` blocks into text placeholders whose MIME/location metadata passes the shared `neutralize_untrusted_tags` guard, so pre-MCP-fix checkpoints heal on the next turn with no migration. Uses the conversion layer's `resource_placeholder_text`/`model_visible_location` for model-visible text; artifact capture scans state messages, so healing adds neither `resource_links` nor handles. See the MCP guide for conversion and the module docstring for healing; malformed URLs degrade to location-less placeholders instead of raising. Registered after DurableContext/summarization, before the date/coalescer tail. +35. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail +36. **Configured extension middlewares** - `extensions.middlewares` in `config.yaml` or `extensions_config.json` optionally accepts `module.path:ClassName` strings or `{class, kwargs}` objects. `deerflow.reflection.resolve_class` loads `AgentMiddleware` classes; import, class, and constructor errors fail agent creation. `kwargs` must be JSON-compatible; YAML dates/timestamps become ISO strings. Order: built-ins/custom and loop/token guards → extensions → terminal-response/safety/clarification tail. Subagents share the list before their safety tail; separate lead/subagent lists are unsupported. Trusted operator config only: paths instantiate arbitrary code. Gateway skill/MCP toggles preserve it in raw JSON; adding an API write path requires explicit trust-boundary review. +37. **TerminalResponseMiddleware** - When a provider returns an empty terminal `AIMessage` after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success +38. **ModelLengthFinishReasonMiddleware** - Match stamps `stop_reason=model_length_capped` and ends tool loop. Suppresses calls, appends notice even with partial text, and stamps `model_length_termination` so downstream guards stand down. Preserves content blocks. +39. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first +40. **ClarificationMiddleware** - Intercepts `ask_clarification`, writes a readable `ToolMessage.content` fallback plus a structured `ToolMessage.artifact.human_input` payload, and interrupts via `Command(goto=END)` (must be last). `after_model` drops same-turn sibling tool calls so they cannot run before the user answers; a malformed `ask_clarification` parked on `invalid_tool_calls` is the same stop signal. `disable_clarification` runs keep the siblings. Payloads are versioned — legacy `free_text`/`choice_with_other` stay `version: 1`; the v2 `form` mode (from `fields`) is `version: 2` so older frontends reject it and fall back to plain text. Field normalization is deterministic and lives in the middleware (it short-circuits before tool execution, so tool-arg typing gives no runtime validation), and it is atomic: any structurally broken entry — non-dict, bad/duplicate name, a name colliding with a JS `Object.prototype` member (`__proto__`/`constructor`), or exceeding the caps (16 fields / 24 options per field / 200 chars per text / `MAX_FORM_SERIALIZED_BYTES` = 16KB UTF-8, the per-item caps alone admitting forms whose IM text fallback overruns channel limits) — degrades the whole form to the legacy option/free-text modes, so a card never renders "complete" while missing a field. Benign issues degrade locally (unknown types — incl. unhashable JSON like `type: []`, which must not raise from the membership probe — and option-less selects become `text`); options are trimmed/deduped with blanks dropped (form- and top-level) since the frontend rejects blank labels. XML-to-dict option payloads are recursively flattened from dict/list containers in source order, scalar leaves kept, residual XML tags stripped before that trimming. Checkboxes are booleans defaulting to "no"; `required` on one means consent semantics. The response protocol is unchanged (v1 `text`/`option`): form cards submit a text summary as `response_kind: "text"`, so journal persistence needs no new allowlist entries. `RunJournal` reconciles visible, unpersisted `ToolMessage`s for current-run lead-agent calls at the next lead-agent `on_chat_model_start` after consumption, or at successful root `on_chain_end`. This preserves any middleware short-circuit, including blocked writes, across later model errors and checkpoint compaction (#4666). `_remember_current_run_tool_calls` excludes subagents; their results stay in `subagent.step`. Human Input Card replies are `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden sources (currently `ask_clarification`) as `llm.human.input`. Inline `skill_references` accepts 1–16 names; empty lists retain slash/plain flow. Resolve the whole batch through user storage, enabled state and agent allowlist diff --git a/backend/packages/harness/deerflow/agents/middlewares/model_content_compatibility_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/model_content_compatibility_middleware.py new file mode 100644 index 0000000000..3ad2811599 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/middlewares/model_content_compatibility_middleware.py @@ -0,0 +1,235 @@ +"""Read-time sanitizer for model-incompatible content blocks in tool results. + +The MCP tool-result conversion layer once persisted URL-sourced ``file`` +content blocks (MCP ``ResourceLink`` results: ``ui://`` MCP App cards, local +files at ``/mnt/user-data/...`` virtual paths, remote non-image links) into +checkpointed ``ToolMessage`` content. When the selected model uses OpenAI Chat +Completions, every subsequent call fails during client-side message serialization: +langchain-core's OpenAI translator raises ``ValueError: OpenAI Chat Completions +does not support file URLs.`` for ANY ``file`` block carrying a ``url`` — the +Chat Completions API accepts file blocks only as base64 or file-id. The agent +loop classifies that as a generic, non-retriable failure, so every subsequent +turn answers with a fallback error: +the thread is permanently bricked. + +The conversion layer no longer emits those blocks for new results. This +middleware covers the other time window — history written before the fix — by +rewriting the *request view* at the model boundary: + +- a ``{"type": "file"}`` block carrying a ``url`` KEY becomes a plain text + placeholder — ``[Resource ({mime or "unknown type"}) available at {url}]`` + when the URL is location-safe, ``[Resource ({mime or "unknown type"})]`` + otherwise (langchain-core raises on the key, not the value) — unless the + selected model's serializer accepts http(s) file URLs (OpenAI Responses + API ``input_file`` / Anthropic ``document`` URL), in which case the block + is kept; +- an ``{"type": "image", "url": ...}`` block whose URL scheme the provider + cannot fetch (anything outside ``http``/``https``/``data``) becomes the same + placeholder — local images are meant to reach the model through the + ``view_image`` tool, not as unresolvable URLs. + +Placeholder text comes from ``resource_placeholder_text`` in +``deerflow.tools.resource_placeholder`` — the same formatter the conversion +layer uses — so a thread healed at read time shows the model the exact +placeholder shape a fresh conversion would have produced. The location segment +goes through the shared ``model_visible_location`` gate, also from that module: +virtual paths and remote schemes stay visible, while a raw ``file://`` URI, a +bare/Windows host path, or a ``data:``/``blob:`` URI (pre-fix blocks could carry +megabytes of inline base64 in a file URL) is withheld from model-visible text — +the same rules the conversion layer applies to fresh results. Interpolated +MIME/location metadata passes through the shared ``neutralize_untrusted_tags`` +guard: this middleware runs after the input/tool-result sanitization passes, +and historical URL blocks never went through them (non-text content is left +untouched), so unguarded ````-style tags or boundary markers +in persisted metadata would otherwise reach the model verbatim. + +The rewrite hooks ``wrap_model_call``/``awrap_model_call`` and hands the +handler an overridden request, exactly like ``ViewImageMiddleware``: nothing is +written to state, so checkpoints keep the original blocks (artifact capture and +other state readers are unaffected) and a poisoned thread heals itself on its +next turn with no migration. Artifact capture scans the original state messages, +so it never sees these request-only placeholders. The healer creates neither +``resource_links`` entries nor artifact handles; any captured references come +from the original tool result under the registry's own referenceability rules. +URLs withheld from the placeholder remain in the original checkpoint blocks; +the healer does not expose them through a new artifact entry. + +The MCP conversion layer separately preserves downgraded ``ResourceLink`` +results in ``ToolMessage.artifact["resource_links"]``, including unresolved host +locations, except ``data:``/``blob:`` links, which it drops. This structured +preservation happens during conversion of new results. + +Scope is deliberately role-agnostic: every request message with list-form +content is rewritten, not just ``ToolMessage`` history. User uploads are +verified to produce base64 blocks — never URL-sourced file blocks — but +imported or cross-client history can carry a URL block on any role, and such a +block is unserializable under Chat Completions regardless of the role carrying +it. Widening the rule beyond ``ToolMessage`` therefore costs nothing and keeps +foreign history from bricking a thread. The rewrite is order-preserving and +idempotent — placeholder text blocks pass through unchanged on any later +application. +""" + +from __future__ import annotations + +import logging +from collections.abc import Awaitable, Callable +from typing import Any, override +from urllib.parse import urlparse + +from langchain.agents import AgentState +from langchain.agents.middleware import AgentMiddleware +from langchain.agents.middleware.types import ModelCallResult, ModelRequest, ModelResponse +from langchain_anthropic import ChatAnthropic +from langchain_core.messages import AnyMessage + +from deerflow.agents.middlewares.input_sanitization_middleware import neutralize_untrusted_tags +from deerflow.tools.resource_placeholder import model_visible_location, resource_placeholder_text + +logger = logging.getLogger(__name__) + +# URL schemes a chat model provider can resolve for image blocks. Everything +# else (``ui://``, scheme-less virtual paths, unknown schemes) is downgraded to +# a text reference. ``data:`` embeds the payload inline, so it is fetchable. +_FETCHABLE_IMAGE_SCHEMES = frozenset({"http", "https", "data"}) + + +def _str_or_none(value: Any) -> str | None: + return value if isinstance(value, str) else None + + +def _neutralize(value: str | None) -> str | None: + """Neutralize framework/injection tokens in untrusted placeholder metadata. + + This middleware runs after the input/tool-result sanitization passes, and + historical URL blocks never went through them (non-text content is left + untouched), so MIME/location values interpolated into the placeholder must + be guarded here — the same shared guard those passes apply. + """ + return neutralize_untrusted_tags(value) if value else value + + +def _is_http_url(url: str) -> bool: + try: + return urlparse(url).scheme.lower() in ("http", "https") + except ValueError: + return False + + +def _accepts_http_file_urls(model: Any) -> bool: + """Whether *model*'s request serializer accepts http(s) file URLs. + + Chat Completions rejects any URL-sourced ``file`` block, but the OpenAI + Responses API (``input_file`` with ``file_url``) and Anthropic + (``document`` with ``source.type == "url"``) accept HTTPS document URLs — + a block persisted for those providers must survive the rewrite, or the + document silently drops out of the model request. + """ + if model is None: + return False + if getattr(model, "use_responses_api", False): + return True + return isinstance(model, ChatAnthropic) + + +def _sanitize_block(block: Any, model: Any) -> Any: + """Return *block* unchanged, or its text-placeholder replacement. + + Identity is the unchanged signal for the caller: the same object comes back + when nothing was rewritten. + """ + if not isinstance(block, dict): + return block + block_type = block.get("type") + if block_type == "file": + # URL-sourced file blocks are rejected by Chat Completions whenever a + # ``url`` KEY is present — langchain-core raises even for an empty + # string or None — so gate on key presence, not truthiness. Base64 / + # file_id blocks carry no ``url`` key and stay untouched. A present but + # unusable URL ("" / None) gets a location-less placeholder. + if "url" not in block: + return block + url = _str_or_none(block.get("url")) + if url and _is_http_url(url) and _accepts_http_file_urls(model): + # The selected provider serializes http(s) file URLs natively + # (Responses API ``input_file`` / Anthropic ``document`` URL): + # keep the block instead of dropping a readable document. + return block + return {"type": "text", "text": resource_placeholder_text(mime_type=_neutralize(_str_or_none(block.get("mime_type"))), url=_neutralize(model_visible_location(url)))} + if block_type == "image": + url = block.get("url") + if not (isinstance(url, str) and url): + return block + try: + scheme = urlparse(url).scheme.lower() + except ValueError: + # Structurally invalid URL (bad IPv6 literal, invalid netloc + # characters): urlparse raises instead of returning a scheme. + # Downgrade like any other non-fetchable location — one malformed + # persisted URL must not raise on every model call and re-brick + # the thread this middleware exists to heal. + scheme = "" + if scheme not in _FETCHABLE_IMAGE_SCHEMES: + return {"type": "text", "text": resource_placeholder_text(mime_type=_neutralize(_str_or_none(block.get("mime_type"))), url=_neutralize(model_visible_location(url)))} + return block + return block + + +def _sanitize_message(message: AnyMessage, model: Any) -> AnyMessage: + """Return *message* unchanged, or a copy with incompatible blocks replaced. + + Role-agnostic on purpose: any message with list-form content can carry a + persisted or imported URL block that Chat Completions cannot serialize. + """ + content = message.content + if not isinstance(content, list): + return message + patched = [_sanitize_block(block, model) for block in content] + if all(new is old for new, old in zip(patched, content, strict=True)): + return message + return message.model_copy(update={"content": patched}) + + +def _sanitize_messages(messages: list[AnyMessage], model: Any) -> list[AnyMessage] | None: + """Rewrite *messages* for the model request, or ``None`` when nothing changed.""" + updated: list[AnyMessage] = [] + changed = False + for message in messages: + patched = _sanitize_message(message, model) + changed = changed or patched is not message + updated.append(patched) + return updated if changed else None + + +class ModelContentCompatibilityMiddleware(AgentMiddleware[AgentState]): + """Downgrades persisted URL-sourced content blocks the model cannot accept. + + Request-view-only rewrite at the model boundary; see the module docstring + for the failure mode and the contract. Stateless and config-free aside + from the selected model's serializer capabilities, which gate only the + http(s) file-URL preserve; the downgrade itself is unconditional. + """ + + @override + def wrap_model_call( + self, + request: ModelRequest, + handler: Callable[[ModelRequest], ModelResponse], + ) -> ModelCallResult: + return handler(self._sanitize_request(request)) + + @override + async def awrap_model_call( + self, + request: ModelRequest, + handler: Callable[[ModelRequest], Awaitable[ModelResponse]], + ) -> ModelCallResult: + # Pure in-memory rewrite: no sandbox or file I/O, so it stays on the loop. + return await handler(self._sanitize_request(request)) + + @staticmethod + def _sanitize_request(request: ModelRequest) -> ModelRequest: + patched = _sanitize_messages(request.messages, request.model) + if patched is None: + return request + return request.override(messages=patched) diff --git a/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py b/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py index e0bc6c653d..fb3eb9bfa9 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py +++ b/backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py @@ -793,6 +793,18 @@ def build_subagent_runtime_middlewares( if summarization_middleware is not None: middlewares.append(summarization_middleware) + # ModelContentCompatibilityMiddleware — the subagent chain replays the + # same checkpointed ToolMessages the lead does, including any written + # before the conversion-layer fix (MCP ResourceLink results persisted as + # URL-sourced file blocks), which the Chat Completions serializer rejects + # on every subsequent model call. Same read-time request-view rewrite as + # the lead chain; placed after DurableContext/summarization so it sees the + # compacted, reinjected message list. Request view only — subagent + # checkpoints are never written. + from deerflow.agents.middlewares.model_content_compatibility_middleware import ModelContentCompatibilityMiddleware + + middlewares.append(ModelContentCompatibilityMiddleware()) + # SubagentDateContextMiddleware (#4781) — inject framework-owned temporal # context before the first model call without registering the lead agent's # DynamicContextMiddleware. The latter also reads user memory, performs a diff --git a/backend/packages/harness/deerflow/mcp/AGENTS.md b/backend/packages/harness/deerflow/mcp/AGENTS.md index f4428ac740..0176a24414 100644 --- a/backend/packages/harness/deerflow/mcp/AGENTS.md +++ b/backend/packages/harness/deerflow/mcp/AGENTS.md @@ -60,6 +60,7 @@ The optional background-task `status` filter flows through `McpTaskSubmitter.lis - **Configured stdio working directory**: A non-empty `McpServerConfig.cwd` is forwarded by `build_server_params` for discovery, pooled Agent calls, and durable task calls. Omit the connection key when the setting is `None` or empty (including unresolved environment references), so the runtime's thread-workspace default still applies; HTTP/SSE connections must not receive it. `tests/test_mcp_cwd.py` exercises the JSON-config-to-subprocess path with relative entrypoints, relative tool inputs, and discovery/pool defaults for omitted, null, empty, and unresolved values. - **Stdio disconnect recovery**: Ordinary Agent tool calls and durable task submit/status/cancel calls that receive the MCP SDK's explicit `Connection closed` error or an AnyIO closed-stream error evict only that `(ownership domain, server_name, incarnation scope)` session when the registered entry is still the same `ClientSession` that failed. A late error from an old concurrent call cannot evict its replacement or a new in-flight creation. The failing call still surfaces its original error and is never replayed automatically; a later retry creates a fresh subprocess/session. Protocol timeouts, normal `isError=true` tool results, and interceptor failures do not evict a healthy stateful session. - **Stdio path translation**: MCP-returned local file references are not copied. If a `ResourceLink` or conservative free-text path resolves to an existing file inside the thread's mounted user-data tree, it is translated deterministically to `/mnt/user-data/...`; paths outside that tree remain unchanged. Bare-filename correlation must not replace a segment of a longer path: both `/` and `\` are separators at either boundary, even when the full-path pass leaves the reference unresolved. A trailing `\` immediately before a newline therefore counts as a separator too; that position is more often a Markdown hard break, but a backslash is also a legal filename character on POSIX, so forgoing that single rewrite is the conservative tradeoff. `tests/test_mcp_file_migration.py` covers this through the result converter with real workspace files. +- **ResourceLink conversion contract**: `_convert_call_tool_result` never emits URL-sourced `file` blocks from `ResourceLink` items — Chat Completions serialization rejects them (langchain-core), and a checkpointed block would fail every later turn. An `http(s)` `image/*` link stays an image block; `data:`/`blob:` URIs are non-referenceable (one embeds its payload, the other names a browser-local object), so they — `image/*` `data:` included — become a location-less `[Resource: name (mime)]` placeholder and stay out of `resource_links` (new inline payloads never enter state; the read-time middleware only preserves persisted `data:` image blocks). Everything else (`ui://` cards, virtual paths, remote non-image links, unknown schemes) becomes a `[Resource: name (mime) available at url]` placeholder, with the link preserved as `{"name", "uri", "mime_type"}` under the artifact's `resource_links` key. Placeholders come from the shared `resource_placeholder_text` formatter, the location gated by the shared `model_visible_location` (both in `deerflow.tools.resource_placeholder`, shared with the read-time middleware): the exact virtual root or a slash-delimited descendant and remote schemes keep the location (a sibling like `/mnt/user-data-backups/...` shares the prefix but is a host path); a raw `file://` URI, bare/Windows host path, or `data:`/`blob:` URI is withheld — the unresolved URI still lands in the artifact channel the model never sees. `tools/artifact_registry.py::extract_artifacts_from_result` consumes `resource_links` through the same referenceability gate. Embedded base64 payloads, text rewriting, and the error-to-`ToolException` path are unchanged. - **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (`PUT /api/mcp/config` keeps whole-payload validation, while `PATCH /api/mcp/config` changes only one server's `enabled` field, normalizes the same `type`/MCP-spec `transport` alias as the runtime config model, and validates the target only when enabling it). MCP, skill, and embedded-client writers hold the process-local `extensions_config_write_lock` plus the sidecar advisory `extensions_config_file_lock` for the complete read-modify-write/reload cycle, then share `atomic_write_extensions_config()`, which writes and fsyncs a same-directory temporary file before `os.replace()` and preserves an existing file's mode and symlink target; failed serialization or replacement leaves the prior config intact and cleans up the temporary file. The five Gateway mutation paths validate the frozen durable-task snapshot before `atomic_write_extensions_config()` and return 409 on rejection. Runtime updates are picked up by the writer's committed `prepare_mcp_reconciliation()` handoff (or a conservative reset on handoff failure); another worker still picks them up through the two-stage staleness check above; a change that leaves the effective MCP slice unchanged (skills-only, middleware-only) no longer retires cached tools or pooled sessions, and when no MCP server is enabled the tool-assembly path calls `refresh_mcp_cache_if_active()`, which applies the same classification (selective retirement of changed bindings, or the conservative full retirement), voids an in-flight initialization whose revision was superseded, and performs no tool discovery or config hashing on a never-initialized process. The Gateway admin reset (`POST /api/mcp/cache/reset`) additionally calls `publish_mcp_tools_cache_reset()`: it writes a random generation to a hidden JSON marker beside the resolved config under the same cross-process lock (`MCP_CACHE_RESET_MARKER`, a `deerflow.config.shared_reset_marker.SharedResetMarker` shared with the skills prompt-cache reset), then retires local state. Every worker mounting that directory records the marker signature when tools publish and invalidates on a later signature, so a remote `tools/list` change converges without a fake config edit; marker drift during discovery fences the stale result. The response reports this transport as `scope=shared_config` rather than claiming an implicit broadcast to replicas with independent filesystems. No resolvable config path keeps the legacy process-local behavior and is reported as `scope=process`. - **Stdio launch policy at the HTTP boundary** (`routers/mcp.py::_validate_mcp_update_request`, shared by `PUT` and the enable branch of `PATCH`): a config file may express anything, but the API is untrusted input, so an API-registered stdio server must (a) name a bare executable from the allowlist — `_DEFAULT_MCP_STDIO_COMMAND_ALLOWLIST` = `{npx, uvx}`, extended by `DEER_FLOW_MCP_STDIO_COMMAND_ALLOWLIST`, with path separators, whitespace, and shell metacharacters rejected in `command`; (b) carry no `args` flag in `_ARBITRARY_EXEC_ARGS`; and (c) set no `env` name in `_CODE_INJECTING_ENV_VARS`. Checks (b) and (c) exist because the command check alone names a binary without constraining what that binary runs. The `env` denylist applies to **every** allowlisted command, and both denylists match `--flag=value` as well as `--flag value`. The `args` denylist's **scope depends on the command**, because where a launcher stops parsing its own flags is what decides whether a token is an exec flag at all: diff --git a/backend/packages/harness/deerflow/mcp/tools.py b/backend/packages/harness/deerflow/mcp/tools.py index a0fc048b26..a85f436b7c 100644 --- a/backend/packages/harness/deerflow/mcp/tools.py +++ b/backend/packages/harness/deerflow/mcp/tools.py @@ -41,6 +41,7 @@ from deerflow.mcp_scope import mcp_session_scope_key, runtime_thread_incarnation from deerflow.reflection import resolve_variable from deerflow.runtime.user_context import resolve_runtime_user_id from deerflow.tools.mcp_metadata import tag_mcp_routing, tag_mcp_tool +from deerflow.tools.resource_placeholder import model_visible_location, resource_placeholder_text from deerflow.tools.sync import make_sync_tool_wrapper from deerflow.tools.types import Runtime @@ -440,6 +441,18 @@ def _convert_call_tool_result( with their cwd/temp pinned inside the mounted tree, so they already live in a servable location. Remote URIs and files outside the thread's user-data tree are left untouched. + + ``ResourceLink`` items never become URL-sourced ``file`` blocks: Chat + Completions message serialization rejects those, and once the result is + checkpointed every later turn of the thread fails. An ``http(s)`` image + link stays an image block; every other link (``ui://`` cards, local files + at virtual paths, remote non-image links, unknown schemes) becomes a short + text placeholder, and the link is preserved as structured data in the + artifact under ``resource_links``. Placeholders never embed host + filesystem paths: a link that stays unresolved (outside the thread's + user-data tree, or no thread context) omits the location segment rather + than leaking a raw ``file://`` URI or bare path, and non-referenceable + ``data:``/``blob:`` URIs are dropped from the artifact channel too. """ from langchain_core.messages import ToolMessage from langchain_core.messages.content import create_file_block, create_image_block, create_text_block @@ -482,6 +495,7 @@ def _convert_call_tool_result( # Convert MCP content blocks to LangChain content blocks. lc_content = [] + resource_links: list[dict[str, Any]] = [] for item in call_tool_result.content: if isinstance(item, TextContent): lc_content.append(create_text_block(text=_resolve_text(item.text))) @@ -490,10 +504,40 @@ def _convert_call_tool_result( elif isinstance(item, ResourceLink): mime = item.mimeType or None url = _resolve_link_url(str(item.uri)) - if mime and mime.startswith("image/"): + if mime and mime.startswith("image/") and url.lower().startswith(("http://", "https://")): lc_content.append(create_image_block(url=url, mime_type=mime)) + elif url.lower().startswith(("data:", "blob:")): + # ``data:``/``blob:`` URIs never enter state: a ``data:`` URI + # embeds the whole payload, so inlining it would put megabytes + # of base64 into model-visible text and into checkpointed + # ``resource_links``; a ``blob:`` URI names a browser-local + # object nothing else can dereference. ``image/*`` links are + # downgraded here too, by design: the conversion layer keeps + # NEW inline payloads out of state, while the read-time + # middleware passes persisted ``data:`` image blocks through + # (it only heals blocks already in checkpoints). Neither is + # referenceable, so the raw URI is dropped from the artifact + # channel and the placeholder omits the location segment. + lc_content.append(create_text_block(text=resource_placeholder_text(name=item.name or "unnamed", mime_type=mime))) else: - lc_content.append(create_file_block(url=url, mime_type=mime)) + # URL-sourced file blocks are rejected by Chat Completions + # serialization (langchain-core), which bricks the thread once + # the result is checkpointed. Downgrade every other link to a + # text placeholder and keep the structured link in the artifact + # channel. The placeholder shows the location only when the + # resolved URL carries no host-path risk — a raw ``file://`` + # URI or bare host path is withheld from model-visible text + # but still recorded in the artifact channel. + lc_content.append( + create_text_block( + text=resource_placeholder_text( + name=item.name or "unnamed", + mime_type=mime, + url=model_visible_location(url), + ) + ) + ) + resource_links.append({"name": item.name, "uri": url, "mime_type": mime}) elif isinstance(item, EmbeddedResource): from mcp.types import BlobResourceContents @@ -518,6 +562,10 @@ def _convert_call_tool_result( artifact = None if call_tool_result.structuredContent is not None: artifact = {"structured_content": call_tool_result.structuredContent} + if resource_links: + if artifact is None: + artifact = {} + artifact["resource_links"] = resource_links return lc_content, artifact diff --git a/backend/packages/harness/deerflow/tools/artifact_registry.py b/backend/packages/harness/deerflow/tools/artifact_registry.py index c835aff6cd..0648e02004 100644 --- a/backend/packages/harness/deerflow/tools/artifact_registry.py +++ b/backend/packages/harness/deerflow/tools/artifact_registry.py @@ -298,14 +298,22 @@ def extract_artifacts_from_result( become concrete ``file``/``task`` entries; when no known key matches, the whole payload becomes a complete JSON ``data`` entry only within the 4096-byte, 1024-node and 32-level limits. Empty/oversized payloads are skipped. - 2. ``content`` blocks of type ``file`` / ``image`` with a URL source become + 2. ``artifact["resource_links"]`` — the ``{"name", "uri", "mime_type"}`` + entries the MCP conversion layer writes for downgraded ``ResourceLink`` + results — become ``file`` entries, subject to the same + referenceability gate as every other ref source (``data:``/``blob:`` + and other unresolvable URIs are skipped). + 3. ``content`` blocks of type ``file`` / ``image`` with a URL source become ``file`` / ``image`` entries. - 3. ``content`` text blocks and plain-string results are scanned + 4. ``content`` text blocks and plain-string results are scanned conservatively for sandbox paths and remote file URLs (gated by ``detect_refs_in_text``). - Error results produce no entries. Every reference from one result gets a - distinct handle via a sequential ordinal. + Error results produce no entries. Every distinct reference from one + result gets its own handle via a sequential ordinal; the same ``real_ref`` + surfaced by several sources (e.g. a ``resource_links`` entry and the + placeholder text the conversion layer wrote for the same link) is kept + only from the first source, so one resource never gets two handles. """ if result.status == "error": return [] @@ -315,6 +323,17 @@ def extract_artifacts_from_result( tool_call_id = result.tool_call_id or "" sink = _EntrySink(thread_id=thread_id, tool_call_id=tool_call_id, call_index=call_index, created_at=now, tool_name=tool_name, occurrence_id=result.id) entries: list[ArtifactEntry] = [] + seen_refs: set[str] = set() + + def _add_unique(**fields: Any) -> None: + # Several sources can surface the same ref for one result — e.g. the + # conversion layer's ``resource_links`` channel plus the placeholder + # text it wrote into a content block. Keep the first entry per ref so + # one resource never gets two handles; earlier (typed) sources win. + if fields["real_ref"] in seen_refs: + return + seen_refs.add(fields["real_ref"]) + entries.append(sink.add(**fields)) artifact = result.artifact if artifact is not None and isinstance(artifact, dict): @@ -326,17 +345,34 @@ def extract_artifacts_from_result( is_task = key in _STRUCTURED_TASK_KEYS if not (_is_referenceable_task_id(value) if is_task else _is_referenceable_url(value)): continue - entries.append(sink.add(artifact_type="task" if is_task else "file", display_name=_display_name_for_ref(value), real_ref=value)) + _add_unique(artifact_type="task" if is_task else "file", display_name=_display_name_for_ref(value), real_ref=value) if not entries and not any(value.startswith(("data:", "blob:")) for _, value in found): encoded = _serialize_bounded_data(structured) if encoded is not None: - entries.append(sink.add(artifact_type="data", display_name=f"{tool_name} structured result", real_ref=encoded)) + _add_unique(artifact_type="data", display_name=f"{tool_name} structured result", real_ref=encoded) + + resource_links = artifact.get("resource_links") + if isinstance(resource_links, list): + for link in resource_links: + if not isinstance(link, dict): + continue + uri = link.get("uri") + if not isinstance(uri, str) or not _is_referenceable_url(uri): + continue + name = link.get("name") + mime = link.get("mime_type") + _add_unique( + artifact_type="file", + display_name=name if isinstance(name, str) and name else _display_name_for_ref(uri), + real_ref=uri, + mime_type=mime if isinstance(mime, str) and mime else None, + ) content = result.content if isinstance(content, str): if detect_refs_in_text and content: for ref in _detect_refs_in_text(content): - entries.append(sink.add(artifact_type=ref["type"], display_name=ref["display"], real_ref=ref["ref"])) + _add_unique(artifact_type=ref["type"], display_name=ref["display"], real_ref=ref["ref"]) return entries if not isinstance(content, list): return entries @@ -352,13 +388,11 @@ def extract_artifacts_from_result( continue url = source.get("url") if isinstance(url, str) and url and _is_referenceable_url(url): - entries.append( - sink.add( - artifact_type="file" if block_type == "file" else "image", - display_name=_display_name_for_ref(url), - real_ref=url, - mime_type=source.get("mime_type") if isinstance(source.get("mime_type"), str) else None, - ) + _add_unique( + artifact_type="file" if block_type == "file" else "image", + display_name=_display_name_for_ref(url), + real_ref=url, + mime_type=source.get("mime_type") if isinstance(source.get("mime_type"), str) else None, ) continue @@ -367,7 +401,7 @@ def extract_artifacts_from_result( if not isinstance(text, str): continue for ref in _detect_refs_in_text(text): - entries.append(sink.add(artifact_type=ref["type"], display_name=ref["display"], real_ref=ref["ref"])) + _add_unique(artifact_type=ref["type"], display_name=ref["display"], real_ref=ref["ref"]) return entries diff --git a/backend/packages/harness/deerflow/tools/resource_placeholder.py b/backend/packages/harness/deerflow/tools/resource_placeholder.py new file mode 100644 index 0000000000..bed3b46e2f --- /dev/null +++ b/backend/packages/harness/deerflow/tools/resource_placeholder.py @@ -0,0 +1,73 @@ +"""Canonical model-visible placeholder text for MCP resource links. + +Both the MCP conversion layer (``deerflow.mcp.tools``) and the read-time +``ModelContentCompatibilityMiddleware`` downgrade resource references that a +chat model cannot consume into a short text placeholder. They share this +formatter so a thread healed at read time shows the model the exact same +placeholder shape a fresh conversion would have produced. Persisted content +blocks carry no resource name, so the name segment is optional and simply +omitted when absent. +""" + +from __future__ import annotations + +from urllib.parse import urlparse + +from deerflow.config.paths import VIRTUAL_PATH_PREFIX + + +def model_visible_location(url: str | None) -> str | None: + """Return *url* when it is safe to show in model-visible placeholder text. + + Shared by the conversion layer and the read-time middleware so both emit + placeholders under the same rules: + + - virtual paths carry no host-path risk and stay visible — but only the + exact virtual root or a slash-delimited descendant: a sibling like + ``/mnt/user-data-backups/...`` shares the prefix yet is a real host + path, so it must collapse to ``None`` like any other host location; + remote schemes (``http(s)``, ``ui://``, ``s3://``, ...) stay visible; + - a raw ``file://`` URI, a bare host path, or a Windows drive path + (urlparse reads the drive letter as a single-character scheme) leaks the + deployment's filesystem layout, so it collapses to ``None`` and the + placeholder omits the location segment (US-16); + - ``data:`` URIs embed their whole base64 payload and ``blob:`` URIs name a + browser-local object nothing else can dereference, so they collapse to + ``None`` too — inlining either would put kilobytes-to-megabytes of + base64 into the model-visible text. + """ + if not url: + return None + if url == VIRTUAL_PATH_PREFIX or url.startswith(VIRTUAL_PATH_PREFIX + "/"): + # Exact virtual root or a slash-delimited descendant. A plain prefix + # match would also admit siblings such as ``/mnt/user-data-backups``, + # leaking the deployment's directory layout into model-visible text. + return url + try: + scheme = urlparse(url).scheme + except ValueError: + return None + if not scheme or scheme in ("file", "data", "blob") or (len(scheme) == 1 and scheme.isalpha()): + # Bare host path, ``file://`` URI, inline payload, browser-local + # object, or Windows drive path. + return None + return url + + +def resource_placeholder_text( + *, + name: str | None = None, + mime_type: str | None = None, + url: str | None = None, +) -> str: + """Return the canonical ``[Resource ...]`` placeholder text. + + ``mime_type`` falls back to ``"unknown type"``; an empty/``None`` ``url`` + omits the location segment entirely (used for host paths that must never + reach model-visible text and for non-referenceable inline schemes). + """ + mime = mime_type or "unknown type" + location = f" available at {url}" if url else "" + if name: + return f"[Resource: {name} ({mime}){location}]" + return f"[Resource ({mime}){location}]" diff --git a/backend/tests/test_artifact_registry.py b/backend/tests/test_artifact_registry.py index 226b924319..4fde91f290 100644 --- a/backend/tests/test_artifact_registry.py +++ b/backend/tests/test_artifact_registry.py @@ -230,6 +230,84 @@ def test_structured_url_key_rejects_non_http_uris(): assert types["job-11"] == "task" +def test_extract_from_resource_links_artifact(): + """The MCP conversion layer preserves ResourceLinks under the artifact's + ``resource_links`` key; a result whose ONLY artifact signal is that key + still yields file entries, named after the link.""" + result = ToolMessage( + content=[{"type": "text", "text": "done"}], + tool_call_id="call-rl", + name="mcp_report_gen", + artifact={ + "resource_links": [ + {"name": "quarterly report", "uri": "https://example.com/report.pdf", "mime_type": "application/pdf"}, + {"name": "", "uri": "/mnt/user-data/outputs/notes.txt", "mime_type": "text/plain"}, + ] + }, + ) + entries = extract_artifacts_from_result(result, thread_id="thread-1") + assert [(entry["artifact_type"], entry["real_ref"]) for entry in entries] == [ + ("file", "https://example.com/report.pdf"), + ("file", "/mnt/user-data/outputs/notes.txt"), + ] + assert entries[0]["display_name"] == "quarterly report" + assert entries[0]["mime_type"] == "application/pdf" + assert entries[1]["display_name"] == "notes.txt" + + +def test_resource_links_reject_unreferenceable_uris(): + """``resource_links`` entries get the same referenceability gate as every + other ref source: embedded-payload and non-fetchable URIs stay out of + thread state.""" + result = ToolMessage( + content=[{"type": "text", "text": "done"}], + tool_call_id="call-rl2", + name="mcp_embedded", + artifact={ + "resource_links": [ + {"name": "blob", "uri": "data:application/pdf;base64," + "A" * 500, "mime_type": "application/pdf"}, + {"name": "card", "uri": "ui://app/card.html", "mime_type": "text/html"}, + {"name": "keep", "uri": "/mnt/user-data/outputs/keep.md", "mime_type": "text/markdown"}, + ] + }, + ) + entries = extract_artifacts_from_result(result, thread_id="thread-1") + assert [entry["real_ref"] for entry in entries] == ["/mnt/user-data/outputs/keep.md"] + + +def test_resource_links_and_placeholder_text_dedup_same_ref(): + """The conversion layer writes both a ``resource_links`` artifact entry and + a model-visible placeholder text for one downgraded ResourceLink: the + registry must not mint two handles for the same ref — the typed + ``resource_links`` entry (with its display name) wins.""" + path_uri = "/mnt/user-data/outputs/page.png" + url_uri = "https://example.com/files/report.pdf" + result = ToolMessage( + content=[ + { + "type": "text", + "text": (f"[Resource: page (image/png) available at {path_uri}]\n[Resource: report (application/pdf) available at {url_uri}]"), + } + ], + tool_call_id="call-dup", + name="mcp_screenshot", + artifact={ + "resource_links": [ + {"name": "page", "uri": path_uri, "mime_type": "image/png"}, + {"name": "report", "uri": url_uri, "mime_type": "application/pdf"}, + ] + }, + ) + entries = extract_artifacts_from_result(result, thread_id="thread-1") + assert [(entry["artifact_type"], entry["real_ref"]) for entry in entries] == [ + ("file", path_uri), + ("file", url_uri), + ] + assert entries[0]["display_name"] == "page" + assert entries[0]["mime_type"] == "image/png" + assert entries[1]["display_name"] == "report" + + def test_extract_from_text_with_path(): result = ToolMessage( content=[{"type": "text", "text": "Report saved to /mnt/user-data/outputs/report.html"}], diff --git a/backend/tests/test_create_deerflow_agent.py b/backend/tests/test_create_deerflow_agent.py index dcc31fe95f..a95c071d66 100644 --- a/backend/tests/test_create_deerflow_agent.py +++ b/backend/tests/test_create_deerflow_agent.py @@ -808,7 +808,9 @@ def test_loop_detection_before_clarification(mock_create_agent): loop_idx = mw_types.index("LoopDetectionMiddleware") clar_idx = mw_types.index("ClarificationMiddleware") assert loop_idx < clar_idx - assert loop_idx == clar_idx - 1 + # The read-time sanitizer owns the slot immediately before Clarification. + assert mw_types[clar_idx - 1] == "ModelContentCompatibilityMiddleware" + assert loop_idx == clar_idx - 2 # --------------------------------------------------------------------------- @@ -851,9 +853,11 @@ def test_loop_detection_custom_middleware(mock_create_agent): mw_types = [type(m).__name__ for m in middleware] # Default LoopDetectionMiddleware must not also appear. assert "LoopDetectionMiddleware" not in mw_types - # Custom replacement sits immediately before TokenBudgetMiddleware and ClarificationMiddleware. + # Custom replacement sits after TokenBudgetMiddleware's slot, with only the + # read-time sanitizer between it and ClarificationMiddleware. assert mw_types[-1] == "ClarificationMiddleware" - assert mw_types[-2] == "MyLoopDetection" + assert mw_types[-2] == "ModelContentCompatibilityMiddleware" + assert mw_types[-3] == "MyLoopDetection" # --------------------------------------------------------------------------- @@ -987,6 +991,7 @@ def test_full_chain_order(mock_create_agent): "ViewImageMiddleware", "SubagentLimitMiddleware", "LoopDetectionMiddleware", + "ModelContentCompatibilityMiddleware", "ClarificationMiddleware", ] assert mw_types == expected_order diff --git a/backend/tests/test_mcp_file_migration.py b/backend/tests/test_mcp_file_migration.py index 804e4dca8f..41f0b130e0 100644 --- a/backend/tests/test_mcp_file_migration.py +++ b/backend/tests/test_mcp_file_migration.py @@ -4,6 +4,12 @@ Regression coverage for GitHub issue #3597: Playwright MCP (and similar stdio servers) write files to a path the sandbox/artifact API cannot resolve. The MCP tool wrapper pins stdio cwd/temp under the thread's mounted user-data tree and rewrites returned file references to ``/mnt/user-data/...`` virtual paths. + +``ResourceLink`` conversion contract: links never become URL-sourced ``file`` +blocks — Chat Completions message serialization rejects those, and a +checkpointed block would fail every later turn of the thread. ``http(s)`` +image links stay image blocks; every other link becomes a text placeholder, +with the original link preserved in the artifact under ``resource_links``. """ import os @@ -719,7 +725,7 @@ class TestResultHasTextContent: class TestConvertCallToolResultRewrites: - def test_resource_link_image_inside_workspace_rewritten(self, paths: Paths): + def test_local_image_resource_link_becomes_text_placeholder(self, paths: Paths): src = _workspace_file(paths, "page.png", content=b"png") result = CallToolResult( content=[ResourceLink(type="resource_link", name="page", uri=src.as_uri(), mimeType="image/png")], @@ -727,12 +733,16 @@ class TestConvertCallToolResultRewrites: ) with _patch_paths(paths): - content, _ = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") - assert content[0]["type"] == "image" - assert content[0]["url"] == f"{VIRTUAL_PATH_PREFIX}/workspace/page.png" + # Local images are downgraded too: the provider cannot fetch virtual + # paths — the view-image tool is the intended path for local images. + virtual = f"{VIRTUAL_PATH_PREFIX}/workspace/page.png" + assert content[0]["type"] == "text" + assert content[0]["text"] == f"[Resource: page (image/png) available at {virtual}]" + assert artifact == {"resource_links": [{"name": "page", "uri": virtual, "mime_type": "image/png"}]} - def test_resource_link_file_inside_outputs_rewritten(self, paths: Paths): + def test_local_file_resource_link_becomes_text_placeholder(self, paths: Paths): outputs = paths.sandbox_outputs_dir("t1", user_id="u1") outputs.mkdir(parents=True) src = outputs / "doc.pdf" @@ -743,12 +753,14 @@ class TestConvertCallToolResultRewrites: ) with _patch_paths(paths): - content, _ = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") - assert content[0]["type"] == "file" - assert content[0]["url"] == f"{VIRTUAL_PATH_PREFIX}/outputs/doc.pdf" + virtual = f"{VIRTUAL_PATH_PREFIX}/outputs/doc.pdf" + assert content[0]["type"] == "text" + assert content[0]["text"] == f"[Resource: doc (application/pdf) available at {virtual}]" + assert artifact == {"resource_links": [{"name": "doc", "uri": virtual, "mime_type": "application/pdf"}]} - def test_resource_link_outside_user_data_untouched(self, tmp_path: Path, paths: Paths): + def test_resource_link_outside_user_data_becomes_text_placeholder(self, tmp_path: Path, paths: Paths): src = tmp_path / "page.png" src.write_bytes(b"png") uri = src.as_uri() @@ -758,11 +770,18 @@ class TestConvertCallToolResultRewrites: ) with _patch_paths(paths): - content, _ = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") - assert content[0]["url"] == uri + # The URI stays unresolved outside the user-data tree, but the link is + # still downgraded — a ``file://`` URL block is equally fatal. The raw + # host path is withheld from model-visible text (US-16); it survives + # only in the artifact channel, which the model never sees. + assert content[0]["type"] == "text" + assert content[0]["text"] == "[Resource: page (image/png)]" + assert all("file://" not in block.get("text", "") for block in content) + assert artifact == {"resource_links": [{"name": "page", "uri": uri, "mime_type": "image/png"}]} - def test_remote_resource_link_untouched(self, paths: Paths): + def test_remote_image_resource_link_stays_image_block(self, paths: Paths): url = "https://example.com/remote.png" result = CallToolResult( content=[ResourceLink(type="resource_link", name="r", uri=url, mimeType="image/png")], @@ -770,9 +789,177 @@ class TestConvertCallToolResultRewrites: ) with _patch_paths(paths): - content, _ = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + assert content[0]["type"] == "image" assert content[0]["url"] == url + assert artifact is None + + def test_remote_image_resource_link_with_uppercase_scheme_stays_image_block(self, paths: Paths): + url = "HTTPS://example.com/remote.png" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="r", uri=url, mimeType="image/png")], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + # Scheme matching is case-insensitive: the link stays an image block, + # not a text placeholder. The block's URL validation normalizes the + # scheme to lowercase; the passthrough itself is what matters. + assert content[0]["type"] == "image" + assert content[0]["url"] == "https://example.com/remote.png" + assert artifact is None + + def test_ui_resource_link_becomes_text_placeholder(self, paths: Paths): + # Regression guard: an MCP Apps UI card must not produce any block the + # Chat Completions translator rejects (any URL-sourced file block). + url = "ui://app/card.html" + mime = "text/html;profile=mcp-app" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="card", uri=url, mimeType=mime)], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + assert content[0]["type"] == "text" + assert content[0]["text"] == f"[Resource: card ({mime}) available at {url}]" + assert all(not (block.get("type") == "file" and "url" in block) for block in content) + assert artifact == {"resource_links": [{"name": "card", "uri": url, "mime_type": mime}]} + + def test_remote_non_image_resource_link_becomes_text_placeholder(self, paths: Paths): + url = "https://example.com/report.pdf" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="report", uri=url, mimeType="application/pdf")], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + assert content[0]["type"] == "text" + assert content[0]["text"] == f"[Resource: report (application/pdf) available at {url}]" + assert artifact == {"resource_links": [{"name": "report", "uri": url, "mime_type": "application/pdf"}]} + + def test_unknown_scheme_resource_link_without_name_or_mime_uses_fallbacks(self, paths: Paths): + url = "s3://bucket/report.pdf" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="", uri=url, mimeType=None)], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + assert content[0]["type"] == "text" + assert content[0]["text"] == f"[Resource: unnamed (unknown type) available at {url}]" + assert artifact == {"resource_links": [{"name": "", "uri": url, "mime_type": None}]} + + def test_data_uri_resource_link_never_enters_model_text_or_state(self, paths: Paths): + payload = "QUFB" + "A" * 500 + url = f"data:application/pdf;base64,{payload}" + result = CallToolResult( + content=[ + ResourceLink(type="resource_link", name="report", uri=url, mimeType="application/pdf"), + ResourceLink(type="resource_link", name="", uri="data:image/png;base64,QUJD", mimeType=None), + ], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + # The URI embeds the whole payload: it must not be inlined into the + # model-visible text nor checkpointed inside resource_links. The + # placeholder is the canonical location-less form. + assert content[0]["type"] == "text" + assert content[0]["text"] == "[Resource: report (application/pdf)]" + assert content[1]["type"] == "text" + assert content[1]["text"] == "[Resource: unnamed (unknown type)]" + assert all(payload not in block.get("text", "") for block in content) + assert all("data:" not in block.get("text", "") for block in content) + assert artifact is None + + def test_image_mime_data_uri_resource_link_is_downgraded(self, paths: Paths): + """An ``image/*`` ``data:`` link is downgraded too, by design: the + conversion layer keeps new inline payloads out of state, even though + the read-time middleware lets persisted ``data:`` image blocks pass.""" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="shot", uri="data:image/png;base64,QUJD", mimeType="image/png")], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + assert content[0]["type"] == "text" + assert content[0]["text"] == "[Resource: shot (image/png)]" + assert all("data:" not in block.get("text", "") for block in content) + assert artifact is None + + def test_blob_uri_resource_link_never_enters_model_text_or_state(self, paths: Paths): + url = "blob:https://example.com/550e8400-e29b-41d4-a716-446655440000" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="report", uri=url, mimeType="application/pdf")], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + # A ``blob:`` URI names a browser-local object nothing else can + # dereference: like ``data:`` it never enters checkpointed state, and + # the placeholder carries no location segment. + assert content[0]["type"] == "text" + assert content[0]["text"] == "[Resource: report (application/pdf)]" + assert all("blob:" not in block.get("text", "") for block in content) + assert artifact is None + + def test_resource_links_merge_with_structured_content_artifact(self, paths: Paths): + url = "https://example.com/report.pdf" + result = CallToolResult( + content=[ + TextContent(type="text", text="done"), + ResourceLink(type="resource_link", name="report", uri=url, mimeType="application/pdf"), + ], + structuredContent={"k": "v"}, + isError=False, + ) + + with _patch_paths(paths): + _, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + assert artifact == { + "structured_content": {"k": "v"}, + "resource_links": [{"name": "report", "uri": url, "mime_type": "application/pdf"}], + } + + def test_multiple_resource_links_preserve_order(self, paths: Paths): + first = "https://example.com/a.pdf" + second = "ui://app/card.html" + result = CallToolResult( + content=[ + ResourceLink(type="resource_link", name="a", uri=first, mimeType="application/pdf"), + ResourceLink(type="resource_link", name="b", uri=second, mimeType="text/html"), + ], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + assert [block["type"] for block in content] == ["text", "text"] + assert content[0]["text"] == f"[Resource: a (application/pdf) available at {first}]" + assert content[1]["text"] == f"[Resource: b (text/html) available at {second}]" + assert artifact == { + "resource_links": [ + {"name": "a", "uri": first, "mime_type": "application/pdf"}, + {"name": "b", "uri": second, "mime_type": "text/html"}, + ] + } def test_text_review_case_rewritten(self, paths: Paths): workspace = paths.sandbox_work_dir("t1", user_id="u1") @@ -819,7 +1006,7 @@ class TestConvertCallToolResultRewrites: assert content[0]["text"] == f"Saved as {VIRTUAL_PATH_PREFIX}/workspace/report with spaces.txt" - def test_no_context_does_not_rewrite(self, paths: Paths): + def test_no_context_resource_link_still_becomes_text_placeholder(self, paths: Paths): src = _workspace_file(paths, "x.png", content=b"png") uri = src.as_uri() result = CallToolResult( @@ -828,9 +1015,32 @@ class TestConvertCallToolResultRewrites: ) with _patch_paths(paths): - content, _ = mcp_tools._convert_call_tool_result(result) + content, artifact = mcp_tools._convert_call_tool_result(result) - assert content[0]["url"] == uri + # Without thread context the URI is not virtualized, but the downgrade + # still applies — a ``file://`` URL image block is not fetchable. The + # host path stays out of model-visible text (US-16). + assert content[0]["type"] == "text" + assert content[0]["text"] == "[Resource: x (image/png)]" + assert all("file://" not in block.get("text", "") for block in content) + assert artifact == {"resource_links": [{"name": "x", "uri": uri, "mime_type": "image/png"}]} + + def test_windows_drive_resource_link_omits_host_path_from_placeholder(self, paths: Paths): + uri = "C:\\Users\\shots\\page.png" + result = CallToolResult( + content=[ResourceLink(type="resource_link", name="page", uri=uri, mimeType="image/png")], + isError=False, + ) + + with _patch_paths(paths): + content, artifact = mcp_tools._convert_call_tool_result(result, thread_id="t1", user_id="u1") + + # urlparse reads the drive prefix as a single-letter scheme; a bare + # Windows host path must not reach model-visible text either. + assert content[0]["type"] == "text" + assert content[0]["text"] == "[Resource: page (image/png)]" + assert all(":\\" not in block.get("text", "") for block in content) + assert artifact == {"resource_links": [{"name": "page", "uri": "c:\\Users\\shots\\page.png", "mime_type": "image/png"}]} def test_text_content_passthrough(self, paths: Paths): result = CallToolResult(content=[TextContent(type="text", text="hello")], isError=False) diff --git a/backend/tests/test_model_content_compatibility_middleware.py b/backend/tests/test_model_content_compatibility_middleware.py new file mode 100644 index 0000000000..ccb85a91ec --- /dev/null +++ b/backend/tests/test_model_content_compatibility_middleware.py @@ -0,0 +1,552 @@ +"""Tests for ModelContentCompatibilityMiddleware (read-time sanitizer). + +The MCP conversion layer used to persist URL-sourced ``file`` content blocks +(e.g. a ResourceLink with a ``ui://`` URI) into checkpointed ToolMessages. Under +the Chat Completions API every subsequent model call in such a thread fails at +client-side serialization — langchain-core's OpenAI translator raises +``ValueError: OpenAI Chat Completions does not support file URLs.`` for ANY +``file`` block carrying a ``url`` — and the error-handling middleware answers +every later turn with a non-retriable fallback. The thread is bricked. + +This middleware heals those threads at read time: just before a request reaches +the model adapter it rewrites, in the request view only (state and checkpoints +are never touched), + +- any ``{"type": "file", "url": ...}`` block in list-form message content into + a plain text placeholder + ``[Resource ({mime or "unknown type"}) available at {url}]``; +- any ``{"type": "image", "url": ...}`` block whose URL scheme is not + fetchable by the provider (not ``http``/``https``/``data``) into the same + placeholder. + +The rules apply to every message role: imported or cross-client history can +carry URL file blocks on HumanMessage/AIMessage too, and such a block is +unserializable regardless of the role carrying it. Everything else — fetchable +image URLs, embedded base64 payloads, text blocks, and string content — passes +through untouched, in order, and a second application is a no-op. +""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +import pytest +from langchain.agents.middleware.types import ModelRequest +from langchain_core.language_models.fake_chat_models import FakeMessagesListChatModel +from langchain_core.messages import AIMessage, AnyMessage, HumanMessage, ToolMessage + +from deerflow.agents.middlewares.model_content_compatibility_middleware import ( + ModelContentCompatibilityMiddleware, +) + +pytestmark = pytest.mark.anyio + + +@pytest.fixture +def anyio_backend(): + return "asyncio" + + +def _model_request(messages: list[AnyMessage], model=None) -> ModelRequest: + """Build a real ModelRequest so `.override()` behaves as it does in the graph.""" + return ModelRequest( + model=model if model is not None else FakeMessagesListChatModel(responses=[AIMessage(content="ok")]), + messages=list(messages), + system_message=None, + tool_choice=None, + tools=[], + response_format=None, + state={"messages": list(messages)}, + runtime=MagicMock(), + model_settings={}, + ) + + +def _tool_message(content, tool_call_id: str = "call_1") -> ToolMessage: + return ToolMessage(content=content, tool_call_id=tool_call_id, name="some_tool") + + +def _run_sync(middleware: ModelContentCompatibilityMiddleware, request: ModelRequest) -> ModelRequest: + """Run wrap_model_call with a capturing handler; return the request the model would see.""" + seen: list[ModelRequest] = [] + middleware.wrap_model_call(request, lambda prepared: seen.append(prepared) or AIMessage(content="ok")) + assert len(seen) == 1 + return seen[0] + + +async def _run_async(middleware: ModelContentCompatibilityMiddleware, request: ModelRequest) -> ModelRequest: + """Run awrap_model_call with a capturing handler; return the request the model would see.""" + seen: list[ModelRequest] = [] + + async def handler(prepared: ModelRequest) -> AIMessage: + seen.append(prepared) + return AIMessage(content="ok") + + await middleware.awrap_model_call(request, handler) + assert len(seen) == 1 + return seen[0] + + +def _url_file_block(url: str, mime_type: str | None = None) -> dict: + """The legacy persisted shape: create_file_block(url=..., mime_type=...).""" + block = {"type": "file", "id": "lc_legacy", "url": url} + if mime_type is not None: + block["mime_type"] = mime_type + return block + + +class TestUrlFileBlocksBecomeTextPlaceholders: + def test_ui_scheme_file_block_becomes_text_placeholder(self): + """The incident regression: an MCP App ``ui://`` card must not reach the model as a file block.""" + message = _tool_message([_url_file_block("ui://weather-app/card", "text/html;profile=mcp-app")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (text/html;profile=mcp-app) available at ui://weather-app/card]"}] + + def test_remote_https_file_block_becomes_text_placeholder(self): + message = _tool_message([_url_file_block("https://example.com/report.pdf", "application/pdf")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf) available at https://example.com/report.pdf]"}] + + def test_local_virtual_path_file_block_becomes_text_placeholder(self): + message = _tool_message([_url_file_block("/mnt/user-data/outputs/notes.txt", "text/plain")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (text/plain) available at /mnt/user-data/outputs/notes.txt]"}] + + def test_file_block_without_mime_type_uses_unknown_type(self): + message = _tool_message([_url_file_block("https://example.com/mystery.bin")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (unknown type) available at https://example.com/mystery.bin]"}] + + def test_no_chat_completions_rejected_block_remains(self): + """Pin the exact incident condition: no ``file`` block carrying a ``url`` survives.""" + message = _tool_message( + [ + _url_file_block("ui://weather-app/card", "text/html;profile=mcp-app"), + _url_file_block("https://example.com/report.pdf", "application/pdf"), + {"type": "text", "text": "done"}, + ] + ) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + content = prepared.messages[0].content + assert isinstance(content, list) + assert not any(isinstance(block, dict) and block.get("type") == "file" and "url" in block for block in content) + + def test_file_block_with_empty_string_url_is_rewritten(self): + """langchain-core raises on ANY ``file`` block carrying a ``url`` key — an + empty string included — so key presence, not truthiness, must drive the + rewrite. With no usable location the placeholder drops the suffix.""" + message = _tool_message([{"type": "file", "url": "", "mime_type": "application/pdf"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + + def test_file_block_with_none_url_is_rewritten(self): + message = _tool_message([{"type": "file", "url": None, "mime_type": "image/png"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (image/png)]"}] + + +class TestImageBlockSchemeGating: + def test_image_block_with_non_fetchable_scheme_is_downgraded(self): + message = _tool_message([{"type": "image", "url": "ui://weather-app/chart", "mime_type": "image/png"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (image/png) available at ui://weather-app/chart]"}] + + def test_local_virtual_path_image_block_is_downgraded(self): + """Local images have no scheme; the model must use the view_image tool instead.""" + message = _tool_message([{"type": "image", "url": "/mnt/user-data/outputs/chart.png", "mime_type": "image/png"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (image/png) available at /mnt/user-data/outputs/chart.png]"}] + + def test_http_and_https_image_blocks_pass_through(self): + blocks = [ + {"type": "image", "url": "http://example.com/a.png", "mime_type": "image/png"}, + {"type": "image", "url": "https://example.com/b.png", "mime_type": "image/png"}, + ] + message = _tool_message(blocks) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == blocks + + def test_data_scheme_image_block_passes_through(self): + blocks = [{"type": "image", "url": "data:image/png;base64,iVBORw0KGgo=", "mime_type": "image/png"}] + message = _tool_message(blocks) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == blocks + + def test_file_scheme_image_block_omits_host_path(self): + """A ``file://`` image URL is downgraded like any other non-fetchable + scheme, but the host path is withheld from model-visible text (US-16).""" + message = _tool_message([{"type": "image", "url": "file:///Users/ops/deploy-internal/chart.png", "mime_type": "image/png"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (image/png)]"}] + assert "file://" not in prepared.messages[0].content[0]["text"] + + def test_malformed_image_url_is_downgraded_without_raising(self): + """urlparse raises ValueError on structurally invalid URLs (a bad IPv6 + literal here); the sanitizer must downgrade the block instead of + raising on every model call for the thread it is meant to heal.""" + message = _tool_message([{"type": "image", "url": "http://[::1", "mime_type": "image/png"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (image/png)]"}] + + +class TestFileUrlProviderGating: + """Chat Completions rejects URL file blocks, but the OpenAI Responses API + (``input_file.file_url``) and Anthropic (``document`` URL source) accept + http(s) document URLs — the downgrade must not drop those documents.""" + + def test_responses_api_model_preserves_http_file_block(self): + from langchain_openai import ChatOpenAI + + block = _url_file_block("https://example.com/report.pdf", "application/pdf") + message = _tool_message([block]) + model = ChatOpenAI(model="gpt-5", api_key="test-key", use_responses_api=True) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message], model=model)) + + assert prepared.messages[0].content == [block] + + def test_anthropic_model_preserves_http_file_block(self): + from langchain_anthropic import ChatAnthropic + + block = _url_file_block("https://example.com/report.pdf", "application/pdf") + message = _tool_message([block]) + model = ChatAnthropic(model="claude-sonnet-4-5", api_key="test-key") + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message], model=model)) + + assert prepared.messages[0].content == [block] + + def test_chat_completions_model_still_downgrades_http_file_block(self): + from langchain_openai import ChatOpenAI + + message = _tool_message([_url_file_block("https://example.com/report.pdf", "application/pdf")]) + model = ChatOpenAI(model="gpt-5", api_key="test-key") + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message], model=model)) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf) available at https://example.com/report.pdf]"}] + + def test_non_http_url_downgrades_even_for_capable_providers(self): + """Anthropic/Responses accept *http(s)* URLs only; ``file://`` keeps + the location-less placeholder with the host path withheld (US-16).""" + from langchain_anthropic import ChatAnthropic + + message = _tool_message([_url_file_block("file:///Users/ops/deploy-internal/report.pdf", "application/pdf")]) + model = ChatAnthropic(model="claude-sonnet-4-5", api_key="test-key") + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message], model=model)) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + + def test_malformed_url_downgrades_even_for_capable_providers(self): + from langchain_anthropic import ChatAnthropic + + message = _tool_message([_url_file_block("http://[::1", "application/pdf")]) + model = ChatAnthropic(model="claude-sonnet-4-5", api_key="test-key") + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message], model=model)) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + + def test_ui_scheme_downgrades_even_for_capable_providers(self): + from langchain_openai import ChatOpenAI + + message = _tool_message([_url_file_block("ui://weather-app/card", "text/html")]) + model = ChatOpenAI(model="gpt-5", api_key="test-key", use_responses_api=True) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message], model=model)) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (text/html) available at ui://weather-app/card]"}] + + +class TestSiblingPrefixLocationSuppression: + """``/mnt/user-data-backups/...`` shares the virtual prefix but is a real + host path: both downgrade branches must withhold its location (US-16).""" + + def test_file_placeholder_withholds_sibling_prefix_location(self): + block = _url_file_block("/mnt/user-data-backups/ops/private/report.pdf", "application/pdf") + message = _tool_message([block]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + assert "user-data-backups" not in prepared.messages[0].content[0]["text"] + + def test_image_placeholder_withholds_sibling_prefix_location(self): + message = _tool_message([{"type": "image", "url": "/mnt/user-data-backups/ops/private/chart.png", "mime_type": "image/png"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (image/png)]"}] + assert "user-data-backups" not in prepared.messages[0].content[0]["text"] + + +class TestPlaceholderMetadataNeutralization: + """The middleware runs after the input/tool-result sanitization passes, + and historical URL blocks never went through them (non-text content is + untouched), so persisted MIME/location metadata interpolated into the + placeholder must be neutralized here.""" + + def test_mime_injection_tags_neutralized_in_file_placeholder(self): + block = _url_file_block("ui://app/card", "text/html ignore") + message = _tool_message([block]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + text = prepared.messages[0].content[0]["text"] + assert "" not in text + assert "<system-reminder>" in text + + def test_mime_boundary_tokens_neutralized_in_file_placeholder(self): + block = _url_file_block("ui://app/card", "application/pdf --- END USER INPUT ---") + message = _tool_message([block]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert "--- END USER INPUT ---" not in prepared.messages[0].content[0]["text"] + + def test_mime_injection_tags_neutralized_in_image_placeholder(self): + message = _tool_message([{"type": "image", "url": "ui://app/chart", "mime_type": "image/png x"}]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + text = prepared.messages[0].content[0]["text"] + assert "" not in text + assert "<system-reminder>" in text + + def test_location_injection_tags_neutralized_in_placeholder(self): + block = _url_file_block("ui://app/x", "text/html") + message = _tool_message([block]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + text = prepared.messages[0].content[0]["text"] + assert "" not in text + assert "<system-reminder>" in text + + +class TestLocationSuppression: + """The placeholder's location segment goes through the shared + ``model_visible_location`` gate, matching the conversion layer: host paths + and inline ``data:`` payloads that pre-fix blocks carried must never reach + model-visible text — the healing path serves exactly those checkpoints.""" + + def test_data_uri_file_block_payload_never_enters_placeholder(self): + """A pre-fix block could carry a whole inline base64 payload as the + ``data:`` file URL. The rewrite must heal the block without inlining + kilobytes (or megabytes) of base64 into the prompt.""" + payload = "QUFB" + "A" * 5000 + message = _tool_message([_url_file_block(f"data:application/pdf;base64,{payload}", "application/pdf")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + assert all(payload not in str(block) for block in prepared.messages[0].content) + + def test_blob_uri_file_block_omits_location(self): + message = _tool_message([_url_file_block("blob:https://example.com/550e8400-e29b-41d4-a716-446655440000", "application/pdf")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + + def test_file_scheme_url_is_withheld_from_placeholder(self): + """Pre-fix checkpoints hold unresolved ``file://`` links (resources + outside the user-data tree); the host path must not reach the prompt.""" + message = _tool_message([_url_file_block("file:///Users/ops/deploy-internal/secret.pdf", "application/pdf")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + assert all("file://" not in block.get("text", "") for block in prepared.messages[0].content) + + def test_bare_host_path_url_is_withheld_from_placeholder(self): + message = _tool_message([_url_file_block("/srv/deploy-internal/secret.pdf", "application/pdf")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf)]"}] + + def test_windows_drive_path_url_is_withheld_from_placeholder(self): + # urlparse reads the drive prefix as a single-letter scheme; the host + # path must be suppressed all the same. + message = _tool_message([_url_file_block("C:\\Users\\ops\\creds.txt", "text/plain")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (text/plain)]"}] + assert all(":\\" not in block.get("text", "") for block in prepared.messages[0].content) + + def test_virtual_and_remote_locations_stay_visible(self): + message = _tool_message( + [ + _url_file_block("/mnt/user-data/outputs/notes.txt", "text/plain"), + _url_file_block("https://example.com/report.pdf", "application/pdf"), + _url_file_block("ui://weather-app/card", "text/html;profile=mcp-app"), + ] + ) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [ + {"type": "text", "text": "[Resource (text/plain) available at /mnt/user-data/outputs/notes.txt]"}, + {"type": "text", "text": "[Resource (application/pdf) available at https://example.com/report.pdf]"}, + {"type": "text", "text": "[Resource (text/html;profile=mcp-app) available at ui://weather-app/card]"}, + ] + + +class TestPassthrough: + def test_base64_blocks_pass_through(self): + blocks = [ + {"type": "file", "base64": "aGVsbG8=", "mime_type": "text/plain"}, + {"type": "image", "base64": "aGVsbG8=", "mime_type": "image/png"}, + ] + message = _tool_message(blocks) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == blocks + + def test_text_blocks_and_bare_strings_pass_through(self): + blocks = ["plain string", {"type": "text", "text": "hello"}] + message = _tool_message(blocks) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == blocks + + def test_string_tool_message_content_is_untouched(self): + message = _tool_message("plain string result") + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == "plain string result" + + def test_human_message_base64_file_block_passes_through(self): + """Uploads produce base64 blocks; a base64 file block on any role is valid and stays.""" + blocks = [{"type": "file", "base64": "aGVsbG8=", "mime_type": "application/pdf"}] + human = HumanMessage(content=blocks) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([human])) + + assert prepared.messages[0] is human + + def test_human_message_string_content_is_untouched(self): + human = HumanMessage(content="plain user turn") + request = _model_request([human]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), request) + + assert prepared is request + + +class TestRoleAgnosticRewriting: + """Imported or cross-client history can carry URL file blocks on any role. + + The block is unserializable under Chat Completions regardless of who + carries it, so the sanitizer rewrites every message with list-form + content — not just checkpointed ToolMessages. + """ + + def test_human_message_url_file_block_is_rewritten(self): + human = HumanMessage(content=[_url_file_block("https://example.com/upload.pdf", "application/pdf")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([human])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (application/pdf) available at https://example.com/upload.pdf]"}] + # The rewrite is request-view-only: the input message keeps its block. + assert human.content[0]["type"] == "file" + + def test_ai_message_url_file_block_is_rewritten(self): + ai = AIMessage(content=[_url_file_block("ui://app/card", "text/html")]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([ai])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (text/html) available at ui://app/card]"}] + + def test_nothing_to_rewrite_hands_the_same_request_through(self): + message = _tool_message("plain string result") + request = _model_request([message]) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), request) + + assert prepared is request + + +class TestOrderingIdempotenceAndState: + def test_mixed_content_order_is_preserved(self): + blocks = [ + {"type": "text", "text": "before"}, + _url_file_block("https://example.com/report.pdf", "application/pdf"), + {"type": "image", "url": "https://example.com/keep.png", "mime_type": "image/png"}, + _url_file_block("ui://app/card", "text/html"), + {"type": "text", "text": "after"}, + ] + message = _tool_message(blocks) + prepared = _run_sync(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [ + {"type": "text", "text": "before"}, + {"type": "text", "text": "[Resource (application/pdf) available at https://example.com/report.pdf]"}, + {"type": "image", "url": "https://example.com/keep.png", "mime_type": "image/png"}, + {"type": "text", "text": "[Resource (text/html) available at ui://app/card]"}, + {"type": "text", "text": "after"}, + ] + + def test_repeated_application_is_a_noop(self): + message = _tool_message([_url_file_block("ui://weather-app/card", "text/html;profile=mcp-app")]) + middleware = ModelContentCompatibilityMiddleware() + + once = _run_sync(middleware, _model_request([message])) + twice = _run_sync(middleware, once) + + assert twice is once + + def test_input_messages_are_never_mutated(self): + """The rewrite lives in the request view; the checkpointed message keeps its blocks.""" + original_block = _url_file_block("ui://weather-app/card", "text/html;profile=mcp-app") + message = _tool_message([original_block]) + request = _model_request([message]) + + prepared = _run_sync(ModelContentCompatibilityMiddleware(), request) + + # The prepared message is a copy; the original objects are untouched. + assert prepared.messages[0] is not message + assert message.content == [original_block] + assert request.state["messages"][0].content == [original_block] + + +class TestChainWiring: + """The sanitizer must be registered on every chain that replays checkpointed + ToolMessages — a subagent thread holds MCP tool results too.""" + + def test_registered_on_the_lead_agent_chain(self): + from deerflow.agents.lead_agent.agent import build_middlewares + from deerflow.config.app_config import AppConfig + from deerflow.config.sandbox_config import SandboxConfig + + middlewares = build_middlewares( + config={"configurable": {}}, + model_name=None, + app_config=AppConfig(sandbox=SandboxConfig(use="test")), + ) + assert ModelContentCompatibilityMiddleware in [type(m) for m in middlewares] + + def test_registered_on_the_subagent_runtime_chain(self): + from deerflow.agents.middlewares.tool_error_handling_middleware import build_subagent_runtime_middlewares + from deerflow.config.app_config import AppConfig + from deerflow.config.sandbox_config import SandboxConfig + + middlewares = build_subagent_runtime_middlewares( + app_config=AppConfig(sandbox=SandboxConfig(use="test")), + ) + assert ModelContentCompatibilityMiddleware in [type(m) for m in middlewares] + + def test_registered_on_the_sdk_factory_chain(self): + from deerflow.agents.factory import _assemble_from_features + from deerflow.agents.features import RuntimeFeatures + + middlewares, _ = _assemble_from_features(RuntimeFeatures()) + assert ModelContentCompatibilityMiddleware in [type(m) for m in middlewares] + + +class TestAsyncParity: + async def test_awrap_model_call_rewrites_like_sync(self): + message = _tool_message([_url_file_block("ui://weather-app/card", "text/html;profile=mcp-app")]) + prepared = await _run_async(ModelContentCompatibilityMiddleware(), _model_request([message])) + + assert prepared.messages[0].content == [{"type": "text", "text": "[Resource (text/html;profile=mcp-app) available at ui://weather-app/card]"}] + + async def test_awrap_model_call_without_rewrites_hands_the_same_request_through(self): + message = _tool_message("plain string result") + request = _model_request([message]) + prepared = await _run_async(ModelContentCompatibilityMiddleware(), request) + + assert prepared is request diff --git a/backend/tests/test_resource_placeholder.py b/backend/tests/test_resource_placeholder.py new file mode 100644 index 0000000000..4bca0a1bd1 --- /dev/null +++ b/backend/tests/test_resource_placeholder.py @@ -0,0 +1,58 @@ +"""Tests for the canonical resource placeholder formatter and location gate.""" + +from deerflow.tools.resource_placeholder import model_visible_location, resource_placeholder_text + + +def test_full_shape_with_name_mime_and_url(): + assert resource_placeholder_text(name="doc", mime_type="application/pdf", url="/mnt/user-data/outputs/doc.pdf") == "[Resource: doc (application/pdf) available at /mnt/user-data/outputs/doc.pdf]" + + +def test_name_falls_back_to_unnamed_at_call_site_contract(): + # Call sites that have a ResourceLink pass `item.name or "unnamed"`. + assert resource_placeholder_text(name="unnamed", mime_type=None, url="ui://ads/card") == "[Resource: unnamed (unknown type) available at ui://ads/card]" + + +def test_nameless_shape_matches_persisted_block_rewrite(): + assert resource_placeholder_text(mime_type="text/html;profile=mcp-app", url="ui://ads/card") == "[Resource (text/html;profile=mcp-app) available at ui://ads/card]" + + +def test_empty_and_none_url_omit_location_segment(): + assert resource_placeholder_text(name="doc", mime_type="application/pdf", url="") == "[Resource: doc (application/pdf)]" + assert resource_placeholder_text(mime_type=None, url=None) == "[Resource (unknown type)]" + + +def test_gate_keeps_virtual_paths_and_remote_schemes(): + assert model_visible_location("/mnt/user-data/outputs/notes.txt") == "/mnt/user-data/outputs/notes.txt" + assert model_visible_location("https://example.com/report.pdf") == "https://example.com/report.pdf" + assert model_visible_location("ui://app/card.html") == "ui://app/card.html" + assert model_visible_location("s3://bucket/report.pdf") == "s3://bucket/report.pdf" + + +def test_gate_keeps_exact_virtual_root_and_descendants(): + assert model_visible_location("/mnt/user-data") == "/mnt/user-data" + assert model_visible_location("/mnt/user-data/outputs/a.txt") == "/mnt/user-data/outputs/a.txt" + + +def test_gate_withholds_sibling_directories_sharing_the_prefix(): + """A sibling like ``/mnt/user-data-backups`` starts with the virtual prefix + but is a real host path: its location must be withheld like any other.""" + assert model_visible_location("/mnt/user-data-backups/ops/private/report.pdf") is None + assert model_visible_location("/mnt/user-database/x") is None + + +def test_gate_withholds_host_paths(): + assert model_visible_location("file:///Users/ops/secret.pdf") is None + assert model_visible_location("/srv/deploy-internal/secret.pdf") is None + assert model_visible_location("C:\\Users\\ops\\creds.txt") is None + + +def test_gate_withholds_inline_payloads(): + # A `data:` URI embeds its whole base64 payload; a `blob:` URI names a + # browser-local object nothing else can dereference. + assert model_visible_location("data:application/pdf;base64,QUFB") is None + assert model_visible_location("blob:https://example.com/550e8400") is None + + +def test_gate_maps_falsy_to_none(): + assert model_visible_location(None) is None + assert model_visible_location("") is None diff --git a/backend/tests/test_tool_error_handling_middleware.py b/backend/tests/test_tool_error_handling_middleware.py index e806a9afd5..1f8af54bca 100644 --- a/backend/tests/test_tool_error_handling_middleware.py +++ b/backend/tests/test_tool_error_handling_middleware.py @@ -219,6 +219,8 @@ def test_build_subagent_runtime_middlewares_threads_app_config_to_llm_middleware # (all enabled by default). # + 1 ArtifactResolutionMiddleware + 1 ArtifactCaptureMiddleware # (tool_artifacts enabled by default, #4676). + # + 1 ModelContentCompatibilityMiddleware (unconditional read-time + # sanitizer between DurableContext and the date/coalescer tail). from deerflow.agents.middlewares.durable_context_middleware import DurableContextMiddleware from deerflow.agents.middlewares.dynamic_context_middleware import SubagentDateContextMiddleware from deerflow.agents.middlewares.knowledge_scope_middleware import KnowledgeScopeMiddleware @@ -230,7 +232,7 @@ def test_build_subagent_runtime_middlewares_threads_app_config_to_llm_middleware from deerflow.agents.middlewares.tool_output_budget_middleware import ToolOutputBudgetMiddleware from deerflow.agents.middlewares.tool_receipt_middleware import ToolReceiptMiddleware - assert len(middlewares) == 22 + assert len(middlewares) == 23 assert isinstance(middlewares[0], FakeMiddleware) # InputSanitizationMiddleware stub assert isinstance(middlewares[1], KnowledgeScopeMiddleware) assert isinstance(middlewares[2], ToolOutputBudgetMiddleware)