diff --git a/README.md b/README.md index 3d0c49750..913d6d982 100644 --- a/README.md +++ b/README.md @@ -778,6 +778,20 @@ This is the difference between a chatbot with tool access and an agent with an a └── outputs/ ← final deliverables ``` +### Agentic Browser Control + +Reading a page is not the same as *using* one. Alongside the read-only `web_fetch` and `web_capture` tools, DeerFlow ships an optional agentic browser tool group that keeps a live, per-conversation browser session so the agent can actually operate a page — navigate, read the interactive elements, click, type, submit forms, and follow multi-step flows on JavaScript-heavy sites. + +Each action returns a fresh snapshot of the page's interactive elements, each addressed by a stable `[ref]` number, so the agent acts on what it just observed instead of guessing selectors. Outbound URLs are SSRF-screened by default. It is powered by Playwright and shipped as an optional extra so the core install stays lean: + +```bash +cd backend +uv sync --extra browser +uv run playwright install chromium +``` + +Then uncomment the `group: browser` tool entries in `config.yaml` (`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_get_text`, `browser_back`, `browser_screenshot`, `browser_close`). `make dev` / Docker startup detects an enabled `browser_navigate` tool and preserves the `browser` extra on dependency syncs. The Gateway fails startup if browser control is configured but Playwright is missing, and `/api/features` hides the Browser UI unless the backend can actually serve it. Keep `headless: true` and `allow_private_addresses: false` for anything but local, trusted debugging. Attaching to an existing Chrome with `cdp_url` cannot enforce DeerFlow's subresource/redirect SSRF guard and therefore fails closed unless `allow_unguarded_cdp: true` explicitly acknowledges that risk; use it only with a trusted local browser. Browser sessions are process-local; keep `GATEWAY_WORKERS=1` while this tool group is enabled because ordinary uvicorn worker dispatch does not provide thread affinity. + ### Context Engineering **Isolated Sub-Agent Context**: Each sub-agent runs in its own isolated context. This means that the sub-agent will not be able to see the context of the main agent or other sub-agents. This is important to ensure that the sub-agent is able to focus on the task at hand and not be distracted by the context of the main agent or other sub-agents. diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 398fc0d8b..14cb4ea6c 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -163,6 +163,11 @@ Boundary check (harness → app import firewall): CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml). +Agentic browser sessions are process-local. The Gateway startup safety gate rejects +`GATEWAY_WORKERS > 1` when `browser_navigate` is configured, because ordinary +uvicorn worker dispatch does not provide thread affinity for browser tools, REST +navigation, and the Live WebSocket. + ## Architecture ### Harness / App Split @@ -329,7 +334,7 @@ Localhost persistence deliberately reads the direct request `Host` and ignores ` | Router | Endpoints | |--------|-----------| | **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details | -| **Features** (`/api/features`) | `GET /` - report config-gated feature availability (currently `agents_api.enabled`) for frontend UI gating | +| **Features** (`/api/features`) | `GET /` - report config-gated feature availability (`agents_api.enabled`, `browser_control.enabled`) for frontend UI gating | | **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured | | **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) | | **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes | @@ -454,6 +459,8 @@ Scheduled-task runtime note: - `firecrawl/` - Web scraping via Firecrawl API - `image_search/` - Image search via DuckDuckGo - `aio_sandbox/` - Docker-based isolation (`AioSandboxProvider`) +- `browser_automation/` - Agentic browser control (stateful `navigate → observe → click/type` loop) via Playwright, distinct from the read-only `web_fetch`/`web_capture` tools. Tools: `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_get_text`, `browser_back`, `browser_screenshot`, `browser_close` (config `group: browser`). A process-local `BrowserSessionManager` owns one private, loop-affine Playwright event-loop thread (same pattern as the BoxLite provider) so a per-thread browser session survives across turns regardless of the caller's loop (Gateway / TUI / test). Each action returns a fresh page snapshot whose interactive elements are addressed by a stable numeric `[ref]` index (stamped as `data-df-ref` during snapshot), so the model acts on what it just observed instead of holding stale handles or guessing selectors. URLs are SSRF-screened via the shared `validate_public_http_url` (opt-out `allow_private_addresses` only for intentional internal targets). CDP attachment cannot install the request guard on an existing Chrome context, so `cdp_url` fails closed unless the operator explicitly sets `allow_unguarded_cdp: true` for a trusted local browser. Browser REST/Live access also requires an exact non-NULL thread owner, rather than the general legacy shared-thread policy, because retained pages may contain authenticated state. Session admission is a hard `max_sessions` cap: pinned Live/operation sessions are never evicted, and a new thread is rejected when no unpinned session can be closed; one Live viewer owns a session at a time. Optional dependency: `cd backend && uv sync --extra browser && uv run playwright install chromium`; `scripts/detect_uv_extras.py` preserves the extra when `config.yaml` enables `browser_navigate`, and Gateway startup fails fast if configured browser control cannot import Playwright. Tests: `tests/test_browser_automation.py` (mocked tools + a real-Chromium integration test guarded by `importorskip`); `tests/manual_browser_live_check.py` is a manual DeepSeek-driven end-to-end check (not collected by pytest). + Live UI input dispatch is kept independent from JPEG capture: non-move actions start a rate-limited background refresh loop, so pointer, wheel, or keyboard input stays responsive while continuous gestures still produce frames throughout the interaction. Additional providers also live here (`boxlite`, `brave`, `browserless`, `crawl4ai`, `ddg_search`, `e2b_sandbox`, `exa`, `fastcrw`, `groundroute`, `infoquest`, `searxng`, `serper`); see each subpackage for specifics. diff --git a/backend/app/gateway/app.py b/backend/app/gateway/app.py index 65b7a35e3..47caf4200 100644 --- a/backend/app/gateway/app.py +++ b/backend/app/gateway/app.py @@ -8,6 +8,7 @@ from fastapi.middleware.cors import CORSMiddleware from app.gateway.auth_disabled import warn_if_auth_disabled_enabled from app.gateway.auth_middleware import AuthMiddleware +from app.gateway.browser_capability import ensure_browser_runtime_available from app.gateway.config import get_gateway_config from app.gateway.csrf_middleware import CSRFMiddleware, get_configured_cors_origins from app.gateway.deps import langgraph_runtime @@ -16,6 +17,7 @@ from app.gateway.routers import ( artifacts, assistants_compat, auth, + browser, channel_connections, channels, console, @@ -181,6 +183,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: try: startup_config = get_app_config() configure_logging(startup_config) + ensure_browser_runtime_available(startup_config) logger.info("Configuration loaded successfully") warn_if_auth_disabled_enabled() except Exception as e: @@ -301,6 +304,23 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: except Exception: logger.exception("Failed to stop scheduled task service") + try: + from deerflow.community.browser_automation import get_browser_session_manager + + closed = await asyncio.wait_for( + get_browser_session_manager().close_all_sessions(), + timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS, + ) + if closed: + logger.info("Closed %d browser session(s)", closed) + except TimeoutError: + logger.warning( + "Browser session shutdown exceeded %.1fs; proceeding with worker exit.", + _SHUTDOWN_HOOK_TIMEOUT_SECONDS, + ) + except Exception: + logger.exception("Failed to close browser sessions") + # Drain the memory backend's pending-update buffer before the worker # exits (best-effort, bounded). IM channels and the scheduler are # already stopped above, so no new IM/scheduler updates arrive during @@ -317,9 +337,10 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: # would (review #6 on the original PR). # # K8s caveat: ``shutdown_flush_timeout_seconds`` must fit inside the - # pod's ``terminationGracePeriodSeconds`` (channel stop + this drain + - # buffer), set on the gateway Helm deployment -- or K8s SIGKILLs the - # drain mid-flight and the loss this is fixing is silently re-introduced. + # pod's ``terminationGracePeriodSeconds`` (channel stop + browser + # session close + this drain + buffer), set on the gateway Helm + # deployment -- or K8s SIGKILLs the drain mid-flight and the loss this + # is fixing is silently re-introduced. try: app_cfg = get_app_config() if app_cfg.memory.enabled: @@ -487,6 +508,9 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for # Artifacts API is mounted at /api/threads/{thread_id}/artifacts app.include_router(artifacts.router) + # Browser API is mounted at /api/threads/{thread_id}/browser + app.include_router(browser.router) + # Uploads API is mounted at /api/threads/{thread_id}/uploads app.include_router(uploads.router) diff --git a/backend/app/gateway/browser_capability.py b/backend/app/gateway/browser_capability.py new file mode 100644 index 000000000..ccfe8e275 --- /dev/null +++ b/backend/app/gateway/browser_capability.py @@ -0,0 +1,72 @@ +"""Shared browser-control capability checks for Gateway surfaces.""" + +from __future__ import annotations + +import importlib.util +from dataclasses import dataclass +from typing import Any + +from deerflow.community.browser_automation.session import browser_multi_worker_error +from deerflow.config.app_config import AppConfig + + +@dataclass(frozen=True) +class BrowserCapability: + """Frontend/API availability for the agentic browser control surface.""" + + configured: bool + available: bool + reason: str | None = None + + +def _tool_config(config: AppConfig) -> Any | None: + get_tool_config = getattr(config, "get_tool_config", None) + if callable(get_tool_config) and callable(getattr(type(config), "get_tool_config", None)): + return get_tool_config("browser_navigate") + return next( + (tool for tool in (getattr(config, "tools", None) or []) if getattr(tool, "name", None) == "browser_navigate"), + None, + ) + + +def _tool_extra(tool_cfg: Any) -> dict[str, Any]: + extra = getattr(tool_cfg, "model_extra", None) + return extra if isinstance(extra, dict) else {} + + +def browser_capability(config: AppConfig) -> BrowserCapability: + """Return whether browser control can actually serve frontend requests.""" + + tool_cfg = _tool_config(config) + if tool_cfg is None: + return BrowserCapability(configured=False, available=False, reason="browser_navigate is not configured") + + worker_error = browser_multi_worker_error() + if worker_error is not None: + return BrowserCapability(configured=True, available=False, reason=worker_error) + + extra = _tool_extra(tool_cfg) + cdp_url = extra.get("cdp_url") + if isinstance(cdp_url, str) and cdp_url.strip() and extra.get("allow_unguarded_cdp") is not True: + return BrowserCapability( + configured=True, + available=False, + reason="cdp_url requires allow_unguarded_cdp: true because DeerFlow cannot enforce the SSRF request guard on a CDP-attached browser", + ) + + if importlib.util.find_spec("playwright") is None or importlib.util.find_spec("playwright.async_api") is None: + return BrowserCapability( + configured=True, + available=False, + reason="Playwright is not installed; install the backend browser extra and run `playwright install chromium`", + ) + + return BrowserCapability(configured=True, available=True) + + +def ensure_browser_runtime_available(config: AppConfig) -> None: + """Fail startup when browser control is configured but cannot run.""" + + capability = browser_capability(config) + if capability.configured and not capability.available: + raise RuntimeError(capability.reason or "Browser automation is not available") diff --git a/backend/app/gateway/deps.py b/backend/app/gateway/deps.py index a41b7d8fa..faf5634d8 100644 --- a/backend/app/gateway/deps.py +++ b/backend/app/gateway/deps.py @@ -27,6 +27,7 @@ from typing import TYPE_CHECKING, TypeVar, cast from fastapi import FastAPI, HTTPException, Request from langgraph.types import Checkpointer +from deerflow.community.browser_automation.session import browser_multi_worker_error from deerflow.config.app_config import AppConfig, get_app_config from deerflow.persistence.feedback import FeedbackRepository from deerflow.runtime import RunContext, RunManager, StreamBridge @@ -46,14 +47,25 @@ logger = logging.getLogger(__name__) _RUN_DRAIN_TIMEOUT_SECONDS = 5.0 +def _browser_tools_enabled_in_config(config: AppConfig) -> bool: + """Return whether process-local agentic browser sessions are configured.""" + get_tool_config = getattr(config, "get_tool_config", None) + if callable(get_tool_config): + return get_tool_config("browser_navigate") is not None + return any(getattr(tool, "name", None) == "browser_navigate" for tool in (getattr(config, "tools", None) or [])) + + def _enforce_postgres_for_multi_worker(config: AppConfig) -> None: - """Refuse to start when GATEWAY_WORKERS > 1 and safety preconditions are not met. + """Refuse unsafe multi-worker configurations before persistence starts. - Two checks (both must pass for multi-worker): + Three checks (all must pass for multi-worker): - 1. The DB backend must be Postgres — SQLite write-locks cannot support + 1. Process-local browser sessions must be disabled. Browser tools keep + Chromium and Playwright objects in one worker's memory, while ordinary + uvicorn dispatch provides no thread-id affinity. + 2. The DB backend must be Postgres — SQLite write-locks cannot support concurrent multi-process access. - 2. ``run_ownership.heartbeat_enabled`` must be True — without heartbeat, + 3. ``run_ownership.heartbeat_enabled`` must be True — without heartbeat, every run has a NULL lease, so reconciliation treats all inflight runs as orphans and Worker B would kill Worker A's live runs on every rolling update or scale-up. @@ -70,6 +82,9 @@ def _enforce_postgres_for_multi_worker(config: AppConfig) -> None: if workers <= 1: return + if _browser_tools_enabled_in_config(config): + raise SystemExit(browser_multi_worker_error(workers)) + backend = getattr(config.database, "backend", None) if backend != "postgres": raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.") diff --git a/backend/app/gateway/routers/__init__.py b/backend/app/gateway/routers/__init__.py index e1750469d..a948c522b 100644 --- a/backend/app/gateway/routers/__init__.py +++ b/backend/app/gateway/routers/__init__.py @@ -1,6 +1,7 @@ from . import ( artifacts, assistants_compat, + browser, input_polish, mcp, models, @@ -15,6 +16,7 @@ from . import ( __all__ = [ "artifacts", "assistants_compat", + "browser", "input_polish", "mcp", "models", diff --git a/backend/app/gateway/routers/browser.py b/backend/app/gateway/routers/browser.py new file mode 100644 index 000000000..ce8e264c8 --- /dev/null +++ b/backend/app/gateway/routers/browser.py @@ -0,0 +1,467 @@ +import asyncio +import contextlib +import json +import logging + +from fastapi import APIRouter, HTTPException, Request, WebSocket, WebSocketDisconnect +from pydantic import BaseModel, Field + +from app.gateway.authz import require_permission +from app.gateway.browser_capability import browser_capability +from deerflow.config.paths import get_paths +from deerflow.runtime.user_context import get_effective_user_id, reset_current_user, set_current_user + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api", tags=["browser"]) + + +class BrowserNavigateRequest(BaseModel): + url: str = Field(..., description="The http(s) URL to open in the thread's live browser session") + + +class BrowserNavigateResponse(BaseModel): + screenshot: str | None = Field(default=None, description="Virtual artifact path of the captured screenshot") + url: str = Field(..., description="The resolved URL after navigation") + title: str = Field(default="", description="The page title after navigation") + + +def _normalize_browser_seed_url(url: str | None) -> str: + return (url or "").split("#", 1)[0].rstrip("/") + + +def _should_apply_browser_seed(current: str | None, seed: str | None) -> bool: + if not seed: + return False + if not current or current == "about:blank": + return True + return _normalize_browser_seed_url(current) != _normalize_browser_seed_url(seed) + + +def _browser_tools_enabled() -> bool: + """Whether the browser tools are turned on in config. + + The live browser HTTP/WS endpoints are an opt-in surface: they must only be + reachable when the operator has enabled the ``browser_navigate`` tool in + ``config.yaml``. Merely having Playwright importable (it may be preinstalled + in a base image) is not sufficient — otherwise the endpoints would expose + server-side browser control the operator never turned on. + """ + from deerflow.config import get_app_config + + with contextlib.suppress(Exception): + capability = browser_capability(get_app_config()) + if capability.configured and not capability.available and capability.reason: + logger.error("browser automation disabled: %s", capability.reason) + return capability.available + return False + + +async def _browser_thread_owned_by(thread_store, thread_id: str, user_id: str) -> bool: + """Require an explicit owner for browser control's high-trust surface. + + General thread reads retain compatibility with legacy ``user_id=NULL`` + rows. A retained browser may contain authenticated cookies and page data, + so REST navigation and the Live socket deliberately use a stricter policy: + only an existing row whose owner exactly matches may drive it. + """ + record = await thread_store.get(thread_id, user_id=user_id) + return record is not None and record.get("user_id") == user_id + + +@router.post( + "/threads/{thread_id}/browser/navigate", + response_model=BrowserNavigateResponse, + summary="Navigate The Live Browser Session", + description="Steer the thread's live browser session to a URL from the UI and capture a screenshot.", +) +@require_permission("threads", "write", owner_check=True, require_existing=True) +async def navigate_browser(thread_id: str, body: BrowserNavigateRequest, request: Request) -> BrowserNavigateResponse: + user_id = str(request.state.auth.user.id) + thread_store = getattr(request.app.state, "thread_store", None) + if thread_store is None or not await _browser_thread_owned_by(thread_store, thread_id, user_id): + raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found") + + if not _browser_tools_enabled(): + raise HTTPException(status_code=404, detail="Browser automation is not enabled") + + try: + from deerflow.community.browser_automation import navigate_and_capture, redact_browser_url + except ImportError as exc: # Playwright is an optional dependency. + raise HTTPException(status_code=501, detail="Browser automation is not available") from exc + + url = body.url.strip() + if not url: + raise HTTPException(status_code=400, detail="URL is required") + + outputs_path = get_paths().sandbox_outputs_dir(thread_id, user_id=get_effective_user_id()) + try: + result = await navigate_and_capture(thread_id=thread_id, url=url, outputs_path=outputs_path) + except ValueError as exc: + # SSRF / URL validation failure. + raise HTTPException(status_code=400, detail=str(exc)) from exc + except Exception as exc: + logger.error( + "Browser navigate failed: thread_id=%s url=%s err_type=%s", + thread_id, + redact_browser_url(url), + type(exc).__name__, + ) + raise HTTPException(status_code=502, detail="Browser navigation failed") from exc + + return BrowserNavigateResponse(**result) + + +async def _authenticate_ws(websocket: WebSocket): + """Resolve the user for a WebSocket, honoring auth-disabled mode. + + WebSocket upgrades bypass ``AuthMiddleware`` (a BaseHTTPMiddleware), so we + replicate the minimal cookie → user resolution here. Returns the user or + ``None`` when unauthenticated. + """ + from app.gateway.auth import decode_token + from app.gateway.auth.errors import TokenError + from app.gateway.auth_disabled import get_auth_disabled_user, is_auth_disabled + from app.gateway.deps import get_local_provider + + access_token = websocket.cookies.get("access_token") + if access_token: + payload = decode_token(access_token) + if not isinstance(payload, TokenError): + provider = get_local_provider() + user = await provider.get_user(payload.sub) + if user is not None and user.token_version == payload.ver: + return user + if is_auth_disabled(): + return get_auth_disabled_user() + return None + + +def _ws_origin_allowed(websocket: WebSocket) -> bool: + """Reject cross-origin WebSocket upgrades (WS-CSRF defense). + + WS upgrades bypass ``CSRFMiddleware`` (also a BaseHTTPMiddleware), so a + cross-origin page could otherwise open this socket riding the victim's + cookie and both observe frames and drive their authenticated browser. The + ``Origin`` header is browser-controlled but always sent on cross-origin + upgrades, so validating it is a standard, cheap mitigation. + + Allow when: no ``Origin`` (non-browser clients such as native ws/tests do + not send it), the origin is an explicitly configured CORS origin, or it is + same-origin with the upgrade target's host. The WS scheme (ws/wss) differs + from the page scheme (http/https), so same-origin compares host[:port]. + """ + from app.gateway.csrf_middleware import ( + _first_header_value, + _normalize_origin, + get_configured_cors_origins, + ) + + origin = websocket.headers.get("origin") + if not origin: + return True + + normalized = _normalize_origin(origin) + if normalized is None: + return False + if normalized in get_configured_cors_origins(): + return True + + target_host = _first_header_value(websocket.headers.get("x-forwarded-host")) or websocket.headers.get("host") + if target_host: + normalized_host = normalized.split("://", 1)[-1] + if normalized_host == target_host.strip().lower(): + return True + return False + + +@router.websocket("/threads/{thread_id}/browser/stream") +async def browser_stream(websocket: WebSocket, thread_id: str) -> None: + """Bidirectional live browser stream. + + Server → client: JSON ``{"type":"frame","data":""}`` frames + captured via CDP screencast. Client → server: input events (click, move, + down, up, wheel, key, text, navigate) that drive the live page. + """ + user = await _authenticate_ws(websocket) + if user is None: + await websocket.close(code=4401) + return + + if not _ws_origin_allowed(websocket): + # Cross-origin upgrade — reject before touching any session (WS-CSRF). + await websocket.close(code=4403) + return + + thread_store = getattr(websocket.app.state, "thread_store", None) + if thread_store is None: + # Fail closed: the live stream drives a real browser (cookies, + # logged-in pages), so if the ownership store can't be resolved we must + # deny rather than let any authenticated caller attach to any thread's + # retained session. + await websocket.close(code=4404) + return + # Browser control is stricter than ordinary legacy-thread access: NULL-owner + # rows are not shared because a retained page may expose cookies or account + # data from a previous authenticated browser session. + if not await _browser_thread_owned_by(thread_store, thread_id, str(user.id)): + await websocket.close(code=4404) + return + + if not _browser_tools_enabled(): + await websocket.close(code=4404) + return + + try: + from deerflow.community.browser_automation import ( + BrowserLiveViewerError, + BrowserSessionCapacityError, + get_browser_session_manager, + validate_browser_url, + ) + except ImportError: + await websocket.close(code=4501) + return + + await websocket.accept() + + token = set_current_user(user) + loop = asyncio.get_running_loop() + frame_queue: asyncio.Queue[str] = asyncio.Queue(maxsize=4) + send_lock = asyncio.Lock() + input_event = asyncio.Event() + input_queue: asyncio.Queue[dict] = asyncio.Queue(maxsize=64) + pending_move: dict | None = None + pending_wheel: dict | None = None + + async def _send_payload(payload: dict) -> None: + async with send_lock: + await websocket.send_text(json.dumps(payload)) + + def _on_frame(data: str) -> None: + # Invoked on the private Playwright loop; hop to this loop and drop the + # oldest frame when the client can't keep up (screencast is lossy). + def _enqueue() -> None: + if frame_queue.full(): + with contextlib.suppress(asyncio.QueueEmpty): + frame_queue.get_nowait() + with contextlib.suppress(asyncio.QueueFull): + frame_queue.put_nowait(data) + + loop.call_soon_threadsafe(_enqueue) + + # Match the tool's session config (headless/viewport/cdp_url) so the live + # stream reuses the same session the agent drives — including CDP-attach to + # the user's real Chrome when configured. + from deerflow.config import get_app_config + + tool_cfg = get_app_config().get_tool_config("browser_navigate") + extra = (tool_cfg.model_extra or {}) if tool_cfg is not None else {} + + def _cfg_int(key: str, default: int) -> int: + value = extra.get(key) + return value if isinstance(value, int) and not isinstance(value, bool) else default + + def _cfg_bool(key: str, default: bool) -> bool: + value = extra.get(key) + return value if isinstance(value, bool) else default + + def _cfg_str(key: str) -> str | None: + value = extra.get(key) + return value.strip() or None if isinstance(value, str) else None + + manager = get_browser_session_manager() + try: + session_lease = manager.acquire_session( + thread_id, + headless=_cfg_bool("headless", True), + timeout_ms=_cfg_int("timeout_ms", 30000), + viewport={"width": _cfg_int("viewport_width", 1280), "height": _cfg_int("viewport_height", 720)}, + cdp_url=_cfg_str("cdp_url"), + allow_unguarded_cdp=_cfg_bool("allow_unguarded_cdp", False), + url_guard=validate_browser_url, + ) + session = session_lease.__enter__() + except BrowserSessionCapacityError: + await websocket.close(code=4429) + reset_current_user(token) + return + except Exception: + await websocket.close(code=4501) + reset_current_user(token) + return + + async def _pump_frames() -> None: + while True: + data = await frame_queue.get() + await _send_payload({"type": "frame", "data": data}) + + async def _send_url() -> None: + # Report the page's real URL so the client's address bar reflects the + # actual location after navigations, redirects, and history moves — not + # the optimistic value the user typed. + with contextlib.suppress(Exception): + url = await session.current_url() + if url: + await _send_payload({"type": "url", "url": url}) + + async def _send_tabs() -> None: + with contextlib.suppress(Exception): + tabs = await session.tabs() + await _send_payload( + { + "type": "tabs", + "tabs": [ + { + "index": tab.index, + "title": tab.title, + "url": tab.url, + "active": tab.active, + } + for tab in tabs + ], + }, + ) + + async def _poll_location() -> None: + # The agent drives the same session through its tools (browser_navigate / + # click / type), which do not flow through this socket's input handler, so + # those location changes would otherwise never reach the address bar/tabs. + # Tool actions push their own settled Live frame after the inline + # screenshot is captured; this poll only keeps URL/tabs metadata in sync. + # Avoid screenshotting here, because opening a Live panel already primes a + # frame and GitHub-style SPAs can generate many URL/render transitions. + last_url: str | None = None + while True: + await asyncio.sleep(1.0) + with contextlib.suppress(Exception): + url = await session.current_url() + if url and url != last_url: + last_url = url + await _send_payload({"type": "url", "url": url}) + await _send_tabs() + + def _queue_input(event: dict) -> None: + nonlocal pending_move, pending_wheel + etype = event.get("type") + if etype == "move": + pending_move = event + elif etype == "wheel": + if pending_wheel is None: + pending_wheel = event + else: + pending_wheel = { + **event, + "dx": float(pending_wheel.get("dx", 0)) + float(event.get("dx", 0)), + "dy": float(pending_wheel.get("dy", 0)) + float(event.get("dy", 0)), + } + else: + pending_move = None + pending_wheel = None + if input_queue.full(): + with contextlib.suppress(asyncio.QueueEmpty): + input_queue.get_nowait() + with contextlib.suppress(asyncio.QueueFull): + input_queue.put_nowait(event) + input_event.set() + + def _has_pending_input() -> bool: + return pending_move is not None or pending_wheel is not None or not input_queue.empty() + + def _take_input() -> dict | None: + nonlocal pending_move, pending_wheel + if not input_queue.empty(): + return input_queue.get_nowait() + if pending_wheel is not None: + event = pending_wheel + pending_wheel = None + return event + if pending_move is not None: + event = pending_move + pending_move = None + return event + return None + + async def _read_inputs() -> None: + while True: + try: + raw = await websocket.receive_text() + except (WebSocketDisconnect, RuntimeError): + return + try: + event = json.loads(raw) + except (json.JSONDecodeError, TypeError): + continue + if isinstance(event, dict): + _queue_input(event) + + async def _process_inputs() -> None: + while True: + await input_event.wait() + while True: + event = _take_input() + if event is None: + input_event.clear() + if _has_pending_input(): + input_event.set() + break + if event.get("type") == "navigate": + # SSRF-screen client-driven navigations with the same policy + # the agent tools enforce; reject rather than dispatch. + url = event.get("url") + reason = validate_browser_url(url) if isinstance(url, str) else "Error: invalid navigation URL" + if reason is not None: + await _send_payload({"type": "nav_rejected", "url": url, "message": reason}) + continue + try: + await session.dispatch_input(event) + except Exception as exc: + logger.warning("browser stream input failed: %s", exc) + else: + # A location may have changed — resync the client's URL bar. + if event.get("type") in {"navigate", "back", "forward", "click", "activate_tab"}: + await _send_url() + await _send_tabs() + + pump_task = asyncio.create_task(_pump_frames()) + input_task: asyncio.Task | None = None + reader_task: asyncio.Task | None = None + poll_task: asyncio.Task | None = None + try: + # Seed the live page from the latest browser_view URL. A thread can have + # a stale browser session from an earlier panel/live attempt; if that + # page differs from the latest visible browser artifact, align Live with + # what the user expects instead of requiring an off/on reconnect. + seed = websocket.query_params.get("seed") + if seed and validate_browser_url(seed) is None: + with contextlib.suppress(Exception): + current = await session.current_url() + if _should_apply_browser_seed(current, seed): + await session.navigate(seed) + try: + await session.start_screencast(_on_frame) + except BrowserLiveViewerError: + await websocket.close(code=4409) + return + await _send_url() + await _send_tabs() + input_task = asyncio.create_task(_process_inputs()) + reader_task = asyncio.create_task(_read_inputs()) + poll_task = asyncio.create_task(_poll_location()) + await reader_task + except WebSocketDisconnect: + pass + except Exception as exc: + logger.exception("browser stream error: thread_id=%s err=%s", thread_id, exc) + finally: + pump_task.cancel() + if input_task is not None: + input_task.cancel() + if reader_task is not None: + reader_task.cancel() + if poll_task is not None: + poll_task.cancel() + with contextlib.suppress(Exception): + await session.stop_screencast(_on_frame) + session_lease.__exit__(None, None, None) + reset_current_user(token) diff --git a/backend/app/gateway/routers/features.py b/backend/app/gateway/routers/features.py index 2b1b38326..51e3f63f3 100644 --- a/backend/app/gateway/routers/features.py +++ b/backend/app/gateway/routers/features.py @@ -9,6 +9,7 @@ on the next request without a restart (config hot-reload boundary). from fastapi import APIRouter, Depends from pydantic import BaseModel, Field +from app.gateway.browser_capability import browser_capability from app.gateway.deps import get_config from deerflow.config.app_config import AppConfig @@ -21,10 +22,17 @@ class AgentsApiFeature(BaseModel): enabled: bool = Field(..., description="Whether the agents_api routes are exposed over HTTP") +class BrowserControlFeature(BaseModel): + """Availability of live agentic browser control.""" + + enabled: bool = Field(..., description="Whether the live browser routes and UI are available") + + class FeaturesResponse(BaseModel): """Frontend-facing feature availability flags.""" agents_api: AgentsApiFeature + browser_control: BrowserControlFeature @router.get( @@ -35,6 +43,8 @@ class FeaturesResponse(BaseModel): ) async def list_features(config: AppConfig = Depends(get_config)) -> FeaturesResponse: """Return availability of optional, config-gated frontend features.""" + browser = browser_capability(config) return FeaturesResponse( agents_api=AgentsApiFeature(enabled=config.agents_api.enabled), + browser_control=BrowserControlFeature(enabled=browser.available), ) diff --git a/backend/app/gateway/routers/threads.py b/backend/app/gateway/routers/threads.py index 5e1f938cb..0ca523f3e 100644 --- a/backend/app/gateway/routers/threads.py +++ b/backend/app/gateway/routers/threads.py @@ -501,6 +501,18 @@ async def delete_thread_data(thread_id: str, request: Request) -> ThreadDeleteRe except Exception: logger.debug("Could not delete thread_meta for %s (not critical)", sanitize_log_param(thread_id)) + # Tear down any live browser session (best-effort). Sessions are keyed only + # by thread_id, so leaving one alive after the owner deletes the thread lets + # a later caller who guesses the id reuse the retained page/cookies. + try: + from deerflow.community.browser_automation import get_browser_session_manager + + await get_browser_session_manager().close_session(thread_id) + except ImportError: + pass # Playwright is an optional dependency. + except Exception: + logger.debug("Could not close browser session for %s (not critical)", sanitize_log_param(thread_id)) + return response diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index f1471b4b5..07706bfa9 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -248,7 +248,7 @@ Notes: - `enabled: false` keeps background polling off by default. - `max_concurrent_runs` is a global cap on active scheduled runs (queued/running run rows); each poll cycle claims only into the remaining budget, so long runs accumulating across cycles cannot exceed it. - All scheduler fields are restart-required; edits need a Gateway restart. -- Multi-worker deployments (`GATEWAY_WORKERS > 1`) must use the Postgres database backend. SQLite silently ignores row-level locks, so multiple workers can double-fire the same task. +- Multi-worker deployments (`GATEWAY_WORKERS > 1`) must use the Postgres database backend. SQLite silently ignores row-level locks, so multiple workers can double-fire the same task. The process-local agentic browser tool group is incompatible with multiple Gateway workers; keep `GATEWAY_WORKERS=1` while `browser_navigate` is enabled. Browser control also requires the backend `browser` extra (`cd backend && uv sync --extra browser && uv run playwright install chromium`); startup detects enabled browser config and fails fast when Playwright is missing, and `/api/features` reports `browser_control.enabled=false` until the runtime is available. - The MVP supports thread reuse and fresh-thread-per-run execution modes. - The MVP supports only `once` and `cron`. - Manual trigger uses the same scheduled-task resource and run lifecycle. diff --git a/backend/packages/harness/deerflow/community/browser_automation/__init__.py b/backend/packages/harness/deerflow/community/browser_automation/__init__.py new file mode 100644 index 000000000..7d87ed0ad --- /dev/null +++ b/backend/packages/harness/deerflow/community/browser_automation/__init__.py @@ -0,0 +1,49 @@ +from .session import ( + BrowserLiveViewerError, + BrowserSession, + BrowserSessionCapacityError, + BrowserSessionManager, + BrowserTab, + PageSnapshot, + SnapshotElement, + browser_multi_worker_error, + get_browser_session_manager, + redact_browser_url, + reset_browser_session_manager, +) +from .tools import ( + browser_back_tool, + browser_click_tool, + browser_close_tool, + browser_get_text_tool, + browser_navigate_tool, + browser_screenshot_tool, + browser_snapshot_tool, + browser_type_tool, + navigate_and_capture, + validate_browser_url, +) + +__all__ = [ + "BrowserSession", + "BrowserSessionCapacityError", + "BrowserSessionManager", + "BrowserTab", + "BrowserLiveViewerError", + "PageSnapshot", + "SnapshotElement", + "browser_multi_worker_error", + "browser_back_tool", + "browser_click_tool", + "browser_close_tool", + "browser_get_text_tool", + "browser_navigate_tool", + "browser_screenshot_tool", + "browser_snapshot_tool", + "browser_type_tool", + "get_browser_session_manager", + "navigate_and_capture", + "redact_browser_url", + "reset_browser_session_manager", + "validate_browser_url", +] diff --git a/backend/packages/harness/deerflow/community/browser_automation/session.py b/backend/packages/harness/deerflow/community/browser_automation/session.py new file mode 100644 index 000000000..5c1be5638 --- /dev/null +++ b/backend/packages/harness/deerflow/community/browser_automation/session.py @@ -0,0 +1,1022 @@ +"""Stateful, loop-affine browser sessions backed by Playwright. + +Playwright's async objects (``Browser``/``BrowserContext``/``Page``) are affine +to the event loop that created them. DeerFlow tools may be awaited on the +Gateway loop, the TUI loop, or a fresh test loop, and a browser session must +survive across turns of the same thread. To decouple Playwright's loop from the +caller's loop, every Playwright operation runs on one private daemon event loop +(same approach the BoxLite provider uses for its loop-affine box handles); async +tools await the result via :func:`asyncio.wrap_future`. + +Playwright itself is an optional dependency — it is imported lazily inside the +private loop so the core harness installs without it. +""" + +from __future__ import annotations + +import asyncio +import base64 +import contextlib +import logging +import os +import threading +import time +from collections.abc import Callable, Coroutine +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, TypeVar +from urllib.parse import urlparse + +if TYPE_CHECKING: + from playwright.async_api import Browser, BrowserContext, Page, Playwright + +logger = logging.getLogger(__name__) + +T = TypeVar("T") + +# Element roles/tags treated as interactive when building a page snapshot. The +# model addresses elements by the ``data-df-ref`` index this snapshot stamps, so +# it never has to guess a CSS selector or hold a stale element handle. +_SNAPSHOT_JS = r""" +() => { + // Clear ref stamps from any previous snapshot first. GitHub-style SPAs keep + // stale (now-hidden) nodes in the DOM carrying old data-df-ref values; if we + // don't strip them, a later click selector like [data-df-ref="5"] can match + // the hidden leftover ahead of the current visible element in DOM order and + // time out waiting for it to become actionable. + for (const stale of document.querySelectorAll("[data-df-ref]")) { + stale.removeAttribute("data-df-ref"); + } + const INTERACTIVE = new Set(["A", "BUTTON", "INPUT", "TEXTAREA", "SELECT"]); + const results = []; + let ref = 0; + const nodes = document.querySelectorAll( + "a, button, input, textarea, select, [role=button], [role=link], [role=tab], [role=checkbox], [onclick]" + ); + for (const el of nodes) { + const rect = el.getBoundingClientRect(); + const visible = rect.width > 0 && rect.height > 0 && + window.getComputedStyle(el).visibility !== "hidden" && + window.getComputedStyle(el).display !== "none"; + if (!visible) continue; + ref += 1; + el.setAttribute("data-df-ref", String(ref)); + const tag = el.tagName.toLowerCase(); + const role = el.getAttribute("role") || ""; + const type = el.getAttribute("type") || ""; + let name = (el.getAttribute("aria-label") || el.getAttribute("name") || + el.getAttribute("placeholder") || el.innerText || el.value || "").trim(); + if (name.length > 120) name = name.slice(0, 120) + "…"; + results.push({ ref, tag, role, type, name }); + if (results.length >= 200) break; + } + return { url: location.href, title: document.title, elements: results }; +} +""" + + +_WHEEL_SCROLL_JS = r""" +({ x, y, dx, dy }) => { + const root = document.scrollingElement || document.documentElement; + const candidates = []; + let node = document.elementFromPoint(x, y); + + while (node && node !== document.documentElement) { + if (node instanceof Element) { + candidates.push(node); + } + node = node.parentElement; + } + candidates.push(root); + + const canScroll = (el, axis, delta) => { + if (!delta || !el) { + return false; + } + const max = + axis === "y" + ? el.scrollHeight - el.clientHeight + : el.scrollWidth - el.clientWidth; + if (max <= 0) { + return false; + } + const current = axis === "y" ? el.scrollTop : el.scrollLeft; + return delta < 0 ? current > 0 : current < max; + }; + + for (const el of candidates) { + if (!canScroll(el, "y", dy) && !canScroll(el, "x", dx)) { + continue; + } + const beforeLeft = el.scrollLeft; + const beforeTop = el.scrollTop; + el.scrollBy({ left: dx, top: dy, behavior: "auto" }); + return el.scrollLeft !== beforeLeft || el.scrollTop !== beforeTop; + } + + const beforeX = window.scrollX; + const beforeY = window.scrollY; + window.scrollBy({ left: dx, top: dy, behavior: "auto" }); + return window.scrollX !== beforeX || window.scrollY !== beforeY; +} +""" + + +# Per-action timeout for clicks. Kept well under the session default (30s) so a +# stale/invalid ref fails fast and the model can re-snapshot instead of blocking +# the whole browsing loop (and tripping the agent loop-detection safety stop). +_CLICK_TIMEOUT_MS = 8000 +# Short, best-effort settle wait after a click. SPA (client-side) navigations +# never fire a fresh load event, so this must never block the action. +_POST_CLICK_LOAD_TIMEOUT_MS = 3000 +_LIVE_FRAME_JPEG_QUALITY = 85 +_MANUAL_LIVE_FRAME_MIN_INTERVAL_S = 0.75 +_LIVE_FRAME_INPUT_INTERVAL_S = 0.05 +_LIVE_FRAME_SETTLE_DELAYS_S = (0.8, 2.0) + +# Bound per-thread Chromium accumulation on a long-running multi-user gateway. +# Sessions unused past the idle timeout are lazily evicted on the next +# get_session call, and the LRU session is closed once the cap is exceeded. +_DEFAULT_MAX_SESSIONS = 32 +_DEFAULT_IDLE_TIMEOUT_S = 30 * 60.0 + + +def browser_multi_worker_error(workers: int | None = None) -> str | None: + """Return the fail-closed reason for process-local browser sessions.""" + if workers is None: + try: + workers = int(os.environ.get("GATEWAY_WORKERS", "1")) + except (TypeError, ValueError): + workers = 1 + if workers <= 1: + return None + return f"GATEWAY_WORKERS={workers} cannot enable agentic browser tools: browser sessions are process-local and uvicorn does not provide thread affinity. Set GATEWAY_WORKERS=1 or disable the browser_navigate tool." + + +def ensure_browser_worker_compatibility() -> None: + """Reject runtime browser use when requests can land in another worker.""" + error = browser_multi_worker_error() + if error is not None: + raise RuntimeError(error) + + +class BrowserSessionCapacityError(RuntimeError): + """Raised when the browser session cap has no evictable slot.""" + + +class BrowserLiveViewerError(RuntimeError): + """Raised when a second Live viewer tries to attach to a session.""" + + +def _is_playwright_timeout_error(exc: Exception) -> bool: + """Recognize Playwright timeouts without requiring Playwright at import time.""" + return exc.__class__.__name__ == "TimeoutError" and exc.__class__.__module__.startswith("playwright.") + + +def redact_browser_url(url: str) -> str: + """Drop query/fragment so a blocked-URL log line can't leak tokens/PII.""" + try: + parsed = urlparse(url) + return f"{parsed.scheme}://{parsed.netloc}{parsed.path}" + except Exception: + return "" + + +class _PlaywrightLoopThread: + """A private asyncio event loop running on a dedicated daemon thread.""" + + def __init__(self) -> None: + self._loop = asyncio.new_event_loop() + self._thread = threading.Thread(target=self._run, name="deerflow-browser-loop", daemon=True) + self._thread.start() + + def _run(self) -> None: + asyncio.set_event_loop(self._loop) + self._loop.run_forever() + + async def run(self, coro: Coroutine[Any, Any, T]) -> T: + """Schedule *coro* on the private loop and await it from any loop.""" + future = asyncio.run_coroutine_threadsafe(coro, self._loop) + return await asyncio.wrap_future(future) + + def submit(self, coro: Coroutine[Any, Any, Any]) -> None: + """Schedule *coro* on the private loop without blocking the caller.""" + future = asyncio.run_coroutine_threadsafe(coro, self._loop) + + def _log_failure(done: Any) -> None: + try: + done.result() + except Exception as exc: + logger.debug("browser background task failed: %s", exc) + + future.add_done_callback(_log_failure) + + def run_sync(self, coro: Coroutine[Any, Any, T], timeout: float | None = None) -> T: + future = asyncio.run_coroutine_threadsafe(coro, self._loop) + return future.result(timeout=timeout) + + +@dataclass +class SnapshotElement: + ref: int + tag: str + role: str + type: str + name: str + + def render(self) -> str: + label = self.role or self.tag + detail = f" type={self.type}" if self.type else "" + name = self.name or "(no text)" + return f"[{self.ref}] {label}{detail}: {name}" + + +@dataclass +class PageSnapshot: + url: str + title: str + elements: list[SnapshotElement] = field(default_factory=list) + + def render(self) -> str: + lines = [f"URL: {self.url}", f"Title: {self.title}", ""] + if not self.elements: + lines.append("No interactive elements detected.") + else: + lines.append("Interactive elements (address them by [ref] number):") + lines.extend(el.render() for el in self.elements) + return "\n".join(lines) + + +@dataclass +class BrowserTab: + index: int + url: str + title: str + active: bool + + +class BrowserSession: + """A single Playwright browser+page bound to the private loop.""" + + def __init__( + self, + loop: _PlaywrightLoopThread, + *, + headless: bool, + timeout_ms: int, + viewport: dict[str, int], + cdp_url: str | None = None, + url_guard: Callable[[str], str | None] | None = None, + on_activity: Callable[[], None] | None = None, + ) -> None: + self._loop = loop + self._headless = headless + self._timeout_ms = timeout_ms + self._viewport = viewport + # Optional SSRF guard applied at the browser request boundary. It returns + # an error string to block a URL (redirect/popup/subresource) or None to + # allow it. The explicit navigate URL is screened by the caller, but + # Playwright follows redirects and issues subresource/popup requests that + # bypass that single check — so we also validate every request the page + # makes here, catching a public URL that 30x-redirects to a private or + # cloud-metadata host. + self._url_guard = url_guard + self._request_guard_bound = False + # When set, attach to an already-running Chrome via the DevTools + # Protocol (like Codex's "connect to your real browser") instead of + # launching a private headless instance. The user watches the agent + # drive their own visible browser, with their real login sessions. + self._cdp_url = cdp_url + self._on_activity = on_activity + self._activity_lock = threading.Lock() + self._active_refs = 0 + self._connected_over_cdp = False + self._playwright: Playwright | None = None + self._browser: Browser | None = None + self._context: BrowserContext | None = None + self._page: Page | None = None + # Tool calls and the Live WebSocket share this session and can observe a + # closed page concurrently. Only one caller may rebuild the browser + # hierarchy; the second check inside the lock reuses its result. + self._ensure_page_lock = asyncio.Lock() + # Live screencast state. When streaming, ``_on_frame`` is retained so the + # screencast can be re-bound to a new page — login/OAuth flows commonly + # open a popup or a fresh tab, and the user must see (and drive) it. + self._on_frame: Callable[[str], None] | None = None + # The page the live screencast's CDP session is currently bound to. Frames + # are captured from ``self._page`` (the live active page), but the CDP + # repaint signal is tied to a specific page; when the active page diverges + # from this one we must rebind so new-page repaints keep driving frames. + self._screencast_page: Page | None = None + # Guards against re-entrant rebinds: while (re)binding the screencast we + # may call _ensure_page (which routes through _set_active_page); without + # this flag that would schedule another rebind and recurse. + self._screencast_binding = False + self._last_manual_live_frame_at = 0.0 + self._settle_live_frames_pending = False + self._input_live_frame_generation = 0 + self._input_live_frame_pending = False + self._page_listener_bound = False + + @property + def active_refs(self) -> int: + with self._activity_lock: + return self._active_refs + + def _pin(self) -> None: + """Keep this session in the manager while a caller owns it.""" + with self._activity_lock: + self._active_refs += 1 + + def _unpin(self) -> None: + with self._activity_lock: + self._active_refs = max(0, self._active_refs - 1) + + @contextlib.contextmanager + def _activity(self): + """Reference a real browser operation and refresh its recency.""" + self._pin() + if self._on_activity is not None: + self._on_activity() + try: + yield + finally: + self._unpin() + + async def _ensure_page(self) -> Page: + if self._page is not None and not self._page.is_closed(): + return self._page + async with self._ensure_page_lock: + if self._page is not None and not self._page.is_closed(): + return self._page + from playwright.async_api import async_playwright + + if self._playwright is None: + self._playwright = await async_playwright().start() + + if self._cdp_url: + # Attach to the user's running Chrome (started with + # --remote-debugging-port). Reuse its default context + an existing + # tab: calling new_context()/new_page() on a CDP-attached real + # Chrome trips "Browser context management is not supported", so we + # adopt the tab Chrome already opened instead. + if self._browser is None or not self._browser.is_connected(): + self._browser = await self._playwright.chromium.connect_over_cdp(self._cdp_url) + self._connected_over_cdp = True + # CDP-attached real Chrome owns its own browsing context, so the + # SSRF request guard is intentionally NOT installed for it (see + # _install_request_guard). Surface that to operators — for this + # session redirects/subresources to private/metadata hosts are + # not aborted; cdp_url is documented local/trusted-only. + logger.warning( + "browser SSRF request guard is disabled for CDP-attached session (cdp_url=%s)", + redact_browser_url(self._cdp_url), + ) + self._context = self._browser.contexts[0] if self._browser.contexts else await self._browser.new_context() + self._context.set_default_timeout(self._timeout_ms) + existing = self._context.pages + self._set_active_page(existing[-1] if existing else await self._context.new_page()) + self._bind_new_page_listener() + return self._page + + if self._browser is None or not self._browser.is_connected(): + self._browser = await self._playwright.chromium.launch(headless=self._headless) + # device_scale_factor=2 renders screenshots at retina density so the + # panel stays crisp when the image is scaled up to fill the view. + self._context = await self._browser.new_context(viewport=self._viewport, device_scale_factor=2) + self._context.set_default_timeout(self._timeout_ms) + await self._install_request_guard() + self._set_active_page(await self._context.new_page()) + self._bind_new_page_listener() + return self._page + + def _set_active_page(self, page: Page) -> None: + """Adopt *page* as the active page and keep the live screencast on it. + + Every path that changes the active page (initial/rebuilt page, popups and + new tabs, explicit tab switches) routes through here so the live stream can + never drift onto a stale page. The CDP screencast's repaint signal is bound + to one page; when the active page diverges from the one the screencast is + bound to, rebind it. Rebinding is scheduled with error handling so a + transient failure cannot leave the callback un-awaited and silently + swallowed (the previous fire-and-forget rebind could strand Live on the + old page — address bar / snapshot moved on, frames did not). + """ + self._page = page + if self._on_frame is not None and not self._screencast_binding and page is not self._screencast_page: + asyncio.ensure_future(self._rebind_screencast_safe()) + + def _bind_new_page_listener(self) -> None: + """Follow popups/new tabs so auth flows stay visible and controllable. + + Login and OAuth consent screens routinely open a popup or a fresh tab. + Without following it the user sees a frozen frame and cannot authorize. + On a new page we adopt it as the active page and, if a screencast is + running, re-bind it so the stream tracks the tab the user must act on. + """ + if self._context is None or self._page_listener_bound: + return + + def _on_new_page(page: Page) -> None: + self._set_active_page(page) + + self._context.on("page", _on_new_page) + self._page_listener_bound = True + + async def _install_request_guard(self) -> None: + """Abort any request whose URL fails the SSRF guard. + + Runs at the context level so it covers the top navigation, every + redirect hop, popups/new tabs, iframes, and subresource fetches — the + paths a one-time initial-URL check cannot see. A public URL that + redirects to ``http://169.254.169.254/...`` is aborted before the + response is exposed through snapshots/text. Skipped for CDP-attached + real Chrome, which owns its own browsing context. + """ + if self._url_guard is None or self._context is None or self._request_guard_bound or self._cdp_url: + return + + guard = self._url_guard + + async def _route(route: Any) -> None: + url = "" + with contextlib.suppress(Exception): + url = route.request.url + if url.startswith(("http://", "https://")) and guard(url) is not None: + logger.warning("browser request blocked by SSRF guard: %s", redact_browser_url(url)) + with contextlib.suppress(Exception): + await route.abort("blockedbyclient") + return + with contextlib.suppress(Exception): + await route.continue_() + + with contextlib.suppress(Exception): + await self._context.route("**/*", _route) + self._request_guard_bound = True + + async def _rebind_screencast(self) -> None: + if self._on_frame is not None: + await self._start_screencast(self._on_frame) + + async def _rebind_screencast_safe(self) -> None: + try: + await self._rebind_screencast() + except Exception as exc: + logger.debug("browser live screencast rebind failed: %s", exc) + + async def _navigate(self, url: str) -> PageSnapshot: + page = await self._ensure_page() + await page.goto(url, wait_until="domcontentloaded") + return await self._snapshot_impl(page) + + async def _snapshot_impl(self, page: Page) -> PageSnapshot: + data = await page.evaluate(_SNAPSHOT_JS) + elements = [SnapshotElement(ref=int(e["ref"]), tag=e["tag"], role=e["role"], type=e["type"], name=e["name"]) for e in data["elements"]] + return PageSnapshot(url=data["url"], title=data["title"], elements=elements) + + async def _snapshot(self) -> PageSnapshot: + page = await self._ensure_page() + return await self._snapshot_impl(page) + + async def _click(self, ref: int) -> PageSnapshot: + page = await self._ensure_page() + selector = f'[data-df-ref="{ref}"]' + base = page.locator(selector) + + # Fast-fail on a stale ref instead of blocking on the 30s session + # default: after a SPA re-render the ref may no longer exist, and the + # model should re-snapshot and retry rather than the browsing loop + # stalling until it trips the agent loop-detection safety stop. + if await base.count() == 0: + raise RuntimeError(f"element [{ref}] is no longer on the page; call browser_snapshot to get fresh refs") + + locator = base.first + # Bring the target into view; harmless if it is already on-screen. + try: + await locator.scroll_into_view_if_needed(timeout=_CLICK_TIMEOUT_MS) + except Exception as exc: + if not _is_playwright_timeout_error(exc): + raise + + try: + await locator.click(timeout=_CLICK_TIMEOUT_MS) + except Exception as exc: + if not _is_playwright_timeout_error(exc): + raise + raise RuntimeError(f"element [{ref}] was not clickable within {_CLICK_TIMEOUT_MS // 1000}s; the page may have changed — call browser_snapshot and retry") from exc + + # SPA (client-side) navigations never fire a fresh load event, so this + # settle wait is best-effort and must never block the snapshot. + try: + await page.wait_for_load_state("domcontentloaded", timeout=_POST_CLICK_LOAD_TIMEOUT_MS) + except Exception as exc: + if not _is_playwright_timeout_error(exc): + raise + + return await self._snapshot_impl(page) + + async def _type(self, ref: int, text: str, submit: bool) -> PageSnapshot: + page = await self._ensure_page() + selector = f'[data-df-ref="{ref}"]' + await page.fill(selector, text) + if submit: + await page.press(selector, "Enter") + # Best-effort settle; a client-side search/submit may never fire a + # fresh load event, so this must not block the snapshot for 30s. + try: + await page.wait_for_load_state("domcontentloaded", timeout=_POST_CLICK_LOAD_TIMEOUT_MS) + except Exception as exc: + if not _is_playwright_timeout_error(exc): + raise + return await self._snapshot_impl(page) + + async def _get_text(self, max_chars: int) -> str: + page = await self._ensure_page() + text = await page.inner_text("body") + return text[:max_chars] + + async def _screenshot_bytes(self, full_page: bool) -> bytes: + page = await self._ensure_page() + return await page.screenshot(full_page=full_page, type="png") + + async def _live_frame(self) -> str: + page = await self._ensure_page() + shot = await page.screenshot(type="jpeg", quality=_LIVE_FRAME_JPEG_QUALITY) + return base64.b64encode(shot).decode("ascii") + + async def _emit_live_frame(self) -> None: + if self._on_frame is None: + return + with self._activity(): + self._on_frame(await self._live_frame()) + self._last_manual_live_frame_at = time.monotonic() + + async def _settle_live_frames(self) -> None: + previous_delay = 0.0 + try: + for delay in _LIVE_FRAME_SETTLE_DELAYS_S: + await asyncio.sleep(max(0.0, delay - previous_delay)) + previous_delay = delay + await self._emit_live_frame() + finally: + self._settle_live_frames_pending = False + + def _schedule_settle_live_frames(self) -> None: + if self._settle_live_frames_pending: + return + self._settle_live_frames_pending = True + asyncio.ensure_future(self._settle_live_frames()) + + async def _push_live_frame(self) -> None: + if self._on_frame is None: + return + elapsed = time.monotonic() - self._last_manual_live_frame_at + if elapsed >= _MANUAL_LIVE_FRAME_MIN_INTERVAL_S: + await self._emit_live_frame() + # One frame is often too early for SPAs: the URL may have changed while + # the page body is still rendering. Add a small, bounded settle burst + # instead of returning to continuous screencast. + self._schedule_settle_live_frames() + + async def _flush_input_live_frames(self) -> None: + try: + # Coalesce the first burst, then keep refreshing at a bounded cadence + # while input continues. A trailing debounce would freeze the visible + # page until a wheel/keyboard gesture stopped. + await asyncio.sleep(_LIVE_FRAME_INPUT_INTERVAL_S) + while self._on_frame is not None: + generation = self._input_live_frame_generation + await self._emit_live_frame() + if generation == self._input_live_frame_generation: + self._schedule_settle_live_frames() + return + await asyncio.sleep(_LIVE_FRAME_INPUT_INTERVAL_S) + finally: + self._input_live_frame_pending = False + + def _schedule_input_live_frame(self) -> None: + self._input_live_frame_generation += 1 + if self._input_live_frame_pending: + return + self._input_live_frame_pending = True + asyncio.ensure_future(self._flush_input_live_frames()) + + async def _back(self) -> PageSnapshot: + page = await self._ensure_page() + await page.go_back(wait_until="domcontentloaded") + return await self._snapshot_impl(page) + + async def _current_url(self) -> str | None: + page = await self._ensure_page() + try: + return page.url + except Exception: + return None + + async def _tabs(self) -> list[BrowserTab]: + await self._ensure_page() + if self._context is None: + return [] + pages = [page for page in self._context.pages if not page.is_closed()] + tabs: list[BrowserTab] = [] + for index, page in enumerate(pages): + try: + title = await page.title() + except Exception: + title = "" + try: + url = page.url + except Exception: + url = "" + tabs.append(BrowserTab(index=index, url=url, title=title, active=page == self._page)) + return tabs + + async def _activate_tab(self, index: int) -> None: + await self._ensure_page() + if self._context is None: + return + pages = [page for page in self._context.pages if not page.is_closed()] + if index < 0 or index >= len(pages): + return + target = pages[index] + await target.bring_to_front() + # This path is async, so rebind the screencast inline (awaited) rather + # than going through _set_active_page's scheduled rebind — that keeps the + # frame source aligned with the switched-to tab before this call returns. + self._page = target + if self._on_frame is not None and target is not self._screencast_page: + await self._rebind_screencast() + + async def _close(self) -> None: + try: + with contextlib.suppress(Exception): + await self._stop_screencast() + if self._connected_over_cdp: + # Attached to the user's real Chrome — never close their browser, + # context, or the tab we adopted. Just disconnect Playwright. + if self._browser is not None: + with contextlib.suppress(Exception): + await self._browser.close() + else: + if self._context is not None: + with contextlib.suppress(Exception): + await self._context.close() + if self._browser is not None: + with contextlib.suppress(Exception): + await self._browser.close() + if self._playwright is not None: + with contextlib.suppress(Exception): + await self._playwright.stop() + finally: + self._playwright = None + self._browser = None + self._context = None + self._page = None + self._screencast_page = None + self._connected_over_cdp = False + self._on_frame = None + self._settle_live_frames_pending = False + self._input_live_frame_generation += 1 + self._page_listener_bound = False + self._request_guard_bound = False + + async def _start_screencast(self, on_frame: Callable[[str], None]) -> None: + """Start Live mode and send an initial JPEG frame. + + This used to attach Chrome's CDP screencast and then turn every repaint + into a high-quality Playwright screenshot. That made GitHub-like pages + expensive to open because a mostly static panel still kept the headless + renderer/GPU busy. Live control now uses on-demand frames instead: + initial connect, browser tool completion, and user input all push a + throttled screenshot. + """ + page = await self._ensure_page() + if self._on_frame is not None and self._on_frame is not on_frame: + raise BrowserLiveViewerError("Browser live stream already has an active viewer") + await self._stop_screencast() + self._on_frame = on_frame + self._screencast_page = page + await self._emit_live_frame() + self._schedule_settle_live_frames() + + async def _stop_screencast(self, on_frame: Callable[[str], None] | None = None) -> None: + if on_frame is not None and self._on_frame is not on_frame: + return + self._on_frame = None + self._settle_live_frames_pending = False + self._input_live_frame_generation += 1 + self._screencast_page = None + + async def _dispatch_input(self, event: dict) -> None: + page = await self._ensure_page() + vw = self._viewport.get("width", 1280) + vh = self._viewport.get("height", 720) + etype = event.get("type") + if etype in {"click", "move", "down", "up"}: + x = float(event.get("nx", 0)) * vw + y = float(event.get("ny", 0)) * vh + if etype == "click": + await page.mouse.click(x, y) + elif etype == "move": + await page.mouse.move(x, y) + elif etype == "down": + await page.mouse.move(x, y) + await page.mouse.down() + elif etype == "up": + await page.mouse.up() + elif etype == "wheel": + dx = float(event.get("dx", 0)) + dy = float(event.get("dy", 0)) + nx = event.get("nx") + ny = event.get("ny") + x = vw / 2 + y = vh / 2 + if nx is not None and ny is not None: + try: + x = max(0.0, min(1.0, float(nx))) * vw + y = max(0.0, min(1.0, float(ny))) * vh + await page.mouse.move(x, y) + except (TypeError, ValueError) as e: + logger.debug("invalid browser wheel coordinates: %s", e) + try: + handled = bool(await page.evaluate(_WHEEL_SCROLL_JS, {"x": x, "y": y, "dx": dx, "dy": dy})) + except Exception as e: + logger.debug("browser js scroll failed: %s", e) + handled = False + if not handled: + await page.mouse.wheel(dx, dy) + elif etype == "key": + key = event.get("key") + if key: + await page.keyboard.press(key) + elif etype == "text": + text = event.get("text") + if text: + await page.keyboard.type(text) + elif etype == "navigate": + url = event.get("url") + if url: + await page.goto(url, wait_until="domcontentloaded") + elif etype == "back": + await page.go_back(wait_until="domcontentloaded") + elif etype == "forward": + await page.go_forward(wait_until="domcontentloaded") + elif etype == "activate_tab": + index = event.get("index") + if isinstance(index, int) and not isinstance(index, bool): + await self._activate_tab(index) + if etype != "move": + self._schedule_input_live_frame() + + # Public API — each marshals onto the private loop. + async def navigate(self, url: str) -> PageSnapshot: + with self._activity(): + return await self._loop.run(self._navigate(url)) + + async def snapshot(self) -> PageSnapshot: + with self._activity(): + return await self._loop.run(self._snapshot()) + + async def click(self, ref: int) -> PageSnapshot: + with self._activity(): + return await self._loop.run(self._click(ref)) + + async def type_text(self, ref: int, text: str, submit: bool = False) -> PageSnapshot: + with self._activity(): + return await self._loop.run(self._type(ref, text, submit)) + + async def get_text(self, max_chars: int = 8000) -> str: + with self._activity(): + return await self._loop.run(self._get_text(max_chars)) + + async def screenshot_bytes(self, full_page: bool = False) -> bytes: + with self._activity(): + return await self._loop.run(self._screenshot_bytes(full_page)) + + async def live_frame(self) -> str: + with self._activity(): + return await self._loop.run(self._live_frame()) + + async def push_live_frame(self) -> None: + with self._activity(): + await self._loop.run(self._push_live_frame()) + + def schedule_live_frames(self) -> None: + self._loop.submit(self._push_live_frame()) + + async def back(self) -> PageSnapshot: + with self._activity(): + return await self._loop.run(self._back()) + + async def current_url(self) -> str | None: + with self._activity(): + return await self._loop.run(self._current_url()) + + async def tabs(self) -> list[BrowserTab]: + with self._activity(): + return await self._loop.run(self._tabs()) + + async def start_screencast(self, on_frame: Callable[[str], None]) -> None: + with self._activity(): + await self._loop.run(self._start_screencast(on_frame)) + + async def stop_screencast(self, on_frame: Callable[[str], None] | None = None) -> None: + with self._activity(): + await self._loop.run(self._stop_screencast(on_frame)) + + async def dispatch_input(self, event: dict) -> None: + with self._activity(): + await self._loop.run(self._dispatch_input(event)) + + async def close(self) -> None: + await self._loop.run(self._close()) + + +class BrowserSessionManager: + """Process-local registry of per-thread browser sessions. + + Sessions are keyed by ``thread_id`` and each owns a headless Chromium + process, so a long-running multi-user gateway would otherwise accumulate one + browser per thread that ever used the tools (a real memory/FD leak). To bound + that, ``get_session`` lazily evicts sessions that have been idle past + ``idle_timeout_s`` and enforces a ``max_sessions`` cap by closing the + least-recently-used unpinned session. Active browser operations and Live + WebSocket leases are reference-counted, so eviction never closes a session + while it is in use. Admission is a hard bound: when every existing session + is pinned, a new thread is rejected instead of exceeding ``max_sessions``. + Eviction is fire-and-forget on the private Playwright loop so it never + blocks the caller; the just-requested thread is always kept. + """ + + def __init__(self, *, max_sessions: int = _DEFAULT_MAX_SESSIONS, idle_timeout_s: float = _DEFAULT_IDLE_TIMEOUT_S) -> None: + self._loop: _PlaywrightLoopThread | None = None + self._sessions: dict[str, BrowserSession] = {} + self._last_used: dict[str, float] = {} + self._max_sessions = max_sessions + self._idle_timeout_s = idle_timeout_s + self._lock = threading.Lock() + + def _touch_session(self, key: str) -> None: + with self._lock: + if key in self._sessions: + self._last_used[key] = time.monotonic() + + def _ensure_loop(self) -> _PlaywrightLoopThread: + if self._loop is None: + self._loop = _PlaywrightLoopThread() + return self._loop + + def get_session( + self, + thread_id: str | None, + *, + headless: bool = True, + timeout_ms: int = 30000, + viewport: dict[str, int] | None = None, + cdp_url: str | None = None, + allow_unguarded_cdp: bool = False, + url_guard: Callable[[str], str | None] | None = None, + pin: bool = False, + ) -> BrowserSession: + ensure_browser_worker_compatibility() + if cdp_url and not allow_unguarded_cdp: + raise RuntimeError("cdp_url uses a browser context where DeerFlow cannot enforce its SSRF request guard; set allow_unguarded_cdp: true only for an explicitly trusted local Chrome session") + key = thread_id or "default" + now = time.monotonic() + evicted: list[BrowserSession] = [] + with self._lock: + session = self._sessions.get(key) + if session is None: + evicted.extend(self._collect_idle_locked(keep_key=key, now=now)) + if self._max_sessions > 0 and len(self._sessions) >= self._max_sessions: + lru = self._pop_lru_unpinned_locked(excluded_keys={key}) + if lru is None: + raise BrowserSessionCapacityError(f"Browser session capacity is full ({self._max_sessions}); close an active Live browser before opening another") + evicted.append(lru) + session = BrowserSession( + self._ensure_loop(), + headless=headless, + timeout_ms=timeout_ms, + viewport=viewport or {"width": 1280, "height": 720}, + cdp_url=cdp_url, + url_guard=url_guard, + on_activity=lambda: self._touch_session(key), + ) + self._sessions[key] = session + if pin: + session._pin() + self._last_used[key] = now + evicted.extend(self._collect_evictable_locked(keep_key=key, now=now)) + for evicted_session in evicted: + self._schedule_close(evicted_session) + return session + + @contextlib.contextmanager + def acquire_session(self, thread_id: str | None, **kwargs: Any): + """Acquire an atomically pinned session for one browser operation.""" + key = thread_id or "default" + session = self.get_session(key, pin=True, **kwargs) + try: + yield session + finally: + self.release_session(key, session) + + def release_session(self, thread_id: str | None, session: BrowserSession) -> None: + """Release a lease and restore idle/LRU bounds when it becomes evictable.""" + key = thread_id or "default" + session._unpin() + now = time.monotonic() + with self._lock: + evicted = self._collect_evictable_locked(keep_key=None, now=now) if self._sessions.get(key) is session else [] + for evicted_session in evicted: + self._schedule_close(evicted_session) + + def _collect_evictable_locked(self, *, keep_key: str | None, now: float) -> list[BrowserSession]: + """Drop idle/over-cap sessions from the registry; return them for close. + + Must be called under ``self._lock``. When set, ``keep_key`` (the + just-touched thread) is never evicted so a new request cannot lose its + session. Lease release passes ``None`` so newly unpinned sessions can + restore the configured bounds without waiting for another request. + """ + to_close: list[BrowserSession] = [] + to_close.extend(self._collect_idle_locked(keep_key=keep_key, now=now)) + if self._max_sessions > 0: + excluded = {keep_key} if keep_key is not None else set() + while len(self._sessions) > self._max_sessions: + session = self._pop_lru_unpinned_locked(excluded_keys=excluded) + if session is None: + break + to_close.append(session) + return to_close + + def _collect_idle_locked(self, *, keep_key: str | None, now: float) -> list[BrowserSession]: + """Drop idle, unpinned sessions from the registry.""" + to_close: list[BrowserSession] = [] + if self._idle_timeout_s > 0: + for other_key, last_used in list(self._last_used.items()): + if other_key == keep_key: + continue + session = self._sessions.get(other_key) + if session is not None and session.active_refs: + continue + if now - last_used >= self._idle_timeout_s: + session = self._sessions.pop(other_key, None) + self._last_used.pop(other_key, None) + if session is not None: + to_close.append(session) + return to_close + + def _pop_lru_unpinned_locked(self, *, excluded_keys: set[str]) -> BrowserSession | None: + candidates = [(last_used, other_key) for other_key, last_used in self._last_used.items() if other_key not in excluded_keys and (session := self._sessions.get(other_key)) is not None and not session.active_refs] + if not candidates: + return None + _, lru_key = min(candidates) + session = self._sessions.pop(lru_key, None) + self._last_used.pop(lru_key, None) + return session + + def _schedule_close(self, session: BrowserSession) -> None: + loop = self._loop + if loop is None: + return + with contextlib.suppress(Exception): + loop.submit(session._close()) + + async def close_session(self, thread_id: str | None) -> bool: + key = thread_id or "default" + with self._lock: + session = self._sessions.pop(key, None) + self._last_used.pop(key, None) + if session is None: + return False + await session.close() + return True + + async def close_all_sessions(self) -> int: + with self._lock: + sessions = list(self._sessions.values()) + self._sessions.clear() + self._last_used.clear() + for session in sessions: + await session.close() + return len(sessions) + + +_manager: BrowserSessionManager | None = None +_manager_lock = threading.Lock() + + +def get_browser_session_manager() -> BrowserSessionManager: + global _manager + if _manager is None: + with _manager_lock: + if _manager is None: + _manager = BrowserSessionManager() + return _manager + + +def reset_browser_session_manager() -> None: + """Test hook: drop the process-local manager without closing sessions.""" + global _manager + with _manager_lock: + _manager = None diff --git a/backend/packages/harness/deerflow/community/browser_automation/tools.py b/backend/packages/harness/deerflow/community/browser_automation/tools.py new file mode 100644 index 000000000..5427486a1 --- /dev/null +++ b/backend/packages/harness/deerflow/community/browser_automation/tools.py @@ -0,0 +1,448 @@ +"""Agentic browser tools — a stateful navigate → observe → act loop. + +Unlike the read-only ``web_fetch`` / ``web_capture`` tools, these keep a live +per-thread browser session (see :mod:`.session`) so the agent can click, type, +submit forms, and follow multi-step flows on JavaScript-heavy or authenticated +pages. Every action returns a fresh page snapshot whose interactive elements are +addressed by a stable ``[ref]`` index, so the model acts on what it just +observed instead of guessing selectors. + +All URLs are SSRF-screened with the shared :func:`validate_public_http_url` +helper (opt-out only for intentional internal targets). +""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import re +from datetime import UTC, datetime +from pathlib import Path +from typing import Annotated + +from langchain.tools import InjectedToolCallId, tool +from langchain_core.messages import ToolMessage +from langgraph.types import Command + +from deerflow.community.url_safety import resolve_host_addresses as _resolve_host_addresses +from deerflow.community.url_safety import validate_public_http_url +from deerflow.config import get_app_config +from deerflow.config.paths import VIRTUAL_PATH_PREFIX +from deerflow.constants import BROWSER_FRAMES_DIRNAME +from deerflow.tools.types import Runtime + +from .session import BrowserSession, BrowserSessionManager, PageSnapshot, get_browser_session_manager + +logger = logging.getLogger(__name__) + +_OUTPUTS_VIRTUAL_PREFIX = f"{VIRTUAL_PATH_PREFIX}/outputs" +# Auto-captured per-step screenshots are live progress feedback (shown in the +# browser panel + inline thumbnails), not deliverables. Keep them in a hidden +# subdir so the workspace-changes review does not list them as file changes. +# The dir name is a shared constant so the scanner's ignore list cannot drift. +_BROWSER_FRAMES_DIRNAME = BROWSER_FRAMES_DIRNAME +_FRAMES_VIRTUAL_PREFIX = f"{_OUTPUTS_VIRTUAL_PREFIX}/{_BROWSER_FRAMES_DIRNAME}" +_SAFE_FILENAME_RE = re.compile(r"[^A-Za-z0-9._-]+") + + +def _get_tool_config(tool_name: str) -> dict: + config = get_app_config().get_tool_config(tool_name) + if config is None: + return {} + return config.model_extra or {} + + +def _as_bool(value: object, default: bool) -> bool: + if isinstance(value, bool): + return value + if isinstance(value, str): + lowered = value.strip().lower() + if lowered in {"1", "true", "yes", "on"}: + return True + if lowered in {"0", "false", "no", "off"}: + return False + return default + + +def _as_int(value: object, default: int) -> int: + if isinstance(value, int) and not isinstance(value, bool): + return value + if isinstance(value, str): + try: + return int(value.strip()) + except ValueError: + return default + return default + + +def _thread_id(runtime: Runtime) -> str | None: + return runtime.context.get("thread_id") if runtime.context else None + + +def _as_str(value: object) -> str | None: + if isinstance(value, str): + trimmed = value.strip() + return trimmed or None + return None + + +class _SessionLease: + """Context manager that keeps a process-local browser session pinned.""" + + def __init__(self, manager: BrowserSessionManager, thread_id: str | None, session: BrowserSession) -> None: + self._manager = manager + self._thread_id = thread_id + self.session = session + + def __enter__(self) -> BrowserSession: + return self.session + + def __exit__(self, exc_type, exc_value, traceback) -> None: + self._manager.release_session(self._thread_id, self.session) + + +def _resolve_session(runtime: Runtime, tool_name: str) -> _SessionLease: + # Launch config (headless/viewport/timeout/cdp_url) is read from a single + # canonical source — always ``browser_navigate`` — regardless of which tool + # first creates the session. ``get_session`` caches per thread and ignores + # these params for later callers, so keying launch config off the calling + # tool made it "first tool to run wins": a ``headless: false`` set only on + # ``browser_navigate`` was silently dropped if another tool (or the live WS) + # initialized the session first. ``tool_name`` is retained for callers that + # read their own non-launch config (e.g. ``browser_get_text``'s max_chars). + del tool_name + cfg = _get_tool_config("browser_navigate") + headless = _as_bool(cfg.get("headless"), True) + timeout_ms = _as_int(cfg.get("timeout_ms"), 30000) + width = _as_int(cfg.get("viewport_width"), 1280) + height = _as_int(cfg.get("viewport_height"), 720) + cdp_url = _as_str(cfg.get("cdp_url")) + manager = get_browser_session_manager() + thread_id = _thread_id(runtime) + session = manager.get_session( + thread_id, + headless=headless, + timeout_ms=timeout_ms, + viewport={"width": width, "height": height}, + cdp_url=cdp_url, + allow_unguarded_cdp=_as_bool(cfg.get("allow_unguarded_cdp"), False), + url_guard=validate_browser_url, + pin=True, + ) + return _SessionLease(manager, thread_id, session) + + +def validate_browser_url(url: str, *, tool_name: str = "browser_navigate") -> str | None: + """SSRF-screen a browser navigation URL using the tool's config policy. + + Returns an ``"Error: ..."`` string when the URL must be rejected, or ``None`` + when navigation may proceed. Shared by the agent tools and the Gateway live + stream so every path that can steer the browser enforces the same allow/deny + policy (``allow_private_addresses`` from the ``browser_navigate`` tool config). + """ + cfg = _get_tool_config(tool_name) + allow_private = _as_bool(cfg.get("allow_private_addresses"), False) + return validate_public_http_url( + url, + allow_private_addresses=allow_private, + action="browse", + resolver=_resolve_host_addresses, + ) + + +def _validate_url(tool_name: str, url: str) -> str | None: + return validate_browser_url(url, tool_name=tool_name) + + +def _snapshot_message(snapshot: PageSnapshot, prefix: str = "") -> str: + body = snapshot.render() + return f"{prefix}\n\n{body}" if prefix else body + + +def _tool_message(content: str, tool_call_id: str) -> Command: + return Command(update={"messages": [ToolMessage(content, tool_call_id=tool_call_id)]}) + + +def _step_screenshot_name(action: str) -> str: + stamp = datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f") + safe = _SAFE_FILENAME_RE.sub("_", action).strip("._-") or "step" + return f"browser-{safe}-{stamp}.png" + + +async def _capture_step_screenshot(runtime: Runtime, session: BrowserSession, action: str) -> str | None: + """Best-effort per-action screenshot saved as hidden live-progress feedback. + + Returns the artifact virtual path (under the hidden ``.browser-frames`` dir so + it stays out of the workspace-changes review), or ``None`` when outputs are + unavailable or capture fails — a failed capture must never break the action. + """ + outputs_path = _thread_outputs_path(runtime) + if isinstance(outputs_path, str): + return None + try: + content = await session.screenshot_bytes(full_page=False) + name = _step_screenshot_name(action) + frames_dir = outputs_path / _BROWSER_FRAMES_DIRNAME + final_name = await asyncio.to_thread(_write_screenshot, frames_dir, name, content) + with contextlib.suppress(Exception): + session.schedule_live_frames() + return f"{_FRAMES_VIRTUAL_PREFIX}/{final_name}" + except Exception as e: + logger.warning(f"browser step screenshot failed: {e}") + return None + + +def _snapshot_command( + runtime: Runtime, + session: BrowserSession, + snapshot: PageSnapshot, + tool_call_id: str, + prefix: str, + screenshot_path: str | None, +) -> Command: + """Build a ToolMessage carrying the text snapshot plus an inline screenshot. + + The screenshot rides both as a thread ``artifacts`` entry (so it opens in the + artifacts side panel) and on ``ToolMessage.additional_kwargs.browser_view`` + (so the chat can render an inline thumbnail per browser step). + """ + text = _snapshot_message(snapshot, prefix) + additional_kwargs: dict = {} + update: dict = {} + if screenshot_path: + additional_kwargs["browser_view"] = {"screenshot": screenshot_path, "url": snapshot.url, "title": snapshot.title} + update["artifacts"] = [screenshot_path] + update["messages"] = [ToolMessage(text, tool_call_id=tool_call_id, additional_kwargs=additional_kwargs)] + return Command(update=update) + + +async def navigate_and_capture(*, thread_id: str | None, url: str, outputs_path: Path) -> dict: + """Drive the per-thread browser session to *url* and capture a screenshot. + + Used by the Gateway browser router so a user can steer the live session from + the UI URL bar. Shares the same per-thread session, SSRF policy, and + screenshot pipeline as :func:`browser_navigate_tool`. + + Returns ``{"screenshot": virtual_path|None, "url": str, "title": str}``. + Raises :class:`ValueError` when the URL fails SSRF validation. + """ + url_error = _validate_url("browser_navigate", url) + if url_error: + raise ValueError(url_error) + cfg = _get_tool_config("browser_navigate") + manager = get_browser_session_manager() + with manager.acquire_session( + thread_id, + headless=_as_bool(cfg.get("headless"), True), + timeout_ms=_as_int(cfg.get("timeout_ms"), 30000), + viewport={"width": _as_int(cfg.get("viewport_width"), 1280), "height": _as_int(cfg.get("viewport_height"), 720)}, + cdp_url=_as_str(cfg.get("cdp_url")), + allow_unguarded_cdp=_as_bool(cfg.get("allow_unguarded_cdp"), False), + url_guard=validate_browser_url, + ) as session: + snapshot = await session.navigate(url) + screenshot_path: str | None = None + try: + content = await session.screenshot_bytes(full_page=False) + name = _step_screenshot_name("navigate") + frames_dir = outputs_path / _BROWSER_FRAMES_DIRNAME + final_name = await asyncio.to_thread(_write_screenshot, frames_dir, name, content) + screenshot_path = f"{_FRAMES_VIRTUAL_PREFIX}/{final_name}" + except Exception as e: + logger.warning(f"browser gateway navigate screenshot failed: {e}") + return {"screenshot": screenshot_path, "url": snapshot.url, "title": snapshot.title} + + +@tool("browser_navigate", parse_docstring=True) +async def browser_navigate_tool(runtime: Runtime, url: str, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command: + """Open a URL in a live browser session and return the page's interactive elements. + + Use this to START a browsing flow. Unlike web_fetch (read-only), this keeps a + stateful browser so you can then click and type on the page. The result lists + interactive elements as ``[ref] role: name`` — use those ``[ref]`` numbers with + browser_click and browser_type. The session persists across tool calls for this + conversation until browser_close. Every navigate/click/type step is + auto-captured as a screenshot the user can see, so you do not need to call + browser_screenshot just to show progress. + URLs must include the scheme, e.g. https://example.com. + + Args: + url: The http(s) URL to open. + """ + try: + url_error = _validate_url("browser_navigate", url) + if url_error: + return _tool_message(url_error, tool_call_id) + with _resolve_session(runtime, "browser_navigate") as session: + snapshot = await session.navigate(url) + screenshot = await _capture_step_screenshot(runtime, session, "navigate") + return _snapshot_command(runtime, session, snapshot, tool_call_id, f"Navigated to {url}.", screenshot) + except Exception as e: + logger.error(f"browser_navigate failed: {e}") + return _tool_message(f"Error: browser navigation failed: {e}", tool_call_id) + + +@tool("browser_snapshot") +async def browser_snapshot_tool(runtime: Runtime, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command: + """Re-read the current page's interactive elements without acting. Use this to refresh the [ref] element list after the page changed on its own (e.g. async content loaded) or when you are unsure of the current state.""" + try: + with _resolve_session(runtime, "browser_snapshot") as session: + snapshot = await session.snapshot() + screenshot = await _capture_step_screenshot(runtime, session, "snapshot") + return _snapshot_command(runtime, session, snapshot, tool_call_id, "", screenshot) + except Exception as e: + logger.error(f"browser_snapshot failed: {e}") + return _tool_message(f"Error: browser snapshot failed: {e}", tool_call_id) + + +@tool("browser_click", parse_docstring=True) +async def browser_click_tool(runtime: Runtime, ref: int, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command: + """Click an interactive element by its ``[ref]`` number from the latest snapshot. + + The ref comes from the numbered element list returned by browser_navigate, + browser_snapshot, browser_click, or browser_type. Returns the updated page + snapshot after the click (with new ``[ref]`` numbers). + + Args: + ref: The element reference number to click. + """ + try: + with _resolve_session(runtime, "browser_click") as session: + snapshot = await session.click(ref) + screenshot = await _capture_step_screenshot(runtime, session, "click") + return _snapshot_command(runtime, session, snapshot, tool_call_id, f"Clicked element [{ref}].", screenshot) + except Exception as e: + logger.error(f"browser_click failed: {e}") + return _tool_message(f"Error: could not click element [{ref}]: {e}", tool_call_id) + + +@tool("browser_type", parse_docstring=True) +async def browser_type_tool( + runtime: Runtime, + ref: int, + text: str, + tool_call_id: Annotated[str, InjectedToolCallId], + submit: bool = False, +) -> Command: + """Type text into an input/textarea element by its ``[ref]`` number. + + Fills the field identified by ref. Set submit=true to press Enter afterward + (e.g. to run a search or submit a form). Returns the updated page snapshot. + + Args: + ref: The element reference number of the input to type into. + text: The text to enter. + submit: When true, press Enter after typing to submit. + """ + try: + with _resolve_session(runtime, "browser_type") as session: + snapshot = await session.type_text(ref, text, submit=submit) + action = f"Typed into element [{ref}] and submitted." if submit else f"Typed into element [{ref}]." + screenshot = await _capture_step_screenshot(runtime, session, "type") + return _snapshot_command(runtime, session, snapshot, tool_call_id, action, screenshot) + except Exception as e: + logger.error(f"browser_type failed: {e}") + return _tool_message(f"Error: could not type into element [{ref}]: {e}", tool_call_id) + + +@tool("browser_get_text") +async def browser_get_text_tool(runtime: Runtime, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command: + """Read the visible text content of the current page. Use this to extract readable text after navigating/interacting, e.g. to quote results or summarize content. Output is truncated for large pages.""" + try: + with _resolve_session(runtime, "browser_get_text") as session: + cfg = _get_tool_config("browser_get_text") + max_chars = _as_int(cfg.get("max_chars"), 8000) + text = await session.get_text(max_chars=max_chars) + return _tool_message(text or "(page has no visible text)", tool_call_id) + except Exception as e: + logger.error(f"browser_get_text failed: {e}") + return _tool_message(f"Error: could not read page text: {e}", tool_call_id) + + +@tool("browser_back") +async def browser_back_tool(runtime: Runtime, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command: + """Go back to the previous page in the browser session's history.""" + try: + with _resolve_session(runtime, "browser_back") as session: + snapshot = await session.back() + screenshot = await _capture_step_screenshot(runtime, session, "back") + return _snapshot_command(runtime, session, snapshot, tool_call_id, "Went back.", screenshot) + except Exception as e: + logger.error(f"browser_back failed: {e}") + return _tool_message(f"Error: could not go back: {e}", tool_call_id) + + +def _safe_screenshot_name(filename: str | None) -> str: + if filename: + stem = Path(filename).stem or "browser-capture" + else: + stem = f"browser-capture-{datetime.now(UTC).strftime('%Y%m%d-%H%M%S')}" + safe = _SAFE_FILENAME_RE.sub("_", stem).strip("._-") or "browser-capture" + return f"{safe[:100]}.png" + + +def _thread_outputs_path(runtime: Runtime) -> Path | str: + if runtime.state is None: + return "Error: Thread runtime state is not available" + thread_data = runtime.state.get("thread_data") or {} + outputs_path = thread_data.get("outputs_path") + if not outputs_path: + return "Error: Thread outputs path is not available" + return Path(outputs_path) + + +def _write_screenshot(outputs_path: Path, name: str, content: bytes) -> str: + outputs_path.mkdir(parents=True, exist_ok=True) + (outputs_path / name).write_bytes(content) + return name + + +@tool("browser_screenshot", parse_docstring=True) +async def browser_screenshot_tool( + runtime: Runtime, + tool_call_id: Annotated[str, InjectedToolCallId], + filename: str | None = None, + full_page: bool = False, +) -> Command: + """Capture a screenshot of the current browser page and save it as an artifact. + + Use this for visual evidence of the current interactive session state (after + clicking/typing), which web_capture cannot provide because it renders a fresh, + stateless page load. + + Args: + filename: Optional output filename (extension is forced to .png). + full_page: Capture the full scrollable page instead of just the viewport. + """ + try: + outputs_path = _thread_outputs_path(runtime) + if isinstance(outputs_path, str): + return _tool_message(outputs_path, tool_call_id) + with _resolve_session(runtime, "browser_screenshot") as session: + content = await session.screenshot_bytes(full_page=full_page) + name = _safe_screenshot_name(filename) + final_name = await asyncio.to_thread(_write_screenshot, outputs_path, name, content) + virtual_path = f"{_OUTPUTS_VIRTUAL_PREFIX}/{final_name}" + return Command( + update={ + "artifacts": [virtual_path], + "messages": [ToolMessage(f"Saved browser screenshot: {virtual_path}", tool_call_id=tool_call_id)], + } + ) + except Exception as e: + logger.error(f"browser_screenshot failed: {e}") + return _tool_message(f"Error: could not capture screenshot: {e}", tool_call_id) + + +@tool("browser_close") +async def browser_close_tool(runtime: Runtime, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command: + """Close the current browser session and free its resources. Call this when done with the browsing flow; a later browser_navigate starts a fresh session.""" + try: + manager = get_browser_session_manager() + closed = await manager.close_session(_thread_id(runtime)) + msg = "Browser session closed." if closed else "No active browser session to close." + return _tool_message(msg, tool_call_id) + except Exception as e: + logger.error(f"browser_close failed: {e}") + return _tool_message(f"Error: could not close browser session: {e}", tool_call_id) diff --git a/backend/packages/harness/deerflow/constants.py b/backend/packages/harness/deerflow/constants.py index 186675ac0..c97abaa0f 100644 --- a/backend/packages/harness/deerflow/constants.py +++ b/backend/packages/harness/deerflow/constants.py @@ -1,3 +1,10 @@ """Shared runtime protocol constants.""" DEFAULT_SKILLS_CONTAINER_PATH = "/mnt/skills" + +# Hidden subdirectory (under a thread's outputs dir) that holds the browser +# tools' per-step screenshots. These are transient live-progress frames, not +# deliverables, so the workspace-changes scanner excludes this directory. Both +# the browser tools (which write here) and the scanner (which ignores it) import +# this single source of truth so the name cannot drift between them. +BROWSER_FRAMES_DIRNAME = ".browser-frames" diff --git a/backend/packages/harness/deerflow/workspace_changes/scanner.py b/backend/packages/harness/deerflow/workspace_changes/scanner.py index de22b0276..e69c3b01a 100644 --- a/backend/packages/harness/deerflow/workspace_changes/scanner.py +++ b/backend/packages/harness/deerflow/workspace_changes/scanner.py @@ -6,6 +6,8 @@ import os from codecs import BOM_UTF16_BE, BOM_UTF16_LE, getincrementaldecoder from pathlib import Path +from deerflow.constants import BROWSER_FRAMES_DIRNAME + from .types import ( DiffUnavailableReason, FileSnapshot, @@ -21,6 +23,10 @@ EXCLUDED_DIR_NAMES = { ".cache", ".next", ".venv", + # Transient per-step browser screenshots: live progress feedback surfaced in + # the browser panel + inline thumbnails, not workspace deliverables. Shared + # constant with the browser tools so the name cannot drift out of sync. + BROWSER_FRAMES_DIRNAME, "__pycache__", "build", "dist", diff --git a/backend/packages/harness/pyproject.toml b/backend/packages/harness/pyproject.toml index d78a19e27..a8349a3bc 100644 --- a/backend/packages/harness/pyproject.toml +++ b/backend/packages/harness/pyproject.toml @@ -68,6 +68,10 @@ boxlite = ["boxlite>=0.9.7"] # Agent observability (Monocle). Optional so a default install stays free of the # OpenTelemetry stack; only pulled in when MONOCLE_TRACING is used. monocle = ["monocle_apptrace>=0.8.8"] +# Agentic browser control (browser_navigate/click/type/... tool group). Optional +# so the core harness install stays lean; import is lazy inside the private +# Playwright loop. After install, run `playwright install chromium` once. +browser = ["playwright>=1.40"] [build-system] requires = ["hatchling"] diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 917d0c42d..3d0e8216b 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -29,6 +29,7 @@ postgres = ["deerflow-harness[postgres]"] redis = ["deerflow-harness[redis]"] discord = ["discord.py>=2.7.0"] monocle = ["deerflow-harness[monocle]"] +browser = ["deerflow-harness[browser]"] [dependency-groups] dev = [ diff --git a/backend/tests/manual_browser_live_check.py b/backend/tests/manual_browser_live_check.py new file mode 100644 index 000000000..8cce37e93 --- /dev/null +++ b/backend/tests/manual_browser_live_check.py @@ -0,0 +1,171 @@ +"""Live end-to-end verification for the agentic browser tools. + +NOT a unit test. Run manually with DEEPSEEK_API_KEY in the environment: + + DEEPSEEK_API_KEY=sk-... PYTHONPATH=. uv run python tests/manual_browser_live_check.py + +It: + 1. serves a tiny local HTML form, + 2. builds an isolated DeerFlow config (DeepSeek model + browser tool group), + 3. runs a real agent turn that must navigate, type, submit, and read the result, + 4. asserts the agent-visible tool trace shows the browser loop actually ran. +""" + +from __future__ import annotations + +import os +import tempfile +import threading +from http.server import BaseHTTPRequestHandler, HTTPServer +from pathlib import Path + +FORM_PAGE = """DeerFlow Browser Test + +

Sign-in demo

+
+ + +
+""" + + +def _welcome_page(username: str) -> str: + return f"""Welcome +

Welcome, {username}!

SECRET-TOKEN-4917

""" + + +class _Handler(BaseHTTPRequestHandler): + def log_message(self, *args): # silence server logs + pass + + def do_GET(self): + from urllib.parse import parse_qs, urlparse + + parsed = urlparse(self.path) + if parsed.path.startswith("/welcome"): + qs = parse_qs(parsed.query) + username = (qs.get("username") or ["friend"])[0] + body = _welcome_page(username) + else: + body = FORM_PAGE + data = body.encode() + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + +def _start_server() -> tuple[HTTPServer, int]: + server = HTTPServer(("127.0.0.1", 0), _Handler) + port = server.server_address[1] + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + return server, port + + +def _write_config(tmp: Path) -> Path: + config = f""" +config_version: 1 +data_dir: {tmp / "data"} + +models: + - name: deepseek-chat + display_name: DeepSeek Chat + use: deerflow.models.patched_deepseek:PatchedChatDeepSeek + model: deepseek-chat + api_key: $DEEPSEEK_API_KEY + timeout: 120.0 + max_retries: 2 + max_tokens: 4096 + +sandbox: + use: deerflow.sandbox.local:LocalSandboxProvider + +tool_groups: + - name: browser + +tools: + - name: browser_navigate + group: browser + use: deerflow.community.browser_automation.tools:browser_navigate_tool + headless: true + allow_private_addresses: true + - name: browser_snapshot + group: browser + use: deerflow.community.browser_automation.tools:browser_snapshot_tool + - name: browser_click + group: browser + use: deerflow.community.browser_automation.tools:browser_click_tool + - name: browser_type + group: browser + use: deerflow.community.browser_automation.tools:browser_type_tool + - name: browser_get_text + group: browser + use: deerflow.community.browser_automation.tools:browser_get_text_tool + - name: browser_close + group: browser + use: deerflow.community.browser_automation.tools:browser_close_tool + +memory: + enabled: false + +title: + enabled: false +""" + path = tmp / "config.yaml" + path.write_text(config) + return path + + +def main() -> int: + if not os.environ.get("DEEPSEEK_API_KEY"): + print("SKIP: DEEPSEEK_API_KEY not set") + return 0 + + server, port = _start_server() + base = f"http://127.0.0.1:{port}/" + tmpdir = Path(tempfile.mkdtemp(prefix="deerflow-browser-live-")) + try: + from deerflow.client import DeerFlowClient + + config_path = _write_config(tmpdir) + # Make config resolution deterministic: get_available_tools() re-resolves + # via get_app_config(), which would otherwise pick up a project-root + # config.yaml. DEER_FLOW_CONFIG_PATH is resolution priority #2. + os.environ["DEER_FLOW_CONFIG_PATH"] = str(config_path) + client = DeerFlowClient(config_path=str(config_path)) + + prompt = ( + f"Use the browser tools to complete this task. " + f"1) Navigate to {base} 2) type the username 'deerbot' into the username field " + f"3) click the Sign in button 4) read the resulting page's text. " + f"Then tell me the exact SECRET token shown on the welcome page." + ) + + tool_calls: list[str] = [] + final_text = "" + for event in client.stream(prompt, thread_id="browser-live-check"): + if event.type == "messages-tuple": + data = event.data or {} + if data.get("type") == "tool": + name = data.get("name") or "" + if name.startswith("browser_"): + tool_calls.append(name) + elif data.get("type") == "ai": + final_text += data.get("content") or "" + + print("Browser tool calls observed:", tool_calls) + print("Final answer:\n", final_text.strip()[:1000]) + + assert "browser_navigate" in tool_calls, "agent never navigated" + assert any(t in tool_calls for t in ("browser_type", "browser_click")), "agent never interacted" + assert "SECRET-TOKEN-4917" in final_text, "agent did not read the post-submit page content" + print("\nLIVE CHECK PASSED") + return 0 + finally: + server.shutdown() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/tests/test_browser_automation.py b/backend/tests/test_browser_automation.py new file mode 100644 index 000000000..a6563fcdb --- /dev/null +++ b/backend/tests/test_browser_automation.py @@ -0,0 +1,1066 @@ +"""Tests for the agentic browser automation tools and session manager. + +The tool tests mock the browser session so they run without Playwright. A small +integration test at the end exercises a real headless Chromium session and is +skipped automatically when Playwright (or its browser binary) is unavailable. +""" + +from __future__ import annotations + +import asyncio +import sys +from types import ModuleType, SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from deerflow.community.browser_automation import session as session_mod +from deerflow.community.browser_automation import tools +from deerflow.community.browser_automation.session import ( + _LIVE_FRAME_JPEG_QUALITY, + BrowserLiveViewerError, + BrowserSession, + BrowserSessionCapacityError, + BrowserSessionManager, + PageSnapshot, + SnapshotElement, +) + +PlaywrightTimeoutError = type( + "TimeoutError", + (TimeoutError,), + {"__module__": "playwright.async_api"}, +) + + +def _runtime(thread_id: str | None = "thread-1", outputs_path: str | None = None): + state = {"thread_data": {"outputs_path": outputs_path}} if outputs_path is not None else {"thread_data": {}} + return SimpleNamespace(context={"thread_id": thread_id}, state=state) + + +def _snapshot() -> PageSnapshot: + return PageSnapshot( + url="https://example.com/", + title="Example", + elements=[ + SnapshotElement(ref=1, tag="a", role="", type="", name="More info"), + SnapshotElement(ref=2, tag="input", role="", type="text", name="Search"), + ], + ) + + +class TestSnapshotRendering: + def test_snapshot_lists_elements_by_ref(self): + rendered = _snapshot().render() + assert "URL: https://example.com/" in rendered + assert "Title: Example" in rendered + assert "[1] a: More info" in rendered + assert "[2] input type=text: Search" in rendered + + def test_empty_snapshot_says_no_elements(self): + snap = PageSnapshot(url="https://x.test/", title="X", elements=[]) + assert "No interactive elements detected." in snap.render() + + +@pytest.mark.asyncio +class TestBrowserTools: + async def _patch_session(self, session): + manager = MagicMock() + manager.get_session.return_value = session + return patch.object(tools, "get_browser_session_manager", return_value=manager), manager + + async def test_navigate_returns_snapshot(self): + session = MagicMock() + session.navigate = AsyncMock(return_value=_snapshot()) + ctx, manager = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_navigate_tool.coroutine( + runtime=_runtime(), + url="https://example.com", + tool_call_id="t1", + ) + content = result.update["messages"][0].content + assert "Navigated to https://example.com." in content + assert "[1] a: More info" in content + session.navigate.assert_awaited_once_with("https://example.com") + manager.get_session.assert_called_once() + + async def test_navigate_emits_screenshot_artifact_and_browser_view(self, tmp_path): + outputs = tmp_path / "outputs" + outputs.mkdir() + session = MagicMock() + session.navigate = AsyncMock(return_value=_snapshot()) + session.screenshot_bytes = AsyncMock(return_value=b"\x89PNG\r\n\x1a\nshot") + session.schedule_live_frames = MagicMock() + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_navigate_tool.coroutine( + runtime=_runtime(outputs_path=str(outputs)), + url="https://example.com", + tool_call_id="t1", + ) + # Screenshot is captured, saved, exposed as an artifact + inline browser_view. + session.screenshot_bytes.assert_awaited_once() + session.schedule_live_frames.assert_called_once() + artifact = result.update["artifacts"][0] + assert artifact.startswith("/mnt/user-data/outputs/.browser-frames/browser-navigate-") + assert artifact.endswith(".png") + saved = list((outputs / ".browser-frames").glob("browser-navigate-*.png")) + assert saved and saved[0].read_bytes() == b"\x89PNG\r\n\x1a\nshot" + meta = result.update["messages"][0].additional_kwargs["browser_view"] + assert meta["screenshot"] == artifact + assert meta["url"] == "https://example.com/" + assert meta["title"] == "Example" + + async def test_navigate_screenshot_failure_does_not_break_action(self, tmp_path): + outputs = tmp_path / "outputs" + outputs.mkdir() + session = MagicMock() + session.navigate = AsyncMock(return_value=_snapshot()) + session.screenshot_bytes = AsyncMock(side_effect=RuntimeError("headless crashed")) + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_navigate_tool.coroutine( + runtime=_runtime(outputs_path=str(outputs)), + url="https://example.com", + tool_call_id="t1", + ) + # Navigation result still returned; no artifact, no browser_view, no crash. + assert "Navigated to https://example.com." in result.update["messages"][0].content + assert "artifacts" not in result.update + assert result.update["messages"][0].additional_kwargs == {} + + async def test_navigate_blocks_private_url(self): + session = MagicMock() + session.navigate = AsyncMock() + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_navigate_tool.coroutine( + runtime=_runtime(), + url="http://169.254.169.254/latest/meta-data/", + tool_call_id="t1", + ) + assert "private, loopback, or metadata" in result.update["messages"][0].content + session.navigate.assert_not_awaited() + + async def test_navigate_rejects_non_http_scheme(self): + session = MagicMock() + session.navigate = AsyncMock() + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_navigate_tool.coroutine( + runtime=_runtime(), + url="file:///etc/passwd", + tool_call_id="t1", + ) + assert "Error" in result.update["messages"][0].content + session.navigate.assert_not_awaited() + + async def test_click_returns_updated_snapshot(self): + session = MagicMock() + session.click = AsyncMock(return_value=_snapshot()) + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_click_tool.coroutine(runtime=_runtime(), ref=1, tool_call_id="t1") + assert "Clicked element [1]." in result.update["messages"][0].content + session.click.assert_awaited_once_with(1) + + async def test_click_error_is_recoverable_message(self): + session = MagicMock() + session.click = AsyncMock(side_effect=RuntimeError("no such element")) + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_click_tool.coroutine(runtime=_runtime(), ref=9, tool_call_id="t1") + assert result.update["messages"][0].content.startswith("Error: could not click element [9]") + + async def test_type_with_submit(self): + session = MagicMock() + session.type_text = AsyncMock(return_value=_snapshot()) + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_type_tool.coroutine( + runtime=_runtime(), + ref=2, + text="hello", + tool_call_id="t1", + submit=True, + ) + assert "Typed into element [2] and submitted." in result.update["messages"][0].content + session.type_text.assert_awaited_once_with(2, "hello", submit=True) + + async def test_get_text_truncates_via_config(self): + session = MagicMock() + session.get_text = AsyncMock(return_value="body text") + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={"max_chars": 1234}): + result = await tools.browser_get_text_tool.coroutine(runtime=_runtime(), tool_call_id="t1") + assert result.update["messages"][0].content == "body text" + session.get_text.assert_awaited_once_with(max_chars=1234) + + async def test_screenshot_writes_artifact(self, tmp_path): + outputs = tmp_path / "outputs" + outputs.mkdir() + session = MagicMock() + session.screenshot_bytes = AsyncMock(return_value=b"\x89PNG\r\n\x1a\npng-bytes") + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_screenshot_tool.coroutine( + runtime=_runtime(outputs_path=str(outputs)), + tool_call_id="t1", + filename="Login Page.png", + ) + artifact = result.update["artifacts"][0] + assert artifact == "/mnt/user-data/outputs/Login_Page.png" + assert (outputs / "Login_Page.png").read_bytes() == b"\x89PNG\r\n\x1a\npng-bytes" + + async def test_screenshot_errors_without_outputs_path(self): + session = MagicMock() + session.screenshot_bytes = AsyncMock(return_value=b"x") + ctx, _ = await self._patch_session(session) + with ctx, patch.object(tools, "_get_tool_config", return_value={}): + result = await tools.browser_screenshot_tool.coroutine(runtime=_runtime(), tool_call_id="t1") + assert "outputs path is not available" in result.update["messages"][0].content + session.screenshot_bytes.assert_not_awaited() + + async def test_close_reports_when_no_session(self): + manager = MagicMock() + manager.close_session = AsyncMock(return_value=False) + with patch.object(tools, "get_browser_session_manager", return_value=manager): + result = await tools.browser_close_tool.coroutine(runtime=_runtime(), tool_call_id="t1") + assert "No active browser session" in result.update["messages"][0].content + manager.close_session.assert_awaited_once_with("thread-1") + + +@pytest.mark.asyncio +async def test_wheel_input_scrolls_target_container_at_pointer_location(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.mouse.move = AsyncMock() + page.mouse.wheel = AsyncMock() + page.evaluate = AsyncMock(return_value=True) + session._ensure_page = AsyncMock(return_value=page) + + await session._dispatch_input( + {"type": "wheel", "nx": 0.25, "ny": 0.75, "dx": 0, "dy": 240}, + ) + + page.mouse.move.assert_awaited_once_with(250.0, 375.0) + page.evaluate.assert_awaited_once() + page.mouse.wheel.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_wheel_input_scrolls_viewport_center_without_pointer_location(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.mouse.move = AsyncMock() + page.mouse.wheel = AsyncMock() + page.evaluate = AsyncMock(return_value=True) + session._ensure_page = AsyncMock(return_value=page) + + await session._dispatch_input({"type": "wheel", "dx": 0, "dy": 240}) + + page.mouse.move.assert_not_awaited() + page.evaluate.assert_awaited_once() + _, payload = page.evaluate.await_args.args + assert payload == {"x": 500.0, "y": 250.0, "dx": 0.0, "dy": 240.0} + page.mouse.wheel.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_wheel_input_falls_back_to_native_wheel_when_js_scroll_fails(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.mouse.move = AsyncMock() + page.mouse.wheel = AsyncMock() + page.evaluate = AsyncMock(return_value=False) + session._ensure_page = AsyncMock(return_value=page) + + await session._dispatch_input( + {"type": "wheel", "nx": 0.25, "ny": 0.75, "dx": 0, "dy": 240}, + ) + + page.mouse.move.assert_awaited_once_with(250.0, 375.0) + page.evaluate.assert_awaited_once() + page.mouse.wheel.assert_awaited_once_with(0.0, 240.0) + + +@pytest.mark.asyncio +async def test_live_frame_returns_base64_jpeg_screenshot(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.screenshot = AsyncMock(return_value=b"\xff\xd8jpeg-bytes") + session._ensure_page = AsyncMock(return_value=page) + + frame = await session._live_frame() + + assert frame == "/9hq cGVnLWJ5dGVz".replace(" ", "") + page.screenshot.assert_awaited_once_with(type="jpeg", quality=_LIVE_FRAME_JPEG_QUALITY) + + +@pytest.mark.asyncio +async def test_input_dispatch_does_not_wait_for_live_frame(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.mouse.click = AsyncMock() + session._ensure_page = AsyncMock(return_value=page) + + refresh_started = asyncio.Event() + release_refresh = asyncio.Event() + + async def slow_live_frame() -> None: + refresh_started.set() + await release_refresh.wait() + + session._on_frame = MagicMock() + session._emit_live_frame = slow_live_frame + session._schedule_settle_live_frames = MagicMock() + dispatch_task = asyncio.create_task( + session._dispatch_input({"type": "click", "nx": 0.25, "ny": 0.75}), + ) + + await asyncio.wait_for(refresh_started.wait(), timeout=0.2) + try: + await asyncio.wait_for(asyncio.shield(dispatch_task), timeout=0.05) + finally: + release_refresh.set() + await dispatch_task + + +@pytest.mark.asyncio +async def test_rapid_inputs_coalesce_live_frame_refresh(monkeypatch): + monkeypatch.setattr(session_mod, "_LIVE_FRAME_INPUT_INTERVAL_S", 0.01) + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.keyboard.press = AsyncMock() + session._ensure_page = AsyncMock(return_value=page) + session._on_frame = MagicMock() + session._emit_live_frame = AsyncMock() + session._schedule_settle_live_frames = MagicMock() + + await session._dispatch_input({"type": "key", "key": "ArrowDown"}) + await session._dispatch_input({"type": "key", "key": "ArrowDown"}) + await session._dispatch_input({"type": "key", "key": "ArrowDown"}) + await asyncio.sleep(0.03) + + assert page.keyboard.press.await_count == 3 + session._emit_live_frame.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_continuous_inputs_refresh_before_input_stops(monkeypatch): + monkeypatch.setattr(session_mod, "_LIVE_FRAME_INPUT_INTERVAL_S", 0.01) + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + page.keyboard.press = AsyncMock() + session._ensure_page = AsyncMock(return_value=page) + session._on_frame = MagicMock() + session._live_frame = AsyncMock(return_value="frame") + session._schedule_settle_live_frames = MagicMock() + + stop = asyncio.Event() + + async def send_continuously() -> None: + while not stop.is_set(): + await session._dispatch_input({"type": "key", "key": "ArrowDown"}) + await asyncio.sleep(0.001) + + producer = asyncio.create_task(send_continuously()) + try: + await asyncio.sleep(0.06) + assert session._on_frame.call_count >= 2 + finally: + stop.set() + await producer + await asyncio.sleep(0.02) + + +@pytest.mark.asyncio +async def test_click_fast_fails_on_stale_ref_without_blocking(): + """A ref missing after a re-render fails immediately with a re-snapshot hint. + + Previously ``page.click`` on a missing selector blocked for the 30s session + default; here the locator reports count 0 so we raise straight away and never + attempt the click, letting the model re-snapshot instead of stalling the loop. + """ + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=30000, + viewport={"width": 1000, "height": 500}, + ) + locator = MagicMock() + locator.count = AsyncMock(return_value=0) + page = MagicMock() + page.locator = MagicMock(return_value=locator) + session._ensure_page = AsyncMock(return_value=page) + + with pytest.raises(RuntimeError, match="no longer on the page"): + await session._click(7) + + page.locator.assert_called_once_with('[data-df-ref="7"]') + + +@pytest.mark.asyncio +async def test_click_tolerates_spa_navigation_without_load_event(): + """A client-side click that never fires a load event still returns a snapshot. + + The post-click settle wait is best-effort: a Playwright timeout on + ``wait_for_load_state`` must be swallowed so SPA navigations (which never emit + a fresh load event) still yield the updated snapshot instead of raising. + """ + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=30000, + viewport={"width": 1000, "height": 500}, + ) + first = MagicMock() + first.scroll_into_view_if_needed = AsyncMock() + first.click = AsyncMock() + locator = MagicMock() + locator.count = AsyncMock(return_value=1) + locator.first = first + page = MagicMock() + page.locator = MagicMock(return_value=locator) + page.wait_for_load_state = AsyncMock(side_effect=PlaywrightTimeoutError("no load")) + session._ensure_page = AsyncMock(return_value=page) + session._snapshot_impl = AsyncMock(return_value=_snapshot()) + + result = await session._click(1) + + assert result.url == "https://example.com/" + first.click.assert_awaited_once() + page.wait_for_load_state.assert_awaited_once() + session._snapshot_impl.assert_awaited_once_with(page) + + +@pytest.mark.asyncio +async def test_ensure_page_serializes_concurrent_rebuilds(): + """Concurrent tool/Live callers must share one rebuilt browser page.""" + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + launch_started = asyncio.Event() + release_launch = asyncio.Event() + page = MagicMock() + page.is_closed.return_value = False + context = MagicMock() + context.new_page = AsyncMock(return_value=page) + context.set_default_timeout = MagicMock() + context.on = MagicMock() + browser = MagicMock() + browser.is_connected.return_value = True + browser.new_context = AsyncMock(return_value=context) + + async def launch(*, headless): + launch_started.set() + await release_launch.wait() + return browser + + chromium = MagicMock() + chromium.launch = AsyncMock(side_effect=launch) + session._playwright = SimpleNamespace(chromium=chromium) + + # ``_ensure_page`` imports Playwright lazily even when a fake runtime is + # already installed. Keep this unit test independent of the optional extra. + fake_async_api = ModuleType("playwright.async_api") + fake_async_api.async_playwright = MagicMock() + fake_playwright = ModuleType("playwright") + fake_playwright.async_api = fake_async_api + with patch.dict(sys.modules, {"playwright": fake_playwright, "playwright.async_api": fake_async_api}): + first = asyncio.create_task(session._ensure_page()) + await asyncio.wait_for(launch_started.wait(), timeout=1.0) + second = asyncio.create_task(session._ensure_page()) + await asyncio.sleep(0) + release_launch.set() + + assert await asyncio.wait_for(asyncio.gather(first, second), timeout=1.0) == [page, page] + chromium.launch.assert_awaited_once_with(headless=True) + browser.new_context.assert_awaited_once() + context.new_page.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_tabs_report_active_page(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page_a = MagicMock() + page_a.is_closed.return_value = False + page_a.title = AsyncMock(return_value="A") + page_a.url = "https://a.example/" + page_b = MagicMock() + page_b.is_closed.return_value = False + page_b.title = AsyncMock(return_value="B") + page_b.url = "https://b.example/" + session._context = SimpleNamespace(pages=[page_a, page_b]) + session._page = page_b + session._ensure_page = AsyncMock(return_value=page_b) + + tabs = await session._tabs() + + assert [(tab.index, tab.title, tab.url, tab.active) for tab in tabs] == [ + (0, "A", "https://a.example/", False), + (1, "B", "https://b.example/", True), + ] + + +@pytest.mark.asyncio +async def test_activate_tab_switches_active_page(): + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page_a = MagicMock() + page_a.is_closed.return_value = False + page_b = MagicMock() + page_b.is_closed.return_value = False + page_b.bring_to_front = AsyncMock() + session._context = SimpleNamespace(pages=[page_a, page_b]) + session._page = page_a + session._ensure_page = AsyncMock(return_value=page_a) + + await session._activate_tab(1) + + assert session._page is page_b + page_b.bring_to_front.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_set_active_page_schedules_rebind_when_screencast_running(): + """Switching the active page while streaming must rebind the screencast. + + The frame source follows ``self._page``; the CDP repaint signal is bound to a + specific page, so a page switch has to reschedule the bind or new-page + repaints stop driving frames (the exact drift that froze Live on the old + page). No rebind should be scheduled when the page is unchanged or idle. + """ + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + old_page = MagicMock() + new_page = MagicMock() + session._screencast_page = old_page + rebind = AsyncMock() + session._rebind_screencast_safe = rebind + + # Not streaming yet: no rebind. + session._set_active_page(new_page) + assert session._page is new_page + rebind.assert_not_called() + + # Streaming and the page actually changed: schedule exactly one rebind. + session._on_frame = lambda _data: None + another_page = MagicMock() + session._set_active_page(another_page) + await asyncio.sleep(0) # let the scheduled task run + rebind.assert_awaited_once() + + # Same page again: no extra rebind. + session._screencast_page = another_page + rebind.reset_mock() + session._set_active_page(another_page) + await asyncio.sleep(0) + rebind.assert_not_called() + + +@pytest.mark.asyncio +async def test_live_frame_screenshots_current_active_page_after_switch(): + """A live frame captures ``self._page``, not a stale handle. + + ``_live_frame`` and the screencast both screenshot the active page, so after + the active page is reassigned (tab/popup handoff) the next frame reflects the + new page — keeping Live aligned with the address bar and snapshot, which also + track ``self._page``. + """ + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + new_page = MagicMock() + new_page.screenshot = AsyncMock(return_value=b"\xff\xd8new-page") + session._ensure_page = AsyncMock(return_value=new_page) + + frame = await session._live_frame() + + new_page.screenshot.assert_awaited_once_with(type="jpeg", quality=_LIVE_FRAME_JPEG_QUALITY) + assert frame # base64 payload of the new page + + +@pytest.mark.asyncio +async def test_stop_screencast_ignores_stale_connection_callback(): + """A closing old WebSocket must not clear the newer Live frame callback.""" + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + + def old_frame(_data: str) -> None: + pass + + def new_frame(_data: str) -> None: + pass + + session._on_frame = new_frame + + await session._stop_screencast(old_frame) + + assert session._on_frame is new_frame + + await session._stop_screencast(new_frame) + + assert session._on_frame is None + + +@pytest.mark.asyncio +async def test_start_screencast_rejects_second_live_viewer(): + """A second Live connection must not silently freeze the first viewer.""" + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + page = MagicMock() + session._ensure_page = AsyncMock(return_value=page) + + def old_frame(_data: str) -> None: + pass + + def new_frame(_data: str) -> None: + pass + + session._on_frame = old_frame + session._screencast_page = page + + with pytest.raises(BrowserLiveViewerError): + await session._start_screencast(new_frame) + + assert session._on_frame is old_frame + + +@pytest.mark.asyncio +class TestSessionManager: + async def test_get_session_is_per_thread_and_cached(self): + manager = BrowserSessionManager() + fake_loop = MagicMock() + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + a1 = manager.get_session("thread-a") + a2 = manager.get_session("thread-a") + b1 = manager.get_session("thread-b") + assert a1 is a2 + assert a1 is not b1 + + async def test_close_session_removes_and_closes(self): + manager = BrowserSessionManager() + fake_loop = MagicMock() + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + session = manager.get_session("thread-a") + session.close = AsyncMock() + assert await manager.close_session("thread-a") is True + session.close.assert_awaited_once() + # Second close is a no-op because the session was dropped. + assert await manager.close_session("thread-a") is False + + async def test_idle_sessions_are_evicted_on_next_get(self): + """A session unused past the idle timeout is dropped + scheduled to close. + + The active (just-requested) thread is always kept; only the stale one is + evicted, so a long-running gateway can't accumulate one Chromium per + thread that ever touched the tools. + """ + manager = BrowserSessionManager(idle_timeout_s=100.0, max_sessions=0) + fake_loop = MagicMock() + manager._loop = fake_loop + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + with patch.object(session_mod.time, "monotonic", return_value=0.0): + stale = manager.get_session("thread-stale") + # Avoid an un-awaited coroutine warning: the fake loop.submit does + # not consume the coroutine _schedule_close would build. + stale._close = MagicMock() + with patch.object(session_mod.time, "monotonic", return_value=1000.0): + fresh = manager.get_session("thread-fresh") + assert "thread-stale" not in manager._sessions + assert "thread-fresh" in manager._sessions + # The evicted session is closed on the private loop (fire-and-forget). + fake_loop.submit.assert_called_once() + assert fresh is manager._sessions["thread-fresh"] + del stale + + async def test_max_sessions_cap_evicts_least_recently_used(self): + manager = BrowserSessionManager(idle_timeout_s=0, max_sessions=2) + fake_loop = MagicMock() + manager._loop = fake_loop + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + with patch.object(session_mod.time, "monotonic", return_value=1.0): + manager.get_session("a") + with patch.object(session_mod.time, "monotonic", return_value=2.0): + evicted = manager.get_session("b") + evicted._close = MagicMock() + # Re-touch "a" so "b" becomes the least-recently-used. + with patch.object(session_mod.time, "monotonic", return_value=3.0): + manager.get_session("a") + with patch.object(session_mod.time, "monotonic", return_value=4.0): + manager.get_session("c") + assert set(manager._sessions) == {"a", "c"} + assert "b" not in manager._sessions + fake_loop.submit.assert_called_once() + + async def test_pinned_session_survives_lru_eviction(self): + manager = BrowserSessionManager(idle_timeout_s=0, max_sessions=2) + fake_loop = MagicMock() + manager._loop = fake_loop + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + with patch.object(session_mod.time, "monotonic", return_value=0.0): + pinned = manager.get_session("pinned", pin=True) + pinned._close = MagicMock() + with patch.object(session_mod.time, "monotonic", return_value=1.0): + lru = manager.get_session("lru") + lru._close = MagicMock() + + with patch.object(session_mod.time, "monotonic", return_value=2.0): + current = manager.get_session("current") + current._close = MagicMock() + + # The pinned stream is the oldest session, but the LRU cap must evict + # the oldest unpinned session instead. + assert "pinned" in manager._sessions + assert "lru" not in manager._sessions + assert current is manager._sessions["current"] + + async def test_pinned_session_makes_capacity_admission_fail_until_release(self): + manager = BrowserSessionManager(idle_timeout_s=0, max_sessions=1) + fake_loop = MagicMock() + manager._loop = fake_loop + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + with patch.object(session_mod.time, "monotonic", return_value=0.0): + pinned = manager.get_session("pinned", pin=True) + pinned._close = MagicMock() + with patch.object(session_mod.time, "monotonic", return_value=1.0): + with pytest.raises(BrowserSessionCapacityError): + manager.get_session("current") + + assert set(manager._sessions) == {"pinned"} + assert fake_loop.submit.call_count == 0 + + manager.release_session("pinned", pinned) + assert set(manager._sessions) == {"pinned"} + + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + with patch.object(session_mod.time, "monotonic", return_value=2.0): + manager.get_session("current") + + assert set(manager._sessions) == {"current"} + fake_loop.submit.assert_called_once() + + async def test_get_session_rejects_runtime_multi_worker_browser_use(self, monkeypatch): + monkeypatch.setenv("GATEWAY_WORKERS", "2") + manager = BrowserSessionManager() + + with pytest.raises(RuntimeError, match="process-local"): + manager.get_session("thread-a") + + async def test_cdp_requires_explicit_unguarded_trust_opt_in(self): + manager = BrowserSessionManager() + + with pytest.raises(RuntimeError, match="allow_unguarded_cdp"): + manager.get_session("thread-a", cdp_url="http://127.0.0.1:9222") + + async def test_cdp_allows_explicit_unguarded_trust_opt_in(self): + manager = BrowserSessionManager() + fake_loop = MagicMock() + + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + session = manager.get_session( + "thread-a", + cdp_url="http://127.0.0.1:9222", + allow_unguarded_cdp=True, + ) + + assert session._cdp_url == "http://127.0.0.1:9222" + + async def test_acquire_session_releases_pin_after_scope(self): + manager = BrowserSessionManager() + fake_loop = MagicMock() + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + with manager.acquire_session("thread-a") as session: + assert session.active_refs == 1 + assert manager._sessions["thread-a"] is session + assert session.active_refs == 0 + + async def test_in_flight_browser_operation_is_not_evicted(self): + manager = BrowserSessionManager(idle_timeout_s=100.0, max_sessions=0) + fake_loop = MagicMock() + manager._loop = fake_loop + with patch.object(manager, "_ensure_loop", return_value=fake_loop): + session = manager.get_session("thread-a") + session._close = MagicMock() + page = MagicMock() + page.is_closed.return_value = False + page.url = "https://example.com/" + session._page = page + started = asyncio.Event() + release = asyncio.Event() + + async def run(coro): + started.set() + await release.wait() + return await coro + + fake_loop.run = run + operation = asyncio.create_task(session.current_url()) + await started.wait() + + with patch.object(session_mod.time, "monotonic", return_value=1000.0): + manager.get_session("thread-b") + assert "thread-a" in manager._sessions + assert session.active_refs == 1 + + release.set() + assert await operation == "https://example.com/" + assert session.active_refs == 0 + + +def test_resolve_session_always_reads_browser_navigate_config(): + """Launch config is deterministic regardless of which tool runs first. + + ``get_session`` caches per thread and ignores launch params for later + callers, so keying config off the calling tool made it "first tool to run + wins" — a ``headless: false`` set only on ``browser_navigate`` was silently + dropped if e.g. ``browser_snapshot`` created the session first. We now always + read ``browser_navigate``'s config as the single canonical source. + """ + configs = { + "browser_navigate": { + "headless": False, + "viewport_width": 1920, + "viewport_height": 1080, + "timeout_ms": 12345, + "cdp_url": "http://127.0.0.1:9222", + "allow_unguarded_cdp": True, + }, + "browser_snapshot": {"headless": True}, + } + captured: dict[str, object] = {} + + class _FakeManager: + def get_session(self, thread_id, **kwargs): + captured.update(kwargs) + captured["thread_id"] = thread_id + return MagicMock() + + with ( + patch.object(tools, "_get_tool_config", lambda name: configs.get(name, {})), + patch.object(tools, "get_browser_session_manager", return_value=_FakeManager()), + ): + # Even when a NON-navigate tool resolves the session, launch config comes + # from browser_navigate. + tools._resolve_session(_runtime(), "browser_snapshot") + + assert captured["headless"] is False + assert captured["timeout_ms"] == 12345 + assert captured["viewport"] == {"width": 1920, "height": 1080} + assert captured["cdp_url"] == "http://127.0.0.1:9222" + assert captured["allow_unguarded_cdp"] is True + + +def test_reset_manager_singleton(): + first = session_mod.get_browser_session_manager() + session_mod.reset_browser_session_manager() + second = session_mod.get_browser_session_manager() + assert first is not second + + +@pytest.mark.asyncio +async def test_real_playwright_navigate_click_type(): + """End-to-end check against real headless Chromium (skipped if unavailable).""" + pytest.importorskip("playwright.async_api") + + session_mod.reset_browser_session_manager() + manager = session_mod.get_browser_session_manager() + session = manager.get_session("it-thread", headless=True, timeout_ms=15000) + + html = "data:text/html,

Hi

" + try: + snap = await session.navigate(html) + assert any(el.tag == "input" for el in snap.elements) + button = next(el for el in snap.elements if "Go" in el.name) + after_click = await session.click(button.ref) + assert after_click.url # still a valid snapshot + + input_ref = next(el for el in snap.elements if el.tag == "input").ref + typed = await session.type_text(input_ref, "hello", submit=False) + assert typed.url + + text = await session.get_text() + assert "clicked" in text + + shot = await session.screenshot_bytes() + assert shot[:4] == b"\x89PNG" + finally: + await manager.close_session("it-thread") + session_mod.reset_browser_session_manager() + + +class _FakeRoute: + def __init__(self, url: str): + self.request = SimpleNamespace(url=url) + self.aborted_with: str | None = None + self.continued = False + + async def abort(self, error_code: str = "failed") -> None: + self.aborted_with = error_code + + async def continue_(self) -> None: + self.continued = True + + +@pytest.mark.asyncio +async def test_request_guard_aborts_blocked_redirect_target(): + """The context request guard aborts any URL the SSRF policy rejects. + + A public initial URL can 30x-redirect to a metadata/private host, and + Playwright follows it automatically; the per-request guard catches those + hops (and subresources/popups) that the one-time initial-URL check misses. + """ + captured: dict[str, object] = {} + + class _FakeContext: + async def route(self, pattern, handler): + captured["pattern"] = pattern + captured["handler"] = handler + + def guard(url: str) -> str | None: + return "Error: blocked" if "169.254.169.254" in url else None + + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + url_guard=guard, + ) + session._context = _FakeContext() + + await session._install_request_guard() + assert session._request_guard_bound is True + assert captured["pattern"] == "**/*" + handler = captured["handler"] + + blocked = _FakeRoute("http://169.254.169.254/latest/meta-data/") + await handler(blocked) + assert blocked.aborted_with == "blockedbyclient" + assert blocked.continued is False + + allowed = _FakeRoute("https://example.com/page") + await handler(allowed) + assert allowed.continued is True + assert allowed.aborted_with is None + + +@pytest.mark.asyncio +async def test_close_continues_when_browser_driver_is_already_disconnected(): + """Shutdown must clear every handle even if the driver died first.""" + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + ) + session._stop_screencast = AsyncMock() + context = MagicMock() + context.close = AsyncMock(side_effect=RuntimeError("driver disconnected")) + browser = MagicMock() + browser.close = AsyncMock(side_effect=RuntimeError("connection closed")) + playwright = MagicMock() + playwright.stop = AsyncMock() + session._context = context + session._browser = browser + session._playwright = playwright + + await session._close() + + context.close.assert_awaited_once() + browser.close.assert_awaited_once() + playwright.stop.assert_awaited_once() + assert session._context is None + assert session._browser is None + assert session._playwright is None + + +@pytest.mark.asyncio +async def test_request_guard_not_installed_for_cdp_sessions(): + """CDP-attached real Chrome owns its own context, so we don't route it.""" + + class _FakeContext: + def __init__(self): + self.routed = False + + async def route(self, pattern, handler): + self.routed = True + + session = BrowserSession( + MagicMock(), + headless=True, + timeout_ms=1000, + viewport={"width": 1000, "height": 500}, + cdp_url="http://127.0.0.1:9222", + url_guard=lambda _url: "Error: blocked", + ) + context = _FakeContext() + session._context = context + + await session._install_request_guard() + assert context.routed is False + assert session._request_guard_bound is False diff --git a/backend/tests/test_browser_router.py b/backend/tests/test_browser_router.py new file mode 100644 index 000000000..6c5f9791a --- /dev/null +++ b/backend/tests/test_browser_router.py @@ -0,0 +1,271 @@ +import logging +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest +from _router_auth_helpers import make_authed_test_app +from fastapi import FastAPI +from fastapi.testclient import TestClient +from starlette.websockets import WebSocketDisconnect + +from app.gateway.auth.models import User +from app.gateway.routers import browser as browser_router +from app.gateway.routers.browser import _should_apply_browser_seed, _ws_origin_allowed + + +class _FakeWebSocket: + """Minimal stand-in exposing only the headers ``_ws_origin_allowed`` reads.""" + + def __init__(self, headers: dict[str, str]): + self.headers = {k.lower(): v for k, v in headers.items()} + + +def _user(user_id: str = "browser-user") -> SimpleNamespace: + return SimpleNamespace(id=user_id) + + +def _browser_ws_app(thread_store=...): + app = FastAPI() + app.include_router(browser_router.router) + if thread_store is not ...: + app.state.thread_store = thread_store + return app + + +def _expect_ws_close(app: FastAPI, code: int, *, headers: dict[str, str] | None = None) -> None: + with TestClient(app) as client: + with pytest.raises(WebSocketDisconnect) as exc_info: + with client.websocket_connect("/api/threads/thread-1/browser/stream", headers=headers or {}): + pass + assert exc_info.value.code == code + + +def test_browser_stream_closes_4401_when_unauthenticated(): + app = _browser_ws_app() + with patch.object(browser_router, "_authenticate_ws", AsyncMock(return_value=None)): + _expect_ws_close(app, 4401) + + +def test_browser_stream_closes_4403_for_cross_origin_upgrade(): + app = _browser_ws_app() + with patch.object(browser_router, "_authenticate_ws", AsyncMock(return_value=_user())): + _expect_ws_close(app, 4403, headers={"origin": "https://evil.example.com"}) + + +def test_browser_stream_closes_4404_when_thread_store_missing(): + app = _browser_ws_app() + with patch.object(browser_router, "_authenticate_ws", AsyncMock(return_value=_user())): + _expect_ws_close(app, 4404) + + +def test_browser_stream_rejects_legacy_null_owner_thread(): + store = MagicMock() + store.check_access = AsyncMock(return_value=True) + store.get = AsyncMock(return_value={"thread_id": "thread-1", "user_id": None}) + app = _browser_ws_app(store) + with ( + patch.object(browser_router, "_authenticate_ws", AsyncMock(return_value=_user())), + patch.object(browser_router, "_browser_tools_enabled", return_value=False), + ): + _expect_ws_close(app, 4404) + store.get.assert_awaited_once_with("thread-1", user_id="browser-user") + + +def test_browser_stream_closes_4404_when_tools_disabled_for_owned_thread(): + store = MagicMock() + store.check_access = AsyncMock(return_value=True) + store.get = AsyncMock(return_value={"thread_id": "thread-1", "user_id": "browser-user"}) + app = _browser_ws_app(store) + with ( + patch.object(browser_router, "_authenticate_ws", AsyncMock(return_value=_user())), + patch.object(browser_router, "_browser_tools_enabled", return_value=False), + ): + _expect_ws_close(app, 4404) + + +def test_browser_stream_closes_4501_when_browser_runtime_unavailable(): + store = MagicMock() + store.check_access = AsyncMock(return_value=True) + store.get = AsyncMock(return_value={"thread_id": "thread-1", "user_id": "browser-user"}) + app = _browser_ws_app(store) + real_import = __import__ + + def fail_browser_runtime_import(name, globals=None, locals=None, fromlist=(), level=0): + if name == "deerflow.community.browser_automation" and "get_browser_session_manager" in fromlist: + raise ImportError("browser runtime unavailable") + return real_import(name, globals, locals, fromlist, level) + + with ( + patch.object(browser_router, "_authenticate_ws", AsyncMock(return_value=_user())), + patch.object(browser_router, "_browser_tools_enabled", return_value=True), + patch("builtins.__import__", side_effect=fail_browser_runtime_import), + ): + _expect_ws_close(app, 4501) + + +def test_browser_navigate_rejects_legacy_null_owner_thread(): + user = User(email="browser@example.com", password_hash="x", system_role="user") + app = make_authed_test_app(user_factory=lambda: user) + app.include_router(browser_router.router) + app.state.thread_store.get = AsyncMock(return_value={"thread_id": "thread-1", "user_id": None}) + + with ( + patch.object(browser_router, "_browser_tools_enabled", return_value=True), + patch("deerflow.community.browser_automation.navigate_and_capture", new=AsyncMock()), + ): + response = TestClient(app).post( + "/api/threads/thread-1/browser/navigate", + json={"url": "https://example.com"}, + ) + + assert response.status_code == 404 + app.state.thread_store.get.assert_awaited_once_with("thread-1", user_id=str(user.id)) + + +def test_browser_tools_disabled_when_cdp_risk_not_explicitly_accepted(): + tool_cfg = SimpleNamespace(name="browser_navigate", model_extra={"cdp_url": "http://127.0.0.1:9222"}) + app_config = SimpleNamespace(tools=[tool_cfg]) + + with ( + patch("deerflow.config.get_app_config", return_value=app_config), + patch("app.gateway.browser_capability.browser_multi_worker_error", return_value=None), + patch("app.gateway.browser_capability.importlib.util.find_spec", return_value=object()), + ): + assert browser_router._browser_tools_enabled() is False + + +def test_browser_tools_enabled_when_cdp_risk_explicitly_accepted(): + tool_cfg = SimpleNamespace( + name="browser_navigate", + model_extra={"cdp_url": "http://127.0.0.1:9222", "allow_unguarded_cdp": True}, + ) + app_config = SimpleNamespace(tools=[tool_cfg]) + + with ( + patch("deerflow.config.get_app_config", return_value=app_config), + patch("app.gateway.browser_capability.browser_multi_worker_error", return_value=None), + patch("app.gateway.browser_capability.importlib.util.find_spec", return_value=object()), + ): + assert browser_router._browser_tools_enabled() is True + + +def test_browser_navigate_redacts_failure_url_from_logs_and_response(caplog): + user = User(email="browser@example.com", password_hash="x", system_role="user") + app = make_authed_test_app(user_factory=lambda: user) + app.include_router(browser_router.router) + app.state.thread_store.get = AsyncMock(return_value={"thread_id": "thread-1", "user_id": str(user.id)}) + failing_url = "https://example.com/callback?code=secret#fragment" + + caplog.set_level(logging.ERROR, logger="app.gateway.routers.browser") + with ( + patch.object(browser_router, "_browser_tools_enabled", return_value=True), + patch( + "deerflow.community.browser_automation.navigate_and_capture", + new=AsyncMock(side_effect=RuntimeError(f"timed out opening {failing_url}")), + ), + ): + response = TestClient(app).post( + "/api/threads/thread-1/browser/navigate", + json={"url": failing_url}, + ) + + assert response.status_code == 502 + assert response.json() == {"detail": "Browser navigation failed"} + assert "https://example.com/callback" in caplog.text + assert "code=secret" not in caplog.text + assert "fragment" not in caplog.text + assert "secret" not in response.text + + +def test_browser_stream_seed_applies_to_blank_page(): + assert _should_apply_browser_seed("about:blank", "https://github.com/bytedance/deer-flow") + + +def test_browser_stream_seed_applies_when_current_url_differs(): + assert _should_apply_browser_seed( + "https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest", + "https://github.com/bytedance/deer-flow", + ) + + +def test_browser_stream_seed_ignores_hash_and_trailing_slash_for_same_page(): + assert not _should_apply_browser_seed( + "https://github.com/bytedance/deer-flow/#readme", + "https://github.com/bytedance/deer-flow/", + ) + + +def test_ws_origin_allowed_without_origin_header(): + # Native ws clients / tests do not send Origin — allow them. + assert _ws_origin_allowed(_FakeWebSocket({"host": "app.example.com"})) is True + + +def test_ws_origin_allowed_same_origin_host(): + # Browser page scheme (https) differs from the ws scheme, so same-origin + # compares host[:port] against the upgrade target Host. + ws = _FakeWebSocket({"origin": "https://app.example.com", "host": "app.example.com"}) + assert _ws_origin_allowed(ws) is True + + +def test_ws_origin_allowed_rejects_cross_origin(): + ws = _FakeWebSocket({"origin": "https://evil.example.com", "host": "app.example.com"}) + assert _ws_origin_allowed(ws) is False + + +def test_ws_origin_allowed_rejects_malformed_origin(): + ws = _FakeWebSocket({"origin": "not-a-url", "host": "app.example.com"}) + assert _ws_origin_allowed(ws) is False + + +def test_ws_origin_allowed_honors_configured_cors_origin(monkeypatch): + monkeypatch.setenv("GATEWAY_CORS_ORIGINS", "https://console.example.com") + ws = _FakeWebSocket({"origin": "https://console.example.com", "host": "gateway.internal"}) + assert _ws_origin_allowed(ws) is True + + +def test_ws_origin_allowed_honors_forwarded_host(): + # Behind a proxy the real upgrade target is X-Forwarded-Host, not Host. + ws = _FakeWebSocket( + { + "origin": "https://app.example.com", + "host": "gateway.internal", + "x-forwarded-host": "app.example.com", + }, + ) + assert _ws_origin_allowed(ws) is True + + +def test_browser_frames_dirname_shared_between_tools_and_scanner(): + """The screenshots dir name must stay identical in the writer and the scanner. + + Both sides import the single ``BROWSER_FRAMES_DIRNAME`` constant; this locks + that they resolve to the same value so the workspace-changes ignore cannot + silently drift away from where the browser tools write frames. + """ + from deerflow.community.browser_automation import tools as browser_tools + from deerflow.constants import BROWSER_FRAMES_DIRNAME + from deerflow.workspace_changes.scanner import EXCLUDED_DIR_NAMES + + assert browser_tools._BROWSER_FRAMES_DIRNAME == BROWSER_FRAMES_DIRNAME + assert BROWSER_FRAMES_DIRNAME in EXCLUDED_DIR_NAMES + + +def test_validate_browser_url_rejects_private_and_non_http(monkeypatch): + """WS seed / navigate events reuse the same SSRF policy as the agent tools. + + With no ``allow_private_addresses`` override the shared validator must reject + loopback / metadata / non-http targets, so the live stream cannot be steered + at internal infrastructure. + """ + from deerflow.community.browser_automation import tools as browser_tools + from deerflow.community.browser_automation import validate_browser_url + + # Isolate from any local config.yaml that may set allow_private_addresses. + monkeypatch.setattr(browser_tools, "_get_tool_config", lambda _tool_name: {}) + + assert validate_browser_url("http://169.254.169.254/latest/meta-data/") is not None + assert validate_browser_url("http://127.0.0.1:8001/") is not None + assert validate_browser_url("file:///etc/passwd") is not None + assert validate_browser_url("ftp://example.com") is not None + # A normal public URL passes (returns None = allowed). + assert validate_browser_url("https://github.com/bytedance/deer-flow") is None diff --git a/backend/tests/test_detect_uv_extras.py b/backend/tests/test_detect_uv_extras.py index 51cb9b1a8..3e1ee3c39 100644 --- a/backend/tests/test_detect_uv_extras.py +++ b/backend/tests/test_detect_uv_extras.py @@ -149,6 +149,22 @@ def test_detect_from_config_redis_via_stream_bridge(tmp_path): assert detect.detect_from_config(cfg) == ["redis"] +def test_detect_from_config_browser_via_browser_navigate_tool(tmp_path): + cfg = tmp_path / "config.yaml" + cfg.write_text( + "tools:\n - name: browser_navigate\n group: browser\n use: deerflow.community.browser_automation.tools:browser_navigate_tool\n", + ) + assert detect.detect_from_config(cfg) == ["browser"] + + +def test_detect_from_config_ignores_commented_browser_tool(tmp_path): + cfg = tmp_path / "config.yaml" + cfg.write_text( + "tools:\n # - name: browser_navigate\n # group: browser\n - name: web_fetch\n group: web\n", + ) + assert detect.detect_from_config(cfg) == [] + + def test_detect_from_config_memory_stream_bridge_returns_no_extras(tmp_path): cfg = tmp_path / "config.yaml" cfg.write_text("stream_bridge:\n type: memory\n queue_maxsize: 256\n") diff --git a/backend/tests/test_features_router.py b/backend/tests/test_features_router.py index e71dafbd1..5deb5353d 100644 --- a/backend/tests/test_features_router.py +++ b/backend/tests/test_features_router.py @@ -1,4 +1,5 @@ from types import SimpleNamespace +from unittest.mock import patch from fastapi import FastAPI from fastapi.testclient import TestClient @@ -7,10 +8,17 @@ from app.gateway.deps import get_config from app.gateway.routers import features -def _app_with_config(*, agents_api_enabled: bool) -> FastAPI: +def _app_with_config(*, agents_api_enabled: bool, browser_enabled: bool = False, browser_extra: dict | None = None) -> FastAPI: app = FastAPI() app.include_router(features.router) - fake_config = SimpleNamespace(agents_api=SimpleNamespace(enabled=agents_api_enabled)) + tools = ( + [ + SimpleNamespace(name="browser_navigate", model_extra=browser_extra or {}), + ] + if browser_enabled + else [] + ) + fake_config = SimpleNamespace(agents_api=SimpleNamespace(enabled=agents_api_enabled), tools=tools) app.dependency_overrides[get_config] = lambda: fake_config return app @@ -19,11 +27,47 @@ def test_features_reports_agents_api_enabled() -> None: with TestClient(_app_with_config(agents_api_enabled=True)) as client: response = client.get("/api/features") assert response.status_code == 200 - assert response.json() == {"agents_api": {"enabled": True}} + assert response.json() == {"agents_api": {"enabled": True}, "browser_control": {"enabled": False}} def test_features_reports_agents_api_disabled() -> None: with TestClient(_app_with_config(agents_api_enabled=False)) as client: response = client.get("/api/features") assert response.status_code == 200 - assert response.json() == {"agents_api": {"enabled": False}} + assert response.json() == {"agents_api": {"enabled": False}, "browser_control": {"enabled": False}} + + +def test_features_reports_browser_control_enabled_when_configured_and_runtime_available() -> None: + with ( + patch("app.gateway.browser_capability.importlib.util.find_spec", return_value=object()), + TestClient(_app_with_config(agents_api_enabled=True, browser_enabled=True)) as client, + ): + response = client.get("/api/features") + assert response.status_code == 200 + assert response.json()["browser_control"] == {"enabled": True} + + +def test_features_reports_browser_control_disabled_when_runtime_missing() -> None: + with ( + patch("app.gateway.browser_capability.importlib.util.find_spec", return_value=None), + TestClient(_app_with_config(agents_api_enabled=True, browser_enabled=True)) as client, + ): + response = client.get("/api/features") + assert response.status_code == 200 + assert response.json()["browser_control"] == {"enabled": False} + + +def test_features_reports_browser_control_disabled_for_unguarded_cdp() -> None: + with ( + patch("app.gateway.browser_capability.importlib.util.find_spec", return_value=object()), + TestClient( + _app_with_config( + agents_api_enabled=True, + browser_enabled=True, + browser_extra={"cdp_url": "http://127.0.0.1:9222"}, + ), + ) as client, + ): + response = client.get("/api/features") + assert response.status_code == 200 + assert response.json()["browser_control"] == {"enabled": False} diff --git a/backend/tests/test_lead_agent_skills.py b/backend/tests/test_lead_agent_skills.py index b78d6b789..7faec55c1 100644 --- a/backend/tests/test_lead_agent_skills.py +++ b/backend/tests/test_lead_agent_skills.py @@ -429,6 +429,47 @@ def test_make_lead_agent_passive_empty_skill_policy_preserves_mcp_and_other_tool assert captured_deferred_setups[0].deferred_names == frozenset({"lightrag_query"}) +def test_default_lead_agent_does_not_apply_installed_skill_allowlists(monkeypatch): + """Installed skills are discoverable but not active for ordinary default chat. + + A public skill with ``allowed-tools`` must not globally hide configured + tools like ``browser_navigate`` before the user has selected a specific + skill-owned workflow. + """ + from unittest.mock import MagicMock + + from deerflow.agents.lead_agent import agent as lead_agent_module + + monkeypatch.setattr(lead_agent_module, "_resolve_model_name", lambda x=None, **kwargs: "default-model") + monkeypatch.setattr(lead_agent_module, "create_chat_model", lambda **kwargs: "model") + monkeypatch.setattr(lead_agent_module, "build_middlewares", lambda *args, **kwargs: []) + monkeypatch.setattr(lead_agent_module, "apply_prompt_template", lambda **kwargs: "mock_prompt") + monkeypatch.setattr(lead_agent_module, "create_agent", lambda **kwargs: kwargs) + monkeypatch.setattr( + lead_agent_module, + "_load_enabled_available_skills", + lambda available_skills, *, app_config, user_id=None: [_make_skill("skill-reviewer", ["review_skill_package"])], + ) + monkeypatch.setattr( + "deerflow.tools.get_available_tools", + lambda **kwargs: [NamedTool("bash"), NamedTool("browser_navigate"), NamedTool("review_skill_package")], + ) + + mock_app_config = MagicMock() + mock_app_config.get_model_config.return_value = SimpleNamespace(supports_thinking=False, supports_vision=False) + mock_app_config.tool_search.enabled = True + mock_app_config.skills.container_path = "/mnt/skills" + mock_app_config.skills.deferred_discovery = True + monkeypatch.setattr(lead_agent_module, "get_app_config", lambda: mock_app_config) + + agent_kwargs = lead_agent_module.make_lead_agent({"configurable": {}}) + + tool_names = [tool.name for tool in agent_kwargs["tools"]] + assert "browser_navigate" in tool_names + assert "bash" in tool_names + assert "describe_skill" in tool_names + + def test_make_lead_agent_fails_closed_when_skill_policy_load_fails(monkeypatch): from unittest.mock import MagicMock diff --git a/backend/tests/test_multi_worker_postgres_gate.py b/backend/tests/test_multi_worker_postgres_gate.py index 0a059ba6f..ced8298d1 100644 --- a/backend/tests/test_multi_worker_postgres_gate.py +++ b/backend/tests/test_multi_worker_postgres_gate.py @@ -19,13 +19,15 @@ import pytest from fastapi import FastAPI from app.gateway.deps import _enforce_postgres_for_multi_worker, langgraph_runtime +from app.gateway.routers.browser import _browser_tools_enabled from deerflow.config.database_config import DatabaseConfig from deerflow.config.run_ownership_config import RunOwnershipConfig -def _config_with_backend(backend: str, *, heartbeat_enabled: bool | None = None) -> SimpleNamespace: +def _config_with_backend(backend: str, *, heartbeat_enabled: bool | None = None, browser_enabled: bool = False) -> SimpleNamespace: run_ownership = RunOwnershipConfig(heartbeat_enabled=heartbeat_enabled) if heartbeat_enabled is not None else None - return SimpleNamespace(database=DatabaseConfig(backend=backend), run_ownership=run_ownership) + tools = [SimpleNamespace(name="browser_navigate")] if browser_enabled else [] + return SimpleNamespace(database=DatabaseConfig(backend=backend), run_ownership=run_ownership, tools=tools) # --------------------------------------------------------------------------- @@ -52,6 +54,25 @@ def test_gate_allows_multi_worker_with_postgres_and_heartbeat(monkeypatch): _enforce_postgres_for_multi_worker(_config_with_backend("postgres", heartbeat_enabled=True)) +def test_gate_rejects_process_local_browser_with_multi_worker(monkeypatch): + monkeypatch.setenv("GATEWAY_WORKERS", "2") + with pytest.raises(SystemExit) as exc_info: + _enforce_postgres_for_multi_worker( + _config_with_backend("postgres", heartbeat_enabled=True, browser_enabled=True), + ) + msg = str(exc_info.value) + assert "process-local" in msg + assert "GATEWAY_WORKERS=1" in msg + + +def test_runtime_browser_surface_stays_disabled_after_incompatible_hot_reload(monkeypatch): + monkeypatch.setenv("GATEWAY_WORKERS", "2") + live_config = SimpleNamespace(tools=[SimpleNamespace(name="browser_navigate", model_extra={})]) + + with patch("deerflow.config.get_app_config", return_value=live_config): + assert _browser_tools_enabled() is False + + def test_gate_rejects_multi_worker_with_sqlite(monkeypatch): monkeypatch.setenv("GATEWAY_WORKERS", "2") with pytest.raises(SystemExit) as exc_info: diff --git a/backend/tests/test_threads_router.py b/backend/tests/test_threads_router.py index 3823db3f3..63ddbd211 100644 --- a/backend/tests/test_threads_router.py +++ b/backend/tests/test_threads_router.py @@ -1,7 +1,7 @@ import asyncio import re from types import SimpleNamespace -from unittest.mock import patch +from unittest.mock import AsyncMock, patch import pytest from _router_auth_helpers import make_authed_test_app @@ -158,6 +158,29 @@ def test_delete_thread_route_cleans_thread_directory(tmp_path): assert not thread_dir.exists() +def test_delete_thread_route_closes_browser_session(tmp_path): + """Deleting a thread tears down its live browser session so a later caller + who reuses the id cannot inherit the retained page/cookies.""" + paths = Paths(tmp_path) + + app = make_authed_test_app() + app.include_router(threads.router) + + manager = SimpleNamespace(close_session=AsyncMock(return_value=True)) + with ( + patch("app.gateway.routers.threads.get_paths", return_value=paths), + patch( + "deerflow.community.browser_automation.get_browser_session_manager", + return_value=manager, + ), + ): + with TestClient(app) as client: + response = client.delete("/api/threads/thread-browser") + + assert response.status_code == 200 + manager.close_session.assert_awaited_once_with("thread-browser") + + def test_delete_thread_route_rejects_invalid_thread_id(tmp_path): paths = Paths(tmp_path) diff --git a/backend/tests/test_workspace_changes.py b/backend/tests/test_workspace_changes.py index be63c8ba3..7648686b6 100644 --- a/backend/tests/test_workspace_changes.py +++ b/backend/tests/test_workspace_changes.py @@ -198,6 +198,20 @@ def test_scan_workspace_roots_skips_excluded_directories(tmp_path): assert "/mnt/user-data/workspace/node_modules/ignored.js" not in snapshot.files +def test_scan_workspace_roots_skips_browser_frames(tmp_path): + roots = _roots(tmp_path) + outputs = roots[1].host_path + (outputs / "report.md").write_text("keep", encoding="utf-8") + frames = outputs / ".browser-frames" + frames.mkdir() + (frames / "browser-navigate-1.png").write_bytes(b"\x89PNG\r\n\x1a\nshot") + + snapshot = scan_workspace_roots(roots) + + assert "/mnt/user-data/outputs/report.md" in snapshot.files + assert "/mnt/user-data/outputs/.browser-frames/browser-navigate-1.png" not in snapshot.files + + def test_scan_workspace_roots_can_skip_text_loading(tmp_path): roots = _roots(tmp_path) workspace = roots[0].host_path diff --git a/backend/uv.lock b/backend/uv.lock index 7249eacca..4579c4eeb 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -796,6 +796,9 @@ dependencies = [ ] [package.optional-dependencies] +browser = [ + { name = "deerflow-harness", extra = ["browser"] }, +] discord = [ { name = "discord-py" }, ] @@ -826,6 +829,7 @@ dev = [ requires-dist = [ { name = "bcrypt", specifier = ">=4.0.0" }, { name = "deerflow-harness", editable = "packages/harness" }, + { name = "deerflow-harness", extras = ["browser"], marker = "extra == 'browser'", editable = "packages/harness" }, { name = "deerflow-harness", extras = ["monocle"], marker = "extra == 'monocle'", editable = "packages/harness" }, { name = "deerflow-harness", extras = ["postgres"], marker = "extra == 'postgres'", editable = "packages/harness" }, { name = "deerflow-harness", extras = ["redis"], marker = "extra == 'redis'", editable = "packages/harness" }, @@ -846,7 +850,7 @@ requires-dist = [ { name = "uvicorn", extras = ["standard"], specifier = ">=0.34.0" }, { name = "wecom-aibot-python-sdk", specifier = ">=0.1.6" }, ] -provides-extras = ["postgres", "redis", "discord", "monocle"] +provides-extras = ["postgres", "redis", "discord", "monocle", "browser"] [package.metadata.requires-dev] dev = [ @@ -907,6 +911,9 @@ dependencies = [ boxlite = [ { name = "boxlite" }, ] +browser = [ + { name = "playwright" }, +] monocle = [ { name = "monocle-apptrace" }, ] @@ -965,6 +972,7 @@ requires-dist = [ { name = "markdownify", specifier = ">=1.2.2" }, { name = "markitdown", extras = ["all", "xlsx"], specifier = ">=0.0.1a2" }, { name = "monocle-apptrace", marker = "extra == 'monocle'", specifier = ">=0.8.8" }, + { name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" }, { name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.3.3" }, { name = "psycopg-pool", marker = "extra == 'postgres'", specifier = ">=3.3.0" }, { name = "pydantic", specifier = ">=2.12.5" }, @@ -977,7 +985,7 @@ requires-dist = [ { name = "textual", marker = "extra == 'tui'", specifier = ">=0.80" }, { name = "tiktoken", specifier = ">=0.8.0" }, ] -provides-extras = ["tui", "groundroute", "ollama", "postgres", "redis", "pymupdf", "boxlite", "monocle"] +provides-extras = ["tui", "groundroute", "ollama", "postgres", "redis", "pymupdf", "boxlite", "monocle", "browser"] [[package]] name = "defusedxml" @@ -3220,6 +3228,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" }, ] +[[package]] +name = "playwright" +version = "1.61.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "greenlet" }, + { name = "pyee" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/44/ee/31e4e0db36588b817a10b299a0285082545fde7d36543c2abe498bb3d61a/playwright-1.61.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:ff138c3a604f69911e9d42fd036e55c2a171e5616edf04c1e7f60a2a285540b0", size = 43421877, upload-time = "2026-06-29T10:32:48.428Z" }, + { url = "https://files.pythonhosted.org/packages/42/35/71395dd3ecc798965be4a3ef8c443217d4abca168e7cb34536304f9489e6/playwright-1.61.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:009588c2a7e499bc5a8b425b61fa65490968bbda9cd69e0cf2cff10f8304659a", size = 42205016, upload-time = "2026-06-29T10:32:52.104Z" }, + { url = "https://files.pythonhosted.org/packages/f4/44/323164cf5cd1647bdefce76ffce27651aadb959d089b48f53ea40918276e/playwright-1.61.0-py3-none-macosx_11_0_universal2.whl", hash = "sha256:9f7de4536088d12037c13a52b7ea34b59270b78926bb56935070597ffac6b1af", size = 43421884, upload-time = "2026-06-29T10:32:55.773Z" }, + { url = "https://files.pythonhosted.org/packages/ab/f8/a35bf179e4ba2522c1893635094a64e407572547bd61528820fc0abc87fe/playwright-1.61.0-py3-none-manylinux1_x86_64.whl", hash = "sha256:54f3b39f6eab832e33458c1dd7da0b5682aedab3b09ae731b5c59fa12fd2024e", size = 47421381, upload-time = "2026-06-29T10:32:59.903Z" }, + { url = "https://files.pythonhosted.org/packages/b7/eb/e3f922348ec17c315f98c463f72faa1181a1c3de0bfe31a8d2edf6561723/playwright-1.61.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93454322ade8c11d5d6c211bfd91bdfb9ffb4810e3e026371bcbc4bec1b7ee4c", size = 47120545, upload-time = "2026-06-29T10:33:03.574Z" }, + { url = "https://files.pythonhosted.org/packages/c2/a6/5be4e52b40a9c0c8a073e7c5b0785c05cf5a9ea8f8a7b5b260e32d970342/playwright-1.61.0-py3-none-win32.whl", hash = "sha256:372d55a6f1248fa1dd47599686980cb8fb5bbe6fcda59eab793eb657c11d8a9b", size = 37844841, upload-time = "2026-06-29T10:33:07.361Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fd/2b78036e5fbe9d5f5645bbe08a1eac7160c51243c0093963edbcf67c35d9/playwright-1.61.0-py3-none-win_amd64.whl", hash = "sha256:35c6cc4589a5d00964a59d7b3e59641e0aac0c02f15479a7af77d20f6bc79597", size = 37844846, upload-time = "2026-06-29T10:33:10.637Z" }, + { url = "https://files.pythonhosted.org/packages/27/0d/1b0f3c4ee4eb0514bc805b5c2f9a223e5b6de4f11a926f5235d51d0fc81b/playwright-1.61.0-py3-none-win_arm64.whl", hash = "sha256:e9fcbffcf557a8620fdedd92491eb59a32d18e23d6f3b4f6214b952be324fe51", size = 33955127, upload-time = "2026-06-29T10:33:14.008Z" }, +] + [[package]] name = "pluggy" version = "1.6.0" diff --git a/config.example.yaml b/config.example.yaml index a61678b6e..078fc0035 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -648,6 +648,7 @@ tool_groups: - name: file:read - name: file:write - name: bash + - name: browser # ============================================================================ # Tools Configuration @@ -801,6 +802,65 @@ tools: # # allow_private_addresses: false # SSRF guard: keep false in production. Set true ONLY to # # # capture internal/private targets (loopback, RFC1918, etc.) + # Agentic browser control (stateful navigate → observe → click/type loop). + # Unlike web_fetch/web_capture (read-only, stateless), these tools keep a live + # per-thread Playwright browser so the agent can interact with JavaScript-heavy + # pages, fill forms, and follow multi-step flows. Elements are addressed by the + # numeric [ref] index returned in each page snapshot. + # + # Requires the optional dependency, installed once per environment: + # cd backend && uv sync --extra browser && uv run playwright install chromium + # make dev / Docker startup auto-detect an active browser_navigate entry and + # preserve the browser extra during dependency sync. + # + # Uncomment the whole block to enable. Launch and URL-policy settings are read + # canonically from browser_navigate and shared by every browser tool + Live. + # - name: browser_navigate + # group: browser + # use: deerflow.community.browser_automation.tools:browser_navigate_tool + # headless: true # Set false only for local, trusted debugging + # timeout_ms: 30000 # Per-action navigation/interaction timeout + # viewport_width: 1280 + # viewport_height: 720 + # # allow_private_addresses: false # SSRF guard: keep false in production + # # cdp_url: http://127.0.0.1:9222 # Attach to YOUR running Chrome (Codex-style) + # # allow_unguarded_cdp: false # REQUIRED true with cdp_url: attaching to an + # # # existing Chrome prevents DeerFlow from + # # # enforcing its subresource/redirect SSRF guard. + # # # instead of launching a private headless one. + # # # Start Chrome with --remote-debugging-port=9222 + # # # so you watch the agent drive your real browser + # # # with your real login sessions. Local/trusted only; + # # # never enable for an untrusted CDP endpoint. + # # # Chrome must be version-matched to the bundled + # # # Playwright; a much newer Chrome can reject the + # # # CDP handshake. If so, use headless: false to launch + # # # Playwright's own visible browser instead. + # # Multi-worker note: browser sessions live in one Gateway worker's memory. + # # Keep GATEWAY_WORKERS=1 while this tool group is enabled. + # - name: browser_snapshot + # group: browser + # use: deerflow.community.browser_automation.tools:browser_snapshot_tool + # - name: browser_click + # group: browser + # use: deerflow.community.browser_automation.tools:browser_click_tool + # - name: browser_type + # group: browser + # use: deerflow.community.browser_automation.tools:browser_type_tool + # - name: browser_get_text + # group: browser + # use: deerflow.community.browser_automation.tools:browser_get_text_tool + # max_chars: 8000 # Truncation cap for browser_get_text output + # - name: browser_back + # group: browser + # use: deerflow.community.browser_automation.tools:browser_back_tool + # - name: browser_screenshot + # group: browser + # use: deerflow.community.browser_automation.tools:browser_screenshot_tool + # - name: browser_close + # group: browser + # use: deerflow.community.browser_automation.tools:browser_close_tool + # Web fetch tool (uses Exa) # NOTE: Only one web_fetch provider can be active at a time. # Comment out the Jina AI web_fetch entry below before enabling this one. diff --git a/docker/nginx/nginx.conf b/docker/nginx/nginx.conf index 9ce9437b5..d20765e51 100644 --- a/docker/nginx/nginx.conf +++ b/docker/nginx/nginx.conf @@ -168,6 +168,26 @@ http { # Disable response buffering to avoid permission errors } + # Live browser stream is a WebSocket upgrade. It must be matched before + # the generic /api/threads regex below (which omits Upgrade/Connection + # forwarding and would downgrade it to plain HTTP). + location ~ ^/api/threads/[^/]+/browser/stream { + proxy_pass http://$gateway_upstream; + proxy_http_version 1.1; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $forwarded_proto; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection 'upgrade'; + proxy_cache_bypass $http_upgrade; + + # Long-lived Live stream: keep the upgraded connection open. + proxy_connect_timeout 600s; + proxy_send_timeout 600s; + proxy_read_timeout 600s; + } + # Custom API: Other endpoints under /api/threads location ~ ^/api/threads { proxy_pass http://$gateway_upstream; diff --git a/docker/nginx/nginx.local.conf b/docker/nginx/nginx.local.conf index c0ae52647..47b2860dd 100644 --- a/docker/nginx/nginx.local.conf +++ b/docker/nginx/nginx.local.conf @@ -171,6 +171,28 @@ http { proxy_cache off; } + # Live browser stream is a WebSocket upgrade. It must be matched before + # the generic /api/threads regex below (which omits Upgrade/Connection + # forwarding and would downgrade it to plain HTTP). + location ~ ^/api/threads/[^/]+/browser/stream { + proxy_pass http://gateway; + proxy_http_version 1.1; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection 'upgrade'; + proxy_cache_bypass $http_upgrade; + + # Long-lived Live stream: keep the upgraded connection open. + proxy_buffering off; + proxy_cache off; + proxy_connect_timeout 600s; + proxy_send_timeout 600s; + proxy_read_timeout 600s; + } + # Custom API: Other endpoints under /api/threads location ~ ^/api/threads { proxy_pass http://gateway; diff --git a/docs/plans/2026-07-10-pluggable-authorization-rfc.md b/docs/plans/2026-07-10-pluggable-authorization-rfc.md index 93336bbd6..7299efe78 100644 --- a/docs/plans/2026-07-10-pluggable-authorization-rfc.md +++ b/docs/plans/2026-07-10-pluggable-authorization-rfc.md @@ -529,5 +529,3 @@ A search of `bytedance/deer-flow` confirms **no RBAC implementation exists**; RB - **[#3929](https://github.com/bytedance/deer-flow/issues/3929)** (open) - sandbox NodePort->ClusterIP (same author family of security hardening RFCs). **Net takeaway:** the upstream has spent real effort plumbing identity into the guardrail execution point (#3665, #3839) and has explicitly deferred the RBAC policy brain. The two-layer design in this RFC is the natural next step the prior work points at - not a competing or redundant proposal. - - diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index 48b96a9a6..07babdb86 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -93,8 +93,10 @@ Tool-calling AI messages can contain user-visible text as well as `tool_calls`. - `src/app/workspace/chats/[thread_id]/page.tsx` owns composer busy-state wiring. - `src/app/workspace/chats/[thread_id]/page.tsx` owns branch-from-turn submission and navigation; sidecar `MessageList` instances do not receive the branch action. +- `src/app/workspace/chats/[thread_id]/page.tsx` gates the Workspace Browser trigger and browser right panel on `/api/features -> browser_control.enabled`; default/failed feature discovery hides the browser control so optional backend installs do not show a dead Live socket. - `src/app/workspace/chats/[thread_id]/page.tsx` and `src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx` own active-goal display state for their composer overlays. - `src/components/workspace/messages/message-list.tsx` owns human-input card answered/latest/pending gating; entry pages only translate a submitted card response into `sendMessage` calls. +- `src/components/workspace/browser-view/browser-view-panel.tsx` forwards each physical pointer click as one `click` input; do not also emit `down`/`up` for the same gesture because the remote Playwright click would run twice. - `src/core/threads/hooks.ts` owns pre-submit upload state and thread submission. ## Code Style diff --git a/frontend/src/app/workspace/chats/[thread_id]/page.tsx b/frontend/src/app/workspace/chats/[thread_id]/page.tsx index 5e8130ac8..e78218a20 100644 --- a/frontend/src/app/workspace/chats/[thread_id]/page.tsx +++ b/frontend/src/app/workspace/chats/[thread_id]/page.tsx @@ -7,6 +7,7 @@ import { toast } from "sonner"; import { type PromptInputMessage } from "@/components/ai-elements/prompt-input"; import { SidebarTrigger } from "@/components/ui/sidebar"; import { ArtifactTrigger } from "@/components/workspace/artifacts"; +import { BrowserTrigger } from "@/components/workspace/browser-view"; import { ChatBox, useSpecificChatMode, @@ -33,6 +34,7 @@ import { TodoList } from "@/components/workspace/todo-list"; import { TokenUsageIndicator } from "@/components/workspace/token-usage-indicator"; import { useActiveGoal } from "@/components/workspace/use-active-goal"; import { Welcome } from "@/components/workspace/welcome"; +import { useBrowserControlEnabled } from "@/core/features"; import { useI18n } from "@/core/i18n/hooks"; import { buildHumanInputResponseText, @@ -68,6 +70,7 @@ export default function ChatPage() { const [isWelcomeMode, setIsWelcomeMode] = useState(isNewThread); const [settings, setSettings] = useThreadSettings(threadId); const [localSettings, setLocalSettings] = useLocalSettings(); + const { enabled: browserControlEnabled } = useBrowserControlEnabled(); const { tokenUsageEnabled } = useModels(); const threadTokenUsage = useThreadTokenUsage( isNewThread || isMock ? undefined : threadId, @@ -240,6 +243,7 @@ export default function ChatPage() { ? localSettings.tokenUsage.inlineMode : "off"; const hasTodos = (thread.values.todos?.length ?? 0) > 0; + const browserEnabled = !isNewThread && browserControlEnabled; const { activeGoal, hasGoal, setLocalGoal } = useActiveGoal( threadId, thread.values.goal, @@ -260,7 +264,7 @@ export default function ChatPage() { context={settings.context} isMock={isMock} > - +
+ {browserEnabled && }
diff --git a/frontend/src/app/workspace/chats/[thread_id]/providers.tsx b/frontend/src/app/workspace/chats/[thread_id]/providers.tsx index 46d4a4cef..ac7eabd54 100644 --- a/frontend/src/app/workspace/chats/[thread_id]/providers.tsx +++ b/frontend/src/app/workspace/chats/[thread_id]/providers.tsx @@ -2,13 +2,16 @@ import { PromptInputProvider } from "@/components/ai-elements/prompt-input"; import { ArtifactsProvider } from "@/components/workspace/artifacts"; +import { BrowserViewProvider } from "@/components/workspace/browser-view"; import { SubtasksProvider } from "@/core/tasks/context"; export function ChatProviders({ children }: { children: React.ReactNode }) { return ( - {children} + + {children} + ); diff --git a/frontend/src/components/workspace/browser-view/api.ts b/frontend/src/components/workspace/browser-view/api.ts new file mode 100644 index 000000000..f9c5eb7e1 --- /dev/null +++ b/frontend/src/components/workspace/browser-view/api.ts @@ -0,0 +1,55 @@ +import { throwGatewayApiError } from "@/core/api/errors"; +import { fetch } from "@/core/api/fetcher"; +import { getBackendBaseURL } from "@/core/config"; + +export interface BrowserNavigateResult { + screenshot: string | null; + url: string; + title: string; +} + +export async function navigateBrowser( + threadId: string, + url: string, +): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/threads/${encodeURIComponent(threadId)}/browser/navigate`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ url }), + }, + ); + if (!response.ok) { + await throwGatewayApiError( + response, + `Failed to navigate browser: ${response.statusText}`, + ); + } + return response.json(); +} + +/** + * Build the WebSocket URL for the live browser stream. + * + * Uses the configured backend base URL when present (split-origin dev/prod), + * otherwise falls back to the current same-origin host (nginx proxies the + * upgrade in the unified deployment). + * + * When ``seedUrl`` is provided, the server aligns the live page to it when the + * current page is blank or points at a different URL — so reconnecting to a + * stale session lands on the page the user expects instead of a white + * about:blank screen or a leftover page. The seed is SSRF-screened server-side. + */ +export function browserStreamURL(threadId: string, seedUrl?: string): string { + const base = getBackendBaseURL(); + const origin = + base && base.length > 0 + ? base + : typeof window !== "undefined" + ? window.location.origin + : ""; + const wsOrigin = origin.replace(/^http/i, "ws"); + const query = seedUrl ? `?seed=${encodeURIComponent(seedUrl)}` : ""; + return `${wsOrigin}/api/threads/${encodeURIComponent(threadId)}/browser/stream${query}`; +} diff --git a/frontend/src/components/workspace/browser-view/browser-trigger.tsx b/frontend/src/components/workspace/browser-view/browser-trigger.tsx new file mode 100644 index 000000000..a51cf5992 --- /dev/null +++ b/frontend/src/components/workspace/browser-view/browser-trigger.tsx @@ -0,0 +1,44 @@ +import { MonitorIcon } from "lucide-react"; + +import { Button } from "@/components/ui/button"; +import { Tooltip } from "@/components/workspace/tooltip"; +import { useI18n } from "@/core/i18n/hooks"; + +import { useMaybeSidecar } from "../sidecar/context"; + +import { useMaybeBrowserView } from "./context"; + +export const BrowserTrigger = () => { + const { t } = useI18n(); + const browserView = useMaybeBrowserView(); + const sidecar = useMaybeSidecar(); + + if (!browserView) { + return null; + } + const browserVisible = browserView.open && !sidecar?.open; + const label = browserVisible ? t.common.close : t.common.showBrowser; + + return ( + + + + ); +}; diff --git a/frontend/src/components/workspace/browser-view/browser-view-panel.tsx b/frontend/src/components/workspace/browser-view/browser-view-panel.tsx new file mode 100644 index 000000000..34631434c --- /dev/null +++ b/frontend/src/components/workspace/browser-view/browser-view-panel.tsx @@ -0,0 +1,462 @@ +"use client"; + +import { + ArrowLeftIcon, + ArrowRightIcon, + GlobeIcon, + Loader2Icon, + MonitorIcon, + RadioIcon, + XIcon, +} from "lucide-react"; +import { useCallback, useEffect, useRef, useState } from "react"; +import { toast } from "sonner"; + +import { ConversationEmptyState } from "@/components/ai-elements/conversation"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { resolveArtifactURL } from "@/core/artifacts/utils"; +import { isIMEComposing } from "@/lib/ime"; +import { cn } from "@/lib/utils"; + +import { navigateBrowser } from "./api"; +import { useMaybeBrowserView } from "./context"; +import { decideBrowserKeyInput } from "./keyboard"; +import { type BrowserInputEvent, useBrowserStream } from "./use-browser-stream"; + +export function BrowserViewPanel({ + threadId, + className, +}: { + threadId: string; + className?: string; +}) { + const browserView = useMaybeBrowserView(); + const frame = browserView?.latestFrame ?? null; + const imageUrl = frame + ? resolveArtifactURL(frame.screenshot, threadId) + : null; + + const [urlInput, setUrlInput] = useState(""); + const [navigating, setNavigating] = useState(false); + const [live, setLive] = useState(true); + const [lastLiveUrl, setLastLiveUrl] = useState(null); + const [liveFallback, setLiveFallback] = useState<{ + frameUrl: string; + url?: string; + } | null>(null); + + const streamSeedUrl = lastLiveUrl ?? liveFallback?.url ?? frame?.url; + const handleNavRejected = useCallback( + (url: string | undefined, message: string | undefined) => { + setNavigating(false); + toast.error( + message?.replace(/^Error:\s*/i, "") ?? + `Cannot open ${url ?? "that URL"}`, + ); + }, + [], + ); + const { status, frameUrl, liveUrl, sendInput } = useBrowserStream( + threadId, + live, + streamSeedUrl, + handleNavRejected, + ); + const panelRef = useRef(null); + const surfaceRef = useRef(null); + const stageRef = useRef(null); + const staticScreenshotRef = useRef(null); + // True while the user is actively editing the URL bar, so background live-URL + // reports do not clobber a half-typed address the user has not submitted yet. + const urlEditingRef = useRef(false); + // Handle for the live-navigate spinner timeout so it can be cleared on + // unmount / re-navigation instead of firing setState after teardown. + const navSpinnerTimerRef = useRef(null); + + useEffect(() => { + return () => { + if (navSpinnerTimerRef.current !== null) { + window.clearTimeout(navSpinnerTimerRef.current); + } + }; + }, []); + + useEffect(() => { + setLastLiveUrl(null); + }, [threadId]); + + useEffect(() => { + if (frame?.url && !urlInput && !liveUrl) { + setUrlInput(frame.url); + } + }, [frame?.url, liveUrl, urlInput]); + + // Keep the URL bar in sync with the latest visible browser state while not + // live. If the user leaves Live mode, the last streamed frame becomes the + // static baseline instead of falling back to an older artifact screenshot. + useEffect(() => { + if (!live) { + const url = liveFallback?.url ?? frame?.url; + if (url) { + setUrlInput(url); + } + } + }, [frame?.url, live, liveFallback?.url]); + + useEffect(() => { + const screenshot = frame?.screenshot ?? null; + if (screenshot && screenshot !== staticScreenshotRef.current) { + staticScreenshotRef.current = screenshot; + setLiveFallback(null); + } + }, [frame?.screenshot]); + + useEffect(() => { + if (frameUrl) { + setLiveFallback((prev) => ({ + frameUrl, + url: liveUrl ?? prev?.url ?? (urlInput || frame?.url), + })); + } + }, [frame?.url, frameUrl, liveUrl, urlInput]); + + // In live mode the server reports the page's real URL (after redirects and + // history moves). Reconcile the address bar + device-shell label + persisted + // frame so the top URL never goes stale after navigation. + useEffect(() => { + if (live && liveUrl) { + setLastLiveUrl(liveUrl); + if (!urlEditingRef.current) { + setUrlInput(liveUrl === "about:blank" ? "" : liveUrl); + } + setNavigating(false); + setLiveFallback((prev) => (prev ? { ...prev, url: liveUrl } : prev)); + } + }, [liveUrl, live]); + + const handleNavigate = async () => { + const target = urlInput.trim(); + if (!target) { + return; + } + const normalized = /^https?:\/\//i.test(target) + ? target + : `https://${target}`; + + // In live mode, steer the streamed page directly over the socket. The + // screencast streams continuously, so there is no single "done" frame to + // key off — show the reload spinner for a brief window so the user gets + // clear feedback that the navigation actually took. Also sync the URL + // everywhere immediately (URL bar, device-shell label, persisted frame). + if (live) { + sendInput({ type: "navigate", url: normalized }); + setUrlInput(normalized); + urlEditingRef.current = false; + setLiveFallback((prev) => (prev ? { ...prev, url: normalized } : prev)); + setNavigating(true); + if (navSpinnerTimerRef.current !== null) { + window.clearTimeout(navSpinnerTimerRef.current); + } + navSpinnerTimerRef.current = window.setTimeout(() => { + navSpinnerTimerRef.current = null; + setNavigating(false); + }, 1200); + return; + } + + if (navigating) { + return; + } + setNavigating(true); + try { + const result = await navigateBrowser(threadId, normalized); + if (result.screenshot) { + setLiveFallback(null); + browserView?.pushFrame({ + screenshot: result.screenshot, + url: result.url, + title: result.title, + }); + } else { + setUrlInput(result.url); + toast.warning("Navigated, but no screenshot could be captured."); + } + browserView?.openPanel(); + } catch (error) { + toast.error(error instanceof Error ? error.message : String(error)); + } finally { + setNavigating(false); + } + }; + + const normalizedPoint = useCallback( + ( + clientX: number, + clientY: number, + options?: { fallbackToCenter?: boolean }, + ): { nx: number; ny: number } | null => { + const el = surfaceRef.current; + if (!el) { + return null; + } + const rect = el.getBoundingClientRect(); + const natW = el.naturalWidth; + const natH = el.naturalHeight; + if (!rect.width || !rect.height || !natW || !natH) { + return null; + } + // The frame is drawn with object-contain: scaled to fit and centered, so + // there are letterbox bars whenever the panel and remote viewport differ + // in aspect. Map the pointer against the actual content box. + const scale = Math.min(rect.width / natW, rect.height / natH); + const contentW = natW * scale; + const contentH = natH * scale; + const offX = (rect.width - contentW) / 2; + const offY = (rect.height - contentH) / 2; + const px = clientX - rect.left - offX; + const py = clientY - rect.top - offY; + if (px < 0 || py < 0 || px > contentW || py > contentH) { + if (options?.fallbackToCenter) { + return { nx: 0.5, ny: 0.5 }; + } + return null; + } + return { nx: px / contentW, ny: py / contentH }; + }, + [], + ); + + const forwardMouse = ( + type: Extract["type"], + event: React.MouseEvent, + ) => { + if (!live) { + return; + } + const point = normalizedPoint(event.clientX, event.clientY); + if (point) { + sendInput({ type, ...point }); + } + }; + + const liveActive = live && status === "open"; + const liveConnecting = live && status === "connecting"; + const displayUrl = live + ? (frameUrl ?? liveFallback?.frameUrl ?? imageUrl) + : (liveFallback?.frameUrl ?? imageUrl); + + // React registers onWheel as a passive listener, so preventDefault() there is + // ignored and the wheel scrolls the host chat page. Bind a native, non-passive + // listener on the always-mounted stage (the may not exist yet when live + // first opens) so scrolling stays captured inside the remote page. + useEffect(() => { + const el = stageRef.current; + if (!el || !liveActive) { + return; + } + let wheelFrame: number | null = null; + let pendingDx = 0; + let pendingDy = 0; + let pendingPoint: { nx: number; ny: number } | null = null; + + const browserWheelDelta = (pixels: number) => { + return Math.abs(pixels) < 0.25 ? 0 : pixels * 2; + }; + + const flushWheel = () => { + wheelFrame = null; + const dx = browserWheelDelta(pendingDx); + const dy = browserWheelDelta(pendingDy); + const point = pendingPoint; + pendingDx = 0; + pendingDy = 0; + pendingPoint = null; + if (dx || dy) { + sendInput({ type: "wheel", dx, dy, ...(point ?? {}) }); + } + }; + + const onWheel = (event: WheelEvent) => { + event.preventDefault(); + event.stopPropagation(); + // Normalize deltaMode (line/page → pixels), then batch one animation + // frame of deltas. A small 2x gain keeps touchpad gestures responsive + // without returning to the jumpy large-step behavior. + const unit = event.deltaMode === 1 ? 16 : event.deltaMode === 2 ? 800 : 1; + pendingDx += event.deltaX * unit; + pendingDy += event.deltaY * unit; + const point = normalizedPoint(event.clientX, event.clientY, { + fallbackToCenter: true, + }); + if (point) { + pendingPoint = point; + } + wheelFrame ??= window.requestAnimationFrame(flushWheel); + }; + el.addEventListener("wheel", onWheel, { + capture: true, + passive: false, + }); + return () => { + el.removeEventListener("wheel", onWheel, { capture: true }); + if (wheelFrame !== null) { + window.cancelAnimationFrame(wheelFrame); + } + }; + }, [liveActive, normalizedPoint, sendInput]); + + return ( +
{ + const target = event.target; + const editableTarget = + target instanceof HTMLInputElement || + target instanceof HTMLTextAreaElement || + (target instanceof HTMLElement && target.isContentEditable); + const input = decideBrowserKeyInput({ + live, + editableTarget, + composing: isIMEComposing(event), + key: event.key, + metaKey: event.metaKey, + ctrlKey: event.ctrlKey, + }); + if (!input) { + return; + } + sendInput(input); + event.preventDefault(); + event.stopPropagation(); + }} + > +
+ + Browser +
+ + +
+
{ + event.preventDefault(); + void handleNavigate(); + }} + > + + { + urlEditingRef.current = true; + setUrlInput(event.target.value); + }} + onFocus={(event) => { + urlEditingRef.current = true; + event.currentTarget.select(); + }} + onBlur={() => { + urlEditingRef.current = false; + }} + onKeyDown={(event) => { + // Keep browser select-all/copy/cut/paste working inside the URL + // bar; stop the panel + global shortcut handlers from swallowing. + event.stopPropagation(); + }} + placeholder="Enter a URL and press Enter" + spellCheck={false} + autoComplete="off" + className="h-8 pl-7 text-xs" + /> + {navigating && ( + + )} + + + +
+
+
{ + panelRef.current?.focus({ preventScroll: true }); + }} + > + {displayUrl ? ( + {frame?.title forwardMouse("click", event)} + /> + ) : ( + } + title={ + live ? "Connecting to live browser…" : "No browser activity yet" + } + description={ + live + ? "Waiting for the first live frame." + : "Enter a URL above or let the agent browse — the live view will appear here." + } + /> + )} + {(navigating || liveConnecting) && displayUrl && ( +
+ +
+ )} +
+
+
+ ); +} diff --git a/frontend/src/components/workspace/browser-view/context.tsx b/frontend/src/components/workspace/browser-view/context.tsx new file mode 100644 index 000000000..866f1b48c --- /dev/null +++ b/frontend/src/components/workspace/browser-view/context.tsx @@ -0,0 +1,76 @@ +"use client"; + +import { + createContext, + useCallback, + useContext, + useMemo, + useState, + type ReactNode, +} from "react"; + +export interface BrowserViewFrame { + screenshot: string; + url?: string; + title?: string; + action?: string; +} + +interface BrowserViewContextValue { + open: boolean; + latestFrame: BrowserViewFrame | null; + pushFrame: (frame: BrowserViewFrame) => void; + openPanel: () => void; + close: () => void; +} + +const BrowserViewContext = createContext(null); + +export function BrowserViewProvider({ children }: { children: ReactNode }) { + const [open, setOpen] = useState(false); + const [latestFrame, setLatestFrame] = useState(null); + + const pushFrame = useCallback((frame: BrowserViewFrame) => { + setLatestFrame((prev) => { + const sameFrame = + prev?.screenshot === frame.screenshot && + prev?.url === frame.url && + prev?.title === frame.title && + prev?.action === frame.action; + if (sameFrame) { + return prev; + } + if (prev?.screenshot !== frame.screenshot) { + // A new browser frame arrived — surface the panel automatically. + setOpen(true); + } + return frame; + }); + }, []); + + const openPanel = useCallback(() => setOpen(true), []); + const close = useCallback(() => setOpen(false), []); + + const value = useMemo( + () => ({ open, latestFrame, pushFrame, openPanel, close }), + [open, latestFrame, pushFrame, openPanel, close], + ); + + return ( + + {children} + + ); +} + +export function useMaybeBrowserView(): BrowserViewContextValue | null { + return useContext(BrowserViewContext); +} + +export function useBrowserView(): BrowserViewContextValue { + const ctx = useContext(BrowserViewContext); + if (!ctx) { + throw new Error("useBrowserView must be used within a BrowserViewProvider"); + } + return ctx; +} diff --git a/frontend/src/components/workspace/browser-view/index.ts b/frontend/src/components/workspace/browser-view/index.ts new file mode 100644 index 000000000..19a4c5492 --- /dev/null +++ b/frontend/src/components/workspace/browser-view/index.ts @@ -0,0 +1,19 @@ +export { BrowserViewPanel } from "./browser-view-panel"; +export { BrowserTrigger } from "./browser-trigger"; +export { + navigateBrowser, + browserStreamURL, + type BrowserNavigateResult, +} from "./api"; +export { + useBrowserStream, + type BrowserTab, + type BrowserInputEvent, + type BrowserStreamStatus, +} from "./use-browser-stream"; +export { + BrowserViewProvider, + useBrowserView, + useMaybeBrowserView, + type BrowserViewFrame, +} from "./context"; diff --git a/frontend/src/components/workspace/browser-view/keyboard.ts b/frontend/src/components/workspace/browser-view/keyboard.ts new file mode 100644 index 000000000..b9a157bb9 --- /dev/null +++ b/frontend/src/components/workspace/browser-view/keyboard.ts @@ -0,0 +1,55 @@ +import type { BrowserInputEvent } from "./use-browser-stream"; + +// Named keys we forward to the remote page as key presses (everything else that +// is a single printable char is forwarded as text). Chosen to cover editing and +// navigation without leaking browser-level shortcuts. +export const FORWARDED_NAMED_KEYS = [ + "Enter", + "Backspace", + "Tab", + "ArrowUp", + "ArrowDown", + "ArrowLeft", + "ArrowRight", + "Escape", + "Delete", +] as const; + +export interface BrowserKeyContext { + /** Panel is in Live (interactive) mode. */ + live: boolean; + /** Focus is on an editable element (URL bar etc.) — keep it local. */ + editableTarget: boolean; + /** An IME composition is active — the keystroke belongs to the composer. */ + composing: boolean; + key: string; + metaKey: boolean; + ctrlKey: boolean; +} + +/** + * Decide how a keydown maps to a remote browser input, or ``null`` to ignore it. + * + * Pure so the forwarding policy (including the IME-composition guard that must + * swallow a composing Enter) can be unit-tested without a DOM. + */ +export function decideBrowserKeyInput( + ctx: BrowserKeyContext, +): BrowserInputEvent | null { + if (!ctx.live || ctx.editableTarget || ctx.composing) { + return null; + } + if ((ctx.ctrlKey || ctx.metaKey) && ctx.key.length === 1) { + return { + type: "key", + key: `${ctx.metaKey ? "Meta" : "Control"}+${ctx.key.toUpperCase()}`, + }; + } + if (ctx.key.length === 1 && !ctx.metaKey && !ctx.ctrlKey) { + return { type: "text", text: ctx.key }; + } + if ((FORWARDED_NAMED_KEYS as readonly string[]).includes(ctx.key)) { + return { type: "key", key: ctx.key }; + } + return null; +} diff --git a/frontend/src/components/workspace/browser-view/use-browser-stream.ts b/frontend/src/components/workspace/browser-view/use-browser-stream.ts new file mode 100644 index 000000000..815aade7b --- /dev/null +++ b/frontend/src/components/workspace/browser-view/use-browser-stream.ts @@ -0,0 +1,237 @@ +"use client"; + +import { useCallback, useEffect, useRef, useState } from "react"; + +import { browserStreamURL } from "./api"; + +export interface BrowserTab { + index: number; + title: string; + url: string; + active: boolean; +} + +export type BrowserInputEvent = + | { type: "click"; nx: number; ny: number } + | { type: "move"; nx: number; ny: number } + | { type: "down"; nx: number; ny: number } + | { type: "up"; nx: number; ny: number } + | { type: "wheel"; dx: number; dy: number; nx?: number; ny?: number } + | { type: "key"; key: string } + | { type: "text"; text: string } + | { type: "navigate"; url: string } + | { type: "back" } + | { type: "forward" } + | { type: "activate_tab"; index: number }; + +export type BrowserStreamStatus = "idle" | "connecting" | "open" | "closed"; + +const RECONNECT_BASE_DELAY_MS = 800; +const RECONNECT_MAX_DELAY_MS = 10_000; +const RECONNECT_MAX_ATTEMPTS = 6; + +function normalizeSeedUrl(url: string | null | undefined): string { + return (url ?? "").split("#", 1)[0]?.replace(/\/+$/, "") ?? ""; +} + +/** + * Manage a live browser screencast WebSocket. + * + * When ``enabled`` is true, opens the stream, exposes the latest JPEG frame as + * a data URL, and returns a ``sendInput`` callback that forwards user input to + * the live page. Closes and cleans up when disabled or unmounted. + * + * ``seedUrl`` is only read when a connection is first established (via a ref, so + * it is NOT a reconnect trigger). A separate effect steers an already-open live + * page toward a changed seed with an in-band ``navigate`` event, so ordinary + * navigations no longer tear down and rebuild the socket. + */ +export function useBrowserStream( + threadId: string, + enabled: boolean, + seedUrl?: string, + onNavRejected?: ( + url: string | undefined, + message: string | undefined, + ) => void, +) { + const [status, setStatus] = useState("idle"); + const [frameUrl, setFrameUrl] = useState(null); + const [liveUrl, setLiveUrl] = useState(null); + const [tabs, setTabs] = useState([]); + const [connectionAttempt, setConnectionAttempt] = useState(0); + const socketRef = useRef(null); + const pendingNavigateRef = useRef | null>(null); + // Read the seed at connect time only; it must not be a reconnect dependency. + const seedRef = useRef(seedUrl); + seedRef.current = seedUrl; + // Latest live page URL reported by the server, used to decide whether an + // open stream already shows the seed target (avoids redundant navigations). + const liveUrlRef = useRef(null); + const onNavRejectedRef = useRef(onNavRejected); + onNavRejectedRef.current = onNavRejected; + + const sendInput = useCallback((event: BrowserInputEvent) => { + const socket = socketRef.current; + if (socket?.readyState === WebSocket.OPEN) { + socket.send(JSON.stringify(event)); + return true; + } + // URL bar submissions are user intent and must not be lost during the + // short Live connection window right after opening the panel. + if (event.type === "navigate") { + pendingNavigateRef.current = event; + } + return false; + }, []); + + useEffect(() => { + pendingNavigateRef.current = null; + }, [threadId]); + + useEffect(() => { + if (enabled) { + return; + } + setConnectionAttempt(0); + setFrameUrl(null); + setLiveUrl(null); + setTabs([]); + liveUrlRef.current = null; + }, [enabled, threadId]); + + useEffect(() => { + if (!enabled) { + setStatus("idle"); + liveUrlRef.current = null; + return; + } + + let closedByEffect = false; + let reconnectTimer: number | null = null; + setStatus("connecting"); + // browserStreamURL treats empty/undefined seed identically (no seed param), + // so the raw ref value is fine here. Record the seed optimistically so the + // "steer to seed" effect below does not fire a duplicate navigate right + // after open (the server already aligns the page to the connect-time seed). + liveUrlRef.current = seedRef.current ?? null; + const socket = new WebSocket(browserStreamURL(threadId, seedRef.current)); + socketRef.current = socket; + + const scheduleReconnect = () => { + if (closedByEffect || !enabled) { + return; + } + if (reconnectTimer !== null) { + return; + } + // Exponential backoff with a ceiling + attempt cap so a server that keeps + // rejecting the upgrade cannot pin the client in a tight reconnect loop. + if (connectionAttempt >= RECONNECT_MAX_ATTEMPTS) { + return; + } + const delay = Math.min( + RECONNECT_BASE_DELAY_MS * 2 ** connectionAttempt, + RECONNECT_MAX_DELAY_MS, + ); + reconnectTimer = window.setTimeout(() => { + setConnectionAttempt((attempt) => attempt + 1); + }, delay); + }; + + socket.onopen = () => { + const pendingNavigate = pendingNavigateRef.current; + if (pendingNavigate) { + socket.send(JSON.stringify(pendingNavigate)); + pendingNavigateRef.current = null; + } + // Reset the reconnect budget on a successful open. Without this the + // cumulative attempt counter never returns to 0 while the panel stays + // mounted, so after RECONNECT_MAX_ATTEMPTS total reconnects — even across + // many healthy connections — scheduleReconnect would bail forever and + // Live would go permanently dead until the panel is toggled off/on. + setConnectionAttempt(0); + setStatus("open"); + }; + socket.onmessage = async (message) => { + try { + const raw = + typeof message.data === "string" + ? message.data + : message.data instanceof Blob + ? await message.data.text() + : message.data instanceof ArrayBuffer + ? new TextDecoder().decode(message.data) + : String(message.data); + // The message may resolve after cleanup (async Blob/ArrayBuffer decode); + // do not write state for a socket the effect already tore down. + if (closedByEffect) { + return; + } + const payload = JSON.parse(raw) as { + type?: string; + data?: string; + url?: string; + message?: string; + tabs?: BrowserTab[]; + }; + if (payload.type === "frame" && payload.data) { + setFrameUrl(`data:image/jpeg;base64,${payload.data}`); + } else if (payload.type === "url" && payload.url) { + liveUrlRef.current = payload.url; + setLiveUrl(payload.url); + } else if (payload.type === "tabs" && Array.isArray(payload.tabs)) { + setTabs(payload.tabs); + } else if (payload.type === "nav_rejected") { + onNavRejectedRef.current?.(payload.url, payload.message); + } + } catch (error) { + console.warn("Ignoring malformed browser stream message", error); + } + }; + socket.onclose = () => { + if (!closedByEffect) { + setStatus("closed"); + scheduleReconnect(); + } + }; + socket.onerror = () => { + if (!closedByEffect) { + setStatus("closed"); + scheduleReconnect(); + } + }; + + return () => { + closedByEffect = true; + if (reconnectTimer !== null) { + window.clearTimeout(reconnectTimer); + } + socketRef.current = null; + socket.close(); + }; + }, [connectionAttempt, enabled, threadId]); + + // Steer an already-open stream toward a changed seed in-band instead of + // rebuilding the socket. Only navigates when the live page differs from the + // seed target, so redirects/history moves the server already reflects do not + // cause a redundant navigation loop. + useEffect(() => { + if (!enabled || status !== "open") { + return; + } + const target = seedUrl?.trim(); + if (!target) { + return; + } + if (normalizeSeedUrl(target) === normalizeSeedUrl(liveUrlRef.current)) { + return; + } + sendInput({ type: "navigate", url: target }); + }, [enabled, seedUrl, sendInput, status]); + + return { status, frameUrl, liveUrl, tabs, sendInput }; +} diff --git a/frontend/src/components/workspace/chats/chat-box.tsx b/frontend/src/components/workspace/chats/chat-box.tsx index e712d32f2..bd3963100 100644 --- a/frontend/src/components/workspace/chats/chat-box.tsx +++ b/frontend/src/components/workspace/chats/chat-box.tsx @@ -4,6 +4,11 @@ import { useEffect, useMemo, useRef, useState } from "react"; import { ConversationEmptyState } from "@/components/ai-elements/conversation"; import { Button } from "@/components/ui/button"; +import { + ResizableHandle, + ResizablePanel, + ResizablePanelGroup, +} from "@/components/ui/resizable"; import { Sheet, SheetContent, @@ -20,17 +25,19 @@ import { ArtifactFileList, useArtifacts, } from "../artifacts"; +import { BrowserViewPanel, useMaybeBrowserView } from "../browser-view"; import { useThread } from "../messages/context"; import { SidecarPanel, useMaybeSidecar } from "../sidecar"; const RIGHT_PANEL_ANIMATION_MS = 280; -type RightPanelKind = "sidecar" | "artifacts"; +type RightPanelKind = "sidecar" | "artifacts" | "browser"; -const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ - children, - threadId, -}) => { +const ChatBox: React.FC<{ + children: React.ReactNode; + threadId: string; + browserEnabled?: boolean; +}> = ({ children, threadId, browserEnabled = true }) => { const { thread } = useThread(); const isMobile = useIsMobile(); const pathname = usePathname(); @@ -47,6 +54,8 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ } = useArtifacts(); const sidecar = useMaybeSidecar(); const sidecarOpen = sidecar?.open ?? false; + const browserView = useMaybeBrowserView(); + const browserViewOpen = browserEnabled && (browserView?.open ?? false); const [autoSelectFirstArtifact, setAutoSelectFirstArtifact] = useState(true); useEffect(() => { @@ -104,9 +113,11 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ const activeRightPanel: RightPanelKind | null = sidecarOpen ? "sidecar" - : artifactPanelOpen - ? "artifacts" - : null; + : browserViewOpen + ? "browser" + : artifactPanelOpen + ? "artifacts" + : null; const rightPanelOpen = activeRightPanel !== null; const [renderedRightPanel, setRenderedRightPanel] = useState(activeRightPanel); @@ -136,7 +147,16 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ } }, [artifactsOpen, setArtifactsOpen, sidecarOpen]); + useEffect(() => { + if (!browserEnabled && browserView?.open) { + browserView.close(); + } + }, [browserEnabled, browserView]); + const rightPanelContent = useMemo(() => { + if (renderedRightPanel === "browser") { + return ; + } if (renderedRightPanel === "sidecar") { return ; } @@ -208,6 +228,9 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ if (sidecarOpen) { sidecar?.close(); } + if (browserViewOpen) { + browserView?.close(); + } if (artifactsOpen) { setArtifactsOpen(false); } @@ -219,7 +242,11 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({ > - {renderedRightPanel === "sidecar" ? "Sidecar" : "Artifacts"} + {renderedRightPanel === "sidecar" + ? "Sidecar" + : renderedRightPanel === "browser" + ? "Browser" + : "Artifacts"} Browse the side panel for this conversation. @@ -236,41 +263,79 @@ const ChatBox: React.FC<{ children: React.ReactNode; threadId: string }> = ({
-
- {children} -
- - {hasSidecarThread ? ( + {hasSidecarThread && ( - ) : ( - // No conversation yet — nothing to delete, so this just discards the - // draft and closes the panel. A plain X (no confirm) keeps it light. - - - )} + + +
diff --git a/frontend/src/core/agents/api.ts b/frontend/src/core/agents/api.ts index f407c565d..c0932defe 100644 --- a/frontend/src/core/agents/api.ts +++ b/frontend/src/core/agents/api.ts @@ -1,5 +1,6 @@ import { fetch } from "@/core/api/fetcher"; import { getBackendBaseURL } from "@/core/config"; +export { fetchAgentsApiEnabled } from "@/core/features/api"; import type { Agent, CreateAgentRequest, UpdateAgentRequest } from "./types"; @@ -87,19 +88,6 @@ export async function deleteAgent(name: string): Promise { if (!res.ok) throw new Error(`Failed to delete agent: ${res.statusText}`); } -interface FeaturesResponse { - agents_api: { enabled: boolean }; -} - -export async function fetchAgentsApiEnabled(): Promise { - const res = await fetch(`${getBackendBaseURL()}/api/features`); - if (!res.ok) { - throw new Error(`Failed to load features: ${res.statusText}`); - } - const data = (await res.json()) as FeaturesResponse; - return data.agents_api.enabled; -} - export async function checkAgentName( name: string, ): Promise<{ available: boolean; name: string }> { diff --git a/frontend/src/core/features/api.ts b/frontend/src/core/features/api.ts new file mode 100644 index 000000000..1a48b4f32 --- /dev/null +++ b/frontend/src/core/features/api.ts @@ -0,0 +1,23 @@ +import { fetch } from "@/core/api/fetcher"; +import { getBackendBaseURL } from "@/core/config"; + +export interface FeaturesResponse { + agents_api: { enabled: boolean }; + browser_control?: { enabled: boolean }; +} + +export async function fetchFeatures(): Promise { + const res = await fetch(`${getBackendBaseURL()}/api/features`); + if (!res.ok) { + throw new Error(`Failed to load features: ${res.statusText}`); + } + return (await res.json()) as FeaturesResponse; +} + +export async function fetchAgentsApiEnabled(): Promise { + return (await fetchFeatures()).agents_api.enabled; +} + +export async function fetchBrowserControlEnabled(): Promise { + return (await fetchFeatures()).browser_control?.enabled ?? false; +} diff --git a/frontend/src/core/features/hooks.ts b/frontend/src/core/features/hooks.ts new file mode 100644 index 000000000..f351edea5 --- /dev/null +++ b/frontend/src/core/features/hooks.ts @@ -0,0 +1,18 @@ +import { useQuery } from "@tanstack/react-query"; + +import { fetchBrowserControlEnabled } from "./api"; + +export function useBrowserControlEnabled() { + const { data, isPending } = useQuery({ + queryKey: ["features", "browser_control"], + queryFn: () => fetchBrowserControlEnabled(), + staleTime: 0, + refetchOnMount: true, + retry: false, + }); + + return { + enabled: data ?? false, + isLoading: isPending, + }; +} diff --git a/frontend/src/core/features/index.ts b/frontend/src/core/features/index.ts new file mode 100644 index 000000000..353123047 --- /dev/null +++ b/frontend/src/core/features/index.ts @@ -0,0 +1,2 @@ +export * from "./api"; +export * from "./hooks"; diff --git a/frontend/src/core/i18n/locales/en-US.ts b/frontend/src/core/i18n/locales/en-US.ts index 4855ad3e3..44a727b6d 100644 --- a/frontend/src/core/i18n/locales/en-US.ts +++ b/frontend/src/core/i18n/locales/en-US.ts @@ -53,6 +53,8 @@ export const enUS: Translations = { regenerate: "Regenerate", branch: "Branch conversation", showArtifacts: "Show artifacts of this conversation", + browser: "Browser", + showBrowser: "Open browser panel", }, // Home @@ -567,6 +569,15 @@ export const enUS: Translations = { clickToViewContent: "Click to view file content", writeTodos: "Update to-do list", skillInstallTooltip: "Install skill and make it available to DeerFlow", + browserNavigate: (url: string) => `Open ${url} in browser`, + browserNavigateGeneric: "Open page in browser", + browserClick: "Click element in browser", + browserType: "Type into browser field", + browserSnapshot: "Read page in browser", + browserGetText: "Read page text in browser", + browserBack: "Go back in browser", + browserScreenshot: "Capture browser screenshot", + browserClose: "Close browser", }, humanInput: { diff --git a/frontend/src/core/i18n/locales/types.ts b/frontend/src/core/i18n/locales/types.ts index 097a1b6c5..844196a5e 100644 --- a/frontend/src/core/i18n/locales/types.ts +++ b/frontend/src/core/i18n/locales/types.ts @@ -42,6 +42,8 @@ export interface Translations { regenerate: string; branch: string; showArtifacts: string; + browser: string; + showBrowser: string; }; home: { @@ -457,6 +459,15 @@ export interface Translations { clickToViewContent: string; writeTodos: string; skillInstallTooltip: string; + browserNavigate: (url: string) => string; + browserNavigateGeneric: string; + browserClick: string; + browserType: string; + browserSnapshot: string; + browserGetText: string; + browserBack: string; + browserScreenshot: string; + browserClose: string; }; humanInput: { diff --git a/frontend/src/core/i18n/locales/zh-CN.ts b/frontend/src/core/i18n/locales/zh-CN.ts index 3a99ee74e..79aaaabe6 100644 --- a/frontend/src/core/i18n/locales/zh-CN.ts +++ b/frontend/src/core/i18n/locales/zh-CN.ts @@ -53,6 +53,8 @@ export const zhCN: Translations = { regenerate: "重新生成", branch: "分叉", showArtifacts: "查看此对话的文件", + browser: "浏览器", + showBrowser: "打开浏览器面板", }, // Home @@ -545,6 +547,15 @@ export const zhCN: Translations = { clickToViewContent: "点击查看文件内容", writeTodos: "更新 To-do 列表", skillInstallTooltip: "安装技能并使其可在 DeerFlow 中使用", + browserNavigate: (url: string) => `在浏览器中打开 ${url}`, + browserNavigateGeneric: "在浏览器中打开页面", + browserClick: "在浏览器中点击元素", + browserType: "在浏览器中输入内容", + browserSnapshot: "在浏览器中读取页面", + browserGetText: "在浏览器中读取页面文本", + browserBack: "在浏览器中返回上一页", + browserScreenshot: "浏览器截图", + browserClose: "关闭浏览器", }, humanInput: { diff --git a/frontend/src/styles/globals.css b/frontend/src/styles/globals.css index bf1f73170..f3d79eaa7 100644 --- a/frontend/src/styles/globals.css +++ b/frontend/src/styles/globals.css @@ -85,6 +85,16 @@ } } + --animate-loading-bar: loading-bar 1s ease-in-out infinite; + @keyframes loading-bar { + 0% { + transform: translateX(-100%); + } + 100% { + transform: translateX(400%); + } + } + --animate-fade-in-up: fade-in-up 0.15s ease-in-out forwards; @keyframes fade-in-up { 0% { diff --git a/frontend/tests/e2e/browser-feature.spec.ts b/frontend/tests/e2e/browser-feature.spec.ts new file mode 100644 index 000000000..d054bedc8 --- /dev/null +++ b/frontend/tests/e2e/browser-feature.spec.ts @@ -0,0 +1,39 @@ +import { expect, test } from "@playwright/test"; + +import { MOCK_THREAD_ID, mockLangGraphAPI } from "./utils/mock-api"; + +test.describe("Browser feature flag", () => { + test("shows browser trigger only when browser_control is enabled", async ({ + page, + }) => { + mockLangGraphAPI(page, { + threads: [{ thread_id: MOCK_THREAD_ID, title: "Browser Enabled" }], + }); + + await page.goto(`/workspace/chats/${MOCK_THREAD_ID}`); + + await expect(page.getByTestId("browser-trigger")).toBeVisible({ + timeout: 15_000, + }); + }); + + test("hides browser trigger when browser_control is disabled", async ({ + page, + }) => { + mockLangGraphAPI(page, { + threads: [{ thread_id: MOCK_THREAD_ID, title: "Browser Disabled" }], + features: { browserControlEnabled: false }, + }); + + const featuresResponse = page.waitForResponse((response) => + response.url().includes("/api/features"), + ); + await page.goto(`/workspace/chats/${MOCK_THREAD_ID}`); + const features = (await (await featuresResponse).json()) as { + browser_control?: { enabled?: boolean }; + }; + expect(features.browser_control?.enabled).toBe(false); + + await expect(page.getByTestId("browser-trigger")).toHaveCount(0); + }); +}); diff --git a/frontend/tests/e2e/utils/mock-api.ts b/frontend/tests/e2e/utils/mock-api.ts index 0e7ec8c39..95cec6651 100644 --- a/frontend/tests/e2e/utils/mock-api.ts +++ b/frontend/tests/e2e/utils/mock-api.ts @@ -87,6 +87,10 @@ export type MockAPIOptions = { max_file_size: number; max_total_size: number; }; + features?: { + agentsApiEnabled?: boolean; + browserControlEnabled?: boolean; + }; }; const DEFAULT_SKILLS: MockSkill[] = [ @@ -126,6 +130,16 @@ function visibleInputMessages(messages: unknown[]) { return messages.filter((message) => !isHiddenInputMessage(message)); } +function mockMessageRunId(message: unknown, fallback: string) { + if (typeof message === "object" && message !== null) { + const runId = Reflect.get(message, "run_id"); + if (typeof runId === "string" && runId.length > 0) { + return runId; + } + } + return fallback; +} + function visibleRunInputMessages(route: Route) { try { const body = route.request().postDataJSON() as { @@ -244,6 +258,10 @@ export function mockLangGraphAPI(page: Page, options?: MockAPIOptions) { max_file_size: 50 * 1024 * 1024, max_total_size: 100 * 1024 * 1024, }; + const featureFlags = { + agentsApiEnabled: options?.features?.agentsApiEnabled ?? true, + browserControlEnabled: options?.features?.browserControlEnabled ?? true, + }; const upsertThread = (thread: MockThread) => { threads = [ @@ -962,7 +980,10 @@ export function mockLangGraphAPI(page: Page, options?: MockAPIOptions) { contentType: "application/json", body: JSON.stringify({ data: (matchingThread?.messages ?? []).map((message, index) => ({ - run_id: `run-${matchingThread?.thread_id ?? "unknown"}`, + run_id: mockMessageRunId( + message, + `run-${matchingThread?.thread_id ?? "unknown"}`, + ), seq: index + 1, content: message, metadata: { caller: "lead_agent" }, @@ -1016,7 +1037,7 @@ export function mockLangGraphAPI(page: Page, options?: MockAPIOptions) { return route.fallback(); }); - // Feature flags — frontend gates UI (e.g. agents) on these. Default to + // Feature flags — frontend gates UI (e.g. agents/browser) on these. Default to // enabled so existing tests exercise the normal path; tests that need the // disabled state override this route after calling mockLangGraphAPI. void page.route("**/api/features", (route) => { @@ -1024,7 +1045,10 @@ export function mockLangGraphAPI(page: Page, options?: MockAPIOptions) { return route.fulfill({ status: 200, contentType: "application/json", - body: JSON.stringify({ agents_api: { enabled: true } }), + body: JSON.stringify({ + agents_api: { enabled: featureFlags.agentsApiEnabled }, + browser_control: { enabled: featureFlags.browserControlEnabled }, + }), }); } return route.fallback(); diff --git a/frontend/tests/unit/components/workspace/browser-view/keyboard.test.ts b/frontend/tests/unit/components/workspace/browser-view/keyboard.test.ts new file mode 100644 index 000000000..3c3dcb7d5 --- /dev/null +++ b/frontend/tests/unit/components/workspace/browser-view/keyboard.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from "@rstest/core"; + +import { decideBrowserKeyInput } from "@/components/workspace/browser-view/keyboard"; + +function ctx( + overrides: Partial[0]> = {}, +) { + return { + live: true, + editableTarget: false, + composing: false, + key: "a", + metaKey: false, + ctrlKey: false, + ...overrides, + }; +} + +describe("decideBrowserKeyInput", () => { + it("forwards printable chars as text", () => { + expect(decideBrowserKeyInput(ctx({ key: "a" }))).toEqual({ + type: "text", + text: "a", + }); + }); + + it("forwards named keys as key presses", () => { + expect(decideBrowserKeyInput(ctx({ key: "Enter" }))).toEqual({ + type: "key", + key: "Enter", + }); + expect(decideBrowserKeyInput(ctx({ key: "ArrowLeft" }))).toEqual({ + type: "key", + key: "ArrowLeft", + }); + }); + + it("forwards modifier combos as a normalized key chord", () => { + expect(decideBrowserKeyInput(ctx({ key: "c", metaKey: true }))).toEqual({ + type: "key", + key: "Meta+C", + }); + expect(decideBrowserKeyInput(ctx({ key: "a", ctrlKey: true }))).toEqual({ + type: "key", + key: "Control+A", + }); + }); + + it("does not forward while an IME composition is active", () => { + // A CJK candidate confirmed with Enter must not submit the remote page. + expect( + decideBrowserKeyInput(ctx({ key: "Enter", composing: true })), + ).toBeNull(); + expect( + decideBrowserKeyInput(ctx({ key: "a", composing: true })), + ).toBeNull(); + }); + + it("ignores keys when not live or focus is on an editable target", () => { + expect( + decideBrowserKeyInput(ctx({ key: "Enter", live: false })), + ).toBeNull(); + expect( + decideBrowserKeyInput(ctx({ key: "Enter", editableTarget: true })), + ).toBeNull(); + }); + + it("ignores unmapped named keys", () => { + expect(decideBrowserKeyInput(ctx({ key: "F5" }))).toBeNull(); + }); +}); diff --git a/frontend/tests/unit/components/workspace/messages/markdown-content.test.ts b/frontend/tests/unit/components/workspace/messages/markdown-content.test.ts index 865e5723a..314024492 100644 --- a/frontend/tests/unit/components/workspace/messages/markdown-content.test.ts +++ b/frontend/tests/unit/components/workspace/messages/markdown-content.test.ts @@ -82,6 +82,13 @@ describe("MarkdownContent streaming code blocks", () => { expect(html).toContain('data-custom-code="true"'); expect(html).toContain("data-streaming-code-block"); }); + + it("does not paint an initial large streaming chunk all at once", () => { + const content = "x".repeat(120); + + expect(renderMarkdown(content, true)).not.toContain(content); + expect(renderMarkdown(content, false)).toContain(content); + }); }); describe("MarkdownContent strikethrough", () => { diff --git a/frontend/tests/unit/components/workspace/messages/message-group.test.ts b/frontend/tests/unit/components/workspace/messages/message-group.test.ts index 0e796680e..4b21c97b1 100644 --- a/frontend/tests/unit/components/workspace/messages/message-group.test.ts +++ b/frontend/tests/unit/components/workspace/messages/message-group.test.ts @@ -1,6 +1,6 @@ import type { Message } from "@langchain/langgraph-sdk"; import { describe, expect, it, rs } from "@rstest/core"; -import { createElement } from "react"; +import { createElement, type ComponentProps } from "react"; import { renderToStaticMarkup } from "react-dom/server"; import { MessageGroup } from "@/components/workspace/messages/message-group"; @@ -121,9 +121,55 @@ describe("MessageGroup", () => { expect(html).toContain("1 more step"); expect(html).not.toContain("Check how processing groups convert messages."); }); + + it("defers browser screenshot previews while the thread is loading", () => { + const messages = [ + { + id: "ai-1", + type: "ai", + content: "", + tool_calls: [ + { + id: "call-1", + name: "browser_navigate", + args: { url: "https://github.com/bytedance/deer-flow" }, + }, + ], + } as Message, + { + id: "tool-1", + type: "tool", + name: "browser_navigate", + tool_call_id: "call-1", + content: "Opened", + additional_kwargs: { + browser_view: { + screenshot: "/mnt/user-data/outputs/browser.png", + url: "https://github.com/bytedance/deer-flow", + }, + }, + } as Message, + ]; + + const visibleHtml = renderGroup(messages, { + threadId: "thread-1", + deferBrowserPreviews: false, + }); + const deferredHtml = renderGroup(messages, { + threadId: "thread-1", + deferBrowserPreviews: true, + }); + + expect(visibleHtml).toContain(", "messages"> = {}, +) { return renderToStaticMarkup( createElement( I18nContext.Provider, @@ -133,7 +179,7 @@ function renderGroup(messages: Message[]) { setLocale: () => undefined, }, }, - createElement(MessageGroup, { messages }), + createElement(MessageGroup, { ...props, messages }), ), ); } diff --git a/frontend/tests/unit/core/agents/features.test.ts b/frontend/tests/unit/core/agents/features.test.ts index a0245c8e6..58cfe90fc 100644 --- a/frontend/tests/unit/core/agents/features.test.ts +++ b/frontend/tests/unit/core/agents/features.test.ts @@ -10,6 +10,7 @@ rs.mock("@/core/config", () => ({ import { fetchAgentsApiEnabled } from "@/core/agents/api"; import { fetch as fetcher } from "@/core/api/fetcher"; +import { fetchBrowserControlEnabled } from "@/core/features/api"; const mockedFetch = rs.mocked(fetcher); @@ -45,3 +46,36 @@ describe("fetchAgentsApiEnabled", () => { await expect(fetchAgentsApiEnabled()).rejects.toThrow(); }); }); + +describe("fetchBrowserControlEnabled", () => { + test("returns true when backend reports browser_control enabled", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + agents_api: { enabled: true }, + browser_control: { enabled: true }, + }), + ); + await expect(fetchBrowserControlEnabled()).resolves.toBe(true); + expect(mockedFetch).toHaveBeenCalledWith("/api/features"); + }); + + test("returns false when browser_control is disabled or omitted", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + agents_api: { enabled: true }, + browser_control: { enabled: false }, + }), + ); + await expect(fetchBrowserControlEnabled()).resolves.toBe(false); + + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { agents_api: { enabled: true } }), + ); + await expect(fetchBrowserControlEnabled()).resolves.toBe(false); + }); + + test("throws when the features request fails", async () => { + mockedFetch.mockResolvedValueOnce(jsonResponse(500, {})); + await expect(fetchBrowserControlEnabled()).rejects.toThrow(); + }); +}); diff --git a/scripts/detect_uv_extras.py b/scripts/detect_uv_extras.py index 078d978e4..f2bdc230a 100755 --- a/scripts/detect_uv_extras.py +++ b/scripts/detect_uv_extras.py @@ -11,6 +11,7 @@ Order of resolution: - database.backend == postgres -> postgres - checkpointer.type == postgres -> postgres - stream_bridge.type == redis -> redis + - tools[].name == browser_navigate -> browser - sandbox.ownership.type == redis -> redis 3. Runtime environment toggles that enable optional backends: - DEER_FLOW_STREAM_BRIDGE_REDIS_URL -> redis @@ -76,6 +77,7 @@ def find_config_file() -> Path | None: _SECTION_RE = re.compile(r"^([A-Za-z_][\w-]*)\s*:\s*$") _INDENTED_SECTION_RE = re.compile(r"^\s+([A-Za-z_][\w-]*)\s*:\s*$") _KEY_RE = re.compile(r"^\s+([A-Za-z_][\w-]*)\s*:\s*(\S.*?)\s*$") +_LIST_ITEM_NAME_RE = re.compile(r"^\s*-\s+name\s*:\s*(\S.*?)\s*$") def _strip_comment(line: str) -> str: @@ -222,6 +224,30 @@ def nested_section_value(lines: list[str], section_path: str, key: str) -> str | return None +def tools_include_name(lines: list[str], tool_name: str) -> bool: + """Return True when the top-level tools list has an active item name.""" + inside = False + for raw in lines: + line = _strip_comment(raw) + if not line.strip(): + continue + sect_match = _SECTION_RE.match(line) + if sect_match: + inside = sect_match.group(1) == "tools" + continue + if not inside: + continue + stripped = line.lstrip() + indent = len(line) - len(stripped) + if indent == 0: + inside = False + continue + name_match = _LIST_ITEM_NAME_RE.match(line) + if name_match and _unquote(name_match.group(1).strip()) == tool_name: + return True + return False + + def detect_from_config(path: Path) -> list[str]: try: text = path.read_text(encoding="utf-8", errors="replace") @@ -239,6 +265,8 @@ def detect_from_config(path: Path) -> list[str]: extras.add("redis") if (nested_section_value(lines, "channels.discord", "enabled") or "").lower() == "true": extras.add("discord") + if tools_include_name(lines, "browser_navigate"): + extras.add("browser") return sorted(extras)