Ryker_Feng fa496c0c8d
feat(browser): add agentic browser control (#4187)
* feat(browser): add agentic browser control

* fix(frontend): format browser view changes

* fix(browser): keep browser optional and isolate sidecar layout

* fix(browser): address PR review security and IME findings

- Nginx: add a browser-stream WebSocket location before the generic
  /api/threads regex so Live upgrades instead of downgrading to HTTP
  (both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
  navigate, and tear down the browser session on thread deletion so a
  later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
  context-level route guard covering redirects, popups, iframes, and
  subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
  CJK candidate with Enter no longer submits the remote page form.

Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.

* fix(frontend): smooth streaming in long tool threads

* Revert "fix(frontend): smooth streaming in long tool threads"

This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.

* fix(browser): address review security and lifecycle findings

- Reject cross-origin WebSocket upgrades on the live browser stream
  (Origin allow-list reuse of CORS/same-origin helpers) to close a
  WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
  off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
  (browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
  and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
  can't permanently stall after the cumulative attempt cap.

* fix(frontend): reduce long tool thread render stalls

Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.

* fix(browser): keep live control responsive during continuous input

Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").

What:
- Frontend forwards one `click` per physical click instead of also emitting
  `down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
  start a rate-limited background refresh loop (leading frame + bounded cadence)
  that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
  inputs coalesce, and continuous input keeps refreshing before it stops.

Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.

* fix(browser): harden worker and session lifecycle

* fix(browser): address latest review feedback

* fix(frontend): preserve optimistic new-chat message

* test(e2e): preserve mocked message run ids

* fix(browser): address capability review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-21 11:46:33 +08:00

82 lines
2.7 KiB
TOML

[project]
name = "deerflow-harness"
version = "2.1.0"
description = "DeerFlow agent harness framework"
requires-python = ">=3.12"
dependencies = [
"agent-client-protocol>=0.4.0",
"agent-sandbox>=0.0.19",
"croniter>=6.0.0",
"dotenv>=0.9.9",
"exa-py>=1.0.0",
"httpx>=0.28.0",
"kubernetes>=30.0.0",
"langchain>=1.2.15",
"langchain-anthropic>=1.4.1",
"langchain-deepseek>=1.0.1",
"langchain-mcp-adapters>=0.2.2",
"langchain-openai>=1.2.1",
"langfuse>=3.4.1",
"langgraph>=1.1.9",
"langgraph-api>=0.8.1",
"langgraph-cli>=0.4.24",
"langgraph-runtime-inmem>=0.28.0",
"markdownify>=1.2.2",
"markitdown[all,xlsx]>=0.0.1a2",
"pydantic>=2.12.5",
"pyyaml>=6.0.3",
"readabilipy>=0.3.0",
"tavily-python>=0.7.17",
"firecrawl-py>=1.15.0",
"tiktoken>=0.8.0",
"ddgs>=9.10.0",
"duckdb>=1.4.4",
"langchain-google-genai>=4.2.1",
"langgraph-checkpoint-sqlite>=3.0.3",
"langgraph-sdk>=0.1.51",
"sqlalchemy[asyncio]>=2.0,<3.0",
"aiosqlite>=0.19",
"alembic>=1.13",
"cryptography>=48.0.1",
"e2b-code-interpreter>=2.8.0",
]
[project.scripts]
deerflow = "deerflow.tui.cli:main"
[project.optional-dependencies]
# Terminal workbench (TUI). Kept optional so the core harness install stays lean;
# the `deerflow` console script degrades to headless help when textual is absent.
tui = ["textual>=0.80"]
# GroundRoute needs no extra packages (httpx is already a core dependency). This
# empty extra exists so the documented `uv add 'deerflow-harness[groundroute]'`
# install command resolves cleanly without an "unknown extra" warning.
groundroute = []
ollama = ["langchain-ollama>=0.3.0"]
postgres = [
"asyncpg>=0.29",
"langgraph-checkpoint-postgres>=3.0.5",
"psycopg[binary]>=3.3.3",
"psycopg-pool>=3.3.0",
]
# Cross-process SSE stream bridge (stream_bridge.type: redis). Optional so
# single-process / memory-bridge installs do not pull redis. The Docker image
# always installs this extra because Docker defaults to the redis bridge.
redis = ["redis>=5.0.0"]
pymupdf = ["pymupdf4llm>=0.0.17"]
boxlite = ["boxlite>=0.9.7"]
# Agent observability (Monocle). Optional so a default install stays free of the
# OpenTelemetry stack; only pulled in when MONOCLE_TRACING is used.
monocle = ["monocle_apptrace>=0.8.8"]
# Agentic browser control (browser_navigate/click/type/... tool group). Optional
# so the core harness install stays lean; import is lazy inside the private
# Playwright loop. After install, run `playwright install chromium` once.
browser = ["playwright>=1.40"]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["deerflow"]