Ryker_Feng fa496c0c8d
feat(browser): add agentic browser control (#4187)
* feat(browser): add agentic browser control

* fix(frontend): format browser view changes

* fix(browser): keep browser optional and isolate sidecar layout

* fix(browser): address PR review security and IME findings

- Nginx: add a browser-stream WebSocket location before the generic
  /api/threads regex so Live upgrades instead of downgrading to HTTP
  (both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
  navigate, and tear down the browser session on thread deletion so a
  later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
  context-level route guard covering redirects, popups, iframes, and
  subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
  CJK candidate with Enter no longer submits the remote page form.

Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.

* fix(frontend): smooth streaming in long tool threads

* Revert "fix(frontend): smooth streaming in long tool threads"

This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.

* fix(browser): address review security and lifecycle findings

- Reject cross-origin WebSocket upgrades on the live browser stream
  (Origin allow-list reuse of CORS/same-origin helpers) to close a
  WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
  off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
  (browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
  and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
  can't permanently stall after the cumulative attempt cap.

* fix(frontend): reduce long tool thread render stalls

Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.

* fix(browser): keep live control responsive during continuous input

Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").

What:
- Frontend forwards one `click` per physical click instead of also emitting
  `down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
  start a rate-limited background refresh loop (leading frame + bounded cadence)
  that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
  inputs coalesce, and continuous input keeps refreshing before it stops.

Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.

* fix(browser): harden worker and session lifecycle

* fix(browser): address latest review feedback

* fix(frontend): preserve optimistic new-chat message

* test(e2e): preserve mocked message run ids

* fix(browser): address capability review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-21 11:46:33 +08:00

108 lines
3.6 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import { describe, expect, it } from "@rstest/core";
import { createElement, type ImgHTMLAttributes } from "react";
import { renderToStaticMarkup } from "react-dom/server";
import { MarkdownContent } from "@/components/workspace/messages/markdown-content";
function renderMarkdown(
content: string,
isLoading: boolean,
components?: Parameters<typeof MarkdownContent>[0]["components"],
) {
return renderToStaticMarkup(
createElement(MarkdownContent, { content, isLoading, components }),
);
}
describe("MarkdownContent streaming code blocks", () => {
it("renders fenced code without Streamdown highlighting while streaming", () => {
const html = renderMarkdown(
["```html", '<main class="report">Hello</main>', "```"].join("\n"),
true,
);
expect(html).toContain("data-streaming-code-block");
expect(html).toContain('data-language="html"');
expect(html).toContain(
"&lt;main class=&quot;report&quot;&gt;Hello&lt;/main&gt;",
);
expect(html).not.toContain('data-streamdown="code-block"');
});
it("keeps inline code inline while streaming", () => {
const html = renderMarkdown("Use `const answer = 42` here.", true);
expect(html).toContain('data-streaming-inline-code="true"');
expect(html).not.toContain("data-streaming-code-block");
});
it("keeps an unlabeled single-line fence as a block while streaming", () => {
const html = renderMarkdown(["```", "x", "```"].join("\n"), true);
expect(html).toContain("data-streaming-code-block");
expect(html).not.toContain('data-streaming-inline-code="true"');
});
it("restores Streamdown highlighting after streaming finishes", () => {
const html = renderMarkdown(
["```html", '<main class="report">Hello</main>', "```"].join("\n"),
false,
);
expect(html).toContain('data-streamdown="code-block"');
expect(html).not.toContain("data-streaming-code-block");
});
it("preserves custom non-code renderers while streaming", () => {
const html = renderMarkdown(
"[Docs](https://example.com)\n\n![Chart](chart.png)",
true,
{
a: ({ children, href }) =>
createElement("a", { "data-custom-link": true, href }, children),
img: (props: ImgHTMLAttributes<HTMLImageElement>) =>
createElement("img", { ...props, "data-custom-image": true }),
},
);
expect(html).toContain('data-custom-link="true"');
expect(html).toContain('data-custom-image="true"');
});
it("preserves a caller-provided code renderer while streaming", () => {
const html = renderMarkdown(
["```html", "<main />", "```"].join("\n"),
true,
{
code: ({ children }) =>
createElement("code", { "data-custom-code": true }, children),
},
);
expect(html).toContain('data-custom-code="true"');
expect(html).toContain("data-streaming-code-block");
});
it("does not paint an initial large streaming chunk all at once", () => {
const content = "x".repeat(120);
expect(renderMarkdown(content, true)).not.toContain(content);
expect(renderMarkdown(content, false)).toContain(content);
});
});
describe("MarkdownContent strikethrough", () => {
it("preserves single tildes in temperature ranges", () => {
const html = renderMarkdown("周六23~30℃周日22~30℃", false);
expect(html).toContain("周六23~30℃周日22~30℃");
expect(html).not.toContain("<del>");
});
it("continues to render double-tilde strikethrough", () => {
const html = renderMarkdown("状态:~~已取消~~", false);
expect(html).toContain("<del>已取消</del>");
});
});