Ryker_Feng
fa496c0c8d
feat(browser): add agentic browser control ( #4187 )
...
* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-21 11:46:33 +08:00
..
2026-07-21 09:09:40 +08:00
2026-07-01 22:49:17 +08:00
2026-07-19 18:26:26 +08:00
2026-07-04 21:31:01 +08:00
2026-05-12 23:18:54 +08:00
2026-06-30 18:58:02 +08:00
2026-04-26 15:09:25 +08:00
2026-06-01 15:50:39 +08:00
2026-07-05 08:45:27 +08:00
2026-07-21 11:46:33 +08:00
2026-06-21 20:56:42 +08:00
2026-06-08 12:35:03 +08:00
2026-03-27 20:03:30 +08:00
2026-07-17 15:11:15 +08:00
2026-07-21 09:09:40 +08:00
2026-07-21 09:09:40 +08:00
2026-05-21 14:44:34 +08:00
2026-07-03 21:52:31 +08:00
2026-06-23 10:39:40 +08:00
2026-07-21 10:14:29 +08:00
2026-07-19 07:31:29 +08:00
2026-05-29 23:05:59 +08:00
2026-05-16 09:24:40 +08:00
2026-04-26 11:08:11 +08:00
2026-06-16 23:20:20 +08:00
2026-07-18 22:17:15 +08:00
2026-06-21 15:47:53 +08:00
2026-07-17 14:49:51 +08:00
2026-07-17 14:49:51 +08:00
2026-07-21 09:23:14 +08:00
2026-06-24 13:57:12 +08:00
2026-07-09 17:21:25 +08:00
2026-07-09 17:21:25 +08:00
2026-07-02 11:00:49 +08:00
2026-07-21 11:46:33 +08:00
2026-07-21 11:46:33 +08:00
2026-07-19 20:08:52 +08:00
2026-07-14 07:37:59 +08:00
2026-06-16 23:04:39 +08:00
2026-06-18 10:15:31 +08:00
2026-07-12 08:54:03 +08:00
2026-05-02 15:19:28 +08:00
2026-07-17 14:49:51 +08:00
2026-06-18 10:45:33 +08:00
2026-07-21 10:02:49 +08:00
2026-04-14 10:29:44 +08:00
2026-07-12 08:15:32 +08:00
2026-07-12 08:15:32 +08:00
2026-07-06 22:34:41 +08:00
2026-03-30 07:41:18 +08:00
2026-04-25 19:40:06 +08:00
2026-04-07 18:21:22 +08:00
2026-07-21 09:39:44 +08:00
2026-07-03 08:01:46 +08:00
2026-03-29 21:03:58 +08:00
2026-05-04 09:56:16 +08:00
2026-07-21 09:39:44 +08:00
2026-05-02 15:04:11 +08:00
2026-06-10 21:36:25 +08:00
2026-07-19 18:56:03 +08:00
2026-07-05 09:40:16 +08:00
2026-07-07 19:55:33 +08:00
2026-04-26 11:05:47 +08:00
2026-07-05 09:54:55 +08:00
2026-03-29 15:31:18 +08:00
2026-07-04 11:27:19 +08:00
2026-05-07 16:15:15 +08:00
2026-06-12 15:24:58 +08:00
2026-07-15 11:21:04 +08:00
2026-07-17 19:16:42 +08:00
2026-06-08 07:59:50 +08:00
2026-07-16 08:29:30 +08:00
2026-07-13 17:57:34 +08:00
2026-06-02 22:43:22 +08:00
2026-06-05 15:21:41 +08:00
2026-07-12 00:17:11 +08:00
2026-07-04 23:09:29 +08:00
2026-06-02 22:43:22 +08:00
2026-07-01 22:49:17 +08:00
2026-07-13 19:26:07 +08:00
2026-07-04 09:21:19 +08:00
2026-05-26 23:30:24 +08:00
2026-07-04 21:31:01 +08:00
2026-07-21 11:46:33 +08:00
2026-06-12 17:16:01 +08:00
2026-06-09 15:29:40 +08:00
2026-07-20 23:00:44 +08:00
2026-06-12 15:24:58 +08:00
2026-06-13 23:27:17 +08:00
2026-03-31 22:19:27 +08:00
2026-07-02 11:00:49 +08:00
2026-07-13 19:26:07 +08:00
2026-07-16 09:39:09 +08:00
2026-07-19 18:45:52 +08:00
2026-04-26 11:09:56 +08:00
2026-04-08 17:13:39 +08:00
2026-07-05 09:54:55 +08:00
2026-07-21 11:46:33 +08:00
2026-07-14 10:43:13 +08:00
2026-07-17 11:18:20 +08:00
2026-07-02 08:24:15 +08:00
2026-07-04 21:31:01 +08:00
2026-07-19 22:12:30 +08:00
2026-07-14 11:57:48 +08:00
2026-05-21 21:18:10 +08:00
2026-05-11 17:38:37 +08:00
2026-07-04 11:27:19 +08:00
2026-07-15 20:25:41 +08:00
2026-06-07 11:24:30 +08:00
2026-07-11 16:05:30 +08:00
2026-07-18 17:50:20 +08:00
2026-07-17 14:49:51 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-19 22:08:48 +08:00
2026-07-04 22:56:24 +08:00
2026-07-14 22:56:50 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-16 09:12:29 +08:00
2026-07-20 08:01:18 +08:00
2026-07-15 22:30:34 +08:00
2026-07-15 20:21:04 +08:00
2026-06-21 15:55:10 +08:00
2026-07-17 14:49:51 +08:00
2026-03-14 22:55:52 +08:00
2026-07-06 07:21:12 +08:00
2026-07-14 10:43:13 +08:00
2026-07-06 22:34:41 +08:00
2026-07-10 11:08:30 +08:00
2026-03-26 14:20:18 +08:00
2026-06-28 16:10:44 +08:00
2026-07-08 17:10:27 +08:00
2026-07-14 11:04:11 +08:00
2026-07-04 13:54:04 +08:00
2026-06-17 15:29:22 +08:00
2026-05-19 22:11:46 +08:00
2026-06-08 23:25:29 +08:00
2026-07-12 11:51:31 +08:00
2026-06-10 16:11:00 +08:00
2026-07-21 09:39:44 +08:00
2026-07-17 23:22:16 +08:00
2026-07-21 11:46:33 +08:00
2026-07-21 08:07:49 +08:00
2026-04-11 16:52:10 +08:00
2026-07-20 23:33:09 +08:00
2026-07-09 17:21:25 +08:00
2026-07-03 07:51:22 +08:00
2026-07-13 00:01:22 +08:00
2026-07-04 13:54:04 +08:00
2026-07-13 00:01:22 +08:00
2026-07-04 13:54:04 +08:00
2026-07-03 08:01:46 +08:00
2026-04-30 22:27:14 +08:00
2026-05-07 16:15:15 +08:00
2026-07-19 17:13:44 +08:00
2026-07-16 08:19:52 +08:00
2026-07-19 22:12:30 +08:00
2026-07-03 15:39:12 +08:00
2026-07-09 16:26:31 +08:00
2026-04-25 09:18:13 +08:00
2026-07-04 13:54:04 +08:00
2026-07-12 11:59:36 +08:00
2026-07-10 07:54:36 +08:00
2026-07-09 16:26:31 +08:00
2026-07-09 16:26:31 +08:00
2026-07-14 09:49:43 +08:00
2026-07-04 09:40:05 +08:00
2026-06-25 22:57:00 +08:00
2026-07-14 09:49:43 +08:00
2026-07-21 09:36:09 +08:00
2026-07-15 20:25:41 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 20:25:41 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 11:21:04 +08:00
2026-07-21 09:36:09 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 11:21:04 +08:00
2026-04-26 11:09:55 +08:00
2026-07-16 14:12:02 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 11:21:04 +08:00
2026-07-20 07:24:34 +08:00
2026-07-16 08:19:52 +08:00
2026-07-04 13:54:04 +08:00
2026-07-17 14:23:44 +08:00
2026-03-22 20:39:26 +08:00
2026-07-14 09:32:53 +08:00
2026-07-14 08:58:06 +08:00
2026-07-21 11:46:33 +08:00
2026-07-16 08:19:52 +08:00
2026-06-22 18:55:20 +08:00
2026-07-18 17:48:42 +08:00
2026-06-21 15:47:53 +08:00
2026-05-28 08:20:52 +08:00
2026-07-11 16:05:30 +08:00
2026-07-04 21:44:22 +08:00
2026-05-28 18:24:32 +08:00
2026-06-08 22:04:38 +08:00
2026-03-26 15:07:05 +08:00
2026-06-09 18:01:43 +08:00
2026-06-12 15:24:58 +08:00
2026-06-24 13:57:12 +08:00
2026-07-16 08:19:52 +08:00
2026-06-24 13:57:12 +08:00
2026-07-16 08:19:52 +08:00
2026-06-24 13:57:12 +08:00
2026-06-24 13:57:12 +08:00
2026-07-16 08:19:52 +08:00
2026-07-21 08:52:31 +08:00
2026-06-24 13:57:12 +08:00
2026-07-19 17:01:28 +08:00
2026-05-21 16:22:09 +08:00
2026-04-26 11:13:01 +08:00
2026-07-20 07:24:34 +08:00
2026-04-10 20:40:30 +08:00
2026-07-09 23:25:38 +08:00
2026-07-14 07:51:59 +08:00
2026-07-21 09:23:14 +08:00
2026-07-06 20:57:41 +08:00
2026-07-11 23:20:53 +08:00
2026-06-17 21:11:44 +08:00
2026-03-14 22:55:52 +08:00
2026-03-14 22:55:52 +08:00
2026-07-03 08:01:46 +08:00
2026-07-14 07:51:59 +08:00
2026-06-08 17:32:41 +08:00
2026-06-09 21:58:31 +08:00
2026-07-14 23:08:33 +08:00
2026-07-11 15:58:07 +08:00
2026-07-14 21:37:47 +08:00
2026-06-24 17:47:04 +08:00
2026-07-02 07:43:09 +08:00
2026-04-26 15:09:25 +08:00
2026-07-12 13:03:02 +08:00
2026-07-16 09:39:09 +08:00
2026-07-16 09:39:09 +08:00
2026-07-16 08:19:52 +08:00
2026-05-21 14:48:28 +08:00
2026-07-14 07:37:59 +08:00
2026-07-13 19:26:07 +08:00
2026-07-20 07:54:15 +08:00
2026-06-24 10:24:36 +08:00
2026-07-03 07:46:04 +08:00
2026-07-19 22:12:30 +08:00
2026-05-22 21:20:28 +08:00
2026-05-22 21:20:28 +08:00
2026-05-22 21:20:28 +08:00
2026-06-25 22:43:28 +08:00
2026-06-03 22:02:27 +08:00
2026-07-14 23:02:26 +08:00
2026-04-09 17:21:23 +08:00
2026-07-21 09:09:40 +08:00
2026-07-21 09:09:40 +08:00
2026-07-13 18:34:41 +08:00
2026-06-25 08:06:17 +08:00
2026-07-12 11:32:41 +08:00
2026-07-12 23:42:58 +08:00
2026-07-04 13:54:04 +08:00
2026-06-12 10:20:38 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-06-12 09:45:26 +08:00
2026-07-20 08:18:23 +08:00
2026-06-19 11:23:07 +08:00
2026-03-26 14:20:18 +08:00
2026-07-21 08:15:33 +08:00
2026-06-27 23:30:54 +08:00
2026-07-07 23:09:06 +08:00
2026-06-21 16:08:25 +08:00
2026-05-12 23:18:54 +08:00
2026-06-14 10:40:16 +08:00
2026-07-05 19:55:39 +08:00
2026-06-19 21:57:03 +08:00
2026-07-04 23:09:29 +08:00
2026-07-04 11:27:19 +08:00
2026-07-04 11:27:19 +08:00
2026-07-04 23:09:29 +08:00
2026-07-20 08:18:23 +08:00
2026-07-13 10:40:22 +08:00
2026-05-28 15:48:32 +08:00
2026-07-17 14:39:35 +08:00
2026-07-16 14:12:02 +08:00
2026-07-11 15:58:07 +08:00
2026-07-11 15:58:07 +08:00
2026-07-04 13:54:04 +08:00
2026-07-16 14:12:02 +08:00
2026-03-25 16:28:33 +08:00
2026-07-16 14:12:02 +08:00
2026-07-07 21:44:28 +08:00
2026-07-19 22:36:15 +08:00
2026-07-16 14:12:02 +08:00
2026-07-04 23:09:29 +08:00
2026-07-17 23:22:16 +08:00
2026-07-21 09:39:44 +08:00
2026-05-07 08:34:43 +08:00
2026-07-21 00:02:16 +08:00
2026-06-18 10:09:46 +08:00
2026-07-08 21:58:33 +08:00
2026-07-16 14:12:02 +08:00
2026-07-13 19:33:40 +08:00
2026-03-30 16:02:23 +08:00
2026-06-12 15:24:58 +08:00
2026-07-12 23:04:59 +08:00
2026-07-11 22:46:10 +08:00
2026-06-14 10:30:45 +08:00
2026-06-08 23:17:22 +08:00
2026-07-14 08:14:21 +08:00
2026-07-17 14:49:51 +08:00
2026-07-13 19:26:07 +08:00
2026-07-14 22:51:21 +08:00
2026-05-02 06:37:49 +08:00
2026-07-11 15:41:57 +08:00
2026-07-11 15:41:57 +08:00
2026-07-13 19:26:07 +08:00
2026-06-19 21:42:42 +08:00
2026-07-08 17:10:27 +08:00
2026-07-15 11:21:04 +08:00
2026-07-07 19:55:33 +08:00
2026-07-19 07:39:52 +08:00
2026-06-24 07:28:17 +08:00
2026-07-17 14:49:51 +08:00
2026-05-21 21:18:10 +08:00
2026-06-18 10:15:31 +08:00
2026-07-11 15:53:14 +08:00
2026-07-10 22:13:50 +08:00
2026-06-21 21:43:43 +08:00
2026-07-14 10:43:13 +08:00
2026-06-12 15:24:58 +08:00
2026-07-14 21:37:47 +08:00
2026-07-04 09:21:19 +08:00
2026-07-18 22:12:57 +08:00
2026-06-02 22:43:22 +08:00
2026-07-14 23:02:26 +08:00
2026-05-22 21:42:14 +08:00
2026-07-21 11:46:33 +08:00
2026-07-08 14:41:16 +08:00
2026-07-15 11:21:04 +08:00
2026-03-14 22:55:52 +08:00
2026-07-11 23:15:03 +08:00
2026-06-12 22:48:47 +08:00
2026-07-08 22:26:06 +08:00
2026-07-04 23:14:46 +08:00
2026-05-10 22:00:57 +08:00
2026-06-23 23:37:41 +08:00
2026-03-26 14:20:18 +08:00
2026-05-10 23:09:03 +08:00
2026-05-19 22:11:46 +08:00
2026-07-21 09:39:44 +08:00
2026-07-04 11:27:19 +08:00
2026-07-16 16:41:04 +08:00
2026-04-06 15:09:57 +08:00
2026-07-06 20:57:41 +08:00
2026-07-06 20:57:41 +08:00
2026-07-12 11:24:24 +08:00
2026-07-14 09:49:43 +08:00
2026-07-19 09:13:51 +08:00
2026-07-19 09:13:51 +08:00
2026-07-14 08:58:06 +08:00
2026-07-14 08:58:06 +08:00
2026-07-03 08:01:46 +08:00
2026-07-20 23:56:37 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-07-20 23:56:37 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-07-15 22:27:00 +08:00
2026-06-25 20:10:49 +08:00
2026-07-20 23:56:37 +08:00
2026-05-12 23:18:54 +08:00
2026-07-16 14:44:22 +08:00
2026-07-01 16:43:56 +08:00
2026-07-14 10:43:13 +08:00
2026-07-19 17:24:31 +08:00
2026-04-26 15:09:25 +08:00
2026-07-16 14:12:02 +08:00
2026-07-11 09:18:12 +08:00
2026-07-14 10:43:13 +08:00
2026-05-02 15:16:16 +08:00
2026-06-09 15:29:40 +08:00
2026-07-14 23:02:26 +08:00
2026-07-14 23:02:26 +08:00
2026-04-06 15:18:34 +08:00
2026-07-04 09:21:19 +08:00
2026-07-06 07:21:12 +08:00
2026-07-20 23:05:47 +08:00
2026-07-18 17:53:30 +08:00
2026-07-12 07:52:45 +08:00
2026-07-19 09:13:51 +08:00
2026-07-12 11:51:31 +08:00
2026-07-21 11:46:33 +08:00
2026-06-07 17:47:11 +08:00