* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): harden artifact and markdown rendering
* fix(#4117): restore allow-scripts for scroll-restore, fix citation XSS bypass
willem-bd identified these issues:
1. [REGRESSION] sandbox removed allow-scripts which broke HTML artifact
scroll-restoration — the injected postMessage script could not run.
The prior config (allow-scripts allow-forms without allow-same-origin,
i.e. opaque origin) was already safe. Restored with corrected comment.
2. [XSS bypass] CitationLink rendered <a href={href}> directly before
isSafeHref ran, so prompt-injected [citation:x](javascript:...)
bypassed the safety check. Moved citation block after the guard.
Also:
- Added mailto: and tel: to SAFE_HREF_PROTOCOLS
- Kept anchor-only attributes off the <span> fallback (React DOM warning)
Note: the loadMessages re-throw originally in this commit was dropped
during the rebase — upstream #4065 rewrote useThreadHistory as a
TanStack useInfiniteQuery that already surfaces fetch failures.
* test(e2e): update sandbox assertion to match allow-scripts allow-forms
The artifact preview iframe's sandbox was restored to 'allow-scripts allow-forms'
for scroll-restoration. Update the E2E test to expect the corrected value.
* fix(#4117): address review — ArtifactLink XSS guard, urlOfArtifact sandbox e2e
- Apply the isSafeHref guard to ArtifactLink (artifact-link.tsx) so a
prompt-injected javascript:/data: href in a .md artifact preview cannot
reach a real anchor in the main document, matching the guard in
createMarkdownLinkComponent.
- Add an e2e asserting the urlOfArtifact iframe (non-code
browser-previewable files like PDF) keeps its empty sandbox.
Note: the loadMessages error-surfacing changes originally in this commit
were dropped during the rebase — upstream #4065 rewrote useThreadHistory
as a TanStack useInfiniteQuery whose queryFn already throws on
!response.ok and surfaces the failure via a toast.
* fix(frontend): let write_file non-code previewable artifacts render in sandboxed iframe
When a write_file artifact such as PDF is clicked in chat, the component receives a path that forced isCodeFile=true, hiding the sandboxed iframe behind the code editor. Now non-code browser-previewable files are detected early so the sandboxed iframe renders correctly.
Fixes the E2E test: renders sandboxed iframe for a browser-previewable non-code file.
* fix(#4117): align PDF artifact route pattern with convention (drop /mock/ prefix)
The test used /mock/api/threads/... for the PDF artifact content route,
but the urlOfArtifact helper generates /api/threads/... (without /mock/)
when isMock=false. The other tests (e.g. presented artifacts) already use
the correct /api/threads/... pattern.
* test(frontend): add render-level coverage for unsafe markdown/artifact links
- Render MarkdownLink and ArtifactLink via renderToStaticMarkup and
assert an unsafe javascript: href produces a disabled <span> (never an
<a>), including through the citation-labelled branch, and that safe
https hrefs render hardened anchors (target=_blank, rel=noopener).
- The new ArtifactLink render test caught the unsafe href leaking onto
the fallback <span> through the {...rest} spread; drop the spread in
both span fallbacks so anchor-only attributes (href/target/rel) and
react-markdown's node prop never reach the span.
- Cover mailto:/tel: in the isSafeHref unit cases and fix the stale
SAFE_HREF_PROTOCOLS docstring that still claimed http/https-only.
* fix(frontend): route the agent save-hint through the safe localStorage facade
Export safeLocalStorage from core/settings/local and use it for the
agent-create save-hint read/write so blocked browser storage (Safari
private mode, strict containers, embedded WebViews) cannot throw from
the effect. core/agents/feature-cache.ts already guards its
localStorage access with try/catch, so settings + agent pages are now
consistently best-effort.
* fix(frontend): allow scheme-less relative markdown links in isSafeHref
* feat(frontend): add structured human input cards for ask_clarification
Implement a reusable Human Input Card flow for ask_clarification while keeping
the existing text fallback for older clients and IM channels.
Backend:
- Add structured ToolMessage.artifact.human_input payloads for clarification requests.
- Preserve ToolMessage.content as the readable Markdown/text fallback.
- Normalize clarification options from native lists, JSON strings, plain strings,
mixed scalar values, None, and missing options.
- Derive input_mode as choice_with_other when options exist, otherwise free_text.
- Keep disable_clarification non-interactive behavior as a plain ToolMessage with
no human_input artifact.
- Cover artifact persistence and Gateway message metadata preservation in tests.
Frontend:
- Add human input protocol types, runtime guards, extractors, response builders,
and thread-state helpers.
- Add reusable HumanInputCard with option buttons, free-text input, pending,
read-only, disabled, and answered states.
- Render structured clarification cards from artifact.human_input, with Markdown
fallback for malformed or legacy tool messages.
- Preserve line breaks in structured question/context/option text.
- Hide submitted clarification bridge messages from the chat UI via
additional_kwargs.hide_from_ui.
- Send structured human_input_response metadata through the fourth sendMessage
options argument, preserving run context in the third argument.
- Wire submissions for normal chats, custom agent chats, agent bootstrap chats,
and sidecar chats.
- Derive answered state from raw thread.messages so hidden replies still update
the original card.
- Clear pending state when the hidden reply arrives, dispatch is dropped, or a
later async stream failure appears on thread.error.
* perf(frontend): optimize HumanInputCard UI interactions
- Support Enter key to submit text input (Shift+Enter for newline)
- Render question and context fields as Markdown instead of plain text
- Replace deprecated FormEventHandler type with structural typing
* test(frontend): add unit test cover optimize HumanInputCard UI interactions
* feat(frontend): disabled chatbox when has new human-input-card
* fix(style): lint error fix
* fix: sanitize hidden human input replies
- Preserve IME composition safety for human input card Enter submits
- Treat hidden human input responses as genuine user messages for sanitization
- Keep hidden card replies in memory filtering while excluding malformed/internal hidden messages
- Add regression coverage for card IME handling and hidden reply sanitization
* fix: tighten human input response validation
- Reject empty hidden human input response values
- Remove invalid list ARIA role from human input card options
- Add backend coverage for empty response payloads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
