* feat(frontend): allow chat replies during clarification
* fix(frontend): unlock input polish during clarification
Remove hasOpenHumanInputCard from inputPolishDisabled so the polish
button stays available when a clarification card is open, matching
the composer unlock behavior. Clean up the now-unused useMemo and
import.
* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(frontend): add structured human input cards for ask_clarification
Implement a reusable Human Input Card flow for ask_clarification while keeping
the existing text fallback for older clients and IM channels.
Backend:
- Add structured ToolMessage.artifact.human_input payloads for clarification requests.
- Preserve ToolMessage.content as the readable Markdown/text fallback.
- Normalize clarification options from native lists, JSON strings, plain strings,
mixed scalar values, None, and missing options.
- Derive input_mode as choice_with_other when options exist, otherwise free_text.
- Keep disable_clarification non-interactive behavior as a plain ToolMessage with
no human_input artifact.
- Cover artifact persistence and Gateway message metadata preservation in tests.
Frontend:
- Add human input protocol types, runtime guards, extractors, response builders,
and thread-state helpers.
- Add reusable HumanInputCard with option buttons, free-text input, pending,
read-only, disabled, and answered states.
- Render structured clarification cards from artifact.human_input, with Markdown
fallback for malformed or legacy tool messages.
- Preserve line breaks in structured question/context/option text.
- Hide submitted clarification bridge messages from the chat UI via
additional_kwargs.hide_from_ui.
- Send structured human_input_response metadata through the fourth sendMessage
options argument, preserving run context in the third argument.
- Wire submissions for normal chats, custom agent chats, agent bootstrap chats,
and sidecar chats.
- Derive answered state from raw thread.messages so hidden replies still update
the original card.
- Clear pending state when the hidden reply arrives, dispatch is dropped, or a
later async stream failure appears on thread.error.
* perf(frontend): optimize HumanInputCard UI interactions
- Support Enter key to submit text input (Shift+Enter for newline)
- Render question and context fields as Markdown instead of plain text
- Replace deprecated FormEventHandler type with structural typing
* test(frontend): add unit test cover optimize HumanInputCard UI interactions
* feat(frontend): disabled chatbox when has new human-input-card
* fix(style): lint error fix
* fix: sanitize hidden human input replies
- Preserve IME composition safety for human input card Enter submits
- Treat hidden human input responses as genuine user messages for sanitization
- Keep hidden card replies in memory filtering while excluding malformed/internal hidden messages
- Add regression coverage for card IME handling and hidden reply sanitization
* fix: tighten human input response validation
- Reject empty hidden human input response values
- Remove invalid list ARIA role from human input card options
- Add backend coverage for empty response payloads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sidecar): make panel button delete the side chat instead of hiding it
The side chat panel's top-right button called `sidecar.close()`, which only
hid the panel — duplicating the header `SidecarTrigger` toggle. Repurpose the
button so the panel owns deletion and the header toggle owns hide/show.
- When a conversation exists, the button shows a trash icon and opens a
confirmation dialog before deleting via `useDeleteThread` (backend cascade).
- In the draft state (references only, no thread yet) the header trigger is not
rendered, so the button falls back to a plain close (X) that discards the
draft without a confirm — there is nothing persisted to delete.
Also fix a latent double-delete bug surfaced by the new dialog:
`useDeleteThread` deletes through the LangGraph route (which drops the
thread_meta row) and then calls `deleteLocalThreadData`, which hits the same
gateway handler whose `require_existing=True` guard now 404s. Treat that 404 as
idempotent success. The recent-chat delete used fire-and-forget `mutate`, so it
swallowed this error; the sidecar's `mutateAsync` surfaced it as a toast.
The e2e mock now mirrors the gateway ownership guard (second DELETE 404s once
the thread is gone) so the regression stays covered. Adds tests for delete,
draft-state close, and header-owned hide/show.
* fix(sidecar): lock delete dialog while deletion is in flight
The delete confirmation dialog disabled Cancel during an in-flight
delete but still allowed dismissal via the overlay, Esc, and the
built-in close (X). Those paths could imply the delete was cancelled
when it was actually still running. Ignore dismissals and drop the
close button while `isDeleting` so only the (disabled) Cancel path
controls the dialog.
* fix(sidecar): correct text-selection toolbar actions inside the side chat
The sidecar panel reuses MessageList, but it did not distinguish the side
chat surface from the main conversation, so selecting text inside the side
chat behaved as if it were the main list:
- The "Ask in side chat" action was shown even though the user is already
in the side chat, which is a no-op interaction.
- "Add to conversation" routed the snippet to the main composer's quotes
(conversationQuotes) instead of the side chat's own composer, so the
reference landed in the wrong input box.
Add a `sidecarSurface` prop to MessageList. On the sidecar surface, hide
"Ask in side chat" and route "Add to conversation" to `sidecar.openContext`
so the snippet attaches to the side chat's own composer (activeReferences).
Main-list behavior is unchanged.
* docs(sidecar): drop AGENTS.md note for the toolbar surface change
The sidecarSurface behavior is self-evident from the code; no dedicated
Interaction Ownership entry is needed.
* feat(frontend): add side conversations for quoted follow-ups
* style(frontend): apply prettier formatting to sidecar-chat files
* fix(frontend): surface sidecar cascade cleanup failures via console.warn
Previously deleteSidecarThreadsForParent silently swallowed both
lookup errors and per-thread deletion failures, so parent thread
deletions could succeed while orphaning sidecar threads with no
signal to the caller. Log a warning that includes the parent id
and the failed thread ids/reasons so the leak is discoverable in
telemetry, matching the existing console.warn/error pattern in
this file.
* fix(frontend): address all sidecar review feedback
Resolve every reviewer comment on PR #3934:
- input-box/hooks/sidecar-panel: clear quoted references only via an
`onSent` callback that fires after the in-flight guard, so a dropped
send no longer silently discards quotes (willem-bd #3550).
- message-list: flip the selection toolbar below the selection when it
would clip above the viewport (willem-bd #3551).
- reference-metadata/thread/input-box: keep referenced ids, roles, and
count arrays 1:1 parallel instead of deduping ids (willem-bd #3552).
- message-list: widen selection containment to the shared assistant-turn
container and hint when a selection crosses messages (willem-bd #3553).
- sidecar/api: coalesce concurrent sidecar creates for one parent behind
a single in-flight promise to prevent duplicates (willem-bd #3554).
- sidecar-trigger/context: force-restore on trigger click so a sidecar
deleted elsewhere self-heals instead of opening a dead thread
(willem-bd #3555).
- threads/hooks: surface sidecar cascade cleanup failures via
console.warn for both lookup and per-thread deletes (Copilot).
Add unit + e2e coverage for parallel metadata, atomic create, and
trigger self-healing.
