fix(voice-input): separate transcript segments for providers that trim final results (#6432)

* fix(voice-input): separate transcript segments for providers that trim final results

readSpeechRecognitionTranscript concatenated final and interim
transcripts with no separator, relying on providers padding final
results with trailing whitespace. WebKit's SpeechRecognition
implementation trims them, so the last finalized word glued onto the
interim hypothesis ("hello world" + "today is" -> "hello worldtoday
is"), both in the live composer and in the submitted message.

Join every adjacent segment pair with an explicit space unless either
side of the boundary is a no-space script or its punctuation (Han, kana,
CJK symbols, full-width forms) — those keep the plain concatenation
Chinese and Japanese dictation always had. Korean is treated as a
space-delimited script, matching its eojeol spacing.

* fix(voice-input): consolidate transcript boundary regressions

Port the remaining regression cases from #6449, retaining equivalent English and Hangul fixtures already present in #6432. Attach leading punctuation and percent signs without losing spacing after sentence punctuation, quoted phrases, or combining accents. Pin whitespace-only segments and document the join contract.

Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

README.md

@@ -1714,7 +1714,7 @@ Unsent Web UI composer drafts survive page reloads and switching between convers
 
 The Web UI composer supports `@` references at the cursor or through the `@` button. Search enabled skills, documents in the current project, and other conversations in a grouped picker; selected references appear inline alongside the draft text and can be removed with normal editing. Multiple skills can be selected in one message (up to 16 unique skills); selecting a checked skill again removes it, with each selection checked against the current user registry and agent allowlist. Project files are attached only after ingestion succeeds; conversation references require the `read_conversation` capability and retain its per-run limit. If capability discovery fails, saved conversation references remain intact and sending waits for a successful retry. Polishing preserves complete reference labels when instructions are reordered. The file group also offers local uploads. `/goal`, `/compact`, and legacy `/skill-name` input remain supported.
 
-The Web UI composer also supports browser-based voice dictation when the browser exposes the Web Speech API. The microphone button transcribes speech into the local draft only; DeerFlow receives only the transcribed text, while audio handling is delegated to the browser or operating system speech-recognition service according to that environment's policy. Users can review or edit the text before sending.
+The Web UI composer also supports browser-based voice dictation when the browser exposes the Web Speech API. The microphone button transcribes speech into the local draft only; DeerFlow receives only the transcribed text, while audio handling is delegated to the browser or operating system speech-recognition service according to that environment's policy. Users can review or edit the text before sending. Dictation keeps words separated when browsers trim transcript segments, preserves Chinese and Japanese joins, and attaches leading punctuation and percent signs without adding spaces.
 
 The Web UI displays a localized AI-generated-content disclaimer below the composer in both standard and custom-agent conversations, reminding users to verify important
 information.

frontend/src/AGENTS.md

@@ -23,7 +23,7 @@ Other preview helpers retain the node test environment.
 Language-map membership checks only own properties; inherited names such as
 `constructor` and `__proto__` must keep the unknown-file download fallback.
 
-1. Optional composer helpers such as `core/input-polish` can rewrite the local draft before submission, and `core/voice-input` can transcribe browser microphone input into that same local draft; confirmed user input then flows to thread hooks (`core/threads/hooks.ts`) → LangGraph SDK streaming
+1. Optional composer helpers such as `core/input-polish` can rewrite the local draft before submission, and `core/voice-input` can transcribe browser microphone input into that same local draft; confirmed user input then flows to thread hooks (`core/threads/hooks.ts`) → LangGraph SDK streaming. Voice transcript joining separates every adjacent pair of trimmed word segments, preserves CJK joins and provider whitespace, and attaches leading punctuation/percent signs while retaining spaces after sentence punctuation. Regression cases live in `tests/unit/core/voice-input/speech-recognition.test.ts`.
 2. Stream events update thread state (messages, artifacts, todos, goal). The main thread stream uses the LangGraph SDK's `throttle: true` mode so updates received in the same macrotask coalesce before React is notified; do not replace it with a numeric delay without validating the SDK's trailing-debounce behavior on a continuous stream.
    File-tool artifact auto-open work must run in an effect with timer cleanup; never schedule timers while rendering streamed `write_file` or `str_replace` updates.
    `ThreadState.artifacts` remains the authoritative artifact list. The artifacts provider persists only thread-scoped panel UI state (`open`, selected path, and a refresh bootstrap cache) in session storage; an initial empty stream value must not overwrite that restored state before history finishes loading.

frontend/src/core/voice-input/speech-recognition.ts

@@ -103,11 +103,49 @@ export function shouldRestartSpeechRecognition(
   return lastError === null || lastError === "no_speech";
 }
 
+// Scripts written without spaces between words and the punctuation that
+// surrounds them: Han (Chinese), kana (Japanese), CJK symbols/punctuation,
+// and their full-width/half-width compatibility forms. Korean Hangul is
+// deliberately absent — Korean uses eojeol spaces, so it is joined like any
+// space-delimited script. Boundaries compare single UTF-16 code units, so
+// supplementary-plane Han (CJK Ext B+) never matches; fine for ASR output.
+const NO_SPACE_SCRIPT_BOUNDARY =
+  /[\u3000-\u303f\u3040-\u30ff\u31f0-\u31ff\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\uff00-\uff9f]/;
+
+// These leading marks attach to the preceding text. Keep the rule directional:
+// a sentence-ending mark on the left still needs a space before the next word.
+const NO_SPACE_BEFORE_PUNCTUATION = /[,.;:!?%…)\]}]/;
+
+function joinTranscriptSegments(left: string, right: string): string {
+  if (!left) {
+    return right;
+  }
+  if (!right) {
+    return left;
+  }
+  const lastChar = left.trimEnd().slice(-1);
+  const firstChar = right.trimStart().slice(0, 1);
+  // Space-delimited scripts need an explicit separator when a provider does
+  // not pad its results (WebKit trims final transcripts), otherwise the last
+  // finalized word glues onto the next segment. No-space scripts (Chinese,
+  // Japanese) keep the plain concatenation they always had.
+  if (
+    !lastChar ||
+    !firstChar ||
+    NO_SPACE_SCRIPT_BOUNDARY.test(lastChar) ||
+    NO_SPACE_SCRIPT_BOUNDARY.test(firstChar) ||
+    NO_SPACE_BEFORE_PUNCTUATION.test(firstChar)
+  ) {
+    return `${left}${right}`;
+  }
+  return `${left} ${right}`;
+}
+
 export function readSpeechRecognitionTranscript(
   results: SpeechRecognitionResultListLike,
 ): { finalText: string; interimText: string; text: string } {
-  let finalText = "";
-  let interimText = "";
+  const finalParts: string[] = [];
+  const interimParts: string[] = [];
 
   for (const result of Array.from(
     { length: results.length },
@@ -115,16 +153,29 @@ export function readSpeechRecognitionTranscript(
   )) {
     const transcript = result?.[0]?.transcript ?? "";
     if (result?.isFinal) {
-      finalText += transcript;
+      finalParts.push(transcript);
     } else {
-      interimText += transcript;
+      interimParts.push(transcript);
     }
   }
 
+  let finalText = "";
+  for (const part of finalParts) {
+    finalText = joinTranscriptSegments(finalText, part);
+  }
+  let interimText = "";
+  for (const part of interimParts) {
+    interimText = joinTranscriptSegments(interimText, part);
+  }
+  let text = "";
+  for (const part of [...finalParts, ...interimParts]) {
+    text = joinTranscriptSegments(text, part);
+  }
+
   return {
     finalText: normalizeSpeechTranscript(finalText),
     interimText: normalizeSpeechTranscript(interimText),
-    text: normalizeSpeechTranscript(`${finalText}${interimText}`),
+    text: normalizeSpeechTranscript(text),
   };
 }
 

frontend/tests/unit/core/voice-input/speech-recognition.test.ts

@@ -57,6 +57,197 @@ describe("speech recognition helpers", () => {
     });
   });
 
+  it("separates unpadded final and interim segments", () => {
+    // WebKit-style providers trim final transcripts; without an explicit
+    // separator the finalized text glues onto the interim hypothesis.
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "hello world" } },
+        1: { isFinal: false, length: 1, 0: { transcript: "today is" } },
+      }),
+    ).toEqual({
+      finalText: "hello world",
+      interimText: "today is",
+      text: "hello world today is",
+    });
+  });
+
+  it("separates consecutive unpadded final segments", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "hello" } },
+        1: { isFinal: true, length: 1, 0: { transcript: "world" } },
+      }),
+    ).toEqual({
+      finalText: "hello world",
+      interimText: "",
+      text: "hello world",
+    });
+  });
+
+  it("adds no separator when the provider already emitted boundary space", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "hello world " } },
+        1: { isFinal: false, length: 1, 0: { transcript: "today" } },
+      }).text,
+    ).toBe("hello world today");
+  });
+
+  it("keeps CJK segments glued when the provider emits no spaces", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "你好世界" } },
+        1: { isFinal: false, length: 1, 0: { transcript: "今天" } },
+      }).text,
+    ).toBe("你好世界今天");
+  });
+
+  it("separates consecutive trimmed finals accumulated over a session", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 3,
+        0: { isFinal: true, length: 1, 0: { transcript: "hello" } },
+        1: { isFinal: true, length: 1, 0: { transcript: "world" } },
+        2: { isFinal: false, length: 1, 0: { transcript: "today" } },
+      }),
+    ).toEqual({
+      finalText: "hello world",
+      interimText: "today",
+      text: "hello world today",
+    });
+  });
+
+  it.each(["cet été", "cafe\u0301"])(
+    "separates accented Latin words after %s",
+    (finalText) => {
+      expect(
+        readSpeechRecognitionTranscript({
+          length: 2,
+          0: { isFinal: true, length: 1, 0: { transcript: finalText } },
+          1: { isFinal: false, length: 1, 0: { transcript: "demain" } },
+        }).text,
+      ).toBe(`${finalText} demain`);
+    },
+  );
+
+  it("keeps digit and symbol boundaries glued", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "up to 50" } },
+        1: { isFinal: false, length: 1, 0: { transcript: "%" } },
+      }).text,
+    ).toBe("up to 50%");
+  });
+
+  it("attaches leading punctuation across final and interim segments", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 4,
+        0: { isFinal: true, length: 1, 0: { transcript: "done" } },
+        1: { isFinal: true, length: 1, 0: { transcript: ", maybe" } },
+        2: { isFinal: false, length: 1, 0: { transcript: "next" } },
+        3: { isFinal: false, length: 1, 0: { transcript: ", please" } },
+      }),
+    ).toEqual({
+      finalText: "done, maybe",
+      interimText: "next, please",
+      text: "done, maybe next, please",
+    });
+  });
+
+  it.each([
+    ["Done.", "Next", "Done. Next"],
+    ["Hello,", "world", "Hello, world"],
+    ["she said", '"hello"', 'she said "hello"'],
+    ['"hello"', "again", '"hello" again'],
+  ])("separates phrases at %s / %s", (left, right, text) => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: left } },
+        1: { isFinal: false, length: 1, 0: { transcript: right } },
+      }).text,
+    ).toBe(text);
+  });
+
+  it("preserves separation across whitespace-only final segments", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 3,
+        0: { isFinal: true, length: 1, 0: { transcript: "hello" } },
+        1: { isFinal: true, length: 1, 0: { transcript: " " } },
+        2: { isFinal: true, length: 1, 0: { transcript: "world" } },
+      }),
+    ).toEqual({
+      finalText: "hello world",
+      interimText: "",
+      text: "hello world",
+    });
+  });
+
+  it("keeps no-space scripts concatenated without inserting separators", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 3,
+        0: { isFinal: true, length: 1, 0: { transcript: "你好" } },
+        1: { isFinal: true, length: 1, 0: { transcript: "世界" } },
+        2: { isFinal: false, length: 1, 0: { transcript: "早上好" } },
+      }),
+    ).toEqual({
+      finalText: "你好世界",
+      interimText: "早上好",
+      text: "你好世界早上好",
+    });
+  });
+
+  it("keeps mixed-script boundaries concatenated", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "你好" } },
+        1: { isFinal: false, length: 1, 0: { transcript: "hello" } },
+      }),
+    ).toEqual({
+      finalText: "你好",
+      interimText: "hello",
+      text: "你好hello",
+    });
+  });
+
+  it("keeps CJK clause-final punctuation boundaries concatenated", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "こんにちは、" } },
+        1: { isFinal: false, length: 1, 0: { transcript: "iPhone" } },
+      }),
+    ).toEqual({
+      finalText: "こんにちは、",
+      interimText: "iPhone",
+      text: "こんにちは、iPhone",
+    });
+  });
+
+  it("joins Korean segments with a space like other space-delimited scripts", () => {
+    expect(
+      readSpeechRecognitionTranscript({
+        length: 2,
+        0: { isFinal: true, length: 1, 0: { transcript: "안녕" } },
+        1: { isFinal: false, length: 1, 0: { transcript: "하세요" } },
+      }),
+    ).toEqual({
+      finalText: "안녕",
+      interimText: "하세요",
+      text: "안녕 하세요",
+    });
+  });
+
   it("appends transcript to an existing draft without duplicating whitespace", () => {
     expect(appendSpeechTranscript("", "  hello  world ")).toBe("hello world");
     expect(appendSpeechTranscript("Draft", "voice text")).toBe(