mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 05:06:59 +00:00
fix(citations): pair inline code within a paragraph, not across blank lines (#5876)
* fix(citations): pair inline code within a paragraph, not across blank lines maskInlineCode paired backticks over the whole document, so a stray backtick in one paragraph and another in a later one formed a span CommonMark never renders. The citation between them was masked out of the Sources list, and the stolen opener simultaneously un-masked a real code span further down. Scope the pairing to blank-line-delimited paragraphs, keeping the captured separator so string length and occurrence indices still align with the original markdown. * fix(citations): end the paragraph on a CRLF blank line too core/messages/utils.ts recognises a blank line as \r?\n[ \t]*\r?\n; the citations scanner now uses the same shape, so CRLF markdown gets the same pairing limit instead of falling back to whole-document pairing. * fix(citations): mask fences in containers and end inline spans at block bounds Follow-up to the paragraph-scoped inline pairing review on #5876. Indented fences inside a list item, and blockquote-prefixed fences, were only blanked by accident before: the whole-document backtick pairing happened to close them. Scoping the pairing to a paragraph removed that safety net and let sample citations surface as sources, so the fence matcher now accepts container prefixes and is marker-aware. Inline spans also end at the other block boundaries CommonMark uses - ATX headings, thematic breaks / setext underlines, interrupting list items and a blockquote's empty continuation line - not only at blank lines. * fix(citations): anchor fence detection and end fences with their container An unanchored container-prefix match turns any line that merely contains a three-backtick run into a fence opener, which masks everything after it and drops live citations. Anchor the match to the line start. A fence also has to end with the container it was opened in: a blockquote loses its fence at the first line without a `>` marker, and an indented fence at the first non-blank line that is less indented than its opener. Without that, the citation following a closed blockquote is masked away, which is the same failure direction as the original bug. * fix(citations): track the container a fence was opened in Separate a fence marker's own indentation from the indentation its container gives it, require whitespace after a list marker before treating it as a container, count blockquote depth instead of testing it as a boolean, and check a closer's indentation against the container before ending the block. Cover the four shapes from the review on #5876 with tests, and check the scanner against remark-parse over twenty fence shapes. * fix(citations): cap an opener's indentation at three columns past its container A marker four columns past the container's content column is an indented code block, so it must not open a fence; otherwise the citation below it is blanked to end of input instead of surfacing as a source. * fix(citations): match fence openers and closers to what their tails allow A closing fence is bare, and a backtick fence's info string carries no backtick. So ` ```text ` inside a ``` block is literal content rather than a closer, and ` ```md `x` ` is a paragraph line rather than an opener. Missing either check sends the scanner the wrong way: the first lets a citation after the fake closer surface as a phantom source, the second blanks a citation the reader can actually click. * fix(citations): measure fence columns in columns, and only close on a closer A tab after a list marker advances to the next four-column stop, so sizing a list item by the character length of its marker put the content column two columns too far left and kept a dedented citation masked inside the fence. A closing fence now comes from its own pattern instead of the opener's: the opener also accepts list markers, which are plain content once a block is open, so `- ``` ` inside a fence closed it early and surfaced the citation under it. The optional space after a block quote marker is padding, not indentation, so measuring a quoted marker from the raw prefix cost the fence one column of its three-column budget and rejected a valid closer. * test(citations): pin the list-nested closer shape as its own case * fix(citations): keep a blockquote fence escape inside an indented code run Losing the `>` closes a quoted fence, but the escaping line is not free text: four columns of indentation at the top level starts an indented code block, so its citations are still code. Without that, the paragraph-scoped inline pairing this scanner replaced leaves them visible as phantom sources. * fix(citations): read blank lines and surviving containers around quote escapes * fix(citations): count a quote that opens behind a list marker BLOCKQUOTE_PREFIX_RE matches from the start of a line, so a fence opened as "- > ```md" recorded quoteDepth 0 even though it sits inside a quote. The escape test then never fired and the fence masked to end of input, blanking citations that remark-parse renders as live links: 35 of the 924 swept container combinations were worse than main, all in that family. Read the opener's own container prefix instead, and park the list stack the quote hides in so a dedented line is measured against the item that survives it. * fix(citations): keep literal quote lines and multi-marker fences contained A fence that was not opened in a block quote has no quote container a later line can leave, so a `>` inside it is literal code: stripping it made an indented content line read as column zero and escape the fence. A line can also open several list items at once, and the fence belongs to the deepest of them, so recording only the first marker put `- - - ```md` four columns past its container and rejected a valid opener. * fix(citations): scan the container prefix in one pass Reading the containers a fence marker may sit behind was a single regex whose alternation could take a space either as a step of its own or as the padding of the next marker, so it described the same prefix exponentially many ways. A line carrying no fence therefore made the engine enumerate every decomposition: the deep-quote shapes tests/e2e/user-message-plain-text.spec.ts renders cost a second at 26 markers, roughly double for each marker added, and never return for the three thousand markers those messages carry, which is why the e2e job timed out waiting for a message that had already rendered on main. The scan now walks each position once. Which shapes open a fence is unchanged: the 32-shape and 35-shape differentials against remark-parse, the 924-combination sweep and this file's 50 existing cases all give the same answers as before. * fix(citations): keep the list a finished quote lived in, and let paragraph text block a non-1 ordered fence maskFencedCodeBlocks cleared the item stack on every blockquote-depth change, so a fence opened after a quote ended measured against the document root and hid a citation the reader had already left the item for. It also pushed any ordered marker as a container, so a `2. ~~~md` line that CommonMark keeps inside the preceding paragraph opened a fence and masked a rendered citation. * fix(citations): test the paragraph boundary on the CR-normalized line A CRLF document keeps its "\r" on every line after the split on "\n", and THEMATIC_BREAK_RE is anchored at the end of the line, so a setext underline "---\r" was not recognised as a leaf block. The paragraph it had already ended stayed live, the following "2. ```md" was read as paragraph text, its fence never opened, and the real closer was then re-read as a top-level opener that masked the rendered citation to EOF. Use the same CR-normalized line the position classification already uses, and pin the LF/CRLF pair plus the cross-depth quote shape against the parser tree. * fix(citations): keep list items and quote markers the reader keeps Three container rules in the fence scanner disagreed with remark-parse: - A dedented line of paragraph text is a lazy continuation, so it does not leave the list item. Popping the item there recorded the next fence at column zero and hid a rendered citation to end of input. - Padding after a list marker tops out at four columns: the fifth belongs to an indented code block inside an item whose content starts one column past the marker, not to a column-six container. - A `>` deeper than the quote markers a quoted fence already consumed is literal code. Stripping it made the content look dedented past the fence's column and ended the fence one line early, surfacing a citation the reader shows as code. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
lihongyuan99
GitHub
parent
582b1f5a28
commit
2ad2132336
@ -245,19 +245,556 @@ export function maskCitationCode(markdown: string): string {
|
||||
return maskInlineCode(maskFencedCodeBlocks(markdown));
|
||||
}
|
||||
|
||||
// A fence can be nested in a list item or a blockquote, so its marker may sit
|
||||
// behind container prefixes and the indentation a container gives it. Only the
|
||||
// column-0 shape was recognised before, and indented fences were blanked by
|
||||
// accident because whole-document backtick pairing happened to close them.
|
||||
// Anchored: a backtick run further into a line is inline code, not an opener.
|
||||
// A list marker only opens a container when whitespace follows it, so `-```md`
|
||||
// is paragraph text rather than a fence opener.
|
||||
//
|
||||
// The container prefix is scanned, not matched. A space can be read either as
|
||||
// padding of the marker it precedes or as a step of its own, so an alternation
|
||||
// over `>`, list markers and spaces describes the same prefix many ways: a line
|
||||
// of repeated markers with no fence after it makes the engine try them all,
|
||||
// which is exponential in the marker count and never finishes for the thousands
|
||||
// of markers one deep-quote message carries. This loop visits each position
|
||||
// once, and skipping padding before a marker loses nothing because a fence
|
||||
// marker is never a space.
|
||||
const CONTAINER_QUOTE_RE = />/y;
|
||||
const CONTAINER_LIST_RE = /(?:[-+*]|\d{1,9}[.)])[ \t]/y;
|
||||
const FENCE_OPEN_MARKER_RE = /(`{3,}|~{3,})/y;
|
||||
|
||||
type FenceOpener = { prefix: string; marker: string; consumed: number };
|
||||
|
||||
function fenceOpener(line: string): FenceOpener | null {
|
||||
let index = 0;
|
||||
for (;;) {
|
||||
while (line[index] === " " || line[index] === "\t") {
|
||||
index += 1;
|
||||
}
|
||||
CONTAINER_QUOTE_RE.lastIndex = index;
|
||||
if (CONTAINER_QUOTE_RE.test(line)) {
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
CONTAINER_LIST_RE.lastIndex = index;
|
||||
if (CONTAINER_LIST_RE.test(line)) {
|
||||
index = CONTAINER_LIST_RE.lastIndex;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
}
|
||||
FENCE_OPEN_MARKER_RE.lastIndex = index;
|
||||
const marker = FENCE_OPEN_MARKER_RE.exec(line)?.[0];
|
||||
if (marker === undefined) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
prefix: line.slice(0, index),
|
||||
marker,
|
||||
consumed: index + marker.length,
|
||||
};
|
||||
}
|
||||
|
||||
const BLOCKQUOTE_PREFIX_RE = /^(?:[ \t]*>)+/;
|
||||
const LIST_ITEM_RE = /^(?:[-+*]|\d{1,9}[.)])[ \t]+/;
|
||||
// The digits of an ordered marker, so the scanner can tell a `1.` item (which
|
||||
// may interrupt a paragraph) from a `2.` item (which may not).
|
||||
const ORDERED_MARKER_DIGITS_RE = /^(\d{1,9})[.)][ \t]+/;
|
||||
|
||||
// A closing fence is indentation and a marker and nothing else, so it cannot
|
||||
// reuse `fenceOpener`: that scan also accepts list markers, which are plain
|
||||
// content once a fence is open.
|
||||
const FENCE_CLOSER_RE = /^([ \t]*)(`{3,}|~{3,})/;
|
||||
|
||||
// Inline spans end at a block boundary, not just at a blank line: see
|
||||
// `inlineSpanStarts` for the shapes and the matching scanner in
|
||||
// core/messages/utils.ts.
|
||||
const BLANK_LINE_RE = /^(?:[ \t]*>)*[ \t]*$/;
|
||||
const ATX_HEADING_RE = /^(?:[ \t]*>)*[ \t]{0,3}#{1,6}(?:[ \t]|$)/;
|
||||
const THEMATIC_BREAK_RE =
|
||||
/^(?:[ \t]*>)*[ \t]{0,3}(?:(?:=+|-+)[ \t]*|(?:\*[ \t]*){3,}|(?:_[ \t]*){3,}|(?:-[ \t]*){3,})$/;
|
||||
const INTERRUPTING_LIST_RE = /^(?:[ \t]*>)*[ \t]{0,3}(?:[-+*]|1[.)])[ \t]+\S/;
|
||||
|
||||
const INLINE_CODE_SPAN_RE = /(`+)[\s\S]*?\1/g;
|
||||
|
||||
// Column reached by advancing from `column` through `text`, expanding a tab to
|
||||
// the next four-column stop the way CommonMark does. Character counts would make
|
||||
// `-\t` two columns wide when it really reaches column four.
|
||||
function advanceColumns(column: number, text: string): number {
|
||||
let reached = column;
|
||||
for (const char of text) {
|
||||
reached += char === "\t" ? 4 - (reached % 4) : 1;
|
||||
}
|
||||
return reached;
|
||||
}
|
||||
|
||||
function indentationColumns(text: string): number {
|
||||
let column = 0;
|
||||
for (const char of text) {
|
||||
if (char !== " " && char !== "\t") {
|
||||
break;
|
||||
}
|
||||
column += char === "\t" ? 4 - (column % 4) : 1;
|
||||
}
|
||||
return column;
|
||||
}
|
||||
|
||||
// A line split into the block quotes it sits in and the text inside the
|
||||
// innermost one. One optional space or tab after the last `>` is the quote
|
||||
// marker's own padding, not indentation, so it is stripped before anything is
|
||||
// measured: counting it left a closer at the three-column limit reading as four
|
||||
// columns and rejected.
|
||||
function quoteContext(line: string): { quoteDepth: number; rest: string } {
|
||||
const quoted = BLOCKQUOTE_PREFIX_RE.exec(line)?.[0];
|
||||
if (!quoted) {
|
||||
return { quoteDepth: 0, rest: line };
|
||||
}
|
||||
return {
|
||||
quoteDepth: quoted.split(">").length - 1,
|
||||
rest: line.slice(quoted.length).replace(/^[ \t]?/, ""),
|
||||
};
|
||||
}
|
||||
|
||||
function splitIndentation(content: string): { indent: number; body: string } {
|
||||
const body = content.replace(/^[ \t]+/, "");
|
||||
return {
|
||||
indent: indentationColumns(content.slice(0, content.length - body.length)),
|
||||
body,
|
||||
};
|
||||
}
|
||||
|
||||
// Where a line sits once its blockquote markers are stripped: how deep in the
|
||||
// quotes it is, the column its content starts at inside them, and the text after
|
||||
// that indentation. A line that loses a `>` has left the blockquote that held
|
||||
// the fence, which is why the depth is counted rather than tested as a boolean.
|
||||
function linePosition(line: string): {
|
||||
quoteDepth: number;
|
||||
indent: number;
|
||||
body: string;
|
||||
} {
|
||||
// After the split on "\n" a CRLF ending still carries its "\r" on the line,
|
||||
// and that "\r" is the other half of the ending rather than content: reading
|
||||
// it as body made a blank CRLF line nonblank, which cleared an indented run
|
||||
// in the middle of a code block.
|
||||
const content = line.endsWith("\r") ? line.slice(0, -1) : line;
|
||||
const { quoteDepth, rest } = quoteContext(content);
|
||||
return { quoteDepth, ...splitIndentation(rest) };
|
||||
}
|
||||
|
||||
// The same measurement with no block quote container assumed: a line inside an
|
||||
// open fence is literal, so a `>` there is code and keeps the columns it
|
||||
// occupies. Used to check containment for a fence that was not opened in a
|
||||
// quote, where stripping the marker made an indented content line read as column
|
||||
// zero and escape the fence.
|
||||
function literalPosition(line: string): {
|
||||
quoteDepth: number;
|
||||
indent: number;
|
||||
body: string;
|
||||
} {
|
||||
const content = line.endsWith("\r") ? line.slice(0, -1) : line;
|
||||
return { quoteDepth: 0, ...splitIndentation(content) };
|
||||
}
|
||||
|
||||
// Column a fence marker starts at, measured from the content of the innermost
|
||||
// blockquote. Container prefixes are part of the offset, so `- ```md` and
|
||||
// ` ```md` both read as column two.
|
||||
function fenceMarkerColumn(line: string, marker: string): number {
|
||||
const rest = quoteContext(line).rest;
|
||||
return advanceColumns(0, rest.slice(0, rest.indexOf(marker)));
|
||||
}
|
||||
|
||||
// The closer a line offers, or null when it is literal content. Only
|
||||
// indentation may precede the marker and nothing but whitespace may follow it.
|
||||
function closingFence(line: string): { marker: string; column: number } | null {
|
||||
const rest = quoteContext(line).rest;
|
||||
const match = FENCE_CLOSER_RE.exec(rest);
|
||||
if (!match) {
|
||||
return null;
|
||||
}
|
||||
const [, indentation = "", marker = ""] = match;
|
||||
if (marker === "" || rest.slice(match[0].length).trim() !== "") {
|
||||
return null;
|
||||
}
|
||||
return { marker, column: indentationColumns(indentation) };
|
||||
}
|
||||
|
||||
// What follows a fence marker on its line. Two CommonMark rules live there: a
|
||||
// closing fence is bare, and a backtick fence's info string carries no backtick.
|
||||
// Both are what the scanner in core/messages/utils.ts already enforces.
|
||||
function fenceTail(line: string, opener: FenceOpener): string {
|
||||
return line.slice(opener.consumed);
|
||||
}
|
||||
|
||||
type OpenFence = { quoteDepth: number; column: number };
|
||||
|
||||
// How far a line has to be indented past its container's content column to be
|
||||
// an indented code block rather than a paragraph. One tab already reaches it.
|
||||
const INDENTED_CODE_COLUMNS = 4;
|
||||
|
||||
// The content column of the deepest list item a line's indentation still keeps
|
||||
// open, popping the items the line has dedented past. The scan loop applies the
|
||||
// same rule before it reads a new list marker.
|
||||
function survivingItemColumn(items: number[], indent: number): number {
|
||||
while (items.length > 0 && indent < (items[items.length - 1] ?? 0)) {
|
||||
items.pop();
|
||||
}
|
||||
return items[items.length - 1] ?? 0;
|
||||
}
|
||||
|
||||
function maskFencedCodeBlocks(markdown: string): string {
|
||||
// Match a fenced block up to its matching closing fence, or — while the
|
||||
// message is still streaming — to end of input when the fence is unclosed.
|
||||
return markdown.replace(
|
||||
/(^|\n)(`{3,}|~{3,})[^\n]*(?:\n[\s\S]*?\n\2[^\n]*(?=\n|$)|[\s\S]*$)/g,
|
||||
maskKeepingNewlines,
|
||||
);
|
||||
// Blank a fenced block from its opener to its matching closer — or, while the
|
||||
// message is still streaming, to end of input when the fence is unclosed.
|
||||
// Marker-aware like the shared FENCE_MARKER_RE: a closer must repeat the
|
||||
// opener character and be at least as long, so a shorter run inside the block
|
||||
// does not close it early.
|
||||
//
|
||||
// A fence also cannot outlive the container it was opened in, and the marker's
|
||||
// own indentation is not part of that container: up to three columns of it are
|
||||
// allowed anywhere, so ` ```md` at the top level keeps swallowing an
|
||||
// unindented sample line, while the same two columns inside a list item whose
|
||||
// content starts at column two end the fence as soon as a line drops back to
|
||||
// column zero. `items` is what tells those two apart.
|
||||
const lines = markdown.split("\n");
|
||||
let openMarker: string | null = null;
|
||||
let fence: OpenFence | null = null;
|
||||
let items: number[] = [];
|
||||
let itemsQuoteDepth = 0;
|
||||
// A quote can sit inside a list item, and the item outlives the quote: the
|
||||
// depth-zero stack is parked here while the quote is scanned, so a fence that
|
||||
// ends with the quote can measure against the item it sat in.
|
||||
let outerItems: number[] | null = null;
|
||||
// Leaving a block quote is not the same as reaching free text: a line four
|
||||
// columns past the surviving list item — or past column zero when there is
|
||||
// none — starts an indented code block, so its citations stay code and must
|
||||
// stay blanked. The quote can also end on a blank line, which decides
|
||||
// nothing; the exit then stays pending until the first nonblank line
|
||||
// classifies it. Blank lines inside such a run need nothing — they carry no
|
||||
// text to hide.
|
||||
let indentedRun = false;
|
||||
// The container column the active run measures its four columns against.
|
||||
let runBase = 0;
|
||||
// Set when a fence ends because its quote did, on a line too blank to tell
|
||||
// an indented code block from ordinary structure.
|
||||
let pendingQuoteExit = false;
|
||||
// The open paragraph a line may or may not interrupt: -1 when the last
|
||||
// structural line ended a block, otherwise the container column its text sits
|
||||
// at and the block quote depth it was read at. A container only starts where
|
||||
// a paragraph may be interrupted, and an ordered item that does not start at 1
|
||||
// may not be: `Intro` then `2. ~~~md` stays one paragraph, so the tilde run on
|
||||
// that line opens no fence. The column is what keeps the rule container-aware —
|
||||
// after `- item` a `2.` at column zero has dedented out of the item's paragraph
|
||||
// and starts a real list — and the depth keeps a line that left the quote (or
|
||||
// never carried its marker) out of it.
|
||||
let paragraphColumn = -1;
|
||||
let paragraphQuote = 0;
|
||||
for (let i = 0; i < lines.length; i += 1) {
|
||||
const line = lines[i]!;
|
||||
// After the split on "\n" a CRLF ending still carries its "\r", and that
|
||||
// "\r" is the other half of the line ending rather than content: the
|
||||
// line-anchored patterns below and `linePosition` have to read the same
|
||||
// CR-normalized line, or a `---\r` setext underline fails the test and keeps
|
||||
// a finished paragraph blocking the list that follows it.
|
||||
const boundary = line.endsWith("\r") ? line.slice(0, -1) : line;
|
||||
const position = linePosition(line);
|
||||
const opener = fenceOpener(line);
|
||||
if (pendingQuoteExit) {
|
||||
if (position.quoteDepth === 0 && position.body === "") {
|
||||
paragraphColumn = -1;
|
||||
continue;
|
||||
}
|
||||
pendingQuoteExit = false;
|
||||
if (position.quoteDepth === 0) {
|
||||
const base = survivingItemColumn(items, position.indent);
|
||||
if (position.indent - base >= INDENTED_CODE_COLUMNS) {
|
||||
runBase = base;
|
||||
indentedRun = true;
|
||||
paragraphColumn = -1;
|
||||
lines[i] = maskKeepingNewlines(line);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (indentedRun) {
|
||||
if (
|
||||
position.quoteDepth === 0 &&
|
||||
(position.body === "" ||
|
||||
position.indent - runBase >= INDENTED_CODE_COLUMNS)
|
||||
) {
|
||||
paragraphColumn = -1;
|
||||
if (position.body !== "") {
|
||||
lines[i] = maskKeepingNewlines(line);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
indentedRun = false;
|
||||
}
|
||||
if (openMarker) {
|
||||
// A fence that was not opened in a quote has no quote container a later
|
||||
// line can leave, and its content lines are literal: measuring them
|
||||
// through `linePosition()` stripped a leading `>` and its padding, so
|
||||
// ` > example` inside a column-two list fence looked dedented to zero.
|
||||
// A `>` deeper than the fence's own quote markers is literal code rather
|
||||
// than a container the line still sits in: stripping it made the content
|
||||
// look dedented past the fence's column and ended the fence one line early.
|
||||
const containment =
|
||||
fence!.quoteDepth === 0 ? literalPosition(line) : position;
|
||||
const overQuoted =
|
||||
fence!.quoteDepth > 0 && position.quoteDepth > fence!.quoteDepth;
|
||||
const escaped =
|
||||
!overQuoted &&
|
||||
(containment.quoteDepth < fence!.quoteDepth ||
|
||||
(containment.body !== "" && containment.indent < fence!.column));
|
||||
if (escaped) {
|
||||
const leftTheQuote = fence!.quoteDepth > 0 && position.quoteDepth === 0;
|
||||
openMarker = null;
|
||||
fence = null;
|
||||
if (leftTheQuote) {
|
||||
// Returning to depth zero hands the line back to the list items the
|
||||
// quote sat in, so the four-column threshold below is measured
|
||||
// against the surviving item rather than the document root.
|
||||
items = outerItems ?? [];
|
||||
outerItems = null;
|
||||
itemsQuoteDepth = 0;
|
||||
if (position.body === "") {
|
||||
pendingQuoteExit = true;
|
||||
} else {
|
||||
runBase = survivingItemColumn(items, position.indent);
|
||||
indentedRun = position.indent - runBase >= INDENTED_CODE_COLUMNS;
|
||||
if (indentedRun) {
|
||||
lines[i] = maskKeepingNewlines(line);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
lines[i] = maskKeepingNewlines(line);
|
||||
paragraphColumn = -1;
|
||||
// A closer is only a closer when nothing but whitespace follows the
|
||||
// marker: ` ```text ` inside a ` ``` ` block is literal content, so
|
||||
// blanking it as a fence end would let a citation on the following
|
||||
// lines surface as a phantom source.
|
||||
const closer = closingFence(line);
|
||||
if (
|
||||
closer &&
|
||||
position.quoteDepth === fence!.quoteDepth &&
|
||||
closer.column - fence!.column <= 3 &&
|
||||
closer.marker.startsWith(openMarker.charAt(0)) &&
|
||||
closer.marker.length >= openMarker.length
|
||||
) {
|
||||
openMarker = null;
|
||||
fence = null;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
// Outside a fence the line is structure again, so it can open or close a
|
||||
// list item. Blank lines neither end an item nor start one.
|
||||
if (position.quoteDepth !== itemsQuoteDepth) {
|
||||
if (position.quoteDepth === 0 && outerItems !== null) {
|
||||
// The quote that held the list items has ended but the items outlive it,
|
||||
// so the stack parked on entry measures the lines after the exit: a
|
||||
// fence opened behind the ending quote still belongs to the item, and a
|
||||
// line that drops to column zero leaves it instead of hiding until EOF.
|
||||
items = outerItems;
|
||||
outerItems = null;
|
||||
} else {
|
||||
if (itemsQuoteDepth === 0 && position.quoteDepth > 0) {
|
||||
outerItems = items;
|
||||
}
|
||||
items = [];
|
||||
}
|
||||
itemsQuoteDepth = position.quoteDepth;
|
||||
}
|
||||
// A line that dedents past the open list items only leaves them when it
|
||||
// starts a block of its own. Unindented paragraph text is a lazy
|
||||
// continuation: the item stays open, so a fence later at the item's content
|
||||
// column still belongs to it and an unindented citation leaves the fence with
|
||||
// the item rather than hiding until end of input.
|
||||
const lazyParagraph =
|
||||
paragraphColumn >= 0 &&
|
||||
paragraphQuote === position.quoteDepth &&
|
||||
position.indent < paragraphColumn &&
|
||||
opener === null &&
|
||||
!ATX_HEADING_RE.test(boundary) &&
|
||||
!THEMATIC_BREAK_RE.test(boundary) &&
|
||||
!INTERRUPTING_LIST_RE.test(boundary);
|
||||
if (position.body !== "" && !lazyParagraph) {
|
||||
survivingItemColumn(items, position.indent);
|
||||
}
|
||||
// A line can open several list items at once and the fence belongs to the
|
||||
// deepest of them: `- - - ```md` has its content column at six, so applying
|
||||
// the marker pattern once recorded column two and the opener check below
|
||||
// rejected a valid fence as four columns past its container.
|
||||
let markerColumn = position.indent;
|
||||
let markerRest = position.body;
|
||||
let markersPushed = 0;
|
||||
let lazyContinuation = false;
|
||||
for (
|
||||
let item = LIST_ITEM_RE.exec(markerRest);
|
||||
item && markerColumn - (items[items.length - 1] ?? 0) <= 3;
|
||||
item = LIST_ITEM_RE.exec(markerRest)
|
||||
) {
|
||||
// Only the line's first marker competes with an open paragraph, and only a
|
||||
// bullet or a `1.` item may interrupt one: `Intro` followed by `2. ~~~md`
|
||||
// is a single paragraph, so the marker is text and the tilde run on that
|
||||
// line opens no fence. After a block boundary the same item is a real
|
||||
// container again.
|
||||
if (
|
||||
markersPushed === 0 &&
|
||||
position.indent >= paragraphColumn &&
|
||||
position.quoteDepth >= paragraphQuote &&
|
||||
paragraphColumn >= 0
|
||||
) {
|
||||
const digits = ORDERED_MARKER_DIGITS_RE.exec(item[0])?.[1];
|
||||
if (digits !== undefined && digits !== "1") {
|
||||
lazyContinuation = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
// The item's content column is where its marker text ends in columns, not
|
||||
// in characters: `-\t` reaches column four, and reading it as two would
|
||||
// keep a citation two spaces in inside a fence the reader already left.
|
||||
// That padding tops out at four columns, though: a fifth belongs to an
|
||||
// indented code block inside an item whose content starts one column past
|
||||
// its marker, so `- code` opens a column-two item, not a column-six one.
|
||||
const padding = /[ \t]+$/.exec(item[0])?.[0] ?? "";
|
||||
const markerEnd = advanceColumns(
|
||||
markerColumn,
|
||||
item[0].slice(0, item[0].length - padding.length),
|
||||
);
|
||||
const paddedColumn = advanceColumns(markerEnd, padding);
|
||||
markerColumn =
|
||||
paddedColumn - markerEnd > INDENTED_CODE_COLUMNS
|
||||
? markerEnd + 1
|
||||
: paddedColumn;
|
||||
items.push(markerColumn);
|
||||
markerRest = markerRest.slice(item[0].length);
|
||||
markersPushed += 1;
|
||||
}
|
||||
if (lazyContinuation) {
|
||||
// The line is paragraph text, so the same paragraph is still open for the
|
||||
// next line to be measured against.
|
||||
continue;
|
||||
}
|
||||
// The opener gets the same three-column budget as the closer: four columns
|
||||
// past the container's content column is an indented code block, so a marker
|
||||
// there cannot open a fence and everything after it keeps rendering.
|
||||
const openerMarker = opener?.marker;
|
||||
const containerColumn = items[items.length - 1] ?? 0;
|
||||
// `BLOCKQUOTE_PREFIX_RE` only matches from the start of a line, so a quote
|
||||
// opened behind a list marker - `- > ```md` - reads as depth zero even
|
||||
// though its fence really is quoted. The opener's own prefix is the truth
|
||||
// there: each `>` in it is a container the fence sits inside, and a line
|
||||
// that drops all of them has left the quote rather than reached free text.
|
||||
const prefixQuoteDepth = (opener?.prefix ?? "").split(">").length - 1;
|
||||
// A backtick fence's info string cannot hold a backtick, so ` ```md `x` `
|
||||
// is paragraph text with an inline span rather than an opener; opening a
|
||||
// fence there would blank a citation the reader can actually click. Tilde
|
||||
// fences take any info string.
|
||||
const openerTail = opener ? fenceTail(line, opener) : "";
|
||||
const infoAllowed =
|
||||
!openerMarker ||
|
||||
openerMarker.startsWith("~") ||
|
||||
!openerTail.includes("`");
|
||||
if (
|
||||
openerMarker &&
|
||||
infoAllowed &&
|
||||
fenceMarkerColumn(line, openerMarker) - containerColumn <= 3
|
||||
) {
|
||||
openMarker = openerMarker;
|
||||
fence = {
|
||||
quoteDepth: Math.max(prefixQuoteDepth, position.quoteDepth),
|
||||
column: containerColumn,
|
||||
};
|
||||
if (prefixQuoteDepth > position.quoteDepth) {
|
||||
// The list items holding that quote are invisible to `quoteDepth`, so
|
||||
// park their stack here: when the fence ends on an unquoted line the
|
||||
// escape below has nothing else to measure its four columns against.
|
||||
outerItems = [...items];
|
||||
}
|
||||
paragraphColumn = -1;
|
||||
lines[i] = maskKeepingNewlines(line);
|
||||
} else {
|
||||
// Everything else that carries text keeps the paragraph open, which is the
|
||||
// state the next line's first container marker has to interrupt. Headings
|
||||
// and thematic breaks (including a setext underline) are leaf blocks, a
|
||||
// blank line ends the paragraph, and the two predicates read the same
|
||||
// CR-normalized line `linePosition` classifies: `THEMATIC_BREAK_RE` is
|
||||
// anchored at the end of the line, so an underline `---\r` failed it and
|
||||
// kept a finished paragraph blocking the list that followed.
|
||||
//
|
||||
// A freshly opened item whose content sits four columns past its marker
|
||||
// starts with an indented code block instead, which is a leaf too.
|
||||
const indentedItemContent =
|
||||
markersPushed > 0 &&
|
||||
position.indent - containerColumn >= INDENTED_CODE_COLUMNS;
|
||||
if (
|
||||
position.body !== "" &&
|
||||
!indentedItemContent &&
|
||||
!ATX_HEADING_RE.test(boundary) &&
|
||||
!THEMATIC_BREAK_RE.test(boundary)
|
||||
) {
|
||||
paragraphColumn = containerColumn;
|
||||
paragraphQuote = position.quoteDepth;
|
||||
} else {
|
||||
paragraphColumn = -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function maskInlineCode(markdown: string): string {
|
||||
// Only mask closed spans: an unclosed backtick run renders as literal text,
|
||||
// so a citation after it is a real, rendered link and must not be masked.
|
||||
return markdown.replace(/(`+)[\s\S]*?\1/g, maskKeepingNewlines);
|
||||
// Pairing stays inside one inline span, because a span cannot reach past the
|
||||
// block boundary that ends its line; without that limit a stray backtick in
|
||||
// an earlier block steals the opener of a later span and mis-pairs both
|
||||
// directions. Chunk offsets tile the input, so occurrence indices stay aligned.
|
||||
const starts = inlineSpanStarts(markdown);
|
||||
if (starts.length === 1) {
|
||||
return markdown.replace(INLINE_CODE_SPAN_RE, maskKeepingNewlines);
|
||||
}
|
||||
return starts
|
||||
.map((start, i) =>
|
||||
markdown
|
||||
.slice(start, starts[i + 1] ?? markdown.length)
|
||||
.replace(INLINE_CODE_SPAN_RE, maskKeepingNewlines),
|
||||
)
|
||||
.join("");
|
||||
}
|
||||
|
||||
// Offsets where a new inline span context starts, mirroring the delimiters the
|
||||
// reasoning scanner in core/messages/utils.ts applies: blank lines (including a
|
||||
// blockquote's empty continuation line), ATX headings, thematic breaks / setext
|
||||
// underlines and interrupting list items. A heading or break ends a span on
|
||||
// both sides of its own line; a list item only starts a new context, since its
|
||||
// own wrapped lines continue the span. An ordered item interrupts only when it
|
||||
// numbers 1, matching CommonMark.
|
||||
function inlineSpanStarts(markdown: string): number[] {
|
||||
const lines = markdown.split("\n");
|
||||
const starts: number[] = [0];
|
||||
let offset = 0;
|
||||
let closesPrevious = false;
|
||||
for (const line of lines) {
|
||||
const compact = line.endsWith("\r") ? line.slice(0, -1) : line;
|
||||
const closes =
|
||||
ATX_HEADING_RE.test(compact) || THEMATIC_BREAK_RE.test(compact);
|
||||
if (
|
||||
closesPrevious ||
|
||||
closes ||
|
||||
BLANK_LINE_RE.test(compact) ||
|
||||
INTERRUPTING_LIST_RE.test(compact)
|
||||
) {
|
||||
if (offset > 0 && offset !== starts[starts.length - 1]) {
|
||||
starts.push(offset);
|
||||
}
|
||||
}
|
||||
closesPrevious = closes;
|
||||
offset += line.length + 1;
|
||||
}
|
||||
return starts;
|
||||
}
|
||||
|
||||
function maskKeepingNewlines(block: string): string {
|
||||
|
||||
@ -109,6 +109,48 @@ describe("extractCitationSources", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a citation whose paragraph sits between two unclosed backtick runs", () => {
|
||||
// An inline span cannot cross a blank line, so neither stray backtick opens
|
||||
// a span here and the citation is a rendered link, not sample code.
|
||||
const markdown = [
|
||||
"Install it with `npm i",
|
||||
"",
|
||||
"The upstream guide is [citation:Docs](https://example.com/docs).",
|
||||
"",
|
||||
"Then run `npm start` to serve it.",
|
||||
].join("\n");
|
||||
|
||||
const sources = extractCitationSources(markdown);
|
||||
|
||||
expect(sources.map((source) => source.url)).toEqual([
|
||||
"https://example.com/docs",
|
||||
]);
|
||||
expect(sources[0]?.occurrences[0]?.index).toBe(
|
||||
markdown.indexOf("[citation:Docs]"),
|
||||
);
|
||||
});
|
||||
|
||||
it("still masks a code span that closes inside its own paragraph", () => {
|
||||
const markdown = [
|
||||
"Install it with `npm i",
|
||||
"",
|
||||
"Example: `[citation:Fake](https://example.com/fake)` then real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends a paragraph on a CRLF blank line, as the inline scanner does", () => {
|
||||
const markdown =
|
||||
"Install it with `npm i\r\n\r\nThe upstream guide is [citation:Docs](https://example.com/docs).\r\n\r\nThen run `npm start` to serve it.";
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/docs",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ignores citations inside an unclosed fenced code block", () => {
|
||||
const markdown = [
|
||||
"Streaming output:",
|
||||
@ -119,6 +161,823 @@ describe("extractCitationSources", () => {
|
||||
expect(extractCitationSources(markdown)).toEqual([]);
|
||||
});
|
||||
|
||||
it("ends an inline span at a heading, thematic break or list marker", () => {
|
||||
// CommonMark ends the paragraph at these block boundaries too, so "Use
|
||||
// `npm" cannot steal the opener of `npm start` two lines later and the
|
||||
// citation between them is a rendered link, not sample code.
|
||||
for (const boundary of ["## Title", "---", "- item"]) {
|
||||
const markdown = [
|
||||
"Use `npm",
|
||||
boundary,
|
||||
"See [citation:X](https://example.com/x) and `npm start`.",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/x",
|
||||
]);
|
||||
}
|
||||
});
|
||||
|
||||
it("ends an inline span at a blank line inside a blockquote", () => {
|
||||
const markdown = [
|
||||
"> Install it with `npm i",
|
||||
">",
|
||||
"> The guide is [citation:Docs](https://example.com/docs) and `npm start`.",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/docs",
|
||||
]);
|
||||
});
|
||||
|
||||
it("masks citations inside a fenced block nested in a list item", () => {
|
||||
// The fence is indented past the list marker, so recognising it needs the
|
||||
// container prefix; the blank line inside keeps it open across paragraphs.
|
||||
const markdown = [
|
||||
"- run:",
|
||||
" ```md",
|
||||
" [citation:Fake1](https://example.com/fake1)",
|
||||
"",
|
||||
" [citation:Fake2](https://example.com/fake2)",
|
||||
" ```",
|
||||
"",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("masks a blockquoted fence, and stops at the line that ends the quote", () => {
|
||||
// Fake1 is inside the fence. The blank line has no `>` marker, so it closes
|
||||
// the blockquote — and the fence with it — which makes Fake2 a rendered link
|
||||
// again. Verified against remark-parse rather than by eye.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
"> [citation:Fake1](https://example.com/fake1)",
|
||||
"",
|
||||
"> [citation:Fake2](https://example.com/fake2)",
|
||||
"> ```",
|
||||
"",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake2",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("masks a dedented citation that lands in an indented code block", () => {
|
||||
// Losing the `>` closes the block quote and the fence inside it, but the
|
||||
// escaping line is not free: four columns of indentation at the top level
|
||||
// is an indented code block, so Fake still renders as code. Verified
|
||||
// against remark-parse, whose tree is quote>code, code, quote>code, paragraph.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("counts a tab-indented escaping line as the same four-column run", () => {
|
||||
// A tab reaches the next tab stop, so `\t` is four columns and `\t\t` eight;
|
||||
// both are indented code, and the marker padding on the opener is irrelevant.
|
||||
const markdown = [
|
||||
">\t```md",
|
||||
"\t[citation:Fake](https://example.com/fake)",
|
||||
">\t```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps an indented run across a blank line inside it", () => {
|
||||
// The CRLF variant pins the blankness classification: after a split on
|
||||
// "\n" the blank line still carries its "\r", and reading it as content
|
||||
// cleared the run and let Fake2 through.
|
||||
for (const eol of ["\n", "\r\n"]) {
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
" [citation:Fake1](https://example.com/fake1)",
|
||||
"",
|
||||
" [citation:Fake2](https://example.com/fake2)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join(eol);
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
}
|
||||
});
|
||||
|
||||
it("masks the indented code block that follows the blank line ending a quoted fence", () => {
|
||||
// The blank line ends the blockquote, and the four-space line after it is a
|
||||
// top-level indented code block, so Fake stays code instead of surfacing as
|
||||
// a phantom source. Verified against remark-parse.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
"",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("renders the paragraph that follows the blank line ending a quoted fence", () => {
|
||||
// Two columns is paragraph territory, so the pending quote exit must not
|
||||
// mask the line that classifies it: this pins the boundary against the
|
||||
// four-column case above.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
"",
|
||||
" [citation:Para](https://example.com/para)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/para",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("measures a quote escape against the list item the quote sat in", () => {
|
||||
// The quote is a child of the list item and the item outlives it, so four
|
||||
// columns of indentation are only two past the item's content column: the
|
||||
// citation is paragraph content, not an indented code block. Verified
|
||||
// against remark-parse.
|
||||
const markdown = [
|
||||
"- item",
|
||||
" > ```md",
|
||||
" [citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("still masks a quote escape indented past the surviving list item", () => {
|
||||
// Six columns are four past the item's content column, so the escaped line
|
||||
// does start an indented code block — and the unindented line after it ends
|
||||
// the item and the run, rendering again.
|
||||
const markdown = [
|
||||
"- item",
|
||||
" > ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends the indented run at a dedent so that citation renders again", () => {
|
||||
// Two columns is inside the item-less paragraph the run gives way to, so the
|
||||
// second citation is a rendered link while the four-column one above stays
|
||||
// code.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
" [citation:Code](https://example.com/code)",
|
||||
" [citation:Para](https://example.com/para)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/para",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("resumes normal scanning once an indented run meets a quoted line", () => {
|
||||
// The run ends at `> [citation:Quoted]`, which is quote content again, and
|
||||
// the `> ``` ` below it opens the fence that the final line escapes.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"> [citation:Quoted](https://example.com/quoted)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/quoted",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not mask an escaping citation three columns in", () => {
|
||||
// Three columns is not enough to start an indented code block, so the line
|
||||
// is a top-level paragraph and its citation is a real rendered link. This
|
||||
// pins the threshold against the four-column case above.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not open a fence from a backtick run in the middle of a line", () => {
|
||||
const markdown = [
|
||||
"Run ```md please",
|
||||
"[citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a fence open across a shorter backtick run inside it", () => {
|
||||
const markdown = [
|
||||
"````md",
|
||||
"```",
|
||||
"[citation:Inside](https://example.com/inside)",
|
||||
"```",
|
||||
"````",
|
||||
"",
|
||||
"Outside [citation:Outside](https://example.com/outside).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/outside",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps an indented top-level fence open for unindented content", () => {
|
||||
// The opener's own indentation is not container indentation: content lines
|
||||
// may sit further left than the marker, and the closer only has to come back
|
||||
// within three columns of the opener.
|
||||
const markdown = [
|
||||
"Intro text.",
|
||||
" ```md",
|
||||
"[citation:Fake](https://example.com/fake)",
|
||||
" ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("treats a four-space indented marker inside a column-zero fence as content", () => {
|
||||
const markdown = [
|
||||
"```md",
|
||||
" ```",
|
||||
"[citation:Fake](https://example.com/fake)",
|
||||
"```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not open a fence from a marker indented four columns at the top level", () => {
|
||||
// Four columns past the container's content column is an indented code
|
||||
// block, not a fence opener, so the citation below it renders as a link.
|
||||
const markdown = [
|
||||
" ```md",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("still opens a fence from a marker indented three columns at the top level", () => {
|
||||
// The limit is three columns, so the next case up pins the boundary rather
|
||||
// than overshooting it.
|
||||
const markdown = [
|
||||
" ```md",
|
||||
"[citation:Fake](https://example.com/fake)",
|
||||
" ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not open a fence from a list marker with no space after it", () => {
|
||||
// `-` has to be followed by indentation to start a list item, so `-```md`
|
||||
// is ordinary paragraph text and nothing behind it is code.
|
||||
const markdown = [
|
||||
"-```md",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("counts a tab after a list marker as columns when it sizes the item", () => {
|
||||
// `-<TAB>` advances to the next four-column stop, so this item's content
|
||||
// starts at column four and the citation two spaces in has left the fence
|
||||
// rather than sitting inside it.
|
||||
const markdown = [
|
||||
"-\t```md",
|
||||
" [citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends a fence when a line drops out of the inner blockquote", () => {
|
||||
const markdown = [
|
||||
">> ```md",
|
||||
">> [citation:Fake](https://example.com/fake)",
|
||||
"> Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("lets a citation after a list-nested quote escape its fence", () => {
|
||||
// `- > ```md` opens the quote behind a list marker, so the line has no
|
||||
// leading `>` for a line-start pattern to find even though the fence really
|
||||
// is quoted. The dedented citation below sits two columns past the item's
|
||||
// content column, which is a paragraph rather than an indented code block.
|
||||
const markdown = [
|
||||
"- > ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends a list-nested quote fence at its first unquoted line", () => {
|
||||
const markdown = [
|
||||
"- > ```md",
|
||||
" [citation:One](https://example.com/one)",
|
||||
" [citation:Two](https://example.com/two)",
|
||||
"\t[citation:Three](https://example.com/three)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/one",
|
||||
"https://example.com/two",
|
||||
"https://example.com/three",
|
||||
]);
|
||||
});
|
||||
|
||||
it("escapes a quote fence nested behind a star marker", () => {
|
||||
const markdown = [
|
||||
"* > ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("escapes a quote fence nested behind an ordered marker", () => {
|
||||
// `1. ` gives the item a content column of three, so four columns of
|
||||
// indentation is one column of content rather than an indented code block.
|
||||
const markdown = [
|
||||
"1. > ```md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a list-nested quote open for a line that carries its marker", () => {
|
||||
// The mirror of the cases above: `>` on the second line means the quote did
|
||||
// not end, so nothing about the escape path applies here.
|
||||
const markdown = [
|
||||
"- >> ```md",
|
||||
">> [citation:Fake](https://example.com/fake)",
|
||||
"> [citation:Mid](https://example.com/mid)",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/mid",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends a list-nested fence at a closer that is itself inside the quote", () => {
|
||||
const markdown = [
|
||||
"- > ```md",
|
||||
"> [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("gives a quoted closer its full three-column budget", () => {
|
||||
// The one optional space after `>` belongs to the block quote marker, not to
|
||||
// the fence, so `>` plus four spaces indents the closer by three columns and
|
||||
// it closes the block.
|
||||
const markdown = [
|
||||
"> ```md",
|
||||
"> [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"> [citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not let a marker with an info string close the fence", () => {
|
||||
// A closing fence is bare, so ` ```text ` here is literal content and the
|
||||
// citation after it stays inside the block instead of becoming a source.
|
||||
const markdown = [
|
||||
"```md",
|
||||
"```text",
|
||||
"[citation:Fake](https://example.com/fake)",
|
||||
"```",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not close a fence from list-looking content inside it", () => {
|
||||
// Inside a fenced block `- ``` ` is literal content: a closing fence may
|
||||
// carry indentation and a marker, nothing else. Reading it as a closer leaks
|
||||
// Fake, and the real closer then masks away Real.
|
||||
const markdown = [
|
||||
"```md",
|
||||
"- ```",
|
||||
"[citation:Fake](https://example.com/fake)",
|
||||
"```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("closes a list-nested fence at the item's content column", () => {
|
||||
// The same rule one container deeper: `- ```md` opens the fence at the list
|
||||
// item's content column, ` - ``` ` stays literal content, and the bare
|
||||
// ` ``` ` at that column closes it. Verified against remark-parse, not by
|
||||
// eye.
|
||||
const markdown = [
|
||||
"- ```md",
|
||||
" - ```",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
" ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not open a backtick fence whose info string holds a backtick", () => {
|
||||
// ` ```md `x` ` is paragraph text, so the citation below it is a link the
|
||||
// reader can reach and must not be masked away.
|
||||
const markdown = [
|
||||
"```md `x`",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("opens a tilde fence whatever its info string says", () => {
|
||||
// Tilde fences take backticks in the info string, so the boundary above is
|
||||
// about the marker character rather than a blanket rule.
|
||||
const markdown = [
|
||||
"~~~md `x`",
|
||||
"[citation:Fake](https://example.com/fake)",
|
||||
"~~~",
|
||||
"Real [citation:Real](https://example.com/real).",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a literal quote marker inside an open list fence", () => {
|
||||
// Once a fence is open its lines are literal, so the `>` here is code rather
|
||||
// than a new block quote. Reading it as a container made the line look
|
||||
// dedented to column zero, which escaped the column-two list fence and
|
||||
// surfaced the sample's citation while reinterpreting the real closer as a
|
||||
// top-level opener.
|
||||
const markdown = [
|
||||
"- ```md",
|
||||
" > example",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
" ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a citation on a literal quote line inside a list fence", () => {
|
||||
// Same shape with the citation on the quoted line itself: it stays code.
|
||||
const markdown = [
|
||||
"- ```md",
|
||||
" > [citation:Fake](https://example.com/fake)",
|
||||
" ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a citation inside a fence opened behind several list markers", () => {
|
||||
// `- - - ```md` is a fence in the third list item, whose content column is
|
||||
// six. Recording only the first marker put the fence at column two, which
|
||||
// rejected the valid opener as four columns past its container.
|
||||
const markdown = [
|
||||
"- - - ```md",
|
||||
" [citation:Fake1](https://example.com/fake1)",
|
||||
"",
|
||||
" [citation:Fake2](https://example.com/fake2)",
|
||||
" ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a literal quote marker inside a quoted list fence", () => {
|
||||
// The fence already consumed the quote it was opened in, so the second `>`
|
||||
// here is literal code. `linePosition()` stripped both markers, which made
|
||||
// the sample line look dedented past the fence's column and ended the fence
|
||||
// one line early, surfacing the citation the renderer shows as code.
|
||||
const markdown = [
|
||||
"> - ```md",
|
||||
"> > example",
|
||||
"> [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps the list item open across a lazy paragraph continuation", () => {
|
||||
// An unindented line of paragraph text continues the item's paragraph rather
|
||||
// than leaving it, so the item's content column still measures the fence two
|
||||
// columns in. Popping the item there recorded the fence at column zero, which
|
||||
// swallowed the unindented citation to end of input.
|
||||
const markdown = [
|
||||
"- note",
|
||||
"lazy continuation",
|
||||
"",
|
||||
" ~~~md",
|
||||
" sample",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads five columns after a list marker as indented code, not padding", () => {
|
||||
// Only up to four columns of whitespace pad a list marker; the fifth belongs
|
||||
// to an indented code block inside an item whose content column is two.
|
||||
// Consuming the whole run recorded column six, so the two-space fence looked
|
||||
// like a dedent that left the item and the citation after it hid to EOF.
|
||||
const markdown = [
|
||||
"- code",
|
||||
"",
|
||||
" ~~~md",
|
||||
" sample",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores the enclosing list when an ordinary quote ends", () => {
|
||||
// The blank line ends `> note`, but the list item outlives the quote, so the
|
||||
// two-space fence still belongs to an item whose content column is two. The
|
||||
// citation at column zero leaves that item and ends the unclosed fence with
|
||||
// it, which makes it a rendered link.
|
||||
const markdown = [
|
||||
"- item",
|
||||
" > note",
|
||||
"",
|
||||
" ~~~md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
" sample",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores the enclosing list when a quote ends at the fence line", () => {
|
||||
// The same exit without a blank line: a fence cannot be lazily continued by
|
||||
// a block quote, so `~~~md` already left the quote and the item's column is
|
||||
// what the following lines measure against.
|
||||
const markdown = [
|
||||
"- item",
|
||||
" > note",
|
||||
" ~~~md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
" sample",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores the enclosing list after a two-deep quote ends", () => {
|
||||
const markdown = [
|
||||
"- item",
|
||||
" >> note",
|
||||
"",
|
||||
" ~~~md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not open a fence from an ordered marker that cannot interrupt a paragraph", () => {
|
||||
// An ordered list that does not start at 1 cannot interrupt a paragraph, so
|
||||
// all three lines are one paragraph and its citation renders.
|
||||
const markdown = [
|
||||
"Intro",
|
||||
"2. ~~~md",
|
||||
" [citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not open a fence from a non-interrupting marker inside an item paragraph", () => {
|
||||
const markdown = [
|
||||
"- item",
|
||||
" 2. ~~~md",
|
||||
" [citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("still opens a fence from a `1.` marker that does interrupt the paragraph", () => {
|
||||
const markdown = [
|
||||
"Intro",
|
||||
"1. ~~~md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown)).toEqual([]);
|
||||
});
|
||||
|
||||
it("still opens a fence from a `2.` marker after a block boundary", () => {
|
||||
// The blank line closes the paragraph, so the ordered item starts a real
|
||||
// container and its fence hides the citation again.
|
||||
const markdown = [
|
||||
"Intro",
|
||||
"",
|
||||
"2. ~~~md",
|
||||
" [citation:Fake](https://example.com/fake)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown)).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the citation a newly entered quote renders as a paragraph", () => {
|
||||
// Pinned against the parser rather than against a rule: remark's tree for
|
||||
// these lines is blockquote > [paragraph "2. ```md", paragraph with the
|
||||
// Fake link, code], so `2. ```md` never becomes a list item inside the
|
||||
// fresh quote and `> ``` ` is the opener of an unclosed fence. Both
|
||||
// citations render, and Sources has to surface both.
|
||||
const markdown = [
|
||||
"Intro",
|
||||
"> 2. ```md",
|
||||
">",
|
||||
"> [citation:Fake](https://example.com/fake)",
|
||||
"> ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/fake",
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends the paragraph for a setext underline over LF", () => {
|
||||
const markdown = [
|
||||
"Intro",
|
||||
"---",
|
||||
"2. ```md",
|
||||
" [citation:Fake1](https://example.com/fake1)",
|
||||
"",
|
||||
" [citation:Fake2](https://example.com/fake2)",
|
||||
" ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("ends the paragraph for a setext underline over CRLF", () => {
|
||||
// The trailing "\r" belongs to the line ending, so `---` is still a setext
|
||||
// underline and the paragraph is over: `2.` opens a real list, both of its
|
||||
// citations are fenced code, and the ` ``` ` line closes that fence rather
|
||||
// than starting a new one that would hide the rendered citation to EOF.
|
||||
const markdown = [
|
||||
"Intro",
|
||||
"---",
|
||||
"2. ```md",
|
||||
" [citation:Fake1](https://example.com/fake1)",
|
||||
"",
|
||||
" [citation:Fake2](https://example.com/fake2)",
|
||||
" ```",
|
||||
"[citation:Real](https://example.com/real)",
|
||||
].join("\r\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/real",
|
||||
]);
|
||||
});
|
||||
|
||||
it("uses the source domain when the citation label is generic", () => {
|
||||
const markdown = "See [citation:Source](https://www.example.com/path).";
|
||||
|
||||
@ -130,6 +989,49 @@ describe("extractCitationSources", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("scans a deeply repeated blockquote chain in one pass", () => {
|
||||
// tests/e2e/user-message-plain-text.spec.ts renders an AI message whose whole
|
||||
// body is one line of thousands of `>` markers, and the citation pass runs on
|
||||
// that text before the renderer does. Reading the containers a fence marker
|
||||
// may sit behind has to reach each position once: an alternation that can take
|
||||
// a space either as a step of its own or as the padding of the next marker
|
||||
// describes the same prefix many ways, so a line carrying no fence leaves the
|
||||
// engine to enumerate them all. Measured against the pattern this file guards,
|
||||
// one check costs about a second at 26 markers and roughly doubles for each
|
||||
// marker added, so a three-thousand-marker line never returns. The budget sits
|
||||
// two orders of magnitude above what a single pass takes, which pins the growth
|
||||
// rather than the machine.
|
||||
const shapes: Array<[string, string]> = [
|
||||
["> ".repeat(26) + "deep", "26-level quote"],
|
||||
["- " + "> ".repeat(26) + "deep-list", "26-level quote behind a list"],
|
||||
["> ".repeat(3000) + "deep", "3000-level quote"],
|
||||
["- " + "> ".repeat(3000) + "deep-list", "3000-level quote behind a list"],
|
||||
];
|
||||
|
||||
for (const [markdown, label] of shapes) {
|
||||
it(`masks nothing in the ${label} shape within one pass of the line`, () => {
|
||||
const started = performance.now();
|
||||
const sources = extractCitationSources(markdown);
|
||||
|
||||
expect(sources).toEqual([]);
|
||||
expect(performance.now() - started).toBeLessThan(500);
|
||||
});
|
||||
}
|
||||
|
||||
it("still reaches a citation that follows a deep quote chain", () => {
|
||||
// Skipping the scan would keep every case above green, so the same shape is
|
||||
// checked with real work to find after it.
|
||||
const markdown = [
|
||||
"> ".repeat(26) + "deep",
|
||||
"[citation:Deep](https://example.com/deep)",
|
||||
].join("\n");
|
||||
|
||||
expect(extractCitationSources(markdown).map((s) => s.url)).toEqual([
|
||||
"https://example.com/deep",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("formatCitationMarkdownReference", () => {
|
||||
it("formats a source as a reusable markdown reference", () => {
|
||||
const [source] = extractCitationSources(
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user