mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-25 14:06:18 +00:00
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Zeren Wang
|
a58ab484a6
|
feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443)
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md (issue #5160, tracker #5129) in the slice order of the spec's §16. Slices: - A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY admission pinning (both server-owned sets + worker hoist); latest-only request-scoped <project> block via DynamicContextMiddleware wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID prefix + marker + provenance, never persisted); journal audit fingerprints; Instructions tab. - B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with locked check-and-set; hash-qualified immutable shelf storage with Paths helpers; upload/list/content/delete-to-trash routes; project delete trashes the shelf in-transaction; request-scoped bounded <documents> index with honest count/shown + actionable overflow note; list_project_documents/read_project_document tools registered only on pinned runs; PAT allowlist + drift guards; blocking-IO anchors. - C: shared thread-upload ingestion service (uploads router refactored to parity); POST from-thread with provenance; attach-to-thread with lock-staged copy (archived source allowed); read-only thread-files view with per-group truncation reporting. - D: restore (restored/merged/not_found/no_target/content_missing; no file moves), purge (continuous row lock across unlink/delete/commit, retryable on FS errors), retention sweep (lazy + startup, 24h orphan guard, row-side reconciliation never deletes). - E: Documents tab (shelf + conversation-files browser, provenance, archived banner, content-missing rows), /workspace/trash route, sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks + specs. Review hardening folded in (10 rounds, all with tests): - force active shelf content (HTML/XML family) to download; nosniff on artifact + content responses; unified unsandboxed-iframe PDF preview (fixes the pre-existing Chromium sandbox blank in the artifact viewer) - scope document trash to the URL project under the document lock - atomic no-overwrite filename reservation for ALL ingestion (seeded claims + os.link commit with suffix retry; same-name re-upload now unique-names instead of replacing); hidden staging only, no visible placeholders; lease cleanup on setup failure - serialize conversion under the document lock with post-lock active revalidation; drain locked filesystem work on cancellation; preserve bytes when an insert's commit state is uncertain (including trashed rows) - original-integrity checks before serving text or cached conversions; content_missing surfaced in list responses (UI reads the flag, no 409-probe); downloads always serve original bytes - bounded streaming document reads with cached char counts; shelf limits declared in middleware release identity - thread-root confinement for from-thread sources; config fallback rejects fractional/infinite values; composer counts staged attachments; pending attachments persist until submission or removal; in-flight instruction/rename edits survive save refetches; shelf and trash pagination; conversation-file and thread-files pages stay subscribed to refetches Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md contracts, config.example.yaml projects block. Review follow-ups (head b4807477 → this revision): - The trash retention sweep is split so repeated lazy triggers stay bounded: the indexed expiry purge still runs on every trigger (GET /api/trash/documents, POST /api/trash/purge) while the O(all rows + all files) reconciliation is throttled to one run per user per 15 minutes (process-local, per-user window). The startup sweep now runs as a background task instead of blocking gateway readiness, and shutdown awaits it (bounded). - The export scrub (stripInternalMarkers) is fence- and indentation-aware like the render path, so a pasted, fenced <project>/<documents> snippet survives markdown export while real injected blocks (never fenced) are still removed. Fence regexes moved to a dependency-free leaf module to avoid the messages↔streamdown import cycle. - The artifact viewer's PDF iframe no longer carries an added title attribute (the upstream e2e contract locates it via :not([title])), and the upstream artifact-preview spec now pins the new contract: PDFs render unsandboxed, images keep sandbox="". * fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title - Shutdown cancelled only the shield around the background startup sweep, so an all-users reconciliation that outlived the 5s budget kept walking rows and files while the document repo and DB engine were disposed underneath it. The wait now lives in `_shutdown_startup_trash_sweep`, which cancels the task and drains it before worker exit: the shield keeps the wait bounded, the cancel makes it final (CancelledError lands at the sweep's next await, and `_run_startup_trash_sweep` only catches `Exception`, so nothing swallows it). - The browser-preview iframe lost `title={getFileName(filepath)}` in the previous fix round, leaving the PDF frame without an accessible name while its siblings keep theirs. Restore it (WCAG frame titles), assert it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]` instead of `iframe:not([title])`. * fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel `Task.cancel()` returns False when the sweep already finished inside the window between the deadline firing and the cancel, so the shutdown log claimed a cancellation that never happened. Branch on that outcome: the warning stays for a real cancel, a late finish is logged at info, and both paths still reap the task before worker exit. * fix(projects): round-4 review — make Empty trash delete what it confirms `POST /api/trash/purge` only ran the retention sweep, and the sweep's candidate selection is age-gated, so a freshly trashed document survived "Empty trash" even though the confirmation promises that every listed document is permanently deleted. With one trashed row the route answered `{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps expired rows before listing, so the visible rows were normally ineligible for the action by construction. Empty trash now drives `purge_all_trashed`: the caller's trashed rows (`list_all_trashed`, no age filter) each go through the same guarded, row-locked `purge` as the single-document delete — bytes first, then the row, in one transaction — so a row restored mid-flight is skipped instead of force-deleted, and an unlink failure rolls that row back and answers 500 with a retryable message. Retention expiry stays where it was: the sweep's `purge_candidates` is now the only age-gated selection, and the lazy retention sweep still runs on the listing and at startup. Tests: the router suite replaces the retention-gated expectation with the reviewer's repro (fresh row purged, bytes unlinked, shelf and other users' trash untouched, a failing unlink stays retryable and 500); a blocking-I/O anchor drives the new entry point through the offload; the mocked e2e covers the action end to end; a new real-backend spec performs it against the real gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE and the phase-2 design docs (en+zh) state the age-independent contract. |
||
|
betterkite
|
b552b5015c
|
fix(messages): drop legacy <uploaded_files> tag handling (#4826)
* fix(messages): drop legacy <uploaded_files> tag handling (#4212) PR #4174 unified upload-context injection on <current_uploads> (IM and web both flow through UploadsMiddleware), and #4632 documented the current path. This removes the remaining backward-compat parsing of the pre-#4174 <uploaded_files> tag, the final cleanup item tracked by the issue: - deermem: only <current_uploads> is stripped from human turns before memory persistence, and the upload-sentence scrubber drops the legacy tag alternative. - mem0: the mirrored message filter recognises only <current_uploads>. - InputSanitizationMiddleware: remove the legacy tag from the blocked-tag denylist (it existed only because deermem parsed the old tag). - frontend: stripUploadedFilesTag / stripInternalMarkers / parseUploadedFiles and the message-list fallback parse only <current_uploads>; demo thread fixtures are migrated to the current tag. Scope decision: a <uploaded_files> block in pre-#4174 history is now treated as ordinary user content (pinned by tests in both layers) instead of being silently dropped or stripped. * style: apply prettier formatting to stripUploadedFilesTag * fix(uploads): keep legacy <uploaded_files> stripping for display/export only Addresses review feedback on #4826: removing the legacy tag from the frontend display layer made pre-#4174 threads render raw <uploaded_files> XML (with server-side upload paths) in chat, copy data, and JSON exports. The backend cleanup stands — memory pipelines and the sanitization denylist treat only <current_uploads> as an internal marker. The frontend keeps the legacy spelling in its display/export-only utilities (stripUploadedFilesTag / INTERNAL_MARKER_TAGS / parseUploadedFiles and the message-list fallback) so old history renders cleanly without leaking internal paths, while the memory/sanitization scope-decision tests remain unchanged. Frontend tests now pin both spellings: <current_uploads> and legacy <uploaded_files> are stripped from copy data, markdown leak-stripping, and JSON exports. * docs(ui): record accepted display-spoof tradeoff for legacy upload tag Review note (willem-bd): since <uploaded_files> is off the sanitization denylist, a live user can type the legacy spelling and fabricate file chips / hide their own message text in display. Display-only and self-inflicted with no backend semantics, so it is accepted for now; documented at both the message-list fallback and stripUploadedFilesTag. Age-gating the legacy spelling remains a possible follow-up. --------- Co-authored-by: betterkite <313258397+betterkite@users.noreply.github.com> |
||
|
qin-chenghan
|
e7d7da9e7c
|
fix(frontend): strip <memory> tags in Streamdown to avoid React console error (#4209)
* fix(frontend): use code-aware stripLeakedSystemTags for all internal marker tags Replace the ad-hoc /<\/?memory>/g in ClipboardSafeStreamdown with a dedicated stripLeakedSystemTags() function in preprocess.ts that: - Covers all INTERNAL_MARKER_TAGS (<memory>, <system-reminder>, <current_date>, <uploaded_files>, <slash_skill_activation>) instead of only <memory> - Is code-aware: skips fenced code blocks (```) and indented code blocks (4-space indent), so user-written meta-discussions about the memory system are not silently stripped - Handles opening tags, closing tags, self-closing tags, and tags with attributes - Adds 15 unit tests covering all the above cases * fix(frontend): use marker-aware fence tracking in stripLeakedSystemTags The boolean insideFence toggle incorrectly closes a fence on any line matching 3+ backticks or tildes, regardless of the actual delimiter. This causes tags inside a tilde-fenced block containing a backtick sub-fence, or a 4-backtick block containing a 3-backtick sub-fence, to be silently stripped. Track the opening fence marker (character and run length) so that only a matching marker with at least the same length closes the fence. Adds 4 new test cases: - Tilde fence with inner backtick fence - 4-backtick fence with inner 3-backtick fence - Shorter tilde closing inside longer tilde fence - Tags stripped after real closing fence * fix(frontend): fix TypeScript strict errors in fence marker tracking - Use non-null assertion (!) on fenceMatch[1] (guaranteed by regex) - Use charAt(0) instead of [0] to avoid string | undefined type |
||
|
Boooobby
|
4a8f94eb28
|
fix(frontend): improve chat math rendering (#3557)
* fix(frontend): improve chat math rendering * fix(frontend): refine math rendering stability * fix(frontend): address review feedback on math preprocessing - Fix escaped-backslash blind spot: consume \\\\ as a unit so \\( and \\[ are not mis-interpreted as math delimiters when the backslash itself is escaped. - Thread inInlineCode state across lines so multi-line backtick code spans are protected from delimiter conversion. - Remove double math normalization: preprocessStreamdownMarkdown now only handles Mermaid; math normalization lives solely in ClipboardSafeStreamdown, preventing non-idempotent double passes. - Wire parseIncompleteMarkdown to isLoading in MarkdownContent so Streamdown's streaming-incomplete-math handling is actually active during streaming. * fix(frontend): address streamdown math review issues * refactor(frontend): move streamdown preprocessing out of ai elements |
||
|
Jiahan Chen
|
6fb22bb311
|
test(frontend): migrate unit tests to rstest (#3703)
* test(frontend): migrate unit tests to rstest * docs: updates AGENTS.md * test(frontend): fix rstest lint formatting |
||
|
Eilen Shin
|
25fbd25b05
|
fix(frontend): cap deeply nested list indentation to prevent render crash (#3393) (#3570)
* fix(frontend): cap deeply nested list indentation to prevent render crash Deeply nested lists make marked's recursive list tokenizer overflow the call stack during Streamdown's lexing useMemo, throwing an uncaught "RangeError: Maximum call stack size exceeded" that replaces the chat route with an error page (issue #3393); on larger stacks the same input exhausts the heap, which the render error boundary cannot catch. Mirror the existing capBlockquoteNesting guard with capListNesting, which clamps leading whitespace to 200 columns (~100 nesting levels) only when pathologically deep indentation is present, leaving normal content and fenced code untouched. Wire both through capMarkdownNesting. * fix(frontend): satisfy prettier format check in preprocess test * fix(frontend): exempt indented code from list-indent cap (PR #3570 review) * fix(frontend): keep capping all deep indentation outside fenced code Revert the indented-code exemption from the PR #3570 review nit. Taken literally the suggested guard (insideFence || INDENTED_CODE_RE.test(line)) no-ops capListNesting, because INDENTED_CODE_RE matches every line with 4+ leading spaces — i.e. exactly the deep-indent lines the cap targets. A context-aware exemption (only treat 4+-space lines as code after a blank line) instead reopens the crash: blank-separated deeply nested list items get exempted and still blow up marked (verified: OOM at depth ~1.5k). Unlike blockquotes (markers take <=3 leading spaces, so deep-quote lines never look like indented code), list vs. indented-code indentation is ambiguous line-by-line, so any exemption is exploitable. Keep capping all deep indentation outside fenced code; the only cost is mild corruption of a >200-column indented-code line, which never occurs in real content and is strictly preferable to a render crash. Add a regression test locking the blank-line case. |
||
|
Xinmin Zeng
|
503eeac788
|
fix(frontend): render user messages as plain text and cap blockquote nesting (#3502)
* fix(frontend): render user messages as plain text and cap blockquote nesting User messages are typed or pasted plain text, not authored Markdown, but they were rendered through the full Streamdown pipeline. Pasted source files got fragmented (indented chunks become code blocks, paragraphs collapse and lose indentation), "$...$" spans were KaTeX-ified, and a message with thousands of nested ">" markers overflowed the call stack in marked's recursive blockquote lexer, permanently crashing the thread on every load. Render human message content verbatim with pre-wrap instead, and cap blockquote nesting at 100 levels at the Streamdown chokepoint so model output cannot trigger the same recursion either. Closes #3500 * fix(frontend): absorb marked lexer crashes with a render fallback boundary Review found two gaps in the nesting cap: marked's list and blockquote tokenizers are mutually recursive, so a list marker in front of the quote chain ("- > > > ...") bypassed the blockquote-only regex and still overflowed the stack; and the line-based rewrite was fence-blind, silently truncating literal ">" runs inside code blocks. Add an error boundary around Streamdown that renders the raw content as plain pre-wrap text when rendering throws (retrying on the next content change), keep the cap as a fast path for the dominant pure-">" case, and make it skip fenced and indented code lines. |