mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-28 14:10:37 +00:00
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md (issue #5160, tracker #5129) in the slice order of the spec's §16. Slices: - A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY admission pinning (both server-owned sets + worker hoist); latest-only request-scoped <project> block via DynamicContextMiddleware wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID prefix + marker + provenance, never persisted); journal audit fingerprints; Instructions tab. - B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with locked check-and-set; hash-qualified immutable shelf storage with Paths helpers; upload/list/content/delete-to-trash routes; project delete trashes the shelf in-transaction; request-scoped bounded <documents> index with honest count/shown + actionable overflow note; list_project_documents/read_project_document tools registered only on pinned runs; PAT allowlist + drift guards; blocking-IO anchors. - C: shared thread-upload ingestion service (uploads router refactored to parity); POST from-thread with provenance; attach-to-thread with lock-staged copy (archived source allowed); read-only thread-files view with per-group truncation reporting. - D: restore (restored/merged/not_found/no_target/content_missing; no file moves), purge (continuous row lock across unlink/delete/commit, retryable on FS errors), retention sweep (lazy + startup, 24h orphan guard, row-side reconciliation never deletes). - E: Documents tab (shelf + conversation-files browser, provenance, archived banner, content-missing rows), /workspace/trash route, sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks + specs. Review hardening folded in (10 rounds, all with tests): - force active shelf content (HTML/XML family) to download; nosniff on artifact + content responses; unified unsandboxed-iframe PDF preview (fixes the pre-existing Chromium sandbox blank in the artifact viewer) - scope document trash to the URL project under the document lock - atomic no-overwrite filename reservation for ALL ingestion (seeded claims + os.link commit with suffix retry; same-name re-upload now unique-names instead of replacing); hidden staging only, no visible placeholders; lease cleanup on setup failure - serialize conversion under the document lock with post-lock active revalidation; drain locked filesystem work on cancellation; preserve bytes when an insert's commit state is uncertain (including trashed rows) - original-integrity checks before serving text or cached conversions; content_missing surfaced in list responses (UI reads the flag, no 409-probe); downloads always serve original bytes - bounded streaming document reads with cached char counts; shelf limits declared in middleware release identity - thread-root confinement for from-thread sources; config fallback rejects fractional/infinite values; composer counts staged attachments; pending attachments persist until submission or removal; in-flight instruction/rename edits survive save refetches; shelf and trash pagination; conversation-file and thread-files pages stay subscribed to refetches Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md contracts, config.example.yaml projects block. Review follow-ups (head b4807477 → this revision): - The trash retention sweep is split so repeated lazy triggers stay bounded: the indexed expiry purge still runs on every trigger (GET /api/trash/documents, POST /api/trash/purge) while the O(all rows + all files) reconciliation is throttled to one run per user per 15 minutes (process-local, per-user window). The startup sweep now runs as a background task instead of blocking gateway readiness, and shutdown awaits it (bounded). - The export scrub (stripInternalMarkers) is fence- and indentation-aware like the render path, so a pasted, fenced <project>/<documents> snippet survives markdown export while real injected blocks (never fenced) are still removed. Fence regexes moved to a dependency-free leaf module to avoid the messages↔streamdown import cycle. - The artifact viewer's PDF iframe no longer carries an added title attribute (the upstream e2e contract locates it via :not([title])), and the upstream artifact-preview spec now pins the new contract: PDFs render unsandboxed, images keep sandbox="". * fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title - Shutdown cancelled only the shield around the background startup sweep, so an all-users reconciliation that outlived the 5s budget kept walking rows and files while the document repo and DB engine were disposed underneath it. The wait now lives in `_shutdown_startup_trash_sweep`, which cancels the task and drains it before worker exit: the shield keeps the wait bounded, the cancel makes it final (CancelledError lands at the sweep's next await, and `_run_startup_trash_sweep` only catches `Exception`, so nothing swallows it). - The browser-preview iframe lost `title={getFileName(filepath)}` in the previous fix round, leaving the PDF frame without an accessible name while its siblings keep theirs. Restore it (WCAG frame titles), assert it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]` instead of `iframe:not([title])`. * fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel `Task.cancel()` returns False when the sweep already finished inside the window between the deadline firing and the cancel, so the shutdown log claimed a cancellation that never happened. Branch on that outcome: the warning stays for a real cancel, a late finish is logged at info, and both paths still reap the task before worker exit. * fix(projects): round-4 review — make Empty trash delete what it confirms `POST /api/trash/purge` only ran the retention sweep, and the sweep's candidate selection is age-gated, so a freshly trashed document survived "Empty trash" even though the confirmation promises that every listed document is permanently deleted. With one trashed row the route answered `{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps expired rows before listing, so the visible rows were normally ineligible for the action by construction. Empty trash now drives `purge_all_trashed`: the caller's trashed rows (`list_all_trashed`, no age filter) each go through the same guarded, row-locked `purge` as the single-document delete — bytes first, then the row, in one transaction — so a row restored mid-flight is skipped instead of force-deleted, and an unlink failure rolls that row back and answers 500 with a retryable message. Retention expiry stays where it was: the sweep's `purge_candidates` is now the only age-gated selection, and the lazy retention sweep still runs on the listing and at startup. Tests: the router suite replaces the retention-gated expectation with the reviewer's repro (fresh row purged, bytes unlinked, shelf and other users' trash untouched, a failing unlink stays retryable and 500); a blocking-I/O anchor drives the new entry point through the offload; the mocked e2e covers the action end to end; a new real-backend spec performs it against the real gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE and the phase-2 design docs (en+zh) state the age-independent contract.
153 lines
5.0 KiB
TypeScript
153 lines
5.0 KiB
TypeScript
import type { SetStateAction } from "react";
|
|
import { useCallback, useEffect, useRef, useState } from "react";
|
|
|
|
import type { AttachProjectDocumentResult } from "./types";
|
|
|
|
/**
|
|
* Cross-route handoff for attach-to-thread (spec §9): the project page's
|
|
* Documents tab stages the *completed* attachment here on success and
|
|
* navigates to the target thread, where the composer shows the file as an
|
|
* already-uploaded attachment. The handoff only ever carries a
|
|
* server-confirmed ingestion result — never an in-flight upload — so the
|
|
* composer receives a completed attachment only on success.
|
|
*
|
|
* The staged set is a per-thread pending LIST that persists until the
|
|
* message is actually sent or the chip is explicitly removed:
|
|
*
|
|
* - Attaching a second document appends to the list (re-attaching the same
|
|
* document refreshes its entry), so navigating back to the project page
|
|
* between attaches cannot drop the earlier attachment.
|
|
* - Mounting the composer only *reads* the list — nothing is consumed on
|
|
* mount, so a reload or a remount before submission keeps every chip.
|
|
* - ``setAttachments`` writes every change through, and submission clears
|
|
* the list in its ``onSent`` callback, which only fires when the send
|
|
* genuinely proceeds.
|
|
*
|
|
* sessionStorage (not module memory) so the pending list also survives a
|
|
* full page load of the target thread.
|
|
*/
|
|
const STORAGE_KEY_PREFIX = "deerflow.project-attachment.";
|
|
|
|
function storageKey(threadId: string): string {
|
|
return `${STORAGE_KEY_PREFIX}${threadId}`;
|
|
}
|
|
|
|
function safeSessionStorage(): Storage | null {
|
|
try {
|
|
return typeof window === "undefined" ? null : window.sessionStorage;
|
|
} catch {
|
|
// Storage can be unavailable (privacy mode, sandboxed frame). The
|
|
// handoff is best-effort — degrading to memory-only state.
|
|
return null;
|
|
}
|
|
}
|
|
|
|
function isAttachment(value: unknown): value is AttachProjectDocumentResult {
|
|
return (
|
|
value !== null &&
|
|
typeof value === "object" &&
|
|
"filename" in value &&
|
|
"virtual_path" in value &&
|
|
typeof (value as AttachProjectDocumentResult).filename === "string" &&
|
|
typeof (value as AttachProjectDocumentResult).virtual_path === "string"
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Stage one thread-confirmed attachment, appending it to that thread's
|
|
* pending list. Re-staging the same document (same ``virtual_path``)
|
|
* refreshes its entry instead of duplicating it.
|
|
*/
|
|
export function stageProjectAttachment(
|
|
threadId: string,
|
|
attachment: AttachProjectDocumentResult,
|
|
): void {
|
|
const storage = safeSessionStorage();
|
|
if (!storage) {
|
|
return;
|
|
}
|
|
const pending = readProjectAttachments(threadId).filter(
|
|
(candidate) => candidate.virtual_path !== attachment.virtual_path,
|
|
);
|
|
storage.setItem(
|
|
storageKey(threadId),
|
|
JSON.stringify([...pending, attachment]),
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Read a thread's pending attachments **without consuming them**. Corrupt
|
|
* payloads are dropped; the pre-list single-object payload shape is read as a
|
|
* one-element list so a mid-session upgrade keeps the staged chip.
|
|
*/
|
|
export function readProjectAttachments(
|
|
threadId: string,
|
|
): AttachProjectDocumentResult[] {
|
|
const storage = safeSessionStorage();
|
|
if (!storage) {
|
|
return [];
|
|
}
|
|
const raw = storage.getItem(storageKey(threadId));
|
|
if (raw === null) {
|
|
return [];
|
|
}
|
|
try {
|
|
const parsed = JSON.parse(raw) as unknown;
|
|
if (Array.isArray(parsed)) {
|
|
return parsed.filter(isAttachment);
|
|
}
|
|
if (isAttachment(parsed)) {
|
|
return [parsed];
|
|
}
|
|
} catch {
|
|
// Corrupt payload: drop it.
|
|
}
|
|
storage.removeItem(storageKey(threadId));
|
|
return [];
|
|
}
|
|
|
|
/**
|
|
* The current thread's pending attachments, held by the mounted composer.
|
|
*
|
|
* The list is loaded from storage on mount and on every thread change, and
|
|
* every mutation is written back immediately, so unmounting (navigating to
|
|
* the project page, reloading) never loses a pending chip. A Strict-Mode
|
|
* effect replay re-reads the same list and re-applies the same state — there
|
|
* is no consume step left to replay wrongly.
|
|
*
|
|
* The returned setter accepts the same value-or-updater shapes as
|
|
* ``useState`` and is stable per thread.
|
|
*/
|
|
export function useStagedProjectAttachments(threadId: string) {
|
|
const [attachments, setAttachmentsState] = useState<
|
|
AttachProjectDocumentResult[]
|
|
>([]);
|
|
const pendingRef = useRef<AttachProjectDocumentResult[]>([]);
|
|
|
|
useEffect(() => {
|
|
const pending = readProjectAttachments(threadId);
|
|
pendingRef.current = pending;
|
|
setAttachmentsState(pending);
|
|
}, [threadId]);
|
|
|
|
const setAttachments = useCallback(
|
|
(value: SetStateAction<AttachProjectDocumentResult[]>) => {
|
|
const next =
|
|
typeof value === "function" ? value(pendingRef.current) : value;
|
|
pendingRef.current = next;
|
|
const storage = safeSessionStorage();
|
|
if (storage) {
|
|
if (next.length === 0) {
|
|
storage.removeItem(storageKey(threadId));
|
|
} else {
|
|
storage.setItem(storageKey(threadId), JSON.stringify(next));
|
|
}
|
|
}
|
|
setAttachmentsState(next);
|
|
},
|
|
[threadId],
|
|
);
|
|
|
|
return [attachments, setAttachments] as const;
|
|
}
|