github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-23 21:16:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/docs/superpowers/plans/2026-09-13-projects-mvp-phase2-implementation-plan.md
Zeren Wang a58ab484a6
feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443) 
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash

Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.

Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
  admission pinning (both server-owned sets + worker hoist); latest-only
  request-scoped <project> block via DynamicContextMiddleware
  wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
  prefix + marker + provenance, never persisted); journal audit
  fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
  locked check-and-set; hash-qualified immutable shelf storage with
  Paths helpers; upload/list/content/delete-to-trash routes; project
  delete trashes the shelf in-transaction; request-scoped bounded
  <documents> index with honest count/shown + actionable overflow note;
  list_project_documents/read_project_document tools registered only on
  pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
  parity); POST from-thread with provenance; attach-to-thread with
  lock-staged copy (archived source allowed); read-only thread-files
  view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
  file moves), purge (continuous row lock across unlink/delete/commit,
  retryable on FS errors), retention sweep (lazy + startup, 24h orphan
  guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
  archived banner, content-missing rows), /workspace/trash route,
  sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
  + specs.

Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
  artifact + content responses; unified unsandboxed-iframe PDF preview
  (fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
  claims + os.link commit with suffix retry; same-name re-upload now
  unique-names instead of replacing); hidden staging only, no visible
  placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
  revalidation; drain locked filesystem work on cancellation; preserve
  bytes when an insert's commit state is uncertain (including trashed
  rows)
- original-integrity checks before serving text or cached conversions;
  content_missing surfaced in list responses (UI reads the flag, no
  409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
  declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
  rejects fractional/infinite values; composer counts staged
  attachments; pending attachments persist until submission or removal;
  in-flight instruction/rename edits survive save refetches; shelf and
  trash pagination; conversation-file and thread-files pages stay
  subscribed to refetches

Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.

Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
  bounded: the indexed expiry purge still runs on every trigger
  (GET /api/trash/documents, POST /api/trash/purge) while the
  O(all rows + all files) reconciliation is throttled to one run per
  user per 15 minutes (process-local, per-user window). The startup
  sweep now runs as a background task instead of blocking gateway
  readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
  like the render path, so a pasted, fenced <project>/<documents> snippet
  survives markdown export while real injected blocks (never fenced) are
  still removed. Fence regexes moved to a dependency-free leaf module to
  avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
  attribute (the upstream e2e contract locates it via :not([title])), and
  the upstream artifact-preview spec now pins the new contract: PDFs
  render unsandboxed, images keep sandbox="".

* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title

- Shutdown cancelled only the shield around the background startup sweep,
  so an all-users reconciliation that outlived the 5s budget kept walking
  rows and files while the document repo and DB engine were disposed
  underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
  which cancels the task and drains it before worker exit: the shield
  keeps the wait bounded, the cancel makes it final (CancelledError lands
  at the sweep's next await, and `_run_startup_trash_sweep` only catches
  `Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
  previous fix round, leaving the PDF frame without an accessible name
  while its siblings keep theirs. Restore it (WCAG frame titles), assert
  it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
  instead of `iframe:not([title])`.

* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel

`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.

* fix(projects): round-4 review — make Empty trash delete what it confirms

`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.

Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.

Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
2026-09-16 18:46:18 +08:00

113 lines
17 KiB
Markdown
Raw Blame History

# DeerFlow Projects MVP — Phase 2 Implementation Plan
**Date**: 2026-09-13
**Spec revision note (2026-09-13)**: the spec's injection strategy was replaced mid-implementation. §7.2 is now **latest-only, request-scoped**: `DynamicContextMiddleware` gains `wrap_model_call`/`awrap_model_call` hooks rendering ONE transient HumanMessage per model call from the admission-pinned snapshot. The persisted identity block, `project_context_revision` in `additional_kwargs`, `__project` corrections, and `<project_update>` notices from the previous revision are abandoned (forbidden by §15 items 18-19). `_build_full_reminder`/`_inject`/date/`__memory` behavior stays byte-identical to main.
**Branch**: `projects-mvp-phase2` (worktree `../deer-flow-phase2`, base `main @ 6f81daef`)
**Scope rule**: every design decision, error mapping, and review-gate is owned by the spec. This document sequences files, symbols, and verification only. Where the spec and code disagree, the spec's deviation register (§10) wins.
## Slice map (spec §16 dependency order)
| Slice | Deliverable | Depends on |
| --- | --- | --- |
| A | Instructions injection end-to-end (config, admission pin, middleware identity half, markers, prompt, journal, frontend Instructions tab) | — |
| B | Shelf storage + delivery + reads (table/migration, Paths, routes, request-scoped `<documents>` block, tools, project-delete trash transition) | A |
| C | Promotion both ways + thread-files view (from-thread, attach-to-thread, thread-files) | B |
| D | Trash completion (restore/purge/empty, retention sweep, reconciliation) | B |
| E | Frontend completion (Documents tab, Trash route, sidebar, copy, e2e) | B/C/D contracts |
Sequencing: A → B → {C, D} → E. C and D are independent of each other once B lands; they still ship serially here because both touch `routers/` mounting, PAT allowlist, and i18n dictionaries.
## Slice A — Instructions injection
**Backend**
1. `config/projects_config.py` (new): `ProjectsConfig(BaseModel)` — `instructions_max_bytes=8192 (256..262144)`, `shelf_index_max_entries=50 (1..500)`, `shelf_index_max_bytes=4096 (512..65536)`, `trash_retention_days=30 (1..3650)`; follow `read_before_write_config.py` shape. Wire `AppConfig.projects` in `config/app_config.py` (invalid/missing → defaults + warning, `_get_upload_limit` idiom). Add `projects:` block to `config.example.yaml`.
2. `routers/projects.py`: enforce UTF-8 byte cap on `instructions` in create + patch → 422 over cap (never truncate). Uses `AppConfig.projects.instructions_max_bytes`.
3. `runtime/context_keys.py`: add `PROJECT_CONTEXT_KEY: Final[str] = "__deerflow_project_context"`. Add to BOTH server-owned sets: `app/gateway/services.py` pop-set (`:435-446`) and `runtime/runs/worker.py::_SERVER_OWNED_RUNTIME_CONTEXT_KEYS`.
4. `deerflow/projects/__init__.py` + `deerflow/projects/context.py` (new): `resolve_project_context(thread_store, thread_id, user-scoped) -> dict | None` — one async resolution: `threads_meta.project_id` → owner-scoped project row `status IN ('active','archived')` → pinned snapshot dict `{project_id, name, instructions}` (shelf fields arrive in B). Failure → warning log + `None` (never fails the run).
5. `app/gateway/services.py`: in shared run-create path (~`:1455-1490`, after `inject_authenticated_user_context`, before `create_or_reject`) call the resolver and set `config["context"][PROJECT_CONTEXT_KEY]` when resolved; non-fatal on error, mirroring `_ensure_thread_metadata`.
6. `DynamicContextMiddleware` (`agents/middlewares/dynamic_context_middleware.py`) — request-scoped project block per new §7.2:
- Gain `wrap_model_call`/`awrap_model_call` hooks backed by PURE rendering helpers in `deerflow/projects/context.py` (no DB/FS I/O, no memory-retrieval dependency). Existing `_build_full_reminder`/`_inject`/date correction/`__memory` persistence UNCHANGED.
- Render at most one transient HumanMessage per model call: `<project id="…" name="…">{instructions}</project>` from `runtime.context[PROJECT_CONTEXT_KEY]` (Slice A: no `<documents>`). Attribute values escape quotes/ampersands/angle brackets; instructions through `neutralize_untrusted_tags`.
- Idempotent request assembly: reserved message-ID prefix + server-owned `deerflow_project_context` marker + provenance in `additional_kwargs` identify the transient message; each assembly removes only its own recognized transient messages from the REQUEST COPY, then inserts one fresh message. Never remove user messages by text or ID-prefix alone. `hide_from_ui` yes; NEVER `dynamic_context_reminder`; never a state update; never in checkpoints.
- Admission strips client-supplied `deerflow_project_context` markers from message metadata (§12).
- Placement: immediately before the genuine current-run user message via server-owned run/message identity; stable across the tool loop (never between assistant tool call and result, never appended per tool result); resumed/internal fallback = after leading SystemMessages; anchor recomputed from the current request after compaction.
- Latest-only semantics: rename/edit/move take effect next run; no corrections, no history comparison. Empty instructions keep identity, omit body; unassigned run renders nothing.
7. `input_sanitization_middleware.py`: `_BLOCKED_TAG_NAMES` gains `"project"` ONLY in Slice A (`"documents"` lands with B). Update `test_input_sanitization_middleware.py::test_denylist_covers_framework_authority_blocks` by classification.
8. `agents/lead_agent/prompt.py:558-564`: rewrite trust passage — distinguish user-role `<memory>` data and request-scoped project data from framework rules; tag escaping grants no system authority.
9. `runtime/journal.py`: `record_memory_context` payload → `{content_sha256: str|None, project_context_revision: str|None, project_shelf_revision: str|None}`. `content_sha256` covers only the existing `__memory` message (null when absent); `project_context_revision` = sha256(rendered `<project>` text) or null; `project_shelf_revision` null in A. Fingerprints only — never compared, never stored in kwargs. Emit once per run at first model-request assembly after delivery is known, when memory OR the project block is supplied (including project-only runs without `__memory`); failed assembly claims nothing; readers tolerate old payloads.
10. `tests/blocking_io/`: middleware already anchored; no new async FS paths in A.
11. `core/messages/utils.ts::INTERNAL_MARKER_TAGS` + `core/threads/export.ts` scrub: add `"project"` and `"documents"` (spec §7.2 end state; `project_update` does not exist in the revised design).
12. Project page `app/workspace/projects/[id]/page.tsx`: tabs (Chats / Documents-placeholder / Instructions / Settings) following `app/workspace/chats/page.tsx:130-138` Tabs pattern. Instructions tab: `Textarea` + save via `usePatchProject`, live UTF-8 byte counter, client guard mirroring 422.
13. i18n: new keys in `locales/types.ts`, `en-US.ts`, `zh-CN.ts` together.
**Tests (A)**: instructions cap boundary (exact/one-over, CJK bytes, create+patch); latest-instructions matrix (rename/edit/clear/move-in/move-out change the NEXT run's request only — no old instructions, no correction messages; empty instructions keep identity; unassigned renders nothing); request-only delivery (exactly one current project message per model call incl. retries; none in checkpoints before/after; idempotent reassembly of an already-decorated request; forged marker/ID cannot suppress/replace; placement before current genuine user message; tool-loop stability; resumed/internal fallback; post-compaction re-anchor); admission strips client-supplied `PROJECT_CONTEXT_KEY` (both sets) + `deerflow_project_context` message markers; resolution failure degrades to unassigned; journal identity (project-only runs, empty instructions, repeated calls, failed assembly, old-payload readers); config defaults/bounds; PAT unchanged (no new routes in A); frontend DOM for Instructions tab.
## Slice B — Shelf storage, delivery, reads
**Backend**
1. `persistence/projects/model.py`: `ProjectDocumentRow` per spec §6.1 table (no `mime`/`is_text`/`version`/`deleted_by`).
2. Migration `0024_project_documents` (renumbered from `0023` after the rebase; head `0023_user_preferences`): `op.create_table` + 4 indexes, `inspector.has_table` guard (`0019_projects.py:22-23` idiom). Update head pin in `tests/test_persistence_forward_revision_compat.py`; `persistence/migrations/AGENTS.md` head note.
3. `persistence/projects/sql.py`: `ProjectDocumentRepository` — `insert_active`, `find_active_by_sha256`, `list_active`, `count_active`, `shelf_snapshot`, `get`, `trash`, `trash_all_for_project` (ContextVar owner filter, `AUTO` sentinel, `projects` row lock with `status='active'` + `with_for_update()`, SQLite `BEGIN IMMEDIATE` idiom from Phase 1).
4. `config/paths.py`: `user_projects_dir`, `user_project_dir`, `project_documents_dir`, `project_document_path` (join + confinement re-check), `_validate_user_id`-style safe-id checks on `project_id`. Layout per §6.2: `{sha256[:2]}/{sha256}/{document_id}/original/{name}`, `derived/converted.md`, `.staging/{uuid}`.
5. `deerflow/projects/documents.py` (new): shelf service — stage→hash→fresh doc id→(tx: lock active project→dedup select→place bytes→insert) per §6.3; dedup hit removes staging, returns existing row (first name wins, §10.9); text detection via extracted shared helper; conversion via `convert_file_to_markdown` honoring `uploads.auto_convert_documents`.
6. Extract `is_text_file_by_content` from `routers/artifacts.py:235` → `deerflow/utils/` shared module; update router import.
7. `routers/project_documents.py` (new, mounted in `app.py:create_app` beside `projects.router`): `GET ""`, `POST ""` (multipart exactly-one-file, 201/200 dedup, 413/400 mirroring uploads router), `GET "/{document_id}/content"` (`download` flag), `DELETE "/{document_id}"` → 204 (trash via repository). 404 fail-closed everywhere; 503 `"Projects"` via `deps.py` `_require` + new `project_document_repo` accessor; archived-project matrix per §8.4/§11.
8. `ProjectRepository.delete`: gain statement 2 (trash all active shelf rows with `trash_origin` snapshot) inside its existing transaction/lock (§8.1).
9. Admission resolver (`projects/context.py` from A): extend pinned snapshot with bounded shelf `{"total", "entries"}` (`shelf_index_max_entries + 1` fetch, `updated_at DESC, id ASC`; +1 never rendered) — `shelf_snapshot` one round trip.
10. Middleware: extend A's transient request-scoped message with the `<documents>` block appended after `</project>` (rendered per model call from the pinned snapshot, never persisted); line-atomic caps (entries + bytes, wrapper/overflow reserved, no partial entry); header `count`/`shown` honest + overflow note naming `list_project_documents`; `project_shelf_revision = sha256(rendered text)` → journal payload only. `"documents"` joins `_BLOCKED_TAG_NAMES` + the drift guard in this slice (spec §16).
11. `deerflow/projects/tools.py` (new): `list_project_documents(offset, limit≤200)`, `read_project_document(document_id, offset, limit≤20000 chars)`; `project_id` from pinned context, `user_id` from `resolve_runtime_user_id`; fail closed (tool error) without pin/session; binary decline naming attach-to-thread; trashed → "no longer on the shelf"; all FS reads offloaded. Conditional registration in `agents/lead_agent/agent.py` — append only when `PROJECT_CONTEXT_KEY` present, via generalized collision-skipping helper (from `_append_memory_tools_without_name_conflicts`); both assembly-descriptor variants pinned. Subagents excluded.
12. `auth/pat.py` allowlist: add B routes; extend `test_pat_auth.py::test_pat_projects_policy_admits_exactly_the_mounted_routes`.
13. `tests/blocking_io/`: anchors for upload staging/rename, conversion, tool reads (mutation-verified).
**Frontend (B)**: delete confirmation copy change (en-US + zh-CN, `en-US.ts:357-358` region); nothing else user-visible until E (Documents tab scaffolding may render shelf read-only if trivial, but full tab is E).
**Tests (B)**: repository (dedup/trashed filter/snapshot ordering), index rendering caps/escaping/overflow-note, shelf delivery (exactly one block per request, never persisted, empty-shelf absence), tools (slice boundaries, clamp, binary decline, conversion on/off, fail-closed, live-edit visibility), archived matrix, project-delete trashes shelf, filename 255/256 boundary, concurrency (upload vs delete, duplicate upload, conversion vs purge-adjacent trash), PAT drift guard, migration upgrade/downgrade, blocking-IO anchors.
## Slice C — Promotion and file browser
**Backend**
1. Extract shared thread-upload ingestion service from `routers/uploads.py` (staging, `claim_unique_filename`, size checks, optional conversion, permissions, non-mounted-provider sandbox sync via authorized lease; denied `sandbox:execute` retains host upload without allocation).
2. `routers/project_documents.py`: `POST "/from-thread"` (`{thread_id, kind, name, shelf_name?}`; source confined to that thread's uploads/outputs; records `source_thread_id/source_kind/source_name`), `POST "/{document_id}/attach-to-thread/{thread_id}"` (stage stable copy under doc lock → ingestion service owns target lifecycle; returns `{filename, size_bytes, virtual_path, artifact_url}` only on success; archived source allowed).
3. `routers/project_thread_files.py` (new): `GET ""` — paged member threads (`thread_limit`≤50), per-thread uploads via `list_files_in_dir(sandbox_uploads_dir)` + NEW outputs listing helper over `sandbox_outputs_dir`; per-thread `file_limit`; `truncated` flag; `next_offset` thread cursor.
4. PAT allowlist + drift guard extension.
**Frontend (C contracts land here, UI in E)**: hooks `usePromoteThreadFile`, `useAttachProjectDocument`, `useProjectThreadFiles` in `core/projects/`.
**Tests (C)**: from-thread confinement/404, provenance columns consumed, attach parity (mounted/non-mounted, denied sandbox:execute, failure cleanup), thread-files paging/truncation, concurrency (attach vs purge serialization).
## Slice D — Trash completion
**Backend**
1. Repository: `restore` (outcome enum `restored|merged|not_found|no_target|content_missing`; lock order target-project→docs in ID order; no file moves; read-only content check under source lock; merge cleanup post-commit), `purge_candidates(retention_days)`, `purge` (continuous row lock across validate→unlink→delete→commit; `FileNotFoundError` = already removed; other unlink errors roll back, retryable).
2. `routers/trash.py` (new): `GET "/documents"` (lazy sweep trigger), `POST "/documents/{id}/restore"`, `POST "/documents/{id}/purge"` (204), `POST "/purge"` (`{purged: int}`). Restore target = body else `trash_origin.project_id` when active/owned; else 404.
3. Retention sweep service: `purge_candidates` + guarded purge; orphan reconciliation (`.staging/*` and unreferenced files under `projects/*/documents/` older than 24h removed; row-side missing/size-mismatch → log warning + `content_missing` surfacing, NEVER auto-delete row); startup trigger in `app.py` lifespan; lazy trigger on trash listing. `rmdir` empty parents best-effort.
4. Content endpoint + tools: explicit `content_missing` error for missing/mismatched files.
5. PAT allowlist + drift guard.
**Tests (D)**: restore outcome matrix incl. 409 `content_missing`, purge ordering (injected `OSError` retains row; missing optional conversion OK; commit failure retryable), retention boundary exactly at `retention_days`, 24h orphan guard, reconciliation never deletes, concurrency (purge vs restore barriers, both lock orders, restore vs project delete), sweep lazy + startup triggers.
## Slice E — Frontend completion
1. Documents tab: shelf area (upload button + drag-drop one-file-per-request loop, rows via `ArtifactFileList` pattern, provenance badge, preview via `ArtifactFilePreview`/`ArtifactViewer`, download, attach-to-thread, move-to-trash with trash-naming confirmation, `content_missing` row rendering) + conversation-files browser (grouped by thread, Save to project).
2. `/workspace/trash` route (NOT under `/workspace/projects/`): rows with origin project + remaining retention, Restore (project picker fallback), Delete-permanently + Empty trash with explicit irreversible confirmations.
3. Sidebar `ProjectsSection` Trash entry; static-demo `null` guards (`isStaticWebsiteOnly()`); move menu untouched.
4. Archived read-only banner on project page Documents area.
5. Interim memory notice (RFC §10 copy) on project page + Documents empty state.
6. Hooks: `useProjectDocuments`, `useUploadProjectDocument`, `useDeleteProjectDocument`, `useTrashDocuments`, `useRestoreDocument`, `usePurgeDocument`, `useEmptyTrash` (+ C's three); query keys per spec §9; invalidation per `invalidateProjectCaches`.
7. i18n keys (types/en-US/zh-CN together); e2e mocks in `tests/e2e/utils/mock-api.ts` + Playwright flows; no `<project>` text leaking into rendered messages.
## Documentation (land with owning slice)
- `README.md` (E), `backend/docs/API.md` (B/C/D routes), `backend/docs/ARCHITECTURE.md` (B shelf layout/pinning/authority split; D trash tier), `persistence/migrations/AGENTS.md` (B), `app/gateway/AGENTS.md` admission-pin contract (A), `config.example.yaml` (A).
## Verification gates
- After each slice: `cd backend && make lint && make test` (targeted first, then full suite at slice end).
- Frontend slices: `cd frontend && pnpm lint && pnpm typecheck`; `BETTER_AUTH_SECRET=local-dev-secret pnpm build` at E.
- Spec §15 review checklist re-read before finalizing each slice (esp. gates 1-8, 11-12, 18-20).
- Concurrency tests are a hard requirement (spec §13), not optional polish.
Reference in New Issue View Git Blame Copy Permalink