* fix(projects): delete read-only documents when purging trash
A permanently deleted project document could never be purged on Windows:
the read-only file attribute makes `Path.unlink` fail with `PermissionError`
(`WinError 5`), where POSIX needs only a writable parent. A restored archive
or a mode-preserving sync sets that attribute, and because the purge re-raises,
the row rolls back and the trashed document keeps coming back with the same
error on every retry.
Clear the attribute and retry the unlink once, mirroring the retry that
`config/paths.py::_clear_read_only_tree` already performs for thread
directories. Any other unlink error still propagates, so the row stays
retryable.
* fix(projects): harden trash purge permission retries
---------
Co-authored-by: Aling33 <along308@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): make read_file_chunks cancellation-safe for worker handoffs
Cancelling a project document read had three unsafe windows around the
dedicated file-I/O worker: (1) a cancellation landing between the open
worker completing and the result reaching the coroutine leaked the
handle (the cleanup path never saw it); (2) the cleanup close could
start while an admitted read was still running; (3) a repeated
cancellation let the caller escape before the close worker settled
(#7491).
Gate every admitted operation with the existing await_drained
primitive: cancellation is delivered only after the operation settles,
the open result is retained in a list cell the moment the worker
returns it (so the cleanup close always has the handle), and waiting
for the next chunk via the yield stays immediately cancellable.
Regression tests gate the real executor with asyncio events at the
worker boundary (real files, no mocks): all three windows are red on
unmodified main and green with the fix; the happy path is unchanged.
The dispatch-spy promotion tests that import app.gateway.routers need
the full backend dependency set and could not run in the minimal local
env; they fail identically on unmodified main (verified).
Fixes#7491
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: fix _gated_offload return annotation to the unpacked tuple
Review nit: the helper returns (calls, gated) and every call site
unpacks it as a pair, but the annotation claimed list[str]. Type it as
the tuple it actually returns.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* test: pin missing-path FileNotFoundError in read_file_chunks
Convergence item from the duplicate-pair verdict (#7493): the
open-failure path must surface FileNotFoundError as-is — the cleanup
finally skips the close because retained is empty (no handle was ever
opened), rather than raising AttributeError from closing None.
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
---------
Co-authored-by: Jay Feng <jayfeng77@outlook.com>
* feat(projects): search shelf documents by keyword
Add search_project_documents: two-tier name/summary then bounded content scan, case-insensitive substring matching with original-text offsets. Never converts on the fly; reports stop_reason, content_scanned, skipped reasons, and capped unscanned/truncated ids.
* fix(projects): derive unscanned count without paging after budget trip
Review feedback: once the content budget trips, stop paging list_active to enumerate unscanned ids — cap ids at the trip page and derive the honest total from one count_active query (tier 1 is exhaustive, so active - tier1 hits - processed = unscanned). Also add a tool-level isolation test pinning that search never crosses projects or users.
* fix(projects): scan the shelf over one stable snapshot, not offset pages
Review feedback: offset paging re-read the live shelf per page, so a concurrent upload/trash between pages could duplicate a match or silently skip one while still claiming complete coverage. Tier 1 now materializes the whole shelf in a single SELECT (one read snapshot) and tier 2 iterates those same rows; the budget-trip unscanned count is derived exactly from the snapshot with no extra query. Mid-scan upload/trash regression tests pin both failure modes.
* fix(projects): check the project before staging shelf uploads
The shelf upload and save-from-thread routes staged the body under
users/{user_id}/projects/{project_id}/ before checking the project. A
missing, foreign, or archived id left an empty {project_id}/documents/
directory that the retention sweep keeps forever, and an id longer than
one path component failed with ENAMETOOLONG (500).
Both routes now require an owned, active project first and answer the
same 404 without writing. The shelf insert still re-checks under the
project row lock.
* test(projects): pin the shelf insert's re-check behind the early project check
With the early check in place, no route test reached the insert's
row-lock re-check any more. Archive the project between the check and
the insert and assert both routes still answer 404 and leave no staging
file.
* docs(changelog): reference #7040 for the shelf upload project check
* docs(changelog): drop the whitespace-only line after the zh #7040 entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Opt-in per-document summary column rendered into the injected
<documents> index, list_project_documents JSON, and the documents API.
Pipeline (projects/summaries.py, behind the narrow enqueue_summary
interface):
- The two shelf write routes enqueue a job on created=True; dedup hits
never regenerate. Off by default (projects.summaries_enabled).
- Bounded worker pool (summary_concurrency + bounded summary_queue_size;
a full queue drops the job) fed by FIFO; shutdown drains queued jobs,
delivers one sentinel per worker via blocking put (queue smaller than
the worker pool cannot strand idle workers), and bounds only
genuinely in-flight jobs with the grace period.
- Jobs carry (document_id, project_id, user_id) explicitly; the job's
owner is bound into the user ContextVar around the model call so
tracing/usage attribution never falls back to 'default'.
- Text source reuses read_text_serving_path (convertible-first rule,
content validation); conversion has no deadline (await_drained), the
model call has a caller-side wait limit — the worker slot is retained
until the provider call actually finishes and late results are
discarded.
- Output contract: strip leading <think> reasoning -> unwrap fence/
quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on
a character boundary; length-capped responses (provider-normalized
detectors) are failures.
- set_summary pins updated_at (literal self-assignment) so a landing
summary never reorders the shelf; owner-predicated UPDATE is a no-op
on rows purged mid-generation.
- Metrics: eligible/success/failed/skipped/queue_dropped/
uncapped_accepted.
Schema/config:
- project_documents.summary (migration 0036, nullable, no backfill;
forward-compat floor unchanged).
- Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS
(captured once at lifespan startup); config_version 57 -> 58.
- run_oneshot_llm gains an optional model_overrides passthrough (passed
only when set, existing callers unchanged) and a detailed variant
exposing the raw response message for termination-signal detection.
Docs: API.md response field, ARCHITECTURE.md pipeline paragraph,
migrations AGENTS.md chain head, root AGENTS.md records the scoped
all-core exception with the lift-out path.
* fix(projects): convert shelf documents outside the SQLite write lock
Lazy conversion ran pymupdf/markitdown inside the BEGIN IMMEDIATE
transaction that serializes the derived/converted.md publish against
trash and purge. On SQLite that lock is database-wide, so every unrelated
writer (run status, thread metadata, the scheduler) waited for the whole
conversion and failed with "database is locked" past busy_timeout (30s).
Conversion now writes into .staging/ outside any transaction. The
document row lock is taken only to revalidate the live row and atomically
rename the output into place (publish_under_live_lock, renamed from
convert_under_live_lock); a row trashed or purged meanwhile discards the
staged output, so nothing is published after purge. A read-only probe
skips converting rows that are already gone, a failed conversion is
still revalidated so a purged row reports content_missing, and the
conversion is drained on cancellation so its staging file is always
removed.
* docs(changelog): reference #6305 in the shelf conversion lock entry
* fix(projects): share one conversion across concurrent first reads
A first-read conversion holds a bounded file-IO pool thread for its
whole duration. With the conversion moved out of the row lock,
concurrent first reads of one document each converted on their own
thread, so a burst of reads could starve unrelated offloaded file work.
Route the conversion through a single-flight keyed by (user, document,
project): the first reader leads and the rest wait on its result
without holding a pool thread. The map holds concurrent.futures.Future
values under a threading.Lock because the read tool also runs on the
isolated subagent loop. Followers wait through asyncio.shield, so a
cancelled follower cannot cancel the shared future, and a leader that
is cancelled or raises hands the work to a follower instead of failing
it.
* fix(uploads): reject filenames reserved for temporary staging
Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace.
* fix(uploads): report reserved names before HTTP batch ingestion
Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation.
* fix(uploads): reject Windows-style staging basenames on POSIX
Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation.
Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
