mirror of
https://github.com/penpot/penpot.git
synced 2026-09-19 10:26:14 +00:00
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Andrey Antukh
|
ad7e035b63
|
✨ Add indirection for upload-chunks storage via new table (#11651)
* ✨ Add upload_session_chunk indirection for chunked uploads Chunks now live in the upload_session_chunk table with non-deleting foreign keys to storage_object and upload_session, instead of tempfile objects with session metadata. Reads go through a JOIN, so chunk state never scans storage_object. Uploads validate the live session, reject duplicate indexes, and store objects in the new upload-session bucket without extra metadata. Assemble removes mappings and marks the session consumed; objects-gc procedurally purges consumed and stalled sessions, touching referenced objects first. Touched-gc and deleted-gc handle the new bucket, and upload-session-gc is removed. Closes #11644 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix quota, give-up and coverage for session chunks Exclude consumed sessions from the sessions-per-profile quota so finished uploads free their slot at once. Remove chunk mappings before the gc-deleted give-up delete to respect the NO ACTION keys. Catch java.sql.SQLException for duplicate chunks. Cover the profile-owned session purge and the UNIQUE race backstop with tests. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Align chunked upload tests with upload_session_chunk Drop the duplicate-index tests written against metadata-backed chunks; the UNIQUE mapping makes those cases unrepresentable and the new tests cover them. Rewrite the rejected-duplicate tests to expect :validation/:chunk-already-exists and assert against the upload_session_chunk table, and scope the chunk-too-large "nothing stored" check to the mapping table. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Use NO ACTION DEFERRABLE session FKs in single migration Fold the profile FK change into 0154 so the feature ships one migration. All three upload session FKs use ON DELETE NO ACTION DEFERRABLE: identical to RESTRICT in normal operation, but deferrable for tooling that relies on SET CONSTRAINTS ALL DEFERRED. Extend the RESTRICT test to the direct profile delete. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Reserve chunk slot before writing blob in upload-chunk Make object_id nullable and insert the mapping with NULL inside the session-locking transaction, then write the blob outside it and link it with a conditional update. A failed write removes the mapping and reraises; a mid-flight death leaves a NULL row and the client starts a new session. AI-assisted-by: muse-spark-1.3-contributor * 🔥 Remove redundant session_id index on upload_session_chunk The UNIQUE(session_id, chunk_index) btree already serves session_id-only lookups and the session FK check through its leftmost column, so the standalone index only taxed the per-chunk INSERT path. Verified with EXPLAIN on an equivalent table shape. AI-assisted-by: muse-spark-1.3-contributor * ⚡ Merge chunk touch and delete into single RETURNING query Replace the SELECT-then-DELETE round-trip in delete-upload-sessions! with DELETE ... RETURNING object_id, touching each returned object. Same semantics, one less query per purged session. Follows the RETURNING pattern already used in file-gc. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Let objects-gc own chunk mapping deletion Assemble-chunks now only marks the session as consumed; the chunk mappings stay until objects-gc purges them (touching the chunk objects first), leaving a single procedural deletion path for consumed, stalled and profile-purge sessions. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix font-deletion GC expectations for chunk objects Update final storage-gc-touched counts to include the two chunk objects touched by objects-gc when purging consumed upload sessions (8/5/5 instead of 6/3/3). AI-assisted-by: muse-spark-1.3-contributor * 🐛 Release chunk reservation when the link UPDATE fails Review feedback on #11651: the link UPDATE in upload-chunk could leave a NULL reservation behind, blocking retries of the same index with :chunk-already-exists. Remove the reservation when the link fails so the client can retry in the same session; the orphaned blob stays touched for touched-gc. Also realign the process-bucket! cond branches in gc-touched. Tests: chunked-upload-link-failure-releases-slot and chunked-upload-null-reservation-blocks-retry. AI-assisted-by: muse-spark-1.3-contributor |
||
|
Andrey Antukh
|
0e388442a1
|
✨ Add storage object status lifecycle and verified dedup (#11345)
* ♻️ Simplify storage GC delays and add skip-delay task params The touched GC no longer applies an extra deletion-delay when marking storage objects as deleted. By the time a storage object is touched, its referencing domain row has already passed its own deletion delay, and the reference scan is the only safety check needed. Touched objects are now marked with deleted_at = now, so the deleted GC removes them on the next run. For the tempfile bucket, upload chunks now set touched-at in the future (1h, aligned with the upload-session-gc TTL) instead of relying on a special-case deletion delay. Task handlers now read their task props: - storage-gc-touched accepts :skip-delay to process all touched objects immediately, bypassing the min-age threshold. - objects-gc accepts :chunk-size and :skip-delay to process recently deleted rows without waiting for the deletion delay. This allows running the deletion cascade immediately from the REPL via run-task! with the skip-delay option. AI-assisted-by: deepseek-v4-flash * ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking Storage object lifecycle hardening: - Add status column ('valid' | 'pending') as write-ahead marker for object creation. put-object! inserts in 'pending' state, writes blob, then promotes to 'valid'. Failed writes remove the pending row. - Add :storage-pending-gc task to reclaim orphaned pending rows (e.g. after crash between blob write and promotion). - Verify blob existence on every dedup hit via exists-object? (fs stat / s3 headObject). Missing blobs mark the row as deleted and create fresh object. - Add deletion_attempts column (migration 0154) to track physical blob deletion attempts. Restructure gc_deleted to use chunked processing with per-chunk transactions (short lock duration). Failed deletions are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent infinite loops. After 7 attempts, give up and accept orphan. - Change del-objects-in-bulk contract to return #{fail-ids} for precise per-id tracking (fs and s3 backends updated). - Use tmp/tempfile for fs atomic writes with cleanup queue registration (crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE POSIX-only assumption. - Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms). - Wrap compensating delete in put-object! catch block to prevent masking original error when connection is aborted. - Fix assert messages in pending_gc.clj and gc_deleted.clj (pool assertion said 'expected valid storage' instead of 'db pool'). - Add pending-objects-excluded-from-gc-deleted test. Use unique path in put-object-write-failure-leaves-no-row test to avoid collisions. AI-assisted-by: qwen3.7-plus * 🐛 Fix review comments on gc-deleted and storage - Fix process-chunk! returning nil causing (+ acc nil) crash - Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent infinite loop when another worker holds locks - Pass :cause to log messages in gc_deleted.clj and s3.clj - Fix extra space in log hint string - Remove unused ::blob-missing? reference from storage memory - Rename test to match actual behavior (leaves pending row) - Add test for gc-deleted giving up after max attempts AI-assisted-by: qwen3.7-plus |
||
|
Andrey Antukh
|
4339d8d244 | 📎 Update serena documentation about backend storage |