mirror of
https://github.com/penpot/penpot.git
synced 2026-04-25 11:18:36 +00:00
6fa440cf92
Introduce a purpose-agnostic three-step session-based upload API that
allows uploading large binary blobs (media files and .penpot imports)
without hitting multipart size limits.
Backend:
- Migration 0147: new `upload_session` table (profile_id, total_chunks,
created_at) with indexes on profile_id and created_at.
- Three new RPC commands in media.clj:
* `create-upload-session` – allocates a session row; enforces
`upload-sessions-per-profile` and `upload-chunks-per-session`
quota limits (configurable in config.clj, defaults 5 / 20).
* `upload-chunk` – stores each slice as a storage object;
validates chunk index bounds and profile ownership.
* `assemble-file-media-object` – reassembles chunks via the shared
`assemble-chunks!` helper and creates the final media object.
- `assemble-chunks!` is a public helper in media.clj shared by both
`assemble-file-media-object` and `import-binfile`.
- `import-binfile` (binfile.clj): accepts an optional `upload-id` param;
when provided, materialises the temp file from chunks instead of
expecting an inline multipart body, removing the 200 MiB body limit
on .penpot imports. Schema updated with an `:and` validator requiring
either `:file` or `:upload-id`.
- quotes.clj: new `upload-sessions-per-profile` quota check.
- Background GC task (`tasks/upload_session_gc.clj`): deletes stalled
(never-completed) sessions older than 1 hour; scheduled daily at
midnight via the cron system in main.clj.
- backend/AGENTS.md: document the background-task wiring pattern.
Frontend:
- New `app.main.data.uploads` namespace: generic `upload-blob-chunked`
helper drives steps 1–2 (create session + upload all chunks with a
concurrency cap of 2) and emits `{:session-id uuid}` for callers.
- `config.cljs`: expose `upload-chunk-size` (default 25 MiB, overridable
via `penpotUploadChunkSize` global).
- `workspace/media.cljs`: blobs ≥ chunk-size go through the chunked path
(`upload-blob-chunked` → `assemble-file-media-object`); smaller blobs
use the existing direct `upload-file-media-object` path.
`handle-media-error` simplified; `on-error` callback removed.
- `worker/import.cljs`: new `import-blob-via-upload` helper replaces the
inline multipart approach for both binfile-v1 and binfile-v3 imports.
- `repo.cljs`: `:upload-chunk` derived as a `::multipart-upload`;
`form-data?` removed from `import-binfile` (JSON params only).
Tests:
- Backend (rpc_media_test.clj): happy path, idempotency, permission
isolation, invalid media type, missing chunks, session-not-found,
chunk-index out-of-range, and quota-limit scenarios.
- Frontend (uploads_test.cljs): session creation and chunk-count
correctness for `upload-blob-chunked`.
- Frontend (workspace_media_test.cljs): direct-upload path for small
blobs, chunked path for large blobs, and chunk-count correctness for
`process-blobs`.
- `helpers/http.cljs`: shared fetch-mock helpers (`install-fetch-mock!`,
`make-json-response`, `make-transit-response`, `url->cmd`).
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
75 lines
2.9 KiB
Clojure
75 lines
2.9 KiB
Clojure
(ns frontend-tests.runner
|
|
(:require
|
|
[cljs.test :as t]
|
|
[frontend-tests.basic-shapes-test]
|
|
[frontend-tests.data.repo-test]
|
|
[frontend-tests.data.uploads-test]
|
|
[frontend-tests.data.viewer-test]
|
|
[frontend-tests.data.workspace-colors-test]
|
|
[frontend-tests.data.workspace-media-test]
|
|
[frontend-tests.data.workspace-texts-test]
|
|
[frontend-tests.data.workspace-thumbnails-test]
|
|
[frontend-tests.helpers-shapes-test]
|
|
[frontend-tests.logic.comp-remove-swap-slots-test]
|
|
[frontend-tests.logic.components-and-tokens]
|
|
[frontend-tests.logic.copying-and-duplicating-test]
|
|
[frontend-tests.logic.frame-guides-test]
|
|
[frontend-tests.logic.groups-test]
|
|
[frontend-tests.logic.pasting-in-containers-test]
|
|
[frontend-tests.main-errors-test]
|
|
[frontend-tests.plugins.context-shapes-test]
|
|
[frontend-tests.svg-fills-test]
|
|
[frontend-tests.tokens.import-export-test]
|
|
[frontend-tests.tokens.logic.token-actions-test]
|
|
[frontend-tests.tokens.logic.token-data-test]
|
|
[frontend-tests.tokens.logic.token-remapping-test]
|
|
[frontend-tests.tokens.style-dictionary-test]
|
|
[frontend-tests.tokens.token-errors-test]
|
|
[frontend-tests.tokens.workspace-tokens-remap-test]
|
|
[frontend-tests.ui.ds-controls-numeric-input-test]
|
|
[frontend-tests.util-object-test]
|
|
[frontend-tests.util-range-tree-test]
|
|
[frontend-tests.util-simple-math-test]
|
|
[frontend-tests.worker-snap-test]))
|
|
|
|
(enable-console-print!)
|
|
|
|
(defmethod cljs.test/report [:cljs.test/default :end-run-tests] [m]
|
|
(if (cljs.test/successful? m)
|
|
(.exit js/process 0)
|
|
(.exit js/process 1)))
|
|
|
|
(defn init
|
|
[]
|
|
(t/run-tests
|
|
'frontend-tests.basic-shapes-test
|
|
'frontend-tests.data.repo-test
|
|
'frontend-tests.main-errors-test
|
|
'frontend-tests.data.uploads-test
|
|
'frontend-tests.data.viewer-test
|
|
'frontend-tests.data.workspace-colors-test
|
|
'frontend-tests.data.workspace-media-test
|
|
'frontend-tests.data.workspace-texts-test
|
|
'frontend-tests.data.workspace-thumbnails-test
|
|
'frontend-tests.helpers-shapes-test
|
|
'frontend-tests.logic.comp-remove-swap-slots-test
|
|
'frontend-tests.logic.components-and-tokens
|
|
'frontend-tests.logic.copying-and-duplicating-test
|
|
'frontend-tests.logic.frame-guides-test
|
|
'frontend-tests.logic.groups-test
|
|
'frontend-tests.logic.pasting-in-containers-test
|
|
'frontend-tests.plugins.context-shapes-test
|
|
'frontend-tests.svg-fills-test
|
|
'frontend-tests.tokens.import-export-test
|
|
'frontend-tests.tokens.logic.token-actions-test
|
|
'frontend-tests.tokens.logic.token-data-test
|
|
'frontend-tests.tokens.logic.token-remapping-test
|
|
'frontend-tests.tokens.style-dictionary-test
|
|
'frontend-tests.tokens.token-errors-test
|
|
'frontend-tests.tokens.workspace-tokens-remap-test
|
|
'frontend-tests.ui.ds-controls-numeric-input-test
|
|
'frontend-tests.util-object-test
|
|
'frontend-tests.util-range-tree-test
|
|
'frontend-tests.util-simple-math-test
|
|
'frontend-tests.worker-snap-test))
|