mirror of
https://github.com/penpot/penpot.git
synced 2026-04-25 11:18:36 +00:00
e5f9c1e863
Introduce a purpose-agnostic three-step session-based upload API that
allows uploading large binary blobs (media files and .penpot imports)
without hitting multipart size limits.
Backend:
- Migration 0147: new `upload_session` table (profile_id, total_chunks,
created_at) with indexes on profile_id and created_at.
- Three new RPC commands in media.clj:
* `create-upload-session` – allocates a session row; enforces
`upload-sessions-per-profile` and `upload-chunks-per-session`
quota limits (configurable in config.clj, defaults 5 / 20).
* `upload-chunk` – stores each slice as a storage object;
validates chunk index bounds and profile ownership.
* `assemble-file-media-object` – reassembles chunks via the shared
`assemble-chunks!` helper and creates the final media object.
- `assemble-chunks!` is a public helper in media.clj shared by both
`assemble-file-media-object` and `import-binfile`.
- `import-binfile` (binfile.clj): accepts an optional `upload-id` param;
when provided, materialises the temp file from chunks instead of
expecting an inline multipart body, removing the 200 MiB body limit
on .penpot imports. Schema updated with an `:and` validator requiring
either `:file` or `:upload-id`.
- quotes.clj: new `upload-sessions-per-profile` quota check.
- Background GC task (`tasks/upload_session_gc.clj`): deletes stalled
(never-completed) sessions older than 1 hour; scheduled daily at
midnight via the cron system in main.clj.
- backend/AGENTS.md: document the background-task wiring pattern.
Frontend:
- New `app.main.data.uploads` namespace: generic `upload-blob-chunked`
helper drives steps 1–2 (create session + upload all chunks with a
concurrency cap of 2) and emits `{:session-id uuid}` for callers.
- `config.cljs`: expose `upload-chunk-size` (default 25 MiB, overridable
via `penpotUploadChunkSize` global).
- `workspace/media.cljs`: blobs ≥ chunk-size go through the chunked path
(`upload-blob-chunked` → `assemble-file-media-object`); smaller blobs
use the existing direct `upload-file-media-object` path.
`handle-media-error` simplified; `on-error` callback removed.
- `worker/import.cljs`: new `import-blob-via-upload` helper replaces the
inline multipart approach for both binfile-v1 and binfile-v3 imports.
- `repo.cljs`: `:upload-chunk` derived as a `::multipart-upload`;
`form-data?` removed from `import-binfile` (JSON params only).
Tests:
- Backend (rpc_media_test.clj): happy path, idempotency, permission
isolation, invalid media type, missing chunks, session-not-found,
chunk-index out-of-range, and quota-limit scenarios.
- Frontend (uploads_test.cljs): session creation and chunk-count
correctness for `upload-blob-chunked`.
- Frontend (workspace_media_test.cljs): direct-upload path for small
blobs, chunked path for large blobs, and chunk-count correctness for
`process-blobs`.
- `helpers/http.cljs`: shared fetch-mock helpers (`install-fetch-mock!`,
`make-json-response`, `make-transit-response`, `url->cmd`).
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
62 lines
2.3 KiB
Clojure
62 lines
2.3 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS INC
|
|
|
|
(ns frontend-tests.helpers.http
|
|
"Helpers for intercepting and mocking the global `fetch` function in
|
|
ClojureScript tests. The underlying HTTP layer (`app.util.http`) calls
|
|
`(js/fetch url params)` directly, so replacing `globalThis.fetch` is the
|
|
correct interception point."
|
|
(:require
|
|
[app.common.transit :as t]
|
|
[clojure.string :as str]))
|
|
|
|
(defn install-fetch-mock!
|
|
"Replaces the global `js/fetch` with `handler-fn`.
|
|
|
|
`handler-fn` is called with `[url opts]` where `url` is a plain string
|
|
such as `\"http://localhost/api/main/methods/some-cmd\"`. It must return
|
|
a JS Promise that resolves to a fetch Response object.
|
|
|
|
Returns the previous `globalThis.fetch` value so callers can restore it
|
|
with [[restore-fetch!]]."
|
|
[handler-fn]
|
|
(let [prev (.-fetch js/globalThis)]
|
|
(set! (.-fetch js/globalThis) handler-fn)
|
|
prev))
|
|
|
|
(defn restore-fetch!
|
|
"Restores `globalThis.fetch` to `orig` (the value returned by
|
|
[[install-fetch-mock!]])."
|
|
[orig]
|
|
(set! (.-fetch js/globalThis) orig))
|
|
|
|
(defn make-json-response
|
|
"Creates a minimal fetch `Response` that returns `body-clj` serialised as
|
|
plain JSON with HTTP status 200."
|
|
[body-clj]
|
|
(let [json-str (.stringify js/JSON (clj->js body-clj))
|
|
headers (js/Headers. #js {"content-type" "application/json"})]
|
|
(js/Response. json-str #js {:status 200 :headers headers})))
|
|
|
|
(defn make-transit-response
|
|
"Creates a minimal fetch `Response` that returns `body-clj` serialised as
|
|
Transit+JSON with HTTP status 200. Use this helper when the code under
|
|
test inspects typed values (UUIDs, keywords, etc.) from the response body,
|
|
since the HTTP layer only decodes transit+json content automatically."
|
|
[body-clj]
|
|
(let [transit-str (t/encode-str body-clj {:type :json-verbose})
|
|
headers (js/Headers. #js {"content-type" "application/transit+json"})]
|
|
(js/Response. transit-str #js {:status 200 :headers headers})))
|
|
|
|
(defn url->cmd
|
|
"Extracts the RPC command keyword from a URL string.
|
|
|
|
Example: `\"http://…/api/main/methods/create-upload-session\"`
|
|
→ `:create-upload-session`."
|
|
[url]
|
|
(when (string? url)
|
|
(keyword (last (str/split url #"/")))))
|