diff --git a/.serena/memories/frontend/dashboard-viewer-subtleties.md b/.serena/memories/frontend/dashboard-viewer-subtleties.md index 14471e7195..4e88a98a5d 100644 --- a/.serena/memories/frontend/dashboard-viewer-subtleties.md +++ b/.serena/memories/frontend/dashboard-viewer-subtleties.md @@ -5,7 +5,9 @@ - Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id. - Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly. - Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications. -- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its header paints the milestone of the job (the stage and its counter) instead of the messages the old SSE stream sent; every file of a container reports the same milestone, so `jp/current-milestone-text` writes it once and the rows never repeat it. Before the job exists, the header paints a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and it writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, no row says anything: the header says the import waits for a slot while no job of it has started. The bar of the header measures the whole import, never a file: indeterminate while checking, waiting for a slot or rebuilding, because the client cannot know a global index and total of operations. The upload is the exception — the chunked upload knows its chunks, so the bar is determinate while the containers are sent, weighted by their size (`imp/upload-pct` over `:container-size`, the only percentage the client really knows). The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line on the rows any more: the rows of the import wizard carry only what is not progress — the name, the size, `Ready` once their container finished uploading, and the error text —, and the button that drops a file only shows while the wizard still offers its import step (the confirmation screen). Above the rows, the import wizard paints a stepper (Check/Upload/Import) plus one phase line: `imp/import-step` derives the step from the dialog status and the entries (upload holds while any file is pending, either `:import-ready` or carrying its own `:upload` milestone), so Check covers analyze, Upload covers the chunked client-side upload, and Import covers queue plus job. A file of a version 3 package shows what it occupies inside its container, computed while analyzing (`imp/file-sizes`): its own `files/…` entries plus the storage objects its media and thumbnails reference, whose metadata is the only link to `objects/…`. The container's own size stays in `:container-size` (what the dropped file weighs) and is what the upload bar weighs, never what a row shows. A file the manifest lists but the container holds no entry of has no size at all, and its row then shows no subtitle: what the package weighs never stands in for it. The subtitle is the size alone, with no extension. While analyzing with no failures nothing renders below the phase header — no rows, no spinner (`imp/awaiting-analysis?`); the header text is the only "Checking…" message. Rows appear once the analysis resolves (one per file inside the package), and a failed analysis takes the error-summary path instead. The ready step and its Continue wait for the end of the analysis (`imp/ready-to-import?`, fed with the completion of `imp/analyze`): a ready file is not enough, because the files still to arrive would never be imported. While rebuilding, the phase title counts the files and `imp/imported-count` counts them done with the job's own `:files` counter, so the number moves while a container rebuilds its files one by one. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients. +- The v3 format carries no size per file: the size of one is the sum of its own entries plus the `objects/` its media and thumbnail metadata reference, the only link between a file and that shared storage. +- Those metadata entries are read one at a time on purpose: a container holds thousands and every read opens its own decompression stream, which makes Chromium fail the analysis — zip.js then masks the cause behind a `Cannot close a ERRORED writable stream` — while Firefox tolerates it. Do not turn this back into a parallel read. The read is best-effort: a reference that cannot be read leaves that file without size, never the whole package. +- The `:error` of an import entry is user-facing text, never an i18n key. The legacy SSE import stays in the backend for other clients. - File creation/duplication strips file `:data` before putting file summaries into dashboard state. - The pending invitations are a team-admin only list (backend `:is-admin` already includes the owner). The dashboard hides the invitations entry from the header tabs and the team-options dropdown for non-admins, `invitations-section-content` keys the section on `:is-admin`, and `fetch-invitations` answers `insufficient-permissions` by emitting `invitations-fetched` with `[]`, so a demoted admin does not keep the cached emails and the header does not paint the invite button from a stale list. The organization `send-invitations` policy only gates inviting and resending, not the list. diff --git a/frontend/src/app/main/data/imports.cljs b/frontend/src/app/main/data/imports.cljs index 6fd079b015..415b204b0b 100644 --- a/frontend/src/app/main/data/imports.cljs +++ b/frontend/src/app/main/data/imports.cljs @@ -15,6 +15,7 @@ (:require [app.common.exceptions :as ex] [app.common.json :as json] + [app.common.logging :as log] [app.common.schema :as sm] [app.common.uuid :as uuid] [app.main.data.jobs :as dj] @@ -25,8 +26,7 @@ [app.util.i18n :as i18n :refer [tr]] [app.util.zip :as uz] [beicon.v2.core :as rx] - [cuerdas.core :as str] - [promesa.core :as p])) + [cuerdas.core :as str])) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; MESSAGES OF A FAILURE @@ -157,25 +157,43 @@ {:files {} :objects {} :references []} entries)) +(defn- read-reference + "The metadata of one of the media or the thumbnails of a file, decoded, + paired with the file it belongs to. A reference that cannot be read is + unknown, never a failure of the package: it only leaves the size of its + file unknown." + [file-id entry] + (->> (rx/from (uz/read-as-text entry)) + (rx/map (fn [text] [file-id (json/decode text)])) + (rx/catch (fn [cause] + (log/warn :msg "cannot read a reference of the package" + :file-id file-id + :entry (.-filename ^js entry) + :cause cause) + (rx/of [file-id ::unreadable]))))) + (defn- read-objects "The objects every file of a container references, read from the - metadata of its media and its thumbnails." + metadata of its media and its thumbnails, one entry at a time. + + Every read opens its own decompression stream and the container of a + big file can hold thousands of references: reading them all at once + opens thousands of streams and is what breaks the analysis on the + browsers that cannot keep up with them." [references] - (->> (map (fn [[file-id entry]] - (->> (uz/read-as-text entry) - (p/fmap (fn [text] [file-id (json/decode text)])))) - references) - (p/all) - (p/fmap - (fn [decoded] - (reduce (fn [result [file-id data]] - (->> [:media-id :thumbnail-id] - (keep #(get data %)) - (reduce (fn [result object-id] - (update result file-id (fnil conj #{}) object-id)) - result))) - {} - decoded))))) + (->> (rx/from references) + (rx/mapcat (fn [[file-id entry]] + (read-reference file-id entry))) + (rx/reduce + (fn [[referenced unreadable] [file-id data]] + (if (= ::unreadable data) + [referenced (conj unreadable file-id)] + [(reduce (fn [referenced object-id] + (update referenced file-id (fnil conj #{}) object-id)) + referenced + (keep #(get data %) [:media-id :thumbnail-id])) + unreadable])) + [{} #{}]))) (defn- file-sizes "How much every file of a container occupies inside it: its own entries @@ -186,13 +204,17 @@ [entries] (let [{:keys [files objects references]} (index-container entries)] (->> (read-objects references) - (p/fmap - (fn [referenced] - (reduce-kv (fn [sizes file-id object-ids] - (update sizes file-id (fnil + 0) - (reduce + 0 (keep objects object-ids)))) - files - referenced)))))) + (rx/map + (fn [[referenced unreadable]] + (let [sizes (reduce-kv (fn [sizes file-id object-ids] + (update sizes file-id (fnil + 0) + (reduce + 0 (keep objects object-ids)))) + files + referenced)] + ;; a file with a reference that could not be read has no + ;; size at all: a number that misses one of its objects is + ;; worse than no number + (reduce dissoc sizes unreadable))))))) (defn- analyze-file "The entries of one file the user dropped: one per file of a version 3 @@ -215,9 +237,8 @@ (rx/merge-map (fn [manifest] (if (= (:type manifest) "penpot/export-files") - (->> (uz/get-entries zip-reader) - (p/mcat file-sizes) - (rx/from) + (->> (rx/from (uz/get-entries zip-reader)) + (rx/mapcat file-sizes) (rx/map (fn [sizes] (-> file (assoc :type :binfile-v3) diff --git a/frontend/test/frontend_tests/data/imports_test.cljs b/frontend/test/frontend_tests/data/imports_test.cljs index e173ec3671..9a58d315dd 100644 --- a/frontend/test/frontend_tests/data/imports_test.cljs +++ b/frontend/test/frontend_tests/data/imports_test.cljs @@ -210,6 +210,112 @@ (t/testing "what the container occupies is kept apart" (t/is (= [10000 10000 10000] (mapv :container-size @seen)))))) +(t/deftest ^:async the-references-of-a-container-are-read-one-at-a-time + (let [file-id (uuid/next) + media-id (str (uuid/next)) + thumb-id (str (uuid/next)) + seen (atom []) + in-flight (atom 0) + peak (atom 0) + entries + #js [#js {:filename "manifest.json" + :compressedSize 100} + #js {:filename (str "files/" file-id ".json") + :compressedSize 200} + #js {:filename (str "files/" file-id "/media/one.json") + :compressedSize 10 + :content (str "{\"mediaId\": \"" media-id "\"}")} + #js {:filename (str "files/" file-id "/thumbnails/t/p/f.json") + :compressedSize 10 + :content (str "{\"thumbnailId\": \"" thumb-id "\"}")} + #js {:filename (str "objects/" media-id ".json") + :compressedSize 20} + #js {:filename (str "objects/" media-id ".png") + :compressedSize 1000} + #js {:filename (str "objects/" thumb-id ".json") + :compressedSize 5} + #js {:filename (str "objects/" thumb-id ".png") + :compressedSize 40}]] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + uz/get-entry (mock/stub (fn [_ _] + (p/resolved #js {:content (manifest [file-id "One"])}))) + uz/get-entries (mock/stub (fn [_] (p/resolved entries))) + ;; every read counts how many are open at the same time + uz/read-as-text + (mock/stub (fn [entry] + (swap! in-flight inc) + (swap! peak max @in-flight) + (->> (p/resolved (.-content entry)) + (p/fmap (fn [content] + (swap! in-flight dec) + content))))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "package.penpot" :size 10000}]) + {:on-next #(swap! seen conj %)})))) + + (t/testing "a container of many references never opens more than one read" + ;; a container of a big file holds thousands of references, and every + ;; read opens its own decompression stream: reading them all at once + ;; is what breaks the analysis in the browser + (t/is (= 1 @peak))) + + (t/testing "every reference is still counted" + ;; 200 + 10 + 10 of its own, and 20 + 1000 and 5 + 40 of the two + ;; objects its media and its thumbnail point at + (t/is (= [1285] (mapv :size @seen)))))) + +(t/deftest ^:async a-reference-that-cannot-be-read-leaves-its-file-without-size + (let [readable-id (uuid/next) + broken-id (uuid/next) + object-id (str (uuid/next)) + broken-path (str "files/" broken-id "/media/two.json") + seen (atom []) + entries + #js [#js {:filename "manifest.json" + :compressedSize 100} + #js {:filename (str "files/" readable-id ".json") + :compressedSize 200} + #js {:filename (str "files/" readable-id "/media/one.json") + :compressedSize 10 + :content (str "{\"mediaId\": \"" object-id "\"}")} + #js {:filename (str "objects/" object-id ".json") + :compressedSize 20} + #js {:filename (str "objects/" object-id ".png") + :compressedSize 1000} + #js {:filename (str "files/" broken-id ".json") + :compressedSize 400} + #js {:filename broken-path + :compressedSize 10}]] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + uz/get-entry (mock/stub (fn [_ _] + (p/resolved #js {:content (manifest [readable-id "Readable"] + [broken-id "Broken"])}))) + uz/get-entries (mock/stub (fn [_] (p/resolved entries))) + uz/read-as-text + (mock/stub (fn [entry] + (if (= broken-path (.-filename entry)) + (p/rejected (ex-info "cannot read the entry" {:path broken-path})) + (p/resolved (.-content entry))))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "package.penpot" :size 10000}]) + {:on-next #(swap! seen conj %)})))) + + (t/testing "one unreadable reference does not fail the package" + (t/is (= [readable-id broken-id] (mapv :file-id @seen))) + (t/is (every? #(= :success (:status %)) @seen))) + + (t/testing "the file whose reference could not be read has no size" + ;; a number that misses one of its objects is worse than no number + (t/is (= [1230 nil] (mapv :size @seen)))))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; IMPORTING A PACKAGE ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;