From 6e4e69c339e825e737d36e7379973651fdcabe35 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Mon, 16 Nov 2026 15:42:15 +0100 Subject: [PATCH] :bug: Read the media and thumbnails of a package one at a time (#13024) The analysis of a package reads the metadata of every media and thumbnail of its container to attribute the shared objects to each file. It read them all at once, one promise per entry, and the container of a big file holds thousands of them: every read opens its own decompression stream, and Chromium fails the whole analysis under that load while Firefox keeps up. The failure is masked by zip.js with a generic "Cannot close a ERRORED writable stream". The references now go through an observable, one entry at a time, and a reference that cannot be read leaves its file without a size instead of failing the package: a number that misses one of the objects of the file is worse than no number. Closes #13022 AI-assisted-by: deepseek-v4.1-flash --- .../frontend/dashboard-viewer-subtleties.md | 4 +- frontend/src/app/main/data/imports.cljs | 77 ++++++++----- .../frontend_tests/data/imports_test.cljs | 106 ++++++++++++++++++ 3 files changed, 158 insertions(+), 29 deletions(-) diff --git a/.serena/memories/frontend/dashboard-viewer-subtleties.md b/.serena/memories/frontend/dashboard-viewer-subtleties.md index 14471e7195..4e88a98a5d 100644 --- a/.serena/memories/frontend/dashboard-viewer-subtleties.md +++ b/.serena/memories/frontend/dashboard-viewer-subtleties.md @@ -5,7 +5,9 @@ - Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id. - Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly. - Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications. -- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its header paints the milestone of the job (the stage and its counter) instead of the messages the old SSE stream sent; every file of a container reports the same milestone, so `jp/current-milestone-text` writes it once and the rows never repeat it. Before the job exists, the header paints a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and it writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, no row says anything: the header says the import waits for a slot while no job of it has started. The bar of the header measures the whole import, never a file: indeterminate while checking, waiting for a slot or rebuilding, because the client cannot know a global index and total of operations. The upload is the exception — the chunked upload knows its chunks, so the bar is determinate while the containers are sent, weighted by their size (`imp/upload-pct` over `:container-size`, the only percentage the client really knows). The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line on the rows any more: the rows of the import wizard carry only what is not progress — the name, the size, `Ready` once their container finished uploading, and the error text —, and the button that drops a file only shows while the wizard still offers its import step (the confirmation screen). Above the rows, the import wizard paints a stepper (Check/Upload/Import) plus one phase line: `imp/import-step` derives the step from the dialog status and the entries (upload holds while any file is pending, either `:import-ready` or carrying its own `:upload` milestone), so Check covers analyze, Upload covers the chunked client-side upload, and Import covers queue plus job. A file of a version 3 package shows what it occupies inside its container, computed while analyzing (`imp/file-sizes`): its own `files/…` entries plus the storage objects its media and thumbnails reference, whose metadata is the only link to `objects/…`. The container's own size stays in `:container-size` (what the dropped file weighs) and is what the upload bar weighs, never what a row shows. A file the manifest lists but the container holds no entry of has no size at all, and its row then shows no subtitle: what the package weighs never stands in for it. The subtitle is the size alone, with no extension. While analyzing with no failures nothing renders below the phase header — no rows, no spinner (`imp/awaiting-analysis?`); the header text is the only "Checking…" message. Rows appear once the analysis resolves (one per file inside the package), and a failed analysis takes the error-summary path instead. The ready step and its Continue wait for the end of the analysis (`imp/ready-to-import?`, fed with the completion of `imp/analyze`): a ready file is not enough, because the files still to arrive would never be imported. While rebuilding, the phase title counts the files and `imp/imported-count` counts them done with the job's own `:files` counter, so the number moves while a container rebuilds its files one by one. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients. +- The v3 format carries no size per file: the size of one is the sum of its own entries plus the `objects/` its media and thumbnail metadata reference, the only link between a file and that shared storage. +- Those metadata entries are read one at a time on purpose: a container holds thousands and every read opens its own decompression stream, which makes Chromium fail the analysis — zip.js then masks the cause behind a `Cannot close a ERRORED writable stream` — while Firefox tolerates it. Do not turn this back into a parallel read. The read is best-effort: a reference that cannot be read leaves that file without size, never the whole package. +- The `:error` of an import entry is user-facing text, never an i18n key. The legacy SSE import stays in the backend for other clients. - File creation/duplication strips file `:data` before putting file summaries into dashboard state. - The pending invitations are a team-admin only list (backend `:is-admin` already includes the owner). The dashboard hides the invitations entry from the header tabs and the team-options dropdown for non-admins, `invitations-section-content` keys the section on `:is-admin`, and `fetch-invitations` answers `insufficient-permissions` by emitting `invitations-fetched` with `[]`, so a demoted admin does not keep the cached emails and the header does not paint the invite button from a stale list. The organization `send-invitations` policy only gates inviting and resending, not the list. diff --git a/frontend/src/app/main/data/imports.cljs b/frontend/src/app/main/data/imports.cljs index 6fd079b015..415b204b0b 100644 --- a/frontend/src/app/main/data/imports.cljs +++ b/frontend/src/app/main/data/imports.cljs @@ -15,6 +15,7 @@ (:require [app.common.exceptions :as ex] [app.common.json :as json] + [app.common.logging :as log] [app.common.schema :as sm] [app.common.uuid :as uuid] [app.main.data.jobs :as dj] @@ -25,8 +26,7 @@ [app.util.i18n :as i18n :refer [tr]] [app.util.zip :as uz] [beicon.v2.core :as rx] - [cuerdas.core :as str] - [promesa.core :as p])) + [cuerdas.core :as str])) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; MESSAGES OF A FAILURE @@ -157,25 +157,43 @@ {:files {} :objects {} :references []} entries)) +(defn- read-reference + "The metadata of one of the media or the thumbnails of a file, decoded, + paired with the file it belongs to. A reference that cannot be read is + unknown, never a failure of the package: it only leaves the size of its + file unknown." + [file-id entry] + (->> (rx/from (uz/read-as-text entry)) + (rx/map (fn [text] [file-id (json/decode text)])) + (rx/catch (fn [cause] + (log/warn :msg "cannot read a reference of the package" + :file-id file-id + :entry (.-filename ^js entry) + :cause cause) + (rx/of [file-id ::unreadable]))))) + (defn- read-objects "The objects every file of a container references, read from the - metadata of its media and its thumbnails." + metadata of its media and its thumbnails, one entry at a time. + + Every read opens its own decompression stream and the container of a + big file can hold thousands of references: reading them all at once + opens thousands of streams and is what breaks the analysis on the + browsers that cannot keep up with them." [references] - (->> (map (fn [[file-id entry]] - (->> (uz/read-as-text entry) - (p/fmap (fn [text] [file-id (json/decode text)])))) - references) - (p/all) - (p/fmap - (fn [decoded] - (reduce (fn [result [file-id data]] - (->> [:media-id :thumbnail-id] - (keep #(get data %)) - (reduce (fn [result object-id] - (update result file-id (fnil conj #{}) object-id)) - result))) - {} - decoded))))) + (->> (rx/from references) + (rx/mapcat (fn [[file-id entry]] + (read-reference file-id entry))) + (rx/reduce + (fn [[referenced unreadable] [file-id data]] + (if (= ::unreadable data) + [referenced (conj unreadable file-id)] + [(reduce (fn [referenced object-id] + (update referenced file-id (fnil conj #{}) object-id)) + referenced + (keep #(get data %) [:media-id :thumbnail-id])) + unreadable])) + [{} #{}]))) (defn- file-sizes "How much every file of a container occupies inside it: its own entries @@ -186,13 +204,17 @@ [entries] (let [{:keys [files objects references]} (index-container entries)] (->> (read-objects references) - (p/fmap - (fn [referenced] - (reduce-kv (fn [sizes file-id object-ids] - (update sizes file-id (fnil + 0) - (reduce + 0 (keep objects object-ids)))) - files - referenced)))))) + (rx/map + (fn [[referenced unreadable]] + (let [sizes (reduce-kv (fn [sizes file-id object-ids] + (update sizes file-id (fnil + 0) + (reduce + 0 (keep objects object-ids)))) + files + referenced)] + ;; a file with a reference that could not be read has no + ;; size at all: a number that misses one of its objects is + ;; worse than no number + (reduce dissoc sizes unreadable))))))) (defn- analyze-file "The entries of one file the user dropped: one per file of a version 3 @@ -215,9 +237,8 @@ (rx/merge-map (fn [manifest] (if (= (:type manifest) "penpot/export-files") - (->> (uz/get-entries zip-reader) - (p/mcat file-sizes) - (rx/from) + (->> (rx/from (uz/get-entries zip-reader)) + (rx/mapcat file-sizes) (rx/map (fn [sizes] (-> file (assoc :type :binfile-v3) diff --git a/frontend/test/frontend_tests/data/imports_test.cljs b/frontend/test/frontend_tests/data/imports_test.cljs index e173ec3671..9a58d315dd 100644 --- a/frontend/test/frontend_tests/data/imports_test.cljs +++ b/frontend/test/frontend_tests/data/imports_test.cljs @@ -210,6 +210,112 @@ (t/testing "what the container occupies is kept apart" (t/is (= [10000 10000 10000] (mapv :container-size @seen)))))) +(t/deftest ^:async the-references-of-a-container-are-read-one-at-a-time + (let [file-id (uuid/next) + media-id (str (uuid/next)) + thumb-id (str (uuid/next)) + seen (atom []) + in-flight (atom 0) + peak (atom 0) + entries + #js [#js {:filename "manifest.json" + :compressedSize 100} + #js {:filename (str "files/" file-id ".json") + :compressedSize 200} + #js {:filename (str "files/" file-id "/media/one.json") + :compressedSize 10 + :content (str "{\"mediaId\": \"" media-id "\"}")} + #js {:filename (str "files/" file-id "/thumbnails/t/p/f.json") + :compressedSize 10 + :content (str "{\"thumbnailId\": \"" thumb-id "\"}")} + #js {:filename (str "objects/" media-id ".json") + :compressedSize 20} + #js {:filename (str "objects/" media-id ".png") + :compressedSize 1000} + #js {:filename (str "objects/" thumb-id ".json") + :compressedSize 5} + #js {:filename (str "objects/" thumb-id ".png") + :compressedSize 40}]] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + uz/get-entry (mock/stub (fn [_ _] + (p/resolved #js {:content (manifest [file-id "One"])}))) + uz/get-entries (mock/stub (fn [_] (p/resolved entries))) + ;; every read counts how many are open at the same time + uz/read-as-text + (mock/stub (fn [entry] + (swap! in-flight inc) + (swap! peak max @in-flight) + (->> (p/resolved (.-content entry)) + (p/fmap (fn [content] + (swap! in-flight dec) + content))))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "package.penpot" :size 10000}]) + {:on-next #(swap! seen conj %)})))) + + (t/testing "a container of many references never opens more than one read" + ;; a container of a big file holds thousands of references, and every + ;; read opens its own decompression stream: reading them all at once + ;; is what breaks the analysis in the browser + (t/is (= 1 @peak))) + + (t/testing "every reference is still counted" + ;; 200 + 10 + 10 of its own, and 20 + 1000 and 5 + 40 of the two + ;; objects its media and its thumbnail point at + (t/is (= [1285] (mapv :size @seen)))))) + +(t/deftest ^:async a-reference-that-cannot-be-read-leaves-its-file-without-size + (let [readable-id (uuid/next) + broken-id (uuid/next) + object-id (str (uuid/next)) + broken-path (str "files/" broken-id "/media/two.json") + seen (atom []) + entries + #js [#js {:filename "manifest.json" + :compressedSize 100} + #js {:filename (str "files/" readable-id ".json") + :compressedSize 200} + #js {:filename (str "files/" readable-id "/media/one.json") + :compressedSize 10 + :content (str "{\"mediaId\": \"" object-id "\"}")} + #js {:filename (str "objects/" object-id ".json") + :compressedSize 20} + #js {:filename (str "objects/" object-id ".png") + :compressedSize 1000} + #js {:filename (str "files/" broken-id ".json") + :compressedSize 400} + #js {:filename broken-path + :compressedSize 10}]] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + uz/get-entry (mock/stub (fn [_ _] + (p/resolved #js {:content (manifest [readable-id "Readable"] + [broken-id "Broken"])}))) + uz/get-entries (mock/stub (fn [_] (p/resolved entries))) + uz/read-as-text + (mock/stub (fn [entry] + (if (= broken-path (.-filename entry)) + (p/rejected (ex-info "cannot read the entry" {:path broken-path})) + (p/resolved (.-content entry))))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "package.penpot" :size 10000}]) + {:on-next #(swap! seen conj %)})))) + + (t/testing "one unreadable reference does not fail the package" + (t/is (= [readable-id broken-id] (mapv :file-id @seen))) + (t/is (every? #(= :success (:status %)) @seen))) + + (t/testing "the file whose reference could not be read has no size" + ;; a number that misses one of its objects is worse than no number + (t/is (= [1230 nil] (mapv :size @seen)))))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; IMPORTING A PACKAGE ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;