From 464756ec593ccb7e59bb8aab036ca3a85b05beb5 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Wed, 7 Oct 2026 21:38:13 +0000 Subject: [PATCH] :sparkles: Rework the import dialog The import wizard now opens with a Check/Upload/Import stepper derived from the dialog status, plus a phase title and description per stage, and its file rows follow the new design: folder icon, file name and a size-plus-extension subtitle. The step reflects the real state: analyze is Check, a milestone in the upload stage is Upload, and queue, start and job milestones are Import. While checking, the dialog shows no rows and no spinner, and the exact files inside a binfile appear once the analysis resolves. Each row paints a progress bar: the upload stage drives it with the real chunk counter, and every import stage keeps the animated bar, because the client cannot know a global percentage of the work. The queued label joins the same status line as the milestone text and now reads "Waiting job slot". The change also drops the context-notification wrapper and the legacy loader around file names, and documents the two CSS pitfalls found on the way: a logical inline-size wins over an inline physical width, and Rumext does not convert dynamic :style maps for plain elements, so inline styles leave CLJS as #js objects. AI-assisted-by: glm-5.3-flash :wrench: Enable NativeMemoryTracking in the backend devenv Add -XX:NativeMemoryTracking=summary to JAVA_OPTS so the backend and REPL JVMs expose native memory breakdowns through jcmd. Useful to diagnose non-heap growth (metaspace, code cache, GC, thread) while working with imports and other JVM-heavy paths. AI-assisted-by: deepseek-v4.1-flash :sparkles: Rework the import wizard progress and the size of its files The milestone in course is written once, in the header, where the message of the phase was: every file of a container reports the same one, so a row repeated it. A row keeps its name and its size, and the button to drop a file only while the wizard still offers its import step, which is where dropping one makes sense. The title counts the files and counts them done with what the job reports about the file it is working on, so the number moves while a container rebuilds its files one by one. The bar measures the whole import: the real bytes of every container while they are uploaded, and animated while the wizard checks, waits for a slot or rebuilds. A file of a version 3 package shows what it occupies inside its container: its own entries plus the storage objects its media and its thumbnails reference, which the container keeps shared under `objects/`. What the container itself occupies stays in `:container-size`, the size the upload of the package is measured on. The ready step waited for every known file instead of for the end of the analysis, so Continue left the files that arrived later out of the import, with a row that never showed anything. The upload step was recognized only once a row carried a milestone, so the wizard said it was importing during the first chunk; it now holds until every file leaves the upload behind. Also fixes a misspelled design system token that left the success color unset. AI-assisted-by: deepseek-v4.1-flash :sparkles: Cover the import sizes and the done counter with tests The size a file of a version 3 package shows is its own entries plus the storage objects its media and its thumbnails reference, and what the container occupies stays apart: the test drives the analysis with a fake container that has both, and checks that the manifest is not charged to any file. The done counter of the header counts the files the job left behind: all of them for a container that finished, the ones before the file it is working on for a container in course, none for one that waits for a slot, and the files of every container add up. The test that answers one entry per file also needed the mock of the entries of a container, which the analysis now reads. AI-assisted-by: deepseek-v4.1-flash :sparkles: Say the wait for a slot in the header, not in every row A row of the import wizard no longer paints a waiting state: while no job of the import has started, the header says it waits for a free slot, and the rows keep their name and their size. Only a file whose container already finished uploading says something of its own ("Ready"), plus the error text when it fails. The clock icon and the `dashboard.import.status.waiting` key of both locales go away with it. The notes of the redesign come along: the description of the wizard in `mem:frontend/dashboard-viewer-subtleties`, and a new `mem:clojure/promesa` with the call styles of the library and how they bridge to rx. AI-assisted-by: deepseek-v4.1-flash --- .serena/memories/clojure/promesa.md | 9 + .serena/memories/critical-info.md | 1 + .../frontend/dashboard-viewer-subtleties.md | 2 +- .../ui-conventions-and-style-system.md | 2 +- backend/scripts/_env | 1 + frontend/src/app/main/data/imports.cljs | 296 ++++++++++++++- .../src/app/main/ui/dashboard/import.cljs | 336 ++++++++++++------ .../src/app/main/ui/dashboard/import.scss | 207 ++++++++++- frontend/src/app/main/ui/jobs/progress.cljs | 48 ++- .../frontend_tests/data/imports_test.cljs | 234 ++++++++++++ .../frontend_tests/ui/jobs_progress_test.cljs | 32 +- frontend/translations/en.po | 74 +++- frontend/translations/es.po | 80 ++++- 13 files changed, 1160 insertions(+), 162 deletions(-) create mode 100644 .serena/memories/clojure/promesa.md diff --git a/.serena/memories/clojure/promesa.md b/.serena/memories/clojure/promesa.md new file mode 100644 index 0000000000..75cb5f6642 --- /dev/null +++ b/.serena/memories/clojure/promesa.md @@ -0,0 +1,9 @@ +# Promesa + +Promise library of the monorepo (frontend CLJS and common, `funcool/promesa` 12.0.1 in `common/deps.edn`). Its API has two call styles and mixing them in one expression is the trap: `fmap`/`mcat`/`merr` are **fn first, promise last** (`->>` friendly) and `then`/`then'`/`catch`/`handle` are **promise first, fn last** (`->` friendly). Convention: `fmap`/`mcat`/`merr` inside `->>` pipelines, `then`/`catch` with `->`. A `then` written inside a `->>` reads backwards and hides whether the step carries a value or a promise. +- `fmap` (alias `map`, both `(f p)`) is functor map: `f` returns a plain value. +- `mcat` (alias `mapcat`, both `(f p)`) is monadic bind: `f` must return a promise, and the result mirrors it. +- `then` also unwraps a promise returned by `f`, which is why it covers both cases at the cost of not naming which one it is; `then'` maps without unwrapping. +- `merr` (fn first) and `catch` (promise first) are the error side. +- `all` takes a collection of promises (a lazy seq works); `let` chains sequential bindings. +- Seam with the rx pipelines of the frontend: `beicon.v2.core/from` builds an observable from "arrays, collections, and promise instance" (`external/beicon`), so `(->> p (p/mcat f) (rx/from) (rx/map g))` is the usual bridge (`app.main.data.imports`, `app.util.zip`). diff --git a/.serena/memories/critical-info.md b/.serena/memories/critical-info.md index 5804c47212..d4284b89bf 100644 --- a/.serena/memories/critical-info.md +++ b/.serena/memories/critical-info.md @@ -8,6 +8,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo. - Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`. - Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics. - Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API. +- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables. # Development workflow diff --git a/.serena/memories/frontend/dashboard-viewer-subtleties.md b/.serena/memories/frontend/dashboard-viewer-subtleties.md index 9c753d8875..ed99fb79fb 100644 --- a/.serena/memories/frontend/dashboard-viewer-subtleties.md +++ b/.serena/memories/frontend/dashboard-viewer-subtleties.md @@ -5,7 +5,7 @@ - Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id. - Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly. - Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications. -- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its rows paint the milestone of the job (the stage and its counter, with the file in course) instead of the messages the old SSE stream sent. Before the job exists, the rows paint a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and the row writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, each entry reports `:queued` ("Queued"); the `:start`/`:retry` events move it to bare progress, and only `:progress` carries a milestone. The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line any more: each row carries its own status. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients. +- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its header paints the milestone of the job (the stage and its counter) instead of the messages the old SSE stream sent; every file of a container reports the same milestone, so `jp/current-milestone-text` writes it once and the rows never repeat it. Before the job exists, the header paints a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and it writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, no row says anything: the header says the import waits for a slot while no job of it has started. The bar of the header measures the whole import, never a file: indeterminate while checking, waiting for a slot or rebuilding, because the client cannot know a global index and total of operations. The upload is the exception — the chunked upload knows its chunks, so the bar is determinate while the containers are sent, weighted by their size (`imp/upload-pct` over `:container-size`, the only percentage the client really knows). The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line on the rows any more: the rows of the import wizard carry only what is not progress — the name, the size, `Ready` once their container finished uploading, and the error text —, and the button that drops a file only shows while the wizard still offers its import step (the confirmation screen). Above the rows, the import wizard paints a stepper (Check/Upload/Import) plus one phase line: `imp/import-step` derives the step from the dialog status and the entries (upload holds while any file is pending, either `:import-ready` or carrying its own `:upload` milestone), so Check covers analyze, Upload covers the chunked client-side upload, and Import covers queue plus job. A file of a version 3 package shows what it occupies inside its container, computed while analyzing (`imp/file-sizes`): its own `files/…` entries plus the storage objects its media and thumbnails reference, whose metadata is the only link to `objects/…`. The container's own size stays in `:container-size` (what the dropped file weighs) and is what the upload bar weighs, never what a row shows. The subtitle is the size alone, with no extension. While analyzing with no failures nothing renders below the phase header — no rows, no spinner (`imp/awaiting-analysis?`); the header text is the only "Checking…" message. Rows appear once the analysis resolves (one per file inside the package), and a failed analysis takes the error-summary path instead. The ready step and its Continue wait for the end of the analysis (`imp/ready-to-import?`, fed with the completion of `imp/analyze`): a ready file is not enough, because the files still to arrive would never be imported. While rebuilding, the phase title counts the files and `imp/imported-count` counts them done with the job's own `:files` counter, so the number moves while a container rebuilds its files one by one. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients. - File creation/duplication strips file `:data` before putting file summaries into dashboard state. ## Viewer diff --git a/.serena/memories/frontend/ui-conventions-and-style-system.md b/.serena/memories/frontend/ui-conventions-and-style-system.md index 1161b55d3a..59390c27e7 100644 --- a/.serena/memories/frontend/ui-conventions-and-style-system.md +++ b/.serena/memories/frontend/ui-conventions-and-style-system.md @@ -12,7 +12,7 @@ - Co-located SCSS modules are preferred. Use `app.main.style/stl` helpers from CLJS and design-system SCSS tokens/mixins instead of legacy global selectors or high-specificity nesting. - Keep CSS specificity low. Avoid nested selectors unless they target elements the component owns; CSS Modules already prevent class-name collisions. -- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer. +- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer. Never mix: a stylesheet logical property (`inline-size`) and an inline physical `width` on the same element cascade separately, and the logical mapping wins — a dynamically driven width must be `width` on both sides (import modal progress bar hit this). And an inline style passed from CLJS must be a JS object (`#js {:width …}`): for plain `[:div …]` elements the Rumext macro does not convert the `:style` map at runtime — a plain CLJS map reaches React raw and React never finds the keys (static style literals are converted at compile time, which is why `:style` maps seem to work elsewhere). - Use named design-system variables/tokens for spacing, borders, fixed dimensions, colors, and typography. Avoid hardcoded px/rem values and deprecated `resources/styles/common/refactor/spacing.scss` variables. - Use component-local CSS custom properties for variants and theming instead of one-off Sass variables when a component has multiple visual states. - Prefer DS typography components (`heading*`, `text*`) and typography mixins instead of plain text wrappers or deprecated typography mixins. diff --git a/backend/scripts/_env b/backend/scripts/_env index f777d16a8a..800ad70a08 100644 --- a/backend/scripts/_env +++ b/backend/scripts/_env @@ -92,6 +92,7 @@ export JAVA_OPTS="\ -XX:+UseShenandoahGC \ -XX:+UseCompactObjectHeaders \ -XX:ShenandoahGCMode=generational \ + -XX:NativeMemoryTracking=summary \ -XX:-OmitStackTraceInFastThrow \ --sun-misc-unsafe-memory-access=allow \ --enable-preview \ diff --git a/frontend/src/app/main/data/imports.cljs b/frontend/src/app/main/data/imports.cljs index 44cdd681d3..7a77de36ad 100644 --- a/frontend/src/app/main/data/imports.cljs +++ b/frontend/src/app/main/data/imports.cljs @@ -25,7 +25,8 @@ [app.util.i18n :as i18n :refer [tr]] [app.util.zip :as uz] [beicon.v2.core :as rx] - [cuerdas.core :as str])) + [cuerdas.core :as str] + [promesa.core :as p])) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; MESSAGES OF A FAILURE @@ -98,10 +99,112 @@ (uz/read-as-text entry)))) (rx/map json/decode))) +(defn- segment-of + "The segment of a zip path that follows `prefix`, up to the next slash, + or nil when the path does not start with it." + [prefix path] + (when (str/starts-with? path prefix) + (let [rest (subs path (count prefix)) + idx (str/index-of rest "/")] + (if (some? idx) (subs rest 0 idx) rest)))) + +(defn- file-of-entry + "The file of a container a zip entry belongs to. A container keeps every + file of a package under its own prefix: `files/.json` for its data + and `files//…` for the rest." + [path] + (some-> (segment-of "files/" path) + (str/replace #"\.json$" ""))) + +(defn- object-of-entry + "The storage object a zip entry holds. A container keeps the media and + the thumbnails of its files shared under `objects/`, and both the + metadata (`objects/.json`) and the binary (`objects/.`) + belong to the same object." + [path] + (when-some [segment (segment-of "objects/" path)] + (let [dot (str/index-of segment ".")] + (if (some? dot) (subs segment 0 dot) segment)))) + +(defn- reference-entry? + "Whether an entry of a file is the metadata of one of the media or the + thumbnails that file references, the only place that links a file to the + objects it occupies." + [path] + (or (str/includes? path "/media/") + (str/includes? path "/thumbnails/"))) + +(defn- index-container + "Read the entries of a container in one pass: what every file of the + package occupies by itself, what every storage object occupies, and the + entries that say which objects a file references." + [entries] + (reduce + (fn [index entry] + (let [path (.-filename ^js entry) + size (.-compressedSize ^js entry) + file-id (file-of-entry path)] + (cond + (some? file-id) + (cond-> (update-in index [:files file-id] (fnil + 0) size) + (reference-entry? path) + (update :references conj [file-id entry])) + + (some? (object-of-entry path)) + (update-in index [:objects (object-of-entry path)] (fnil + 0) size) + + :else + index))) + {:files {} :objects {} :references []} + entries)) + +(defn- read-objects + "The objects every file of a container references, read from the + metadata of its media and its thumbnails." + [references] + (->> (map (fn [[file-id entry]] + (->> (uz/read-as-text entry) + (p/fmap (fn [text] [file-id (json/decode text)])))) + references) + (p/all) + (p/fmap + (fn [decoded] + (reduce (fn [result [file-id data]] + (->> [:media-id :thumbnail-id] + (keep #(get data %)) + (reduce (fn [result object-id] + (update result file-id (fnil conj #{}) object-id)) + result))) + {} + decoded))))) + +(defn- file-sizes + "How much every file of a container occupies inside it: its own entries + plus the storage objects it references, which the container keeps shared + and which are only linked from the file. The manifest carries neither a + size, so the parts of a package are the only account of what it + occupies." + [entries] + (let [{:keys [files objects references]} (index-container entries)] + (->> (read-objects references) + (p/fmap + (fn [referenced] + (reduce-kv (fn [sizes file-id object-ids] + (update sizes file-id (fnil + 0) + (reduce + 0 (keep objects object-ids)))) + files + referenced)))))) + (defn- analyze-file "The entries of one file the user dropped: one per file of a version 3 - package, one for a version 1 file, and an error for anything else." - [{:keys [uri] :as file}] + package, one for a version 1 file, and an error for anything else. + + A container carries its own weight and the weight of every file in it: + the entry of a file of a version 3 package sizes the file inside its + container, and keeps what the container itself occupies in + `:container-size`, which is what the upload of the package is measured + on." + [{:keys [uri size] :as file}] (let [stream (->> (slurp-uri uri :buffer) (rx/merge-map (fn [body] @@ -110,16 +213,23 @@ (= "application/zip" mtype) (let [zip-reader (uz/reader body)] (->> (read-zip-manifest zip-reader) - (rx/map + (rx/merge-map (fn [manifest] (if (= (:type manifest) "penpot/export-files") - (let [manifest (decode-manifest manifest)] - (assoc file :type :binfile-v3 :files (:files manifest))) - (assoc file :type :unknown)))) + (->> (uz/get-entries zip-reader) + (p/mcat file-sizes) + (rx/from) + (rx/map (fn [sizes] + (-> file + (assoc :type :binfile-v3) + (assoc :container-size size) + (assoc :files (:files (decode-manifest manifest))) + (assoc :sizes sizes))))) + (rx/of (assoc file :type :unknown))))) (rx/finalize (partial uz/close zip-reader)))) (= "application/octet-stream" mtype) - (rx/of (assoc file :type :binfile-v1)) + (rx/of (assoc file :type :binfile-v1 :container-size size)) :else (rx/of (assoc file :type :unknown)))))) @@ -137,19 +247,28 @@ (->> stream (rx/filter (fn [entry] (= :binfile-v3 (:type entry)))) - (rx/merge-map (fn [{:keys [files] :as entry}] + (rx/merge-map (fn [{:keys [files sizes] :as entry}] (->> (rx/from files) (rx/map (fn [file] (-> entry - (dissoc :files) + (dissoc :files :sizes) (assoc :name (:name file)) (assoc :file-id (:id file)) + ;; the size the file itself + ;; occupies inside the + ;; container, never the + ;; container's + (assoc :size (get sizes (str (:id file)) (:size entry))) (assoc :status :success)))))))) (->> stream (rx/filter (fn [data] (= :unknown (:type data)))) (rx/map (fn [_] {:uri (:uri file) + :name (:name file) + :size (:size file) + :container-size (:size file) + :type :unknown :status :error :error (tr "dashboard.import.analyze-error")})))) @@ -273,6 +392,163 @@ (rx/from (entry-messages entries {:status :error :error (cause-message cause (tr "labels.error"))})))))))) +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; WIZARD STEP, UPLOAD AND FILE SIZE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn format-file-size + "Human size of a dropped file, like `234MB` or `512KB`. + Returns nil when the size is unknown, so the row shows no subtitle." + [size] + (when (and (some? size) (number? size) (>= size 0)) + (let [kb 1024 + mb (* kb 1024) + gb (* mb 1024)] + (cond + (< size kb) + (str (js/Math.round size) "B") + + (< size mb) + (let [v (/ size kb)] + (str (if (< v 10) + (-> (.toFixed v 1) + (str/replace #"\.0$" "")) + (str (js/Math.round v))) + "KB")) + + (< size gb) + (let [v (/ size mb)] + (str (if (< v 10) + (-> (.toFixed v 1) + (str/replace #"\.0$" "")) + (str (js/Math.round v))) + "MB")) + + :else + (let [v (/ size gb)] + (str (if (< v 10) + (-> (.toFixed v 1) + (str/replace #"\.0$" "")) + (str (js/Math.round v))) + "GB")))))) + +(defn- upload-ratio + "The fraction of the container of an entry already sent: the chunks + uploaded over the chunks the file has, or nil while its upload has not + started." + [entry] + (let [{:keys [current total]} (get-in entry [:progress :counters :upload])] + (when (and (some? current) (some? total) (pos? total)) + (/ current total)))) + +(defn- uploaded-ratio + "The fraction of the container of an entry already uploaded. A container + is uploaded whole before its job exists, so from the moment its entries + leave the upload stage they count as sent: only an entry still sending + chunks, or one that has not started, says less than everything." + [entry] + (cond + (= :upload (get-in entry [:progress :stage])) (or (upload-ratio entry) 0) + (contains? #{:analyze :import-ready} (:status entry)) 0 + :else 1)) + +(defn upload-done? + "Whether the container of an entry finished uploading: the chunks sent + reached the chunks the file has." + [entry] + (let [{:keys [current total]} (get-in entry [:progress :counters :upload])] + (and (some? current) (some? total) (= current total)))) + +(defn upload-pct + "The percentage of the whole upload: the bytes already sent over the + bytes of every container. The containers are weighted by their size, + never averaged, because a `.penpot` is uploaded whole: a container of + 200MB moves the bar as much as four of 50MB." + [entries] + (let [containers (map (comp first :entries) (packages entries)) + total (reduce + 0 (keep :container-size containers))] + (when (pos? total) + (let [sent (reduce + 0 (map (fn [{:keys [container-size] :as entry}] + (* container-size (uploaded-ratio entry))) + containers))] + (js/Math.round (* 100 (/ sent total))))))) + +(defn- files-done + "How many files of a container are already rebuilt. A container that + finished leaves all of its files behind; while it runs, its milestone + says which file it is working on, one based, so the ones before it are + done." + [entries] + (if (every? #(= :import-success (:status %)) entries) + (count entries) + (let [{:keys [current total]} (get-in (first entries) [:progress :counters :files]) + total (or total (count entries))] + (-> (or current 1) dec (max 0) (min total))))) + +(defn imported-count + "How many of the files being imported are already rebuilt: the files of + the containers that finished plus the ones a running container left + behind. A container rebuilds its files one by one, so counting only the + finished containers (or only their files) would leave the number still + for the whole import." + [entries] + (reduce + 0 (map (comp files-done :entries) (packages entries)))) + +(defn waiting-slot? + "Whether every file of the wizard waits for a free slot: the jobs of the + containers exist, but no worker picked them up yet." + [entries] + (and (pos? (count entries)) + (every? #(= :import-queued (:status %)) entries))) + +(defn- pending-upload? + "Whether a file is still to be uploaded, or is being uploaded right now: + the wizard is in its upload step while any file is pending." + [entry] + (or (= :import-ready (:status entry)) + (= :upload (get-in entry [:progress :stage])))) + +(defn import-step + "The wizard step the header paints: `:check` while the entries are + analyzed, `:upload` until every file of every container is uploaded, and + `:import` once the jobs run or the wizard reaches its outcome. + + A file leaves the upload behind when its job exists: the queued message + carries no milestone, and the upload one disappears with it, so the + presence of a milestone is not enough to keep the step in the upload." + [status entries] + (case status + (:analyze :import-ready) + :check + + :import-progress + (if (some pending-upload? entries) + :upload + :import) + + (:import-success :import-error :library-resolution :library-summary) + :import + + :check)) + +(defn awaiting-analysis? + "Whether the wizard hides the rows and shows the checking placeholder + instead: the dialog is still analyzing and no entry failed yet. Entries + that fail during analyze take the error-summary path instead of rows." + [status errors?] + (and (= :analyze status) (not errors?))) + +(defn ready-to-import? + "Whether the wizard can offer its import step: the analysis is over and + every file it found can be imported. A ready file is not enough on its + own: until the analysis ends the wizard does not know how many files the + dropped containers carry, and the ones still to arrive would never be + imported." + [entries analysis-ended?] + (and analysis-ended? + (pos? (count entries)) + (every? #(= :import-ready (:status %)) entries))) + (defn import-files "Import the entries the wizard analyzed, one job per package. diff --git a/frontend/src/app/main/ui/dashboard/import.cljs b/frontend/src/app/main/ui/dashboard/import.cljs index 347364ff67..e5d5ffe2a1 100644 --- a/frontend/src/app/main/ui/dashboard/import.cljs +++ b/frontend/src/app/main/ui/dashboard/import.cljs @@ -29,7 +29,6 @@ [app.main.ui.ds.foundations.typography.heading :refer [heading*]] [app.main.ui.ds.foundations.typography.text :refer [text*]] [app.main.ui.ds.notifications.context-notification :refer [context-notification*]] - [app.main.ui.ds.product.loader :refer [loader*]] [app.main.ui.icons :as deprecated-icon] [app.main.ui.jobs.progress :as jp] [app.util.dom :as dom] @@ -52,6 +51,7 @@ (let [entries (->> entries (mapv (fn [file] {:name (.-name file) + :size (.-size file) :uri (wapi/create-uri file)})) (not-empty))] (when entries @@ -171,7 +171,7 @@ pending))))) (defn- analyze-entries - [state entries] + [state entries on-analyze-end] (->> (imp/analyze entries) (rx/mapcat #(rx/delay emit-delay (rx/of %))) (rx/filter some?) @@ -180,7 +180,11 @@ (when (some? (:error message)) (st/emit! (ev/event {::ev/name "import-files-error" :error (:error message)}))) - (swap! state update-with-analyze-result message))))) + (swap! state update-with-analyze-result message)) + (constantly nil) + ;; the wizard cannot offer its import step until it knows every + ;; file of every container + on-analyze-end))) (defn- import-files [state library-resolution-data* project-id entries on-job] @@ -202,30 +206,31 @@ (mf/defc import-entry* {::mf/memo true ::mf/private true} - [{:keys [entries entry edition can-be-deleted is-progress on-edit on-change on-delete]}] + [{:keys [entries entry edition on-edit on-change on-delete]}] (let [status (:status entry) ;; FIXME: rename to format format (:type entry) - loading? (or (= :analyze status) - (= :import-queued status) - (= :import-progress status) - (and is-progress (= :import-ready status))) analyze-error? (= :analyze-error status) - import-success? (= :import-success status) import-error? (= :import-error status) import-ready? (= :import-ready status) - level (cond - import-success? :success - import-ready? :success - import-error? :error - analyze-error? :error - loading? nil - :else :default) + progress (:progress entry) + upload? (= :upload (:stage progress)) + + ;; the container the file came in is fully sent, but its job does + ;; not exist yet: the wizard is still uploading the other files + ready? (and (= :import-progress status) + upload? + (imp/upload-done? entry)) + + ;; a file can be dropped while the wizard has not started yet: once + ;; the import runs, its entries are committed + can-be-deleted? (and import-ready? + (> (count entries) 1)) is-shared? (:shared entry) - progress (:progress entry) + subtitle (imp/format-file-size (:size entry)) file-id (:file-id entry) editing? (and (some? file-id) (= edition file-id)) @@ -265,44 +270,42 @@ [:div {:class (stl/css-case :file-entry true - :loading loading? - :success import-success? + :ready ready? + :success (= :import-success status) :error (or import-error? analyze-error?) :editable (and import-ready? (not editing?)))} - [:div {:class (stl/css :file-name)} - (when loading? [:> loader* {:width 26 :title (tr "labels.loading")}]) + [:div {:class (stl/css :file-main)} + [:div {:class (stl/css :file-icon) + :aria-hidden "true"} + [:> icon* {:icon-id i/folder :size "m"}]] - (if editing? - [:div {:class (stl/css :file-name-edit)} - [:input {:type "text" - :auto-focus true - :class (stl/css :file-name-input) - :aria-label (tr "dashboard.import.file-name-label") - :default-value (:name entry) - :on-key-press on-edit-key-press - :on-blur on-edit-blur}]] + [:div {:class (stl/css :file-meta)} + (if editing? + [:div {:class (stl/css :file-name-edit)} + [:input {:type "text" + :auto-focus true + :class (stl/css :file-name-input) + :aria-label (tr "dashboard.import.file-name-label") + :default-value (:name entry) + :on-key-press on-edit-key-press + :on-blur on-edit-blur}]] - [:div {:class (stl/css :file-name-label)} - (if loading? - [:> text* {:class (stl/css :file-name-label) - :as "span" - :typography t/body-medium} - (:name entry) - (when ^boolean is-shared? - [:> icon* {:icon-id i/library :class (stl/css :file-label-icon)}])] - [:> context-notification* - {:level level - :appearance :ghost - :class (stl/css :file-name-notification)} - [:> text* {:class (stl/css :file-name-label) + [:* + [:> text* {:class (stl/css :file-name-label) + :as "span" + :typography t/body-medium} + [:span {:class (stl/css :file-name-text)} + (:name entry)] + (when ^boolean is-shared? + [:> icon* {:icon-id i/library :class (stl/css :file-label-icon)}])] + (when (some? subtitle) + [:> text* {:class (stl/css :file-subtitle) :as "span" - :typography t/body-medium} - (:name entry) - (when ^boolean is-shared? - [:> icon* {:icon-id i/library :class (stl/css :file-label-icon)}])]])]) + :typography t/body-small} + subtitle])])] - (when ^boolean (or editable? can-be-deleted) + (when ^boolean (or editable? can-be-deleted?) [:div {:class (stl/css :edit-entry-buttons)} (when ^boolean editable? [:> icon-button* {:on-click on-edit' @@ -310,38 +313,36 @@ :icon-size "s" :aria-label (tr "labels.edit") :icon i/curve}]) - (when ^boolean can-be-deleted + (when ^boolean can-be-deleted? [:> icon-button* {:on-click on-delete' :variant "ghost" :icon-size "s" :aria-label (tr "labels.delete") - :icon i/delete}])])] - (cond - analyze-error? - [:> text* {:class (stl/css :error-message) - :as "span" - :typography t/body-small} - ;; the message is user-facing text already (a hint or a translated - ;; label), never a key - (if (some? (:error entry)) - (:error entry) - (tr "dashboard.import.analyze-error"))] + :icon i/delete}])]) - import-error? - [:> text* {:class (stl/css :error-message) - :as "span" - :typography t/body-small} - (if (some? (:error entry)) - (:error entry) - (tr "labels.error"))] + ;; a row carries its name and its size, and only the state that is not + ;; progress: the milestone in course and the files already imported are + ;; the business of the header, which writes the milestone once for + ;; every file of a container + (when ^boolean ready? + [:div {:class (stl/css :file-status)} + [:> icon* {:icon-id i/status-tick :size "s"}] + [:> text* {:as "span" + :typography t/body-medium} + (tr "dashboard.import.status.ready")]])] - (and (= :import-queued status) (not import-success?)) - [:div {:class (stl/css :progress-message)} - (tr "jobs.queued")] - - (and (not import-success?) (some? progress)) - [:div {:class (stl/css :progress-message)} - (jp/milestone-text progress :file? true)]) + (when ^boolean (or analyze-error? import-error?) + [:div {:class (stl/css :file-error)} + [:> text* {:class (stl/css :error-message) + :as "span" + :typography t/body-small} + ;; the message is user-facing text already (a hint or a translated + ;; label), never a key + (if (some? (:error entry)) + (:error entry) + (if analyze-error? + (tr "dashboard.import.analyze-error") + (tr "labels.error")))]]) ;; This is legacy code, will be removed when legacy-zip format is removed [:div {:class (stl/css :linked-libraries)} @@ -561,13 +562,126 @@ ;; ── Stage components ──────────────────────────────────────────────── +(mf/defc import-steps* + {::mf/private true} + [{:keys [step]}] + (let [order [:check :upload :import] + label {:check (tr "dashboard.import.steps.check") + :upload (tr "dashboard.import.steps.upload") + :import (tr "dashboard.import.steps.import")} + current (count (take-while #(not= % step) order))] + [:ol {:class (stl/css :import-steps) + :aria-label (tr "dashboard.import.steps.label")} + (for [[idx id] (d/enumerate order)] + (let [done? (< idx current) + active? (= idx current)] + [:li {:key (name id) + :class (stl/css-case :import-step true + :done done? + :active active?) + :aria-current (when active? "step")} + [:span {:class (stl/css :step-marker) + :aria-hidden "true"} + (if done? + [:> icon* {:icon-id i/status-tick :size "s"}] + (str (inc idx)))] + [:> text* {:as "span" + :typography t/body-medium + :class (stl/css :step-label)} + (get label id)] + (when (< idx 2) + [:span {:class (stl/css :step-line) :aria-hidden "true"}])]))])) + +(defn- importing-title + "The title of the phase that rebuilds the files, with the files the + wizard is working on and the ones already done." + [total done] + (dm/str (tr "dashboard.import.importing.title" (i18n/c total)) + " · " + (tr "dashboard.import.importing.done" (i18n/c done)))) + +(mf/defc import-phase-header* + {::mf/private true} + [{:keys [status step entries template]}] + (let [total (+ (count entries) (if (some? template) 1 0)) + ;; the file the job is working on is reported by the job itself, so + ;; the counter moves while a container rebuilds its files one by one + done (imp/imported-count entries) + + phase (cond + (= :analyze status) :check + (= :import-ready status) :ready + (not= :import-progress status) nil + (= :upload step) :upload + (imp/waiting-slot? entries) :queued + :else :importing) + + ;; the bar measures the whole import, never a single file: the + ;; upload of every container put together while they are sent, and + ;; the animated one from there on, because the stages of a job are + ;; not comparable one with another + [title desc pct] + (case phase + :check [(tr "dashboard.import.check.title") + (tr "dashboard.import.check.message") + nil] + :ready [(tr "dashboard.import.ready.title") + (tr "dashboard.import.ready.message") + nil] + :upload [(tr "dashboard.import.upload.title") + (tr "dashboard.import.upload.message") + (imp/upload-pct entries)] + :queued [(tr "dashboard.import.queued.title") + (tr "dashboard.import.queued.message") + nil] + :importing [(importing-title total done) + (tr "dashboard.import.importing.message") + nil] + [nil nil nil]) + + ;; the milestone in course takes the place of the message: the files + ;; of a container share it, so writing it here keeps it once + desc + (or (jp/current-milestone-text entries) desc) + + ;; the bar paints what is running: indeterminate while the wizard + ;; works —checking, waiting for a slot, rebuilding— and determinate + ;; while it uploads, which is the only percentage it really knows + bar? (contains? #{:check :upload :queued :importing} phase)] + + (when (some? phase) + [:div {:class (stl/css :phase-header)} + [:> heading* {:level 3 + :typography t/headline-medium + :class (stl/css :phase-title)} + title] + [:> text* {:as "p" + :typography t/body-medium + :class (stl/css :phase-message)} + desc] + (when bar? + [:div {:class (stl/css-case :progress-bar true + :determinate (some? pct)) + :role "progressbar" + :aria-valuemin 0 + :aria-valuemax 100 + :aria-valuenow pct} + [:div {:class (stl/css :progress-fill) + :style (when (some? pct) + #js {:width (str pct "%")})}]])]))) + (mf/defc import-files-stage* {::mf/private true} - [{:keys [entries template status errors? import-success-total auto-linked-count + [{:keys [entries template status step errors? import-success-total auto-linked-count edition on-edit on-change on-delete on-cancel on-cancel-import on-continue on-accept pending-analysis?]}] [:* + [:> import-steps* {:step step}] [:div {:class (stl/css :modal-content)} + [:> import-phase-header* {:status status + :step step + :entries entries + :template template}] (when (and (= :analyze status) errors?) [:> context-notification* {:level :warning @@ -590,7 +704,8 @@ :class (stl/css :context-notification-error)} (tr "dashboard.import.import-error.disclaimer")]) - (if (or (= :import-error status) (and (= :analyze status) errors?)) + (cond + (or (= :import-error status) (and (= :analyze status) errors?)) [:div {:class (stl/css :import-error-disclaimer)} [:div (tr "dashboard.import.import-error.message1")] [:ul {:class (stl/css :import-error-list)} @@ -614,20 +729,24 @@ (tr "dashboard.import.import-error.unknown-error"))])]))] [:div (tr "dashboard.import.import-error.message2")]] + (imp/awaiting-analysis? status errors?) + ;; while checking, the phase header carries the message: no rows, + ;; no spinners + nil + + :else (for [entry entries] [:> import-entry* {:edition edition :key (dm/str (:uri entry) "/" (:file-id entry)) :entry entry :entries entries - :is-progress (= :import-progress status) :on-edit on-edit :on-change on-change - :on-delete on-delete - :can-be-deleted (> (count entries) 1)}])) + :on-delete on-delete}])) - (when (some? template) - [:> import-entry* {:entry (assoc template :status status) - :can-be-deleted false}])] + (when (and (some? template) + (not (imp/awaiting-analysis? status errors?))) + [:> import-entry* {:entry (assoc template :status status)}])] [:div {:class (stl/css :modal-footer)} [:div {:class (stl/css :action-buttons)} @@ -736,6 +855,13 @@ status* (mf/use-state :analyze) status (deref status*) + ;; the analysis of the dropped files reached its end: only then the + ;; wizard knows every file it is going to import + analysis-ended* + (mf/use-state false) + analysis-ended? + (deref analysis-ended*) + ;; Jobs created while importing, to cancel them on demand, and ;; the subscription to their messages, to stop listening. jobs* (mf/use-state #{}) @@ -971,13 +1097,12 @@ (every? #(contains? visited (:id %)) unresolved-files)) (reset! status* :library-summary))) - (mf/with-effect [entries resolution] + (mf/with-effect [entries resolution analysis-ended?] (cond (some? template) (reset! status* :import-ready) - (and (seq entries) - (every? #(= :import-ready (:status %)) entries)) + (imp/ready-to-import? entries analysis-ended?) (reset! status* :import-ready) (and (seq entries) @@ -994,7 +1119,7 @@ ;; Run analyze operation on component mount (mf/with-effect [] - (let [sub (analyze-entries state* entries)] + (let [sub (analyze-entries state* entries #(reset! analysis-ended* true))] (partial rx/dispose! sub))) [:div {:class (stl/css :modal-overlay)} @@ -1003,7 +1128,7 @@ [:> heading* {:level 2 :typography t/headline-large :class (stl/css :modal-title)} - (tr "dashboard.import")] + (tr "dashboard.import.title")] [:> icon-button* {:variant "ghost" :aria-label (tr "labels.close") :on-click on-cancel @@ -1016,6 +1141,7 @@ {:entries entries :template template :status status + :step (imp/import-step status entries) :errors? errors? :import-success-total import-success-total :auto-linked-count auto-linked-count @@ -1030,23 +1156,27 @@ :pending-analysis? pending-analysis?}] :library-resolution - [:> import-library-resolution-stage* - {:current-unresolved-file current-unresolved-file - :selection selection - :on-select manage-on-select - :on-disconnect manage-on-disconnect - :visited visited - :last-file? last-unresolved-file? - :on-wizard-prev on-wizard-prev - :on-wizard-next on-wizard-next - :on-wizard-skip on-wizard-skip}] + [:* + [:> import-steps* {:step :import}] + [:> import-library-resolution-stage* + {:current-unresolved-file current-unresolved-file + :selection selection + :on-select manage-on-select + :on-disconnect manage-on-disconnect + :visited visited + :last-file? last-unresolved-file? + :on-wizard-prev on-wizard-prev + :on-wizard-next on-wizard-next + :on-wizard-skip on-wizard-skip}]] :library-summary - [:> import-library-summary-stage* - {:resolution resolution - :selection selection - :visited visited - :on-summary-back on-summary-back - :on-confirm-library-links on-confirm-library-links}] + [:* + [:> import-steps* {:step :import}] + [:> import-library-summary-stage* + {:resolution resolution + :selection selection + :visited visited + :on-summary-back on-summary-back + :on-confirm-library-links on-confirm-library-links}]] nil)]])) \ No newline at end of file diff --git a/frontend/src/app/main/ui/dashboard/import.scss b/frontend/src/app/main/ui/dashboard/import.scss index eafdc1ecc6..b67a2fe80a 100644 --- a/frontend/src/app/main/ui/dashboard/import.scss +++ b/frontend/src/app/main/ui/dashboard/import.scss @@ -34,7 +34,7 @@ background-color: var(--color-background-primary); border: $b-2 solid var(--color-background-quaternary); min-block-size: $sz-192; - inline-size: px2rem(640); + inline-size: px2rem(720); max-block-size: px2rem(800); } @@ -56,6 +56,104 @@ .modal-title { color: var(--color-foreground-primary); + text-transform: uppercase; + letter-spacing: 0.02em; +} + +// ################################ +// WIZARD STEPS +// ################################ + +.import-steps { + display: flex; + align-items: center; + gap: 0; + list-style: none; + margin: 0; + padding: 0; +} + +.import-step { + --step-fg-color: var(--color-foreground-secondary); + --step-marker-border: var(--color-foreground-secondary); + + display: flex; + align-items: center; + gap: var(--sp-s); + flex-shrink: 0; + + &:not(:last-child) { + flex: 1 1 auto; + } + + &.active { + --step-fg-color: var(--color-foreground-primary); + --step-marker-border: var(--color-accent-primary); + } + + &.done { + --step-fg-color: var(--color-foreground-secondary); + --step-marker-border: var(--color-accent-primary); + --step-marker-bg-color: var(--color-accent-primary); + --step-marker-fg-color: var(--color-background-secondary); + } +} + +.step-marker { + display: flex; + justify-content: center; + align-items: center; + block-size: $sz-24; + inline-size: $sz-24; + flex-shrink: 0; + border-radius: 50%; + border: $b-1 solid var(--step-marker-border); + background-color: var(--step-marker-bg-color, transparent); + color: var(--step-marker-fg-color, var(--step-marker-border)); + font-size: px2rem(12); + font-weight: 500; + + svg { + block-size: $sz-16; + inline-size: $sz-16; + } +} + +.step-label { + color: var(--step-fg-color); + white-space: nowrap; +} + +.step-line { + flex: 1 1 auto; + block-size: $b-1; + min-inline-size: $sz-32; + margin-inline: var(--sp-m); + background-color: var(--color-background-quaternary); +} + +.import-step.active .step-line, +.import-step.done .step-line { + background-color: var(--color-background-quaternary); +} + +// ################################ +// PHASE HEADER +// ################################ + +.phase-header { + display: flex; + flex-direction: column; + gap: var(--sp-xs); +} + +.phase-title { + color: var(--color-foreground-primary); +} + +.phase-message { + color: var(--color-foreground-secondary); + margin: 0; } .modal-close-btn { @@ -86,8 +184,9 @@ flex-direction: column; gap: var(--sp-m); + &.ready, &.success { - --file-entry-fg-color: var(--color-accent-sucess); + --file-entry-fg-color: var(--color-accent-success); } &.error { @@ -99,16 +198,10 @@ } } -.error-message, -.progress-message { - display: flex; - align-items: center; - min-block-size: $sz-32; - color: var(--file-entry-fg-color); -} - .error-message { + display: flex; align-items: flex-start; + color: var(--file-entry-fg-color); white-space: pre-wrap; overflow-wrap: anywhere; } @@ -153,6 +246,86 @@ color: var(--file-entry-fg-color); } +.file-main { + display: flex; + align-items: center; + gap: var(--sp-m); +} + +.file-icon { + display: flex; + justify-content: center; + align-items: center; + block-size: px2rem(40); + inline-size: px2rem(40); + flex-shrink: 0; + border-radius: $br-8; + background-color: var(--color-background-tertiary); + color: var(--color-foreground-secondary); +} + +.file-meta { + display: flex; + flex-direction: column; + gap: var(--sp-xxs); + flex: 1 1 auto; + min-inline-size: 0; +} + +.file-subtitle { + color: var(--color-foreground-secondary); +} + +.file-status { + display: flex; + align-items: center; + gap: var(--sp-s); + flex-shrink: 0; + color: var(--file-entry-fg-color); + white-space: nowrap; +} + +.file-error { + padding-inline-start: calc(px2rem(40) + var(--sp-m)); +} + +.progress-bar { + block-size: px2rem(4); + border-radius: $br-8; + background-color: var(--color-background-tertiary); + overflow: hidden; +} + +.progress-fill { + block-size: 100%; + border-radius: $br-8; + background-color: var(--color-accent-primary); + + // the client cannot know a global percentage of the import work, so + // the fill slides through the bar while the jobs run; the upload is + // the exception, its chunk counters are real and the inline width + // style drives the fill then. The rule is written as physical `width` + // on purpose: the determinate fill gets its width from an inline + // style, and a logical `inline-size` here would map after the cascade + // and win over it. + width: 40%; + animation: progress-slide 1.2s ease-in-out infinite; + + .determinate & { + animation: none; + } +} + +@keyframes progress-slide { + from { + transform: translateX(-100%); + } + + to { + transform: translateX(250%); + } +} + .edit-entry-buttons { display: flex; align-items: center; @@ -175,7 +348,15 @@ display: flex; align-items: center; gap: var(--sp-m); - flex-grow: 1; + min-inline-size: 0; + color: var(--color-foreground-primary); +} + +// the status sits at the end of the same line: a long name gives way to it +.file-name-text { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; } .file-label-icon { @@ -214,10 +395,6 @@ --context-notification-bg-color: var(--color-background-primary); } -.file-name-notification { - flex-grow: 1; -} - .import-error-disclaimer { color: var(--color-foreground-secondary); } diff --git a/frontend/src/app/main/ui/jobs/progress.cljs b/frontend/src/app/main/ui/jobs/progress.cljs index 83967aa978..c3f1404a66 100644 --- a/frontend/src/app/main/ui/jobs/progress.cljs +++ b/frontend/src/app/main/ui/jobs/progress.cljs @@ -5,8 +5,8 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns app.main.ui.jobs.progress - "How a milestone of a job is written in a row: the stage in course and - its counter. + "How the milestone of a job is written: the stage in course and its + counter. The stage is a stable key of the vocabulary of the worker, never a display string, so it is resolved here with a `case` of literal `tr` @@ -42,28 +42,38 @@ (str current "/" total) (str current)))) +(defn upload-pct + "The whole percentage of the file already uploaded. The chunked upload + knows how many chunks the file has and which one is in course, so + this is the only counter the client can trust as a percentage." + [{:keys [counters]}] + (let [{:keys [current total]} (get counters :upload)] + (when (and (some? current) (some? total) (pos? total)) + (js/Math.round (* 100 (/ current total)))))) + (defn- upload-label "The upload of a file counts chunks, not units of work, so it is written as the percentage of the file already sent instead of a counter." - [{:keys [current total]}] - (when (and (some? current) (some? total) (pos? total)) - (str "(" (js/Math.round (* 100 (/ current total))) "%)"))) + [milestone] + (when-some [pct (upload-pct milestone)] + (str "(" pct "%)"))) (defn milestone-text - "The text of a milestone: the stage in course and its counter, and with - `:file?` the file in course behind it, when the milestone carries its - counter. The upload stage is the exception: it counts chunks, so it is - written as a percentage of the file." - [milestone & {:keys [file?]}] + "The text of a milestone: the stage in course and its counter. The + upload of a file counts chunks, not units of work, so it is written as + the percentage of the file already sent instead of a counter." + [milestone] (when-some [stage (:stage milestone)] - (let [counters (:counters milestone) - own (if (= :upload stage) - (upload-label (get counters :upload)) - (counter-label (get counters stage)))] + (let [own (if (= :upload stage) + (upload-label milestone) + (counter-label (get (:counters milestone) stage)))] (str (stage-label stage) (when (some? own) - (str " " own)) - (when-let [files (and file? - (not= :files stage) - (counter-label (get counters :files)))] - (str " · " (stage-label :files) " " files)))))) + (str " " own)))))) + +(defn current-milestone-text + "The text of the milestone in course, for a caller that writes the + progress once for a whole group of files: every file of a container + reports the same milestone, so a row would repeat it." + [entries] + (some (comp milestone-text :progress) entries)) diff --git a/frontend/test/frontend_tests/data/imports_test.cljs b/frontend/test/frontend_tests/data/imports_test.cljs index e73802fe38..63b3e764de 100644 --- a/frontend/test/frontend_tests/data/imports_test.cljs +++ b/frontend/test/frontend_tests/data/imports_test.cljs @@ -137,6 +137,7 @@ (mock/with-mocks* {http/send! (slurp-stub zip-magic) uz/get-entry (mock/stub (fn [_ _] (p/resolved ::entry))) + uz/get-entries (mock/stub (fn [_] (p/resolved #js []))) uz/read-as-text (mock/stub (fn [_] (p/resolved (manifest [first-id "First"] [second-id "Second"])))) uz/close (mock/stub (fn [_] nil))} @@ -153,6 +154,55 @@ (t/testing "all of them share the upload of the file that was dropped" (t/is (every? #(= uri (:uri %)) @seen))))))) +(t/deftest ^:async a-version-3-package-sizes-every-file-with-what-it-occupies + (let [first-id (uuid/next) + second-id (uuid/next) + object-id (str (uuid/next)) + seen (atom []) + ;; the entries of a container: every file keeps its own under + ;; `files/`, the binaries of its media shared under `objects/`, + ;; and the manifest belongs to the container itself + entries + #js [#js {:filename "manifest.json" + :compressedSize 100} + #js {:filename (str "files/" first-id ".json") + :compressedSize 200} + #js {:filename (str "files/" first-id "/pages/one.json") + :compressedSize 300} + #js {:filename (str "files/" first-id "/media/one.json") + :compressedSize 10 + :content (str "{\"mediaId\": \"" object-id "\"}")} + #js {:filename (str "objects/" object-id ".json") + :compressedSize 20} + #js {:filename (str "objects/" object-id ".png") + :compressedSize 1000} + #js {:filename (str "files/" second-id ".json") + :compressedSize 400}]] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + ;; the manifest is read through the same reader as the rest of the + ;; entries, so the fake carries its text and the mock answers what + ;; every entry holds + uz/get-entry (mock/stub (fn [_ _] + (p/resolved #js {:content (manifest [first-id "First"] + [second-id "Second"])}))) + uz/get-entries (mock/stub (fn [_] (p/resolved entries))) + uz/read-as-text (mock/stub (fn [entry] (p/resolved (.-content entry)))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "package.penpot" :size 10000}]) + {:on-next #(swap! seen conj %)})))) + + (t/testing "a file occupies its own entries and the objects it references" + ;; 200 + 300 + 10 of its own, and 20 + 1000 of the object its media + ;; points at; the 100 of the manifest belong to the container + (t/is (= [1530 400] (mapv :size @seen)))) + + (t/testing "what the container occupies is kept apart" + (t/is (= [10000 10000] (mapv :container-size @seen)))))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; IMPORTING A PACKAGE ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -510,6 +560,190 @@ (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) (await done)))))) +(t/deftest format-file-size-writes-human-sizes + (t/testing "unknown sizes show nothing" + (t/is (nil? (imp/format-file-size nil))) + (t/is (nil? (imp/format-file-size -1)))) + + (t/testing "bytes, kilobytes, megabytes and gigabytes" + (t/is (= "0B" (imp/format-file-size 0))) + (t/is (= "512B" (imp/format-file-size 512))) + (t/is (= "1KB" (imp/format-file-size 1024))) + (t/is (= "1.5KB" (imp/format-file-size 1536))) + (t/is (= "512KB" (imp/format-file-size (* 512 1024)))) + (t/is (= "234MB" (imp/format-file-size (* 234 1024 1024)))) + (t/is (= "1.2GB" (imp/format-file-size (* 1.2 1024 1024 1024)))))) + +(defn- uploading + "A file of the wizard while the container it came in is uploaded: the + chunks sent out of the chunks the container has." + [uri size current total] + {:file-id (uuid/next) + :uri uri + :size (* 1024 1024) + :container-size size + :type :binfile-v3 + :status :import-progress + :progress {:stage :upload :counters {:upload {:current current :total total}}}}) + +(t/deftest upload-pct-weighs-every-container-by-its-size + (let [small (* 100 1024 1024) + big (* 300 1024 1024)] + + (t/testing "the bytes sent of every container over the bytes of all of them" + ;; 100MB of 100MB and half of 300MB: 250 of 400, not the average of + ;; 100% and 50% + (t/is (= 63 (imp/upload-pct [(uploading "a" small 4 4) + (uploading "b" big 1 2)])))) + + (t/testing "a container that has not started counts as nothing sent" + (t/is (= 25 (imp/upload-pct [(uploading "a" small 4 4) + {:uri "b" :container-size big :type :binfile-v3 + :status :import-ready}])))) + + (t/testing "the files of one container are one container" + (t/is (= 100 (imp/upload-pct [(uploading "a" big 1 1) + (uploading "a" big 1 1)])))) + + (t/testing "a container with no size leaves the bar indeterminate" + (t/is (nil? (imp/upload-pct []))) + (t/is (nil? (imp/upload-pct [{:uri "a" :type :binfile-v3 :status :import-ready}])))))) + +(t/deftest a-container-past-the-upload-counts-whole + ;; the job of a container is created with its upload as its input, so the + ;; moment the entries leave the upload stage the container is sent + (t/is (= 100 (imp/upload-pct [{:uri "a" :name "a.penpot" :container-size (* 100 1024 1024) + :type :binfile-v1 :status :import-queued} + {:uri "b" :name "b.penpot" :container-size (* 300 1024 1024) + :type :binfile-v1 :status :import-success}])))) + +(t/deftest upload-done?-tells-when-the-chunks-of-a-container-are-sent + (t/testing "the last chunk closes the upload" + (t/is (true? (imp/upload-done? (uploading "a" 100 3 3)))) + (t/is (false? (imp/upload-done? (uploading "a" 100 2 3))))) + + (t/testing "a file that is not uploading yet is not done" + (t/is (false? (imp/upload-done? {:status :import-ready}))) + (t/is (false? (imp/upload-done? {}))))) + +(t/deftest imported-count-counts-the-files-the-job-left-behind + (let [file (fn [uri status progress] + (cond-> {:uri uri :type :binfile-v3 :status status} + (some? progress) (assoc :progress progress)))] + + (t/testing "a container that finished counts all of its files" + (t/is (= 2 (imp/imported-count [(file "a" :import-success nil) + (file "a" :import-success nil)])))) + + (t/testing "a running container counts the files it left behind" + ;; its milestone says it is working on the second file of three + (t/is (= 1 (imp/imported-count [(file "a" :import-progress + {:stage :pages + :counters {:files {:current 2 :total 3} + :pages {:current 1 :total 5}}})])))) + + (t/testing "a container that has not started has nothing done" + (t/is (= 0 (imp/imported-count [(file "a" :import-queued nil)]))) + (t/is (= 0 (imp/imported-count [(file "a" :import-progress + {:stage :pages :counters {}})])))) + + (t/testing "the files of every container add up" + (t/is (= 3 (imp/imported-count [(file "a" :import-success nil) + (file "a" :import-success nil) + (file "b" :import-progress + {:stage :pages + :counters {:files {:current 2 :total 4}}})])))) + + (t/testing "no files, nothing done" + (t/is (= 0 (imp/imported-count [])))))) + +(t/deftest waiting-slot?-is-true-until-a-worker-picks-the-jobs-up + (t/testing "every file queued is a wizard waiting for a slot" + (t/is (true? (imp/waiting-slot? [{:status :import-queued} + {:status :import-queued}])))) + + (t/testing "a job in course leaves the wait behind" + (t/is (false? (imp/waiting-slot? [{:status :import-queued} + {:status :import-progress}])))) + + (t/testing "a file already imported is not waiting either" + (t/is (false? (imp/waiting-slot? [{:status :import-success}])))) + + (t/testing "no files, no wait" + (t/is (false? (imp/waiting-slot? []))))) + +(t/deftest awaiting-analysis-hides-the-rows-while-checking + (t/testing "only a clean analysis shows the placeholder" + (t/is (true? (imp/awaiting-analysis? :analyze false))) + (t/is (false? (imp/awaiting-analysis? :analyze true)))) + (t/testing "any later stage shows the rows" + (t/is (false? (imp/awaiting-analysis? :import-ready false))) + (t/is (false? (imp/awaiting-analysis? :import-progress false))) + (t/is (false? (imp/awaiting-analysis? :import-success false))))) + +(t/deftest ready-to-import?-waits-for-the-whole-analysis + (t/testing "every file found and ready, with the analysis over" + (t/is (true? (imp/ready-to-import? [{:status :import-ready} + {:status :import-ready}] + true)))) + + (t/testing "a file that has not arrived keeps the wizard checking" + ;; the files of a container are only known once its manifest is read, so + ;; offering the import step here would leave the missing one behind + (t/is (false? (imp/ready-to-import? [{:status :import-ready}] false)))) + + (t/testing "a file that cannot be imported is not ready either" + (t/is (false? (imp/ready-to-import? [{:status :import-ready} + {:status :analyze-error}] + true)))) + + (t/testing "no files at all is not ready" + (t/is (false? (imp/ready-to-import? [] true))))) + +(t/deftest import-step-follows-the-wizard-status + (t/testing "analysis paints the check step" + (t/is (= :check (imp/import-step :analyze []))) + (t/is (= :check (imp/import-step :import-ready [])))) + + (t/testing "an upload milestone paints the upload step" + (let [entries [{:file-id (uuid/next) + :status :import-progress + :progress {:stage :upload :counters {:upload {:current 1 :total 2}}}}]] + (t/is (= :upload (imp/import-step :import-progress entries))))) + + (t/testing "a file that has not reported anything yet is still uploading" + ;; the wizard is uploading from the click, before the first chunk arrives + (let [entries [{:file-id (uuid/next) :status :import-ready} + {:file-id (uuid/next) + :status :import-progress + :progress {:stage :upload :counters {:upload {:current 1 :total 2}}}}]] + (t/is (= :upload (imp/import-step :import-progress entries))))) + + (t/testing "a finished upload with its job still to come is uploading" + (let [entries [{:file-id (uuid/next) + :status :import-progress + :progress {:stage :upload :counters {:upload {:current 2 :total 2}}}} + {:file-id (uuid/next) + :status :import-progress + :progress {:stage :upload :counters {:upload {:current 1 :total 4}}}}]] + (t/is (= :upload (imp/import-step :import-progress entries))))) + + (t/testing "queued or job milestones paint the import step" + (let [queued [{:file-id (uuid/next) :status :import-queued}] + started [{:file-id (uuid/next) :status :import-progress}] + pages [{:file-id (uuid/next) + :status :import-progress + :progress {:stage :pages :counters {:pages {:current 1 :total 8}}}}]] + (t/is (= :import (imp/import-step :import-progress queued))) + (t/is (= :import (imp/import-step :import-progress started))) + (t/is (= :import (imp/import-step :import-progress pages))))) + + (t/testing "terminal and resolution stages paint the import step" + (t/is (= :import (imp/import-step :import-success []))) + (t/is (= :import (imp/import-step :import-error []))) + (t/is (= :import (imp/import-step :library-resolution []))) + (t/is (= :import (imp/import-step :library-summary []))))) + (t/deftest ^:async cancel-job-asks-the-server-and-ignores-failures (let [calls (atom []) job-id (uuid/next)] diff --git a/frontend/test/frontend_tests/ui/jobs_progress_test.cljs b/frontend/test/frontend_tests/ui/jobs_progress_test.cljs index 9d25f852af..55937e561e 100644 --- a/frontend/test/frontend_tests/ui/jobs_progress_test.cljs +++ b/frontend/test/frontend_tests/ui/jobs_progress_test.cljs @@ -5,8 +5,7 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns frontend-tests.ui.jobs-progress-test - "How a job milestone is written: the stage and its counter, and the - upload stage as a percentage of the file." + "How a job milestone is written: the stage and its counter." (:require [app.main.ui.jobs.progress :as jp] [app.util.i18n :as i18n] @@ -24,15 +23,7 @@ (t/is (= "TR:jobs.progress.stage.pages 3" (jp/milestone-text {:stage :pages :counters {:pages {:current 3}}})))))) -(t/deftest a-file-milestone-appends-the-file-counter - (with-redefs [i18n/tr stub-tr] - (t/is (= "TR:jobs.progress.stage.pages 3/8 · TR:jobs.progress.stage.files 1/2" - (jp/milestone-text {:stage :pages - :counters {:pages {:current 3 :total 8} - :files {:current 1 :total 2}}} - :file? true))))) - -(t/deftest an-upload-milestone-writes-the-percentage-of-the-file +(t/deftest the-upload-milestone-writes-the-percentage-of-the-file (with-redefs [i18n/tr stub-tr] (t/testing "the chunks sent become a whole percentage" (t/is (= "TR:jobs.progress.stage.upload (33%)" @@ -42,3 +33,22 @@ (t/testing "an upload with no total keeps the bare label" (t/is (= "TR:jobs.progress.stage.upload" (jp/milestone-text {:stage :upload :counters {:upload {:current 1}}})))))) + +(t/deftest the-milestone-in-course-is-the-one-of-the-files-that-report-it + (with-redefs [i18n/tr stub-tr] + (t/testing "the files of a container share their milestone, so one is enough" + (t/is (= "TR:jobs.progress.stage.pages 4/5" + (jp/current-milestone-text + [{:status :import-queued} + {:progress {:stage :pages :counters {:pages {:current 4 :total 5}}}} + {:progress {:stage :pages :counters {:pages {:current 4 :total 5}}}}])))) + + (t/testing "a file with nothing to report says nothing" + (t/is (nil? (jp/current-milestone-text [{:status :import-ready} + {:status :import-queued}]))) + (t/is (nil? (jp/current-milestone-text [])))))) + +(t/deftest a-milestone-without-a-stage-writes-nothing + (with-redefs [i18n/tr stub-tr] + (t/is (nil? (jp/milestone-text nil))) + (t/is (nil? (jp/milestone-text {}))))) diff --git a/frontend/translations/en.po b/frontend/translations/en.po index 9e4020a717..5a33b78c61 100644 --- a/frontend/translations/en.po +++ b/frontend/translations/en.po @@ -673,6 +673,70 @@ msgstr "Connect selected libraries" msgid "dashboard.import.file-name-label" msgstr "File name" +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.title" +msgstr "Import file" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.status.ready" +msgstr "Ready" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.label" +msgstr "Import steps" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.check" +msgstr "Check" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.upload" +msgstr "Upload" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.import" +msgstr "Import" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.check.title" +msgstr "Checking your files..." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.check.message" +msgstr "Making sure each file can be imported. This only takes a moment." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.ready.title" +msgstr "Files ready" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.ready.message" +msgstr "Your files can be imported. Continue to upload them." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.upload.title" +msgstr "Uploading your files..." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.upload.message" +msgstr "Sending your files. Keep this dialog open until the upload ends." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.importing.title" +msgid_plural "dashboard.import.importing.title" +msgstr[0] "Importing 1 file" +msgstr[1] "Importing %s files" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.importing.done" +msgid_plural "dashboard.import.importing.done" +msgstr[0] "1 done" +msgstr[1] "%s done" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.importing.message" +msgstr "Rebuilding pages, components and assets in your project." + #: src/app/main/ui/dashboard/import.cljs:619 msgid "dashboard.import.import-error.check-error" msgstr "We couldn't verify this file." @@ -715,6 +779,14 @@ msgstr "Next file" msgid "dashboard.import.no-perms" msgstr "You don’t have permission to import to this team" +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.queued.title" +msgstr "Waiting to start..." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.queued.message" +msgstr "Your files are uploaded. The import starts as soon as a slot is free." + #: src/app/main/ui/dashboard/import.cljs:443 msgid "dashboard.import.resolve-libraries" msgstr "" @@ -2668,7 +2740,7 @@ msgstr "Uploading file…" #: src/app/main/ui/dashboard/import.cljs:328, src/app/main/ui/exports/files.cljs:103 msgid "jobs.queued" -msgstr "Queued" +msgstr "Waiting job slot" #: src/app/main/ui/dashboard/comments.cljs:94 msgid "label.mark-all-as-read" diff --git a/frontend/translations/es.po b/frontend/translations/es.po index 1cf8ab2e51..35a7c50984 100644 --- a/frontend/translations/es.po +++ b/frontend/translations/es.po @@ -672,6 +672,16 @@ msgstr[1] "%s bibliotecas fueron vinculadas automáticamente por nombre." msgid "dashboard.import.bad-url" msgstr "La importación ha fallado. La URL de la plantilla es incorrecta" +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.check.message" +msgstr "" +"Estamos comprobando que cada archivo se puede importar. Solo tardará un " +"momento." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.check.title" +msgstr "Comprobando tus archivos..." + #: src/app/main/ui/dashboard/import.cljs:733 msgid "dashboard.import.confirm-library-links" msgstr "Confirmar vínculos de biblioteca" @@ -718,6 +728,22 @@ msgstr[1] "%s ficheros se han importado correctamente." msgid "dashboard.import.import-warning" msgstr "Algunos ficheros contenían objetos erroneos que no han sido importados." +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.importing.title" +msgid_plural "dashboard.import.importing.title" +msgstr[0] "Importando 1 archivo" +msgstr[1] "Importando %s archivos" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.importing.done" +msgid_plural "dashboard.import.importing.done" +msgstr[0] "1 hecho" +msgstr[1] "%s hechos" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.importing.message" +msgstr "Reconstruyendo páginas, componentes y recursos en tu proyecto." + #: src/app/main/ui/dashboard/import.cljs:710 msgid "dashboard.import.next-file" msgstr "Siguiente archivo" @@ -726,6 +752,24 @@ msgstr "Siguiente archivo" msgid "dashboard.import.no-perms" msgstr "No tienes permisos para importar en este equipo" +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.queued.title" +msgstr "Esperando a empezar..." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.queued.message" +msgstr "" +"Tus archivos ya están subidos. La importación empieza en cuanto haya un hueco " +"libre." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.ready.message" +msgstr "Tus archivos se pueden importar. Continúa con la subida." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.ready.title" +msgstr "Archivos listos" + #: src/app/main/ui/dashboard/import.cljs:443 msgid "dashboard.import.resolve-libraries" msgstr "" @@ -744,6 +788,26 @@ msgstr "conectar a" msgid "dashboard.import.resolve-libraries.original-library" msgstr "biblioteca original" +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.status.ready" +msgstr "Listo" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.check" +msgstr "Comprobación" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.import" +msgstr "Importación" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.label" +msgstr "Pasos de la importación" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.steps.upload" +msgstr "Subida" + #: src/app/main/ui/dashboard/import.cljs:521 msgid "dashboard.import.summary.linked" msgstr "auto-vinculada" @@ -764,6 +828,20 @@ msgstr "Ninguna biblioteca seleccionada" msgid "dashboard.import.summary.original" msgstr "Original" +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.title" +msgstr "Importar archivo" + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.upload.message" +msgstr "" +"Enviando tus archivos. Mantén este diálogo abierto hasta que termine la " +"subida." + +#: src/app/main/ui/dashboard/import.cljs +msgid "dashboard.import.upload.title" +msgstr "Subiendo tus archivos..." + #: src/app/main/ui/dashboard/team.cljs:881 msgid "dashboard.invitation-modal.delete" msgstr "Vas a eliminar las invitaciones para:" @@ -2706,7 +2784,7 @@ msgstr "Subiendo archivo…" #: src/app/main/ui/dashboard/import.cljs:328, src/app/main/ui/exports/files.cljs:103 msgid "jobs.queued" -msgstr "En cola" +msgstr "Esperando hueco para el job" #: src/app/main/ui/dashboard/comments.cljs:94 msgid "label.mark-all-as-read"