github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-10-29 14:00:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Rework the import dialog

Browse Source 
The import wizard now opens with a Check/Upload/Import stepper derived
from the dialog status, plus a phase title and description per stage,
and its file rows follow the new design: folder icon, file name and a
size-plus-extension subtitle.

The step reflects the real state: analyze is Check, a milestone in the
upload stage is Upload, and queue, start and job milestones are Import.
While checking, the dialog shows no rows and no spinner, and the exact
files inside a binfile appear once the analysis resolves.

Each row paints a progress bar: the upload stage drives it with the
real chunk counter, and every import stage keeps the animated bar,
because the client cannot know a global percentage of the work. The
queued label joins the same status line as the milestone text and now
reads "Waiting job slot".

The change also drops the context-notification wrapper and the legacy
loader around file names, and documents the two CSS pitfalls found on
the way: a logical inline-size wins over an inline physical width, and
Rumext does not convert dynamic :style maps for plain elements, so
inline styles leave CLJS as #js objects.

AI-assisted-by: glm-5.3-flash

🔧 Enable NativeMemoryTracking in the backend devenv

Add -XX:NativeMemoryTracking=summary to JAVA_OPTS so the backend and
REPL JVMs expose native memory breakdowns through jcmd.

Useful to diagnose non-heap growth (metaspace, code cache, GC, thread)
while working with imports and other JVM-heavy paths.

AI-assisted-by: deepseek-v4.1-flash

✨ Rework the import wizard progress and the size of its files

The milestone in course is written once, in the header, where the
message of the phase was: every file of a container reports the same
one, so a row repeated it. A row keeps its name and its size, and the
button to drop a file only while the wizard still offers its import
step, which is where dropping one makes sense.

The title counts the files and counts them done with what the job
reports about the file it is working on, so the number moves while a
container rebuilds its files one by one. The bar measures the whole
import: the real bytes of every container while they are uploaded, and
animated while the wizard checks, waits for a slot or rebuilds.

A file of a version 3 package shows what it occupies inside its
container: its own entries plus the storage objects its media and its
thumbnails reference, which the container keeps shared under
`objects/`. What the container itself occupies stays in
`:container-size`, the size the upload of the package is measured on.

The ready step waited for every known file instead of for the end of
the analysis, so Continue left the files that arrived later out of the
import, with a row that never showed anything. The upload step was
recognized only once a row carried a milestone, so the wizard said it
was importing during the first chunk; it now holds until every file
leaves the upload behind.

Also fixes a misspelled design system token that left the success
color unset.

AI-assisted-by: deepseek-v4.1-flash

✨ Cover the import sizes and the done counter with tests

The size a file of a version 3 package shows is its own entries plus
the storage objects its media and its thumbnails reference, and what
the container occupies stays apart: the test drives the analysis with a
fake container that has both, and checks that the manifest is not
charged to any file.

The done counter of the header counts the files the job left behind:
all of them for a container that finished, the ones before the file it
is working on for a container in course, none for one that waits for a
slot, and the files of every container add up.

The test that answers one entry per file also needed the mock of the
entries of a container, which the analysis now reads.

AI-assisted-by: deepseek-v4.1-flash

✨ Say the wait for a slot in the header, not in every row

A row of the import wizard no longer paints a waiting state: while no
job of the import has started, the header says it waits for a free
slot, and the rows keep their name and their size. Only a file whose
container already finished uploading says something of its own
("Ready"), plus the error text when it fails.

The clock icon and the `dashboard.import.status.waiting` key of both
locales go away with it.

The notes of the redesign come along: the description of the wizard in
`mem:frontend/dashboard-viewer-subtleties`, and a new
`mem:clojure/promesa` with the call styles of the library and how they
bridge to rx.

AI-assisted-by: deepseek-v4.1-flash
This commit is contained in:
Andrey Antukh 2026-10-07 21:38:13 +00:00
parent fcf9c27635
commit 464756ec59
13 changed files with 1160 additions and 162 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.serena/memories/clojure/promesa.md Normal file
View File

@ -0,0 +1,9 @@
# Promesa
Promise library of the monorepo (frontend CLJS and common, `funcool/promesa` 12.0.1 in `common/deps.edn`). Its API has two call styles and mixing them in one expression is the trap: `fmap`/`mcat`/`merr` are **fn first, promise last** (`->>` friendly) and `then`/`then'`/`catch`/`handle` are **promise first, fn last** (`->` friendly). Convention: `fmap`/`mcat`/`merr` inside `->>` pipelines, `then`/`catch` with `->`. A `then` written inside a `->>` reads backwards and hides whether the step carries a value or a promise.
- `fmap` (alias `map`, both `(f p)`) is functor map: `f` returns a plain value.
- `mcat` (alias `mapcat`, both `(f p)`) is monadic bind: `f` must return a promise, and the result mirrors it.
- `then` also unwraps a promise returned by `f`, which is why it covers both cases at the cost of not naming which one it is; `then'` maps without unwrapping.
- `merr` (fn first) and `catch` (promise first) are the error side.
- `all` takes a collection of promises (a lazy seq works); `let` chains sequential bindings.
- Seam with the rx pipelines of the frontend: `beicon.v2.core/from` builds an observable from "arrays, collections, and promise instance" (`external/beicon`), so `(->> p (p/mcat f) (rx/from) (rx/map g))` is the usual bridge (`app.main.data.imports`, `app.util.zip`).

1
.serena/memories/critical-info.md
View File

@ -8,6 +8,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
- Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables.
# Development workflow

2
.serena/memories/frontend/dashboard-viewer-subtleties.md
View File

@ -5,7 +5,7 @@
- Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id.
- Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly.
- Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications.
- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its rows paint the milestone of the job (the stage and its counter, with the file in course) instead of the messages the old SSE stream sent. Before the job exists, the rows paint a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and the row writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, each entry reports `:queued` ("Queued"); the `:start`/`:retry` events move it to bare progress, and only `:progress` carries a milestone. The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line any more: each row carries its own status. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients.
- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its header paints the milestone of the job (the stage and its counter) instead of the messages the old SSE stream sent; every file of a container reports the same milestone, so `jp/current-milestone-text` writes it once and the rows never repeat it. Before the job exists, the header paints a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and it writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, no row says anything: the header says the import waits for a slot while no job of it has started. The bar of the header measures the whole import, never a file: indeterminate while checking, waiting for a slot or rebuilding, because the client cannot know a global index and total of operations. The upload is the exception — the chunked upload knows its chunks, so the bar is determinate while the containers are sent, weighted by their size (`imp/upload-pct` over `:container-size`, the only percentage the client really knows). The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line on the rows any more: the rows of the import wizard carry only what is not progress — the name, the size, `Ready` once their container finished uploading, and the error text —, and the button that drops a file only shows while the wizard still offers its import step (the confirmation screen). Above the rows, the import wizard paints a stepper (Check/Upload/Import) plus one phase line: `imp/import-step` derives the step from the dialog status and the entries (upload holds while any file is pending, either `:import-ready` or carrying its own `:upload` milestone), so Check covers analyze, Upload covers the chunked client-side upload, and Import covers queue plus job. A file of a version 3 package shows what it occupies inside its container, computed while analyzing (`imp/file-sizes`): its own `files/<id>…` entries plus the storage objects its media and thumbnails reference, whose metadata is the only link to `objects/<id>…`. The container's own size stays in `:container-size` (what the dropped file weighs) and is what the upload bar weighs, never what a row shows. The subtitle is the size alone, with no extension. While analyzing with no failures nothing renders below the phase header — no rows, no spinner (`imp/awaiting-analysis?`); the header text is the only "Checking…" message. Rows appear once the analysis resolves (one per file inside the package), and a failed analysis takes the error-summary path instead. The ready step and its Continue wait for the end of the analysis (`imp/ready-to-import?`, fed with the completion of `imp/analyze`): a ready file is not enough, because the files still to arrive would never be imported. While rebuilding, the phase title counts the files and `imp/imported-count` counts them done with the job's own `:files` counter, so the number moves while a container rebuilds its files one by one. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients.
- File creation/duplication strips file `:data` before putting file summaries into dashboard state.
## Viewer

2
.serena/memories/frontend/ui-conventions-and-style-system.md
View File

@ -12,7 +12,7 @@
- Co-located SCSS modules are preferred. Use `app.main.style/stl` helpers from CLJS and design-system SCSS tokens/mixins instead of legacy global selectors or high-specificity nesting.
- Keep CSS specificity low. Avoid nested selectors unless they target elements the component owns; CSS Modules already prevent class-name collisions.
- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer.
- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer. Never mix: a stylesheet logical property (`inline-size`) and an inline physical `width` on the same element cascade separately, and the logical mapping wins — a dynamically driven width must be `width` on both sides (import modal progress bar hit this). And an inline style passed from CLJS must be a JS object (`#js {:width …}`): for plain `[:div …]` elements the Rumext macro does not convert the `:style` map at runtime — a plain CLJS map reaches React raw and React never finds the keys (static style literals are converted at compile time, which is why `:style` maps seem to work elsewhere).
- Use named design-system variables/tokens for spacing, borders, fixed dimensions, colors, and typography. Avoid hardcoded px/rem values and deprecated `resources/styles/common/refactor/spacing.scss` variables.
- Use component-local CSS custom properties for variants and theming instead of one-off Sass variables when a component has multiple visual states.
- Prefer DS typography components (`heading*`, `text*`) and typography mixins instead of plain text wrappers or deprecated typography mixins.

1
backend/scripts/_env
View File

@ -92,6 +92,7 @@ export JAVA_OPTS="\
-XX:+UseShenandoahGC \
-XX:+UseCompactObjectHeaders \
-XX:ShenandoahGCMode=generational \
-XX:NativeMemoryTracking=summary \
-XX:-OmitStackTraceInFastThrow \
--sun-misc-unsafe-memory-access=allow \
--enable-preview \

296
frontend/src/app/main/data/imports.cljs
View File

@ -25,7 +25,8 @@
[app.util.i18n :as i18n :refer [tr]]
[app.util.zip :as uz]
[beicon.v2.core :as rx]
[cuerdas.core :as str]))
[cuerdas.core :as str]
[promesa.core :as p]))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; MESSAGES OF A FAILURE
@ -98,10 +99,112 @@
(uz/read-as-text entry))))
(rx/map json/decode)))
(defn- segment-of
"The segment of a zip path that follows `prefix`, up to the next slash,
or nil when the path does not start with it."
[prefix path]
(when (str/starts-with? path prefix)
(let [rest (subs path (count prefix))
idx (str/index-of rest "/")]
(if (some? idx) (subs rest 0 idx) rest))))
(defn- file-of-entry
"The file of a container a zip entry belongs to. A container keeps every
file of a package under its own prefix: `files/<id>.json` for its data
and `files/<id>/…` for the rest."
[path]
(some-> (segment-of "files/" path)
(str/replace #"\.json$" "")))
(defn- object-of-entry
"The storage object a zip entry holds. A container keeps the media and
the thumbnails of its files shared under `objects/`, and both the
metadata (`objects/<id>.json`) and the binary (`objects/<id>.<ext>`)
belong to the same object."
[path]
(when-some [segment (segment-of "objects/" path)]
(let [dot (str/index-of segment ".")]
(if (some? dot) (subs segment 0 dot) segment))))
(defn- reference-entry?
"Whether an entry of a file is the metadata of one of the media or the
thumbnails that file references, the only place that links a file to the
objects it occupies."
[path]
(or (str/includes? path "/media/")
(str/includes? path "/thumbnails/")))
(defn- index-container
"Read the entries of a container in one pass: what every file of the
package occupies by itself, what every storage object occupies, and the
entries that say which objects a file references."
[entries]
(reduce
(fn [index entry]
(let [path (.-filename ^js entry)
size (.-compressedSize ^js entry)
file-id (file-of-entry path)]
(cond
(some? file-id)
(cond-> (update-in index [:files file-id] (fnil + 0) size)
(reference-entry? path)
(update :references conj [file-id entry]))
(some? (object-of-entry path))
(update-in index [:objects (object-of-entry path)] (fnil + 0) size)
:else
index)))
{:files {} :objects {} :references []}
entries))
(defn- read-objects
"The objects every file of a container references, read from the
metadata of its media and its thumbnails."
[references]
(->> (map (fn [[file-id entry]]
(->> (uz/read-as-text entry)
(p/fmap (fn [text] [file-id (json/decode text)]))))
references)
(p/all)
(p/fmap
(fn [decoded]
(reduce (fn [result [file-id data]]
(->> [:media-id :thumbnail-id]
(keep #(get data %))
(reduce (fn [result object-id]
(update result file-id (fnil conj #{}) object-id))
result)))
{}
decoded)))))
(defn- file-sizes
"How much every file of a container occupies inside it: its own entries
plus the storage objects it references, which the container keeps shared
and which are only linked from the file. The manifest carries neither a
size, so the parts of a package are the only account of what it
occupies."
[entries]
(let [{:keys [files objects references]} (index-container entries)]
(->> (read-objects references)
(p/fmap
(fn [referenced]
(reduce-kv (fn [sizes file-id object-ids]
(update sizes file-id (fnil + 0)
(reduce + 0 (keep objects object-ids))))
files
referenced))))))
(defn- analyze-file
"The entries of one file the user dropped: one per file of a version 3
package, one for a version 1 file, and an error for anything else."
[{:keys [uri] :as file}]
package, one for a version 1 file, and an error for anything else.
A container carries its own weight and the weight of every file in it:
the entry of a file of a version 3 package sizes the file inside its
container, and keeps what the container itself occupies in
`:container-size`, which is what the upload of the package is measured
on."
[{:keys [uri size] :as file}]
(let [stream (->> (slurp-uri uri :buffer)
(rx/merge-map
(fn [body]
@ -110,16 +213,23 @@
(= "application/zip" mtype)
(let [zip-reader (uz/reader body)]
(->> (read-zip-manifest zip-reader)
(rx/map
(rx/merge-map
(fn [manifest]
(if (= (:type manifest) "penpot/export-files")
(let [manifest (decode-manifest manifest)]
(assoc file :type :binfile-v3 :files (:files manifest)))
(assoc file :type :unknown))))
(->> (uz/get-entries zip-reader)
(p/mcat file-sizes)
(rx/from)
(rx/map (fn [sizes]
(-> file
(assoc :type :binfile-v3)
(assoc :container-size size)
(assoc :files (:files (decode-manifest manifest)))
(assoc :sizes sizes)))))
(rx/of (assoc file :type :unknown)))))
(rx/finalize (partial uz/close zip-reader))))
(= "application/octet-stream" mtype)
(rx/of (assoc file :type :binfile-v1))
(rx/of (assoc file :type :binfile-v1 :container-size size))
:else
(rx/of (assoc file :type :unknown))))))
@ -137,19 +247,28 @@
(->> stream
(rx/filter (fn [entry] (= :binfile-v3 (:type entry))))
(rx/merge-map (fn [{:keys [files] :as entry}]
(rx/merge-map (fn [{:keys [files sizes] :as entry}]
(->> (rx/from files)
(rx/map (fn [file]
(-> entry
(dissoc :files)
(dissoc :files :sizes)
(assoc :name (:name file))
(assoc :file-id (:id file))
;; the size the file itself
;; occupies inside the
;; container, never the
;; container's
(assoc :size (get sizes (str (:id file)) (:size entry)))
(assoc :status :success))))))))
(->> stream
(rx/filter (fn [data] (= :unknown (:type data))))
(rx/map (fn [_]
{:uri (:uri file)
:name (:name file)
:size (:size file)
:container-size (:size file)
:type :unknown
:status :error
:error (tr "dashboard.import.analyze-error")}))))
@ -273,6 +392,163 @@
(rx/from (entry-messages entries {:status :error
:error (cause-message cause (tr "labels.error"))}))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; WIZARD STEP, UPLOAD AND FILE SIZE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn format-file-size
"Human size of a dropped file, like `234MB` or `512KB`.
Returns nil when the size is unknown, so the row shows no subtitle."
[size]
(when (and (some? size) (number? size) (>= size 0))
(let [kb 1024
mb (* kb 1024)
gb (* mb 1024)]
(cond
(< size kb)
(str (js/Math.round size) "B")
(< size mb)
(let [v (/ size kb)]
(str (if (< v 10)
(-> (.toFixed v 1)
(str/replace #"\.0$" ""))
(str (js/Math.round v)))
"KB"))
(< size gb)
(let [v (/ size mb)]
(str (if (< v 10)
(-> (.toFixed v 1)
(str/replace #"\.0$" ""))
(str (js/Math.round v)))
"MB"))
:else
(let [v (/ size gb)]
(str (if (< v 10)
(-> (.toFixed v 1)
(str/replace #"\.0$" ""))
(str (js/Math.round v)))
"GB"))))))
(defn- upload-ratio
"The fraction of the container of an entry already sent: the chunks
uploaded over the chunks the file has, or nil while its upload has not
started."
[entry]
(let [{:keys [current total]} (get-in entry [:progress :counters :upload])]
(when (and (some? current) (some? total) (pos? total))
(/ current total))))
(defn- uploaded-ratio
"The fraction of the container of an entry already uploaded. A container
is uploaded whole before its job exists, so from the moment its entries
leave the upload stage they count as sent: only an entry still sending
chunks, or one that has not started, says less than everything."
[entry]
(cond
(= :upload (get-in entry [:progress :stage])) (or (upload-ratio entry) 0)
(contains? #{:analyze :import-ready} (:status entry)) 0
:else 1))
(defn upload-done?
"Whether the container of an entry finished uploading: the chunks sent
reached the chunks the file has."
[entry]
(let [{:keys [current total]} (get-in entry [:progress :counters :upload])]
(and (some? current) (some? total) (= current total))))
(defn upload-pct
"The percentage of the whole upload: the bytes already sent over the
bytes of every container. The containers are weighted by their size,
never averaged, because a `.penpot` is uploaded whole: a container of
200MB moves the bar as much as four of 50MB."
[entries]
(let [containers (map (comp first :entries) (packages entries))
total (reduce + 0 (keep :container-size containers))]
(when (pos? total)
(let [sent (reduce + 0 (map (fn [{:keys [container-size] :as entry}]
(* container-size (uploaded-ratio entry)))
containers))]
(js/Math.round (* 100 (/ sent total)))))))
(defn- files-done
"How many files of a container are already rebuilt. A container that
finished leaves all of its files behind; while it runs, its milestone
says which file it is working on, one based, so the ones before it are
done."
[entries]
(if (every? #(= :import-success (:status %)) entries)
(count entries)
(let [{:keys [current total]} (get-in (first entries) [:progress :counters :files])
total (or total (count entries))]
(-> (or current 1) dec (max 0) (min total)))))
(defn imported-count
"How many of the files being imported are already rebuilt: the files of
the containers that finished plus the ones a running container left
behind. A container rebuilds its files one by one, so counting only the
finished containers (or only their files) would leave the number still
for the whole import."
[entries]
(reduce + 0 (map (comp files-done :entries) (packages entries))))
(defn waiting-slot?
"Whether every file of the wizard waits for a free slot: the jobs of the
containers exist, but no worker picked them up yet."
[entries]
(and (pos? (count entries))
(every? #(= :import-queued (:status %)) entries)))
(defn- pending-upload?
"Whether a file is still to be uploaded, or is being uploaded right now:
the wizard is in its upload step while any file is pending."
[entry]
(or (= :import-ready (:status entry))
(= :upload (get-in entry [:progress :stage]))))
(defn import-step
"The wizard step the header paints: `:check` while the entries are
analyzed, `:upload` until every file of every container is uploaded, and
`:import` once the jobs run or the wizard reaches its outcome.
A file leaves the upload behind when its job exists: the queued message
carries no milestone, and the upload one disappears with it, so the
presence of a milestone is not enough to keep the step in the upload."
[status entries]
(case status
(:analyze :import-ready)
:check
:import-progress
(if (some pending-upload? entries)
:upload
:import)
(:import-success :import-error :library-resolution :library-summary)
:import
:check))
(defn awaiting-analysis?
"Whether the wizard hides the rows and shows the checking placeholder
instead: the dialog is still analyzing and no entry failed yet. Entries
that fail during analyze take the error-summary path instead of rows."
[status errors?]
(and (= :analyze status) (not errors?)))
(defn ready-to-import?
"Whether the wizard can offer its import step: the analysis is over and
every file it found can be imported. A ready file is not enough on its
own: until the analysis ends the wizard does not know how many files the
dropped containers carry, and the ones still to arrive would never be
imported."
[entries analysis-ended?]
(and analysis-ended?
(pos? (count entries))
(every? #(= :import-ready (:status %)) entries)))
(defn import-files
"Import the entries the wizard analyzed, one job per package.

336
frontend/src/app/main/ui/dashboard/import.cljs
View File

@ -29,7 +29,6 @@
[app.main.ui.ds.foundations.typography.heading :refer [heading*]]
[app.main.ui.ds.foundations.typography.text :refer [text*]]
[app.main.ui.ds.notifications.context-notification :refer [context-notification*]]
[app.main.ui.ds.product.loader :refer [loader*]]
[app.main.ui.icons :as deprecated-icon]
[app.main.ui.jobs.progress :as jp]
[app.util.dom :as dom]
@ -52,6 +51,7 @@
(let [entries (->> entries
(mapv (fn [file]
{:name (.-name file)
:size (.-size file)
:uri (wapi/create-uri file)}))
(not-empty))]
(when entries
@ -171,7 +171,7 @@
pending)))))
(defn- analyze-entries
[state entries]
[state entries on-analyze-end]
(->> (imp/analyze entries)
(rx/mapcat #(rx/delay emit-delay (rx/of %)))
(rx/filter some?)
@ -180,7 +180,11 @@
(when (some? (:error message))
(st/emit! (ev/event {::ev/name "import-files-error"
:error (:error message)})))
(swap! state update-with-analyze-result message)))))
(swap! state update-with-analyze-result message))
(constantly nil)
;; the wizard cannot offer its import step until it knows every
;; file of every container
on-analyze-end)))
(defn- import-files
[state library-resolution-data* project-id entries on-job]
@ -202,30 +206,31 @@
(mf/defc import-entry*
{::mf/memo true
::mf/private true}
[{:keys [entries entry edition can-be-deleted is-progress on-edit on-change on-delete]}]
[{:keys [entries entry edition on-edit on-change on-delete]}]
(let [status (:status entry)
;; FIXME: rename to format
format (:type entry)
loading? (or (= :analyze status)
(= :import-queued status)
(= :import-progress status)
(and is-progress (= :import-ready status)))
analyze-error? (= :analyze-error status)
import-success? (= :import-success status)
import-error? (= :import-error status)
import-ready? (= :import-ready status)
level (cond
import-success? :success
import-ready? :success
import-error? :error
analyze-error? :error
loading? nil
:else :default)
progress (:progress entry)
upload? (= :upload (:stage progress))
;; the container the file came in is fully sent, but its job does
;; not exist yet: the wizard is still uploading the other files
ready? (and (= :import-progress status)
upload?
(imp/upload-done? entry))
;; a file can be dropped while the wizard has not started yet: once
;; the import runs, its entries are committed
can-be-deleted? (and import-ready?
(> (count entries) 1))
is-shared? (:shared entry)
progress (:progress entry)
subtitle (imp/format-file-size (:size entry))
file-id (:file-id entry)
editing? (and (some? file-id) (= edition file-id))
@ -265,44 +270,42 @@
[:div {:class (stl/css-case
:file-entry true
:loading loading?
:success import-success?
:ready ready?
:success (= :import-success status)
:error (or import-error? analyze-error?)
:editable (and import-ready? (not editing?)))}
[:div {:class (stl/css :file-name)}
(when loading? [:> loader* {:width 26 :title (tr "labels.loading")}])
[:div {:class (stl/css :file-main)}
[:div {:class (stl/css :file-icon)
:aria-hidden "true"}
[:> icon* {:icon-id i/folder :size "m"}]]
(if editing?
[:div {:class (stl/css :file-name-edit)}
[:input {:type "text"
:auto-focus true
:class (stl/css :file-name-input)
:aria-label (tr "dashboard.import.file-name-label")
:default-value (:name entry)
:on-key-press on-edit-key-press
:on-blur on-edit-blur}]]
[:div {:class (stl/css :file-meta)}
(if editing?
[:div {:class (stl/css :file-name-edit)}
[:input {:type "text"
:auto-focus true
:class (stl/css :file-name-input)
:aria-label (tr "dashboard.import.file-name-label")
:default-value (:name entry)
:on-key-press on-edit-key-press
:on-blur on-edit-blur}]]
[:div {:class (stl/css :file-name-label)}
(if loading?
[:> text* {:class (stl/css :file-name-label)
:as "span"
:typography t/body-medium}
(:name entry)
(when ^boolean is-shared?
[:> icon* {:icon-id i/library :class (stl/css :file-label-icon)}])]
[:> context-notification*
{:level level
:appearance :ghost
:class (stl/css :file-name-notification)}
[:> text* {:class (stl/css :file-name-label)
[:*
[:> text* {:class (stl/css :file-name-label)
:as "span"
:typography t/body-medium}
[:span {:class (stl/css :file-name-text)}
(:name entry)]
(when ^boolean is-shared?
[:> icon* {:icon-id i/library :class (stl/css :file-label-icon)}])]
(when (some? subtitle)
[:> text* {:class (stl/css :file-subtitle)
:as "span"
:typography t/body-medium}
(:name entry)
(when ^boolean is-shared?
[:> icon* {:icon-id i/library :class (stl/css :file-label-icon)}])]])])
:typography t/body-small}
subtitle])])]
(when ^boolean (or editable? can-be-deleted)
(when ^boolean (or editable? can-be-deleted?)
[:div {:class (stl/css :edit-entry-buttons)}
(when ^boolean editable?
[:> icon-button* {:on-click on-edit'
@ -310,38 +313,36 @@
:icon-size "s"
:aria-label (tr "labels.edit")
:icon i/curve}])
(when ^boolean can-be-deleted
(when ^boolean can-be-deleted?
[:> icon-button* {:on-click on-delete'
:variant "ghost"
:icon-size "s"
:aria-label (tr "labels.delete")
:icon i/delete}])])]
(cond
analyze-error?
[:> text* {:class (stl/css :error-message)
:as "span"
:typography t/body-small}
;; the message is user-facing text already (a hint or a translated
;; label), never a key
(if (some? (:error entry))
(:error entry)
(tr "dashboard.import.analyze-error"))]
:icon i/delete}])])
import-error?
[:> text* {:class (stl/css :error-message)
:as "span"
:typography t/body-small}
(if (some? (:error entry))
(:error entry)
(tr "labels.error"))]
;; a row carries its name and its size, and only the state that is not
;; progress: the milestone in course and the files already imported are
;; the business of the header, which writes the milestone once for
;; every file of a container
(when ^boolean ready?
[:div {:class (stl/css :file-status)}
[:> icon* {:icon-id i/status-tick :size "s"}]
[:> text* {:as "span"
:typography t/body-medium}
(tr "dashboard.import.status.ready")]])]
(and (= :import-queued status) (not import-success?))
[:div {:class (stl/css :progress-message)}
(tr "jobs.queued")]
(and (not import-success?) (some? progress))
[:div {:class (stl/css :progress-message)}
(jp/milestone-text progress :file? true)])
(when ^boolean (or analyze-error? import-error?)
[:div {:class (stl/css :file-error)}
[:> text* {:class (stl/css :error-message)
:as "span"
:typography t/body-small}
;; the message is user-facing text already (a hint or a translated
;; label), never a key
(if (some? (:error entry))
(:error entry)
(if analyze-error?
(tr "dashboard.import.analyze-error")
(tr "labels.error")))]])
;; This is legacy code, will be removed when legacy-zip format is removed
[:div {:class (stl/css :linked-libraries)}
@ -561,13 +562,126 @@
;; ── Stage components ────────────────────────────────────────────────
(mf/defc import-steps*
{::mf/private true}
[{:keys [step]}]
(let [order [:check :upload :import]
label {:check (tr "dashboard.import.steps.check")
:upload (tr "dashboard.import.steps.upload")
:import (tr "dashboard.import.steps.import")}
current (count (take-while #(not= % step) order))]
[:ol {:class (stl/css :import-steps)
:aria-label (tr "dashboard.import.steps.label")}
(for [[idx id] (d/enumerate order)]
(let [done? (< idx current)
active? (= idx current)]
[:li {:key (name id)
:class (stl/css-case :import-step true
:done done?
:active active?)
:aria-current (when active? "step")}
[:span {:class (stl/css :step-marker)
:aria-hidden "true"}
(if done?
[:> icon* {:icon-id i/status-tick :size "s"}]
(str (inc idx)))]
[:> text* {:as "span"
:typography t/body-medium
:class (stl/css :step-label)}
(get label id)]
(when (< idx 2)
[:span {:class (stl/css :step-line) :aria-hidden "true"}])]))]))
(defn- importing-title
"The title of the phase that rebuilds the files, with the files the
wizard is working on and the ones already done."
[total done]
(dm/str (tr "dashboard.import.importing.title" (i18n/c total))
" · "
(tr "dashboard.import.importing.done" (i18n/c done))))
(mf/defc import-phase-header*
{::mf/private true}
[{:keys [status step entries template]}]
(let [total (+ (count entries) (if (some? template) 1 0))
;; the file the job is working on is reported by the job itself, so
;; the counter moves while a container rebuilds its files one by one
done (imp/imported-count entries)
phase (cond
(= :analyze status) :check
(= :import-ready status) :ready
(not= :import-progress status) nil
(= :upload step) :upload
(imp/waiting-slot? entries) :queued
:else :importing)
;; the bar measures the whole import, never a single file: the
;; upload of every container put together while they are sent, and
;; the animated one from there on, because the stages of a job are
;; not comparable one with another
[title desc pct]
(case phase
:check [(tr "dashboard.import.check.title")
(tr "dashboard.import.check.message")
nil]
:ready [(tr "dashboard.import.ready.title")
(tr "dashboard.import.ready.message")
nil]
:upload [(tr "dashboard.import.upload.title")
(tr "dashboard.import.upload.message")
(imp/upload-pct entries)]
:queued [(tr "dashboard.import.queued.title")
(tr "dashboard.import.queued.message")
nil]
:importing [(importing-title total done)
(tr "dashboard.import.importing.message")
nil]
[nil nil nil])
;; the milestone in course takes the place of the message: the files
;; of a container share it, so writing it here keeps it once
desc
(or (jp/current-milestone-text entries) desc)
;; the bar paints what is running: indeterminate while the wizard
;; works —checking, waiting for a slot, rebuilding— and determinate
;; while it uploads, which is the only percentage it really knows
bar? (contains? #{:check :upload :queued :importing} phase)]
(when (some? phase)
[:div {:class (stl/css :phase-header)}
[:> heading* {:level 3
:typography t/headline-medium
:class (stl/css :phase-title)}
title]
[:> text* {:as "p"
:typography t/body-medium
:class (stl/css :phase-message)}
desc]
(when bar?
[:div {:class (stl/css-case :progress-bar true
:determinate (some? pct))
:role "progressbar"
:aria-valuemin 0
:aria-valuemax 100
:aria-valuenow pct}
[:div {:class (stl/css :progress-fill)
:style (when (some? pct)
#js {:width (str pct "%")})}]])])))
(mf/defc import-files-stage*
{::mf/private true}
[{:keys [entries template status errors? import-success-total auto-linked-count
[{:keys [entries template status step errors? import-success-total auto-linked-count
edition on-edit on-change on-delete
on-cancel on-cancel-import on-continue on-accept pending-analysis?]}]
[:*
[:> import-steps* {:step step}]
[:div {:class (stl/css :modal-content)}
[:> import-phase-header* {:status status
:step step
:entries entries
:template template}]
(when (and (= :analyze status) errors?)
[:> context-notification*
{:level :warning
@ -590,7 +704,8 @@
:class (stl/css :context-notification-error)}
(tr "dashboard.import.import-error.disclaimer")])
(if (or (= :import-error status) (and (= :analyze status) errors?))
(cond
(or (= :import-error status) (and (= :analyze status) errors?))
[:div {:class (stl/css :import-error-disclaimer)}
[:div (tr "dashboard.import.import-error.message1")]
[:ul {:class (stl/css :import-error-list)}
@ -614,20 +729,24 @@
(tr "dashboard.import.import-error.unknown-error"))])]))]
[:div (tr "dashboard.import.import-error.message2")]]
(imp/awaiting-analysis? status errors?)
;; while checking, the phase header carries the message: no rows,
;; no spinners
nil
:else
(for [entry entries]
[:> import-entry* {:edition edition
:key (dm/str (:uri entry) "/" (:file-id entry))
:entry entry
:entries entries
:is-progress (= :import-progress status)
:on-edit on-edit
:on-change on-change
:on-delete on-delete
:can-be-deleted (> (count entries) 1)}]))
:on-delete on-delete}]))
(when (some? template)
[:> import-entry* {:entry (assoc template :status status)
:can-be-deleted false}])]
(when (and (some? template)
(not (imp/awaiting-analysis? status errors?)))
[:> import-entry* {:entry (assoc template :status status)}])]
[:div {:class (stl/css :modal-footer)}
[:div {:class (stl/css :action-buttons)}
@ -736,6 +855,13 @@
status* (mf/use-state :analyze)
status (deref status*)
;; the analysis of the dropped files reached its end: only then the
;; wizard knows every file it is going to import
analysis-ended*
(mf/use-state false)
analysis-ended?
(deref analysis-ended*)
;; Jobs created while importing, to cancel them on demand, and
;; the subscription to their messages, to stop listening.
jobs* (mf/use-state #{})
@ -971,13 +1097,12 @@
(every? #(contains? visited (:id %)) unresolved-files))
(reset! status* :library-summary)))
(mf/with-effect [entries resolution]
(mf/with-effect [entries resolution analysis-ended?]
(cond
(some? template)
(reset! status* :import-ready)
(and (seq entries)
(every? #(= :import-ready (:status %)) entries))
(imp/ready-to-import? entries analysis-ended?)
(reset! status* :import-ready)
(and (seq entries)
@ -994,7 +1119,7 @@
;; Run analyze operation on component mount
(mf/with-effect []
(let [sub (analyze-entries state* entries)]
(let [sub (analyze-entries state* entries #(reset! analysis-ended* true))]
(partial rx/dispose! sub)))
[:div {:class (stl/css :modal-overlay)}
@ -1003,7 +1128,7 @@
[:> heading* {:level 2
:typography t/headline-large
:class (stl/css :modal-title)}
(tr "dashboard.import")]
(tr "dashboard.import.title")]
[:> icon-button* {:variant "ghost"
:aria-label (tr "labels.close")
:on-click on-cancel
@ -1016,6 +1141,7 @@
{:entries entries
:template template
:status status
:step (imp/import-step status entries)
:errors? errors?
:import-success-total import-success-total
:auto-linked-count auto-linked-count
@ -1030,23 +1156,27 @@
:pending-analysis? pending-analysis?}]
:library-resolution
[:> import-library-resolution-stage*
{:current-unresolved-file current-unresolved-file
:selection selection
:on-select manage-on-select
:on-disconnect manage-on-disconnect
:visited visited
:last-file? last-unresolved-file?
:on-wizard-prev on-wizard-prev
:on-wizard-next on-wizard-next
:on-wizard-skip on-wizard-skip}]
[:*
[:> import-steps* {:step :import}]
[:> import-library-resolution-stage*
{:current-unresolved-file current-unresolved-file
:selection selection
:on-select manage-on-select
:on-disconnect manage-on-disconnect
:visited visited
:last-file? last-unresolved-file?
:on-wizard-prev on-wizard-prev
:on-wizard-next on-wizard-next
:on-wizard-skip on-wizard-skip}]]
:library-summary
[:> import-library-summary-stage*
{:resolution resolution
:selection selection
:visited visited
:on-summary-back on-summary-back
:on-confirm-library-links on-confirm-library-links}]
[:*
[:> import-steps* {:step :import}]
[:> import-library-summary-stage*
{:resolution resolution
:selection selection
:visited visited
:on-summary-back on-summary-back
:on-confirm-library-links on-confirm-library-links}]]
nil)]]))

207
frontend/src/app/main/ui/dashboard/import.scss vendored
View File

@ -34,7 +34,7 @@
background-color: var(--color-background-primary);
border: $b-2 solid var(--color-background-quaternary);
min-block-size: $sz-192;
inline-size: px2rem(640);
inline-size: px2rem(720);
max-block-size: px2rem(800);
}
@ -56,6 +56,104 @@
.modal-title {
color: var(--color-foreground-primary);
text-transform: uppercase;
letter-spacing: 0.02em;
}
// ################################
// WIZARD STEPS
// ################################
.import-steps {
display: flex;
align-items: center;
gap: 0;
list-style: none;
margin: 0;
padding: 0;
}
.import-step {
--step-fg-color: var(--color-foreground-secondary);
--step-marker-border: var(--color-foreground-secondary);
display: flex;
align-items: center;
gap: var(--sp-s);
flex-shrink: 0;
&:not(:last-child) {
flex: 1 1 auto;
}
&.active {
--step-fg-color: var(--color-foreground-primary);
--step-marker-border: var(--color-accent-primary);
}
&.done {
--step-fg-color: var(--color-foreground-secondary);
--step-marker-border: var(--color-accent-primary);
--step-marker-bg-color: var(--color-accent-primary);
--step-marker-fg-color: var(--color-background-secondary);
}
}
.step-marker {
display: flex;
justify-content: center;
align-items: center;
block-size: $sz-24;
inline-size: $sz-24;
flex-shrink: 0;
border-radius: 50%;
border: $b-1 solid var(--step-marker-border);
background-color: var(--step-marker-bg-color, transparent);
color: var(--step-marker-fg-color, var(--step-marker-border));
font-size: px2rem(12);
font-weight: 500;
svg {
block-size: $sz-16;
inline-size: $sz-16;
}
}
.step-label {
color: var(--step-fg-color);
white-space: nowrap;
}
.step-line {
flex: 1 1 auto;
block-size: $b-1;
min-inline-size: $sz-32;
margin-inline: var(--sp-m);
background-color: var(--color-background-quaternary);
}
.import-step.active .step-line,
.import-step.done .step-line {
background-color: var(--color-background-quaternary);
}
// ################################
// PHASE HEADER
// ################################
.phase-header {
display: flex;
flex-direction: column;
gap: var(--sp-xs);
}
.phase-title {
color: var(--color-foreground-primary);
}
.phase-message {
color: var(--color-foreground-secondary);
margin: 0;
}
.modal-close-btn {
@ -86,8 +184,9 @@
flex-direction: column;
gap: var(--sp-m);
&.ready,
&.success {
--file-entry-fg-color: var(--color-accent-sucess);
--file-entry-fg-color: var(--color-accent-success);
}
&.error {
@ -99,16 +198,10 @@
}
}
.error-message,
.progress-message {
display: flex;
align-items: center;
min-block-size: $sz-32;
color: var(--file-entry-fg-color);
}
.error-message {
display: flex;
align-items: flex-start;
color: var(--file-entry-fg-color);
white-space: pre-wrap;
overflow-wrap: anywhere;
}
@ -153,6 +246,86 @@
color: var(--file-entry-fg-color);
}
.file-main {
display: flex;
align-items: center;
gap: var(--sp-m);
}
.file-icon {
display: flex;
justify-content: center;
align-items: center;
block-size: px2rem(40);
inline-size: px2rem(40);
flex-shrink: 0;
border-radius: $br-8;
background-color: var(--color-background-tertiary);
color: var(--color-foreground-secondary);
}
.file-meta {
display: flex;
flex-direction: column;
gap: var(--sp-xxs);
flex: 1 1 auto;
min-inline-size: 0;
}
.file-subtitle {
color: var(--color-foreground-secondary);
}
.file-status {
display: flex;
align-items: center;
gap: var(--sp-s);
flex-shrink: 0;
color: var(--file-entry-fg-color);
white-space: nowrap;
}
.file-error {
padding-inline-start: calc(px2rem(40) + var(--sp-m));
}
.progress-bar {
block-size: px2rem(4);
border-radius: $br-8;
background-color: var(--color-background-tertiary);
overflow: hidden;
}
.progress-fill {
block-size: 100%;
border-radius: $br-8;
background-color: var(--color-accent-primary);
// the client cannot know a global percentage of the import work, so
// the fill slides through the bar while the jobs run; the upload is
// the exception, its chunk counters are real and the inline width
// style drives the fill then. The rule is written as physical `width`
// on purpose: the determinate fill gets its width from an inline
// style, and a logical `inline-size` here would map after the cascade
// and win over it.
width: 40%;
animation: progress-slide 1.2s ease-in-out infinite;
.determinate & {
animation: none;
}
}
@keyframes progress-slide {
from {
transform: translateX(-100%);
}
to {
transform: translateX(250%);
}
}
.edit-entry-buttons {
display: flex;
align-items: center;
@ -175,7 +348,15 @@
display: flex;
align-items: center;
gap: var(--sp-m);
flex-grow: 1;
min-inline-size: 0;
color: var(--color-foreground-primary);
}
// the status sits at the end of the same line: a long name gives way to it
.file-name-text {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.file-label-icon {
@ -214,10 +395,6 @@
--context-notification-bg-color: var(--color-background-primary);
}
.file-name-notification {
flex-grow: 1;
}
.import-error-disclaimer {
color: var(--color-foreground-secondary);
}

48
frontend/src/app/main/ui/jobs/progress.cljs
View File

@ -5,8 +5,8 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.main.ui.jobs.progress
"How a milestone of a job is written in a row: the stage in course and
its counter.
"How the milestone of a job is written: the stage in course and its
counter.
The stage is a stable key of the vocabulary of the worker, never a
display string, so it is resolved here with a `case` of literal `tr`
@ -42,28 +42,38 @@
(str current "/" total)
(str current))))
(defn upload-pct
"The whole percentage of the file already uploaded. The chunked upload
knows how many chunks the file has and which one is in course, so
this is the only counter the client can trust as a percentage."
[{:keys [counters]}]
(let [{:keys [current total]} (get counters :upload)]
(when (and (some? current) (some? total) (pos? total))
(js/Math.round (* 100 (/ current total))))))
(defn- upload-label
"The upload of a file counts chunks, not units of work, so it is written
as the percentage of the file already sent instead of a counter."
[{:keys [current total]}]
(when (and (some? current) (some? total) (pos? total))
(str "(" (js/Math.round (* 100 (/ current total))) "%)")))
[milestone]
(when-some [pct (upload-pct milestone)]
(str "(" pct "%)")))
(defn milestone-text
"The text of a milestone: the stage in course and its counter, and with
`:file?` the file in course behind it, when the milestone carries its
counter. The upload stage is the exception: it counts chunks, so it is
written as a percentage of the file."
[milestone & {:keys [file?]}]
"The text of a milestone: the stage in course and its counter. The
upload of a file counts chunks, not units of work, so it is written as
the percentage of the file already sent instead of a counter."
[milestone]
(when-some [stage (:stage milestone)]
(let [counters (:counters milestone)
own (if (= :upload stage)
(upload-label (get counters :upload))
(counter-label (get counters stage)))]
(let [own (if (= :upload stage)
(upload-label milestone)
(counter-label (get (:counters milestone) stage)))]
(str (stage-label stage)
(when (some? own)
(str " " own))
(when-let [files (and file?
(not= :files stage)
(counter-label (get counters :files)))]
(str " · " (stage-label :files) " " files))))))
(str " " own))))))
(defn current-milestone-text
"The text of the milestone in course, for a caller that writes the
progress once for a whole group of files: every file of a container
reports the same milestone, so a row would repeat it."
[entries]
(some (comp milestone-text :progress) entries))

234
frontend/test/frontend_tests/data/imports_test.cljs
View File

@ -137,6 +137,7 @@
(mock/with-mocks*
{http/send! (slurp-stub zip-magic)
uz/get-entry (mock/stub (fn [_ _] (p/resolved ::entry)))
uz/get-entries (mock/stub (fn [_] (p/resolved #js [])))
uz/read-as-text (mock/stub (fn [_] (p/resolved (manifest [first-id "First"]
[second-id "Second"]))))
uz/close (mock/stub (fn [_] nil))}
@ -153,6 +154,55 @@
(t/testing "all of them share the upload of the file that was dropped"
(t/is (every? #(= uri (:uri %)) @seen)))))))
(t/deftest ^:async a-version-3-package-sizes-every-file-with-what-it-occupies
(let [first-id (uuid/next)
second-id (uuid/next)
object-id (str (uuid/next))
seen (atom [])
;; the entries of a container: every file keeps its own under
;; `files/<id>`, the binaries of its media shared under `objects/`,
;; and the manifest belongs to the container itself
entries
#js [#js {:filename "manifest.json"
:compressedSize 100}
#js {:filename (str "files/" first-id ".json")
:compressedSize 200}
#js {:filename (str "files/" first-id "/pages/one.json")
:compressedSize 300}
#js {:filename (str "files/" first-id "/media/one.json")
:compressedSize 10
:content (str "{\"mediaId\": \"" object-id "\"}")}
#js {:filename (str "objects/" object-id ".json")
:compressedSize 20}
#js {:filename (str "objects/" object-id ".png")
:compressedSize 1000}
#js {:filename (str "files/" second-id ".json")
:compressedSize 400}]]
(await
(mock/with-mocks*
{http/send! (slurp-stub zip-magic)
;; the manifest is read through the same reader as the rest of the
;; entries, so the fake carries its text and the mock answers what
;; every entry holds
uz/get-entry (mock/stub (fn [_ _]
(p/resolved #js {:content (manifest [first-id "First"]
[second-id "Second"])})))
uz/get-entries (mock/stub (fn [_] (p/resolved entries)))
uz/read-as-text (mock/stub (fn [entry] (p/resolved (.-content entry))))
uz/close (mock/stub (fn [_] nil))}
(await (hva/observe (imp/analyze [{:uri uri :name "package.penpot" :size 10000}])
{:on-next #(swap! seen conj %)}))))
(t/testing "a file occupies its own entries and the objects it references"
;; 200 + 300 + 10 of its own, and 20 + 1000 of the object its media
;; points at; the 100 of the manifest belong to the container
(t/is (= [1530 400] (mapv :size @seen))))
(t/testing "what the container occupies is kept apart"
(t/is (= [10000 10000] (mapv :container-size @seen))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMPORTING A PACKAGE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@ -510,6 +560,190 @@
(rx/push! ws-stream (message (event job-id :end {:outcome "completed"})))
(await done))))))
(t/deftest format-file-size-writes-human-sizes
(t/testing "unknown sizes show nothing"
(t/is (nil? (imp/format-file-size nil)))
(t/is (nil? (imp/format-file-size -1))))
(t/testing "bytes, kilobytes, megabytes and gigabytes"
(t/is (= "0B" (imp/format-file-size 0)))
(t/is (= "512B" (imp/format-file-size 512)))
(t/is (= "1KB" (imp/format-file-size 1024)))
(t/is (= "1.5KB" (imp/format-file-size 1536)))
(t/is (= "512KB" (imp/format-file-size (* 512 1024))))
(t/is (= "234MB" (imp/format-file-size (* 234 1024 1024))))
(t/is (= "1.2GB" (imp/format-file-size (* 1.2 1024 1024 1024))))))
(defn- uploading
"A file of the wizard while the container it came in is uploaded: the
chunks sent out of the chunks the container has."
[uri size current total]
{:file-id (uuid/next)
:uri uri
:size (* 1024 1024)
:container-size size
:type :binfile-v3
:status :import-progress
:progress {:stage :upload :counters {:upload {:current current :total total}}}})
(t/deftest upload-pct-weighs-every-container-by-its-size
(let [small (* 100 1024 1024)
big (* 300 1024 1024)]
(t/testing "the bytes sent of every container over the bytes of all of them"
;; 100MB of 100MB and half of 300MB: 250 of 400, not the average of
;; 100% and 50%
(t/is (= 63 (imp/upload-pct [(uploading "a" small 4 4)
(uploading "b" big 1 2)]))))
(t/testing "a container that has not started counts as nothing sent"
(t/is (= 25 (imp/upload-pct [(uploading "a" small 4 4)
{:uri "b" :container-size big :type :binfile-v3
:status :import-ready}]))))
(t/testing "the files of one container are one container"
(t/is (= 100 (imp/upload-pct [(uploading "a" big 1 1)
(uploading "a" big 1 1)]))))
(t/testing "a container with no size leaves the bar indeterminate"
(t/is (nil? (imp/upload-pct [])))
(t/is (nil? (imp/upload-pct [{:uri "a" :type :binfile-v3 :status :import-ready}]))))))
(t/deftest a-container-past-the-upload-counts-whole
;; the job of a container is created with its upload as its input, so the
;; moment the entries leave the upload stage the container is sent
(t/is (= 100 (imp/upload-pct [{:uri "a" :name "a.penpot" :container-size (* 100 1024 1024)
:type :binfile-v1 :status :import-queued}
{:uri "b" :name "b.penpot" :container-size (* 300 1024 1024)
:type :binfile-v1 :status :import-success}]))))
(t/deftest upload-done?-tells-when-the-chunks-of-a-container-are-sent
(t/testing "the last chunk closes the upload"
(t/is (true? (imp/upload-done? (uploading "a" 100 3 3))))
(t/is (false? (imp/upload-done? (uploading "a" 100 2 3)))))
(t/testing "a file that is not uploading yet is not done"
(t/is (false? (imp/upload-done? {:status :import-ready})))
(t/is (false? (imp/upload-done? {})))))
(t/deftest imported-count-counts-the-files-the-job-left-behind
(let [file (fn [uri status progress]
(cond-> {:uri uri :type :binfile-v3 :status status}
(some? progress) (assoc :progress progress)))]
(t/testing "a container that finished counts all of its files"
(t/is (= 2 (imp/imported-count [(file "a" :import-success nil)
(file "a" :import-success nil)]))))
(t/testing "a running container counts the files it left behind"
;; its milestone says it is working on the second file of three
(t/is (= 1 (imp/imported-count [(file "a" :import-progress
{:stage :pages
:counters {:files {:current 2 :total 3}
:pages {:current 1 :total 5}}})]))))
(t/testing "a container that has not started has nothing done"
(t/is (= 0 (imp/imported-count [(file "a" :import-queued nil)])))
(t/is (= 0 (imp/imported-count [(file "a" :import-progress
{:stage :pages :counters {}})]))))
(t/testing "the files of every container add up"
(t/is (= 3 (imp/imported-count [(file "a" :import-success nil)
(file "a" :import-success nil)
(file "b" :import-progress
{:stage :pages
:counters {:files {:current 2 :total 4}}})]))))
(t/testing "no files, nothing done"
(t/is (= 0 (imp/imported-count []))))))
(t/deftest waiting-slot?-is-true-until-a-worker-picks-the-jobs-up
(t/testing "every file queued is a wizard waiting for a slot"
(t/is (true? (imp/waiting-slot? [{:status :import-queued}
{:status :import-queued}]))))
(t/testing "a job in course leaves the wait behind"
(t/is (false? (imp/waiting-slot? [{:status :import-queued}
{:status :import-progress}]))))
(t/testing "a file already imported is not waiting either"
(t/is (false? (imp/waiting-slot? [{:status :import-success}]))))
(t/testing "no files, no wait"
(t/is (false? (imp/waiting-slot? [])))))
(t/deftest awaiting-analysis-hides-the-rows-while-checking
(t/testing "only a clean analysis shows the placeholder"
(t/is (true? (imp/awaiting-analysis? :analyze false)))
(t/is (false? (imp/awaiting-analysis? :analyze true))))
(t/testing "any later stage shows the rows"
(t/is (false? (imp/awaiting-analysis? :import-ready false)))
(t/is (false? (imp/awaiting-analysis? :import-progress false)))
(t/is (false? (imp/awaiting-analysis? :import-success false)))))
(t/deftest ready-to-import?-waits-for-the-whole-analysis
(t/testing "every file found and ready, with the analysis over"
(t/is (true? (imp/ready-to-import? [{:status :import-ready}
{:status :import-ready}]
true))))
(t/testing "a file that has not arrived keeps the wizard checking"
;; the files of a container are only known once its manifest is read, so
;; offering the import step here would leave the missing one behind
(t/is (false? (imp/ready-to-import? [{:status :import-ready}] false))))
(t/testing "a file that cannot be imported is not ready either"
(t/is (false? (imp/ready-to-import? [{:status :import-ready}
{:status :analyze-error}]
true))))
(t/testing "no files at all is not ready"
(t/is (false? (imp/ready-to-import? [] true)))))
(t/deftest import-step-follows-the-wizard-status
(t/testing "analysis paints the check step"
(t/is (= :check (imp/import-step :analyze [])))
(t/is (= :check (imp/import-step :import-ready []))))
(t/testing "an upload milestone paints the upload step"
(let [entries [{:file-id (uuid/next)
:status :import-progress
:progress {:stage :upload :counters {:upload {:current 1 :total 2}}}}]]
(t/is (= :upload (imp/import-step :import-progress entries)))))
(t/testing "a file that has not reported anything yet is still uploading"
;; the wizard is uploading from the click, before the first chunk arrives
(let [entries [{:file-id (uuid/next) :status :import-ready}
{:file-id (uuid/next)
:status :import-progress
:progress {:stage :upload :counters {:upload {:current 1 :total 2}}}}]]
(t/is (= :upload (imp/import-step :import-progress entries)))))
(t/testing "a finished upload with its job still to come is uploading"
(let [entries [{:file-id (uuid/next)
:status :import-progress
:progress {:stage :upload :counters {:upload {:current 2 :total 2}}}}
{:file-id (uuid/next)
:status :import-progress
:progress {:stage :upload :counters {:upload {:current 1 :total 4}}}}]]
(t/is (= :upload (imp/import-step :import-progress entries)))))
(t/testing "queued or job milestones paint the import step"
(let [queued [{:file-id (uuid/next) :status :import-queued}]
started [{:file-id (uuid/next) :status :import-progress}]
pages [{:file-id (uuid/next)
:status :import-progress
:progress {:stage :pages :counters {:pages {:current 1 :total 8}}}}]]
(t/is (= :import (imp/import-step :import-progress queued)))
(t/is (= :import (imp/import-step :import-progress started)))
(t/is (= :import (imp/import-step :import-progress pages)))))
(t/testing "terminal and resolution stages paint the import step"
(t/is (= :import (imp/import-step :import-success [])))
(t/is (= :import (imp/import-step :import-error [])))
(t/is (= :import (imp/import-step :library-resolution [])))
(t/is (= :import (imp/import-step :library-summary [])))))
(t/deftest ^:async cancel-job-asks-the-server-and-ignores-failures
(let [calls (atom [])
job-id (uuid/next)]

32
frontend/test/frontend_tests/ui/jobs_progress_test.cljs
View File

@ -5,8 +5,7 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns frontend-tests.ui.jobs-progress-test
"How a job milestone is written: the stage and its counter, and the
upload stage as a percentage of the file."
"How a job milestone is written: the stage and its counter."
(:require
[app.main.ui.jobs.progress :as jp]
[app.util.i18n :as i18n]
@ -24,15 +23,7 @@
(t/is (= "TR:jobs.progress.stage.pages 3"
(jp/milestone-text {:stage :pages :counters {:pages {:current 3}}}))))))
(t/deftest a-file-milestone-appends-the-file-counter
(with-redefs [i18n/tr stub-tr]
(t/is (= "TR:jobs.progress.stage.pages 3/8 · TR:jobs.progress.stage.files 1/2"
(jp/milestone-text {:stage :pages
:counters {:pages {:current 3 :total 8}
:files {:current 1 :total 2}}}
:file? true)))))
(t/deftest an-upload-milestone-writes-the-percentage-of-the-file
(t/deftest the-upload-milestone-writes-the-percentage-of-the-file
(with-redefs [i18n/tr stub-tr]
(t/testing "the chunks sent become a whole percentage"
(t/is (= "TR:jobs.progress.stage.upload (33%)"
@ -42,3 +33,22 @@
(t/testing "an upload with no total keeps the bare label"
(t/is (= "TR:jobs.progress.stage.upload"
(jp/milestone-text {:stage :upload :counters {:upload {:current 1}}}))))))
(t/deftest the-milestone-in-course-is-the-one-of-the-files-that-report-it
(with-redefs [i18n/tr stub-tr]
(t/testing "the files of a container share their milestone, so one is enough"
(t/is (= "TR:jobs.progress.stage.pages 4/5"
(jp/current-milestone-text
[{:status :import-queued}
{:progress {:stage :pages :counters {:pages {:current 4 :total 5}}}}
{:progress {:stage :pages :counters {:pages {:current 4 :total 5}}}}]))))
(t/testing "a file with nothing to report says nothing"
(t/is (nil? (jp/current-milestone-text [{:status :import-ready}
{:status :import-queued}])))
(t/is (nil? (jp/current-milestone-text []))))))
(t/deftest a-milestone-without-a-stage-writes-nothing
(with-redefs [i18n/tr stub-tr]
(t/is (nil? (jp/milestone-text nil)))
(t/is (nil? (jp/milestone-text {})))))

74
frontend/translations/en.po
View File

@ -673,6 +673,70 @@ msgstr "Connect selected libraries"
msgid "dashboard.import.file-name-label"
msgstr "File name"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.title"
msgstr "Import file"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.status.ready"
msgstr "Ready"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.label"
msgstr "Import steps"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.check"
msgstr "Check"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.upload"
msgstr "Upload"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.import"
msgstr "Import"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.check.title"
msgstr "Checking your files..."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.check.message"
msgstr "Making sure each file can be imported. This only takes a moment."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.ready.title"
msgstr "Files ready"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.ready.message"
msgstr "Your files can be imported. Continue to upload them."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.upload.title"
msgstr "Uploading your files..."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.upload.message"
msgstr "Sending your files. Keep this dialog open until the upload ends."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.importing.title"
msgid_plural "dashboard.import.importing.title"
msgstr[0] "Importing 1 file"
msgstr[1] "Importing %s files"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.importing.done"
msgid_plural "dashboard.import.importing.done"
msgstr[0] "1 done"
msgstr[1] "%s done"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.importing.message"
msgstr "Rebuilding pages, components and assets in your project."
#: src/app/main/ui/dashboard/import.cljs:619
msgid "dashboard.import.import-error.check-error"
msgstr "We couldn't verify this file."
@ -715,6 +779,14 @@ msgstr "Next file"
msgid "dashboard.import.no-perms"
msgstr "You don’t have permission to import to this team"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.queued.title"
msgstr "Waiting to start..."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.queued.message"
msgstr "Your files are uploaded. The import starts as soon as a slot is free."
#: src/app/main/ui/dashboard/import.cljs:443
msgid "dashboard.import.resolve-libraries"
msgstr ""
@ -2668,7 +2740,7 @@ msgstr "Uploading file…"
#: src/app/main/ui/dashboard/import.cljs:328, src/app/main/ui/exports/files.cljs:103
msgid "jobs.queued"
msgstr "Queued"
msgstr "Waiting job slot"
#: src/app/main/ui/dashboard/comments.cljs:94
msgid "label.mark-all-as-read"

80
frontend/translations/es.po
View File

@ -672,6 +672,16 @@ msgstr[1] "%s bibliotecas fueron vinculadas automáticamente por nombre."
msgid "dashboard.import.bad-url"
msgstr "La importación ha fallado. La URL de la plantilla es incorrecta"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.check.message"
msgstr ""
"Estamos comprobando que cada archivo se puede importar. Solo tardará un "
"momento."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.check.title"
msgstr "Comprobando tus archivos..."
#: src/app/main/ui/dashboard/import.cljs:733
msgid "dashboard.import.confirm-library-links"
msgstr "Confirmar vínculos de biblioteca"
@ -718,6 +728,22 @@ msgstr[1] "%s ficheros se han importado correctamente."
msgid "dashboard.import.import-warning"
msgstr "Algunos ficheros contenían objetos erroneos que no han sido importados."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.importing.title"
msgid_plural "dashboard.import.importing.title"
msgstr[0] "Importando 1 archivo"
msgstr[1] "Importando %s archivos"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.importing.done"
msgid_plural "dashboard.import.importing.done"
msgstr[0] "1 hecho"
msgstr[1] "%s hechos"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.importing.message"
msgstr "Reconstruyendo páginas, componentes y recursos en tu proyecto."
#: src/app/main/ui/dashboard/import.cljs:710
msgid "dashboard.import.next-file"
msgstr "Siguiente archivo"
@ -726,6 +752,24 @@ msgstr "Siguiente archivo"
msgid "dashboard.import.no-perms"
msgstr "No tienes permisos para importar en este equipo"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.queued.title"
msgstr "Esperando a empezar..."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.queued.message"
msgstr ""
"Tus archivos ya están subidos. La importación empieza en cuanto haya un hueco "
"libre."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.ready.message"
msgstr "Tus archivos se pueden importar. Continúa con la subida."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.ready.title"
msgstr "Archivos listos"
#: src/app/main/ui/dashboard/import.cljs:443
msgid "dashboard.import.resolve-libraries"
msgstr ""
@ -744,6 +788,26 @@ msgstr "conectar a"
msgid "dashboard.import.resolve-libraries.original-library"
msgstr "biblioteca original"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.status.ready"
msgstr "Listo"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.check"
msgstr "Comprobación"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.import"
msgstr "Importación"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.label"
msgstr "Pasos de la importación"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.steps.upload"
msgstr "Subida"
#: src/app/main/ui/dashboard/import.cljs:521
msgid "dashboard.import.summary.linked"
msgstr "auto-vinculada"
@ -764,6 +828,20 @@ msgstr "Ninguna biblioteca seleccionada"
msgid "dashboard.import.summary.original"
msgstr "Original"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.title"
msgstr "Importar archivo"
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.upload.message"
msgstr ""
"Enviando tus archivos. Mantén este diálogo abierto hasta que termine la "
"subida."
#: src/app/main/ui/dashboard/import.cljs
msgid "dashboard.import.upload.title"
msgstr "Subiendo tus archivos..."
#: src/app/main/ui/dashboard/team.cljs:881
msgid "dashboard.invitation-modal.delete"
msgstr "Vas a eliminar las invitaciones para:"
@ -2706,7 +2784,7 @@ msgstr "Subiendo archivo…"
#: src/app/main/ui/dashboard/import.cljs:328, src/app/main/ui/exports/files.cljs:103
msgid "jobs.queued"
msgstr "En cola"
msgstr "Esperando hueco para el job"
#: src/app/main/ui/dashboard/comments.cljs:94
msgid "label.mark-all-as-read"