mirror of
https://github.com/penpot/penpot.git
synced 2026-10-29 14:00:37 +00:00
⚡ The frontend exports assets through the jobs substrate
Every shape, frame and simple export of the UI is one `:export-assets` job now: the backend creates it with the items frozen, the external worker renders it, and the client follows its life on the same websocket channel the binfile export has ridden since it became a job. The widget shows the real milestones of the worker (`:preparing`, `:rendering`, `:packaging`, with the `objects` and `pages` counters, two new translated stage labels), downloads the artifact of the completed row, and its cancel asks the backend to stop the job. The saturation of the export queue answers as the exporter's own busy state, so the user is told to retry. The three doors of the legacy surface are gone: `repo` no longer carries `:export`, `:create-export-job` or `:cancel-export-job` — there is no `/api/export*` call left in the frontend — and their other two callers (copy-as-image and the plugin export) fetch their blob from the row's `result` resource now. The single-object wasm render stays client-side: it never leaves the browser, so it keeps downloading locally without a job. A viewer that exports through a public share names the share in the items, and the creation validates every item the way the share fetches do (the same 4-arity `check-read-permissions!` `get-file-fragment` uses); the command stays authenticated, the same as the exporter cookie it replaces, so an anonymous visitor is refused. Also here: the management `complete`/`fail` settle closes the named render session even when they lose the write, the piece the exporter's cancel work left pending. Two new translated strings (`exporting-preparing`, `exporting-packaging`), a share-aware permission test of the assets creation, and the data flow tests rewritten over the jobs channel. AI-assisted-by: glm-5.3-flash
This commit is contained in:
Andrey Antukh
parent
dba39c953d
commit
3a6ee0003f
@ -67,7 +67,7 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i
|
||||
- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
|
||||
- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
|
||||
- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.
|
||||
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
|
||||
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job`, `create-export-assets-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part — binfile checks the files it reads by id, and assets checks every item on its own: an item carrying `:share-id` validates through the share-link permissions the same way `get-file-fragment` does (a logged viewer of the share is enough; the command stays authenticated, so the anonymous visitor is refused, the same as the exporter cookie it replaces) —, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
|
||||
- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is the plain JSON the row holds, so a uuid comes back as the text it is on disk (a result has no schema of its own like the params do). The event log still has no read surface: a client that follows a job takes its progress from the websocket.
|
||||
- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow.
|
||||
- `job.error` conforms to `app.jobs/schema:job-error` (`type` and `code` keywords, `hint` text, extra details allowed). Read it back with `jobs/decode-job-error`; a plain `db/decode-json-pgobject` leaves the keywords as strings. `complete` takes named options and an optional `resource-id`, which is only ever set, never replaced: it reads the row locked inside its own transaction and applies the rules in a fixed order, skipping a job that is no longer completable before refusing one for its resource, so the answer is a function of the row and not of the order two callers arrive in.
|
||||
|
||||
@ -11,13 +11,24 @@
|
||||
- Exporter test conventions and CI: `mem:exporter/testing`.
|
||||
- Promise chains in the exporter (management client, auth) follow promesa's two API families; arg orders, what each fn must return, the `->>` preference and the async-test silent-green trap: `mem:clojure/promesa` — read before touching promise code.
|
||||
|
||||
## HTTP and browser pool
|
||||
## HTTP and browser pool (legacy surface, doomed)
|
||||
|
||||
- POST body limit is about 60 MB. Exporter supports `application/transit+json`; request params merge query params and body params.
|
||||
- Map response bodies are Transit JSON and force HTTP 200; nil 200 bodies become 204.
|
||||
- Auth token comes from cookie `auth-token`, then uploads use Bearer auth plus the management shared key.
|
||||
- Each export job gets a fresh Playwright browser context. On success, the context closes and the browser returns to the pool; on error, the browser is destroyed instead of reused.
|
||||
- Borrow validates browser connection. Pool acquire timeout is about 10s; font loading timeout logs a warning and continues after about 15s.
|
||||
- The `http` role needs `PENPOT_EXPORTER_ROLES=http`. Default is gone once the HTTP deletion task lands; the scheduler config keys (`PENPOT_EXPORTER_MAX_CONCURRENT_JOBS` and friends) belong to this surface.
|
||||
|
||||
## Worker (consumer of the jobs substrate)
|
||||
|
||||
- Roles: `PENPOT_EXPORTER_ROLES` selects `http` and/or `worker`; **default is `worker`** (consumer-only process, no HTTP server to preserve): `app.consumer.config`, all accessors read live config.
|
||||
- `worker` boots K pollers (`app.consumer.worker`): one Redis connection per poller (connection = slot + backpressure), BLPOP on `penpot.worker.queue:<tenant>:exporter`, payload `[job-id scheduled-at]` JSON; corrupt payload warns and drops, dead Redis delays 1s and retries. `K = PENPOT_EXPORTER_WORKER_CONCURRENCY` default 2, floor 1; replicas × K is the total concurrency. Pollers start after pools in `core.cljs` and stop before redis in the shutdown.
|
||||
- Management client (`app.consumer.api`): claim/progress/create-job-session/complete-job (JSON and multipart via undici FormData)/fail-job, POST to `<internal-uri>/api/management/methods/<method>` with `X-Shared-Key: exporter <key>` (key = `PENPOT_EXPORTER_SHARED_KEY` or HKDF-derived from the secret); transit bodies, non-2xx errors carry `:status` and the error body.
|
||||
- Runner (`app.consumer.exports`, `run-export!`): mints a render session (`create-job-session`), plans via `make-plan` (single → artifact of its type; multi → zip; frames → pdf via pdfunite), settles via `complete-job-with-artifact` multipart or `fail-job`; the settle never rejects.
|
||||
- Beats: per-object milestones `:preparing/:rendering/:packaging` with `objects`/`pages` counters, throttled at 250ms; a watchdog repeats the last milestone every 1s.
|
||||
- Cancellation: a `skip` answer on any beat means the row is terminal (cancelled, aborted, settled). The runner raises `:job-cancelled` between units of work; mid-Skia the watchdog fires the local port once (`jobs/mark-cancelled`: flag + SharedArrayBuffer + terminate callbacks registered by `renderer.wasm/with-scope`), killing the leased worker thread. The runner owns the local registry entry via `jobs/register!` and releases it on settle.
|
||||
- The legacy job machinery (`app.jobs`, scheduler, store) only serves the `http` surfaces plus the local cancel arm; the worker reuses pools and `job.utils/track!/release!` for temp files.
|
||||
|
||||
## Export batching and async behavior
|
||||
|
||||
|
||||
@ -5,7 +5,7 @@ Frontend: CLJS SPA; React/Rumext; Potok; RxJS; okulary refs; SCSS modules; share
|
||||
## Stable namespace map
|
||||
|
||||
- `app.main.ui.*`: Rumext/React UI components for workspace, dashboard, viewer, settings, auth, nitrate, etc.
|
||||
- `app.main.data.*`: Potok event handlers and side effects.
|
||||
- `app.main.data.*`: Potok event handlers and side effects. Exports live in `app.main.data.exports.*`: assets (shapes and frames) and files (binfile) both run as jobs of the jobs substrate (`create-export-*-job` + `dj/watch-job` on the websocket; the widget `ui.exports.assets/progress-widget` shows milestones `{stage counters}`, downloads on the completed row's `result` and cancels by job id). The single-object wasm render stays client-side: it renders and downloads locally and never creates a job.
|
||||
- `app.main.refs`: reactive refs/lenses over store and derived workspace data.
|
||||
- `app.main.store`: Potok store and `emit!`.
|
||||
- `app.plugins.*` and `app.plugins`: CLJS implementation of Plugin JS API proxies.
|
||||
|
||||
@ -121,7 +121,8 @@
|
||||
(defn- check-file-permissions
|
||||
"The files the job will read, checked before anything is stored: a
|
||||
refusal arrives synchronously. The handler checks it again before
|
||||
reading, because a permission can be revoked while the job waits."
|
||||
reading, because a permission can be revoked while the job waits.
|
||||
This is the binfile flavour: whole files, no share ids."
|
||||
[cfg profile-id file-ids]
|
||||
(when (empty? file-ids)
|
||||
(ex/raise :type :validation
|
||||
@ -130,6 +131,21 @@
|
||||
(doseq [file-id file-ids]
|
||||
(files/check-read-permissions! cfg profile-id file-id)))
|
||||
|
||||
(defn- check-item-permissions
|
||||
"The assets flavour: the items name files, and a viewer that reached
|
||||
them through a share link names the share, so the permission goes the
|
||||
same way every share fetch goes (a logged viewer of a public share is
|
||||
allowed; the workspace path stays for the plain items). The command
|
||||
is not anonymous: an anonymous visitor has no profile to own the
|
||||
artifact, the same as the exporter cookie this cutover replaces."
|
||||
[cfg profile-id items]
|
||||
(doseq [item items]
|
||||
(let [file-id (:file-id item)
|
||||
share-id (:share-id item)]
|
||||
(if (some? share-id)
|
||||
(files/check-read-permissions! cfg profile-id file-id share-id)
|
||||
(files/check-read-permissions! cfg profile-id file-id)))))
|
||||
|
||||
(sv/defmethod ::create-export-binfile-job
|
||||
"Create a durable job that exports a set of files as a `.penpot`
|
||||
package.
|
||||
@ -188,8 +204,7 @@
|
||||
;; before the command ran
|
||||
params (jobs/validate-params job-def (:params envelope))]
|
||||
|
||||
(check-file-permissions cfg profile-id
|
||||
(into #{} (map :file-id) (:exports params)))
|
||||
(check-item-permissions cfg profile-id (:exports params))
|
||||
|
||||
(quotes/check! cfg {::quotes/id ::quotes/export-jobs-per-profile
|
||||
::quotes/profile-id profile-id})
|
||||
|
||||
@ -253,7 +253,9 @@
|
||||
object is written: a lost race against a cancel never stores bytes.
|
||||
|
||||
With `:session-id`, the render session the worker minted through
|
||||
`create-job-session` is deleted after the completion commits.
|
||||
`create-job-session` is deleted once the completion is answered,
|
||||
whatever it says: the settle of the run is what ends the life of the
|
||||
session it minted, even a settle that lost against a cancel.
|
||||
|
||||
A completion of a job that already ended answers `{:action :skip,
|
||||
:status ...}` and changes nothing."
|
||||
@ -271,7 +273,10 @@
|
||||
(do
|
||||
(check-artifact! content)
|
||||
(complete-with-artifact cfg job content filename mtype session-id))
|
||||
(skip-with-status cfg job-id))
|
||||
(do
|
||||
;; a completion that lost the race is not a session to keep either
|
||||
(close-job-session! cfg session-id)
|
||||
(skip-with-status cfg job-id)))
|
||||
|
||||
;; the JSON form: nothing to store here, the resource (if any)
|
||||
;; already lives in storage
|
||||
@ -283,7 +288,9 @@
|
||||
(do
|
||||
(close-job-session! cfg session-id)
|
||||
{:action :run})
|
||||
(skip-with-status cfg job-id))))
|
||||
(do
|
||||
(close-job-session! cfg session-id)
|
||||
(skip-with-status cfg job-id)))))
|
||||
|
||||
;; ---- RPC METHOD: FAIL-JOB
|
||||
|
||||
@ -308,10 +315,12 @@
|
||||
::sm/result schema:fail-job-result
|
||||
::rpc/auth false} ;; shared-key enforced by route resolver
|
||||
[cfg {:keys [job-id error session-id]}]
|
||||
;; The session is closed whatever the write says: the worker that
|
||||
;; settles (even a settle that loses against a cancel it cannot see)
|
||||
;; is what ends the life of the render session it minted.
|
||||
(close-job-session! cfg session-id)
|
||||
(if (pos? (jobs/fail cfg job-id error))
|
||||
(do
|
||||
(close-job-session! cfg session-id)
|
||||
{:action :run})
|
||||
{:action :run}
|
||||
(skip-with-status cfg job-id)))
|
||||
|
||||
;; ---- RPC METHOD: CREATE-JOB-SESSION
|
||||
|
||||
@ -255,6 +255,51 @@
|
||||
(t/testing "and nothing was created"
|
||||
(t/is (zero? (count-jobs))))))
|
||||
|
||||
(t/deftest create-export-assets-job-accepts-a-file-nameable-through-a-share
|
||||
;; the viewer that renders a file through a public share exports the
|
||||
;; same way every share fetch goes: the share id names the file, the
|
||||
;; share-link permissions answer (a logged viewer is enough), and the
|
||||
;; workspace membership is never touched
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
viewer (th/create-profile* 2 {:is-active true})
|
||||
file-id (first (:file-ids (import-fixture! owner)))
|
||||
share (th/command! {::th/type :create-share-link
|
||||
::rpc/profile-id (:id owner)
|
||||
:file-id file-id
|
||||
:pages #{(uuid/next)}
|
||||
:who-comment "team"
|
||||
:who-inspect "all"})
|
||||
share-id (:id (:result share))
|
||||
|
||||
out (create-export-assets-job (:id viewer)
|
||||
[(export-item file-id
|
||||
:share-id share-id)])]
|
||||
|
||||
(t/testing "a logged viewer of the share can have the job"
|
||||
(t/is (th/success? out))
|
||||
(t/is (= 1 (count-jobs)))))
|
||||
|
||||
(t/testing "a share id that names another file is not a pass"
|
||||
(let [owner (th/create-profile* 3 {:is-active true})
|
||||
viewer (th/create-profile* 4 {:is-active true})
|
||||
file-a (first (:file-ids (import-fixture! owner)))
|
||||
file-b (first (:file-ids (import-fixture! viewer)))
|
||||
share (th/command! {::th/type :create-share-link
|
||||
::rpc/profile-id (:id viewer)
|
||||
:file-id file-b
|
||||
:pages #{(uuid/next)}
|
||||
:who-comment "team"
|
||||
:who-inspect "all"})
|
||||
share-id (:id (:result share))
|
||||
out (create-export-assets-job (:id viewer)
|
||||
[(export-item file-a
|
||||
:share-id share-id)])]
|
||||
(t/is (not (th/success? out)))
|
||||
(t/is (= :not-found (th/ex-type (:error out))))
|
||||
(t/testing "and nothing was created for it"
|
||||
;; only the accepted share job of the first case exists
|
||||
(t/is (= 1 (count-jobs)))))))
|
||||
|
||||
(t/deftest create-export-assets-job-adds-the-job-id-to-the-audit-props
|
||||
(let [profile (th/create-profile* 1)
|
||||
file-id (first (:file-ids (import-fixture! profile)))
|
||||
|
||||
@ -5,19 +5,33 @@
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.main.data.exports.assets
|
||||
"The shapes and frames exportation, over the jobs substrate.
|
||||
|
||||
Every export here is one `:export-assets` job: the backend creates it
|
||||
(`create-export-assets-job`) and freezes the items, the external
|
||||
exporter worker renders them, and the client follows its life on the
|
||||
websocket with `dj/watch-job`, the same channel the file exports of
|
||||
`data.exports.files` ride since they became jobs. The milestones of
|
||||
the job (`:preparing`, `:rendering`, `:packaging`, with the `objects`
|
||||
and `pages` counters) become the figures the progress widget shows;
|
||||
the completed row carries the artifact under its `result`, for the
|
||||
download; and the X of the widget cancels the job by its id.
|
||||
|
||||
The client-side wasm render of a single object is the exception: it
|
||||
never leaves the browser, so it keeps rendering and downloading
|
||||
locally."
|
||||
(:require
|
||||
[app.common.time :as ct]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.main.data.event :as ev]
|
||||
[app.main.data.exports.wasm :as wasm.exports]
|
||||
[app.main.data.helpers :as dsh]
|
||||
[app.main.data.jobs :as dj]
|
||||
[app.main.data.modal :as modal]
|
||||
[app.main.data.persistence :as dwp]
|
||||
[app.main.features :as features]
|
||||
[app.main.repo :as rp]
|
||||
[app.main.store :as st]
|
||||
[app.util.dom :as dom]
|
||||
[app.util.websocket :as ws]
|
||||
[beicon.v2.core :as rx]
|
||||
[cuerdas.core :as str]
|
||||
[potok.v2.core :as ptk]))
|
||||
@ -34,12 +48,6 @@
|
||||
[exports]
|
||||
(mapv normalize-export exports))
|
||||
|
||||
(defn- normalize-export-shapes-params
|
||||
[{:keys [exports] :as params}]
|
||||
(cond-> params
|
||||
(seq exports)
|
||||
(assoc :exports (normalize-exports exports))))
|
||||
|
||||
(defn toggle-detail-visibililty
|
||||
[]
|
||||
(ptk/reify ::toggle-detail-visibililty
|
||||
@ -66,7 +74,6 @@
|
||||
state
|
||||
(dissoc state :export))))))
|
||||
|
||||
|
||||
(defn show-workspace-export-dialog
|
||||
[{:keys [selected origin]}]
|
||||
(ptk/reify ::show-workspace-export-dialog
|
||||
@ -115,7 +122,7 @@
|
||||
(cond-> share-id (assoc :share-id share-id))))]
|
||||
(rx/of (modal/show :export-shapes {:exports (vec exports)
|
||||
:origin "viewer"
|
||||
:name name})))))) #_TODO
|
||||
:name name}))))))
|
||||
|
||||
(defn show-workspace-export-frames-dialog
|
||||
[frames]
|
||||
@ -140,69 +147,112 @@
|
||||
:origin "workspace:menu"
|
||||
:name page-name}))))))
|
||||
|
||||
;; ---- THE PROGRESS WIDGET STATE ------------------------------------
|
||||
;; The events this namespace emits for the export progress widget. The
|
||||
;; job's channel (`dj/watch-job`) turns into them on its way up.
|
||||
|
||||
(defn- initialize-export-status
|
||||
"`job` is only present on the job API path; without it the widget counts the
|
||||
exports the client submitted, exactly as it always has."
|
||||
[exports cmd resource {:keys [job-id total status backend] :as job}]
|
||||
"The state a just created job starts at: the widget shows it queued,
|
||||
with no figures yet — the counter comes with the first milestone the
|
||||
worker publishes, and counts itself down from the items meanwhile."
|
||||
[exports cmd {:keys [id status]}]
|
||||
(ptk/reify ::initialize-export-status
|
||||
ptk/UpdateEvent
|
||||
(update [_ state]
|
||||
(assoc state :export (cond-> {:in-progress true
|
||||
:resource-id (:id resource)
|
||||
:healthy? true
|
||||
:error false
|
||||
:progress 0
|
||||
:widget-visible true
|
||||
:detail-visible true
|
||||
:exports exports
|
||||
:last-update (ct/now)
|
||||
:cmd cmd}
|
||||
(some? job)
|
||||
(assoc :job-id job-id
|
||||
:total total
|
||||
:status status
|
||||
:backend backend))))))
|
||||
(assoc state :export {:in-progress true
|
||||
:healthy? true
|
||||
:error false
|
||||
:progress 0
|
||||
:widget-visible true
|
||||
:detail-visible true
|
||||
:exports exports
|
||||
:last-update (ct/now)
|
||||
:cmd cmd
|
||||
:job-id id
|
||||
:status status}))))
|
||||
|
||||
(defn- update-export-status
|
||||
[{:keys [done total status resource-uri filename mtype] :as data}]
|
||||
(ptk/reify ::update-export-status
|
||||
ptk/UpdateEvent
|
||||
(update [_ state]
|
||||
(let [time-diff (ct/diff-ms (get-in state [:export :last-update]) (ct/now))
|
||||
healthy? (< time-diff 6000)
|
||||
;; The legacy path has no server-side figures to track; it keeps
|
||||
;; reporting progress over the client's own list.
|
||||
job? (some? (get-in state [:export :job-id]))]
|
||||
(cond-> state
|
||||
job?
|
||||
(update :export assoc :status status)
|
||||
(defn- counter-of
|
||||
"The figures of one milestone: `objects` counts the shapes and
|
||||
`pages` counts the frames, both as `current` exported and `total`
|
||||
planned. A job publishes one kind or the other; this takes whatever
|
||||
it carries."
|
||||
[counters]
|
||||
(or (:objects counters) (:pages counters)))
|
||||
|
||||
(and job? (some? total))
|
||||
(update :export assoc :total total)
|
||||
(defn- add-milestone
|
||||
"One beat of the job into the widget: what has landed and what is
|
||||
coming, named by the stage the run is in. A milestone that arrives
|
||||
once the widget settled from a cancel or an error is ignored."
|
||||
[payload]
|
||||
(let [figures (counter-of (:counters payload))]
|
||||
(ptk/reify ::add-milestone
|
||||
ptk/UpdateEvent
|
||||
(update [_ state]
|
||||
(if-not (get-in state [:export :in-progress])
|
||||
state
|
||||
(update state :export assoc
|
||||
:status "running"
|
||||
:stage (:stage payload)
|
||||
:last-update (ct/now)
|
||||
:progress (:current figures)
|
||||
:total (or (:total figures)
|
||||
(get-in state [:export :total]))))))))
|
||||
|
||||
(= status "running")
|
||||
(update :export assoc :progress done :last-update (ct/now) :healthy? healthy?)
|
||||
(defn- add-outcome
|
||||
"The last row of the job as the end of the widget. A completed job
|
||||
downloads its artifact right away and the widget goes away after a
|
||||
moment; a failed one stays, naming the error; a cancelled one just
|
||||
settles; and a cancelling step keeps the widget alive while the
|
||||
cancel flies."
|
||||
[{:keys [status result error]}]
|
||||
(let [downloadable? (or (= "completed" status) (= "ended" status))]
|
||||
(ptk/reify ::add-outcome
|
||||
ptk/UpdateEvent
|
||||
(update [_ state]
|
||||
(update state :export (fn [export]
|
||||
(if-not (and (map? export)
|
||||
(or (:in-progress export)
|
||||
(= "cancelling" (:status export))))
|
||||
export
|
||||
(cond
|
||||
(= "cancelling" status)
|
||||
(assoc export :status "cancelling"
|
||||
:last-update (ct/now))
|
||||
|
||||
(= status "error")
|
||||
(update :export assoc :in-progress false :error (:cause data) :last-update (ct/now) :healthy? healthy?)
|
||||
downloadable?
|
||||
(assoc export :status "ended"
|
||||
:in-progress false
|
||||
:last-update (ct/now))
|
||||
|
||||
(= status "cancelling")
|
||||
(update :export assoc :last-update (ct/now) :healthy? healthy?)
|
||||
(= "failed" status)
|
||||
(assoc export :status "error"
|
||||
:in-progress false
|
||||
:error (:hint error)
|
||||
:error-code (:code error)
|
||||
:last-update (ct/now))
|
||||
|
||||
(= status "cancelled")
|
||||
(update :export assoc :in-progress false :last-update (ct/now) :healthy? healthy?)
|
||||
(= "cancelled" status)
|
||||
(assoc export :status "cancelled"
|
||||
:in-progress false
|
||||
:last-update (ct/now))
|
||||
|
||||
(= status "ended")
|
||||
(update :export assoc :in-progress false :progress done :last-update (ct/now) :healthy? healthy?))))
|
||||
:else export)))))
|
||||
|
||||
ptk/WatchEvent
|
||||
(watch [_ _ _]
|
||||
(when (= status "ended")
|
||||
(dom/trigger-download-uri filename mtype resource-uri)))))
|
||||
ptk/WatchEvent
|
||||
(watch [_ _ _]
|
||||
(when downloadable?
|
||||
(dom/trigger-download-uri (:filename result)
|
||||
(:mtype result)
|
||||
(:resource-uri result)))
|
||||
(when (or downloadable? (= "cancelled" status))
|
||||
;; the widget has its moment naming what happened, and goes
|
||||
;; away
|
||||
(->> (rx/of (clear-export-state nil))
|
||||
(rx/delay default-timeout)))))))
|
||||
|
||||
;; The exporter is at capacity. Not a crash: the widget says so and the user
|
||||
;; retries, instead of the generic error dialog.
|
||||
(def ^:private saturation-codes #{:queue-full})
|
||||
(def ^:private saturation-codes #{:queue-full :max-quote-reached})
|
||||
|
||||
(defn- export-failed
|
||||
"Reports a failure that happened before the export ever started, so the widget
|
||||
@ -224,32 +274,72 @@
|
||||
:last-update (ct/now)}))))
|
||||
|
||||
(defn cancel-export
|
||||
"Stops the running export. Only reachable on the job API path, where the
|
||||
exporter can actually abort the work.
|
||||
|
||||
The widget settles from here rather than from the job's `cancelled` message:
|
||||
the outcome is known once the request returns, and waiting on a round trip
|
||||
through redis and the websocket would leave it stuck whenever that message is
|
||||
missed."
|
||||
"Stops the running export: the X of the widget asks the backend to
|
||||
cancel its job — fire and forget, the job may have just ended on its
|
||||
own — and settles the widget itself, because the outcome is known
|
||||
once the cancel is dispatched; waiting on the row and the websocket
|
||||
would leave it stuck whenever that message is missed. The widget
|
||||
keeps its cancelled state on display a moment, and goes away."
|
||||
[]
|
||||
(ptk/reify ::cancel-export
|
||||
ptk/WatchEvent
|
||||
(watch [_ state _]
|
||||
(when-let [job-id (get-in state [:export :job-id])]
|
||||
(let [resource-id (get-in state [:export :resource-id])
|
||||
settle (rx/concat
|
||||
(rx/of (update-export-status {:status "cancelled"}))
|
||||
(->> (rx/of (clear-export-state resource-id))
|
||||
(rx/delay default-timeout)))]
|
||||
(rx/concat
|
||||
;; Stopping is not instantaneous: the request has to reach the
|
||||
;; exporter and the work has to unwind.
|
||||
(rx/of (update-export-status {:status "cancelling"}))
|
||||
(->> (rp/cmd! :cancel-export-job {:job-id job-id})
|
||||
(rx/mapcat (fn [_] settle))
|
||||
;; Already finished, or the exporter is gone; either way
|
||||
;; there is nothing left to stop.
|
||||
(rx/catch (fn [_] settle)))))))))
|
||||
(dj/cancel-job job-id)
|
||||
;; Stopping is not instantaneous: the row is marked, and the
|
||||
;; worker stops at its next beat.
|
||||
(rx/of (add-outcome {:status "cancelling"}))))))
|
||||
|
||||
;; ---- THE JOB FLOW -------------------------------------------------
|
||||
|
||||
(defn- ->widget-event
|
||||
"One emission of the job taken from `dj/watch-job` is the event the
|
||||
widget consumes: the payload of a progress event carries the
|
||||
milestone, and the row at the end answers with the outcome."
|
||||
[emission]
|
||||
(cond
|
||||
(and (= :progress (:kind emission))
|
||||
(map? (:payload emission)))
|
||||
(add-milestone (:payload emission))
|
||||
|
||||
(contains? emission :status)
|
||||
(add-outcome emission)))
|
||||
|
||||
(defn- export-stream!
|
||||
"The whole life of one export job as a stream of widget events: the
|
||||
creation of the job, then its channel (`dj/watch-job`) mapped into
|
||||
the vocabulary of the widget, and the stream over once the outcome
|
||||
lands. The start and retry events of the channel are dropped by the
|
||||
mapper.
|
||||
|
||||
A creation that could not even happen (the export queue saturated,
|
||||
mostly) reports on the widget and, unless it is saturation, rethrows
|
||||
for the global error handling of the store."
|
||||
[ws-conn exports cmd {:keys [force-multiple name is-wasm]}]
|
||||
(let [params (cond-> {:exports exports
|
||||
:is-wasm is-wasm}
|
||||
(some? name)
|
||||
(assoc :name name)
|
||||
|
||||
(some? force-multiple)
|
||||
(assoc :force-multiple force-multiple))]
|
||||
|
||||
(->> (rp/cmd! :create-export-assets-job {:params params})
|
||||
(rx/mapcat
|
||||
(fn [job]
|
||||
(rx/concat
|
||||
(rx/of (initialize-export-status exports cmd job))
|
||||
(->> (dj/watch-job ws-conn (:id job))
|
||||
(rx/map ->widget-event)
|
||||
(rx/filter some?)))))
|
||||
(rx/catch (fn [cause]
|
||||
(let [failed (export-failed exports cmd cause)]
|
||||
(if (contains? saturation-codes (:code (ex-data cause)))
|
||||
(rx/of failed)
|
||||
(rx/concat (rx/of failed)
|
||||
(rx/throw cause))))))
|
||||
(rx/finalize (fn []
|
||||
(swap! st/ongoing-tasks disj :export))))))
|
||||
|
||||
(def ^:private wasm-export-types #{:jpeg :webp :png :pdf :svg})
|
||||
|
||||
@ -278,33 +368,16 @@
|
||||
[{:keys [export]}]
|
||||
(let [export (normalize-export export)]
|
||||
(ptk/reify ::request-simple-export
|
||||
ptk/UpdateEvent
|
||||
(update [_ state]
|
||||
(cond-> state
|
||||
(not (use-wasm-export? state export))
|
||||
(update :export assoc :in-progress true :id uuid/zero)))
|
||||
|
||||
ptk/WatchEvent
|
||||
(watch [_ state _]
|
||||
(if (use-wasm-export? state export)
|
||||
(rx/of (request-simple-export-wasm export))
|
||||
(let [profile-id (:profile-id state)
|
||||
params (normalize-export-shapes-params {:exports [export]
|
||||
:profile-id profile-id
|
||||
:cmd :export-shapes
|
||||
:wait true
|
||||
:is-wasm (wasm-export-enabled? state)})]
|
||||
(let [ws-conn (:ws-conn state)]
|
||||
(swap! st/ongoing-tasks conj :export)
|
||||
(rx/concat
|
||||
(dwp/force-persist-and-wait 400)
|
||||
|
||||
(->> (rp/cmd! :export params)
|
||||
(rx/map (fn [{:keys [filename mtype uri]}]
|
||||
(dom/trigger-download-uri filename mtype uri)
|
||||
(clear-export-state uuid/zero)))
|
||||
(rx/catch (fn [cause]
|
||||
(rx/concat
|
||||
(rx/of (clear-export-state uuid/zero))
|
||||
(rx/throw cause))))))))))))
|
||||
(export-stream! ws-conn [export] nil
|
||||
{:is-wasm (wasm-export-enabled? state)}))))))))
|
||||
|
||||
(defn request-multiple-export
|
||||
[{:keys [exports cmd name]
|
||||
@ -314,85 +387,16 @@
|
||||
(ptk/reify ::request-multiple-export
|
||||
ptk/WatchEvent
|
||||
(watch [_ state _]
|
||||
(let [resource-id (volatile! nil)
|
||||
profile-id (:profile-id state)
|
||||
ws-conn (:ws-conn state)
|
||||
params (cond->
|
||||
{:exports exports
|
||||
:cmd cmd
|
||||
:profile-id profile-id
|
||||
:force-multiple true
|
||||
:is-wasm (wasm-export-enabled? state)}
|
||||
(some? name)
|
||||
(assoc :name name))
|
||||
|
||||
progress-stream
|
||||
(->> (ws/get-rcv-stream ws-conn)
|
||||
(rx/filter ws/message-event?)
|
||||
(rx/map :payload)
|
||||
(rx/filter #(= :export-update (:type %)))
|
||||
(rx/filter #(= @resource-id (:resource-id %)))
|
||||
(rx/share))
|
||||
|
||||
stopper
|
||||
(rx/filter #(or (= "ended" (:status %))
|
||||
(= "error" (:status %))
|
||||
(= "cancelled" (:status %)))
|
||||
progress-stream)]
|
||||
|
||||
(let [ws-conn (:ws-conn state)]
|
||||
(swap! st/ongoing-tasks conj :export)
|
||||
|
||||
(rx/merge
|
||||
;; Force that all data is persisted; best effort.
|
||||
(rx/of ::dwp/force-persist)
|
||||
|
||||
;; Launch the exportation process and stores the resource id
|
||||
;; locally. With wasm export active the job API is used instead: it
|
||||
;; answers with the exporter's own object count and gives a handle
|
||||
;; to cancel.
|
||||
(->> (if (wasm-export-enabled? state)
|
||||
(->> (rp/cmd! :create-export-job params)
|
||||
(rx/map (fn [{job-id :id :keys [total] :as job}]
|
||||
(vreset! resource-id (:resource-id job))
|
||||
(initialize-export-status exports cmd
|
||||
{:id (:resource-id job)}
|
||||
{:job-id job-id
|
||||
:total total
|
||||
:status (:state job)
|
||||
:backend (:backend job)}))))
|
||||
(->> (rp/cmd! :export params)
|
||||
(rx/map (fn [{:keys [id] :as resource}]
|
||||
(vreset! resource-id id)
|
||||
(initialize-export-status exports cmd resource nil)))))
|
||||
(rx/catch (fn [cause]
|
||||
;; Saturation is an answer, not a fault.
|
||||
(if (contains? saturation-codes (:code (ex-data cause)))
|
||||
(rx/of (export-failed exports cmd cause))
|
||||
(rx/concat
|
||||
(rx/of (export-failed exports cmd cause))
|
||||
(rx/throw cause))))))
|
||||
|
||||
;; We proceed to update the export state with incoming
|
||||
;; progress updates. We delay the stopper for give some time
|
||||
;; to update the status with ended or errored status before
|
||||
;; close the stream.
|
||||
(->> progress-stream
|
||||
(rx/map update-export-status)
|
||||
(rx/take-until (rx/delay 500 stopper))
|
||||
(rx/finalize (fn []
|
||||
(swap! st/ongoing-tasks disj :export))))
|
||||
|
||||
;; We hide need to hide the ui elements of the export after
|
||||
;; some interval. We also delay a little bit more the stopper
|
||||
;; for ensure that after some security time, the stream is
|
||||
;; completely closed.
|
||||
(->> progress-stream
|
||||
(rx/filter #(or (= "ended" (:status %))
|
||||
(= "cancelled" (:status %))))
|
||||
(rx/take 1)
|
||||
(rx/delay default-timeout)
|
||||
(rx/map #(clear-export-state @resource-id))
|
||||
(rx/take-until (rx/delay 6000 stopper)))))))))
|
||||
(->> (export-stream! ws-conn exports cmd
|
||||
{:force-multiple true
|
||||
:name name
|
||||
:is-wasm (wasm-export-enabled? state)}))))))))
|
||||
|
||||
(defn request-export
|
||||
[{:keys [exports] :as params}]
|
||||
|
||||
@ -37,6 +37,7 @@
|
||||
[app.main.data.exports.assets :as de]
|
||||
[app.main.data.exports.wasm :as wasm.exports]
|
||||
[app.main.data.helpers :as dsh]
|
||||
[app.main.data.jobs :as dj]
|
||||
[app.main.data.notifications :as ntf]
|
||||
[app.main.data.persistence :as dps]
|
||||
[app.main.data.workspace.media :as dwm]
|
||||
@ -1208,11 +1209,17 @@
|
||||
;; Call exporter to get image URI, then fetch blob and resolve the deferred.
|
||||
(->> (if (features/active-feature? state "render-wasm/v1")
|
||||
(rx/of {:uri (wasm.exports/export-image-uri export)})
|
||||
(rp/cmd! :export
|
||||
{:exports [export]
|
||||
:profile-id (:profile-id state)
|
||||
:cmd :export-shapes
|
||||
:wait true}))
|
||||
|
||||
;; One export job, its artifact under the completed row's
|
||||
;; result: the URI of the resource is what the blob fetch
|
||||
;; reads now, as the download of the exports does.
|
||||
(->> (rp/cmd! :create-export-assets-job
|
||||
{:params {:exports [export]}})
|
||||
(rx/mapcat (fn [{job-id :id}]
|
||||
(->> (dj/watch-job (:ws-conn state) job-id)
|
||||
(rx/filter #(= "completed" (:status %)))
|
||||
(rx/map :result))))
|
||||
(rx/map (fn [result] {:uri (:resource-uri result)}))))
|
||||
|
||||
(rx/mapcat (fn [{:keys [uri]}]
|
||||
(http/send! {:method :get
|
||||
|
||||
@ -267,45 +267,6 @@
|
||||
(rx/map http/conditional-decode-transit)
|
||||
(rx/mapcat handle-response))))
|
||||
|
||||
(defn- send-export
|
||||
[{:keys [blob?] :as params}]
|
||||
(->> (http/send! {:method :post
|
||||
:uri (u/join cf/public-uri "api/export")
|
||||
:body (http/transit-data (dissoc params :blob?))
|
||||
:headers {"x-external-session-id" (cf/external-session-id)
|
||||
"x-event-origin" (::ev/origin (meta params))}
|
||||
:credentials "include"
|
||||
:response-type (if blob? :blob :text)})
|
||||
(rx/map http/conditional-decode-transit)
|
||||
(rx/mapcat handle-response)))
|
||||
|
||||
(defmethod cmd! :export
|
||||
[_ params]
|
||||
(let [default {:wait false :blob? false}]
|
||||
(send-export (merge default params))))
|
||||
|
||||
(defmethod cmd! :create-export-job
|
||||
[_ params]
|
||||
(->> (http/send! {:method :post
|
||||
:uri (u/join cf/public-uri "api/export/jobs")
|
||||
:body (http/transit-data params)
|
||||
:headers {"x-external-session-id" (cf/external-session-id)
|
||||
"x-event-origin" (::ev/origin (meta params))}
|
||||
:credentials "include"
|
||||
:response-type :text})
|
||||
(rx/map http/conditional-decode-transit)
|
||||
(rx/mapcat handle-response)))
|
||||
|
||||
(defmethod cmd! :cancel-export-job
|
||||
[_ {:keys [job-id]}]
|
||||
(->> (http/send! {:method :delete
|
||||
:uri (u/join cf/public-uri "api/export/jobs/" (str job-id))
|
||||
:headers {"x-external-session-id" (cf/external-session-id)}
|
||||
:credentials "include"
|
||||
:response-type :text})
|
||||
(rx/map http/conditional-decode-transit)
|
||||
(rx/mapcat handle-response)))
|
||||
|
||||
(defn- multipart-upload
|
||||
[id params]
|
||||
(->> (http/send! {:method :post
|
||||
|
||||
@ -220,7 +220,8 @@
|
||||
error? (:error state)
|
||||
;; The exporter is at capacity: worth its own wording, so the user
|
||||
;; knows retrying later is the thing to do.
|
||||
busy? (= :queue-full (:error-code state))
|
||||
busy? (contains? #{:queue-full :max-quote-reached}
|
||||
(:error-code state))
|
||||
healthy? (:healthy? state)
|
||||
detail-visible? (:detail-visible state)
|
||||
widget-visible? (:widget-visible state)
|
||||
@ -228,13 +229,19 @@
|
||||
items (:exports state)
|
||||
job-id (:job-id state)
|
||||
status (:status state)
|
||||
;; The stages of the job, named: what the worker reports beats
|
||||
;; a single "exporting" wording.
|
||||
stage-title (case (:stage state)
|
||||
:preparing (tr "workspace.options.exporting-preparing")
|
||||
:packaging (tr "workspace.options.exporting-packaging")
|
||||
nil)
|
||||
queued? (and (some? job-id) (= "queued" status))
|
||||
cancelling? (and (some? job-id) (= "cancelling" status))
|
||||
cancelled? (and (some? job-id) (= "cancelled" status))
|
||||
;; Only the wasm backend can actually stop: a browser render holds its
|
||||
;; pool slot until playwright gives up.
|
||||
;; The job is cancellable while it runs: the mark reaches the
|
||||
;; row and the worker unwinds at its next beat.
|
||||
cancellable? (and (some? job-id)
|
||||
(= "wasm" (:backend state))
|
||||
(= "running" status)
|
||||
(:in-progress state)
|
||||
(not cancelling?))
|
||||
total (or (:total state) (count items))
|
||||
@ -270,6 +277,7 @@
|
||||
cancelled? (tr "workspace.options.exporting-cancelled")
|
||||
queued? (tr "workspace.options.exporting-queued")
|
||||
complete? (tr "workspace.options.exporting-complete")
|
||||
(and healthy? (some? stage-title)) stage-title
|
||||
healthy? (tr "workspace.options.exporting-object")
|
||||
(not healthy?) (tr "workspace.options.exporting-object-slow"))
|
||||
|
||||
|
||||
@ -35,6 +35,7 @@
|
||||
[app.common.uuid :as uuid]
|
||||
[app.main.data.exports.assets :as de]
|
||||
[app.main.data.exports.wasm :as wasm.exports]
|
||||
[app.main.data.jobs :as dj]
|
||||
[app.main.data.persistence :as dwp]
|
||||
[app.main.data.plugins :as dp]
|
||||
[app.main.data.workspace :as dw]
|
||||
@ -1630,20 +1631,18 @@
|
||||
(rx/tap #(st/emit! (se/event plugin-id "export-shapes" :method "wasm")))
|
||||
(rx/subs! resolve reject)))))
|
||||
|
||||
;; Render via the exporter service.
|
||||
(let [shape (u/locate-shape file-id page-id id)
|
||||
payload
|
||||
{:cmd :export-shapes
|
||||
:profile-id (:profile-id @st/state)
|
||||
:wait true
|
||||
:is-wasm wasm-enabled?
|
||||
:exports [(de/normalize-export {:file-id file-id
|
||||
:page-id page-id
|
||||
:object-id id
|
||||
:name (:name shape)
|
||||
:type export-type
|
||||
:suffix (:suffix value "")
|
||||
:scale (:scale value 1)})]}]
|
||||
;; One export job of the named item: the worker
|
||||
;; renders it, the completed row carries the artifact
|
||||
;; under its result, and the blob fetch reads its
|
||||
;; resource uri.
|
||||
(let [shape (u/locate-shape file-id page-id id)
|
||||
item (de/normalize-export {:file-id file-id
|
||||
:page-id page-id
|
||||
:object-id id
|
||||
:name (:name shape)
|
||||
:type export-type
|
||||
:suffix (:suffix value "")
|
||||
:scale (:scale value 1)})]
|
||||
(js/Promise.
|
||||
(fn [resolve reject]
|
||||
;; Exporter reads persisted file state; flush first
|
||||
@ -1653,10 +1652,16 @@
|
||||
(->> (rx/concat
|
||||
(->> (dwp/wait-persisted 5000)
|
||||
(rx/ignore))
|
||||
(rp/cmd! :export payload))
|
||||
(rx/mapcat (fn [{:keys [uri]}]
|
||||
(->> (rp/cmd! :create-export-assets-job
|
||||
{:params {:exports [item]
|
||||
:is-wasm wasm-enabled?}})
|
||||
(rx/mapcat (fn [{job-id :id}]
|
||||
(->> (dj/watch-job (:ws-conn @st/state) job-id)
|
||||
(rx/filter #(= "completed" (:status %)))
|
||||
(rx/map :result))))))
|
||||
(rx/mapcat (fn [result]
|
||||
(->> (http/send! {:method :get
|
||||
:uri uri
|
||||
:uri (:resource-uri result)
|
||||
:response-type :blob
|
||||
:omit-default-headers true})
|
||||
(rx/map :body))))
|
||||
|
||||
@ -5,14 +5,17 @@
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns frontend-tests.data.exports-assets-test
|
||||
"The assets export over the jobs substrate: one job per run, its
|
||||
milestones into the widget, the artifact of the completed row as the
|
||||
download, and the cancel from the widget."
|
||||
(:require
|
||||
[app.common.uuid :as uuid]
|
||||
[app.main.data.exports.assets :as de]
|
||||
[app.main.data.jobs :as dj]
|
||||
[app.main.data.persistence :as dwp]
|
||||
[app.main.repo :as repo]
|
||||
[app.main.store :as st]
|
||||
[app.util.dom :as dom]
|
||||
[app.util.websocket :as ws]
|
||||
[beicon.v2.core :as rx]
|
||||
[cljs.test :as t :include-macros true]
|
||||
[frontend-tests.helpers.events :as the]
|
||||
@ -30,9 +33,31 @@
|
||||
(merge export {:name name}))
|
||||
|
||||
(defn- test-state
|
||||
[]
|
||||
{:profile-id (:id export)
|
||||
:ws-conn nil})
|
||||
([] (test-state nil))
|
||||
([export-state]
|
||||
(cond-> {:profile-id (:id export)
|
||||
:ws-conn nil}
|
||||
(some? export-state)
|
||||
(assoc :export export-state))))
|
||||
|
||||
(def ^:private fake-job
|
||||
"What the creation of the job answers: enough to follow it."
|
||||
{:id (uuid/next) :status "queued" :name "export-assets"})
|
||||
|
||||
(defn- milestone
|
||||
[stage current total counter-kind]
|
||||
{:kind :progress
|
||||
:payload {:stage stage
|
||||
:counters {counter-kind {:current current :total total}}}})
|
||||
|
||||
(defn- fake-watch
|
||||
"The channel of the job, as a stream of emissions the flow reads."
|
||||
[emissions]
|
||||
(apply rx/concat (map rx/of emissions)))
|
||||
|
||||
(defn- completed-emission
|
||||
[result]
|
||||
{:status "completed" :result result})
|
||||
|
||||
(t/deftest normalize-export-preserves-existing-name
|
||||
(t/is (= (export-with-name "Layer 1")
|
||||
@ -46,54 +71,151 @@
|
||||
(t/is (= (export-with-name (str (:object-id export)))
|
||||
(de/normalize-export (assoc export :name "")))))
|
||||
|
||||
(t/deftest request-simple-export-sends-normalized-export
|
||||
(t/deftest request-simple-export-runs-one-assets-job
|
||||
(t/async done
|
||||
(let [export (export-with-name "")
|
||||
observed (atom nil)]
|
||||
(mock/with-mocks {repo/cmd! (mock/stub (fn [_ params]
|
||||
(reset! observed params)
|
||||
(rx/of {:filename "export.png"
|
||||
:mtype "image/png"
|
||||
:uri "blob:export"})))
|
||||
dwp/force-persist-and-wait (mock/stub (fn [_] (rx/of ::force-persisted)))
|
||||
dom/trigger-download-uri (mock/stub (fn [& _] nil))}
|
||||
(let [export (export-with-name "")
|
||||
observed (atom nil)
|
||||
downloaded (atom nil)]
|
||||
(mock/with-mocks
|
||||
{repo/cmd! (mock/stub (fn [cmd params]
|
||||
(reset! observed {:cmd cmd :params (:params params)})
|
||||
(rx/of fake-job)))
|
||||
dj/watch-job (mock/stub (fn [_ _]
|
||||
(fake-watch
|
||||
[(milestone :rendering 1 1 :objects)
|
||||
(milestone :packaging 1 1 :objects)
|
||||
(completed-emission {:resource-uri "https://resources/uri"
|
||||
:filename "export.png"
|
||||
:mtype "image/png"})])))
|
||||
dwp/force-persist-and-wait (mock/stub (fn [_] (rx/of ::force-persisted)))
|
||||
dom/trigger-download-uri (mock/stub (fn [filename mtype uri]
|
||||
(when filename
|
||||
(reset! downloaded
|
||||
{:filename filename :mtype mtype :uri uri}))))
|
||||
st/ongoing-tasks (atom #{})}
|
||||
(fn [done']
|
||||
(let [completed (fn [_state]
|
||||
(t/is (= (export-with-name (str (:object-id export)))
|
||||
(-> @observed :exports first))))]
|
||||
(let [completed (fn [state]
|
||||
(t/testing "the creation froze the normalized export"
|
||||
(t/is (= :create-export-assets-job (:cmd @observed)))
|
||||
(t/is (= [(export-with-name (str (:object-id export)))]
|
||||
(get-in @observed [:params :exports]))))
|
||||
|
||||
(t/is (= 1 (get-in state [:export :progress])))
|
||||
|
||||
(t/testing "the widget settled the completed job"
|
||||
(t/is (false? (get-in state [:export :in-progress])))
|
||||
(t/is (= "ended" (get-in state [:export :status])))
|
||||
(t/is (= 1 (get-in state [:export :total]))))
|
||||
|
||||
(t/testing "the artifact of the row is the download"
|
||||
(t/is (= {:filename "export.png"
|
||||
:mtype "image/png"
|
||||
:uri "https://resources/uri"}
|
||||
@downloaded))))]
|
||||
(ptk/emit! (the/prepare-store (test-state) done' completed)
|
||||
(de/request-simple-export {:export export})
|
||||
:the/end)))
|
||||
done))))
|
||||
|
||||
(t/deftest request-multiple-export-sends-normalized-enabled-exports
|
||||
(t/deftest request-simple-export-reads-the-pages-of-a-frame-export
|
||||
;; A frame names no `type`: it is the pdf of its page, and its
|
||||
;; milestones count pages, not objects.
|
||||
(t/async done
|
||||
(let [exports [{:id "enabled-1"
|
||||
:object-id "enabled-1"
|
||||
:shape {:id "enabled-1"}
|
||||
:type :png
|
||||
:suffix ""
|
||||
:scale 1
|
||||
:enabled true
|
||||
:name ""}]
|
||||
observed (atom nil)]
|
||||
(mock/with-mocks {repo/cmd! (mock/stub (fn [_ params]
|
||||
(reset! observed params)
|
||||
(rx/of {:id (:id export)})))
|
||||
ws/get-rcv-stream (mock/stub (fn [_] (rx/empty)))
|
||||
st/ongoing-tasks (atom #{})}
|
||||
(let [frame {:id (uuid/next)
|
||||
:object-id (uuid/next)
|
||||
:suffix ""
|
||||
:scale 1
|
||||
:name "Frame 1"}
|
||||
downloaded (atom nil)]
|
||||
(mock/with-mocks
|
||||
{repo/cmd! (mock/stub (fn [_ _] (rx/of fake-job)))
|
||||
dj/watch-job (mock/stub (fn [_ _]
|
||||
(fake-watch
|
||||
[(milestone :rendering 1 2 :pages)
|
||||
(milestone :packaging 2 2 :pages)
|
||||
(completed-emission {:resource-uri "https://resources/uri"
|
||||
:filename "file.pdf"
|
||||
:mtype "application/pdf"})])))
|
||||
dwp/force-persist-and-wait (mock/stub (fn [_] (rx/of ::force-persisted)))
|
||||
dom/trigger-download-uri (mock/stub (fn [filename _mtype _uri]
|
||||
(reset! downloaded filename)))
|
||||
st/ongoing-tasks (atom #{})}
|
||||
(fn [done']
|
||||
(let [completed (fn [_state]
|
||||
(t/is (= [{:id "enabled-1"
|
||||
:object-id "enabled-1"
|
||||
:shape {:id "enabled-1"}
|
||||
:type :png
|
||||
:suffix ""
|
||||
:scale 1
|
||||
:enabled true
|
||||
:name "enabled-1"}]
|
||||
(:exports @observed))))]
|
||||
(let [completed (fn [state]
|
||||
(t/is (= 2 (get-in state [:export :progress])))
|
||||
(t/is (= 2 (get-in state [:export :total])))
|
||||
(t/is (= "file.pdf" @downloaded)))]
|
||||
(ptk/emit! (the/prepare-store (test-state) done' completed)
|
||||
(de/request-multiple-export {:exports exports})
|
||||
(de/request-simple-export {:export frame})
|
||||
:the/end)))
|
||||
done))))
|
||||
|
||||
(t/deftest request-multiple-export-freezes-the-run-of-many
|
||||
(t/async done
|
||||
(let [items [(assoc export :name "" :enabled true)]
|
||||
observed (atom nil)]
|
||||
(mock/with-mocks
|
||||
{repo/cmd! (mock/stub (fn [cmd params]
|
||||
(reset! observed {:cmd cmd :params (:params params)})
|
||||
(rx/of fake-job)))
|
||||
dj/watch-job (mock/stub (fn [_ _]
|
||||
(fake-watch
|
||||
[(milestone :rendering 1 1 :objects)
|
||||
(completed-emission {:resource-uri "https://resources/uri"
|
||||
:filename "export.zip"
|
||||
:mtype "application/zip"})])))
|
||||
dom/trigger-download-uri (mock/stub (fn [& _] nil))
|
||||
st/ongoing-tasks (atom #{})}
|
||||
(fn [done']
|
||||
(let [completed (fn [state]
|
||||
(t/testing "the creation froze the multiple run"
|
||||
(t/is (= true (get-in @observed [:params :force-multiple]))))
|
||||
(t/testing "the widget followed the job"
|
||||
(t/is (false? (get-in state [:export :in-progress])))
|
||||
(t/is (= "ended" (get-in state [:export :status])))))]
|
||||
(ptk/emit! (the/prepare-store (test-state) done' completed)
|
||||
(de/request-multiple-export {:exports items})
|
||||
:the/end)))
|
||||
done))))
|
||||
|
||||
(t/deftest request-multiple-export-reports-the-saturation
|
||||
;; The creation refused because the queue is full: an answer, not a
|
||||
;; fault — the widget names it and offers the retry.
|
||||
(t/async done
|
||||
(let [items [(assoc export :name "" :enabled true)]]
|
||||
(mock/with-mocks
|
||||
{repo/cmd! (mock/stub (fn [_ _]
|
||||
(rx/throw (ex-info "quote exceeded" {:code :max-quote-reached}))))
|
||||
st/ongoing-tasks (atom #{})}
|
||||
(fn [done']
|
||||
(let [completed (fn [state]
|
||||
(t/testing "the widget names the exporter busy"
|
||||
(t/is (false? (get-in state [:export :in-progress])))
|
||||
(t/is (= :max-quote-reached (get-in state [:export :error-code])))))]
|
||||
(ptk/emit! (the/prepare-store (test-state) done' completed)
|
||||
(de/request-multiple-export {:exports items})
|
||||
:the/end)))
|
||||
done))))
|
||||
|
||||
(t/deftest cancel-export-asks-the-backend-and-settles
|
||||
(t/async done
|
||||
(let [job-id (uuid/next)
|
||||
observed (atom nil)]
|
||||
(mock/with-mocks
|
||||
{repo/cmd! (mock/stub (fn [_ params]
|
||||
(reset! observed params)
|
||||
(rx/of nil)))}
|
||||
(fn [done']
|
||||
(let [completed (fn [state]
|
||||
(t/testing "the cancel of the job the widget tracks"
|
||||
(t/is (= {:id job-id} @observed)))
|
||||
(t/testing "and the widget shows the cancel flying"
|
||||
(t/is (= "cancelling" (get-in state [:export :status])))
|
||||
(t/is (true? (get-in state [:export :in-progress])))))]
|
||||
(ptk/emit! (the/prepare-store (test-state {:job-id job-id
|
||||
:in-progress true
|
||||
:status "running"})
|
||||
done' completed)
|
||||
(de/cancel-export)
|
||||
:the/end)))
|
||||
done))))
|
||||
|
||||
@ -152,9 +152,10 @@
|
||||
{:file-ids #{job-id} :export-type :detach-libraries})))
|
||||
(await (hva/->promise (rp/cmd! :create-import-binfile-job {:project-id (str job-id)})))
|
||||
(await (hva/->promise (rp/cmd! :get-job {:id (str job-id)})))
|
||||
(await (hva/->promise (rp/cmd! :create-export-job {})))
|
||||
(await (hva/->promise (rp/cmd! :create-export-assets-job
|
||||
{:params {:exports []}})))
|
||||
|
||||
(let [[export import read exporter] @requests]
|
||||
(let [[export import read assets] @requests]
|
||||
(t/testing "the binfile export asks the backend, not the exporter service"
|
||||
(t/is (= :post (:method export)))
|
||||
(t/is (str/ends-with? (:path (:uri export))
|
||||
@ -171,6 +172,7 @@
|
||||
"/api/main/methods/get-job"))
|
||||
(t/is (= (str job-id) (:id (:query read)))))
|
||||
|
||||
(t/testing "and the exporter keeps its own name"
|
||||
(t/is (= :post (:method exporter)))
|
||||
(t/is (str/ends-with? (:path (:uri exporter)) "/api/export/jobs"))))))))
|
||||
(t/testing "and the assets export joins the same door"
|
||||
(t/is (= :post (:method assets)))
|
||||
(t/is (str/ends-with? (:path (:uri assets))
|
||||
"/api/main/methods/create-export-assets-job"))))))))
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -7339,6 +7339,14 @@ msgstr "Exportación lenta"
|
||||
msgid "workspace.options.exporting-queued"
|
||||
msgstr "Esperando..."
|
||||
|
||||
#: src/app/main/ui/exports/assets.cljs:179
|
||||
msgid "workspace.options.exporting-packaging"
|
||||
msgstr "Empaquetando archivos…"
|
||||
|
||||
#: src/app/main/ui/exports/assets.cljs:177
|
||||
msgid "workspace.options.exporting-preparing"
|
||||
msgstr "Preparando..."
|
||||
|
||||
#: src/app/main/ui/workspace/sidebar/options/menus/fill.cljs:108, src/app/main/ui/workspace/sidebar/options/menus/fill.cljs:238
|
||||
msgid "workspace.options.fill"
|
||||
msgstr "Relleno"
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user