mirror of
https://github.com/penpot/penpot.git
synced 2026-10-31 23:10:31 +00:00
🎉 Give each render backend its own scope namespace
The wasm lease moves to exporter.wasm.scope (run over cfg plus an on-scope body, low-level render-on with asserted cfg), and the browser batch gains exporter.browser.scope with the mirrored run over cfg plus exports. The facade validates once, partitions by backend and joins; sys is now cfg everywhere, headless? is wasm? and the batch resolves nil. A misresolved key also stops masking a missing timeout as the 300s default. AI-assisted-by: muse-spark-1.3-contributor
This commit is contained in:
Andrey Antukh
parent
a2dd3caaf3
commit
b94174f1d1
@ -41,10 +41,12 @@ on this side: the row of the job, its cancel and its result are the backend's.
|
||||
- First ported service is `exporter.browser` (whole `app.browser` at once): the running instance is the raw pool, the public API takes it first (`exec` fails fast on a nil pool), the browser factory is injected through config for hermetic tests, and the new tree carries no promesa.
|
||||
- `exporter.main` mirrors the backend's `app.main`: a `system-config` def reading env at the wiring layer, a public `system` atom with the running map, `start`/`stop`/`restart` (`^:async`, resolving `:started`/`:stopped`/`:restarted`), `start-custom` as the test seam, idempotent `install-process-handlers` for the `uncaughtException`/SIGTERM/SIGINT wiring (runs on `start`, never on namespace load), `^:dev/before-load-async`/`^:dev/after-load-async` hooks bridging the promise lifecycle to shadow's `done` callbacks (a hook failure logs, never sticks the reload), and `main` as the entry point: shadow boots `exporter.main/main`, and the old `app.core` reload path (`stop` and the `:devtools` wiring) sits commented out until `app.*` is fully ported. The entry leaves info logging enabled for the new tree (`setup!` on the `exporter` logger, mirroring the old `{:app :info}`); without it the default level filters everything below fatal.
|
||||
- Service config maps take plain (non-namespaced) param keys (`{:exporter.browser/pool {:max 2}}`): the top-level key already scopes the component, so namespacing the params only adds noise. Component keys and error reasons stay namespaced.
|
||||
- `exporter.wasm.pool` mirrors the browser service for `worker_threads`: the running instance is `{:pool :timeout-ms}` (the pool plus the silence timeout its renders await), `with-worker`/`render-on`/`terminate` take the pool first, the worker factory is injected through config, and the creation handshake (`await-ready`) is its own unit with `:worker-not-ready`/`boot-error` paths.
|
||||
- `exporter.wasm.pool` follows the backend `db/run!` shape: `run [cfg f & args]` resolves the pool from `cfg` (the `:exporter.wasm.pool/pool` service instance, or a raw pool), acquires a worker, applies `f` to the cfg carrying it under `:exporter.wasm.pool/worker` plus any extra args, and returns the worker — or destroys it when `f` fails, like a rollback. Pure pool mechanics only (`init`/`halt`, `terminate`, `await-ready`); the render protocol lives one level up.
|
||||
- `exporter.wasm.scope` owns the lease: `run [sys on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the check is mandatory at the lease entry and runs here when configured.
|
||||
- `exporter.renderer.browser` merges the bitmap/pdf/svg flows into one `render [ctx params on-object]` (branching by type, `on-object` always awaited, unknown types raise); `ctx` carries what the job params do not freeze (`:browser-pool`, `:base-uri`, `:public-uri`, `:svgo?`). Dead xml predicates from the old tree did not survive the merge.
|
||||
- `exporter.renderer.wasm` drives headless renders off the wasm pool instance (`{:pool :timeout-ms}`): `with-scope [sys lease]` leases one worker and serializes its renders, where the lease map carries `on-scope` (fn of the leased render fn, firing every headless export through it), `check-cancelled` and `watch-interval` (tests only). Cancel is fully driver-owned: the injected zero-arg check (raising `:job-cancelled`) runs before each render, and a watchdog interval polls it mid-render — on cancel the leased worker aborts (per-lease SharedArrayBuffer flipped, thread terminated) and the render rejects. The buffer is born in the lease; nobody outside knows what it is. A nil check arms no interval and wires no cancel.
|
||||
- `exporter.renderer` is the malli-validated facade with one entry point: `render [sys & task]` takes a task map (`:exports :on-object :check-cancelled`) and fires the whole batch at once, the headless exports sharing one leased worker, the browser ones on their own checkouts. `sys` is task-free infrastructure with process lifetime (pools, uris, flags); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[sys params on-object check]` shape and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The scope fn survives one level down only, as the lease primitive inside `exporter.renderer.wasm`.
|
||||
- `exporter.browser.scope` owns the browser batch: `run [sys task]` fires every export through its own checkout, all at once, bounded by the pool. No lease — a browser batch shares nothing but the pool, so fan-out only. The facade validates once, partitions by backend and joins both scopes.
|
||||
- `exporter.renderer.wasm` is a thin single-shot over the scope lease. The scope fn survives one level down, as `exporter.wasm.scope/run`.
|
||||
- `exporter.renderer` is the malli-validated facade with one entry point: `render [cfg & task]` takes a task map (`:exports :on-object :check-cancelled`) and fires the whole batch at once — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). `cfg` is task-free infrastructure with process lifetime (pools, uris, flags); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`.
|
||||
- The runner stays backend-blind: plan → render batch → settle, with beats/watchdog/settle untouched. Its `on-object` calls the same injected check the drivers use. When `exporter.jobs` is ported, its only cancel surface toward renderers is one zero-arg check constructor per job; the old `cancel-fns`/terminate-registration apparatus does not survive (abort ownership moved to the lease holder).
|
||||
- `exporter.main` wires both pools (`:exporter.browser/pool` and `:exporter.wasm.pool/pool`, each sized to the worker concurrency). No test boots the production wiring: the wasm pool warms one worker eagerly and the worker script in a test process would be the test bundle itself; the wiring is asserted as data and the lifecycle rides fake configs.
|
||||
- Async test vars prove their chain only when really awaited: a test passing with 0 assertions means the chain broke upstream.
|
||||
|
||||
26
exporter/src/exporter/browser/scope.cljs
Normal file
26
exporter/src/exporter/browser/scope.cljs
Normal file
@ -0,0 +1,26 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter.browser.scope
|
||||
"One batch through browsers: every export gets its own checkout,
|
||||
all at once, bounded by the pool. No lease — a browser batch shares
|
||||
nothing but the pool, so this level owns fan-out only, the mirror of
|
||||
`exporter.wasm.scope` holding a lease instead. Task keys ride in
|
||||
`cfg` under `:exporter.renderer/*`, set by whoever built the batch."
|
||||
(:require
|
||||
[exporter.renderer.browser :as driver]))
|
||||
|
||||
(defn ^:async run
|
||||
"Renders every export through its own browser checkout. Rejects on
|
||||
the first failure; progress and cancel flow per export through the
|
||||
task's own callbacks."
|
||||
[cfg exports]
|
||||
(let [on-object (:exporter.renderer/on-object cfg)
|
||||
check-cancelled (:exporter.renderer/check-cancelled cfg)]
|
||||
(await (js/Promise.all
|
||||
(mapv (fn [params]
|
||||
(driver/render cfg params on-object check-cancelled))
|
||||
exports)))))
|
||||
@ -7,31 +7,32 @@
|
||||
(ns exporter.renderer
|
||||
"Common renderer interface, validated with malli from day one.
|
||||
|
||||
One entry point: `render` takes `sys` plus a task map
|
||||
(`:exports`, `:on-object`, `:check-cancelled`) and fires the whole
|
||||
batch at once, dispatching each export to the browser or the
|
||||
headless driver by its `:is-wasm` flag. A single export is a batch
|
||||
of one. The headless ones share one leased worker (serialized on
|
||||
it); the browser ones check out their own browser each (bounded by
|
||||
the pool). Rendered objects travel through `on-object`, so the
|
||||
batch itself resolves a vector of nils.
|
||||
One entry point: `render` takes `cfg` plus a task map
|
||||
(`:exports`, `:on-object`, `:check-cancelled`), validates it, and
|
||||
fires the whole batch at once — the browser exports through
|
||||
`exporter.browser.scope`, the wasm ones through one leased
|
||||
worker in `exporter.wasm.scope`. A single export is a batch of one.
|
||||
Rendered objects travel through `on-object`; the batch itself
|
||||
resolves nil once every export landed.
|
||||
|
||||
`sys` is task-free infrastructure with process lifetime — the
|
||||
browser pool and render uris, the wasm pool instance, flags — what
|
||||
could one day come straight from the running system map. The task
|
||||
`cfg` is task-free infrastructure with process lifetime: the pool
|
||||
services under their system keys (`:exporter.browser/pool`,
|
||||
`:exporter.wasm.pool/pool`) plus the static render config
|
||||
(`:base-uri`, `:public-uri`, `:svgo?`) — a view over the running
|
||||
system map that each domain reads with its own keys. The task
|
||||
map holds everything per-task: the `exports` data plus the two
|
||||
injected indirections, `on-object` (the per-object collector) and
|
||||
`check-cancelled` (the zero-arg cancel check raising
|
||||
`check-cancelled` (the zero-arg check-cancelled raising
|
||||
`:job-cancelled`). One closed schema validates the whole task, so a
|
||||
bad batch — or a non-fn injection — never touches a pool. Each
|
||||
driver takes the same `[sys params on-object check]` shape and
|
||||
driver takes the same `[cfg params on-object check-cancelled]` shape and
|
||||
implements cancel internally: cooperative checkpoints on the
|
||||
browser, checkpoints plus a mid-render abort of its own leased
|
||||
worker on wasm."
|
||||
(:require
|
||||
[app.common.schema :as sm]
|
||||
[exporter.renderer.browser :as browser]
|
||||
[exporter.renderer.wasm :as wasm]))
|
||||
[exporter.browser.scope :as bscope]
|
||||
[exporter.wasm.scope :as scope]))
|
||||
|
||||
(def schema:type
|
||||
[:enum :png :jpeg :webp :pdf :svg])
|
||||
@ -70,28 +71,29 @@
|
||||
(def ^:private check-task
|
||||
(sm/check-fn schema:task :hint "invalid render task"))
|
||||
|
||||
(defn headless?
|
||||
(defn wasm?
|
||||
"Whether `params` renders with render-wasm rather than a browser."
|
||||
[{:keys [is-wasm]}]
|
||||
(boolean is-wasm))
|
||||
|
||||
(defn ^:async render
|
||||
"Renders every export, firing them all at once. The whole task is
|
||||
validated before anything is leased, so a bad task never touches a
|
||||
pool. Rejects on the first failure."
|
||||
[sys & {:as task}]
|
||||
(let [{:keys [exports on-object check-cancelled] :as task}
|
||||
(check-task task)
|
||||
|
||||
do-render
|
||||
(fn [wasm-render]
|
||||
(js/Promise.all
|
||||
(mapv (fn [params]
|
||||
(if (and (headless? params) (fn? wasm-render))
|
||||
(wasm-render params on-object)
|
||||
(browser/render sys params on-object check-cancelled)))
|
||||
exports)))]
|
||||
|
||||
(if (some headless? exports)
|
||||
(await (wasm/with-scope sys (assoc task :on-scope do-render)))
|
||||
(await (do-render nil)))))
|
||||
"Renders every export, firing them all at once: the browser ones
|
||||
through their own scope, the wasm ones through one leased
|
||||
worker. The whole task is validated before anything is leased, so a
|
||||
bad task never touches a pool. Rejects on the first failure."
|
||||
[cfg & {:as task}]
|
||||
(let [{:keys [exports on-object check-cancelled]} (check-task task)
|
||||
cfg (assoc cfg
|
||||
::on-object on-object
|
||||
::check-cancelled check-cancelled)
|
||||
wasm-exports (filterv wasm? exports)
|
||||
browser-exports (remove wasm? exports)]
|
||||
(await (js/Promise.all
|
||||
(cond-> []
|
||||
(seq browser-exports)
|
||||
(conj (bscope/run cfg browser-exports))
|
||||
(seq wasm-exports)
|
||||
(conj (scope/run cfg
|
||||
(fn [render-fn]
|
||||
(js/Promise.all (mapv render-fn wasm-exports))))))))
|
||||
nil))
|
||||
|
||||
@ -12,9 +12,9 @@
|
||||
with each object carrying its `:path`. `on-object` may return a
|
||||
plain value or a promise; it is always awaited.
|
||||
|
||||
`sys` is task-free infrastructure: the `:browser-pool`, the
|
||||
`cfg` is task-free infrastructure: the `:exporter.browser/pool`, the
|
||||
render `:base-uri` and `:public-uri`, and the `:svgo?` flag. The
|
||||
per-task injections (`on-object`, `check`) ride positional. There
|
||||
per-task injections (`on-object`, `check-cancelled`) ride positional. There
|
||||
are no promesa chains in this namespace."
|
||||
(:require
|
||||
["@penpot/svgo" :as svgo]
|
||||
@ -34,14 +34,18 @@
|
||||
|
||||
(l/set-level! :trace)
|
||||
|
||||
(defn- check-cancelled
|
||||
"Cooperative checkpoint over the injected zero-arg check. This
|
||||
(defn- raise-if-cancelled
|
||||
"Cooperative checkpoint over the injected zero-arg check-cancelled. This
|
||||
backend never blocks indefinitely (every step yields), so
|
||||
yielding-point checks are the whole cancel story here; a nil check
|
||||
wires no cancel."
|
||||
[check]
|
||||
(when check
|
||||
(check)))
|
||||
yielding-point checks are the whole cancel story here. The check-cancelled is
|
||||
mandatory: without one it raises `:check-cancelled-missing` instead
|
||||
of running unprotected."
|
||||
[check-cancelled]
|
||||
(if (fn? check-cancelled)
|
||||
(check-cancelled)
|
||||
(throw (ex/error :type :assertion
|
||||
:code :check-cancelled-missing
|
||||
:hint "browser render needs a zero-arg check-cancelled"))))
|
||||
|
||||
(defn- prepare-options
|
||||
[uri token scale]
|
||||
@ -91,15 +95,15 @@
|
||||
nil)
|
||||
|
||||
(defn- ^:async render-bitmap
|
||||
[sys {:keys [file-id page-id share-id token scale type objects skip-children]} on-object]
|
||||
[cfg {:keys [file-id page-id share-id token scale type objects skip-children]} on-object]
|
||||
(let [query {:file-id file-id
|
||||
:page-id page-id
|
||||
:share-id share-id
|
||||
:object-id (mapv :id objects)
|
||||
:route "objects"
|
||||
:skip-children skip-children}
|
||||
uri (render-uri (:base-uri sys) query)]
|
||||
(await (browser/exec (:browser-pool sys)
|
||||
uri (render-uri (:base-uri cfg) query)]
|
||||
(await (browser/exec (:exporter.browser/pool cfg)
|
||||
(prepare-options uri token scale)
|
||||
(fn [page] (render-bitmap-page uri page type objects on-object))))))
|
||||
|
||||
@ -151,20 +155,20 @@
|
||||
(await (on-object (assoc object :path path))))))
|
||||
|
||||
(defn- ^:async render-pdf-page
|
||||
[base-uri page params objects on-object check]
|
||||
[base-uri page params objects on-object check-cancelled]
|
||||
(doseq [object objects]
|
||||
(check-cancelled check)
|
||||
(raise-if-cancelled check-cancelled)
|
||||
(await (render-pdf-object page base-uri params object on-object)))
|
||||
nil)
|
||||
|
||||
(defn- ^:async render-pdf
|
||||
[sys {:keys [token scale] :as params} on-object check]
|
||||
(let [base-uri (-> (:base-uri sys)
|
||||
[cfg {:keys [token scale] :as params} on-object check-cancelled]
|
||||
(let [base-uri (-> (:base-uri cfg)
|
||||
(u/ensure-path-slash))]
|
||||
(await (browser/exec (:browser-pool sys)
|
||||
(await (browser/exec (:exporter.browser/pool cfg)
|
||||
(prepare-options base-uri token scale)
|
||||
(fn [page]
|
||||
(render-pdf-page base-uri page params (:objects params) on-object check))))))
|
||||
(render-pdf-page base-uri page params (:objects params) on-object check-cancelled))))))
|
||||
|
||||
;; --- SVG
|
||||
|
||||
@ -440,7 +444,7 @@
|
||||
(str/replace content " " " "))
|
||||
|
||||
(defn- ^:async render-svg-object
|
||||
[sys page type object on-object]
|
||||
[cfg page type object on-object]
|
||||
(let [path (await (sh/tempfile :prefix "penpot.tmp.render.svg." :suffix (mime/get-extension type)))
|
||||
node (browser/select page (str/concat "#screenshot-" (:id object)))]
|
||||
(await (browser/wait-for node))
|
||||
@ -449,56 +453,56 @@
|
||||
result (replace-text-nodes xmldata txtdata)
|
||||
result (sanitize-svg-content result)
|
||||
|
||||
result (if (:svgo? sys)
|
||||
result (if (:svgo? cfg)
|
||||
(svgo/optimize result svgo/defaultOptions)
|
||||
result)
|
||||
|
||||
result (replace-internal-uris result (:base-uri sys) (:public-uri sys))]
|
||||
result (replace-internal-uris result (:base-uri cfg) (:public-uri cfg))]
|
||||
(await (sh/write-file path result))
|
||||
(await (on-object (assoc object :path path))))))
|
||||
|
||||
(defn- ^:async render-svg-page
|
||||
[sys uri page type objects on-object]
|
||||
[cfg uri page type objects on-object]
|
||||
(l/info :uri uri)
|
||||
;; navigate to the page and perform basic setup
|
||||
(await (browser/nav page (str uri)))
|
||||
(await (browser/sleep page 1000)) ; the good old fix with sleep
|
||||
(await (browser/wait-for-fonts page))
|
||||
;; take the screnshot of requested objects, one by one
|
||||
(await (js/Promise.all (mapv (fn [object] (render-svg-object sys page type object on-object))
|
||||
(await (js/Promise.all (mapv (fn [object] (render-svg-object cfg page type object on-object))
|
||||
objects)))
|
||||
nil)
|
||||
|
||||
(defn- ^:async render-svg
|
||||
[sys {:keys [file-id page-id share-id token scale type objects]} on-object]
|
||||
[cfg {:keys [file-id page-id share-id token scale type objects]} on-object]
|
||||
(let [query {:file-id file-id
|
||||
:page-id page-id
|
||||
:share-id share-id
|
||||
:render-embed true
|
||||
:object-id (mapv :id objects)
|
||||
:route "objects"}
|
||||
uri (render-uri (:base-uri sys) query)]
|
||||
(await (browser/exec (:browser-pool sys)
|
||||
uri (render-uri (:base-uri cfg) query)]
|
||||
(await (browser/exec (:exporter.browser/pool cfg)
|
||||
(prepare-options uri token scale)
|
||||
(fn [page] (render-svg-page sys uri page type objects on-object))))))
|
||||
(fn [page] (render-svg-page cfg uri page type objects on-object))))))
|
||||
|
||||
(defn ^:async render
|
||||
"Renders one partition of objects through a browser page: bitmap
|
||||
(png, jpeg, webp), pdf or svg depending on the type in `params`.
|
||||
Same shape as `exporter.renderer.wasm/render`: `sys` is task-free
|
||||
infrastructure (pools, uris, flags), while `on-object` and `check`
|
||||
Same shape as `exporter.renderer.wasm/render`: `cfg` is task-free
|
||||
infrastructure (pools, uris, flags), while `on-object` and `check-cancelled`
|
||||
ride positional as the per-task injections — the per-object
|
||||
collector and the zero-arg cancel check raising `:job-cancelled`.
|
||||
The check runs here at fail-fast entry plus the sequential pdf
|
||||
loop; the concurrent flows rely on the entry check and the caller's
|
||||
collector and the zero-arg check-cancelled raising `:job-cancelled`.
|
||||
The check-cancelled runs here at fail-fast entry plus the sequential pdf
|
||||
loop; the concurrent flows rely on the entry check-cancelled and the caller's
|
||||
`on-object`. Calls `on-object` with each object carrying its
|
||||
`:path`, awaiting it every time, and resolves nil."
|
||||
[sys {:keys [type] :as params} on-object check]
|
||||
(check-cancelled check)
|
||||
[cfg {:keys [type] :as params} on-object check-cancelled]
|
||||
(raise-if-cancelled check-cancelled)
|
||||
(case type
|
||||
(:png :jpeg :webp) (await (render-bitmap sys params on-object))
|
||||
:pdf (await (render-pdf sys params on-object check))
|
||||
:svg (await (render-svg sys params on-object))
|
||||
(:png :jpeg :webp) (await (render-bitmap cfg params on-object))
|
||||
:pdf (await (render-pdf cfg params on-object check-cancelled))
|
||||
:svg (await (render-svg cfg params on-object))
|
||||
(ex/raise :type :validation
|
||||
:code :unknown-render-type
|
||||
:hint (str "unknown render type: " type))))
|
||||
|
||||
@ -5,84 +5,18 @@
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter.renderer.wasm
|
||||
"Main-thread side of the headless renderer.
|
||||
|
||||
Renders run on pooled workers because Skia calls are synchronous and
|
||||
would block the event loop for every other export. Each batch keeps
|
||||
one worker for all its renders, sharing its caches and pool slot.
|
||||
|
||||
`sys` is task-free infrastructure (the wasm pool instance, the
|
||||
`{:pool :timeout-ms}` map the pool service owns — hence unwrapping
|
||||
one level to reach the raw pool). The lease map holds the rest:
|
||||
`on-scope`, a fn receiving the leased 2-arg render fn and firing
|
||||
every headless export through it, and `check-cancelled`, the
|
||||
zero-arg cancel check raising `:job-cancelled` once the job ended.
|
||||
Cancel is fully driver-owned: the check runs before each render,
|
||||
and a watchdog interval polls it mid-render — on cancel the leased
|
||||
worker aborts (shared signal flipped, thread terminated) and the
|
||||
render rejects. The buffer the worker polls is born in the lease
|
||||
itself; nobody outside knows what it is. A nil check arms no
|
||||
interval and wires no cancel."
|
||||
"Single-shot headless renders: builds the renderer task keys and
|
||||
delegates the lease to `exporter.wasm.scope`."
|
||||
(:require
|
||||
[app.common.logging :as l]
|
||||
[exporter.wasm.pool :as pool]))
|
||||
|
||||
(defn- serializer
|
||||
"Chains thunks so a job's renders run one at a time on its worker. A
|
||||
failure is isolated: it doesn't break the chain for the next one."
|
||||
[]
|
||||
(let [queue (atom (js/Promise.resolve nil))]
|
||||
(fn [thunk]
|
||||
(let [result (.then @queue (fn [_] (thunk)) (fn [_] (thunk)))]
|
||||
(reset! queue (.then result (fn [_] nil) (fn [_] nil)))
|
||||
result))))
|
||||
|
||||
(defn ^:async with-scope
|
||||
"Leases one worker for the batch: `on-scope` fires every headless
|
||||
export through the leased render fn, one at a time on the same
|
||||
worker, so the check runs as each render's turn comes up. The
|
||||
interval dies with the lease; a truly stuck thread is backstopped
|
||||
by the pool's own silence watchdog."
|
||||
[sys {:keys [check-cancelled on-scope watch-interval] :or {watch-interval 1000}}]
|
||||
(await
|
||||
(pool/with-worker (:pool (:wasm-pool sys))
|
||||
(fn [worker]
|
||||
((^:async fn []
|
||||
(let [chain (serializer)
|
||||
live (volatile! worker)
|
||||
signal (js/Int32Array. (js/SharedArrayBuffer. 4))
|
||||
timer (volatile! nil)
|
||||
opts {:cancel-buffer (.-buffer signal)
|
||||
:check-cancelled check-cancelled
|
||||
:timeout-ms (:timeout-ms (:wasm-pool sys))}]
|
||||
(vreset! timer
|
||||
(when check-cancelled
|
||||
(js/setInterval
|
||||
(fn []
|
||||
(try
|
||||
(check-cancelled)
|
||||
(catch :default _cause
|
||||
;; One-shot: the lease is doomed anyway.
|
||||
;; Between objects the worker sees the
|
||||
;; flag; inside a render only terminating
|
||||
;; the thread stops it.
|
||||
(some-> @timer js/clearInterval)
|
||||
(js/Atomics.store signal 0 1)
|
||||
(some-> @live pool/terminate))))
|
||||
watch-interval)))
|
||||
(try
|
||||
(await (on-scope (fn [params on-object]
|
||||
(chain (fn [] (pool/render-on worker params on-object opts))))))
|
||||
(finally
|
||||
(some-> @timer js/clearInterval)
|
||||
(vreset! live nil))))))))))
|
||||
[exporter.wasm.scope :as scope]))
|
||||
|
||||
(defn ^:async render
|
||||
"Same shape as `exporter.renderer.browser/render`: one export as a
|
||||
batch of one."
|
||||
[sys params on-object check]
|
||||
[cfg params on-object check-cancelled]
|
||||
(l/info :hint "render" :type (:type params) :backend "wasm")
|
||||
(await (with-scope sys {:exports [params]
|
||||
:on-object on-object
|
||||
:check-cancelled check
|
||||
:on-scope (fn [render-wasm] (render-wasm params on-object))})))
|
||||
(await (scope/run (assoc cfg
|
||||
:exporter.renderer/on-object on-object
|
||||
:exporter.renderer/check-cancelled check-cancelled)
|
||||
(fn [render-fn] (render-fn params)))))
|
||||
|
||||
@ -31,7 +31,6 @@
|
||||
[app.common.data :as d]
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.transit :as t]
|
||||
[exporter.utils.system :as system]))
|
||||
|
||||
(l/set-level! :info)
|
||||
@ -148,106 +147,40 @@
|
||||
(l/info :hint "finalizing render worker pool")
|
||||
(drain-pool pool)))
|
||||
|
||||
(defn ^:async with-worker
|
||||
"Acquires one worker for the whole of `f`, a fn of that worker. On
|
||||
failure the worker is destroyed instead of reused: the module may be
|
||||
aborted or mid-write, and a terminated worker cannot come back."
|
||||
[pool f]
|
||||
(let [worker (await (.acquire ^js pool))]
|
||||
(try
|
||||
(let [result (await (f worker))]
|
||||
(await (.release ^js pool worker))
|
||||
result)
|
||||
(catch :default cause
|
||||
(try
|
||||
(await (.destroy ^js pool worker))
|
||||
(catch :default cause'
|
||||
(l/warn :hint "render worker destroy failed"
|
||||
:cause cause')))
|
||||
(throw cause)))))
|
||||
(defn- pool?
|
||||
"A raw generic-pool: anything lendable through acquire and release."
|
||||
[x]
|
||||
(and (some? x)
|
||||
(fn? (unchecked-get x "acquire"))
|
||||
(fn? (unchecked-get x "release"))))
|
||||
|
||||
(defn- render-on-worker
|
||||
"Settles when the worker reports the render finished, failed, or the
|
||||
thread went away. That last case matters: a terminated worker (how a
|
||||
cancel stops a render mid-Skia) emits `exit` and never `error`, and a
|
||||
promise left pending there would keep its pool slot borrowed for the
|
||||
life of the process."
|
||||
[^js worker params cancel-buffer on-object timeout-ms]
|
||||
(js/Promise.
|
||||
(fn [resolve reject]
|
||||
(let [timer (volatile! nil)]
|
||||
(letfn [(disarm []
|
||||
(when-let [t @timer]
|
||||
(js/clearTimeout t)
|
||||
(vreset! timer nil)))
|
||||
|
||||
(rearm []
|
||||
(disarm)
|
||||
(vreset! timer (js/setTimeout
|
||||
(fn []
|
||||
(l/error :hint "render worker went silent, terminating"
|
||||
:worker-id (unchecked-get worker "__id"))
|
||||
(cleanup)
|
||||
;; Terminating is what frees the pool slot:
|
||||
;; the `exit` it raises has no listener left.
|
||||
(unchecked-set worker "__alive" false)
|
||||
(.terminate ^js worker)
|
||||
(reject (ex/error :type :internal
|
||||
:code :render-timeout
|
||||
:hint "render worker stopped responding")))
|
||||
timeout-ms)))
|
||||
|
||||
(cleanup []
|
||||
(disarm)
|
||||
(.off worker "message" on-message)
|
||||
(.off worker "error" on-error)
|
||||
(.off worker "exit" on-exit))
|
||||
|
||||
(on-error [cause]
|
||||
(cleanup)
|
||||
(reject cause))
|
||||
|
||||
(on-exit [code]
|
||||
(cleanup)
|
||||
(reject (ex/error :type :internal
|
||||
:code :worker-exited
|
||||
:hint (str "render worker exited with code " code))))
|
||||
|
||||
(on-message [data]
|
||||
(rearm)
|
||||
(case (unchecked-get data "type")
|
||||
;; A failure while the main thread handles the object
|
||||
;; (moving the file, appending to the zip) has to end
|
||||
;; the render too, or nothing ever settles this promise.
|
||||
"object" (try
|
||||
(on-object (t/decode-str (unchecked-get data "payload")))
|
||||
(catch :default cause
|
||||
(cleanup)
|
||||
(reject cause)))
|
||||
"done" (do (cleanup) (resolve nil))
|
||||
"error" (do (cleanup)
|
||||
(reject (ex/error :type :internal
|
||||
:code (or (some-> (unchecked-get data "code") keyword)
|
||||
:wasm-render-error)
|
||||
:hint (unchecked-get data "message"))))
|
||||
nil))]
|
||||
|
||||
(.on worker "message" on-message)
|
||||
(.once worker "error" on-error)
|
||||
(.once worker "exit" on-exit)
|
||||
(rearm)
|
||||
(.postMessage worker #js {:type "render"
|
||||
:params (t/encode-str params)
|
||||
:cancel cancel-buffer}))))))
|
||||
|
||||
(defn ^:async render-on
|
||||
"Renders `params` on an already acquired worker. The check runs
|
||||
before posting: a render whose turn comes up cancelled rejects
|
||||
without touching the worker."
|
||||
[worker params on-object {:keys [cancel-buffer check-cancelled timeout-ms]}]
|
||||
(when check-cancelled
|
||||
(check-cancelled))
|
||||
(await (render-on-worker worker params cancel-buffer on-object timeout-ms)))
|
||||
(defn ^:async run
|
||||
"Lends one worker: resolves the pool from `cfg` (the `::pool`
|
||||
service instance, or a raw pool), acquires a worker, applies `f` to
|
||||
the cfg carrying it under `::worker` plus any extra `args`, and
|
||||
returns the worker — or destroys it when `f` fails, the way a failed
|
||||
transaction rolls back instead of returning its connection: a failed
|
||||
worker may be aborted or mid-write, and a terminated worker cannot
|
||||
come back."
|
||||
[cfg f & args]
|
||||
(let [cfg (if (pool? cfg) {::pool {:pool cfg}} cfg)
|
||||
pool (:pool (::pool cfg))]
|
||||
(when-not (pool? pool)
|
||||
(throw (ex/error :type :assertion
|
||||
:code :invalid-pool-cfg
|
||||
:hint "no pool in cfg")))
|
||||
(let [worker (await (.acquire ^js pool))]
|
||||
(try
|
||||
(let [result (await (apply f (assoc cfg ::worker worker) args))]
|
||||
(await (.release ^js pool worker))
|
||||
result)
|
||||
(catch :default cause
|
||||
(try
|
||||
(await (.destroy ^js pool worker))
|
||||
(catch :default cause'
|
||||
(l/warn :hint "render worker destroy failed"
|
||||
:cause cause')))
|
||||
(throw cause))))))
|
||||
|
||||
(defn terminate
|
||||
[worker]
|
||||
|
||||
171
exporter/src/exporter/wasm/scope.cljs
Normal file
171
exporter/src/exporter/wasm/scope.cljs
Normal file
@ -0,0 +1,171 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter.wasm.scope
|
||||
"One leased render worker for a whole batch.
|
||||
|
||||
The pool owns lending; this level owns what a lease means: every
|
||||
headless export runs on the same worker, one at a time, sharing its
|
||||
caches and pool slot, with the cancel abort armed for the life of
|
||||
the lease. Task keys arrive in `cfg` under `:exporter.renderer/*`
|
||||
(set by whoever built the batch); the pool instance under the
|
||||
pool's own key."
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.transit :as t]
|
||||
[exporter.wasm.pool :as pool]))
|
||||
|
||||
(defn- serializer
|
||||
"Chains thunks so a job's renders run one at a time on its worker. A
|
||||
failure is isolated: it doesn't break the chain for the next one."
|
||||
[]
|
||||
(let [queue (atom (js/Promise.resolve nil))]
|
||||
(fn [thunk]
|
||||
(let [result (.then @queue (fn [_] (thunk)) (fn [_] (thunk)))]
|
||||
(reset! queue (.then result (fn [_] nil) (fn [_] nil)))
|
||||
result))))
|
||||
|
||||
(defn- render-on-worker-
|
||||
"Settles when the worker reports the render finished, failed, or the
|
||||
thread went away. That last case matters: a terminated worker (how a
|
||||
cancel stops a render mid-Skia) emits `exit` and never `error`, and a
|
||||
promise left pending there would keep its pool slot borrowed for the
|
||||
life of the process."
|
||||
[^js worker params cancel-buffer on-object timeout-ms]
|
||||
(js/Promise.
|
||||
(fn [resolve reject]
|
||||
(let [timer (volatile! nil)]
|
||||
(letfn [(disarm []
|
||||
(when-let [t @timer]
|
||||
(js/clearTimeout t)
|
||||
(vreset! timer nil)))
|
||||
|
||||
(rearm []
|
||||
(disarm)
|
||||
(vreset! timer (js/setTimeout
|
||||
(fn []
|
||||
(l/error :hint "render worker went silent, terminating"
|
||||
:worker-id (unchecked-get worker "__id"))
|
||||
(cleanup)
|
||||
;; Terminating is what frees the pool slot:
|
||||
;; the `exit` it raises has no listener left.
|
||||
(unchecked-set worker "__alive" false)
|
||||
(.terminate ^js worker)
|
||||
(reject (ex/error :type :internal
|
||||
:code :render-timeout
|
||||
:hint "render worker stopped responding")))
|
||||
timeout-ms)))
|
||||
|
||||
(cleanup []
|
||||
(disarm)
|
||||
(.off worker "message" on-message)
|
||||
(.off worker "error" on-error)
|
||||
(.off worker "exit" on-exit))
|
||||
|
||||
(on-error [cause]
|
||||
(cleanup)
|
||||
(reject cause))
|
||||
|
||||
(on-exit [code]
|
||||
(cleanup)
|
||||
(reject (ex/error :type :internal
|
||||
:code :worker-exited
|
||||
:hint (str "render worker exited with code " code))))
|
||||
|
||||
(on-message [data]
|
||||
(rearm)
|
||||
(case (unchecked-get data "type")
|
||||
;; A failure while the main thread handles the object
|
||||
;; (moving the file, appending to the zip) has to end
|
||||
;; the render too, or nothing ever settles this promise.
|
||||
"object" (try
|
||||
(on-object (t/decode-str (unchecked-get data "payload")))
|
||||
(catch :default cause
|
||||
(cleanup)
|
||||
(reject cause)))
|
||||
"done" (do (cleanup) (resolve nil))
|
||||
"error" (do (cleanup)
|
||||
(reject (ex/error :type :internal
|
||||
:code (or (some-> (unchecked-get data "code") keyword)
|
||||
:wasm-render-error)
|
||||
:hint (unchecked-get data "message"))))
|
||||
nil))]
|
||||
|
||||
(.on worker "message" on-message)
|
||||
(.once worker "error" on-error)
|
||||
(.once worker "exit" on-exit)
|
||||
(rearm)
|
||||
(.postMessage worker #js {:type "render"
|
||||
:params (t/encode-str params)
|
||||
:cancel cancel-buffer}))))))
|
||||
|
||||
(defn ^:async render-on
|
||||
"Low-level render on the worker leased in `cfg`. A missing worker
|
||||
or timeout raises instead of failing obscurely (`:worker-not-leased`,
|
||||
`:timeout-ms-missing`): both are programmer errors, and a missing
|
||||
timeout must never degrade silently into the default silence budget."
|
||||
[cfg params]
|
||||
(let [worker (::pool/worker cfg)
|
||||
timeout-ms (or (:timeout-ms (:exporter.wasm.pool/pool cfg))
|
||||
(throw (ex/error :type :assertion
|
||||
:code :timeout-ms-missing
|
||||
:hint "render-on needs :timeout-ms in cfg")))]
|
||||
(when-not worker
|
||||
(throw (ex/error :type :assertion
|
||||
:code :worker-not-leased
|
||||
:hint "render-on needs a leased worker in cfg")))
|
||||
(when-let [check-cancelled (:exporter.renderer/check-cancelled cfg)]
|
||||
(check-cancelled))
|
||||
(await (render-on-worker- worker
|
||||
params
|
||||
(::cancel-buffer cfg)
|
||||
(:exporter.renderer/on-object cfg)
|
||||
timeout-ms))))
|
||||
|
||||
(defn ^:async run
|
||||
"Leases one worker for `on-scope`, a fn of a single-arg render fn.
|
||||
Every headless export fires through it, one at a time on the same
|
||||
worker, so the check-cancelled runs as each render's turn comes up. The check-cancelled
|
||||
is mandatory: without one the lease raises `:check-cancelled-
|
||||
missing` before acquiring anything. The per lease signal buffer is
|
||||
born here; the interval dies with the lease. A truly stuck thread
|
||||
is backstopped by the pool's own silence watchdog."
|
||||
[cfg on-scope]
|
||||
(let [check-cancelled (:exporter.renderer/check-cancelled cfg)]
|
||||
(when-not (fn? check-cancelled)
|
||||
(throw (ex/error :type :assertion
|
||||
:code :check-cancelled-missing
|
||||
:hint "scope/run needs a zero-arg check-cancelled in cfg")))
|
||||
(await
|
||||
(pool/run cfg
|
||||
(^:async fn [cfg]
|
||||
(let [worker (::pool/worker cfg)
|
||||
chain (serializer)
|
||||
live (volatile! worker)
|
||||
signal (js/Int32Array. (js/SharedArrayBuffer. 4))
|
||||
timer (volatile! nil)
|
||||
render-cfg (assoc cfg
|
||||
::cancel-buffer (.-buffer signal))]
|
||||
(vreset! timer
|
||||
(js/setInterval
|
||||
(fn []
|
||||
(try
|
||||
(check-cancelled)
|
||||
(catch :default _cause
|
||||
;; One-shot: the lease is doomed anyway.
|
||||
;; Between objects the worker sees the
|
||||
;; flag; inside a render only terminating
|
||||
;; the thread stops it.
|
||||
(some-> @timer js/clearInterval)
|
||||
(js/Atomics.store signal 0 1)
|
||||
(some-> @live pool/terminate))))
|
||||
(::watch-interval cfg 1000)))
|
||||
(try
|
||||
(await (on-scope (fn [params] (chain (fn [] (render-on render-cfg params))))))
|
||||
(finally
|
||||
(some-> @timer js/clearInterval)
|
||||
(vreset! live nil)))))))))
|
||||
95
exporter/test/exporter_tests/browser_scope_test.cljs
Normal file
95
exporter/test/exporter_tests/browser_scope_test.cljs
Normal file
@ -0,0 +1,95 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.browser-scope-test
|
||||
"The browser batch: every export through its own checkout, all at
|
||||
once, against a stub pool with stub pages."
|
||||
(:require
|
||||
["generic-pool" :as gp]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[cljs.test :as t :include-macros true]
|
||||
[exporter.browser.scope :as scope]))
|
||||
|
||||
(defn- stub-locator
|
||||
[]
|
||||
#js {:waitFor (fn [_opts] (js/Promise.resolve nil))
|
||||
:screenshot (fn [_opts] (js/Promise.resolve nil))})
|
||||
|
||||
(defn- stub-page
|
||||
[calls]
|
||||
#js {:goto (fn [url _opts]
|
||||
(swap! calls conj [:goto (str url)])
|
||||
(js/Promise.resolve nil))
|
||||
:waitForTimeout (fn [ms] (swap! calls conj [:sleep ms]) (js/Promise.resolve nil))
|
||||
:waitForFunction (fn [_src _arg _opts] (js/Promise.resolve nil))
|
||||
:locator (fn [sel] (swap! calls conj [:select sel]) (stub-locator))
|
||||
:$$ (fn [sel]
|
||||
(swap! calls conj [:select-all sel])
|
||||
(js/Promise.resolve #js []))
|
||||
:evaluate (fn [_f & _args] (js/Promise.resolve nil))})
|
||||
|
||||
(defn- stub-context
|
||||
[page]
|
||||
#js {:newPage (fn [] (js/Promise.resolve page))
|
||||
:close (fn [] (js/Promise.resolve nil))})
|
||||
|
||||
(defn- stub-browser
|
||||
[page]
|
||||
#js {:newContext (fn [_opts] (js/Promise.resolve (stub-context page)))
|
||||
:isConnected (fn [] true)
|
||||
:close (fn [] (js/Promise.resolve nil))})
|
||||
|
||||
(defn- stub-pool
|
||||
[calls]
|
||||
(let [page (stub-page calls)]
|
||||
(gp/createPool #js {:create (fn [] (js/Promise.resolve (stub-browser page)))
|
||||
:destroy (fn [browser] (.close ^js browser))
|
||||
:validate (fn [_] true)}
|
||||
#js {:max 2 :min 0})))
|
||||
|
||||
(defn- test-cfg
|
||||
[calls]
|
||||
{:exporter.browser/pool (stub-pool calls)
|
||||
:base-uri (cf/get-internal-uri)
|
||||
:public-uri (cf/get :public-uri)
|
||||
:svgo? false})
|
||||
|
||||
(defn- test-params
|
||||
[id]
|
||||
{:file-id (uuid/next)
|
||||
:page-id (uuid/next)
|
||||
:token "token"
|
||||
:scale 1
|
||||
:type :png
|
||||
:objects [{:id id :name id :suffix ".png" :filename (str id ".png")}]})
|
||||
|
||||
(t/deftest ^:async run-fires-every-export-through-its-own-checkout
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
seen (atom [])]
|
||||
(await (scope/run (assoc (test-cfg calls)
|
||||
:exporter.renderer/on-object (fn [object] (swap! seen conj object) nil)
|
||||
:exporter.renderer/check-cancelled (fn [] nil))
|
||||
[(test-params "a") (test-params "b")]))
|
||||
(t/is (= ["a" "b"] (mapv :id @seen)))
|
||||
(t/is (= 2 (count (filter #(and (vector? %) (= :goto (first %))) @calls)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async run-rejects-on-the-first-failure
|
||||
(try
|
||||
(let [calls (atom [])]
|
||||
(try
|
||||
(await (scope/run (assoc (test-cfg calls)
|
||||
:exporter.renderer/on-object (fn [_] (throw (ex-info "on-object broke" {})))
|
||||
:exporter.renderer/check-cancelled (fn [] nil))
|
||||
[(test-params "a")]))
|
||||
(t/is false "run should have rejected a failing on-object")
|
||||
(catch :default cause
|
||||
(t/is (= "on-object broke" (ex-message cause))))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
@ -62,12 +62,12 @@
|
||||
:validate (fn [_] true)}
|
||||
#js {:max 1 :min 0}))
|
||||
|
||||
(defn- test-sys
|
||||
(defn- test-cfg
|
||||
[pool]
|
||||
{:browser-pool pool
|
||||
:base-uri (cf/get-internal-uri)
|
||||
:public-uri (cf/get :public-uri)
|
||||
:svgo? false})
|
||||
{:exporter.browser/pool pool
|
||||
:base-uri (cf/get-internal-uri)
|
||||
:public-uri (cf/get :public-uri)
|
||||
:svgo? false})
|
||||
|
||||
(defn- never-cancelled
|
||||
[]
|
||||
@ -77,7 +77,7 @@
|
||||
[calls raise-on-call]
|
||||
(let [seen (atom 0)]
|
||||
(fn []
|
||||
(swap! calls conj :check)
|
||||
(swap! calls conj :check-cancelled)
|
||||
(when (= raise-on-call (swap! seen inc))
|
||||
(throw (ex-info "export job was cancelled"
|
||||
{:type :internal
|
||||
@ -109,7 +109,7 @@
|
||||
page (stub-page calls (stub-locator calls (fn [_] nil)))
|
||||
pool (test-pool calls page)
|
||||
{:keys [seen fn]} (collect-on-object)]
|
||||
(await (render/render (test-sys pool) (test-params :png ["a" "b"]) fn (never-cancelled)))
|
||||
(await (render/render (test-cfg pool) (test-params :png ["a" "b"]) fn (never-cancelled)))
|
||||
(t/is (= 2 (count @seen)))
|
||||
(t/is (every? #(str/ends-with? (:path %) ".png") @seen))
|
||||
(t/is (= ["a" "b"] (mapv :id @seen)))
|
||||
@ -128,7 +128,7 @@
|
||||
page (stub-page calls (stub-locator calls (fn [_] nil)))
|
||||
pool (test-pool calls page)
|
||||
{:keys [seen fn]} (collect-on-object)]
|
||||
(await (render/render (test-sys pool) (test-params :pdf ["a" "b"]) fn (never-cancelled)))
|
||||
(await (render/render (test-cfg pool) (test-params :pdf ["a" "b"]) fn (never-cancelled)))
|
||||
(t/is (= 2 (count @seen)))
|
||||
(t/is (every? #(str/ends-with? (:path %) ".pdf") @seen))
|
||||
(t/testing "one navigation per object"
|
||||
@ -142,9 +142,9 @@
|
||||
(let [calls (atom [])
|
||||
page (stub-page calls (stub-locator calls (fn [_] nil)))
|
||||
pool (test-pool calls page)
|
||||
sys (test-sys pool)]
|
||||
cfg (test-cfg pool)]
|
||||
(try
|
||||
(await (render/render sys (test-params :png ["a"])
|
||||
(await (render/render cfg (test-params :png ["a"])
|
||||
(fn [_] (swap! calls conj :object))
|
||||
(raising-check calls 1)))
|
||||
(t/is false "render should have rejected a cancelled render")
|
||||
@ -162,10 +162,10 @@
|
||||
pool (test-pool calls page)
|
||||
;; entry check is the first call, then one per object:
|
||||
;; the cancel lands as the second object comes up
|
||||
sys (test-sys pool)
|
||||
cfg (test-cfg pool)
|
||||
{:keys [seen fn]} (collect-on-object)]
|
||||
(try
|
||||
(await (render/render sys (test-params :pdf ["a" "b"]) fn (raising-check calls 3)))
|
||||
(await (render/render cfg (test-params :pdf ["a" "b"]) fn (raising-check calls 3)))
|
||||
(t/is false "render should have rejected a cancelled batch")
|
||||
(catch :default cause
|
||||
(t/is (= :job-cancelled (-> cause ex-data :code)))
|
||||
@ -182,7 +182,7 @@
|
||||
page (stub-page calls locator)
|
||||
pool (test-pool calls page)
|
||||
{:keys [seen fn]} (collect-on-object)
|
||||
_ (await (render/render (test-sys pool) (test-params :svg ["a"]) fn (never-cancelled)))
|
||||
_ (await (render/render (test-cfg pool) (test-params :svg ["a"]) fn (never-cancelled)))
|
||||
path (:path (first @seen))
|
||||
content (await (sh/read-file path))]
|
||||
(t/is (= 1 (count @seen)))
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
|
||||
(ns exporter-tests.renderer-test
|
||||
"The renderer facade: malli validation and dispatch to the browser
|
||||
or headless drivers, against a stub browser pool and a stub wasm
|
||||
or wasm drivers, against a stub browser pool and a stub wasm
|
||||
pool."
|
||||
(:require
|
||||
["generic-pool" :as gp]
|
||||
@ -87,16 +87,16 @@
|
||||
(defn- stub-check
|
||||
[calls]
|
||||
(fn []
|
||||
(swap! calls conj :check)))
|
||||
(swap! calls conj :check-cancelled)))
|
||||
|
||||
(defn- test-sys
|
||||
(defn- test-cfg
|
||||
[calls]
|
||||
(let [worker (stub-worker calls)]
|
||||
{:browser-pool (stub-browser-pool calls)
|
||||
:base-uri (cf/get-internal-uri)
|
||||
:public-uri (cf/get :public-uri)
|
||||
:svgo? false
|
||||
:wasm-pool {:pool (stub-wasm-pool calls worker) :timeout-ms 5000}}))
|
||||
{:exporter.browser/pool (stub-browser-pool calls)
|
||||
:base-uri (cf/get-internal-uri)
|
||||
:public-uri (cf/get :public-uri)
|
||||
:svgo? false
|
||||
:exporter.wasm.pool/pool {:pool (stub-wasm-pool calls worker) :timeout-ms 5000}}))
|
||||
|
||||
(defn- never-cancelled
|
||||
[calls]
|
||||
@ -112,16 +112,16 @@
|
||||
:objects [{:id "a" :name "a" :suffix ".png" :filename "a.png"}]}
|
||||
opts))
|
||||
|
||||
(t/deftest headless-reads-the-wasm-flag
|
||||
(t/is (true? (renderer/headless? {:is-wasm true :type :png})))
|
||||
(t/is (false? (renderer/headless? {:type :png})))
|
||||
(t/is (false? (renderer/headless? {:is-wasm false :type :png}))))
|
||||
(t/deftest wasm-reads-the-wasm-flag
|
||||
(t/is (true? (renderer/wasm? {:is-wasm true :type :png})))
|
||||
(t/is (false? (renderer/wasm? {:type :png})))
|
||||
(t/is (false? (renderer/wasm? {:is-wasm false :type :png}))))
|
||||
|
||||
(t/deftest ^:async render-routes-bitmaps-to-the-browser
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
seen (atom [])]
|
||||
(await (renderer/render (test-sys calls)
|
||||
(await (renderer/render (test-cfg calls)
|
||||
:exports [(test-params :png nil)]
|
||||
:on-object (fn [object] (swap! seen conj object) nil)
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
@ -131,18 +131,18 @@
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-routes-headless-to-the-worker
|
||||
(t/deftest ^:async render-routes-wasm-to-the-worker
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
job-id (uuid/next)
|
||||
seen (atom [])]
|
||||
(await (renderer/render (test-sys calls)
|
||||
(await (renderer/render (test-cfg calls)
|
||||
:exports [(test-params :png {:is-wasm true :job-id job-id})]
|
||||
:on-object (fn [object] (swap! seen conj object) nil)
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
(t/is (= [{:id "w1" :path "/tmp/w1.png"}] @seen))
|
||||
(t/is (some #{[:post "render"]} @calls))
|
||||
(t/is (some #{:check} @calls))
|
||||
(t/is (some #{:check-cancelled} @calls))
|
||||
(t/is (not-any? #(and (vector? %) (= :goto (first %))) @calls)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
@ -151,7 +151,7 @@
|
||||
(try
|
||||
(let [calls (atom [])]
|
||||
(try
|
||||
(await (renderer/render (test-sys calls)
|
||||
(await (renderer/render (test-cfg calls)
|
||||
:exports [(dissoc (test-params :png nil) :objects)]
|
||||
:on-object (fn [_] nil)
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
@ -165,7 +165,7 @@
|
||||
(try
|
||||
(let [calls (atom [])]
|
||||
(try
|
||||
(await (renderer/render (test-sys calls)
|
||||
(await (renderer/render (test-cfg calls)
|
||||
:exports [(test-params :png nil)]
|
||||
:on-object :not-a-fn
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
@ -175,30 +175,30 @@
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-shares-one-worker-across-headless-renders
|
||||
(t/deftest ^:async render-shares-one-worker-across-wasm-renders
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
job-id (uuid/next)
|
||||
sys (test-sys calls)
|
||||
cfg (test-cfg calls)
|
||||
seen (atom [])]
|
||||
(await (renderer/render sys
|
||||
:exports [(test-params :png {:is-wasm true :job-id job-id})
|
||||
(test-params :png {:is-wasm true :job-id job-id})
|
||||
(test-params :png {:job-id job-id})]
|
||||
:on-object (fn [object] (swap! seen conj object) nil)
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
(t/is (nil? (await (renderer/render cfg
|
||||
:exports [(test-params :png {:is-wasm true :job-id job-id})
|
||||
(test-params :png {:is-wasm true :job-id job-id})
|
||||
(test-params :png {:job-id job-id})]
|
||||
:on-object (fn [object] (swap! seen conj object) nil)
|
||||
:check-cancelled (never-cancelled calls)))))
|
||||
(t/is (= 1 (count (filter #{:acquire} @calls))))
|
||||
(t/is (= 1 (count (filter #{:release} @calls))))
|
||||
(t/is (= 3 (count @seen))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-without-headless-renders-directly
|
||||
(t/deftest ^:async render-without-wasm-renders-directly
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
sys (test-sys calls)
|
||||
cfg (test-cfg calls)
|
||||
seen (atom [])]
|
||||
(await (renderer/render sys
|
||||
(await (renderer/render cfg
|
||||
:exports [(test-params :png nil)]
|
||||
:on-object (fn [object] (swap! seen conj object) nil)
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
@ -211,9 +211,9 @@
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
job-id (uuid/next)
|
||||
sys (test-sys calls)]
|
||||
cfg (test-cfg calls)]
|
||||
(try
|
||||
(await (renderer/render sys
|
||||
(await (renderer/render cfg
|
||||
:exports [(test-params :png {:is-wasm true :job-id job-id})
|
||||
(dissoc (test-params :png nil) :objects)]
|
||||
:on-object (fn [_] nil)
|
||||
@ -230,7 +230,7 @@
|
||||
(try
|
||||
(let [calls (atom [])]
|
||||
(try
|
||||
(await (renderer/render (test-sys calls)
|
||||
(await (renderer/render (test-cfg calls)
|
||||
:exports [(test-params :png nil)]
|
||||
:on-objec (fn [_] nil)
|
||||
:check-cancelled (never-cancelled calls)))
|
||||
|
||||
@ -5,9 +5,8 @@
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.renderer-wasm-test
|
||||
"The wasm renderer driver: leased workers, sequential renders,
|
||||
failure isolation and driver-owned cancel, against a stub pool,
|
||||
stub workers and an injected zero-arg check."
|
||||
"The wasm renderer driver: a single shot over the scope lease,
|
||||
against a stub pool, stub workers and a positional zero-arg check."
|
||||
(:require
|
||||
["node:events" :as events]
|
||||
[app.common.transit :as transit]
|
||||
@ -46,92 +45,28 @@
|
||||
(defn- stub-check
|
||||
[calls cancelled?]
|
||||
(fn []
|
||||
(swap! calls conj :check)
|
||||
(swap! calls conj :check-cancelled)
|
||||
(when @cancelled?
|
||||
(throw (ex-info "export job was cancelled"
|
||||
{:type :internal
|
||||
:code :job-cancelled
|
||||
:hint "export job was cancelled"})))))
|
||||
|
||||
(defn- test-sys
|
||||
(defn- test-cfg
|
||||
[calls worker]
|
||||
{:wasm-pool {:pool (stub-pool calls worker) :timeout-ms 5000}})
|
||||
{:exporter.wasm.pool/pool {:pool (stub-pool calls worker) :timeout-ms 5000}})
|
||||
|
||||
(defn- test-params
|
||||
[]
|
||||
{:type :png
|
||||
:objects [{:id "o1"}]})
|
||||
|
||||
(defn- posted?
|
||||
[calls]
|
||||
(some #(and (vector? %) (= :post (first %))) calls))
|
||||
|
||||
(t/deftest ^:async with-scope-runs-renders-on-the-leased-worker
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1" :path "/tmp/o1.png"}))
|
||||
seen (atom [])]
|
||||
(await (render/with-scope (test-sys calls worker)
|
||||
{:check-cancelled (stub-check calls (atom false))
|
||||
:on-scope (^:async fn [render*]
|
||||
(await (render* (test-params)
|
||||
(fn [object] (swap! seen conj object) nil)))
|
||||
(await (render* (test-params)
|
||||
(fn [object] (swap! seen conj object) nil))))}))
|
||||
(t/is (= [{:id "o1" :path "/tmp/o1.png"}
|
||||
{:id "o1" :path "/tmp/o1.png"}]
|
||||
@seen))
|
||||
(t/is (= 1 (count (filter #{:release} @calls))))
|
||||
(t/testing "the check runs as each render's turn comes up"
|
||||
(t/is (= 2 (count (filter #{:check} @calls)))))
|
||||
(t/testing "a quiet lease leaves its worker alone"
|
||||
(t/is (not-any? #{:terminate} @calls))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async cancelled-render-rejects-before-posting
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))]
|
||||
(try
|
||||
(await (render/with-scope (test-sys calls worker)
|
||||
{:check-cancelled (stub-check calls (atom true))
|
||||
:on-scope (fn [render*]
|
||||
(render* (test-params) (fn [_] nil)))}))
|
||||
(t/is false "with-scope should have rejected a cancelled render")
|
||||
(catch :default cause
|
||||
(t/is (= :job-cancelled (-> cause ex-data :code)))
|
||||
(t/is (not (posted? @calls))))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async cancel-aborts-the-leased-worker-mid-render
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (fn [_ _]))
|
||||
cancelled (atom false)]
|
||||
(try
|
||||
(await (render/with-scope (test-sys calls worker)
|
||||
{:check-cancelled (stub-check calls cancelled)
|
||||
:watch-interval 20
|
||||
:on-scope (fn [render*]
|
||||
;; the cancel lands mid-render, once
|
||||
;; the worker is listening
|
||||
(js/setTimeout #(reset! cancelled true) 10)
|
||||
(render* (test-params) (fn [_] nil)))}))
|
||||
(t/is false "with-scope should have rejected an aborted render")
|
||||
(catch :default cause
|
||||
(t/is (= :worker-exited (-> cause ex-data :code)))
|
||||
(t/is (some #{:terminate} @calls)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-leases-and-releases-for-one-render
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))
|
||||
seen (atom [])]
|
||||
(await (render/render (test-sys calls worker)
|
||||
(await (render/render (test-cfg calls worker)
|
||||
(test-params)
|
||||
(fn [object] (swap! seen conj object) nil)
|
||||
(stub-check calls (atom false))))
|
||||
@ -140,36 +75,19 @@
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-without-check-wires-no-cancel
|
||||
(t/deftest ^:async render-without-check-rejects-before-leasing
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))
|
||||
seen (atom [])]
|
||||
(await (render/render (test-sys calls worker)
|
||||
(test-params)
|
||||
(fn [object] (swap! seen conj object) nil)
|
||||
nil))
|
||||
(t/is (= [{:id "o1"}] @seen))
|
||||
(t/is (not-any? #{:terminate} @calls)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async failed-render-does-not-break-the-chain
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))
|
||||
seen (atom [])]
|
||||
(await (render/with-scope (test-sys calls worker)
|
||||
{:check-cancelled (stub-check calls (atom false))
|
||||
:on-scope (^:async fn [render*]
|
||||
(try
|
||||
(await (render* (test-params)
|
||||
(fn [_] (throw (ex-info "on-object broke" {})))))
|
||||
(catch :default _cause
|
||||
(swap! calls conj :first-failed)))
|
||||
(await (render* (test-params)
|
||||
(fn [object] (swap! seen conj object) nil))))}))
|
||||
(t/is (some #{:first-failed} @calls))
|
||||
(t/is (= [{:id "o1"}] @seen)))
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))]
|
||||
(try
|
||||
(await (render/render (test-cfg calls worker)
|
||||
(test-params)
|
||||
(fn [_] nil)
|
||||
nil))
|
||||
(t/is false "render should have rejected a missing check")
|
||||
(catch :default cause
|
||||
(t/is (= :assertion (-> cause ex-data :type)))
|
||||
(t/is (= :check-cancelled-missing (-> cause ex-data :code)))
|
||||
(t/is (not-any? #{:acquire} @calls)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
@ -11,6 +11,7 @@
|
||||
[clojure.string :as str]
|
||||
[clojure.tools.cli :refer [parse-opts]]
|
||||
[exporter-tests.api-test]
|
||||
[exporter-tests.browser-scope-test]
|
||||
[exporter-tests.browser-test]
|
||||
[exporter-tests.consumer-config-test]
|
||||
[exporter-tests.consumer-plan-test]
|
||||
@ -24,6 +25,7 @@
|
||||
[exporter-tests.system-test]
|
||||
[exporter-tests.wasm-pool-service-test]
|
||||
[exporter-tests.wasm-pool-test]
|
||||
[exporter-tests.wasm-scope-test]
|
||||
[exporter-tests.worker-test]
|
||||
[goog.object :as gobj]))
|
||||
|
||||
@ -37,7 +39,8 @@
|
||||
(println (str "UNHANDLED-REJECTION: " reason))))
|
||||
|
||||
(def test-namespaces
|
||||
['exporter-tests.browser-test
|
||||
['exporter-tests.browser-scope-test
|
||||
'exporter-tests.browser-test
|
||||
'exporter-tests.consumer-plan-test
|
||||
'exporter-tests.jobs-test
|
||||
'exporter-tests.main-test
|
||||
@ -51,7 +54,8 @@
|
||||
'exporter-tests.shell-test
|
||||
'exporter-tests.system-test
|
||||
'exporter-tests.wasm-pool-service-test
|
||||
'exporter-tests.wasm-pool-test])
|
||||
'exporter-tests.wasm-pool-test
|
||||
'exporter-tests.wasm-scope-test])
|
||||
|
||||
(assert (every? find-ns-obj test-namespaces)
|
||||
"test-namespaces contains a namespace that isn't required in runner.cljs")
|
||||
|
||||
@ -6,11 +6,10 @@
|
||||
|
||||
(ns exporter-tests.wasm-pool-service-test
|
||||
"The wasm worker pool service: lifecycle through the system, the
|
||||
lend protocol, the render protocol, and the creation handshake,
|
||||
against fake workers."
|
||||
lend protocol analogy to a transaction (return on success, destroy
|
||||
on failure), and the creation handshake, against fake workers."
|
||||
(:require
|
||||
["node:events" :as events]
|
||||
[app.common.transit :as transit]
|
||||
[cljs.test :as t :include-macros true]
|
||||
[exporter.utils.system :as system]
|
||||
[exporter.wasm.pool :as pool]))
|
||||
@ -34,15 +33,6 @@
|
||||
(swap! calls conj :create)
|
||||
(js/Promise.resolve (fake-worker calls behavior))))
|
||||
|
||||
(defn- render-behavior
|
||||
[payload]
|
||||
(fn [^js emitter _msg]
|
||||
(js/setTimeout
|
||||
(fn []
|
||||
(.emit emitter "message" #js {:type "object" :payload (transit/encode-str payload)})
|
||||
(.emit emitter "message" #js {:type "done"}))
|
||||
0)))
|
||||
|
||||
(defn- silent-behavior
|
||||
[_emitter _msg])
|
||||
|
||||
@ -67,74 +57,68 @@
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async with-worker-lends-and-reclaims
|
||||
(t/deftest ^:async run-lends-and-reclaims
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
sys (await (system/init (test-config calls silent-behavior 5000)))
|
||||
pool (pool-of sys)
|
||||
seen (atom nil)
|
||||
result (await (pool/with-worker pool (fn [worker] (reset! seen worker) :used)))]
|
||||
result (await (pool/run pool (fn [cfg] (reset! seen cfg) :used)))]
|
||||
(t/is (= :used result))
|
||||
(t/is (some? (unchecked-get @seen "__id")))
|
||||
(t/is (some? (unchecked-get (:exporter.wasm.pool/worker @seen) "__id")))
|
||||
(t/testing "a second use reuses the returned worker: nothing is created again"
|
||||
(await (pool/with-worker pool (fn [_] :again)))
|
||||
(await (pool/run pool (fn [_] :again)))
|
||||
(t/is (= 1 (count (filter #{:create} @calls)))))
|
||||
(await (system/halt sys)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-on-delivers-objects-and-done
|
||||
(t/deftest ^:async run-reads-the-pool-from-cfg
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
sys (await (system/init (test-config calls
|
||||
(render-behavior {:id "o1" :path "/tmp/o1.png"})
|
||||
5000)))
|
||||
seen (atom [])]
|
||||
(t/is (nil? (await (pool/with-worker (pool-of sys)
|
||||
(fn [worker]
|
||||
(pool/render-on worker
|
||||
{}
|
||||
(fn [object] (swap! seen conj object))
|
||||
{:timeout-ms 5000}))))))
|
||||
(t/is (= [{:id "o1" :path "/tmp/o1.png"}] @seen))
|
||||
(let [calls (atom [])
|
||||
sys (await (system/init (test-config calls silent-behavior 5000)))
|
||||
seen (atom nil)
|
||||
result (await (pool/run sys
|
||||
(fn [cfg] (reset! seen cfg) :used)))]
|
||||
(t/is (= :used result))
|
||||
(t/is (some? (unchecked-get (:exporter.wasm.pool/worker @seen) "__id")))
|
||||
(await (system/halt sys)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-on-rejects-cancelled-upfront
|
||||
(t/deftest ^:async run-threads-extra-args-to-the-body
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
sys (await (system/init (test-config calls silent-behavior 5000)))]
|
||||
(t/is (= [:a :b] (await (pool/run (pool-of sys)
|
||||
(fn [_ x y] [x y])
|
||||
:a :b))))
|
||||
(await (system/halt sys)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async run-rejects-cfg-without-pool
|
||||
(try
|
||||
(try
|
||||
(await (pool/run {:timeout-ms 5000} (fn [_] :used)))
|
||||
(t/is false "run should have rejected a cfg without pool")
|
||||
(catch :default cause
|
||||
(t/is (= :invalid-pool-cfg (-> cause ex-data :code)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async run-destroys-the-worker-on-body-failure
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
sys (await (system/init (test-config calls silent-behavior 5000)))]
|
||||
(try
|
||||
(await (pool/with-worker (pool-of sys)
|
||||
(fn [worker]
|
||||
(pool/render-on worker
|
||||
{}
|
||||
(fn [_])
|
||||
{:check-cancelled (fn []
|
||||
(throw (ex-info "export job was cancelled"
|
||||
{:code :job-cancelled})))
|
||||
:timeout-ms 5000}))))
|
||||
(t/is false "render-on should have rejected a cancelled render")
|
||||
(await (pool/run (pool-of sys)
|
||||
(fn [_] (throw (ex-info "body broke" {})))))
|
||||
(t/is false "run should have rejected a failed body")
|
||||
(catch :default cause
|
||||
(t/is (= :job-cancelled (-> cause ex-data :code)))
|
||||
(t/is (not-any? #(and (vector? %) (= :post (first %))) @calls))))
|
||||
(await (system/halt sys)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async silent-worker-times-out-and-is-terminated
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
sys (await (system/init (test-config calls silent-behavior 50)))]
|
||||
(try
|
||||
(await (pool/with-worker (pool-of sys)
|
||||
(fn [worker]
|
||||
(pool/render-on worker {} (fn [_]) {:timeout-ms 50}))))
|
||||
(t/is false "render-on should have rejected a silent worker")
|
||||
(catch :default cause
|
||||
(t/is (= :render-timeout (-> cause ex-data :code)))
|
||||
(t/is (some #{:terminate} @calls))))
|
||||
(t/is (= "body broke" (ex-message cause)))
|
||||
(t/is (some #{:terminate} @calls))
|
||||
(t/is (not-any? #{:release} @calls))))
|
||||
(await (system/halt sys)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
199
exporter/test/exporter_tests/wasm_scope_test.cljs
Normal file
199
exporter/test/exporter_tests/wasm_scope_test.cljs
Normal file
@ -0,0 +1,199 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.wasm-scope-test
|
||||
"The headless lease: one shared worker, sequential renders, failure
|
||||
isolation and driver-owned cancel, against a stub pool and stub
|
||||
workers. Task keys ride in cfg under `:exporter.renderer/*`."
|
||||
(:require
|
||||
["node:events" :as events]
|
||||
[app.common.transit :as transit]
|
||||
[cljs.test :as t :include-macros true]
|
||||
[exporter.wasm.scope :as scope]))
|
||||
|
||||
(defn- stub-worker
|
||||
[calls behavior]
|
||||
(let [^js emitter (events/EventEmitter.)]
|
||||
(unchecked-set emitter "postMessage"
|
||||
(fn [msg]
|
||||
(swap! calls conj [:post (unchecked-get msg "type")])
|
||||
(behavior emitter msg)))
|
||||
(unchecked-set emitter "terminate"
|
||||
(fn []
|
||||
(swap! calls conj :terminate)
|
||||
(.emit emitter "exit" 1)
|
||||
(js/Promise.resolve nil)))
|
||||
emitter))
|
||||
|
||||
(defn- render-behavior
|
||||
[payload]
|
||||
(fn [^js emitter _msg]
|
||||
(js/setTimeout
|
||||
(fn []
|
||||
(.emit emitter "message" #js {:type "object" :payload (transit/encode-str payload)})
|
||||
(.emit emitter "message" #js {:type "done"}))
|
||||
0)))
|
||||
|
||||
(defn- silent-behavior
|
||||
[_emitter _msg])
|
||||
|
||||
(defn- stub-pool
|
||||
[calls worker]
|
||||
#js {:acquire (fn [] (js/Promise.resolve worker))
|
||||
:release (fn [_] (swap! calls conj :release) (js/Promise.resolve nil))
|
||||
:destroy (fn [_] (swap! calls conj :destroy) (js/Promise.resolve nil))})
|
||||
|
||||
(defn- stub-check
|
||||
[calls cancelled?]
|
||||
(fn []
|
||||
(swap! calls conj :check-cancelled)
|
||||
(when @cancelled?
|
||||
(throw (ex-info "export job was cancelled"
|
||||
{:type :internal
|
||||
:code :job-cancelled
|
||||
:hint "export job was cancelled"})))))
|
||||
|
||||
(defn- test-cfg
|
||||
([calls worker on-object cancelled?]
|
||||
(test-cfg calls worker on-object cancelled? nil))
|
||||
([calls worker on-object cancelled? watch-interval]
|
||||
(cond-> {:exporter.wasm.pool/pool {:pool (stub-pool calls worker) :timeout-ms 5000}
|
||||
:exporter.renderer/on-object on-object
|
||||
:exporter.renderer/check-cancelled (stub-check calls cancelled?)}
|
||||
watch-interval (assoc :exporter.wasm.scope/watch-interval watch-interval))))
|
||||
|
||||
(defn- test-params
|
||||
[]
|
||||
{:type :png
|
||||
:objects [{:id "o1"}]})
|
||||
|
||||
(defn- posted?
|
||||
[calls]
|
||||
(some #(and (vector? %) (= :post (first %))) calls))
|
||||
|
||||
(t/deftest ^:async run-shares-one-worker-across-renders
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1" :path "/tmp/o1.png"}))
|
||||
seen (atom [])
|
||||
cfg (test-cfg calls worker
|
||||
(fn [object] (swap! seen conj object) nil)
|
||||
(atom false))]
|
||||
(await (scope/run cfg
|
||||
(^:async fn [render-fn]
|
||||
(await (render-fn (test-params)))
|
||||
(await (render-fn (test-params))))))
|
||||
(t/is (= [{:id "o1" :path "/tmp/o1.png"}
|
||||
{:id "o1" :path "/tmp/o1.png"}]
|
||||
@seen))
|
||||
(t/is (= 1 (count (filter #{:release} @calls))))
|
||||
(t/testing "the check runs as each render's turn comes up"
|
||||
(t/is (= 2 (count (filter #{:check-cancelled} @calls)))))
|
||||
(t/testing "a quiet lease leaves its worker alone"
|
||||
(t/is (not-any? #{:terminate} @calls))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async cancelled-render-rejects-before-posting
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))
|
||||
cfg (test-cfg calls worker (fn [_] nil) (atom true))]
|
||||
(try
|
||||
(await (scope/run cfg (fn [render-fn] (render-fn (test-params)))))
|
||||
(t/is false "run should have rejected a cancelled render")
|
||||
(catch :default cause
|
||||
(t/is (= :job-cancelled (-> cause ex-data :code)))
|
||||
(t/is (not (posted? @calls))))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async cancel-aborts-the-leased-worker-mid-render
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls silent-behavior)
|
||||
cancelled (atom false)
|
||||
cfg (test-cfg calls worker (fn [_] nil) cancelled 20)]
|
||||
(try
|
||||
(await (scope/run cfg
|
||||
(fn [render-fn]
|
||||
;; the cancel lands mid-render, once
|
||||
;; the worker is listening
|
||||
(js/setTimeout #(reset! cancelled true) 10)
|
||||
(render-fn (test-params)))))
|
||||
(t/is false "run should have rejected an aborted render")
|
||||
(catch :default cause
|
||||
(t/is (= :worker-exited (-> cause ex-data :code)))
|
||||
(t/is (some #{:terminate} @calls)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async failed-render-does-not-break-the-chain
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls (render-behavior {:id "o1"}))
|
||||
seen (atom [])
|
||||
thrown (atom false)
|
||||
cfg (test-cfg calls worker
|
||||
(fn [object]
|
||||
(if (compare-and-set! thrown false true)
|
||||
(throw (ex-info "on-object broke" {}))
|
||||
(do (swap! seen conj object) nil)))
|
||||
(atom false))]
|
||||
(await (scope/run cfg
|
||||
(^:async fn [render-fn]
|
||||
(try
|
||||
(await (render-fn (test-params)))
|
||||
(catch :default _cause
|
||||
(swap! calls conj :first-failed)))
|
||||
(await (render-fn (test-params))))))
|
||||
(t/is (some #{:first-failed} @calls))
|
||||
(t/is (= [{:id "o1"}] @seen)))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-on-without-lease-raises
|
||||
(try
|
||||
(try
|
||||
(await (scope/render-on {:exporter.renderer/on-object (fn [_] nil)
|
||||
:exporter.renderer/check-cancelled (fn [] nil)
|
||||
:exporter.wasm.pool/pool {:pool nil :timeout-ms 50}}
|
||||
(test-params)))
|
||||
(t/is false "render-on should have rejected a missing lease")
|
||||
(catch :default cause
|
||||
(t/is (= :worker-not-leased (-> cause ex-data :code)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async render-on-without-timeout-raises
|
||||
(try
|
||||
(try
|
||||
(await (scope/render-on {:exporter.wasm.pool/worker #js {}
|
||||
:exporter.renderer/on-object (fn [_] nil)
|
||||
:exporter.renderer/check-cancelled (fn [] nil)
|
||||
:exporter.wasm.pool/pool {:pool nil}}
|
||||
(test-params)))
|
||||
(t/is false "render-on should have rejected a missing timeout")
|
||||
(catch :default cause
|
||||
(t/is (= :assertion (-> cause ex-data :type)))
|
||||
(t/is (= :timeout-ms-missing (-> cause ex-data :code)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
|
||||
(t/deftest ^:async silent-worker-times-out-and-is-terminated
|
||||
(try
|
||||
(let [calls (atom [])
|
||||
worker (stub-worker calls silent-behavior)
|
||||
cfg (-> (test-cfg calls worker (fn [_] nil) (atom false))
|
||||
(assoc-in [:exporter.wasm.pool/pool :timeout-ms] 50))]
|
||||
(try
|
||||
(await (scope/run cfg (fn [render-fn] (render-fn (test-params)))))
|
||||
(t/is false "run should have rejected a silent worker")
|
||||
(catch :default cause
|
||||
(t/is (= :render-timeout (-> cause ex-data :code)))
|
||||
(t/is (some #{:terminate} @calls)))))
|
||||
(catch :default cause
|
||||
(t/is false (str "unexpected failure: " (ex-message cause))))))
|
||||
Loading…
x
Reference in New Issue
Block a user