diff --git a/.serena/memories/exporter/core.md b/.serena/memories/exporter/core.md index 6bbe196bea..f90d0314c3 100644 --- a/.serena/memories/exporter/core.md +++ b/.serena/memories/exporter/core.md @@ -41,10 +41,12 @@ on this side: the row of the job, its cancel and its result are the backend's. - First ported service is `exporter.browser` (whole `app.browser` at once): the running instance is the raw pool, the public API takes it first (`exec` fails fast on a nil pool), the browser factory is injected through config for hermetic tests, and the new tree carries no promesa. - `exporter.main` mirrors the backend's `app.main`: a `system-config` def reading env at the wiring layer, a public `system` atom with the running map, `start`/`stop`/`restart` (`^:async`, resolving `:started`/`:stopped`/`:restarted`), `start-custom` as the test seam, idempotent `install-process-handlers` for the `uncaughtException`/SIGTERM/SIGINT wiring (runs on `start`, never on namespace load), `^:dev/before-load-async`/`^:dev/after-load-async` hooks bridging the promise lifecycle to shadow's `done` callbacks (a hook failure logs, never sticks the reload), and `main` as the entry point: shadow boots `exporter.main/main`, and the old `app.core` reload path (`stop` and the `:devtools` wiring) sits commented out until `app.*` is fully ported. The entry leaves info logging enabled for the new tree (`setup!` on the `exporter` logger, mirroring the old `{:app :info}`); without it the default level filters everything below fatal. - Service config maps take plain (non-namespaced) param keys (`{:exporter.browser/pool {:max 2}}`): the top-level key already scopes the component, so namespacing the params only adds noise. Component keys and error reasons stay namespaced. -- `exporter.wasm.pool` mirrors the browser service for `worker_threads`: the running instance is `{:pool :timeout-ms}` (the pool plus the silence timeout its renders await), `with-worker`/`render-on`/`terminate` take the pool first, the worker factory is injected through config, and the creation handshake (`await-ready`) is its own unit with `:worker-not-ready`/`boot-error` paths. +- `exporter.wasm.pool` follows the backend `db/run!` shape: `run [cfg f & args]` resolves the pool from `cfg` (the `:exporter.wasm.pool/pool` service instance, or a raw pool), acquires a worker, applies `f` to the cfg carrying it under `:exporter.wasm.pool/worker` plus any extra args, and returns the worker — or destroys it when `f` fails, like a rollback. Pure pool mechanics only (`init`/`halt`, `terminate`, `await-ready`); the render protocol lives one level up. +- `exporter.wasm.scope` owns the lease: `run [sys on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the check is mandatory at the lease entry and runs here when configured. - `exporter.renderer.browser` merges the bitmap/pdf/svg flows into one `render [ctx params on-object]` (branching by type, `on-object` always awaited, unknown types raise); `ctx` carries what the job params do not freeze (`:browser-pool`, `:base-uri`, `:public-uri`, `:svgo?`). Dead xml predicates from the old tree did not survive the merge. -- `exporter.renderer.wasm` drives headless renders off the wasm pool instance (`{:pool :timeout-ms}`): `with-scope [sys lease]` leases one worker and serializes its renders, where the lease map carries `on-scope` (fn of the leased render fn, firing every headless export through it), `check-cancelled` and `watch-interval` (tests only). Cancel is fully driver-owned: the injected zero-arg check (raising `:job-cancelled`) runs before each render, and a watchdog interval polls it mid-render — on cancel the leased worker aborts (per-lease SharedArrayBuffer flipped, thread terminated) and the render rejects. The buffer is born in the lease; nobody outside knows what it is. A nil check arms no interval and wires no cancel. -- `exporter.renderer` is the malli-validated facade with one entry point: `render [sys & task]` takes a task map (`:exports :on-object :check-cancelled`) and fires the whole batch at once, the headless exports sharing one leased worker, the browser ones on their own checkouts. `sys` is task-free infrastructure with process lifetime (pools, uris, flags); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[sys params on-object check]` shape and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The scope fn survives one level down only, as the lease primitive inside `exporter.renderer.wasm`. +- `exporter.browser.scope` owns the browser batch: `run [sys task]` fires every export through its own checkout, all at once, bounded by the pool. No lease — a browser batch shares nothing but the pool, so fan-out only. The facade validates once, partitions by backend and joins both scopes. +- `exporter.renderer.wasm` is a thin single-shot over the scope lease. The scope fn survives one level down, as `exporter.wasm.scope/run`. +- `exporter.renderer` is the malli-validated facade with one entry point: `render [cfg & task]` takes a task map (`:exports :on-object :check-cancelled`) and fires the whole batch at once — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). `cfg` is task-free infrastructure with process lifetime (pools, uris, flags); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`. - The runner stays backend-blind: plan → render batch → settle, with beats/watchdog/settle untouched. Its `on-object` calls the same injected check the drivers use. When `exporter.jobs` is ported, its only cancel surface toward renderers is one zero-arg check constructor per job; the old `cancel-fns`/terminate-registration apparatus does not survive (abort ownership moved to the lease holder). - `exporter.main` wires both pools (`:exporter.browser/pool` and `:exporter.wasm.pool/pool`, each sized to the worker concurrency). No test boots the production wiring: the wasm pool warms one worker eagerly and the worker script in a test process would be the test bundle itself; the wiring is asserted as data and the lifecycle rides fake configs. - Async test vars prove their chain only when really awaited: a test passing with 0 assertions means the chain broke upstream. diff --git a/exporter/src/exporter/browser/scope.cljs b/exporter/src/exporter/browser/scope.cljs new file mode 100644 index 0000000000..de28d1ec3f --- /dev/null +++ b/exporter/src/exporter/browser/scope.cljs @@ -0,0 +1,26 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns exporter.browser.scope + "One batch through browsers: every export gets its own checkout, + all at once, bounded by the pool. No lease — a browser batch shares + nothing but the pool, so this level owns fan-out only, the mirror of + `exporter.wasm.scope` holding a lease instead. Task keys ride in + `cfg` under `:exporter.renderer/*`, set by whoever built the batch." + (:require + [exporter.renderer.browser :as driver])) + +(defn ^:async run + "Renders every export through its own browser checkout. Rejects on + the first failure; progress and cancel flow per export through the + task's own callbacks." + [cfg exports] + (let [on-object (:exporter.renderer/on-object cfg) + check-cancelled (:exporter.renderer/check-cancelled cfg)] + (await (js/Promise.all + (mapv (fn [params] + (driver/render cfg params on-object check-cancelled)) + exports))))) diff --git a/exporter/src/exporter/renderer.cljs b/exporter/src/exporter/renderer.cljs index e3b134f94e..847c1329b4 100644 --- a/exporter/src/exporter/renderer.cljs +++ b/exporter/src/exporter/renderer.cljs @@ -7,31 +7,32 @@ (ns exporter.renderer "Common renderer interface, validated with malli from day one. - One entry point: `render` takes `sys` plus a task map - (`:exports`, `:on-object`, `:check-cancelled`) and fires the whole - batch at once, dispatching each export to the browser or the - headless driver by its `:is-wasm` flag. A single export is a batch - of one. The headless ones share one leased worker (serialized on - it); the browser ones check out their own browser each (bounded by - the pool). Rendered objects travel through `on-object`, so the - batch itself resolves a vector of nils. + One entry point: `render` takes `cfg` plus a task map + (`:exports`, `:on-object`, `:check-cancelled`), validates it, and + fires the whole batch at once — the browser exports through + `exporter.browser.scope`, the wasm ones through one leased + worker in `exporter.wasm.scope`. A single export is a batch of one. + Rendered objects travel through `on-object`; the batch itself + resolves nil once every export landed. - `sys` is task-free infrastructure with process lifetime — the - browser pool and render uris, the wasm pool instance, flags — what - could one day come straight from the running system map. The task + `cfg` is task-free infrastructure with process lifetime: the pool + services under their system keys (`:exporter.browser/pool`, + `:exporter.wasm.pool/pool`) plus the static render config + (`:base-uri`, `:public-uri`, `:svgo?`) — a view over the running + system map that each domain reads with its own keys. The task map holds everything per-task: the `exports` data plus the two injected indirections, `on-object` (the per-object collector) and - `check-cancelled` (the zero-arg cancel check raising + `check-cancelled` (the zero-arg check-cancelled raising `:job-cancelled`). One closed schema validates the whole task, so a bad batch — or a non-fn injection — never touches a pool. Each - driver takes the same `[sys params on-object check]` shape and + driver takes the same `[cfg params on-object check-cancelled]` shape and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm." (:require [app.common.schema :as sm] - [exporter.renderer.browser :as browser] - [exporter.renderer.wasm :as wasm])) + [exporter.browser.scope :as bscope] + [exporter.wasm.scope :as scope])) (def schema:type [:enum :png :jpeg :webp :pdf :svg]) @@ -70,28 +71,29 @@ (def ^:private check-task (sm/check-fn schema:task :hint "invalid render task")) -(defn headless? +(defn wasm? "Whether `params` renders with render-wasm rather than a browser." [{:keys [is-wasm]}] (boolean is-wasm)) (defn ^:async render - "Renders every export, firing them all at once. The whole task is - validated before anything is leased, so a bad task never touches a - pool. Rejects on the first failure." - [sys & {:as task}] - (let [{:keys [exports on-object check-cancelled] :as task} - (check-task task) - - do-render - (fn [wasm-render] - (js/Promise.all - (mapv (fn [params] - (if (and (headless? params) (fn? wasm-render)) - (wasm-render params on-object) - (browser/render sys params on-object check-cancelled))) - exports)))] - - (if (some headless? exports) - (await (wasm/with-scope sys (assoc task :on-scope do-render))) - (await (do-render nil))))) + "Renders every export, firing them all at once: the browser ones + through their own scope, the wasm ones through one leased + worker. The whole task is validated before anything is leased, so a + bad task never touches a pool. Rejects on the first failure." + [cfg & {:as task}] + (let [{:keys [exports on-object check-cancelled]} (check-task task) + cfg (assoc cfg + ::on-object on-object + ::check-cancelled check-cancelled) + wasm-exports (filterv wasm? exports) + browser-exports (remove wasm? exports)] + (await (js/Promise.all + (cond-> [] + (seq browser-exports) + (conj (bscope/run cfg browser-exports)) + (seq wasm-exports) + (conj (scope/run cfg + (fn [render-fn] + (js/Promise.all (mapv render-fn wasm-exports)))))))) + nil)) diff --git a/exporter/src/exporter/renderer/browser.cljs b/exporter/src/exporter/renderer/browser.cljs index b28ba49d17..d31d60a074 100644 --- a/exporter/src/exporter/renderer/browser.cljs +++ b/exporter/src/exporter/renderer/browser.cljs @@ -12,9 +12,9 @@ with each object carrying its `:path`. `on-object` may return a plain value or a promise; it is always awaited. - `sys` is task-free infrastructure: the `:browser-pool`, the + `cfg` is task-free infrastructure: the `:exporter.browser/pool`, the render `:base-uri` and `:public-uri`, and the `:svgo?` flag. The - per-task injections (`on-object`, `check`) ride positional. There + per-task injections (`on-object`, `check-cancelled`) ride positional. There are no promesa chains in this namespace." (:require ["@penpot/svgo" :as svgo] @@ -34,14 +34,18 @@ (l/set-level! :trace) -(defn- check-cancelled - "Cooperative checkpoint over the injected zero-arg check. This +(defn- raise-if-cancelled + "Cooperative checkpoint over the injected zero-arg check-cancelled. This backend never blocks indefinitely (every step yields), so - yielding-point checks are the whole cancel story here; a nil check - wires no cancel." - [check] - (when check - (check))) + yielding-point checks are the whole cancel story here. The check-cancelled is + mandatory: without one it raises `:check-cancelled-missing` instead + of running unprotected." + [check-cancelled] + (if (fn? check-cancelled) + (check-cancelled) + (throw (ex/error :type :assertion + :code :check-cancelled-missing + :hint "browser render needs a zero-arg check-cancelled")))) (defn- prepare-options [uri token scale] @@ -91,15 +95,15 @@ nil) (defn- ^:async render-bitmap - [sys {:keys [file-id page-id share-id token scale type objects skip-children]} on-object] + [cfg {:keys [file-id page-id share-id token scale type objects skip-children]} on-object] (let [query {:file-id file-id :page-id page-id :share-id share-id :object-id (mapv :id objects) :route "objects" :skip-children skip-children} - uri (render-uri (:base-uri sys) query)] - (await (browser/exec (:browser-pool sys) + uri (render-uri (:base-uri cfg) query)] + (await (browser/exec (:exporter.browser/pool cfg) (prepare-options uri token scale) (fn [page] (render-bitmap-page uri page type objects on-object)))))) @@ -151,20 +155,20 @@ (await (on-object (assoc object :path path)))))) (defn- ^:async render-pdf-page - [base-uri page params objects on-object check] + [base-uri page params objects on-object check-cancelled] (doseq [object objects] - (check-cancelled check) + (raise-if-cancelled check-cancelled) (await (render-pdf-object page base-uri params object on-object))) nil) (defn- ^:async render-pdf - [sys {:keys [token scale] :as params} on-object check] - (let [base-uri (-> (:base-uri sys) + [cfg {:keys [token scale] :as params} on-object check-cancelled] + (let [base-uri (-> (:base-uri cfg) (u/ensure-path-slash))] - (await (browser/exec (:browser-pool sys) + (await (browser/exec (:exporter.browser/pool cfg) (prepare-options base-uri token scale) (fn [page] - (render-pdf-page base-uri page params (:objects params) on-object check)))))) + (render-pdf-page base-uri page params (:objects params) on-object check-cancelled)))))) ;; --- SVG @@ -440,7 +444,7 @@ (str/replace content " " " ")) (defn- ^:async render-svg-object - [sys page type object on-object] + [cfg page type object on-object] (let [path (await (sh/tempfile :prefix "penpot.tmp.render.svg." :suffix (mime/get-extension type))) node (browser/select page (str/concat "#screenshot-" (:id object)))] (await (browser/wait-for node)) @@ -449,56 +453,56 @@ result (replace-text-nodes xmldata txtdata) result (sanitize-svg-content result) - result (if (:svgo? sys) + result (if (:svgo? cfg) (svgo/optimize result svgo/defaultOptions) result) - result (replace-internal-uris result (:base-uri sys) (:public-uri sys))] + result (replace-internal-uris result (:base-uri cfg) (:public-uri cfg))] (await (sh/write-file path result)) (await (on-object (assoc object :path path)))))) (defn- ^:async render-svg-page - [sys uri page type objects on-object] + [cfg uri page type objects on-object] (l/info :uri uri) ;; navigate to the page and perform basic setup (await (browser/nav page (str uri))) (await (browser/sleep page 1000)) ; the good old fix with sleep (await (browser/wait-for-fonts page)) ;; take the screnshot of requested objects, one by one - (await (js/Promise.all (mapv (fn [object] (render-svg-object sys page type object on-object)) + (await (js/Promise.all (mapv (fn [object] (render-svg-object cfg page type object on-object)) objects))) nil) (defn- ^:async render-svg - [sys {:keys [file-id page-id share-id token scale type objects]} on-object] + [cfg {:keys [file-id page-id share-id token scale type objects]} on-object] (let [query {:file-id file-id :page-id page-id :share-id share-id :render-embed true :object-id (mapv :id objects) :route "objects"} - uri (render-uri (:base-uri sys) query)] - (await (browser/exec (:browser-pool sys) + uri (render-uri (:base-uri cfg) query)] + (await (browser/exec (:exporter.browser/pool cfg) (prepare-options uri token scale) - (fn [page] (render-svg-page sys uri page type objects on-object)))))) + (fn [page] (render-svg-page cfg uri page type objects on-object)))))) (defn ^:async render "Renders one partition of objects through a browser page: bitmap (png, jpeg, webp), pdf or svg depending on the type in `params`. - Same shape as `exporter.renderer.wasm/render`: `sys` is task-free - infrastructure (pools, uris, flags), while `on-object` and `check` + Same shape as `exporter.renderer.wasm/render`: `cfg` is task-free + infrastructure (pools, uris, flags), while `on-object` and `check-cancelled` ride positional as the per-task injections — the per-object - collector and the zero-arg cancel check raising `:job-cancelled`. - The check runs here at fail-fast entry plus the sequential pdf - loop; the concurrent flows rely on the entry check and the caller's + collector and the zero-arg check-cancelled raising `:job-cancelled`. + The check-cancelled runs here at fail-fast entry plus the sequential pdf + loop; the concurrent flows rely on the entry check-cancelled and the caller's `on-object`. Calls `on-object` with each object carrying its `:path`, awaiting it every time, and resolves nil." - [sys {:keys [type] :as params} on-object check] - (check-cancelled check) + [cfg {:keys [type] :as params} on-object check-cancelled] + (raise-if-cancelled check-cancelled) (case type - (:png :jpeg :webp) (await (render-bitmap sys params on-object)) - :pdf (await (render-pdf sys params on-object check)) - :svg (await (render-svg sys params on-object)) + (:png :jpeg :webp) (await (render-bitmap cfg params on-object)) + :pdf (await (render-pdf cfg params on-object check-cancelled)) + :svg (await (render-svg cfg params on-object)) (ex/raise :type :validation :code :unknown-render-type :hint (str "unknown render type: " type)))) diff --git a/exporter/src/exporter/renderer/wasm.cljs b/exporter/src/exporter/renderer/wasm.cljs index cae8480ca1..779f150105 100644 --- a/exporter/src/exporter/renderer/wasm.cljs +++ b/exporter/src/exporter/renderer/wasm.cljs @@ -5,84 +5,18 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns exporter.renderer.wasm - "Main-thread side of the headless renderer. - - Renders run on pooled workers because Skia calls are synchronous and - would block the event loop for every other export. Each batch keeps - one worker for all its renders, sharing its caches and pool slot. - - `sys` is task-free infrastructure (the wasm pool instance, the - `{:pool :timeout-ms}` map the pool service owns — hence unwrapping - one level to reach the raw pool). The lease map holds the rest: - `on-scope`, a fn receiving the leased 2-arg render fn and firing - every headless export through it, and `check-cancelled`, the - zero-arg cancel check raising `:job-cancelled` once the job ended. - Cancel is fully driver-owned: the check runs before each render, - and a watchdog interval polls it mid-render — on cancel the leased - worker aborts (shared signal flipped, thread terminated) and the - render rejects. The buffer the worker polls is born in the lease - itself; nobody outside knows what it is. A nil check arms no - interval and wires no cancel." + "Single-shot headless renders: builds the renderer task keys and + delegates the lease to `exporter.wasm.scope`." (:require [app.common.logging :as l] - [exporter.wasm.pool :as pool])) - -(defn- serializer - "Chains thunks so a job's renders run one at a time on its worker. A - failure is isolated: it doesn't break the chain for the next one." - [] - (let [queue (atom (js/Promise.resolve nil))] - (fn [thunk] - (let [result (.then @queue (fn [_] (thunk)) (fn [_] (thunk)))] - (reset! queue (.then result (fn [_] nil) (fn [_] nil))) - result)))) - -(defn ^:async with-scope - "Leases one worker for the batch: `on-scope` fires every headless - export through the leased render fn, one at a time on the same - worker, so the check runs as each render's turn comes up. The - interval dies with the lease; a truly stuck thread is backstopped - by the pool's own silence watchdog." - [sys {:keys [check-cancelled on-scope watch-interval] :or {watch-interval 1000}}] - (await - (pool/with-worker (:pool (:wasm-pool sys)) - (fn [worker] - ((^:async fn [] - (let [chain (serializer) - live (volatile! worker) - signal (js/Int32Array. (js/SharedArrayBuffer. 4)) - timer (volatile! nil) - opts {:cancel-buffer (.-buffer signal) - :check-cancelled check-cancelled - :timeout-ms (:timeout-ms (:wasm-pool sys))}] - (vreset! timer - (when check-cancelled - (js/setInterval - (fn [] - (try - (check-cancelled) - (catch :default _cause - ;; One-shot: the lease is doomed anyway. - ;; Between objects the worker sees the - ;; flag; inside a render only terminating - ;; the thread stops it. - (some-> @timer js/clearInterval) - (js/Atomics.store signal 0 1) - (some-> @live pool/terminate)))) - watch-interval))) - (try - (await (on-scope (fn [params on-object] - (chain (fn [] (pool/render-on worker params on-object opts)))))) - (finally - (some-> @timer js/clearInterval) - (vreset! live nil)))))))))) + [exporter.wasm.scope :as scope])) (defn ^:async render "Same shape as `exporter.renderer.browser/render`: one export as a batch of one." - [sys params on-object check] + [cfg params on-object check-cancelled] (l/info :hint "render" :type (:type params) :backend "wasm") - (await (with-scope sys {:exports [params] - :on-object on-object - :check-cancelled check - :on-scope (fn [render-wasm] (render-wasm params on-object))}))) + (await (scope/run (assoc cfg + :exporter.renderer/on-object on-object + :exporter.renderer/check-cancelled check-cancelled) + (fn [render-fn] (render-fn params))))) diff --git a/exporter/src/exporter/wasm/pool.cljs b/exporter/src/exporter/wasm/pool.cljs index cacb3b69ff..054e16bdfd 100644 --- a/exporter/src/exporter/wasm/pool.cljs +++ b/exporter/src/exporter/wasm/pool.cljs @@ -31,7 +31,6 @@ [app.common.data :as d] [app.common.exceptions :as ex] [app.common.logging :as l] - [app.common.transit :as t] [exporter.utils.system :as system])) (l/set-level! :info) @@ -148,106 +147,40 @@ (l/info :hint "finalizing render worker pool") (drain-pool pool))) -(defn ^:async with-worker - "Acquires one worker for the whole of `f`, a fn of that worker. On - failure the worker is destroyed instead of reused: the module may be - aborted or mid-write, and a terminated worker cannot come back." - [pool f] - (let [worker (await (.acquire ^js pool))] - (try - (let [result (await (f worker))] - (await (.release ^js pool worker)) - result) - (catch :default cause - (try - (await (.destroy ^js pool worker)) - (catch :default cause' - (l/warn :hint "render worker destroy failed" - :cause cause'))) - (throw cause))))) +(defn- pool? + "A raw generic-pool: anything lendable through acquire and release." + [x] + (and (some? x) + (fn? (unchecked-get x "acquire")) + (fn? (unchecked-get x "release")))) -(defn- render-on-worker - "Settles when the worker reports the render finished, failed, or the - thread went away. That last case matters: a terminated worker (how a - cancel stops a render mid-Skia) emits `exit` and never `error`, and a - promise left pending there would keep its pool slot borrowed for the - life of the process." - [^js worker params cancel-buffer on-object timeout-ms] - (js/Promise. - (fn [resolve reject] - (let [timer (volatile! nil)] - (letfn [(disarm [] - (when-let [t @timer] - (js/clearTimeout t) - (vreset! timer nil))) - - (rearm [] - (disarm) - (vreset! timer (js/setTimeout - (fn [] - (l/error :hint "render worker went silent, terminating" - :worker-id (unchecked-get worker "__id")) - (cleanup) - ;; Terminating is what frees the pool slot: - ;; the `exit` it raises has no listener left. - (unchecked-set worker "__alive" false) - (.terminate ^js worker) - (reject (ex/error :type :internal - :code :render-timeout - :hint "render worker stopped responding"))) - timeout-ms))) - - (cleanup [] - (disarm) - (.off worker "message" on-message) - (.off worker "error" on-error) - (.off worker "exit" on-exit)) - - (on-error [cause] - (cleanup) - (reject cause)) - - (on-exit [code] - (cleanup) - (reject (ex/error :type :internal - :code :worker-exited - :hint (str "render worker exited with code " code)))) - - (on-message [data] - (rearm) - (case (unchecked-get data "type") - ;; A failure while the main thread handles the object - ;; (moving the file, appending to the zip) has to end - ;; the render too, or nothing ever settles this promise. - "object" (try - (on-object (t/decode-str (unchecked-get data "payload"))) - (catch :default cause - (cleanup) - (reject cause))) - "done" (do (cleanup) (resolve nil)) - "error" (do (cleanup) - (reject (ex/error :type :internal - :code (or (some-> (unchecked-get data "code") keyword) - :wasm-render-error) - :hint (unchecked-get data "message")))) - nil))] - - (.on worker "message" on-message) - (.once worker "error" on-error) - (.once worker "exit" on-exit) - (rearm) - (.postMessage worker #js {:type "render" - :params (t/encode-str params) - :cancel cancel-buffer})))))) - -(defn ^:async render-on - "Renders `params` on an already acquired worker. The check runs - before posting: a render whose turn comes up cancelled rejects - without touching the worker." - [worker params on-object {:keys [cancel-buffer check-cancelled timeout-ms]}] - (when check-cancelled - (check-cancelled)) - (await (render-on-worker worker params cancel-buffer on-object timeout-ms))) +(defn ^:async run + "Lends one worker: resolves the pool from `cfg` (the `::pool` + service instance, or a raw pool), acquires a worker, applies `f` to + the cfg carrying it under `::worker` plus any extra `args`, and + returns the worker — or destroys it when `f` fails, the way a failed + transaction rolls back instead of returning its connection: a failed + worker may be aborted or mid-write, and a terminated worker cannot + come back." + [cfg f & args] + (let [cfg (if (pool? cfg) {::pool {:pool cfg}} cfg) + pool (:pool (::pool cfg))] + (when-not (pool? pool) + (throw (ex/error :type :assertion + :code :invalid-pool-cfg + :hint "no pool in cfg"))) + (let [worker (await (.acquire ^js pool))] + (try + (let [result (await (apply f (assoc cfg ::worker worker) args))] + (await (.release ^js pool worker)) + result) + (catch :default cause + (try + (await (.destroy ^js pool worker)) + (catch :default cause' + (l/warn :hint "render worker destroy failed" + :cause cause'))) + (throw cause)))))) (defn terminate [worker] diff --git a/exporter/src/exporter/wasm/scope.cljs b/exporter/src/exporter/wasm/scope.cljs new file mode 100644 index 0000000000..3bec169eae --- /dev/null +++ b/exporter/src/exporter/wasm/scope.cljs @@ -0,0 +1,171 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns exporter.wasm.scope + "One leased render worker for a whole batch. + + The pool owns lending; this level owns what a lease means: every + headless export runs on the same worker, one at a time, sharing its + caches and pool slot, with the cancel abort armed for the life of + the lease. Task keys arrive in `cfg` under `:exporter.renderer/*` + (set by whoever built the batch); the pool instance under the + pool's own key." + (:require + [app.common.exceptions :as ex] + [app.common.logging :as l] + [app.common.transit :as t] + [exporter.wasm.pool :as pool])) + +(defn- serializer + "Chains thunks so a job's renders run one at a time on its worker. A + failure is isolated: it doesn't break the chain for the next one." + [] + (let [queue (atom (js/Promise.resolve nil))] + (fn [thunk] + (let [result (.then @queue (fn [_] (thunk)) (fn [_] (thunk)))] + (reset! queue (.then result (fn [_] nil) (fn [_] nil))) + result)))) + +(defn- render-on-worker- + "Settles when the worker reports the render finished, failed, or the + thread went away. That last case matters: a terminated worker (how a + cancel stops a render mid-Skia) emits `exit` and never `error`, and a + promise left pending there would keep its pool slot borrowed for the + life of the process." + [^js worker params cancel-buffer on-object timeout-ms] + (js/Promise. + (fn [resolve reject] + (let [timer (volatile! nil)] + (letfn [(disarm [] + (when-let [t @timer] + (js/clearTimeout t) + (vreset! timer nil))) + + (rearm [] + (disarm) + (vreset! timer (js/setTimeout + (fn [] + (l/error :hint "render worker went silent, terminating" + :worker-id (unchecked-get worker "__id")) + (cleanup) + ;; Terminating is what frees the pool slot: + ;; the `exit` it raises has no listener left. + (unchecked-set worker "__alive" false) + (.terminate ^js worker) + (reject (ex/error :type :internal + :code :render-timeout + :hint "render worker stopped responding"))) + timeout-ms))) + + (cleanup [] + (disarm) + (.off worker "message" on-message) + (.off worker "error" on-error) + (.off worker "exit" on-exit)) + + (on-error [cause] + (cleanup) + (reject cause)) + + (on-exit [code] + (cleanup) + (reject (ex/error :type :internal + :code :worker-exited + :hint (str "render worker exited with code " code)))) + + (on-message [data] + (rearm) + (case (unchecked-get data "type") + ;; A failure while the main thread handles the object + ;; (moving the file, appending to the zip) has to end + ;; the render too, or nothing ever settles this promise. + "object" (try + (on-object (t/decode-str (unchecked-get data "payload"))) + (catch :default cause + (cleanup) + (reject cause))) + "done" (do (cleanup) (resolve nil)) + "error" (do (cleanup) + (reject (ex/error :type :internal + :code (or (some-> (unchecked-get data "code") keyword) + :wasm-render-error) + :hint (unchecked-get data "message")))) + nil))] + + (.on worker "message" on-message) + (.once worker "error" on-error) + (.once worker "exit" on-exit) + (rearm) + (.postMessage worker #js {:type "render" + :params (t/encode-str params) + :cancel cancel-buffer})))))) + +(defn ^:async render-on + "Low-level render on the worker leased in `cfg`. A missing worker + or timeout raises instead of failing obscurely (`:worker-not-leased`, + `:timeout-ms-missing`): both are programmer errors, and a missing + timeout must never degrade silently into the default silence budget." + [cfg params] + (let [worker (::pool/worker cfg) + timeout-ms (or (:timeout-ms (:exporter.wasm.pool/pool cfg)) + (throw (ex/error :type :assertion + :code :timeout-ms-missing + :hint "render-on needs :timeout-ms in cfg")))] + (when-not worker + (throw (ex/error :type :assertion + :code :worker-not-leased + :hint "render-on needs a leased worker in cfg"))) + (when-let [check-cancelled (:exporter.renderer/check-cancelled cfg)] + (check-cancelled)) + (await (render-on-worker- worker + params + (::cancel-buffer cfg) + (:exporter.renderer/on-object cfg) + timeout-ms)))) + +(defn ^:async run + "Leases one worker for `on-scope`, a fn of a single-arg render fn. + Every headless export fires through it, one at a time on the same + worker, so the check-cancelled runs as each render's turn comes up. The check-cancelled + is mandatory: without one the lease raises `:check-cancelled- + missing` before acquiring anything. The per lease signal buffer is + born here; the interval dies with the lease. A truly stuck thread + is backstopped by the pool's own silence watchdog." + [cfg on-scope] + (let [check-cancelled (:exporter.renderer/check-cancelled cfg)] + (when-not (fn? check-cancelled) + (throw (ex/error :type :assertion + :code :check-cancelled-missing + :hint "scope/run needs a zero-arg check-cancelled in cfg"))) + (await + (pool/run cfg + (^:async fn [cfg] + (let [worker (::pool/worker cfg) + chain (serializer) + live (volatile! worker) + signal (js/Int32Array. (js/SharedArrayBuffer. 4)) + timer (volatile! nil) + render-cfg (assoc cfg + ::cancel-buffer (.-buffer signal))] + (vreset! timer + (js/setInterval + (fn [] + (try + (check-cancelled) + (catch :default _cause + ;; One-shot: the lease is doomed anyway. + ;; Between objects the worker sees the + ;; flag; inside a render only terminating + ;; the thread stops it. + (some-> @timer js/clearInterval) + (js/Atomics.store signal 0 1) + (some-> @live pool/terminate)))) + (::watch-interval cfg 1000))) + (try + (await (on-scope (fn [params] (chain (fn [] (render-on render-cfg params)))))) + (finally + (some-> @timer js/clearInterval) + (vreset! live nil))))))))) diff --git a/exporter/test/exporter_tests/browser_scope_test.cljs b/exporter/test/exporter_tests/browser_scope_test.cljs new file mode 100644 index 0000000000..53c100f28c --- /dev/null +++ b/exporter/test/exporter_tests/browser_scope_test.cljs @@ -0,0 +1,95 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns exporter-tests.browser-scope-test + "The browser batch: every export through its own checkout, all at + once, against a stub pool with stub pages." + (:require + ["generic-pool" :as gp] + [app.common.uuid :as uuid] + [app.config :as cf] + [cljs.test :as t :include-macros true] + [exporter.browser.scope :as scope])) + +(defn- stub-locator + [] + #js {:waitFor (fn [_opts] (js/Promise.resolve nil)) + :screenshot (fn [_opts] (js/Promise.resolve nil))}) + +(defn- stub-page + [calls] + #js {:goto (fn [url _opts] + (swap! calls conj [:goto (str url)]) + (js/Promise.resolve nil)) + :waitForTimeout (fn [ms] (swap! calls conj [:sleep ms]) (js/Promise.resolve nil)) + :waitForFunction (fn [_src _arg _opts] (js/Promise.resolve nil)) + :locator (fn [sel] (swap! calls conj [:select sel]) (stub-locator)) + :$$ (fn [sel] + (swap! calls conj [:select-all sel]) + (js/Promise.resolve #js [])) + :evaluate (fn [_f & _args] (js/Promise.resolve nil))}) + +(defn- stub-context + [page] + #js {:newPage (fn [] (js/Promise.resolve page)) + :close (fn [] (js/Promise.resolve nil))}) + +(defn- stub-browser + [page] + #js {:newContext (fn [_opts] (js/Promise.resolve (stub-context page))) + :isConnected (fn [] true) + :close (fn [] (js/Promise.resolve nil))}) + +(defn- stub-pool + [calls] + (let [page (stub-page calls)] + (gp/createPool #js {:create (fn [] (js/Promise.resolve (stub-browser page))) + :destroy (fn [browser] (.close ^js browser)) + :validate (fn [_] true)} + #js {:max 2 :min 0}))) + +(defn- test-cfg + [calls] + {:exporter.browser/pool (stub-pool calls) + :base-uri (cf/get-internal-uri) + :public-uri (cf/get :public-uri) + :svgo? false}) + +(defn- test-params + [id] + {:file-id (uuid/next) + :page-id (uuid/next) + :token "token" + :scale 1 + :type :png + :objects [{:id id :name id :suffix ".png" :filename (str id ".png")}]}) + +(t/deftest ^:async run-fires-every-export-through-its-own-checkout + (try + (let [calls (atom []) + seen (atom [])] + (await (scope/run (assoc (test-cfg calls) + :exporter.renderer/on-object (fn [object] (swap! seen conj object) nil) + :exporter.renderer/check-cancelled (fn [] nil)) + [(test-params "a") (test-params "b")])) + (t/is (= ["a" "b"] (mapv :id @seen))) + (t/is (= 2 (count (filter #(and (vector? %) (= :goto (first %))) @calls))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async run-rejects-on-the-first-failure + (try + (let [calls (atom [])] + (try + (await (scope/run (assoc (test-cfg calls) + :exporter.renderer/on-object (fn [_] (throw (ex-info "on-object broke" {}))) + :exporter.renderer/check-cancelled (fn [] nil)) + [(test-params "a")])) + (t/is false "run should have rejected a failing on-object") + (catch :default cause + (t/is (= "on-object broke" (ex-message cause)))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) diff --git a/exporter/test/exporter_tests/renderer_browser_test.cljs b/exporter/test/exporter_tests/renderer_browser_test.cljs index 0bf64296dd..14dc93c310 100644 --- a/exporter/test/exporter_tests/renderer_browser_test.cljs +++ b/exporter/test/exporter_tests/renderer_browser_test.cljs @@ -62,12 +62,12 @@ :validate (fn [_] true)} #js {:max 1 :min 0})) -(defn- test-sys +(defn- test-cfg [pool] - {:browser-pool pool - :base-uri (cf/get-internal-uri) - :public-uri (cf/get :public-uri) - :svgo? false}) + {:exporter.browser/pool pool + :base-uri (cf/get-internal-uri) + :public-uri (cf/get :public-uri) + :svgo? false}) (defn- never-cancelled [] @@ -77,7 +77,7 @@ [calls raise-on-call] (let [seen (atom 0)] (fn [] - (swap! calls conj :check) + (swap! calls conj :check-cancelled) (when (= raise-on-call (swap! seen inc)) (throw (ex-info "export job was cancelled" {:type :internal @@ -109,7 +109,7 @@ page (stub-page calls (stub-locator calls (fn [_] nil))) pool (test-pool calls page) {:keys [seen fn]} (collect-on-object)] - (await (render/render (test-sys pool) (test-params :png ["a" "b"]) fn (never-cancelled))) + (await (render/render (test-cfg pool) (test-params :png ["a" "b"]) fn (never-cancelled))) (t/is (= 2 (count @seen))) (t/is (every? #(str/ends-with? (:path %) ".png") @seen)) (t/is (= ["a" "b"] (mapv :id @seen))) @@ -128,7 +128,7 @@ page (stub-page calls (stub-locator calls (fn [_] nil))) pool (test-pool calls page) {:keys [seen fn]} (collect-on-object)] - (await (render/render (test-sys pool) (test-params :pdf ["a" "b"]) fn (never-cancelled))) + (await (render/render (test-cfg pool) (test-params :pdf ["a" "b"]) fn (never-cancelled))) (t/is (= 2 (count @seen))) (t/is (every? #(str/ends-with? (:path %) ".pdf") @seen)) (t/testing "one navigation per object" @@ -142,9 +142,9 @@ (let [calls (atom []) page (stub-page calls (stub-locator calls (fn [_] nil))) pool (test-pool calls page) - sys (test-sys pool)] + cfg (test-cfg pool)] (try - (await (render/render sys (test-params :png ["a"]) + (await (render/render cfg (test-params :png ["a"]) (fn [_] (swap! calls conj :object)) (raising-check calls 1))) (t/is false "render should have rejected a cancelled render") @@ -162,10 +162,10 @@ pool (test-pool calls page) ;; entry check is the first call, then one per object: ;; the cancel lands as the second object comes up - sys (test-sys pool) + cfg (test-cfg pool) {:keys [seen fn]} (collect-on-object)] (try - (await (render/render sys (test-params :pdf ["a" "b"]) fn (raising-check calls 3))) + (await (render/render cfg (test-params :pdf ["a" "b"]) fn (raising-check calls 3))) (t/is false "render should have rejected a cancelled batch") (catch :default cause (t/is (= :job-cancelled (-> cause ex-data :code))) @@ -182,7 +182,7 @@ page (stub-page calls locator) pool (test-pool calls page) {:keys [seen fn]} (collect-on-object) - _ (await (render/render (test-sys pool) (test-params :svg ["a"]) fn (never-cancelled))) + _ (await (render/render (test-cfg pool) (test-params :svg ["a"]) fn (never-cancelled))) path (:path (first @seen)) content (await (sh/read-file path))] (t/is (= 1 (count @seen))) diff --git a/exporter/test/exporter_tests/renderer_test.cljs b/exporter/test/exporter_tests/renderer_test.cljs index 7ad56b0a3e..dff034e37e 100644 --- a/exporter/test/exporter_tests/renderer_test.cljs +++ b/exporter/test/exporter_tests/renderer_test.cljs @@ -6,7 +6,7 @@ (ns exporter-tests.renderer-test "The renderer facade: malli validation and dispatch to the browser - or headless drivers, against a stub browser pool and a stub wasm + or wasm drivers, against a stub browser pool and a stub wasm pool." (:require ["generic-pool" :as gp] @@ -87,16 +87,16 @@ (defn- stub-check [calls] (fn [] - (swap! calls conj :check))) + (swap! calls conj :check-cancelled))) -(defn- test-sys +(defn- test-cfg [calls] (let [worker (stub-worker calls)] - {:browser-pool (stub-browser-pool calls) - :base-uri (cf/get-internal-uri) - :public-uri (cf/get :public-uri) - :svgo? false - :wasm-pool {:pool (stub-wasm-pool calls worker) :timeout-ms 5000}})) + {:exporter.browser/pool (stub-browser-pool calls) + :base-uri (cf/get-internal-uri) + :public-uri (cf/get :public-uri) + :svgo? false + :exporter.wasm.pool/pool {:pool (stub-wasm-pool calls worker) :timeout-ms 5000}})) (defn- never-cancelled [calls] @@ -112,16 +112,16 @@ :objects [{:id "a" :name "a" :suffix ".png" :filename "a.png"}]} opts)) -(t/deftest headless-reads-the-wasm-flag - (t/is (true? (renderer/headless? {:is-wasm true :type :png}))) - (t/is (false? (renderer/headless? {:type :png}))) - (t/is (false? (renderer/headless? {:is-wasm false :type :png})))) +(t/deftest wasm-reads-the-wasm-flag + (t/is (true? (renderer/wasm? {:is-wasm true :type :png}))) + (t/is (false? (renderer/wasm? {:type :png}))) + (t/is (false? (renderer/wasm? {:is-wasm false :type :png})))) (t/deftest ^:async render-routes-bitmaps-to-the-browser (try (let [calls (atom []) seen (atom [])] - (await (renderer/render (test-sys calls) + (await (renderer/render (test-cfg calls) :exports [(test-params :png nil)] :on-object (fn [object] (swap! seen conj object) nil) :check-cancelled (never-cancelled calls))) @@ -131,18 +131,18 @@ (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async render-routes-headless-to-the-worker +(t/deftest ^:async render-routes-wasm-to-the-worker (try (let [calls (atom []) job-id (uuid/next) seen (atom [])] - (await (renderer/render (test-sys calls) + (await (renderer/render (test-cfg calls) :exports [(test-params :png {:is-wasm true :job-id job-id})] :on-object (fn [object] (swap! seen conj object) nil) :check-cancelled (never-cancelled calls))) (t/is (= [{:id "w1" :path "/tmp/w1.png"}] @seen)) (t/is (some #{[:post "render"]} @calls)) - (t/is (some #{:check} @calls)) + (t/is (some #{:check-cancelled} @calls)) (t/is (not-any? #(and (vector? %) (= :goto (first %))) @calls))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) @@ -151,7 +151,7 @@ (try (let [calls (atom [])] (try - (await (renderer/render (test-sys calls) + (await (renderer/render (test-cfg calls) :exports [(dissoc (test-params :png nil) :objects)] :on-object (fn [_] nil) :check-cancelled (never-cancelled calls))) @@ -165,7 +165,7 @@ (try (let [calls (atom [])] (try - (await (renderer/render (test-sys calls) + (await (renderer/render (test-cfg calls) :exports [(test-params :png nil)] :on-object :not-a-fn :check-cancelled (never-cancelled calls))) @@ -175,30 +175,30 @@ (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async render-shares-one-worker-across-headless-renders +(t/deftest ^:async render-shares-one-worker-across-wasm-renders (try (let [calls (atom []) job-id (uuid/next) - sys (test-sys calls) + cfg (test-cfg calls) seen (atom [])] - (await (renderer/render sys - :exports [(test-params :png {:is-wasm true :job-id job-id}) - (test-params :png {:is-wasm true :job-id job-id}) - (test-params :png {:job-id job-id})] - :on-object (fn [object] (swap! seen conj object) nil) - :check-cancelled (never-cancelled calls))) + (t/is (nil? (await (renderer/render cfg + :exports [(test-params :png {:is-wasm true :job-id job-id}) + (test-params :png {:is-wasm true :job-id job-id}) + (test-params :png {:job-id job-id})] + :on-object (fn [object] (swap! seen conj object) nil) + :check-cancelled (never-cancelled calls))))) (t/is (= 1 (count (filter #{:acquire} @calls)))) (t/is (= 1 (count (filter #{:release} @calls)))) (t/is (= 3 (count @seen)))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async render-without-headless-renders-directly +(t/deftest ^:async render-without-wasm-renders-directly (try (let [calls (atom []) - sys (test-sys calls) + cfg (test-cfg calls) seen (atom [])] - (await (renderer/render sys + (await (renderer/render cfg :exports [(test-params :png nil)] :on-object (fn [object] (swap! seen conj object) nil) :check-cancelled (never-cancelled calls))) @@ -211,9 +211,9 @@ (try (let [calls (atom []) job-id (uuid/next) - sys (test-sys calls)] + cfg (test-cfg calls)] (try - (await (renderer/render sys + (await (renderer/render cfg :exports [(test-params :png {:is-wasm true :job-id job-id}) (dissoc (test-params :png nil) :objects)] :on-object (fn [_] nil) @@ -230,7 +230,7 @@ (try (let [calls (atom [])] (try - (await (renderer/render (test-sys calls) + (await (renderer/render (test-cfg calls) :exports [(test-params :png nil)] :on-objec (fn [_] nil) :check-cancelled (never-cancelled calls))) diff --git a/exporter/test/exporter_tests/renderer_wasm_test.cljs b/exporter/test/exporter_tests/renderer_wasm_test.cljs index a1ef0825c8..6bfec38445 100644 --- a/exporter/test/exporter_tests/renderer_wasm_test.cljs +++ b/exporter/test/exporter_tests/renderer_wasm_test.cljs @@ -5,9 +5,8 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns exporter-tests.renderer-wasm-test - "The wasm renderer driver: leased workers, sequential renders, - failure isolation and driver-owned cancel, against a stub pool, - stub workers and an injected zero-arg check." + "The wasm renderer driver: a single shot over the scope lease, + against a stub pool, stub workers and a positional zero-arg check." (:require ["node:events" :as events] [app.common.transit :as transit] @@ -46,92 +45,28 @@ (defn- stub-check [calls cancelled?] (fn [] - (swap! calls conj :check) + (swap! calls conj :check-cancelled) (when @cancelled? (throw (ex-info "export job was cancelled" {:type :internal :code :job-cancelled :hint "export job was cancelled"}))))) -(defn- test-sys +(defn- test-cfg [calls worker] - {:wasm-pool {:pool (stub-pool calls worker) :timeout-ms 5000}}) + {:exporter.wasm.pool/pool {:pool (stub-pool calls worker) :timeout-ms 5000}}) (defn- test-params [] {:type :png :objects [{:id "o1"}]}) -(defn- posted? - [calls] - (some #(and (vector? %) (= :post (first %))) calls)) - -(t/deftest ^:async with-scope-runs-renders-on-the-leased-worker - (try - (let [calls (atom []) - worker (stub-worker calls (render-behavior {:id "o1" :path "/tmp/o1.png"})) - seen (atom [])] - (await (render/with-scope (test-sys calls worker) - {:check-cancelled (stub-check calls (atom false)) - :on-scope (^:async fn [render*] - (await (render* (test-params) - (fn [object] (swap! seen conj object) nil))) - (await (render* (test-params) - (fn [object] (swap! seen conj object) nil))))})) - (t/is (= [{:id "o1" :path "/tmp/o1.png"} - {:id "o1" :path "/tmp/o1.png"}] - @seen)) - (t/is (= 1 (count (filter #{:release} @calls)))) - (t/testing "the check runs as each render's turn comes up" - (t/is (= 2 (count (filter #{:check} @calls))))) - (t/testing "a quiet lease leaves its worker alone" - (t/is (not-any? #{:terminate} @calls)))) - (catch :default cause - (t/is false (str "unexpected failure: " (ex-message cause)))))) - -(t/deftest ^:async cancelled-render-rejects-before-posting - (try - (let [calls (atom []) - worker (stub-worker calls (render-behavior {:id "o1"}))] - (try - (await (render/with-scope (test-sys calls worker) - {:check-cancelled (stub-check calls (atom true)) - :on-scope (fn [render*] - (render* (test-params) (fn [_] nil)))})) - (t/is false "with-scope should have rejected a cancelled render") - (catch :default cause - (t/is (= :job-cancelled (-> cause ex-data :code))) - (t/is (not (posted? @calls)))))) - (catch :default cause - (t/is false (str "unexpected failure: " (ex-message cause)))))) - -(t/deftest ^:async cancel-aborts-the-leased-worker-mid-render - (try - (let [calls (atom []) - worker (stub-worker calls (fn [_ _])) - cancelled (atom false)] - (try - (await (render/with-scope (test-sys calls worker) - {:check-cancelled (stub-check calls cancelled) - :watch-interval 20 - :on-scope (fn [render*] - ;; the cancel lands mid-render, once - ;; the worker is listening - (js/setTimeout #(reset! cancelled true) 10) - (render* (test-params) (fn [_] nil)))})) - (t/is false "with-scope should have rejected an aborted render") - (catch :default cause - (t/is (= :worker-exited (-> cause ex-data :code))) - (t/is (some #{:terminate} @calls))))) - (catch :default cause - (t/is false (str "unexpected failure: " (ex-message cause)))))) - (t/deftest ^:async render-leases-and-releases-for-one-render (try (let [calls (atom []) worker (stub-worker calls (render-behavior {:id "o1"})) seen (atom [])] - (await (render/render (test-sys calls worker) + (await (render/render (test-cfg calls worker) (test-params) (fn [object] (swap! seen conj object) nil) (stub-check calls (atom false)))) @@ -140,36 +75,19 @@ (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async render-without-check-wires-no-cancel +(t/deftest ^:async render-without-check-rejects-before-leasing (try (let [calls (atom []) - worker (stub-worker calls (render-behavior {:id "o1"})) - seen (atom [])] - (await (render/render (test-sys calls worker) - (test-params) - (fn [object] (swap! seen conj object) nil) - nil)) - (t/is (= [{:id "o1"}] @seen)) - (t/is (not-any? #{:terminate} @calls))) - (catch :default cause - (t/is false (str "unexpected failure: " (ex-message cause)))))) - -(t/deftest ^:async failed-render-does-not-break-the-chain - (try - (let [calls (atom []) - worker (stub-worker calls (render-behavior {:id "o1"})) - seen (atom [])] - (await (render/with-scope (test-sys calls worker) - {:check-cancelled (stub-check calls (atom false)) - :on-scope (^:async fn [render*] - (try - (await (render* (test-params) - (fn [_] (throw (ex-info "on-object broke" {}))))) - (catch :default _cause - (swap! calls conj :first-failed))) - (await (render* (test-params) - (fn [object] (swap! seen conj object) nil))))})) - (t/is (some #{:first-failed} @calls)) - (t/is (= [{:id "o1"}] @seen))) + worker (stub-worker calls (render-behavior {:id "o1"}))] + (try + (await (render/render (test-cfg calls worker) + (test-params) + (fn [_] nil) + nil)) + (t/is false "render should have rejected a missing check") + (catch :default cause + (t/is (= :assertion (-> cause ex-data :type))) + (t/is (= :check-cancelled-missing (-> cause ex-data :code))) + (t/is (not-any? #{:acquire} @calls))))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) diff --git a/exporter/test/exporter_tests/runner.cljs b/exporter/test/exporter_tests/runner.cljs index a827f182a5..b5f200a129 100644 --- a/exporter/test/exporter_tests/runner.cljs +++ b/exporter/test/exporter_tests/runner.cljs @@ -11,6 +11,7 @@ [clojure.string :as str] [clojure.tools.cli :refer [parse-opts]] [exporter-tests.api-test] + [exporter-tests.browser-scope-test] [exporter-tests.browser-test] [exporter-tests.consumer-config-test] [exporter-tests.consumer-plan-test] @@ -24,6 +25,7 @@ [exporter-tests.system-test] [exporter-tests.wasm-pool-service-test] [exporter-tests.wasm-pool-test] + [exporter-tests.wasm-scope-test] [exporter-tests.worker-test] [goog.object :as gobj])) @@ -37,7 +39,8 @@ (println (str "UNHANDLED-REJECTION: " reason)))) (def test-namespaces - ['exporter-tests.browser-test + ['exporter-tests.browser-scope-test + 'exporter-tests.browser-test 'exporter-tests.consumer-plan-test 'exporter-tests.jobs-test 'exporter-tests.main-test @@ -51,7 +54,8 @@ 'exporter-tests.shell-test 'exporter-tests.system-test 'exporter-tests.wasm-pool-service-test - 'exporter-tests.wasm-pool-test]) + 'exporter-tests.wasm-pool-test + 'exporter-tests.wasm-scope-test]) (assert (every? find-ns-obj test-namespaces) "test-namespaces contains a namespace that isn't required in runner.cljs") diff --git a/exporter/test/exporter_tests/wasm_pool_service_test.cljs b/exporter/test/exporter_tests/wasm_pool_service_test.cljs index 01aca5a01b..dd74f3310f 100644 --- a/exporter/test/exporter_tests/wasm_pool_service_test.cljs +++ b/exporter/test/exporter_tests/wasm_pool_service_test.cljs @@ -6,11 +6,10 @@ (ns exporter-tests.wasm-pool-service-test "The wasm worker pool service: lifecycle through the system, the - lend protocol, the render protocol, and the creation handshake, - against fake workers." + lend protocol analogy to a transaction (return on success, destroy + on failure), and the creation handshake, against fake workers." (:require ["node:events" :as events] - [app.common.transit :as transit] [cljs.test :as t :include-macros true] [exporter.utils.system :as system] [exporter.wasm.pool :as pool])) @@ -34,15 +33,6 @@ (swap! calls conj :create) (js/Promise.resolve (fake-worker calls behavior)))) -(defn- render-behavior - [payload] - (fn [^js emitter _msg] - (js/setTimeout - (fn [] - (.emit emitter "message" #js {:type "object" :payload (transit/encode-str payload)}) - (.emit emitter "message" #js {:type "done"})) - 0))) - (defn- silent-behavior [_emitter _msg]) @@ -67,74 +57,68 @@ (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async with-worker-lends-and-reclaims +(t/deftest ^:async run-lends-and-reclaims (try (let [calls (atom []) sys (await (system/init (test-config calls silent-behavior 5000))) pool (pool-of sys) seen (atom nil) - result (await (pool/with-worker pool (fn [worker] (reset! seen worker) :used)))] + result (await (pool/run pool (fn [cfg] (reset! seen cfg) :used)))] (t/is (= :used result)) - (t/is (some? (unchecked-get @seen "__id"))) + (t/is (some? (unchecked-get (:exporter.wasm.pool/worker @seen) "__id"))) (t/testing "a second use reuses the returned worker: nothing is created again" - (await (pool/with-worker pool (fn [_] :again))) + (await (pool/run pool (fn [_] :again))) (t/is (= 1 (count (filter #{:create} @calls))))) (await (system/halt sys))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async render-on-delivers-objects-and-done +(t/deftest ^:async run-reads-the-pool-from-cfg (try - (let [calls (atom []) - sys (await (system/init (test-config calls - (render-behavior {:id "o1" :path "/tmp/o1.png"}) - 5000))) - seen (atom [])] - (t/is (nil? (await (pool/with-worker (pool-of sys) - (fn [worker] - (pool/render-on worker - {} - (fn [object] (swap! seen conj object)) - {:timeout-ms 5000})))))) - (t/is (= [{:id "o1" :path "/tmp/o1.png"}] @seen)) + (let [calls (atom []) + sys (await (system/init (test-config calls silent-behavior 5000))) + seen (atom nil) + result (await (pool/run sys + (fn [cfg] (reset! seen cfg) :used)))] + (t/is (= :used result)) + (t/is (some? (unchecked-get (:exporter.wasm.pool/worker @seen) "__id"))) (await (system/halt sys))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) -(t/deftest ^:async render-on-rejects-cancelled-upfront +(t/deftest ^:async run-threads-extra-args-to-the-body + (try + (let [calls (atom []) + sys (await (system/init (test-config calls silent-behavior 5000)))] + (t/is (= [:a :b] (await (pool/run (pool-of sys) + (fn [_ x y] [x y]) + :a :b)))) + (await (system/halt sys))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async run-rejects-cfg-without-pool + (try + (try + (await (pool/run {:timeout-ms 5000} (fn [_] :used))) + (t/is false "run should have rejected a cfg without pool") + (catch :default cause + (t/is (= :invalid-pool-cfg (-> cause ex-data :code))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async run-destroys-the-worker-on-body-failure (try (let [calls (atom []) sys (await (system/init (test-config calls silent-behavior 5000)))] (try - (await (pool/with-worker (pool-of sys) - (fn [worker] - (pool/render-on worker - {} - (fn [_]) - {:check-cancelled (fn [] - (throw (ex-info "export job was cancelled" - {:code :job-cancelled}))) - :timeout-ms 5000})))) - (t/is false "render-on should have rejected a cancelled render") + (await (pool/run (pool-of sys) + (fn [_] (throw (ex-info "body broke" {}))))) + (t/is false "run should have rejected a failed body") (catch :default cause - (t/is (= :job-cancelled (-> cause ex-data :code))) - (t/is (not-any? #(and (vector? %) (= :post (first %))) @calls)))) - (await (system/halt sys))) - (catch :default cause - (t/is false (str "unexpected failure: " (ex-message cause)))))) - -(t/deftest ^:async silent-worker-times-out-and-is-terminated - (try - (let [calls (atom []) - sys (await (system/init (test-config calls silent-behavior 50)))] - (try - (await (pool/with-worker (pool-of sys) - (fn [worker] - (pool/render-on worker {} (fn [_]) {:timeout-ms 50})))) - (t/is false "render-on should have rejected a silent worker") - (catch :default cause - (t/is (= :render-timeout (-> cause ex-data :code))) - (t/is (some #{:terminate} @calls)))) + (t/is (= "body broke" (ex-message cause))) + (t/is (some #{:terminate} @calls)) + (t/is (not-any? #{:release} @calls)))) (await (system/halt sys))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) diff --git a/exporter/test/exporter_tests/wasm_scope_test.cljs b/exporter/test/exporter_tests/wasm_scope_test.cljs new file mode 100644 index 0000000000..5bc5ef3802 --- /dev/null +++ b/exporter/test/exporter_tests/wasm_scope_test.cljs @@ -0,0 +1,199 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns exporter-tests.wasm-scope-test + "The headless lease: one shared worker, sequential renders, failure + isolation and driver-owned cancel, against a stub pool and stub + workers. Task keys ride in cfg under `:exporter.renderer/*`." + (:require + ["node:events" :as events] + [app.common.transit :as transit] + [cljs.test :as t :include-macros true] + [exporter.wasm.scope :as scope])) + +(defn- stub-worker + [calls behavior] + (let [^js emitter (events/EventEmitter.)] + (unchecked-set emitter "postMessage" + (fn [msg] + (swap! calls conj [:post (unchecked-get msg "type")]) + (behavior emitter msg))) + (unchecked-set emitter "terminate" + (fn [] + (swap! calls conj :terminate) + (.emit emitter "exit" 1) + (js/Promise.resolve nil))) + emitter)) + +(defn- render-behavior + [payload] + (fn [^js emitter _msg] + (js/setTimeout + (fn [] + (.emit emitter "message" #js {:type "object" :payload (transit/encode-str payload)}) + (.emit emitter "message" #js {:type "done"})) + 0))) + +(defn- silent-behavior + [_emitter _msg]) + +(defn- stub-pool + [calls worker] + #js {:acquire (fn [] (js/Promise.resolve worker)) + :release (fn [_] (swap! calls conj :release) (js/Promise.resolve nil)) + :destroy (fn [_] (swap! calls conj :destroy) (js/Promise.resolve nil))}) + +(defn- stub-check + [calls cancelled?] + (fn [] + (swap! calls conj :check-cancelled) + (when @cancelled? + (throw (ex-info "export job was cancelled" + {:type :internal + :code :job-cancelled + :hint "export job was cancelled"}))))) + +(defn- test-cfg + ([calls worker on-object cancelled?] + (test-cfg calls worker on-object cancelled? nil)) + ([calls worker on-object cancelled? watch-interval] + (cond-> {:exporter.wasm.pool/pool {:pool (stub-pool calls worker) :timeout-ms 5000} + :exporter.renderer/on-object on-object + :exporter.renderer/check-cancelled (stub-check calls cancelled?)} + watch-interval (assoc :exporter.wasm.scope/watch-interval watch-interval)))) + +(defn- test-params + [] + {:type :png + :objects [{:id "o1"}]}) + +(defn- posted? + [calls] + (some #(and (vector? %) (= :post (first %))) calls)) + +(t/deftest ^:async run-shares-one-worker-across-renders + (try + (let [calls (atom []) + worker (stub-worker calls (render-behavior {:id "o1" :path "/tmp/o1.png"})) + seen (atom []) + cfg (test-cfg calls worker + (fn [object] (swap! seen conj object) nil) + (atom false))] + (await (scope/run cfg + (^:async fn [render-fn] + (await (render-fn (test-params))) + (await (render-fn (test-params)))))) + (t/is (= [{:id "o1" :path "/tmp/o1.png"} + {:id "o1" :path "/tmp/o1.png"}] + @seen)) + (t/is (= 1 (count (filter #{:release} @calls)))) + (t/testing "the check runs as each render's turn comes up" + (t/is (= 2 (count (filter #{:check-cancelled} @calls))))) + (t/testing "a quiet lease leaves its worker alone" + (t/is (not-any? #{:terminate} @calls)))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async cancelled-render-rejects-before-posting + (try + (let [calls (atom []) + worker (stub-worker calls (render-behavior {:id "o1"})) + cfg (test-cfg calls worker (fn [_] nil) (atom true))] + (try + (await (scope/run cfg (fn [render-fn] (render-fn (test-params))))) + (t/is false "run should have rejected a cancelled render") + (catch :default cause + (t/is (= :job-cancelled (-> cause ex-data :code))) + (t/is (not (posted? @calls)))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async cancel-aborts-the-leased-worker-mid-render + (try + (let [calls (atom []) + worker (stub-worker calls silent-behavior) + cancelled (atom false) + cfg (test-cfg calls worker (fn [_] nil) cancelled 20)] + (try + (await (scope/run cfg + (fn [render-fn] + ;; the cancel lands mid-render, once + ;; the worker is listening + (js/setTimeout #(reset! cancelled true) 10) + (render-fn (test-params))))) + (t/is false "run should have rejected an aborted render") + (catch :default cause + (t/is (= :worker-exited (-> cause ex-data :code))) + (t/is (some #{:terminate} @calls))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async failed-render-does-not-break-the-chain + (try + (let [calls (atom []) + worker (stub-worker calls (render-behavior {:id "o1"})) + seen (atom []) + thrown (atom false) + cfg (test-cfg calls worker + (fn [object] + (if (compare-and-set! thrown false true) + (throw (ex-info "on-object broke" {})) + (do (swap! seen conj object) nil))) + (atom false))] + (await (scope/run cfg + (^:async fn [render-fn] + (try + (await (render-fn (test-params))) + (catch :default _cause + (swap! calls conj :first-failed))) + (await (render-fn (test-params)))))) + (t/is (some #{:first-failed} @calls)) + (t/is (= [{:id "o1"}] @seen))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async render-on-without-lease-raises + (try + (try + (await (scope/render-on {:exporter.renderer/on-object (fn [_] nil) + :exporter.renderer/check-cancelled (fn [] nil) + :exporter.wasm.pool/pool {:pool nil :timeout-ms 50}} + (test-params))) + (t/is false "render-on should have rejected a missing lease") + (catch :default cause + (t/is (= :worker-not-leased (-> cause ex-data :code))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async render-on-without-timeout-raises + (try + (try + (await (scope/render-on {:exporter.wasm.pool/worker #js {} + :exporter.renderer/on-object (fn [_] nil) + :exporter.renderer/check-cancelled (fn [] nil) + :exporter.wasm.pool/pool {:pool nil}} + (test-params))) + (t/is false "render-on should have rejected a missing timeout") + (catch :default cause + (t/is (= :assertion (-> cause ex-data :type))) + (t/is (= :timeout-ms-missing (-> cause ex-data :code))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async silent-worker-times-out-and-is-terminated + (try + (let [calls (atom []) + worker (stub-worker calls silent-behavior) + cfg (-> (test-cfg calls worker (fn [_] nil) (atom false)) + (assoc-in [:exporter.wasm.pool/pool :timeout-ms] 50))] + (try + (await (scope/run cfg (fn [render-fn] (render-fn (test-params))))) + (t/is false "run should have rejected a silent worker") + (catch :default cause + (t/is (= :render-timeout (-> cause ex-data :code))) + (t/is (some #{:terminate} @calls))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause))))))