🐛 Fix the review findings of the exporter worker

Await the wasm callback chain before settling a render, so a
multi-object export cannot close its zip with entries still pending.
Hand a payload popped during shutdown back to its queue instead of
running an export the stop is about to cut, and let stop wait for the
poller turn before quitting the connections. Destroy the zip write
stream on the failure paths: it leaked one descriptor per cancelled or
failed job, and releasing the temp path does not close it.

Gate complete-job on the tenant in a single read at the entry, so a
foreign row no longer reveals its status. Normalize the exporter tenant
before validating it: an upper-case value derived another key and
popped another queue. Share the queue payload shape between the
dispatcher and the worker in app.common.types.job, require the export
suffix in the contract, and validate the payload before decoding it.

Count dangling promise rejections as failures in the exporter test
runner, release the export flag when the persist fails, and pin the
progress answer of a missing job. Memory, README and the pool docstring
stop describing behaviour the code no longer has.

AI-assisted-by: deepseek-v4.1-flash
This commit is contained in:
Andrey Antukh 2026-11-04 18:52:23 +00:00
parent 676701f74c
commit 2d80945965
27 changed files with 558 additions and 130 deletions

View File

@ -70,7 +70,7 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i
- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.
- The `:exporter` queue has no runner in the backend: its consumer is the exporter process (a separate Node worker; see `mem:exporter/core`). The backend serves it through `app.rpc.management.jobs`, shared-key authenticated (`X-Shared-Key: exporter[:<tenant>] <key>`, the key the exporter derives from the same secret with a per-tenant salt; the bare `exporter` id is the legacy form): `claim-job {job-id scheduled-at}` (exact `scheduled-at` match; a stale payload answers `{:action :skip :status ...}`; the row's params come back decoded with the job-def, so the worker gets typed values — a row whose def is gone answers the stored shapes), `report-job-progress {job-id progress {stage counters}}` (the worker owns the milestone words; the substrate validates the shape), `create-job-session {job-id}` (a real `http_session_v2` of the owner profile, so the assets fetch as the user; TTL `:job-session-ttl` default 1h), `complete-job` (the params form carries `result`/`resource-id`; the artifact form carries a multipart `content` part, the single source of `filename`/`mtype`, declared with `::rpc/multipart-only-params` so a transit body cannot name a server path — path confined by `media.v/check-upload-path`, allowlist `common.media/export-artifact-types`, cap `:exporter-max-result-size`; the checks run before any bytes are stored and a lost race stores nothing) and `fail-job {job-id error session-id}`. Every method answers `{:action :skip :status ...}` on a row it cannot write and reports the row status, and a settle named with `session-id` closes that render session whatever the write says — only when the session belongs to the job's `profile-id`. Every method also checks the row belongs to the calling exporter's tenant (suffix of a per-tenant key id, own tenant for the bare id): anything else answers a bare `{:action :skip}` (no status, no write, no session close), so no oracle leaks tenant rows. `report-job-progress` carries the tenant in the heartbeat write itself (`:tenant` on `jobs/heartbeat`, filtered in the touch and progress statements), so the hot path needs no row read; a filter that matches nothing raises `:interrupt` with `:reason :foreign-tenant`. The tenant is normalized to lowercase when the configuration is read (`PENPOT_TENANT=ACME` behaves as `acme`), and `app.setup.keys/derive-exporter-key` is the single definition of the exporter key on the backend (registry and middleware both call it).
- The `:exporter` queue has no runner in the backend: its consumer is the exporter process (a separate Node worker; see `mem:exporter/core`). The backend serves it through `app.rpc.management.jobs`, shared-key authenticated (`X-Shared-Key: exporter[:<tenant>] <key>`, the key the exporter derives from the same secret with a per-tenant salt; the bare `exporter` id is the legacy form): `claim-job {job-id scheduled-at}` (exact `scheduled-at` match; a stale payload answers `{:action :skip :status ...}`; the row's params come back decoded with the job-def, so the worker gets typed values — a row whose def is gone answers the stored shapes), `report-job-progress {job-id progress {stage counters}}` (the worker owns the milestone words; the substrate validates the shape), `create-job-session {job-id}` (a real `http_session_v2` of the owner profile, so the assets fetch as the user; TTL `:job-session-ttl` default 1h), `complete-job` (the params form carries `result`/`resource-id`; the artifact form carries a multipart `content` part, the single source of `filename`/`mtype`, declared with `::rpc/multipart-only-params` so a transit body cannot name a server path — path confined by `media.v/check-upload-path`, allowlist `common.media/export-artifact-types`, cap `:exporter-max-result-size`; the checks run before any bytes are stored and a lost race stores nothing) and `fail-job {job-id error session-id}`. Every method answers `{:action :skip :status ...}` on a row it cannot write and reports the row status, and a settle named with `session-id` closes that render session whatever the write says — only when the session belongs to the job's `profile-id`. Every method also checks the row belongs to the calling exporter's tenant (suffix of a per-tenant key id, own tenant for the bare id): anything else answers a bare `{:action :skip}` (no status, no write, no session close), so no oracle leaks tenant rows; `complete-job` reads the row once at entry and gates there, before the completable/owned branches, so a foreign row that already ended or has no owner leaks nothing either. `report-job-progress` carries the tenant in the heartbeat write itself (`:tenant` on `jobs/heartbeat`, filtered in the touch and progress statements), so the hot path needs no row read; a filter that matches nothing raises `:interrupt` with `:reason :foreign-tenant`. The tenant is normalized to lowercase when the configuration is read (`PENPOT_TENANT=ACME` behaves as `acme`), and `app.setup.keys/derive-exporter-key` is the single definition of the exporter key on the backend (registry and middleware both call it).
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job`, `create-export-assets-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part — binfile checks the files it reads by id, and assets checks every item on its own: an item carrying `:share-id` validates through the share-link permissions the same way `get-file-fragment` does (a logged viewer of the share is enough; the command stays authenticated, so the anonymous visitor is refused, the same as the exporter cookie it replaces) —, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` (or ten minutes when the creation says `:transient`, the single-use exports of the clipboard and the Plugin API) and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. Assets params carry an optional `:kind` (`:shapes`/`:frames`, declared by the frames dialog; absent means shapes): the worker routes on it instead of inferring the export from the items. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is then decoded with the decoder the substrate compiled from the result schema the job-def the row names declares (`decode-result`; a def without one, or a row whose job-def is gone, keeps the plain JSON the row holds: a uuid as the text it is on disk, a uuid used as a map key as a keyword). The event log still has no read surface: a client that follows a job takes its progress from the websocket.
- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow.
@ -94,7 +94,10 @@ dispatcher's guts. Verified against `app.worker/queue-key`,
- Dispatcher order: the rows are marked `scheduled` FIRST, pushed LAST
(RPUSH). A crash between the two leaves `scheduled` rows that
`reschedule-lost-jobs` re-queues; the reverse order would leave a
duplicate payload nothing deduplicates.
duplicate payload nothing deduplicates. That sweep is on its way out:
`niwinz-consumer-generalization-1` replaces it with an abort sweep
(grace 5 min, cap 24 h, `LLEN` only inside the grace), so a consumer
must never rely on a lost payload being re-dispatched.
- The `scheduled-at` in the payload must be handed back AS-IS on `claim-job`:
the claim demands an exact match and answers `{:action :skip :status ...}`
for a stale payload. That is what makes a claim idempotent and what turns

View File

@ -10,7 +10,8 @@
- `app.common.logic.*`: higher-level workflows/algorithms over files, shapes, components, variants, libraries, tokens, etc.
- `app.common.geom.*`: geometry helpers and transformations.
- `app.common.schema` / `app.common.schema.*`: Malli abstraction layer.
- `app.common.export`: asset-export job item/params schemas, the single source of truth the backend job-def and the exporter worker validate and decode against.
- `app.common.export`: asset-export job item/params schemas, the single source of truth the backend job-def and the exporter worker validate and decode against. `:suffix` is required (the renderer builds the file name with it) and `schema:params` references `schema:export-items` instead of restating the vector.
- `app.common.types.job`: the wire shape of a queue payload (`schema:queue-payload`, `[job-id scheduled-at]` as two strings), validated by the backend dispatcher before it encodes and by the exporter before it decodes.
- `app.common.math`, `app.common.time`, `app.common.uuid`, `app.common.json`, etc.: cross-runtime utilities.
- `app.common.test_helpers.*`: test builders and production-path helpers.

View File

@ -31,7 +31,7 @@ on this side: the row of the job, its cancel and its result are the backend's.
- Cancellation: a `skip` answer on any beat means the row is terminal (cancelled, aborted, settled). The runner raises `:job-cancelled` between units of work; mid-Skia the per-lease interval fires the SharedArrayBuffer signal and terminates the leased worker thread. The runner owns the local registry entry via `exporter.jobs` and releases it on settle.
- `exporter.jobs` is the **local cancel arm only**: one flag per claimed job (`register`/`mark-cancelled` returning true/nil/`cancelled?`/`release`) plus one zero-arg check constructor per job (`check-cancelled`, raising `:job-cancelled`). No terminate callbacks and no shared-array signal travel through here — the abort belongs to whoever holds the render (the per-lease interval of `exporter.wasm.scope` terminates its own worker).
- Temp files: `exporter.tmpdir/track`/`release` per job id; the `:exporter/tmpdir` init cleans orphans (aged `:job-ttl`). No redis-side store, cancel topic or abandoned-job sweep: an unclaimed-in-time row goes `aborted` by the backend's own lease GC.
- Run artifacts: `exporter.resources` (`create [tmpdir type name]`, `create-zip`/`add-to-zip`/`close-zip`); no per-file deletion timers — the tmpdir registry owns every path.
- Run artifacts: `exporter.resources` (`create [tmpdir type name]`, `create-zip`/`add-to-zip`/`close-zip`/`abort-zip`); no per-file deletion timers — the tmpdir registry owns every path. The write stream of a zip lives on the archiver and `abort-zip` destroys it: a render that fails or is cancelled would otherwise leak one descriptor per job, invisible because releasing the temp path does not close the handle. `close-zip` and `abort-zip` are idempotent with each other.
## Running it (devenv and deploy)
@ -44,15 +44,16 @@ on this side: the row of the job, its cancel and its result are the backend's.
- Lifecycle is `exporter.utils.system`: async-aware Integrant-style init/halt (`init-key`/`halt-key` multimethods, `ref`s, `weavejester.dependency` ordering, `^:async` + `cljs.core/await`, no promesa chains); tests are `^:async deftest` with `await` directly in the body.
- `exporter.main` mirrors the backend's `app.main`: a `system-config` def reading env at the wiring layer, a public `system` atom with the running map, `start`/`stop`/`restart` (`^:async`, resolving `:started`/`:stopped`/`:restarted`), a thread guard in `start` (a render worker thread runs this same bundle and boots the `exporter.wasm.worker` engine instead of the system — without it a warmed pool worker would nest pools and pollers and never post `ready`), `start-custom` as the test seam, idempotent `install-process-handlers` for the `uncaughtException`/SIGTERM/SIGINT wiring (signals drain through `shutdown`, whose `exit` is a test seam; runs on `start`, never on namespace load), `^:dev/before-load-async`/`^:dev/after-load-async` hooks bridging the promise lifecycle to shadow's `done` callbacks (a hook failure logs, never sticks the reload), and `main` as the entry point: shadow boots `exporter.main/main`. The entry leaves info logging enabled for the new tree (`setup!` on the `exporter` logger); without it the default level filters everything below fatal.
- Service config maps take plain (non-namespaced) param keys (`{:exporter.browser/pool {:max 2}}`): the top-level key already scopes the component, so namespacing the params only adds noise. Component keys and error reasons stay namespaced.
- `exporter.config` is the process configuration: `PENPOT_`-prefixed env over defaults, malli-decoded and validated; `get`/`get-internal-uri`/`management-key` (HKDF)/`flags`/`version`. `prepare-config` over an explicit env object is pure and throws `:invalid-config` (tests exercise it); only the wiring entry `load-config!` reads the live environment and exits on it.
- `exporter.config` is the process configuration: `PENPOT_`-prefixed env over defaults, malli-decoded and validated; `get`/`get-internal-uri`/`management-key` (HKDF)/`flags`/`version`. `prepare-config` over an explicit env object is pure and throws `:invalid-config` (tests exercise it); only the wiring entry `load-config!` reads the live environment and exits on it. The tenant is lowercased before validation (schema `^[a-z0-9-]+$`, the backend rule), so `PENPOT_TENANT=ACME` and `acme` derive the same management key and pop the same queue.
- `exporter.browser` is the browser pool service: the running instance is the raw pool, the public API takes it first (`exec` fails fast on a nil pool), the browser factory is injected through config for hermetic tests.
- `exporter.wasm.pool` follows the backend `db/run!` shape: `run [cfg f & args]` resolves the pool from `cfg` (the `:exporter.wasm/pool` service instance, or a raw pool), acquires a worker, applies `f` to the cfg carrying it under `:exporter.wasm.pool/worker` plus any extra args, and returns the worker — or destroys it when `f` fails, like a rollback. Pure pool mechanics only (`init`/`halt`, `terminate`, `await-ready`); the render protocol lives one level up.
- `exporter.wasm.scope` owns the lease: `run [cfg on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the check is mandatory at the lease entry and runs here when configured.
- `exporter.wasm.scope` owns the lease: `run [cfg on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the `object`/`done` messages of a render go through a serializer, so `done` settles only after the last `on-object` resolved (the worker posts every object and the settle back to back); the check is mandatory at the lease entry and runs here when configured.
- `exporter.renderer.browser` merges the bitmap/pdf/svg flows into one `render [cfg params on-object check-cancelled]` (branching by type, `on-object` always awaited, unknown types raise); `cfg` carries what the job params do not freeze (`:exporter.browser/pool`, `:base-uri`, `:public-uri`, `:svgo?`).
- `exporter.browser.scope` owns the browser batch: `run [cfg exports]` fires every export through its own checkout, all at once, bounded by the pool. No lease — a browser batch shares nothing but the pool, so fan-out only. The facade validates once, partitions by backend and joins both scopes.
- The wasm driver is `exporter.wasm.scope/run`, called from the facade; there is no `exporter.renderer.wasm` wrapper.
- `exporter.renderer` is the malli-validated facade with one entry point, the `render-fn` closure over `cfg`: it validates the task map (`:exports :on-object :check-cancelled`) and dispatches it through `render [cfg task]`, which assumes a valid task and only partitions and joins — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). The `:exporter/renderer` service builds the closure from pools plus tmpdir (refs) and static config (`:base-uri`, `:public-uri`, `:management-key`, `:image-cache-size`, `:svgo?`). The facade merges the render env (`::wrender/env`) into every wasm export's params before the lease: the worker thread reads no config and no globals. The consumer renders through the injected fn and never requires the facade. `cfg` is task-free infrastructure with process lifetime (pools, uris, key, flags, tmpdir); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`.
- `exporter.consumer.worker` is the queue poller (`read-payload`, `process`, `poll-once`, `start`/`stop` with a running flag, no promesa): K connections over the `:exporter` queue, claim → `run-export` through the injected `:renderer` and `:exporter/tmpdir` → settle before the next pop. The worker config carries the queue knobs plus both by ref; no render statics nest inside it. `stop` quits every connection and ends every loop; the temp area boots clean through the tmpdir init.
- `exporter.renderer` is the malli-validated facade with one entry point, the `render-fn` closure over `cfg`: it validates the task map (`:exports :on-object :check-cancelled`) and dispatches it through `render [cfg task]`, which assumes a valid task and only partitions and joins — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). The `:exporter/renderer` service builds the closure from pools plus tmpdir (refs) and static config (`:base-uri`, `:public-uri`, `:image-cache-size`, `:svgo?`; the render paths carry no shared key). The facade merges the render env (`::wrender/env`) into every wasm export's params before the lease: the worker thread reads no config and no globals. The consumer renders through the injected fn and never requires the facade. `cfg` is task-free infrastructure with process lifetime (pools, uris, key, flags, tmpdir); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`.
- The queue payload (`[job-id scheduled-at]`, a JSON array of two strings) is defined once in `app.common.types.job` (`schema:queue-payload`): the backend dispatcher validates before encoding and the worker validates before decoding, so a drift fails at push time. `sm/decoder` returns an invalid value untouched, so the worker checks the shape first.
- `exporter.consumer.worker` is the queue poller (`read-payload`, `process`, `poll-once`, `start`/`stop` with a running flag, no promesa): K connections over the `:exporter` queue, claim → `run-export` through the injected `:renderer` and `:exporter/tmpdir` → settle before the next pop. The worker config carries the queue knobs plus both by ref; no render statics nest inside it. `stop` falls the running flag, lets every poller finish the turn it already started (bounded by the pop timeout), and only then quits the connections; a payload popped after the flag fell goes back to its queue with `LPUSH`, because the shutdown must not start an export it is about to cut. The temp area boots clean through the tmpdir init. Do not assume the dispatcher re-queues a lost `scheduled` row: the `reschedule-lost` sweep of develop is replaced by an abort sweep in `niwinz-consumer-generalization-1`, so the payload must come back to the queue at the consumer.
- `exporter.consumer.config` holds the consumer knobs (`concurrency` floored at 1, `queue-key`, `queue-name`): the runner, the worker and `exporter.main` read it.
- `exporter.consumer.api` is the backend management client (claim/progress/session/settle, transit bodies, multipart artifact), with the same `fetch` def seam for hermetic tests.
- `exporter.wasm` is the headless render-wasm driver under Node (artifact load + headless boot, font provisioning, shape rendering); `exporter.wasm.enums` binds the generated `shared.js` the `render-wasm/build export` step emits next to it (not committed). `exporter.wasm.text`/`serialize` own the browser-free call sequencing over the shared serializers.

View File

@ -277,22 +277,20 @@
content)
(defn- completable-job
"The row a completion may land on: running (or retrying), whatever
the JSON form asks for."
[cfg job-id]
(let [job (jobs/get-job cfg job-id)]
(when (and (some? job)
(contains? #{"running" "retry"} (:status job)))
job)))
"The row a completion may land on: running (or retrying), whatever the
form asks for."
[job]
(when (contains? #{"running" "retry"} (:status job))
job))
(defn- artifact-owner
"The job an artifact completion may land on: completable AND owned,
"The row an artifact completion may land on: completable AND owned,
because the object needs an owner and a job without one owns nothing
(it may only settle without an artifact). Read before anything is
stored: a lost race with a cancel never turns the bytes of the worker
into an object."
[cfg job-id]
(let [job (completable-job cfg job-id)]
[job]
(let [job (completable-job job)]
(when (and (some? job)
(uuid? (:profile-id job)))
job)))
@ -368,43 +366,43 @@
::rpc/multipart-only-params [:content]
::rpc/allowed-shared-keys exporter-key?}
[cfg {:keys [job-id result resource-id content session-id] :as params}]
(cond
;; the multipart form stores the artifact itself: only for a job
;; that can still complete and has an owner, because the object
;; needs one
(some? content)
(if-let [job (artifact-owner cfg job-id)]
(if (tenant-ok? (::rpc/auth-key-id params) job)
(do
(check-artifact! content)
(complete-with-artifact cfg job content session-id))
;; a job of another tenant answers like a missing row, and its
;; session is not ours to close
{:action :skip})
(do
;; a completion that lost the race is not a session to keep either
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))
;; the params form: nothing to store here, the resource (if any)
;; already lives in storage
:else
(if-let [job (completable-job cfg job-id)]
(if (tenant-ok? (::rpc/auth-key-id params) job)
(if (pos? (jobs/complete cfg
:job-id job-id
:result result
:resource-id resource-id))
;; One read, one gate: a job of another tenant answers like a missing
;; row before any storage, write or session close, so neither its
;; existence nor its state leaks.
(let [row (jobs/get-job cfg job-id)]
(if-not (tenant-ok? (::rpc/auth-key-id params) row)
{:action :skip}
(cond
;; the multipart form stores the artifact itself: only for a job
;; that can still complete and has an owner, because the object
;; needs one
(some? content)
(if-let [job (artifact-owner row)]
(do
(close-job-session! cfg job-id session-id)
{:action :run})
(check-artifact! content)
(complete-with-artifact cfg job content session-id))
(do
;; a completion that lost the race is not a session to keep either
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))
{:action :skip})
(do
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))))
;; the params form: nothing to store here, the resource (if any)
;; already lives in storage
:else
(if (completable-job row)
(if (pos? (jobs/complete cfg
:job-id job-id
:result result
:resource-id resource-id))
(do
(close-job-session! cfg job-id session-id)
{:action :run})
(do
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))
(do
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))))))
;; ---- RPC METHOD: FAIL-JOB

View File

@ -11,6 +11,7 @@
[app.common.logging :as l]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.common.types.job :as ctj]
[app.config :as cf]
[app.db :as db]
[app.jobs :as jobs]
@ -100,8 +101,17 @@ RETURNING job.id, job.queue, job.name, job.created_at")
SELECT unnest(?::uuid[]), 'end', ?::jsonb")
(defn- encode-payload
"The payload the queue carries: the job id and its `scheduled-at`, in
the shape the consumer decodes (`app.common.types.job`). A drift
fails here, on the producer, instead of at the claim."
[{:keys [id scheduled-at]}]
(json/encode [(str id) (ct/format-inst scheduled-at)]))
(let [payload [(str id) (ct/format-inst scheduled-at)]]
(when-not (ctj/valid-queue-payload? payload)
(throw (ex-info "invalid queue payload"
{:type :assertion
:code :invalid-queue-payload
:payload payload})))
(json/encode payload)))
(defn- reschedule-lost-jobs
[{:keys [::db/conn ::timestamp ::wrk/tenant] :as cfg}]

View File

@ -223,8 +223,8 @@
:object-id (or object-id (uuid/next))
:type (or type :png)
:name (or name "shape")
:suffix (or suffix "")
:scale (or scale 1)}
suffix (assoc :suffix suffix)
share-id (assoc :share-id share-id)))
(defn- create-export-assets-job

View File

@ -216,6 +216,18 @@
(t/is (= {:action :skip :status "completed"} (:result out)))
(t/is (= [] (get-progresss job-id)))))
(t/deftest report-job-progress-skips-a-missing-row
;; The worker sends the beat with the id it claimed: a row that is
;; gone answers a bare skip, the same contract the claim pins.
(let [job-id (uuid/next)
out (mgmt :report-job-progress {:job-id job-id
:progress {:stage :pages
:counters {:pages {:current 1
:total 2}}}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/is (zero? (count-job-events job-id)))))
(t/deftest report-job-progress-persists-rapid-reports
(let [job-id (mk-job {})
_ (jobs/claim (make-cfg) job-id (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at)))
@ -688,6 +700,33 @@
(t/is (= "running" (:status (get-row job-id))))
(t/is (nil? (:completed-at (get-row job-id))))))
(t/deftest complete-job-skips-a-terminal-job-of-another-tenant
;; The row already ended, so the multipart form does not even reach
;; the "completable" branch: the tenant gate still answers first, and
;; the state of a foreign row never leaks.
(let [profile (mk-profile)
job-id (mk-job {:status "completed"
:profile-id (:id profile)
:tenant "other-tenant"})
owner-sess (session/create-session (get th/*system* :app.http.session/manager)
{:profile-id (:id profile)})
out (multipart-complete job-id (artifact-tempfile "penpot")
:session-id (:id owner-sess))]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/testing "nothing was stored, written or closed"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "completed" (:status (get-row job-id))))
(t/is (some? (get-session-row (:id owner-sess)))))))
(t/deftest complete-job-params-form-skips-a-terminal-job-of-another-tenant
(let [job-id (mk-job {:status "completed" :tenant "other-tenant"})
out (mgmt :complete-job {:job-id job-id
:result {:x 1}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/is (= "completed" (:status (get-row job-id))))))
(t/deftest complete-job-refuses-an-upload-outside-multipart
;; The upload map names a server path: only a request the multipart
;; parser flagged may carry one. A transit body is refused before any

View File

@ -31,7 +31,7 @@
;; a plain string on purpose, empty included: the export form sends
;; `""` when the user types no suffix, and the legacy `/api/export`
;; surface took it that way; the worker concatenates it as-is
[:suffix {:optional true} :string]
[:suffix :string]
[:scale ::sm/number]
[:share-id {:optional true} ::sm/uuid]])
@ -48,7 +48,7 @@
joins pages into one pdf); absent it is a shapes export. The worker
reads it as declared instead of inferring it from the items."
[:map {:title "export-assets-params" :closed true}
[:exports [:vector {:min 1} schema:export-item]]
[:exports schema:export-items]
[:kind {:optional true} [::sm/one-of #{:shapes :frames}]]
[:name {:optional true} ::sm/text]
[:skip-children {:optional true} ::sm/boolean]

View File

@ -0,0 +1,27 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.common.types.job
"The wire shapes of the job queue, shared by the process that pushes a
payload (the backend dispatcher) and the process that pops it (the
exporter worker). One definition on purpose: a drift between the two
fails at push time instead of at claim time."
(:require
[app.common.schema :as sm]))
(def schema:queue-payload
"The two-element JSON array the dispatcher pushes and the consumer
pops: the job id as text and the `scheduled-at` of the row when it was
pushed, in `ct/format-inst` shape. Fewer than two items, other types
or a different container is not a payload: the consumer drops it
instead of claiming an invalid job."
[:tuple :string :string])
(def valid-queue-payload?
(sm/validator schema:queue-payload))
(def decode-queue-payload
(sm/decoder schema:queue-payload sm/string-transformer))

View File

@ -52,6 +52,14 @@
:scale 2}]
(t/is (= typed (decode typed)))))
(t/deftest an-item-without-suffix-is-not-a-valid-export
;; The renderer needs the suffix always (it builds the file name), so
;; the contract requires it: the job is refused at creation instead of
;; failing at render time.
(let [item (-> stored-item decode (dissoc :suffix))]
(t/is (not (sm/valid? export/schema:params {:exports [item]})))
(t/is (sm/valid? export/schema:params {:exports [(assoc item :suffix "")]}))))
(t/deftest params-schema-owns-kind-and-options
(let [decode-params (sm/decoder export/schema:params sm/json-transformer)
params (decode-params {:exports [stored-item]

View File

@ -101,6 +101,7 @@
[common-tests.types.container-test]
[common-tests.types.fill-test]
[common-tests.types.font-test]
[common-tests.types.job-test]
[common-tests.types.modifiers-test]
[common-tests.types.objects-map-test]
[common-tests.types.organization-test]
@ -211,6 +212,7 @@
'common-tests.types.container-test
'common-tests.types.fill-test
'common-tests.types.font-test
'common-tests.types.job-test
'common-tests.types.modifiers-test
'common-tests.types.objects-map-test
'common-tests.types.organization-test

View File

@ -0,0 +1,33 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.types.job-test
"The queue payload both processes share: a two-element array of the
job id and the instant it was pushed, and nothing else."
(:require
[app.common.types.job :as ctj]
[clojure.test :as t]))
(t/deftest a-payload-is-two-strings
(t/testing "the dispatcher shape validates"
(t/is (ctj/valid-queue-payload?
["03b7300f-11e6-8131-8008-e4b09b780a0e" "2026-10-07T09:00:00Z"])))
(t/testing "the decoder keeps it a vector of two strings"
(t/is (= ["03b7300f-11e6-8131-8008-e4b09b780a0e" "2026-10-07T09:00:00Z"]
(ctj/decode-queue-payload
["03b7300f-11e6-8131-8008-e4b09b780a0e" "2026-10-07T09:00:00Z"])))))
(t/deftest anything-else-is-not-a-payload
(t/testing "an empty object is not the array the dispatcher pushes"
(t/is (not (ctj/valid-queue-payload? {}))))
(t/testing "nil is not a payload"
(t/is (not (ctj/valid-queue-payload? nil))))
(t/testing "an empty array has no job id"
(t/is (not (ctj/valid-queue-payload? []))))
(t/testing "two items of another type are not a payload"
(t/is (not (ctj/valid-queue-payload? [1 2]))))
(t/testing "a third item is not part of the contract"
(t/is (not (ctj/valid-queue-payload? ["a" "b" "c"])))))

View File

@ -58,14 +58,14 @@ packs it for the image.
| `PENPOT_PUBLIC_URI` | — | The frontend the assets render through |
| `PENPOT_INTERNAL_URI` | `public-uri` | Same network's frontend; the management API too |
| `PENPOT_REDIS_URI` | `redis://redis/0` | The queue and the rest of redis |
| `PENPOT_TENANT` | `default` | Queue prefix (workspace name in devenv) |
| `PENPOT_TENANT` | `default` | Queue prefix (workspace name in devenv); lowercased on read |
| `PENPOT_TEMPDIR` | `/tmp/penpot` | Temp area owned by the tmpdir service |
| `PENPOT_SECRET_KEY` | — | Required; the management key derives from it when no shared key is set |
| `PENPOT_EXPORTER_SHARED_KEY` | — | The shared key; derived from the secret otherwise |
| `PENPOT_EXPORTER_JOB_TTL` | `3600` | Lifetime of the temp files a job owns, in seconds |
| `PENPOT_EXPORTER_WORKER_CONCURRENCY` | `2` | `K`: pollers, running exports, pool sizes |
| `PENPOT_WASM_WORKER_POOL_MIN` | `1` | Render workers kept warm; clamped to the max |
| `PENPOT_WASM_WORKER_IDLE_TIMEOUT` | `300` | Silence before a worker is terminated, in seconds |
| `PENPOT_WASM_WORKER_IDLE_TIMEOUT` | `300` | How long an idle pooled worker is kept before eviction, in seconds |
| `PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE` | `134217728` | Per-worker image cache budget, in bytes |
## Inspecting the backend

View File

@ -50,7 +50,7 @@
[:public-uri {:optional true} ::sm/uri]
[:internal-uri {:optional true} ::sm/uri]
[:exporter-shared-key {:optional true} :string]
[:tenant {:optional true} :string]
[:tenant {:optional true} [:re #"^[a-z0-9-]+$"]]
[:flags {:optional true} [::sm/set :keyword]]
[:redis-uri {:optional true} :string]
[:tempdir {:optional true} :string]
@ -73,6 +73,15 @@
[config]
(flags/parse (:flags config)))
(defn- normalize-tenant
"Lowercase the configured tenant before it is validated, so `ACME` and
`acme` name the same tenant: the key derivation, the key id and the
queue key all compare it as an exact string, and the backend
normalizes its side too."
[config]
(cond-> config
(string? (:tenant config)) (update :tenant str/lower)))
(defn read-env
"The `PENPOT_`-prefixed variables of `env` as kebab-case keys, the
rest ignored. Takes the env object explicitly so tests pass a fake."
@ -95,6 +104,7 @@
environment."
[env]
(let [data (-> (merge defaults (d/without-nils (read-env env)))
(normalize-tenant)
(decode-config))]
(when-not (valid-config? data)
(throw (ex/error :type :internal

View File

@ -261,18 +261,23 @@
plan/sanitize-file-regex
"_"))
nil)]
(check-beat (await (beat :rendering (counter counter-kind 0 total) :force? true)))
(await (render {:exports (mapv #(cond-> (assoc % :job-id job-id)
(some? (:skip-children plan))
(assoc :skip-children (:skip-children plan)))
prepared)
:on-object append
:check-cancelled check-cancelled}))
(when-let [cause @failure]
(throw cause))
(await (rsc/close-zip zip))
(check-beat (await (beat :packaging (counter counter-kind total total) :force? true)))
resource))
(try
(check-beat (await (beat :rendering (counter counter-kind 0 total) :force? true)))
(await (render {:exports (mapv #(cond-> (assoc % :job-id job-id)
(some? (:skip-children plan))
(assoc :skip-children (:skip-children plan)))
prepared)
:on-object append
:check-cancelled check-cancelled}))
(when-let [cause @failure]
(throw cause))
(await (rsc/close-zip zip))
(check-beat (await (beat :packaging (counter counter-kind total total) :force? true)))
resource
(finally
;; A render that failed or was cancelled never reaches the close
;; above: destroying the stream is what releases the descriptor.
(rsc/abort-zip zip)))))
(defn- ^:async join-pdf
"The pages render one file per page; `pdfunite` stitches one pdf out

View File

@ -19,10 +19,11 @@
`penpot.worker.queue:<tenant>:exporter`. Corrupt payloads are logged
and dropped; a Redis hiccup is a lost turn, not a lost job: the
poller waits a bit and pops again (what was already delivered is
claimed, and the loop does not stop the process). Stopping quits
every connection and ends every loop: unlike the legacy worker,
whose loops kept failing turns after the stop, no poller survives
its handle.
claimed, and the loop does not stop the process). Stopping falls the
running flag, lets every poller finish the turn it already started,
and only then quits the connections: a payload that arrives in that
window goes back to its queue instead of starting an export the
shutdown is about to cut, and no poller survives its handle.
The worker is a system component (`:exporter.consumer/worker`): its
config carries the queue knobs (`:concurrency`, `:queue-key`, the
@ -32,6 +33,7 @@
(:require
["ioredis" :as redis]
[app.common.logging :as l]
[app.common.types.job :as ctj]
[exporter.config :as cf]
[exporter.consumer :as consumer]
[exporter.consumer.api :as api]
@ -49,12 +51,19 @@
(defn read-payload
"The JSON payload of the dispatcher: `job-id` and the `scheduled-at`
of the row when it was pushed. Anything else is a corrupt payload:
of the row when it was pushed, decoded against the shape both sides
share (`app.common.types.job`). Anything else is a corrupt payload:
none, and the caller drops it."
[payload]
(try
(let [[job-id scheduled-at] (js->clj (js/JSON.parse payload))]
{:job-id job-id :scheduled-at scheduled-at})
(let [decoded (js->clj (js/JSON.parse payload))]
;; `sm/decoder` hands an invalid value back untouched, so the shape
;; is checked first: a payload that is not the pair is dropped, not
;; turned into a claim with nil ids.
(if (ctj/valid-queue-payload? decoded)
(let [[job-id scheduled-at] (ctj/decode-queue-payload decoded)]
{:job-id job-id :scheduled-at scheduled-at})
(throw (ex-info "unexpected queue payload" {:payload decoded}))))
(catch :default cause
(l/warn :hint "corrupt queue payload dropped"
:payload (str payload) :cause cause)
@ -81,14 +90,21 @@
(defn ^:async poll-once
"One turn: pop the queue, claim what came, and wait for the settle.
An empty pop and a dropped payload both resolve; a claimed job is
awaited the whole run."
[conn key on-payload]
awaited the whole run. A payload popped after the stop flag fell goes
back to the queue: the shutdown must not start an export it is about
to cut, and the next worker (or the next start) claims it."
[handle conn key on-payload]
(let [answer (await (.blpop ^js conn key poll-timeout-s))
payload (second answer)
claim (when (some? payload) (read-payload payload))]
(when (some? claim)
(await (on-payload claim))
nil)))
payload (second answer)]
(if (and (some? payload) (not @(:running handle)))
(do
(await (.lpush ^js conn key payload))
(l/info :hint "payload handed back to the queue" :key key)
nil)
(let [claim (when (some? payload) (read-payload payload))]
(when (some? claim)
(await (on-payload claim))
nil)))))
(defn- ^:async poll-loop
"The wait of one poller, turn after turn, until the handle stops. A
@ -101,7 +117,7 @@
(loop []
(when @(:running handle)
(try
(await (poll-once conn key on-payload))
(await (poll-once handle conn key on-payload))
(catch :default cause
(l/error :hint "poller turn failed" :key key :cause cause)
(await (wait reconnect-delay-ms))))
@ -123,7 +139,8 @@
[{:keys [concurrency queue-key connect] :as cfg}]
(let [running (atom true)
inflight (atom 0)
handle {:running running :conns (atom []) :inflight inflight}
handle {:running running :conns (atom []) :loops (atom [])
:inflight inflight}
on-payload (^:async fn [claim]
(swap! inflight inc)
(try
@ -136,7 +153,7 @@
(doseq [_ (range k)]
(let [conn (open)]
(swap! (:conns handle) conj conn)
(poll-loop handle conn queue-key on-payload)))
(swap! (:loops handle) conj (poll-loop handle conn queue-key on-payload))))
handle))
(defn- ^:async drain-inflight
@ -148,14 +165,25 @@
(defn ^:async stop
"Ends the pollers, closing their connections and their loops: the
running flag falls first, so a turn that ends after the quit does
not pop again, then the in-flight runs drain within the grace
running flag falls first, so a turn that ends after the stop hands
its payload back instead of starting an export, then the pollers
finish the turn they already began (bounded by the pop timeout, so a
running export never holds the shutdown here) and only after that
the connections quit. The in-flight runs drain within the grace
period before the pools halt behind them. Idempotent: a second stop
quits nothing."
([handle] (stop handle nil))
([handle {:keys [grace-ms] :or {grace-ms default-shutdown-grace-ms}}]
([handle {:keys [grace-ms turn-ms]
:or {grace-ms default-shutdown-grace-ms
turn-ms (* 1000 (inc poll-timeout-s))}}]
(when (and (some? handle)
(compare-and-set! (:running handle) true false))
;; The blocking pop cannot be cancelled, so the wait is bounded by
;; its own timeout: after it, every poller is either idle or
;; running an export, which the drain below handles.
(await (js/Promise.race
#js [(js/Promise.allSettled @(or (:loops handle) (atom [])))
(wait turn-ms)]))
(await (js/Promise.allSettled
(mapv (fn [conn] (.quit ^js conn)) @(:conns handle))))
(let [inflight (or (:inflight handle) (atom 0))]

View File

@ -18,6 +18,7 @@
["archiver" :as arc]
["node:fs" :as fs]
["node:path" :as path]
[app.common.exceptions :as ex]
[app.common.uuid :as uuid]
[cljs.core :as c]
[cuerdas.core :as str]
@ -43,6 +44,8 @@
(let [^js zip (new arc/ZipArchive)
^js out (fs/createWriteStream (:path resource))
on-complete (or on-complete (constantly nil))
on-progress (or on-progress (constantly nil))
on-error (or on-error (constantly nil))
progress (atom 0)]
(.on zip "error" on-error)
(.on zip "end" on-complete)
@ -51,6 +54,9 @@
num (swap! progress inc)]
(on-progress {:done num :filename name}))))
(.pipe zip out)
;; The stream belongs to the archiver: `abort-zip` needs it to close
;; the descriptor of a run that never reaches the happy path.
(unchecked-set zip "__out" out)
zip))
(defn add-to-zip
@ -73,4 +79,19 @@
(reject cause))]
(.on ^js zip "close" (fn [] (done)))
(.on ^js zip "error" (fn [cause] (fail cause)))
(.finalize ^js zip)))))))
(.finalize ^js zip)))))
(unchecked-set zip "__closed" true)
nil))
(defn abort-zip
"Abort `zip` and destroy the write stream behind it, so a render that
fails or is cancelled does not leave the descriptor open: releasing
the temp path removes the name, not the handle. Idempotent, and safe
after `close-zip` (the archiver is then already settled)."
[zip]
(when (and (some? zip) (not (unchecked-get zip "__closed")))
(ex/ignoring (.abort ^js zip))
(when-let [out (unchecked-get zip "__out")]
(ex/ignoring (.destroy ^js out)))
(unchecked-set zip "__closed" true))
nil)

View File

@ -12,9 +12,10 @@
behave the same way for both render backends. A worker is expensive
to build (it boots its own render-wasm module), hence the pooling.
Acquisition is not capped: the admission scheduler is the
backpressure, and the idle watchdog guarantees a wedged worker gives
its slot back.
Acquisition is not capped beyond the pool maximum: the K pollers of
the worker are the real concurrency, and a worker that stops answering
is cut by the silence watchdog of `exporter.wasm.scope`, which
terminates it and frees the slot.
Workers run the same bundle as the main thread. There is always at
least one worker, since a headless render has nowhere else to go.

View File

@ -38,7 +38,12 @@
[^js worker params cancel-buffer on-object timeout-ms]
(js/Promise.
(fn [resolve reject]
(let [timer (volatile! nil)]
(let [timer (volatile! nil)
;; The worker posts every object and `done` back to back, so
;; the message handler must wait for `on-object` before it
;; settles the render: the callback may still be appending to
;; the artifact. A rejected callback rejects the render.
chain (serializer)]
(letfn [(disarm []
(when-let [t @timer]
(js/clearTimeout t)
@ -82,12 +87,20 @@
;; A failure while the main thread handles the object
;; (moving the file, appending to the zip) has to end
;; the render too, or nothing ever settles this promise.
"object" (try
(on-object (t/decode-str (unchecked-get data "payload")))
(catch :default cause
(cleanup)
(reject cause)))
"done" (do (cleanup) (resolve nil))
"object" (-> (chain (fn []
(on-object (t/decode-str (unchecked-get data "payload")))))
(.catch (fn [cause]
(cleanup)
(reject cause))))
;; `done` queues behind every object of this render, so
;; the settle never overtakes a pending append.
"done" (-> (chain (fn [] nil))
(.then (fn [_]
(cleanup)
(resolve nil)))
(.catch (fn [cause]
(cleanup)
(reject cause))))
"error" (do (cleanup)
(reject (ex/error :type :internal
:code (or (some-> (unchecked-get data "code") keyword)

View File

@ -23,6 +23,7 @@
[exporter.consumer.api :as api]
[exporter.jobs :as jobs]
[exporter.renderer :as renderer]
[exporter.resources :as rsc]
[exporter.shell :as shell]))
;; ---- THE MANAGEMENT FAKE
@ -361,6 +362,34 @@
(finally
(restore-fetch)))))
(t/deftest ^:async a-failed-multi-run-aborts-the-zip
;; A render that fails after an object landed must destroy the write
;; stream: removing the temp path does not close the handle.
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
original rsc/abort-zip
aborted (atom 0)
render (fn [{:keys [on-object]}]
(let [path (shell/tempfile (os/tmpdir) :prefix "penpot.abort."
:suffix ".png")]
(-> (.then (fsp/writeFile path "rendered!")
(fn [_] (on-object {:path path :filename "one.png"})))
(.then (fn [_] (js/Promise.reject (ex-info "render boom" {})))))))
job-id (uuid/next)]
(set! rsc/abort-zip (fn [zip] (swap! aborted inc) (original zip)))
(try
(await (consumer/run-export render (os/tmpdir) {:job-id job-id}
{:exports [(one-shape "one") (one-shape "two")]
:name "the export"}))
(t/testing "the run settled as failed"
(t/is (= "fail-job" (last (steps-of)))))
(t/testing "the zip was aborted, not left open"
(t/is (= 1 @aborted)))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(set! rsc/abort-zip original)
(restore-fetch)))))
(t/deftest ^:async cancel-before-the-render-settles-cancelled
(let [restore-fetch (fake-fetch
(fn [method _body]

View File

@ -79,14 +79,19 @@
(worker/read-payload (encoded-claim job-id
"2026-10-07T09:00:00Z"))))))
(t/testing "a corrupt payload is dropped, not thrown at the loop"
(t/is (nil? (worker/read-payload "{not a payload")))))
(t/is (nil? (worker/read-payload "{not a payload"))))
(t/testing "valid JSON of another shape is corrupt too"
(doseq [payload ["{}" "null" "[]" "\"foo\"" "[1,2]" "[\"a\",\"b\",\"c\"]"]]
(t/is (nil? (worker/read-payload payload)) payload))))
(t/deftest ^:async poll-once-awaits-what-it-claims
(try
(let [queue (atom [(encoded-claim (uuid/next) "2026-10-07T09:00:00Z")])
conn (fake-conn queue)
handle {:running (atom true)}
handled (atom [])]
(await (worker/poll-once conn
(await (worker/poll-once handle
conn
"the.queue.key"
(fn [claim]
(swap! handled conj claim)
@ -95,15 +100,42 @@
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async poll-once-hands-back-a-payload-popped-after-the-stop
;; The pop cannot be cancelled, so the payload it returns during the
;; shutdown goes back to the queue: the export is not started by a
;; process that is about to halt.
(try
(let [payload (encoded-claim (uuid/next) "2026-10-07T09:00:00Z")
queue (atom [payload])
conn (fake-conn queue)
pushed (atom [])
handle {:running (atom false)}]
(set! (.-lpush conn)
(fn [key value]
(swap! pushed conj [key value])
(js/Promise.resolve 1)))
(await (worker/poll-once handle conn "the.queue.key"
(fn [_claim]
(t/is false "a stopping poller must not claim"))))
(t/testing "the payload went back to the same queue, untouched"
(t/is (= [["the.queue.key" payload]] @pushed)))
(t/testing "and nothing was handled"
(t/is (empty? @queue))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async poll-once-drops-empty-pops-and-corrupt-payloads
(try
(let [queue (atom ["{corrupt"])
conn (fake-conn queue)]
(await (worker/poll-once conn
(let [queue (atom ["{corrupt"])
conn (fake-conn queue)
handle {:running (atom true)}]
(await (worker/poll-once handle
conn
"the.queue.key"
(fn [_claim]
(t/is false "corrupt must not be handled"))))
(await (worker/poll-once conn
(await (worker/poll-once handle
conn
"the.queue.key"
(fn [_claim]
(t/is false "a corrupt pop has nothing to handle"))))
@ -188,10 +220,10 @@
c))
handle (await (worker/start (test-cfg {:concurrency 2 :connect conn})))]
(t/is (= 2 (count @opened)))
(await (worker/stop handle))
(await (worker/stop handle {:turn-ms 20}))
(t/is (= 2 (count @quit)))
(t/testing "a second stop is a no-op"
(await (worker/stop handle))
(await (worker/stop handle {:turn-ms 20}))
(t/is (= 2 (count @quit)))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))

View File

@ -10,7 +10,8 @@
schema, with the management key derived the way the backend expects."
(:require
[cljs.test :as t :include-macros true]
[exporter.config :as cf]))
[exporter.config :as cf]
[exporter.consumer.config :as ccfg]))
(t/deftest read-env-keeps-only-the-prefixed-keys
(let [env (cf/read-env #js {"PENPOT_SECRET_KEY" "s"
@ -41,6 +42,29 @@
(t/is (some? cause))
(t/is (= :invalid-config (:code (ex-data cause))))))
(t/deftest prepare-config-normalizes-the-tenant-to-lowercase
;; The backend lowercases the tenant it reads; a worker configured with
;; another spelling would derive another key and pop another queue.
(let [config (cf/prepare-config #js {"PENPOT_SECRET_KEY" "s"
"PENPOT_TENANT" "ACME"})]
(t/is (= "acme" (:tenant config)))
(t/testing "the key derived from the normalized value is the same"
(t/is (= (cf/derive-management-key "s" "acme")
(cf/derive-management-key "s" (:tenant config)))))
(t/testing "and the queue key lands where the dispatcher pushes"
(t/is (= "penpot.worker.queue:acme:exporter"
(ccfg/queue-key (:tenant config)))))))
(t/deftest prepare-config-refuses-a-tenant-outside-the-label-form
(let [cause (try
(cf/prepare-config #js {"PENPOT_SECRET_KEY" "s"
"PENPOT_TENANT" "acme!"})
nil
(catch :default cause
cause))]
(t/is (some? cause))
(t/is (= :invalid-config (:code (ex-data cause))))))
(t/deftest derive-management-key-matches-the-tenant-formula
;; the backend derives the same key from the same secret and tenant
;; (HKDF blake2b512 over "exporter:<tenant>", 32 bytes, base64url):

View File

@ -54,6 +54,40 @@
(finally
(await (fsp/rm dir #js {:recursive true :force true}))))))
(t/deftest ^:async abort-zip-destroys-the-write-stream
(let [dir (test-area)
_ (await (fsp/mkdir dir #js {:recursive true}))
resource (rsc/create dir :zip "the export")
zip (rsc/create-zip :resource resource)
^js out (unchecked-get zip "__out")]
(try
(t/testing "the stream is open before the abort"
(t/is (false? (.-destroyed out))))
(rsc/abort-zip zip)
(t/testing "the abort releases the descriptor"
(t/is (true? (.-destroyed out))))
(t/testing "a second abort is harmless"
(t/is (nil? (rsc/abort-zip zip)))
(t/is (true? (.-destroyed out))))
(finally
(await (fsp/rm dir #js {:recursive true :force true}))))))
(t/deftest ^:async abort-zip-after-close-is-a-no-op
(let [dir (test-area)
_ (await (fsp/mkdir dir #js {:recursive true}))
resource (rsc/create dir :zip "the export")
member (path/join dir "member.txt")
_ (await (fsp/writeFile member "member!"))
zip (rsc/create-zip :resource resource)]
(try
(rsc/add-to-zip zip member "member.txt")
(await (rsc/close-zip zip {:timeout-ms 1000}))
(t/testing "an abort after the close does not disturb the artifact"
(t/is (nil? (rsc/abort-zip zip)))
(t/is (pos? (.-size (await (fsp/stat (:path resource)))))))
(finally
(await (fsp/rm dir #js {:recursive true :force true}))))))
(defn- fake-zip
"An event-emitter stand-in for `close-zip`: records its listeners and
runs `on-finalize` with them once finalized."

View File

@ -39,10 +39,21 @@
;; The tests that ride the mock http must never leave a rejected promise
;; dangling: a dangling one kills the whole run with ERR_UNHANDLED_REJECTION
;; and without any way to point at the chain that broke. Log and keep going.
;; and without any way to point at the chain that broke. The run keeps
;; going so it can report every test, but the summary counts each one as
;; a failure: a broken chain is a regression, not a log line.
(def ^:private dangling-rejections (atom []))
(js/process.on "unhandledRejection"
(fn [reason _promise]
(println (str "UNHANDLED-REJECTION: " reason))))
(swap! dangling-rejections conj reason)))
(defn- rejection-text
[reason]
(let [stack (unchecked-get reason "stack")]
(if (string? stack)
(str reason "\n" stack)
(str reason))))
(def test-namespaces
['exporter-tests.browser-scope-test
@ -194,6 +205,11 @@
(partial merge-with +)
(:report-counters (t/get-current-env))))
(fn []
(let [dangling @dangling-rejections]
(when (seq dangling)
(doseq [reason dangling]
(println (str "UNHANDLED-REJECTION: " (rejection-text reason))))
(vswap! summary update :fail + (count dangling))))
(t/report @summary)
(t/report (assoc @summary :type :end-run-tests)))]))))

View File

@ -74,6 +74,18 @@
[calls]
(some #(and (vector? %) (= :post (first %))) calls))
(defn- multi-behavior
"Posts every object back to back and then the settle, the way the real
worker does: nothing waits for the main thread between messages."
[payloads]
(fn [^js emitter _msg]
(js/setTimeout
(fn []
(doseq [payload payloads]
(.emit emitter "message" #js {:type "object" :payload (transit/encode-str payload)}))
(.emit emitter "message" #js {:type "done"}))
0)))
(t/deftest ^:async run-shares-one-worker-across-renders
(try
(let [calls (atom [])
@ -155,6 +167,47 @@
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async a-slow-on-object-settles-before-the-render
;; The worker posts every object and `done` in one burst; the settle
;; must wait for the callbacks, or a multi-object export can close its
;; artifact with entries still pending.
(try
(let [calls (atom [])
worker (stub-worker calls (multi-behavior [{:id "o1"} {:id "o2"}]))
seen (atom [])
cfg (test-cfg calls worker
(fn [object]
;; resolves on a later tick, like a beat does
(js/Promise.
(fn [resolve]
(js/setTimeout
(fn []
(swap! seen conj (:id object))
(resolve nil))
5))))
(atom false))]
(await (scope/run cfg (fn [render-fn] (render-fn (test-params)))))
(t/testing "every object landed before the render resolved"
(t/is (= ["o1" "o2"] @seen))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async a-rejected-on-object-rejects-the-render
(try
(let [calls (atom [])
worker (stub-worker calls (multi-behavior [{:id "o1"}]))
cfg (test-cfg calls worker
(fn [_object]
(js/Promise.reject (ex-info "append broke" {})))
(atom false))]
(try
(await (scope/run cfg (fn [render-fn] (render-fn (test-params)))))
(t/is false "a rejected on-object must reject the render")
(catch :default cause
(t/is (= "append broke" (ex-message cause))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async render-on-without-lease-raises
(try
(try

View File

@ -94,10 +94,15 @@
:hint (tr "errors.export-failed")))))
(defn normalize-export
[{:keys [object-id name] :as export}]
(assoc export :name (if (str/blank? name)
(str object-id)
name)))
"The export item of the shared contract: the name filled from the
object id when blank, and the suffix defaulted to empty for items
created before the contract required it."
[{:keys [object-id name suffix] :as export}]
(assoc export
:name (if (str/blank? name)
(str object-id)
name)
:suffix (or suffix "")))
(defn- normalize-exports
[exports]
@ -534,11 +539,13 @@
(let [ws-conn (:ws-conn state)
file-ids (export-file-ids [export])]
(swap! st/ongoing-tasks conj :export)
(rx/concat
(dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms)
(rx/of (warn-unsaved-changes file-ids))
(export-stream! ws-conn [export] :export-shapes
{:is-wasm (wasm-export-enabled? state)}))))))))
(->> (rx/concat
(dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms)
(rx/of (warn-unsaved-changes file-ids))
(export-stream! ws-conn [export] :export-shapes
{:is-wasm (wasm-export-enabled? state)}))
(rx/finalize (fn []
(swap! st/ongoing-tasks disj :export))))))))))
(defn- export-file-page-ids
[state exports]
@ -575,7 +582,7 @@
:type type
:scale (or scale 1)
:name name
:suffix suffix})
:suffix (or suffix "")})
exports))
(defn- request-local-export-wasm
@ -676,14 +683,16 @@
;; export is a shapes one
kind (if (= :export-frames cmd) :frames :shapes)]
(swap! st/ongoing-tasks conj :export)
(rx/concat
(dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms)
(rx/of (warn-unsaved-changes file-ids))
(export-stream! ws-conn exports cmd
{:force-multiple true
:name name
:kind kind
:is-wasm (wasm-export-enabled? state)}))))))
(->> (rx/concat
(dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms)
(rx/of (warn-unsaved-changes file-ids))
(export-stream! ws-conn exports cmd
{:force-multiple true
:name name
:kind kind
:is-wasm (wasm-export-enabled? state)}))
(rx/finalize (fn []
(swap! st/ongoing-tasks disj :export))))))))
(defn request-multiple-export
[{:keys [exports cmd]

View File

@ -427,6 +427,37 @@
;; Scenario: an export starts while its file has a change not saved yet.
;; Proves: the user is warned that the export may miss it, and the export
;; runs anyway.
(t/deftest ^:async a-failed-persist-releases-the-export-flag
;; The persist runs before the job exists: if it fails, the flag must
;; not stay set, or the leave-page warning never turns off.
(let [tasks (atom #{})
store-errors (atom [])
store (export-store (test-state) store-errors)]
(await
(mock/with-mocks*
{dwp/force-persist-files-and-wait
(mock/stub (fn [_ _] (rx/throw (ex-info "persist failed" {:type :internal}))))
st/ongoing-tasks tasks}
(ptk/emit! store (de/request-simple-export {:export export}))
(await (async/wait-for #(not (contains? @tasks :export))
"the leave-page warning is released"))
(t/is (empty? @tasks))))))
(t/deftest ^:async a-failed-persist-releases-the-export-flag-of-a-multiple-run
(let [tasks (atom #{})
store-errors (atom [])
store (export-store (test-state) store-errors)]
(await
(mock/with-mocks*
{dwp/force-persist-files-and-wait
(mock/stub (fn [_ _] (rx/throw (ex-info "persist failed" {:type :internal}))))
st/ongoing-tasks tasks}
(ptk/emit! store (de/request-multiple-export
{:exports [(assoc export :enabled true)]}))
(await (async/wait-for #(not (contains? @tasks :export))
"the leave-page warning is released"))
(t/is (empty? @tasks))))))
(t/deftest ^:async export-warns-when-the-file-has-unsaved-changes
(let [file-id (uuid/next)
downloaded (atom false)